Sensibeat's Col-based Transparency Tutorial

So, as many people are asking, and because it may be difficult to find every informations in the SF4 modding thread (31 pages already), I translated a tutorial I made some time ago on a french forum.

I used Sakura as an example.

To make transparency only using col.emz file, 2 steps are required:
- editing the col.emz file with a hexadecimal editor
- editing the Alpha layer of the dds where the texture is

1) edit the col.emz file

First I use piecemontee's SF4Explorer to identify the objects names and find them easier in the hex-editor

link: Free File Hosting Made Simple - MediaFire (you can get a newer version but this one is enough for names)

Reminder: to see objects names in SF4Explorer we are going to look at the cos.emz file, but it's the col.emz file we are going to edit (both are linked).
This step isn't required, but it's such easier when you know the true names of things, instead of having to look for them one by one, doing blind tests...

Ok... there the skirt was obviously "Skirt", but "Hatimaki" for her hairband was quite harder to figure...

Now you know the name of the thing you want to edit, open the col.emz of your mod (not the original col.emz, the unarchived one -dat, emb, name it... I rename mine .emz just after I unarchive them to .emb with quickbms-) under a hex-editor and do a search on that name:

Now scroll down a bit to the line "BrushA":

See those 40 characters outligned? You'll have to edit them:

For full transparency (removing objects, holes, ripped clothes) change them by those 40:

(I'll test on Sakura's skirt)

For semi transparency (Rose's pants) change them by those 40:

(I'll test on Sakura's hairband)

Note: if the object you want to edit doesn't have BrushA or B or C or D... you'll have to find something else to replace (that don't affect the mod too much...)
(you may ask in the SF4 modding thread, tests have already been done on some)

Ok, now that first step is done, save your col.emz and let's edit that texture.

2) edit the .dds file

Firstly what is an Alpha layer?

It's a layer in addition to basic color layers (RGB) that will create transparency (a white layer without colors, only black & white -and shades of grey-).

You find it here in Photoshop:

If it's here, no problem, your .dds file is in DXT5 format.
If it isn't there, the .dds file should be in DXT1, you'll have to add one yourself.

To know the DXT format of your .dds files I recommend downloading Windows Texture Viewer from the NVIDIA site: it shows the DXT.
link: Windows Texture Viewer

Warning: note the DXT format of your dds, a character can have some of different formats (Sakura's hairband is located on a DXT5, her skirt on a DXT1).
If you want to edit a DXT1 (normal without alpha), you'll have to save it as a DXT1 with 1bit alpha (make backups, it erases the texture under the alpha layer edit) or as a DXT5 matching the DXT1 weight (by losing some quality on your textures)
*
Full transparency works if you save in both formats, DXT1 1bit alpha or DXT5.
Semi transparency only works if you save in DXT5.

To show you the effect of semi transparency, I put a big rectangle with a gradient from white (visible) to black (invisible) on Sakura's hairband:

(on screen: Alpha layer alone)

To show you full transparency, I draw some simple shapes on her skirt:

(on screen: RGB layers and Alpha layer displayed together: the black of the Alpha layer is shown in transparent red under this display mode in Photoshop)

Then I inject the .dds back in the col.emz file...

And here is the result:

Have fun!

Credits goes to sn00pee for all his finds on the subject.

edit:

*forgot to mention an important thing:
Full transparency works if you save in both formats, DXT1 1bit alpha or DXT5.
Semi transparency only works if you save in DXT5.

sn00pee (again) found a way to inject bigger files in .emz without crashing the game here.
(I think the guy that created Rose Ninja used that method on the *cough*second color*cough*)