How to display and export Vertex alpha in Ofusion/Ogre

Hi, I have a shader in shaderFX that uses vertex alpha to blend two normal maps, and add shinyness to one.



Were having problems converting the shader to ogre because were unsure whether the mesh is being exported with vertex alpha whether the problem is with our code. At the moment we get this in Ogre:


I can get vertex colors to work in ofusion, but not alpha. The shader doesn't work properly if you use R,G or B instead of alpha.

Are there any special settings required to get the vertex alpha to show in ofusion, I need to be able to confirm that they are being exported with the mesh.

Questions:

how do I see vertex alpha?

Do I need to add something in the material file that enables them?



Here's my material file for the above shader:

material blend
{
technique pass
{
pass light1
{
//cull_hardware anticlockwise

vertex_program_ref texture_vs
{
}

fragment_program_ref texture_ps
{
}

texture_unit DiffMapASampler
{
texture MIDNITE4.dds
}

texture_unit DiffMapBSampler
{
texture streambed.dds
}

texture_unit NormMapASampler
{
texture MIDNITE4_N2.png
}

texture_unit NormMapBSampler
{
texture streambed_N2.png
}

}
}

}

Here's the shader source adapted from shaderFX. It's a slightly older onw with an extra inverse alpha node that wasn't really needed.

Being an artist I understand very little, perhaps someone can tell me if our coder who's new to shaders is correctly reading the vertex Alpha?

Source:

// Leadfoot Productions/Kinetic Realities -> jeremyAlessi -> ShaderFX -> OGREConversion

// input from application
struct a2v
{
float4 position : POSITION;
float2 texCoord : TEXCOORD0;
float4 vertcol : COLOR0;
float3 tangent : TANGENT;
float3 binormal : BINORMAL;
float3 normal : NORMAL;
};

// output to fragment program
struct v2f
{
float4 position : POSITION;
float3 lightVec : TEXCOORD0;
float3 eyeVec : TEXCOORD1;

float2 texCoord : TEXCOORD2;
float4 color : COLOR;
float3 worldTangent : TEXCOORD3;
float3 worldBinormal : TEXCOORD4;
float3 worldNormal : TEXCOORD5;

};

//Diffuse and Specular Pass Vertex Shader
v2f v(a2v In, uniform float4 lightPosition, uniform float4x4 wvp, uniform float4x4 worldIT, uniform float4x4 viewInv, uniform float4x4 world)
{

v2f Out = (v2f)0;
Out.position = mul(wvp, In.position);

//transform vert position to homogeneous clip space
//this code was added by the standard material

float3 worldSpacePos = mul(world, In.position);

//world space position
//this line was added by the standard material

Out.lightVec = lightPosition - worldSpacePos;

//light vector in world space
//this code was added by the standard material

Out.eyeVec = viewInv[3] - worldSpacePos;

//eye vector in world space
//this code was added by the texture map Node

Out.texCoord = In.texCoord;

//pass through texture coordinates from channel 1
//this code was added by the Vertex Color Node

Out.color = In.vertcol;

//this code was added by the Texture Map Node

Out.worldNormal = mul(worldIT, In.normal).xyz;

//compute world space normal

Out.worldBinormal = mul(worldIT, In.binormal).xyz;

//compute world space binormal

Out.worldTangent = mul(worldIT, In.tangent).xyz;

//compute world space tangent
//this code was added by the Texture Map Node

return Out;
}


//Diffuse and Specular Pass Pixel Shader
float4 f(v2f In, uniform float4 lightColor, uniform sampler DiffMapASampler, uniform sampler DiffMapBSampler, uniform sampler NormMapASampler, uniform sampler NormMapBSampler) : COLOR
{

float3 ret = float3(0,0,0);
float3 V = normalize(In.eyeVec);

//creating the eye vector

float3 L = normalize(In.lightVec);

//creating the light vector

float4 DiffMapA = tex2D(DiffMapASampler, In.texCoord.xy);
float4 DiffMapB = tex2D(DiffMapBSampler, In.texCoord.xy);
float4 VertVCol_AInput = In.color;

//bringing in the vertex color

float3 DiffMapInterp = lerp(DiffMapA.rgb, DiffMapB.rgb, VertVCol_AInput.a);
float3 input2 = DiffMapInterp;


float InvertAlpha = 1 - VertVCol_AInput.a;

//Subtract the input from one

float input4 = InvertAlpha;


float4 NormMapA = tex2D(NormMapASampler, In.texCoord.xy);
NormMapA = NormMapA * 2 - 1;

//expand to -1 to 1 range
//this code was added by the Tex2d Node

float3 Nn = normalize(In.worldNormal);

//input the vectors required for tangent to world space transform

float3 Tn = normalize(In.worldTangent);
float3 Bn = normalize(In.worldBinormal);
NormMapA = float4((Nn * NormMapA.z) + (NormMapA.x * Tn + NormMapA.y * -Bn), NormMapA.w);
//create a per-pixel normal for Y up

NormMapA.rgb = normalize(NormMapA.rgb);

//normalized the normal vector

float4 NormMapB = tex2D(NormMapBSampler, In.texCoord.xy);
NormMapB = NormMapB * 2 - 1;

//expand to -1 to 1 range

NormMapB = float4((Nn * NormMapB.z) + (NormMapB.x * Tn + NormMapB.y * -Bn), NormMapB.w);
//create a per-pixel normal for Y up

NormMapB.rgb = normalize(NormMapB.rgb);

//normalized the normal vector

float3 NormMapInterp = lerp(NormMapA.rgb, NormMapB.rgb, VertVCol_AInput.a);
float3 input8 = NormMapInterp;

float3 N = input8;

//using the Normal socket

float3 diffuseColor = input2;

//using the Diffuse Color socket

float diffuse = saturate(dot(N,L));

//calculate the diffuse

diffuseColor *= diffuse;

//the resulting diffuse color

ret += diffuseColor;

//add diffuse light to final color

float specularColor = input4;

//the Specular Color socket was empty - using Specular Level instead

float glossiness = 20;

//the Glossiness socket was empty - using default value

float3 H = normalize(L + V);

//Compute the half angle

float NdotH = saturate(dot(N,H));

//Compute NdotH

specularColor *= pow(NdotH, glossiness);

//Raise to glossiness power and compute final specular color

ret += specularColor;

//add specular light to final color

ret *= lightColor;

//multiply by the color of the light

float4 done = float4(ret, 1);

return done;

}

.program file

// Leadfoot Productions/Kinetic Realities -> jeremyAlessi -> ShaderFX -> OGREConversion

vertex_program texture_vs hlsl
{
source vertexASNBWorld.hlsl
entry_point v
target vs_2_0
//profiles vs_2_0 arbvp1

default_params
{
param_named_auto lightPosition LIGHT_POSITION 0
param_named_auto wvp WORLDVIEWPROJ_MATRIX
param_named_auto worldIT INVERSE_TRANSPOSE_WORLD_MATRIX
param_named_auto viewInv INVERSE_VIEW_MATRIX
param_named_auto world WORLD_MATRIX


}
}

fragment_program texture_ps hlsl
{
source vertexASNBWorld.hlsl
entry_point f
target ps_2_0
//profiles ps_2_0 arbvp1 fp20

default_params
{
param_named_auto lightColor LIGHT_DIFFUSE_COLOUR 0
}
}

Were using the same .material file as in the post above.

I will take a look at this asap, and provide you with a working shader

Thanks Lioric, If you want the scene that was used with the vertex Alpha in, a max9 one and an ogre mesh is in this archive. It's the mesh that was used in the screengrab above, with the shaderFX material displayed.

http://www.kineticrealities.com/Vertexcolor.zip

If you need anything else let me know.

Really appreciate the help

There's a slightly simpler version of the same shader in .fx format, I swapped the lerp texture order around so that I didn't have to add an invert vertex alpha node for the specular effect to come out on the correct texture.

It looks exactly the same though.

http://www.kineticrealities.com/vertexASNB_Simpler.fx

Anyway, will be good to know where the vertex alpha problem lies, thanks

We redid the shader using red vertex colors, and then with some experimentation found we had to link up all RGB vertex color channels in order to get the shader to mask correctly.

Is this how it should work?

Now we have masking working with the red vertex colors, the lighting seems very different to the shaderFX version. Anyone know why the specular level thats done with the same vertex color value as the mask does not appear as strong as the Max DX9 render?



I was wondering whether linking up all the vertex color channels diminished the specualr level?.

We don't have any extra lighting parameters in the shader (ambient etc) but neither does the shaderFX material.

As noted i will help you with this as soon as i get some free time (hope to be in the next days)

A brief look at your shader shows that you are using the TANGENT an BINORMAL semantics, the mesh will contain the tangents buffer in the TEXCOORDx buffer (depends on the number of textures, usualy in the next buffer, if you have 1 texture coords then the tangent buffer will be 2, the log will display where the tangents are placed)

The Binormal needs to be calculated in your shader, is not part of the mesh data, see the RenderMonkey shader tutorial, it contains the code (a single line) you need to add to the shader to calculate the binormal (and information on the tangents)

This is needed to get the lighting correct

thanks lioric, were just learning shaders, so everythings new and confusing.

To get the red vertex color to work, I thought we only needed VertexRed.r only. To get our shader to work we ended up having to use all 3 channels to get the desired result.

float3 LinearInterpD = lerp(DifTexA.rgb, DifTexB.rgb, VertexRed.r + VertexRed.g + VertexRed.b);

Is that normal? might explain why vertex alpha wouldn't work

Anyway, we look forward to seeing how you do it, been a tricky couple of days for us .

If you are using the red color to define where the texture blends, then you dont need to add the other color components, there must be some other issue affecting the results (in your example adding the other colors dont alter the color value, as you are using red color only, adding r = 1, b = 0 and g = 0 produces r+g+b = 1 == r component)

Thanks for clearing the color question up. I really have to learn more about the basic fundamental on the workings of shaders before I can understand your tutorial properly. I am picking some things up, VERY slowly.

I can read much of whats going on, just fall flat on how to use the syntax and language to write my own, as well as missing few very basic concepts that are still a bit vague.

Our coders will be going through those as they get better aquainted with shaders themeselves

well, we got the binormal part fixed so everything looks correct now, but were still having to send all 3 RGB vertex channels in order for the red to work.

Vertex alpha is still being illusive, but its cool that we got the result we wanted.

Here's as far as we got with the vertex red mask, it uses all 3 RGB channels, but it works.

// Leadfoot Productions/Kinetic Realities -> jeremyAlessi -> ShaderFX -> OGREConversion

// input from application
struct a2v
{
float4 position : POSITION;
float3 tangent : TEXCOORD1;
//float3 binormal : BINORMAL;
float3 normal : NORMAL;

float2 texCoord : TEXCOORD0;
float4 vertcol : COLOR0;
};

// output to fragment program
struct v2f
{
float4 position : POSITION;
float3 lightVec : TEXCOORD0;
float3 eyeVec : TEXCOORD1;

float2 texCoord : TEXCOORD2;
float4 color : COLOR;
};

//Diffuse and Specular Pass Vertex Shader
v2f v(a2v In, uniform float4 lightPosition, uniform float4x4 wvp, uniform float4x4 worldI, uniform float3 camPos)
{

float3 binormal = cross( In.tangent, In.normal );
v2f Out = (v2f)0;
Out.position = mul(wvp, In.position);
//transform vert position to homogeneous clip space
//this code was added by the standard material

float3x3 objTangentXf;
//build object to tangent space transform matrix

objTangentXf[0] = In.tangent;
objTangentXf[1] = -binormal;
objTangentXf[2] = In.normal;
//this code was added by the standard material

float4 osLPos = mul(worldI, lightPosition);
//put world space light position in object space

float3 osLVec = osLPos.xyz - In.position.xyz;
//object space light vector

Out.lightVec = mul(objTangentXf, osLVec);
//tangent space light vector passed out
//this code was added by the standard material

float4 osIPos = mul(worldI, camPos);
//put world space eye position in object space

float3 osIVec = osIPos.xyz - In.position.xyz;
//object space eye vector

Out.eyeVec = mul(objTangentXf,osIVec);
//tangent space eye vector passed out

//this code was added by the texture map Node
Out.texCoord = In.texCoord;
//pass through texture coordinates from channel 1
//this code was added by the Vertex Color Node

Out.color = In.vertcol;

return Out;
}

//Diffuse and Specular Pass Pixel Shader
float4 f(v2f In, uniform float4 lightColor, uniform sampler DifTexASampler, uniform sampler DifTexBSampler, uniform sampler NormTexASampler, uniform sampler NormTexBSampler) : COLOR
{
float3 ret = float3(0,0,0);
float3 V = normalize(In.eyeVec);
//creating the eye vector

float3 L = normalize(In.lightVec);
//creating the light vector

float4 DifTexA = tex2D(DifTexASampler, In.texCoord.xy);
float4 DifTexB = tex2D(DifTexBSampler, In.texCoord.xy);
float4 VertexRed = In.color;
//bringing in the vertex color

float3 LinearInterpD = lerp(DifTexA.rgb, DifTexB.rgb, VertexRed.r + VertexRed.g + VertexRed.b );
float3 input2 = LinearInterpD;

float input4 = VertexRed.r + VertexRed.g + VertexRed.b;

float4 NormTexA = tex2D(NormTexASampler, In.texCoord.xy);
NormTexA = NormTexA * 2 - 1;
//expand to -1 to 1 range

NormTexA.rgb = normalize(NormTexA.rgb);
//normalized the normal vector

float4 NormTexB = tex2D(NormTexBSampler, In.texCoord.xy);
NormTexB = NormTexB * 2 - 1;
//expand to -1 to 1 range

NormTexB.rgb = normalize(NormTexB.rgb);
//normalized the normal vector

float3 LinearInterpN = lerp(NormTexA.rgb, NormTexB.rgb, VertexRed.r + VertexRed.g + VertexRed.b);
float3 input8 = LinearInterpN;

float3 N = input8;
//using the Normal socket

float3 diffuseColor = input2;
//using the Diffuse Color socket

float diffuse = saturate(dot(N,L));
//calculate the diffuse

diffuseColor *= diffuse;
//the resulting diffuse color

ret += diffuseColor;
//add diffuse light to final color

float specularColor = input4;
//the Specular Color socket was empty - using Specular Level instead

float glossiness = 20;
//the Glossiness socket was empty - using default value

float3 H = normalize(L + V);
//Compute the half angle

float NdotH = saturate(dot(N,H));
//Compute NdotH

specularColor *= pow(NdotH, glossiness);
//Raise to glossiness power and compute final specular color

ret += specularColor;
//add specular light to final color

ret *= lightColor;
//multiply by the color of the light

float4 done = float4(ret, 1);

return done;
}

I have reviewed your shader using the vertex alpha value of the object and it seems to be working correctly (i will review why your scene was not using the alpha value asap)

I created a plane, set the alpha value to top:100%, middle:50% and lower:0%, modified the sahder ot use the In.color.a (the vertex alpha value) and added a glossiness parameter to control the specular level


Click for bigger image

thanks lioric,

I don't actually know anything much about shader coding, but I have picked up a simple method of cutting and pasting chunks of code from an FX file, referencing our coders mul and binormal tweaks and modifying as needed. In most cases this works quite well.

But I'm a bit stumped with these vertex colors, alpha in particular. When you find the problem in the shaderFX code, I'd be very interested in seeing what has to be changed so that it works. Especialy with the vertex alpha that no one over has managed to fix.

Evak,

I would be interested to know your actual process of copy-pasting from ShaderFX to Ogre-compatible shaders I am not into shader yet, but will do so in near future.

I'll try and post something soon, at the moment it still takes me 20 - 30 minutes to convert a single shader, and some of it has to be done step by step by looking at the compile errors in the ogre log.

When I have a better grasp of it I'll post something for you guys. Hopefully there will be a simple explanation for how to fix ShaderFX vertex output easily.

From liorics last post it looks like the problem doesn't lie with Ofusion. Hoping theres a simple method that I can add to my current conversion process to fix this issue too .

Hi Lioric, incase it makes it easier I made a simpler vertex alpha that blends two diffuse textures shader in SFX, converted it as far as I could. includes the SFX file, .FX file my mostly converted ogre files up to the point where Vertex alpha needs to be fixed

Theres a little app that you can run to check the shader running on a mesh (press esc to quit)

and the ogre log file is in the /conf directory.

http://www.kineticrealities.com/vablend.rar



You don't have to use it if you don't want too, but should make things simpler.