shaders [Guide] Creating a custom entity shader using CESapi.

[-LukaS-]

Hello!

To start off, this tutorial is NOT about making generic shaders for DST. I'll make a guide on that someday. This tutorial is strictly about how to make entity shaders with the use of CESapi. I will try my best to keep it as simple as possible however it is always better to have some experience with Lua and modding, and it's even better if you know a thing or two about OpenGL and it's shading language, GLSL ES.

So, with that out of the way...

 

1. What even is CESapi?

Spoiler

CESapi, which stands for Custom Entity Shaders api, is a tool designed help modders easily create more interesting shaders for entities. It opens a way for entity shaders to use custom uniform variables and texture/shader samplers, which is normally not possible. It is a fully client sided api but can be used for both client and non-client sided mods.

The current version (v0.6.0 at the time of posting this) is still in BETA however I am continuously working to add more features, fix bugs and improve the mod in general. If you have any questions feel free to ask me, either under this post or personally on discord (username: lukas_t_)

2. Preparing your mod

Spoiler

CESapi is extremely easy to set up, all you need do is download and enable it:

[CESapi] Custom Entity Shaders API BETA

Doing so will allow you to use a few new functions, the most important one being CESAPI.GenerateMaskingShaders(name, asset_table) which is used to generate appropriate masking shaders (you don't need to worry about them for now).

3. Creating your shader

Spoiler

In this tutorial I will go over how to create one of the shaders I packaged with CESapi. The enchantment glint effect. There are 4 prepackaged and fully documented shaders in the api itself, feel free to check them out if you're curious.

Here's how it's supposed to look:

Looks familiar?

So, let's begin!

Shaders are held inside .ksh files, Klei Shader files. These files are created by combining vertex shaders (.vs) and fragment shaders (.ps) and are compiled using the shader compiler packaged with the Don't Starve Mod Tools on Steam.

 

The vertex shader:

This shader is used to manipulate vertices and is run for every vertex of a model. I call it a "model" but don't let that name fool you. Remember that most entities in DST are 2D, while they do have vertices it's not the same as the vertices on a real 3D model, like the one you might make in blender. However that is a mute point because this shader will not even be used on the entities vertices. For CESapi shaders the vertex shader will always look the same.

Here's how it should look:

Spoiler

attribute vec3 POSITION;
attribute vec2 TEXCOORD0;
varying vec2 PS_TEXCOORD0;

void main()
{
    gl_Position = vec4(POSITION.xyz, 1.0);
    PS_TEXCOORD0.xy = TEXCOORD0.xy;
}

That's all you need and that's all you will ever need. Any shader you create in the future will most likely have this exact vertex shader behind it.
We start of by declaring 2 attributes, which are like variables that are passed in from the engine (we can't create them ourselves). POSITION is a vec3 variable, vec3 stands for `vector of length 3`. GLSL ES allows for vectors of lengths 2, 3 and 4. POSITION holds information about the 3 dimentional position of the vertex. TEXCOORD0 is a vec2 variable that holds information about the XY coordinate of the screen. PS_TEXCOORD0 is a varying variable, which basically means it gets passed from the vertex shader to the fragment shader. As you can see we will later pass in the TEXCOORD0 values into PS_TEXCOORD0.
But before that we pass in values to gl_Position. gl_Position is a reserved GLSL variable that defines the position of the vertex which can be manipulated however you like. In this case all we are doing is passing in the POSITION into it, however, as a vec4. That is required by OpenGL.
The values of vector variables in GLSL can be accessed similarly to how values in tables can be accessed in Lua. GLSL allows 3 different letters for accessing vector values.

For all 4 values, in order, you can use:

• x, y, z, w
  or
• r, g, b, a
  or
• s, t, p, q

There is no difference between these so you may use them how you like. I like using them depending on the context of the vector, for example, I prefer using r, g, b, a when it's representing a color, and x, y, z, w when it's representing a position.
So, back to the shader, all we're doing is passing in the POSITION into gl_Position as a vec4 and then TEXCOORD0 into PS_TEXCOORD0.

 

The fragment shader:

This is where the fun begins. Fragment shaders run for each pixel of the model. Explaining how we go from vertices to pixels is way above my pay grade ~0.00$/h however there's plenty of websites providing information about the rendering pipeline of OpenGL if you're interested.
Fragment shaders is where we can do the most in terms of interesting effects. It is also where you'll be spending most of your time writing code. Here's how the enchantment glint's fragment shader looks like:

Spoiler

#ifdef GL_ES
    precision mediump float;
#endif

uniform sampler2D SAMPLER[3];
#define SRC_IMAGE      SAMPLER[0]
#define MASKED_SAMPLER SAMPLER[1]
#define EFFECT_SAMPLER SAMPLER[2]

uniform vec4 SCREEN_PARAMS;
#define WINDOW_WIDTH  SCREEN_PARAMS.x
#define WINDOW_HEIGHT SCREEN_PARAMS.y

uniform vec4 TIMEPARAMS;
uniform float GLINT_TIME_SCALE;
uniform float GLINT_EFFECT_SCALE;

const float effectWidth = 128.0;
const float effectHeight = 128.0;

varying vec2 PS_TEXCOORD0;

void main()
{
    vec4 bgColor = texture2D(SRC_IMAGE, PS_TEXCOORD0);
    vec4 textureColor = texture2D(MASKED_SAMPLER, PS_TEXCOORD0);

    vec2 effectCoord = vec2((PS_TEXCOORD0.x * WINDOW_WIDTH) / effectWidth,
                            (PS_TEXCOORD0.y * WINDOW_HEIGHT) / effectHeight)
                       / GLINT_EFFECT_SCALE;
    effectCoord.x += TIMEPARAMS.x * GLINT_TIME_SCALE;

    vec4 effectColor = texture2D(EFFECT_SAMPLER, effectCoord);

    textureColor.rgb = mix(textureColor.rgb, effectColor.rgb, effectColor.a);
    gl_FragColor.rgb = mix(bgColor.rgb, textureColor.rgb, textureColor.a);
}

Quite sizable. Let's start from the top. The first thing we need to do is define the float precision.

#ifdef GL_ES
    precision mediump float;
#endif

We can choose between lowp, mediump and highp. Mediump should be enough in most cases. Next we have some new variables.

uniform sampler2D SAMPLER[3];
#define SRC_IMAGE      SAMPLER[0]
#define MASKED_SAMPLER SAMPLER[1]
#define EFFECT_SAMPLER SAMPLER[2]

The first one is a uniform variable. Since fragment shaders run for every pixel, uniform variables are a way to have a variable that will always stay the same for each pixel. That's why they are called uniform. You can think of them as constant values that can be modified with code from the Lua side.
In this case we're creating a uniform sampler2D array of length 3. sampler2D is a sampler type. Simply put, it contains a 2D image we can pull from.
CESapi lets you create and pass in however many samplers you want however the first sampler is always the unaltered screen (minus any UI elements), called SRC_IMAGE in this case but you can name it whatever you want. Any other samplers will be passed in using Lua. In this case the 2nd sampler will be the entity "cutout" and the 3rd will be our enchantment texture. I call the 2nd sampler a "cutout" because it is basically the entire screen again but with only the entity affected by the shader while everything else is black.

uniform vec4 SCREEN_PARAMS;
#define WINDOW_WIDTH  SCREEN_PARAMS.x
#define WINDOW_HEIGHT SCREEN_PARAMS.y

Next uniform is SCREEN_PARAMS. It holds information about the screens size in pixels and so SCREEN_PARAMS.x would be the width, SCREEN_PARAMS.y the height. We'll use them to calculate the correct coordinates for the enchantment texture.

uniform vec4 TIMEPARAMS;

Next we have TIMEPARAMS which holds information about the games time. TIMEPARAMS.x is the games current time.

uniform float GLINT_TIME_SCALE;
uniform float GLINT_EFFECT_SCALE;

And here we have our own uniform variables that we will define in our mod a bit later in the Lua section of the tutorial.

const float effectWidth = 128.0;
const float effectHeight = 128.0;

Here we define 2 constant values which are the enchantment texture width and height.

varying vec2 PS_TEXCOORD0;

And here we have the aforementioned PS_TEXCOORD0 that we passed in from the vertex shader.

And finally, the main function:

void main()
{
    vec4 bgColor = texture2D(SRC_IMAGE, PS_TEXCOORD0);
    vec4 textureColor = texture2D(MASKED_SAMPLER, PS_TEXCOORD0);

    vec2 effectCoord = vec2((PS_TEXCOORD0.x * WINDOW_WIDTH) / effectWidth,
                            (PS_TEXCOORD0.y * WINDOW_HEIGHT) / effectHeight)
                       / GLINT_EFFECT_SCALE;
    effectCoord.x += TIMEPARAMS.x * GLINT_TIME_SCALE;

    vec4 effectColor = texture2D(EFFECT_SAMPLER, effectCoord);

    textureColor.rgb = mix(textureColor.rgb, effectColor.rgb, effectColor.a);
    gl_FragColor.rgb = mix(bgColor.rgb, textureColor.rgb, textureColor.a);
}

texture2D is a GLSL function that takes in a sampler and XY coordinates and returns a vec4 representing a RGBA color. It works by taking the color of the pixel the coordinates are pointing to. The coordinates are normalized meaning they range from 0 to 1. For X 0 is left and 1 is right, for Y 0 is bottom 1 is top.
The first thing we're doing is taking both the screen pixel and the "cutout" pixel and putting them in bgColor and textureColor respectively. The next part is a bit more complex. We need to calculate the correct coordinates for the enchantment texture using the screen coordinates from PS_TEXCOORD0. We can do it by multiplying PS_TEXCOORD0 by WINDOW_WIDTH for X and by WINDOW_HEIGHT for Y and effectively turning the normalized coordinates ranging from 0 to 1 into coordinates ranging from 0 to WINDOW_WIDTH/WINDOW_HEIGHT. Then we divide these values by the textures width/height giving us the correct coordinates for the texture.
Lastly we divide the whole vec2 by GLINT_EFFECT_SCALE to apply scaling which we will be able to manipulate using Lua later down the line.
Next we're going to add the current time (TIMEPARAMS.x) multiplied by GLINT_TIME_SCALE which will make it look like the texture is constantly moving to the left. GLINT_TIME_SCALE will work as our adjustable speed scaler.
Now we can use these coordinates to pull the pixel color from the EFFECT_SAMPLER.
The last two lines introduce a very useful function called mix. It mixes 2 vectors based on a value between 0 and 1.
0 means take the first vector, 1 means take the second vector. Anything in between mixes the 2 vectors based on the value.

textureColor.rgb = mix(textureColor.rgb, effectColor.rgb, effectColor.a);

This line changes the stored textureColor.rgb using mix to apply the enchantment texture based on the textures opacity.

gl_FragColor.rgb = mix(bgColor.rgb, textureColor.rgb, textureColor.a);

gl_FragColor is another GLSL reserved variable. It's the current pixels color. Here we're setting it's rgb values to the mix of the screen and the "cutout" based on the "cutout"s opacity.

That's it for the shaders.

4. Compiling your shader

Spoiler

To compile your shader you need to download the mod tools on steam and use the shader compiler. You can find it in .\Don't Starve Mod Tools\mod_tools\tools\bin\ShaderCompiler.exe.

To use the compiler you need to run it through command prompt. Here's the command you need to use:

`compiler_path` -little `shader_name` `vs_path` `ps_path` `shader_file_name` -oglsl

• compiler_path - the path to the compiler, you can paste it in by dragging the compiler .exe onto the command prompt
• shader_name - the name of the shader
• vs_path - path to the vertex shader
• ps_path - path to the fragment shader
• shader_file_path - the path to the .ksh file that gets generated

If you don't want to type out the entire command every time you want to compile your shaders you can create a .bat file that automatically runs the command when you run the file. It's also good to finish the command off with `pause` to keep command prompt from closing automatically. Helps with catching compile errors.

Successfully compiling your shader should leave you with a .ksh file. I set my shader name to enchantmentglint.

5. Using your shader

Spoiler

To use your shader, first, put it in your mods' shaders folder. You can then import it by adding it as an Asset in your mods Assets table. In that same table place the enchantment texture file. The file itself can be found in the examplemod folder inside CESapi's mod folder.

Assets = {
    Asset("SHADER", "shaders/enchantmentglint.ksh"),
    Asset("IMAGE", "images/enchanted_glint_entity.tex")
}

Now, it's time to use some CESapi functions. The most important part is generating masking shaders. They are required for CESapi shaders to work. To generate masking shaders simply run:

GLOBAL.CESAPI.GenerateMaskingShaders("enchantmentglint", Assets)

This function will generate 2 masking shader, one called enchantmentglint_mask, and one called pp_enchantmentglint_mask. We'll use them later.

After preparing the masking shaders we'll have to create a new post processing shader. Here's the code for that:

AddPrefabPostInit("world", function()
    if not GLOBAL.TheNet:IsDedicated() then
        GLOBAL.TexSamplers.GLINT_TEX = GLOBAL.PostProcessor:AddTextureSampler(GLOBAL.resolvefilepath("images/enchanted_glint_entity.tex"))
        GLOBAL.PostProcessor:SetTextureSamplerState(GLOBAL.TexSamplers.GLINT_TEX, GLOBAL.WRAP_MODE.WRAP)
        GLOBAL.PostProcessor:SetTextureSamplerFilter(GLOBAL.TexSamplers.GLINT_TEX, GLOBAL.FILTER_MODE.POINT, GLOBAL.FILTER_MODE.POINT, GLOBAL.MIP_FILTER_MODE.NONE)

        GLOBAL.PostProcessorEffects.EnchantmentGlint = GLOBAL.PostProcessor:AddPostProcessEffect(GLOBAL.resolvefilepath("shaders/enchantmentglint.ksh"))
        GLOBAL.PostProcessor:AddSampler(GLOBAL.PostProcessorEffects.EnchantmentGlint, GLOBAL.SamplerEffectBase.Shader, GLOBAL.SamplerEffects["pp_enchantmentglint_mask"])
        GLOBAL.PostProcessor:AddSampler(GLOBAL.PostProcessorEffects.EnchantmentGlint, GLOBAL.SamplerEffectBase.Texture, GLOBAL.TexSamplers.GLINT_TEX)

        GLOBAL.UniformVariables.GLINT_TIME_SCALE = GLOBAL.PostProcessor:AddUniformVariable("GLINT_TIME_SCALE", 1)
        GLOBAL.UniformVariables.GLINT_EFFECT_SCALE = GLOBAL.PostProcessor:AddUniformVariable("GLINT_EFFECT_SCALE", 1)
        GLOBAL.PostProcessor:SetEffectUniformVariables(GLOBAL.PostProcessorEffects.EnchantmentGlint, GLOBAL.UniformVariables.GLINT_TIME_SCALE, GLOBAL.UniformVariables.GLINT_EFFECT_SCALE)
        GLOBAL.PostProcessor:SetUniformVariable(GLOBAL.UniformVariables.GLINT_TIME_SCALE, 0.1)
        GLOBAL.PostProcessor:SetUniformVariable(GLOBAL.UniformVariables.GLINT_EFFECT_SCALE, 4)

        GLOBAL.PostProcessor:SetPostProcessEffectBefore(GLOBAL.PostProcessorEffects.EnchantmentGlint, GLOBAL.PostProcessorEffects.Bloom)
        GLOBAL.PostProcessor:EnablePostProcessEffect(GLOBAL.PostProcessorEffects.EnchantmentGlint, true)
    end
end)

It might look scary at first but it's quite simple. Due to some bug in the games engine, in order to use custom textures in shaders we have to run the code inside a post init of the world. We also need to make sure the code doesn't get run on dedicated servers, they don't need it.
After that we can create a texture sampler using PostProcessor:AddTextureSampler where we specify the texture we want. It is important that you use resolvefilepath whenever creating any new samplers.
Next, we'll set the texture samplers wrap mode to wrap meaning the texture will loop infinitely. Then we specify filter modes.
After preping our texture sampler we can create a new post processing effect using PostProcessor:AddPostProcessEffect. I called mine EnchantmentGlint. Here you need to specify your shader inside resolvefilepath.
Next we pass in the 2 samplers that the shader will use. PostProcessor:AddSampler accepts a few parameters. The first one defines what shader we're passing the samplers into, the second one is the type of sampler we're passing. The only 2 we'll be using are SamplerEffectBase.Shader and SamplerEffectBase.Texture. Then we just specify the sampler we want to pass.
The first sampler is the auto-generated pp_enchantmentglint_mask. This is the "cutout" I talked about earlier. The second one is the texture sampler.
After adding the samplers we'll create the 2 uniform variables used by the shader. PostProcessor:AddUniformVariable accepts 2 parameters. The first one being the name of the variable, the second one being the "size", which just means whether the uniform variable is a float, a vec2, a vec3 or a vec4. 1 = float, 2 = vec2 and so on...
You can then assign the uniform variables to our shader using PostProcessor:SetEffectUniformVariables. Then you can use PostProcessor:SetUniformVariable to set the variables values. You can do this during runtime to manipulate your shaders dynamically.
The last thing we need to do is to sort our shader in the stack of post processor effects and enable it. Make sure you always sort your shaders to be before PostProcessorEffects.Bloom.

And that's it! Now you just need to apply your shader to an entity. Which you can do using CESAPI.SetCustomEffect(animstate, name) where animstate is the AnimState of the entity you want the shader to be applied to and name is the name of the shader. You can run this function inside a prefab file or through the console or wherever you might want.

To clear the shader you can use CESAPI.ClearCustomEffect(animstate).

And if you're interested in the ever so popular bloom effect that's already in the game you can either use the default AnimState.SetBloomEffectHandle("shaders/anim.ksh") or use CESAPI.SetDefaultBloomEffect(animstate).

And in the same way you can also clear this effect using CESAPI.ClearDefaultBloomEffect(animstate).

6. What if I don't know if the player has CESapi?

Spoiler

Well, no worry! As long as you keep your CESapi shader code inside an if statement checking whether CESapi is enabled no crash will happen.

if GLOBAL.KnownModIndex:IsModEnabledAny("workshop-3345578390") then -- 3345578390 is the exact CESapi mod ID
  -- your shader code here
end

It's a good practice to always prepare your code for both situations: when the user has CESapi and when they don't!

7. Closing thoughts

Spoiler

 

Edited June 24, 2025 by -LukaS-