Lesson 1 - using HelixMod
Summary
This lesson demonstrates how to use the HelixMod debugger to find shaders, and how to save shader ASM code.
Level of difficulty: Easy
Time required: 15 minutes
Video Walkthrough on YouTube
Video Walkthrough direct download link
Objective
We'll run the HelixMod debugger and learn the commands to hunt shaders.
We'll create and learn about the DX9Settings.ini file with good default settings.
We'll find a shader and save and look at the ASM file.
Quiz
Save and share an ASM shader file from the game.
Now that we have HelixMod installed with the red text, we are clearly running the debugger version. HelixMod operates using the NumberPad keys. The keys can be changed if necessary or desired. The default settings use Numpad 1,2,3 for Pixel Shader, NumPad 4,5,6 for Vertex Shader.
In general, Vertex Shader specifies position, including 3D position in space. And the Pixel Shader determines the color of the resulting on screen pixel. It gets more complicated than that, but that is all we need for now.
The basic strategy is navigate the list of shaders in a game, looking for the specific one that is causing problems. You can hold down the hunting key to have it cycle to the next. As you cycle through shaders, you will see parts of the game image blink off, then back on. When a shader blinks off, you've found the one you want, and can save it using another key.
Let's see how to navigate the HelixMod UI using the default keys.
- Shader hunting keys
- Run The Ball, in 3D with HelixMod installed.
- Make sure we have red text. On left, VS and PS counts and CRC (hex numbers).
- In middle, convergence and separation settings. Use normal Alt-F3/F4 and Alt-F5/F6 to see them change.
- Hit NumPad 1, then NumPad 4 to cycle backwards one shader.
- Note the count of the shaders for each type. This simple game has almost 5000 shaders.
- Hold down both the NumPad 1 and NumPad 4 key watching for changes on screen.
- When you see a change in the image, let go and then cycle up and down with 1,2 and 4,5 to get a feel for it.
- We aren't looking for a specific shader just yet, so let's move on. Quit the game.
It should be clear that hunting specific shaders can be fairly tedious, especially if there are a lot of them. Let's see how to create a DX9Settings.ini file, to add some better defaults to HelixMod.
A particularly valuable feature is the UseRenderedShaders=true, which will limit the set of shaders to cycle to only those being actively used in the current scene, not every shader in the game. This can take this list of shaders down from 5000 to 100. This doesn't work with every game, and is not available in older versions of the HelixMod.
We will also set DumpAll=true which will generate ASM text files for every shader seen in the game. We won't use these for this Lesson, but it's a valuable option.
We also want to take a quick look at the LOG.txt file generated by HelixMod as it's often very useful and necessary to help solve problems when things go wrong. There will be a lot of debug lines talking about missing Lua scripts, which is OK. That's for a future advanced lesson.
We are going to make a very small but important default file. In later Lessons we'll need to add more information to the DX9Settings.ini.
- Editing DX9Settings.ini
- Run NotePad++ and create a new document.
- Type or paste in [General]. On the next line enter UseRenderedShaders=true. On the third line enter DumpAll=true.
- Save the file on the Desktop next to your d3d9.dll.
- Navigate to the game folder for The Ball.
- C:\Program Files (x86)\Steam\SteamApps\common\The Ball Demo\Binaries\Win32\
- While we are here, make a convenience shortcut to that directory on your Desktop, as we'll come back here often.
- Copy the default DX9Settings.ini file from the Desktop and put it next to the d3d9.dll in the game directory.
- While in the game directory, open the LOG.txt file HelixMod generated.
- The LOG.txt is extremely important, as you need it when things aren't working correctly. Read through it a bit, to get a feel for it.
- optional: Read about the Original DX9Settings discussion with key defaults.
Now that we are set up to use UseRenderedShaders=true, let's see how that improves our shader hunting, to only cycle shaders in the current frame.
- Improved shader hunting
- Run The Ball in 3D.
- Note the red debug text. There won't be any obvious indication that the DX9Settings.ini was loaded.
- Cycle backwards one step like before, using Numpad 1 and Numpad 4.
- Look for the maximum numbers of shaders, and note that it is a lot smaller.
- Hold down the Numpad 1 key to cycle through all the Pixel Shaders, just to see how it works and get a feel for it.
In Lesson 2, we will disable an effect, but for now we will just see how the ASM text files are created, and where.
As a general rule, if we are going to disable an effect, the safest approach is to disable a PixelShader, because they tend to have more specific roles in the image. Also any given VertexShader can often be used for multiple purposes, so disabling those can sometimes disable other parts of the image that we want to keep.
Let's choose any shader that you find interesting, and have HelixMod save the ASM text.
- Shader saving
- While in game, cycle through Pixel Shaders, and find one that is interesting.
- With the shader blanked out, showing it's currently the marked shader, press Numpad 3.
- Nothing will change on-screen, but we just saved the ASM text for that shader.
- Quit the game.
- Use your game directory shortcut to get to:
- C:\Program Files (x86)\Steam\SteamApps\common\The Ball Demo\Binaries\Win32\
- Look for the Helix generated shader folder of Dumps.
- That will have ..\Dumps\AllShaders\ and ..\Dumps\SingleShaders\
- AllShaders was generated from our DumpAll=true command. Examine the contents to see all PixelShaders and all VertexShaders.
- Go to the ..\Dumps\SingleShaders\PixelShader\ folder, and look for the file we created.
- Open that file with the editor, and marvel at the Klingon-esque ASM.
Now that we have generated an ASM text file for a PixelShader, it's time for the Quiz.
Demonstrate that you have this all working by sharing your shader file to the Wiki.
- Save shader text file
- Navigate to your user page on the Wiki, by clicking your name in top right.
- Edit that page using the top-middle Edit navbar.
- Below your text and image for Lesson 0, enter text of <br> to make space.
- Then enter the text == Lesson 1 ==. The "==" tag makes it a header.
- Add any text you'd like to add as description or notes. Modify the Lesson 0 line to be a header too.
- Enter the Wiki formatting tags of <nowiki></nowiki>
- Make sure there is one space before <nowiki>
- Switch to your editor and copy the text of the your ASM PixelShader.
- Paste that text in between <nowiki> and </nowiki>.
- Click Show Preview on your page to see how it looks.
- When you are satisfied, click Save page to publish your work.
If you sucessfully saved the ASM text file, then congratulations! You now know how to use the HelixMod debugger to hunt shaders, and how to find the shaders that you mark.
If you have any questions or suggestions, please use the Discussion tab, and start a new topic.
