Additive, subtractive, and premul blending do not handle fog properly.

[jitspoe]

Tested versions

4.0-4.4, probably 3.x as well.

System information

Godot v4.5.dev (e244b3d) - Windows 10 (build 19045) - Multi-window, 2 monitors - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 3080 (NVIDIA; 32.0.15.6603) - AMD Ryzen 7 3700X 8-Core Processor (16 threads)

Issue description

If a material is using something other than mix blend mode, the fog behavior does not look correct. For example, additive blending will cause the whole quad of a sprite to become visible, since the fog is setting the color on the whole thing and the color is then being added.

The fire on the enemy here is using blend_add and the fire with smoke from the flamethrower is using blend_premul_alpha. Both have the same issue, but the solution is slightly different. I think additive blending can simply be solved by using a fog color of black, while the premul will need black where alpha is 0 and fade toward the color based on the alpha value.

Steps to reproduce

Create a scene with fog. Add some materials with different blend modes and observe them from a distance.

Minimal reproduction project (MRP)

@Calinou made a test project here https://github.com/user-attachments/files/15509973/test_material_fog.zip

This is to continue the issue discussion from the blend_premul_alpha issue here: godotengine/godot-proposals#3431 (comment)

I thought we had a separate issue created for this, but apparently not.

[Calinou]

I think additive blending can simply be solved by using a fog color of black, while the premul will need black where alpha is 0 and fade toward the color based on the alpha value.

Is there a correct way to fix this for Subtract and Multiply as well? As I understand it, blend modes are implemented on the hardware, so we can't override how they work. All we could do is try to compute fog differently on materials with a blend mode other than Mix to try to compensate for the different blending formula.