-
Notifications
You must be signed in to change notification settings - Fork 66
Genericize the CMake source generator for embedding files #1849
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Once this is merged, here are example of things you can do in a C++ file: #include "DaemonEmbeddedFiles/EngineShaders.h"
void test()
{
const char* filename = "common.glsl";
// Reading a file content by its name:
const char* file1 = EngineShaders::ReadFile(filename);
// Using a file directly using its symbol:
const unsigned char* file2 = EngineShaders::common_glsl;
// Alternatively:
const char* file3 = (const char*) EngineShaders::common_glsl;
// Testing the presence of a file with a given name:
Log::Debug("File %s is %s", filename,
EngineShaders::ReadFile(filename) == nullptr ? "absent" : "present");
// Iterating the files:
for ( auto it : EngineShaders::FileMap )
{
Log::Debug("Embedded file %s", it.first);
}
}The |
Listing files to embed:set(GLSL_EMBED_DIR "${ENGINE_DIR}/renderer/glsl_source")
set(GLSL_EMBED_LIST
# Common shader libraries
common.glsl
common_cp.glsl
fogEquation_fp.glsl
shaderProfiler_vp.glsl
shaderProfiler_fp.glsl
# …
)Generating the embedded files, etc: # Generate GLSL include files.
daemon_embed_files("EngineShaders" "GLSL" "TEXT" "client-objects")First option ("EngineShaders") is the wanted NameSpace, second option ("GLSL") is the SLUG used in Those variable and list should be named This And that's all. It's even not needed anymore to add the entry points by hand like this: set(RENDERERLIST
${DAEMON_EMBEDDED_DIR}/EngineShaders.cpp
${DAEMON_EMBEDDED_DIR}/EngineShaders.h
${ENGINE_DIR}/renderer/BufferBind.h
# …
) |
9b96078 to
1313666
Compare
a7ff720 to
945511f
Compare
|
So this will be used for the Vulkan binary shaders list? I guess the separate EMBED_DIR and EMBED_LIST thing was designed for VFS embedding; if we don't need that it seems over-complicated. Just use a single list of absolute paths like everything else. Note that the GLSL map that we have now is coded in a rather inefficient way. The map object stores the files as It doesn't make sense to me that some auxiliary functions return |
Vulkan will use functionality that was already added in #1845, which is now rebased on this pr. |
95677da to
ff5cf6c
Compare
I don't know yet if that's usable for the Vulkan binary, but with the latest iteration the embedded file should be directly accessible as a
Not only, it's easier to paste the path without having to add the prefix boiler plate. I prefer to set the DIR once than copypasting the DIR for every file.
Ouch that's awful! That should be better now, the map is now a map of a name and a pair of
I removed the Our other It's unsafe to access the |
|
Here is an example of how the current implementation works in the C++ code: void test()
{
// Using a file directly using its symbol:
const char* file = (const char*) EngineShaders::common_glsl;
// Loading a file by its name:
const char* filename = "common.glsl";
auto found = EngineShaders::FileMap.find(filename);
const char* foundfile = (const char*) found->second.data;
// Iterating the files:
for ( auto it : EngineShaders::FileMap )
{
Log::Debug("Embedded file %s at address %p with size %d", it.first, it.second.data, it.second.size);
}
} |
|
The map is just a convenience for accessing the files, the map just stores the name, the size and the pointer to the data, so I assume the compiler will drop the map when the data is only accessed directly. This should fit the need for:
This code was tested with my builtin pak implementation, so the only thing I haven't tested is the direct access (more than compiling the above code) is the direct access, but that should work. |
src/common/EmbeddedFile.h
Outdated
| @@ -0,0 +1,7 @@ | |||
| struct embeddedFileMapEntry_t | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can use Str::StringRef. There's no reason to be using unsigned char instead of char for the data
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's no reason to be using
unsigned charinstead ofcharfor the data
I tried:
In file included from GeneratedSource/DaemonEmbeddedFiles/EngineShaders.cpp:7:
GeneratedSource/DaemonEmbeddedFiles/EngineShaders/fogEquation_fp_glsl.h:137:1:
error: narrowing conversion of ‘226’ from ‘int’ to ‘char’ [-Wnarrowing]
137 | };
| ^
GeneratedSource/DaemonEmbeddedFiles/EngineShaders/fogEquation_fp_glsl.h:137:1:
error: narrowing conversion of ‘137’ from ‘int’ to ‘char’ [-Wnarrowing]
GeneratedSource/DaemonEmbeddedFiles/EngineShaders/fogEquation_fp_glsl.h:137:1:
error: narrowing conversion of ‘164’ from ‘int’ to ‘char’ [-Wnarrowing]
GeneratedSource/DaemonEmbeddedFiles/EngineShaders/fogEquation_fp_glsl.h:137:1:
error: narrowing conversion of ‘226’ from ‘int’ to ‘char’ [-Wnarrowing]
GeneratedSource/DaemonEmbeddedFiles/EngineShaders/fogEquation_fp_glsl.h:137:1:
error: narrowing conversion of ‘136’ from ‘int’ to ‘char’ [-Wnarrowing]
GeneratedSource/DaemonEmbeddedFiles/EngineShaders/fogEquation_fp_glsl.h:137:1:
error: narrowing conversion of ‘154’ from ‘int’ to ‘char’ [-Wnarrowing]
GeneratedSource/DaemonEmbeddedFiles/EngineShaders/fogEquation_fp_glsl.h:137:1:
error: narrowing conversion of ‘226’ from ‘int’ to ‘char’ [-Wnarrowing]
GeneratedSource/DaemonEmbeddedFiles/EngineShaders/fogEquation_fp_glsl.h:137:1:
error: narrowing conversion of ‘137’ from ‘int’ to ‘char’ [-Wnarrowing]
GeneratedSource/DaemonEmbeddedFiles/EngineShaders/fogEquation_fp_glsl.h:137:1:
error: narrowing conversion of ‘165’ from ‘int’ to ‘char’ [-Wnarrowing]
GeneratedSource/DaemonEmbeddedFiles/EngineShaders/fogEquation_fp_glsl.h:137:1:
error: narrowing conversion of ‘226’ from ‘int’ to ‘char’ [-Wnarrowing]
GeneratedSource/DaemonEmbeddedFiles/EngineShaders/fogEquation_fp_glsl.h:137:1:
error: narrowing conversion of ‘136’ from ‘int’ to ‘char’ [-Wnarrowing]
GeneratedSource/DaemonEmbeddedFiles/EngineShaders/fogEquation_fp_glsl.h:137:1:
error: narrowing conversion of ‘154’ from ‘int’ to ‘char’ [-Wnarrowing]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah yes that's very inconvenient to work around at the level of defining the actual array, but we should reinterpret_cast any pointers to char as soon as possible because unsigned char is worse for all purposes.
cmake/EmbedText.cmake
Outdated
| endif() | ||
|
|
||
| # Add null terminator. | ||
| string(REGEX REPLACE ",$" ",0x00," contents "${contents}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This shouldn't be using a regex replace since it is in a fixed location at the end.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right, fixed.
cmake/DaemonSourceGenerator.cmake
Outdated
| @@ -0,0 +1,139 @@ | |||
| set(DAEMON_TEXT_EMBEDDER "${CMAKE_CURRENT_SOURCE_DIR}/cmake/EmbedText.cmake") | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This "Text" naming scheme doesn't make that much sense now that it can handle binary files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, but I didn't want to rename that one in that PR for now. I was going to name it DaemonEmbeddedFile or something like that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
|
|
||
| file(MAKE_DIRECTORY "${embed_dir}") | ||
|
|
||
| foreach(kind CPP H) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The code in this loop is too confusing. I shouldn't have to sit here half an hour trying to decipher this code using multiple variable indirection to do something that probably could have been done by just setting a variable to a hard-coded string.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This kind of indirection saves me a lot of time when rewriting and re-rewriting across my various iterations , and it already saved me a lot of time. It only looks better to hardcode those strings when all the work was done by someone else, because then you don't know all the hundreds of different strings that have been used before being presented to the final state. Those indirections aren't only written to reduce the amount of lines to edit when redesigning things, but also to avoid the introduction of mistakes when editing the alternative expanded boiler plate.
And we know well who is doing all those iterations and who will suffer from editing the duplicated boiler plate once it is expanded:
git ls-files | grep -i cmake | while read f; do git blame -w -M -C -C --line-porcelain "$f" | grep -I '^author '; done | sort -f | uniq -ic | sort -n --reverse
2135 author Thomas Debesse
1286 author TimePath
612 author slipher
334 author Daniel Maloney
243 author Corentin Wallez
175 author Darren Salt
161 author dolcetriade
135 author VReaperV
108 author Amanieu d'Antras
107 author perturbed
43 author Tsarevich Dmitry
29 author Morel Bérenger
12 author Mattia Basaglia
7 author Tim
3 author cmf028
2 author Jesper B. Christensen
1 author maek
1 author Keziolio
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It only looks better to hardcode those strings when all the work was done by someone else,
Yes that's the point. Other people besides you should be able to read the code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I meant: You don't know yet what's the cost of maintaining it is.
I can tell you it's easier this way, because I already maintained it.
Said other way: this is the solution I have chosen because I first tried the hardcoded way and it was painful and doing it this way made it easier. Lessons got learned, I share you the result, I save you time.
| endif() | ||
|
|
||
| include(DaemonBuildInfo) | ||
| include(DaemonSourceGenerator) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Name seems inapt, how about DaemonFileEmbedding? Or alternatively something about "resources" since that is a commonly used term for embedding files in this way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just DaemonEmbed? That's what I did in #1845 until I rebased on this pr.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
DaemonEmbed or a variant of it (DamonEmbeddedFile?) is the naming I'm planing to use for EmbedText that doesn't only produce embedded text files anymore.
Also this cmake does more code generation than just embedding files (like the buildinfo thing).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cmake/DaemonSourceGenerator.cmake
Outdated
| endmacro() | ||
|
|
||
| macro(daemon_embed_files basename slug format targetname) | ||
| set(embed_source_dir "${slug}_EMBED_DIR") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This variable name concatenation will make the code hard to understand later. For example if you search for GLSL_EMBED_DIR you will not find anything; the variable seems to be unused.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's better than boilerplate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This mechanism also makes possible to embed files with paths having directories in them, the previous code only supported basename and it was then also making assumptions in the back of the user that were more painful and more limiting.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This mechanism also makes possible to embed files with paths having directories in them, the previous code only supported basename and it was then also making assumptions in the back of the user that were more painful and more limiting.
That's a separate question. You can have directory names in the path without magical variable concatenation. The function can take two arguments - base dir variable and file list variable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I modified it to pass variables instead.
ff5cf6c to
6978bf5
Compare
c5da974 to
2bdccd1
Compare
|
This now uses That now looks ready to me. |
|
Looking pretty good but I still want to ask that |
2bdccd1 to
b3d0517
Compare
|
I don't know how to make it use a |
A StringRef could be constructed like |
|
This is accepted by the compiler: const Str::StringRef screen_fp_glsl = reinterpret_cast<const char*>( data_screen_fp_glsl );This isn't: const Str::StringRef screen_fp_glsl = { reinterpret_cast<const char*>( data_screen_fp_glsl ),
sizeof ( data_screen_fp_glsl ) - 1 };But the first one has then Also with the first implementation (without the size), with some more elaborated embedded file code than the GLSL one I get some errors about |
|
It looks like I need to add a new |
|
Oops, I didn't realize we were missing that constructor. See #1872 |
|
Yes, I just added a new constructor on my end (but without the assert) and it worked. |
|
Yours has a chicken-and-egg problem about the ASSERT definition. I'm currently using the simpler constructor without assert while waiting for the assert-featured one to work. |
77fee60 to
a0fd2b3
Compare
|
I modified the code to be standalone (no dependency from |
56b6260 to
e38f914
Compare
|
For some unknown reasons code becomes simpler with that dedicated struct than with For example when testing this code with #1842, here is what I need to do with for (auto& p : builtinPakMap[pak.name])
{
if (p.first == path)
{
const Str::StringRef& entry = p.second;
std::string out;
out.resize(entry.size());
memcpy(&out[0], entry.data(), entry.size());
return out;
}
}While I can do this simpler code with the simple dedicated struct: const embeddedFileMapEntry_t& entry = builtinPakMap[pak.name][path];
std::string out;
out.resize(entry.size);
memcpy(&out[0], entry.data, entry.size);
return out;With |
c2d2350 to
5c7bc3f
Compare
StringRef has automatic conversion to string so you could do Anyway it's fine with me as long as the interface doesn't used |
|
So, here are usage examples of the API that is implemented in the code that has been merged: #include "DaemonEmbeddedFiles/EngineShaders.h"
void test()
{
// Using a file directly using its symbol:
const char* file = EngineShaders::common_glsl.data;
// Loading a file by its name:
const char* filename = "common.glsl";
auto found = EngineShaders::FileMap.find(filename);
const char* foundfile = found->second.data;
// Iterating the files:
for ( auto it : EngineShaders::FileMap )
{
Log::Debug("Embedded file %s at address %p with size %d", it.first, it.second.data, it.second.size);
}
} |
Extracted from #1842:
Extracted and polished. The design is now stable enough to be submitted for merging.
The bug I face in #1842 is unrelated to that code.
Usage example: