[question] self.export_sources_folder in def export_sources(self) method

[AroshikaFernando]

I am learning about conan and try migrate conan 1.x recipes to conan 2.x.
Could you please someone tell me where the self.export_sources_folder tis folder is define in following recipe file?

What is the value of self.export_sources_folder for following scenario?

import os
from conan import ConanFile
from conan.tools.files import load, copy
from conan.tools.cmake import CMake


class PkgSay(ConanFile):
    name = "say"
    version = "1.0"
    settings = "os", "compiler", "build_type", "arch"
    generators = "CMakeToolchain"

    def layout(self):
        # The root of the project is one level above
        self.folders.root = ".." 
        # The source of the project (the root CMakeLists.txt) is the source folder
        self.folders.source = "."  
        self.folders.build = "build"

    def export_sources(self):
        # The path of the CMakeLists.txt and sources we want to export are one level above
        folder = os.path.join(self.recipe_folder, "..")
        copy(self, "*.txt", folder, self.export_sources_folder)
        copy(self, "src/*.cpp", folder, self.export_sources_folder)
        copy(self, "include/*.h", folder, self.export_sources_folder)
    
    def source(self):
        # Check that we can see that the CMakeLists.txt is inside the source folder
        cmake_file = load(self, "CMakeLists.txt")

    def build(self):
        # Check that the build() method can also access the CMakeLists.txt in the source folder
        path = os.path.join(self.source_folder, "CMakeLists.txt")
        cmake_file = load(self, path)

        cmake = CMake(self)
        cmake.configure()
        cmake.build()

    def package(self):
        cmake = CMake(self)
        cmake.install()

[memsharded]

Hi @AroshikaFernando

The exports_sources is a folder in the Conan cache, that contains files that belong to the recipe that will be needed later to build the package binaries from source.

In the example above, the export_sources() method is copying files from your current user folder (user space, the folder that you git clone and contains your conanfile.py) to the Conan cache folder of the say/1.0 that stores the recipe sources.

Does this clarify a bit?

[AroshikaFernando]

Hi @memsharded,

So, Don't we need to define what self.export_sources_folder is in layout() method or anywhere?

Because, in my case, I'll put a print line in my conanfile.py for self.export_sources_folder, but it's printed as export_sources_folder is folder in my local folder which have conanfile.py.

How do I change that?

[memsharded]

Yes, the self.export_sources_folder is read-only, Conan defines it automatically, it cannot be defined by recipes or users, and also it is not something that could be indirectly customized via layout().

The self.export_sources_folder will only take the right value inside the export_sources() method, when you do a conan install for example, it can be the local folder, indeed.

Why would you need to change it? It shouldn't be necessary to change this, ever.

[AroshikaFernando]

Thank you @memsharded
No. I don't want to change it.
I got confused cause print line gave me different output other than the folder in the Conan cache.

And I have another concern about my folder structure.

My project folder as follows.

Main Project Folder (which is from git clone)
├── build Directory => which is for build the source code and containing build outputs (libraries and outputs)
├── Sub Component
└──src
├ └──build_config
├ └── conanfile.py
├── *.h files and *.cpp files

How do I invoke build directory from conanfile.py?
And Could you please clarify what would be the self.build_folder, self.source_folder in above scenario?

When I print self.build_folder and self.source_folder inside the package(), it both gives path to connafile.py
main_project_folder/ sub_somponent/src/build_config

[memsharded]

Quick question: why are you putting the conanfile.py inside src folder? It is making your life more difficult. While there are layout() features to model this, like self.folders.root = "..", that I see you are using, in general the recommended approach is to have the conanfile.py in the root of your repo.

It is also confusing that you have self.folders.source = ".", but then have a src folder, and you have the CMakeLists.txt in the root of the repo. What is the "sub component"? is it another conan package? Just more source .cpp files? Are the actual .h and .cpp files above inside the src folder?

All the folders that you ask for, the answer is it depends. It depends on your layout() method mostly, but can also be altered by --output-folder (not necessary in most cases, the recommended approach would be to use the recipe layout() method)

[AroshikaFernando]

Hi @memsharded

I'll add our repo structure little bit more clear.
And I can't change that structure. I have to continue the Conan migration to 2.0 with this structure. That's why I'm having questions about my folder structure.

Our main project have several sub component. Each sub component has own conanfile.py file, CMakeLists.txt file.

Detailed Repo Structure of our project as follows,

I am building each sub component separately or all the sub components at once in the build folder, which stated above. And that build folder having all the build outputs like binaries and libraries.

And how can I access that build (Build directory) folder from my conanfile.py file?

And should self.folders.root be a folder that includes conanfile.py?

[memsharded]

This layout should make it:

def test_project_subcomponents_layout():
    c = TestClient()
    pkg = textwrap.dedent("""
        import os
        from conan import ConanFile
        from conan.tools.files import copy

        class Pkg(ConanFile):
            version = "0.1"

            def export_sources(self):
                copy(self, "*.cpp",
                     src=os.path.join(self.recipe_folder, "..", ".."),
                     dst=os.path.join(self.export_sources_folder, self.name))

            def layout(self):
                self.folders.root = "../../.."
                self.folders.source = f"{self.name}/src"
                self.folders.build = f"mybuild/{self.name}"
                self.folders.generators = os.path.join(self.folders.build, "generators")

            def source(self):
                self.output.info(f"SOURCE-SOURCE FOLDER {self.source_folder}")
                assert os.path.exists(os.path.join(self.source_folder, f"{self.name}.cpp"))

            def generate(self):
                self.output.info(f"GENERATE-SOURCE FOLDER {self.source_folder}")
                assert os.path.exists(os.path.join(self.source_folder, f"{self.name}.cpp"))

            def build(self):
                self.output.info(f"BUILD-SOURCE FOLDER {self.source_folder}")
                self.output.info(f"BUILD FOLDER {self.build_folder}")
                assert os.path.exists(os.path.join(self.source_folder, f"{self.name}.cpp"))

            def package_info(self):
                self.output.info(f"PKG package_folder {self.package_folder}")
        """)
    c.save({"comp1/src/build_config/conanfile.py": pkg,
            "comp1/src/comp1.cpp": "",
            "comp2/src/build_config/conanfile.py": pkg,
            "comp2/src/comp2.cpp": ""
            })

    # This works without problems
    c.run("create comp1/src/build_config --name=comp1")

    # local development also puts things in the build folder
    c.run("build comp1/src/build_config --name=comp1")
    c.run("build comp2/src/build_config --name=comp2")
    print(c.current_folder)

Check the layout() method and the export_sources(), no much magic there, but explicitly defining the folders according to your above requirements.

[AroshikaFernando]

Thank you very much @memsharded

[memsharded]

Did the above work? Please let us know, maybe we can close the issue? Thanks.

[AroshikaFernando]

Hi @memsharded

self.folders.build = f"mybuild/{self.name}"
What is mybuild folder you are referring to? Is it build folder in the above attached image inside the mtech folder?
If it's same, is there any method to access the build folder without using "build" folder name?

self.folders.generators = os.path.join(self.folders.build, "generators")
And could you please explain the use of this self.folders.generators in my case?

And is it must to define source(), generate(), build() also in my recipe file?

[memsharded]

What is mybuild folder you are referring to?

I am using "mybuild" to make sure it is the value I defined, instead of some other default (specially while debugging). So, yes, you can rename it to "build" and it will be your folder above

f it's same, is there any method to access the build folder without using "build" folder name?

Do you mean self.build_folder? Not sure what is the question

And could you please explain the use of this self.folders.generators in my case?

Conan generates a bunch of files for the build. It is convenient to have them all in a known place. It could be just the build folder too, but that is dirtier

And is it must to define source(), generate(), build() also in my recipe file?

they are there just for the test and to validate with the asserts that everything is good, no need to define them in your recipes if you don't need them.

[AroshikaFernando]

Thank you @memsharded for the explanation.

Do you mean self.build_folder? Not sure what is the question

Yes, can we access this self.folders.build = f"mybuild/{self.name}" without using "mybuild" folder name?

Because I'm in a situation like, I can't put restrictions to folder names.
In my case build folder name created by users who build the code.
If I use this "mybuild" build folder name in my recipe file, all the users of that recipe file also should be having build folder name "mybuild". It's not practical for my case.

And How to copy files in the root folder?
I mean my ProjectInfo.yml is in the root folder and I want to copy it to the self.export_folder.
copy(self, "ProjectInfo.yml", src = ".", dst = os.path.join(self.export_folder, "."))
What is the src would be in above scenario?

[memsharded]

Yes, can we access this self.folders.build = f"mybuild/{self.name}" without using "mybuild" folder name?

You can use self.folders.build = f"build/{self.name}", no need to use "mybuild"

What is the src would be in above scenario?

The current dir is the recipe dir, so you need to go up

def export_sources(self):
      copy(self, "*.cpp",
           src=os.path.join(self.recipe_folder, "..", ".."),
           dst=os.path.join(self.export_sources_folder, self.name))
      copy(self, "somefile.txt",
           src=os.path.join(self.recipe_folder, "..", "..", ".."),
           dst=self.export_sources_folder)

[AroshikaFernando]

You can use self.folders.build = f"build/{self.name}", no need to use "mybuild"

So, is it necessary to use folder name like above right?

The current dir is the recipe dir, so you need to go up

Okay. Thank you.

[AroshikaFernando]

You can use self.folders.build = f"build/{self.name}", no need to use "mybuild"

self.folders.build = f"<build folder name>/{self.name}"
is there any method to define self.folders.build without using this folder name?

[memsharded]

is there any method to define self.folders.build without using this folder name?

I am not sure what you mean. The folder is called "build", you need to specify at least that self.folders.build = "build". The {self.name} part is recommended to avoid collisions between different subprojects, otherwise sharing the same build folders can produce problems (for example, the Conan generated files could ovewrite each other)

[AroshikaFernando]

Okay. Thank you very much @memsharded .

My issue is sorted for now. I'll Close this issue with this comment.

[AroshikaFernando]

Hi @memsharded,

How can I access folders in my root folder in package()?

As example I want to copy files from scripts folder to package folder in conan cache and that scripts folder is in my root folder.
In above folder structure, that script folder is in the mtech folder as follows

mtech
├──build
├──scripts
├──sub component1
├──sub component2

When I try this in the package() os.path.join(self.recipe_folder, "..", "..", "..") It'll give wrong path.

copy(self, "*", src = <script folder>, dst = dst = os.path.join(self.package_folder,"lib")

What is the src in above scenario?

[memsharded]

Typically you don't. You access self.source_folder and self.build_folder. The previous steps should have located or prepared things to be in those folders. Also, it is different if you are running a conan create and package() is in the Conan cache or if you are running a conan export-pkg and it is packaging from your user folder.

[AroshikaFernando]

Okay thank you very much.