[question] Packages build caching

[omgronny]

What is your question?

Hello, I use conan2 in my projects and I face an unexpected bahavior with building a packages (builds are caching even if I change build settings):

I have a boost stacktrace dependency, and if I build my project in release version w/o sanitizers, it compiles fine

If I run rm -rf ~/.conan2, and then build my project with asan, I get an error:

boost/1.83.0: WARN: Boost component 'stacktrace_backtrace' is missing libraries. Try building boost with '-o boost:without_stacktrace_backtrace'. (Option is not guaranteed to exist)
ERROR: boost/1.83.0: Error in package_info() method, line 2011
	raise ConanException(f"These libraries were expected to be built, but were not built: {non_built}")
	ConanException: These libraries were expected to be built, but were not built: {'boost_stacktrace_backtrace'}

This is a known boost error, so it is expected behahior

But if I build my project w/o asan, and then build it with asan (w/o removing .conan2), it compiles fine. So I think some of my projects actually compiling w/o sanitizers because conan just use their built versions

I use --build missing flag in conan install because if compiler settings hasn't changed, I want conan too use build versions

my conanfile.py:

from conan import ConanFile
from pathlib import Path
from conan.tools.cmake import CMakeToolchain, CMake, CMakeDeps

class RepoConan(ConanFile):
    name = "repo"
    settings = ("os", "build_type", "compiler", "sanitizer")
    options = {}
    requires = [
        "arrow/18.1.0",
    ]

    def generate(self):
        deps = CMakeDeps(self)
        deps.generate()
        tc = CMakeToolchain(self, generator="Ninja")
        tc.generate()

    def build(self):
        cmake = CMake(self)
        cmake.configure()
        cmake.build()

    def package(self):
        cmake = CMake(self)
        cmake.install()

    def config_options(self):
        asan = ["-fsanitize=address,undefined"]
        tsan = ["-fsanitize=thread"]

        if self.settings.sanitize or self.settings.sanitizer == "address":
            self.conf.define("tools.build:cflags", asan)
            self.conf.define("tools.build:cxxflags", asan)
            self.conf.define("tools.build:exelinkflags", asan)
            self.conf.define("tools.build:sharedlinkflags", asan)
        elif self.settings.sanitizer == "thread":
            self.conf.define("tools.build:cflags", tsan)
            self.conf.define("tools.build:cxxflags", tsan)
            self.conf.define("tools.build:exelinkflags", tsan)
            self.conf.define("tools.build:sharedlinkflags", tsan)

    def requirements(self):
        self.requires("boost/1.83.0", override=True, transitive_headers=True)
        self.requires("boost::headers/1.83.0", override=True, transitive_headers=True)

    default_options = {
        "arrow/*:fPIC": False,
        "arrow/*:shared": False,
        "arrow/*:parquet": True,
        "arrow/*:with_csv": True,
        "arrow/*:with_zstd": True,
        "boost/*:fPIC": False,
        "boost/*:shared": False,
        "boost/*:with_stacktrace_backtrace": True,
        "boost/*:zstd": True,
    }

def package_info(self):
    self.cpp_info.requires.append("boost::headers")

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[omgronny]

And if I change --build=missing to --build='*', I get:

Requirements
    arrow/18.1.0#032d83f98246ca1d0facc6413141392e:83cc3a618565e3632c429aa4bd1efc0c05105c92 - Missing
    boost/1.83.0#dd39934d1af6ee91b3bd0d5668b3b93e:aa6a95d37134d263e1525ba22443b2fd037620d3#0497ad166b23b6474be6c833cb6adf67 - Cache
Build requirements
Skipped binaries
ERROR: Missing binary: arrow/18.1.0:83cc3a618565e3632c429aa4bd1efc0c05105c92

ERROR: Missing prebuilt package for 'arrow/18.1.0'. You can try:
    - List all available packages using 'conan list "arrow/18.1.0:*" -r=remote'
    - Explain missing binaries: replace 'conan install ...' with 'conan graph explain ...'
    - Try to build locally from sources using the '--build=missing' argument

More Info at 'https://docs.conan.io/2/knowledge/faq.html#error-missing-prebuilt-package'

And I'm also confused by the fact that boost have been found in cache instead of building from source

[memsharded]

Thanks for your question @omgronny

The bit that you might be missing is this one, that is from the Conan 1 docs (we haven't had time yet to update it for the new Conan 2 docs), but that it would still be valid for Conan 2: https://docs.conan.io/1/howtos/sanitizers.html#adding-custom-settings

The key points are:

• You are using settings = ("os", "build_type", "compiler", "sanitizer") in your package
• You should most like add the arch there too
• But the boost recipe is not defined that settings, it will typically define settings = ("os", "build_type", "compiler", "arch")
• So boost binaries will not depend on sanitizer. If you wanted that there are 2 alternatives:
  • Modify the boost recipe to include sanitizer there.
  • Or use sanitizer as a sub-setting to the compiler as shown in https://docs.conan.io/1/howtos/sanitizers.html#adding-a-list-of-commonly-used-values, instead of having sanitizer as a first-level setting that needs to be added explicitly to settings in recipe code.

[omgronny]

Thanks for your answer

I also use conan profiles

for release:

[settings]
os=Linux
arch=x86_64
compiler=clang
compiler.cppstd=20
compiler.version=17
compiler.libcxx=libc++
build_type=Release
[options]
[buildenv]
CC=/usr/bin/clang-17
CXX=/usr/bin/clang++-17
CPLUS_INCLUDE_PATH=/usr/lib/llvm-17/include/c++/v1/

and for asan:

[settings]
os=Linux
arch=x86_64
compiler=clang-san
compiler.cppstd=20
compiler.version=17
compiler.libcxx=libc++
build_type=Release
boost/*:compiler=clang
boost/*:compiler.cppstd=20
boost/*:compiler.version=17
boost/*:compiler.libcxx=libc++
[options]
[buildenv]
CC=/usr/bin/clang-17
CXX=/usr/bin/clang++-17
CFLAGS=-fsanitize=address,undefined
CXXFLAGS=-fsanitize=address,undefined -fexperimental-library -stdlib=libc++

So I should add compiler.sanitizer=AddressUndefinedBehavior and it should help?
Or, I guess, boost/*:compiler.sanitizer=AddressUndefinedBehavior?

[omgronny]

Yes, It seems to me that adding boost/*:compiler.sanitizer=AddressUndefinedBehavior has helped.
Thank you

[memsharded]

• compiler=clang-san is not a recognized compiler, it might do unexpected things in different build system integrations
• I'd recommend to use compiler=clang and compiler.sanitizer=san (or the value you have defined in your custom settings) if anything
• Yes compiler.sanitizer=AddressUndefinedBehavior might be good if you want to build all your packages and dependencies with the sanitizer activated (some sanitizers require to be consistently used across the whole project and dependencies)
• The boost/*:compiler.sanitizer=AddressUndefinedBehavior is ok, if the sanitizer doesn't require the whole project and dependencies to be compiled with the same sanitizer settings.

[omgronny]

Thank you for helping

I have one more related question. What is the proper way to build my project for specific arch so that all the imported packages will also be built fot this arch? Should I set arch (or arch_build or arch_target?) in my conan profile?

And is it possible to set arch to sapphirerapids?

[memsharded]

I have one more related question. What is the proper way to build my project for specific arch so that all the imported packages will also be built fot this arch? Should I set arch (or arch_build or arch_target?) in my conan profile?

arch_build doesn't exist in Conan 2, maybe you refer settings_build.arch?

The way that Conan has to define architeucture is with the settings, so defining arch=xxxx in your profile is the way to go, that applies automatically to all packages in the dependency graph.

Conan default settings can be extended, it is possible to add new architectures, and other stuff there (see the settings_user.yml file, for example. But adding random values there like sapphirerapids (what architecture is that?) will not be understood by the build-system integrations like CMakeDeps/CMakeToolchain, MesonDeps/MesonToolchain, etc. So probably not recommended in most cases.

[omgronny]

sapphirerapids is an architecture of new intel xeon cpus: https://reviews.llvm.org/D86503

I have one more question:
Is it possible to define march so that if some package has the builds for this arch (sapphirerapids in my case), it will use them, and use x86-64 otherwise?

For example, abseil can be compiled with -march=sapphirerapids, but cmake cannot, so I want the abseil to be compiled with -march=sapphirerapids, and cmake with -march=x86-64?

[memsharded]

But that is a micro-architecture, not an architecture, isn't it?

For micro-architectures, the recommendation would be to model them as sub-settings of the corresponding architecture, but leave the architecture as x86_64 or whatever it belongs to. That will do the right package_id computation, differentiating binaries built with different micro-architectures. If you want to define fallback logic across binaries built with different settings, that is possible to be done in the compatibility.py plugin: https://docs.conan.io/2/reference/extensions/binary_compatibility.html

Then you might to define the compiler flags accordingly, as the Conan built-in build-system integrations are not aware of such subsetting definition. So you might need to pass the specific flag, maybe via tools.build:cxxflags or something like that. You can define different confs for different packages with the mypkg/*:tools.build:... patterns.

[omgronny]

So in my settings.yml it should be something like

arch:
    x86_64:
        ???: [x86_64, sapphirerapids]

And then in conan profile

arch.???=sapphirerapids

But what "???" should be?

And I didn't actually get a pint with tools.build:cxxflags. I should set -march for my dependencies packages using it?

[memsharded]

You can define settings_user.yml to avoid having to copy and maintain the whole file.

Then in it, define any name that you like, something like:

arch:
    x86_64:
        micro: [null, sapphirerapids]

You can refer to it in recipes as self.settings.arch.micro == "xxx" and it will get the validation features as all the other settings.

And I didn't actually get a pint with tools.build:cxxflags. I should set -march for my dependencies packages using it?

The tools.build:cxxflags is a mechanism to inject cxxflags, yes, for any package that is built that matches the possible pattern of that conf (or all packages if no pattern is specified).

[omgronny]

Okay, now I get the idea of specifying micro, thank you

Just to be sure (because I don't know how to properly check it myself)

    def config_options(self):
        if self.settings.arch.micro == "sapphirerapids":
            self.conf.define("tools.build:cxxflags", ["-march=sapphirerapids"])

This is what conanfile.py should look like to add this flag to package dependencies?

[memsharded]

The configuration comes from profiles, you cannot define it in recipes. See https://docs.conan.io/2/knowledge/guidelines.html

Settings and configuration (conf) are read-only in recipes: The settings and configuration cannot be defined or assigned values in recipes. Something like self.settings.compiler = "gcc" in recipes shouldn’t be done. That is undefined behavior and can crash at any time, or just be ignored. Settings and configuration can only be defined in profiles, in command line arguments or in the profile.py plugin.

If you want to condition something on a setting for the current package build, do it in the generate() method, in the CMakeToolchain.extra_xxxxx flags. If you want to do it for all packages, put it in a profile.