infra: Roll OUR_LLVM_REVISION

[maflcko]

From #11707 with the branch renamed

[maflcko]

Ref googleprojectzero/SockFuzzer#16

[maflcko]

tensorflow/tensorflow#64673

[maflcko]

This needs https://chromium.googlesource.com/chromium/src/+/9bfbbffdba73668fdb483e5a850911d2b64c35d7%5E%21/#F0

[nico]

Alternatively, it needs an even newer toolchain, since this is fixed in newer libc++s.

[maflcko]

The error:

Found clang 18.0
clang>=8, using the major as version
Profile created with detected settings: /root/.conan/profiles/default
+ conan profile update settings.compiler.version=18 default
+ conan profile update settings.compiler.libcxx=libc++ default
+ conan profile update settings.compiler.fpo=False default
+ conan profile update settings.compiler.address_sanitizer=True default
+ conan profile update settings.compiler.fuzzer_sanitizer=True default
+ sed -i 's/\[settings\]/include(libfuzzer_base)\n\n[settings]/' /root/.conan/profiles/default
+ echo 'CFLAGS=$BASE_CFLAGS'
+ echo 'CXXFLAGS=$BASE_CXXFLAGS'
+ echo 'LDFLAGS=$BASE_LDFLAGS'
+ echo 'OrbitProfiler:CFLAGS=$BASE_CFLAGS -O1 -fno-omit-frame-pointer -gline-tables-only -Wno-error=enum-constexpr-conversion -Wno-error=incompatible-function-pointer-types -Wno-error=int-conversion -Wno-error=deprecated-declarations -Wno-error=implicit-function-declaration -Wno-error=implicit-int -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link'
+ echo 'OrbitProfiler:CXXFLAGS=$BASE_CFLAGS -O1 -fno-omit-frame-pointer -gline-tables-only -Wno-error=enum-constexpr-conversion -Wno-error=incompatible-function-pointer-types -Wno-error=int-conversion -Wno-error=deprecated-declarations -Wno-error=implicit-function-declaration -Wno-error=implicit-int -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -stdlib=libc++'
+ echo 'OrbitProfiler:LDFLAGS=$BASE_LDFLAGS '
+ echo 'llvm-core:CFLAGS=$BASE_CFLAGS -O1 -fno-omit-frame-pointer -gline-tables-only -Wno-error=enum-constexpr-conversion -Wno-error=incompatible-function-pointer-types -Wno-error=int-conversion -Wno-error=deprecated-declarations -Wno-error=implicit-function-declaration -Wno-error=implicit-int -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link'
+ echo 'llvm-core:CXXFLAGS=$BASE_CXXFLAGS -O1 -fno-omit-frame-pointer -gline-tables-only -Wno-error=enum-constexpr-conversion -Wno-error=incompatible-function-pointer-types -Wno-error=int-conversion -Wno-error=deprecated-declarations -Wno-error=implicit-function-declaration -Wno-error=implicit-int -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -stdlib=libc++'
+ echo 'llvm-core:LDFLAGS=$BASE_LDFLAGS '
+ /src/orbit/build.sh default
ERROR: Invalid setting '18' is not a valid 'settings.compiler.version' value.
Possible values are ['3.3', '3.4', '3.5', '3.6', '3.7', '3.8', '3.9', '4.0', '5.0', '6.0', '7.0', '7.1', '8', '9', '10', '11', '12', '13', '14', '15', '16']
Read "http://docs.conan.io/en/latest/faq/troubleshooting.html#error-invalid-setting"
ERROR:__main__:Building fuzzers failed.

[maflcko]

The error:

/src/immer/immer/detail/hamts/node.hpp:229:26: runtime error: constructor call on address 0x55b5a324ff80 with insufficient space for an object of type 'node_t' (aka 'immer::detail::hamts::node<std::pair<unsigned long, int>, immer::map<unsigned long, int, colliding_hash_t, std::equal_to<void>, immer::memory_policy<immer::heap_policy<immer::cpp_heap>, immer::unsafe_refcount_policy, immer::no_lock_policy, immer::no_transience_policy, false>>::hash_key, immer::map<unsigned long, int, colliding_hash_t, std::equal_to<void>, immer::memory_policy<immer::heap_policy<immer::cpp_heap>, immer::unsafe_refcount_policy, immer::no_lock_policy, immer::no_transience_policy, false>>::equal_key, immer::memory_policy<immer::heap_policy<immer::cpp_heap>, immer::unsafe_refcount_policy, immer::no_lock_policy, immer::no_transience_policy, false>, 5>')
0x55b5a324ff80: note: pointer points here
 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00
              ^ 
    #0 0x55b5a0cc6222 in make_inner_n /src/immer/immer/detail/hamts/node.hpp:229:18
    #1 0x55b5a0cc6222 in immer::detail::hamts::champ<std::__1::pair<unsigned long, int>, immer::map<unsigned long, int, colliding_hash_t, std::__1::equal_to<void>, immer::memory_policy<immer::heap_policy<immer::cpp_heap>, immer::unsafe_refcount_policy, immer::no_lock_policy, immer::no_transience_policy, false, true>, 5u>::hash_key, immer::map<unsigned long, int, colliding_hash_t, std::__1::equal_to<void>, immer::memory_policy<immer::heap_policy<immer::cpp_heap>, immer::unsafe_refcount_policy, immer::no_lock_policy, immer::no_transience_policy, false, true>, 5u>::equal_key, immer::memory_policy<immer::heap_policy<immer::cpp_heap>, immer::unsafe_refcount_policy, immer::no_lock_policy, immer::no_transience_policy, false, true>, 5u>::empty() /src/immer/immer/detail/hamts/champ.hpp:142:34
    #2 0x55b5a0cc4170 in map /src/immer/immer/map.hpp:547:20
    #3 0x55b5a0cc4170 in LLVMFuzzerTestOneInput /src/immer/extra/fuzzer/map-st.cpp:36:46
    #4 0x55b5a0c26af0 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:614:13
    #5 0x55b5a0c27ff1 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:807:3
    #6 0x55b5a0c285d7 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:867:3
    #7 0x55b5a0c16be6 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:914:6
    #8 0x55b5a0c43112 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #9 0x7f11aab5b082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 87b331c034a6458c64ce09c03939e947212e18ce)
    #10 0x55b5a0c07d5d in _start (/tmp/not-out/tmpr0lgzuu6/map-st+0x2bd5d)

DEDUP_TOKEN: make_inner_n--immer::detail::hamts::champ<std::__1::pair<unsigned long, int>, immer::map<unsigned long, int, colliding_hash_t, std::__1::equal_to<void>, immer::memory_policy<immer::heap_policy<immer::cpp_heap>, immer::unsafe_refcount_policy, immer::no_lock_policy, immer::no_transience_policy, false, true>, 5u>::hash_key, immer::map<unsigned long, int, colliding_hash_t, std::__1::equal_to<void>, immer::memory_policy<immer::heap_policy<immer::cpp_heap>, immer::unsafe_refcount_policy, immer::no_lock_policy, immer::no_transience_policy, false, true>, 5u>::equal_key, immer::memory_policy<immer::heap_policy<immer::cpp_heap>, immer::unsafe_refcount_policy, immer::no_lock_policy, immer::no_transience_policy, false, true>, 5u>::empty()--map
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /src/immer/immer/detail/hamts/node.hpp:229:26 in 
MS: 0 ; base unit: 0000000000000000000000000000000000000000


artifact_prefix='./'; Test unit written to ./crash-da39a3ee5e6b4b0d3255bfef95601890afd80709
Base64: 

ERROR: 57.89473684210526% of fuzz targets seem to be broken. See the list above for a detailed information.
ERROR:__main__:Check build failed.

[maflcko]

The error:

/src/tarantool/src/lib/core/trigger.h:121:2: runtime error: member access within null pointer of type 'typeof (*trigger)' (aka 'struct trigger')
    #0 0x5604a358c3be in trigger_destroy /src/tarantool/src/lib/core/trigger.h:121:2
    #1 0x5604a356cc35 in fiber_destroy /src/tarantool/src/lib/core/fiber.c:1649:2
    #2 0x5604a356c9a7 in cord_destroy /src/tarantool/src/lib/core/fiber.c:1953:2
    #3 0x5604a3577d06 in fiber_free /src/tarantool/src/lib/core/fiber.c:2268:2
    #4 0x5604a3519112 in LLVMFuzzerTestOneInput /src/tarantool/test/fuzz/swim_proto_meta_fuzzer.c:41:5
    #5 0x5604a345d1a0 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:614:13
    #6 0x5604a345e6a1 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:807:3
    #7 0x5604a345ec87 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:867:3
    #8 0x5604a344c646 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:914:6
    #9 0x5604a34797c2 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #10 0x7fb394a92082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 87b331c034a6458c64ce09c03939e947212e18ce)
    #11 0x5604a343d7bd in _start (/tmp/not-out/tmpku8u9l2_/swim_proto_meta_fuzzer+0x1d17bd)

DEDUP_TOKEN: trigger_destroy--fiber_destroy--cord_destroy
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /src/tarantool/src/lib/core/trigger.h:121:2 in 

[jonathanmetzman]

BoringSSL says this is probably a false positive. https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=68441

[maflcko]

BoringSSL says this is probably a false positive. https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=68441

I don't have permission to see the issue. I presume it is an msan warning?

[maflcko]

If it is an msan warning, see #11880 for a potential fix.

If not, please share more details, so that I can take a look.

[davidben]

It is indeed an MSan warning.

[kleisauke]

I assume that after this we can safely remove the OLD_LLVMPASS environment variable? For example:

oss-fuzz/projects/libvips/Dockerfile

Lines 53 to 55 in c7e622f

	# This is to fix Fuzz Introspector build by using LLVM old pass manager
	# re https://github.com/ossf/fuzz-introspector/issues/305
	ENV OLD_LLVMPASS 1

(many other projects also define this env variable)

[DavidKorczynski]

I assume that after this we can safely remove the OLD_LLVMPASS environment variable? For example:

oss-fuzz/projects/libvips/Dockerfile

Lines 53 to 55 in c7e622f

	# This is to fix Fuzz Introspector build by using LLVM old pass manager
	# re https://github.com/ossf/fuzz-introspector/issues/305
	ENV OLD_LLVMPASS 1

(many other projects also define this env variable)

Yes, we should do that now -- will try and remove these

[catenacyber]

Hello @maflcko

I am not sure this is related, but the timing matches :
https://oss-fuzz.com/testcase-detail/5167091775766528 and https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=68317 do not get closed even if the bug was fixed... on April 30 by OISF/suricata@c53e9ac

The detailed report seems frozen :

Status: Pending, refreshes every 5 minutes.

The last tested revision shown is the one before the 30 of April :
OISF/suricata@ad4185b

I triggered a manual check if bug still reproduces
but it seems frozen as well :
[2024-05-06 14:44:21 UTC] [email protected]: Redo task(s): progression
[2024-05-06 15:58:18 UTC] oss-fuzz-linux-zone4-host-v79g-9: Progression task started.

Could we get this bug closed ?

[jonathanmetzman]

Hello @maflcko

I am not sure this is related, but the timing matches : https://oss-fuzz.com/testcase-detail/5167091775766528 and https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=68317 do not get closed even if the bug was fixed... on April 30 by OISF/suricata@c53e9ac

The detailed report seems frozen :

Status: Pending, refreshes every 5 minutes.

The last tested revision shown is the one before the 30 of April : OISF/suricata@ad4185b

I triggered a manual check if bug still reproduces but it seems frozen as well : [2024-05-06 14:44:21 UTC] [email protected]: Redo task(s): progression [2024-05-06 15:58:18 UTC] oss-fuzz-linux-zone4-host-v79g-9: Progression task started.

Could we get this bug closed ?

I think the failure you are seeing is because the suricata build is too big.

[catenacyber]

I think the failure you are seeing is because the suricata build is too big.

Ok, is there a way to fix it ? (besides building less fuzzers)

[jonathanmetzman]

I think the failure you are seeing is because the suricata build is too big.

Ok, is there a way to fix it ? (besides building less fuzzers)

I'm looking into whether we can give individual projects more disk

[guidovranken]

When trying to build some projects with the pinned base builder (gcr.io/oss-fuzz-base/base-builder@sha256:19782f7fe8092843368894dbc471ce9b30dd6a2813946071a36e8b05f5b1e27e) I get these download errors:

Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/d/distro-info-data/distro-info-data_0.43ubuntu1.15_all.deb  404  Not Found [IP: 185.125.190.39 80]

When I try to use the current gcr.io/oss-fuzz-base/base-builder (without the sha256 hash) I'm getting a lot of linking errors, that don't make sense to me. Try building the nettle project for example with FROM gcr.io/oss-fuzz-base/base-builder (e.g. the new version). The libgmp build fails with:

`ctz_table' referenced in section `.text' of mpn/.libs/gcd_11.o: defined in discarded section `.rodata.foo[foo]' of mpn/.libs/gcd_11.o
`ctz_table' referenced in section `.text' of mpn/.libs/gcd_22.o: defined in discarded section `.rodata.foo[foo]' of mpn/.libs/gcd_22.o

But curiously if you remove this line:

oss-fuzz/projects/nettle/build.sh

Line 44 in 204d784

autoreconf -ivf

It works. But then Nettle still fails to build with:

/usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_aes192_encrypt_aesni'
/usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_poly1305_set_key'
/usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_salsa20_2core'
/usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_poly1305_blocks'
/usr/bin/ld: ../libnettle.so: undefined reference to `nettle_serpent_decrypt'
/usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_memxor_sse2'
/usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_umac_nh_n'
/usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_camellia_crypt'
etc etc

Then if I add --disable-shared to Nettle's ./configure, I get:

/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x25
../libnettle.a(fat-x86_64.o): in function `fat_init':
fat-x86_64.c:(.text.fat_init[fat_init]+0x42c): undefined reference to `_nettle_cbc_aes256_encrypt_aesni'
/usr/bin/ld: fat-x86_64.c:(.text.fat_init[fat_init]+0x433): undefined reference to `_nettle_cbc_aes192_encrypt_aesni'
/usr/bin/ld: fat-x86_64.c:(.text.fat_init[fat_init]+0x43a): undefined reference to `_nettle_cbc_aes128_encrypt_aesni'
/usr/bin/ld: fat-x86_64.c:(.text.fat_init[fat_init]+0x441): undefined reference to `_nettle_aes256_decrypt_aesni'
/usr/bin/ld: fat-x86_64.c:(.text.fat_init[fat_init]+0x448): undefined reference to `_nettle_aes256_encrypt_aesni'
/usr/bin/ld: fat-x86_64.c:(.text.fat_init[fat_init]+0x44f): undefined reference to `_nettle_aes192_decrypt_aesni'
etc etc

It seems related to sanitizer usage, build succeeds with -fsanitize=fuzzer-no-link but not with -fsanitize=address.

It's all very counter-intuitive. I've already spent hours on this to no avail and I don't really have that much time for this at the moment. Can someone please make the old version work again, or make the new version sane?

[jonathanmetzman]

According to https://oss-fuzz-build-logs.storage.googleapis.com/index.html#nettle the last successful build was on May 3rd,
this PR landed on April 29th. Are you sure the breakage is related?

[guidovranken]

According to https://oss-fuzz-build-logs.storage.googleapis.com/index.html#nettle the last successful build was on May 3rd, this PR landed on April 29th. Are you sure the breakage is related?

It started failing on the 4th with

Step #1:   404  Not Found [IP: 91.189.91.83 80]
Step #1: Fetched 33.1 MB in 1s (24.6 MB/s)
Step #1: �[91mE: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/d/distro-info-data/distro-info-data_0.43ubuntu1.15_all.deb  404  Not Found [IP: 91.189.91.83 80]

I don't know what that is about, but if that could be fixed, it would fix all my projects.

[jonathanmetzman]

Ah I think I know the issue. One sec

[jonathanmetzman]

#11943