[OSRM] Add new binary, second try... #8893

jeremiahpslewis · 2024-06-14T08:03:58Z

No description provided.

jeremiahpslewis · 2024-06-14T10:17:37Z

Error on x86-64-linux-gnu-cxx11:

[ Info: Checking shared library lib/libosrm.so
--
ERROR: could not load library "/cache/build/yggy-amdci7-8/julialang/yggdrasil/O/OSRM/build/x86_64-linux-gnu-cxx11/ZsPx5jB0/x86_64-linux-gnu-libgfortran5-cxx11/destdir/lib/libosrm.so"
  | /cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.7/julia-1.7-latest-linux-x86_64/bin/../lib/julia/libstdc++.so.6: version `GLIBCXX_3.4.31' not found (required by /cache/build/yggy-amdci7-8/julialang/yggdrasil/O/OSRM/build/x86_64-linux-gnu-cxx11/ZsPx5jB0/x86_64-linux-gnu-libgfortran5-cxx11/destdir/lib/libosrm.so)
  | ┌ Warning: lib/libosrm.so cannot be dlopen()'ed

jeremiahpslewis · 2024-06-14T10:19:28Z

Error on *-apple-* builds:

ld64.lld: error: undefined symbol: osrm::extractor::CompressedEdgeContainer::GetBucketReference(unsigned int) const

imciner2 · 2024-06-14T10:19:54Z

Error on x86-64-linux-gnu-cxx11

That is because it is using GCC 13, so it gets linked against a newer glibcxx version because that is what ships with it. You need to lower the GCC version requested to get around this.

jeremiahpslewis · 2024-06-15T08:28:30Z

Error on x86-64-linux-gnu-cxx11

That is because it is using GCC 13, so it gets linked against a newer glibcxx version because that is what ships with it. You need to lower the GCC version requested to get around this.

If I lower GCC below v12, other target builds break. Is there another strategy for dealing with this issue (I don't exactly understand what the problem is...all of the dependencies were built with earlier GCC versions).

jeremiahpslewis · 2024-06-15T20:53:28Z

Hey @mattwigway!

you have write access to this branch and should feel free to modify it as necessary.
I added your gcc modification to my CMakeList.txt patch
Using clang for all builds seems strictly better
The remaining errors which I encounter don't seem related to boost versions. They are as follows:

-apple-

[20:49:20] ld64.lld: error: undefined symbol: osrm::extractor::CompressedEdgeContainer::GetBucketReference(unsigned int) const

-mingw-

warning "__STRICT_ANSI__ seems to have been undefined; this is not supported"�[39m

This is a problem, as the mingw symbols which we need to build are only available when __STRICT_ANSI__ is undefined. Until Ygg ships a more recent version of mingw, (maybe with the upcoming BinaryBuilder redo) it's probably best to just drop windows.

x86_64-linux-gnu-cxx11

<head></head>
[ Info: Checking shared library lib/libosrm_extract.so
--
  | ERROR: could not load library "/cache/build/yggy-amdci7-14/julialang/yggdrasil/O/OSRM/build/x86_64-linux-gnu-cxx11/Sjda2mna/x86_64-linux-gnu-libgfortran5-cxx11/destdir/lib/libosrm_extract.so"
  | /cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.7/julia-1.7-latest-linux-x86_64/bin/../lib/julia/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /cache/build/yggy-amdci7-14/julialang/yggdrasil/O/OSRM/build/x86_64-linux-gnu-cxx11/Sjda2mna/x86_64-linux-gnu-libgfortran5-cxx11/destdir/lib/libosrm_extract.so)
  | ┌ Warning: lib/libosrm_extract.so cannot be dlopen()'ed

~~- dlopen issues, which I wonder if we can solve by setting the appropriate binaries to dont_dlopen=true~~

Worth seeing whether we can get this thing to build with gcc10

giordano · 2024-06-15T21:00:03Z

dlopen issues, which I wonder if we can solve by setting the appropriate binaries to dont_dlopen=true

Ignoring an issue doesn't solve it. You simply can't build a library with a recent compiler and load it with an older runtime. There's no way around it: just don't do it.

jeremiahpslewis · 2024-06-15T21:03:49Z

dlopen issues, which I wonder if we can solve by setting the appropriate binaries to dont_dlopen=true

Ignoring an issue doesn't solve it. You simply can't build a library with a recent compiler and load it with an older runtime. There's no way around it: just don't do it.

Why do I get an older runtime when I set the build gcc to v12?

giordano · 2024-06-15T21:13:44Z

Why do I get an older runtime when I set the build gcc to v12?

The runtime is used....at run time, not during the build. It's the gcc runtime shipped with Julia v1.7, which was released months before GCC 12 was a thing. This is the whole point of the compatibility section in the CONTRIBUTING.md file.

mattwigway · 2024-06-15T22:58:28Z

So based on the contributing.md document, it seems like Clang doesn't have the same problem with runtime version incompatibilities that gcc does, so maybe that's a solution? Otherwise I'm okay making this only supported on more recent Julia, as it's a fairly specialized package.

giordano · 2024-06-15T23:06:19Z

it seems like Clang doesn't have the same problem with runtime version incompatibilities that gcc does, so maybe that's a solution?

Wrong, clang on Linux uses gcc runtime.

mattwigway · 2024-06-15T23:58:37Z

FWIW, here are the missing symbols in the Mac build, demangled (with llvm-cxxfilt):

tbb::detail::r1::initialize(tbb::detail::d1::task_group_context&)
tbb::detail::r1::allocate_memory(unsigned long)
tbb::detail::r1::set_end_of_input(tbb::detail::d1::base_filter&)
tbb::detail::r1::deallocate_memory(void*)
tbb::detail::r1::parallel_pipeline(tbb::detail::d1::task_group_context&, unsigned long, tbb::detail::d1::filter_node const&)
tbb::detail::r1::destroy(tbb::detail::d1::task_group_context&)
osrm::extractor::SuffixTable::isSuffix(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) const
osrm::extractor::RestrictionGraph::GetRestrictions(unsigned long long) const
osrm::extractor::WayRestrictionMap::IsRestricted(unsigned long long, unsigned int) const
osrm::extractor::WayRestrictionMap::IsViaWayEdge(unsigned int, unsigned int) const
osrm::extractor::WayRestrictionMap::GetRestrictions(unsigned long long, unsigned int) const
osrm::extractor::WayRestrictionMap::DuplicatedNodeIDs(unsigned int, unsigned int) const
osrm::extractor::CompressedEdgeContainer::HasEntryForID(unsigned int) const
osrm::extractor::CompressedEdgeContainer::GetBucketReference(unsigned int) const

I'm not sure why these are missing, all the osrm:: ones I've checked are defined in both .cpp and .h files. I don't have time tonight, but I'm going to try the build locally and look at the built artifacts to see if anything looks suspicious.

mattwigway · 2024-06-17T15:16:04Z

So, the Mac build failures do appear to be linker problems. All of the osrm::* missing symbols are in fact defined in compiled .o files. The tbb ones are not, but I wouldn't expect them to be—they should be in .so files (I haven't specifically looked for them there). I'm going to see if I can figure out why the linker isn't finding the symbols.

mattwigway · 2024-06-17T15:30:47Z

So, what these symbols have in common is that they're all symbols that come from other dynamic libraries, rather than from .o files being linked into the current dynamic library. So I think this is an rpath issue.

it is not supported with Julia 1.7 used during build, but it does work with julia 1.8+ specified in our compat entry.

mattwigway · 2024-06-18T16:15:19Z

Builds are now succeeding on macOS and Linux. We had to set dont_dlopen because OSRM can only be built with GCC 12+, and that's not compatible with the Julia 1.7 used in CI. I confirmed that the built artifacts work on Julia 1.8, and julia_compat is set to 1.8. Perhaps in a future BinaryBuilder the dlopen() test could happen with the julia version specified in compat—I'm happy to take a look at that if desired.

There are a few more steps before merge:

We're using a significantly different build configuration than stock OSRM (single monolithic dylib, instead of five or six circularly-dependent dylibs), I'm waiting for upstream feedback on whether this will get mainlined (Link libosrm_guidance correctly and work around circular dependencies Project-OSRM/osrm-backend#6955)
To use OSRM.jl from Julia requires a C++ shim. I'm building packages locally and am going to confirm that the C++ shim works with the built packages.
I assume we probably want to squash all the commits to avoid polluting the main branch (maybe create a copy of this branch first for archival reasons)

jeremiahpslewis added 12 commits June 14, 2024 10:02

Add back OSRM

897c40d

try gcc 10

2887931

Try gcc 13

b9c0a37

Comment out mingw patch

5c2c9b8

Add c++20 compatible Mac sdk

9cb97a6

try later version of boost

9f00d18

Drop 32 bit systems

b051e8c

use clang for all builds

38a965b

Try boost 1.76 again

40a522c

set mingw flag according to onetbb

7780ba6

Revert to boost 1.79

40fa381

better accommodate mingw

ae75969

jeremiahpslewis added 10 commits June 14, 2024 11:34

Fix mingw patch

ed4073c

Add missing image line to devcontainer.json

86c71a7

Drop unnecessary part of patch

3ad0b11

update patch

95916f4

gcc 12 confirmed as lower bound

2142a39

Augment mingw patch

191a720

Better mingw patch

f325649

tweak mingw patch

fc080a6

Try earlier gcc

b2e4911

Revert to gcc12

84324c4

jeremiahpslewis added 4 commits June 15, 2024 08:50

Tweak mingw patch

985b35a

fix patch

673d301

update patch

9933996

attempt more extreme cmake patch

b1f4443

jeremiahpslewis added 3 commits June 15, 2024 20:30

Incorporate tweak from @mattwigway

d4b7e84

Try using gcc again...

6cd51b4

Revert back to clang

198f815

jeremiahpslewis added 2 commits June 15, 2024 22:54

Try clang lld fix

551b7ad

Drop windows for now

a6c629a

Add zlib dependency

9a07122

jeremiahpslewis added 4 commits June 15, 2024 21:14

Fix zlib?

f9c584e

Go back to gcc 10

c927929

Try julia compat = 1.8 and gcc 12

74e1ec3

Tweak params

aedc225

jeremiahpslewis and others added 10 commits June 17, 2024 22:13

Add osmium as dependency

4ed8af5

Try boost 1.76 for libosmium compat

09d6ac1

[OSRM] use libosmium dependency

0816ba0

[OSRM] fix mac build

cc089ef

Cleanup script

1e72189

Drop libosmium so we can use boost 1.79

87e70aa

update guidance build patch so that osrm-extract binary works

ad333e2

[OSRM] try monolithic libosrm fork

381a2ca

[OSRM] set libraryproducts correctly for monolithic libosrm

bf842b6

[OSRM] don't dlopen libosrm.so

44fdc5a

it is not supported with Julia 1.7 used during build, but it does work with julia 1.8+ specified in our compat entry.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[OSRM] Add new binary, second try... #8893

[OSRM] Add new binary, second try... #8893

jeremiahpslewis commented Jun 14, 2024

jeremiahpslewis commented Jun 14, 2024

jeremiahpslewis commented Jun 14, 2024

imciner2 commented Jun 14, 2024

jeremiahpslewis commented Jun 15, 2024

jeremiahpslewis commented Jun 15, 2024 •

edited

Loading

giordano commented Jun 15, 2024

jeremiahpslewis commented Jun 15, 2024

giordano commented Jun 15, 2024

mattwigway commented Jun 15, 2024

giordano commented Jun 15, 2024

mattwigway commented Jun 15, 2024

mattwigway commented Jun 17, 2024

mattwigway commented Jun 17, 2024

mattwigway commented Jun 18, 2024 •

edited

Loading

[OSRM] Add new binary, second try... #8893

Are you sure you want to change the base?

[OSRM] Add new binary, second try... #8893

Conversation

jeremiahpslewis commented Jun 14, 2024

jeremiahpslewis commented Jun 14, 2024

jeremiahpslewis commented Jun 14, 2024

imciner2 commented Jun 14, 2024

jeremiahpslewis commented Jun 15, 2024

jeremiahpslewis commented Jun 15, 2024 • edited Loading

giordano commented Jun 15, 2024

jeremiahpslewis commented Jun 15, 2024

giordano commented Jun 15, 2024

mattwigway commented Jun 15, 2024

giordano commented Jun 15, 2024

mattwigway commented Jun 15, 2024

mattwigway commented Jun 17, 2024

mattwigway commented Jun 17, 2024

mattwigway commented Jun 18, 2024 • edited Loading

jeremiahpslewis commented Jun 15, 2024 •

edited

Loading

mattwigway commented Jun 18, 2024 •

edited

Loading