Cannot seem to provoke genuinely necessary overlinking warning

[h-vetinari]

Overlinking and overdepending are opposites of each other (I'd be the first to agree that the naming isn't great BTW). In short: overlinking is when the package is linking to a library that is not found in a direct runtime dependency; overdepending is if you have runtime dependencies that aren't actually being linked by anything (obviously there's a good amount of stuff that causes false positives here)

I was actually writing some docs related to this and realized that I can't seem to manage to get rattler-build to actually produce an overlinking warning for lodepng corresponding to conda-build's

   INFO: sysroot: '/home/conda/feedstock_root/build_artifacts/mujoco_1749065357629/_build_env/x86_64-conda-linux-gnu/sysroot/' files: '['usr/share/zoneinfo', 'usr/share/locale/zh_TW/LC_MESSAGES/libc.mo', 'usr/share/locale/zh_CN/LC_MESSAGES/libc.mo', 'usr/share/locale/vi/LC_MESSAGES/libc.mo']'
   INFO (mujoco-simulate,bin/mujoco-simulate): Needed DSO lib/libmujoco.so.3.3.2 found in local/linux-64::libmujoco==3.3.2=hebd7970_1
WARNING (mujoco-simulate,bin/mujoco-simulate): Needed DSO lib/liblodepng.so found in ['conda-forge/linux-64::lodepng==20220109=h924138e_0']
WARNING (mujoco-simulate,bin/mujoco-simulate): .. but ['conda-forge/linux-64::lodepng==20220109=h924138e_0'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
   INFO (mujoco-simulate,bin/mujoco-simulate): Needed DSO lib/libglfw.so.3 found in conda-forge/linux-64::glfw==3.4=hd590300_0
   INFO (mujoco-simulate,bin/mujoco-simulate): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/libpthread.so.0 found in CDT/compiler package conda-forge/noarch::sysroot_linux-64==2.17=h0157908_18
   INFO (mujoco-simulate,bin/mujoco-simulate): Needed DSO lib/libstdc++.so.6 found in conda-forge/linux-64::libstdcxx==15.1.0=h8f9b012_2
   INFO (mujoco-simulate,bin/mujoco-simulate): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/libm.so.6 found in CDT/compiler package conda-forge/noarch::sysroot_linux-64==2.17=h0157908_18
   INFO (mujoco-simulate,bin/mujoco-simulate): Needed DSO lib/libgcc_s.so.1 found in conda-forge/linux-64::libgcc==15.1.0=h767d61c_2
   INFO (mujoco-simulate,bin/mujoco-simulate): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/libc.so.6 found in CDT/compiler package conda-forge/noarch::sysroot_linux-64==2.17=h0157908_18
WARNING (mujoco-simulate): dso library package conda-forge/linux-64::libgl==1.7.0=ha4b6fd6_2 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)

Even if I purposefully ignore the liblodepng run-export (so it's present in host: to build, but not present in run:, which should lead to a warning), rattler-build gives me a pristine-looking

 │ │ [bin/mujoco-simulate] links against:
 │ │  ├─ lib/liblodepng.so (lodepng)
 │ │  ├─ lib/libmujoco.so.3.3.6 (libmujoco)
 │ │  ├─ lib/libgcc_s.so.1 (libgcc)
 │ │  ├─ lib/libstdc++.so.6.0.34 (libstdcxx)
 │ │  ├─ libc.so.6 (system)
 │ │  ├─ libpthread.so.0 (system)
 │ │  ├─ lib/libglfw.so.3.4 (glfw)
 │ │  └─ libm.so.6 (system)
 │ │  ⚠ warning Overdepending against libgl

Originally posted by @h-vetinari in #1949

[wolfv]

I am actually curious: do you agree that a missing dependency is more likely than "overlinking"? I mean, who writes their build script / makefile by hand?

I think the wording should be different and we shoudl say something "libfoo.so is linking to liblodepng, but it's not in the run requirements".

[h-vetinari]

do you agree that a missing dependency is more likely than "overlinking"?

That distinction is pretty academic? I don't like the naming in the first place, but if the build is linking to a library that would be in a dependency that is not present in run, it is overlinking (linking to too many things compared to its declared deps) as well as a missing dependency (compared to what it's linking to)?

AFAIU, it's two sides of the same coin. Or can you explain the difference between the two?

[wolfv]

Overlinking implies that the user "manually" linked to too many libraries. I don't think a user does that often. It would be more often that a dependency is forgotten

[h-vetinari]

I don't understand what you mean by "manually linked too many libraries". I also don't see how it's relevant whether the linkage was created by a tool or by a human. If the binary requires symbols from that linkage (and we can mostly detect whether that's the case or not), then that library needs to be present at runtime.

Said differently, what do we gain by distinguishing the two cases? They seem identical to me.

[wolfv]

Yeah they are identical but overlinking makes it sound like you'd do "-lfoobar" yourself

[h-vetinari]

I see "overlinking" as a term of art that describes "this binary links to libraries that are not present in the direct dependencies of this package". Who added -lfoobar is irrelevant IMO¹, and likewise it doesn't matter how exactly the dependency was added. What matters is that both sides of the ledger (link surface & content of declared deps) need to match, and IMO we shouldn't try to ascribe a mismatch as being the fault of one side or the other.

Now, I'll be the first to admit that it took me a very long time to arrive at this interpretation (and I did not understand overlinking vs. overdepending at all for many years), so perhaps there's opportunities for better naming here. But that should probably be a separate discussion?

Footnotes

1. especially since it will be ignored if the binary doesn't actually link anything from libfoobar ↩

[JeanChristopheMorinPerso]

I see "overlinking" as a term of art that describes "this binary links to libraries that are not present in the direct dependencies of this package". Who added -lfoobar is irrelevant IMO1, and likewise it doesn't matter how exactly the dependency was added. What matters is that both sides of the ledger (link surface & content of declared deps) need to match, and IMO we shouldn't try to ascribe the mistake to one side or the other.

I agree with this statement. It doesn't matter who added the linker flag. It could be added by the project being packaged, by a packager, by a transitive dependency (via cmake config files or pkgconfig files, etc).

As for the names, they are not great, but if you search online, you'll see that there are other ecosystems that use the same terms for exactly the same thing as in the conda ecosystem. And they are the type of thing that doesn't make sense if someone don't know or understand the concepts, but it makes sense if they understand the concept. Changing the names might actually cause more confusion.

[wolfv]

Fixed!

[h-vetinari]

Great, thanks a lot! :)