Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Executables from k3s maybe be incorrectly flagged as malware #9738

Closed
benklett opened this issue Mar 14, 2024 · 19 comments
Closed

Executables from k3s maybe be incorrectly flagged as malware #9738

benklett opened this issue Mar 14, 2024 · 19 comments

Comments

@benklett
Copy link

benklett commented Mar 14, 2024

Environmental Info:
K3s Version:

$ k3s -v
k3s version v1.29.2+k3s1 (86f10213)
go version go1.21.7

affected are the channels from v1.23 until v1.29

Node(s) CPU architecture, OS, and Version:

$ uname -a
Linux ben-Virtual-Machine 5.15.0-27-generic #28-Ubuntu SMP Thu Apr 14 04:55:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

single node cluster

Describe the bug:
A lot of files in /var/lib/rancher/k3s/data/current/bin/ get flagged by various Antvirus Software, e. g. ipset:

https://www.virustotal.com/gui/file/b52d5de7999a5b5b08ecc0bad539f99263b00bc6129aeb7327ead0e8f8b98bcd

SHA256: b52d5de7999a5b5b08ecc0bad539f99263b00bc6129aeb7327ead0e8f8b98bcd

This is the same sha for every current version in the channels v1.23 until latest.

The ipset file from version v1.22.17+k3s1:

https://www.virustotal.com/gui/file/618d9c1952ecfc13ce4358afec73e9283b7c24f968208656ad6fe2b3c7b3f8ef
SHA256: 618d9c1952ecfc13ce4358afec73e9283b7c24f968208656ad6fe2b3c7b3f8ef

List of files matched by F-Secure:

blkid
coreutils
aux/ebtablesd
aux/ebtables-legacy
find
fuse-overlayfs
ip
ipset
losetup
nsenter
pigz
slirp4netns
aux/xtables-legacy-multi
aux/xtables-nft-multi

Steps To Reproduce:

Expected behavior:
No results on virustotal.com
image

Actual behavior:
Some Hits on virustotal.com
image

Additional context / logs:

@thejan2009
Copy link

These packages are sourced from k3s-root, the same ipset SHA can be found there in the last two releases https://github.com/k3s-io/k3s-root/releases/download/v0.12.2/k3s-root-amd64.tar and https://github.com/k3s-io/k3s-root/releases/download/v0.13.0/k3s-root-amd64.tar

@benklett
Copy link
Author

These packages are sourced from k3s-root, the same ipset SHA can be found there in the last two releases https://github.com/k3s-io/k3s-root/releases/download/v0.12.2/k3s-root-amd64.tar and https://github.com/k3s-io/k3s-root/releases/download/v0.13.0/k3s-root-amd64.tar

Thank you for the hint. I have tried to build my own binaries with this repo, but the resulting ipset< binary has the exact same SHA hash.

k3s-root seems to use https://github.com/buildroot/buildroot to build the required binaries.

@cwayne18
Copy link
Member

This is a false positive, for some reason it happens sometimes on the ML-driven VirusTotal scanners. We will ensure they are properly marked as safe, thank you for the heads up!

@cwayne18
Copy link
Member

@benklett
Copy link
Author

benklett commented Mar 14, 2024

That is the older file version, which was not affected.

The current version is still affected:
https://www.virustotal.com/gui/file/b52d5de7999a5b5b08ecc0bad539f99263b00bc6129aeb7327ead0e8f8b98bcd

@benklett
Copy link
Author

blkid
coreutils
aux/ebtablesd
aux/ebtables-legacy
find
fuse-overlayfs
ip
ipset
losetup
nsenter
pigz
slirp4netns
aux/xtables-legacy-multi
aux/xtables-nft-multi

@cwayne18 Can you tell me if you already submitted the files as false positive or do I have to do that?

@cwayne18
Copy link
Member

Our security team has already submitted them, thank you for offering!

@cippaciong
Copy link

Hello @cwayne18, sorry for asking the same question as @benklett, it's just to be 100% sure I understood correctly.

We have a similar report for the pigz file included in K3s (https://www.virustotal.com/gui/file/d7ac92d1df715d2cb5ffa7a8a49b12be738e6e5808e2d8bcadadd799aaad2d8e/detection), do you confirm that this and all other files affected have been submitted as false positives to VirusTotal?

Thanks

@mjrist
Copy link

mjrist commented Mar 20, 2024

@cwayne18 we are also investigating this. You claim that the security team has submitted these as false positives to VirusTotal. Any idea on the timeline here? When will these no longer be flagged?

@rjshrjndrn
Copy link

Can we keep this issue open till the scanners clear the flag? And, anybody who stumbles upon this issue will have visibility.

@brandond
Copy link
Member

The false positives have been reported. There is no additional work to be done on our side at this time, and no timeline for when these low-quality scanners will address their issues.

@mjrist
Copy link

mjrist commented Mar 21, 2024

The false positives have been reported. There is no additional work to be done on our side at this time, and no timeline for when these low-quality scanners will address their issues.

Hey @brandond, many of us have customers who are inquiring into this. So having some visibility on what is happening here would be very helpful.

By the way, if I compile these executables (e.g. pigz) from source the scanners don't complain. My assumption is that they don't like something about the way they are being sourced from buildroot.

Is there an FAQ/known issues documentation that notes on this could be added to? Open to other ideas. In essence just looking for something I can point customers to: "This is not an issue. Here is why."

Thanks.

@brandond
Copy link
Member

brandond commented Mar 21, 2024

Unfortunately we don't have any visibility into these ML-based scanners either. We report issues when they pop up, they go away after a period of time. These vendors have no SLA on their free services, and offer no assistance other than some vague warnings about false positives in their ToS. It seems to happen about once a year or so they decide some random thing (usually busybox) is a virus or hacking tool, and we have to go through it again.

Feel free to point your users at this issue for context.

@benklett
Copy link
Author

I have just read, that somehow signing could help with the issue. Would it be possible to always sign the released binaries to not let this happen again?

@cwayne18
Copy link
Member

Signing the binaries wouldn't likely solve this issue, it's a bit out of our hands at the moment but we have reached out to Virustotal

@maxksec
Copy link

maxksec commented Apr 4, 2024

xtables-nft-multi is still flaged by virus total :(

k3s version v1.29.3+k3s1 (8aecc26b)
/var/lib/rancher/k3s/data/3fcd4fcf3ae2ba4d577d4ee08ad7092538cd7a7f0da701efa2a8807d44a25f66/bin/aux$ sha256sum xtables-nft-multi
60efc5a86f68103c5cf263ba61bdd2a468f93685a947e987f018cac8b95497f2  xtables-nft-multi

https://www.virustotal.com/gui/file/60efc5a86f68103c5cf263ba61bdd2a468f93685a947e987f018cac8b95497f2?nocache=1

Screenshot 2024-04-04 at 13 26 53

@uniit
Copy link

uniit commented Apr 16, 2024

Hi there,

k3s --version
k3s version v1.27.5+k3s1 (8d074ec)
go version go1.20.7

We received a notification from rapid7 about xtables-legacy-multi:

  "process": {
...
    "name": "xtables-legacy-multi",
...
    "exe_path": "/var/lib/rancher/k3s/data/b2c9b831d16a6f5e7d0224148df7be9b67491fc0f36e7a9a58e1e5b0c42b7258/bin/aux/xtables-legacy-multi",
    "cmd_line": "/var/lib/rancher/k3s/data/b2c9b831d16a6f5e7d0224148df7be9b67491fc0f36e7a9a58e1e5b0c42b7258/bin/aux/xtables-legacy-multi --help",
...
    "hash_reputation": {
      "reputation": "Malicious",
      "threat_level": "Critical",
      "reliability": "Low",
...
      "classification": {
        "family_name": "LuckyMouse",
        "type": "Trojan",
        "platform": "Linux"
      }
...

https://www.virustotal.com/gui/file/8cc834717af150b879f9cb9892236fd0fd4695d54bc50fc82a8d31f66a1bfa84

@brandond
Copy link
Member

Still a false positive. Report it to your AV vendor and get them to fix it.

@burlyunixguy
Copy link

Regarding: https://www.virustotal.com/gui/file/b52d5de7999a5b5b08ecc0bad539f99263b00bc6129aeb7327ead0e8f8b98bcd/detection

We finally got a response from our vendor that confirmed ipset (b52d5de7999a5b5b08ecc0bad539f99263b00bc6129aeb7327ead0e8f8b98bcd) is a false positive. VirusTotal still has non-zero community score, but at least we can move on.

@brandond brandond pinned this issue Jun 7, 2024
@brandond brandond changed the title Executables from k3s get flagged as malware by Antvirus F-Secure and on virustotal.com Executables from k3s maybe be incorrectly flagged as malware Jun 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

No branches or pull requests

10 participants