Cargo package says the repo is dirty, but the files are gitignored

[scullionw]

Problem

I have a folder called "extra" that is in my .gitignore. git status confirms it

Cargo package says the repo is dirty, but those files are gitignored.

Note: In my Cargo.toml I am adding this extra folder to the include field, because i want this folder present in the package (but not in git).

Steps

1. Add a folder with some data to the .gitignore
2. Include that folder in the cargo package using the include field in the Cargo.toml

Possible Solution(s)

No response

Notes

Normally I worked around the bug with "--allow-dirty", but now that i've switched to cargo release to publish my workspace, I cannot (it doesn't support that flag)

Version

cargo version --verbose
cargo 1.70.0 (ec8a8a0ca 2023-04-25)
release: 1.70.0
commit-hash: ec8a8a0cabb0e0cadef58902470f6c7ee7868bdc
commit-date: 2023-04-25
host: aarch64-apple-darwin
libgit2: 1.6.3 (sys:0.17.0 vendored)
libcurl: 7.87.0 (sys:0.4.61+curl-8.0.1 system ssl:(SecureTransport) LibreSSL/3.3.6)
os: Mac OS 13.3.0 [64-bit]

[weihanglo]

Thanks for filing this report, and sorry for the inconvenience.

The function check_repo_state() is how Cargo checks VCS (Git) status to determine whether the repo is dirty. Roughly it compares src files, which are what you list in include directive, to git status/untracked/ignored files. The implementation was done in #9478. I wasn't there but so far as I can tell, it is intentional to report “dirty” when a file is gitignore'd but still included in include. Here is a test verifying this behavior:

cargo/tests/testsuite/package.rs

Lines 1080 to 1100 in 26b7725

	#[cargo_test]
	fn dirty_ignored() {
	// Cargo warns about an ignored file that will be published.
	let (p, repo) = git::new_repo("foo", |p| {
	p.file(
	"Cargo.toml",
	r#"
	[package]
	name = "foo"
	version = "0.1.0"
	description = "foo"
	license = "foo"
	documentation = "foo"
	include = ["src", "build"]
	"#,
	)
	.file("src/lib.rs", "")
	.file(".gitignore", "build")
	});
	// Example of adding a file that is confusingly ignored by an overzealous
	// gitignore rule.

I guess the intention was making files being packaged and file tracked by Git in sync. Despite that, this “smart” behaivor does break tools like cargo-release and maybe cargo-smart-release as well.

@epage, @Byron, as you're the maintainers of those tools. Do you have any thought on this behavior? How to make cargo and releases tools inline with each others?

[Byron]

The behaviour of cargo publish here is desired, even though it might lack refinement for this case: a file is ignored by git and not tracked by git yet - in this case it could probably just be added to the package anyway. Maybe there is a test for that as well and I didn't see it. Maybe this is something that could be improved when porting this area over to gitoxide, then I should have a better understanding of the tests as well and how it should work in detail.

Speaking only for cargo-smart-release, it doesn't try to re-implement any ignore logic but relies completely on cargo package - doing a git status check is merely for convenience, and for that it only relies on git itself without extra logic so I think it's fine in this case.

[scullionw]

I see!

I guess the intention was making files being packaged and file tracked by Git in sync

I was under the impression (can't find the docs where I read this) that there was no guarantee that a cargo package matches a git repo 1-1. I feel that is also implied by the include/exclude?

For reference I am including some build artifacts for a couple targets that speed up compilation (the crate implements cxx bindings to a c++ library). The crate is not usable otherwise (10sec vs 20min compilation).. This is in a private registry.

I understand this is an edge case though.

In case someone else runs into this: another workaround I am using now is when publishing in CI, I make a dummy commit that I don't push just before the publish.

Anyways thanks for looking into this!

[weihanglo]

#13592 is now merged. We have file listing support in gitoxide. I wonder if that is a step toward fixing this with gitoxide
cc @Byron

[Byron]

Would you have a rustup-based cargo invocation handy that invokes a dry-run publish with gitoxide for file listing? I would have left it here myself, but seem to have lost my old invocations. Thanks in advance.

[gaykitty]

I am experiencing this problem too, but the files are not in Cargo.toml's include. No matter what I do cargo thinks the files in .direnv are not commit despite that directory being ignored.

[Byron]

My guess would be that the .gitignore file contains something like this:

.direnv/

And it might work if it looks like this:

.direnv/
.direnv

Does that work for you?

[gaykitty]

No, cargo package --list still complains with either variation.