Repository navigation
CRC of objects with references is not comparable across separate databases #74
Description
Summary
objects.crc32 is meant to be a content fingerprint, and the comparing-builds workflow expects you can analyze two builds into two separate databases and diff CRCs to find which objects changed. That works for leaf assets (Texture2D/Mesh/AudioClip — no references), but it is broken for any object that contains references (Materials, prefabs/GameObjects, MonoBehaviours, etc.): identical content produces different CRCs in two separate analyze runs.
Cause
When PPtrAndCrcProcessor.ExtractPPtr folds a reference into the CRC, it uses the resolved analyzer/database object id returned by the callback, not the PPtr's own identity:
var refId = m_Callback(m_ObjectId, fileId, pathId, ...); // analyzer db id
m_Crc32 = Crc32Algorithm.Append(m_Crc32, <refId bytes>);That id comes from ObjectIdProvider.GetId((m_LocalToDbFileId[fileId], pathId)), and both the serialized-file id and the object id are assigned sequentially per analyze run. So the same logical object gets different ids in db1 vs db2 → different CRC for identical content → cross-database comparison reports spurious differences for every object that has references.
Why we can't just hash the raw PPtr (the tradeoff)
The obvious fix is to hash the raw on-disk PPtr (fileId + pathId) instead of the resolved id. But the resolved id is currently what makes within-database duplicate detection (view_potential_duplicates) work across bundles: two copies of the same object in different bundles reference the same target, and resolving through m_LocalToDbFileId (keyed by filename) + pathId normalizes them to the same id → same CRC → detected as duplicates.
fileId is a local index into a serialized file's external-reference list, so two copies of an object in different bundles can have different fileId values for the same target. Hashing the raw PPtr would therefore weaken duplicate detection. Deduplication is an important feature and is probably not well covered by tests yet, so we don't want to risk regressing it.
Options to evaluate
- Raw PPtr (
fileId+pathId) — simplest; fixes cross-db comparison in the common case; risks weakeningview_potential_duplicates(localfileIddiffers between bundles). - Stable target identity +
pathId— resolvefileIdto a stable identifier for the target file and hash that +pathId, so it is independent of the local index. This fixes cross-db comparison AND preserves cross-bundle duplicate detection, but the "stable identifier" differs by source:- Build output external references carry a path (e.g.
archive:/CAB-...), not a GUID. - Editor / Library references carry a GUID (the source asset's GUID).
So the CRC needs to mix in whichever ofExternalReference.Path/ExternalReference.Guidis populated (and a fixed marker for local refs,fileId == 0). Relies on those fields being present and stable.
More code: thread the external-reference info fromsf.ExternalReferencesinto the CRC.
- Build output external references carry a path (e.g.
- Status quo — cross-db comparison stays broken for referenced objects.
Prerequisite
Add test coverage for view_potential_duplicates / cross-bundle deduplication before changing the CRC, so a fix can be validated to not regress it.
Context
Discovered while reviewing #73 / #70. Note that this is independent of the CRC changes made there (the ManagedReferenceData size fix, the ComputeCRC chunking fix, and the cah:/ stream hashing) — those also change CRC values vs. older tool versions, so CRCs are not comparable across tool versions regardless.
Related: #44 (refs table).
Activity
[#70] Note CRC is a within-database fingerprint (ref #74)
SkowronskiAndrew commentedon Sep 16, 2026
AGENT: Some findings from the Unity 6.7 work in #145 that bear on this, plus a fourth option that came up while discussing it.
What the CRC actually covers today
Worth writing down precisely, because "content fingerprint" suggests something simpler than what
PPtrAndCrcProcessorcomputes. It is not a hash of the object's bytes. Walking the TypeTree, it appends:AppendCrc(m_Offset, node.Size))StreamingInfo/StreamedResource— the referenced stream's bytes, read from the companion.resS/.resource, or thecah:/<hash>path itself for content-addressed streamsand it never appends:
m_Offset = (m_Offset + 3) & ~3with noAppendCrc, so those bytes are skipped. Normally zeros, but nothing guarantees it.So two objects can have equal CRCs while differing in bytes, and vice versa. That matters for the options below.
What #145 changed
From SerializedFile v25 the
[SerializeReference]registry is a self-delimiting frame in the object's data that no TypeTree node describes. #145 CRCs the frame's header and tables as the raw bytes they are, then walks each instance's blob through its own TypeTree with the usual PPtr substitution.Two consequences:
ProcessManagedReferenceDataappends each of the three FQN strings aslength + charactersand skips the alignment padding after each. In v3 the whole header-and-tables range goes in as one contiguous span, padding included. Both are "the registry's raw bytes", but not byte-identical treatment.Neither changes the analysis in the issue body. The PPtr substitution is untouched, so cross-database comparison is broken exactly as before.
Option 4: CRC the object's raw bytes
Raised by Andrew on #145: instead of deriving a CRC from the walk, hash the object's byte range directly (
obj.Offset .. obj.Offset + obj.Size), plus any external stream content it references.Arguments for:
crc32already is.Arguments against, and one correction to an assumption:
m_FileID, which is a local index into that serialized file's external-reference list. Two copies of the same object in different bundles can carry differentfileIdvalues for the same target, so their raw bytes differ andview_potential_duplicatesstops seeing them as duplicates. This is the same tradeoff the issue already identifies for option 1, so the prerequisite (test coverage for dedup first) applies equally.So option 4 is best read as option 1 plus padding, minus the walk-derived-ness — the same duplicate-detection risk, with extra robustness and simplicity in exchange. It is not a way around the tradeoff.
A wrinkle for option 2
Option 2 wants a stable identity for the target of a reference. Note that from v26 the registry frame carries the fully qualified type name (class / namespace / assembly) of every
[SerializeReference]instance inline, in its Types table, and those bytes are already in the CRC. That is a stable, build-independent identity, so for registry-held instances the "stable identifier" problem is already solved by the format — the open question is only about PPtrs to other serialized objects.Suggestion
The prerequisite in the issue body still looks like the right first move, and #145 does not change it: add coverage for
view_potential_duplicates/ cross-bundle deduplication before touching the CRC. Until that exists, options 1, 2 and 4 all carry the same unmeasured risk, and there is no way to tell which of them actually costs us something.One more thing worth deciding explicitly when this is picked up: whether
crc32should be one column trying to serve both "did this object change between builds" and "are these two objects the same content", since those two questions pull in opposite directions on exactly the PPtr treatment. Two columns would let each be answered correctly.SkowronskiAndrew commentedon Sep 29, 2026
This ticket has a lot of AI mumble jumble on it. Its still not clear to me if there truly is a bug or not "in practice" when comparing builds.
And it seems the actual meaning of the CRC value is part of the confusion.
So it will be worth investigating this so that we can better reenforce our support for comparing builds and quickly narrowing down which objects have changed