Standardize Blobentry

[Affie]

Blobentry Standardization

Field Comparison Table

	Old Field	New Field	Reason for Change
✓	× id::Union{UUID, Nothing}	(removed)	REMOVE - label serves as key within parent struct
✓	✎ blobId::UUID	blobid::UUID	Standard use of capitals
✓	label::Symbol	label::Symbol	No change needed
✓	blobstore::Symbol	blobstore::Symbol	No change needed - matches Blobstore struct naming
✓	✎ hash::String		Replaced with separate fields for CRC and SHA hashes
✓		crchash::Union{UInt32, Nothing}
✓		shahash::Union{Vector{UInt8}, Nothing}
✓	origin::String	origin::String	see discussion
✓	✎ size::String	size::Int64	Type safety
✓	description::String	description::String	No change needed
✓	✎ mimeType::String	mimetype::MIME	Use built-in MIME type and standard use of capitals
☐	✎ metadata::String	metadata::JSONText	see #1105 and #1172
☐	✎ timestamp::ZonedDateTime	✎ timestamp::NanoDate	TODO
✓	× createdTimestamp	(removed)	REMOVE - see discussion
✓	× lastUpdatedTimestamp	(removed)	REMOVE - see discussion
✓	✎ _version::VersionNumber	version::VersionNumber	Remove underscore prefix #1177

Key Improvements

1. Hash Standardization: Split hash into two standardized fields - crcHash for fast error detection and shaHash for strong integrity
2. Type Safety: Use Int64 for size instead of string representation and MIME for mime types (needs StructUtils#16)
3. Metadata Structure: Replace base64 encoded strings with JDONText
4. Remove DB Implementation Details: Remove id field since label serves as the key within parent structures
5. Remove Unused Timestamps: Remove createdTimestamp and lastUpdatedTimestamp to avoid breaking serialization when we decide to use and finalise. (so not DFGv1 but v1.x)
6. GraphQL Compatibility: Structure fields for easy GraphQL querying and JSON serialization

[Affie]

size::Int64 - was a String due to a serialization issue with 64-bit. We can try using the new JSONv1 features to still serialize to a String but keep the correct type.

[dehann]

I'm assuming JSON.jl with new v1 PR automatically does the necessary struct conversions to support things like ::MIME, from raw jsonstrings coming in from the wild.

I'll first post all the ones I'm fine with so we can lock those down asap:

• removing id
• blobId::UUID no change
• label::Symbol no change
• blobStore::Symbol no change
• crcHash::String new
• shaHash::String new
• drop hash
• description::String no change like user scratch space for human readable notes
• mimeType::MIME fix type
• metadata::Dict{String,String}, happy after long discussion at All node metadata re-design #1105
• timestamp::ZonedDateTime no change
• version::VersionNumber fix type

[dehann]

Now for the issues:

• size::Int64/String not all JSON libraries supported Int64 and likely still the case, that is what motivated Jim to change to string (if I recall correctly). So great that Julia has Int64 support, but we need to think of general usage. I recall this error originally - when HTTP requests were made, the Int64 info was being truncated, so it became string as a workaround. These were likely from webservers, browsers, other language json libraries etc. The only way we can go to Int64 is to confirm that all json libraries out there can support more than Int32. What is for sure is that we need numbers in size that are larger than Int32 can hold. We might be forced into String regardless. So the task here is to find out if ECMA-404 is a viable compliance standard, but again -- my sense is it is risky to force Int64 given past experience.
• origin::String we already spoke about this on the phone, and my sense is @Affie has not used the field much but I almost always use it. I don't like flattening an mushing all data into metadata, because the MongoDB argument is everything is just a dict, so why structure data at all. That goes back to the fundamental argument about duck typing. If everything were just jammed in .metadata then you're forced to build a DB in a DB -- e.g. CRUD/ACID on .metadata is a reflection of why not just force an entire Blobentry to be immutable and use only the Bloblet approach instead. That way CRUD/ACID comes for free. So for me the "why not put it in metadata" is the polar opposite of "we must have a strict Schema". The motive for each of these fields need to be that they either are natively used to organize the data, or are used so often that it makes sense to have a dedicated field. All data has an origin in a distributed system. It worries me that complexities of origin of data conflict with CRUD/ACID on metadata and furthermore the risk increases if inconsistencies can develop on proliferation of blobentries (high degree of duplication as per design). So in my view origin::String field is MASSIVELY important -- regardless whether there is a string convention about what goes in .origin. E.g. might be "NvaPWA:user", "NvaSDK.rs/py/jl/js/c/cpp/java/...", "CustomApp007", etc.
• let me think more on the other timestamps, but if you want mutations on blobentries just for fields like .metadata, you will likely need at least two timetamp fields. The reason is blobentries are going to proliferate (as per design). Now someone mutates a blobentry somewhere. How is a user supposed to know that two of the same blobentry.label=>blobIds are not the same -- the minimum safety net i can think of is to compare created vs. updated timestamps (or lock blobentry to be fully immutable). Then there is the issue of wall time vs. sensor create time. In ROS these are Header stamps vs wall time. My understanding thus far has been that we had consensus on needing 3 timestamps because:
  • timestamp as per a sensor's internal clock, independent of the compute moving the data
  • createdTimestamp when a blobentry.label => blobId pair was first created anywhere in the distributed system. I would expect these to be hard copy the same across the proliferated copies of all the same blobentry.label=>blobId. e.g. computer A generates the blobentry for the first time, captures the sensor and created time in the blobentry. Some time later computer B does something which copies the blobentry to a different node in some other graph.label and then proceeds to mutate the copy's metadata. Computer C only ever sees the second copy, but at least computer C knows hey sensor event and mutation happened ( but blind to if a copy happened) along the way. Maybe one improvement is to rename originTimestamp instead of createdTimestamp.
  • lastUpdatedTimestamp a way to detect if an individual blobentry was modified (e.g. added fields to .metadata), but all the other proliferated copies of blobentries pointing to the same blobId will then obviously not have that mutation. Maybe also rename this to modifiedTimestamp instead of lastUpdatedTimestamp.

---

at least two timestamps is common:

[Affie]

• size::Int64/String

DFG type will go back to Int64 but still serialized as a String. I changed it to String in #1095 (DFG v0.25)

Edit: here is the implementation in case it helps

size::Int64 = -1 &(json=(lower=string, lift=x->parse(Int64, x)))

[Affie]

• origin::String

I don't have a problem with origin as a dedicated field, I would just like it defined better then so everyone uses it for the same thing. The definition was:

Context from which a Blobentry=>Blob was first created. E.g. agent|graph|varlabel.

(what I understood from this and the implementation was "who is my parent" not "where were I born")

I edited the first post to remove TODO.

and my sense is @Affie has not used the field much but I almost always use it

This is how it was used in dfg: "$(getAgentLabel(dfg))|$(getGraphLabel(dfg))|$label", so yes, I've never found it useful.

[dehann]

I would just like it defined better then so everyone uses it for the same thing

yes definitely agree.

"where [was the data blob] born"

can we make that the working definition for Blobentry.origin?

This is how it was used in dfg: "$(getAgentLabel(dfg))|$(getGraphLabel(dfg))|$label", so yes, I've never found it useful.

Yes, I added that line and that had nothing to do with "who is the parent", it was always meant as "where is the data from". I added that as a standard default in case the user does not overwrite.

Again, I don't think we need to define a convention for the origin string. I'm fine with whatever happens in that string. The documentation on motive for .origin should definitely be accurate though.

[dehann]

go back to Int64 but still serialized as a String

Okay, so size is then also resolved i think -- yes, will be serialized as a string.

[dehann]

Just the 3 timestamps are left to sort out?