Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 50 additions & 4 deletions crates/iceberg/src/io/object_cache.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ use crate::encryption::EncryptionManager;
use crate::io::FileIO;
use crate::spec::{
FormatVersion, Manifest, ManifestFile, ManifestList, ManifestListReader, ManifestReader,
SchemaId, SnapshotRef, TableMetadataRef,
SnapshotRef, TableMetadataRef,
};
use crate::{Error, ErrorKind, Result};

Expand All @@ -36,7 +36,7 @@ pub(crate) enum CachedItem {

#[derive(Clone, Debug, Hash, Eq, PartialEq)]
pub(crate) enum CachedObjectKey {
ManifestList((String, FormatVersion, SchemaId)),
ManifestList((String, FormatVersion)),

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another alternative is to continue to use Option<SchemaId> - but it doesn't seem necessary

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not for this PR, but now that schema_id is gone I'm a little curious what format_version is still buying us. Each ObjectCache is bound to a single table, so the format version is fixed for the cache's lifetime and can't disambiguate two lookups — the location is already unique.

Given xanderbailey's note that Java keys on location alone, I'd either drop it in a follow-up or leave a one-liner on why it stays, so the next reader isn't left guessing. wdyt?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think format_version should stay. The Java and Rust caching models have different semantics. The Java implementation caches raw bytes and parses after retrieval, so the cached value is version-agnostic. Rust implementation caches the parsed values, so the cached value is version-specific.

If we ever move Rust impelementation to a byte-cache like Java's, format_version could then come out of the key too, but that's a separate change.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think we should we should remove FormatVersion, a manifest list could be uniquely identified by a path. IIRC they were initially added for parsing manifest list.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked into this and I think you are right. ManifestList::parse_with_version(bytes, version) is a pure function of its two inputs, and the manifest-list bytes are immutable and uniquely identified by the location.

format_version would only disambiguate if one cache instance saw the same location under two versions. That can happen across an in-place format upgrade. Consider the following scenario:

  1. Table is at v1, and cache is hydrated.
  2. Table gets upgraded to v2.
  3. Now read happens. Question is what happens here.

So if format_version is in the cache key, 3 will always see a cache miss. It will then reparse v1 file as v2. If format_version is not in the cache key, the cache will serve the correct v1 parse output, so arguably more correct.

I'll drop it in a follow-up so this PR stays scoped to the schema-id panic fix.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#3259 as followup

// The manifest-level `first_row_id` is part of the key because the parsed
// manifest inherits it onto its entries: the same physical manifest can be
// referenced with different offsets across snapshots and branches, so it
Expand Down Expand Up @@ -159,7 +159,6 @@ impl ObjectCache {
let key = CachedObjectKey::ManifestList((
snapshot.manifest_list().to_string(),
table_metadata.format_version,
snapshot.schema_id().unwrap(),
));
let cache_entry = self
.cache
Expand Down Expand Up @@ -214,6 +213,7 @@ impl ObjectCache {

#[cfg(test)]
mod tests {
use std::collections::HashMap;
use std::fs;

use minijinja::value::Value;
Expand All @@ -226,7 +226,8 @@ mod tests {
use crate::io::{FileIO, OutputFile};
use crate::spec::{
DataContentType, DataFileBuilder, DataFileFormat, Literal, ManifestEntry,
ManifestListWriter, ManifestStatus, ManifestWriterBuilder, Struct, TableMetadata,
ManifestListWriter, ManifestStatus, ManifestWriterBuilder, Operation, Snapshot, Struct,
Summary, TableMetadata,
};
use crate::table::Table;
use crate::test_utils::test_runtime;
Expand Down Expand Up @@ -444,6 +445,51 @@ mod tests {
);
}

#[tokio::test]
async fn test_get_manifest_list_with_no_schema_id() {
let mut fixture = TableTestFixture::new();
fixture.setup_manifest_files().await;

let current_snapshot = fixture.table.metadata().current_snapshot().unwrap();

// The spec marks `schema-id` optional in every version (v1-v3), so a
// snapshot may omit it; fetching its manifest list must not depend on the
// schema-id being present.
let snapshot_without_schema_id: SnapshotRef = Snapshot::builder()
.with_snapshot_id(current_snapshot.snapshot_id())
.with_sequence_number(current_snapshot.sequence_number())
.with_timestamp_ms(current_snapshot.timestamp_ms())
.with_manifest_list(current_snapshot.manifest_list())
.with_summary(Summary {
operation: Operation::Append,
additional_properties: HashMap::new(),
})
.build()
.into();
assert!(snapshot_without_schema_id.schema_id().is_none());
assert!(current_snapshot.schema_id().is_some());

let object_cache = ObjectCache::new(fixture.table.file_io().clone(), None);

// Cold miss: the schema-id-less snapshot populates the cache.
let inserted = object_cache
.get_manifest_list(&snapshot_without_schema_id, &fixture.table.metadata_ref())
.await
.unwrap();
assert_eq!(inserted.entries().len(), 1);

// Warm hit: the original snapshot carries a schema-id but points at the same
// manifest-list location, so it returns the same cached entry.
let cached = object_cache
.get_manifest_list(current_snapshot, &fixture.table.metadata_ref())
.await
.unwrap();
assert!(
Arc::ptr_eq(&inserted, &cached),
"snapshots with and without schema-id at one location must share a cache entry"
);
}

#[tokio::test]
async fn test_get_manifest_keys_on_first_row_id() {
use crate::spec::{NestedField, PartitionSpec, PrimitiveType, Schema, SchemaRef, Type};
Expand Down
Loading