Is there a fast way to estimate data size for the lance dataset?

[SaintBacchus]

The data size was very useful for cost based optimizer in compute engine like spark.

But now the lance has to open LanceFileReader to get the data size for one fragment. If we use this sdk, it will have a lot of io requests for a large lance dataset.

Is there an other way to get the data size (approximate data size was also fine) like the count_rows function?

[wjones127]

I think we are trying to expose some stats here: #3221

If we want to use them for planning purposes, we might consider caching at least some of them in the manifest.

I assume you want the size per column, right? Could you provide a link to the Spark API you are talking about?

[SaintBacchus]

Spark needs the Statistics from this interface.
The Statistics has three interfaces, sizeInBytes and numRows are useful for simple optimizer rules. The columnStats is also useful for CBO rules but it has lots of column statistics. I'm not sure the storage statistics can provide them all

public interface Statistics {
  OptionalLong sizeInBytes();
  OptionalLong numRows();
  default Map<NamedReference, ColumnStatistics> columnStats() {
    return new HashMap<>();
  }
}