Add warehouse parameter to the REST Catalog doc #2066
Conversation
| @@ -346,6 +346,7 @@ catalog: | |||
| | rest.signing-name | execute-api | The service signing name to use when SigV4 signing a request | | |||
| | oauth2-server-uri | <https://auth-service/cc> | Authentication URL to use for client credentials authentication (default: uri + 'v1/oauth/tokens') | | |||
| | snapshot-loading-mode | refs | The snapshots to return in the body of the metadata. Setting the value to `all` would return the full set of snapshots currently valid for the table. Setting the value to `refs` would load all snapshots referenced by branches or tags. | | |||
| | warehouse | myCatalog | For some catalog implementations like Databricks Unity, the warehouse is mandatory to identify which top level container (e.g. catalog, metastore...) is accessed. Namespaces are within this container. | | |||
There was a problem hiding this comment.
This configuration is used across all catalog's, so wdyt about saying something closer to the root directory in storage where new Iceberg tables are created, also it may be worth mentioning that some REST clients may override this path via the getConfig api.
https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L97
We should also have a storage path be the example.
There was a problem hiding this comment.
Hello, I am not sure to understand the comment and if some rework is requested.
There was a problem hiding this comment.
@geruh While that is true, the meaning of warehouse is different for the REST catalog and the others. For example, for Hive a path s3://bucket/warehouse is expected.
There was a problem hiding this comment.
Ohh so it doesn't have to be a path for the warehouse in rest?
There was a problem hiding this comment.
warehouse is kinda confusing for rest catalog. its used as an identifier
polaris uses it to specify the "polaris catalog name" since it supports multiple catalogs https://polaris.apache.org/in-dev/unreleased/polaris-spark-client/
s3 tables uses it to specify the specific s3 tables arn https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-open-source.html
| @@ -346,6 +346,7 @@ catalog: | |||
| | rest.signing-name | execute-api | The service signing name to use when SigV4 signing a request | | |||
| | oauth2-server-uri | <https://auth-service/cc> | Authentication URL to use for client credentials authentication (default: uri + 'v1/oauth/tokens') | | |||
| | snapshot-loading-mode | refs | The snapshots to return in the body of the metadata. Setting the value to `all` would return the full set of snapshots currently valid for the table. Setting the value to `refs` would load all snapshots referenced by branches or tags. | | |||
| | warehouse | myCatalog | For some catalog implementations like Databricks Unity, the warehouse is mandatory to identify which top level container (e.g. catalog, metastore...) is accessed. Namespaces are within this container. | | |||
There was a problem hiding this comment.
| | warehouse | myCatalog | For some catalog implementations like Databricks Unity, the warehouse is mandatory to identify which top level container (e.g. catalog, metastore...) is accessed. Namespaces are within this container. | | |
| | warehouse | myWarehouse | Warehouse location or identifier to request from the catalog service. May be used to determine server-side overrides, such as the warehouse location. | |
how about something like this? this aligns with what the rest client use warehouse for. and is taken from the openapi spec
https://github.com/apache/iceberg/blob/17f9a9fd28bbcb37745f13ebfcd57cd5a96e0a5d/open-api/rest-catalog-open-api.yaml#L78
There was a problem hiding this comment.
Warehouse is only one of the terms used by providers to identify the top level containers. Some are using catalog, other metastore. As it is quite important I suggest to be more explicit.
There was a problem hiding this comment.
Some are using catalog, other metastore. As it is quite important I suggest to be more explicit.
It seems like catalog/metastore are terms specific to UC. I would like to keep the section generic for all REST implementations. Implementation can map the warehouse identifier to its internal construct. WDYT?
Rationale for this change
Missing parameter in REST Catalog documentation
Are these changes tested?
Doc only
Are there any user-facing changes?
Doc only