Make export usable on large accounts (10k SHOW cap) - #69
Conversation
905c490 to
20ae8a3
Compare
jacodegroothydrab
changed the title
On a large account (~15k tables, ~90k grants) `snowcap export --all` does not
complete. Three independent causes, all in the read path.
1. `SHOW ... IN ACCOUNT` is capped at 10,000 rows by Snowflake:
090153 (22000): The result set size exceeded the max number of
rows(10000) supported for SHOW statements. on SHOW TABLES IN ACCOUNT
Past that, list_tables / list_views / list_stages fail outright and take the
whole export or plan with them. They now read SNOWFLAKE.ACCOUNT_USAGE, which
has no cap, filtering `deleted IS NULL` so dropped objects are not
resurrected. Databases are compared as ResourceName, the way the SHOW path
does, so quoted and mixed-case databases are not silently excluded.
ACCOUNT_USAGE lags live state by up to ~2 hours, so an object created moments
ago can be missed. On an account past the cap a slightly stale answer beats no
answer, so this stays the default, announced once per run, with
`--no-use-account-usage` as the escape hatch to real-time SHOW. Nothing tries
to detect truncation by counting rows; the only automatic fallback is the
pre-existing one, on missing IMPORTED PRIVILEGES or a failed query.
2. `_show_all_grants_to_role` re-filtered the entire cached ACCOUNT_USAGE grant
list on every call. A 300-grant profile showed 32.7M `str.upper()` calls.
Now indexed by grantee, once per session.
3. `_fetch_grant_to_role` scanned a role's whole grant list, constructing two
ResourceName objects per candidate. Now indexed by
(granted_on, privilege, name).
ResourceName cannot key a dict directly: __hash__ is hash(str(self)), but
__eq__ treats quoted "FOO" and unquoted FOO as equal while str() keeps the
quotes, so the two disagree. _grant_name_key normalises to "exact if quoted,
upper-cased otherwise", reproducing __eq__ across all four quoted/unquoted
combinations. Object types go through _granted_on_label, as the scan it
replaces did, so reported synonyms (CORTEX_AGENT_SERVER for MCP SERVER) still
match.
Both indexes are derived state, so reset_account_usage_caches clears them
alongside the caches they are built from.
Measured on the affected account, 5,000-grant sample, steady state:
before 48.32 ms/grant -> ~74 min projected for the grant phase alone
after 0.01 ms/grant -> ~1 second
Full `export --all --exclude=table,view`: did not complete -> 125 seconds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
20ae8a3 to
abb15f3
Compare
|
There are two cache/configuration lifecycle issues to address:
Please also substantially shorten the long rationale comments added throughout |
…o reset_cache Two lifecycle issues from review, plus a comment trim. 1. Sync-mode listing did not forward the configured source. list_tables, list_views and list_stages had defaulted to ACCOUNT_USAGE, so `use_account_usage: false` still read a listing that lags live state by up to ~2 hours -- and sync mode drops whatever remote state omits. fetch_remote_state now passes self._config.use_account_usage to list_resource, which drops the kwarg for list functions that do not take it. The provider-level defaults go back to False, so a direct caller keeps the real-time SHOW it had before; `snowcap export` still opts in, as its --use-account-usage/--no-use-account-usage flag already did. 2. _GRANT_LOOKUP_INDEX_CACHE outlived reset_cache(). The index is built from cacheable SHOW GRANTS rows, but only reset_account_usage_caches() cleared it, so a second plan on the same session -- Blueprint.plan() calls reset_cache() and nothing else -- was answered from the first plan's grants. client.register_cache_reset_hook lets a module invalidate a cache derived from the execution cache without client importing it back; data_provider registers the index there, and reset_cache() now fires the hooks. Regression tests cover both: sync-mode tables issue SHOW TABLES IN ACCOUNT and no ACCOUNT_USAGE query when the flag is false, and a changed grant is observed after reset_cache() while the index still survives within one plan. Also shortened the rationale comments through data_provider.py. The performance history lives in the PR description; what stays in the code is the quoted-name normalization, the object-type synonyms, and first-match behaviour -- the invariants a later edit could silently break. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Both fixed in c6f1e06. Sync listing — Grant index — Tests: sync-mode tables issue Comments in |
2 participants
Replaces #47, cut to the scope of your review.
snowcap export --alldoes not complete on a large account (~15k tables, ~90k grants). We faced this challenge after importing from a legacy SAGE instance. Beyond lifting this cap, it also contains a number of performance fixes that speed up the code base, and would benefit users with a large number of tables too:Three issues:
SHOW ... IN ACCOUNTis capped at 10,000 rows (090153), solist_tables/list_views/list_stagesfail outright past that. They now readSNOWFLAKE.ACCOUNT_USAGE, filteringdeleted IS NULL._show_all_grants_to_rolerescanned the whole grant cache per call — 32.7Mstr.upper()calls on a 300-grant profile. Now indexed by grantee, once per session._fetch_grant_to_rolescanned a role's grants linearly, building twoResourceNameobjects per candidate. Now indexed by(granted_on, privilege, name).ResourceNamecan't key a dict:__hash__ishash(str(self))but__eq__treats quoted"FOO"and unquotedFOOas equal, andstr()keeps the quotes._grant_name_keynormalises to "exact if quoted, upper-cased otherwise", reproducing__eq__across all four combinations. Types go through_granted_on_label, as the scan did, so the synonyms #66 fixed still match. Both indexes are cleared byreset_account_usage_caches.Your review:
ResourceNamespace.test_a_quoted_database_is_keptfails against the old comparison.ACCOUNT_USAGEstays the default, warned once per run, with--use-account-usage/--no-use-account-usageonexportreaching down to the listers. No cap-detection; the only automatic fallback is the pre-existing missing-IMPORTED PRIVILEGES-or-query-failed one._grant_name_keyagainstResourceName.__eq__;_grant_lookup_index(quoting, account grants, type synonyms, first-duplicate-wins, one read per role, per-role isolation, rebuild after reset);_grants_by_role_index(grouping, case-insensitive grantee, non-role grantees, served without a query, cleared on reset); the listing helper (quoted/upper-case databases kept, unmanaged and system databases dropped, opt-out and no-access returnNoneunqueried, failed query stops retrying, warned exactly once); the three listers preferringACCOUNT_USAGE, falling back, and forwarding the opt-out.ruff,black,codespell,mypyclean; unit suite passes.Not carried over from #47:
--threads(not what fixed this) and the schema-scoped export qualification (separate concern — happy to open it on its own).Results — 5,000-grant sample, steady state:
export --all --exclude=table,view: did not complete → 125 seconds.🤖 Generated with Claude Code