Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
276 changes: 276 additions & 0 deletions PLAN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,276 @@
# Comprehensive Plan: Fix Remaining Tests

## Current Status
- **Tests passing:** 6,005 (88.0%)
- **Tests skipped:** 819 (12.0%)
- Parser failures: 173 tests
- Explain mismatches: 331 tests
- Other (metadata skip/explain=false): ~315 tests

## Phase 1: Parser Fixes (High Impact)

### 1.1 `view()` Table Function (~50 tests)
**Problem:** The `view(SELECT ...)` table function with inline subquery fails to parse.
```sql
SELECT * FROM view(SELECT 1 as id);
```
**Files:** `parser/parser.go` (parseTableExpression, parseFunctionCall)
**Solution:** When parsing a function call and the function name is `view`, check if the first argument starts with SELECT/WITH and parse it as a subquery instead of expression list.

### 1.2 Complex Type Casts with Named Parameters (~30 tests)
**Problem:** `::Tuple(a UInt32, b String)` with named fields fails
```sql
SELECT tuple(42, 42)::Tuple(a UInt32, b UInt32);
```
**Files:** `parser/expression.go` (parseDataType)
**Solution:** Extend parseDataType to handle named parameters in type constructors like `Tuple(name Type, ...)`.

### 1.3 DESCRIBE on Table Functions (~20 tests)
**Problem:** `desc format()`, `desc url()`, `desc s3Cluster()` fail
```sql
desc format(CSV, '"value"');
```
**Files:** `parser/parser.go` (parseDescribe)
**Solution:** Handle table function after DESC/DESCRIBE by calling parseTableExpression.

### 1.4 INSERT INTO FUNCTION (~15 tests)
**Problem:** INSERT INTO FUNCTION with file paths and settings fails
```sql
insert into function file(02458_data.jsonl) select * settings engine_file_truncate_on_insert=1;
```
**Files:** `parser/parser.go` (parseInsert)
**Solution:** Handle TABLE FUNCTION keyword and parse function call with settings.

### 1.5 CREATE USER / FUNCTION / DICTIONARY (~10 tests)
**Problem:** These CREATE variants are not supported
```sql
CREATE USER test_user GRANTEES ...;
CREATE DICTIONARY d0 (c1 UInt64) PRIMARY KEY c1;
```
**Files:** `parser/parser.go` (parseCreate)
**Solution:** Add cases for USER, FUNCTION, DICTIONARY in parseCreate switch.

### 1.6 SHOW SETTINGS (~5 tests)
**Problem:** SHOW SETTINGS LIKE syntax not supported
```sql
show settings like 'send_timeout';
```
**Files:** `parser/parser.go` (parseShow)
**Solution:** Handle SETTINGS keyword after SHOW.

### 1.7 PASTE JOIN (~3 tests)
**Problem:** PASTE JOIN is not recognized
```sql
SELECT * FROM t1 PASTE JOIN t2;
```
**Files:** `parser/parser.go` (parseTableExpression or join parsing)
**Solution:** Add PASTE as a valid join type.

### 1.8 `any()` Subquery Syntax (~2 tests)
**Problem:** `== any (SELECT ...)` syntax not supported
```sql
select 1 == any (select number from numbers(10));
```
**Files:** `parser/expression.go`
**Solution:** Handle `any(subquery)` as a special expression form after comparison operators.

---

## Phase 2: Explain Layer Fixes (Medium Impact)

### 2.1 INDEX Clause in CREATE TABLE (~50 tests)
**Problem:** INDEX definitions are skipped but should produce explain output
```sql
CREATE TABLE t (x UInt8, INDEX i x TYPE hypothesis GRANULARITY 100);
```
**Files:** `parser/parser.go` (parseCreateTable), `internal/explain/statements.go`
**Solution:**
1. Parse INDEX into an ast.IndexDefinition struct
2. Add explain output for index definitions

### 2.2 SETTINGS Inside Function Arguments (~40 tests)
**Problem:** SETTINGS in table functions should create a Set child
```sql
SELECT * FROM icebergS3(s3_conn, SETTINGS key='value');
```
**Files:** `parser/expression.go` (parseFunctionCall), `internal/explain/functions.go`
**Solution:** Capture SETTINGS as a Set node attached to the function call, output in explain.

### 2.3 WITH FILL Clause (~30 tests)
**Problem:** ORDER BY ... WITH FILL is not captured
```sql
SELECT nan ORDER BY 1 WITH FILL;
```
**Files:** `parser/parser.go` (parseOrderByItem), `internal/explain/select.go`
**Solution:** Add WithFill field to OrderItem, parse WITH FILL, output in explain.

### 2.4 Column CODEC Clause (~20 tests)
**Problem:** CODEC(GCD, LZ4) in columns not captured
```sql
CREATE TABLE t (col UInt32 CODEC(GCD, LZ4));
```
**Files:** `parser/parser.go` (parseColumnDeclaration), `internal/explain/statements.go`
**Solution:** Parse CODEC clause into ColumnDeclaration, output in explain.

### 2.5 Column EPHEMERAL Modifier (~15 tests)
**Problem:** EPHEMERAL keyword not captured
```sql
CREATE TABLE t (a Int EPHEMERAL);
```
**Files:** `parser/parser.go` (parseColumnDeclaration)
**Solution:** Add Ephemeral field to ColumnDeclaration, parse and explain.

### 2.6 CREATE TABLE ... AS function() (~15 tests)
**Problem:** CREATE TABLE AS s3Cluster(...) should have Function child
```sql
CREATE TABLE test AS s3Cluster('cluster', 'url');
```
**Files:** `parser/parser.go` (parseCreateTable), `internal/explain/statements.go`
**Solution:** Parse AS clause when followed by function call, store as TableFunction field.

### 2.7 WithElement Wrapper for CTEs (~20 tests)
**Problem:** Some CTEs need WithElement wrapper in output
```sql
WITH sub AS (SELECT ...) SELECT ...;
```
**Files:** `internal/explain/select.go`
**Solution:** Output WithElement wrapper when appropriate for CTE definitions.

### 2.8 Float Scientific Notation (~15 tests)
**Problem:** Very small/large floats should use scientific notation
```sql
SELECT 2.2250738585072014e-308;
```
**Files:** `internal/explain/format.go`
**Solution:** Format floats using scientific notation when appropriate.

### 2.9 Negative Literals in Arrays (~10 tests)
**Problem:** Arrays with negatives may output Function instead of Literal
```sql
SELECT [-10000, 5750];
```
**Files:** `internal/explain/expressions.go`
**Solution:** Properly detect and format negative integer literals in arrays.

### 2.10 Parameterized View Placeholders (~10 tests)
**Problem:** `{name:Type}` parameters in views
```sql
create view v as select number where number%2={parity:Int8};
```
**Files:** `internal/explain/expressions.go`
**Solution:** Output Parameter nodes correctly with type info.

### 2.11 Column TTL (~10 tests)
**Problem:** TTL expression on columns not captured
```sql
CREATE TABLE t (c Int TTL expr());
```
**Files:** `parser/parser.go` (parseColumnDeclaration)
**Solution:** Parse TTL clause into ColumnDeclaration.

---

## Phase 3: Lower Priority Fixes

### 3.1 GROUPING SETS (~5 tests)
```sql
SELECT ... GROUP BY GROUPING SETS ((a), (b));
```

### 3.2 QUALIFY Clause (~5 tests)
```sql
SELECT x QUALIFY row_number() OVER () = 1;
```

### 3.3 INTO OUTFILE TRUNCATE (~3 tests)
```sql
SELECT 1 INTO OUTFILE '/dev/null' TRUNCATE FORMAT Npy;
```

### 3.4 INTERVAL with Dynamic Type (~3 tests)
```sql
SELECT INTERVAL c0::Dynamic DAY;
```

### 3.5 ALTER TABLE with Multiple Operations (~3 tests)
```sql
ALTER TABLE t (DELETE WHERE ...), (UPDATE ... WHERE ...);
```

### 3.6 EXPLAIN SYNTAX for SYSTEM commands (~2 tests)
```sql
explain syntax system drop schema cache for hdfs;
```

---

## Implementation Order (Recommended)

1. **Week 1: Parser Fundamentals**
- 1.2 Complex Type Casts (unlocks many tests)
- 1.1 view() Table Function (high impact)
- 1.3 DESCRIBE on Table Functions

2. **Week 2: Parser Completeness**
- 1.4 INSERT INTO FUNCTION
- 1.5 CREATE USER/FUNCTION/DICTIONARY
- 1.6 SHOW SETTINGS
- 1.7 PASTE JOIN
- 1.8 any() Subquery

3. **Week 3: Explain Layer - CREATE TABLE**
- 2.1 INDEX Clause
- 2.4 CODEC Clause
- 2.5 EPHEMERAL Modifier
- 2.6 CREATE TABLE AS function()
- 2.11 Column TTL

4. **Week 4: Explain Layer - SELECT**
- 2.2 SETTINGS in Functions
- 2.3 WITH FILL
- 2.7 WithElement for CTEs
- 2.10 Parameterized View Placeholders

5. **Week 5: Explain Layer - Formatting**
- 2.8 Float Scientific Notation
- 2.9 Negative Literals in Arrays

6. **Week 6: Remaining Items**
- Phase 3 lower priority items

---

## Estimated Impact

| Phase | Tests Fixed | New Pass Rate |
|-------|-------------|---------------|
| 1.1-1.4 | ~115 | ~90% |
| 1.5-1.8 | ~20 | ~90.5% |
| 2.1-2.6 | ~140 | ~93% |
| 2.7-2.11 | ~65 | ~94% |
| Phase 3 | ~20 | ~94.5% |

---

## Files to Modify

### Parser Layer
- `parser/parser.go` - Main parser (CREATE, INSERT, DESCRIBE, SHOW, joins)
- `parser/expression.go` - Expression parsing (type casts, functions, special syntax)
- `ast/ast.go` - AST node definitions (IndexDefinition, new fields)

### Explain Layer
- `internal/explain/statements.go` - CREATE TABLE explain
- `internal/explain/select.go` - SELECT explain (WITH FILL, CTEs)
- `internal/explain/functions.go` - Function explain (SETTINGS)
- `internal/explain/expressions.go` - Expression explain (literals, parameters)
- `internal/explain/format.go` - Output formatting (scientific notation)

---

## Testing Strategy

1. Run tests frequently: `go test ./parser -timeout 5s`
2. After each fix, verify no regressions: compare PASS count
3. Check specific test cases: `go test ./parser -v -run "TestParser/test_name"`
4. Monitor for infinite loops (timeout protection already in place)
Loading