Skip to content

Conversation

@adamziel
Copy link
Collaborator

@adamziel adamziel commented Nov 21, 2025

Adapts WP_MySQL_Naive_Query_Stream to support multiline SQL queries in the runSql step. With this PR, the following call works:

await runSql(php, {
	sql: new File(
		[
			`SELECT * FROM 
				wp_users
				-- users table
			;
			 SELECT * FROM wp_posts;`
		]
		'no-trailing-newline.sql'
	),
});

Whereas before this PR, the runSql step assumed every line of a SQL file is a separate query and would fail on the above call.

Implementation details

See WordPress/sqlite-database-integration#264. Tl;dr we tokenize the query and treat ; and EOF tokens as query separators. The stream is only "naive" in that every query must be smaller than 15MB. It might fail for some very large WordPress posts, but should work most of the time. Once the lexer provides an explicit distinction between syntax errors and incomplete input, we'll be able to support arbitrarily large queries.

Testing Instructions (or ideally a Blueprint)

Tests have been updated to verify multiline query handling, SQL comment preservation, and queries with subqueries. The streaming parser correctly handles edge cases like empty lines, semicolon-only lines, and queries split across chunk boundaries.

cc @JanJakes

… support

The runSql step now processes SQL files using proper streaming semantics instead of loading entire files into memory. This change enables handling large SQL dumps efficiently while adding support for multiline queries, SQL comments, and complex query structures.

The implementation uses WP_MySQL_Naive_Query_Stream to parse SQL incrementally in 8KB chunks. As each chunk arrives, the parser identifies complete queries (those ending with semicolons) and executes them immediately. This approach keeps memory usage constant regardless of file size and provides better feedback during long-running imports.

Query parsing leverages the MySQL lexer from the sqlite-database-integration plugin, which understands MySQL syntax including string literals, comments, and nested statements. The bundled WP_MySQL_Naive_Query_Stream class wraps the lexer to handle streaming scenarios where queries may span multiple chunks.

Tests have been updated to verify multiline query handling, SQL comment preservation, and queries with subqueries. The streaming parser correctly handles edge cases like empty lines, semicolon-only lines, and queries split across chunk boundaries.
Removed loading of PHP classes from sqlite-database-integration repository in tests.
@adamziel adamziel merged commit 8d7b5f8 into trunk Nov 21, 2025
30 of 32 checks passed
@adamziel adamziel deleted the multiline-run-sql-command branch November 21, 2025 22:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants