preprocessing.log

2025-03-02 19:44:10,612 - INFO - Starting preprocessing setup
2025-03-02 19:44:10,613 - INFO - Downloading NLTK resources...
2025-03-02 19:44:17,856 - INFO - NLTK resources ready
2025-03-02 19:44:17,859 - INFO - Total exclusion words: 198
2025-03-02 19:44:17,859 - INFO - Tokenizer initialized
2025-03-02 19:45:02,554 - INFO - Starting preprocessing setup
2025-03-02 19:45:02,554 - INFO - Downloading NLTK resources...
2025-03-02 19:45:02,751 - INFO - NLTK resources ready
2025-03-02 19:45:02,752 - INFO - Total exclusion words: 198
2025-03-02 19:45:02,752 - INFO - Tokenizer initialized
2025-03-02 19:45:02,770 - INFO - Created view of Parquet file: casts.parquet
2025-03-02 19:45:02,771 - INFO - Testing tokenization on sample texts:
2025-03-02 19:45:31,108 - INFO - Starting preprocessing setup
2025-03-02 19:45:31,108 - INFO - Downloading NLTK resources...
2025-03-02 19:45:31,285 - INFO - NLTK resources ready
2025-03-02 19:45:31,286 - INFO - Total exclusion words: 198
2025-03-02 19:45:31,286 - INFO - Tokenizer initialized
2025-03-02 19:45:31,300 - INFO - Created view of Parquet file: casts.parquet
2025-03-02 19:45:31,300 - INFO - Testing tokenization on sample texts:
2025-03-02 19:45:32,128 - INFO - Sample 1:
2025-03-02 19:45:32,128 - INFO -   Original: Hello world! This is a test #farcaster message with @user mentions.
2025-03-02 19:45:32,128 - INFO -   Tokens: ['hello', 'world', 'test', 'farcaster', 'message', 'user', 'mention']
2025-03-02 19:45:32,128 - INFO - ---
2025-03-02 19:45:32,128 - INFO - Sample 2:
2025-03-02 19:45:32,128 - INFO -   Original: Check out https://farcaster.xyz for more information about #crypto and #web3!
2025-03-02 19:45:32,128 - INFO -   Tokens: ['check', 'information', 'crypto', 'web3']
2025-03-02 19:45:32,128 - INFO - ---
2025-03-02 19:45:32,128 - INFO - Sample 3:
2025-03-02 19:45:32,128 - INFO -   Original: Running and ran are both forms of run, but flies and flying are different from fly.
2025-03-02 19:45:32,128 - INFO -   Tokens: ['running', 'ran', 'form', 'run', 'fly', 'flying', 'different', 'fly']
2025-03-02 19:45:32,128 - INFO - ---
2025-03-02 19:45:32,128 - INFO - Sample 4:
2025-03-02 19:45:32,128 - INFO -   Original: The quick brown fox jumps over the lazy dog. #animals
2025-03-02 19:45:32,128 - INFO -   Tokens: ['quick', 'brown', 'fox', 'jump', 'lazy', 'dog', 'animal']
2025-03-02 19:45:32,129 - INFO - ---
2025-03-02 19:45:32,129 - INFO - Preprocessing setup complete
2025-03-02 19:47:05,512 - INFO - Starting Farcaster 'First Said' token extraction pipeline
2025-03-02 19:47:05,513 - INFO - Downloading NLTK resources...
2025-03-02 19:47:05,711 - INFO - NLTK resources ready
2025-03-02 19:47:05,711 - INFO - Total exclusion words: 198
2025-03-02 19:47:05,711 - INFO - Tokenizer initialized
2025-03-02 19:47:05,716 - WARNING - Parquet file not found: farcaster.db
2025-03-02 19:47:05,722 - INFO - Creating initial tables from casts.parquet
2025-03-02 19:47:38,201 - INFO - Loaded 157,054,986 rows from parquet file
2025-03-02 19:49:17,844 - ERROR - Error creating initial tables: Query interrupted
2025-03-02 19:51:00,898 - INFO - Starting Farcaster 'First Said' token extraction pipeline
2025-03-02 19:51:00,899 - INFO - Downloading NLTK resources...
2025-03-02 19:51:01,121 - INFO - NLTK resources ready
2025-03-02 19:51:01,122 - INFO - Total exclusion words: 198
2025-03-02 19:51:01,122 - INFO - Tokenizer initialized
2025-03-02 19:51:01,126 - WARNING - Parquet file not found: farcaster.db
2025-03-02 19:51:01,131 - INFO - Creating initial tables from casts.parquet
2025-03-02 19:51:05,362 - INFO - Loaded 157,054,986 rows from parquet file
2025-03-02 19:58:49,722 - ERROR - Error creating initial tables: Query interrupted
2025-03-02 20:00:11,583 - INFO - Starting Farcaster 'First Said' token extraction pipeline
2025-03-02 20:00:11,585 - INFO - Downloading NLTK resources...
2025-03-02 20:00:11,800 - INFO - NLTK resources ready
2025-03-02 20:00:11,801 - INFO - Total exclusion words: 198
2025-03-02 20:00:11,801 - INFO - Tokenizer initialized
2025-03-02 20:00:11,806 - WARNING - Parquet file not found: farcaster.db
2025-03-02 20:00:11,811 - INFO - Creating initial tables from casts.parquet
2025-03-02 20:00:11,836 - INFO - Total rows in parquet file: 157,054,986
2025-03-02 20:00:22,544 - ERROR - Error creating initial tables: Query interrupted
2025-03-02 20:01:23,939 - INFO - Starting Farcaster 'First Said' token extraction pipeline
2025-03-02 20:01:23,940 - INFO - Downloading NLTK resources...
2025-03-02 20:01:24,131 - INFO - NLTK resources ready
2025-03-02 20:01:24,132 - INFO - Total exclusion words: 198
2025-03-02 20:01:24,132 - INFO - Tokenizer initialized
2025-03-02 20:01:24,137 - WARNING - Parquet file not found: farcaster.db
2025-03-02 20:01:24,142 - INFO - Creating initial tables from casts.parquet
2025-03-02 20:01:24,167 - INFO - Total rows in parquet file: 157,054,986
2025-03-02 20:01:24,169 - ERROR - Error creating initial tables: Parser Error: syntax error at or near "BY"
2025-03-02 20:01:57,228 - INFO - Starting Farcaster 'First Said' token extraction pipeline
2025-03-02 20:01:57,228 - INFO - Downloading NLTK resources...
2025-03-02 20:01:57,419 - INFO - NLTK resources ready
2025-03-02 20:01:57,420 - INFO - Total exclusion words: 198
2025-03-02 20:01:57,420 - INFO - Tokenizer initialized
2025-03-02 20:01:57,424 - WARNING - Parquet file not found: farcaster.db
2025-03-02 20:01:57,429 - INFO - Creating initial tables from casts.parquet
2025-03-02 20:01:57,455 - INFO - Total rows in parquet file: 157,054,986
2025-03-02 20:03:55,419 - ERROR - Error creating initial tables: Query interrupted
2025-03-02 20:07:05,898 - INFO - Starting Farcaster 'First Said' token extraction pipeline
2025-03-02 20:07:05,899 - INFO - Downloading NLTK resources...
2025-03-02 20:07:06,080 - INFO - NLTK resources ready
2025-03-02 20:07:06,082 - INFO - Total exclusion words: 198
2025-03-02 20:07:06,082 - INFO - Tokenizer initialized
2025-03-02 20:07:58,991 - INFO - Starting Farcaster 'First Said' token extraction pipeline
2025-03-02 20:07:58,992 - INFO - Downloading NLTK resources...
2025-03-02 20:07:59,174 - INFO - NLTK resources ready
2025-03-02 20:07:59,175 - INFO - Total exclusion words: 198
2025-03-02 20:07:59,175 - INFO - Tokenizer initialized
2025-03-02 20:07:59,179 - INFO - Connected to DuckDB database: farcaster.db
2025-03-02 20:08:57,710 - INFO - Starting Farcaster 'First Said' token extraction pipeline
2025-03-02 20:08:57,711 - INFO - Downloading NLTK resources...
2025-03-02 20:08:57,896 - INFO - NLTK resources ready
2025-03-02 20:08:57,896 - INFO - Total exclusion words: 198
2025-03-02 20:08:57,896 - INFO - Tokenizer initialized
2025-03-02 20:08:57,901 - INFO - Connected to DuckDB database: farcaster.db
2025-03-02 20:08:57,907 - INFO - Creating tables in the persistent database
2025-03-02 20:09:12,217 - INFO - Creating indexes...
2025-03-02 20:11:49,411 - INFO - Tables created in persistent database successfully
2025-03-02 20:11:49,428 - INFO - Starting processing of 157,054,986 casts in chunks of 250,000
2025-03-02 20:11:56,446 - ERROR - Error during processing at offset 0: Conversion Error: invalid timestamp field format: "0", expected format is (YYYY-MM-DD HH:MM:SS[.US][±HH:MM| ZONE])
2025-03-02 20:14:24,359 - INFO - Starting Farcaster 'First Said' token extraction pipeline
2025-03-02 20:14:24,359 - INFO - Downloading NLTK resources...
2025-03-02 20:14:24,554 - INFO - NLTK resources ready
2025-03-02 20:14:24,555 - INFO - Total exclusion words: 198
2025-03-02 20:14:24,565 - INFO - Tokenizer initialized
2025-03-02 20:14:24,571 - INFO - Connected to DuckDB database: farcaster.db
2025-03-02 20:14:24,577 - INFO - Using existing tables in database: farcaster.db
2025-03-02 20:14:24,578 - INFO - Starting processing of 157,054,986 casts in chunks of 250,000
2025-03-02 20:14:24,578 - INFO - Resuming from checkpoint: offset = 0
2025-03-02 20:14:32,209 - ERROR - Error during processing at offset 0: Conversion Error: invalid timestamp field format: "0", expected format is (YYYY-MM-DD HH:MM:SS[.US][±HH:MM| ZONE])
2025-03-02 20:15:45,309 - INFO - Starting Farcaster 'First Said' token extraction pipeline
2025-03-02 20:15:45,310 - INFO - Downloading NLTK resources...
2025-03-02 20:15:45,491 - INFO - NLTK resources ready
2025-03-02 20:15:45,492 - INFO - Total exclusion words: 198
2025-03-02 20:15:45,492 - INFO - Tokenizer initialized
2025-03-02 20:15:45,497 - INFO - Connected to DuckDB database: farcaster.db
2025-03-02 20:15:45,502 - INFO - Creating tables in the persistent database
2025-03-02 20:16:41,188 - INFO - Starting Farcaster 'First Said' token extraction pipeline
2025-03-02 20:16:41,189 - INFO - Downloading NLTK resources...
2025-03-02 20:16:41,380 - INFO - NLTK resources ready
2025-03-02 20:16:41,380 - INFO - Total exclusion words: 198
2025-03-02 20:16:41,380 - INFO - Tokenizer initialized
2025-03-02 20:16:41,385 - INFO - Connected to DuckDB database: farcaster.db
2025-03-02 20:16:41,390 - INFO - Creating tables in the persistent database
2025-03-02 20:16:53,865 - INFO - Creating indexes...
2025-03-02 20:17:30,459 - INFO - Tables created in persistent database successfully
2025-03-02 20:17:30,478 - INFO - Starting processing of 157,054,986 casts in chunks of 250,000
2025-03-02 20:17:30,480 - INFO - Resuming from checkpoint: offset = 0
2025-03-02 20:18:22,767 - INFO - Progress: 1,000,000/157,054,986 rows (0.6%)
2025-03-02 20:18:22,768 - INFO - Performance: 19186.6 rows/sec overall, 18688.8 rows/sec this chunk
2025-03-02 20:18:22,768 - INFO - Tokens per row: 8.01, Total tokens: 8,010,286
2025-03-02 20:19:19,449 - INFO - Progress: 2,000,000/157,054,986 rows (1.3%)
2025-03-02 20:19:19,449 - INFO - Performance: 18382.2 rows/sec overall, 20207.8 rows/sec this chunk
2025-03-02 20:19:19,449 - INFO - Tokens per row: 7.41, Total tokens: 14,813,810
2025-03-02 20:20:11,720 - INFO - Progress: 3,000,000/157,054,986 rows (1.9%)
2025-03-02 20:20:11,723 - INFO - Performance: 18625.2 rows/sec overall, 17558.8 rows/sec this chunk
2025-03-02 20:20:11,723 - INFO - Tokens per row: 6.05, Total tokens: 18,141,662
2025-03-02 20:21:16,131 - INFO - Progress: 4,000,000/157,054,986 rows (2.5%)
2025-03-02 20:21:16,131 - INFO - Performance: 17739.7 rows/sec overall, 13936.1 rows/sec this chunk
2025-03-02 20:21:16,131 - INFO - Tokens per row: 5.41, Total tokens: 21,656,251
2025-03-02 20:22:30,494 - INFO - Progress: 5,000,000/157,054,986 rows (3.2%)
2025-03-02 20:22:30,494 - INFO - Performance: 16675.2 rows/sec overall, 12787.2 rows/sec this chunk
2025-03-02 20:22:30,494 - INFO - Tokens per row: 5.06, Total tokens: 25,292,861
2025-03-02 20:23:57,507 - INFO - Progress: 6,000,000/157,054,986 rows (3.8%)
2025-03-02 20:23:57,507 - INFO - Performance: 15509.5 rows/sec overall, 11165.3 rows/sec this chunk
2025-03-02 20:23:57,507 - INFO - Tokens per row: 4.88, Total tokens: 29,261,035
2025-03-02 20:25:30,471 - INFO - Progress: 7,000,000/157,054,986 rows (4.5%)
2025-03-02 20:25:30,471 - INFO - Performance: 14588.7 rows/sec overall, 10283.3 rows/sec this chunk
2025-03-02 20:25:30,471 - INFO - Tokens per row: 4.67, Total tokens: 32,667,029
2025-03-02 20:27:14,374 - INFO - Progress: 8,000,000/157,054,986 rows (5.1%)
2025-03-02 20:27:14,374 - INFO - Performance: 13705.1 rows/sec overall, 8899.2 rows/sec this chunk
2025-03-02 20:27:14,374 - INFO - Tokens per row: 4.48, Total tokens: 35,839,321
2025-03-02 20:29:12,628 - INFO - Progress: 9,000,000/157,054,986 rows (5.7%)
2025-03-02 20:29:12,631 - INFO - Performance: 12820.9 rows/sec overall, 8355.1 rows/sec this chunk
2025-03-02 20:29:12,631 - INFO - Tokens per row: 4.34, Total tokens: 39,047,303
2025-03-02 20:31:23,038 - INFO - Progress: 10,000,000/157,054,986 rows (6.4%)
2025-03-02 20:31:23,039 - INFO - Performance: 12013.6 rows/sec overall, 7418.8 rows/sec this chunk
2025-03-02 20:31:23,039 - INFO - Tokens per row: 4.25, Total tokens: 42,505,329
2025-03-02 20:33:40,485 - INFO - Progress: 11,000,000/157,054,986 rows (7.0%)
2025-03-02 20:33:40,485 - INFO - Performance: 11342.1 rows/sec overall, 7128.0 rows/sec this chunk
2025-03-02 20:33:40,485 - INFO - Tokens per row: 4.18, Total tokens: 45,972,660
2025-03-02 20:36:04,626 - INFO - Progress: 12,000,000/157,054,986 rows (7.6%)
2025-03-02 20:36:04,631 - INFO - Performance: 10772.2 rows/sec overall, 6726.9 rows/sec this chunk
2025-03-02 20:36:04,631 - INFO - Tokens per row: 4.12, Total tokens: 49,451,692
2025-03-02 20:38:41,439 - INFO - Progress: 13,000,000/157,054,986 rows (8.3%)
2025-03-02 20:38:41,440 - INFO - Performance: 10229.9 rows/sec overall, 6387.1 rows/sec this chunk
2025-03-02 20:38:41,440 - INFO - Tokens per row: 4.07, Total tokens: 52,899,743
2025-03-02 20:41:31,505 - INFO - Progress: 14,000,000/157,054,986 rows (8.9%)
2025-03-02 20:41:31,506 - INFO - Performance: 9716.4 rows/sec overall, 5559.2 rows/sec this chunk
2025-03-02 20:41:31,506 - INFO - Tokens per row: 4.02, Total tokens: 56,271,784
2025-03-02 20:44:29,999 - INFO - Progress: 15,000,000/157,054,986 rows (9.6%)
2025-03-02 20:44:30,000 - INFO - Performance: 9263.0 rows/sec overall, 5791.3 rows/sec this chunk
2025-03-02 20:44:30,000 - INFO - Tokens per row: 3.98, Total tokens: 59,734,206
2025-03-02 20:47:48,253 - INFO - Progress: 16,000,000/157,054,986 rows (10.2%)
2025-03-02 20:47:48,253 - INFO - Performance: 8802.8 rows/sec overall, 4691.7 rows/sec this chunk
2025-03-02 20:47:48,253 - INFO - Tokens per row: 3.96, Total tokens: 63,286,705
2025-03-02 20:51:18,126 - INFO - Progress: 17,000,000/157,054,986 rows (10.8%)
2025-03-02 20:51:18,127 - INFO - Performance: 8384.8 rows/sec overall, 4603.6 rows/sec this chunk
2025-03-02 20:51:18,127 - INFO - Tokens per row: 3.93, Total tokens: 66,860,369
2025-03-02 20:54:53,035 - INFO - Progress: 18,000,000/157,054,986 rows (11.5%)
2025-03-02 20:54:53,035 - INFO - Performance: 8027.2 rows/sec overall, 4700.2 rows/sec this chunk
2025-03-02 20:54:53,035 - INFO - Tokens per row: 3.91, Total tokens: 70,467,418
2025-03-02 20:58:33,580 - INFO - Progress: 19,000,000/157,054,986 rows (12.1%)
2025-03-02 20:58:33,582 - INFO - Performance: 7714.4 rows/sec overall, 4228.9 rows/sec this chunk
2025-03-02 20:58:33,582 - INFO - Tokens per row: 3.90, Total tokens: 74,101,088
2025-03-02 21:02:35,607 - INFO - Progress: 20,000,000/157,054,986 rows (12.7%)
2025-03-02 21:02:35,608 - INFO - Performance: 7393.8 rows/sec overall, 4449.8 rows/sec this chunk
2025-03-02 21:02:35,608 - INFO - Tokens per row: 3.89, Total tokens: 77,824,609
2025-03-02 21:06:47,231 - INFO - Progress: 21,000,000/157,054,986 rows (13.4%)
2025-03-02 21:06:47,232 - INFO - Performance: 7102.8 rows/sec overall, 4141.2 rows/sec this chunk
2025-03-02 21:06:47,232 - INFO - Tokens per row: 3.88, Total tokens: 81,566,057
2025-03-02 21:10:58,107 - INFO - Progress: 22,000,000/157,054,986 rows (14.0%)
2025-03-02 21:10:58,107 - INFO - Performance: 6859.0 rows/sec overall, 4115.6 rows/sec this chunk
2025-03-02 21:10:58,107 - INFO - Tokens per row: 3.87, Total tokens: 85,201,667
2025-03-02 21:15:06,488 - INFO - Progress: 23,000,000/157,054,986 rows (14.6%)
2025-03-02 21:15:06,489 - INFO - Performance: 6655.4 rows/sec overall, 3978.3 rows/sec this chunk
2025-03-02 21:15:06,489 - INFO - Tokens per row: 3.86, Total tokens: 88,866,947
2025-03-02 21:19:31,062 - INFO - Progress: 24,000,000/157,054,986 rows (15.3%)
2025-03-02 21:19:31,063 - INFO - Performance: 6450.9 rows/sec overall, 3923.7 rows/sec this chunk
2025-03-02 21:19:31,063 - INFO - Tokens per row: 3.85, Total tokens: 92,463,526
2025-03-02 21:23:53,193 - INFO - Progress: 25,000,000/157,054,986 rows (15.9%)
2025-03-02 21:23:53,194 - INFO - Performance: 6277.4 rows/sec overall, 3910.7 rows/sec this chunk
2025-03-02 21:23:53,194 - INFO - Tokens per row: 3.84, Total tokens: 95,940,728
2025-03-02 21:28:31,503 - INFO - Progress: 26,000,000/157,054,986 rows (16.6%)
2025-03-02 21:28:31,504 - INFO - Performance: 6102.1 rows/sec overall, 3198.7 rows/sec this chunk
2025-03-02 21:28:31,504 - INFO - Tokens per row: 3.83, Total tokens: 99,521,721
2025-03-02 21:33:22,218 - INFO - Progress: 27,000,000/157,054,986 rows (17.2%)
2025-03-02 21:33:22,222 - INFO - Performance: 5932.0 rows/sec overall, 2984.4 rows/sec this chunk
2025-03-02 21:33:22,222 - INFO - Tokens per row: 3.82, Total tokens: 103,076,169
2025-03-02 21:38:10,903 - INFO - Progress: 28,000,000/157,054,986 rows (17.8%)
2025-03-02 21:38:10,904 - INFO - Performance: 5784.8 rows/sec overall, 3598.6 rows/sec this chunk
2025-03-02 21:38:10,904 - INFO - Tokens per row: 3.81, Total tokens: 106,555,192
2025-03-02 21:43:07,349 - INFO - Progress: 29,000,000/157,054,986 rows (18.5%)
2025-03-02 21:43:07,350 - INFO - Performance: 5645.6 rows/sec overall, 3452.3 rows/sec this chunk
2025-03-02 21:43:07,350 - INFO - Tokens per row: 3.79, Total tokens: 109,977,742
2025-03-02 21:48:02,701 - INFO - Progress: 30,000,000/157,054,986 rows (19.1%)
2025-03-02 21:48:02,702 - INFO - Performance: 5522.8 rows/sec overall, 3480.7 rows/sec this chunk
2025-03-02 21:48:02,702 - INFO - Tokens per row: 3.78, Total tokens: 113,466,164
2025-03-02 21:52:54,361 - INFO - Progress: 31,000,000/157,054,986 rows (19.7%)
2025-03-02 21:52:54,362 - INFO - Performance: 5416.1 rows/sec overall, 3476.3 rows/sec this chunk
2025-03-02 21:52:54,362 - INFO - Tokens per row: 3.77, Total tokens: 117,000,911
2025-03-02 21:57:56,074 - INFO - Progress: 32,000,000/157,054,986 rows (20.4%)
2025-03-02 21:57:56,078 - INFO - Performance: 5310.8 rows/sec overall, 3244.7 rows/sec this chunk
2025-03-02 21:57:56,078 - INFO - Tokens per row: 3.77, Total tokens: 120,572,634
2025-03-02 22:03:24,546 - INFO - Progress: 33,000,000/157,054,986 rows (21.0%)
2025-03-02 22:03:24,547 - INFO - Performance: 5193.7 rows/sec overall, 2899.7 rows/sec this chunk
2025-03-02 22:03:24,547 - INFO - Tokens per row: 3.76, Total tokens: 124,083,695
2025-03-02 22:08:52,848 - INFO - Progress: 34,000,000/157,054,986 rows (21.6%)
2025-03-02 22:08:52,849 - INFO - Performance: 5088.1 rows/sec overall, 3146.2 rows/sec this chunk
2025-03-02 22:08:52,849 - INFO - Tokens per row: 3.75, Total tokens: 127,463,922
2025-03-02 22:14:34,047 - INFO - Progress: 35,000,000/157,054,986 rows (22.3%)
2025-03-02 22:14:34,048 - INFO - Performance: 4983.3 rows/sec overall, 2894.5 rows/sec this chunk
2025-03-02 22:14:34,048 - INFO - Tokens per row: 3.74, Total tokens: 130,888,646
2025-03-02 22:19:55,325 - INFO - Progress: 36,000,000/157,054,986 rows (22.9%)
2025-03-02 22:19:55,325 - INFO - Performance: 4901.5 rows/sec overall, 3058.1 rows/sec this chunk
2025-03-02 22:19:55,325 - INFO - Tokens per row: 3.74, Total tokens: 134,550,495
2025-03-02 22:25:39,149 - INFO - Progress: 37,000,000/157,054,986 rows (23.6%)
2025-03-02 22:25:39,150 - INFO - Performance: 4812.4 rows/sec overall, 2968.8 rows/sec this chunk
2025-03-02 22:25:39,150 - INFO - Tokens per row: 3.73, Total tokens: 138,189,492
2025-03-02 22:31:16,418 - INFO - Progress: 38,000,000/157,054,986 rows (24.2%)
2025-03-02 22:31:16,419 - INFO - Performance: 4734.7 rows/sec overall, 2948.8 rows/sec this chunk
2025-03-02 22:31:16,419 - INFO - Tokens per row: 3.73, Total tokens: 141,767,955
2025-03-02 22:37:18,472 - INFO - Progress: 39,000,000/157,054,986 rows (24.8%)
2025-03-02 22:37:18,473 - INFO - Performance: 4649.6 rows/sec overall, 2534.6 rows/sec this chunk
2025-03-02 22:37:18,473 - INFO - Tokens per row: 3.73, Total tokens: 145,333,463
2025-03-02 22:43:30,320 - INFO - Progress: 40,000,000/157,054,986 rows (25.5%)
2025-03-02 22:43:30,322 - INFO - Performance: 4566.4 rows/sec overall, 2716.1 rows/sec this chunk
2025-03-02 22:43:30,322 - INFO - Tokens per row: 3.72, Total tokens: 148,834,732
2025-03-02 22:49:29,171 - INFO - Progress: 41,000,000/157,054,986 rows (26.1%)
2025-03-02 22:49:29,172 - INFO - Performance: 4496.3 rows/sec overall, 2799.8 rows/sec this chunk
2025-03-02 22:49:29,172 - INFO - Tokens per row: 3.71, Total tokens: 152,240,439
2025-03-02 22:55:37,243 - INFO - Progress: 42,000,000/157,054,986 rows (26.7%)
2025-03-02 22:55:37,244 - INFO - Performance: 4427.3 rows/sec overall, 2829.3 rows/sec this chunk
2025-03-02 22:55:37,244 - INFO - Tokens per row: 3.70, Total tokens: 155,594,523
2025-03-02 23:01:56,297 - INFO - Progress: 43,000,000/157,054,986 rows (27.4%)
2025-03-02 23:01:56,298 - INFO - Performance: 4358.6 rows/sec overall, 2616.5 rows/sec this chunk
2025-03-02 23:01:56,298 - INFO - Tokens per row: 3.70, Total tokens: 158,956,390
2025-03-02 23:08:04,027 - INFO - Progress: 44,000,000/157,054,986 rows (28.0%)
2025-03-02 23:08:04,027 - INFO - Performance: 4299.7 rows/sec overall, 2743.8 rows/sec this chunk
2025-03-02 23:08:04,027 - INFO - Tokens per row: 3.69, Total tokens: 162,391,459
2025-03-02 23:14:22,456 - INFO - Progress: 45,000,000/157,054,986 rows (28.7%)
2025-03-02 23:14:22,457 - INFO - Performance: 4240.6 rows/sec overall, 2532.2 rows/sec this chunk
2025-03-02 23:14:22,457 - INFO - Tokens per row: 3.69, Total tokens: 165,906,154
2025-03-02 23:20:40,450 - INFO - Progress: 46,000,000/157,054,986 rows (29.3%)
2025-03-02 23:20:40,451 - INFO - Performance: 4185.7 rows/sec overall, 2687.7 rows/sec this chunk
2025-03-02 23:20:40,451 - INFO - Tokens per row: 3.68, Total tokens: 169,383,019
2025-03-02 23:27:18,715 - INFO - Progress: 47,000,000/157,054,986 rows (29.9%)
2025-03-02 23:27:18,716 - INFO - Performance: 4127.1 rows/sec overall, 2323.2 rows/sec this chunk
2025-03-02 23:27:18,716 - INFO - Tokens per row: 3.68, Total tokens: 172,929,593
2025-03-02 23:33:49,827 - INFO - Progress: 48,000,000/157,054,986 rows (30.6%)
2025-03-02 23:33:49,827 - INFO - Performance: 4075.0 rows/sec overall, 2563.5 rows/sec this chunk
2025-03-02 23:33:49,827 - INFO - Tokens per row: 3.68, Total tokens: 176,420,923
2025-03-02 23:40:22,266 - INFO - Progress: 49,000,000/157,054,986 rows (31.2%)
2025-03-02 23:40:22,266 - INFO - Performance: 4025.8 rows/sec overall, 2570.6 rows/sec this chunk
2025-03-02 23:40:22,266 - INFO - Tokens per row: 3.67, Total tokens: 179,994,060
2025-03-02 23:47:15,649 - INFO - Progress: 50,000,000/157,054,986 rows (31.8%)
2025-03-02 23:47:15,650 - INFO - Performance: 3973.0 rows/sec overall, 2421.4 rows/sec this chunk
2025-03-02 23:47:15,650 - INFO - Tokens per row: 3.67, Total tokens: 183,525,796
2025-03-02 23:54:20,061 - INFO - Progress: 51,000,000/157,054,986 rows (32.5%)
2025-03-02 23:54:20,062 - INFO - Performance: 3920.2 rows/sec overall, 2404.9 rows/sec this chunk
2025-03-02 23:54:20,062 - INFO - Tokens per row: 3.67, Total tokens: 186,994,610
2025-03-03 00:01:19,881 - INFO - Progress: 52,000,000/157,054,986 rows (33.1%)
2025-03-03 00:01:19,882 - INFO - Performance: 3872.1 rows/sec overall, 2378.6 rows/sec this chunk
2025-03-03 00:01:19,882 - INFO - Tokens per row: 3.66, Total tokens: 190,485,197
2025-03-03 00:08:26,266 - INFO - Progress: 53,000,000/157,054,986 rows (33.7%)
2025-03-03 00:08:26,267 - INFO - Performance: 3825.2 rows/sec overall, 2366.6 rows/sec this chunk
2025-03-03 00:08:26,267 - INFO - Tokens per row: 3.66, Total tokens: 194,091,223
2025-03-03 00:15:37,244 - INFO - Progress: 54,000,000/157,054,986 rows (34.4%)
2025-03-03 00:15:37,244 - INFO - Performance: 3779.8 rows/sec overall, 2317.6 rows/sec this chunk
2025-03-03 00:15:37,244 - INFO - Tokens per row: 3.66, Total tokens: 197,850,474
2025-03-03 00:25:05,175 - INFO - Progress: 55,000,000/157,054,986 rows (35.0%)
2025-03-03 00:25:05,176 - INFO - Performance: 3702.6 rows/sec overall, 2112.5 rows/sec this chunk
2025-03-03 00:25:05,176 - INFO - Tokens per row: 3.67, Total tokens: 201,823,754
2025-03-03 00:32:42,068 - INFO - Progress: 56,000,000/157,054,986 rows (35.7%)
2025-03-03 00:32:42,070 - INFO - Performance: 3657.4 rows/sec overall, 2247.8 rows/sec this chunk
2025-03-03 00:32:42,070 - INFO - Tokens per row: 3.67, Total tokens: 205,515,032
2025-03-03 00:40:26,000 - INFO - Progress: 57,000,000/157,054,986 rows (36.3%)
2025-03-03 00:40:26,001 - INFO - Performance: 3613.2 rows/sec overall, 2146.9 rows/sec this chunk
2025-03-03 00:40:26,001 - INFO - Tokens per row: 3.67, Total tokens: 209,252,019
2025-03-03 00:48:12,731 - INFO - Progress: 58,000,000/157,054,986 rows (36.9%)
2025-03-03 00:48:12,732 - INFO - Performance: 3571.0 rows/sec overall, 2205.9 rows/sec this chunk
2025-03-03 00:48:12,732 - INFO - Tokens per row: 3.67, Total tokens: 213,058,786
2025-03-03 00:56:01,931 - INFO - Progress: 59,000,000/157,054,986 rows (37.6%)
2025-03-03 00:56:01,931 - INFO - Performance: 3530.5 rows/sec overall, 2266.1 rows/sec this chunk
2025-03-03 00:56:01,931 - INFO - Tokens per row: 3.67, Total tokens: 216,816,102
2025-03-03 01:03:19,398 - INFO - Progress: 60,000,000/157,054,986 rows (38.2%)
2025-03-03 01:03:19,401 - INFO - Performance: 3498.8 rows/sec overall, 2273.4 rows/sec this chunk
2025-03-03 01:03:19,401 - INFO - Tokens per row: 3.68, Total tokens: 220,617,029
2025-03-03 01:10:37,410 - INFO - Progress: 61,000,000/157,054,986 rows (38.8%)
2025-03-03 01:10:37,411 - INFO - Performance: 3468.5 rows/sec overall, 2302.0 rows/sec this chunk
2025-03-03 01:10:37,411 - INFO - Tokens per row: 3.68, Total tokens: 224,401,352
2025-03-03 01:17:49,191 - INFO - Progress: 62,000,000/157,054,986 rows (39.5%)
2025-03-03 01:17:49,191 - INFO - Performance: 3440.9 rows/sec overall, 2344.3 rows/sec this chunk
2025-03-03 01:17:49,191 - INFO - Tokens per row: 3.68, Total tokens: 228,032,885
2025-03-03 01:25:12,580 - INFO - Progress: 63,000,000/157,054,986 rows (40.1%)
2025-03-03 01:25:12,581 - INFO - Performance: 3412.4 rows/sec overall, 2187.4 rows/sec this chunk
2025-03-03 01:25:12,581 - INFO - Tokens per row: 3.68, Total tokens: 231,573,846
2025-03-03 01:32:41,808 - INFO - Progress: 64,000,000/157,054,986 rows (40.8%)
2025-03-03 01:32:41,810 - INFO - Performance: 3384.2 rows/sec overall, 2236.8 rows/sec this chunk
2025-03-03 01:32:41,810 - INFO - Tokens per row: 3.67, Total tokens: 235,168,106
2025-03-03 01:40:11,283 - INFO - Progress: 65,000,000/157,054,986 rows (41.4%)
2025-03-03 01:40:11,284 - INFO - Performance: 3357.3 rows/sec overall, 2240.6 rows/sec this chunk
2025-03-03 01:40:11,284 - INFO - Tokens per row: 3.68, Total tokens: 238,898,269
2025-03-03 01:48:07,308 - INFO - Progress: 66,000,000/157,054,986 rows (42.0%)
2025-03-03 01:48:07,308 - INFO - Performance: 3327.2 rows/sec overall, 2003.9 rows/sec this chunk
2025-03-03 01:48:07,309 - INFO - Tokens per row: 3.68, Total tokens: 242,790,192
2025-03-03 01:55:39,383 - INFO - Progress: 67,000,000/157,054,986 rows (42.7%)
2025-03-03 01:55:39,384 - INFO - Performance: 3302.3 rows/sec overall, 2219.6 rows/sec this chunk
2025-03-03 01:55:39,384 - INFO - Tokens per row: 3.68, Total tokens: 246,546,289
2025-03-03 02:03:19,113 - INFO - Progress: 68,000,000/157,054,986 rows (43.3%)
2025-03-03 02:03:19,114 - INFO - Performance: 3277.4 rows/sec overall, 2197.1 rows/sec this chunk
2025-03-03 02:03:19,114 - INFO - Tokens per row: 3.68, Total tokens: 250,297,733
2025-03-03 02:11:00,188 - INFO - Progress: 69,000,000/157,054,986 rows (43.9%)
2025-03-03 02:11:00,189 - INFO - Performance: 3253.3 rows/sec overall, 2195.8 rows/sec this chunk
2025-03-03 02:11:00,189 - INFO - Tokens per row: 3.68, Total tokens: 254,045,646
2025-03-03 02:18:44,061 - INFO - Progress: 70,000,000/157,054,986 rows (44.6%)
2025-03-03 02:18:44,062 - INFO - Performance: 3229.8 rows/sec overall, 2096.8 rows/sec this chunk
2025-03-03 02:18:44,062 - INFO - Tokens per row: 3.68, Total tokens: 257,712,488
2025-03-03 02:26:36,286 - INFO - Progress: 71,000,000/157,054,986 rows (45.2%)
2025-03-03 02:26:36,288 - INFO - Performance: 3206.0 rows/sec overall, 2206.9 rows/sec this chunk
2025-03-03 02:26:36,288 - INFO - Tokens per row: 3.68, Total tokens: 261,403,361
2025-03-03 02:34:32,643 - INFO - Progress: 72,000,000/157,054,986 rows (45.8%)
2025-03-03 02:34:32,644 - INFO - Performance: 3182.7 rows/sec overall, 2015.8 rows/sec this chunk
2025-03-03 02:34:32,644 - INFO - Tokens per row: 3.68, Total tokens: 265,207,944
2025-03-03 02:42:26,058 - INFO - Progress: 73,000,000/157,054,986 rows (46.5%)
2025-03-03 02:42:26,060 - INFO - Performance: 3160.8 rows/sec overall, 2114.4 rows/sec this chunk
2025-03-03 02:42:26,060 - INFO - Tokens per row: 3.68, Total tokens: 268,981,780
2025-03-03 02:50:24,310 - INFO - Progress: 74,000,000/157,054,986 rows (47.1%)
2025-03-03 02:50:24,312 - INFO - Performance: 3139.1 rows/sec overall, 2046.7 rows/sec this chunk
2025-03-03 02:50:24,312 - INFO - Tokens per row: 3.69, Total tokens: 272,724,834
2025-03-03 02:58:21,779 - INFO - Progress: 75,000,000/157,054,986 rows (47.8%)
2025-03-03 02:58:21,780 - INFO - Performance: 3118.4 rows/sec overall, 2089.0 rows/sec this chunk
2025-03-03 02:58:21,780 - INFO - Tokens per row: 3.69, Total tokens: 276,643,246
2025-03-03 03:06:26,972 - INFO - Progress: 76,000,000/157,054,986 rows (48.4%)
2025-03-03 03:06:26,973 - INFO - Performance: 3097.4 rows/sec overall, 2051.3 rows/sec this chunk
2025-03-03 03:06:26,973 - INFO - Tokens per row: 3.69, Total tokens: 280,704,227
2025-03-03 03:14:31,512 - INFO - Progress: 77,000,000/157,054,986 rows (49.0%)
2025-03-03 03:14:31,513 - INFO - Performance: 3077.4 rows/sec overall, 2061.9 rows/sec this chunk
2025-03-03 03:14:31,513 - INFO - Tokens per row: 3.70, Total tokens: 284,736,952
2025-03-03 03:22:58,358 - INFO - Progress: 78,000,000/157,054,986 rows (49.7%)
2025-03-03 03:22:58,359 - INFO - Performance: 3055.5 rows/sec overall, 1962.4 rows/sec this chunk
2025-03-03 03:22:58,359 - INFO - Tokens per row: 3.70, Total tokens: 288,943,794
2025-03-03 03:31:15,197 - INFO - Progress: 79,000,000/157,054,986 rows (50.3%)
2025-03-03 03:31:15,198 - INFO - Performance: 3035.6 rows/sec overall, 2011.7 rows/sec this chunk
2025-03-03 03:31:15,198 - INFO - Tokens per row: 3.71, Total tokens: 293,307,674
2025-03-03 03:39:39,902 - INFO - Progress: 80,000,000/157,054,986 rows (50.9%)
2025-03-03 03:39:39,904 - INFO - Performance: 3015.5 rows/sec overall, 1997.4 rows/sec this chunk
2025-03-03 03:39:39,904 - INFO - Tokens per row: 3.73, Total tokens: 298,192,410
2025-03-03 03:48:09,958 - INFO - Progress: 81,000,000/157,054,986 rows (51.6%)
2025-03-03 03:48:09,959 - INFO - Performance: 2995.6 rows/sec overall, 1965.9 rows/sec this chunk
2025-03-03 03:48:09,959 - INFO - Tokens per row: 3.74, Total tokens: 302,832,323
2025-03-03 03:56:42,592 - INFO - Progress: 82,000,000/157,054,986 rows (52.2%)
2025-03-03 03:56:42,593 - INFO - Performance: 2976.2 rows/sec overall, 1952.5 rows/sec this chunk
2025-03-03 03:56:42,593 - INFO - Tokens per row: 3.75, Total tokens: 307,491,835
2025-03-03 04:05:21,546 - INFO - Progress: 83,000,000/157,054,986 rows (52.8%)
2025-03-03 04:05:21,547 - INFO - Performance: 2956.8 rows/sec overall, 1921.5 rows/sec this chunk
2025-03-03 04:05:21,547 - INFO - Tokens per row: 3.76, Total tokens: 311,931,813
2025-03-03 04:13:58,911 - INFO - Progress: 84,000,000/157,054,986 rows (53.5%)
2025-03-03 04:13:58,912 - INFO - Performance: 2938.3 rows/sec overall, 1908.9 rows/sec this chunk
2025-03-03 04:13:58,912 - INFO - Tokens per row: 3.77, Total tokens: 316,643,777
2025-03-03 04:22:46,164 - INFO - Progress: 85,000,000/157,054,986 rows (54.1%)
2025-03-03 04:22:46,165 - INFO - Performance: 2919.4 rows/sec overall, 1878.3 rows/sec this chunk
2025-03-03 04:22:46,165 - INFO - Tokens per row: 3.78, Total tokens: 321,184,213
2025-03-03 04:31:36,002 - INFO - Progress: 86,000,000/157,054,986 rows (54.8%)
2025-03-03 04:31:36,004 - INFO - Performance: 2901.0 rows/sec overall, 1893.0 rows/sec this chunk
2025-03-03 04:31:36,004 - INFO - Tokens per row: 3.79, Total tokens: 325,894,548
2025-03-03 04:40:40,141 - INFO - Progress: 87,000,000/157,054,986 rows (55.4%)
2025-03-03 04:40:40,142 - INFO - Performance: 2881.8 rows/sec overall, 1940.6 rows/sec this chunk
2025-03-03 04:40:40,142 - INFO - Tokens per row: 3.80, Total tokens: 331,000,348
2025-03-03 04:49:29,853 - INFO - Progress: 88,000,000/157,054,986 rows (56.0%)
2025-03-03 04:49:29,854 - INFO - Performance: 2864.7 rows/sec overall, 1898.0 rows/sec this chunk
2025-03-03 04:49:29,854 - INFO - Tokens per row: 3.82, Total tokens: 336,003,248
2025-03-03 04:58:19,040 - INFO - Progress: 89,000,000/157,054,986 rows (56.7%)
2025-03-03 04:58:19,041 - INFO - Performance: 2848.1 rows/sec overall, 1895.8 rows/sec this chunk
2025-03-03 04:58:19,041 - INFO - Tokens per row: 3.83, Total tokens: 340,740,131
2025-03-03 05:07:05,437 - INFO - Progress: 90,000,000/157,054,986 rows (57.3%)
2025-03-03 05:07:05,438 - INFO - Performance: 2832.4 rows/sec overall, 1888.4 rows/sec this chunk
2025-03-03 05:07:05,438 - INFO - Tokens per row: 3.84, Total tokens: 345,702,120
2025-03-03 05:16:02,099 - INFO - Progress: 91,000,000/157,054,986 rows (57.9%)
2025-03-03 05:16:02,100 - INFO - Performance: 2816.3 rows/sec overall, 1863.3 rows/sec this chunk
2025-03-03 05:16:02,100 - INFO - Tokens per row: 3.85, Total tokens: 350,602,646
2025-03-03 05:25:03,999 - INFO - Progress: 92,000,000/157,054,986 rows (58.6%)
2025-03-03 05:25:04,000 - INFO - Performance: 2800.3 rows/sec overall, 1834.6 rows/sec this chunk
2025-03-03 05:25:04,000 - INFO - Tokens per row: 3.87, Total tokens: 355,661,083
2025-03-03 05:34:03,500 - INFO - Progress: 93,000,000/157,054,986 rows (59.2%)
2025-03-03 05:34:03,501 - INFO - Performance: 2785.0 rows/sec overall, 1846.3 rows/sec this chunk
2025-03-03 05:34:03,501 - INFO - Tokens per row: 3.88, Total tokens: 360,494,480
2025-03-03 05:43:09,200 - INFO - Progress: 94,000,000/157,054,986 rows (59.9%)
2025-03-03 05:43:09,201 - INFO - Performance: 2769.7 rows/sec overall, 1836.9 rows/sec this chunk
2025-03-03 05:43:09,201 - INFO - Tokens per row: 3.89, Total tokens: 365,407,600
2025-03-03 05:52:16,431 - INFO - Progress: 95,000,000/157,054,986 rows (60.5%)
2025-03-03 05:52:16,432 - INFO - Performance: 2754.8 rows/sec overall, 1822.0 rows/sec this chunk
2025-03-03 05:52:16,432 - INFO - Tokens per row: 3.90, Total tokens: 370,057,219
2025-03-03 06:01:25,024 - INFO - Progress: 96,000,000/157,054,986 rows (61.1%)
2025-03-03 06:01:25,026 - INFO - Performance: 2740.2 rows/sec overall, 1791.0 rows/sec this chunk
2025-03-03 06:01:25,026 - INFO - Tokens per row: 3.91, Total tokens: 374,907,126
2025-03-03 06:10:32,993 - INFO - Progress: 97,000,000/157,054,986 rows (61.8%)
2025-03-03 06:10:32,994 - INFO - Performance: 2726.1 rows/sec overall, 1826.0 rows/sec this chunk
2025-03-03 06:10:32,994 - INFO - Tokens per row: 3.91, Total tokens: 379,621,777
2025-03-03 06:19:48,078 - INFO - Progress: 98,000,000/157,054,986 rows (62.4%)
2025-03-03 06:19:48,079 - INFO - Performance: 2711.9 rows/sec overall, 1809.3 rows/sec this chunk
2025-03-03 06:19:48,079 - INFO - Tokens per row: 3.92, Total tokens: 384,393,014
2025-03-03 06:29:09,038 - INFO - Progress: 99,000,000/157,054,986 rows (63.0%)
2025-03-03 06:29:09,039 - INFO - Performance: 2697.7 rows/sec overall, 1829.9 rows/sec this chunk
2025-03-03 06:29:09,039 - INFO - Tokens per row: 3.94, Total tokens: 389,777,191
2025-03-03 06:38:46,979 - INFO - Progress: 100,000,000/157,054,986 rows (63.7%)
2025-03-03 06:38:46,980 - INFO - Performance: 2682.7 rows/sec overall, 1573.6 rows/sec this chunk
2025-03-03 06:38:46,980 - INFO - Tokens per row: 3.95, Total tokens: 394,851,739
2025-03-03 06:48:06,047 - INFO - Progress: 101,000,000/157,054,986 rows (64.3%)
2025-03-03 06:48:06,048 - INFO - Performance: 2669.5 rows/sec overall, 1795.2 rows/sec this chunk
2025-03-03 06:48:06,048 - INFO - Tokens per row: 3.96, Total tokens: 400,078,346
2025-03-03 06:57:27,796 - INFO - Progress: 102,000,000/157,054,986 rows (64.9%)
2025-03-03 06:57:27,797 - INFO - Performance: 2656.4 rows/sec overall, 1799.6 rows/sec this chunk
2025-03-03 06:57:27,797 - INFO - Tokens per row: 3.98, Total tokens: 405,753,126
2025-03-03 07:10:13,547 - INFO - Progress: 103,000,000/157,054,986 rows (65.6%)
2025-03-03 07:10:13,548 - INFO - Performance: 2630.0 rows/sec overall, 1671.7 rows/sec this chunk
2025-03-03 07:10:13,548 - INFO - Tokens per row: 3.99, Total tokens: 411,321,058
2025-03-03 07:19:44,902 - INFO - Progress: 104,000,000/157,054,986 rows (66.2%)
2025-03-03 07:19:44,903 - INFO - Performance: 2617.4 rows/sec overall, 1777.2 rows/sec this chunk
2025-03-03 07:19:44,903 - INFO - Tokens per row: 4.01, Total tokens: 416,941,007
2025-03-03 07:29:43,595 - INFO - Progress: 105,000,000/157,054,986 rows (66.9%)
2025-03-03 07:29:43,597 - INFO - Performance: 2603.3 rows/sec overall, 1760.6 rows/sec this chunk
2025-03-03 07:29:43,597 - INFO - Tokens per row: 4.03, Total tokens: 422,695,752
2025-03-03 07:39:42,318 - INFO - Progress: 106,000,000/157,054,986 rows (67.5%)
2025-03-03 07:39:42,319 - INFO - Performance: 2589.7 rows/sec overall, 1734.2 rows/sec this chunk
2025-03-03 07:39:42,319 - INFO - Tokens per row: 4.05, Total tokens: 429,123,127
2025-03-03 07:49:35,101 - INFO - Progress: 107,000,000/157,054,986 rows (68.1%)
2025-03-03 07:49:35,102 - INFO - Performance: 2576.8 rows/sec overall, 1597.6 rows/sec this chunk
2025-03-03 07:49:35,102 - INFO - Tokens per row: 4.07, Total tokens: 435,506,601
2025-03-03 07:59:21,109 - INFO - Progress: 108,000,000/157,054,986 rows (68.8%)
2025-03-03 07:59:21,112 - INFO - Performance: 2564.7 rows/sec overall, 1708.6 rows/sec this chunk
2025-03-03 07:59:21,112 - INFO - Tokens per row: 4.09, Total tokens: 442,193,800
2025-03-03 08:09:10,382 - INFO - Progress: 109,000,000/157,054,986 rows (69.4%)
2025-03-03 08:09:10,383 - INFO - Performance: 2552.7 rows/sec overall, 1697.4 rows/sec this chunk
2025-03-03 08:09:10,383 - INFO - Tokens per row: 4.11, Total tokens: 448,367,801
2025-03-03 08:19:16,958 - INFO - Progress: 110,000,000/157,054,986 rows (70.0%)
2025-03-03 08:19:16,959 - INFO - Performance: 2540.0 rows/sec overall, 1699.7 rows/sec this chunk
2025-03-03 08:19:16,959 - INFO - Tokens per row: 4.14, Total tokens: 454,992,237
2025-03-03 08:29:26,703 - INFO - Progress: 111,000,000/157,054,986 rows (70.7%)
2025-03-03 08:29:26,704 - INFO - Performance: 2527.5 rows/sec overall, 1719.1 rows/sec this chunk
2025-03-03 08:29:26,704 - INFO - Tokens per row: 4.16, Total tokens: 461,462,605
2025-03-03 08:39:44,035 - INFO - Progress: 112,000,000/157,054,986 rows (71.3%)
2025-03-03 08:39:44,036 - INFO - Performance: 2515.0 rows/sec overall, 1561.4 rows/sec this chunk
2025-03-03 08:39:44,036 - INFO - Tokens per row: 4.18, Total tokens: 468,069,725
2025-03-03 08:49:38,415 - INFO - Progress: 113,000,000/157,054,986 rows (71.9%)
2025-03-03 08:49:38,415 - INFO - Performance: 2504.0 rows/sec overall, 1701.7 rows/sec this chunk
2025-03-03 08:49:38,415 - INFO - Tokens per row: 4.20, Total tokens: 475,075,775
2025-03-03 08:59:34,079 - INFO - Progress: 114,000,000/157,054,986 rows (72.6%)
2025-03-03 08:59:34,081 - INFO - Performance: 2493.3 rows/sec overall, 1670.4 rows/sec this chunk
2025-03-03 08:59:34,081 - INFO - Tokens per row: 4.23, Total tokens: 482,202,701
2025-03-03 09:09:32,332 - INFO - Progress: 115,000,000/157,054,986 rows (73.2%)
2025-03-03 09:09:32,333 - INFO - Performance: 2482.6 rows/sec overall, 1679.5 rows/sec this chunk
2025-03-03 09:09:32,333 - INFO - Tokens per row: 4.25, Total tokens: 489,091,672
2025-03-03 09:19:41,457 - INFO - Progress: 116,000,000/157,054,986 rows (73.9%)
2025-03-03 09:19:41,458 - INFO - Performance: 2471.7 rows/sec overall, 1511.9 rows/sec this chunk
2025-03-03 09:19:41,459 - INFO - Tokens per row: 4.27, Total tokens: 495,546,889
2025-03-03 09:29:50,728 - INFO - Progress: 117,000,000/157,054,986 rows (74.5%)
2025-03-03 09:29:50,728 - INFO - Performance: 2461.1 rows/sec overall, 1651.7 rows/sec this chunk
2025-03-03 09:29:50,728 - INFO - Tokens per row: 4.29, Total tokens: 502,028,678
2025-03-03 09:39:48,135 - INFO - Progress: 118,000,000/157,054,986 rows (75.1%)
2025-03-03 09:39:48,138 - INFO - Performance: 2451.3 rows/sec overall, 1670.2 rows/sec this chunk
2025-03-03 09:39:48,138 - INFO - Tokens per row: 4.31, Total tokens: 508,915,066
2025-03-03 09:49:45,431 - INFO - Progress: 119,000,000/157,054,986 rows (75.8%)
2025-03-03 09:49:45,433 - INFO - Performance: 2441.8 rows/sec overall, 1656.9 rows/sec this chunk
2025-03-03 09:49:45,433 - INFO - Tokens per row: 4.33, Total tokens: 515,607,483
2025-03-03 09:59:44,346 - INFO - Progress: 120,000,000/157,054,986 rows (76.4%)
2025-03-03 09:59:44,347 - INFO - Performance: 2432.4 rows/sec overall, 1630.8 rows/sec this chunk
2025-03-03 09:59:44,347 - INFO - Tokens per row: 4.35, Total tokens: 521,999,876
2025-03-03 10:09:53,035 - INFO - Progress: 121,000,000/157,054,986 rows (77.0%)
2025-03-03 10:09:53,036 - INFO - Performance: 2422.8 rows/sec overall, 1740.7 rows/sec this chunk
2025-03-03 10:09:53,037 - INFO - Tokens per row: 4.37, Total tokens: 528,535,707
2025-03-03 10:19:50,842 - INFO - Progress: 122,000,000/157,054,986 rows (77.7%)
2025-03-03 10:19:50,845 - INFO - Performance: 2413.9 rows/sec overall, 1670.9 rows/sec this chunk
2025-03-03 10:19:50,845 - INFO - Tokens per row: 4.38, Total tokens: 534,939,087
2025-03-03 10:29:40,441 - INFO - Progress: 123,000,000/157,054,986 rows (78.3%)
2025-03-03 10:29:40,443 - INFO - Performance: 2405.6 rows/sec overall, 1705.5 rows/sec this chunk
2025-03-03 10:29:40,443 - INFO - Tokens per row: 4.41, Total tokens: 542,318,813
2025-03-03 10:39:30,358 - INFO - Progress: 124,000,000/157,054,986 rows (79.0%)
2025-03-03 10:39:30,359 - INFO - Performance: 2397.5 rows/sec overall, 1735.2 rows/sec this chunk
2025-03-03 10:39:30,359 - INFO - Tokens per row: 4.43, Total tokens: 549,139,623
2025-03-03 10:49:11,869 - INFO - Progress: 125,000,000/157,054,986 rows (79.6%)
2025-03-03 10:49:11,870 - INFO - Performance: 2390.0 rows/sec overall, 1683.4 rows/sec this chunk
2025-03-03 10:49:11,870 - INFO - Tokens per row: 4.45, Total tokens: 556,043,383
2025-03-03 10:59:24,839 - INFO - Progress: 126,000,000/157,054,986 rows (80.2%)
2025-03-03 10:59:24,840 - INFO - Performance: 2381.2 rows/sec overall, 1523.8 rows/sec this chunk
2025-03-03 10:59:24,840 - INFO - Tokens per row: 4.47, Total tokens: 563,417,575
2025-03-03 11:09:07,063 - INFO - Progress: 127,000,000/157,054,986 rows (80.9%)
2025-03-03 11:09:07,064 - INFO - Performance: 2374.0 rows/sec overall, 1718.5 rows/sec this chunk
2025-03-03 11:09:07,064 - INFO - Tokens per row: 4.49, Total tokens: 570,832,015
2025-03-03 11:19:28,214 - INFO - Progress: 128,000,000/157,054,986 rows (81.5%)
2025-03-03 11:19:28,215 - INFO - Performance: 2365.2 rows/sec overall, 1584.6 rows/sec this chunk
2025-03-03 11:19:28,215 - INFO - Tokens per row: 4.52, Total tokens: 579,108,466
2025-03-03 11:29:15,655 - INFO - Progress: 129,000,000/157,054,986 rows (82.1%)
2025-03-03 11:29:15,656 - INFO - Performance: 2358.1 rows/sec overall, 1733.1 rows/sec this chunk
2025-03-03 11:29:15,656 - INFO - Tokens per row: 4.56, Total tokens: 587,617,315
2025-03-03 11:39:07,894 - INFO - Progress: 130,000,000/157,054,986 rows (82.8%)
2025-03-03 11:39:07,895 - INFO - Performance: 2350.9 rows/sec overall, 1702.4 rows/sec this chunk
2025-03-03 11:39:07,895 - INFO - Tokens per row: 4.58, Total tokens: 595,468,743
2025-03-03 11:48:42,059 - INFO - Progress: 131,000,000/157,054,986 rows (83.4%)
2025-03-03 11:48:42,060 - INFO - Performance: 2344.7 rows/sec overall, 1769.2 rows/sec this chunk
2025-03-03 11:48:42,060 - INFO - Tokens per row: 4.61, Total tokens: 603,528,051
2025-03-03 11:58:24,160 - INFO - Progress: 132,000,000/157,054,986 rows (84.0%)
2025-03-03 11:58:24,161 - INFO - Performance: 2338.2 rows/sec overall, 1821.7 rows/sec this chunk
2025-03-03 11:58:24,161 - INFO - Tokens per row: 4.63, Total tokens: 611,316,521
2025-03-03 12:12:48,828 - INFO - Progress: 133,000,000/157,054,986 rows (84.7%)
2025-03-03 12:12:48,832 - INFO - Performance: 2320.4 rows/sec overall, 1373.5 rows/sec this chunk
2025-03-03 12:12:48,832 - INFO - Tokens per row: 4.65, Total tokens: 618,539,528
2025-03-03 12:23:28,169 - INFO - Progress: 134,000,000/157,054,986 rows (85.3%)
2025-03-03 12:23:28,170 - INFO - Performance: 2312.0 rows/sec overall, 1635.9 rows/sec this chunk
2025-03-03 12:23:28,170 - INFO - Tokens per row: 4.67, Total tokens: 625,474,963
2025-03-03 12:34:05,058 - INFO - Progress: 135,000,000/157,054,986 rows (86.0%)
2025-03-03 12:34:05,060 - INFO - Performance: 2304.0 rows/sec overall, 1643.7 rows/sec this chunk
2025-03-03 12:34:05,060 - INFO - Tokens per row: 4.69, Total tokens: 632,707,942
2025-03-03 12:44:58,966 - INFO - Progress: 136,000,000/157,054,986 rows (86.6%)
2025-03-03 12:44:58,967 - INFO - Performance: 2295.4 rows/sec overall, 1485.6 rows/sec this chunk
2025-03-03 12:44:58,967 - INFO - Tokens per row: 4.71, Total tokens: 640,014,583
2025-03-03 12:55:55,349 - INFO - Progress: 137,000,000/157,054,986 rows (87.2%)
2025-03-03 12:55:55,350 - INFO - Performance: 2287.0 rows/sec overall, 1603.6 rows/sec this chunk
2025-03-03 12:55:55,350 - INFO - Tokens per row: 4.73, Total tokens: 648,078,742
2025-03-03 13:06:18,095 - INFO - Progress: 138,000,000/157,054,986 rows (87.9%)
2025-03-03 13:06:18,096 - INFO - Performance: 2280.0 rows/sec overall, 1603.1 rows/sec this chunk
2025-03-03 13:06:18,096 - INFO - Tokens per row: 4.75, Total tokens: 655,757,098
2025-03-03 13:17:02,303 - INFO - Progress: 139,000,000/157,054,986 rows (88.5%)
2025-03-03 13:17:02,305 - INFO - Performance: 2272.3 rows/sec overall, 1429.9 rows/sec this chunk
2025-03-03 13:17:02,305 - INFO - Tokens per row: 4.77, Total tokens: 662,907,695
2025-03-03 13:27:21,670 - INFO - Progress: 140,000,000/157,054,986 rows (89.1%)
2025-03-03 13:27:21,672 - INFO - Performance: 2265.7 rows/sec overall, 1629.6 rows/sec this chunk
2025-03-03 13:27:21,672 - INFO - Tokens per row: 4.79, Total tokens: 670,927,804
2025-03-03 13:37:38,952 - INFO - Progress: 141,000,000/157,054,986 rows (89.8%)
2025-03-03 13:37:38,954 - INFO - Performance: 2259.3 rows/sec overall, 1652.4 rows/sec this chunk
2025-03-03 13:37:38,954 - INFO - Tokens per row: 4.82, Total tokens: 678,994,619
2025-03-03 13:47:57,366 - INFO - Progress: 142,000,000/157,054,986 rows (90.4%)
2025-03-03 13:47:57,368 - INFO - Performance: 2253.0 rows/sec overall, 1646.7 rows/sec this chunk
2025-03-03 13:47:57,368 - INFO - Tokens per row: 4.84, Total tokens: 687,717,048
2025-03-03 13:58:29,387 - INFO - Progress: 143,000,000/157,054,986 rows (91.1%)
2025-03-03 13:58:29,388 - INFO - Performance: 2246.4 rows/sec overall, 1402.2 rows/sec this chunk
2025-03-03 13:58:29,388 - INFO - Tokens per row: 4.87, Total tokens: 696,825,876
2025-03-03 14:08:21,252 - INFO - Progress: 144,000,000/157,054,986 rows (91.7%)
2025-03-03 14:08:21,253 - INFO - Performance: 2241.2 rows/sec overall, 1758.0 rows/sec this chunk
2025-03-03 14:08:21,253 - INFO - Tokens per row: 4.90, Total tokens: 705,793,911
2025-03-03 14:17:59,348 - INFO - Progress: 145,000,000/157,054,986 rows (92.3%)
2025-03-03 14:17:59,351 - INFO - Performance: 2236.7 rows/sec overall, 1738.6 rows/sec this chunk
2025-03-03 14:17:59,352 - INFO - Tokens per row: 4.92, Total tokens: 713,885,496
2025-03-03 14:27:45,008 - INFO - Progress: 146,000,000/157,054,986 rows (93.0%)
2025-03-03 14:27:45,009 - INFO - Performance: 2231.9 rows/sec overall, 1749.3 rows/sec this chunk
2025-03-03 14:27:45,009 - INFO - Tokens per row: 4.95, Total tokens: 722,396,569
2025-03-03 14:37:15,225 - INFO - Progress: 147,000,000/157,054,986 rows (93.6%)
2025-03-03 14:37:15,228 - INFO - Performance: 2227.8 rows/sec overall, 1673.7 rows/sec this chunk
2025-03-03 14:37:15,228 - INFO - Tokens per row: 4.97, Total tokens: 731,225,704
2025-03-03 14:46:52,343 - INFO - Progress: 148,000,000/157,054,986 rows (94.2%)
2025-03-03 14:46:52,344 - INFO - Performance: 2223.5 rows/sec overall, 1775.6 rows/sec this chunk
2025-03-03 14:46:52,344 - INFO - Tokens per row: 5.00, Total tokens: 739,689,480
2025-03-03 14:56:09,376 - INFO - Progress: 149,000,000/157,054,986 rows (94.9%)
2025-03-03 14:56:09,377 - INFO - Performance: 2219.9 rows/sec overall, 1782.7 rows/sec this chunk
2025-03-03 14:56:09,378 - INFO - Tokens per row: 5.02, Total tokens: 748,479,844
2025-03-03 15:06:00,626 - INFO - Progress: 150,000,000/157,054,986 rows (95.5%)
2025-03-03 15:06:00,627 - INFO - Performance: 2215.3 rows/sec overall, 1754.9 rows/sec this chunk
2025-03-03 15:06:00,627 - INFO - Tokens per row: 5.05, Total tokens: 757,511,050
2025-03-03 15:15:10,217 - INFO - Progress: 151,000,000/157,054,986 rows (96.1%)
2025-03-03 15:15:10,218 - INFO - Performance: 2212.1 rows/sec overall, 1864.7 rows/sec this chunk
2025-03-03 15:15:10,218 - INFO - Tokens per row: 5.08, Total tokens: 766,498,851
2025-03-03 15:23:59,227 - INFO - Progress: 152,000,000/157,054,986 rows (96.8%)
2025-03-03 15:23:59,228 - INFO - Performance: 2209.7 rows/sec overall, 1949.0 rows/sec this chunk
2025-03-03 15:23:59,228 - INFO - Tokens per row: 5.10, Total tokens: 775,498,870
2025-03-03 15:32:36,538 - INFO - Progress: 153,000,000/157,054,986 rows (97.4%)
2025-03-03 15:32:36,539 - INFO - Performance: 2207.6 rows/sec overall, 1954.6 rows/sec this chunk
2025-03-03 15:32:36,539 - INFO - Tokens per row: 5.13, Total tokens: 784,797,600
2025-03-03 15:41:06,959 - INFO - Progress: 154,000,000/157,054,986 rows (98.1%)
2025-03-03 15:41:06,962 - INFO - Performance: 2205.8 rows/sec overall, 1953.8 rows/sec this chunk
2025-03-03 15:41:06,962 - INFO - Tokens per row: 5.16, Total tokens: 794,334,630
2025-03-03 15:49:34,507 - INFO - Progress: 155,000,000/157,054,986 rows (98.7%)
2025-03-03 15:49:34,508 - INFO - Performance: 2204.1 rows/sec overall, 1985.0 rows/sec this chunk
2025-03-03 15:49:34,508 - INFO - Tokens per row: 5.18, Total tokens: 803,626,071
2025-03-03 15:58:13,984 - INFO - Progress: 156,000,000/157,054,986 rows (99.3%)
2025-03-03 15:58:13,988 - INFO - Performance: 2202.0 rows/sec overall, 1985.0 rows/sec this chunk
2025-03-03 15:58:13,988 - INFO - Tokens per row: 5.21, Total tokens: 812,715,878
2025-03-03 16:06:35,526 - INFO - Progress: 157,000,000/157,054,986 rows (100.0%)
2025-03-03 16:06:35,528 - INFO - Performance: 2200.6 rows/sec overall, 1994.2 rows/sec this chunk
2025-03-03 16:06:35,528 - INFO - Tokens per row: 5.23, Total tokens: 821,416,430
2025-03-03 16:08:27,642 - INFO - Progress: 157,054,986/157,054,986 rows (100.0%)
2025-03-03 16:08:27,642 - INFO - Performance: 2197.9 rows/sec overall, 490.4 rows/sec this chunk
2025-03-03 16:08:27,642 - INFO - Tokens per row: 5.23, Total tokens: 821,886,361
2025-03-03 16:10:15,871 - INFO - Processing complete: 157,054,986 rows processed in 71565.2 seconds
2025-03-03 16:10:15,872 - INFO - Total tokens extracted: 821,886,361
2025-03-03 16:10:15,872 - INFO - Average performance: 2194.6 rows/sec
2025-03-03 16:11:00,126 - INFO - Token extraction pipeline complete
2025-03-03 16:13:52,010 - INFO - Starting Farcaster 'First Said' identification process
2025-03-03 16:20:09,291 - INFO - Starting Farcaster 'First Said' identification process
2025-03-03 16:29:39,763 - INFO - Connected to DuckDB database: farcaster.db
2025-03-03 16:29:39,850 - INFO - Found 821,886,361 tokens in the database
2025-03-03 16:29:39,850 - INFO - Identifying first usage of each word...
2025-03-03 16:29:39,850 - INFO - Creating table of earliest word usage...
2025-03-03 16:31:17,559 - INFO - Starting Farcaster 'First Said' identification process
2025-03-03 16:38:58,652 - INFO - Starting Farcaster 'First Said' identification process
2025-03-03 16:47:59,971 - INFO - Connected to DuckDB database: farcaster.db
2025-03-03 17:27:31,147 - INFO - Starting Farcaster 'First Said' identification process
2025-03-03 17:36:12,646 - INFO - Connected to DuckDB database: farcaster.db
2025-03-03 17:56:48,732 - INFO - Starting Farcaster 'First Said' identification process
2025-03-03 18:05:31,506 - INFO - Connected to DuckDB database: farcaster.db
2025-03-03 18:05:31,588 - INFO - Found 821,886,361 tokens in the database
2025-03-03 18:05:31,588 - INFO - Identifying first usage of each word...
2025-03-03 18:05:31,588 - INFO - Creating table of earliest word usage...