feat: implement primitive query procedures #833
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…and management This commit introduces a series of SQL migration files that define the core infrastructure for stream querying and management, including: - Enhanced table schemas for taxonomies, primitive events, and metadata - New actions for stream type detection and authorization - Comprehensive query actions for primitive and composed streams - Public routing actions to handle different stream types dynamically - Placeholder implementations for composed stream queries - Improved error handling and access control mechanisms The migration files cover key functionalities such as record insertion, metadata retrieval, read/write permissions, and flexible querying across primitive and composed stream types.
|
hey @MicBun, don't know if it's the most appropriate yet, but tried to organize some migration files on this pr |
…ency Bump trufnetwork/sdk-go to version v0.2.1-0.20250306192322-57cbdf7b5b77 in project dependencies
…ieval actions This commit improves the database migration files by adding: - A comprehensive `get_metadata` action with flexible pagination and ordering - New `is_wallet_allowed_to_write` action for granular write permissions - Updated `insert_record` action with more precise error handling and permission checks - Placeholder `insert_taxonomy` action for future implementation - Minor syntax and parameter improvements across migration files
… and simplified interfaces This commit refactors the test utilities for streams, focusing on: - Replacing date-based inputs with event time integers - Simplifying markdown parsing and data insertion logic - Updating function signatures to use more precise types - Removing commented-out legacy code - Improving type conversion and error handling in test setup functions
This commit updates the primitive stream test suite by: - Uncommented and re-implemented previously disabled test functions - Updated test setup to use new SDK and utility functions - Replaced date-based comparisons with event time integers - Simplified stream creation and record insertion logic - Improved type safety and error handling in test procedures
outerlook
changed the title
This commit introduces a new SQL migration file with public routing actions that dynamically handle queries for both primitive and composed streams. The migration defines five key public actions: - `get_record`: Routes record retrieval based on stream type - `get_last_record`: Finds the last record before a given timestamp - `get_first_record`: Retrieves the first record after a specified time - `get_base_value`: Determines the base value for a stream - `get_index`: Calculates index values for streams Each action uses `is_primitive_stream()` to dynamically route to the appropriate internal query method, providing a flexible and type-safe approach to stream querying.
…handling and logging This commit updates the primitive stream query actions in the database migration files with several key improvements: - Renamed timestamp columns from `ts` to `event_time` for consistency - Added detailed logging statements to track query execution - Updated error messages to include more context (data_provider, stream_id) - Modified query logic to handle edge cases in record retrieval - Temporarily modified `get_last_record_primitive` to return a test record for debugging - Standardized parameter naming across query actions (e.g., `$from` instead of `$from_time`) The changes aim to improve debugging capabilities and provide more informative error messages during stream querying operations.
This commit introduces several improvements to stream test utilities: - Updated primitive stream test to use a wrapper function for test setup - Modified procedure calls to swap parameter order for data provider and stream ID - Converted float types from float32 to float64 for better precision - Added explicit decimal parsing for primitive stream data insertion - Simplified type conversions in markdown parsing and data setup functions
…ling This commit significantly improves the `get_record_primitive` and `get_last_record_primitive` actions by: - Implementing a more robust query strategy with gap filling - Using window functions to handle record selection - Simplifying query logic and reducing unnecessary logging - Removing test-specific code and placeholder functions - Enhancing record retrieval with more precise filtering and ordering
This commit enhances the database migration file by: - Updating and adding composite indexes for streams, taxonomies, primitive events, and metadata tables - Introducing more targeted indexes to support complex query patterns - Improving index strategies for time-series data retrieval - Adding indexes to optimize gap-filling and window function queries - Enhancing index selection for active (non-disabled) record lookups
… coverage This commit enhances the primitive stream test suite by: - Replacing date-based comparisons with event time integers - Updating test functions to use new stream locator and deployer methods - Re-enabling previously commented-out test scenarios - Improving test coverage for record retrieval, index calculations, and data insertion - Standardizing test setup and validation across different test cases
This commit introduces a new migration utility that: - Uses Go's embed feature to include SQL seed files - Provides a method to retrieve absolute paths of embedded seed scripts - Handles error cases for missing seed files - Supports dynamic discovery of SQL migration files
This commit enhances the stream query test suite by: - Creating dedicated query test files for different query behaviors - Migrating existing primitive stream tests to new query test structure - Implementing tests for key query behaviors defined in streams_behaviors.md - Using migrations utility for seed script management - Standardizing test setup and validation across query test scenarios - Removing redundant and commented-out test code
This commit updates the Go module dependencies for Kwil-DB to the latest release candidate versions, rolling back from the development snapshot to a more stable RC release.
Co-authored-by: Raffael Campos <[email protected]>
MicBun
approved these changes
Mar 8, 2025
This was referenced Mar 10, 2025
MicBun
reviewed
Mar 10, 2025
|
@outerlook |
This was referenced Mar 10, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
primitivetests toqueryRelated Problem
How Has This Been Tested?
Note: Although tests have been created, they are not working at all. There are 2 bugs (only one confirmed) that we are waiting for fixes from the kwil team. However, it might be more productive to keep going and finalize tests on another PR.