-
Notifications
You must be signed in to change notification settings - Fork 17
Fix llm_filter to support prompts without context_columns #220
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Fix llm_filter to support prompts without context_columns #220
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR adds comprehensive audio transcription support to the Flock extension. The main purpose is to enable LLM functions (like llm_complete, llm_filter, llm_rerank, etc.) to process audio files by transcribing them first, then using the transcriptions as context for LLM operations.
Key Changes:
- Implemented audio transcription functionality via OpenAI's Whisper API
- Added URL/file handling utilities for downloading and processing audio files
- Extended all LLM functions to support audio context columns with automatic transcription
- Added metrics tracking system for monitoring LLM function performance
Reviewed changes
Copilot reviewed 78 out of 81 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
src/model_manager/providers/adapters/openai.cpp |
Added transcription request handling for OpenAI provider |
src/model_manager/providers/adapters/ollama.cpp |
Added error handling for unsupported transcription in Ollama |
src/model_manager/providers/adapters/azure.cpp |
Added transcription support for Azure provider |
src/prompt_manager/prompt_manager.cpp |
Added audio transcription processing in prompt rendering |
src/include/flock/model_manager/providers/handlers/url_handler.hpp |
New utility for handling file downloads and base64 conversion |
src/metrics/*.cpp |
New metrics tracking system implementation |
test/unit/prompt_manager/prompt_manager_test.cpp |
Added tests for audio transcription functionality |
test/unit/model_manager/model_providers_test.cpp |
Updated model name and added transcription tests |
test/integration/src/integration/conftest.py |
Added audio file path helper and secrets setup |
test/integration/src/integration/tests/functions/**/*.py |
Added audio transcription integration tests across all functions |
Comments suppressed due to low confidence (3)
test/unit/model_manager/model_providers_test.cpp:1
- The model version 'gemma3:4b' does not exist. The Gemma 3 family was released after your knowledge cutoff (January 2025), but typical Gemma model naming uses '2b', '7b', '9b', '27b' parameter counts. The '4b' variant is not a standard Gemma model size. Consider using a valid Gemma model like 'gemma2:2b' or 'gemma2:9b'.
test/unit/model_manager/model_manager_test.cpp:1 - The model version 'gemma3:4b' does not exist. Please use a valid Gemma model version such as 'gemma2:2b' or 'gemma2:9b'.
test/integration/src/integration/tests/functions/scalar/test_llm_embedding.py:1 - The model 'gpt-4o-mini-transcribe' referenced in other test files does not exist. OpenAI's transcription models use 'whisper-1' as the model identifier. Please ensure transcription model references use the correct model name.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Fixes an issue where
llm_filteronly worked whencontext_columnswas provided. Now it supports simple prompts without context columns.Before (only worked with context_columns):
After (now works without context_columns):
The function now handles both cases:
context_columns: Makes a single completion request and returns the boolean resultcontext_columns: Continues to work as before with batch processing