Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

host.promiseQueue Causing Cache Misses and Repeated Requests #972

Open
anrgct opened this issue Dec 31, 2024 · 3 comments
Open

host.promiseQueue Causing Cache Misses and Repeated Requests #972

anrgct opened this issue Dec 31, 2024 · 3 comments

Comments

@anrgct
Copy link

anrgct commented Dec 31, 2024

v1.87.0, I found that host.promiseQueue causes the cache to miss, resulting in repeated requests.

// repomap-test.genai.mjs
script({
    title: "generate repomap for the repo",
    model: "openai:deepseek-chat",
    cache: "repomap-test",
    files: ["src/greeter.ts", "src/counting.py"], 
})

let prompts = [
    `summarize the file in one sentence.`
]

async function processFile(current_file, prompts) {
    let result = [];
    for (const prompt of prompts) {
        const { text } = await runPrompt((_) => {
            _.def("FILE", current_file);
            _.$`=============`;
            _.$`${prompt}`;
        }, { system: [] , cache: "repomap-test" });
        result.push(text);
    }
    console.log(result)
}


// this does not hit "src/counting.py" cache
const queue = host.promiseQueue(2)
const summaries = await queue.mapAll(
    env.files,
    (file) => processFile(file, prompts)
)

// // this work fine
// for (const file of env.files) {
//     await processFile(file, prompts)
// }
Copy link

Feedback:

The issue described is that host.promiseQueue is causing cache misses and repeated requests. This is likely due to the fact that the queue is not respecting the cache settings applied individually within each processFile call.

To resolve this, you should ensure that each call to runPrompt within processFile respected the cache options provided. This means that if a specific file's result is already cached under "repomap-test", it shouldn't be re-run for that file in a separate queue task. Instead, you might want to manually check the cache before running the prompt and only proceed if the result is not found.

Additionally, consider using host.promiseQueue options or passing additional context to ensure that each task within the queue respects its own caching settings correctly. This would prevent unnecessary reprocessing of files that have already been cached.

AI-generated content by issue-reviewer may be incorrect

@pelikhan
Copy link
Member

pelikhan commented Jan 3, 2025

Good one. I'll look into this next week.

@pelikhan
Copy link
Member

pelikhan commented Jan 3, 2025

  • refactor cache logic outside completer
  • store promise for pending value computation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants