Make load test more generic for other LM tasks #53
Conversation
|
@dagrayvid Definitely no rush on this, but whenever you have a second could you take a look at this PR and let me know what you think in terms of where this could should/could go? It restructures the current llm-load-test model quite a bit to make the embedding stuff fit, so it might not fit with the vision for llm-load-test. Happy to keep this as a fork otherwise :) |
| RUN git switch $GIT_BRANCH | ||
| RUN pip3 install -r requirements.txt | ||
|
|
||
| CMD python3 load_test.py -c $LLM_LOAD_TEST_CONFIG -log $LLM_LOAD_TEST_LOG_LEVEL |
There was a problem hiding this comment.
Can you add here an ENV for the output files?
There was a problem hiding this comment.
Not 100% sure what you mean, but the path for the output files should be included in the llm-load-test config file
|
@ccamacho @drewrip for now, I'm thinking that testing embedding models is out-of-scope for llm-load-test, and this should remain as a separate fork for now. One reason is that it requires making the output processing "pluggable", which is otherwise unnecessary so far. Another reason is that I think a tool made specifically for embedding tasks would probably have a different type of dataset. Does that make sense? Open to more discussion on this |
In order to better use llm-load-test to evaluate embedding tasks I had to chop the code up a bit. I don't think this even necessarily should or needs to be merged, but I wanted to open this for discussion and publicize this code as I work on it.