Directly apply SAP on outputs without LLM call

[iiLaurens]

Given an output from some language model, is it possible to directly apply the SAP parsing? I'm specifically interested in Python.

This can be useful in situations where there is a propietary API layer on top of an LLM. Obviously I cannot expect BAML to support this. But I can implement the call to the LLMs myself, collect the outputs and post process it with BAMLs SAP

Another situation in which this is useful is when batch processing. First I saturate the LLM and collect outputs, then I can shut down the GPU to save cost and move on to the next stage with BAML.

Any suggestions how to approach this?

[sxlijin]

Howdy! It sounds like there's a specific provider you're using that you want to integrate with - are there more specifics you can offer?

We've tossed around some ideas here, but haven't really found one where we're happy with the usability tradeoff, which is why we haven't done this yet. Your point about supporting batch outputs is a good one though.

[iiLaurens]

It really is just an in-house wrapper around external APIs (think Gemini, Claude and Gpt4). Companies in high risk sectors such as health care and finance will often do this, because there is very stringent security and monitoring requirements and generally none of the open source solutions will tick all boxes. So some internal team will create an API that does not match any of the standards. That's unfortunately what I'm dealing with.

Even more so is that I have hundreds of thousands offline documents that might need processing. I would really need the fastest solution out there, and preferably saturate GPU usage with batch processing so that I don't hog compute resources.

In terms of tradeoffs, I think this is quite a niche feature and some tradeoffs will be acceptable. An user that would need thus should be experienced enough to weigh the scale on whether the tradeoff is worth it, I suppose.