How to send image data to vision models?

[leandrumartin]

What's the appropriate way to use the Foundry Local SDK to send an image to one of the image processing models (particularly in Python)? There don't seem to be any SDK methods surfacing this functionality.

The only way I can think of is to use the Hugging Face Transformers library to create processors and send it to the models that way, treating the models as any other Hugging Face model. That would require getting the folder path of the model weight data, and while that's easy enough to retrieve, it's not directly exposed by anything in the SDK, so it feels...messy.

So is there an official way to send image data to the vision models that's different from what I've just described?

---

EDIT:

Using the Transformers library didn't work because the downloaded model didn't have a preprocessor_config.json in the folder. I'll also add that I've also tried simply formatting the messages in the usual OpenAI format, e.g.

{
  "role": "user",
  "content": [
    {
      "type": "text",
      "text": "Describe this image in one sentence."
    },
    {
      "type": "image_url",
      "image_url": {
        "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
      }
    }
  ]
}

but the model does not access or transform the image. If I take an image, convert it to base64, and have that be the passed image content, then that also doesn't work, and the models only tells me "I see you've pasted some base64 image data, but I can't process anything other than text" (or it just hallucinates).

---

EDIT 2:

It looks like the C++ example for vision input (which is the only language in which a vision input sample is provided, for some reason?) does what I did with the base64 conversion, but routes it through the web service instead of directly feeding it through the "complete chat" function, which is what I was trying to do. I guess I'll try this method with Python and report back whether or not it works.

If that is in fact the only way to add vision input...it should really be documented better than this.

---

EDIT 3:

Didn't work.

I'm guessing that there just isn't a way to send image data yet, despite vision models being available? But the documentation should really clarify this.