Better LLM assistance for Statamic and Antlers syntax?

[jensolafkoch]

I’m curious about your experiences with current LLMs like Claude 3.5 Sonnet or GPT-4o in relation to Statamic and its Antlers syntax, particularly when it comes to tags, modifiers, and inline PHP.

It seems that Statamic-specific material is sparse in the training data, as one might expect. Do you know of any models that are better suited or specifically trained for Statamic and Antlers? Alternatively, are you aware of any initiatives aimed at fine-tuning LLMs for this purpose?

There’s always the option of injecting such knowledge into the context manually—has anyone explored or shared their experiences with this approach?

BTW, I mostly use ChatGPT Plus and Codeium in PhpStorm, as well as experimenting with the new and promising Windsurf editor from Codeium.

[edalzell]

We tried to feed the Statamic docs into an LLM, but produced wrong results very often, so we stopped trying.

What have you tried searching for in our docs that you haven't found?

[jensolafkoch]

It’s not about not finding information in the docs—I do. 🙂

I’m looking for a way to get better LLM results when dealing with Antlers. Automated workflows, especially in Codeium Windsurf, are quite powerful, at least for PHP/Laravel and broadly used tech stacks, but not for Antlers.

I tried today to give the LLM a system message to always check the docs (the locally cloned docs repo, that is) before generating Antlers code, but the results are suboptimal, to say the least. 🙂

In what way did you try to feed the Statamic docs? As full context, as instructions to search the docs online, or via fine-tuning a model?