Need the option to mask the input and output of the LLM API in Datadog LLM observability

[Gekko0114]

The input and output of the LLM API contain sensitive information, so I don't want to send them to Datadog.
I would like to have an option to send data to Datadog with the input and output masked.
If it's okay, I want to create a PR for this.

Previously I raised the same question in this issue #10517 .
In this issue, I heard that APM integration dashboards should be used for these four providers (OpenAI/Bedrock/LangChain/Anthropic).
However, I would like to monitor Gemini.
According to #10971, Gemini will not be supported in APM integration because LLM obs will support Gemini.

Therefore I need an option to mask the input and output of the LLM API in Datadog LLM observability.
If it is okay, I would like to create a PR for this.

[Gekko0114]

Hi @Yun-Kim
Gently ping

[Gekko0114]

Hi,
It would be nice if you could give me any feedback.

[Kyle-Verhoog]

Hi @Gekko0114! Are you using VertexAI to use Gemini? If so, we have an integration coming in the next few weeks which will include controls to omit the input and output. Also, we're looking for people to try the integration as we develop it, would you be interested in partnering with us for this?

[Gekko0114]

Are you using VertexAI to use Gemini? If so, we have an integration coming in the next few weeks which will include controls to omit the input and output

Yes, I am using VertexAI. Sounds great!

Also, we're looking for people to try the integration as we develop it, would you be interested in partnering with us for this?

No problem! What should I do?

[Kyle-Verhoog]

Awesome, @Gekko0114 are you in our public Slack? If not could you join and we can follow up with you once we have a build ready for you to try!

[Gekko0114]

Hi @Kyle-Verhoog
Thanks, I've just joined the slack, but I couldn't find you because this channel has a lot of people named Kyle.
Which channel should I join? And could you let me know your account name?

[yj-ang]

Not sure if it's off topic. I tried with setting env DD_LANGCHAIN_SPAN_PROMPT_COMPLETION_SAMPLE_RATE="0.0" stated in the LangChain integration but I could still see all the content (input and output) visible under the LLM Obs Traces dashboard.

I want to collect the metrics but hide the input completely or having option to mask the sensitive information like phone number and email, I wonder if we could get flexibility to control over that.

[Kyle-Verhoog]

@Gekko0114 you should be able to find me under "Kyle Verhoog", send me a message!

[Kyle-Verhoog]

@yj-ang this sounds like a bug 🤔. I will do some investigation and get back to you.