-
Notifications
You must be signed in to change notification settings - Fork 427
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge remote-tracking branch 'origin/master' into built-mobile-replay
- Loading branch information
Showing
313 changed files
with
8,191 additions
and
3,981 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,202 @@ | ||
| --- | ||
| date: 2024-10-03 | ||
| title: Did you know AI is answering our community questions? | ||
| rootPage: /blog | ||
| featuredImage: >- | ||
| https://res.cloudinary.com/dmukukwp6/image/upload/max_ai_poster_b829564bbf.png | ||
| featuredImageType: full | ||
| author: | ||
| - cory-watilo | ||
| category: PostHog news | ||
| --- | ||
|
|
||
| AI. You may have heard of it. | ||
|
|
||
| Sure, ChatGPT is pretty cool, but when it comes to AI chatbots that try to replace a human in a product support context, I've generally been underwhelmed. The experience generally isn't great, and I'm often left with more frustration than before I skeptically asked the question in the first place. | ||
|
|
||
| But it's not all the fault of LLMs. It also has to do with the available information about a product that an LLM consumes. And for a technical product like PostHog, the documentation has to be extensive enough to answer very nuanced questions. (This is tricky as information is constantly changing and there's always room for improvement!) | ||
|
|
||
| As the ~~product owner~~ _totally official_ "webmaster" of PostHog.com, I've contemplated how we can make use of AI to help answer questions... and until now, I've lacked a good solution. | ||
|
|
||
| ## Why AI chatbots have a bad rap | ||
|
|
||
| There are a lot of _"wrong"_ ways to do AI chatbots – and there are many examples of this even before we collectively started using true AI. | ||
|
|
||
| How many times have you tried to create a support ticket but then: | ||
|
|
||
| - been first prompted to review a list of "related questions" that may solve your issue | ||
| - been _blocked_ from submitting until you explicitly confirm that none of the suggested answers are helpful | ||
|
|
||
| And now in a world of _actual_ AI, we see a new set of issues – largely centered around LLMs not understanding the question, the context, or straight up hallucinating. | ||
|
|
||
| In no world would we want to subject our very technical audience to these types of dreaded experiences. | ||
|
|
||
| This isn't to say _all_ AI chatbots are bad. A couple of exceptions come top of mind mainly Intercom and Shopify. But the status quo that the industry has to overcome is the negative connotation of historically bad chatbot interactions. | ||
|
|
||
| ## Our approach | ||
|
|
||
| The reason we've been so cautious to introduce AI is that we're very intentional about earning (and not losing) the trust of our audience. We do this in a handful of ways: | ||
|
|
||
| 1. We don't use fancy words that don't add actual value | ||
| 1. We try to be relatable and show a personality | ||
| 1. We traded in the [value-based sales model](/sales) for a no-BS approach (self-service, fair, usage-based pricing) | ||
| 1. We avoid gimmicky marketing tactics (like popups and jamming CTAs into every available whitespace) | ||
|
|
||
| So for an AI solution to mesh well with our brand's vibe (yes, we're about the vibes), it needs to be accurate and succinct in its replies, and never make up a false answer. | ||
|
|
||
| After all, if you interact with an AI chatbot and don't get the answer you're looking for, you're likely to write it off entirely. And that's exactly what I want to avoid. | ||
|
|
||
| ## Where we decided to implement AI | ||
|
|
||
| The most natural place to include an AI chatbot would be in our [docs](/docs). But I was hesitant to start here, as the conversational nature of AI meant we'd lack control over the quality and accuracy of the responses. So I decided to try something different. | ||
|
|
||
| Our [community questions](/questions) section is where users can ask questions directly in the docs. And we currently have... a lot of open (unanswered) questions. But because they're posted in a forum-style format, there's no expectation of a live chat experience. And this presented an opportunity... | ||
|
|
||
| Today customers may wait hours (or even longer) for a response from a human. **So what if AI could answer _only_ the questions it was highly confident it could answer, then leave the rest for a human?** | ||
|
|
||
| Using this approach, AI can _add_ value by instantly responding if it has a high degree of confidence, but it won't _detract_ from the experience if it doesn't have the solution, simply leaving those questions open as they normally would. | ||
|
|
||
| ## Introducing Inkeep | ||
|
|
||
| This whole idea came about from an [Inkeep](https://inkeep.com?utm_source=posthog) ad on LinkedIn – which is incredible to say because I'm generally allergic to ads. But Inkeep's ad spoke to me. | ||
|
|
||
| <ProductScreenshot | ||
| imageLight = "https://res.cloudinary.com/dmukukwp6/image/upload/inkeep_ad_9b53d43fda.png" | ||
| classes="rounded" | ||
| alt="Inkeep's LinkedIn ad" | ||
| /> | ||
|
|
||
| I took the bait, and they quickly set me up with [a custom sandbox](https://share.inkeep.com/posthog/3003538eb7fd) to try them out. They're indexing our docs, tutorials, blog posts, handbook, existing community answers, and even GitHub repos. (This also means it's aware of bug reports, feature requests, and open pull requests - _huge!_) | ||
|
|
||
| <ProductScreenshot | ||
| imageLight = "https://res.cloudinary.com/dmukukwp6/image/upload/inkeep_sandbox_4ca8513bf0.png" | ||
| classes="rounded" | ||
| alt="Inkeep sandbox" | ||
| /> | ||
|
|
||
| They even themed the sandbox to match our brand. <3 | ||
|
|
||
| <ProductScreenshot | ||
| imageLight = "https://res.cloudinary.com/dmukukwp6/image/upload/inkeep_email_f22b3b7112.png" | ||
| classes="rounded" | ||
| alt="Inkeep email" | ||
| /> | ||
|
|
||
| To vet the quality of the answers, I started copy/pasting unanswered community questions into the sandbox and seeing what Inkeep could produce. And to say I wasn't let down should tell you something. It was good enough to where I was willing to try it out on our real questions. | ||
|
|
||
| ## How it works | ||
|
|
||
| > Inkeep's pre-built library of UI components works great for most companies. But out of the kindness of their hearts, they went the extra mile for us. | ||
| **Here's how our setup works:** | ||
|
|
||
| 1. When a new question is posted, we send it to Inkeep's LLM. | ||
| 1. Typically within ~20 seconds, we get a response. During this time, we show that we're searching for an answer. | ||
| <ProductScreenshot | ||
| imageLight = "https://res.cloudinary.com/dmukukwp6/image/upload/inkeep_in_progress_5599f03642.png" | ||
| classes="rounded" | ||
| alt="AI answer in progress" | ||
| /> | ||
| 1. In the response, we get a `confidence` score. | ||
|
|
||
| ``` | ||
| const KnownAnswerConfidence = z.enum([ | ||
| "very_confident", | ||
| "somewhat_confident", | ||
| "not_confident", | ||
| "no_sources", | ||
| "other", | ||
| ]); | ||
| ``` | ||
|
|
||
| <p><strong className="text-green">If the confidence score is high enough,</strong> we show the answer to the user, credited to our hedgehog mascot, <TeamMember name="Max AI" photo />.</p> | ||
| <p><strong className="text-red">If Inkeep <em>isn't</em> confident in the answer,</strong> we show a message that says we couldn't find an answer, and that a human will appear as soon as possible.</p> | ||
|
|
||
| When an AI answer is presented, we show feedback buttons so the original poster can tell us if the answer was helpful. We use this information to improve the quality of the answers. | ||
|
|
||
| <ProductScreenshot | ||
| imageLight = "https://res.cloudinary.com/dmukukwp6/image/upload/inkeep_review_6a6510b750.png" | ||
| classes="rounded" | ||
| alt="How's the AI answer?" | ||
| /> | ||
|
|
||
| ## Demo | ||
|
|
||
| Here's what it looks like to submit a question and receive an AI answer from Inkeep. | ||
|
|
||
| <ProductVideo | ||
| videoLight= "https://res.cloudinary.com/dmukukwp6/video/upload/inkeep_demo_71fa820a65.mp4" | ||
| alt="Inkeep answers a PostHog question" | ||
| classes="rounded shadow-xl" | ||
| autoPlay={false} | ||
| /> | ||
|
|
||
| ## How we're tracking feedback | ||
|
|
||
| As you can imagine, every time a customer rates a response, we're sending the information into PostHog as a custom event. Along with the event, it sends properties like the sentiment ("helpful" or "not helpful") and the category of the question. (This will help us identify if certain areas of our documentation needs to improve.) | ||
|
|
||
| <ProductScreenshot | ||
| imageLight = "https://res.cloudinary.com/dmukukwp6/image/upload/inkeep_event_957c7b340d.png" | ||
| classes="rounded bg-light dark:bg-dark" | ||
| alt="Feedback event" | ||
| /> | ||
|
|
||
| Now that this is live, we're monitoring feedback in Slack. PostHog's recently introduced support for [realtime destinations](https://posthog.com/changelog/2024#realtime-destinations-now-in-open-beta) in data pipelines. | ||
|
|
||
| <ProductScreenshot | ||
| imageLight = "https://res.cloudinary.com/dmukukwp6/image/upload/inkeep_slack_config_b865638953.png" | ||
| classes="rounded" | ||
| alt="Realtime Slack notification config" | ||
| /> | ||
|
|
||
| This allowed me to configure notifications when we get feedback about an AI answer. | ||
|
|
||
| <ProductScreenshot | ||
| imageLight = "https://res.cloudinary.com/dmukukwp6/image/upload/inkeep_slack_973620665a.png" | ||
| classes="rounded" | ||
| alt="Slack notification" | ||
| /> | ||
|
|
||
| ## So how's it working? | ||
|
|
||
| Overall, it's working pretty well! In fact, it worked so well [in one case](/questions/autocapture-event-bubbling) that the original poster replied to share his delight. | ||
|
|
||
| <ProductScreenshot | ||
| imageLight = "https://res.cloudinary.com/dmukukwp6/image/upload/positive_reply_1_ce48f0fc3a.png" | ||
| classes="rounded" | ||
| alt="Customer approved" | ||
| /> | ||
|
|
||
| In other examples, we were receiving feedback that the answer _wasn't_ helpful. But after reviewing the response, the best way to accurately summarize the bulk of feedback is: "The answer is _correct,_ but I didn't _like_ the answer." (This is usually the case when a feature isn't yet supported and it's not the answer the customer was hoping for.) | ||
|
|
||
| This resulted in a lower overall success rate. So we modified the response options to better understand the different reasons why a response might not meet the mark. | ||
|
|
||
| <ProductScreenshot | ||
| imageLight = "https://res.cloudinary.com/dmukukwp6/image/upload/ai_responses_cc819a5877.png" | ||
| classes="rounded" | ||
| alt="Response options" | ||
| /> | ||
|
|
||
| With these additional feedback options, we're able to use it to inform areas for improvement. For example, if someone says, _"I think this is a bug",_ we'll take a closer look. Or if they choose the _"Answer is wrong"_ option, we'll take a look into what might be missing from the documentation. | ||
|
|
||
| It's also won over some internal support from developers on the PostHog team: | ||
|
|
||
| <ProductScreenshot | ||
| imageLight = "https://res.cloudinary.com/dmukukwp6/image/upload/positive_reply_2_90d983c0e5.png" | ||
| classes="rounded" | ||
| alt="Developer approved" | ||
| /> | ||
|
|
||
| --- | ||
|
|
||
| We're still early in our days of exploring AI – and don't worry, AI _is_ coming to the PostHog app when we feel it's ready. | ||
|
|
||
| But I couldn't give you a better rationale for a slow rollout than how a colleague explained it to me: | ||
|
|
||
| <ProductScreenshot | ||
| imageLight = "https://res.cloudinary.com/dmukukwp6/image/upload/anonymous_colleague_ec9787d2ed.png" | ||
| classes="rounded" | ||
| alt="Anonymous quote" | ||
| /> | ||
|
|
||
| Hopefully the next time you [ask a question](/questions) in our community, you'll get the answer you're looking for. And even better – if you get it even faster than a human could answer. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.