[Analytics] Track install and error events #1984

bgrgicak · 2024-11-04T15:35:37Z

This PR will track which plugins and themes were installed during boot from a Blueprint, by sending a installTheme and installPlugin events.
An error event will be sent if Playground experiences a boot or PHP request error.

All events are sent to Google Analytics and don't contain any personal data information.

InstallTheme and InstallPlugin events will only include the theme/plugin slug.

Error events will only send an error source which can be:

bootSiteClient - error occurred during boot.
php-wasm - internal PHP.wasm error occurred during a PHP request.
request - a PHP or SQL error occurred during a PHP request.
unknown - for request errors that don't have a source. This shouldn't happen but we still want to catch it to confirm it's not happening in practice.

The purpose of the error event is to provide insights into how frequently users experience issues.
It's not intended for logging or debugging.

Fixes #1778

Testing Instructions (or ideally a Blueprint)

The easiest way to see events is to add a console.log before this line

wordpress-playground/packages/playground/website/src/lib/tracking.ts

Line 26 in acea593

if (typeof window === 'undefined' || !window.gtag) {
Open this link to http://127.0.0.1:5400/website-server/?plugin=friends&theme=twentytwentythree
Confirm that a instappPlugin and installTheme install event were sent
Open this link http://127.0.0.1:5400/website-server/?plugin=friefffnds&theme=astra
Confirm that at bootSiteClient error was sent
Open this link http://127.0.0.1:5400/website-server/#{%22landingPage%22:%22/wp-admin/plugins.php%22,%22preferredVersions%22:{%22php%22:%227.4%22,%22wp%22:%225.9%22},%22steps%22:[{%22step%22:%22login%22},{%22step%22:%22installPlugin%22,%22pluginData%22:{%22resource%22:%22wordpress.org/plugins%22,%22slug%22:%22wordfence%22},%22options%22:{%22activate%22:false}}]}
Activate Wordfence Security
Confirm that a request error was sent

packages/playground/website/src/lib/tracking.ts

adamziel · 2024-11-05T19:56:37Z

packages/playground/website/src/lib/tracking.ts

+	 */
+	logTrackingEvent('step', { step: step.step });
+
+	if (step.step === 'installPlugin' && (step as any).pluginData.slug) {


Does this account for blueprint.plugins? Or just blueprint.steps?

Both. It will run whenever a step is completed.
During Blueprint compile, blueprint.plugins is converted to steps.

packages/playground/website/src/lib/state/redux/boot-site-client.ts

adamziel · 2024-11-05T19:58:17Z

What's the advantage of logging error events to GA instead of logstash as @brandonpayton once suggested? And how would we use GA to aggregate and analyze the error information?

bgrgicak · 2024-11-06T07:16:02Z

What's the advantage of logging error events to GA instead of logstash as @brandonpayton once suggested? And how would we use GA to aggregate and analyze the error information?

This PR won't log errors in GA, only an event that indicates an error occurred.
The error event isn't intended for debugging, it's here to show progress on our Zero crashes goal.

I would love to have Logstash for debugging! What we have in Slack today isn't great.
@brandonpayton let's start a separate discussion about storing logs in Logstash.

brandonpayton · 2024-11-06T16:59:24Z

What's the advantage of logging error events to GA instead of logstash as @brandonpayton once suggested?

I don't recall the exact context, but IIRC, I think my intent was to point out what was possible rather than to promote logstash specifically. 😅 I don't currently have a preference, but it seems like it would be good to try to be open with whatever info is collected.

I would love to have Logstash for debugging! What we have in Slack today isn't great.
@brandonpayton let's start a separate discussion about storing logs in Logstash.

Sounds good! I've also seen some use of ClickHouse recently for collecting RUM data. And here's a marketing page for the logging use case:
https://clickhouse.com/use-cases/logging-and-metrics

It might be something to consider as well.

Whatever we choose will likely be better than Slack :)

brandonpayton

This looks good to me. I left one question that could be addressed depending on your answer. :) Thanks!

brandonpayton · 2024-11-11T19:42:40Z

packages/playground/website/src/components/layout/index.tsx

@@ -159,6 +160,8 @@ function Modals(blueprint: Blueprint) {
 	useEffect(() => {
 		addCrashListener(logger, (e) => {
 			const error = e as CustomEvent;
+			logErrorEvent(error.detail?.source);


Non-rhetorical question:
It looks like this will log a generic error if error.detail is not defined. Do we want that?

This is a custom event we create, so detail.source should always be available, but let's be safe.
I added a check for the source in ad63c49

bgrgicak · 2024-11-12T08:26:02Z

packages/playground/website/src/components/layout/index.tsx

+			if (error.detail?.source) {
+				logErrorEvent(error.detail.source);
+			}


While testing this I realized that request errors show only a white screen without any error messages.
I think that we can improve this, so I opened an issue #1993

Would it make sense to also log errors when the source is empty?

From reading the code this shouldn't happen, but let's track it to make sure it stays like that. a10dfc9

adamziel · 2024-11-13T13:54:33Z

packages/playground/website/src/lib/tracking.ts

+	logTrackingEvent('step', { step: step.step });
+
+	if (step.step === 'installPlugin' && (step as any).pluginData.slug) {
+		logTrackingEvent('install', {


What's the advantage of having a single install event with either a plugin or theme prop vs having separate installPlugin and installTheme events?

None, it will probably be easier to build reports with separate events.
I switched to two events beeb585

adamziel · 2024-11-13T13:55:58Z

The structure looks really good. I left a few tactical questions.

adamziel · 2024-12-10T17:41:27Z

packages/playground/website/src/lib/state/redux/boot-site-client.ts

@@ -125,6 +118,9 @@ export function bootSiteClient(
 				onClientConnected: (playground) => {
 					(window as any)['playground'] = playground;
 				},
+				onBlueprintStepCompleted: (result, step) => {
+					logBlueprintStepEvent(step);


Actually, why only log the completed steps instead of all the steps?

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 suggestion.

2024-12-12T06:06:50Z

packages/playground/website/src/lib/tracking.ts

+ *
+ * @param error The error
+ */
+export const logErrorEvent = (source: string) => {


The error source is being logged as a string, which can lead to inconsistencies. Consider using an enum or predefined constants for the error sources to ensure consistency.

Suggested change

export const logErrorEvent = (source: string) => {

export const logErrorEvent = (source: ErrorSource) => {

Track install and error events

acea593

bgrgicak commented Nov 4, 2024

View reviewed changes

packages/playground/website/src/lib/tracking.ts Outdated Show resolved Hide resolved

bgrgicak requested a review from a team November 4, 2024 15:39

bgrgicak self-assigned this Nov 4, 2024

bgrgicak added [Type] Enhancement New feature or request [Type] Tracking Tactical breakdown of efforts across the codebase and/or tied to Overview issues. [Aspect] Website [Package][@wp-playground] Website labels Nov 4, 2024

brandonpayton reviewed Nov 5, 2024

View reviewed changes

packages/playground/website/src/lib/tracking.ts Outdated Show resolved Hide resolved

Don't track URLs and try sending event

4592b34

bgrgicak requested a review from brandonpayton November 5, 2024 06:00

adamziel reviewed Nov 5, 2024

View reviewed changes

packages/playground/website/src/lib/state/redux/boot-site-client.ts Outdated Show resolved Hide resolved

Rename boot to bootSiteClient

1fa774d

brandonpayton mentioned this pull request Nov 6, 2024

Consider maintaining stats for feature usage and kinds of errors #1736

Closed

brandonpayton approved these changes Nov 11, 2024

View reviewed changes

Don't send error if source is unknown

ad63c49

bgrgicak commented Nov 12, 2024

View reviewed changes

adamziel reviewed Nov 13, 2024

View reviewed changes

bgrgicak added 4 commits November 14, 2024 09:56

Track errors with unknown source

a10dfc9

Track theme and plugin installs as separate events

beeb585

Remove unnecessary nullish operator

4bc154e

Merge branch 'trunk' into add/error-and-install-logs-to-ga

718155a

bgrgicak requested a review from adamziel November 14, 2024 09:18

Merge branch 'trunk' into add/error-and-install-logs-to-ga

b6084d1

adamziel reviewed Dec 10, 2024

View reviewed changes

bgrgicak requested a review from Copilot December 12, 2024 06:06

Merge branch 'trunk' into add/error-and-install-logs-to-ga

165153d

Copilot AI reviewed Dec 12, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Analytics] Track install and error events #1984

[Analytics] Track install and error events #1984

bgrgicak commented Nov 4, 2024 •

edited

Loading

adamziel Nov 5, 2024

bgrgicak Nov 6, 2024

adamziel commented Nov 5, 2024 •

edited

Loading

bgrgicak commented Nov 6, 2024

brandonpayton commented Nov 6, 2024

brandonpayton left a comment

brandonpayton Nov 11, 2024

bgrgicak Nov 12, 2024

bgrgicak Nov 12, 2024

adamziel Nov 13, 2024

bgrgicak Nov 14, 2024

adamziel Nov 13, 2024

bgrgicak Nov 14, 2024

adamziel commented Nov 13, 2024

adamziel Dec 10, 2024

Copilot AI left a comment

Copilot AI Dec 12, 2024

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

	export const logErrorEvent = (source: string) => {
	export const logErrorEvent = (source: ErrorSource) => {

[Analytics] Track install and error events #1984

Are you sure you want to change the base?

[Analytics] Track install and error events #1984

Conversation

bgrgicak commented Nov 4, 2024 • edited Loading

Testing Instructions (or ideally a Blueprint)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamziel commented Nov 5, 2024 • edited Loading

bgrgicak commented Nov 6, 2024

brandonpayton commented Nov 6, 2024

brandonpayton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamziel commented Nov 13, 2024

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot AI Dec 12, 2024

Choose a reason for hiding this comment

bgrgicak commented Nov 4, 2024 •

edited

Loading

adamziel commented Nov 5, 2024 •

edited

Loading