Error handling (#4)

* remove logging

d'oh.

* reimplement credential validation

previously, we validated credentials by using /import ... this was not an effectively strategy for mp => sheets as consumer permissions can still view reports.

now auth checks should give clear error messages if the credentials are bad.

* standardize tracking model

* serialize errors for tracking

* don't assume time is valid

* add help docs to UI

* surfacing errors to user

* language tweak

* handle status codes

better handling of status codes for mp => sheet; a few UI odds + ends

* tests bad input errors

* vocabulary

* docs refresh

good documentation... finally.

Author: ak--47

Code.js

@@ -31,25 +31,63 @@ each UI has a simple user interface, and is essentially a form you fill out that
 
 ## 🗺️ mappings (sheet → mixpanel)
 
-choose the type of data you are importing and then use the visual mapper to connect the events in your **currently active** spreadsheet to the **required fields** for the type of mixpanel data you are importing: 
+sheet → mixpanel queries your **currently active sheet** to get your sheet's **column headers**.
 
-<img src="https://aktunes.neocities.org/sheets-mixpanel/sheet-to-mp-2.png">
+once you choose the type of data you are importing, you will use the visual mapper to connect the **column headers** from your sheet to the **required fields** for the type of mixpanel data you are importing: 
+
+<img src="https://aktunes.neocities.org/sheets-mixpanel/mappings.png">
+
+as a brief summary of [the documentation](https://developer.mixpanel.com/docs/data-structure-deep-dive#anatomy-of-an-event) mixpanel's data model for events requires fields for:
+- **event name** :  what to call each event in mixpanel
+- **distinct_id** : the unique user identifier to whom the event is attributed
+- **time** : a valid *date* or *time*; if the sheet recognizes your chosen column as a 'date' or time', it should work as intended
+- **insert_id** : a value used to deduplicate records (optional) 
 
 all other columns in your spreadsheet will get sent as **properties** (event, user, or group)
 
-finally, provide some project information and authentication details.
+you'll also need to provide :
+- [project id](https://help.mixpanel.com/hc/en-us/articles/115004490503-Project-Settings#project-id)
+- [project token](https://help.mixpanel.com/hc/en-us/articles/115004490503-Project-Settings#project-token)
+- [project region](https://help.mixpanel.com/hc/en-us/articles/360039135652-Data-Residency-in-EU#enabling-eu-residency)
+- either: 
+	- [service account](https://developer.mixpanel.com/reference/service-accounts) (admin or higher)
+	OR
+	- [API secret](https://help.mixpanel.com/hc/en-us/articles/115004490503-Project-Settings#api-secret)
+
+note: since v1.12 **syncs** are **not supported** for events.
+
+next, read about [runs + syncs](#sync)
 
 
 <div id="export"></div>
 
 ## 💽 exports (mixpanel → sheet)
+mixpanel → sheet queries your mixpanel project for a **report** or **cohort** and makes the results available in a new sheet.
+
+note that this will be identical to the functionality would get when exporting a CSV file from the mixpanel UI:
+
+<img src="https://aktunes.neocities.org/sheets-mixpanel/exportcsv.png">
+
+there are a number of parameters needed to fetch a CSV from mixpanel; the simplest way to gather those parameters is to paste the URL of the report/cohort you wish to sync from your mixpanel project, and the app should find them:
 
-provide authentication details along with a URL of a report or cohort from your mixpanel project that you wish to sync:
-<img src="https://aktunes.neocities.org/sheets-mixpanel/mp-to-sheet-2.png">
+<img src="https://aktunes.neocities.org/sheets-mixpanel/urlMapper.gif">
 
-the UI will try to resolve all the relevant Ids; in case it cannot, you can add in your information to specify the particular report you want to sync
+in case the URL does not contain all the values you need, the UI requires:
+- a [service account](https://developer.mixpanel.com/reference/service-accounts) (consumer or higher) 
+- a URL with either `mixpanel.com` or `eu.mixpanel.com` (to resolve data residency)
+- [your project id](https://help.mixpanel.com/hc/en-us/articles/115004490503-Project-Settings#project-id)
+- [your workspace id](https://developer.mixpanel.com/reference/query-api-authentication#:~:text=Projects%20with%20Data,a%20request%20parameter.)
 
-only **report** and **cohort** syncs are supported. currently, you can not sync flows reports.
+- and either:
+	- [your report id](https://developer.mixpanel.com/reference/insights-query#:~:text=The%20ID%20of%20your%20Insights%20report%20can%20be%20found%20from%20the%20url%3A%20https%3A//mixpanel.com/report/1/insights%23report/%3CYOUR_BOOKMARK_ID%3E/example%2Dreport) 
+	OR 
+	- [your cohort id](https://developer.mixpanel.com/reference/cohorts-list)
+
+you can manually type these values in after pasting in a URL.
+
+note: as of v1.12 **insights**, **funnels**, & **retention** are the only supported reports
+
+next, read about [runs + syncs](#sync)
 
 
 <div id="sync"></div>
@@ -63,10 +101,18 @@ each UI has a similar user interface for you to input your details with **four**
 - **Run**: run the current configuration **once**; results are display in the UI
  - **Sync**: run the current configuration **every hour**; run receipts are stored in a log sheet
  - **Save**: store the current configuration
- - **Clear**: delete document syncs and delete the current configuration
+ - **Clear**: delete this sheet's sync and reload the UI
 
 you may only have **one sync** active per sheet at a time.
 
+if you are planning to sync data from your sheet to mixpanel, it is recommended that you do a "run" first.
+
+once created, syncs will run on an hourly schedule; they can _also_ be manually triggered from the main menu by choosing **Sync Now!**:
+
+<img src="https://aktunes.neocities.org/sheets-mixpanel/sync%20now.png">
+
+note: since v1.12 **syncs** are **not supported** for events.
+
 
 
 <div id="dev"></div>
@@ -224,6 +270,6 @@ no other sensitive scopes are requested by the application.
 
 ## 💬 motivation
 
-google sheets are databases. mixpanel is a database. it should be easy to make these things interoperable. 
+google sheets are databases. mixpanel is a database. it should be easy to make these things interoperable. now it is!
 
 that's it for now. have fun!

README.md

@@ -11,11 +11,10 @@ DATA OUT OF MP
  * @returns {[string, ReportMeta | CohortMeta]} string + metadata `[csv, {}]`
  */
 function exportData(config) {
-    const startTime = Date.now();
-    const runId = Math.random();
     //use last known config if unset
-    //@ts-ignore
     if (!config) config = getConfig();
+
+    //@ts-ignore
     if (!config.auth) config.auth = validateCreds(config);
 
     const type = config.entity_type;
@@ -55,7 +54,9 @@ function getParams(config) {
     const { project_id, workspace_id, region, report_id, auth } = config;
     let subdomain = ``;
     if (region === "EU") subdomain = `eu.`;
-    const URL = `https://${subdomain}mixpanel.com/api/app/workspaces/${Number(workspace_id)}/bookmarks/${Number(report_id)}?v=2`;
+    const URL = `https://${subdomain}mixpanel.com/api/app/workspaces/${Number(workspace_id)}/bookmarks/${Number(
+        report_id
+    )}?v=2`;
 
     /** @type {GoogleAppsScript.URL_Fetch.URLFetchRequestOptions} */
     const options = {
@@ -67,21 +68,41 @@ function getParams(config) {
         muteHttpExceptions: true
     };
 
-    const res = UrlFetchApp.fetch(URL, options).getContentText();
-    const data = JSON.parse(res);
+    const res = UrlFetchApp.fetch(URL, options);
+    const statusCode = res.getResponseCode();
+    switch (statusCode) {
+        case 200:
+            //noop
+            break;
+        case 404:
+			throw `the report ${report_id || ""} could not be found; check your project, workspace, and report id's and try again`
+            break;
+		case 429:
+			throw `your project has been rate limited; this should resolve by itself`
+			break;
+		case 410:
+			throw ``
+		case 500:
+			throw `mixpanel server error; report ${report_id || ""} may no longer exist`;
+        default:
+			throw `an unknown error has occurred when getting report ${report_id || ""}'s parameters`;
+            break;
+    }
+    const text = res.getContentText();
+
+    const data = JSON.parse(text).results;
     const result = {
         meta: {
-            report_type: data?.results?.type || data?.results?.original_type,
-            report_name: data?.results?.name,
-            report_desc: data?.results?.description,
-            report_id: data?.results?.id,
-            project_id: data?.results?.project_id,
-            dashboard_id: data?.results?.dashboard_id,
-            workspace_id: data?.results?.workspace_id,
-            report_creator:
-                data?.results?.creator_name || data?.results?.email || data?.results?.creator_id || "unknown"
+            report_type: data.type || data.original_type,
+            report_name: data.name,
+            report_desc: data.description,
+            report_id: data.id,
+            project_id: data.project_id,
+            dashboard_id: data.dashboard_id,
+            workspace_id: data.workspace_id,
+            report_creator: data.creator_name || data.email || data.creator_id || "unknown"
         },
-        payload: data?.results?.params
+        payload: data.params
     };
 
     return result;
@@ -100,15 +121,17 @@ function getReportCSV(report_type, params, config) {
     if (region === "EU") subdomain = `eu.`;
 
     if (!["insights", "funnels", "retention"].includes(report_type)) {
-        throw `${report_type} reports are not currently supported for CSV export`;
+        throw `${report_type || "your supplied"} report is not currently supported for CSV export`;
     }
 
     let route = report_type;
     if (report_type === "funnels") {
         route = `arb_funnels`;
     }
 
-    const URL = `https://${subdomain}mixpanel.com/api/query/${route}?workspace_id=${Number(workspace_id)}&project_id=${Number(project_id)}`;
+    const URL = `https://${subdomain}mixpanel.com/api/query/${route}?workspace_id=${Number(
+        workspace_id
+    )}&project_id=${Number(project_id)}`;
     const payload = {
         bookmark: params,
         use_query_cache: false,
@@ -121,7 +144,7 @@ function getReportCSV(report_type, params, config) {
         headers: {
             Authorization: `Basic ${auth}`
         },
-        muteHttpExceptions: true,
+        muteHttpExceptions: false,
         payload: JSON.stringify(payload)
     };
 
@@ -204,7 +227,7 @@ function getCohortMeta(config) {
             Authorization: `Basic ${auth}`,
             Accept: `application/json`
         },
-        muteHttpExceptions: true
+        muteHttpExceptions: false
     };
 
     const res = JSON.parse(UrlFetchApp.fetch(URL, options).getContentText());

components/dataExport.js

@@ -12,14 +12,12 @@ DATA INTO MP
  * @returns {[ImportResponse[], ImportResults]}
  */
 function importData(config, sheet) {
-    const runId = Math.random();
     //use last known config if unset
-    //@ts-ignore
     if (!config) config = getConfig();
+    
+	//@ts-ignore
     if (!config.auth) config.auth = validateCreds(config);
-    const { record_type } = config;
 
-    // console.log('SYNC');
     const startTime = Date.now();
     let endTime;
     const mappings = getMappings(config);
@@ -29,13 +27,12 @@ function importData(config, sheet) {
     const sourceData = getJSON(sheet);
 
     // diffing
-    // todo
     const hash = MD5(JSON.stringify(sourceData));
     let priorHashes;
     try {
-        priorHashes = JSON.parse(config.hashes); // this isn't working
+        priorHashes = JSON.parse(config.hashes); // only on sync
     } catch (e) {
-        priorHashes = [];
+        priorHashes = []; // always on 'run'
     }
 
     if (priorHashes.includes(hash)) {
@@ -182,7 +179,7 @@ function getTransformType(config) {
     if (config.record_type === "user") return modelMpUsers;
     if (config.record_type === "group") return modelMpGroups;
     if (config.record_type === "table") return modelMpTables;
-    throw new Error(`${config.record_type} is not a supported record type`);
+    throw `${config.record_type} is not a supported record type`;
 }
 
 /**
@@ -233,11 +230,12 @@ function summarizeImport(cleanConfig, responses, startTime, endTime, targetData)
 
 if (typeof module !== "undefined") {
     module.exports = { importData };
-    const { getConfig } = require("../utilities/storage.js");
+    const { getConfig, appendConfig } = require("../utilities/storage.js");
     const { validateCreds } = require("../utilities/validate.js");
     const { flushToMixpanel } = require("../utilities/flush.js");
     const { getJSON } = require("../utilities/toJson.js");
     const { clone } = require("../utilities/misc.js");
+	const { MD5 } = require('../utilities/md5.js')
     const { modelMpEvents } = require("../models/modelEvents.js");
     const { modelMpUsers } = require("../models/modelUsers.js");
     const { modelMpTables } = require("../models/modelTables.js");