Adding recent troubleshooting guidance

TROUBLESHOOTING.md

@@ -48,6 +48,57 @@
 
     In this case, use the databricks CLI to upload the jar to the expected location to avoid changes in the file name.
 
+* ### Internal Error Resolving Secrets
+
+    For the demo deployment, if your cluster fails and returns the error "Internal Error resolving secrets" and "Failed to fetch secrets referred to in Spark Conf", the deployment script may have failed to add an Access Policy to the Azure Key Vault or the secret scope was not created.
+
+    **Solution**: Update the values in the below script and execute it in the cloud shell. This script deletes the demo deployment's secret scope and then recreates it. After executing the script, you should see an access policy for "AzureDatabricks" in your Azure Key Vault.
+
+    ```bash
+    adb_ws_url=adb-DATABRICKS_WORKSPACE.ID.azuredatabricks.net
+    global_adb_token=$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d -o tsv --query '[accessToken]')
+    adb_ws_id=/subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME/providers/Microsoft.Databricks/workspaces/DATABRICKS_WORKSPACE_NAME
+    subscription_id=123acb-456-def
+    akv_name=AKV_NAME
+    akv_resource_id=/subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME/providers/Microsoft.KeyVault/vaults/AKV_NAME
+
+    # Remove the Secret Scope if it exists
+    cat << EOF > delete-scope.json
+    {
+    "scope": "purview-to-adb-kv"
+    }
+    EOF
+
+    curl \
+    -X POST https://$adb_ws_url/api/2.0/secrets/scopes/delete \
+    -H "Authorization: Bearer $global_adb_token" \
+    -H "X-Databricks-Azure-Workspace-Resource-Id: $adb_ws_id" \
+    --data @delete-scope.json
+
+    # If the above fails, that's okay
+    # Ultimately, we just need a clean slate
+
+    cat << EOF > create-scope.json
+    {
+    "scope": "purview-to-adb-kv",
+    "scope_backend_type": "AZURE_KEYVAULT",
+    "backend_azure_keyvault":
+    {
+        "resource_id": "$akv_resource_id",
+        "dns_name": "https://$akv_name.vault.azure.net/"
+    },
+    "initial_manage_principal": "users"
+    }
+    EOF
+
+
+    curl \
+    -X POST https://$adb_ws_url/api/2.0/secrets/scopes/create \
+    -H "Authorization: Bearer $global_adb_token" \
+    -H "X-Databricks-Azure-Workspace-Resource-Id: $adb_ws_id" \
+    --data @create-scope.json
+    ```
+
 ## <a id="no-lineage" />I don't see lineage in Microsoft Purview
 
 * ### Try Refreshing the Page

deploy-base.md

@@ -117,6 +117,8 @@ From the [Azure Portal](https://portal.azure.com)
 
   echo $purview_type_resp_custom_type
   ```
+
+  If you need a Powershell alternative, see the [docs](./docs/powershell-alternatives.md#upload-custom-types).
 
 ## <a id="download-openlineage" />Download the OpenLineage Spark agent and configure with your Azure Databricks clusters
 

deploy-demo.md

@@ -120,3 +120,7 @@ purview_type_resp_custom_type=$(curl -s -X POST $purview_endpoint/catalog/api/at
 
 echo $purview_type_resp_custom_type
 ```
+
+If you need a Powershell alternative, see the [docs](./docs/powershell-alternatives.md#upload-custom-types).
+
+You should now be able to run your demo notebook and receive lineage.

docs/powershell-alternatives.md

@@ -0,0 +1,25 @@
+# Powershell Alternative Scripts
+
+In some cases, you're not able to use the cloud shell or you don't have access to a machine that can run wsl / curl.  This doc provides alternatives to select 
+
+## Upload Custom Types
+
+Assumes you are in the `deployment/infra` folder of the repo.
+
+```powershell
+$purview_endpoint="https://PURVIEW_ACCOUNT_NAME.purview.azure.com"
+$TENANT_ID="TENANT_ID" 
+$CLIENT_ID="CLIENT_ID" 
+$CLIENT_SECRET="CLIENT_SECRET"
+
+$get_token=(Invoke-RestMethod -Method 'Post' -Uri "https://login.microsoftonline.com/$TENANT_ID/oauth2/token"  -Body "resource=https://purview.azure.net&client_id=$CLIENT_ID&client_secret=$CLIENT_SECRET&grant_type=client_credentials")
+$token=$get_token.access_token
+$body=(Get-Content -Path .\Custom_Types.json)
+$headers = @{
+'Content-Type'='application/json'
+'Authorization'= "Bearer $token"
+}
+
+Invoke-RestMethod -Method 'Post' -Uri "$purview_endpoint/catalog/api/atlas/v2/types/typedefs"  -Body $body -Headers $headers
+
+```