azure-collector

AlertLogic Office 365 Log Collector

Overview

This repository contains Azure Web application Node.js source code and an ARM template for setting up a data collector in Azure which will collect and forward Office 365 log data to the Alert Logic Cloud Defender Log Manager (LM) feature.

Installation

Installation requires the following steps:

1. Register a new O365 web application in O365 portal for collecting O365 logs.
2. Set up the required Active Directory security permissions for the application to authorize it to read threat intelligence data and activity reports for your organization.
3. Create an Access Key that will allow the application to connect to the Alert Logic Cloud Defender and Cloud Insight backend.
4. Download and deploy a custom ARM template to Microsoft Azure to create functions for collecting and managing O365 log data
5. Verify the installation was successful using the Alert Logic UI under Configuration -> Deployments -> All Deployments -> Log Sources -> Filter by Push (Office 365, CloudWatch) collection method.

Register a New O365 Web Application in O365

In order to install O365 Log collector:

1. Log into O365 portal as AD tenant administrator.

2. Navigate to Admin Centers and Azure AD.

3. On the left side panel click Azure Active Directory then select App Registrations.

4. Click + New application registration, fill in configuration parameters and click Create:

   • Name - for example alo365collector.
   • Select Web app/ API as Application type.
   • In Sign-on URL enter some URL, for example http://alo365collector.com. Note, it is not used anywhere within your subscription.

5. After application is created select All apps, click on the application name that was created and make a note of the Application ID, for example, a261478c-84fb-42f9-84c2-de050a4babe3

Set Up the Required Active Directory Security Permissions

1. On the Settings panel under the newly created Application, select Required permissions and click + Add
2. Hit Select an API and chose Office 365 Management APIs, click Select
3. In Application permissions select Read service health information for your organization, Read activity data for your organization, Read threat intelligence data for your organization and Read activity reports for your organization. Click Select and then Done.
4. Click the Grant Permissions button and confirm by clicking Yes. Note: only AD tenant admin can grant permissions to an Azure AD application.
5. On the Settings panel of the application, select Keys.
6. Enter key Description and set Duration to Never expires then click Save. Note: please save the key value, it is needed later during template deployment.
7. Get the Service Principal ID associated with the application by navigating back to the Registered App blade, click on the link under Managed application in local directory then click Properties. The Service Principal ID is labeled as Object ID on the properties page. Caution: This is not the same Object ID found under the Registered app view or under the Settings.

Create an Alert Logic Access Key

From the Bash command line in Azure Cloud Shell run the following commands, where <username> is your Alert Logic user and <password> is your Alert Logic password:

export AL_USERNAME='<username>'
auth=$(curl -SX POST -u $AL_USERNAME https://api.global-services.global.alertlogic.com/aims/v1/authenticate); export AL_ACCOUNT_ID=$(echo $auth | jq -r '.authentication.account.id'); export AL_USER_ID=$(echo $auth | jq -r '.authentication.user.id'); export AL_TOKEN=$(echo $auth | jq -r '.authentication.token'); if [ -z $AL_TOKEN ]; then echo "Authentication failure"; else roles=$(curl -SX GET -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/roles | jq -r '.roles[].name'); if [ "$roles" != "Administrator" ]; then echo "The $AL_USERNAME doesn’t have Administrator role. Assigned role is '$roles'"; else curl -SX POST -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/access_keys | jq .; fi; fi; unset AL_USERNAME;

For accounts where MFA is enabled:

export AL_USERNAME='<username>'
auth=$(curl -SX POST -d '{"mfa_code": "<mfa_code_here>" }' -u $AL_USERNAME https://api.global-services.global.alertlogic.com/aims/v1/authenticate); export AL_ACCOUNT_ID=$(echo $auth | jq -r '.authentication.account.id'); export AL_USER_ID=$(echo $auth | jq -r '.authentication.user.id'); export AL_TOKEN=$(echo $auth | jq -r '.authentication.token'); if [ -z $AL_TOKEN ]; then echo "Authentication failure"; else roles=$(curl -SX GET -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/roles | jq -r '.roles[].name'); if [ "$roles" != "Administrator" ]; then echo "The $AL_USERNAME doesn’t have Administrator role. Assigned role is '$roles'"; else curl -SX POST -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/access_keys | jq .; fi; fi; unset AL_USERNAME;

An example of a successful response is:

{
  "access_key_id": "712c0b413eef41f6",
  "secret_key": "1234567890b3eea8880d292fb31aa96902242a076d3d0e320cc036eb51bf25ad"
}

Note: if the output is blank please double-check the Alert Logic user permission, you should have administrator access. More details about AIMS APIs can be found here.

Make a note of the access_key_id and secret_key values for use in the deployment steps below.

Note: Only five access keys can be created per user. If you get a "limit exceeded" response you will need to delete some keys in order to create new ones. Use the following command to list access keys:

curl -s -X GET -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/access_keys | jq

Then use the selected access_key_id in the following curl command to delete it:

curl -X DELETE -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/access_keys/<ACCESS_KEY_ID_HERE>

Function deployment

Log into Azure portal. Note, In order to perform steps below you should have an active Azure subscription, to find out visit Azure subscriptions blade.

If multiple Active Directory tenants are used within your organization please login into the same tenant where application registration was created during Register a New O365 Web Application in O365. How to find Office365 tenant id see here.

Deploy via the Custom ARM Template in an Azure Subscription

Fill in required template parameters and click the Purchase button to start a deployment:

• Name - This is the name of the log source that will show in the Alert Logic UI
• Storage Name - Any Storage Account name (that does not currently exist)
• Alert Logic Access Key ID - access_key_id returned from AIMs above
• Alert Logic Secret Key - secret_key returned from AIMs above
• Alert Logic API endpoint - leave it as api.global-services.global.alertlogic.com
• Alert Logic Data Residency - leave it as default
• Office365 Content Streams - The list of streams you would like to collect. Valid values are:
  • ["Audit.AzureActiveDirectory","Audit.Exchange","Audit.SharePoint","Audit.General", "DLP.All"]
• Service Principal ID - The Object ID of the application that created the subscription. You can obtain it from Azure -> AD -> App registrations -> Your app name -> Link under Managed application in local directory -> Properties -> Object ID
• App Client ID - The GUID of your application that created the subscription. You can obtain it from Azure -> AD -> App registrations -> Your app name
• App Client Secret - The secret key of your application from App Registrations

Deploy via Azure CLI

You can use either Azure Cloud Shell or local installation of Azure CLI.

1. Create a new resource group in, for example, the "Central US" location by executing following command:

   az group create --name <new-resource-group-name> --location "Central US"
   

2. Once created go to Resource groups blade and select the resource group.
3. Select Access Control (IAM) and add Website Contributor role to AD application identity created above.
4. Deploy a template by using following command, during its execution enter required parameters when asked.

   az group deployment create \
       --resource-group <new-resource-group-name> \
       --template-uri "https://raw.githubusercontent.com/alertlogic/azure-collector/master/template.json"
   

Wait until it is deployed successfully.

Verify the Installation

1. Go to Function Apps and choose the Alert Logic O365 collector function. The recent log entry under Functions-> Master-> Monitor should read an OK status (Example: O365 source checkin OK) and should not contain any error messages.
2. Log into Alert Logic UI and navigate into Configuration -> Deployments -> All Deployments -> Log Sources -> Filter by Push (Office 365, CloudWatch) collection method. Check new O365 log source (with a name provided during az group deployment create above) has been created and source status is ok.

How It Works

Note: the following Azure functions use Application/O365 tenant id (APP_TENANT_ID web application setting) as a PublisherIdentifier during O365 management API requests. More info about PublisherIdentifier can be found here.

Master Function

The Master function is a timer trigger function which is responsible for:

• registering the Azure web app in Alertlogic backend;
• reporting health-checks to the backed;
• performing log source configuration updates, which happen via Alertlogic UI.

Note: When releasing a new version of the collector please remember to increment the version number in npm package.json file. To display the current version locally, issue npm run local-version

Updater Function

The Updater is a timer triggered function runs deployment sync operation every 12 hours in order to keep entire Web application up to date.

O365WebHook Function

The O365WebHook function exposes an HTTP API endpoint https://<app-name>/o365/webhook which is registered as an Office 365 webhook and processes O365 activity notifications. Below is a notification example,

[
  {
    "contentType": "Audit.AzureActiveDirectory",
    "contentId": "20170721121608709004422$20170721121608709004422$audit_azureactivedirectory$Audit_AzureActiveDirectory$IsFromNotification",
    "contentUri": "https://manage.office.com/api/v1.0/bf8d32d3-1c13-4487-af02-80dba2236485/activity/feed/audit/20170721121608709004422$20170721121608709004422$audit_azureactivedirectory$Audit_AzureActiveDirectory$IsFromNotification",
    "notificationStatus": "Succeeded",
    "contentCreated": "2017-07-21T12:16:56.798Z",
    "notificationSent": "2017-07-21T12:16:56.798Z",
    "contentExpiration": "2017-07-28T12:16:08.709Z"
  },
  {
    "contentType": "Audit.AzureActiveDirectory",
    "contentId": "20170721121625590007449$20170721121625590007449$audit_azureactivedirectory$Audit_AzureActiveDirectory$IsFromNotification",
    "contentUri": "https://manage.office.com/api/v1.0/bf8d32d3-1c13-4487-af02-80dba2236485/activity/feed/audit/20170721121625590007449$20170721121625590007449$audit_azureactivedirectory$Audit_AzureActiveDirectory$IsFromNotification",
    "notificationStatus": "Succeeded",
    "contentCreated": "2017-07-21T12:16:56.798Z",
    "notificationSent": "2017-07-21T12:16:56.798Z",
    "contentExpiration": "2017-07-28T12:16:25.590Z"
  }
]

A notification contains a link to the actual data which is retrieved by the O365WebHook, wrapped into a protobuf structure and is sent into Alertlogic Ingest service.

Note: it can take up to 24 hours before audit content is available. Please follow this link to find the time it takes for the different services in Office 365.

Local Development

1. Clone repo git clone [email protected]:alertlogic/azure-collector.git
2. cd azure-collector
3. Run ./local_dev/setup.sh
4. Edit ./local_dev/dev_config.js
5. Run Master function locally: npm run local-master
6. Run Updater function locally: npm run local-updater
7. Run O365WebHook function locally: npm run local-o365webhook
8. Run npm test in order to perform code analysis and unit tests.

Please use the following code style as much as possible.

Setting environment in dev_config.js

• process.env.APP_TENANT_ID - The GUID of the tenant i.e. 'alazurealertlogic.onmicrosoft.com'
• process.evn.APP_RESOURCE_GROUP - The name of the resource group where your application is deployed.
• process.env.CUSTOMCONNSTR_APP_CLIENT_ID - The GUID of your application that created the subscription. You can obtain it from Azure -> AD -> App registrations -> Your app name
• process.env.CUSTOMCONNSTR_APP_CLIENT_SECRET - A secret key of your application from App Registrations.
• process.env.CUSTOMCONNSTR_APP_CI_ACCESS_KEY_ID - access key returned from AIMs above.
• process.env.CUSTOMCONNSTR_APP_CI_SECRET_KEY- secret key returned from AIMs above.

Known Issues/ Open Questions

• Sometimes deployments fail after siteSync action. We need better updater to handle that in order not to wait for 12 hours for the next update attempt.

Useful Links

• Node.js static code analysis tool
• How to monitor Azure functions?
• Server application to Web API authentication.
• Office 365 Management API reference.
• Azure Web apps API reference