[WIP] feat: delete unconfirmed users

rfcs/inactive_users_cleanup.md

@@ -0,0 +1,80 @@
+# Inactive user cleanup
+
+## Overview and motivation
+For users who didn't pass the activation process, the service using TTL keyspace cleanup and only `Internal Data` deleted.
+A lot of linked keys remain in the database, and this leads to keyspace pollution.
+For better data handling and clean database structure, I introduce some changes in service logic:
+
+## General defs
+  - `inactive-users`
+  Redis database sorted set key. Assigned to `USER_ACTIVATE` constant.
+  Record contains `userId` as value and `timestamp` as score. 
+  - `user-audiences` [Described here](update_metadata_lua.md#audience-list-update)
+  - `deleteInactivatedUsers` Redis script, handling all cleanup logic.
+
+## Organization Members
+The `organization add member` process doesn't have validation whether the user passed activation and allows
+inviting inactive users into an organization. The script checks whether inactivated user assigned to any organization
+and deletes users from organization members and user->organization bindings.
+
+## Registration and activation
+Every Activation and Registration request event executes  `users:cleanup`  hook.
+The Activation request executes the hook first this strategy saves from inactive
+users that hit TTL but tried to pass the activation process.
+The Registration request executes the hook after `username` exists check.
+
+## Registration process
+When the user succeeds registration but activation not requested, the new entry added to `inactive-users`.
+Record contains `userId` and `current timestamp`.
+
+## Activation process
+When the user succeeds activation `userId`,the entry deleted from `inactive-users`.
+
+## `users:cleanup` hook `cleanUsers(suppress?)`
+`suppress` parameter defines function error behavior. If parameter set, the function throws errors,
+otherwise, function calls `log.error` with `error.message` as message.
+Default value is `true`. IMHO User shouldn't know about our problems.
+
+Other option, is to define new config parameter as object and move `config.deleteInactiveAccounts` into it:
+```javascript
+const conf = {
+  deleteInactiveUsers: {
+    ttl: seconds, // replaces deleteInactiveAccounts
+    suppressErrors: true || false,
+  },
+}
+```
+Calls `deleteInactivatedUsers` script with TTL parameter from `service.config.deleteInactiveAccounts`.
+When script finished, calls TokenManager to delete Action tokens(`USER_ACTION_*`, ``). 
+*NOTE*: Should we update TokenManager to add support of pipeline?
+
+## Redis Delete Inactive User Script
+When the service connects to the Redis server and fires event "plugin:connect:$redisType" `utils/inactive/defineCommand.js` executed.
+Function rendering `deleteInactivatedUsers.lua.hbs` and evals resulting script into IORedis.
+The Script using a dozen constants, keys, and templates, so all these values rendered inside of the template using template context.
+Returns list of deleted users.
+
+*NOTE*: Using experimental `fs.promises.readFile` API function. On `node` 10.x it's an experimental feature,
+on `node` >= 11.x function becomes stable without any changes in API.
+
+#### deleteInactivatedUsers `USERS_ACTIVATED` `TTL` as seconds
+##### Script paramerters:
+1. KEYS[1] Sorted Set name containing the list of users that didn't pass activation.
+2. ARGS[1] TTL in seconds.
+
+##### When started:
+1. Gets UserId's from ZSET `USERS_ACTIVATED` where score < `now() - TTL * 1000` and iterates over them.
+2. Gets dependent userData such as username, alias, and SSO provider information used in delete procedure and calls [Delete process](#delete-process).
+3. Deletes processed user ids from `USER_ACTIVATED` key.
+
+##### Delete process 
+The main logic is based on `actions/removeUsers.js`.
+Using the username, id, alias and SSO providers fields, script checks and removes dependent data from the database:
+* Alias to id binding.
+* Username to id binding.
+* All assigned metadata. Key names rendered from the template and `user-audiences`.
+* SSO provider to id binding. Using `SSO_PROVIDERS` items as the field name decodes and extracts UID's from Internal data.
+* User tokens.
+* Private and public id indexes
+* Links and data used in Organization assignment
+

rfcs/update_metadata_lua.md

@@ -0,0 +1,77 @@
+# User/Organization metadata update rework
+## Overview and Motivation
+When user or organization metadata needs to be updated, the Service uses the Redis pipeline javascript code.
+For each assigned meta hash always exists a single `audience`, but there is no list of `audiences` assigned to the user or company.
+To achieve easier audience tracking and a combined metadata update, I advise using a Lua based script.
+
+## Audience lists
+Audiences stored in sets formed from `USERS_AUDIENCE` or `ORGANISATION_AUDIENCE` constants and `Id`
+(eg: `{ms-users}10110110111!audiences`). Both keys contain `audience` names that are currently have assigned values.
+
+## utils/updateMetadata.js
+Almost all logic in this file removed and ported into LUA Script.
+This Function checks the consistency of the provided `opts`. If `opts.metadata` and `opts.audiences` are objects, script transforming them to an array containing these objects. Checks count of meta operations and audiences to equal each other.
+Organization meta update request `utils/setOrganizationMetadata.js` uses the same functionality, so the same changes applied to it.
+
+After commands execution result returned from the script, decoded from JSON string.
+
+## script/updateMetadata.lua
+Script repeats all logic including custom scripts support.
+
+### Script parameters:
+1. KEYS[1] Audiences key template.
+2. KEYS[2] used as metadata key template, eg: "{ms-users}{id}!metadata!{audience}".
+3. ARGV[1] Id - organization or user-id.
+4. ARGV[2] JSON encoded opts parameter opts.{script, metadata, audiences}.
+
+### Depending on metadata or script set:
+If `opt.metadata` set:
+ * Script starts iterating audiences.
+ * On each audience, creates metadata key from provided template.
+ * Iterates operations from `opt.metadata`, based on index of `opts.audiences`.
+    ```javascript
+     const opts = {
+       audiences: ['first', 'second'],
+       metadata: [{
+         // first audience commands
+       }, {
+         // second audience commands
+       }], 
+     }
+    ```
+   Commands execute in order: `audiences[0]` => `metadata[0]`,`audiences[1]` => `metadata[1]`, 
+
+If `opt.script` set:
+* Script iterates `audiences` and creates metadata keys from provided template
+ * Iterates `opt.script`:
+    * EVAL's script from `script.lua` and executes with params generated from: metadata keys(look to the previous step)
+     and passed `script.argv`.
+    * If script evaluation fails, script returns redis.error witch description.
+
+When operations/scripts processed, the script forms JSON object like
+```javascript
+const metaResponse = [
+  //forEach audience
+  {
+    '$incr': {
+      field: 'result', // result returned from HINCRBY command
+    },
+    '$remove': intCount, // count of deleted fields
+    '$set': "OK", // or cmd hset result.
+  },
+];
+
+const scriptResponse = {
+  'scriptName': [
+    // values returned from script
+  ],
+};
+```
+
+### Audience list update
+When all update operations succeeded:
+* Script get's current list of user's or organization's audiences from HSET `KEYS[1]`,
+unions them with `opts.audiences` and generates full list metadata keys.
+* Iterates over them to check whether some data exists.
+* If no data exists, the script deletes the corresponding audience from HSET `KEYS[1]`.
+

scripts/deleteInactivatedUsers.lua.hbs

@@ -0,0 +1,190 @@
+local usersInactiveKey = KEYS[1]
+local exTime = ARGV[1]
+---
+-- var defs
+---
+local delimiter = '{{ KEY_SEPARATOR }}'
+local keyPrefix = '{{ keyPrefix }}'
+
+local ssoProviders = {
+    {{#each sso}}
+        "{{ this }}",
+    {{/each}}
+};
+
+-- key templates
+local usersDataKeyTemplate = '{{ keyTemplates.USERS_DATA }}'
+local usersMetaKeyTemplate = '{{ keyTemplates.USERS_METADATA }}'
+local usersTokenKeyTemplate = '{{ keyTemplates.USERS_TOKENS }}'
+local usersAudienceKeyTemplate = '{{ keyTemplates.USERS_AUDIENCE }}'
+
+local usersOrganizationsKeyTemplate = '{{ keyTemplates.USERS_ORGANIZATIONS }}'
+local organizationsMembersKeyTemplate = '{{ keyTemplates.ORGANIZATIONS_MEMBERS }}'
+local organizationsMemberKeyTemplate = '{{ keyTemplates.ORGANIZATIONS_MEMBER }}'
+local organizationMemberTemplate = '{{ templates.ORGANIZATIONS_MEMBER }}'
+
+-- simple keys
+local usersAliasToIDKey = '{{ keys.USERS_ALIAS_TO_ID }}'
+local usersUsernameToIDKey = '{{ keys.USERS_USERNAME_TO_ID }}'
+local usersSSOToIDKey = '{{ keys.USERS_SSO_TO_ID }}'
+
+-- indexes
+local organizationsInvitationIndex = '{{ keys.ORGANIZATIONS_INVITATIONS_INDEX }}'
+local usersPublicIndex = '{{ keys.USERS_PUBLIC_INDEX }}'
+local usersIndex = '{{ keys.USERS_INDEX }}'
+
+-- fields
+local usersUsernameField = '{{ fields.USERS_USERNAME_FIELD }}'
+local usersAliasField = '{{ fields.USERS_ALIAS_FIELD }}'
+
+
+---
+-- Helper functions
+---
+local function isempty(s)
+  return s == nil or s == '' or s == false;
+end
+
+--decodes json
+local function decode(strval)
+  if type(strval) == "string" then
+    return cjson.decode(strval)
+  else
+    return {}
+  end
+end
+
+-- key generator
+local function key(...)
+  return table.concat(arg, delimiter)
+end
+
+-- key from template generator
+local function makeKey(template, templateValues)
+  local str = template
+  for param, value in pairs(templateValues) do
+    str = str:gsub('{'..param..'}', value, 1)
+  end
+  return str
+end
+
+-- gets user data
+local function getData(key)
+  local fields = { usersUsernameField, usersAliasField, unpack(ssoProviders) }
+  local data = redis.call("HMGET", key, unpack(fields))
+
+  if #data > 0 then
+    local result = {};
+    --convert to table
+    for i = 1, #data, 1 do
+      result[fields[i]] = data[i]
+    end
+    return result;
+  end
+
+  return nil
+end
+
+---
+-- Script logic functions
+---
+
+-- deletes organization bindings
+local function deleteOrganizationMember(username)
+  local userOrganizationsKey = makeKey(usersOrganizationsKeyTemplate, { username = username })
+  local organizationIds = redis.call("HKEYS", userOrganizationsKey)
+
+  redis.call('SREM', organizationsInvitationIndex, username)
+
+  for _, orgId in pairs(organizationIds) do
+    local organizationMembersKey = makeKey(organizationsMembersKeyTemplate, { orgid = orgId})
+    local organizationMemberKey = makeKey(organizationsMemberKeyTemplate, { orgid = orgId, username = username })
+    local organizationMember = makeKey(organizationMemberTemplate, {orgid = orgId, username = username })
+
+    redis.call("DEL", organizationMemberKey)
+    redis.call("HDEL", userOrganizationsKey, orgId)
+    redis.call("ZREM", organizationMembersKey, organizationMember )
+  end
+
+end
+
+-- handles emails (usernames)
+local inactiveUserNames = {}
+
+-- delete logic
+local function deleteUser(userID, userData)
+  local alias = userData[usersAliasField]
+  local username = userData[usersUsernameField]
+
+  -- save username
+  table.insert(inactiveUserNames, username)
+
+  -- delete alias
+  if isempty(alias) == false then
+    redis.call("HDEL", usersAliasToIDKey, alias, string.lower(alias))
+  end
+
+  redis.call("HDEL", usersUsernameToIDKey, username)
+
+  -- if user assigned to organization
+  deleteOrganizationMember(username)
+
+  -- delete SSO data
+  for k, provider in pairs(ssoProviders) do
+    local rawData = userData[provider]
+    local providerData = decode(rawData)
+    if isempty(providerData['uid']) == false then
+      redis.call("HDEL", usersSSOToIDKey, providerData['uid'])
+    end
+  end
+
+  -- clean indicies
+  redis.call("HDEL", usersPublicIndex, userID )
+  redis.call("SREM", usersIndex, userID)
+
+  -- delete user data
+  local userDataKey = makeKey(usersDataKeyTemplate, { id = userID })
+  redis.call("DEL", userDataKey)
+
+  -- delete meta data
+  local usersAudienceKey = makeKey(usersAudienceKeyTemplate, { id = userID })
+  local userAudiences = redis.call("SMEMBERS", usersAudienceKey)
+
+  for k, audience in pairs(userAudiences) do
+    local metaKey = makeKey(usersMetaKeyTemplate, { id = userID, audience = audience })
+    redis.call("DEL", metaKey)
+  end
+
+  -- delete USERS_TOKENS
+  local userTokensKey = makeKey(usersTokenKeyTemplate, { id = userID })
+  redis.call("DEL", userTokensKey)
+
+  -- delete user audiences list
+  redis.call("DEL", usersAudienceKey)
+
+end
+
+---
+-- Script Logic
+---
+
+redis.replicate_commands();
+
+local inactiveUsers = redis.call("ZRANGEBYSCORE", usersInactiveKey, '-inf', exTime)
+
+for key, userID in pairs(inactiveUsers) do
+  local internalDatakey = makeKey(usersDataKeyTemplate, { id = userID })
+  local userData = getData(internalDatakey)
+
+  if userData ~= nil then
+    deleteUser(userID, userData)
+  end
+end
+
+for _ , userID in pairs(inactiveUsers) do
+  redis.call('ZREM', usersInactiveKey, userID)
+end
+
+-- returns emails of deleted users
+-- helps to remove TokenManager tokens
+return inactiveUserNames