Prevent overlapping state transitions in workflows

[MatanYadaev]

Type

enhancement, bug_fix

---

Description

• Refactored WorkflowService and related repositories to support transactions, enhancing data consistency and error handling.
• Added DEEP_MERGE_CONTEXT event to allow deep merging of workflow context data with support for various array merge strategies.
• Implemented integration tests for new features and enhancements in WorkflowService.
• Developed a utility function in workflow-core package for deep merging objects, supporting different strategies for merging arrays.
• Integrated deep merge context functionality into the workflow runner, enabling complex state management scenarios.

---

Changes walkthrough

	Relevant files
Enhancement	workflow.service.ts Enhance Workflow Service with Transaction Support and Deep Merge Context Event services/workflows-service/src/workflow/workflow.service.ts Refactored to include PrismaTransaction in various methods for transaction support. Added transaction support to methods for updating workflow runtime data. Utilized beginTransactionIfNotExistCurry for transactional operations. Implemented DEEP_MERGE_CONTEXT event handling for deep merging context data. +841/-1020 workflow-runtime-data.repository.ts Update Workflow Runtime Data Repository for Transaction Support services/workflows-service/src/workflow/workflow-runtime-data.repository.ts Added methods for locking workflow runtime data for update operations. Updated methods to support transactions. Removed unused types and methods. +116/-70 deep-merge-with-options.ts Implement Deep Merge Utility Function with Array Merge Options packages/workflow-core/src/lib/utils/deep-merge-with-options.ts Implemented a utility function for deep merging objects with various array merge strategies. +139/-0  workflow-runner.ts Integrate Deep Merge Context Event Handling in Workflow Runner packages/workflow-core/src/lib/workflow-runner.ts Integrated DEEP_MERGE_CONTEXT event handling in the workflow runner. Adjusted plugin invocation to pass additional parameters. +45/-18
Tests	workflow.service.intg.test.ts Add Integration Tests for WorkflowService Enhancements      services/workflows-service/src/workflow/workflow.service.intg.test.ts Added integration tests for WorkflowService. Tested DEEP_MERGE_CONTEXT event for various scenarios including nested objects and arrays. +1175/-0

---

✨ PR-Agent usage:
Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

[changeset-bot]

⚠️ No Changeset found

Latest commit: 451db86

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[github-actions]

PR Description updated to latest commit (736412f)

[github-actions]

PR Review

⏱️ Estimated effort to review [1-5]	4, due to the extensive changes across multiple files, including the addition of new functionality, refactoring, and integration tests. The complexity of the changes, especially those related to transaction management and deep merging logic, requires careful review to ensure correctness and maintainability.
🧪 Relevant tests	Yes
🔍 Possible issues	The implementation of locking in lockWorkflowHierarchyForUpdate might lead to deadlocks if not carefully managed, especially with recursive CTEs in PostgreSQL.
	The deep merge utility function's handling of arrays and objects during merge operations could introduce unexpected behavior if the merging logic does not align with the intended use cases, particularly with the ARRAY_MERGE_OPTION.BY_ID strategy.
	The extensive use of raw SQL queries and operations, especially in WorkflowRuntimeDataRepository, increases the risk of SQL injection if not properly sanitized and could lead to maintenance challenges.
🔒 Security concerns	No

Code feedback:

---

relevant file	services/workflows-service/src/workflow/workflow.service.ts
suggestion	Consider implementing error handling for the beginTransactionIfNotExistCurry utility function to ensure that transactions are correctly managed, especially in cases where the transaction might fail to start. This is crucial to prevent potential data inconsistencies and ensure the application's stability. [important]
relevant line	private readonly prismaService: PrismaService,

---

relevant file	packages/workflow-core/src/lib/utils/deep-merge-with-options.ts
suggestion	It's recommended to add type checks or guards within the deepMergeWithOptions function to ensure that the inputs (val1, val2) are of the expected types before proceeding with the merge operation. This can help prevent runtime errors and ensure the function's robustness. [important]
relevant line	export const deepMergeWithOptions = (

---

relevant file	services/workflows-service/src/workflow/workflow-runtime-data.repository.ts
suggestion	For the lockWorkflowHierarchyForUpdate method, consider adding logging to capture the start and end of the lock operation, including any errors that may occur. This will be invaluable for debugging purposes and understanding the locking behavior in production environments. [medium]
relevant line	private async lockWorkflowHierarchyForUpdate(

---

relevant file	services/workflows-service/src/workflow/workflow.service.ts
suggestion	When integrating the deep merge functionality into the workflow runner, ensure that the merge strategy aligns with the business logic and data structure expectations. It may be beneficial to expose a configuration option for specifying the merge strategy at the workflow definition level. [medium]
relevant line	private readonly prismaService: PrismaService,

---

---

✨ Review tool usage guide:

---

Overview:
The review tool scans the PR code changes, and generates a PR review. The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on any PR.
When commenting, to edit configurations related to the review tool (pr_reviewer section), use the following template:

/review --pr_reviewer.some_config1=... --pr_reviewer.some_config2=...

With a configuration file, use the following template:

[pr_reviewer]
some_config1=...
some_config2=...

Utilizing extra instructions The review tool can be configured with extra instructions, which can be used to guide the model to a feedback tailored to the needs of your project. Be specific, clear, and concise in the instructions. With extra instructions, you are the prompter. Specify the relevant sub-tool, and the relevant aspects of the PR that you want to emphasize. Examples for extra instructions: [pr_reviewer] # /review # extra_instructions=""" In the 'possible issues' section, emphasize the following: - Does the code logic cover relevant edge cases? - Is the code logic clear and easy to understand? - Is the code logic efficient? ... """ Use triple quotes to write multi-line instructions. Use bullet points to make the instructions more readable.

How to enable\disable automation When you first install PR-Agent app, the default mode for the review tool is: pr_commands = ["/review", ...] meaning the review tool will run automatically on every PR, with the default configuration. Edit this field to enable/disable the tool, or to change the used configurations

Auto-labels The review tool can auto-generate two specific types of labels for a PR: a possible security issue label, that detects possible security issues (enable_review_labels_security flag) a Review effort [1-5]: x label, where x is the estimated effort to review the PR (enable_review_labels_effort flag)

Extra sub-tools The review tool provides a collection of possible feedbacks about a PR. It is recommended to review the possible options, and choose the ones relevant for your use case. Some of the feature that are disabled by default are quite useful, and should be considered for enabling. For example: require_score_review, require_soc2_ticket, and more.

Auto-approve PRs By invoking: /review auto_approve The tool will automatically approve the PR, and add a comment with the approval. To ensure safety, the auto-approval feature is disabled by default. To enable auto-approval, you need to actively set in a pre-defined configuration file the following: [pr_reviewer] enable_auto_approval = true (this specific flag cannot be set with a command line argument, only in the configuration file, committed to the repository) You can also enable auto-approval only if the PR meets certain requirements, such as that the estimated_review_effort is equal or below a certain threshold, by adjusting the flag: [pr_reviewer] maximal_review_effort = 5

More PR-Agent commands To invoke the PR-Agent, add a comment using one of the following commands: /review: Request a review of your Pull Request. /describe: Update the PR title and description based on the contents of the PR. /improve [--extended]: Suggest code improvements. Extended mode provides a higher quality feedback. /ask <QUESTION>: Ask a question about the PR. /update_changelog: Update the changelog based on the PR's contents. /add_docs 💎: Generate docstring for new components introduced in the PR. /generate_labels 💎: Generate labels for the PR based on the PR's contents. /analyze 💎: Automatically analyzes the PR, and presents changes walkthrough for each component. See the tools guide for more details. To list the possible configuration parameters, add a /config comment.

See the review usage page for a comprehensive guide on using this tool.

[github-actions]

PR Code Suggestions

	Suggestions
enhancement	Add error handling to the mutation operation.                                 Consider adding error handling for the mutation operation. This can be achieved by using the onError callback in the mutation options to handle any errors that might occur during the mutation process. apps/backoffice-v2/src/domains/entities/hooks/mutations/useApproveTaskByIdMutation/useApproveTaskByIdMutation.tsx [9-12] export const useApproveTaskByIdMutation = (workflowId: string) => { const queryClient = useQueryClient(); const filterId = useFilterId(); const workflowById = workflowsQueryKeys.byId({ workflowId, filterId }); + // Example error handling + const mutation = useMutation(mutationFn, { + onError: (error, variables, context) => { + // Handle error + }, + }); Add error handling for useRevisionTaskByIdMutation call.         Consider handling errors when calling useRevisionTaskByIdMutation() to ensure robust error handling and user feedback. apps/backoffice-v2/src/lib/blocks/variants/KybExampleBlocks/hooks/useKybExampleBlocksLogic/useKybExampleBlocksLogic.tsx [81-82] -const { mutate: mutateRevisionTaskById, isLoading: isLoadingReuploadNeeded } = +const { mutate: mutateRevisionTaskById, isLoading: isLoadingReuploadNeeded, error } = useRevisionTaskByIdMutation(); +if (error) { + console.error("Error in useRevisionTaskByIdMutation:", error); + // Handle error appropriately +} Add error handling for child workflow action invocation.                      Ensure that any errors thrown by the action method are caught and handled appropriately to avoid unhandled promise rejections. packages/workflow-core/src/lib/plugins/common-plugin/child-workflow-plugin.ts [30-44] try { await this.action({ parentWorkflowRuntimeId: this.parentWorkflowRuntimeId, definitionId: this.definitionId, initOptions: { context: childWorkflowContext, event: this.initEvent, config: { language: this.parentWorkflowRuntimeConfig.language, }, }, }); -} finally { - return Promise.resolve(); +} catch (error) { + console.error("Error invoking child workflow action:", error); + // Handle error appropriately } Destructure imported objects for cleaner code.                                Consider destructuring BUILT_IN_EVENT and ARRAY_MERGE_OPTION directly in the import statement for cleaner code and easier readability. services/workflows-service/src/collection-flow/controllers/collection-flow.controller.ts [18] -import { BUILT_IN_EVENT, ARRAY_MERGE_OPTION } from '@ballerine/workflow-core'; +import { BUILT_IN_EVENT: { DEEP_MERGE_CONTEXT }, ARRAY_MERGE_OPTION: { BY_ID } } from '@ballerine/workflow-core'; Remove redundant array wrapping around parameters.                            Remove redundant array wrapping around tokenScope.projectId in the event method call for resubmitFlow. services/workflows-service/src/collection-flow/controllers/collection-flow.controller.ts [156] -[tokenScope.projectId], +
best practice	Destructure workflow object properties for cleaner code.         To improve code readability and maintainability, consider destructuring workflow object properties when they are used multiple times. This can make the code cleaner and easier to read. apps/backoffice-v2/src/lib/blocks/hooks/useDirectorsBlocks/useDirectorsBlocks.tsx [45] -const { mutate: removeDecisionById } = useRemoveDecisionTaskByIdMutation(workflow?.id); +const { id: workflowId } = workflow; +const { mutate: removeDecisionById } = useRemoveDecisionTaskByIdMutation(workflowId); Ensure consistent use of mutation hooks and their loading states.             For consistency and to avoid potential issues, ensure that all mutation hooks are used in a similar manner, especially regarding the handling of their loading states. apps/backoffice-v2/src/lib/blocks/hooks/useDocumentBlocks/useDocumentBlocks.tsx [80-81] const { mutate: mutateApproveTaskById, isLoading: isLoadingApproveTaskById } = useApproveTaskByIdMutation(workflow?.id); -const { isLoading: isLoadingRejectTaskById } = useRejectTaskByIdMutation(workflow?.id); +const { mutate: mutateRejectTaskById, isLoading: isLoadingRejectTaskById } = useRejectTaskByIdMutation(workflow?.id); Use a more specific type for the context in the updateContext and deepMergeContext functions. Consider using a more specific type than Record<PropertyKey, unknown> for the context in the updateContext and deepMergeContext functions. A more specific type can help with type safety and code readability. packages/workflow-core/src/lib/workflow-runner.ts [443-465] context, event: { payload: { - context: Record<PropertyKey, unknown>; + context: SpecificContextType; // Replace SpecificContextType with the actual type }; }, Add error handling for the token creation process.                            Add error handling for the create method in WorkflowTokenRepository to gracefully handle potential errors during the token creation process. services/workflows-service/src/auth/workflow-token/workflow-token.repository.ts [18-21] -return await transaction.workflowRuntimeDataToken.create({ - data: { - ...data, - projectId, +try { + return await transaction.workflowRuntimeDataToken.create({ + data: { + ...data, + projectId, + }); +} catch (error) { + // Handle error appropriately +} Implement error handling for the business update process.                     Add error handling for the updateById method in BusinessRepository to manage potential exceptions and provide feedback. services/workflows-service/src/business/business.repository.ts [80-83] -return await transaction.business.update({ - where: { id }, - ...args, +try { + return await transaction.business.update({ + where: { id }, + ...args, + }); +} catch (error) { + // Handle error appropriately +} Ensure consistent use of async/await.                                         Use async/await consistently in resubmitFlow method for better readability and error handling. services/workflows-service/src/collection-flow/controllers/collection-flow.controller.ts [154] -await this.workflowService.event( +return await this.workflowService.event( Add explicit return types to methods for better type safety.                  Ensure the return type of updateFlowLanguage and resubmitFlow methods are consistent with other methods in the controller for better type safety and readability. services/workflows-service/src/collection-flow/controllers/collection-flow.controller.ts [109] -async updateFlowLanguage( +async updateFlowLanguage(): Promise<ReturnType> {
possible issue	Ensure useRevisionTaskByIdMutation is used correctly by passing necessary arguments. To ensure that the useRevisionTaskByIdMutation hook is used correctly and efficiently, consider passing necessary arguments if the mutation function requires them, similar to other mutation hooks in the project. apps/backoffice-v2/src/lib/blocks/hooks/useDefaultBlocksLogic/useDefaultBlocksLogic.tsx [66] -const { mutate: mutateRevisionTaskById, isLoading: isLoadingReuploadNeeded } = useRevisionTaskByIdMutation(); +// Assuming the mutation requires a workflowId, similar to other mutations +const { mutate: mutateRevisionTaskById, isLoading: isLoadingReuploadNeeded } = useRevisionTaskByIdMutation(workflow?.id); Add validation for iterationParams structure in IterativePlugin. Validate that iterationParams contains the expected structure before proceeding with the iteration logic to prevent runtime errors. packages/workflow-core/src/lib/plugins/common-plugin/iterative-plugin.ts [29-30] if (!Array.isArray(iterationParams)) { console.error('Iterative plugin could not find iterate on param'); + return; // Add return statement to halt execution } +// Further validation can be added here to ensure each item in iterationParams has the expected structure
maintainability	Make workflow state check more flexible for future additions.                 Replace direct comparison with CommonWorkflowStates.MANUAL_REVIEW with a more flexible check that allows for future state additions without changing this logic. apps/backoffice-v2/src/pages/Entity/components/Case/hooks/usePendingRevisionEvents/usePendingRevisionEvents.tsx [21] -pendingWorkflowEvent?.workflowState === CommonWorkflowStates.MANUAL_REVIEW; +[CommonWorkflowStates.MANUAL_REVIEW].includes(pendingWorkflowEvent?.workflowState); Abstract the event sending logic into a separate method.                      Replace the direct use of BUILT_IN_EVENT.UPDATE_CONTEXT with a method that abstracts event sending, to improve maintainability and readability. services/workflows-service/src/collection-flow/collection-flow.service.ts [218-228] -return await this.workflowService.event( - { - id: tokenScope.workflowRuntimeDataId, - name: BUILT_IN_EVENT.UPDATE_CONTEXT, - payload: { - context: payload.data.context, - }, - }, - [tokenScope.projectId], - tokenScope.projectId, +return await this.sendUpdateContextEvent(tokenScope, payload.data.context); +// Where sendUpdateContextEvent is a method that encapsulates the event sending logic Add explanatory comments for the always condition in the state transition rule. Consider adding documentation or comments to the generateBaseCaseLevelStatesAutoTransitionOnRevision function to explain the purpose and usage of the always condition with jmespath. This can help future maintainers understand the logic and intention behind this specific state transition rule. services/workflows-service/scripts/workflows/generate-base-case-level-states.ts [49-57] +// This 'always' transition ensures that if the condition specified by the 'jmespath' rule is met, +// the state automatically transitions to the defaultState. The rule checks if the number of documents +// with a decision status is less than the total number of documents, indicating that not all documents +// have been reviewed, thus requiring a transition back to the review state. always: { target: defaultState, cond: { type: 'jmespath', options: { rule: 'length(documents[?decision.status]) < length(documents)', }, }, }, Use constants for event names for consistency and maintainability.            Consider using a constant for the event name 'RESUBMITTED' to maintain consistency and facilitate changes in the future. services/workflows-service/src/collection-flow/controllers/collection-flow.controller.ts [155] -{ id: tokenScope.workflowRuntimeDataId, name: 'RESUBMITTED' }, +{ id: tokenScope.workflowRuntimeDataId, name: BUILT_IN_EVENT.RESUBMITTED },
performance	Optimize array merging by ID using a Map for efficiency.                      Optimize the mergeArraysById function by using a Map for faster lookups instead of filtering the combined array for each id. packages/workflow-core/src/lib/utils/deep-merge-with-options.ts [21-28] -const combined = [...arr1, ...arr2]; -const ids = Array.from(new Set(combined.map(item => item.id))); -return ids.map(id => { - const sameIdItems = combined.filter(item => item.id === id); - return sameIdItems.reduce(mergeObjects, {}); +const idToItemMap = new Map(); +[...arr1, ...arr2].forEach(item => { + idToItemMap.set(item.id, mergeObjects(idToItemMap.get(item.id) || {}, item)); }); +return Array.from(idToItemMap.values());

---

✨ Improve tool usage guide:

---

Overview:
The improve tool scans the PR code changes, and automatically generates suggestions for improving the PR code. The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on a PR.
When commenting, to edit configurations related to the improve tool (pr_code_suggestions section), use the following template:

/improve --pr_code_suggestions.some_config1=... --pr_code_suggestions.some_config2=...

With a configuration file, use the following template:

[pr_code_suggestions]
some_config1=...
some_config2=...

Enabling\disabling automation When you first install the app, the default mode for the improve tool is: pr_commands = ["/improve --pr_code_suggestions.summarize=true", ...] meaning the improve tool will run automatically on every PR, with summarization enabled. Delete this line to disable the tool from running automatically.

Utilizing extra instructions Extra instructions are very important for the improve tool, since they enable to guide the model to suggestions that are more relevant to the specific needs of the project. Be specific, clear, and concise in the instructions. With extra instructions, you are the prompter. Specify relevant aspects that you want the model to focus on. Examples for extra instructions: [pr_code_suggestions] # /improve # extra_instructions=""" Emphasize the following aspects: - Does the code logic cover relevant edge cases? - Is the code logic clear and easy to understand? - Is the code logic efficient? ... """ Use triple quotes to write multi-line instructions. Use bullet points to make the instructions more readable.

A note on code suggestions quality While the current AI for code is getting better and better (GPT-4), it's not flawless. Not all the suggestions will be perfect, and a user should not accept all of them automatically. Suggestions are not meant to be simplistic. Instead, they aim to give deep feedback and raise questions, ideas and thoughts to the user, who can then use his judgment, experience, and understanding of the code base. Recommended to use the 'extra_instructions' field to guide the model to suggestions that are more relevant to the specific needs of the project, or use the custom suggestions 💎 tool With large PRs, best quality will be obtained by using 'improve --extended' mode.

More PR-Agent commands To invoke the PR-Agent, add a comment using one of the following commands: /review: Request a review of your Pull Request. /describe: Update the PR title and description based on the contents of the PR. /improve [--extended]: Suggest code improvements. Extended mode provides a higher quality feedback. /ask <QUESTION>: Ask a question about the PR. /update_changelog: Update the changelog based on the PR's contents. /add_docs 💎: Generate docstring for new components introduced in the PR. /generate_labels 💎: Generate labels for the PR based on the PR's contents. /analyze 💎: Automatically analyzes the PR, and presents changes walkthrough for each component. See the tools guide for more details. To list the possible configuration parameters, add a /config comment.

See the improve usage page for a more comprehensive guide on using this tool.