fix(job-executor): Adding helpful debug logs to improve job acquisition and job execution logging. Issue#2666. #4217

prakashpalanisamy · 2024-03-22T18:02:50Z

This code change adds additional logging to Camunda Job Executor and scoped to the Job Acquisition and Job Execution.

#2666

…on and job execution logging. Issue#2666.

…n logging. Issue#2666

CLAassistant · 2024-03-22T18:02:55Z

All committers have signed the CLA.

yanavasileva · 2024-03-25T07:35:55Z

Hi @prakashpalanisamy,

Thank you for raising this.
We will have a look at it and get back to you.

Best,
Yana

yanavasileva

@prakashpalanisamy, nice work!
I left a few comments for the code formatting (check all of the code for missing spaces) and covering the non-Spring use cases.

In addition to the changes, could you have have a look at adding a few test cases for the logging. The logging rule can be helpful for them. You can see examples here:

yanavasileva · 2024-04-08T09:11:43Z

engine/src/main/java/org/camunda/bpm/engine/impl/jobexecutor/JobExecutor.java

    }
  }

+  public void logJobExecutionInfo(ProcessEngineImpl engine, int executionQueueSize, int executionQueueCapacity, int maxExecutionThreads, int activeExecutionThreads) {


❓ The code is called only for Spring use cases. Have you considered how to log the same data for non-Spring scenario? As I don't think the ticket is created only for Spring use cases.
Hint:

camunda-bpm-platform/engine/src/main/java/org/camunda/bpm/engine/impl/jobexecutor/DefaultJobExecutor.java

Lines 43 to 45 in 5df0514

protected int queueSize = 3;

protected int corePoolSize = 3;

protected int maxPoolSize = 10;

❓ Could you also have a look at https://github.com/camunda/camunda-bpm-platform/blob/master/engine-cdi/core/src/main/java/org/camunda/bpm/engine/cdi/impl/ManagedJobExecutor.java?

engine/src/main/java/org/camunda/bpm/engine/impl/jobexecutor/JobExecutor.java

.../src/main/java/org/camunda/bpm/engine/impl/jobexecutor/SequentialJobAcquisitionRunnable.java

...re/src/main/java/org/camunda/bpm/engine/spring/components/jobexecutor/SpringJobExecutor.java

engine/src/main/java/org/camunda/bpm/engine/impl/jobexecutor/JobExecutorLogger.java

prakashpalanisamy · 2024-04-08T14:48:15Z

@yanavasileva , Thank you for taking your time to review. Ill look at the comments and update you. Thanks!!

yanavasileva · 2024-05-22T06:36:30Z

Closing due to inactivity. The PR can be reopened again when you get back to the topic.

prakashpalanisamy · 2024-06-28T08:43:58Z

Hi @yanavasileva Requesting you to please reopen the PR. I would like to continue the contribution to this PR. Thank you!

prakashpalanisamy · 2024-07-16T06:47:08Z

@yanavasileva , hope you are doing good. I have updated the PR with additional commits. Can you please take a look into it when you get some time. Thanks!!

yanavasileva · 2024-07-16T07:02:19Z

I will try to have a look and provide input later this week.

prakashpalanisamy · 2024-08-14T13:13:29Z

@yanavasileva Please let me know on the comments once you have finished reviewing, I will have them addressed. Thank you! :)

yanavasileva

❌ Formatting is off almost everywhere. You can find the latest formatters here:
https://github.com/camunda/camunda-bpm-platform/tree/master/settings
👍 Changes look good so far, thank you for considering my feedback.
I added two more notes below. I will validate the added tests, run some further scenarios, and get back to you with my findings.

Thank you for your patience.

yanavasileva · 2024-08-22T16:05:22Z

engine/src/main/java/org/camunda/bpm/engine/impl/jobexecutor/JobExecutorLogger.java

+  public void availableThreadsCalculationError() {
+    logWarn(
+            "039",
+            "Arithmetic exception occurred while computing remaining available thread count for logging.");
+  }
+
+  public void totalQueueCapacityCalculationError() {
+    logWarn(
+            "040",
+            "Arithmetic exception occurred while computing total queue capacity for logging.");
+  }


❌ Change severity to debug since the related logs are also logged with it + formatting:

Suggested change

public void availableThreadsCalculationError() {

logWarn(

"039",

"Arithmetic exception occurred while computing remaining available thread count for logging.");

}

public void totalQueueCapacityCalculationError() {

logWarn(

"040",

"Arithmetic exception occurred while computing total queue capacity for logging.");

}

public void availableThreadsCalculationError() {

logDebug("039", "Arithmetic exception occurred while computing remaining available thread count for logging.");

}

public void totalQueueCapacityCalculationError() {

logDebug("040", "Arithmetic exception occurred while computing total queue capacity for logging.");

}

yanavasileva · 2024-08-22T16:06:48Z

engine/src/main/java/org/camunda/bpm/engine/impl/jobexecutor/JobExecutorLogger.java

+  public void numJobsInQueue(String processEngine, int numJobsInQueue, int maxQueueSize) {
+    logDebug(
+            "038",
+            "Jobs currently in queue to be executed for the process engine '{}' is {} out of the max queue size : {}", processEngine, numJobsInQueue, maxQueueSize);


🔧 Adjust the wording for better readability:

Suggested change

"Jobs currently in queue to be executed for the process engine '{}' is {} out of the max queue size : {}", processEngine, numJobsInQueue, maxQueueSize);

"Jobs currently in queue to be executed for the process engine '{}': {} (out of the max queue size : {});

yanavasileva · 2024-08-22T16:36:45Z

engine/src/main/java/org/camunda/bpm/engine/impl/jobexecutor/JobExecutor.java

    }
  }

+  public void logJobExecutionInfo(ProcessEngineImpl engine, int executionQueueSize, int executionQueueCapacity, int maxExecutionThreads, int activeExecutionThreads) {


❓ Could you also have a look at https://github.com/camunda/camunda-bpm-platform/blob/master/engine-cdi/core/src/main/java/org/camunda/bpm/engine/cdi/impl/ManagedJobExecutor.java?

yanavasileva

Adding some further adjustments related to the tests.

yanavasileva · 2024-08-23T07:53:45Z

engine/src/main/java/org/camunda/bpm/engine/impl/jobexecutor/JobExecutor.java

+      LOG.numJobsInQueue(engine.getName(), executionQueueSize, executionQueueCapacity);
+      LOG.currentJobExecutions(engine.getName(), activeExecutionThreads);


🔧 Let's switch the order, I think it better to log executions first:

Suggested change

LOG.numJobsInQueue(engine.getName(), executionQueueSize, executionQueueCapacity);

LOG.currentJobExecutions(engine.getName(), activeExecutionThreads);

LOG.currentJobExecutions(engine.getName(), activeExecutionThreads);

LOG.numJobsInQueue(engine.getName(), executionQueueSize, executionQueueCapacity);

yanavasileva · 2024-08-23T08:20:33Z

engine/src/test/java/org/camunda/bpm/engine/test/jobexecutor/JobAcquisitionLoggingTest.java

+  @After
+  public void tearDown() {
+    List<Job> jobs = managementService.createJobQuery().processDefinitionKey("simpleAsyncProcess").list();
+
+    // remove simple async process jobs
+    for (Job job : jobs) {
+      managementService.deleteJob(job.getId());
+    }
+  }
+


❌ This is taken care of during the test destroy and deletion of the deployment.

Suggested change

@After

public void tearDown() {

List<Job> jobs = managementService.createJobQuery().processDefinitionKey("simpleAsyncProcess").list();

// remove simple async process jobs

for (Job job : jobs) {

managementService.deleteJob(job.getId());

}

}

yanavasileva · 2024-08-23T08:21:43Z

engine/src/test/java/org/camunda/bpm/engine/test/jobexecutor/JobAcquisitionLoggingTest.java

+import org.camunda.bpm.engine.runtime.Job;
+import org.camunda.bpm.engine.test.Deployment;
+import org.camunda.bpm.engine.test.util.ProcessEngineTestRule;
+import org.camunda.bpm.engine.test.util.ProvidedProcessEngineRule;
+import org.camunda.commons.testing.ProcessEngineLoggingRule;
+import org.junit.After;


❌ See above, not needed

Suggested change

import org.camunda.bpm.engine.runtime.Job;

import org.camunda.bpm.engine.test.Deployment;

import org.camunda.bpm.engine.test.util.ProcessEngineTestRule;

import org.camunda.bpm.engine.test.util.ProvidedProcessEngineRule;

import org.camunda.commons.testing.ProcessEngineLoggingRule;

import org.junit.After;

import org.camunda.bpm.engine.test.Deployment;

import org.camunda.bpm.engine.test.util.ProcessEngineTestRule;

import org.camunda.bpm.engine.test.util.ProvidedProcessEngineRule;

import org.camunda.commons.testing.ProcessEngineLoggingRule;

yanavasileva · 2024-08-23T08:23:36Z

engine/src/test/java/org/camunda/bpm/engine/test/jobexecutor/JobAcquisitionLoggingTest.java

+
+    // given max-job-acquisition threshold to three
+    processEngineConfiguration.getJobExecutor().setMaxJobsPerAcquisition(3);


❌ No need of this:

Suggested change

// given max-job-acquisition threshold to three

processEngineConfiguration.getJobExecutor().setMaxJobsPerAcquisition(3);

yanavasileva · 2024-08-23T08:26:02Z

engine/src/test/java/org/camunda/bpm/engine/test/jobexecutor/JobAcquisitionLoggingTest.java

+  private RuntimeService runtimeService;
+  private ManagementService managementService;
+  private ProcessEngineConfigurationImpl processEngineConfiguration;
+
+  @Before
+  public void init() {
+    runtimeService = engineRule.getRuntimeService();
+    processEngineConfiguration = engineRule.getProcessEngineConfiguration();
+    managementService = engineRule.getProcessEngine().getManagementService();
+  }


🔧 Clean up:

Suggested change

private RuntimeService runtimeService;

private ManagementService managementService;

private ProcessEngineConfigurationImpl processEngineConfiguration;

@Before

public void init() {

runtimeService = engineRule.getRuntimeService();

processEngineConfiguration = engineRule.getProcessEngineConfiguration();

managementService = engineRule.getProcessEngine().getManagementService();

}

protected RuntimeService runtimeService;

protected ProcessEngineConfigurationImpl processEngineConfiguration;

@Before

public void init() {

runtimeService = engineRule.getRuntimeService();

processEngineConfiguration = engineRule.getProcessEngineConfiguration();

}

yanavasileva · 2024-08-23T08:40:19Z

engine/src/test/java/org/camunda/bpm/engine/test/jobexecutor/JobAcquisitionLoggingTest.java

+
+import ch.qos.logback.classic.Level;
+import ch.qos.logback.classic.spi.ILoggingEvent;
+import org.camunda.bpm.engine.ManagementService;


Suggested change

import org.camunda.bpm.engine.ManagementService;

yanavasileva · 2024-08-23T08:40:58Z

engine/src/test/java/org/camunda/bpm/engine/test/jobexecutor/JobExecutionLoggingTest.java

+import org.camunda.bpm.engine.ManagementService;
+import org.camunda.bpm.engine.RuntimeService;
+import org.camunda.bpm.engine.impl.cfg.ProcessEngineConfigurationImpl;
+import org.camunda.bpm.engine.impl.jobexecutor.CallerRunsRejectedJobsHandler;
+import org.camunda.bpm.engine.impl.jobexecutor.DefaultJobExecutor;
+import org.camunda.bpm.engine.runtime.Job;
+import org.camunda.bpm.engine.test.Deployment;
+import org.camunda.bpm.engine.test.util.ProcessEngineTestRule;
+import org.camunda.bpm.engine.test.util.ProvidedProcessEngineRule;
+import org.camunda.commons.testing.ProcessEngineLoggingRule;
+import org.junit.After;


Clean up:

Suggested change

import org.camunda.bpm.engine.ManagementService;

import org.camunda.bpm.engine.RuntimeService;

import org.camunda.bpm.engine.impl.cfg.ProcessEngineConfigurationImpl;

import org.camunda.bpm.engine.impl.jobexecutor.CallerRunsRejectedJobsHandler;

import org.camunda.bpm.engine.impl.jobexecutor.DefaultJobExecutor;

import org.camunda.bpm.engine.runtime.Job;

import org.camunda.bpm.engine.test.Deployment;

import org.camunda.bpm.engine.test.util.ProcessEngineTestRule;

import org.camunda.bpm.engine.test.util.ProvidedProcessEngineRule;

import org.camunda.commons.testing.ProcessEngineLoggingRule;

import org.junit.After;

import org.camunda.bpm.engine.RuntimeService;

import org.camunda.bpm.engine.impl.cfg.ProcessEngineConfigurationImpl;

import org.camunda.bpm.engine.impl.jobexecutor.CallerRunsRejectedJobsHandler;

import org.camunda.bpm.engine.impl.jobexecutor.DefaultJobExecutor;

import org.camunda.bpm.engine.test.Deployment;

import org.camunda.bpm.engine.test.util.ProcessEngineTestRule;

import org.camunda.bpm.engine.test.util.ProvidedProcessEngineRule;

import org.camunda.commons.testing.ProcessEngineLoggingRule;

yanavasileva · 2024-08-23T08:41:48Z

engine/src/test/java/org/camunda/bpm/engine/test/jobexecutor/JobExecutionLoggingTest.java

+import org.junit.rules.RuleChain;
+
+import java.util.List;
+import java.util.concurrent.*;


❌ Apply project's formatter:

Suggested change

import java.util.concurrent.*;

import java.util.concurrent.ArrayBlockingQueue;

import java.util.concurrent.BlockingQueue;

import java.util.concurrent.RejectedExecutionException;

import java.util.concurrent.ThreadPoolExecutor;

import java.util.concurrent.TimeUnit;

yanavasileva · 2024-08-23T08:42:53Z

engine/src/test/java/org/camunda/bpm/engine/test/jobexecutor/JobExecutionLoggingTest.java

+  private ManagementService managementService;
+  private ProcessEngineConfigurationImpl processEngineConfiguration;
+
+  @Before
+  public void init() {
+    runtimeService = engineRule.getRuntimeService();
+    processEngineConfiguration = engineRule.getProcessEngineConfiguration();
+    managementService = engineRule.getProcessEngine().getManagementService();
+  }
+
+  @After
+  public void tearDown() {
+    List<Job> simpleAsyncProcessJobs = managementService.createJobQuery().processDefinitionKey("simpleAsyncProcess").list();
+    List<Job> testProcessJobs = managementService.createJobQuery().processDefinitionKey("testProcess").list();
+
+    // remove simple async process jobs
+    for (Job job : simpleAsyncProcessJobs) {
+      managementService.deleteJob(job.getId());
+    }
+
+    // remove test process jobs
+    for (Job job : testProcessJobs) {
+      managementService.deleteJob(job.getId());
+    }
+  }


❌ Removing of the jobs is taken care of the test rule during the deletion of the deployment:

Suggested change

private ManagementService managementService;

private ProcessEngineConfigurationImpl processEngineConfiguration;

@Before

public void init() {

runtimeService = engineRule.getRuntimeService();

processEngineConfiguration = engineRule.getProcessEngineConfiguration();

managementService = engineRule.getProcessEngine().getManagementService();

}

@After

public void tearDown() {

List<Job> simpleAsyncProcessJobs = managementService.createJobQuery().processDefinitionKey("simpleAsyncProcess").list();

List<Job> testProcessJobs = managementService.createJobQuery().processDefinitionKey("testProcess").list();

// remove simple async process jobs

for (Job job : simpleAsyncProcessJobs) {

managementService.deleteJob(job.getId());

}

// remove test process jobs

for (Job job : testProcessJobs) {

managementService.deleteJob(job.getId());

}

}

private ProcessEngineConfigurationImpl processEngineConfiguration;

@Before

public void init() {

runtimeService = engineRule.getRuntimeService();

processEngineConfiguration = engineRule.getProcessEngineConfiguration();

}

yanavasileva · 2024-08-23T08:45:06Z

engine/src/test/java/org/camunda/bpm/engine/test/jobexecutor/JobExecutionLoggingTest.java

+    processEngineConfiguration.getJobExecutor().shutdown();
+
+    // look for filled queue logs
+    List<ILoggingEvent> filteredLogList = loggingRule.getFilteredLog("Jobs currently in queue to be executed for the process engine 'default' is 2 out of the max queue size : 2");


🔧 Adjust wording:

Suggested change

List<ILoggingEvent> filteredLogList = loggingRule.getFilteredLog("Jobs currently in queue to be executed for the process engine 'default' is 2 out of the max queue size : 2");

List<ILoggingEvent> filteredLogList = loggingRule.getFilteredLog("Jobs currently in queue to be executed for the process engine 'default': 2 (out of the max queue size : 2)");

yanavasileva

❌ I have ran the tests a couple of times but they are unstable. Could you please iterate on them to make them reliable?

yanavasileva · 2024-08-23T12:35:58Z

engine/src/test/java/org/camunda/bpm/engine/test/jobexecutor/JobAcquisitionLoggingTest.java

❌ Missing license headers.

yanavasileva · 2024-08-23T12:36:06Z

engine/src/test/java/org/camunda/bpm/engine/test/jobexecutor/JobExecutionLoggingTest.java

❌ Missing license headers.

yanavasileva · 2024-08-23T12:51:32Z

engine/src/test/java/org/camunda/bpm/engine/test/jobexecutor/JobAcquisitionLoggingTest.java

+    List<ILoggingEvent> filteredLogList = loggingRule.getFilteredLog("Attempting to acquire 3 jobs for the process engine 'default'");
+
+    // then find the expected logs
+    assertThat(filteredLogList.size()).isGreaterThan(4);


This check if flaky.

yanavasileva · 2024-08-23T12:51:40Z

engine/src/test/java/org/camunda/bpm/engine/test/jobexecutor/JobAcquisitionLoggingTest.java

+    List<ILoggingEvent> filteredLogList = loggingRule.getFilteredLog("Jobs failed to Lock during Acquisition of jobs for the process engine 'default' : 0");
+
+    // then get logs that states there were no faults during job acquisition locks
+    assertThat(filteredLogList.size()).isGreaterThan(4);


This check if flaky.

yanavasileva · 2024-08-24T07:35:09Z

engine/src/test/java/org/camunda/bpm/engine/test/jobexecutor/JobExecutionLoggingTest.java

+    List<ILoggingEvent> filteredLogList = loggingRule.getFilteredLog("Jobs currently in queue to be executed for the process engine 'default' is 2 out of the max queue size : 2");
+
+    // then 3 instances of filled queue logs will be available as there is 2 additional threads possible to reach max-pool-size
+    assertThat(filteredLogList.size()).isEqualTo(3);


This check if flaky.

yanavasileva · 2024-08-24T07:38:08Z

engine/src/test/java/org/camunda/bpm/engine/test/jobexecutor/JobExecutionLoggingTest.java

+    List<ILoggingEvent> filteredLogList = loggingRule.getFilteredLog("Jobs currently in execution for the process engine 'default' : 3");
+
+    // then one count of 'three jobs in execution' will be available as 2 out of 5 jobs should be queued.
+    assertThat(filteredLogList.size()).isEqualTo(1);


This check is flaky.

yanavasileva · 2024-09-07T14:06:36Z

@prakashpalanisamy, heads up, we approach the code freeze of the minor release.
❗ If you want the feature to be included in 7.22.0, please apply the feedback by 16th of September.

prakashpalanisamy · 2024-09-09T11:52:54Z

@prakashpalanisamy, heads up, we approach the code freeze of the minor release. ❗ If you want the feature to be included in 7.22.0, please apply the feedback by 16th of September.

Sure @yanavasileva I'll take this up on priority. Was tied up due to personal matters. I may need your inputs on the Tests though. I observed that some of the new logs and already present logs were getting into the log file while the JE gets loaded. sometime twice. So the number of occurrences were varying on my system as well. Hence I was opting for greater than checks. Will test that once more and reach out to you with specifics for directions. Thank you for assisting me with this PR :)

prakashpalanisamy · 2024-09-10T14:10:14Z

@yanavasileva - quick question on the formatting. When I apply the formatting, it also suggest changes to prior code that other authors have contributed in the past, on the files that I have changes in. Do you suggest that I apply them along with my changes? else only to the lines that Im contributing and ignoring other formatting suggestions? Thanks!!

yanavasileva · 2024-09-10T14:28:41Z

@prakashpalanisamy,

You can mark only your code and format only that. This will be easier to review by me.

prakashpalanisamy · 2024-09-11T13:24:21Z

@yanavasileva - I need your directions for the ManagedJobExecutor changes. Since it will be using the app servers' thread pool, should I set it up in wildfly distribution with ManagedExecutorService configuration? if so, any pointers on how to do the configurations? If you could point to any reference links, it will help me.
Basically, once i put the code in, I want to get it up and running and test manually with some process of mine having async scenarios with java delegates.

yanavasileva · 2024-09-18T08:55:35Z

I won't be able to provide input on this, we can create a follow up ticket to explore further ManagedJobExecutor. Feel free to apply the rest of the feedback.

For the tests, I also saw that the log entries can vary. Maybe you can do the assertion for minimum occurrences?

prakashpalanisamy added 3 commits March 8, 2024 21:26

fix(job-executor): Adding helpful debug logs to improve job acquisiti…

9cb4278

…on and job execution logging. Issue#2666.

Adding helpful debug logs to improve job acquisition and job executio…

648af28

…n logging. Issue#2666

refactoring done.

513e3c8

prakashpalanisamy mentioned this pull request Mar 22, 2024

Improve job acquisition job execution logging #2666

Open

yanavasileva assigned prakashpalanisamy Mar 25, 2024

yanavasileva self-requested a review March 25, 2024 07:35

yanavasileva reviewed Apr 8, 2024

View reviewed changes

prakashpalanisamy added 3 commits April 10, 2024 15:31

Committing the Formatting suggestions received during review.

6d6540d

cleanup

1ae927e

job-acquisition-execution-logging-improvement-tomcat

4219778

yanavasileva closed this May 22, 2024

prakashpalanisamy added 3 commits May 27, 2024 17:00

job-acquisition-logging-defaultexecutorimpl-and-formatting

946b893

added test cases for the job-acquisition and job-execution logs

72d6e77

minor updates and comments added to the test cases

dff8540

tasso94 reopened this Jun 28, 2024

updated the formatting

1622149

brianwarner force-pushed the 2666-job-acquisition-execution-logging-improvement branch from b8380ab to 1622149 Compare July 11, 2024 16:13

yanavasileva self-requested a review August 14, 2024 13:21

yanavasileva requested changes Aug 22, 2024

View reviewed changes

yanavasileva reviewed Aug 23, 2024

View reviewed changes

yanavasileva requested changes Aug 24, 2024

View reviewed changes

	protected int queueSize = 3;
	protected int corePoolSize = 3;
	protected int maxPoolSize = 10;

	"Jobs currently in queue to be executed for the process engine '{}' is {} out of the max queue size : {}", processEngine, numJobsInQueue, maxQueueSize);
	"Jobs currently in queue to be executed for the process engine '{}': {} (out of the max queue size : {});

		LOG.numJobsInQueue(engine.getName(), executionQueueSize, executionQueueCapacity);
		LOG.currentJobExecutions(engine.getName(), activeExecutionThreads);


		// given max-job-acquisition threshold to three
		processEngineConfiguration.getJobExecutor().setMaxJobsPerAcquisition(3);

-import java.util.concurrent.*;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;

	List<ILoggingEvent> filteredLogList = loggingRule.getFilteredLog("Jobs currently in queue to be executed for the process engine 'default' is 2 out of the max queue size : 2");
	List<ILoggingEvent> filteredLogList = loggingRule.getFilteredLog("Jobs currently in queue to be executed for the process engine 'default': 2 (out of the max queue size : 2)");

fix(job-executor): Adding helpful debug logs to improve job acquisition and job execution logging. Issue#2666. #4217

Are you sure you want to change the base?

fix(job-executor): Adding helpful debug logs to improve job acquisition and job execution logging. Issue#2666. #4217

Conversation

prakashpalanisamy commented Mar 22, 2024 • edited by yanavasileva Loading

CLAassistant commented Mar 22, 2024 • edited Loading

yanavasileva commented Mar 25, 2024

yanavasileva left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prakashpalanisamy commented Apr 8, 2024

yanavasileva commented May 22, 2024

prakashpalanisamy commented Jun 28, 2024

prakashpalanisamy commented Jul 16, 2024

yanavasileva commented Jul 16, 2024

prakashpalanisamy commented Aug 14, 2024

yanavasileva left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yanavasileva Aug 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yanavasileva left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yanavasileva left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yanavasileva commented Sep 7, 2024

prakashpalanisamy commented Sep 9, 2024

prakashpalanisamy commented Sep 10, 2024

yanavasileva commented Sep 10, 2024

prakashpalanisamy commented Sep 11, 2024

yanavasileva commented Sep 18, 2024

prakashpalanisamy commented Mar 22, 2024 •

edited by yanavasileva

Loading

CLAassistant commented Mar 22, 2024 •

edited

Loading

yanavasileva Aug 22, 2024 •

edited

Loading