Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

java.util.ConcurrentModificationException in the #155

Open
PeterTeunissen opened this issue Aug 2, 2022 · 6 comments
Open

java.util.ConcurrentModificationException in the #155

PeterTeunissen opened this issue Aug 2, 2022 · 6 comments

Comments

@PeterTeunissen
Copy link

PeterTeunissen commented Aug 2, 2022

We have build the ccd-validator and are calling the rest endpoint with different documents (with different validationObjectives) concurrently under pretty high volume.

We see the below ConcurrentModificationException in the code. Looking at the code the unload/reload methods on the Mu2consolePackage.eINSTANCE look like a static operation. That makes me wonder if the ccd validator is thread safe?

java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.remove(HashMap.java:1483)
	at org.hl7.security.ds4p.contentprofile.impl.CONTENTPROFILEPackageImpl.unload(CONTENTPROFILEPackageImpl.java:435)
	at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator.validateDocumentByTypeUsingMDHTApi(ReferenceCCDAValidator.java:160)
	at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator.validateFile(ReferenceCCDAValidator.java:108)
	at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator$$FastClassBySpringCGLIB$$fb042745.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
	at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator$$EnhancerBySpringCGLIB$$c5ebceb.validateFile(<generated>)
	at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.doMDHTValidation(ReferenceCCDAValidationService.java:202)
	at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.runValidators(ReferenceCCDAValidationService.java:127)
	at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDAImplementation(ReferenceCCDAValidationService.java:79)
	at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDA(ReferenceCCDAValidationService.java:69)
<<shortened>>
@swmuir
Copy link
Contributor

swmuir commented Aug 2, 2022 via email

@PeterTeunissen
Copy link
Author

PeterTeunissen commented Aug 2, 2022

It is sporadic (but persistent). I will try to recreate and update here. We have this integrated with another process, so the call is direct to the Rest endpoint (not through the UI). We will try some jmeter setup.

@PeterTeunissen
Copy link
Author

There are two more flavors of these errors:

 java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.remove(HashMap.java:1483)
	at org.openhealthtools.mdht.uml.cda.mu2consol.impl.Mu2consolPackageImpl.unload(Mu2consolPackageImpl.java:303)
	at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator.validateDocumentByTypeUsingMDHTApi(ReferenceCCDAValidator.java:159)
	at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator.validateFile(ReferenceCCDAValidator.java:108)
	at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator$$FastClassBySpringCGLIB$$fb042745.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
	at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator$$EnhancerBySpringCGLIB$$c5ebceb.validateFile(<generated>)
	at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.doMDHTValidation(ReferenceCCDAValidationService.java:202)
	at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.runValidators(ReferenceCCDAValidationService.java:127)
	at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDAImplementation(ReferenceCCDAValidationService.java:79)
	at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDA(ReferenceCCDAValidationService.java:69)
	at com.cerner.ccdavalidator.service.CCDAValidationService.validateCCDA(CCDAValidationService.java:117)
	at com.cerner.ccdavalidator.rest.CCDAValidationController.doValidationWithDomainNameAndLogicalDomainId(CCDAValidationController.java:106)

and

java.util.ConcurrentModificationException
	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:911)
	at java.util.ArrayList$Itr.remove(ArrayList.java:875)
	at org.sitenv.vocabularies.validation.services.VocabularyValidationService.limitConfiguredExpressionsBySeverity(VocabularyValidationService.java:174)
	at org.sitenv.vocabularies.validation.services.VocabularyValidationService.validate(VocabularyValidationService.java:219)
	at org.sitenv.vocabularies.validation.services.VocabularyValidationService.validate(VocabularyValidationService.java:136)
	at org.sitenv.vocabularies.validation.services.VocabularyValidationService.validate(VocabularyValidationService.java:105)
	at org.sitenv.referenceccda.validators.vocabulary.VocabularyCCDAValidator.doValidation(VocabularyCCDAValidator.java:78)
	at org.sitenv.referenceccda.validators.vocabulary.VocabularyCCDAValidator.validateFileImplementation(VocabularyCCDAValidator.java:66)
	at org.sitenv.referenceccda.validators.vocabulary.VocabularyCCDAValidator.validateFile(VocabularyCCDAValidator.java:54)
	at org.sitenv.referenceccda.validators.vocabulary.VocabularyCCDAValidator$$FastClassBySpringCGLIB$$b32d0b9.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
	at org.sitenv.referenceccda.validators.vocabulary.VocabularyCCDAValidator$$EnhancerBySpringCGLIB$$2d929969.validateFile(<generated>)
	at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.doVocabularyValidation(ReferenceCCDAValidationService.java:210)
	at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.runValidators(ReferenceCCDAValidationService.java:143)
	at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDAImplementation(ReferenceCCDAValidationService.java:79)
	at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDA(ReferenceCCDAValidationService.java:69)

@swmuir
Copy link
Contributor

swmuir commented Aug 2, 2022 via email

@PeterTeunissen
Copy link
Author

PeterTeunissen commented Aug 10, 2022

We have setup jmeter, with 4 different thread groups, each with an HTTP Request with a different XML document.
Each thread group makes 10 concurrent requests. So we hit the validators with 40 requests at the same time.
We have 2 instances of the validator running behind a load balancer, so each instance would be hit with roughly 20 requests concurrently.

Give it enough time and we get the ConcurrentModificationException.

@swmuir We could try to help fix this, but would need some pointers. You mentioned something in the first comment about maybe not unloading validation libraries?

@drbgfc
Copy link
Contributor

drbgfc commented Aug 16, 2022

In order to work as one WAR, we had to add a load/unload procedure to switch between MU2 and non-MU2 validation. If you make a separate WAR for MU2, or, you have no use for MU2 validation and never run it (2014 is pretty old at this point), then it should be a non-issue. We could certainly look into fixing this (or removing it). We recently had a discussion where we agreed to sunset MU2 in general. So, I'm not sure we will have the resources to change the operation. It would be interesting to know if you are using MU2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants