Skip to content

Conversation

@STKMakita
Copy link

@STKMakita STKMakita commented Oct 24, 2025

概要

GraalJSのエラーで以下のメッセージに一致する場合リトライを実行

Multi threaded access requested

参考:エラー発生時のログ

https://ma-digdag.zozo-inc.com/sessions/1234795
Caused by: com.oracle.truffle.polyglot.PolyglotIllegalStateException: Multi threaded access requested by thread Thread[0057@[0:zozo-notification:1234795:1236957]+delivery_workflow_starter+start_workflows^sub+get_waiting_request_ids,5,main] but is not allowed for language(s) js.と出力されている

2025-10-13 23:36:17.259 +0000 [ERROR] (0057@[0:zozo-notification:1234795:1236957]+delivery_workflow_starter+start_workflows^sub+get_waiting_request_ids) io.digdag.core.agent.OperatorManager: Task failed with unexpected error: Failed to process variables
java.lang.RuntimeException: Failed to process variables
	at io.digdag.core.agent.OperatorManager.runWithWorkspace(OperatorManager.java:246)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.lambda$runWithHeartbeat$2(OperatorManager.java:152)
	at io.digdag.core.agent.ExtractArchiveWorkspaceManager.withExtractedArchive(ExtractArchiveWorkspaceManager.java:75)
	at io.digdag.core.agent.OperatorManager.runWithHeartbeat(OperatorManager.java:150)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.run(OperatorManager.java:133)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.MultiThreadAgent.lambda$null$0(MultiThreadAgent.java:132)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.oracle.truffle.polyglot.PolyglotIllegalStateException: Multi threaded access requested by thread Thread[0057@[0:zozo-notification:1234795:1236957]+delivery_workflow_starter+start_workflows^sub+get_waiting_request_ids,5,main] but is not allowed for language(s) js.
	at com.oracle.truffle.polyglot.PolyglotContextImpl.throwDeniedThreadAccess(PolyglotContextImpl.java:638)
	at com.oracle.truffle.polyglot.PolyglotContextImpl.checkAllThreadAccesses(PolyglotContextImpl.java:541)
	at com.oracle.truffle.polyglot.PolyglotContextImpl.enterThreadChanged(PolyglotContextImpl.java:468)
	at com.oracle.truffle.polyglot.PolyglotEngineImpl.enter(PolyglotEngineImpl.java:1531)
	at com.oracle.truffle.polyglot.PolyglotEngineImpl.enterIfNeeded(PolyglotEngineImpl.java:1507)
	at com.oracle.truffle.polyglot.PolyglotLanguageContext.getHostBindings(PolyglotLanguageContext.java:293)
	at com.oracle.truffle.polyglot.PolyglotContextImpl.getBindings(PolyglotContextImpl.java:675)
	at org.graalvm.polyglot.Context.getBindings(Context.java:407)
	at io.digdag.core.agent.GraalJsEngine$GraalEvaluator.evaluate(GraalJsEngine.java:176)
	at io.digdag.core.agent.GraalJsEngine$GraalEvaluatorWithRetry.evaluate(GraalJsEngine.java:130)
	at io.digdag.core.agent.ConfigEvalEngine$Context.evalValue(ConfigEvalEngine.java:239)
	at io.digdag.core.agent.ConfigEvalEngine$Context.evalObjectRecursive(ConfigEvalEngine.java:195)
	at io.digdag.core.agent.ConfigEvalEngine$Context.access$300(ConfigEvalEngine.java:164)
	at io.digdag.core.agent.ConfigEvalEngine.eval(ConfigEvalEngine.java:321)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.evalConfig(OperatorManager.java:226)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.runWithWorkspace(OperatorManager.java:240)
	... 16 common frames omitted

動作確認

  • ビルドが通ることを確認
    • 参考:https://qiita.com/sonots/items/626b5d08b5876a9c3e35
    • 指定したdocker image:
      - image: digdag/digdag-build:20210121T160201-c21ea363746ab5ef7f7503a1a6212a37b79a9943
    • ビルド結果
      Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
      Use '--warning-mode all' to show the individual deprecation warnings.
      See https://docs.gradle.org/6.9.2/userguide/command_line_interface.html#sec:command_line_warnings
      
      BUILD SUCCESSFUL in 1m 9s
      37 actionable tasks: 2 executed, 35 up-to-date
      
      スクリーンショット 2025-10-28 11 54 58
  • マージ後にStaging環境でdelivery_workflow_starterを実行し正常終了することを確認

リリース後の対応

タグv0.10.5_patched_3作成
buildした成果物のjarを添付
https://github.com/st-tech/ma-gcp-infra/blob/main/docker/digdag/Dockerfile#L15
imageのhashを書き換えてリリースで完了
ref: https://github.com/st-tech/ma-gcp-infra/pull/1373

@STKMakita STKMakita requested a review from hnarimiya October 24, 2025 07:39
@STKMakita STKMakita changed the title GraalJSエラー時にリトライ(Multi threaded access requested〜) アラート対応:GraalJSエラー時にリトライ(Multi threaded access requested〜) Oct 24, 2025
Copy link

@hnarimiya hnarimiya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

こちら厳密なテストはいらないのですが、javaで

String message = "Multi threaded access requested by thread Thread[0057@[0:zozo-notification:1234795:1236957]+delivery_wo...";

if (message.startsWith("Multi threaded access requested")) {
  System.out.println("ok");
}

的なコードを書いて念の為確認お願いします

*/
String message = e.getMessage();
if ("Engine is already closed.".equals(message) || "The Context is already closed.".equals(message)) {
if ("Engine is already closed.".equals(message) || "The Context is already closed.".equals(message) || message.startsWith("Multi threaded access requested")) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

以下のような実行ごとに変わるメッセージのため、startsWithで比較していると理解しました。

 Multi threaded access requested by thread Thread[0057@[0:zozo-notification:1234795:1236957]+delivery_wo

@STKMakita
Copy link
Author

こちら厳密なテストはいらないのですが、javaで

String message = "Multi threaded access requested by thread Thread[0057@[0:zozo-notification:1234795:1236957]+delivery_wo...";

if (message.startsWith("Multi threaded access requested")) {
  System.out.println("ok");
}

的なコードを書いて念の為確認お願いします

サンプルを組んで実行しstartsWithでキャッチできることを確認しました。
■ソース

public class GraalJSTest
{
    public static void main(String[] args) {
        String message = "Multi threaded access requested by thread Thread[0057@[0:zozo-notification:1234795:1236957]+delivery_wo...";

        if (message.startsWith("Multi threaded access requested")) {
            System.out.println("input:"+message);
            System.out.println("ok");
        }
    }
}

■出力結果

> Task :digdag-core:io.digdag.core.GraalJSTest.main()
input:Multi threaded access requested by thread Thread[0057@[0:zozo-notification:1234795:1236957]+delivery_wo...
ok

@hnarimiya hnarimiya merged commit b337804 into v0.10.5_patched Nov 4, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants