[CI Breakage] More than 10 PRs failing due to E2E runtime test failures - "torch._dynamo.exc.InternalTorchDynamoError: TimeoutError: Timeout"

[ivangarcia44]

There are many fx_importer tests failing with the error below. Various pull requests have been affected starting at least from this one:

#4186

torch._dynamo.exc.InternalTorchDynamoError: TimeoutError: Timeout

 FAIL - "AdaptiveMaxPool3dDynamicWithIndices_basic"
      Runtime error: Traceback (most recent call last):
        File "/home/runner/_work/torch-mlir/torch-mlir/build/tools/torch-mlir/python_packages/torch_mlir/torch_mlir_e2e_test/framework.py", line 354, in compile_and_run_test
          trace = config.run(compiled, golden_trace)
        File "/home/runner/_work/torch-mlir/torch-mlir/build/tools/torch-mlir/python_packages/torch_mlir/torch_mlir_e2e_test/configs/fx_importer_backend.py", line 61, in run
          self._export_run(artifact, trace)
        File "/home/runner/_work/torch-mlir/torch-mlir/build/tools/torch-mlir/python_packages/torch_mlir/torch_mlir_e2e_test/configs/fx_importer_backend.py", line 137, in _export_run
          prog: ExportedProgram = torch.export.export(
        File "/home/runner/.local/lib/python3.10/site-packages/torch/export/__init__.py", line 319, in export
          raise e
        File "/home/runner/.local/lib/python3.10/site-packages/torch/export/__init__.py", line 286, in export
          return _export(
        File "/home/runner/.local/lib/python3.10/site-packages/torch/export/_trace.py", line 1153, in wrapper
          raise e
        File "/home/runner/.local/lib/python3.10/site-packages/torch/export/_trace.py", line 1119, in wrapper
          ep = fn(*args, **kwargs)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/export/exported_program.py", line 123, in wrapper
          return fn(*args, **kwargs)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/export/_trace.py", line 2175, in _export
          ep = _export_for_training(
        File "/home/runner/.local/lib/python3.10/site-packages/torch/export/_trace.py", line 1153, in wrapper
          raise e
        File "/home/runner/.local/lib/python3.10/site-packages/torch/export/_trace.py", line 1119, in wrapper
          ep = fn(*args, **kwargs)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/export/exported_program.py", line 123, in wrapper
          return fn(*args, **kwargs)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/export/_trace.py", line 2036, in _export_for_training
          export_artifact = export_func(
        File "/home/runner/.local/lib/python3.10/site-packages/torch/export/_trace.py", line 1410, in _strict_export
          gm_torch_level = _export_to_torch_ir(
        File "/home/runner/.local/lib/python3.10/site-packages/torch/export/_trace.py", line 800, in _export_to_torch_ir
          gm_torch_level, _ = torch._dynamo.export(
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1849, in inner
          result_traced = opt_f(*args, **kwargs)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 381, in __call__
          return super().__call__(*args, **kwargs)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
          return self._call_impl(*args, **kwargs)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
          return forward_call(*args, **kwargs)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 749, in compile_wrapper
          return fn(*args, **kwargs)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
          return self._call_impl(*args, **kwargs)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
          return forward_call(*args, **kwargs)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1562, in __call__
          return self._torchdynamo_orig_callable(
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 648, in __call__
          return _compile(
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1198, in _compile
          raise InternalTorchDynamoError(
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1137, in _compile
          guarded_code = compile_inner(code, one_graph, hooks, transform)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_utils_internal.py", line 98, in wrapper_function
          return function(*args, **kwargs)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 816, in compile_inner
          return _compile_inner(code, one_graph, hooks, transform)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 967, in _compile_inner
          check_fn = CheckFunctionManager(
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 2787, in __init__
          builder, guard_manager = self.build_guards(
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 3045, in build_guards
          guard.create(builder)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_guards.py", line 360, in create
          return self.create_fn(builder, self)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 2222, in SHAPE_ENV
          clib = CppCodeCache.load(func_str)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 2591, in load
          return cls.load_async(*args, **kwargs)()
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 2457, in load_async
          "vec_isa": pick_vec_isa(),
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_inductor/cpu_vec_isa.py", line 432, in pick_vec_isa
          _valid_vec_isa_list: list[VecISA] = valid_vec_isa_list()
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_inductor/cpu_vec_isa.py", line 419, in valid_vec_isa_list
          isa_list.extend(
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_inductor/cpu_vec_isa.py", line 419, in <genexpr>
          isa_list.extend(
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_inductor/cpu_vec_isa.py", line 147, in __bool__
          return self.__bool__impl(config.cpp.vec_isa_ok)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_inductor/cpu_vec_isa.py", line 157, in __bool__impl
          return self.check_build(VecISA._avx_code)
        File "/home/runner/.local/lib/python3.10/site-packages/torch/_inductor/cpu_vec_isa.py", line 108, in check_build
          with lock:
        File "/home/runner/.local/lib/python3.10/site-packages/torch/utils/_filelock.py", line 25, in __enter__
          result = super().__enter__()
        File "/home/runner/.local/lib/python3.10/site-packages/filelock/_api.py", line 376, in __enter__
          self.acquire()
        File "/home/runner/.local/lib/python3.10/site-packages/filelock/_api.py", line 344, in acquire
          time.sleep(poll_interval)
        File "/home/runner/_work/torch-mlir/torch-mlir/build/tools/torch-mlir/python_packages/torch_mlir/torch_mlir_e2e_test/framework.py", line 316, in handle_timeout
          raise TimeoutError(self.error_message)
      torch._dynamo.exc.InternalTorchDynamoError: TimeoutError: Timeout

The following tests fail for similar reason:

ElementwiseDivScalarRoundingModeTruncModule_basic
ElementwiseClampMaxModule_basic
ElementwiseAtenLogicalNotOpPromoteModule_basic
ElementwiseAtenIsneginfOpModule_basic
ElementwiseAsinhModule_basic
BoolTensorHandleSignless_basic
BatchNorm1DModule_basic
AvgPool3dCountIncludePadFalse_basic
AtenMmQuint8_basic
AtenIntTensorCharDtypeModule_basic
ArgmaxModule_with_dim
AddmmModuleFloat_basic

[ivangarcia44]

@powderluv @sjain-stanford @asaadaldien @ashay @sahas3 @vinitdeodhar @amd-vivekag @justin-ngo-arm

[ivangarcia44]

This issue was fixed here: #4271