8349772: C2 hits MemLimit during incremental inlining

[dafedafe]

The issue was very intermittent but an old run that reproduced (for the crash with -XX:-TieredCompilation -XX:+AlwaysIncrementalInline) suggested the following was happening:
The test builds a very long MH adapter chain (in some runs up to ~10,000 transformations). Many small MH adapters are late-inlined, and C2 performs repeated incremental-inline/prune/IGVN rounds.
In each round, parser/IR/support data is allocated from compile arenas. Even when pruning removes dead nodes (live node count seems to remain moderate, e.g. ~13k), arena usage appears to be reclaimed only when the compilation ends.
So, for runs with long/complex chains, peak arena usage seems to cross MemLimit, which triggers a crash in debug (or a compilation bailout in product/non-crash mode).
Disabling/stopping late inlining at that point would probably only postpone the resource failure to a later phase.
JDK-8349820 disabled the explicit test-level MemLimit override. This change keeps a safeguard by using a larger explicit limit (2G) in debug builds.

Testing: tier1-3+

---

• I confirm that I make this contribution in accordance with the OpenJDK Interim AI Policy.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8349772: C2 hits MemLimit during incremental inlining (Bug - P4)

Reviewers

• Christian Hagedorn (@chhagedorn - Reviewer) 🔄 Re-review required (review applies to bbc67b51)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/30555/head:pull/30555
$ git checkout pull/30555

Update a local copy of the PR:
$ git checkout pull/30555
$ git pull https://git.openjdk.org/jdk.git pull/30555/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 30555

View PR using the GUI difftool:
$ git pr show -t 30555

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/30555.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back dfenacci! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@dafedafe The following label will be automatically applied to this pull request:

• hotspot-compiler

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 01: Full - Incremental (1617f869)
• 00: Full (bbc67b51)

[dean-long]

Might as well use the new ${test.main.class} for these two @run commands.

[dafedafe]

👍, fixed that.

[chhagedorn]

That looks reasonable to me.

[dafedafe]

/template append

[openjdk]

@dafedafe The pull request template has been appended to the pull request body