Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: auto set GOMEMLIMIT when memory is limited #14882

Merged
merged 6 commits into from
Dec 12, 2024

Conversation

kruskall
Copy link
Member

@kruskall kruskall commented Dec 9, 2024

Motivation/summary

By default memlimit uses all the memory available on the host but if cgroups are used and the process is being limited we want to use the correct max memory.
Use automemlimit lib to set memory limit
Run check every 30s
Similar approach as automaxprocs

Test has to be performed manually (see below) because we can't really restrict the process memory :(

Checklist

For functional changes, consider:

  • Is it observable through the addition of either logging or metrics?
  • Is its use being published in telemetry to enable product improvement?
  • Have system tests been added to avoid regression?

How to test these changes

  • make apm-server
  • docker run -it -v ./:/apm-server --memory=4096m golang
  • cd /apm-server && ./apm-server -e
  • observe the the log line changing the memlimit

Related issues

Closes #14475

use automemlimit lib to set memory limit
run check every 30s
@kruskall kruskall requested a review from a team as a code owner December 9, 2024 21:28
Copy link
Contributor

mergify bot commented Dec 9, 2024

This pull request does not have a backport label. Could you fix it @kruskall? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-7.17 is the label to automatically backport to the 7.17 branch.
  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
  • backport-8.x is the label to automatically backport to the 8.x branch.

Copy link
Contributor

mergify bot commented Dec 9, 2024

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label.

@mergify mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Dec 9, 2024
@kruskall kruskall added backport-skip Skip notification from the automated backport with mergify and removed backport-8.x Automated backport to the 8.x branch with mergify labels Dec 9, 2024
@kruskall kruskall requested review from marclop and 1pkg December 12, 2024 11:56
@kruskall kruskall enabled auto-merge (squash) December 12, 2024 16:59
@kruskall kruskall merged commit 4940a7e into elastic:main Dec 12, 2024
11 checks passed
@kruskall kruskall deleted the feat/automemlimit branch December 12, 2024 17:08
@@ -348,6 +350,11 @@ func (b *Beat) Run(ctx context.Context) error {
return adjustMaxProcs(ctx, 30*time.Second, logger)
})

slogger := slog.New(zapslog.NewHandler(logger.Core()))
if err := adjustMemlimit(1*time.Second, slogger); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed this in the first review, but isn't a 1 second refresh interval a bit on the aggressive side? I haven't delved into the details of the library, but it seems like a waste of CPU cycles.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aaah, this was there for testing. It should be 30s like the max procs check 😭

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-skip Skip notification from the automated backport with mergify
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Set GOMEMLIMIT when running in a containerized environment
3 participants