pyroscope.ebpf: Retry after eBPF errors and buffer config updates.#3381
Merged
simonswine merged 5 commits intomainfrom May 23, 2025
Merged
pyroscope.ebpf: Retry after eBPF errors and buffer config updates.#3381simonswine merged 5 commits intomainfrom
simonswine merged 5 commits intomainfrom
Conversation
simonswine
added
the
area/pyroscope
simonswine
force-pushed
the
20250416_pyroscope.ebpf-error-propagation
branch
from
312558c to
9e1854d
Compare
simonswine
changed the title
simonswine
changed the title
simonswine
commented
Apr 17, 2025
Contributor
Author
|
@JohanLindvall wonder if this fixes your problem and/or gives you a better error message? |
@simonswine With these changes our Alloy Pyroscope scrapers have been running for a few hours without any issues. I have not seen any of the new error messages logged. |
Contributor
|
This PR has not had any activity in the past 30 days, so the |
If the eBPF component encounter an error, it stops the main processing loops, which then caused issues because any config reload for alloy would block the whole agent. Fixes #3332 Also for long running scrapes we were blocking the component evaluation as we are waiting for the config to be updated. I switched this to be a buffered channel, so the config updated are unblocked. In case of multiple updated buffered it will skip the in between updates and go straight to the latest version. Fixes #1371
simonswine
force-pushed
the
20250416_pyroscope.ebpf-error-propagation
branch
from
9e1854d to
19591e0
Compare
Contributor
Author
|
I have built an image with the changes from this PR: In case someone wants to try: |
simonswine
force-pushed
the
20250416_pyroscope.ebpf-error-propagation
branch
from
19591e0 to
8d79504
Compare
aleks-p
approved these changes
May 23, 2025
Contributor
aleks-p
left a comment
aleks-p
left a comment
There was a problem hiding this comment.
Looks good to me. I noticed the added error logging which is great. I would consider adding Info and Debug logs in a few select places to better understand what happens during important state transitions.
Contributor
|
The backport to To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-3381-to-release/v1.8 origin/release/v1.8
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 9c21229be370614d21f9cc4251aae85b4e689cba
# Push it to GitHub
git push --set-upstream origin backport-3381-to-release/v1.8
git switch main
# Remove the local backport branch
git branch -D backport-3381-to-release/v1.8Then, create a pull request where the |
simonswine
added a commit
to simonswine/alloy
that referenced
this pull request
May 23, 2025
…rafana#3381) * fix: Keep ebpf component loop running If the eBPF component encounter an error, it stops the main processing loops, which then caused issues because any config reload for alloy would block the whole agent. Fixes grafana#3332 Also for long running scrapes we were blocking the component evaluation as we are waiting for the config to be updated. I switched this to be a buffered channel, so the config updated are unblocked. In case of multiple updated buffered it will skip the in between updates and go straight to the latest version. Fixes grafana#1371 * Fix data racec in test * Limit tries to start an ebpf session * Add info logs at session start/end * Pass context around to shutdown properly (cherry picked from commit 9c21229)
simonswine
added a commit
to simonswine/alloy
that referenced
this pull request
May 23, 2025
…rafana#3381) * fix: Keep ebpf component loop running If the eBPF component encounter an error, it stops the main processing loops, which then caused issues because any config reload for alloy would block the whole agent. Fixes grafana#3332 Also for long running scrapes we were blocking the component evaluation as we are waiting for the config to be updated. I switched this to be a buffered channel, so the config updated are unblocked. In case of multiple updated buffered it will skip the in between updates and go straight to the latest version. Fixes grafana#1371 * Fix data racec in test * Limit tries to start an ebpf session * Add info logs at session start/end * Pass context around to shutdown properly (cherry picked from commit 9c21229)
wildum
pushed a commit
that referenced
this pull request
May 26, 2025
…3381) * fix: Keep ebpf component loop running If the eBPF component encounter an error, it stops the main processing loops, which then caused issues because any config reload for alloy would block the whole agent. Fixes #3332 Also for long running scrapes we were blocking the component evaluation as we are waiting for the config to be updated. I switched this to be a buffered channel, so the config updated are unblocked. In case of multiple updated buffered it will skip the in between updates and go straight to the latest version. Fixes #1371 * Fix data racec in test * Limit tries to start an ebpf session * Add info logs at session start/end * Pass context around to shutdown properly
wildum
added a commit
that referenced
this pull request
May 26, 2025
* pyroscope.ebpf: Retry after eBPF errors and buffer config updates. (#3381) * fix: Keep ebpf component loop running If the eBPF component encounter an error, it stops the main processing loops, which then caused issues because any config reload for alloy would block the whole agent. Fixes #3332 Also for long running scrapes we were blocking the component evaluation as we are waiting for the config to be updated. I switched this to be a buffered channel, so the config updated are unblocked. In case of multiple updated buffered it will skip the in between updates and go straight to the latest version. Fixes #1371 * Fix data racec in test * Limit tries to start an ebpf session * Add info logs at session start/end * Pass context around to shutdown properly * fix(pyroscope.scrape): godeltaprof scraping (#3542) * fix(pyroscope.scrape): godeltaprof scraping * cl * update changelog --------- Co-authored-by: Christian Simon <simon@swine.de> Co-authored-by: Tolya Korniltsev <korniltsev.anatoly@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
If the eBPF component encounters an error, it stops the main processing loops, which then caused issues because any config reload for alloy would block the whole agent.
Fixes #3332
Also for long running scrapes we were blocking the component evaluation as we are waiting for the config to be updated. I switched this to be a buffered channel, so the config updated are unblocked. In case of multiple updated buffered it will skip the in between updates and go straight to the latest version.
Fixes #1371