-
Notifications
You must be signed in to change notification settings - Fork 627
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docs: update rust example and readme #3775
Merged
Merged
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
49ab6b6
docs: update rust example and readme
korniltsev f142d44
Update examples/language-sdk-instrumentation/rust/README.md
korniltsev c1c6b4c
Update examples/language-sdk-instrumentation/rust/README.md
korniltsev 43d6f44
Update examples/language-sdk-instrumentation/rust/README.md
korniltsev 408f708
Update examples/language-sdk-instrumentation/rust/README.md
korniltsev af4d3ca
Update examples/language-sdk-instrumentation/rust/README.md
korniltsev fb35c02
Update examples/language-sdk-instrumentation/rust/README.md
korniltsev 2d0832d
Update examples/language-sdk-instrumentation/rust/README.md
korniltsev c7aebe1
Update examples/language-sdk-instrumentation/rust/README.md
korniltsev 7839cd9
Update examples/language-sdk-instrumentation/rust/README.md
korniltsev d67c803
Update examples/language-sdk-instrumentation/rust/README.md
korniltsev c29ea27
Update examples/language-sdk-instrumentation/rust/README.md
korniltsev 7500f92
Update examples/language-sdk-instrumentation/rust/README.md
korniltsev 870a126
Update examples/language-sdk-instrumentation/rust/README.md
korniltsev File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,148 @@ | ||
## Continuous Profiling for Rust applications | ||
|
||
### Profiling a Rust Rideshare App with Pyroscope | ||
|
||
![Image](https://github.com/user-attachments/assets/9a6e50a1-b8df-4923-9632-79ace3fea216) | ||
|
||
> [!NOTE] | ||
> For documentation on Pyroscope's Rust integration, refer to the [Rust push mode](https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/rust/) documentation. | ||
|
||
## Background | ||
|
||
This example shows a simplified, basic use case of Pyroscope that uses a "ride share" company which has three | ||
endpoints found in `main.rs`: | ||
|
||
- `/bike` : calls the `order_bike(search_radius)` function to order a bike | ||
- `/car` : calls the `order_car(search_radius)` function to order a car | ||
- `/scooter` : calls the `order_scooter(search_radius)` function to order a scooter | ||
|
||
The example also simulates running 3 distinct servers in 3 different regions ( | ||
via [docker-compose.yml](https://github.com/grafana/pyroscope/blob/main/examples/language-sdk-instrumentation/rust/rideshare/docker-compose.yml)): | ||
|
||
- us-east | ||
- eu-north | ||
- ap-south | ||
|
||
Pyroscope lets you tag your data in a way that is meaningful to you. In | ||
this case, there are two natural divisions, and so data is "tagged" to represent them: | ||
|
||
- `region`: statically tags the region of the server running the code | ||
- `vehicle`: dynamically tags the endpoint (similar to how one might tag a controller) | ||
|
||
## Tagging static region | ||
|
||
Tagging something static, like the `region`, can be done using `PyroscopeAgentBuilder#tags` method in the initialization | ||
code in the `main` function: | ||
|
||
```rust | ||
let agent = PyroscopeAgent::builder(server_address, app_name.to_owned()) | ||
.backend(pprof_backend(PprofConfig::new().sample_rate(100))) | ||
.tags(vec![("region", ®ion)]) | ||
.build()?; | ||
``` | ||
|
||
## Tagging dynamically within functions | ||
|
||
Tagging something more dynamically can be done using `PyroscopeAgent#tag_wrapper`. For example, you'd use code like this for the `vehicle` tag: | ||
|
||
```rust | ||
let (add_tag, remove_tag) = agent_running.tag_wrapper(); | ||
let add = Arc::new(add_tag); | ||
let remove = Arc::new(remove_tag); | ||
let car = warp::path("car").map(move || { | ||
add("vehicle".to_string(), "car".to_string()); | ||
order_car(3); | ||
remove("vehicle".to_string(), "car".to_string()); | ||
"Car ordered" | ||
}); | ||
``` | ||
|
||
This block does the following: | ||
|
||
1. Add the label `vehicle=car` | ||
2. Execute the `order_car` function | ||
3. Remove the label `vehicle=car` | ||
|
||
## Resulting flame graph / performance results from the example | ||
|
||
### Running the example | ||
|
||
To run the example, use the following commands: | ||
|
||
``` | ||
# Pull latest pyroscope and grafana images: | ||
docker pull grafana/pyroscope:latest | ||
docker pull grafana/grafana:latest | ||
|
||
# Run the example project: | ||
docker-compose up --build | ||
|
||
# Reset the database (if needed): | ||
# docker-compose down | ||
``` | ||
|
||
This example runs all the code mentioned above and also sends some mock-load to the 3 servers as well as | ||
their respective 3 endpoints. If you select `rust-ride-sharing-app` from the dropdown, you should see a | ||
flame graph that looks like this (below). Wait 20-30 seconds for the flame graph to update, and then click the | ||
refresh button to see 3 functions at the bottom of the flame graph taking CPU `resources _proportional` to the `size_` | ||
of their respective `search_radius` parameters. | ||
|
||
[//]: # (http://localhost:3000/a/grafana-pyroscope-app/profiles-explorer?searchText=&panelType=time-series&layout=grid&hideNoData=off&explorationType=flame-graph&var-serviceName=rust-ride-sharing-app&var-profileMetricId=process_cpu:cpu:nanoseconds:cpu:nanoseconds&var-dataSource=local-pyroscope&var-groupBy=all&var-filters=&maxNodes=16384&from=now-5m&to=now&var-filtersBaseline=&var-filtersComparison=) | ||
|
||
## Where's the performance bottleneck? | ||
|
||
![Image](https://github.com/user-attachments/assets/d4b0f85d-cc8d-4058-b019-1c5198849676) | ||
|
||
To analyze a profile outputted from your application, take note of the _largest node_ which is | ||
where your application is spending the most resources. In this case, it happens to be the `order_car` function. | ||
|
||
ThePyroscope package lets you investigate further as to _why_ the `order_car()` | ||
function is problematic. Tagging both `region` and `vehicle` allows us to test two good hypotheses: | ||
|
||
- Something is wrong with the `/car` endpoint code | ||
- Something is wrong with one of our regions | ||
|
||
To analyze this, select one or more tags on the "Labels" page: | ||
|
||
![Image](https://github.com/user-attachments/assets/3e5cb3ac-609e-493a-ae4d-248de150a33b) | ||
|
||
## Narrowing in on the Issue Using Tags | ||
|
||
Since you know there is an issue with the `order_car` function, select that tag. After inspecting | ||
multiple `region` tags, the timeline shows that there is an issue with the `eu-north` region, | ||
where it alternates between high-cpu times and low-cpu times. | ||
|
||
Note that the `mutex_lock()` function is consuming almost 70% of CPU resources during this time period. | ||
|
||
![Image](https://github.com/user-attachments/assets/12fc0912-8b65-4c24-9284-b0aa1eef45ba) | ||
|
||
## Visualizing Diff Between Two Flame graphs | ||
|
||
While the difference _in this case_ is stark enough to see in the comparison view, sometimes the diff between the two | ||
flame graphs is better visualized with them overlayed over each other. Without changing any parameters, you can | ||
select the diff view tab and see the difference represented in a color-coded diff flame graph. | ||
|
||
![Image](https://github.com/user-attachments/assets/97f6e51c-4211-4a0a-8f11-d2ee0402e396) | ||
|
||
### More use cases | ||
|
||
We have been beta testing this feature with several different companies and some of the ways that we've seen companies | ||
tag their performance data: | ||
|
||
- Tagging Kubernetes attributes | ||
- Tagging controllers | ||
- Tagging regions | ||
- Tagging jobs from a queue | ||
- Tagging commits | ||
- Tagging staging / production environments | ||
- Tagging different parts of their testing suites | ||
- Etc... | ||
|
||
### Future Roadmap | ||
|
||
We would love for you to try out this example and see what ways you can adapt this to your Rust application. Continuous | ||
profiling has become an increasingly popular tool for the monitoring and debugging of performance issues (arguably the | ||
fourth pillar of observability). | ||
|
||
We'd love to continue to improve our Rust integrations, and so we would love to hear what features _you would like to | ||
see_. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
target/ |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we include a suggested method for contacting us?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
idk, This was copypasted from somewhere else. There used to be a twitter handle on one of screenshots. If we need to suggest a method, it should be done everywhere