Skip to content

Commit 03d6e80

Browse files
committed
Introduce tokio-trace (tokio-rs#827)
<!-- Thank you for your Pull Request. Please provide a description above and review the requirements below. Bug fixes and new features should include tests. Contributors guide: https://github.com/tokio-rs/tokio/blob/master/CONTRIBUTING.md --> ## Motivation In asynchronous systems like Tokio, interpreting traditional log messages can often be quite challenging. Since individual tasks are multiplexed on the same thread, associated events and log lines are intermixed making it difficult to trace the logic flow. Currently, none of the available logging frameworks or libraries in Rust offer the ability to trace logical paths through a futures-based program. There also are complementary goals that can be accomplished with such a system. For example, metrics / instrumentation can be tracked by observing emitted events, or trace data can be exported to a distributed tracing or event processing system. In addition, it can often be useful to generate this diagnostic data in a structured manner that can be consumed programmatically. While prior art for structured logging in Rust exists, it is not currently standardized, and is not "Tokio-friendly". ## Solution This branch adds a new library to the tokio project, `tokio-trace`. `tokio-trace` expands upon logging-style diagnostics by allowing libraries and applications to record structured events with additional information about *temporality* and *causality* --- unlike a log message, a span in `tokio-trace` has a beginning and end time, may be entered and exited by the flow of execution, and may exist within a nested tree of similar spans. In addition, `tokio-trace` spans are *structured*, with the ability to record typed data as well as textual messages. The `tokio-trace-core` crate contains the core primitives for this system, which are expected to remain stable, while `tokio-trace` crate provides a more "batteries-included" API. In particular, it provides macros which are a superset of the `log` crate's `error!`, `warn!`, `info!`, `debug!`, and `trace!` macros, allowing users to begin the process of adopting `tokio-trace` by performing a drop-in replacement. ## Notes Work on this project had previously been carried out in the [tokio-trace-prototype] repository. In addition to the `tokio-trace` and `tokio-trace-core` crates, the `tokio-trace-prototype` repo also contains prototypes or sketches of adapter, compatibility, and utility crates which provide useful functionality for `tokio-trace`, but these crates are not yet ready for a release. When this branch is merged, that repository will be archived, and the remaining unstable crates will be moved to a new `tokio-trace-nursery` repository. Remaining issues on the `tokio-trace-prototype` repo will be moved to the appropriate new repo. The crates added in this branch are not _identical_ to the current head of the `tokio-trace-prototype` repo, as I did some final clean-up and docs polish in this branch prior to merging this PR. [tokio-trace-prototype]: https://github.com/hawkw/tokio-trace-prototype Closes: tokio-rs#561 Signed-off-by: Eliza Weisman <[email protected]>
1 parent 767ec8c commit 03d6e80

34 files changed

+6340
-83
lines changed

Cargo.toml

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
[workspace]
22

33
members = [
4+
".",
45
"tokio-trace-proc-macros",
56
"tokio-trace-fmt",
67
"tokio-trace-futures",
@@ -13,3 +14,34 @@ members = [
1314
"tokio-trace-subscriber",
1415
"tokio-trace-serde",
1516
]
17+
18+
[package]
19+
name = "tokio-trace"
20+
version = "0.0.1"
21+
authors = ["Eliza Weisman <[email protected]>"]
22+
license = "MIT"
23+
repository = "https://github.com/tokio-rs/tokio"
24+
homepage = "https://tokio.rs"
25+
description = """
26+
A scoped, structured logging and diagnostics system.
27+
"""
28+
categories = ["development-tools::debugging", "asynchronous"]
29+
keywords = ["logging", "tracing"]
30+
31+
# Not yet ready for production.
32+
publish = false
33+
34+
[dependencies]
35+
tokio-trace-core = { path = "tokio-trace-core" }
36+
37+
[dev-dependencies]
38+
ansi_term = "0.11"
39+
humantime = "1.1.1"
40+
futures = "0.1"
41+
log = "0.4"
42+
43+
# These are used for the "basic" example from the tokio-trace-prototype repo,
44+
# which is currently not included as it used the `tokio-trace-log` crate, and
45+
# that crate is currently unstable.
46+
# env_logger = "0.5"
47+
# tokio-trace-log = { path = "../tokio-trace-log" }

LICENSE

Lines changed: 21 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,25 @@
1-
MIT License
2-
31
Copyright (c) 2019 Tokio Contributors
42

5-
Permission is hereby granted, free of charge, to any person obtaining a copy
6-
of this software and associated documentation files (the "Software"), to deal
7-
in the Software without restriction, including without limitation the rights
8-
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9-
copies of the Software, and to permit persons to whom the Software is
10-
furnished to do so, subject to the following conditions:
3+
Permission is hereby granted, free of charge, to any
4+
person obtaining a copy of this software and associated
5+
documentation files (the "Software"), to deal in the
6+
Software without restriction, including without
7+
limitation the rights to use, copy, modify, merge,
8+
publish, distribute, sublicense, and/or sell copies of
9+
the Software, and to permit persons to whom the Software
10+
is furnished to do so, subject to the following
11+
conditions:
1112

12-
The above copyright notice and this permission notice shall be included in all
13-
copies or substantial portions of the Software.
13+
The above copyright notice and this permission notice
14+
shall be included in all copies or substantial portions
15+
of the Software.
1416

15-
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16-
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17-
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18-
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19-
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20-
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21-
SOFTWARE.
17+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
18+
ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
19+
TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
20+
PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
21+
SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
22+
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
23+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
24+
IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
25+
DEALINGS IN THE SOFTWARE.

README.md

Lines changed: 154 additions & 66 deletions
Original file line numberDiff line numberDiff line change
@@ -1,71 +1,161 @@
1-
# tokio-trace-nursery
1+
# tokio-trace
22

3-
Less-stable utility crates for [`tokio-trace`].
4-
5-
[![MIT licensed][mit-badge]][mit-url]
6-
[![Build Status][travis-badge]][travis-url]
7-
[![Gitter chat][gitter-badge]][gitter-url]
8-
9-
[mit-badge]: https://img.shields.io/badge/license-MIT-blue.svg
10-
[mit-url]: LICENSE
11-
[travis-badge]: https://travis-ci.org/tokio-rs/tokio-trace-nursery.svg?branch=master
12-
[travis-url]: https://travis-ci.org/tokio-rs/tokio-trace-nursery/branches
13-
[gitter-badge]: https://img.shields.io/gitter/room/tokio-rs/tokio.svg
14-
[gitter-url]: https://gitter.im/tokio-rs/tokio
15-
16-
[Website](https://tokio.rs) |
17-
[Chat](https://gitter.im/tokio-rs/tokio)
3+
A scoped, structured logging and diagnostics system.
184

195
## Overview
206

21-
[`tokio-trace`] is a framework for instrumenting Rust programs to collect
22-
structured, event-based diagnostic information. This repository contains a set
23-
of utility and compatibility crates for use with `tokio-trace`.
24-
25-
### Stability
26-
27-
While `tokio-trace` and `tokio-trace-core` have been published on crates.io and
28-
adhere to the same stability policies as the rest of the Tokio project, the
29-
crates in the nursery are generally less stable. Many of these crates are not
30-
yet released and are undergoing active development. Therefore, users are warned
31-
that breaking changes may occur.
32-
33-
In general, when depending on a crate from the nursery as a git dependency,
34-
users are advised to pin to a specific git revision using the [`rev`] Cargo key.
35-
This prevents your build from breaking should a breaking change to that crate be
36-
merged to master.
37-
38-
[`rev`]: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#specifying-dependencies-from-git-repositories
39-
40-
## Getting Help
41-
42-
First, see if the answer to your question can be found in the API documentation.
43-
If the answer is not there, there is an active community in
44-
the [Tokio Gitter channel][chat]. We would be happy to try to answer your
45-
question. Last, if that doesn't work, try opening an [issue] with the question.
46-
47-
[chat]: https://gitter.im/tokio-rs/tokio
48-
[issue]: https://github.com/tokio-rs/tokio-trace-nursery/issues/new
49-
50-
## Contributing
51-
52-
:balloon: Thanks for your help improving the project! We are so happy to have
53-
you! We have a [contributing guide][guide] to help you get involved in the Tokio
54-
project.
55-
56-
[guide]: CONTRIBUTING.md
57-
<!--
58-
## Project layout
59-
TODO: add this
60-
-->
61-
62-
## Supported Rust Versions
63-
64-
Tokio is built against the latest stable, nightly, and beta Rust releases. The
65-
minimum version supported is the stable release from three months before the
66-
current stable release version. For example, if the latest stable Rust is 1.29,
67-
the minimum version supported is 1.26. The current Tokio version is not
68-
guaranteed to build on Rust versions earlier than the minimum supported version.
7+
`tokio-trace` is a framework for instrumenting Rust programs to collect
8+
structured, event-based diagnostic information.
9+
10+
In asynchronous systems like Tokio, interpreting traditional log messages can
11+
often be quite challenging. Since individual tasks are multiplexed on the same
12+
thread, associated events and log lines are intermixed making it difficult to
13+
trace the logic flow. `tokio-trace` expands upon logging-style diagnostics by
14+
allowing libraries and applications to record structured events with additional
15+
information about *temporality* and *causality* — unlike a log message, a span
16+
in `tokio-trace` has a beginning and end time, may be entered and exited by the
17+
flow of execution, and may exist within a nested tree of similar spans. In
18+
addition, `tokio-trace` spans are *structured*, with the ability to record typed
19+
data as well as textual messages.
20+
21+
The `tokio-trace` crate provides the APIs necessary for instrumenting libraries
22+
and applications to emit trace data.
23+
24+
## Usage
25+
26+
First, add this to your `Cargo.toml`:
27+
28+
```toml
29+
[dependencies]
30+
tokio-trace = { git = "https://github.com/tokio-rs/tokio" }
31+
```
32+
33+
Next, add this to your crate:
34+
35+
```rust
36+
#[macro_use]
37+
extern crate tokio_trace;
38+
```
39+
40+
This crate provides macros for creating `Span`s and `Event`s, which represent
41+
periods of time and momentary events within the execution of a program,
42+
respectively.
43+
44+
As a rule of thumb, _spans_ should be used to represent discrete units of work
45+
(e.g., a given request's lifetime in a server) or periods of time spent in a
46+
given context (e.g., time spent interacting with an instance of an external
47+
system, such as a database). In contrast, _events_ should be used to represent
48+
points in time within a span — a request returned with a given status code,
49+
_n_ new items were taken from a queue, and so on.
50+
51+
`Span`s are constructed using the `span!` macro, and then _entered_
52+
to indicate that some code takes place within the context of that `Span`:
53+
54+
```rust
55+
// Construct a new span named "my span".
56+
let mut span = span!("my span");
57+
span.enter(|| {
58+
// Any trace events in this closure or code called by it will occur within
59+
// the span.
60+
});
61+
// Dropping the span will close it, indicating that it has ended.
62+
```
63+
64+
The `Event` type represent an event that occurs instantaneously, and is
65+
essentially a `Span` that cannot be entered. They are created using the `event!`
66+
macro:
67+
68+
```rust
69+
use tokio_trace::Level;
70+
event!(Level::INFO, "something has happened!");
71+
```
72+
73+
Users of the [`log`] crate should note that `tokio-trace` exposes a set of macros for
74+
creating `Event`s (`trace!`, `debug!`, `info!`, `warn!`, and `error!`) which may
75+
be invoked with the same syntax as the similarly-named macros from the `log`
76+
crate. Often, the process of converting a project to use `tokio-trace` can begin
77+
with a simple drop-in replacement.
78+
79+
Let's consider the `log` crate's yak-shaving example:
80+
81+
```rust
82+
#[macro_use]
83+
extern crate tokio_trace;
84+
use tokio_trace::field;
85+
86+
pub fn shave_the_yak(yak: &mut Yak) {
87+
// Create a new span for this invocation of `shave_the_yak`, annotated
88+
// with the yak being shaved as a *field* on the span.
89+
span!("shave_the_yak", yak = field::debug(&yak)).enter(|| {
90+
// Since the span is annotated with the yak, it is part of the context
91+
// for everything happening inside the span. Therefore, we don't need
92+
// to add it to the message for this event, as the `log` crate does.
93+
info!(target: "yak_events", "Commencing yak shaving");
94+
95+
loop {
96+
match find_a_razor() {
97+
Ok(razor) => {
98+
// We can add the razor as a field rather than formatting it
99+
// as part of the message, allowing subscribers to consume it
100+
// in a more structured manner:
101+
info!({ razor = field::display(razor) }, "Razor located");
102+
yak.shave(razor);
103+
break;
104+
}
105+
Err(err) => {
106+
// However, we can also create events with formatted messages,
107+
// just as we would for log records.
108+
warn!("Unable to locate a razor: {}, retrying", err);
109+
}
110+
}
111+
}
112+
})
113+
}
114+
```
115+
116+
You can find examples showing how to use this crate in the examples directory.
117+
118+
### In libraries
119+
120+
Libraries should link only to the `tokio-trace` crate, and use the provided
121+
macros to record whatever information will be useful to downstream consumers.
122+
123+
### In executables
124+
125+
In order to record trace events, executables have to use a `Subscriber`
126+
implementation compatible with `tokio-trace`. A `Subscriber` implements a way of
127+
collecting trace data, such as by logging it to standard output.
128+
129+
Unlike the `log` crate, `tokio-trace` does *not* use a global `Subscriber` which
130+
is initialized once. Instead, it follows the `tokio` pattern of executing code
131+
in a context. For example:
132+
133+
```rust
134+
#[macro_use]
135+
extern crate tokio_trace;
136+
137+
let my_subscriber = FooSubscriber::new();
138+
139+
tokio_trace::subscriber::with_default(subscriber, || {
140+
// Any trace events generated in this closure or by functions it calls
141+
// will be collected by `my_subscriber`.
142+
})
143+
```
144+
145+
This approach allows trace data to be collected by multiple subscribers within
146+
different contexts in the program. Alternatively, a single subscriber may be
147+
constructed by the `main` function and all subsequent code executed with that
148+
subscriber as the default. Any trace events generated outside the context of a
149+
subscriber will not be collected.
150+
151+
The executable itself may use the `tokio-trace` crate to instrument itself as
152+
well.
153+
154+
The [`tokio-trace-nursery`] repository contains less stable crates designed to
155+
be used with the `tokio-trace` ecosystem. It includes a collection of
156+
`Subscriber` implementations, as well as utility and adapter crates.
157+
158+
[`log`]: https://docs.rs/log/0.4.6/log/
69159

70160
## License
71161

@@ -76,5 +166,3 @@ This project is licensed under the [MIT license](LICENSE).
76166
Unless you explicitly state otherwise, any contribution intentionally submitted
77167
for inclusion in Tokio by you, shall be licensed as MIT, without any additional
78168
terms or conditions.
79-
80-
[`tokio-trace`]: https://github.com/tokio-rs/tokio/tree/master/tokio-trace

benches/no_subscriber.rs

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
#![feature(test)]
2+
#[macro_use]
3+
extern crate tokio_trace;
4+
#[macro_use]
5+
extern crate log;
6+
extern crate test;
7+
use test::Bencher;
8+
9+
#[bench]
10+
fn bench_span_no_subscriber(b: &mut Bencher) {
11+
b.iter(|| {
12+
span!("span");
13+
});
14+
}
15+
16+
#[bench]
17+
fn bench_log_no_logger(b: &mut Bencher) {
18+
b.iter(|| {
19+
log!(log::Level::Info, "log");
20+
});
21+
}
22+
23+
#[bench]
24+
fn bench_costly_field_no_subscriber(b: &mut Bencher) {
25+
b.iter(|| {
26+
span!(
27+
"span",
28+
foo = tokio_trace::field::display(format!("bar {:?}", 2))
29+
);
30+
});
31+
}
32+
33+
#[bench]
34+
fn bench_no_span_no_subscriber(b: &mut Bencher) {
35+
b.iter(|| {});
36+
}
37+
38+
#[bench]
39+
fn bench_1_atomic_load(b: &mut Bencher) {
40+
// This is just included as a baseline.
41+
use std::sync::atomic::{AtomicUsize, Ordering};
42+
let foo = AtomicUsize::new(1);
43+
b.iter(|| foo.load(Ordering::Relaxed));
44+
}

0 commit comments

Comments
 (0)