Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ docs/ AsciiDoc documentation, file format spec, diagrams.
- Macros use `__VA_OPT__` (C++20/C23/GCC extension). NEVER replace with `##__VA_ARGS__` -- they are not equivalent.
- `CLLTK_TRACEPOINT_DUMP` takes `(buffer, message, address, size)` -- NOT a printf format. Do not confuse with `CLLTK_TRACEPOINT`.
- `CLLTK_DYN_TRACEPOINT` takes a string buffer name (not an identifier) and binds at runtime. Slower than static tracepoints.
- `CLLTK_TRACEPOINT_FMT` (meta type 5) is C++20-only; a `char*` argument is always recorded as a string ({} has no %p/%s ambiguity) - cast to `void*` for the pointer value. Format validation happens at compile time via a never-executed `_clltk_fmt_check` call.
- `CLLTK_SPAN_BEGIN` is a GNU statement expression that evaluates to the new span id (`clltk_span_id_t`, a plain uint64; 0 = no parent). Span begin/end are meta entry types 3/4 and carry their ids as ordinary uint64 arguments; decoders correlate begin/end by id, not by nesting.

### ELF Sections
Expand Down
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,15 @@ The name is used to associated tracepoint with this tracebuffer and is also the
The tracepoints `CLLTK_TRACEPOINT` defines first the target tracebuffer, than the format string, followed by the arguments.


### fmt-style format strings (C++20)

`CLLTK_TRACEPOINT_FMT` accepts std::format style `{}` placeholders, validated
against the argument types at compile time:

```cpp
CLLTK_TRACEPOINT_FMT(MyFirstTracebuffer, "loaded {} in {}ms", name, duration);
```

### Spans

Spans track scoped work with begin/end events and a carryable id:
Expand Down
9 changes: 8 additions & 1 deletion VERSION.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
1.5.1
1.6.0

# Change log
## 1.6.0
- feat: fmt-style tracepoints (C++20 only). `CLLTK_TRACEPOINT_FMT(buffer, "loaded {} in {}ms",
name, ms)` uses std::format {} placeholders, validated against the argument types at compile
time via std::format_string. Argument encoding is unchanged; both decoders render the new
meta entry type 5 with their native {} formatters. A char* argument is always recorded as a
string (no %p/%s ambiguity). New meta type = file layout change, hence the minor bump.
Kernel: not applicable (C API).
## 1.5.1
- feat: `clltk export` converts trace files into Chrome/Perfetto trace event JSON
(open in ui.perfetto.dev): tracepoints as instant events, spans as async begin/end
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ namespace CommonLowLevelTracingKit::decoder {
Printf = 1,
Dump = 2,
SpanBegin = 3,
SpanEnd = 4
SpanEnd = 4,
Fmt = 5
};

struct EXPORT MetaEntryInfo {
Expand Down
4 changes: 3 additions & 1 deletion decoder_tool/cpp/source/Tracepoint.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,8 @@ const std::string_view TracepointStatic::msg() const {
std::span<const uint8_t> args_raw{arg_start, arg_size};
if (m_type == MetaType::printf) [[likely]] {
m_msg = source::formatter::printf(format(), m_arg_types, args_raw, e->foreignEndian());
} else if (m_type == MetaType::fmt) {
m_msg = source::formatter::fmt(format(), m_arg_types, args_raw, e->foreignEndian());
} else if (m_type == MetaType::dump) {
m_msg = source::formatter::dump(format(), m_arg_types, args_raw, e->foreignEndian());
} else if (m_type == MetaType::span_begin) {
Expand All @@ -161,7 +163,7 @@ const std::string_view TracepointStatic::msg() const {
CLLTK_DECODER_THROW(
exception::InvalidMeta,
"Invalid meta data type: " + std::to_string(static_cast<uint8_t>(m_type)) +
" (expected printf=1, dump=2, span_begin=3, or span_end=4)");
" (expected printf=1, dump=2, span_begin=3, span_end=4, or fmt=5)");
}
}
return m_msg;
Expand Down
5 changes: 3 additions & 2 deletions decoder_tool/cpp/source/TracepointInternal.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -106,10 +106,11 @@ namespace CommonLowLevelTracingKit::decoder {
printf = 1,
dump = 2,
span_begin = 3,
span_end = 4
span_end = 4,
fmt = 5
};
static constexpr CONST_INLINE MetaType toMetaType(uint8_t a) {
if (a <= static_cast<uint8_t>(MetaType::span_end)) return static_cast<MetaType>(a);
if (a <= static_cast<uint8_t>(MetaType::fmt)) return static_cast<MetaType>(a);
return MetaType::undefined;
}
class TracepointStatic final : public TraceEntryHead {
Expand Down
110 changes: 110 additions & 0 deletions decoder_tool/cpp/source/low_level/formatter.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
#include <cctype>
#include <cstring>
#include <ffi.h>
#include <format>
#include <stdio.h>
#include <variant>

Expand Down Expand Up @@ -337,6 +338,115 @@ std::string formatter::printf(const std::string_view format, const std::span<con
return msg;
}

// Variant type that can hold any decoded argument value for std::vformat.
// std::make_format_args requires lvalues so we store into a variant array first.
using fmt_arg_value = std::variant<uint64_t, int64_t, double, std::string>;

// Read a POD value of type T from raw memory at clltk_arg, byte-swapping if needed.
template <typename T>
INLINE static T read_raw(uintptr_t clltk_arg, size_t remaining, bool foreign_endian) {
if (sizeof(T) > remaining) [[unlikely]]
CLLTK_DECODER_THROW(FormattingFailed, "out of range access for fmt formatter");
T value{};
memcpy(&value, reinterpret_cast<const void *>(clltk_arg), sizeof(T));
if constexpr (CommonLowLevelTracingKit::decoder::source::internal::ByteSwappable<T>) {
if (foreign_endian)
value = CommonLowLevelTracingKit::decoder::source::internal::byteswapValue(value);
}
return value;
}

INLINE static fmt_arg_value clltk_arg_to_fmt_value(const char clltk_type, const uintptr_t clltk_arg,
size_t remaining, bool foreign_endian) {
switch (clltk_type) {
case 'c': return static_cast<uint64_t>(read_raw<uint8_t>(clltk_arg, remaining, foreign_endian));
case 'w':
return static_cast<uint64_t>(read_raw<uint16_t>(clltk_arg, remaining, foreign_endian));
case 'i':
return static_cast<uint64_t>(read_raw<uint32_t>(clltk_arg, remaining, foreign_endian));
case 'l':
return static_cast<uint64_t>(read_raw<uint64_t>(clltk_arg, remaining, foreign_endian));
case 'C': return static_cast<int64_t>(read_raw<int8_t>(clltk_arg, remaining, foreign_endian));
case 'W': return static_cast<int64_t>(read_raw<int16_t>(clltk_arg, remaining, foreign_endian));
case 'I': return static_cast<int64_t>(read_raw<int32_t>(clltk_arg, remaining, foreign_endian));
case 'L': return static_cast<int64_t>(read_raw<int64_t>(clltk_arg, remaining, foreign_endian));
case 'f': return static_cast<double>(read_raw<float>(clltk_arg, remaining, foreign_endian));
case 'd': return static_cast<double>(read_raw<double>(clltk_arg, remaining, foreign_endian));
case 'p':
return static_cast<uint64_t>(read_raw<uint64_t>(clltk_arg, remaining, foreign_endian));
case 's': {
// string: uint32 length prefix then null-terminated chars; skip the length prefix
return std::string{reinterpret_cast<const char *>(clltk_arg + sizeof(uint32_t))};
}
default: CLLTK_DECODER_THROW(FormattingFailed, "unknown type for fmt");
}
}

// Wrapper so one std::formatter specialization can format any decoded value.
// Dispatching the concrete types at the make_format_args call site would need
// nested std::visit over all argument slots, instantiating vformat for every
// type combination (4^10). Instead the wrapper's formatter captures the format
// spec during parse() and applies it to the variant's active alternative with
// a single std::visit per argument.
struct FmtDecodedArg {
fmt_arg_value value{};
};

template <> struct std::formatter<FmtDecodedArg, char> {
std::string spec;

constexpr auto parse(std::format_parse_context &ctx) {
auto it = ctx.begin();
const auto end = ctx.end();
while (it != end && *it != '}') { spec += *it++; }
return it;
}

auto format(const FmtDecodedArg &arg, std::format_context &ctx) const {
const std::string one_arg_format = "{:" + spec + "}";
return std::visit(
[&](const auto &value) {
return std::vformat_to(ctx.out(), one_arg_format, std::make_format_args(value));
},
arg.value);
}
};

std::string formatter::fmt(const std::string_view format, const std::span<const char> &types_raw,
const std::span<const uint8_t> &args_raw, bool foreign_endian) {
if (format.empty()) return "";

// Decode all arguments into a variant array so we have lvalues for make_format_args.
static constexpr size_t max_fmt_args = 10;
std::array<FmtDecodedArg, max_fmt_args> decoded{};
const size_t arg_count = types_raw.size();
if (arg_count > max_fmt_args) CLLTK_DECODER_THROW(FormattingFailed, "too many fmt args");

size_t raw_offset = 0;
for (size_t i = 0; i < arg_count; ++i) {
const char type = types_raw[i];
if (raw_offset >= args_raw.size())
CLLTK_DECODER_THROW(FormattingFailed, "out of range access for fmt formatter");
const uintptr_t current = std::bit_cast<uintptr_t>(&args_raw[raw_offset]);
const size_t remaining = args_raw.size() - raw_offset;
// Validate and compute size first (clltk_arg_to_size checks string bounds).
const size_t arg_size = clltk_arg_to_size(type, current, remaining, foreign_endian);
decoded[i].value = clltk_arg_to_fmt_value(type, current, remaining, foreign_endian);
raw_offset += arg_size;
}
if (raw_offset != args_raw.size()) [[unlikely]]
CLLTK_DECODER_THROW(FormattingFailed, "raw args invalid");

// Pass all slots; std::vformat ignores arguments the format string does not
// reference, so the unused trailing (default-constructed) slots are harmless.
try {
return std::vformat(format,
std::make_format_args(decoded[0], decoded[1], decoded[2], decoded[3],
decoded[4], decoded[5], decoded[6], decoded[7],
decoded[8], decoded[9]));
} catch (const std::format_error &) { return std::string(format) + " fmt-error"; }
}

std::string formatter::dump(const std::string_view format, const std::span<const char> &types_raw,
const std::span<const uint8_t> &args_raw, bool foreign_endian) {
if (types_raw.size() != 1 || types_raw[0] != 'x')
Expand Down
2 changes: 2 additions & 0 deletions decoder_tool/cpp/source/low_level/formatter.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,7 @@ namespace CommonLowLevelTracingKit::decoder::source::formatter {
const std::span<const uint8_t> &args_raw, bool foreign_endian = false);
std::string dump(const std::string_view format, const std::span<const char> &types,
const std::span<const uint8_t> &args_raw, bool foreign_endian = false);
std::string fmt(const std::string_view format, const std::span<const char> &types,
const std::span<const uint8_t> &args_raw, bool foreign_endian = false);

} // namespace CommonLowLevelTracingKit::decoder::source::formatter
3 changes: 3 additions & 0 deletions decoder_tool/cpp/source/low_level/meta_parser.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,8 @@ namespace CommonLowLevelTracingKit::decoder::source {
entry.type = MetaEntryType::SpanBegin;
} else if (type_byte == static_cast<uint8_t>(MetaEntryType::SpanEnd)) {
entry.type = MetaEntryType::SpanEnd;
} else if (type_byte == static_cast<uint8_t>(MetaEntryType::Fmt)) {
entry.type = MetaEntryType::Fmt;
} else {
entry.type = MetaEntryType::Unknown;
}
Expand Down Expand Up @@ -106,6 +108,7 @@ namespace CommonLowLevelTracingKit::decoder {
case MetaEntryType::Dump: return "dump";
case MetaEntryType::SpanBegin: return "span_begin";
case MetaEntryType::SpanEnd: return "span_end";
case MetaEntryType::Fmt: return "fmt";
case MetaEntryType::Unknown:
default: return "unknown";
}
Expand Down
38 changes: 37 additions & 1 deletion decoder_tool/python/clltk_decoder.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ class MetaEntryType(Enum):
Dump = 2
SpanBegin = 3
SpanEnd = 4
Fmt = 5

class Endian(Enum):
Big = "big"
Expand Down Expand Up @@ -436,7 +437,8 @@ def __init__(self, entry: Ringbuffer_Entry, meta: Stack_Entry):
self.type = get_int(meta.body, self.metaentry_in_metablob + 5, 1)

assert self.type in [MetaEntryType.Printf.value, MetaEntryType.Dump.value,
MetaEntryType.SpanBegin.value, MetaEntryType.SpanEnd.value], \
MetaEntryType.SpanBegin.value, MetaEntryType.SpanEnd.value,
MetaEntryType.Fmt.value], \
f"invalid meta data, unsupported type {self.type}"

self.meta_size = get_int(meta.body, self.metaentry_in_metablob + 1, 4)
Expand Down Expand Up @@ -501,6 +503,40 @@ def __init__(self, entry: Ringbuffer_Entry, meta: Stack_Entry):
self.args = [span_id]
return

# Type 5 (Fmt): C++20 std::format-style {} placeholders.
# Supported subset: positional {} in order, and format specs that
# Python's str.format also understands ({:x}, {:08.3f}, {:>10}, etc.).
# Specs are passed through unchanged to str.format.
# Arguments are encoded identically to printf (type 1) entries; char*
# args are always length-prefixed strings (never pointer form).
# Falls back to "<fmt-error: <format> <args>>" when str.format raises
# for specs outside the common subset, so no crash on unknown specs.
if self.type == MetaEntryType.Fmt.value:
self.args = []
# Synthetic format dict for get_arg: char* is always a string in
# fmt mode (no %p variant), so pass conversion_specifier='s'.
_fmt_format_dict = {"conversion_specifier": "s"}
try:
offset = 0
for i in range(self.argument_count):
(offset, value) = StaticTraceentry.get_arg(
entry.args_raw, offset, self.arg_types[i], _fmt_format_dict)
self.args.append(value)
except Exception as e:
error_str = (f'Extracting fmt argument failed "{self.format}" '
f'from {self.file}:{self.line} with '
f'{e.with_traceback(e.__traceback__)} in {traceback.format_exc()}')
logging.error(error_str)
self.formatted = error_str
return
logging.debug(f"fmt tp args = {self.args}")
try:
self.formatted = self.format.format(*self.args) if self.argument_count else self.format
except Exception as e:
self.formatted = f"fmt-error: {self.format} {self.args}"
logging.error(f'fmt format failed "{self.format}" from {self.file}:{self.line} with {e}')
return

formats = [m.groupdict() for m in format_regex.finditer(self.format)]

self.args = []
Expand Down
2 changes: 2 additions & 0 deletions docs/file_specification.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,8 @@ The meta_entry_type differentiates the tracepoint styles:
* 3 = span begin (the string is the span name; two uint64 arguments: span id,
parent span id or 0)
* 4 = span end (empty string; one uint64 argument: span id)
* 5 = fmt tracepoint (format string with C++20 std::format {} placeholders;
argument encoding identical to printf tracepoints)

Decoders skip entries with unknown types, so the list can grow without
breaking old decoders.
Expand Down
11 changes: 11 additions & 0 deletions docs/technical_documentation.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,17 @@ rendering and correlation differ by meta type. Span ids come from
32 bit counter, unique without cross-process coordination, never 0
(0 means "no parent").

=== fmt-style tracepoints
`CLLTK_TRACEPOINT_FMT` (meta entry type 5) stores the {} format string
verbatim and encodes arguments exactly like printf tracepoints - the library
types arguments independently of the format syntax. Compile-time validation
happens through a never-executed call to a helper taking
`std::format_string<Args...>`, whose consteval constructor checks the literal
against the argument types. The runtime call goes through an entry point
without the printf format attribute, since {} strings must not be checked as
printf formats. Decoders render type 5 with their native {} formatters
(python `str.format`, C++ `std::vformat`).

=== caching process and thread id
Each call to `syscall(SYS_getpid)` and `syscall(SYS_gettid)` requires a context switch to kernel and therefore burdens a heavy overhead. To remove this context switch, pid (process id) and tid (thread id) should be cached. +
It's possible to detect if a code is executed inside a new thread with storage specifier `__thread` for tid and detect forking with `pthread_atfork`.
Expand Down
1 change: 1 addition & 0 deletions examples/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ add_subdirectory(./with_libraries)
add_subdirectory(./spans_c)
add_subdirectory(./simple_cpp)
add_subdirectory(./spans_cpp)
add_subdirectory(./fmt_cpp)
add_subdirectory(./complex_cpp)
add_subdirectory(./process_threads)
add_subdirectory(./snapshot_cpp)
Expand Down
16 changes: 16 additions & 0 deletions examples/fmt_cpp/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Copyright (c) 2026, International Business Machines
# SPDX-License-Identifier: BSD-2-Clause-Patent

add_executable(example-fmt_cpp
./fmt_cpp.cpp
)

target_link_libraries(example-fmt_cpp
PRIVATE
clltk_tracing
clltk_example_flags
)

add_dependencies(example-fmt_cpp
clean_up_traces
)
27 changes: 27 additions & 0 deletions examples/fmt_cpp/fmt_cpp.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
// Copyright (c) 2026, International Business Machines
// SPDX-License-Identifier: BSD-2-Clause-Patent

// fmt-style tracepoints (C++20): format strings use {} placeholders and are
// validated against the argument types at compile time via std::format_string.
// Same speed and mechanics as CLLTK_TRACEPOINT - only the syntax differs.

#include "CommonLowLevelTracingKit/tracing/tracing.h"

#include <string>

CLLTK_TRACEBUFFER(FMT_CPP, 4096)

int main()
{
CLLTK_TRACEPOINT_FMT(FMT_CPP, "loaded {} in {}ms", "module-a", 42);
CLLTK_TRACEPOINT_FMT(FMT_CPP, "hex {:x} padded {:>8} float {:.2f}", 255u, 7, 3.5);

// a char* argument is always recorded as a string; record the pointer
// value by casting to void*
const char *name = "pointer-vs-string";
CLLTK_TRACEPOINT_FMT(FMT_CPP, "string {} pointer {}", name, (const void *)name);

// mixing with printf-style tracepoints in the same buffer is fine
CLLTK_TRACEPOINT(FMT_CPP, "printf style %s", "still works");
return 0;
}
6 changes: 6 additions & 0 deletions tests/decoder.tests/Meta.tests.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,9 @@ INSTANTIATE_TEST_SUITE_P(TypeMappings, MetaEntryTypeParamTest,
::testing::Values(std::make_tuple(0, MetaEntryType::Unknown),
std::make_tuple(1, MetaEntryType::Printf),
std::make_tuple(2, MetaEntryType::Dump),
std::make_tuple(3, MetaEntryType::SpanBegin),
std::make_tuple(4, MetaEntryType::SpanEnd),
std::make_tuple(5, MetaEntryType::Fmt),
std::make_tuple(99, MetaEntryType::Unknown),
std::make_tuple(255, MetaEntryType::Unknown)));

Expand All @@ -229,6 +232,9 @@ TEST_F(MetaEntryInfoTest, typeToString)
{
EXPECT_EQ(MetaEntryInfo::typeToString(MetaEntryType::Printf), "printf");
EXPECT_EQ(MetaEntryInfo::typeToString(MetaEntryType::Dump), "dump");
EXPECT_EQ(MetaEntryInfo::typeToString(MetaEntryType::SpanBegin), "span_begin");
EXPECT_EQ(MetaEntryInfo::typeToString(MetaEntryType::SpanEnd), "span_end");
EXPECT_EQ(MetaEntryInfo::typeToString(MetaEntryType::Fmt), "fmt");
EXPECT_EQ(MetaEntryInfo::typeToString(MetaEntryType::Unknown), "unknown");
}

Expand Down
2 changes: 2 additions & 0 deletions tests/golden/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ regenerated from source.
| `golden-1.3.0-be-s390x.o` / `.so` | ELF objects compiled with library 1.3.0 headers on s390x |
| `golden-1.5.0-le-aarch64.clltk_trace` | library 1.5.0 (first with span events), aarch64 |
| `golden-1.5.0-be-s390x.clltk_trace` | library 1.5.0 (first with span events), s390x (big endian) |
| `golden-1.6.0-fmt-le-aarch64.clltk_trace` | library 1.6.0 (first with fmt tracepoints), aarch64 |
| `golden-1.6.0-fmt-be-s390x.clltk_trace` | library 1.6.0 (first with fmt tracepoints), s390x (big endian) |

`tests/test_golden.py` decodes every fixture and asserts the formatted
tracepoint messages. When the trace file format changes, add a new fixture
Expand Down
Loading
Loading