Releases: YutaroHayakawa/ipftrace2
v0.10.1
4bd7dcf
YutaroHayakawa
Yutaro Hayakawa
This is a minor fix release.
Full Changelog: v0.10.0...v0.10.1
v0.10.0
Release Highlight
- Support "format specifier" in BPF-based extension module
- Fix a symbol resolution issue when Intel IBT is enabled
Full Change Log: v0.9.0...v0.10.0
Support "format specifier" in the BPF-based extension module
In the BPF-based extension module, we couldn't change the value format. For example, when we want to dump bit flags in human-visible format (such as hex decimal or name of the flags), we can't. This release introduced a new mechanism to specify the value format with a "format specifier" annotation. We can now format your output with a special annotation like this.
struct event {
__u32 gso_type __ipft_fmt_hex;
} __ipft_event_struct;The __ipft_fmt_hex formats the value of the gso_type field as hex decimal. This produces the output like this.
809146393994497 004 vmlinux dev_hard_start_xmit ( gso_type: 0x1 )
Cool, but we may want to display a more human-readable name. In that case, we can use __ipft_fmt_enum_flags like this.
enum {
tcpv4 = 1 << 0,
dodgy = 1 << 1,
tcp_ecn = 1 << 2,
tcp_fixedid = 1 << 3,
tcpv6 = 1 << 4,
fcoe = 1 << 5,
gre = 1 << 6,
gre_csum = 1 << 7,
ipxip4 = 1 << 8,
ipxip6 = 1 << 9,
udp_tunnel = 1 << 10,
udp_tunnel_csum = 1 << 11,
partial = 1 << 12,
tunnel_remcsum = 1 << 13,
sctp = 1 << 14,
esp = 1 << 15,
udp = 1 << 16,
udp_l4 = 1 << 17,
flaglist = 1 << 18,
} __ipft_ref(gso_types);
struct event {
__u32 gso_type __ipft_fmt_enum_flags(gso_types);
} __ipft_event_struct;Now we have a new enum annotated with __ipft_ref(gso_types). This annotation makes a "reference" named gso_types and __ipft_fmt_enum_flags refers to that. This produces an output like this.
809146393994497 004 vmlinux dev_hard_start_xmit ( gso_type: tcpv4 )
809146393997960 004 ipip ipip_tunnel_xmit ( gso_type: tcpv4 )
809146394001327 004 vmlinux iptunnel_handle_offloads ( gso_type: tcpv4 )
809146394004583 004 ip_tunnel ip_tunnel_xmit ( gso_type: tcpv4|ipxip4 )
809146394008133 004 ip_tunnel tnl_update_pmtu ( gso_type: tcpv4|ipxip4 )
Now we can get more human-readable output. Nice! If the enum is not a bit flag, we also have __ipft_fmt_enum(ref) format specifier.
v0.9.0
Release Highlight
- Support an experimental BPF-based extension module 1bfbbcf
Full Changelog: v0.8.0...v0.9.0
An experimental BPF-based extension module
ipftrace2 is used to provide the extension mechanism that can customize your output with a custom BPF program and Lua script. In this extension mechanism, we needed to write both of BPF custom data collector and Lua event decoder, which was sometimes annoying since we needed to go BPF and Lua back and forth.
In the new BPF-based extension mechanism, we only need to write the BPF program, but we can get a similar output. What you need to do in addition is putting some annotation to your struct definition.
#include <linux/types.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>
#define __ipft_sec_skip __attribute__((section("__ipft_skip")))
#define __ipft_event_struct __ipft_event_struct __ipft_sec_skip
struct event {
unsigned int len;
} __ipft_event_struct; // a special annotation
struct sk_buff {
unsigned int len;
};
__hidden int
module(void *ctx, struct sk_buff *skb, __u8 data[64])
{
struct event *ev = (struct event *)data;
ev->len = BPF_CORE_READ(skb, len);
return 0;
}With this program, you can get an output like this. I believe it's much easier than writing both BPF and Lua.
464048546191357 005 nf_conntrack ipv4_conntrack_in ( len: 2688 )
464048546194045 005 nf_conntrack nf_conntrack_in ( len: 2688 )
464048546194858 005 nf_conntrack resolve_normal_ct ( len: 2688 )
464048546195703 005 nf_conntrack nf_conntrack_handle_packet ( len: 2688 )
464048546196407 005 nf_conntrack nf_conntrack_tcp_packet ( len: 2688 )
464048546197050 005 vmlinux nf_checksum ( len: 2688 )
464048546197694 005 vmlinux nf_ip_checksum ( len: 2688 )
464048546198546 005 nf_nat nf_nat_ipv4_pre_routing ( len: 2688 )
464048546199226 005 nf_nat nf_nat_inet_fn ( len: 2688 )
464048546207009 005 vmlinux tcp_v4_early_demux ( len: 2688 )
464048546208553 005 vmlinux ip_local_deliver ( len: 2688 )
464048546209310 005 vmlinux nf_hook_slow ( len: 2688 )
For more information, please see the guide. You an also find more advanced use cases in GSO example. Enjoy!
v0.8.0
Release Highlight
- Display module name to trace output 36986c4
- Module-based regex filtering 0e220c2
- libbpf v1.0.0 support 5da963a
Full Changelog: v0.7.0...v0.8.0
Display module name to trace output
Now both of aggregated (default) output and JSON output includes kernel module name by default.
4383592522664911 006 nf_conntrack ipv4_conntrack_in
4383592522667659 006 nf_conntrack nf_conntrack_in
4383592522668702 006 nf_conntrack resolve_normal_ct
4383592522670093 006 nf_conntrack nf_conntrack_handle_packet
4383592522671173 006 nf_conntrack nf_conntrack_tcp_packet
4383592522672257 006 vmlinux nf_checksum
4383592522673321 006 vmlinux nf_ip_checksum
4383592522674599 006 nf_nat nf_nat_ipv4_pre_routing
4383592522675668 006 nf_nat nf_nat_inet_fn
4383592522677146 006 vmlinux tcp_v4_early_demux
4383592522678949 006 vmlinux ip_local_deliver
4383592522679910 006 vmlinux nf_hook_slow
4383592522680986 006 nf_tables nft_do_chain_ipv4
4383592522682869 006 nf_nat nf_nat_ipv4_local_in
4383592522683810 006 nf_nat nf_nat_inet_fn
4383592522684916 006 nf_conntrack ipv4_confirm
4383592522686317 006 nf_conntrack nf_confirm
4383592522687451 006 vmlinux ip_local_deliver_finish
4383592522688806 006 vmlinux ip_protocol_deliver_rcu
{"packet_id":"0xffff998bdbeec800","timestamp":4383908208987951,"processor_id":0,"module":"vmlinux","function":"nf_ip_checksum","is_return":false}
{"packet_id":"0xffff998bdbeec800","timestamp":4383908208989663,"processor_id":0,"module":"nf_nat","function":"nf_nat_ipv4_pre_routing","is_return":false}
{"packet_id":"0xffff998bdbeec800","timestamp":4383908208990837,"processor_id":0,"module":"nf_nat","function":"nf_nat_inet_fn","is_return":false}
{"packet_id":"0xffff998bdbeec800","timestamp":4383908208992985,"processor_id":0,"module":"vmlinux","function":"tcp_v4_early_demux","is_return":false}
{"packet_id":"0xffff998bdbeec800","timestamp":4383908208995575,"processor_id":0,"module":"vmlinux","function":"ip_local_deliver","is_return":false}
{"packet_id":"0xffff998bdbeec800","timestamp":4383908208996525,"processor_id":0,"module":"vmlinux","function":"nf_hook_slow","is_return":false}
{"packet_id":"0xffff998bdbeec800","timestamp":4383908208997448,"processor_id":0,"module":"nf_tables","function":"nft_do_chain_ipv4","is_return":false}
{"packet_id":"0xffff998bdbeec800","timestamp":4383908208999436,"processor_id":0,"module":"nf_nat","function":"nf_nat_ipv4_local_in","is_return":false}
{"packet_id":"0xffff998bdbeec800","timestamp":4383908209000178,"processor_id":0,"module":"nf_nat","function":"nf_nat_inet_fn","is_return":false}
{"packet_id":"0xffff998bdbeec800","timestamp":4383908209001170,"processor_id":0,"module":"nf_conntrack","function":"ipv4_confirm","is_return":false}
{"packet_id":"0xffff998bdbeec800","timestamp":4383908209002501,"processor_id":0,"module":"nf_conntrack","function":"nf_confirm","is_return":false}
Module-based regex filtering
As well as displaying the module name, we now allow filtering the symbols to trace by module name with --module-regex. For example, if you want to trace Netfilter-related functions you can do like below. As you can see, Netfilter functions are not always prefixed with nf_ (e.g. resolve_normal_ct), so it was impossible with symbol-based regex filtering.
$ sudo ipft -l --module-regex "nf_"
NAME MOD ADDR SKB_POSITION
nft_data_dump nf_tables 0xffffffffc1e34380 0
nft_verdict_dump nf_tables 0xffffffffc1e34290 0
nft_dump_register nf_tables 0xffffffffc1e22370 0
<skip...>
__nf_nat_decode_session nf_nat 0xffffffffc1ebf250 0
nf_nat_icmpv6_reply_translation nf_nat 0xffffffffc1ec13e0 0
nf_nat_icmp_reply_translation nf_nat 0xffffffffc1ec18a0 0
<skip...>
nf_tproxy_laddr4 nf_tproxy_ipv4 0xffffffffc242b320 0
nf_tproxy_handle_time_wait6 nf_tproxy_ipv6 0xffffffffc2431390 0
nf_tproxy_laddr6 nf_tproxy_ipv6 0xffffffffc2431540 0
nf_send_unreach nf_reject_ipv4 0xffffffffc1d5b6a0 0
nf_reject_fill_skb_dst nf_reject_ipv4 0xffffffffc1d5b090 0
<skip...>
resolve_normal_ct nf_conntrack 0xffffffffc1e769e0 1
ipv6_conntrack_local nf_conntrack 0xffffffffc1e7b1a0 1
ipv6_conntrack_in nf_conntrack 0xffffffffc1e7b1c0 1
libbpf v1.0.0 support
It may not be interesting for users, but we fully migrated to libbpf v1.0.0.
New Contributors
Thanks for your contribution!
Contributors
Assets 3
v0.7.0
Release Highlight
- Better support of
BPF_TRACE_KPROBE_MULTIeccfe71 - Display some symbol metadata with
-loption 32fcb3a - Some bug fixes
Better support of BPF_TRACE_KPROBE_MULTI
In the previous release, we declared the support of BPF_TRACE_KPROBE_MULTI. However, the way of using that was actually wrong. Fixing that produces great performance again and now the attach takes less than 1 second.
Display some symbol metadata with -l option
Display some useful metadata with symbol in -l option. This may break the existing programs that parse the output of -l option. The new output looks like below. It displays function address and skb position in the argument. The header is written into stderr and the rest are written into stdout.
NAME ADDR SKB_POSITION
audit_receive 0xffffffffb0f86c40 0
audit_receive_msg 0xffffffffb0f85940 0
kauditd_send_multicast_skb 0xffffffffb0f833f0 0
kauditd_retry_skb 0xffffffffb0f83a60 0
kauditd_hold_skb 0xffffffffb0f83ab0 0
kauditd_rehold_skb 0xffffffffb0f83490 0
kauditd_printk_skb 0xffffffffb0f833b0 0
audit_list_rules_send 0xffffffffb0f887e0 0
taskstats_user_cmd 0xffffffffb0fa6400 0
cgroupstats_user_cmd 0xffffffffb0fa5fa0 0
mk_reply 0xffffffffb0fa6310 0
send_reply 0xffffffffb0fa5d30 0
v0.6.1
Some minor bug fixes.
v0.6.0
Release Highlight
- Support TCP-based readiness probe 3f7b9f3
- Support speeding up attach time with
BPF_TRACE_KPROBE_MULTI87ed510
TCP-based readiness probe
Previously, it was difficult to automate the ipftrace2 tracing because it was missing the way to signal ready state (Trace Ready! text is for humans, not for machines). To resolve this problem, we introduce a TCP-based readiness probe feature. When the --enable-probe-server option is specified, ipftrace2 starts the TCP server (it closes the connection immediately after accepting) when it becomes ready for tracing. For example, we can do something like this.
# Launch function tracer
ipft -m 1 --enable-probe-server > /tmp/trace.txt &
# Wait for tracer to be ready
until nc -z 127.0.0.1 13720; do sleep 2s; done
# Make some traffic
ping -c 1 -m 1 127.0.0.1
# Stop tracer
pkill -e -TERM ipft
# Wait for ipft to finish
wait
# Dump trace
cat /tmp/trace.txtWhen we want to change the port number, we can use --probe-server-port.
BPF_TRACE_KPROBE_MULTI support
BPF_TRACE_KPROBE_MULTI is a new attach type for kprobe programs that allows us to attach multiple program at once and dramatically improves the time takes to attach. ipftrace2 is a perfect use case of this feature. Now when we use the function tracer, ipftrace2 automatically probes the kernel feature and uses BPF_TRACE_KPROBE_MULTI when possible. Otherwise, falls back to bare kprobe. We can also explicitly specify them with -b option.
The below video clips show the difference between kprobe and kprobe-multi attach and detach time in my local development environment. With kprobe-multi, we can see dramatic improvement for both attach and detach time.
v0.5.1
v0.5.0
Release Highlight
- Support experimental function graph tracing
Function Graph Tracing (Experimental)
Function graph tracing is a new tracing method similar to ftrace's function_graph tracer. It allows us to trace not only function calls, but also function returns. As a result, we can visualize the depth of the call like below. This feature is currently experimental. It can be removed or modified with breaking change. Please play around it and feel free to give us some feedback.
Note that it relies on a relatively new kernel feature (bpf_get_func_ip helper), it only works for Linux v5.15 or newer.
$ sudo ipft -m 0xdeadbeef -t function_graph
<skip...>
26956966578343 000 ip_output() {
26956966579469 000 nf_hook_slow() {
26956966580830 000 selinux_ipv4_postroute() {
26956966582109 000 selinux_ip_postroute() {
26956966583280 000 }
26956966584417 000 }
26956966585569 000 }
26956966586827 000 ip_finish_output() {
26956966594966 000 __cgroup_bpf_run_filter_skb() {
26956966596126 000 }
26956966597257 000 __ip_finish_output() {
26956966598533 000 ip_finish_output2() {
26956966599826 000 dev_queue_xmit() {
26956966601079 000 __dev_queue_xmit() {
26956966602573 000 netdev_core_pick_tx() {
26956966603819 000 }
26956966605195 000 validate_xmit_skb_list() {
26956966606320 000 validate_xmit_skb() {
26956966607635 000 netif_skb_features() {
26956966608921 000 passthru_features_check() {
26956966610149 000 }
26956966611460 000 skb_network_protocol() {
26956966612565 000 }
26956966613636 000 }
26956966614933 000 skb_csum_hwoffload_help() {
26956966616047 000 }
26956966617349 000 validate_xmit_xfrm() {
26956966618506 000 }
26956966619576 000 }
26956966620746 000 }
26956966621909 000 dev_hard_start_xmit() {
26956966623503 000 skb_clone_tx_timestamp() {
26956966624695 000 }
26956966626115 000 skb_to_sgvec() {
26956966627231 000 __skb_to_sgvec() {
26956966628426 000 }
26956966629630 000 }
26956966631129 000 }
26956966632560 000 }
26956966633720 000 }
26956966634835 000 }
26956966635963 000 }
26956966637099 000 }
26956966638213 000 }