Sysprof

The default profiling options for LuaJIT are not fine enough to get an understanding of performance. For example, perf only able to show host stack, so all the Lua calls are seen as single pcall. Oppositely, the jit.p module provided with LuaJIT is not able to give any information about the host stack. Here comes the sysprof: it is able to capture both guest and host stacks simultaneously, along with virtual machine states.

The C API

Configuration:

• int luaM_sysprof_set_writer(sp_writer writer). Sets writer function for sysprof.

• int luaM_sysprof_set_on_stop(sp_on_stop on_stop). Sets on stop callback for sysprof to clear resources.

• int luaM_sysprof_set_backtracer(sp_backtracer backtracer). Sets backtracking function. If backtracer arg is NULL, the default backtracer is set.
  There is no need to call the configuration functions multiple times, if you are starting and stopping profiler several times in a single program. Also, it is not necessary to configure sysprof for the default mode, however, one MUST configure it for the other modes.

• int luaM_sysprof_start(lua_State *L, const struct luam_Sysprof_Options *opt)

• int luaM_sysprof_stop(lua_State *L)

• int luaM_sysprof_report(struct luam_Sysprof_Counters *counters). Writes profiling counters for each vmstate.

All of the functions return 0 on success and an error code on failure.

The configuration types are:

/* Profiler configurations. */
/*
** Writer function for profile events. Must be async-safe, see also
** `man 7 signal-safety`.
** Should return amount of written bytes on success or zero in case of error.
** Setting *data to NULL means end of profiling.
** For details see <lj_wbuf.h>.
*/
typedef size_t (*sp_writer)(const void **data, size_t len, void *ctx);
/*
** Callback on profiler stopping. Required for correctly cleaning
** at VM finalization when profiler is still running.
** Returns zero on success.
*/
typedef int (*sp_on_stop)(void *ctx, uint8_t *buf);
/*
** Backtracing function for the host stack. Should call `frame_writer` on
** each frame in the stack in the order from the stack top to the stack
** bottom. The `frame_writer` function is implemented inside the sysprof
** and will be passed to the `backtracer` function. If `frame_writer` returns
** NULL, backtracing should be stopped. If `frame_writer` returns not NULL,
** the backtracing should be continued if there are frames left.
*/
typedef void (*sp_backtracer)(void *(*frame_writer)(int frame_no, void *addr));

Profiler options are the following:

struct luam_Sysprof_Options {
  /* Profiling mode. */
  uint8_t mode;
  /* Sampling interval in msec. */
  uint64_t interval;
  /* Custom buffer to write data. */
  uint8_t *buf;
  /* The buffer's size. */
  size_t len;
  /* Context for the profile writer and final callback. */
  void *ctx;
};

Profiling modes:

/*
** DEFAULT mode collects only data for luam_sysprof_counters, which is stored
** in memory and can be collected with luaM_sysprof_report after profiler
** stops.
*/
#define LUAM_SYSPROF_DEFAULT 0
/*
** LEAF mode = DEFAULT + streams samples with only top frames of host and
** guests stacks in format described in <lj_sysprof.h>
*/
#define LUAM_SYSPROF_LEAF 1
/*
** CALLGRAPH mode = DEFAULT + streams samples with full callchains of host
** and guest stacks in format described in <lj_sysprof.h>
*/
#define LUAM_SYSPROF_CALLGRAPH 2

Counters structure for the luaM_Sysprof_Report:

struct luam_Sysprof_Counters {
  uint64_t vmst_interp;
  uint64_t vmst_lfunc;
  uint64_t vmst_ffunc;
  uint64_t vmst_cfunc;
  uint64_t vmst_gc;
  uint64_t vmst_exit;
  uint64_t vmst_record;
  uint64_t vmst_opt;
  uint64_t vmst_asm;
  uint64_t vmst_trace;
  /*
  ** XXX: Order of vmst counters is important: it should be the same as the
  ** order of the vmstates.
  */
  uint64_t samples;
};

Caveats:

• Providing writers, backtracers, etc; in the Default mode is pointless, since
  it just collect counters.
• There is NO default configuration for sysprof, so the luaM_Sysprof_Configure
  must be called before the first run of the sysprof.
• Mind the async-safety

The Lua API

• misc.sysprof.start(opts)
• misc.sysprof.stop()
• misc.sysprof.report()

First two functions return boolean res and err, which is nil on success and contains an error message on failure

misc.sysprof.report returns a lua table containing the following counters:

{
  "samples" = int,
  "INTERP" = int,
  "LFUNC" = int,
  "FFUNC" = int,
  "CFUNC" = int,
  "GC" = int,
  "EXIT" = int,
  "RECORD" = int,
  "OPT" = int,
  "ASM" = int,
  "TRACE" = int
}

Parameter opts for the misc.sysprof.start can contain the following parameters:

{
  mode = 'D'/'L'/'C', -- 'D' = DEFAULT, 'L' = LEAF, 'C' = CALLGRAPH
  interval = 10, -- sampling interval in msec.
  path = '/path/to/file' -- location to store profile data.
}

Mode MUST be provided always, interval and path are optional. The default
interval is 10 msec, path -- sysprof.bin

Parsers

The default profile data parser is provided. Its output is flamegraph.pl-suitable,
so one can do this:

$ luajit-parse-sysprof /path/to/profile/data > tmp
$ flamegraph.pl tmp > graph.svg

Пример использования

1. Подключаемся к экземпляру

2. Запускаем трассировку:

   misc.sysprof.start({
       mode = 'C', -- 'D' = DEFAULT, 'L' = LEAF, 'C' = CALLGRAPH
       interval = 10, -- sampling interval in msec.
       path = '/path/to/save/file.bin' -- location to store profile data.
   })

   Вывод:

   ---
   - true
   ...
   

3. Останавливаем трассировку:

   misc.sysprof.stop()

   Вывод:

   ---
   - true
   ...
   

4. Выходим из экземпляра и выполняем:

   tarantool -e 'require("sysprof")(arg[1])' - /path/to/save/file.bin > /path/to/save/out.bin
   ./flamegraph.pl /path/to/save/out.bin > /path/to/save/out.svg
   

   flamegraph.pl

5. Смотрим SVG

А что насчёт подробностей в ошибках?
Сейчас вот получил ошибку profiler_misuse и сижу гадаю на кофейной гуще, что же пошло не так. Посмотрел в исходниках, насчитал около 10 мест, откуда возвращается PROFILE_ERRUSE, который потом превращается в эту ошибку.

А что насчёт подробностей в ошибках? Сейчас вот получил ошибку profiler_misuse и сижу гадаю на кофейной гуще, что же пошло не так. Посмотрел в исходниках, насчитал около 10 мест, откуда возвращается PROFILE_ERRUSE, который потом превращается в эту ошибку.

Эта ошибка говорит о некорректно введенных параметрах, либо о том, что у вас сборка без sysprof.
Если вызов вида misc.sysprof.start({mode = 'C', interval = 10, path = 'profile.bin'}) возвращает ошибку, то можно уверенно сказать, что в вашей сборке нет профилировщика.

что в вашей сборке нет профилировщика.

Ну то есть в стандартном докер образе тарантула отсутствует sysprof? Чтож, звучит печально. Особенно на фоне того, что gperftools там все таки присутствует.

@mkokryashkin, consider adding misc.sysprof.available() similar to the one provided in uJIT profiler. It's worth adding it for memprof as well.