Skip to content

FAQ: How to submit kernel panic message

ABC edited this page Nov 3, 2013 · 9 revisions

Please attach full kernel message log with actual ipt_NETFLOW.ko binary.

  • Kernel message (/var/log/kern on some systems, or console output, or dmesg output) most preferable is to be since module load and including module version string. Binary is important and required to decode stack trace, code dump, and meaning of registers.

  • To get kernel messages on serial console add console=ttyS0,115200 to kernel boot command line.

  • To get kernel messages in separate log file, ex. /var/log/kern, edit your /etc/syslog.conf to add line like: kern.* /var/log/kern, and restart syslogd.

  • ipt_NETFLOW.ko binary is usually resides in /lib/modules/$(uname -r)/extra dir, or where you compiled it.

  • Example kernel message:

Nov  3 15:28:21 x kernel: [82720.255973] ipt_NETFLOW version v1.8-75-g626dbdd, srcversion E97F99E2F369A4484CA0A30
Nov  3 15:28:21 x kernel: [82720.255979] ipt_NETFLOW: hashsize 2097152
Nov  3 15:28:21 x kernel: [82720.267075] netflow: registered: /proc/net/stat/ipt_netflow
Nov  3 15:28:21 x kernel: [82720.267094] netflow: registered: sysctl net.netflow
Nov  3 15:28:21 x kernel: [82720.267298] ipt_NETFLOW: added destination 127.0.0.1:2055
Nov  3 15:28:21 x kernel: [82720.267332] ipt_NETFLOW protocol version 5 (NetFlow) enabled.
Nov  3 15:28:21 x kernel: [82720.267341] ipt_NETFLOW is loaded.
Nov  3 15:29:56 x kernel: [82815.104006] INFO: rcu_sched self-detected stall on CPU { 7}  (t=15000 jiffies g=173289 c=173288 q=3308)
Nov  3 15:29:56 x kernel: [82815.104008] sending NMI to all CPUs:
Nov  3 15:29:56 x kernel: [82815.104008] NMI backtrace for cpu 7
Nov  3 15:29:56 x kernel: [82815.104008] CPU: 7 PID: 0 Comm: swapper/7 Tainted: GF          O 3.10.17-matrix-imq-debug #1
Nov  3 15:29:56 x kernel: [82815.104008] Hardware name: Dell Inc. PowerEdge 1950/0TT740, BIOS 2.6.1 04/20/2009
Nov  3 15:29:56 x kernel: [82815.104008] task: ffff880129b3c000 ti: ffff880129b36000 task.ti: ffff880129b36000
Nov  3 15:29:56 x kernel: [82815.104008] RIP: 0010:[<ffffffff81027642>]  [<ffffffff81027642>] arch_trigger_all_cpu_backtrace+0x82/0x90
Nov  3 15:29:56 x kernel: [82815.104008] RSP: 0018:ffff88012fdc34e0  EFLAGS: 00000046
Nov  3 15:29:56 x kernel: [82815.104008] RAX: 0000000000000000 RBX: 0000000000002710 RCX: 0000000000000004
Nov  3 15:29:56 x kernel: [82815.104008] RDX: ffffffff81c20f20 RSI: 0000000000000100 RDI: ffffffff81d01ca0
Nov  3 15:29:56 x kernel: [82815.104008] RBP: ffff88012fdc34e8 R08: 0000000000000001 R09: 0000000000000001
Nov  3 15:29:56 x kernel: [82815.104008] R10: 0000000000000001 R11: 00000000000028af R12: ffffffff81c50500
Nov  3 15:29:56 x kernel: [82815.104008] R13: ffffffff81c50500 R14: ffff88012fdcdca0 R15: 0000000000000cec
Nov  3 15:29:56 x kernel: [82815.104008] FS:  0000000000000000(0000) GS:ffff88012fdc0000(0000) knlGS:0000000000000000
Nov  3 15:29:56 x kernel: [82815.104008] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov  3 15:29:56 x kernel: [82815.104008] CR2: 000000000069e000 CR3: 0000000127375000 CR4: 00000000000007e0
Nov  3 15:29:56 x kernel: [82815.104008] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov  3 15:29:56 x kernel: [82815.104008] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov  3 15:29:56 x kernel: [82815.104008] Stack:
Nov  3 15:29:56 x kernel: [82815.104008]  ffffffff81d021e0 ffff88012fdc3548 ffffffff810e705d ffffffff810ea90c
Nov  3 15:29:56 x kernel: [82815.104008]  ffff88012fdc3538 ffffffff8107e7ca ffff880129b36000 0000000000000007
Nov  3 15:29:56 x kernel: [82815.104008]  ffff880129b3c000 0000000000000000 0000000000000007 ffff88012fdcd580
Nov  3 15:29:56 x kernel: [82815.104008] Call Trace:
Nov  3 15:29:56 x kernel: [82815.104008]  <IRQ>
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff810e705d>] rcu_check_callbacks+0x30d/0x5e0
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff810ea90c>] ? acct_account_cputime+0x1c/0x20
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff8107e7ca>] ? account_system_time+0xda/0x190
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff810559b8>] update_process_times+0x48/0x80
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff810a1efe>] tick_sched_handle.isra.10+0x2e/0x70
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff810a208c>] tick_sched_timer+0x4c/0x80
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff8106ec12>] __run_hrtimer+0x72/0x1b0
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff810a2040>] ? tick_sched_do_timer+0x40/0x40
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff8106f517>] hrtimer_interrupt+0xf7/0x230
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff816e21a9>] smp_apic_timer_interrupt+0x69/0x99
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff816e13ca>] apic_timer_interrupt+0x6a/0x70
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa047cfb1>] ? netflow_target+0x5c1/0x10e8 [ipt_NETFLOW]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa047d627>] ? netflow_target+0xc37/0x10e8 [ipt_NETFLOW]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa019b19d>] ? ipt_do_table+0x30d/0x5e5 [ip_tables]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff816e20c3>] ? do_IRQ+0x63/0xe0
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa019b156>] ipt_do_table+0x2c6/0x5e5 [ip_tables]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa01920d3>] iptable_filter_hook+0x33/0x64 [iptable_filter]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff815f1ac6>] nf_iterate+0x86/0xb0
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa0466770>] ? br_nf_dev_queue_xmit+0x90/0x90 [bridge]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff815f1b8c>] nf_hook_slow+0x9c/0x180
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa0466770>] ? br_nf_dev_queue_xmit+0x90/0x90 [bridge]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa0466f21>] br_nf_forward_ip+0x301/0x3c0 [bridge]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff815f1ac6>] nf_iterate+0x86/0xb0
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa045f750>] ? br_flood+0x130/0x130 [bridge]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff815f1b8c>] nf_hook_slow+0x9c/0x180
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa045f750>] ? br_flood+0x130/0x130 [bridge]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa045f7b0>] ? br_forward_finish+0x60/0x60 [bridge]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa045f860>] __br_forward+0xb0/0xf0 [bridge]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff815b6a33>] ? skb_clone+0x63/0xb0
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa045f7b0>] ? br_forward_finish+0x60/0x60 [bridge]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa045f30e>] deliver_clone+0x3e/0x60 [bridge]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa045f6fa>] br_flood+0xda/0x130 [bridge]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa045fb15>] br_flood_forward+0x15/0x20 [bridge]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa04607c4>] br_handle_frame_finish+0x1a4/0x330 [bridge]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa04677ad>] br_nf_pre_routing_finish+0x1bd/0x510 [bridge]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa0467df7>] br_nf_pre_routing+0x2f7/0x660 [bridge]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff815f1ac6>] nf_iterate+0x86/0xb0
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa0460620>] ? br_handle_local_finish+0x50/0x50 [bridge]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff815f1b8c>] nf_hook_slow+0x9c/0x180
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa0460620>] ? br_handle_local_finish+0x50/0x50 [bridge]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffffa0460ae0>] br_handle_frame+0x190/0x270 [bridge]
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff815c2c32>] __netif_receive_skb_core+0x3b2/0x850
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff815c2949>] ? __netif_receive_skb_core+0xc9/0x850
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff815c3219>] ? process_backlog+0xd9/0x170
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff815c30f1>] __netif_receive_skb+0x21/0x70
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff815c31e4>] process_backlog+0xa4/0x170
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff815c3ac9>] net_rx_action+0x139/0x220
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff8104c077>] __do_softirq+0xe7/0x230
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff816e1a3c>] call_softirq+0x1c/0x30
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff810049d5>] do_softirq+0x55/0x90
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff8104c325>] irq_exit+0xa5/0xb0
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff816e20c3>] do_IRQ+0x63/0xe0
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff816d822a>] common_interrupt+0x6a/0x6a
Nov  3 15:29:56 x kernel: [82815.104008]  <EOI>
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff815686e3>] ? cpuidle_enter_state+0x63/0xe0
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff81568819>] cpuidle_idle_call+0xb9/0x200
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff8100b24e>] arch_cpu_idle+0xe/0x30
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff81098310>] cpu_startup_entry+0xd0/0x250
Nov  3 15:29:56 x kernel: [82815.104008]  [<ffffffff816c1367>] start_secondary+0x1df/0x1e4
Nov  3 15:29:56 x kernel: [82815.104008] Code: 00 eb 15 66 0f 1f 44 00 00 bf 58 89 41 00 e8 b6 51 32 00 83 eb 01 74 15 be 00 01 00 00 48 c7 c7 a0 1c d0 81 e8 70 7c 32 00 85 c0 <74> dc f0 80 25 34 d8 e4 00 fe 5b 5d c3 90 66 66 66 66 90 55 48

Please include full content if it's longer in your case.

Clone this wiki locally