Investigating Archlinux Systemd Coredumps

So, something unusual occurred recently. Anytime I opened certain github pages in firefox with waffle.io open at the same time, firefox crashed and systemd wrote large coredumps, eating up half of my processor cores.

Turns out, this gent had a similar issue.and here's the bug report. Apparently its a systemd 216 problem. Oddly, it doesn't occur on my desktop.

bash init --version

``` systemd 216 +PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN

```

Indeed this is the affected version.

I think this presents a good opportunity to dig a little deeper into the system.

From falcondidy on the bug report page,

"When a program is terminated either by SIGABRT or SIGSEGV, the kernel sends the coredump to userspace. The value in /proc/sys/kernel/corepattern determines what userspace does with it. In this particular case, systemd-coredump is spawned and the kernel pipes the coredump to systemd-coredump's stdin. Based on the settings in /etc/systemd/coredump.conf, systemd is writing it to /var/lib/systemd/coredump."

bash ls /var/lib/systemd/coredump

core.firefox.1000.somehash.xz ... core.plugin-containe.1000.somehash.xz

Indeed that's where these coredumps are being written.There's also some plugin coredumps that I didn't notice in the process list when the firefox dumps were being written. They happen to match each firefox coredump per the date and time of each file. So I'll look into these as a pair.

bash coredumpctl info

Gives info of all coredumps, the last two, with some variables omitted, contain

``` PID: 12709 (plugin-containe) Signal: 11 (SEGV) Command Line: /usr/lib/firefox/plugin-container /usr/lib/mozilla/plugins/libflashplayer.so -greomni /usr/lib/firefox/omni.ja -appomni /usr/lib/firefox/browser/omni.ja Executable: /usr/lib/firefox/plugin-container Control Group: /user.slice/user-1000.slice/session-c1.scope Unit: session-c1.scope Message: Process 12709 (plugin-containe) of user 1000 dumped core.

   PID: 2866 (firefox)
Signal: 6 (ABRT)

Executable: /usr/lib/firefox/firefox Unit: session-c1.scope Coredump: /var/lib/systemd/coredump/core.firefox.1000.d75ca0fef4df4ffa8f30ac7498fa5221.2866.1414950918000000.xz Message: Process 2866 (firefox) of user 1000 dumped core. ```

So, firefox is segfaulting and the flash plugin container is aborting.I think either could be the cause of the other.

That's as much as I can muster tonight. Further steps are,

  • to configure systemd to not write these coredump files
  • investigate the coredump further to try and get to the root cause

Coredump analysis sounds like an interesting topic and useful skill. gdb is a utility one can use to dig into the coredumps, and a strategy is available from redhat. And, you can actually invoke gdb straigt from coredumpctl as follows

bash coredumpctl gdb 2866

replacing 2866 with the PID of your dump. See man coredumpctl for more info.

Good times.

For now I'm just going to hope the next firefox/flash updates solve this issue. It is avoidable, and not a major nuisance when it occurs.

Go Top
comments powered by Disqus