Part 2: Monitoring Package File Integrity

There are some errors here, but I leave it for people searching for why their timers may not be working. See the working version in the next post.

Recently I worked on a script to check changes in package file integrity. I want to do automate the check and email myeslf the results. I considered using cron, but it turns out that systemd has a timer service. It’s Good time to learn some more about systemd.

Ah, systemd. I’ve read about it, and the arguments in support and against systemd. From what I recall, it’s convenient way to manage things, but the fear is that it manages things too well, and linux risks becoming sysdnix as more tasks fall under systemd’s domain. Vague eh. I had little linux experience pre-systemd. And that’s why I think systemd will continue to permeate the distros and system space; it’s what new users will get to know.

Did I imply thatI know anything about systemd? Better try to back that up now.

systemd units are configuration files that store information about system utilities (e.g. devices, timers, sockets). Unit files have sections [unit], [service], and [install]. If you use Arch, you’ve likely setup service files for wifi connections, maybe through a command line front-end (e.g. wifi-menu). For systemd running in --sytstem mode, unit files are stored in

  • /usr/lib/systemd/system/ for package units
  • /run/systemd/system/*
  • /etc/systemd/system/ for sysadmin units

For example

ls /usr/lib/systemd/system/ | head -n 5
alsa-restore.service
alsa-state.service
alsa-store.service
autovt@.service
avahi-daemon.service

For those who are interested, there’s a bit about the difference in --system and --user modes on the Archwiki.

Check out man systemd.timer Timer units require a .timer and automatically invoke a matching .service file. I’m putting them in /etc/systemd/system/, since this is the location for sysadmin units.

Service file (see manpage for args and format).

/etc/systemd/system/pacqkk.service

[Unit]
Description=Pacman -Qkk change observer
OnFailure=status-email-jotham@%n.service

[Service]
Type=simple
ExecStart=/home/joth/bin/pacqkk_check.sh

Timer files (and manpage)

/etc/systemd/system/pacqkk.timer

[Unit]
Description=Pacman -Qkk change observer

[Timer]
Persistent=True
OnCalendar=*-*-* 08:00:00
Unit=pacqkk.service

[Install]
WantedBy=timers.target

See man systemd.time for how calendar times are specified.

I saw different specifications for WantedBy, namely, ‘multi-user.target’ and I’m not sure why one should be used over another. Drop a comment if you know…

I created another service to email me if a job fails

/etc/systemd/system/status-email-jotham@.service

[Unit]
Description=status email for %I to jotham

[Service]
Type=oneshot
ExecStart=/usr/bin/systemd-email jothamapaloo@gmail.com %i
User=nobody
Group=systemd-journal

That refers to a bash script to do the actual email

#!/bin/bash
/usr/bin/sendmail -t <<ERRMAIL
To: "$1"
From: systemd <"root@$HOSTNAME">
Subject: "$2"
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8

$(systemctl status --full "$2")
ERRMAIL

Which is available at the archwiki. Note also the list of specifiers, such as the %i, for unit files near the bottom of this page.

Finally, start the timer and enable it so it can run at boot. The email service does not need to be started and enabled.

systemctl start pacqkk.timer
systemctl enable pacqkk.timer
# test that you can run the email service
systemctl start status-email-jotham@pacqkk.service
systemctl list-timers | grep pacqkk

… The next day …

Ack, the service failed, but the script runs fine from the command line. And, to my dismay, I didn’t get the failure notification email. Wow, double fail.

What of the service?

journalctl -u pacqkk

And houw about the email?

systemctl status status-email-jotham@pacqkk.service
● status-email-jotham@pacqkk.service - status email for pacqkk to jotham
   Loaded: loaded (/etc/systemd/system/status-email-jotham@.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2015-04-08 20:42:59 EDT; 1min 11s ago
  Process: 11054 ExecStart=/home/joth/bin/systemd-email.sh jothamapaloo@gmail.com %i (code=exited, status=203/EXEC)
 Main PID: 11054 (code=exited, status=203/EXEC)

Apr 08 20:42:59 archLenFlex systemd[11054]: Failed at step EXEC spawning /home/joth/bin/systemd-email.sh: Permission denied
Apr 08 20:42:59 archLenFlex systemd[1]: status-email-jotham@pacqkk.service: main process exited, code=exited, status=203/EXEC
Apr 08 20:42:59 archLenFlex systemd[1]: Failed to start status email for pacqkk to jotham.
Apr 08 20:42:59 archLenFlex systemd[1]: Unit status-email-jotham@pacqkk.service entered failed state.
Apr 08 20:42:59 archLenFlex systemd[1]: status-email-jotham@pacqkk.service failed.

These likely relate to --user-mode, as it looks like the environment vars $HOME and $EMAIL were not expanded and the user which systemd is running as doesn’t have perms for the executable.

Does systemd run as user systemd or root? Drop a comment.

Update: sorted it out in the next post!

Go Top
comments powered by Disqus