40

I'm particularly interested in this for looking at the output of oneshot services that run on a timer. The --unit flag is close, but it concatenates all the runs of the service together. The most obvious way I can think of would be to filter on PID, but that makes me worry about PID reuse / services that fork, and getting the last PID is pretty inconvenient. Is there some other identifier that corresponds to a single run of a service, that I could use to filter the logs?

EDIT: I would happily accept an authoritative "no" if that's the real answer.

Jack O'Connor
  • 811

6 Answers6

31

Since systemd version 232, we have the concept of invocation ID. Each time a unit is ran, it has a unique 128 bit invocation ID. Unlike MainPID which can be recycled, or ActiveEnterTimestamp which can have resolution troubles, it is a failsafe way to get all the log of a particular systemd unit invocation.

To obtain the latest invocation ID of a unit

$ systemctl show --value -p InvocationID openipmi
bd3eb84c3aa74169a3dcad2af183885b

To obtain the journal of the latest invocation of, say, openipmi, whether it failed or not, you can use the one liner

$ journalctl _SYSTEMD_INVOCATION_ID=`systemctl show -p InvocationID --value openipmi.service`
-- Logs begin at Thu 2018-07-26 12:09:57 IDT, end at Mon 2019-07-08 01:32:50 IDT. --
Jun 21 13:03:13 build03.lbits openipmi[1552]:  * Starting ipmi drivers
Jun 21 13:03:13 build03.lbits openipmi[1552]:    ...fail!
Jun 21 13:03:13 build03.lbits openipmi[1552]:    ...done.

(Note that the --value is available since systemd 230, older than InvocationID)

Elazar Leibovich
  • 516
  • 1
    If someone's trying to investigate this: journalctl --user -u UNITFILE -f -o json-pretty might be helpful; you're looking for MESSAGE fields in particular.

    I found out that you might also need USER_INVOCATION_ID, and also some messages don't have any invocation id attached to them, so can't be filtered through this mechanism. Not sure why, maybe my logging is misconfigured..

     – karlicoss Jan 24 '20 at 21:52
  • The first command returns nothing for me (systemd 245) – OrangeDog Jun 04 '20 at 10:25
14

I'm not sure which timestamp makes the most sense but this works for me. Hopefully there is a better way of working with the timestamps from systemctl show than awk - could not figure out how to control the format of timestamps.

unit=foo.service

ts=$(systemctl show -p ActiveEnterTimestamp $unit)

echo $ts
ActiveEnterTimestamp=Fri 2016-11-11 12:30:01 MST

journalctl -u $unit --since "$(echo $ts | awk '{print $2 $3}')"
Erik Stephens
  • 141
  • Just in case somebody needs it as a one-liner: journalctl --since "systemctl show -p ActiveEnterTimestamp thermo.service | awk '{print $2 \" \" $3}'" -fu thermo.service | less – DimanNe Nov 04 '18 at 07:11
  • 1
    You can also use systemctl show -p ActiveEnterTimestamp --value $unit, so there's not need for extra awk – karlicoss Jan 24 '20 at 22:00
5

You can use the boot flag to fetch only the logs from that boot. for instance 

journalctl _SYSTEMD_UNIT=avahi-daemon.service -b 5
Nikolaidis Fotis
  • 2,032
  • 11
  • 13
  • 3
    This is similar to what I want, but it doesn't work in situations like: 1) if the machine has been rebooted since the last time the service ran, or 2) if the service has run multiple times since the last boot. – Jack O'Connor Apr 18 '16 at 19:03
  • I 'm not sure why it doesn't work for the 1st case. If it has been rebooted, the will be rebooted too. You just have to go to that specific boot and fetch your info. Regarding the second ... you are right. The noise logs depends on the number of times the service has been rebooted. But once you have spotted your service pid you can filter it using _PID=XXX argument. The chances of reusing the same pid for the same service on the same boot cycle are ..... no idea ... but rather close to impossible. – Nikolaidis Fotis Apr 20 '16 at 08:19
  • I'm interested in handling services that don't necessarily run on boot, either because they're on timers or because they're one-off commands. – Jack O'Connor Oct 20 '16 at 19:22
5

These might help you:

  • journalctl -u foo.service | tail -n 2

    or replace 2 with expected number of lines

  • journalctl -u foo.service --since='2016-04-11 13:00:00'

You can as well combine them to get firstly the last run time timestamp, and then use that timestamp with the --since switch.

Breign
  • 116
  • This feels like a workaround similar to the PID approach, but it's very manual. If my service runs over many seconds, and spits out many log lines, I have to go searching for the first line that has the starting timestamp I care about. That won't work very well in a script. – Jack O'Connor Apr 18 '16 at 19:06
5

You can use field filters with Journalctl. E.g.

journalctl _PID=1234

Get a list of all the available fields using:

journalctl --fields --unit kubelet

One available field is _PID.

You can get the PID of a running process using pidof or systemctl show --property MainPID <SERVICE_NAME>

So here's how I get the logs from the current Kubernetes kubelet process:

# journalctl --unit kubelet _PID=$(systemctl show --property MainPID kubelet 2>/dev/null | cut -d= -f2) | head

Now tell me why I Kubernetes is so hard to install :-(

richardw
  • 151
0

journalctl -r | grep -m1 foo.service

nieman
  • 1
  • 1
    Welcome to ServerFault. Can you explain how this is better or different than the other answers that were already here? https://serverfault.com/help/how-to-answer – chicks Aug 21 '18 at 23:10