Bash: Extract Data from logfile x hours ago, starting from last loginput

Question

I am new to Bash, and I have an assignment to extract data from an logfile, depending on the timestamp. I want to be able to see inputs in the logfile for the last h hours, starting from the latest input in the file. I have some code but It does not work since it writes EVERYTHING in the logfile.

A part of my logfile looks like this:

213.64.56.208 - - [01/Jan/2003:10:14:34 +0100] "GET
213.64.56.208 - - [01/Jan/2003:10:14:36 +0100] "GET
213.64.56.208 - - [01/Jan/2003:10:14:39 +0100] "GET
213.64.56.208 - - [01/Jan/2003:10:14:42 +0100] "GET
213.64.56.208 - - [01/Jan/2003:10:14:47 +0100] "GET
213.64.56.208 - - [01/Jan/2003:10:14:49 +0100] "GET
213.64.56.208 - - [01/Jan/2003:10:14:52 +0100] "GET
213.64.56.208 - - [01/Jan/2003:10:14:57 +0100] "GET
213.67.145.223 - - [01/Jan/2003:11:00:06 +0100] "HEAD
213.46.27.204 - - [01/Jan/2003:12:55:15 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:15 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:17 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:17 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:18 +0100] "GET

My code is supposed to get the timestamp of the last input, and compare to the others, but the comparison does not seem to work. Here is the code:

if [ $h -gt 0 ]
then    
    echo " A specified time is set! "
    TimeInSeconds=$((h*60*60)) # set to seconds instead of hours
    last=$(tail -n1 thttpd.log |awk -F'[][]' '{ gsub(/\//," ",$2); sub(/:/," ",$2); "date +%s -d \""$2"\""| getline d; print d;}')
    awk -F'[][]' -v last=$last -v x=$TimeInSeconds '{ gsub(/\//," ",$2); sub(/:/," ",$2); "date +%s -d \""$2"\""|getline d; if (last-date<=x)print $1 "[" $2 "]"  }' thttpd.log

As I said, it does not print the correct timespan, I am sure there's an easy solution to this, but I can't see it.

Does anyone see the error?

If you want the "last `h` hours" is that supposed to be `h` hours ago from the current time, or the last `h` hours of the file? In other words, if there's been no activity for the last `h` hours, should the output be nothing? — lurker, Sep 15 '13 at 11:25
Last h hours of the file, since I am supposed to be able to check old log files. — Katja, Sep 15 '13 at 12:41

score 1 · Accepted Answer · edited May 23 '17 at 11:50

Your question is very similar to this thread. I'm forwarding the solution here but just with minor modifications respecting to your requirement.

#!/bin/bash

H=1  ## Hours
LOGFILE=/path/to/logfile.txt

X=$(( H * 60 * 60 )) ## Hours converted to seconds

function get_ts {
    DATE="${1%%\]*}"; DATE="${DATE##*\[}"; DATE=${DATE/:/ }; DATE=${DATE//\// }
    TS=$(date -d "$DATE" '+%s')
}

get_ts "$(tail -n 1 "$LOGFILE")"
LAST=$TS

while read -r LINE; do
    get_ts "$LINE"
    (( (LAST - TS) <= X )) && echo "$LINE"
done < "$LOGFILE"

Run this script with bash script.sh.

Example output:

213.46.27.204 - - [01/Jan/2003:12:55:15 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:15 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:17 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:17 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:18 +0100] "GET

If you want you can let it accept arguments instead:

#!/bin/bash

H=$1
LOGFILE=$2

...

And run bash script.sh h logfile where h is the number of hours and logfile is the path to your logfile.

score 1 · Answer 2 · answered Sep 15 '13 at 13:41

1

The problem is that on the last line, getline is reading into the variable d but in the comparison you are using the variable date.

answered Sep 15 '13 at 13:41

Jester

56,577
4
81
125

Ed Morton · Answer 3 · 2013-09-15T13:58:43.490

With GNU awk for it's time functions:

$ cat tst.awk
function time2secs(time,        t) {
    split(time,t,/[/:]/)
    t[2] = (match("JanFebMarAprMayJunJulAugSepOctNovDec",t[2])+2)/3
    return mktime(t[3]" "t[2]" "t[1]" "t[4]" "t[5]" "t[6])
}
BEGIN{ FS="[[ ]"; ARGV[ARGC++] = ARGV[ARGC-1]; xs= x * 60 * 60 }
FNR == NR { lasttime = $5; next }
FNR ==  1 { tstamp = time2secs(lasttime) - xs }
time2secs($5) >= tstamp
$
$ awk -v x=1 -f tst.awk file
213.46.27.204 - - [01/Jan/2003:12:55:15 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:15 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:17 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:17 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:18 +0100] "GET
$
$ awk -v x=2 -f tst.awk file
213.67.145.223 - - [01/Jan/2003:11:00:06 +0100] "HEAD
213.46.27.204 - - [01/Jan/2003:12:55:15 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:15 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:17 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:17 +0100] "GET
213.46.27.204 - - [01/Jan/2003:12:55:18 +0100] "GET

Bash: Extract Data from logfile x hours ago, starting from last loginput

3 Answers3