2

I'm reading this post and I don't fully follow the answer.

I wondered if I could modify this, what's it called, command redirect?

What I would like:

  1. Don't hang, use ampersand & to move on to the next command
  2. Output the data generated by the command to file data.json
  3. Any problems, error messages or issues running the command, output them to bqlog.txt

Is this possible?

E.g. I currently have this command:

bq query --max_rows=1000000 --use_legacy_sql=false --format=json --batch=true --headless=true 'select * from `ga4-extract.analytics_123456789.events_'"$RUNDATE"'`;' > data.json

It's failing after about 30 minutes and I cannot see the output or what the issue is. This is on a github actions runner and the console output of the workflow just gives a failed status but, unique to this workflow, no console output indicating the error.

I wanted to modify my command so that I can output to data.json like I already have, but then also move on to the next step of monitoring the bqlog.txt file with tail -F bqlog.txt

Doug Fir
  • 155

1 Answers1

6

Yes, this is possible.

The 2>&1 means the following:

2 : stderr
> : output redirect
& : copy file descriptor
1 : file descriptor 1

So: stderr output redirect to wherever file descriptor 1 (stdout) is directed. Default: the screen.

This means, stderr is outputted to the screen.

Your question is, how can I redirect the output of my command to file 1, and the stderr to another?

Your command becomes:

command >output1.txt 2>output2.txt

To use your own code, you would write (I've added enters to avoid a scrollbar and making it readable. The same command without the enters and scrollbar is below:

bq query --max_rows=1000000 --use_legacy_sql=false --format=json 
   --batch=true --headless=true 
   'select * from `ga4-extract.analytics_123456789.events_'"$RUNDATE"'`;
   ' >data.json 2>bglog.txt

or without enters:

bq query --max_rows=1000000 --use_legacy_sql=false --format=json --batch=true --headless=true 'select * from `ga4-extract.analytics_123456789.events_'"$RUNDATE"'`;' >data.json 2>bglog.txt

Do note, this command will create an empty file if there is no output, and it will overwrite the file on every command. If you want to append, use >> instead. Eg:

>>data.json 2>>bglog.txt

In addition you may want to add the following:

echo --------------------- >> data.json
echo --------------------- >> bglog.txt

This will write ---- to both files, helping you to see when an old execution and new exection takes place.

ilkkachu
  • 1,221
LPChip
  • 61,264
  • Ah thank you! I can output err direct to console then? Don't even need to follow the log file? Is that right? – Doug Fir Jul 30 '22 at 21:28
  • 4
    by default stderr is logged to console, you don't need to do anything for that. the >txt only logs normal output. By using 2>&1 you say, I want stderr to be redirected to whatever my output already is. so >txt 2>&1 basically makes stderr also output to txt – LPChip Jul 30 '22 at 21:30
  • 1
    & does not mean ‘append to’, that would be >>. – user3840170 Jul 31 '22 at 06:24
  • Regarding multi-line - you can use ^ to break long lines in bat/cmd scripts. – Jonathan Jul 31 '22 at 07:08
  • "append" has a particular meaning when outputting to a file: it means all writes go the very end of the (existing) file, regardless of anything else. You can create a redirection in append mode with >>, it's also different from a regular one (>), also in that the regular mode truncates the file first. & only tells to copy an existing file descriptor, it does not change between regular and append mode. (the answer also uses the proper meaning of the word later) – ilkkachu Jul 31 '22 at 09:05