Job randomly fails when executing a shell script on Jenkins

Question

I have a jenkins server running on Ubuntu 16.04.2 x64 but recently the jobs started to fail randomly when a shell script is executed. I can execute the script manually on the server and it works but when jenkins executes then it fails for these reasons:

Fail 1

JavaProcess leaked file descriptors. See https://jenkins.io/redirect/troubleshooting/process-leaked-file-descriptors for more information
Build step 'Execute shell' marked build as failure

Fail 2

# gradle script that downloads some jars..
Download https://jcenter.bintray.com/org/pegdown/pegdown/1.6.0/pegdown-1.6.0.jar
Build step 'Execute shell' marked build as failure
Skipped archiving because build is not successful

It's not always the same jar that fails to get downloaded in Fail 2.

I have tried to update jenkins to latest version (2.164.2) but still same problem.

I have tried the solutions mentioned in Process leaked file descriptors error on JENKINS for Fail 1 but no success, still fails the same way.

Update 1

I have localized the problem in my shell script to this part:

# Run script in docker container
chmod +x tmp.sh
docker run --entrypoint=/bin/bash -v $(pwd):/src --workdir=/build mydocker/myimage -c "/src/tmp.sh"
rm -f tmp.sh

So when docker is executing my tmp.sh through Jenkins "Execute shell" it fails randomly.

Log from docker journalctl -u docker.service

May 21 10:33:23 jenkins dockerd[1332]: time="2019-05-21T10:33:23.886024261Z" level=error msg="attach: stdout: write unix /var/run/docker.sock->@: write: broken pipe"
May 21 10:33:25 jenkins dockerd[1332]: time="2019-05-21T10:33:25.186663914Z" level=error msg="attach: stderr: write unix /var/run/docker.sock->@: write: broken pipe"
May 21 10:33:25 jenkins dockerd[1332]: time="2019-05-21T10:33:25.186715731Z" level=error msg="attach failed with error: write unix /var/run/docker.sock->@: write: broken pipe"

Log from docker container docker container logs cranky_cori

BUILD SUCCESSFUL

Total time: 38.273 secs

This build could be faster, please consider using the Gradle Daemon: https://docs.gradle.org/2.14/userguide/gradle_daemon.html

Conclusion

Pipe to stdout and stderr gets broken and jenkins reports this as failed build but the docker container logs shows that docker continued executing and successfully finished my gradle script.

@yarin as it says in my description - I've tried running the command manually and it doesn't fail — vato, May 16 '19 at 14:14
thanks @yarin, I tried as jenkins user and root - it still works when running it manually. — vato, May 17 '19 at 12:13
@yarin I have updated with some more information, see update1. Do you have any idea of what might cause the problem now? Thanks in advance! — vato, May 21 '19 at 08:40

score 0 · Accepted Answer · answered May 21 '19 at 14:14

After unsuccessfully finding the cause of my issue I though of upgrading my OS, and boy I was lucky!

Configuration not working

Ubuntu 16.04.2 x64
----------------------------
- Docker version 17.03.1-ce
- Jenkins 2.64.2
NOR 
- Docker version 18.09.6
- Jenkins 2.64.2
- Jenkins 2.64.3

After upgrading Ubuntu this worked for me

Ubuntu 18.04.2 LTS 
----------------------------
- Docker version 18.09.6
- Jenkins 2.64.3

Job randomly fails when executing a shell script on Jenkins

1 Answers1