3

Since today I haven't been able to push new signed images to a Docker Hub private repository due to image signing failing. I have Docker Content Trust enabled. I don't know of any significant changes in my environment that could affect this, except routinely installing the latest security updates to Ubuntu a couple days ago. But signed image pushing did work after those upgrades.

My question is, how to go about debugging signing related problems? There seem to be not much available by googling or duckduckgoing.

I tried running the notary CLI but it didn't seem to provide much help to me. The various options of different commands are not very well documented.

Environment:

  • OS: Ubuntu 18.04.1 LTS
  • Docker version 18.06.1-ce, build e68fc7a

  • relevant environment variables:

    DOCKER_CONTENT_TRUST=1 DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE=[undisclosed] DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE=[undisclosed]

Failing command:

docker push xyz/abc:def

Sample output (obfuscated from the unrelevant parts):

user@machine:~$ source .docker-signing-credentials
user@machine:~$ export DOCKER_CONTENT_TRUST=1
user@machine:~$ docker push myorg/myproject:myimage_v1.38.0
The push refers to repository [docker.io/myorg/myproject]
c72506834af4: Layer already exists
043ae531d76e: Layer already exists
... 8< ... snip ... 8< ...
af840f32f0a2: Layer already exists
8decd5535924: Layer already exists
myimage_v1.38.0: digest: sha256:baa3e1148e0100df8cbb0aab46200be2bdf600d7802d7cddb3a23c12053af82d size: 8883
Signing and pushing trust metadata
failed to sign docker.io/myorg/myproject:myimage_v1.38.0: An error occurred during validation: rpc error: code = 14 desc = grpc: RPC failed fast due to transport failure

When I unset DOCKER_CONTENT_TRUST, there is no problem with pushing the images.

Matti Mäki
  • 31
  • 3
  • I have also used some hours today to try to solve this without any progress. – Ville Laitila Oct 03 '18 at 13:28
  • Although the root cause for the issue was found out and resolved, I won't mark any of the answers because my question was "how to go about debugging signing related problems" and that remains yet a mystery to me. – Matti Mäki Oct 08 '18 at 10:56

2 Answers2

0

There is an issue open with this same exact description in:

https://github.com/docker/hub-feedback/issues/1646

it might be a good idea to join to this issue.

Jorge Ramirez
  • 1
  • While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes – Jeroen Heier Oct 03 '18 at 17:33
0

The root cause was the issue of degraded performance in Docker Hub Notary. See the resolution by Docker support.

Ville Laitila
  • 1,187
  • 11
  • 18