Solvedkaniko [0.21.0] Permission denied when using GCR caching

Actual behavior
After updating to 0.21.0, I get access denied (I'm using GCR).
I haven't tried 0.20.0, because of the bugs present on this one.
But it was working with 0.19, 0.18, 0.17, 0.16.

Expected behavior
Being able to push layers through GCR.

To Reproduce
Steps to reproduce the behavior:

  1. Try to build an image with GCR

Additional Information

  • Dockerfile
FROM python:3.7.5

ENV PATH /usr/local/nvidia/bin/:$PATH
ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64

ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
LABEL com.nvidia.volumes.needed="nvidia_driver"

ENV CUDA_VISIBLE_DEVICES=0

ENV PACKAGES="\
build-essential \
libssl-dev \
liblzma-dev \
libmagickwand-dev \
curl \
zip \
"
RUN apt -y update \
    && apt -y upgrade \
    && apt -y install ${PACKAGES}
INFO[0157] Pushing layer eu.gcr.io/{PROJECT}/{REPO}/cache:aa9e858784087f68c07788a90e987a85f0e102442d2cc080745fe19dbee270e0 to cache now 
E0506 05:32:34.870621      11 metadata.go:154] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
E0506 05:32:34.873013      11 metadata.go:166] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
WARN[0159] error uploading layer to cache: failed to push to destination eu.gcr.io/{PROJECT}/{REPO}/cache:4e242b84587a99fb7635dd599ce64c320231736fa6738bbe3f7942b267a5c928: PATCH https://eu.gcr.io/v2/{PROJECT}/{REPO}/cache/blobs/uploads/AJnulEH5k9X4qLOJKDk-D0QqRVMYP61rQ4XPeRewSOT23RDJNQ68Ne_FHk-iacL9uA8FxhruPQ2iC12YXQYywPc: DENIED: Access denied. 
error pushing image: failed to push to destination eu.gcr.io/{PROJECT}/{REPO}:deploy-cache-fix-4e98c20555c3b6459ad511637b771a2cda57b460: PATCH https://eu.gcr.io/v2/{PROJECT}/{REPO}/blobs/uploads/AJnulEEKHdOiEGXQKIq8VaS2hKHziCksXlEa7qer2NOQFUImIWOjkZgEW9jzBmTs8hz-aCOM72rnVm87OjlE5N8: DENIED: Access denied.

Triage Notes for the Maintainers

Description Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

Please let me know if you need anything else !

13 Answers

✔️Accepted Answer

Our systems are also impacted by this, with the same permission denied error. I'm assuming this issue will become very popular in the next hours :)

If you're encountering this, you are probably not specifying a version of Kaniko in your cloudbuild.yml, and therefore, always using the latest version.
As a quick fix, just specify yesterday's version as so:

# Please update kaniko version occasionally
- name: 'gcr.io/kaniko-project/executor:v0.20.0'

More Issues: