Solvedskaffold Skaffold rebuilds all images on every run (GitHub Action) - also the unaffected ones

I wish to have a GitHub Action that deploys all services in GKE if the service has been changed (either in manifest, source code or dependency). I wish that the action uses cache for the building.

The action looks like this

name: Deploy to staging
      - develop
      - "k8s-manifest/**/*"
      - "lib/**"
      - "services/**"
      - "skaffold.yaml"

  GKE_CLUSTER: parallax-cluster
  GKE_ZONE: europe-west3-a

    name: Deploy
    runs-on: ubuntu-latest
      - name: Check out repository on develop branch
        uses: actions/checkout@v2
          ref: develop
      - name: Install gcloud
        uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
          version: '309.0.0'
          service_account_key: ${{ secrets.STAGING_SA_BUILD_AND_DEPLOY }}
          project_id: ${{ secrets.STAGING_PROJECT }}
      - name: Install kustomize, kubectl and skaffold
        uses: daisaru11/setup-cd-tools@v1
          kubectl: '1.19.2'
          kustomize: '3.8.4'
          skaffold: '1.14.0'
      - name: Configure docker
        run: |
          gcloud --quiet auth configure-docker
      - name: Connect to cluster
        run: |
          gcloud container clusters get-credentials "$GKE_CLUSTER" --zone "$GKE_ZONE"
      - name: Build and deploy to cluster
        run: |
          skaffold run \

while my skaffold.yaml looks like this

apiVersion: skaffold/v2alpha2
kind: Config
    concurrency: 0
    - image: python-backend
      context: .
        dockerfile: services/backend/Dockerfile
    - image: horizon-worker
      context: .
        dockerfile: services/horizon_worker/Dockerfile
    - image: react-frontend
      context: services/frontend

  - name: staging
        path: k8s-manifest/staging
  - name: production
        path: k8s-manifest/production
  - name: local
        path: k8s-manifest/local
      - kubeContext: minikube

Expected behavior

I'd expect the above action to only rebuild affected images

Actual behavior

The above action rebuilds all images - also the ones that already exist in the default repo and which hasn't had any changes. I'd expect that skaffold run would realize that the unaffected images wouldn't be rebuild (works if I do skaffold run from my local machine).

I have considered setting up a self-hosted runner which always will have the cache available but I imagine that my task is very common for the community so I'm probably just missing something essential here to do. What is the best practice?

24 Answers

βœ”οΈAccepted Answer

I run skaffold run -v debug in debug mode and found some interesting logs.

2020-10-02T23:16:13.7729379Z time="2020-10-02T23:16:13Z" level=debug msg="Could not import artifact from Docker, building instead (import of missing images disabled)"

The whole issue can be resolved by setting tryImportMissing πŸ˜„

apiVersion: skaffold/v2beta8
kind: Config
    sha256: {}
    concurrency: 0
    tryImportMissing: true

Why isn't it true by default? We should also improve the docs around local caching.

Related Issues:

skaffold Pass --network=host during docker build
I'm having the same issue when skaffold build during jenkins the container have no internet access a...
skaffold Skaffold rebuilds all images on every run (GitHub Action) - also the unaffected ones
I run skaffold run -v debug in debug mode and found some interesting logs I wish to have a GitHub Ac...
distribution Private registry push fail: server gave HTTP response to HTTPS client
I get helped from [
kubernetes deleting namespace stuck at "Terminating" state
@ManifoldFR I had the same issue as yours and I managed to make it work by making an API call with j...
moby The name "/data-container-name" is already used by container <hash>. You have to remove (or rename) that container to be able to reuse that name.
I have a helper function to nuke everything so that our Continuous blah cycle can be tested erm.. co...
kubernetes PV is stuck at terminating after PVC is deleted
I got rid of this issue by performing the following actions: Then I manually edited the pv individua...
kubernetes x509 cert issues after kubeadm init
do you have $KUBECONFIG pointing to /etc/kubernetes/kubelet.conf? BUG REPORT: (I think?) What happen...
kubernetes Ingress: Allow for multiple hosts
I also would like to see this feature but as a workaround I use YAML ids Here is how it would look f...
kubernetes The connection to the server localhost:8080 was refused - did you specify the right host or port?
Run these commands solved this issue: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HO...
minikube minikube start - Error starting host, machine does not exist
On macOS Sierra rm -rf ~/.minikube fixed it for me After that minikube start worked as expected mini...
kubernetes Force pods to re-pull an image without changing the image tag
@yujuhong Sometimes it's very useful to be able to do this For instance Problem A frequent question ...
moby docker-engine 1.10.2-0~trusty can't install on clean Ubuntu 64-bit 14.04.3
I seem to have resolved this by putting deb trusty main in /etc/...
moby Docker service update --image "could not accessed on a registry to record its digest"
When updating services that need credentials to pull the image you need to pass --with-registry-auth...
kubernetes 'unknown revision v0.0.0' errors, seemingly due to 'require v0.0.0'
For anyone else who hits this issue after much weeping and gnashing of teeth this is the little scri...
minikube kube-proxy configmap update: timed out (unknown root cause)
I had this error when upgrading from 0.25 to 0.26.1 Simply performing minikube delete and then re-cr...
kubernetes JSONpath fails to return keys containing dots in a map
Escaping dots works now To revisit the example in my original message: Closing πŸŽ‰ ...
moby docker daemon unable to access registry - Client.Timeout exceeded while awaiting headers
I found out that the problem might be in /etc/resolv.conf I had: but moving the non-working (yet) 10...
kubernetes "Failed to setup network for pod \ using network plugins \"cni\": no IP addresses available in network: podnet; Skipping pod"
I had a simliar issue while testing kubernetes with kubeadm This started to happen after I did a kub...
kubernetes Kubectl cp gives "tar: removing leading '/' from member names" warning
Something I found is that if I do not put a / at the beginning of my path following : in <pod>:<path...
moby docker.service Failed with result 'start-limit-hit'.
I had the same error message once because of an empty /etc/docker/daemon.json file Delete it if you ...
kubernetes no kind "Deployment" is registered for version "apps/v1beta2"
Small tip: To find out what exact apps/xxx api version your cluster supports use kubectl api-version...
minikube Can't pull images from an insecure registry in Minikube VM
I just tried this with minikube v0.10.0 and --insecure-registry='' wa...
rancher Namespace created by rancher can't delete
This is a known issue with removing an imported cluster (and in the process of being fixed) but you ...
che event-stream-3.3.6.tgz not found in Yarnpkg.
this is due to : to...
moby containerd: start container" error="oci runtime error: fork/exec /usr/bin/docker (deleted): no such file or directory: "
Not sure if this is the same cause but this might help someone Steps to reproduce the issue: Install...
moby Error response from daemon: Get
may be you are behind a firewall/proxy server i was also behind my office firewall so i tried below ...
kubernetes Scale down a deployment by removing specific pods
In my company we have the exact same case of that of @antoinne85: workers that pick tasks from a que...
kubernetes Ubuntu 16.04 LTS - Unable to locate package kubelet, kubeadm, kubectl
Hi @monique77 On Ubuntu 16.04: Add key for new repository: Add repository: Install Kubernetes: Regar...
vscode remote release VS Code Server for WSL closed unexpectedly: Input/output error
I just got this issue too I installed the Windows May 2020 Update yesterday and then installed Ubunt...
kubernetes Kubernetes-cni issue with 1.9.0 - no ip address available in range
I had the same issue on Ubuntu 16.04.1 LTS (using flannel for the networking) Is this a BUG REPORT o...
minikube Error getting state for host: exit status 126
As a referential point for anyone who run into this issue on El Capitan and found this via Google (l...
minikube Can't use Minikube on VPN
I know this is closed but I'm adding this comment to describe how we made this work: Set port forwar...
moby linux spec user: Unable to find user xxx
Why not use the numeric ID of the user instead of a user-name; doing so you don't have to bind-mount...
minikube Failed to restart crio.service: Unit crio.service not found.
Ran into the same issue However I am working on Linux The following steps did the trick for me: Unin...
moby 20.10.0-beta1@Fedora 33: Failed to program NAT chain: ZONE_CONFLICT: 'docker0' already bound to a zone
I've got a similar error after upgrading docker to version 20.10 on Fedora 32 To fix this ...
kubernetes Flannel (NetworkPlugin cni) error: /run/flannel/subnet.env: no such file or directory
Just got the same problem - fixed it by manually adding the file: /run/flannel/subnet.env ...
kubernetes Pods stuck on terminating
I have the same issue on Kubernetes 1.8.2 on IBM Cloud After new pods are started the old pods are s...
moby Named Volumes in Dockerfile
This is by design With docker-compose I can use named volumes like this: What about Dockerfile? I tr...
minikube No example to use minikube with private registry
For Google Container Registry (using oauth2) this works for me: kubectl create secret docker-registr...
moby Add daemon options for Docker network subnets
This is a feature request and I support it! docker daemon should have two additional options: --net-...
kubernetes Deployment with recreate strategy does not remove old replica set
Thought this might help anyone looking at how to remove inactive replicasets (spec.replicas set to z...
moby Trying to connect to Docker API for shell commands throws error
This is likely due to either the Virtual Machine not being started (try docker-machine start default...
kubernetes network plugin is not ready: cni config uninitialized
removing $KUBELET_NETWORK_ARGS not work with me. Hello I want to do a fresh install of kubernetes vi...
kops etcd-manager certificate rotation
The certs and keys in S3 are only for the CA certificates which have a 10 year expiration ...
kubernetes gcloud container get-credentials not authenticating service account
Previously gcloud would have configured kubectl to use the cluster's static client certificate to au...
kubernetes kube-proxy iptables load-balancing probability is not equal
This is intended because these iptables rules will be examined from top to bottom Take your case as ...
kops kubelet-api is Forbidden to logs pods
I'm not sure this is the right solution 1 What kops version are you running? The command kops versio...
kubernetes (1.17) Kubelet won't reconnect to Apiserver after NIC failure (use of closed network connection)
I've fixed it by running this bash script every 5 minutes: We've just upgrade our production cluster...