Solvedcloud builders How do I run kubectl after image push?

It would seem that a CI/CD pipeline would require the ability to deploy an image into the gke cluster after it was built/tested/pushed.

Without this, it seems like Builder is incomplete.

31 Answers

✔️Accepted Answer

Hey!

Good news, the up-scoping feature is now live.

That means that you can add permission for Container Engine to your Cloudbuild service account, and then you can use kubectl directly in your build step:

steps:
- name: 'gcr.io/cloud-builders/docker'
  args: ["build", "-t", "gcr.io/my-project/frontend", "."]
- name: 'gcr.io/cloud-builders/docker'
  args: ["push", "gcr.io/my-project/frontend"]
- name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    gcloud components install kubectl
    gcloud container clusters get-credentials <my-cluster> --zone <my-zone>
    kubectl set image deployment/frontend frontend=gcr.io/my-project/frontend

Enjoy!

Other Answers:

To expand on the above solution, I think these steps will work at this time.
In addition to the above link, this thread was useful Here. In particular a comment by jlowdermilk.

You begin by going to IAM > Service accounts and creating a creating a service account and downloading the json key file. ex. cloudbuild@my-project.iam.gserviceaccount.com

Then you add this service account to IAM and give it project editor access (the same as cloud builder, the difference is that later on this account will have the missing scopes).

  1. On a machine with gcloud:
export CLOUDSDK_CONTAINER_USE_APPLICATION_DEFAULT_CREDENTIALS=true
gcloud container clusters get-credentials mycluster
kubectl config view --minify --flatten > kubeconfig
  1. Copy these this file and the json key file to Cloud Storage. I removed all privleges except 'Owner' and then granted the service account (cloudbuild@my-project.iam.gserviceaccount.com) read access. Not that this matters because this whole process is not very secure.

  2. Then add a build step to cloudbuild.yaml

- name: 'gcr.io/cloud-builders/docker'
  args: ["push", "gcr.io/$PROJECT_ID/imagename:$BRANCH_NAME-$REVISION_ID"]
- name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    gcloud components install kubectl
    gsutil cp gs://owner_secrets/cloud_build/$PROJECT_ID-cloudbuild.json .
    gsutil cp gs://owner_secrets/cloud_build/$PROJECT_ID-kubeconfig .
    export GOOGLE_APPLICATION_CREDENTIALS=$PROJECT_ID-cloudbuild.json
    export KUBECONFIG=$PROJECT_ID-kubeconfig
    kubectl set image deployment/deploy_name imagename=gcr.io/$PROJECT_ID/imagename:$BRANCH_NAME-$REVISION_ID --namespace=$BRANCH_NAME

That seemed to work for me at this time.

In case people are curious, here is my cloudbuild.yaml file that builds my image, pushes it to docker registry, then updates kubernetes:

My kubernetes file references my image like this:
- image: gcr.io/my-google-project/my-project:${TAG}
And is then set up with the following cloudbuild.yaml

steps:
- name: 'gcr.io/$PROJECT_ID/envsubst'
  args: ['/workspace/kubernetes.yaml', '/workspace/kubernetes.yaml']
  env:
  - 'TAG=$TAG_NAME'
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '--build-arg', 'PROJECT_ID=$PROJECT_ID', '--cache-from', 'gcr.io/$PROJECT_ID/my-project:latest', '-t', 'gcr.io/$PROJECT_ID/my-project:$TAG_NAME', '-t', 'gcr.io/$PROJECT_ID/my-project:latest', '.']
- name: 'gcr.io/cloud-builders/docker'
  args: ['push', 'gcr.io/$PROJECT_ID/my-project:$TAG_NAME']
- name: 'gcr.io/cloud-builders/kubectl'
  env: ['CLOUDSDK_COMPUTE_ZONE=us-central1-a', 'CLOUDSDK_CONTAINER_CLUSTER=my-cluster']
  args: ['apply', '-f', '/workspace/kubernetes.yaml']
images: ['gcr.io/$PROJECT_ID/my-project:latest']

The envsubst Dockerfile:

FROM alpine:3.6
MAINTAINER Nick Richardson <nick.richardson@mediapixeldesign.com>
RUN apk --update add gettext-dev

ADD envsubst-file.sh /
RUN chmod +x /envsubst-file.sh

ENTRYPOINT ["/envsubst-file.sh"]

The envsubst-file.sh:

#!/bin/sh

FILENAME=$1
FILENAME_NEW=$2

if [ ! $FILENAME ]
then
  echo 'No filename argument provided'
  exit -1
fi
if [ ! $FILENAME2 ]
then
  FILENAME_NEW=$FILENAME
fi
echo "Processing $FILENAME ..."

envsubst < $FILENAME > $FILENAME-bak
echo "Finished modifying file"
mv $FILENAME-bak $FILENAME_NEW

Yes, there are plans for this, landing soon...

At the moment, you will have trouble using kubectl from within the container builder service, since the credentials used do not have the proper scopes. We will address this issue soon.

For pushing before running another step, you can always run docker push as its own step (either instead of or in addition to listing the image in the images field).

steps:
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'gcr.io/$PROJECT_ID/foo', '.']
- name: 'gcr.io/cloud-builders/docker'
  args: ['push', 'gcr.io/$PROJECT_ID/foo']
- name: 'gcr.io/cloud-builders/kubectl'
  args: ['run', 'foo-pod', '--image=gcr.io/$PROJECT_ID/foo']
images: ['gcr.io/$PROJECT_ID/foo']

(for an upcoming kubectl step that is waiting on the scope issue)

Related Issues:

37
cloud builders Mounting files in /workspace to docker run steps
Ah ok; I think I've got it This is somewhat related to #345 Following the steps in this comment work...
20
cloud builders How do I run kubectl after image push?
Hey! Good news the up-scoping feature is now live That means that you can add permission for Contain...
16
cloud builders Fetch repositories with submodules
You can use the source repositories of gcloud instead of github It is unclear how repositories with ...
16
cloud builders Document how to connect to a database
Cloud Build uses cloudbuild Docker network so we can minimize use of raw docker run without Docker C...
11
cloud builders Google Cloud Build for Github does not fetch submodules
Try this (I use BitBucket but change the url appropriately): It works for me Thanks ...
224
bazel Bazel 0.8.1: Build fails with "Xcode version must be specified to use an Apple CROSSTOOL" on OSX High Sierra
FYI: I encountered similar issue recently This is similar to #3063 but not quite the same so I'm add...
104
bazel Ubuntu 16.04 PPA key expired
I can confirm that reinstalling the key resolves the issue; sudo apt-get update runs successfully Th...
60
msbuild Cannot find reference assemblies for .NET 3.5 or lower using core msbuild
I got this working by adding the following to the .csproj file: Unfortunately dotnet/sdk supports ta...
31
msbuild Add support for non-string assembly attributes
Sorry for being late to the party on this I know it's closed but I think it's worth considering reop...
30
msbuild "MSB3823: Non-string resources require the property GenerateResourceUsePreserializedResources to be set to true" if TargetFramework=net48
I did not quite understand We have a few projects that muti target .net core and the old fashioned ....
24
bazel Error while building tensorflow 0.11.0 - cache (directory not empty)
I attempted both solutions suggest by @sfincke and @yselivonchyk but without luck Finally ...
21
bazel Bazel installer stopped working after d3f8efc, if project has .bazelversion file
We would like to kindly request this issue to be reopened as it does appears to have broken both Baz...
21
bazel a way at start a repl
@damienmg I think this is pretty standard for any build tool whose language has a repl: Clojure: lei...
19
tensorflow windows wheel failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
I haven't seen this error before Try set CUDA_VISIBLE_DEVICES=0 Hi ...
18
bazel Compatibility with Xcode and Command Line Developer Tools on macOS
I've just hit this on a fresh Bazel clone (bc5a9b1): Environment: macOS 10.14.6 Xcode 10.3 I can con...
12
msbuild Setting BaseIntermediateOutputPath correctly in a SDK-based project is hard
To summarize if I want to override my intermediate and output folders NuGet restore drops the projec...
5
msbuild MSBuild ToolsVersions missing from hklm\software\Microsoft\MSBuild\ToolsVersions registry key
This a critical issue All previous versions of MSBuild left a registry entry in hklm\software\Micros...
4
bazel bazel-built protoc segfaults on mac os monterey
I have had success working around this with the command line option --incompatible_linkopts_to_linkl...
4
msbuild Warning AL1073 when .resx files is compiled under x64
To be clear for those hitting this the workaround should be very similar to the workaround in the OP...
4
pants 2.3.0rc0 - ProcessExecutionFailure: Process 'Extracting plugin locations' failed with exit code 2
@jsirois Thanks! Can confirm this works for me now I should also note that in plugin_resolver.py ...
3
assemble yfm in partials doesn't work
Since you're using the built-in Handlebars syntax for the partial: {{> foo }} assemble does not merg...
3
bazel Make maven_jar and friends smarter by re-using previously fetched artifacts across different projects
0590483 now lets you use --experimental_repository_cache=$HOME/some/path to cache downloaded artifac...
684
laradock Mysql. The server requested authentication method unknown to the client [caching_sha2_password]
alter user 'username'@'localhost' identified with mysql_native_password by 'password'; would fix it....
654
nvidia docker OpenCV Docker error "ImportError: libSM.so.6: cannot open shared object file: No such file or directory"
I fixed this problem on nvcr.io/nvidia/tensorflow:18.12-py3 with (using solution above): ...
457
compose Docker-compose up failing because "port is already allocated"
I ran into the same issue today (with a postgres container) and despite having tried docker-compose ...
447
moby The name "/data-container-name" is already used by container <hash>. You have to remove (or rename) that container to be able to reuse that name.
I have a helper function to nuke everything so that our Continuous blah cycle can be tested erm.. co...
371
compose Compose error "HTTP request took too long to complete"
By simply restarting the docker service via sudo service docker restart I was able to get the aforem...
369
compose error on launching docker-compose by piping to sh ( echo 'docker-compose ... ' | sh )
I could get it to work by adding the -T parameter to not create a Pseudo-TTY docker-compose exec -T ...
337
compose docker-compose up fails if network attached to container is removed
Thanks for the report! I think there are several things to note here: First and foremost ...
316
nvidia docker docker: Error response from daemon: Unknown runtime specified nvidia.
I've also installed correctly but forgot to restart daemon in ubuntu it may resolve your error. ...
297
compose Error when trying to run docker-compose up. "oci runtime error: container_linux.go:247..."
you gotta make the docker-entrypoint.sh an executable before building the image: otherwise it cant b...
292
laradock SQLSTATE[HY000] [2054] The server requested authentication method unknown to the client
+1 I'm having the same problem here. Info: Docker version ($ docker --version): Docker version 17.12...
257
compose docker-compose up doesn't pull down latest image if the image exists locally
Imagine that git didn't have pull because git fetch && git merge origin/master is functionally ident...
236
nvidia docker could not select device driver "" with capabilities: [[gpu]].
Hello! If you didn't already make sure you've installed the nvidia-container-toolkit If this doesn't...
205
moby docker-engine 1.10.2-0~trusty can't install on clean Ubuntu 64-bit 14.04.3
I seem to have resolved this by putting deb http://cz.archive.ubuntu.com/ubuntu trusty main in /etc/...
183
moby Docker service update --image "could not accessed on a registry to record its digest"
When updating services that need credentials to pull the image you need to pass --with-registry-auth...
178
laradock MySQL Container fails to start
I had the same issue last night I think it's the mysql version problem What I did was edited laradoc...
169
compose Docker Compose mounts named volumes as 'root' exclusively
Actually I come here with news it seems what I am trying to achieve is doable but I don't know if th...
149
compose INTERNAL ERROR: cannot create temporary directory!
Confirming this happened to me Today Was running low on space: After removing a container.. it works...
147
cookiecutter django No support for python3? I am getting: invalid syntax: raise ValueError, "No frame marked with %s." % fname
For me the issue was that I installed the environ package instead of the django-environ package. ...
147
compose docker-compose up -d doesn't expose ports when defined with build directive
oh you didn't specify but I'm assuming you're using run instead of up? If so you need --service-port...
142
compose How does compose chooses subnet for default network?
I'm also running into this issue Another way around this is to set the default-address-pools in your...
139
ddev In WSL2 ddev start fails at docker-credential-desktop.exe, "error listing credentials"
I had to set credsStore: in my ~/.docker/config.json .. it was previously set to credentials.exe ...
139
docker touch: cannot touch ‘/var/jenkins_home/copy_reference_file.log’: Permission denied
as mentioned there you need to figure out your volume mapping permissions ie. I have the same issue ...
138
kubernetes ingress 413 Request Entity Too Large
FYI the annotation has changed and is now: Also I had to restart the nginx pod for the effect to tak...
127
moby docker daemon unable to access registry - Client.Timeout exceeded while awaiting headers
I found out that the problem might be in /etc/resolv.conf I had: but moving the non-working (yet) 10...
120
cli How to skip one stage from multi-stage docker build
Docker 18.06 has been released Description We have Multistage docker build that creates rpm in each ...
119
compose ERROR: for db Cannot start service db: driver failed programming external connectivity on endpoint ltg_db_1
The following worked for me when i do the following : ± docker-compose up Starting ltg_db_1 ERROR: f...