Solvedcloud builders Google Cloud Build for Github does not fetch submodules

I am using the Google Cloud Build for Github and it seems that it checkouts a copy (non git) of source code excluding submodules into the work directory.

Is there a way to have the submodules checked out as well?

15 Answers

✔️Accepted Answer

Try this (I use BitBucket but change the url appropriately):

- id: git-submodule
  name: 'gcr.io/cloud-builders/git'
  entrypoint: 'bash'
  env: ['GIT_DISCOVERY_ACROSS_FILESYSTEM=1']
  args:
  - '-c'
  - |
    git init
    git config -f .gitmodules submodule.packages.url https://source.developers.google.com/p/[project]/r/bitbucket_[submodule]
    git submodule init
    git submodule update

It works for me.
Thanks

Other Answers:

I'm using the github trigger, and submodules are not being cloned. Here is my workaround, derived from the private repo tutorial

I put id_rsa.enc and known_hosts in storage, but feel free to put them in your build

e.g.

gsutil ls  gs://${PROJECT_ID}_cloudbuild/deps/root/.ssh/
gs://XXXXX_cloudbuild/deps/root/.ssh/
gs://XXXXX_cloudbuild/deps/root/.ssh/id_rsa.enc
gs://XXXXX_cloudbuild/deps/root/.ssh/known_hosts
steps:
- name: 'gcr.io/cloud-builders/gsutil'
  args:
  - 'cp'
  - '-r'
  - 'gs://${PROJECT_ID}_cloudbuild/deps/root/.ssh'
  - '/root/'
  volumes:
  - name: 'ssh'
    path: /root/.ssh
- name: 'gcr.io/cloud-builders/gcloud'
  args:
  - kms
  - decrypt
  - --ciphertext-file=/root/.ssh/id_rsa.enc
  - --plaintext-file=/root/.ssh/id_rsa
  - --location=global
  - --keyring=${PROJECT_ID}
  - --key=github-key
  volumes:
  - name: 'ssh'
    path: /root/.ssh
# Set up git with key and domain.
- name: 'gcr.io/cloud-builders/git'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    chmod 600 /root/.ssh/id_rsa
    cat <<EOF >/root/.ssh/config
    Hostname github.com
    IdentityFile /root/.ssh/id_rsa
    EOF
  volumes:
  - name: 'ssh'
    path: /root/.ssh
- name: 'gcr.io/cloud-builders/git'
  entrypoint: 'bash'
  args:
   - '-c'
   - |
      git submodule init
      git submodule update
  volumes:
  - name: 'ssh'
    path: /root/.ssh

Also seeing this, not sure why the --recurse-submodules flag is not included in the default git clone hook. For my use case of simply building the contained Dockerfile, the following cloudbuild config works (though note I only have public modules so YMMV):

steps:
- name: gcr.io/cloud-builders/git
  args: ['submodule', 'update', '--init', '--recursive']
- name: gcr.io/cloud-builders/docker
  args: ['build', '-t', 'gcr.io/$PROJECT_ID/code', '.']
images: ['gcr.io/$PROJECT_ID/code']

Related Issues:

37
cloud builders Mounting files in /workspace to docker run steps
Ah ok; I think I've got it This is somewhat related to #345 Following the steps in this comment work...
20
cloud builders How do I run kubectl after image push?
Hey! Good news the up-scoping feature is now live That means that you can add permission for Contain...
16
cloud builders Fetch repositories with submodules
You can use the source repositories of gcloud instead of github It is unclear how repositories with ...
16
cloud builders Document how to connect to a database
Cloud Build uses cloudbuild Docker network so we can minimize use of raw docker run without Docker C...
11
cloud builders Google Cloud Build for Github does not fetch submodules
Try this (I use BitBucket but change the url appropriately): It works for me Thanks ...
224
bazel Bazel 0.8.1: Build fails with "Xcode version must be specified to use an Apple CROSSTOOL" on OSX High Sierra
FYI: I encountered similar issue recently This is similar to #3063 but not quite the same so I'm add...
104
bazel Ubuntu 16.04 PPA key expired
I can confirm that reinstalling the key resolves the issue; sudo apt-get update runs successfully Th...
60
msbuild Cannot find reference assemblies for .NET 3.5 or lower using core msbuild
I got this working by adding the following to the .csproj file: Unfortunately dotnet/sdk supports ta...
31
msbuild Add support for non-string assembly attributes
Sorry for being late to the party on this I know it's closed but I think it's worth considering reop...
30
msbuild "MSB3823: Non-string resources require the property GenerateResourceUsePreserializedResources to be set to true" if TargetFramework=net48
I did not quite understand We have a few projects that muti target .net core and the old fashioned ....
24
bazel Error while building tensorflow 0.11.0 - cache (directory not empty)
I attempted both solutions suggest by @sfincke and @yselivonchyk but without luck Finally ...
21
bazel Bazel installer stopped working after d3f8efc, if project has .bazelversion file
We would like to kindly request this issue to be reopened as it does appears to have broken both Baz...
21
bazel a way at start a repl
@damienmg I think this is pretty standard for any build tool whose language has a repl: Clojure: lei...
19
tensorflow windows wheel failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
I haven't seen this error before Try set CUDA_VISIBLE_DEVICES=0 Hi ...
18
bazel Compatibility with Xcode and Command Line Developer Tools on macOS
I've just hit this on a fresh Bazel clone (bc5a9b1): Environment: macOS 10.14.6 Xcode 10.3 I can con...
12
msbuild Setting BaseIntermediateOutputPath correctly in a SDK-based project is hard
To summarize if I want to override my intermediate and output folders NuGet restore drops the projec...
5
msbuild MSBuild ToolsVersions missing from hklm\software\Microsoft\MSBuild\ToolsVersions registry key
This a critical issue All previous versions of MSBuild left a registry entry in hklm\software\Micros...
4
bazel bazel-built protoc segfaults on mac os monterey
I have had success working around this with the command line option --incompatible_linkopts_to_linkl...
4
msbuild Warning AL1073 when .resx files is compiled under x64
To be clear for those hitting this the workaround should be very similar to the workaround in the OP...
4
pants 2.3.0rc0 - ProcessExecutionFailure: Process 'Extracting plugin locations' failed with exit code 2
@jsirois Thanks! Can confirm this works for me now I should also note that in plugin_resolver.py ...
3
assemble yfm in partials doesn't work
Since you're using the built-in Handlebars syntax for the partial: {{> foo }} assemble does not merg...
3
bazel Make maven_jar and friends smarter by re-using previously fetched artifacts across different projects
0590483 now lets you use --experimental_repository_cache=$HOME/some/path to cache downloaded artifac...
684
laradock Mysql. The server requested authentication method unknown to the client [caching_sha2_password]
alter user 'username'@'localhost' identified with mysql_native_password by 'password'; would fix it....
654
nvidia docker OpenCV Docker error "ImportError: libSM.so.6: cannot open shared object file: No such file or directory"
I fixed this problem on nvcr.io/nvidia/tensorflow:18.12-py3 with (using solution above): ...
457
compose Docker-compose up failing because "port is already allocated"
I ran into the same issue today (with a postgres container) and despite having tried docker-compose ...
447
moby The name "/data-container-name" is already used by container <hash>. You have to remove (or rename) that container to be able to reuse that name.
I have a helper function to nuke everything so that our Continuous blah cycle can be tested erm.. co...
371
compose Compose error "HTTP request took too long to complete"
By simply restarting the docker service via sudo service docker restart I was able to get the aforem...
369
compose error on launching docker-compose by piping to sh ( echo 'docker-compose ... ' | sh )
I could get it to work by adding the -T parameter to not create a Pseudo-TTY docker-compose exec -T ...
337
compose docker-compose up fails if network attached to container is removed
Thanks for the report! I think there are several things to note here: First and foremost ...
316
nvidia docker docker: Error response from daemon: Unknown runtime specified nvidia.
I've also installed correctly but forgot to restart daemon in ubuntu it may resolve your error. ...
297
compose Error when trying to run docker-compose up. "oci runtime error: container_linux.go:247..."
you gotta make the docker-entrypoint.sh an executable before building the image: otherwise it cant b...
292
laradock SQLSTATE[HY000] [2054] The server requested authentication method unknown to the client
+1 I'm having the same problem here. Info: Docker version ($ docker --version): Docker version 17.12...
257
compose docker-compose up doesn't pull down latest image if the image exists locally
Imagine that git didn't have pull because git fetch && git merge origin/master is functionally ident...
236
nvidia docker could not select device driver "" with capabilities: [[gpu]].
Hello! If you didn't already make sure you've installed the nvidia-container-toolkit If this doesn't...
205
moby docker-engine 1.10.2-0~trusty can't install on clean Ubuntu 64-bit 14.04.3
I seem to have resolved this by putting deb http://cz.archive.ubuntu.com/ubuntu trusty main in /etc/...
183
moby Docker service update --image "could not accessed on a registry to record its digest"
When updating services that need credentials to pull the image you need to pass --with-registry-auth...
178
laradock MySQL Container fails to start
I had the same issue last night I think it's the mysql version problem What I did was edited laradoc...
169
compose Docker Compose mounts named volumes as 'root' exclusively
Actually I come here with news it seems what I am trying to achieve is doable but I don't know if th...
149
compose INTERNAL ERROR: cannot create temporary directory!
Confirming this happened to me Today Was running low on space: After removing a container.. it works...
147
cookiecutter django No support for python3? I am getting: invalid syntax: raise ValueError, "No frame marked with %s." % fname
For me the issue was that I installed the environ package instead of the django-environ package. ...
147
compose docker-compose up -d doesn't expose ports when defined with build directive
oh you didn't specify but I'm assuming you're using run instead of up? If so you need --service-port...
142
compose How does compose chooses subnet for default network?
I'm also running into this issue Another way around this is to set the default-address-pools in your...
139
ddev In WSL2 ddev start fails at docker-credential-desktop.exe, "error listing credentials"
I had to set credsStore: in my ~/.docker/config.json .. it was previously set to credentials.exe ...
139
docker touch: cannot touch ‘/var/jenkins_home/copy_reference_file.log’: Permission denied
as mentioned there you need to figure out your volume mapping permissions ie. I have the same issue ...
138
kubernetes ingress 413 Request Entity Too Large
FYI the annotation has changed and is now: Also I had to restart the nginx pod for the effect to tak...
127
moby docker daemon unable to access registry - Client.Timeout exceeded while awaiting headers
I found out that the problem might be in /etc/resolv.conf I had: but moving the non-working (yet) 10...
120
cli How to skip one stage from multi-stage docker build
Docker 18.06 has been released Description We have Multistage docker build that creates rpm in each ...
119
compose ERROR: for db Cannot start service db: driver failed programming external connectivity on endpoint ltg_db_1
The following worked for me when i do the following : ± docker-compose up Starting ltg_db_1 ERROR: f...