Solvedcloud builders Fetch repositories with submodules

It is unclear how repositories with submodules can be built.

This is what the build currently does:

Initialized empty Git repository in /workspace/.git/
... <fetches from the single specified repository> ...

It is not clear if it is possible to use the cloud-builders/git container somehow to fetch all of the repositories. Is the workspace cleared between each buildstep? What if they are executed in parallel?

18 Answers

✔️Accepted Answer

You can use the source repositories of gcloud instead of github. Read my comment in the #44
I perform a step in my cloudbuild.yaml which rename the repositories :

- id: renameGitHubUrl
  name: debian
  - sed
  - -i
  - '\(.*\)\.git#\1#g'
  - .gitmodules

As soon as your private github repository are well named in gcloud source repositories, it will be correct. After this step you can submodule init and update.

Other Answers:


For anyone coming across this: An easier way than creating deploy keys, encrypting them with kms and adding multiple additional steps to your pipeline is to simply have a Google mirror the submodule as well. Then run two commands to temporarily change the submodule config on disk to point to the Google mirror before updating the submodule. Like so:

# Update submodules
- name: ''
  entrypoint: 'bash'
  - '-c'
  - |
    git config -f .git/config[project_id]/r/[repo_one]
    git config -f .gitmodules[project_id]/r/[repo_one]
    git submodule init
    git submodule update

Wondering if this is scheduled to be fixed anytime soon. @imjasonh

For what it's worth, pst's comment saved the day. THANKS!

That said, I must admit I had to poke around to make sense of it, so perhaps these more detailed steps will save the next person time...

To mirror the submodule, go to Cloud Source Repositories. That gave me a repo like:[my-project]/github_[my-sub-repo-name]

In order for Cloud Build to gain access, I needed to rewrite like:[my-project]/r/github_[my-sub-repo-name]

With that new url, I followed the example above... but I did not need to update .git/config, only .gitmodules.


- name: ''
  entrypoint: 'bash'
  - '-c'
  - |
    git config -f .gitmodules submodule.[my-sub-repo-name].url[my-project]/r/github_[my-sub-repo-name]
    git submodule init
    git submodule update

I added this to my cloudbuild.yaml as the first step. Cloud Build then had no issues cloning the submodule during the build.

Your mileage may vary with a different setup. I do hope Google adds proper support to do this automatically in the near future. What a pain.

Related Issues:

cloud builders Mounting files in /workspace to docker run steps
Ah ok; I think I've got it This is somewhat related to #345 Following the steps in this comment work...
cloud builders How do I run kubectl after image push?
Hey! Good news the up-scoping feature is now live That means that you can add permission for Contain...
cloud builders Fetch repositories with submodules
You can use the source repositories of gcloud instead of github It is unclear how repositories with ...
cloud builders Document how to connect to a database
Cloud Build uses cloudbuild Docker network so we can minimize use of raw docker run without Docker C...
cloud builders Google Cloud Build for Github does not fetch submodules
Try this (I use BitBucket but change the url appropriately): It works for me Thanks ...
bazel Bazel 0.8.1: Build fails with "Xcode version must be specified to use an Apple CROSSTOOL" on OSX High Sierra
FYI: I encountered similar issue recently This is similar to #3063 but not quite the same so I'm add...
bazel Ubuntu 16.04 PPA key expired
I can confirm that reinstalling the key resolves the issue; sudo apt-get update runs successfully Th...
msbuild Cannot find reference assemblies for .NET 3.5 or lower using core msbuild
I got this working by adding the following to the .csproj file: Unfortunately dotnet/sdk supports ta...
msbuild Add support for non-string assembly attributes
Sorry for being late to the party on this I know it's closed but I think it's worth considering reop...
msbuild "MSB3823: Non-string resources require the property GenerateResourceUsePreserializedResources to be set to true" if TargetFramework=net48
I did not quite understand We have a few projects that muti target .net core and the old fashioned ....
bazel Error while building tensorflow 0.11.0 - cache (directory not empty)
I attempted both solutions suggest by @sfincke and @yselivonchyk but without luck Finally ...
bazel Bazel installer stopped working after d3f8efc, if project has .bazelversion file
We would like to kindly request this issue to be reopened as it does appears to have broken both Baz...
bazel a way at start a repl
@damienmg I think this is pretty standard for any build tool whose language has a repl: Clojure: lei...
tensorflow windows wheel failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
I haven't seen this error before Try set CUDA_VISIBLE_DEVICES=0 Hi ...
bazel Compatibility with Xcode and Command Line Developer Tools on macOS
I've just hit this on a fresh Bazel clone (bc5a9b1): Environment: macOS 10.14.6 Xcode 10.3 I can con...
msbuild Setting BaseIntermediateOutputPath correctly in a SDK-based project is hard
To summarize if I want to override my intermediate and output folders NuGet restore drops the projec...
msbuild MSBuild ToolsVersions missing from hklm\software\Microsoft\MSBuild\ToolsVersions registry key
This a critical issue All previous versions of MSBuild left a registry entry in hklm\software\Micros...
bazel bazel-built protoc segfaults on mac os monterey
I have had success working around this with the command line option --incompatible_linkopts_to_linkl...
msbuild Warning AL1073 when .resx files is compiled under x64
To be clear for those hitting this the workaround should be very similar to the workaround in the OP...
pants 2.3.0rc0 - ProcessExecutionFailure: Process 'Extracting plugin locations' failed with exit code 2
@jsirois Thanks! Can confirm this works for me now I should also note that in ...
assemble yfm in partials doesn't work
Since you're using the built-in Handlebars syntax for the partial: {{> foo }} assemble does not merg...
bazel Make maven_jar and friends smarter by re-using previously fetched artifacts across different projects
0590483 now lets you use --experimental_repository_cache=$HOME/some/path to cache downloaded artifac...
laradock Mysql. The server requested authentication method unknown to the client [caching_sha2_password]
alter user 'username'@'localhost' identified with mysql_native_password by 'password'; would fix it....
nvidia docker OpenCV Docker error "ImportError: cannot open shared object file: No such file or directory"
I fixed this problem on with (using solution above): ...
compose Docker-compose up failing because "port is already allocated"
I ran into the same issue today (with a postgres container) and despite having tried docker-compose ...
moby The name "/data-container-name" is already used by container <hash>. You have to remove (or rename) that container to be able to reuse that name.
I have a helper function to nuke everything so that our Continuous blah cycle can be tested erm.. co...
compose Compose error "HTTP request took too long to complete"
By simply restarting the docker service via sudo service docker restart I was able to get the aforem...
compose error on launching docker-compose by piping to sh ( echo 'docker-compose ... ' | sh )
I could get it to work by adding the -T parameter to not create a Pseudo-TTY docker-compose exec -T ...
compose docker-compose up fails if network attached to container is removed
Thanks for the report! I think there are several things to note here: First and foremost ...
nvidia docker docker: Error response from daemon: Unknown runtime specified nvidia.
I've also installed correctly but forgot to restart daemon in ubuntu it may resolve your error. ...
compose Error when trying to run docker-compose up. "oci runtime error: container_linux.go:247..."
you gotta make the an executable before building the image: otherwise it cant b...
laradock SQLSTATE[HY000] [2054] The server requested authentication method unknown to the client
+1 I'm having the same problem here. Info: Docker version ($ docker --version): Docker version 17.12...
compose docker-compose up doesn't pull down latest image if the image exists locally
Imagine that git didn't have pull because git fetch && git merge origin/master is functionally ident...
nvidia docker could not select device driver "" with capabilities: [[gpu]].
Hello! If you didn't already make sure you've installed the nvidia-container-toolkit If this doesn't...
moby docker-engine 1.10.2-0~trusty can't install on clean Ubuntu 64-bit 14.04.3
I seem to have resolved this by putting deb trusty main in /etc/...
moby Docker service update --image "could not accessed on a registry to record its digest"
When updating services that need credentials to pull the image you need to pass --with-registry-auth...
laradock MySQL Container fails to start
I had the same issue last night I think it's the mysql version problem What I did was edited laradoc...
compose Docker Compose mounts named volumes as 'root' exclusively
Actually I come here with news it seems what I am trying to achieve is doable but I don't know if th...
compose INTERNAL ERROR: cannot create temporary directory!
Confirming this happened to me Today Was running low on space: After removing a container.. it works...
cookiecutter django No support for python3? I am getting: invalid syntax: raise ValueError, "No frame marked with %s." % fname
For me the issue was that I installed the environ package instead of the django-environ package. ...
compose docker-compose up -d doesn't expose ports when defined with build directive
oh you didn't specify but I'm assuming you're using run instead of up? If so you need --service-port...
compose How does compose chooses subnet for default network?
I'm also running into this issue Another way around this is to set the default-address-pools in your...
ddev In WSL2 ddev start fails at docker-credential-desktop.exe, "error listing credentials"
I had to set credsStore: in my ~/.docker/config.json .. it was previously set to credentials.exe ...
docker touch: cannot touch ‘/var/jenkins_home/copy_reference_file.log’: Permission denied
as mentioned there you need to figure out your volume mapping permissions ie. I have the same issue ...
kubernetes ingress 413 Request Entity Too Large
FYI the annotation has changed and is now: Also I had to restart the nginx pod for the effect to tak...
moby docker daemon unable to access registry - Client.Timeout exceeded while awaiting headers
I found out that the problem might be in /etc/resolv.conf I had: but moving the non-working (yet) 10...
cli How to skip one stage from multi-stage docker build
Docker 18.06 has been released Description We have Multistage docker build that creates rpm in each ...
compose ERROR: for db Cannot start service db: driver failed programming external connectivity on endpoint ltg_db_1
The following worked for me when i do the following : ± docker-compose up Starting ltg_db_1 ERROR: f...