Solvedcookiecutter How to update a project with changes from its cookiecutter?


I maintain several personal cookiecutters, and I found that I spend a lot of time keeping projects in sync with updates on the cookiecutter.

I started trying a workflow to deal with this, as follows:

  1. Every project I render is assumed to be versioned by git. In the project repo, there is a branch template, that contains the result of cookiecutter rendering, without any other changes.
  2. Every cookiecutter renders a file .cookiecutter.json at the root of the project containing the original context used to render that project, in json format.
  3. Every project has a Makefile containing at least an update-template target. This make target does the following:
    • Create a temporary checkout (via git worktree) of the template branch.
    • Use the stored context in .cookiecutter.json to regenerate the project on top of the temporary checkout, overriding previous content.
    • Commit the newly rendered project in the template branch.

This way, the template branch tracks the changes from the cookiecutter, and I can use git-merges into master to update the current project.

The key thing is that by using git merges, I can update template files that got havily modified in the project. Most of the time the merge is clean, but from time to time a conflict will arise, which is easily resolved.

I've used it for a while and works pretty well. There are at least a couple of pain points though, that could benefit from changes in cookiecutter upstream, which may be of independent interest.

  1. The template I use to generate a .cookiecutter.json is a bit hacky. I'd like to do something like this:

    {{ cookiecutter | jsonify }}

    Would you accept distributing this jsonify as a default extension? I can provide the code in an other PR.

  2. In order to get updates from the cookiecutter, the project stores its url in a _template field on its context. Right now, I hard-code this url in the cookiecutter itself. I would love to free my cookiecutters from hard-coding their url, That could be achieved if cookiecutter itself would inject a _template field into the context with the url used at the time of rendering. I submitted #774 to that effect.

Additionally, maybe a functionality like the update-template I described above could be moved into cookiecutter upstream. That would require some thinking, I guess...

Of course, any suggestions, or other ways in which a workflow like this could be better supported by cookiecutter would be very welcome!

28 Answers

✔️Accepted Answer

Alternatively just use cruft

Other Answers:

I'd like to chime in with a solution I came up with:

cookiecutter REPO_URL --output-dir .. --config-file .cookiecutter.yaml --no-input --overwrite-if-exists
  • --output-dir .. allows us to run the cookiecutter in the repo root vs. the parent directory because cookiecutter will always render output in a child directory of the output-dir.
  • --config-file cookiecutter.yaml will run the cookiecutter with the original inputs which are persisted to .cookiecutter.yaml in the repo root (this is generated by the cookiecutter).
  • --no-input will skip the cookiecutter prompts.
  • --overwrite-if-exists will replace our previous cookiecutter output with the most up-to-date output.

Err, nobody's asking for a magic, do-what-I-mean solution. This is a three-way merge. You do the merges you can, and then flag the conflicts for human review. This general concept is already implemented in multiple other places, including Debian for config files in /etc, and more specifically here, other projects built on top of cookiecutter (like cruft: It's fine for the project to decide this is out of scope. But it's clearly possible, because this problem is being solved elsewhere.

Would have a need for this also. What's needed for this to move forward? Any opposition from the cookiecutter maintainers for integrating a solution to this into cookiecutter itself? Because at the moment, many people seem to be reinventing that wheel.

Assuming there's no opposition to the idea of adding this feature, do we still need to discuss the approach to take here, or would we be at the stage where a pull request would be the best way to move this forward? Thanks!

(One implementation of this feature is here:

Related Issues:

cookiecutter How to update a project with changes from its cookiecutter?
Alternatively just use cruft Hi I maintain several personal cookiecut...
cookiecutter django No support for python3? I am getting: invalid syntax: raise ValueError, "No frame marked with %s." % fname
For me the issue was that I installed the environ package instead of the django-environ package. ...
cookiecutter django Invalid HTTP_HOST header: <ip>. You may need to add <ip> to ALLOWED_HOSTS.
@piyushmaurya23 if you do set ALLOWED_HOSTS = ['*'] for the love of God don't leave it as such once ...
cookiecutter data science Integration with dvc
@isms good questions! I've prepared a PR #159 to show how the first step would look like Would be gr...
cookiecutter django docker/celery exits because celery is run as root
It's only running locally where it's a problem In my dev.yml I added C_FORCE_ROOT to the environment...
keras Loading model with custom loss function: ValueError: 'Unknown loss function'
I solved this problem by adding 'custom_bojects' my loss function: I trained and saved a model that ...
scrapy ' error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 '
@euler16 for scrapy with Python 3 you'll need with Python 2 you'll need I wanted to install scrapy i...
keras Tensorflow backend - bug in model._make_predict_function(...)
I had this problem when doing inference in a different thread than where I loaded my model ...
mysqlclient Can't install mysqlclient on MacOS
You can set ssl library path explicitly. Hi I'm on MacOSX Sierra ...
keras Creating Model variable throws "AttributeError: 'Tensor' object has no attribute '_keras_history'"
@zaid478 oh I assumed you were doing it on coursera Anyway Hello all setup: Keras 2.0 Tensorflow 1.0...
tensorflow ImportError: cannot open shared object file: No such file or directory
@rhinsall I just found this out myself not sure if it's common knowledge but got around this by doin...
ipython Last jedi release (0.18.0) is incompatible with ipython (7.19 and 7.18 tested); reason - column arg was deprecated, and now removed
As a temporary fix for anyone just trying to get things working again: It would be really nice if yo...
pyodbc pip install pyodbc failing: error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
src/pyodbc.h:56:17: fatal error: sql.h: No such file or directory #include <sql.h> ^ Your machine do...
keras "<tensor> is not an element of this graph." when loading model.
I had a problem similar to that of @piraka9011 which was solved by calling model._make_predict_funct...
psycopg2 psycopg2-binary fails to install on macOS Big Sur 11.0.1 and Python 3.9.0 (with possible workaround)
Actually figured it out Haven't worked on a Mac in a while and forgot about the PostgreSQL dependenc...
pytorch I can't import PyTorch, libomp.dylib can't be loaded.
brew install libomp solves the problem. 🐛 Bug I tried to install PyTorch but I can't use it ...
pytorch RuntimeError: cuda runtime error (59) : device-side assert triggered when running transfer_learning_tutorial
@soumith Hi I find the error it is due to my labels contain value -1 which is unacceptable in criter...
Zappa bad magic number in 'application': b'\x03\xf3\r\n': ImportError
I was able to fix this too by doing find -name \*.pyc -delete Mine definitely wasn't caused by PY2 -...
tensorflow At Runtime : "Error while reading resource variable softmax/kernel from Container: localhost"
I had the same issue in tensorflow 1.13.1 which I have resolved by creating a reference to the sessi...
tensorflow Mac + Python 3.6.1: Attempting to download mnist data results in CERTIFICATE_VERIFY_FAILED error
This might have also been because I recently switched to a new machine and it's still only partially...
pyenv WARNING: pyenv init - no longer sets PATH.
I saw the same message on an RH derivative Linux using bash 4.4.19 To resolve After running pyenv up...
pyenv unable to install python 3.8.0 on macox 11
Try to install brew reinstall zlib bzip2 and in the ~/.zshrc file add the following: in case you are...
psutil psutil fails to install on python3.6 and Ubuntu16.04
I was just about to comment the same after I read the install file:
psycopg2 Installing psycopg2-binary with Python:3.6.4-alpine doesn't work
I found this solution which worked quite fine Edit The above compiles the package from source so I w...
pyenv Unable to build Python on macOS Big Sur with Xcode 12 beta
Thank you HomeBrew team for all the hard work you did to find the patch that python team is brewing ...
Pillow ImportError: cannot import name 'PILLOW_VERSION' from 'PIL' (unknown location)
I'm getting the same error right now after conda install pillow on Win 10 It installed pillow 6.2.1 ...
pip ImportError in system pip wrappers after an upgrade
I followed some instructions elsewhere and ran and hit upon this issue fixed it returning me to the ...
tensorflow UnrecognizedFlagError: Unknown command line flag 'f'
I solved this issue by adding the line:'f' '' 'kernel') what this essent...
tensorflow FutureWarning: Deprecated numpy API calls in tf.python.framework.dtypes
@Akeaakar And for the time being pip install numpy<1.17 to revert to numpy version 1.16.4 ...
certbot Client with the currently selected authenticator does not support any combination of challenges that will satisfy the CA
Unfortunately My operating system is (include version): Ubuntu 16.04 I installed Certbot with (certb...
pyinstaller TypeError: an integer is required (got type bytes) when compiling any script
pip install Direct install the lat...
287 Sphinx error: master file [..]/checkouts/latest/contents.rst not found
Encountered the same problem here I'm using sphinx 2.0.1 I’m still new to and have n...
277 Cannot connect to host certificate verify failed
I looked into this some more Apparently 3.6 changed the installation process on OS X ...
tensorflow ERROR: Cannot uninstall 'wrapt'. during upgrade
This worked for me: pip install wrapt --upgrade --ignore-installed pip install tensorflow ...
tqdm Jupyterlab and tqdm_notebook
I confirm this problem on my environment but it is upstream so IMHO nothing can be done on tqdm Plea...
virtualenv Error creating virtualenv with python3.6
The original poster's problem is due to not having the 'python3.6-venv' package installed ...
pip pip 18.0 install fails with AttributeError: 'NoneType' object has no attribute 'bytes'
try: python -m pip install -U --force-reinstall pip inside your venv Environment pip version: 10.0.1...
pytorch ModuleNotFoundError: No module named 'torch._C'
Solution the problem is that you have a folder called torch in the same directory which is being pic...
tensorflow Keras application - Tensor is not an element of this graph on eval after train
K.clear_session() did not work for me however what worked was : While predicting use the same graph ...
pip ImportError: cannot import name 'sysconfig' (Ubuntu 16.04)
I also saw a similar error However by installing python3-distutils it was possible to avoid this err...
pyenv Install failed, "zlib not available" on macOS Mojave
Solved it by running: sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_hea...
pyenv Python 3.6 cannot be installed on Mac OS 11.0.1
I solved this issue by the method provided by this blog Hi,Morning: First of all thank you very much...
keras Quick Question: can a model be fit for multiple times?
Yes successive calls to fit will incrementally train the model. I'm now having something around 5000...
ipython NameError: name 'sys' is not defined
Ok that was easy The Jedi release on 12/25 version 0.18.0 is breaking tab completion pip install --u...
pyodbc fatal error: sql.h: No such file or directory
You don't have the required ODBC header files on your machine For example on Ubuntu you would need t...
pip IOError: [Errno 13] Permission denied: '/usr/local/bin/pip'
You should add the --user option to install without needing root permission. Pip version:1.5.4 Pytho...
tensorflow InternalError: Blas GEMM launch failed
Make sure you have no other processes using the GPU running Run nvidia-smi to check this. ...
pytorch RuntimeError: DataLoader worker is killed by signal: Killed.
I've encountered the same problem recently If you're using the docker to run the PyTorch program wit...