Solvedcookiecutter How to update a project with changes from its cookiecutter?

Hi,

I maintain several personal cookiecutters, and I found that I spend a lot of time keeping projects in sync with updates on the cookiecutter.

I started trying a workflow to deal with this, as follows:

  1. Every project I render is assumed to be versioned by git. In the project repo, there is a branch template, that contains the result of cookiecutter rendering, without any other changes.
  2. Every cookiecutter renders a file .cookiecutter.json at the root of the project containing the original context used to render that project, in json format.
  3. Every project has a Makefile containing at least an update-template target. This make target does the following:
    • Create a temporary checkout (via git worktree) of the template branch.
    • Use the stored context in .cookiecutter.json to regenerate the project on top of the temporary checkout, overriding previous content.
    • Commit the newly rendered project in the template branch.

This way, the template branch tracks the changes from the cookiecutter, and I can use git-merges into master to update the current project.

The key thing is that by using git merges, I can update template files that got havily modified in the project. Most of the time the merge is clean, but from time to time a conflict will arise, which is easily resolved.

I've used it for a while and works pretty well. There are at least a couple of pain points though, that could benefit from changes in cookiecutter upstream, which may be of independent interest.

  1. The template I use to generate a .cookiecutter.json is a bit hacky. I'd like to do something like this:

    {{ cookiecutter | jsonify }}
    

    Would you accept distributing this jsonify as a default extension? I can provide the code in an other PR.

  2. In order to get updates from the cookiecutter, the project stores its url in a _template field on its context. Right now, I hard-code this url in the cookiecutter itself. I would love to free my cookiecutters from hard-coding their url, That could be achieved if cookiecutter itself would inject a _template field into the context with the url used at the time of rendering. I submitted #774 to that effect.

Additionally, maybe a functionality like the update-template I described above could be moved into cookiecutter upstream. That would require some thinking, I guess...

Of course, any suggestions, or other ways in which a workflow like this could be better supported by cookiecutter would be very welcome!

28 Answers

✔️Accepted Answer

Alternatively just use cruft https://cruft.github.io/cruft/

Other Answers:

I'd like to chime in with a solution I came up with:

cookiecutter REPO_URL --output-dir .. --config-file .cookiecutter.yaml --no-input --overwrite-if-exists
  • --output-dir .. allows us to run the cookiecutter in the repo root vs. the parent directory because cookiecutter will always render output in a child directory of the output-dir.
  • --config-file cookiecutter.yaml will run the cookiecutter with the original inputs which are persisted to .cookiecutter.yaml in the repo root (this is generated by the cookiecutter).
  • --no-input will skip the cookiecutter prompts.
  • --overwrite-if-exists will replace our previous cookiecutter output with the most up-to-date output.

Err, nobody's asking for a magic, do-what-I-mean solution. This is a three-way merge. You do the merges you can, and then flag the conflicts for human review. This general concept is already implemented in multiple other places, including Debian for config files in /etc, and more specifically here, other projects built on top of cookiecutter (like cruft: https://github.com/timothycrosley/cruft/). It's fine for the project to decide this is out of scope. But it's clearly possible, because this problem is being solved elsewhere.

Would have a need for this also. What's needed for this to move forward? Any opposition from the cookiecutter maintainers for integrating a solution to this into cookiecutter itself? Because at the moment, many people seem to be reinventing that wheel.

Assuming there's no opposition to the idea of adding this feature, do we still need to discuss the approach to take here, or would we be at the stage where a pull request would be the best way to move this forward? Thanks!

(One implementation of this feature is here: https://github.com/senseyeio/cupper)

Related Issues:

12
cookiecutter How to update a project with changes from its cookiecutter?
Alternatively just use cruft https://cruft.github.io/cruft/ Hi I maintain several personal cookiecut...
147
cookiecutter django No support for python3? I am getting: invalid syntax: raise ValueError, "No frame marked with %s." % fname
For me the issue was that I installed the environ package instead of the django-environ package. ...
22
cookiecutter django Invalid HTTP_HOST header: <ip>. You may need to add <ip> to ALLOWED_HOSTS.
@piyushmaurya23 if you do set ALLOWED_HOSTS = ['*'] for the love of God don't leave it as such once ...
13
cookiecutter data science Integration with dvc
@isms good questions! I've prepared a PR #159 to show how the first step would look like Would be gr...
11
cookiecutter django docker/celery exits because celery is run as root
It's only running locally where it's a problem In my dev.yml I added C_FORCE_ROOT to the environment...
926
keras Loading model with custom loss function: ValueError: 'Unknown loss function'
I solved this problem by adding 'custom_bojects' my loss function: I trained and saved a model that ...
731
scrapy ' error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 '
@euler16 for scrapy with Python 3 you'll need with Python 2 you'll need I wanted to install scrapy i...
633
keras Tensorflow backend - bug in model._make_predict_function(...)
I had this problem when doing inference in a different thread than where I loaded my model ...
575
mysqlclient Can't install mysqlclient on MacOS
You can set ssl library path explicitly. Hi I'm on MacOSX Sierra ...
516
keras Creating Model variable throws "AttributeError: 'Tensor' object has no attribute '_keras_history'"
@zaid478 oh I assumed you were doing it on coursera Anyway Hello all setup: Keras 2.0 Tensorflow 1.0...
456
tensorflow ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory
@rhinsall I just found this out myself not sure if it's common knowledge but got around this by doin...
423
ipython Last jedi release (0.18.0) is incompatible with ipython (7.19 and 7.18 tested); reason - column arg was deprecated, and now removed
As a temporary fix for anyone just trying to get things working again: It would be really nice if yo...
416
pyodbc pip install pyodbc failing: error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
src/pyodbc.h:56:17: fatal error: sql.h: No such file or directory #include <sql.h> ^ Your machine do...
395
keras "<tensor> is not an element of this graph." when loading model.
I had a problem similar to that of @piraka9011 which was solved by calling model._make_predict_funct...
392
psycopg2 psycopg2-binary fails to install on macOS Big Sur 11.0.1 and Python 3.9.0 (with possible workaround)
Actually figured it out Haven't worked on a Mac in a while and forgot about the PostgreSQL dependenc...
387
pytorch I can't import PyTorch, libomp.dylib can't be loaded.
brew install libomp solves the problem. 🐛 Bug I tried to install PyTorch but I can't use it ...
380
pytorch RuntimeError: cuda runtime error (59) : device-side assert triggered when running transfer_learning_tutorial
@soumith Hi I find the error it is due to my labels contain value -1 which is unacceptable in criter...
373
Zappa bad magic number in 'application': b'\x03\xf3\r\n': ImportError
I was able to fix this too by doing find -name \*.pyc -delete Mine definitely wasn't caused by PY2 -...
356
tensorflow At Runtime : "Error while reading resource variable softmax/kernel from Container: localhost"
I had the same issue in tensorflow 1.13.1 which I have resolved by creating a reference to the sessi...
355
tensorflow Mac + Python 3.6.1: Attempting to download mnist data results in CERTIFICATE_VERIFY_FAILED error
This might have also been because I recently switched to a new machine and it's still only partially...
335
pyenv WARNING: pyenv init - no longer sets PATH.
I saw the same message on an RH derivative Linux using bash 4.4.19 To resolve After running pyenv up...
333
pyenv unable to install python 3.8.0 on macox 11
Try to install brew reinstall zlib bzip2 and in the ~/.zshrc file add the following: in case you are...
331
psutil psutil fails to install on python3.6 and Ubuntu16.04
I was just about to comment the same after I read the install file: https://github.com/giampaolo/psu...
326
psycopg2 Installing psycopg2-binary with Python:3.6.4-alpine doesn't work
I found this solution which worked quite fine Edit The above compiles the package from source so I w...
325
pyenv Unable to build Python on macOS Big Sur with Xcode 12 beta
Thank you HomeBrew team for all the hard work you did to find the patch that python team is brewing ...
320
Pillow ImportError: cannot import name 'PILLOW_VERSION' from 'PIL' (unknown location)
I'm getting the same error right now after conda install pillow on Win 10 It installed pillow 6.2.1 ...
317
pip ImportError in system pip wrappers after an upgrade
I followed some instructions elsewhere and ran and hit upon this issue fixed it returning me to the ...
308
tensorflow UnrecognizedFlagError: Unknown command line flag 'f'
I solved this issue by adding the line: tf.app.flags.DEFINE_string('f' '' 'kernel') what this essent...
298
tensorflow FutureWarning: Deprecated numpy API calls in tf.python.framework.dtypes
@Akeaakar And for the time being pip install numpy<1.17 to revert to numpy version 1.16.4 ...
295
certbot Client with the currently selected authenticator does not support any combination of challenges that will satisfy the CA
Unfortunately My operating system is (include version): Ubuntu 16.04 I installed Certbot with (certb...
293
pyinstaller TypeError: an integer is required (got type bytes) when compiling any script
pip install https://github.com/pyinstaller/pyinstaller/archive/develop.tar.gz Direct install the lat...
287
readthedocs.org Sphinx error: master file [..]/checkouts/latest/contents.rst not found
Encountered the same problem here I'm using sphinx 2.0.1 I’m still new to readthedocs.org and have n...
277
discord.py Cannot connect to host discordapp.com certificate verify failed
I looked into this some more Apparently 3.6 changed the installation process on OS X ...
265
tensorflow ERROR: Cannot uninstall 'wrapt'. during upgrade
This worked for me: pip install wrapt --upgrade --ignore-installed pip install tensorflow ...
264
tqdm Jupyterlab and tqdm_notebook
I confirm this problem on my environment but it is upstream so IMHO nothing can be done on tqdm Plea...
259
virtualenv Error creating virtualenv with python3.6
The original poster's problem is due to not having the 'python3.6-venv' package installed ...
254
pip pip 18.0 install fails with AttributeError: 'NoneType' object has no attribute 'bytes'
try: python -m pip install -U --force-reinstall pip inside your venv Environment pip version: 10.0.1...
254
pytorch ModuleNotFoundError: No module named 'torch._C'
Solution the problem is that you have a folder called torch in the same directory which is being pic...
252
tensorflow Keras application - Tensor is not an element of this graph on eval after train
K.clear_session() did not work for me however what worked was : While predicting use the same graph ...
240
pip ImportError: cannot import name 'sysconfig' (Ubuntu 16.04)
I also saw a similar error However by installing python3-distutils it was possible to avoid this err...
234
pyenv Install failed, "zlib not available" on macOS Mojave
Solved it by running: sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_hea...
224
pyenv Python 3.6 cannot be installed on Mac OS 11.0.1
I solved this issue by the method provided by this blog Hi,Morning: First of all thank you very much...
204
keras Quick Question: can a model be fit for multiple times?
Yes successive calls to fit will incrementally train the model. I'm now having something around 5000...
202
ipython NameError: name 'sys' is not defined
Ok that was easy The Jedi release on 12/25 version 0.18.0 is breaking tab completion pip install --u...
202
pyodbc fatal error: sql.h: No such file or directory
You don't have the required ODBC header files on your machine For example on Ubuntu you would need t...
201
pip IOError: [Errno 13] Permission denied: '/usr/local/bin/pip'
You should add the --user option to install without needing root permission. Pip version:1.5.4 Pytho...
196
tensorflow InternalError: Blas GEMM launch failed
Make sure you have no other processes using the GPU running Run nvidia-smi to check this. ...
194
pytorch RuntimeError: DataLoader worker is killed by signal: Killed.
I've encountered the same problem recently If you're using the docker to run the PyTorch program wit...