Solvednixpkgs ZERO Hydra Failures 20.09

UPDATE after initial 20.09 Release:
Fixing broken packages will always be possible throughout the lifetime of the 20.09 release. However, you may need to remove the broken = true; attr on the package. Otherwise please follow normal back-port conventions. :)

Old Post:


Every time we branch off a release we stabilize the release branch.
Our goal here is to get as little as possible jobs failing on the release-20.09 jobsets.
I'd like to heighten, while it's great to focus on zero as our goal, it's essentially to
have all deliverables that worked in the previous release work here also.

How many failing jobs are there?

At the opening of this issue we have the main x86_64-linux jobset at 1153 failing jobs, x86_64-darwin at >7130, and aarch64-linux at 7573+.

Previous releases first evals

19.09 had 1654 failing jobs.
20.03 had 1204 failing jobs,
20.09 had 1153 failing jobs,
So we're actually getting better at maintaining a more stable "unstable" channel.

However, our darwin story isn't as good (we need more darwin reviewers/contributors)
20.03 had 1384 failing jobs,
20.09 had >7130 failing jobs,

How to help (textual)

  1. Select an evaluation of the release-20.09 jobset by #id
    Screenshot from 2020-02-08 15 20 41

  2. Find a failed job ❌️
    Screenshot from 2020-02-08 15 26 47

  3. Troubleshoot why it's failing and fix it

  4. Create a Pull Request with the fix targeting master, wait for it to be merged.
    Generally the job fails on master also, you can verify that on Hydra - example URL:
    That means most PR's should be target the master branch, however, if your PR causes around 500+ rebuilds, it's preferred to target staging to avoid compute and storage churn.

Always reference this issue in the body of your PR:

ZHF: #97479
  1. After the master PR was merged, select the commit(s) from the master branch and cherry-pick the commit(s) into a backport PR. See for exact details on how that should be performed. An example can be found here: Master PR and 20.09 PR

Please ping @NixOS/nixos-release-managers on the PR.
If you're unable to because you're not a member of the NixOS org please ping
@jonringer and @worldofpeace (the same people in the team).

How to help video tutorial

@jonringer has made a video on YouTube to guide anyone through how fixing something for ZHF will look like:

New to nixpkgs?

@jonringer created some videos to help get people started with nixpkgs:

Also be sure to check out other resources at:

Packages that don't get fixed

The remaining packages will be marked as broken before the release (on the failing platforms).
You can do this like:

meta = {
  # ref to issue/explanation
  # `true` is for everything
  broken = stdenv.isDarwin 

These are the utility flags used to test the type of platform.


This is a great way to help NixOS, and it was some of my earliest contributions.
Let's go ✌️

✨️ @worldofpeace and @jonringer

cc @NixOS/nixpkgs-committers @NixOS/nixpkgs-maintainers

Related Issues

18 Answers

✔️Accepted Answer

We have our first 20.09 stable evaluation completed: Thus this should be closed.

Fixing broken packages will always be possible throughout the lifetime of the 20.09 release. However, you may need to remove the broken = true; attr on the package.

Other Answers:

Since last time this was useful to many, here's one way to build all packages which have you as maintainer:

  • apply #97514 Pull from master or release-20.09 so you have the latest build.nix mentioned below
  • run nix-build maintainers/scripts/build.nix --argstr maintainer fgaz (adjust the maintainer name to yourself)

Also, people can use @samueldr 's nix-review-tools to create a report which will show which packages are causing the most failures.

I usually do something like:

mkdir cache
cd cache
../eval-report $evaluation_id > index.html
xdg-open index.html

It can also be useful to refer to, a master evaluation taken around the time of the branchoff, which will have a much longer history you can use to dig back to a package's last successful build and get an indication of the changes which might have caused it to break.

More Issues: