Solvedvirtual environments Ubuntu images are running out of disk space

Describe the bug
Jobs running in Ubuntu-16.04 images are starting to throw error related with low disk space

Area for Triage: Servers

Question, Bug, or Feature?: Bug

Virtual environments affected

  • macOS 10.15
  • Ubuntu 16.04 LTS
  • Ubuntu 18.04 LTS
  • Windows Server 2016 R2
  • Windows Server 2019

Expected behavior
The disk should have enough free space for custom user software needed for building.

Actual behavior

  1. We can take as example this build: https://dev.azure.com/jneira/haskell-ide-engine/_build/results?buildId=758
  2. At start, after only checkout the project, disk usage(du -h) is:
Filesystem      Size  Used Avail Use% Mounted on
udev            3.4G     0  3.4G   0% /dev
tmpfs           695M  8.9M  686M   2% /run
/dev/sda1        84G   72G   12G  86% /
tmpfs           3.4G  8.0K  3.4G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.4G     0  3.4G   0% /sys/fs/cgroup
/dev/loop0       40M   40M     0 100% /snap/hub/43
/dev/loop1       94M   94M     0 100% /snap/core/8935
/dev/sda15      105M  3.6M  101M   4% /boot/efi
/dev/sdb1        14G   35M   13G   1% /mnt

Just before the error due to not enough disk space:

Filesystem      Size  Used Avail Use% Mounted on
udev            3.4G     0  3.4G   0% /dev
tmpfs           695M  8.9M  686M   2% /run
/dev/sda1        84G   80G  3.6G  96% /
tmpfs           3.4G  8.0K  3.4G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.4G     0  3.4G   0% /sys/fs/cgroup
/dev/loop0       40M   40M     0 100% /snap/hub/43
/dev/loop1       94M   94M     0 100% /snap/core/8935
/dev/sda15      105M  3.6M  101M   4% /boot/efi
/dev/sdb1        14G   35M   13G   1% /mnt

The concrete error is:

Unpacking GHC into /home/vsts/.stack/programs/x86_64-linux/ghc-8.8.1.temp/ ...
Configuring GHC ...
Installing GHC ...
/home/vsts/.stack/programs/x86_64-linux/ghc-8.8.1/lib/ghc-8.8.1/Cabal-3.0.0.0/.copyFile74954-1209.tmp: copyFile: resource exhausted (No space left on device)
make[1]: *** [install_packages] Error 1
make: *** [install] Error 2
Received ExitFailure 2 when running
Raw command: /usr/bin/make install
Run from: /home/vsts/.stack/programs/x86_64-linux/ghc-8.8.1.temp/ghc-8.8.1/

But it is not important, whatever command that needs disk space will fail

17 Answers

✔️Accepted Answer

Hello @jneira , it is expected behavior. Based on official documentation, hosted agents provide at least 10 GB of storage for your source and build outputs.
At the begin of your build, there are 12 GB of free space.

As a possible workarounds, you can remove the part of pre-installed software in runtime. For example, this command will release 5+ of free space:

sudo rm -rf "/usr/local/share/boost"
sudo rm -rf "$AGENT_TOOLSDIRECTORY"

Other Answers:

@matthewfeickert fwiw, I would also view it as a bug - because you're not exceeding the space we make available, and because things all of a sudden changed and broke you. We try hard for this not to happen but it happens sometimes. We'll figure out a solution as soon as possible (probably will take a day or two to replicate everywhere) so you can remove the workaround.

So I'm pretty sure this is an actual new bug. I started running into it on Ubuntu 18.04 just a few days ago. If you look a the output of df, you will notice that the build isn't using any disk space at all on /mnt, which is where most of the disk space is available. I am guessing that /home/runner/work used to be located on /mnt, but now is being left on /, and so we have gone from having 14GB for source + build outputs and a few gigabytes for extra software we install to having only a few gigabytes total for the entire build operation. I can manually move my builds to try to use space on /mnt, but it seems like this really did change this week, I guess first for Ubuntu 16.04 and soon thereafter for Ubuntu 18.04, and that it would be better if the runner was automatically placed on /mnt. (I wish I could re-open this bug, but sadly I can't; if no one sees this comment I guess I'll open a new bug and reference this one? @maxim-lobanov)

@alepauly, Does it make sense to extend disk space on Ubuntu 16.04/18.04 from 84gb to at least 128gb?

Ubuntu 16.04:
/dev/sda1 84G 75G 8.8G 90% /

Ubuntu 18.04:
/dev/sda1 84G 75G 8.3G 91% /

@maxim-lobanov thanks a lot for posting this workaround.
my tests have started to fail as well because of this issue. In my case, I'm running the tests in a docker container. If anybody else is in the same situation, there is a target: host flag to run things on the host, instead of the container.

EDIT: this is the change: apache/flink@cb8ae28
This is my "clean up script": https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh