SolvedImageMagick Memory issues

Wasn't sure whether to open 3 issues for these.

I want to convert 119 700x950px jpg images each 81K large into a gif, so in total 9.5M. But when I try to do that I get

convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `000081.jpg' @ error/cache.c/OpenPixelCache/3945.
convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `000082.jpg' @ error/cache.c/OpenPixelCache/3945.
...
convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `000118.jpg' @ error/cache.c/OpenPixelCache/3945.
convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `000119.jpg' @ error/cache.c/OpenPixelCache/3945.

Meaning it fails after the 80th image for some reason.
So in my opinion a first performance bug is already, that ImageMagick seems to cache all images first instead of just streaming them. Also why does this exceed my memory limit in the first place. It seems to cache them uncompressed.

Looking at the limits I get:

identify -list resource
    Resource limits:
      Width: 16KP
      Height: 16KP
      Area: 128MP
      Memory: 256MiB
      Map: 512MiB
      Disk: 1GiB
      File: 768
      Thread: 4
      Throttle: 0
      Time: unlimited

So another problem I have is that for some reason memory is limited to 256MiB, but I have 16GiB in total and 10GiB free:

free -m
                  total        used        free      shared  buff/cache   available
    Mem:          15935        5265        2154         122        8515       10418
    Swap:             0           0           0

Also I have a bit more disk space

df -h
    Filesystem              Size  Used Avail Use% Mounted on
    /dev/sdc1                95G   78G   13G  87% /

So I tried setting it higher with an old code snippet I found here

identify -limit memory 8000mb -list resource
    Resource limits:
      Width: 16KP
      Height: 16KP
      Area: 128MP
      Memory: 7.81KiB
      Map: 512MiB
      Disk: 1GiB
      File: 768
      Thread: 4
      Throttle: 0
      Time: unlimited

So instead of setting it to 8000 MB like I intended it sets it to 8000 Byte. In my opinion the unit specifier shouldn't be case-sensitive. And even if you want to enforce a case-sensitive CLI, there should be a warning or even error message if the unit specifier could not be recognized!

It works in uppercase:

identify -limit memory 80MB -list resource
    Resource limits:
      Memory: 76.29MiB

BUT trying to set it higher than the default 256MiB does not work for some weird reason which is the most pressing bug for me:

identify -limit memory 8000MiB -list resource
    Resource limits:
      Memory: 256MiB
20 Answers

✔️Accepted Answer

For anyone who is still facing this problem even after increasing the memory limit, try commenting out the limit tags for policymap. Eg:

<policymap>
  <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
  <!-- <policy domain="resource" name="memory" value="2GiB"/> -->
  <!-- <policy domain="resource" name="map" value="4GiB"/> -->
  <!-- <policy domain="resource" name="area" value="1GB"/> -->
  <!-- <policy domain="resource" name="disk" value="16EB"/> -->
  <!-- <policy domain="resource" name="file" value="768"/> -->
  <!-- <policy domain="resource" name="thread" value="4"/> -->
  <!-- <policy domain="resource" name="throttle" value="0"/> -->
  <!-- <policy domain="resource" name="time" value="3600"/> -->
  <!-- <policy domain="system" name="precision" value="6"/> -->
  <policy domain="cache" name="shared-secret" value="passphrase"/>
  <policy domain="coder" rights="none" pattern="EPHEMERAL" />
  <policy domain="coder" rights="none" pattern="URL" />
  <policy domain="coder" rights="none" pattern="HTTPS" />
  <policy domain="coder" rights="none" pattern="MVG" />
  <policy domain="coder" rights="none" pattern="MSL" />
  <policy domain="coder" rights="none" pattern="TEXT" />
  <policy domain="coder" rights="none" pattern="SHOW" />
  <policy domain="coder" rights="none" pattern="WIN" />
  <policy domain="coder" rights="none" pattern="PLT" />
  <policy domain="path" rights="none" pattern="@*" />
</policymap>

Other Answers:

Thank you, increasing the memory limit in /etc/ImageMagic-6/policy.xml works for me.

Bzzz
213

HOW TO POLITELY ASK QUESTIONS ON THE INTERWEBS?

For some reason, my ImageMagick came with very low limits:

Resource limits:
  Width: 16KP
  Height: 16KP
  Area: 128MP
  Memory: 256MiB
  Map: 512MiB
  Disk: 1GiB
  File: 768
  Thread: 4
  Throttle: 0
  Time: unlimited

I changed /etc/ImageMagick-6/policy.xml to more sensible defaults:

Resource limits:
  Width: 128KP
  Height: 128KP
  Area: 1.0737GP
  Memory: 2GiB
  Map: 4GiB
  Disk: 8GiB
  File: 768
  Thread: 4
  Throttle: 0
  Time: unlimited

and it doesn't choke anymore with the new limits when I try to manipulate more than a few dozen scanned pages.

For anyone who is still facing this problem even after increasing the memory limit, try commenting out the limit tags for policymap. Eg:

<policymap>
  <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
  <!-- <policy domain="resource" name="memory" value="2GiB"/> -->
  <!-- <policy domain="resource" name="map" value="4GiB"/> -->
  <!-- <policy domain="resource" name="area" value="1GB"/> -->
  <!-- <policy domain="resource" name="disk" value="16EB"/> -->
  <!-- <policy domain="resource" name="file" value="768"/> -->
  <!-- <policy domain="resource" name="thread" value="4"/> -->
  <!-- <policy domain="resource" name="throttle" value="0"/> -->
  <!-- <policy domain="resource" name="time" value="3600"/> -->
  <!-- <policy domain="system" name="precision" value="6"/> -->

Remove all of these would be a bad idea.

The main problem is the area config, which means "Use up to _ disk space before failure"

(memory means "Use up to _ of memory before doing mmap" and map means "Use up to _ mmaps before caching to disk")

More Issues: