Solvedjulia Precompile everything after Pkg.update() and friends

This issue is meant as a discussion starter, I'm actually not sure it should actually be done ;)

Here is my situation: I normally Pkg.update() in the morning. I know it will take some time, so I typically do it when I don't need a julia prompt the next second. Later that day someone comes by the office, and I want to show him/her how cool and fast julia is. I start my neat example code, and it all starts by spending a couple of minutes precompiling things. And I stutter "uh, oh, precompile, you know...". It bites me every time I try to demo julia.

At least for me I would much prefer if precompile happend with Pkg.update(). I'm already in a mood that things will take a while when I run Pkg.update(), so if they take a little longer, no problem. Whereas precompile on demand seems to hit me always in moments when it is really, really inconvenient. Having precompile as part of Pkg.update() would make things much more predictable, which is something I would value.

For this example, precompile should probably also be triggered by all the other Pkg.* functions that change packages.

Alternatives:

  • a flag like Pkg.update(precompile=true). Would work but I would probably forget it, and then the precompile cost would again hit me in an inconvenient moment.
  • a global configuration flag. Could maybe go into the user julia startup script or something like that.

Thoughts?

51 Answers

✔️Accepted Answer

Hi, sorry for the bump, but I think this warrants further discussion. I would consider myself a beginner, and my use-case (and I think a lot of people in the sciences like myself) want to do something like use Julia in a REPL or Jupyter Notebook to do some nice data exploration type tasks (like how people use Matlab). Opening up Jupyter and then running using SymPy, DataFrames, Plots and having to wait five minutes actually makes me drop Julia and just open up Python instead. I like the features of Julia for data work, but I'll just use numpy instead if it means I can actually get work done. I have the same problem as other people in this thread, I really like Julia and I want to show my colleagues the cool stuff that makes my work easier but opening it up and going "no wait, it just does this for a few minutes and then it's fast" leaves a really bad impression.

I think that having Pkg.update() precompile (including dependents) by default is the best option. In my experience, non-programmers don't often update, and are probably fine with it taking longer if it means they can do work faster after. Updates usually take a long time (look at R or MATLAB) anyway, so I really don't think Pkg.update() being slow is an issue. As other people have brought up, doing the work in parallel would be a good idea.

Other Answers:

Pkg.update_and_compile() ?

Pkg.update(precompile=true) seems better to me. I also think precompilation as default seems aggressive.

@davidanthoff, in the meantime you can always add this to your .juliarc.jl:

function recompile()
    for pkg in Pkg.available()
        try
            pkgsym = Symbol(pkg)
            eval(:(using $pkgsym))
        catch
        end
    end
end

pkgupdate() = (Pkg.update(); recompile())

Not perfect, but pretty useful.

This would also be helpful for MPI programs on clusters with a shared file system. It would be better to have a single process precompile everything before launching the MPI tasks. Currently, every MPI task precompiles every package.

What's the problem with precompiling packages on install by default? According to the "vote" at the top, this makes a lot of sense.

The issue with leaving it opt-in is that only advanced users will find out about this, which will give the impression that Julia is slow any time they will show it to their colleagues after an update.

More Issues: