Julia Environment
This notebook describes and creates the default Julia environments in Nextjournal. It is based on the Bash environment. Check out the showcase if you want to see what the environment contains. To see how it’s built, see setup.
Showcase
These packages are included in Nextjournal's Julia 1.4 environment.
]st
System Packages and Basics
A wide variety of support libraries are installed, as well as gcc v7 and ImageMagick.
Version
Julia packages are installed normally, using Pkg in a Julia cell. Please refer to the Julia section of Installing Software and Packages for more detailed information.
Plotting
The default environment comes with GR v
Plots
The JuliaPlots framework provides a unified interface to multiple graphical backends. The default backend is GR, which efficiently generates static plots and animations.
using Plots; gr()
# define the Lorenz attractor
mutable struct Lorenz
dt; σ; ρ; β; x; y; z
end
function step!(l::Lorenz)
dx = l.σ*(l.y - l.x) ; l.x += l.dt * dx
dy = l.x*(l.ρ - l.z) - l.y ; l.y += l.dt * dy
dz = l.x*l.y - l.β*l.z ; l.z += l.dt * dz
end
attractor =
Lorenz((dt = 0.02, σ = 10., ρ = 28., β = 8//3, x = 1., y = 1., z = 1.)...)
# initialize a 3D plot with 1 empty series
plt = plot3d(1, xlim=(-25,25), ylim=(-25,25), zlim=(0,50),
title = "Lorenz Attractor", marker = 2)
# build an animated gif by pushing new points to the plot, saving every 10th frame
for i=1:1500
step!(attractor)
push!(plt, attractor.x, attractor.y, attractor.z)
end every 10
Switching to the Plotly backend adds some interactivity to the output.
plotly(lw=3)
x = -100:100
Plots.plot(x, 100x.^2)
Plots.plot!(x, x.^3 - x.^2)
Makie
The WGLMakie package can generate beautiful 3D visualizations with WebGL.
using WGLMakie, AbstractPlotting, LinearAlgebra
Attributes(font = "Open Sans", resolution = (600, 500))
n = 20
set_theme!(resolution = (600, 500))
f = (x,y,z) -> x*exp(cos(y)*z)
∇f = (x,y,z) -> Point3f0(exp(cos(y)*z), -sin(y)*z*x*exp(cos(y)*z), x*cos(y)*exp(cos(y)*z))
∇ˢf = (x,y,z) -> ∇f(x,y,z) - Point3f0(x,y,z)*dot(Point3f0(x,y,z), ∇f(x,y,z))
θ = [0;(0.5:n-0.5)/n;1]
φ = [(0:2n-2)*2/(2n-1);2]
x = [cospi(φ)*sinpi(θ) for θ in θ, φ in φ]
y = [sinpi(φ)*sinpi(θ) for θ in θ, φ in φ]
z = [cospi(θ) for θ in θ, φ in φ]
pts = vec(Point3f0.(x, y, z))
∇ˢF = vec(∇ˢf.(x, y, z)) .* 0.1f0
AbstractPlotting.surface(x, y, z)
AbstractPlotting.arrows!(pts, ∇ˢF, arrowsize = 0.03, linecolor = (:white, 0.6), linewidth = 3)
VegaLite
VegaLite can also provide interactive plotting.
using VegaLite, VegaDatasets
vega_plot = dataset("cars") |>
(
:point,
x = :Horsepower,
y = :Miles_per_Gallon,
color = :Origin,
width = 650,
height = 400
) |> VegaLite.interactive()
Data Structures
Several data-related packages are installed in the default environment.
For handling their relative datatypes and I/O for associated files, we install a variety of packages, including JLD v
"0.10.0", CSV v"0.7.7", HDF5 v"0.13.6", and JSON v"0.21.1".The FileIO v
"1.4.4"framework provides a unified interface for loading files of many types via multiple backends, including ImageMagick.DataFrames v
"0.10.0"defines and handles objects similar to those found in R and the Python pandas toolkit, with a comparable interface.
File Handling
The HDF5 package provides a Julia interface to the HDF5 library. It is also used by the JLD and MAT packages. Let's look at some example temperature data.
using HDF5
# Some methods to traverse the first path in an h5 file.
dd(node::HDF5File) = dd(node[names(node)[1]])
dd(node::HDF5Group) = dd(node[names(node)[1]])
dd(node::HDF5Dataset) = node
dspath = h5open(NEONDSTowerTemperatureData.hdf5) do h5
name(dd(h5))
end
print("First path: $dspath.")
data = h5read(NEONDSTowerTemperatureData.hdf5, dspath)
using Plots
Plots.plot([x.date for x in data],[y.mean for y in data],
label="Temperature (Min/Max)",
ribbon=[(y.min,y.max) for y in data], xrotation=45)
The JLD package provides a type-preserving way to save Julia objects to file.
using JLD
struct testData
x::Int64
y::String
end
foo = testData(7,"test")
save("/tmp/blah.jld", "foo", foo)
load("/tmp/blah.jld")
FileIO
FileIO provides a unified set of methods to access data in files: query()
, load()
, and save()
, as well as loadstreaming()
and savestreaming()
for large files. The functions can automatically recognize files using headers or extensions, but you can also provide format information as below, where we load the Iris Dataset from a CSV file.
using FileIO
load(File(format"CSV",iris.csv))
DataFrames
DataFrame objects represent tabular data as a set of vectors.
using DataFrames
df = DataFrame(A = 1:5, B = 1:2:10, C = ["a","b","c","d","q"])
A | B | C |
---|---|---|
1 | 1 | a |
2 | 3 | b |
3 | 5 | c |
4 | 7 | d |
5 | 9 | q |
Column names are referenced via accesing the fields.
df.A .+ df.B
JSON
Import and export JSON using the JSON package, which is always loaded on Nextjournal. In the example below, a Julia data structure input results in JSON output. The change from nothing
to null
is a clear indicator.
json(["foo", Dict("bar" => ("baz", nothing, 1.0, 2))])
Setup
Julia 1.x
Build a Minimal Julia 1.x Environment
We'll base our environment off of our Minimal Python image, which has a small conda
Python setup. Note that the Julia version is set as an environment variable on the runtime.
PATH | /usr/local/julia/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin |
JULIA_PATH | /usr/local/julia |
JULIA_VERSION | 1.5.2 |
CONDA_JL_HOME | /opt/conda |
JSSERVE_LISTEN_URL | 0.0.0.0 |
The exact version we're installing is
tarArch="x86_64"
dirArch="x64"
BASEURL="https://julialang-s3.julialang.org/bin"
tarfile="julia-${JULIA_VERSION}-linux-${tarArch}.tar.gz"
dir="${dirArch}/${JULIA_VERSION%[.-]*}"
tarurl="${BASEURL}/linux/${dir}/${tarfile}"
sigfile="${tarfile}.asc"
sigurl="${tarurl}.asc"
shafile="julia-${JULIA_VERSION}.sha256"
shaurl="${BASEURL}/checksums/${shafile}"
keyfile="juliareleases.asc"
keyurl="https://julialang.org/assets/${keyfile}"
wget -q --show-progress --progress=bar:force \
$tarurl $sigurl $shaurl $keyurl
sha256sum -c --ignore-missing $shafile
export GNUPGHOME="$(mktemp -d)"
gpg --import $keyfile
gpg --batch --verify $sigfile $tarfile
gpgconf --kill all
cp $tarfile /results/
rm -rf "$GNUPGHOME" $tarfile $sigfile $shafile $keyfile
Install from saved archive.
mkdir -p "$JULIA_PATH"
tar -xzf julia-1.5.2-linux-x86_64.tar.gz -C "$JULIA_PATH" --strip-components 1
And verify it runs.
julia -v
The JSON package is required to run on Nextjournal.
julia -e 'using Pkg; pkg"up; add JSON; precompile; test JSON"'
Install Conda.jl, which is set to use the existing Conda installation via the CONDA_JL_HOME
environment variable.
julia -e 'using Pkg; pkg"add Conda; build Conda; precompile"'
Check size.
du -hsx /
Build the Default Julia 1.x Environment
We'll add a number of packages as well as the libraries and support programs required by them. The major packages installed are JuliaPlots along with the GR and PlotlyJS backends, WGLMakie, and the DataFrames, CSV, and HDF5 data-handling packages.
First we'll check that the minimal Julia env works.
"$VERSION"
Install a few system tools and libraries.
apt-get -qq update
DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends \
imagemagick libhdf5-dev hdf5-tools mesa-utils \
libxt6 libxrender1 libgl1-mesa-glx libqt5widgets5 `# for GR` \
libhttp-parser2.7.1 `# for PlotlyJS`
apt-get clean
rm -r /var/lib/apt/lists/* # Clear package list so it isn't stale
Next the Julia package installs.
]up
]add SoftGlobalScope DataFrames JLD CSV CSVFiles BSON NRRD MeshIO HDF5 MAT FileIO JSExpr CSSUtil StatsBase StatsPlots Observables Parameters Unitful Interact WebSockets HTTP Blink WebIO PlotlyBase PlotlyJS RecipesBase GR Plots ImageCore ImageShow ImageMagick Colors ProgressMeter BenchmarkTools PackageCompiler FixedPointNumbers OffsetArrays IJulia WGLMakie AbstractPlotting JSCall VegaLite VegaDatasets RDatasets GeometryBasics
Build all packages.
]build
Precompile any qualifying packages.
]precompile
Finally, add font defaults for JuliaPlots and Makie. These named Code Listings will be mounted as files to the runtime's filesystem, and saved with the environment.
PLOTS_DEFAULTS = Dict(:fontfamily => "Open Sans")
Makie caches some fonts on first using
, so we make sure that they're already cached.
using AbstractPlotting, WGLMakie
for res in (AbstractPlotting.High, AbstractPlotting.Low)
AbstractPlotting.set_glyph_resolution!(res)
AbstractPlotting.cached_load();
end
Test.
julia -v
du -hsx /
Test Default 1.x Env
Test-load all packages.
using SoftGlobalScope, DataFrames, JLD, CSV, CSVFiles, BSON, NRRD, MeshIO, HDF5, MAT, FileIO, JSExpr, CSSUtil, StatsBase, StatsPlots, Observables, Parameters, Unitful, Interact, WebSockets, HTTP, Blink, WebIO, PlotlyBase, PlotlyJS, RecipesBase, GR, Plots, ImageCore, ImageShow, ImageMagick, Colors, ProgressMeter, BenchmarkTools, PackageCompiler, FixedPointNumbers, OffsetArrays, IJulia, WGLMakie, AbstractPlotting, JSCall, VegaLite, VegaDatasets, RDatasets, GeometryBasics
IJulia.verbose
Update, and add some mixed packages to test dependency resolution and installation.
]up
]add Flux ArrayFire FFTW Images ImageFiltering Adapt GPUArrays CUDA NNlib QuantumOptics LinearAlgebra DifferentialEquations BoundaryValueDiffEq DelayDiffEq DiffEqBase DiffEqCallbacks DiffEqFinancial DiffEqJump DiffEqNoiseProcess DiffEqPhysics DimensionalPlotRecipes OrdinaryDiffEq SteadyStateDiffEq StochasticDiffEq
Print package versions.
"$(Pkg.installed()["SoftGlobalScope"])"
"$(Pkg.installed()["GR"])"
"$(Pkg.installed()["PlotlyBase"])"
"$(Pkg.installed()["PlotlyJS"])"
"$(Pkg.installed()["Plots"])"
"$(Pkg.installed()["AbstractPlotting"])"
"$(Pkg.installed()["JLD"])"
"$(Pkg.installed()["CSV"])"
"$(Pkg.installed()["HDF5"])"
"$(Pkg.installed()["JSON"])"
"$(Pkg.installed()["FileIO"])"
"$(Pkg.installed()["DataFrames"])"