API

mutable struct ReplicatedSuperMatrix{T, I, S, F, J, G} <: SuperLUDIST.AbstractSuperMatrix{T, I, S}

SuperMatrix that has been globally replicated on all ranks in a communicator. The local storage contains the full matrix, the Julia representation may be accessed by A.store::SparseBase.CSCStore{T, CIndex{<:Int}}.

Extended

• supermatrix: Internal: Reference to the C SuperMatrix

• format: Internal: Reference to the C storage format NCformat held by supermatrix

• store: Julia-side storage, of type SparseBase.CSCStore. Keeps memory alive.

• grid

mutable struct DistributedSuperMatrix{T, I, S, F, J, G} <: SuperLUDIST.AbstractSuperMatrix{T, I, S}

SuperMatrix distributed by row between all ranks in a communicator. The Julia representation may be accessed by A.store::SparseBase.CSRStore{T, CIndex{<:Int}}.

Each local matrix stores a subset of rows specified by A.globalrows

Extended

• supermatrix: Internal: Reference to the C SuperMatrix

• format: Internal: Reference to the C storage format NRformat_loc held by supermatrix

• store: Julia-side storage, of type SparseBase.CSRStore. Keeps memory alive.

• globalsize

• first_row

• grid

cscstore(Tv, Ti, A)

Copy the sparse matrix A (anything with SparseMatrixCSC's colptr, rowval and nzval fields, such as the result of MatrixMarket.mmread) into a SparseBase.CSCStore with the 0-based CIndex{Ti} indices that SuperLU_DIST expects. The values are always copied, since SuperLU_DIST may scale and permute the matrix in place.

coostore(Tv, Ti, A)

Like cscstore, but copy A into a SparseBase.CoordinateStore.

pgssvx!(
    A::ReplicatedSuperMatrix{Tv, Ti},
    b::Union{Array{Tv, 1}, Array{Tv, 2}};
    options,
    perm,
    LU,
    stat,
    berr
) -> Tuple{Any, Any}

Solve the replicated sparse linear system Ax = b, overwriting b with the solution.

Returns b and the factor object which may be used to solve with different b or reuse parts of the factorization with different values for A.

pgssvx!(
    A::DistributedSuperMatrix{Tv, Ti},
    b::Union{Array{Tv, 1}, Array{Tv, 2}};
    options,
    Solve,
    perm,
    LU,
    stat,
    berr
) -> Tuple{Any, Any}

Solve the distributed sparse linear system Ax = b, overwriting b with the solution. On a Grid3D this uses SuperLU_DIST's 3D pgssvx3d drivers.

Returns b and the factor object which may be used to solve with different b or reuse parts of the factorization with different values for A.

lu!(A::Union{ReplicatedSuperMatrix, DistributedSuperMatrix}, b_local = ones(...); kwargs...)

Factorize A with SuperLU_DIST, and return a factorization object F that solves with F \ b or ldiv!(F, b). A distributed b holds this rank's rows of the right hand side.

On a Grid3D, later solves run in blocks of as many columns as b_local has, so pass a b_local with the number of right-hand sides you will usually solve with.

The keyword arguments are passed on to pgssvx!, for example options.

SuperLUFactorization

The factorization returned by pgssvx! and lu!. It holds the matrix, the options and SuperLU_DIST's factors, and solves further right-hand sides with F \ b or ldiv!(F, b).

isfactored(F::SuperLUFactorization)

Whether F holds a computed factorization, so that further solves reuse it.

Grid{Ti}(nprow, npcol, comm = MPI.COMM_WORLD; batch = false, usermap = nothing)

A 2D nprow × npcol process grid for SuperLU_DIST with index type Ti ∈ (Int32, Int64), built from the first nprow * npcol ranks of comm. MPI is initialized if necessary.

With batch = true the ranks are mapped onto the grid by usermap, a nprow × npcol matrix of ranks (default: row-major order).

Fields of the underlying gridinfo_t, such as iam, nprow and npcol, are available as properties.

Grid3D{Ti}(nprow, npcol, npdep, comm = MPI.COMM_WORLD)

A 3D nprow × npcol × npdep process grid for SuperLU_DIST's communication-avoiding 3D factorization, with index type Ti ∈ (Int32, Int64), built from the first nprow * npcol * npdep ranks of comm. It consists of npdep layers of nprow × npcol 2D grids. npdep must be a power of two. MPI is initialized if necessary.

A DistributedSuperMatrix on a Grid3D is solved with SuperLU_DIST's pgssvx3d drivers. Its rows must be distributed over all the ranks of the 3D grid.

Fields of the underlying gridinfo3d_t, such as iam, nprow, npcol, npdep, grid2d (this rank's layer) and zscp (zscp.Iam is the layer number), are available as properties.

AbstractGrid{Ti}

A SuperLU_DIST process grid with index type Ti: a 2D Grid or a 3D Grid3D.

Options{Ti}(; kwargs...)

Solver options (superlu_dist_options_t) for SuperLU_DIST with index type Ti. They start from SuperLU_DIST's defaults (set_default_options_dist), and any field may be set with a keyword argument or as a property afterwards, for example Options{Int64}(; ColPerm = SuperLUDIST.Common.MMD_AT_PLUS_A, PrintStat = SuperLUDIST.Common.NO).

set_gpu!(gpu::Symbol)

Choose which build of SuperLU_DIST to use:

• :none (the default) uses the CPU build from SuperLU_DIST_jll.
• :cuda uses the CUDA build from SuperLU_DIST_CUDA_jll. That package is not installed with SuperLUDIST, so add it to your environment first. The CUDA build needs x86_64 Linux, an NVIDIA GPU and driver, and MPI.jl configured with MPItrampoline, and SuperLUDIST reports an error when it is loaded where it can't be used.

Libraries set with set_libraries! take precedence. HAVE_CUDA tells which build is in use.

The choice is stored with Preferences.jl and takes effect after restarting Julia.

HAVE_CUDA

Whether SuperLUDIST is set up for a CUDA build of SuperLU_DIST. With it, factorizations and solves are offloaded to the GPU unless options.superlu_acc_offload is set to 0. If the CUDA build can't be used where SuperLUDIST is loaded, an error is logged and the libraries are not set. See set_gpu!.

set_libraries!(; libsuperlu_dist_Int32 = nothing, libsuperlu_dist_Int64 = nothing)

Use your own builds of SuperLU_DIST instead of the ones from SuperLU_DIST_jll or SuperLU_DIST_CUDA_jll, for example to link against a system MPI on an HPC cluster. The libraries must be SuperLU_DIST 9.3.x, built with 32-bit and 64-bit int_t respectively, and linked against the same MPI that MPI.jl uses. They may be CPU or CUDA builds, but both must be the same kind. Either may be omitted, in which case matrices with that index type are unavailable. Calling set_libraries!() with no arguments reverts to the jll builds.

The choice is stored with Preferences.jl and takes effect after restarting Julia.

superlu_set_num_threads(Ti, n)

Set the number of OpenMP threads used by the SuperLU_DIST library with index type Ti on this rank.

superlu_set_num_threads(Ti, n)

Set the number of OpenMP threads used by the SuperLU_DIST library with index type Ti on this rank.

mmread_and_generatesolution(Tv, Ti, nrhs, path, grid::AbstractGrid; root = 0)

Read the Matrix Market file path on rank root, and generate a test problem for it, as SuperLU_DIST's example drivers do. Returns (coo, b, xtrue), where coo is the matrix as a SparseBase.CoordinateStore on root (nothing elsewhere) and b = A * xtrue holds nrhs right-hand sides on every rank.

GenXtrue_dist!(xtrue::Matrix, Ti = Int)

Fill xtrue with SuperLU_DIST's test solution (GenXtrue_dist), for example to build a right-hand side with FillRHS_dist!. Ti selects the SuperLU_DIST library.

FillRHS_dist!(b, A::ReplicatedSuperMatrix, xtrue; trans = false)

Compute the right-hand side b = A * xtrue (or transpose(A) * xtrue) with SuperLU_DIST's FillRHS_dist, and return b.

inf_norm_error_dist(x::Matrix, xtrue::Matrix, grid::AbstractGrid)

Print the relative infinity norm error of each column of the solution x against xtrue, with SuperLU_DIST's inf_norm_error_dist. It only uses this rank's data and does no communication, so with a distributed x each rank prints the error of its own rows.

PStatPrint(F::SuperLUFactorization)

Print SuperLU_DIST's statistics (timings, flops and memory) for the factorization F. This is collective: every rank in the grid must call it.

lu!(A::Union{ReplicatedSuperMatrix, DistributedSuperMatrix}, b_local = ones(...); kwargs...)

Factorize A with SuperLU_DIST, and return a factorization object F that solves with F \ b or ldiv!(F, b). A distributed b holds this rank's rows of the right hand side.

On a Grid3D, later solves run in blocks of as many columns as b_local has, so pass a b_local with the number of right-hand sides you will usually solve with.

The keyword arguments are passed on to pgssvx!, for example options.

mmread(DistributedSuperMatrix{Tv, Ti}, filename::String, grid::AbstractGrid{Ti}; root = 0, partitioner = distribute_evenly)

Read an mtx file on rank root and distribute its rows between the ranks in grid. The grid must use all the ranks of grid.comm.

Arguments

• arg1<:Type{DistributedSuperMatrix{Tv, Ti}}
  • Tv <: Union{Float32, Float64, ComplexF64}: Non-optional eltype.
  • Ti <: Union{Int32, Int64}: Non-optional index type.
• filename::String: .mtx file accessible by the root process.
• grid::AbstractGrid{Ti}: The 2D or 3D process grid on which to distribute the matrix.

Keyword Arguments

• root::Int = 0: The MPI process on which to read the file.
• partitioner = SparseBase.Communication.distribute_evenly: Computes the number of rows

on each rank from the number of rows and ranks.

Example

using MPI, SuperLUDIST, MatrixMarket
grid = SuperLUDIST.Grid{Int32}(2, 2)
A = MatrixMarket.mmread(DistributedSuperMatrix{Float64, Int32}, "matrix.mtx", grid)

mmread(ReplicatedSuperMatrix{Tv, Ti}, filename::String, grid::Grid{Ti}; root = 0)

Read an mtx file on rank root into a compressed sparse column format and broadcast it to all ranks in grid.

Arguments

• arg1<:Type{ReplicatedSuperMatrix{Tv, Ti}}
  • Tv <: Union{Float32, Float64, ComplexF64}: Non-optional eltype.
  • Ti <: Union{Int32, Int64}: Non-optional index type.
• filename::String: .mtx file accessible by the root process.
• grid::Grid{Ti}: The process grid on which to replicate the matrix.

Keyword Arguments

• root::Int = 0: The MPI process on which to read the file.

Example

using MPI, SuperLUDIST, MatrixMarket
grid = SuperLUDIST.Grid{Int32}(1, 1)
A = MatrixMarket.mmread(ReplicatedSuperMatrix{Float64, Int32}, "matrix.mtx", grid)

scatterstore!(
    rstore::DistributedSuperMatrix{Tv, Ti},
    sstore::Union{Nothing, SparseBase.SinglyCompressedStore{Tvalues, StorageOrders.RowMajor()} where Tvalues},
    chunksizes;
    root,
    comm
) -> DistributedSuperMatrix

Distribute the rows of sstore, given on rank root, over the ranks of comm, which must be the ranks of the grid of rstore, with chunksizes[i] rows on rank i - 1.

cref(T)

A zero-initialized Ref{T} for the C struct T.

libmodule(::Type{Ti})

The module holding the generated bindings for SuperLU_DIST built with index type Ti.

setfield_ref!(r::Ref{T}, name::Symbol, v)

Set field name of the immutable struct held in r to v.