High-level parallelization of SfePy using petsc4py

Robert Cimrman

Audience level:


We illustrate a high level parallelization of the partial differential equation solution package SfePy (http://sfepy.org), enabled by the code capability to integrate an equation over an arbitrary subdomain. Several processes, each responsible for its own subdomain, then assemble their contributions to the global linear system. For the parallel solution we use the PETSc library via petsc4py.


SfePy (Simple Finite Elements in Python, http://sfepy.org) is a framework for solving various kinds of problems (mechanics, physics, biology, ...) described by partial differential equations (PDEs) in one, two or three space dimensions. It is written mostly in Python and relies on the scientific Python software stack, most notably on fast vectorized operations provided by NumPy arrays, with heavy use of advanced broadcasting and "index tricks" features. C and Cython are used in places where vectorization is not possible, or is too difficult or unreadable. Other scientific Python packages used include: SciPy solvers and algorithms, Matplotlib for 2D and simple 3D plots, Mayavi for 3D plots and simple postprocessing GUI, SymPy for symbolic operations/code generation, PyTables for storing solutions and related data, etc. Besides the finite element method (FEM) [1], it is possible to use also a spline-based version of FEM, called isogeometric analysis [2].

A user can specify a PDE to solve by selecting one ore more terms from the extensive built-in library. The terms correspond to integrals over the domain of interest or its subdomains - inner or boundary. In the poster we illustrate a relatively straightforward way of the high level parallelization of the PDE solution, that makes use of the code capability to integrate a term over an arbitrary subdomain. Several processes of the solver can be spawned, each responsible for its own subdomain, so that the entire domain is covered. Each process then assembles its local contributions - a sparse matrix and a right-hand side vector - to the global linear system that is solved in parallel. For this purpose, we use the PETSc library [3] and its large collection of parallel solvers and data structures, that can be used in Python thanks to the petsc4py package [4].


[1] Thomas J. R. Hughes. The Finite Element Method: Linear Static and Dynamic Finite Element Analysis, Dover Publications, 2000.

[2] J. Austin Cottrell, Thomas J.R. Hughes, Yuri Bazilevs. Isogeometric Analysis: Toward Integration of CAD and FEA. John Wiley & Sons. 2009

[3] Satish Balay, Shrirang Abhyankar, Mark F. Adams, Jed Brown, Peter Brune, Kris Buschelman, Victor Eijkhout, William D. Gropp, Dinesh Kaushik, MatthewG. Knepley, Lois Curfman McInnes, Karl Rupp, Barry F. Smith, Hong Zhang. PETSc Web page, http://www.mcs.anl.gov/petsc. 2014

[4] Lisandro D. Dalcin, Rodrigo R. Paz, Pablo A. Kler, Alejandro Cosimo. Parallel Distributed Computing using Python, Advances in Water Resources, 34(9):1124-1139, 2011. http://dx.doi.org/10.1016/j.advwatres.2011.04.013