Parsl
Brief for “Parallel Scripting Library,” Parsl permits you to take computing jobs and break up them throughout a number of techniques utilizing roughly the identical syntax as Python’s current Pool objects. It additionally permits you to sew collectively completely different computing duties into multi-step workflows, which may run in parallel, in sequence, or through map/cut back operations.
Parsl permits you to execute native Python purposes, but in addition run every other exterior utility by the use of instructions to the shell. Your Python code is written like regular Python code, save for a particular operate decorator that marks the entry level to your work. The job-submission system additionally offers you fine-grained management over how issues run on the targets—for instance, the variety of cores per employee, how a lot reminiscence per employee, CPU affinity controls, how typically to ballot for timeouts, and so forth.
One glorious function Parsl provides is a set of prebuilt templates to dispatch work to quite a lot of high-end computing sources. This not solely consists of staples like AWS or Kubernetes clusters, however supercomputing sources (assuming you’ve entry) like Blue Waters, ASPIRE 1, Frontera, and so forth. (Parsl was co-developed with assistance from lots of the establishments that constructed such {hardware}.)