Release v1.8.3 (Getting Started)

License PyPI Version Python Versions Documentation Downloads

Hyper-shell is an elegant, cross-platform, high-performance computing utility for processing shell commands over a distributed, asynchronous queue.


Simple, Scalable, Elastic

With hyper-shell you can take a listing of shell commands and process them in parallel. Use the available cluster mode to process locally, automatically scale out using SSH or MPI, or configure Parsl to scale elastically using an HPC scheduler like Slurm or in the cloud with Kubernetes.

➜ seq -w 10000 | hyper-shell cluster -N24 -t 'echo {}' | tail -4


The novel feature of hyper-shell, however, is the ability to independently stand up the server on one machine and then connect to that server using the client from a different environment.

Start the hyper-shell server and set the bind address to to allow remote connections. The server acts like a sieve, reading tasks from a file (or stdin), publishing them to the queue, and recording failed commands to a file (or stdout).

➜ hyper-shell server -H -k MY_KEY < TASKFILE > TASKFILE.failed

Connect to the running server from a different host (even from a different platform, e.g., Windows). You can connect with any number of clients from any number of hosts. The separate client connections will each pull individual tasks off the queue asynchronously, balancing the load.

➜ hyper-shell client -H host-1 -k MY_KEY > TASKFILE.out


Special variables are automatically defined for each individual task. For example, TASK_ID gives a unique integer identifier for each task (regardless of which client executes the task).

Further, any environment variable defined with the HYPERSHELL_ prefix will be injected into the environment of each task, sans prefix.

Use -t (short for --template) to execute a template, “{}” can be used to insert the incoming task argument (alternatively, use TASK_ARG). Be sure to use single quotes to delay the variable expansion.

➜ hyper-shell cluster -t '{} >outputs/$TASK_ID.out'

Table of Contents