Funnel is a toolkit for distributed task execution with a simple API.
A task describes metadata, state, input/output files, resource requests, commands, and logs. The task API has four actions: create, get, list, and cancel. Given a task, Funnel will queue it, schedule it to a worker, and track its state and logs. A worker will download input files, run a sequence of Docker containers, upload output files, and emits events and logs along the way.
A wide variety of options make Funnel easily adaptable: - BoltDB - Elasticsearch - MongoDB - AWS Batch, S3, DynamoDB - OpenStack Swift - Google Cloud Storage, Datastore - Kafka - HPC support: HTCondor, Slurm, etc. - and more