Funnel

Apr 27, 2016
Go to Project Site

https://ohsu-comp-bio.github.io/funnel/

Funnel is a toolkit for distributed task execution with a simple API.

A task describes metadata, state, input/output files, resource requests, commands, and logs. The task API has four actions: create, get, list, and cancel. Given a task, Funnel will queue it, schedule it to a worker, and track its state and logs. A worker will download input files, run a sequence of Docker containers, upload output files, and emits events and logs along the way.

A wide variety of options make Funnel easily adaptable:

  • BoltDB
  • Elasticsearch
  • MongoDB
  • AWS Batch, S3, DynamoDB
  • OpenStack Swift
  • Google Cloud Storage, Datastore
  • Kafka
  • HPC support: HTCondor, Slurm, etc.
  • and more
Data Systems
Kyle Ellrott
Kyle Ellrott
Assistant Professor

Researching Cancer Genomics Data Science