Workflow tools

[christophernhill]

What workflow tools are people using, learning. What features are useful, what are missing.

[Aariq]

I've been learning and using the targets R package for workflow management, and I really love it. It's not language agnostic, of course, but it's really nice for a lot of other reasons including relatively easy setup to work with HPC.

[jmelot]

We use Apache Airflow and Beam on the google cloud platform and although they are not without their quirks they have worked well for us for automating and scaling ETL, model predictions, etc. for a few years now. I'd definitely be curious to hear about other people's data workflows/pipelines!