Kestra

Kestra is a workflow orchestration platform built around a simple idea: one engine for every team. Data engineers, infrastructure teams, and AI practitioners often end up maintaining separate tools that don't talk to each other. Kestra replaces that fragmentation with a single, declarative platform where all workflows live, run, and get monitored together.

Workflows are written in YAML, which means they're version-controlled, reviewable, and deployable through standard CI/CD pipelines. You don't have to rewrite existing code. Tasks can run in Python, Bash, Node.js, Go, or containers, so teams keep their existing logic and just wire it into Kestra's orchestration layer.

Key capabilities:

• 1400+ plugins covering cloud providers, databases, messaging systems, data tools (dbt, Airbyte, Spark), CI/CD, and SaaS services
• Event-driven and scheduled triggers including cron, webhooks, and message queue events, all handled by the same engine
• Built-in governance with retries, timeouts, SLAs, RBAC, and audit logs baked in, not bolted on
• Dual interface where developers work in code and non-engineers use the UI, both staying in sync
• API-first design so executions and workflow management can be automated or integrated into other systems
• for orchestrating RAG pipelines, model retraining, evaluation, and agent workflows with full observability