Scaling Nextflow pipelines and AI agents on Verily Workbench

To ensure researchers can easily incorporate advanced analytics tools into their work, Verily Health recently added Nextflow into Verily Workbench. By bringing this powerful workflow framework directly into Workbench’s enterprise-grade Trusted Research Environment (TRE),researchers can now seamlessly scale both massive genomic pipeline and high-throughput AI agent evaluations with secure co-analysis and data governance.

Nextflow has become a go-to tool for scalable, reproducible scientific workflows due to its ability to handle complex dependencies and make pipelines portable across clouds and HPC systems. However, running Nextflow as a separate deployment can require significant operational overhead requiring teams to manage infrastructure and ensure proper governance and data access for users.

By handling the heavy infrastructure layer behind the scenes, Verily Workbench further expands the value of Nextflow across four core capabilities:

• Unified workflow management: Verily Workbench consolidates the entire workflow lifecycle into a single, clean user interface (UI). Researchers can register pipelines, configure inputs, and submit runs, while tracking real-time progress, outputs, and engine logs all in one organized workspace.

• Click-and-go setup: Whether you are using your own custom processes or popular, ready-to-use community workflows (e.g. nf-core), Workbench makes it easy to get started.

• Built for reproducibility and scale: Every run captures crucial metadata including inputs, sample sheets and profiles. A centralized dashboard allows for real-time tracking of jobs and individual tasks, making it easy to spot bottlenecks or failures.

• Integrated data and analysis: Workbench integrates Nextflow with its Data Collections, allowing teams to manage their governed multi-modal datasets in one place. Once a job completes, researchers can launch JupyterLab,VS Code or other custom environments within the platform in order to continue their analysis and securely share results with collaborators.

In our whitepaper, we highlight how researchers at Bangor University migrated their wastewater-based epidemiology pipeline to Verily Workbench to track high-risk antimicrobial resistance (AMR) genes. Historically, running this analysis on a shared institutional supercomputer resulted in queue times of up to 48 hours. On Workbench, the team is reducing those delays, utilizing dynamic cloud provisioning to scale resources up and down automatically for cost-optimized execution.

Crucially, this integration extends well beyond traditional bioinformatics. The whitepaper also explores how Nextflow on Workbench orchestrates modern engineering workloads, such as parallel AI agent evaluations. Developing robust AI assistants can often require simulating thousands of user conversations and grading them simultaneously. Because Nextflow allows you to build and test code locally on a small scale before instantly scaling it across thousands of cloud nodes without changing a single line of code, teams can evaluate AI models at a fraction of the traditional time and cost.