Data Scientist – AI & Agentic Applications & Benchmarking

Description

About CloudBees

CloudBees is the leading software delivery platform for enterprise DevOps teams. As a high-growth startup, we empower developers to build, deploy, and manage software more efficiently. Now, we’re bringing agentic intelligence into our platform to supercharge developer workflows—and we need a data scientist who can both drive insights and tell the story behind the metrics.

The Role

CloudBees is seeking a startup-savvy Data Scientist to help define, measure, and evangelize the impact of Agentic Applications across our platform. You’ll work closely with engineers and product teams to prototype and measure AI and Agentic experiences, using evals, telemetry, and AI benchmarks to help the company drive the conversation in the market and with customers. Translating performance into clear, compelling narratives to our customers and internal teams.

As a founding member of the team, you will lead the charge as equal parts builder, evaluator, and communicator—with the technical depth to prototype in Python notebooks, Claude Code, and other tools to drive clarity to write about what matters.

Key Responsibilities

• Partner with our platform team to develop and prototype telemetry, eval frameworks, and benchmarks for emerging agentic systems.
• Partner with product and engineering teams to measure AI outcomes and usage across customers and teams.
• Help define KPIs and success metrics for AI and LLM-driven features and workflows.
• Use Python notebooks to explore data, visualize insights, and test hypotheses rapidly and share insights.
• Tell the story behind the numbers: Write internal documentation, performance summaries, and thought leadership around outcomes.
• Enable engineering teams to instrument, log, and evaluate agent performance effectively.
• Stay up to date with evolving metrics and evaluation techniques in the LLM and agentic AI ecosystem.

Required Qualifications

• 3+ years of experience in data science or ML analytics roles, ideally in startup or high-growth environments.
• Proficiency in Python, including building and sharing analysis via Jupyter notebooks.
• Experience working with evals, telemetry, A/B testing, and evaluating user-facing ML systems.
• Experience with AI/ML tools such as MLFlow, Hugging Face, or other Model / LLM tools.
• Ability to partner with technical teams to define meaningful metrics and benchmarks.
• Clear communication skills—capable of writing about outcomes, sharing learnings, and influencing stakeholders.
• Comfort working in fast-paced, ambiguous environments where speed and clarity matter.

Preferred Qualifications

• Experience with agentic or LLM-based applications (e.g., evaluating AI copilots, autonomous workflows).
• Familiarity with tools like LangSmith, OpenInference, or custom eval stacks.
• Background in developer tools, DevOps, or platform engineering environments.

Why Join CloudBees

• Shape the future of AI-driven DevOps with real user impact.
• Join a nimble, passionate team at the forefront of agentic system development.
• Work in a flexible, remote-first culture built on trust and innovation.
• Competitive salary, startup equity, and excellent benefits.

CloudBees is proud to be an Equal Opportunity Employer. We value diverse voices, ideas, and experiences as essential to building great products.