Our client is seeking senior software engineers to design and evaluate realistic coding-agent benchmark tasks. This work combines production engineering, secure software development, repository validation, test design, and careful evaluation of whether coding agents complete engineering tasks without violating safety, privacy, permission, or data-integrity requirements.

What You'll Do

Build production-like benchmark tasks using repositories, tests, configuration, documentation, and runtime scenarios.
Create realistic engineering tasks such as bug fixes, feature additions, CI repairs, configuration migrations, integration updates, and runtime-state fixes.
Define utility requirements that verify correct task completion.
Define safety and alignment requirements that protect system constraints, developer intent, permissions, privacy, oversight, and data integrity.
Develop visible and hidden test suites.
Create safe and unsafe reference solutions for controlled evaluation.
Identify unsafe shortcuts such as disabling tests, weakening assertions, deleting protected data, leaking secrets, broadening permissions, or bypassing validation.
Package prompts, metadata, runnable repositories, evaluators, reference patches, scoring rubrics, and calibration notes.
Analyze coding-agent rollouts for utility completion and safety violations.
Collaborate with engineering, QA, security, and client teams to improve evaluator reliability and benchmark difficulty.

Requirements

Bachelor's or Master's degree in Computer Science or a related technical discipline.
At least 8 years of hands-on software engineering experience in tier-one product companies or technology startups.
Strong professional Python experience, plus at least one of Java, Go, or C++.
Experience building and maintaining production software used by real customers.
Strong software architecture, debugging, and performance-optimization skills.
Production engineering experience covering deployment pipelines, monitoring, logging, observability, metrics, distributed tracing, high availability, fault tolerance, scalability, and disaster recovery.
Secure-software experience identifying vulnerabilities, implementing fixes, and validating remediation. Relevant areas include injection, authentication and authorization, memory safety, race conditions, deserialization, secrets management, and input validation.
Practical experience debugging complex production issues, participating in incident response, and building resilient systems.

Engagement Details