Software Engineer - Benchmarking

Responsibilities

• Prepare, clean, convert, validate, and maintain benchmark datasets and flag ambiguities that could compromise results.
• Build infrastructure and pipelines that run evaluations consistently across model APIs and terminal agents.
• Create lightweight containerized evaluation environments and viewers for tasking and tool-use assessment.
• Support fine-tuning experiments and compare baseline with post-training model performance.
• Build scoreboards and leaderboards showing model-, benchmark-, task-, domain-, and rubric-level results.
• Develop tools to identify failure modes, compare models and agent scaffolds, and track capability changes over time.
• Build HTML viewers and internal tools for task authoring, expert annotation, review, quality control, and structured data collection.
• Collaborate with researchers and engineers to ensure evaluation data and published outputs are accurate, consistent, and integrated.

Requirements

• 4+ years of professional experience building and maintaining complex systems.
• Strong Python engineering skills and experience working in existing codebases and infrastructure.
• Experience preparing, cleaning, validating, and maintaining datasets.
• Experience with Docker and reproducible execution environments.
• Ability to collaborate with researchers and scientists and translate methodology into working systems.
• Hands-on AI evaluation experience or experience with Harbor, Terminal-Bench, or Inspect is a strong plus.
• Experience fine-tuning or post-training open-source LLMs, agentic or tool-use evaluation, LLM-as-judge or rubric-based grading, data-heavy dashboards, leaderboards, visualizations, or related open-source or published benchmark work is preferred.

Benefits

• Medical, dental, and vision coverage.
• 401(k).
• Monthly wellness and fitness stipend.
• Paid time off and company holidays.
• Annual company off-sites in Tahoe, Mendocino, Mexico City, San Diego, and Park City.
• Parent-friendly policies, paid family leave, and remote flexibility.
• Fully remote or hybrid work from the San Francisco or New York City offices.