Apertus Engineer: Evaluations

ETH Zurich seeks a Mid-level Apertus Engineer to manage evaluation pipelines for large-scale open foundation models on the Alps supercomputer in Zürich.

This position supports the development of the Apertus project, which trains massive open-source models on the Alps supercomputing infrastructure. The engineer will ensure that evaluation processes are robust, scalable, and reliable, providing critical data to guide training and release decisions while collaborating with researchers and infrastructure teams.

Responsibilities

• Manage and improve the codebase and pipelines used to evaluate Apertus models throughout their lifecycle.
• Optimize evaluation execution for speed and scale, utilizing parallel processing on the Alps cluster and efficient inference backends.
• Ensure consistency in tokenization, prompting, and sampling across different evaluation harnesses and inference engines to align training and serving metrics.
• Troubleshoot failures, regressions, and inconsistencies within evaluation systems.
• Integrate and maintain reliable evaluation workflows for text, image, and audio modalities.
• Onboard new benchmarks in collaboration with academic partners, validating their metrics and implementation quality.
• Assess third-party services and other models using standardized benchmark suites to produce reproducible, comparable results.
• Generate reports, dashboards, and data insights that inform decisions on data mixtures, ablations, and model releases.
• Collaborate with safety, deployment, and community-focused engineers to integrate their specific evaluation needs into the shared pipeline.

Requirements

• MSc or PhD in Computer Science, Data Science, AI, or a related field; exceptional BSc holders with strong engineering experience are also considered.
• Proficient in Python and software engineering, with a track record of building robust data or evaluation pipelines.
• Demonstrated experience in LLM evaluation, whether using established harnesses like lm-evaluation-harness or custom tooling.
• Proven ability to collaborate effectively across research and engineering disciplines.
• Hands-on experience in relevant domains, gained through projects, studies, or professional work.
• Adaptability to shifting priorities, tools, and tasks driven by training schedules and rapid field developments.
• Experience running large-scale evaluations on GPU clusters using Slurm or similar schedulers, along with inference engines such as vLLM or SGLang.
• Familiarity with agentic evaluation frameworks, including tool use, sandboxed environments, and benchmarks like SWE-bench.
• Experience evaluating multimodal models, specifically those involving image or audio data.

Nice to have

• Strong attention to statistical rigor, including understanding variance, prompt sensitivity, and significance testing.
• Published research or deep familiarity with recent academic developments in relevant AI domains.
• Experience with LLM-as-judge pipelines and their calibration.
• Knowledge of practices for detecting and mitigating benchmark contamination.
• Ability to visualize and communicate evaluation findings clearly to research teams.

About the company

The Apertus team consists of over a dozen full-time engineers working alongside leading researchers from EPFL and ETH Zürich. They have released the Apertus 1 and 1.5 models and collaborate with more than thirty academic partners to deliver fully open, responsibly trained, multilingual, and multimodal AI models for both research and industry use.

• Collaborative environment with full-time engineers and leading academic researchers from top Swiss institutions.
• Focus on delivering fully open-source, responsibly trained AI models.
• Strong academic partnerships with over thirty collaborators.
• Work on cutting-edge multimodal and multilingual foundation models.