Senior Storage & Data Engineer

ETH Zurich seeks a Senior Storage & Data Engineer in Lugano to design and manage petabyte-scale storage systems and data pipelines that support scientific research and large-scale model training.

The role focuses on bridging the gap between raw data ingestion and its practical use by designing the underlying pipelines, metadata structures, and access layers. This work ensures that data is findable and consumable for training jobs and simulations, directly enabling published scientific discoveries and frontier-scale model development.

Responsibilities

• Establish data lineage and provenance to trace any dataset, checkpoint, or result back to its original inputs and transformations.
• Tune parallel filesystems like Lustre and GPFS, alongside object storage, to handle the specific concurrency, small-file, and large-checkpoint demands of distributed GPU training and HPC simulations.
• Architect and operate multi-petabyte storage environments that guarantee the integrity and availability required for scientific work, utilizing erasure coding and tiering strategies from hot to archival storage.
• Implement storage and data services as code, deploying and scaling them programmatically.
• Monitor storage health, capacity trends, and pipeline performance to proactively identify and resolve issues before they impact users.
• Translate real-world access patterns from domain scientists and ML engineers into technical requirements, while challenging requests that could negatively impact downstream systems.

Requirements

• Hold a technical degree in computer science or engineering, or possess equivalent experience demonstrating comparable depth.
• Demonstrate strong expertise in storage systems, including block and object filesystems, performance tuning, and redundancy mechanisms like RAID and erasure coding.
• Be proficient in Python and comfortable automating infrastructure using tools such as Ansible or Terraform.
• Understand how ML and scientific workloads consume data, including patterns involving billions of small files, large checkpoints, and sharding, and recognize why naive layouts fail.
• Possess a clear perspective on data lineage, provenance, or reproducibility, with practical experience using relevant tooling.

Nice to have

• Experience with parallel filesystems like Lustre, Spectrum Scale, or GPFS, or distributed storage systems such as Ceph or VAST.
• Familiarity with scientific data formats including HDF5, Zarr, and Parquet, along with insights into their appropriate use cases.
• Experience interfacing object storage (S3) with ML frameworks like PyTorch or TensorFlow.
• Knowledge of orchestration platforms like Kubernetes or Argo, and data-movement tooling.
• Background in data versioning or cataloguing tools such as DVC, lakeFS, or metadata catalogs, plus familiarity with FAIR data principles.
• Experience with CI/CD and provisioning tools including GitLab CI, HashiCorp Vault, and MAAS.

What the company offers

• Access to unique hardware and scale that exceeds typical enterprise IT environments, offering complex problems without standard vendor solutions.
• Opportunities to actively shape data management strategies in an environment that prioritizes data integrity and innovation.
• Public transport season tickets and car-sharing benefits.
• Access to a wide range of sports facilities and programs through ASVZ.
• Childcare support and attractive pension benefits.

About the company

ETH Zurich fosters a culture built on curiosity, openness, courage, support, and integrity. The institution encourages deep learning, effective collaboration, and tackling difficult problems. It is committed to building a diverse and inclusive engineering team, particularly encouraging applications from underrepresented groups. The university supports professional development and contributes to positive societal change.

• Curiosity: Enjoying learning and understanding systems deeply.
• Openness: Collaborating effectively and valuing different perspectives.
• Courage: Willingness to tackle difficult or unfamiliar problems.
• Supportive: Helping colleagues and users succeed.
• Integrity: Acting responsibly, reliably, and transparently.
• Commitment to diversity and inclusion in engineering.
• Focus on professional development and societal impact.