- Pay
- $142,300 — $263,300 USD per year
Employer Job Description
From Apple's posting. “We” and “our” refer to the employer.
Apple is where individual imaginations gather together, committing to the values that lead to great work. Every new product we build, service we create, or experience we deliver is the result of us making each other's ideas stronger.
Description
Our team, part of Apple Services Engineering, is building the scientific foundation for how AI systems are evaluated across Apple. Our goal is to create the evidence that teams can act on: what to measure, whether a change helped, why a system failed, and whether it is ready to ship. We are seeking a Measurement Scientist to ensure that our evaluation methods are not just sophisticated, but scientifically valid and trustworthy.
In this role, you will apply apply state-of-the-art measurement methods, validity frameworks, experimental design, and statistical rigor to establish measurement standards for AI evaluation, ensuring that when we claim an evaluator measures "helpfulness" or "safety," it actually does. We are looking for individuals across a range of experience levels.
This role uniquely bridges measurement science and cutting-edge AI evaluation. This role uniquely bridges measurement science and cutting-edge AI evaluation. Our team is building AI tools and capabilities to help developers do evaluation better and faster. You will develop methods for validating these tools: making sure that they generate evaluation artifacts and workflows that give developers the right signals to make sound engineering decisions and build great products for Apple customers.
You will work on an interdisciplinary team with with other measurement scientists and ML researchers to solve new problems in AI evaluation. Your work will be both published at top measurement and ML venues and productionized into the evaluation SDK used across Apple. The successful candidate will have deep expertise in psychometrics or other measurement disciplines, or in experimental design and causal inference, with the ability to apply these principles to novel AI evaluation challenges.
Our team deliberately combines ML science and measurement science. We especially welcome people who bridge those areas, including computer scientists who specialize in measurement, and measurement scientists who have worked on ML problems. You will also work collaboratively with platform engineers and evaluation practitioners to translate measurement science into practical tools that scale across the organization.
Responsibilities
- Design validity frameworks for AI evaluation, ensuring that automated metrics, LLM-as-judge systems, and human evaluation protocols measure what they claim to measure across diverse contexts.
- Develop and apply methods from measurement science to assessing the quality of benchmarks, for example drawing on frameworks like item response theory (IRT).
- Create calibration and bias detection systems for automated evaluators, ensuring LLM-as-judge scores are interpretable, consistent, and free from systematic biases.
- Build robust statistical tools for practitioners for sample-size planning, comparing system versions, quantifying uncertainty, controlling error rates, and visualizing data.
- Connect offline evaluation to real-world outcomes: test whether offline score changes predict what users experience, design randomized audits, and develop methods for setting defensible ship thresholds that engineering teams can apply.
- Establish measurement standards for evaluator transfer and generalization, including methods to quantify or predict when evaluators will maintain validity across domains, languages, or contexts.
- Validate novel evaluation methods in collaboration with ML researchers, ensuring intelligent search algorithms discover statistically meaningful patterns, failure analyses estimate how often problems occur with uncertainty, and synthetic data generation produces representative samples.
- Collaborate with platform engineers to productionize measurement methods into evaluation infrastructure, creating self-service tools for validity checking, reliability testing, and interpretable outputs (report cards, warnings, confidence metrics).
- Publish research at top measurement venues and/or ML conferences (NeurIPS, ICML, ICLR), advancing both measurement science and AI evaluation.
- Collaborate across disciplines with ML researchers developing novel methods, platform engineers building scalable infrastructure, and evaluation practitioners using these tools in production.
Minimum Qualifications
- PhD in Computer Science or Machine Learning with a measurement or evaluation focus, Psychometrics, Educational Measurement, Quantitative Psychology, Statistics, Biostatistics, Econometrics, or equivalent research/work experience.
- Deep expertise in modeling test or benchmark data (IRT, latent variable, or other response models) and construct validation, in benchmark design and validity for AI systems, or in experimental design and causal inference (surrogate outcomes, observational causal methods).
- Strong statistical foundation including experimental design, power analysis, sampling theory, and uncertainty quantification.
- Track record of designing and validating measurement instruments or experiments as demonstrated through publications or applied work.
- Proficiency in Python (preferred) or R for statistical analysis, psychometric modeling, and method implementation.
- Strong working knowledge of generative AI technology.
- Excellent communication skills with the ability to explain complex measurement concepts to engineers, ML researchers, and non-technical stakeholders.
Preferred Qualifications
- Research or experience applying measurement science to AI/ML evaluation, benchmark validity, automated scoring systems, or computational assessment.
- Knowledge of modern ML evaluation challenges including LLM-as-judge, automated metrics, benchmark design, and agentic systems.
- Publications at measurement venues or top ML conferences (NeurIPS, ICML, ICLR).
- Expertise in computational social or behavioral science using generative AI.
- Experience collaborating with engineers to turn research methods into production tools and scalable infrastructure.
- Experience with online experimentation or with validating offline metrics against real-world outcomes (surrogate or proxy metrics).
- Experience estimating rare events or making decisions when one kind of error costs far more than another.
- Pay & Benefits
At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $142,300 and $263,300, and your base pay will depend on your skills, qualifications, experience, and location.
Apple employees also have the opportunity to become an Apple shareholder through participation in Apple’s discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple’s Employee Stock Purchase Plan.
You’ll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses — including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation.
Learn more about Apple Benefits
Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.
About This Listing
LAYIQ is an independent job-discovery service. This listing does not imply a partnership with or endorsement by the employer. Review the original posting for current details and availability.
Employer posted:
Listing Check
- Employer Posted
- · 1 day ago
- Status
- Listed as a current opening
- Apply Goes To
- jobs.apple.com, the employer's own site
How to spot a ghost jobGhost Job Checker: check another job link
The checker shows whether the employer still lists a job.





