Subject Matter Expert (AI)

TL;DR

Subject Matter Expert (AI): Evaluating and auditing AI model outputs to ensure high-quality standards and accuracy with an accent on domain-specific rigor and quality framework development. Focus on identifying systemic issues, defining benchmarking criteria, and providing actionable insights to improve AI performance.

Company

Scale AI provides high-quality data and full-stack technologies that power the world's leading AI models for enterprises and governments.

What you will do

• Conduct structured audits of task outputs to evaluate accuracy and adherence to project guidelines.
• Define and refine quality benchmarks, scoring rubrics, and evaluation criteria.
• Analyze audit results to generate actionable insights on quality trends and performance gaps.
• Contribute to the development of gold standards and reference materials.
• Partner with internal stakeholders to calibrate on quality expectations and scoring consistency.
• Recommend improvements to workflows and evaluation processes based on audit findings.

Requirements

• Deep domain expertise in a specialized field such as medicine, law, or engineering.
• Strong analytical skills with the ability to evaluate complex outputs against defined standards.
• Experience in quality assurance, auditing, or evaluation frameworks.
• Ability to produce clear, structured, and actionable feedback.
• High attention to detail and commitment to accuracy and consistency.

Culture & Benefits

• Flexible, project-based work schedule allowing you to set your own hours.
• Opportunity to contribute to the development of cutting-edge AI systems.
• Inclusive and equal opportunity workplace committed to diversity.
• Independent work environment with minimal oversight.