Senior QA Automation Engineer (AI)

TL;DR

Senior QA Automation Engineer (AI): Leading validation and verification strategies for AI transformation with an accent on non-deterministic testing and LLM reliability. Focus on building automated frameworks to validate AI behaviors against ground truth datasets and ensuring compliance with formal systems engineering standards.

Location: Must be based in or able to work from Abu Dhabi (Hybrid)

Company

hirify.global is a recruitment and talent consultancy firm specializing in technical and executive placements.

What you will do

• Architect automated frameworks to evaluate Generative AI outputs for hallucination, consistency, and factual accuracy.
• Implement automated metrics to verify RAG pipelines and ensure regulatory documentation compliance.
• Design regression suites to monitor prompt drift and maintain quality in AI-generated engineering documents.
• Validate data consistency between AI agents and enterprise systems like SAP S/4HANA.
• Map automated test cases to system requirements to create digital evidence for formal V&V reports.
• Manage the defect lifecycle and trace AI logic defects back to specific model versions.

Requirements

• 5+ years of experience in QA Automation with 2+ years focused on complex data-driven applications, ML models, or AI agents.
• Expert proficiency in Python (Pytest) and standard automation libraries.
• Hands-on experience with LLM evaluation frameworks and ground truth dataset management.
• Proficiency in performance engineering and data validation using SQL and data quality tools.
• Strong experience integrating quality gates into GitLab CI/CD pipelines.
• Deep understanding of modern QE practices including Shift Left and Test Pyramid.

Nice to have

• Background in Defense, Aerospace, or highly regulated industries.
• Familiarity with IV&V (Independent Verification and Validation) processes.

Culture & Benefits

• Opportunity to work on cutting-edge AI transformation projects in the defense and enterprise sectors.
• Engagement in high-stakes systems engineering and formal verification processes.
• Professional environment focused on reliability and rigorous quality standards.