Forward Deployed Engineer – AI Assurance

About the role

We are looking for a Forward Deployed Engineer specialized in AI Assurance to own the quality of AI‑native applications. You will combine functional testing with AI‑specific evaluation such as accuracy, drift and hallucination, ensuring every AI feature meets rigorous standards before reaching production.

Key responsibilities

• Build test plans and automation for AI‑native application features, covering both functional and AI‑specific aspects.
• Design evaluation harnesses for model and agent outputs, measuring accuracy, consistency and hallucination rates.
• Run regression testing across model, prompt and configuration changes to detect silent quality drift.
• Red‑team AI features for edge cases and adversarial inputs where relevant.
• Build automated evaluation pipelines integrated into CI/CD.
• Partner with AI Architects to define testability requirements before development starts.
• Own the quality gate before any AI feature ships to production.
• Communicate quality risks to delivery leadership in actionable business terms.
• Train delivery teams on AI‑specific testing practices.
• Maintain the evaluation framework, including golden datasets, scoring rubrics, LLM‑as‑judge calibration and versioned benchmarks per use case.
• Define acceptance thresholds for evaluations and gate releases accordingly.
• Develop evaluation engineering tooling – dataset curation, trace capture, offline/online runs and dashboards for teams.
• Instrument production evaluations and drift monitoring, feeding failures back into golden datasets.

Required profile

• 5–9 years of QA/test engineering experience, with at least 2 years focused on AI/ML‑powered features.
• Strong test automation expertise using Python‑based frameworks and CI/CD integration.
• Deep understanding of AI‑specific failure modes such as hallucination, bias, drift and non‑determinism.
• Statistical literacy to interpret model evaluation metrics beyond simple pass/fail.
• Familiarity with red‑teaming methodologies for AI systems.
• Clear, assertive communicator willing to block releases over quality concerns.
• Detail‑oriented and methodical under delivery‑timeline pressure.
• Collaborative yet independent, able to explain quality risk in business‑impact terms.

Required skills

• Python
• Test automation frameworks
• CI/CD pipelines
• OpenAI Evals
• Ragas
• DeepEval
• LangSmith
• Azure AI Foundry evaluations
• Statistical analysis
• Red‑teaming methodologies
• RAG evaluation metrics
• Dataset curation and evaluation dashboarding