Hiring: AI Quality Engineer Company: Dealstitch AI Location: United States Job Posted Time: 2026-09-10 10:13:11 Target Skills & Keywords : CI/CD, LLM, Python, RAG, REST About the job Experience: •40+ years of combined private-market technology experience. You'll work on problems that matter, with direct access to decision-makers, and see the impact of your work in weeks, not quarters. Required Skills: •Own the evaluation pipelines that gate every change we ship, keeping them automated and running in CI/CD rather than leaning on manual spot-checks. •Keep agents scored at the step level, not just the final answer, across tool selection, planning and reasoning chains, and retrieval quality, so subtle regressions surface before clients do. •Extend the eval toolkit as new failure modes appear: LLM-as-a-judge, RAG faithfulness and relevance, hallucination detection, prompt and agent regression suites, and red-teaming for safety. •Maintain the golden datasets that mirror real production inputs and the tracing and observability that watch our agents once they are live. •Own the quality bar for what ships and raise it over time with the rest of the engineering team. •Strong software engineering with Python, and a feel for testing non-deterministic systems, where classic assertion-based tests are not enough. •Fluency with RAG and retrieval quality, plus observability and tracing for agents in production. •A bias for measurement: you define what "good" means, then build the harness that proves it. Compensation: •Bonus: data-pipeline testing experience, or exposure to private markets, financial services, or enterprise data Interested candidates, please apply directly through the job posting on company's career page or try via AI auto apply on this platform. Don't miss this opportunity to join a forward-thinking team!