Hiring: Data Scientist Company: Programmers.io Location: Louisville, KY Job Posted Time: 2026-09-02 18:02:23 Employment Type: Hybrid Target Skills & Keywords : Compliance, LLM, Machine Learning, Python, SAFe About the job Experience: •3+ years of experience in data science, machine learning, analytics, or AI model evaluation. Required Skills: •We are seeking a Data Scientist to help build and maintain a robust evaluation framework for conversational AI systems. This role will focus on developing automated AI quality measurement solutions, including LLM-as-a-Judge systems, to assess hallucination rates, intent accuracy, transcript quality, and responsible AI metrics at scale. •As a key member of our AI Quality team, you will work closely with annotation specialists, product teams, and governance stakeholders to ensure our AI experiences are accurate, safe, compliant, and customer-centric. •Design and develop LLM-as-a-Judge evaluation frameworks for conversational AI systems. •Build automated monitoring solutions to evaluate millions of voice and chat-based customer interactions. •Collaborate with user experience teams to identify core metrics and evaluation criteria. •Validate model performance against human-labeled datasets using statistical best practices. •Measure and report precision, recall, false-positive rates, and false-negative rates. •Develop statistical sampling methodologies and quality measurement frameworks. •Continuously calibrate and improve evaluation models as production AI systems evolve. •Analyze hallucination rates, intent classification accuracy, transcript quality (WER), guardrail effectiveness, vulnerability to jailbreak attempts, and fairness metrics. •Create dashboards and reporting to support governance, legal, compliance, and Responsible AI reviews. •Collaborate with annotation teams to improve label quality and gold-standard datasets. •Support future multilingual AI evaluation initiatives. •Master's degree or PhD in Data Science, Statistics, Computer Science, Machine Learning, or related field, or equivalent experience. •3+ years of experience in data science, machine learning, analytics, or AI model evaluation. Interested candidates, please apply directly through the job posting on company's career page or try via AI auto apply on this platform. Don't miss this opportunity to join a forward-thinking team!