Hiring: Senior Software Engineer - Model Evaluation & AI Systems Company: Deepgram Location: United States Job Posted Time: 2026-09-10 10:21:12 Target Skills & Keywords : CI/CD, Cloudflare, Grafana, LLM, Python, RAG, React Native, Rust, Twilio About the job Experience: •000 years of audio and transcribed more than 1 trillion words. There is no organization in the world that understands voice better than Deepgram. •5+ years of professional software or QA engineering experience, with a track record of shipping test infrastructure or evaluation systems (senior candidates with significantly deeper experience welcome). Required Skills: •Deepgram is looking for a Senior Software Engineer - Model Evaluation & AI Systems to join the team responsible for validating the quality of our speech, audio, and multilingual models before they reach customers. •Define and build evaluation methodologies for Deepgram's models, spanning speech-to-text, text-to-speech, and emerging LLM, RAG, agent, and multimodal systems. •Design, build, and maintain automated evaluation pipelines across batch and streaming (e.g. WER, runaway/hallucination detection, latency and time-to-first-byte), with a focus on correctness, reproducibility, and ease of adoption. •Build scalable, reproducible evaluation infrastructure — harnesses, orchestration, and result-aggregation pipelines — running against production models and, where needed, large GPU clusters. •Translate Research benchmarks and expected model metrics into automated, enforceable pass/fail gates. •Build and operate canaries and continuous-monitoring systems that detect quality regressions in production before they reach customers. •Partner with DevOps/Infra to stand up ephemeral test environments and results-aggregation infrastructure. •Work alongside Research, model training, inference, and product teams to provide trusted evaluation signals that inform release and optimization decisions. Interested candidates, please apply directly through the job posting on company's career page or try via AI auto apply on this platform. Don't miss this opportunity to join a forward-thinking team!