Hiring: Product Owner (AI & Prompt Strategy) Company: Kaleidoscope Innovation Location: Fort Worth, TX Job Posted Time: 2026-09-16 17:45:09 Employment Type: Hybrid Target Skills & Keywords : A/B Testing, AWS, LLM, Product Management, Product Strategy, Quality Assurance, Stakeholder Management About the job Experience: •7 years of Product Management or Product Ownership experience, ideally within mortgage, fintech, or complex customer servicing environments. Required Skills: •Strategy & Orchestration: Lead the overall product vision for conversational AI agents, owning the company-wide persona, brand voice, and overarching prompt strategy. •Iterative Prompt Refinement: Continuously develop, test, and tune complex system prompts, system instructions, and few-shot examples. Optimize these prompts across the AWS tech stack to balance conversational quality, token efficiency, and response latency. •Prompt Effectiveness Tracking & Analytics: Establish robust tracking frameworks to measure prompt and agent efficacy at a granular level. Monitor specialized metrics such as hallucination rates, contextual accuracy, task completion/containment rates, and user drop-off to drive continuous prompt optimization cycles. •Stakeholder Alignment: Serve as the central orchestrator across CDL/Sales, Servicing, and Fulfillment teams to negotiate and refine business requirements, ensuring parallel development without technical bottlenecks. •Testing & Quality Assurance: Define the "Golden Dataset" of test scenarios to feed into engineering’s automated LLM-as-a-judge evaluation pipelines. Oversee structured A/B testing initiatives and data analysis to iteratively optimize agent responses against baseline configurations. •Performance & ROI: Define product strategy and track ROI by aligning AI agent performance to key business KPIs, including Customer Satisfaction (CSAT) and Average Handle Time (AHT). •Cross-Functional Collaboration: Partner with the engineering team to integrate prompt designs with the core infrastructure, latency management pipelines, and real-time safety guardrails. •5–7 years of Product Management or Product Ownership experience, ideally within mortgage, fintech, or complex customer servicing environments. Interested candidates, please apply directly through the job posting on company's career page or try via AI auto apply on this platform. Don't miss this opportunity to join a forward-thinking team!