About the role
IDT Technologies is seeking a Senior QA Engineer to own quality for an AI voice virtual agent. You will test end-to-end across LLM, ASR, TTS, and telephony, designing non-deterministic evaluation strategies and robust automation.
Role involves provisioning SIP devices, analyzing traces, and driving CI/CD with Jira, Playwright, and Jenkis/GitHub Actions. Remote-friendly with a focus on international collaboration and high-quality release delivery.
Embed fully within a Scrum team and maintain associated artifacts. Own the end-to-end test strategy for the AI voice virtual agent across the LLM, speech, and telephony layers. Design test scenarios for complex conversational behavior, including multi-turn dialogues, interruptions, and escalations. Build and maintain an evaluation framework for non-deterministic output and regression detection. Adversarially test the agent against hallucinations, grounding issues, and data leakage. Prepare and provision SIP devices, endpoints, and call scenarios for automation. Diagnose failures across LLM, ASR, TTS, media path, and routing layers. Extend automated coverage and integrate test suites into CI/CD as release gates. Define and track core quality metrics (task completion, containment, intent accuracy, latency). Maintain documented test coverage in the TMS with traceability to requirements. Communicate quality risks to stakeholders with supporting data. 5+ years in QA with ownership of a complex, integrated product. Practical experience testing SIP/VoIP systems including SIP traces or packet captures. Creative test design for live scenarios and probabilistic outputs. Proficiency in Python for automation. Experience with Jira and formal test management tools. Fluent English with overlap to US Eastern business hours. Remote work opportunity B2B Employment (gross) Competitive salary with annual reviews Long-term growth opportunity Professional training support Referral program Company-supported English classes