About the role
Robots & Pencils is an applied AI engineering firm building AI co-workers that integrate into enterprise operations. This role focuses on owning testing duties across automation and AI, shaping quality strategy, and taking real responsibility from day one.
You will write automated tests, validate AI/ML pipelines, and integrate quality practices into delivery with cross-functional engineers.
Write and maintain automated test scripts for UI, API, and integration testing. Validate ML model outputs for accuracy and consistency against criteria. Test data pipelines and AI workflows with data quality checks. Run automated test suites, analyze results, and log defects in Jira/TestRail. Integrate tests into CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins). Support model evaluation and end-to-end AI workflow testing. Test AI/ML systems including LLM evaluation, RAG, model drift, and safety. Collaborate with developers, ML engineers, and data scientists on quality goals. Participate in sprint ceremonies and quality strategy discussions. Provide clear testing reports and communicate status to the team. 2+ years of experience in test automation or AI/ML testing. Experience with test automation frameworks for UI, API, or ML workflows. Understanding of AI/ML concepts and the ML lifecycle, with the ability to validate model outputs. Experience with CI/CD pipelines and version control. Working knowledge of API testing and data validation. SDLC/STLC and Agile methodologies knowledge. Bug-tracking and test management tools proficiency. Familiarity with AI/ML testing: LLM evaluation, RAG, model drift, safety. Hands-on use of AI-forward tools like Claude and Cursor. Strong problem-solving skills and ambiguity handling.