AI testing Engineer
Posted: 07/14/2026
Job Number: NktsT0dYNWFaSG13TzRmYytRUGRhUT09
Pay Rate: 100000-110000 Yearly USD
Job Description
AI testing Engineer
Experience: 8+ years
100% Remote (Canada & US Both)
Video Testing experience with AWS environment
- We are seeking an AI Testing Engineer to validate and assure quality of AI/ML systems powering real-time contextual advertising on live video streams.
- The role involves testing multi-modal AI outputs (video + text), ensuring accuracy of detected moments, and automating validation workflows.
- This role is critical for native moments validation and system-level QA in a real-time streaming environment.
Key Responsibilities
- Validate AI-generated outputs against labelled datasets and expected outcomes
- Perform content validation for native moments detection across video streams
- Design and implement automated testing frameworks (Python-based)
Build validation harness for:
- JSON parsing and comparison
- Structured output validation using defined schemas
- Conduct multi-modal validation (video, audio, transcript outputs)
- Execute system integration and end-to-end testing
- Track accuracy metrics, defect logging, and root cause analysis
- Automate regression and validation pipelines across datasets
- Collaborate with Applied Scientists to validate LLM and GenAI outputs
- Work within AWS environment using services such as: S3 (data storage).Lambda (processing).CloudWatch (monitoring)
Required Skills & Experience:
- Strong Python scripting for automation and testing
- Experience in AI/ML model validation and QA workflows.
- Hands-on with: JSON parsing, schema comparison Data validation pipelines
- Experience in testing: Multi-modal AI systems (vision + NLP + audio) LLM outputs and generative AI pipelines. Good understanding of: Structured output validation AI accuracy evaluation techniques Familiarity with AWS cloud environment (S3, Lambda, logging/monitoring) Experience with media/video content analysis workflows preferred Ability to handle mixed manual + automated testing approaches