QA Engineer Job at Storm3, New York, NY

  • Storm3
  • New York, NY

Job Description

Senior QA Lead (AI Systems)

About the Company

We’re a fast-growing Series A AI startup building software for the life sciences industry. Our platform helps enterprise customers create and review complex, regulated content using AI-powered workflows. We’re working with leading organizations in the space and are building at the forefront of LLMs, agentic systems, and applied AI.

About the Role

We’re hiring our first dedicated QA Lead to own quality across our AI platform. This is a hands-on engineering role focused on building testing infrastructure, evaluation systems, and automation that ensure both product reliability and AI output quality.

You’ll be responsible for everything from LLM evaluations and agent testing to Playwright automation, CI/CD quality gates, and release sign-off. The ideal candidate can write production-quality code, build scalable test systems, and think critically about how to measure quality in non-deterministic AI environments.

What You’ll Do
  • Build evaluation frameworks for LLM and agent-based systems
  • Design regression tests for model, prompt, and workflow changes
  • Create automated testing infrastructure using Python and Playwright
  • Own end-to-end testing across web applications and APIs
  • Develop CI/CD quality gates and automated release workflows
  • Perform manual and exploratory testing on new features
  • Monitor production systems and convert failures into repeatable tests
  • Test agent workflows for failure modes such as tool misuse, looping, hallucinations, and prompt injection
  • Partner with engineering and product teams to establish quality standards
  • Define and scale the QA function as the first dedicated hire
Required Experience
  • Strong Python engineering skills
  • Extensive Playwright and test automation experience
  • Experience building and maintaining CI/CD testing pipelines
  • Hands-on manual and exploratory testing expertise
  • Familiarity with AI, ML, or LLM-based systems
  • Experience designing evaluation methodologies and quality metrics
  • Strong debugging, root-cause analysis, and error investigation skills
  • Ability to operate independently and build processes from scratch
Preferred Experience
  • Experience with LangChain, LangGraph, or agent frameworks
  • Familiarity with LangSmith, Langfuse, Arize, Braintrust, Promptfoo, OpenAI Evals, or similar tools
  • TypeScript and React experience
  • Adversarial testing or red-team experience
  • Backend experience with async Python services and job queues
  • Experience in regulated industries such as healthcare, pharma, or finance
  • MLOps or LLMOps experience
Compensation & Benefits
  • $160,000-$300,000 base salary
  • Meaningful equity package
  • Comprehensive medical, dental, and vision coverage
  • 401(k) with company match
  • Visa sponsorship available
  • Relocation support
  • Unlimited PTO
  • Professional development stipend
Location

New York City 5 days. Relocation assistance available

#J-18808-Ljbffr

Job Tags

Relocation package

Similar Jobs

Jordan Valley Health

Addiction Medicine/Primary Care Physician Job at Jordan Valley Health

 ...qualifies for a $10,000 per contract year sign on bonus* Physician must be board-certified. About Jordan Valley Community Health...  ...primary care specialty. Missouri State License to practice medicine. Current BLS Certification required. Federal DEA number... 

Eyouth

Voice Over Artist Job at Eyouth

Voice Over Artist AStudio Al Araby Cairo Remote A role for a capable professional to work as Voice Over Artist within general operations. The position focuses on delivering reliable day-to-day results, collaborating with relevant teams, and supporting business objectives...

American Consultants

Travel CT Tech - $2,429 per week in Santa Monica, CA Job at American Consultants

 ...Silicon Valley), entertainment (Hollywood), agriculture, and aerospace. Home to the world's oldest and tallest treesbristlecone pines and redwoods. Home to Hollywood film industry and numerous major universities. Requirements Required for Onboarding ~ BLS... 

Poppy Bank

Housekeeper (Room Attendant) - WoodSpring Suites Job at Poppy Bank

 ...stay.Pay: $14.00 - $16.00 per hour (based on experience)J&P Hospitality Management, LLCJob DescriptionJob Title: Housekeeper...  ...authorized to work in the United States without current of future visa sponsorship.Employment is subject to E-Verify verification.... 

HYATT Hotels

Housekeeper/Room Attendant ($27.05/hr) Job at HYATT Hotels

 ...At Hyatt, we believe our guests select Hyatt because of our caring and attentive associates who are focused on providing efficient service and meaningful experiences. The Room Attendant is responsible for maintaining the cleanliness of the guest rooms assigned. Previous...