Skills : Data Analyst, Python, vLLM,
Builds and maintains the evaluation frameworks that determine whether an AI system is good enough to ship — designing test sets, scoring rubrics, automated graders, and regression suites. A role most product companies currently can't hire fast enough for.
Job Description:
Builds and maintains the evaluation frameworks that determine whether an AI system is good enough to ship — designing test sets, scoring rubrics, automated graders, and regression suites. A role most product companies currently can't hire fast enough for.
Key Skills:
- Eval framework design (golden datasets, LLM-as-judge, human-in-the-loop review)
- Statistics fundamentals for measuring model quality and significance
- Python, data analysis (pandas), eval tooling (Braintrust, LangSmith, custom harnesses)
- Domain knowledge to define what "good" output looks like for a specific use case
- Bias/fairness and safety-aware evaluation design
Experience Required:
- 2–5 years in ML engineering, data science, or QA with an analytical bent
- Experience building or maintaining a test/eval suite for any software system is transferable