LLM Evaluator (Model Response Analyst)

About the role

We are looking for a detail‑oriented LLM Evaluator to assess and improve the performance of large language models. Working remotely, you will analyze AI‑generated text for factual accuracy, coherence, safety, bias, and alignment with our guidelines.

Key responsibilities

• Evaluate and rank model outputs using complex rubrics covering factuality, coherence, safety, instruction‑following, and creativity.
• Compare multiple responses to the same prompt and justify the preferred output.
• Provide concise feedback to modeling and training teams on recurring failure patterns.
• Design adversarial prompts to expose biased or harmful behavior and help patch safety gaps.
• Collaborate with QA to refine evaluation guidelines and calibrate scoring across a global team.
• Investigate underperforming outputs, hypothesize root causes, and flag novel model behaviors.

Required profile

• Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP quality assurance, or cognitive science.
• Bachelor’s degree in Computer Science or a related field.
• Proven ability to craft prompts that test model limits and to explain why outputs are good or bad.
• Experience with Reinforcement Learning from Human Feedback (RLHF) data collection and evaluation calibration.

Required skills

• Prompt engineering
• Reinforcement Learning from Human Feedback (RLHF)
• A/B testing concepts applied to AI
• Data annotation and cleaning for LLM fine‑tuning