LLM Evaluation Engineer (all genders)

Remote (EU time zones) / Hybrid (Germany) Full-time · Part-time Open until 30 Nov 2026

About the role

If you can’t measure it, you can’t ship it. You build the evaluation infrastructure that tells us – and our clients – whether a model, prompt or agent is actually good enough for production.

What you’ll do

  • Build automated eval pipelines for LLM features and agents (offline + online).
  • Design metrics, golden datasets and LLM-as-judge setups – and validate them against human ratings.
  • Run regression tests on every model, prompt or tool change.
  • Turn human trainer feedback into scalable, versioned datasets.
  • Visualise results so product teams can make fast, informed decisions.

What you bring

  • Strong Python and data skills; solid statistics fundamentals.
  • Hands-on experience with LLM APIs, prompt engineering and evaluation.
  • Experience with experiment tracking and CI/CD.
  • Pragmatic, quality-obsessed and good at explaining results.
  • Very good English.

Nice to have

  • Experience with fine-tuning or RLHF pipelines.
  • Familiarity with Inspect, OpenAI Evals, Promptfoo or similar.

What we offer

  • Remote-first with flexible hours – and an office whenever you want a desk.
  • Premium AI tooling on us: Claude, ChatGPT, Codex, Cursor & co. – no budget discussions.
  • Real ownership: what you ship reaches 15,000+ customers and 45M+ end users.
  • An international team across Germany, India, Australia, Estonia and Ecuador.
  • Learning budget, conferences and our own events – grow as fast as the field does.
  • ISO 27001 & ISO 9001 processes: structure without bureaucracy.

Tools & stack

PythonPandasClaude APIOpenAI APIPromptfooGitHub ActionsPostgres

Apply now

Apply for this role

LLM Evaluation Engineer (all genders) · jocapps GmbH

Takes about 3 minutes. CV or LinkedIn profile is enough.

AI tools you use regularly
CV & documents
* Required