LLM Evaluation Engineer (all genders)
Remote (EU time zones) / Hybrid (Germany)
Full-time · Part-time
Open until 30 Nov 2026
About the role
If you can’t measure it, you can’t ship it. You build the evaluation infrastructure that tells us – and our clients – whether a model, prompt or agent is actually good enough for production.
What you’ll do
- Build automated eval pipelines for LLM features and agents (offline + online).
- Design metrics, golden datasets and LLM-as-judge setups – and validate them against human ratings.
- Run regression tests on every model, prompt or tool change.
- Turn human trainer feedback into scalable, versioned datasets.
- Visualise results so product teams can make fast, informed decisions.
What you bring
- Strong Python and data skills; solid statistics fundamentals.
- Hands-on experience with LLM APIs, prompt engineering and evaluation.
- Experience with experiment tracking and CI/CD.
- Pragmatic, quality-obsessed and good at explaining results.
- Very good English.
Nice to have
- Experience with fine-tuning or RLHF pipelines.
- Familiarity with Inspect, OpenAI Evals, Promptfoo or similar.
What we offer
- Remote-first with flexible hours – and an office whenever you want a desk.
- Premium AI tooling on us: Claude, ChatGPT, Codex, Cursor & co. – no budget discussions.
- Real ownership: what you ship reaches 15,000+ customers and 45M+ end users.
- An international team across Germany, India, Australia, Estonia and Ecuador.
- Learning budget, conferences and our own events – grow as fast as the field does.
- ISO 27001 & ISO 9001 processes: structure without bureaucracy.
Tools & stack
PythonPandasClaude APIOpenAI APIPromptfooGitHub ActionsPostgres
Apply now
Apply for this role
LLM Evaluation Engineer (all genders) · jocapps GmbH
Takes about 3 minutes. CV or LinkedIn profile is enough.
