Legal questions cannot be answered dependably by loading a few PDFs into a vector database and asking the latest language model to write over them. The decisive work happens earlier: Are the relevant decisions present in the corpus? Does search retrieve the correct judgments for a specialist question? And do practising lawyers assess the result in the same way as a technical metric?
I developed an internal research and answer system for a large German law firm that makes published German court decisions searchable. The corpus included decisions from the Federal Court of Justice and several higher administrative courts. Two practising lawyers contributed reference cases and specialist evaluation.
The first measurable version achieved approximately 60 percent in this project-specific assessment by lawyers. After building better training data and a long iterative fine-tuning and evaluation process, the result exceeded 90 percent. This figure is not a universal legal-AI benchmark. It describes the task and assessment method used in this project. That is precisely what makes it useful: it was not copied from a public leaderboard but measured against the intended legal workflow.
The real product started with the corpus
Retrieval cannot provide dependable evidence without complete and properly processed sources. Court decisions do not arrive as one uniform training dataset. Publication routes, metadata, HTML structures and document formats differ. Decisions may be updated, referenced in several locations or accompanied by headnotes of varying detail.
Crawling was therefore a core system component rather than a one-off import. For the selected courts and sources, crawlers needed to:
- detect newly published decisions,
- retrieve documents and metadata reproducibly,
- normalise court, date, docket number and document type,
- consolidate duplicates and alternative references,
- divide full text into stable sections,
- expose failures, gaps and structural source changes.
“Finding all cases” did not mean possessing every decision ever issued in Germany. It meant verifiable coverage of the relevant publicly accessible sources within the defined subject and court scope. This limitation matters: a system must not conceal missing sources behind confident language.
The data pipeline therefore stored provenance and retrieval status alongside content. Later, we could distinguish between a judgment that did not fit and one that was absent from the corpus. That distinction is fundamental in legal research.
An answer system begins as a search system
The generative model is the most visible component, but retrieval determines which evidence it can see. If the correct decision is absent from the top results, a stronger prompt and more fluent language model can only help so much. In the worst case, the system writes a plausible answer based on the wrong cases.
We therefore treated the technical stages separately:
- Prepare the specialist question for retrieval.
- Use an embedding model to place the question and decision passages in a vector space.
- Apply filters such as court, period or document class where the task requires them.
- Optionally assess the most relevant candidates again.
- Only then provide selected sources to the answer model.
- Display the answer and concrete references together.
This separation made failures diagnosable. When an answer was wrong, we could establish whether the relevant decision was missing, the embedding model ranked it too low, a later stage displaced it or the language model interpreted a correct source incorrectly.
Two lawyers turned examples into training data
Automatically generated questions would have produced volume quickly, but they would not reliably represent how future users reasoned. Legal questions contain abbreviated facts, specialist vocabulary, implicit jurisdiction and nuances that can become unnaturally tidy in synthetic data.
Collaboration with two practising lawyers created dependable specialist references instead. For representative questions, they assessed which decisions were relevant and whether the retrieved sources supported a useful answer. Disagreement was not useless noise. It identified cases where the task, corpus or expected outcome needed a more precise definition.
This work produced several kinds of data:
- a question and matching decision or passage as a positive pair,
- similar-sounding but professionally irrelevant decisions as negative examples,
- particularly difficult candidates as hard negatives,
- questions for which the corpus provided no dependable basis,
- and a separate evaluation set for honest comparison.
The embedding model did not learn “German law” in the abstract. It learned a narrower, testable task: for questions from this workflow, relevant published decisions should rank ahead of semantically similar but unsuitable texts.
Why the base model achieved about 60 percent
A general German or multilingual embedding model already represents many linguistic similarities. Legal relevance does not always follow the strongest lexical similarity. Two judgments may use identical terminology while deciding different legal questions. Conversely, a relevant decision may use language that differs from the query.
The first evaluation exposed this gap. The system often found documents about the same general topic but not consistently the cases expected by the lawyers. Simply returning more results might have improved recall while feeding more irrelevant context into the answer model. The goal was not “somewhere among the first hundred results”, but a ranking useful in professional work.
Public benchmarks help select a base model, but they do not replace a task-specific retrieval test. The MTEB benchmark itself shows substantial variation across tasks and found no single method that dominated every embedding task. German legal research is more specific again than aggregate performance on a general benchmark.
The path from 60 to over 90 percent
Improvement did not come from one magic hyperparameter. It came from a loop of data analysis, training, retrieval and specialist review.
After each run, we did more than count wrong results. We examined the failure type:
- Was the correct decision present in the corpus?
- Was the question too broad, or did it contain the decisive facts?
- Did a lexically similar but legally irrelevant case outrank the correct one?
- Was a relevant long judgment segmented poorly?
- Could court or time period provide a useful filter?
- Did the training data contain contradictory positive and negative examples?
Hard negatives were particularly useful: decisions that appeared linguistically and topically close but which the lawyers judged irrelevant to the particular question. They force a model to learn finer boundaries than keyword similarity. The current Sentence Transformers hard-negative-mining documentation describes this data shape and how close candidates can be turned into training material.
Training, development and test cases remained separate. Otherwise, the model might have seen the same questions or near-identical decisions during training, turning apparent improvement into leakage rather than generalisation. The score above 90 percent was measured on held-out, specialist-reviewed cases.
Small hardware was not a blocker
The project was built in an early GPT-4 era, using comparatively modest local hardware for data processing, experimentation and fine-tuning. This was practical because we were not training the entire generative system from scratch. We adapted an existing embedding model suitable for German to a concrete retrieval task.
Modest hardware encourages discipline:
- Select model size and sequence length according to the task.
- Fit batches, gradient accumulation and caching to available memory.
- Reuse embeddings and candidate lists instead of recomputing every comparison.
- Validate data and loss with small experiments before a long run.
- Compare every checkpoint against the same evaluation set.
The current Sentence Transformers training documentation supports this workflow much more conveniently through trainers, evaluators and several loss functions. Base models, multilingual embeddings, hard-negative tooling and efficient training methods have also advanced since the original project.
I therefore expect that the same use case could now be delivered faster and probably at a higher ceiling with stronger starting models and tooling. That remains a hypothesis rather than a retroactively measured figure. It becomes evidence only when modern candidates are evaluated against the same lawyer-reviewed test set.
Why over 90 percent does not mean automatically legally correct
The project score assessed whether the system retrieved professionally appropriate cases and a useful research basis for the defined questions. It does not mean that every generated legal statement is correct, that every German judgment is present or that legal review can be removed.
The production system therefore retained several boundaries:
- References remained visible beside the answer.
- Users could open the underlying decision.
- Missing evidence was not meant to be replaced by unconstrained generation.
- Access and processing followed the internal use context.
- Specialist assessment and approval remained with practising lawyers.
This distinction is what makes such a system professional. The embedding model accelerates selection of relevant evidence. It does not make legally binding decisions.
From experiment to internal tool
Model improvement still left conventional product work. Crawlers needed monitoring, new decisions required indexing, failures needed logging and access controls needed enforcement. Model version, index and corpus state had to stay compatible. A new source or segmentation rule could not be allowed to degrade established quality silently.
The resulting system is now used internally by a large German law firm. For confidentiality, I do not identify the firm or publish internal questions, case data or model artefacts. The technical and methodological work can still be described: source acquisition, data pipeline, lawyer-curated training and test data, fine-tuning, measured retrieval improvement and integration into a usable system.
The general process for ongoing quality control is described in evaluating RAG systems with real evals. This project provides concrete evidence for why retrieval, answer generation and specialist assessment need to remain distinguishable.
Conclusion
The most important model improvement in this project did not begin with larger hardware. It began with a controlled corpus and two lawyers who translated relevance into testable training and evaluation data. That increased project-specific, specialist-rated performance from approximately 60 to over 90 percent.
This is the kind of AI work I deliver as an AI and LLM freelancer: acquire data sources, make quality measurable, adapt models deliberately and integrate the result into a secure production workflow. If you need specialist search, an internal answer system or better retrieval for an existing RAG product, you can describe the use case confidentially.