Fine-tuning embedding models with domain data: from expert judgement to training pairs

Fine-tuning an embedding model initially sounds like a model task. In production search systems, it is primarily a data and evaluation task. A training run only becomes useful after defining which texts should be close for a given query, which apparently similar documents are specifically unsuitable and how improvement will be measured independently.

This distinction matters because a general leaderboard does not represent a company's specialist task. Product search, support, patent research and legal case research can use the same embedding model while producing completely different failures. Fine-tuning is valuable when domain-specific examples can describe those failures more precisely.

That is how I built an internal answer system for a large German law firm. Two practising lawyers assessed questions and relevant court decisions. A base model suitable for German started at approximately 60 percent in the project's evaluation; after data work and iterative fine-tuning, specialist-rated quality exceeded 90 percent. This article explains the transferable method behind that result.

Fine-tuning starts with a retrieval definition

“Find similar text” is too vague for training. First define which two object types are being compared. In asymmetric search, one side may be a short question and the other a long document passage. In duplicate detection, both sides have a similar shape. These tasks do not necessarily require the same model configuration or evaluation.

A useful definition answers:

• What does the user submit?
• What is the smallest useful retrievable unit?
• Can one question have several correct documents?
• Which results need to appear among the first positions?
• Which metadata may filter before or after vector search?
• Which professionally similar results would still be wrong?

In the legal system, a question had to retrieve relevant decisions rather than texts containing the same legal vocabulary. This made the task asymmetric: short, fact-oriented questions were ranked against longer, formally written court decisions.

The MTEB benchmark deliberately separates retrieval, semantic textual similarity, clustering, reranking and other tasks. Its findings also show that no single embedding method dominates every type of task. A strong aggregate score identifies candidates; it does not accept a product.

Build a gold set before training data

Before examples are used for training, the project needs a small untouched test set. Otherwise, the score may rise simply because the model recognises familiar questions more effectively. The gold data should contain real or realistic queries and be judged by the specialists responsible for eventual quality.

One case may contain:

Version the test set and do not quietly reuse it as training material. With small datasets, variants of the same underlying case must not be split between training and testing either. The model could otherwise recognise wording or near-identical documents without generalising to new cases.

Assessment by two lawyers was particularly valuable in the law-firm project. It exposed places where technical similarity and legal relevance diverged. When the specialists disagreed, we did not automatically create a majority label. We first established whether several decisions were defensible or the question lacked necessary context.

Positive pairs are the start, not the complete dataset

The simplest training shape associates a query with a matching text:

{
  "query": "Which requirements apply in this situation?",
  "positive": "Relevant, publicly available passage from a decision"
}

These pairs teach the model to bring matching content together. They say little about the confusions that are expensive in the product. If training uses only easily recognised positives, training loss can improve while the ranking of difficult candidates remains nearly unchanged.

Positive data should not blindly reproduce the natural frequency distribution. Common subjects would overwhelm rare but important queries. Case groups are more useful: frequent standard questions, rare specialist questions, short and long queries, multiple relevant documents and cases without a dependable result.

Document segmentation is also a data decision. A complete judgment may be too long and contain several subjects; an individual sentence may lose its reasoning and context. Evaluate segmentation strategy, headings, headnotes and metadata together with the model.

Hard negatives carry specialist knowledge

A hard negative is a document that appears similar to the query and ranks highly under the base model, but is not professionally suitable evidence. This is often where expert feedback provides the greatest value.

In legal research, two decisions could concern the same statute, use similar language or describe neighbouring fact patterns while answering different legal questions. A randomly chosen negative document would be too easy for the model. A similar-sounding but incorrect case marked the specialist boundary it needed to learn.

The training example becomes a triplet:

{
  "anchor": "Specialist question from the intended workflow",
  "positive": "Passage judged relevant by specialists",
  "negative": "Similar passage that is unsuitable for this question"
}

Hard negatives can be mined from the current model's incorrect high-ranked results and then reviewed by specialists. The Sentence Transformers hard-negative-mining utility supports finding nearby candidates and producing pair or triplet datasets.

Automated mining must not accidentally label positive documents as negative. The risk is particularly high for questions with several defensible sources. Candidate selection therefore needs margins, known-positive lists and specialist sampling.

Match the loss and batch to the data shape

The loss function defines which geometric relation the model optimises. Contrastive methods are common for query-document pairs: the positive document should be closer to its query than other documents in the batch or explicit negative examples.

Three practical issues matter:

• Small batches provide fewer implicit negatives.
• Larger batches consume more memory and can contain false negatives.
• Overly aggressive hard negatives can destabilise training or push valid alternatives apart.

The current Sentence Transformers loss overview covers Multiple Negatives Ranking Loss, cached variants and explicit hard negatives. The appropriate choice still follows from data shape and hardware rather than a universal recipe.

A small pilot run should first prove that data loads correctly, token lengths are understood and metrics are calculated properly. Only then is a long training run justified. Perfect training loss with stagnant test retrieval is not progress.

Evaluate retrieval with several metrics

“Accuracy” needs explanation in search. Depending on the project, it may mean that at least one relevant result appears within the top k, that the first result is relevant or that specialists consider the entire result list useful.

A report should therefore accompany the project-specific score with technical metrics:

• Recall@k: How many expected relevant documents appear among the first k results?
• Precision@k: What proportion of those results is relevant?
• MRR: How early does the first relevant result appear?
• nDCG: How closely does the ranking reflect graded relevance judgements?
• Abstention quality: Does the system remain controlled when evidence is absent?

In the law-firm project, the lawyers' judgement was the decisive product metric. The increase from approximately 60 to over 90 percent refers to that project-specific assessment framework, not the universal legal correctness of a model. Individual technical metrics helped explain changes but could not replace specialist review.

Break results down by case group as well. A strong average can conceal regression in a rare but business-critical subject. A new model is only released when agreed core groups retain their minimum quality.

Version corpus, model and index together

A fine-tuned model does not remain good automatically when data changes. New documents, a different segmentation strategy or corrected metadata alter the candidate set. At least four versions therefore belong to every reproducible run:

• raw sources and acquisition state,
• normalised corpus and segmentation logic,
• model checkpoint and encoding parameters,
• index and filter configuration.

When new content is crawled regularly, the pipeline needs quality controls: unexpectedly low document counts, empty full text, changed HTML structures, unusual duplicate rates or missing metadata. Model training cannot repair a broken source.

Embeddings must also be recreated following relevant model or preprocessing changes. Queries and documents need compatible model versions and the intended prefixes or prompts. Mixed indexes cause failures that look like weak model quality.

Use modest hardware effectively

Domain-specific fine-tuning does not automatically require a large GPU cluster. Model size, sequence length, batch strategy and the quantity of genuinely useful data determine the requirement. The legal project ran on comparatively modest hardware even though today's convenient tooling and strong embedding base models were not available in the same form.

Useful techniques for limited hardware include:

• a compact base model covering the required language,
• mixed precision where model and hardware support it dependably,
• gradient accumulation or cached contrastive losses,
• precomputed candidates for data analysis,
• short pilot runs before complete epochs,
• early stopping based on the real evaluation metric.

Larger hardware can test more models and parameter combinations in parallel. It cannot replace specialist labels. In many projects, another hour with a domain expert is worth more than another unguided training run.

When fine-tuning is not the first intervention

Weak retrieval does not always require an adapted model. First check that the relevant document exists in the corpus, filters work correctly and segmentation keeps the required context together. Lexical hybrid search, better metadata or a reranker may produce a faster improvement.

Fine-tuning is particularly plausible when:

• the same specialist confusions recur,
• enough expert-reviewed pairs or triplets exist,
• a stable evaluation set can demonstrate improvement,
• privacy or operations favour a compact owned model,
• and expected value justifies ongoing data maintenance.

It is less plausible when the corpus changes fundamentally all the time, only a few ambiguous examples exist or missing documents are the real cause. Repair the information system first in those situations.

The production loop

After initial release, a controlled loop begins. Specialist users flag poor or missing results. Following sanitisation and review, suitable cases become new evaluations or training examples. A challenger model competes against the production version. Only a traceable improvement leads to a new index.

Answer generation is evaluated separately. Better embeddings can retrieve more relevant sources while a language model still summarises them incorrectly. Conversely, a more fluent answer can hide poor retrieval. Evaluating RAG systems with real evals describes this second quality axis in detail.

Conclusion

Embedding fine-tuning translates specialist knowledge into a measurable ranking. Positive pairs identify what belongs together. Hard negatives mark the difficult domain boundaries. An untouched, expert-reviewed test set determines whether the model truly generalises.

The project for the German law firm demonstrates the potential effect: approximately 60 to over 90 percent within the defined lawyer-led assessment, implemented on modest hardware and later used in an internal tool. As an AI and LLM freelancer, I connect crawling, data pipelines, training, evaluation and product integration. If you want to adapt an embedding model with proprietary specialist data—or first identify the most effective improvement—describe the use case without obligation.