“Which model is best?” rarely leads to good enterprise architecture. The relevant question is which model completes a defined task with sufficient quality, acceptable cost and suitable data protection. In many projects the answer is not one model family, but a controlled hybrid setup.
Start with data classes
Classify the processed information before comparing benchmarks. Public product copy has different protection needs from unreleased source code, employee records, legal documents or details of an unresolved vulnerability.
| Data class | Example | Possible operation |
|---|---|---|
| public | published documentation | external API possible |
| internal | process knowledge without personal data | contractually reviewed API or local |
| confidential | source code, contracts, customer data | local or strongly controlled |
| highly critical | credentials, vulnerabilities, secrets | exclude from model or isolate locally |
Classification does not replace a legal privacy assessment. It creates a technical rule the application can enforce. Users should not need to decide for every request whether a document may leave the controlled environment.
Measure quality on the actual task
Public benchmarks show general capability under defined conditions. They say little about internal abbreviations, document structure or tool contracts in a specific product. Selection therefore needs a project evaluation set.
Useful dimensions include:
- specialist correctness and completeness,
- grounding and appropriate refusal,
- structured output and tool calling,
- latency and throughput,
- context size and memory demand,
- cost per successful task.
A cheap call becomes expensive when people regularly correct it. The strongest model is equally uneconomic when a smaller local model performs a narrow workflow reliably.
What local inference actually provides
Running a model on owned hardware initially prevents prompts from being sent to an external model provider for inference. Security still depends on the complete operating system:
- Model artefacts and containers come from traceable sources.
- APIs and administration are not publicly exposed.
- Users, services and projects have separate rights.
- Prompts, outputs and logs are retained according to data class.
- Tools and data sources have their own authorisation.
- Updates, monitoring and capacity limits are defined.
Local inference is not a single privacy switch. It is a system that needs operation and review.
Use frontier models deliberately
External frontier models often provide strong general capability, large contexts and new functionality without owning GPU infrastructure. They can accelerate development and processing for public or suitably sanitised work.
Before use, review contract terms, data location, retention, training use, subprocessors and deletion options. API credentials belong in a server-side secret store, never in a browser or mobile application. Cost and rate limits prevent uncontrolled consumption.
Make hybrid routing explicit
A hybrid policy routes work by task and data class. Public summarisation can use an external model. Confidential code review uses local inference. Highly critical credentials are removed before any model sees the request.
The route must be observable. Every run should record the model class, data sources and policy used. A silent fallback from local to external inference is a privacy failure even when the response is correct.
Evaluate owned hardware economically
Local systems involve purchase, electricity, maintenance and capacity planning. In return they offer controlled data paths, predictable use costs and persistent availability for internal work. A high-memory multi-GPU workstation can cover compact and medium models plus parallel evaluation. Systems with large unified memory extend model and context capacity.
The most impressive parameter count is not the goal. Quantisation, batch size, context and required throughput determine practical performance. Measure real tasks before selecting architecture.
A decision matrix instead of model allegiance
Every candidate runs against the same test set and operating limits. Weighting is fixed before the test:
| Criterion | Measurement | Example weight |
|---|---|---|
| Task quality | passed reference cases, with critical errors reported separately | 35% |
| Privacy | permitted data classes and actual data route | exclusion criterion |
| Latency | p50 and p95 per completed task | 15% |
| Throughput | successful tasks per minute at target load | 10% |
| Cost | infrastructure and API cost per successful task | 20% |
| Operations | updates, monitoring, failure route and required expertise | 20% |
Quantised and unquantised variants are separate candidates. An automatic fallback may only use a route with an equal or stricter data permission. This matrix also explains a hybrid decision: not “model A is better”, but “route A meets this task and protection class at this cost”.
For knowledge systems, the RAG evaluation guide supplies quality metrics. For autonomous workflows, securing AI agents adds tool, budget and approval controls.
Sources and further reading
- NIST AI Risk Management Framework
- BSI: Generative AI Models
- OWASP Top 10 for LLM Applications
- European Commission: AI Act
Conclusion
Model choice follows task and protection need. Frontier models can deliver speed, local models provide controlled data paths and a clear hybrid design combines both. Quality and privacy must be measurable technical rules, not promises made during a sales call.