Stockholm MLOps #40: The Production Decision Starts With Constraints

October 21, 2026 | Reaktorhallen, Stockholm
Insights by Stockholm MLOps

An embedding model can perform well in English and still struggle badly when a user asks the same kind of question in Finnish or Swedish.

At Stockholm MLOps #40, ConfidentialMind showed how large that difference could be. On a sampled FIQA retrieval benchmark, English queries against English documents scored 0.582 nDCG@10. Finnish queries against the same English document set fell to 0.285, while Swedish queries scored 0.370.

The interesting part was not simply that multilingual performance differed.

The response was also revealing.

Instead of rebuilding the entire retrieval system or replacing the underlying embedding model, SampoTron adapted the query side of the pipeline and preserved the existing English document index. The reported result was a 20.9 percent relative improvement in mean nDCG@10 across the tested cross lingual retrieval cases.

That example captured something important about Stockholm MLOps #40.

The event was called Orchestration vs Model Selection, but the evening did not produce a winner.

Instead, it showed that production AI decisions tend to begin with constraints.

Sometimes the constraint is model capability.

Sometimes it is language.

Sometimes it is whether a particular model or execution environment is even allowed to receive the data.

Sometimes the problem is not model selection at all, but how software is specified, verified, deployed and maintained.

And sometimes the important decision sits another layer down, where an interactive inference service and a tightly coupled scientific workload require fundamentally different infrastructure schedulers.

Across ConfidentialMind, CHI Software, Hopsworks and Sweden AI Factory, the same lesson appeared from different directions:

Before asking which model or orchestration layer is best, you have to understand what the workload actually requires.

Summary: Key Insights

Model quality still matters, especially when the task becomes specific. ConfidentialMind showed a substantial retrieval gap between English, Swedish and Finnish queries even when the underlying document collection remained English. Model selection has not disappeared as an engineering problem.

Adaptation can sometimes be better than replacement. SampoTron improved the part of the retrieval pipeline that was failing while preserving the existing document index and the original path where it already worked. That is a more surgical production decision than replacing an entire model stack.

The highest performing model is not always an eligible model. CHI Software's architecture placed data boundaries and organizational policy ahead of ordinary optimization. Cost and quality only matter inside the set of execution paths an organization is actually permitted to use.

“Orchestration” described several completely different engineering problems during the evening. Policy aware execution, multilingual query routing, software lifecycle automation and infrastructure scheduling all involve orchestration, but they operate at different layers and solve different problems.

Automation does not remove verification. Hopsworks' software factory extended beyond code generation into requirements, specifications, testing, deployment, monitoring and maintenance, while explicitly retaining human review before implementation.

Infrastructure should follow workload behavior. Sweden AI Factory showed why a stateless inference service and a tightly coordinated HPC workload have different scheduling requirements. The meaningful question is not simply Slurm or Kubernetes. It is what kind of work is being scheduled.

The room itself reinforced the importance of evaluation and constraints. When 53 respondents were asked what determines which model handles a task, 17 selected accuracy on internal evaluations and 16 selected data and compliance boundaries. Together, those accounted for roughly 62 percent of responses.

The broader lesson from #40 is therefore not that orchestration has replaced model selection. It is that production AI is becoming a sequence of constrained decisions, each made at a different layer of the system.

What This Event Was Really About

The title Orchestration vs Model Selection suggests a contest.

The event itself suggested something more complicated.

ConfidentialMind began with a measurable model behavior problem. The same embedding model that produced reasonable English retrieval performance degraded substantially when queries arrived in Finnish or Swedish. That is not an orchestration abstraction. It is a concrete capability difference that has to be measured.

But the resulting engineering decision was not simply to choose a different model.

SampoTron introduced a query side adaptation path. That preserved the existing English index and retained the original embedding route where it still worked. The important architectural idea was selective intervention rather than wholesale replacement.

CHI Software then approached selection from a completely different direction. Its sovereign execution architecture asked which models, tools and execution environments were permitted to process particular information. A model could be excellent for the task and still be unavailable because organizational policy prohibited sending the data there.

Hopsworks moved even further away from model routing. Jim Dowling's presentation framed orchestration around the entire lifecycle of creating and operating ML systems. Requirements, specifications, implementation, verification, infrastructure, deployment and monitoring were coordinated through a software factory in which agents performed substantial work but humans still reviewed the specification before implementation.

Sweden AI Factory then shifted the conversation into infrastructure. Daniel Medeiros distinguished interactive inference serving from traditional HPC and mixed scientific AI workflows. A stateless vLLM service can be replicated and exposed through Kubernetes, while tightly coupled computational jobs have very different requirements around scheduling and coordination.

These are not competing implementations of the same idea.

They are four different decisions happening at four different levels.

That is why the word orchestration needs to be treated carefully. It can describe routing a query, enforcing policy, managing a software development lifecycle or scheduling compute resources. A production system could use all four without any of them replacing the others.

The more useful question after #40 is therefore not whether orchestration matters more than model selection.

It is:

What decision are we actually trying to make, what constraint governs that decision, and what evidence tells us the decision is correct?

1. A Benchmark That Exposed the Language Gap

ConfidentialMind provided the evening's clearest quantitative example of model behavior affecting architecture.

The sampled FIQA benchmark contained 194 question identities and 14,330 English passages. English queries against those passages achieved 0.582 nDCG@10. Finnish queries against the same English material fell to 0.285, while Swedish queries reached 0.370.

Those numbers should not be generalized into a claim about all Finnish or Swedish retrieval systems. This was a particular benchmark, model and evaluation setup.

But the gap itself matters.

A system can look strong when measured in English and still behave very differently when the user changes language.

That creates a practical problem for Nordic applications. A model leaderboard or aggregate multilingual benchmark can hide a weakness that becomes obvious once the actual language pair used in production is measured.

SampoTron's response was based around NVIDIA Nemotron 3 Embed 1B and query side adaptation. The reported result was a 20.9 percent relative mean improvement in nDCG@10 for the cross lingual benchmark.

The lesson is not that routing always beats model selection.

It is that you cannot select or route intelligently until you have measured the failure you are trying to fix.

2. When Adaptation Beats Replacement

The most interesting architectural feature of SampoTron may be what it avoids changing.

A retrieval system with an established document index represents an operational investment. Replacing the embedding model can require regenerating embeddings across the entire collection, moving data and validating the new retrieval behavior.

SampoTron instead adapts the query side.

The slide summarized the principle clearly:

“Measure the language pair. Adapt the query, reuse the index. Keep the original path where it works.”

ConfidentialMind, SampoTron slide

That last part is important.

The architecture does not assume that every query needs intervention. It preserves the known good path and applies adaptation selectively.

This is a useful production pattern because optimization does not necessarily mean replacing everything with the newest component.

Sometimes the better system is the one that changes only the layer responsible for the observed weakness.

That approach has costs of its own. Language detection, routing logic, adapter maintenance and evaluation across language pairs all become new operational responsibilities.

But it introduces a useful design question:

Can we improve the failing part without rebuilding the part that already works?

3. The Model You Are Allowed to Use

CHI Software's presentation introduced another dimension of model selection that is often lost in benchmark comparisons.

The best model is not always the model an organization is allowed to use.

Ihor Tanenkov's sovereign execution architecture placed organizational eligibility and data boundaries before ordinary considerations such as price or raw model capability.

The system could potentially use custom models, open weight models and deterministic tools across different steps of a task. But execution first had to remain inside approved organizational and data boundaries.

This changes the order of the decision.

A conventional model router might ask:

Which model gives us the best combination of quality, latency and cost?

A policy aware execution layer has to ask another question first:

Which execution paths are permitted to receive this data at all?

Only after that constraint has been applied does the ordinary optimization problem begin.

This turns sovereignty into something more operational than simply deciding where a model is hosted. In a multi step system, different parts of the task may carry different data, use different tools and require different execution environments.

Control therefore becomes part of runtime architecture.

The important limitation is that the material from #40 did not quantify the additional engineering cost, latency or operational complexity of maintaining these policies. Nor did it provide measured policy violation rates or failure incidents.

The architecture is compelling.

The economics still need measurement.

4. There Was No Single Orchestration Problem

One of the most useful things #40 did was expose how imprecisely the word orchestration is often used.

CHI Software was orchestrating permitted execution paths.

The decision involved organizational policy, data classification, models, deterministic tools and potentially different execution environments.

ConfidentialMind was orchestrating query representation.

Its router decided when language adaptation should be introduced before retrieval.

Hopsworks was orchestrating the software lifecycle.

Requirements, specification, coding sessions, verification, deployment, infrastructure and monitoring were coordinated as part of a repeatable factory process.

Sweden AI Factory was orchestrating compute resources and workloads.

Slurm, Kubernetes, vLLM services and mixed scientific workflows sit at a completely different layer of the architecture.

These systems share a general idea: something is deciding how work should proceed.

But the similarity ends quickly.

They maintain different state.

They make different decisions.

They fail differently.

They optimize different outcomes.

And they introduce different operational complexity.

That matters because saying an organization “needs orchestration” is almost meaningless until we know what is being orchestrated.

5. A Software Factory Needs More Than a Coding Agent

Hopsworks pushed orchestration into yet another domain.

The problem was not which model should answer a request.

It was how an ML system moves from a requirement into working, verified and maintainable software.

Jim Dowling presented Hopsable and a software factory architecture incorporating requirements, specifications, coding sessions, infrastructure, verification, deployment and monitoring. The slides showed GitHub, JIRA, Claude Code sessions, Hopsworks and infrastructure tooling working within the broader process.

The key distinction is that generating code is only one stage.

A coding agent can produce software without proving that the software corresponds to the actual requirement, behaves correctly after deployment or remains maintainable over time.

One of the slides made the human role explicit:

“People stay in the loop; they review the spec before anything is built…”

Hopsworks software factory slide

That is not evidence that automation failed.

It is evidence that human review is being designed into the automated process.

This is a more useful way to think about human involvement in production AI. The question is not whether a human remains somewhere in the workflow. It is where human judgment creates the most leverage.

In the Hopsworks example, that point sits before implementation, where the specification itself can be reviewed.

The presentation made substantial productivity claims around the software factory approach, but the material available from #40 does not provide a sufficiently controlled comparison to turn those claims into a general productivity benchmark.

The architecture is the stronger Insight.

6. Why Workloads Choose Schedulers

The Sweden AI Factory presentation introduced a deceptively simple infrastructure question:

When should you use Slurm, and when should you use Kubernetes?

The answer was not that one is modern and the other is legacy.

It depended on the workload.

Daniel Medeiros used vLLM inference serving as an example of a stateless service where replicas can be exchanged. Kubernetes fits naturally because it provides service endpoints, scaling and fault tolerance around that deployment model.

But an HPC workload can look very different.

A tightly coordinated computational job may require multiple resources to operate together, with different assumptions about communication, scheduling and job completion.

Mixed scientific AI workflows complicate the picture further. The ECMWF forecasting example combined observation processing, model computation and forecast generation into one operational pipeline.

The inference slide also made clear that infrastructure decisions eventually surface as user experience and economics:

“We have to worry about user-QoS (p=95, p=99) and about our money.”

Sweden AI Factory slide

That is the production problem in one sentence.

A scheduler is not selected because one platform wins a popularity contest.

It is selected because the workload has particular requirements around latency, resource coordination, availability and cost.

7. Evaluation Is Becoming an Operational Discipline

Evaluation appeared repeatedly during #40, but in several different forms.

ConfidentialMind used evaluation diagnostically. The retrieval benchmark revealed that a system behaving acceptably in English had a meaningful cross lingual weakness.

Hopsworks treated verification as part of the software delivery lifecycle. Generated software still needed to be checked against its specification.

The audience then reinforced the same concern from another direction.

When 53 Mentimeter respondents were asked what determines which model handles a task, 17 selected accuracy on internal evaluations, making it the most frequently chosen answer. Data and compliance boundaries followed with 16.

That is significant because it suggests this practitioner audience is not primarily choosing models from public leaderboards or price lists.

They want to know whether the system works on their own task, with their own constraints.

There is an important boundary to preserve here.

#40 did not demonstrate a production control plane continuously evaluating models and dynamically routing every request based on live scores.

That would be a much stronger claim.

What the event did show is that evaluation increasingly sits close to important production decisions. It informs model selection, exposes task specific weaknesses and becomes part of verification before software moves forward.

Insights by Stockholm MLOps: Signals From the Room

The Mentimeter results provide a useful second evidence stream because they tell us what practitioners in this particular room reported about their own environments.

They should not be treated as industry survey data. Response counts varied significantly between questions, and some screenshots may have captured polls before every participant had responded.

But the patterns are still useful.

Internal Evaluation and Compliance Are Driving Model Decisions

The question most directly connected to the event asked what determines which model handles a task.

Among 53 recorded responses, 17 selected accuracy on internal evaluations and 16 selected data and compliance boundaries.

Together, those two categories accounted for roughly 62 percent of responses.

That is a powerful counterweight to the idea that model assignment is mainly an exercise in choosing the cheapest endpoint.

For this room, measured task quality and operational eligibility were much more prominent.

And both appeared directly in the presentations.

SampoTron showed why internal evaluation can reveal a capability weakness that broader benchmarks might hide.

CHI Software showed why the theoretically best model may not belong to the eligible execution set in the first place.

There Is No Single Production Infrastructure

When participants were asked about their primary production AI topology, managed public cloud APIs and hybrid cloud plus on premises infrastructure were tied at 10 responses each.

Five respondents reported cloud hosted open weight models, four reported sovereign or private infrastructure, and two reported edge or physical deployment.

That distribution matters because it mirrors the central technical lesson from the talks.

Different organizations are already operating under different deployment constraints.

There is no single obvious production architecture from which every team is gradually converging.

Software Engineering Is Where the Room Is Seeing the Most Value

When participants were asked where their organizations were seeing realized AI value, software engineering and code generation received 21 selections, substantially ahead of the other categories shown in the results. Internal productivity followed with nine.

That creates an interesting connection to Hopsworks.

The poll does not establish that these teams need a software factory, nor does it prove the productivity claims made during the presentation.

But it does show that software development is already one of the clearest AI value areas for this audience.

The important next question may therefore be how those increasingly AI assisted development workflows become repeatable, governed and verifiable.

The Community Wants More Evaluation and Guardrails

We also asked what Stockholm MLOps should explore next.

Evaluation, Guardrails & LLMOps ranked first.

Physical AI and Embedded Edge Systems ranked second, followed by Agentic Systems and Orchestration, Inference Optimization and SLMs, and Sovereign AI and Air Gapped Deployments.

That ranking is particularly interesting after an event called Orchestration vs Model Selection.

The room did not ask primarily for another discussion about which model is best.

It asked to go deeper into how we determine whether these increasingly complex systems can actually be trusted in production.

Only Said Here: What Practitioners Actually Taught Us

Some of the most valuable details from #40 were not sweeping conclusions.

They were specific engineering observations.

A system can perform well in English while hiding a substantial Nordic language retrieval problem.

A query side adapter can improve that weakness without requiring an organization to rebuild an existing document index.

Routing can preserve a known good path rather than forcing every request through a new architecture.

A model can be technically capable and still be operationally ineligible because of the data it would need to receive.

A software factory can automate large parts of delivery while deliberately keeping humans at the specification boundary.

An inference service and a tightly coordinated scientific workload can both be called “AI infrastructure” while requiring fundamentally different schedulers.

And p95 or p99 latency can become just as important to infrastructure design as raw computational throughput.

These are much more useful observations than saying orchestration matters.

They tell us where it matters, what is being decided and why another part of the system cannot simply make the decision disappear.

The Costs We Still Do Not Know

One of the strengths of #40 is also one of its limitations.

Several architectures were technically convincing without providing enough evidence to compare their total production economics.

SampoTron avoids rebuilding an existing index, but the material does not quantify the compute, engineering and operational savings of doing so.

CHI Software's policy aware architecture provides a clear argument for execution eligibility, but we do not have measured latency, failure rates or the operating cost of maintaining the policy layer.

Hopsworks presented a much broader software lifecycle automation system, but the evidence available does not support a generalized claim that teams become ten times more productive.

And Sweden AI Factory clearly distinguished infrastructure workload types without providing a controlled Slurm versus Kubernetes cost or latency benchmark.

Those missing numbers should not weaken the article.

They help define what we still need to understand.

As AI systems become more orchestrated, the cost of the orchestration itself becomes part of the production question.

Routing, adaptation, evaluation, policy enforcement, verification and additional infrastructure can improve the system while also creating more system to operate.

Architecture needs economics eventually.

What Stockholm MLOps #40 Signals

Stockholm MLOps #40 did not show that orchestration has displaced model selection.

It showed why the two are difficult to separate into a simple contest.

Model capability mattered when ConfidentialMind measured a real cross lingual retrieval gap.

Orchestration mattered when SampoTron selectively adapted the failing path instead of rebuilding everything.

Policy mattered when CHI Software placed execution eligibility ahead of ordinary model optimization.

Verification mattered when Hopsworks extended AI assisted development into specifications, testing, deployment and maintenance.

Workload shape mattered when Sweden AI Factory showed why an interactive inference service and a tightly coupled HPC job should not automatically share the same scheduling architecture.

And the room added one more piece of evidence: when practitioners were asked what determines model assignment, internal evaluations and compliance boundaries dominated the responses.

Taken together, those findings point toward a more useful production principle.

There is no single “best model” independent of the conditions under which that model has to operate.

A model has to be good enough for the actual task.

It has to be eligible to receive the data.

It has to fit inside a system that can be verified.

Its workload has to run on infrastructure suited to its operational behavior.

And the organization needs evidence that the resulting system performs well enough to trust.

That is why the title question ultimately became less interesting than the questions underneath it.

Not:

Orchestration or model selection?

But:

What actually constrains this workload, and which layer of the system is responsible for resolving that constraint?

Once that is understood, model selection and orchestration stop looking like competing strategies.

They become different decisions inside the same production system.

Speakers and Organizations

Ihor Tanenkov, CHI Software presented a sovereign AI execution architecture in which model and tool eligibility is determined by data boundaries and organizational policy before ordinary optimization begins.

Jim Dowling, Hopsworks presented Hopsable and the concept of a domain specific ML software factory spanning requirements, specification, implementation, verification, deployment and operations.

Akshay Jain, ConfidentialMind presented SampoTron, a query adaptation and routing approach designed to improve cross lingual retrieval for Nordic languages while preserving an existing English document index.

Daniel Medeiros, Sweden AI Factory explored the differences between Slurm, Kubernetes and mixed workload orchestration across HPC, AI inference and scientific computing.

Related Insights

Stockholm MLOps #39: The Bottleneck Moves

#39 showed that inference performance depends on the complete system around the processor, including model structure, workload shape, memory and data movement. #40 continues that line of reasoning at another level: production architecture also depends on language performance, policy, verification and workload behavior.

Stockholm MLOps #38: The Agent Era Is Becoming an Infrastructure Era

#38 explored the infrastructure required once agents become operational actors. #40 adds an important qualification: before an intelligent system can act, organizations increasingly need explicit rules for which models, tools, data and execution paths are eligible for each step.

Sovereign AI & Control AI

CHI Software's presentation strengthens the operational definition of sovereignty. Control is not only about where infrastructure is physically located. It can also mean determining at runtime which systems are permitted to receive specific information and perform particular actions.

Event Details

Stockholm MLOps #40
Orchestration vs Model Selection
October 21, 2026
Reaktorhallen, Stockholm

Featuring CHI Software, Hopsworks, ConfidentialMind and Sweden AI Factory.

The evening explored multilingual retrieval, sovereign execution, software factories and orchestration across HPC, AI and cloud infrastructure.