Stockholm MLOps #39: The Bottleneck Moves
September 24, 2026 | AI Sweden, Stockholm
Insights by Stockholm MLOps
HPE added another cache tier to its inference stack and performance got worse. Time to first token went from roughly 37 seconds using host memory to 74 seconds when the KV cache was pushed through a conventional S3 and TCP/IP path. Then the transport architecture changed, and the same workload fell to 14.1 seconds.
The most interesting number was not 14.1. It was 74, because that regression exposed the real lesson of Stockholm MLOps #39. Optimization does not simply make a system faster. It changes where the limiting factor sits.
That pattern appeared throughout the evening. Multiverse Computing changed the structure of the model itself, reducing its memory footprint and potentially changing the class of hardware on which it could run. Intel showed why CPU versus GPU is often the wrong way to frame an inference decision, because the answer depends on workload shape, concurrency, precision, memory behavior and how much of the application is actually serial. HPE then moved outside the processor entirely and showed that long context inference could become a problem of memory capacity, cache reuse and data transport rather than raw arithmetic.
Then the room added another dimension.
When we asked attendees what was actually preventing their teams from moving AI and LLMs into production, the largest single answer was not compute, data or regulation. Among 35 respondents, 13 selected organizational readiness and culture, compared with nine who selected high compute costs or difficulty justifying ROI.
That does not mean the technical problems have disappeared. Collectively, technical, economic and governance issues accounted for most responses. But it was a useful reminder at an evening spent deep inside AI infrastructure: a faster system does not automatically create an organization ready to put it into production.
Sweden AI Factory provided another important counterpoint. Europe is still investing heavily in new compute capacity. Optimization is not replacing the need for more infrastructure. What #39 showed is that adding compute and using compute efficiently are separate engineering problems.
The bottleneck moves.
Summary: Key Insights
Production inference is increasingly a systems problem. Multiverse, Intel and HPE approached different parts of the stack, but each showed that performance depends on more than the nominal processor. Model structure, precision, memory capacity, data movement, cache placement, concurrency and workload shape can all change what infrastructure makes sense.
The workload itself is not fixed. Structural compression can reduce model size and memory requirements enough to alter the infrastructure options available to a team. But compression still has to be validated against the quality metric that matters for the actual application.
CPU versus GPU is too simple a framing. Intel's argument was not that CPUs replace accelerators. Large parallel matrix operations remain natural accelerator workloads, while serial, irregular and orchestration heavy execution can behave very differently. Modern applications increasingly contain both.
Concurrency can change the answer. Hardware that looks clearly superior for one request may behave differently under production load. The CPU result presented alongside Multiverse became interesting at higher concurrency, not because CPU suddenly became universally faster.
Data movement deserves the same attention as arithmetic. HPE's benchmark showed this unusually clearly. Moving KV cache to conventional object storage made performance worse. Changing the transport architecture then made external cache useful.
The room was thinking about economics as much as performance. In our Mentimeter poll, cost was the most selected inference optimization priority, with 21 responses. Data privacy and sovereignty followed with 12, while latency received five and throughput four.
And the biggest single production blocker was organizational. Among the 35 people who answered that question, 37 percent selected organizational readiness and culture. The technology may be moving quickly, but production adoption remains as much an organizational problem as an engineering one.
What This Event Was Really About
Stockholm MLOps #39 was nominally about model optimization and CPU based inference, but the technical discussion ended up somewhere more interesting. The speakers repeatedly changed the boundary around what we normally call the inference problem.
Multiverse started by questioning whether the model itself should be treated as fixed. Franco Serra presented CompactifAI as a structural compression approach that profiles a model, identifies redundancy, reduces parts of the architecture and then retrains or distills the resulting model to recover useful capability. This is different from simply storing the same weights at lower numerical precision. The aim is to change the computational object itself.
That matters because infrastructure planning usually begins from assumptions about the model as it exists today. If the model can be made substantially smaller while preserving the quality required by the workload, then the original hardware decision may no longer apply. A workload that looked like a multi GPU problem may fit on a smaller accelerator. A deployment that looked impractical on CPU may become viable. An edge use case may suddenly become realistic.
Intel then moved the discussion from model structure to workload structure. Jonas Svennebring and Theo Charitidis did not argue that CPUs should replace GPUs. Their point was that “AI workload” is too broad a category to determine hardware placement. Large transformer operations, small latency sensitive models, database access, operating system calls, agent orchestration and sequential tool execution have very different characteristics.
Theo captured that particularly well:
“Agentic AI has a lot of operations that are not AI operations.”
That observation matters because modern AI applications increasingly alternate between neural inference and conventional systems work. Benchmarking only the transformer may therefore tell us surprisingly little about the behavior of the complete application.
HPE moved the boundary again. Johan Fondin showed a workload where the problem was no longer primarily arithmetic. Long context and high concurrency created enough KV cache pressure that useful state had to be evicted from GPU memory. Once that happened, the system could either reconstruct state it had already calculated or retrieve it from somewhere else.
The failed S3 result is what makes the story particularly useful. Moving the state outside GPU memory did not automatically improve performance. Conventional object storage actually made the benchmark worse. Only after the data path itself changed did external cache become the fastest tested configuration.
Taken together, the evening pointed toward a more mature way of thinking about inference. The production problem is increasingly deciding what work should happen at all, where state should live and which part of the system should perform each remaining operation.
1. The Workload Is Not Fixed
Multiverse Computing's presentation challenged one of the assumptions behind infrastructure sizing: that a model arrives as a fixed object and the engineer's job is simply to find enough hardware to run it.
CompactifAI instead starts by profiling the model. Franco described examining sensitivity across different parts of the architecture, identifying redundancy and reducing model structure before retraining or distilling the compressed system. Tensor network methods are used to preserve important correlations while removing parameters that contribute less to the target workload.
In specific deployments discussed during the evening, those reductions were substantial. The material around Telefónica included reductions of up to 80 percent in model size and significant energy savings for particular Llama workloads. Those are important results, but they should not be generalized into a claim that every model can lose 80 percent of its size without consequence. Multiverse's underlying research makes clear that increasingly aggressive compression can discard useful information and reduce accuracy.
The production question is therefore not simply whether a model can be compressed. It is how much this particular model can be changed for this particular workload before the quality metric that actually matters starts to break.
That has an infrastructure consequence. Hardware sizing performed only against the original model may be premature, because optimization can change the hardware decision itself.
2. CPU Versus GPU Is the Wrong First Question
At an event explicitly focused on CPU based inference, Intel gave perhaps the least satisfying answer for anyone hoping for a simple benchmark winner.
There is no universal winner.
The reason lies in the difference between general purpose execution and specialized parallel execution. CPU cores contain substantial machinery for branch prediction, speculative execution, caching, prefetching and irregular control flow. Accelerators can devote far more silicon to parallel arithmetic, which is precisely why they perform so well on large matrix operations.
Theo put the CPU side of that distinction simply:
“The core is very good at serial workloads.”
This becomes particularly relevant as AI applications grow more complex. An agent may perform a neural inference step, query a database, make a tool call, wait on a dependency, interact with an operating system and then return to the model. The complete application therefore contains different kinds of computation with different hardware characteristics.
Intel also discussed AMX as an example of bringing matrix acceleration directly into Xeon CPUs, further weakening the usefulness of treating “CPU” and “AI accelerator” as completely separate worlds.
The better engineering question is which parts of the workload require massive parallelism, which are serial, which are latency sensitive and how those parts interact under real production conditions.
3. Concurrency Changes the Answer
One of the most dangerous habits in AI infrastructure is taking a benchmark from one workload and turning it into an architectural rule.
The material presented at #39 showed exactly why that can fail. At low concurrency, the GPU configuration in Franco's comparison was substantially faster. As concurrency increased, however, a compressed CPU based configuration became considerably more competitive.
The interesting result is not that CPU “beat” GPU. It is that the answer changed when the workload changed.
Production infrastructure rarely serves one isolated request. Teams may instead need to optimize for hundreds of simultaneous users, aggregate throughput, power consumption, memory footprint, infrastructure cost or latency under a particular service level.
A configuration that looks clearly inferior at concurrency one can look very different at concurrency 256. That is why processor benchmarks need context. Model size, precision, sequence length, batch size, concurrency, latency target and the surrounding application all matter.
Without that context, a benchmark number is often just a number.
4. The Memory Wall Arrives at Inference
HPE's presentation brought the discussion into another part of the system entirely.
Johan separated inference into two broad phases:
“The pre-fill phase is compute-bound.”
“The decode phase is memory-bound.”
Prefill processes the incoming context and creates the attention state the model needs. Decode then repeatedly accesses that state while generating new tokens.
As context grows, the KV cache grows with it. As concurrency rises, multiple requests compete for finite HBM capacity. Eventually useful state gets evicted.
“When the memory fills, the oldest parts of the memory get evicted.”
If that state is needed again, the system may have to recreate it.
“We have to do a full pre-fill recomputation.”
At that point the bottleneck has changed character. The question is no longer simply how quickly the GPU can perform arithmetic. It becomes whether previously calculated state still exists close enough to the accelerator to be reused economically.
GPU memory is extremely fast but finite. Host memory offers more capacity but sits further away. External storage offers dramatically more capacity again, but introduces still more distance and transport overhead.
Inference becomes a memory hierarchy problem.
5. The Optimization That Made Things Worse
HPE's benchmark produced one of the most useful results of the evening precisely because it included a failed optimization.
Under the presented high concurrency and long context workload, the approximate time to first token results were:
KV strategyTime to first tokenRecomputation baseline288 secHost DRAM cache37 secConventional S3 over TCP/IP74 secRDMA backed external cache14.1 sec
The test environment was highly specific, including eight NVIDIA H200 NVL GPUs, a 70B Nemotron model, 96 concurrent requests and roughly 50K tokens of input context. Those conditions matter. These numbers should not be treated as universal inference performance.
What makes the benchmark valuable is the middle of the table. Moving the cache from host DRAM to a conventional S3 and TCP/IP path made the system almost twice as slow.
Johan was explicit:
“If we add the S3 standard TCP/IP object store as a tier, we actually regress to 74 seconds.”
That negative result prevents an easy conclusion such as “put KV cache on storage.” Externalizing state was not the optimization. The benefit only appeared after changing how the state moved.
The storage device itself was therefore not the whole optimization. The data path was.
6. When Storage Becomes Part of Inference
HPE's next step was to change that transport architecture.
The system combined vLLM, LMCache, RDMA enabled networking, GPU Direct Storage and HPE Alletra Storage MP X10000. The aim was to reduce the conventional CPU, operating system and TCP/IP overhead involved in moving KV state between external storage and the accelerator.
Johan described the intervention this way:
“We bypass the CPU, the DRAM and the OS kernel entirely.”
The exact technical boundaries of that statement matter, but the broader production principle is straightforward. Once the data path became efficient enough, retrieving previously calculated state became cheaper than rebuilding it.
Under the tested workload, time to first token fell to approximately 14.1 seconds.
“Time to first token collapsed to 14 seconds, 14.1.”
This does not mean external storage is faster than HBM. It means that retrieving previously calculated KV state from the external cache was much faster than asking the GPUs to recompute evicted prefixes under this particular workload.
That distinction changes the role of storage. Instead of acting only as passive persistence, it becomes part of the active inference memory hierarchy.
“Storage can actually act as a realistic context cache for the model.”
That is a much more interesting development than simply building a faster object store.
7. Stop Paying GPUs to Remember
The HPE architecture raises a broader utilization question: what should expensive GPUs actually spend their time doing?
If significant amounts of HBM are occupied by state that can economically live elsewhere, that memory cannot be used for other work. If GPUs repeatedly reconstruct prefixes that have already been calculated, the accelerators are spending time recreating history rather than producing new output.
Johan summarized the desired outcome clearly:
“The GPU generates new tokens, which the GPU is meant to do, instead of recalculating prefixes.”
That connects unexpectedly well with Multiverse. Multiverse asks which model parameters need to remain at all. HPE asks which calculations need to be repeated. Intel then asks which type of processor should execute the work that remains.
These are different technical approaches, but they point toward the same engineering discipline: remove unnecessary work before buying faster hardware to perform it.
That does not mean faster hardware is unnecessary. It means utilization deserves the same attention as capacity.
8. More Compute Is Still Part of the Answer
It would be easy to turn the previous sections into an argument against GPU expansion. That would contradict the evidence from the same evening.
Ashwin V. Mohanan presented Sweden AI Factory as infrastructure plus expertise, providing access to AI optimized supercomputing together with training, workshops, expert support and help navigating Swedish and European compute resources.
Sweden's Arrhenius system entered production in 2026 as the country's first supercomputer within the EuroHPC collaboration, with hundreds of GPU nodes based on NVIDIA Grace Hopper systems together with CPU infrastructure and a large parallel filesystem.
Europe is building that capacity because aggregate demand for compute continues to grow. Optimization does not eliminate that demand.
The more useful conclusion is that capacity and utilization solve different problems. More compute increases what organizations can attempt, while better optimization determines how effectively that compute is used.
Stockholm MLOps #39 therefore did not produce either of the easy conclusions. It did not tell us that we no longer need more GPUs, and it did not tell us simply to add more GPUs. Production AI increasingly needs both greater capacity and more deliberate use of that capacity.
Insights by Stockholm MLOps: Signals From the Room
The Mentimeter results add an important layer to #39 because they tell us something about the people actually trying to move AI into production, not only the technologies being presented on stage.
The biggest single production bottleneck was organizational
When we asked “What is currently the single biggest bottleneck preventing your team from moving AI/LLMs into production?”, 35 people responded.
Organizational readiness and culture came first with 13 responses, roughly 37 percent. High compute costs and ROI justification followed with nine, while data quality and pipeline hygiene and security, privacy and data sovereignty received four each. Model evaluation and observability received three, and governance and regulatory compliance two.
The result should not be interpreted as saying technology no longer matters. Most responses, taken together, still pointed to technical, economic or governance constraints.
What is notable is that no individual technical category was selected as often as organizational readiness.
At an event spent deep inside model compression, processors, memory hierarchy and storage architecture, that is an important reality check. Better infrastructure does not automatically produce organizational capability.
Production architecture is already heterogeneous
We also asked respondents about the primary infrastructure topology for their production AI workloads. Among 55 responses, pure public cloud using managed services was the largest single category at 18, but hybrid infrastructure was close behind at 16.
Another 10 respondents reported sovereign on premises or private infrastructure, nine were running open weights in cloud environments, and only two identified edge or physical hardware as their primary topology.
In other words, only about one third of respondents described pure managed public cloud as their main production architecture. The rest were already operating across open weight cloud, private infrastructure, hybrid systems or edge environments.
That fits #39 remarkably well. There is no single production topology because there is no single AI workload.
The most realized value is still close to the people building and operating software
When 59 attendees were asked where their organizations were seeing the most realized value from AI in production, the two most frequently selected areas were software engineering and code generation, with 28 selections, and internal productivity and automation, with 25.
Customer support and operations followed with 10 selections, while customer facing product features and domain specific R&D each received six. Eight respondents said their organization did not yet have AI in production.
Because respondents could select more than one area, these numbers should not be treated as market shares. But the pattern is useful. For this room, the strongest realized value was still concentrated in developer workflows and internal productivity rather than in customer facing AI products.
That also helps explain why architecture and infrastructure questions are becoming more relevant. These are no longer only experimental systems. A meaningful part of the room is already using AI inside day to day engineering and operational workflows.
The community wants to go deeper on inference optimization
We asked what Stockholm MLOps should deep dive into next. Inference Optimization and Small Language Models ranked first overall, followed by Agentic Systems and Orchestration. Physical AI and Embedded Edge Systems came third, Evaluation, Guardrails and LLMOps fourth, and Sovereign AI and Air Gapped Deployments fifth.
That ranking is particularly interesting after #39. The evening did not close the inference optimization discussion. It appears to have made the community want more of it.
The ranking also reinforces the broader direction of Stockholm MLOps. The conversation is increasingly moving from whether AI works toward how these systems behave, scale and operate under real constraints.
Cost is the dominant optimization concern
Finally, when we asked what mattered most when optimizing AI inference in production, cost was the strongest response, with 21 selections. Data privacy and sovereignty followed with 12, while latency received five, throughput four, energy efficiency four and memory footprint one among the visible responses.
This adds useful context to the technical discussions from Multiverse, Intel and HPE. Engineers may talk about throughput, TTFT, memory hierarchy and numerical precision, but those optimizations ultimately have to translate into something the organization values.
For many teams in this room, that value is economic.
The Bottleneck Moves
Viewed separately, the talks covered very different parts of the AI infrastructure stack. Sweden AI Factory focused on access to large scale compute and expertise. Multiverse changed model structure. Intel looked at processor architecture and workload shape. HPE focused on memory hierarchy, KV cache and data transport.
Yet the same engineering pattern appeared repeatedly.
Once one bottleneck was reduced, another became visible. Compress the model and the hardware placement decision changes. Change the workload and the appropriate processor changes. Increase context and concurrency and HBM capacity becomes constrained. Move state out of HBM and transport becomes the problem. Improve the transport path and external storage can become part of the active inference system.
The Mentimeter results add an almost uncomfortable final step to that pattern. Even if engineers solve every one of those infrastructure problems, the bottleneck may simply move again, this time into the organization itself.
That is why peak compute alone is an increasingly incomplete way to reason about production AI. FLOPS still matter, but they coexist with model structure, numerical precision, memory capacity, bandwidth, state reuse, network transport, concurrency, cost, deployment topology and organizational readiness.
The performance of the system emerges from all of them.
Only Said Here: What Practitioners Actually Taught Us
Some of the most useful intelligence from Stockholm MLOps events appears in details that rarely survive into polished benchmark slides.
HPE openly showed the optimization that failed. Conventional external cache was slower than DRAM, and that negative result made the eventual RDMA result more credible because it demonstrated that the team had not simply discovered a magical storage tier. They had discovered that the data path determined whether the tier was useful.
Intel, at an event explicitly framed around CPU based inference, described workloads that clearly belong on accelerators. That refusal to turn an engineering discussion into a simple CPU sales pitch was valuable in itself.
Multiverse's CPU result also depended heavily on concurrency. The GPU performed better in the low concurrency case, while the economics changed as the workload scaled. That nuance is far more useful to production teams than simply demonstrating that an LLM can run on a CPU.
And Johan challenged his own favorable cache story with a question that any practitioner should ask:
“What if we have a 100% cache miss?”
That matters because cache benchmarks are only useful if we understand what happens when the cache does not help.
The Q&A revealed another operational detail. After a presentation involving HBM, RDMA, object storage and GPU direct access, someone in the audience asked about virtualization. Johan explained that the storage platform internally uses containers and Kubernetes, but that implementation is hidden from the user.
“The software within the storage platform actually runs on containers and Kubernetes.”
“That's hidden from you as a user.”
That distinction is easy to overlook. Sophisticated infrastructure may exist underneath a product without automatically becoming the customer's operational burden.
What Stockholm MLOps #39 Signals
There was no CPU victory over GPU at Stockholm MLOps #39.
There was something more useful: evidence that hardware performance cannot be separated from the system around it.
Multiverse showed that the computational workload itself can be changed before deployment. Intel showed that different parts of an AI application have fundamentally different execution characteristics. HPE showed that inference performance can depend on whether previously computed state has to be recreated, where that state is stored and how it moves through the system. Sweden AI Factory reminded us that all of this optimization is happening alongside a major expansion of European compute capacity.
The community data adds another layer. The room is already operating across public cloud, open weight cloud, sovereign infrastructure and hybrid deployments. The most realized production value is concentrated in software engineering and internal productivity. Cost dominates the inference optimization discussion. And despite all the technical sophistication in the room, organizational readiness remains the largest single reported blocker to production.
The common lesson is therefore not simply “optimize.” It is to understand the full system.
What is being computed? Which parts do not need to be computed again? What can be compressed? What can be reused? Where does state live? How often does it move? Which work is serial? Which work is massively parallel? What happens when concurrency rises? What does the infrastructure cost? Where does the workload need to run? And is the organization actually prepared to operate what has been built?
The processor remains important, but it is only one part of that system.
Perhaps the most useful question after Stockholm MLOps #39 is therefore not simply:
Which accelerator should we buy?
It is:
Where is the bottleneck now?
Because #39 showed that once you solve one, the answer has a habit of changing.
Speakers and Organizations
Ashwin V. Mohanan, Sweden AI Factory presented Sweden AI Factory as a combination of AI optimized compute, expertise, training and support for organizations looking to use Swedish and European infrastructure.
Franco Serra, Multiverse Computing presented CompactifAI and showed how structural model compression can change memory requirements, energy use and potentially the hardware needed for deployment.
Jonas Svennebring and Theo Charitidis, Intel focused on processor architecture, precision and why hardware decisions depend on workload structure rather than simply whether an application contains AI.
Johan Fondin, HPE explored KV cache as an inference memory problem and showed how the transport path determined whether externalized cache improved or degraded performance.
Related Insights
Stockholm MLOps #38: The Agent Era Is Becoming an Infrastructure Era
#38 explored the infrastructure required once agents become operational actors. #39 goes deeper into that infrastructure and shows why even inference cannot be treated as one compute problem.
Builders & Brews Stockholm: The Hard Questions Arrive Earlier
The hackathon showed builders reaching architectural and product constraints much earlier. #39 shows the same dynamic deeper in the stack: once one technical limitation disappears, the next one becomes visible.
Sovereign AI & Control AI
Sweden AI Factory and the Mentimeter results create a genuine connection to the continuing European infrastructure story. Ten of 55 respondents described sovereign on premises or private infrastructure as their primary production topology, while another 16 were operating hybrid environments. Sovereignty was not the main thesis of #39, but deployment control is clearly part of the production reality of this community.
Event Details
Stockholm MLOps #39
September 24, 2026
AI Sweden, Stockholm
Featuring Sweden AI Factory, Multiverse Computing, Intel and HPE.
The evening focused on model optimization, CPU based inference and the wider systems architecture required to run AI efficiently in production.