Apple Silicon Unified Memory as On-Premise AI Infrastructure (Jan 2025 – Aug 2026)

Narrative

Independent commentary on Apple Silicon as an AI substrate splits cleanly into two camps: hands-on benchmarkers documenting what a Mac cluster can actually do, and strategy essayists arguing about who captures value if it works. The benchmarking camp is anchored by Jeff Geerling's December 2025 post on RDMA over Thunderbolt 5, which found that enabling RDMA on a four-node Mac Studio cluster took Exo from roughly 5 tokens/sec to 32 tokens/sec on Kimi K2, a workload too large for any single consumer GPU. Creative Strategies' "Running a 1T parameter model on a $40K Mac Studio Cluster" documents the same RDMA transition, reporting that Apple enabled Thunderbolt 5 RDMA in macOS 26.2 and that Kimi K2 throughput moved from 5 to 25 tokens/sec once RDMA bypassed the network stack. Independent developer awni Hannun's ">20 tokens/second" DeepSeek-V3-0324 4-bit result on a 512GB M3 Ultra, first surfaced on social media in March 2025 and republished across Hardware Corner and Slashdot, remains the most-cited single benchmark in this space, and several later posts (oMLX's DeepSeek-V4-Flash test, SomeOddCodeGuy's dev.to write-up) extend or partially reproduce it rather than independently re-deriving it from scratch, which is worth flagging as a citation chain rather than a fully replicated result.

A second cluster of practitioner blogs (Starlog, Contra Collective, GK Servis, RunAIHome, Awesome Agents, Medium author Michael Hannecke) converges on a consistent empirical picture: Exo and MLX Distributed can shard trillion-parameter MoE models like Kimi K2 and DeepSeek across three to four Mac Studios via Thunderbolt 5 RDMA, achieving single-digit-microsecond node latency versus 100+ microseconds over standard networking, but the resulting throughput (25-32 tokens/sec) serves essentially one user stream, not a concurrent-request API. Contra Collective's May 2026 piece on vLLM-on-Metal is the most direct technical rebuttal to any equivalence with CUDA infrastructure: it reports the vLLM CPU backend on an M5 Max hitting only 3-5 tokens/sec aggregate versus llama.cpp's native Metal backend at 92 tokens/sec on the same hardware, and states plainly that Metal Performance Shaders does not provide the primitives PagedAttention depends on. Skorppio's DGX Spark comparison and its Neural Engine deep-dive independently argue that Apple's ANE is "a dead end for LLMs" architecturally, built for CNN-era graph execution rather than iterative transformer attention, with Apple instead routing matmul work to new per-core GPU Neural Accelerators in the M5 generation.

On the strategy side, Substack essays diverge sharply on whether any of this matters commercially. Ben Pouladian's bepresearch.substack.com "Other Memory Wall, Part 2" and the limitededitionjonathan.substack.com piece "Apple Is the King of AI and Nobody Knows It" both argue Apple is building an "infrastructure moat" on the edge side of a two-sided memory wall that NVIDIA owns on the data-centre side, with the latter noting Jensen Huang's own musings about "an AI supercomputer" becoming common in the home as evidence NVIDIA is hedging into Apple's territory. Dave Friedman's davefriedman.substack.com piece is more clinical, itemising real vulnerabilities in the on-device thesis: Apple's roughly $1 billion annual Gemini licensing dependency, undisclosed OpenAI economics, and the open question of whether a 3-billion-parameter on-device model stays adequate as frontier capability keeps advancing. Longyield's substack frames the central unresolved question precisely as who captures surplus value if local AI becomes technically feasible at scale, the model lab or the device platform that routes the interaction, concluding the evidence is "directionally favorable" for Apple's device-layer thesis but conceding Apple is "losing the perception argument" even while "winning the capability argument in silicon." Dana Blankenhorn's substack, by contrast, expresses second-guessing of even a bullish Apple position, noting Apple is the only former "Cloud Czar" not building AI data centres and questioning whether that restraint will still look wise.

Simon Willison's ongoing apple-tagged coverage on his personal weblog provides the most technically grounded independent tracking of the developer-facing stack: his llm-mlx plugin post, his notes on the Foundation Models framework's 3-billion-parameter, 2-bit-quantized on-device model with constrained-decoding "guided generation," and his coverage of WWDC 2026's opening of the Foundation Models framework to third-party and MLX-community models via Hugging Face all track Apple's stack maturing in near-real time without editorializing about the data-centre framing. IronCore Labs' analysis of Private Cloud Compute is the sharpest independent security take, conceding Apple's transparency and bug-bounty approach to PCC is comparatively strong specifically because "OpenAI, Microsoft, Google, and other big players have done nothing" comparable on GenAI privacy, while still flagging that software bugs remain inevitable regardless of architecture.