Core Labs R&DWhitepaper / Preprint
Daniele Trovato — Core Labs R&D
Modern language-model deployments fragment across heterogeneous execution targets: small on-device models, mid-sized models on local-network accelerators, self-hosted server models, and third-party APIs. Users and applications are forced to choose a single backend, trading off latency, capability, cost, and — critically — data sovereignty. We present Molly OS, a model-agnostic inference orchestration layer that routes each request to the best available execution target while keeping data under user control by default. Molly OS unifies five mechanisms: (i) capability-based routing with cascades and fallback across on-device, local-network, and remote backends; (ii) concurrent specialist-adapter serving, in which a single quantized base model exposes many domain experts via hot/cold-managed low-rank adapters; (iii) a sovereign execution policy that enforces on-device-first processing and explicit data-residency constraints; (iv) a continuous specialization loop that converts interaction traces into new specialist adapters via evaluation and distillation; and (v) multimodal generation behind one interface. We describe the architecture, the routing and adapter-serving subsystems, and the federated improvement protocol. An evaluation across approximately 100 domains, scored by a neutral held-out LLM judge, shows that specialist-adapter orchestration improves output quality over an unspecialized base model across domains — with the largest gains in generative and AI/ML tasks — demonstrating that sovereign, on-device-first orchestration is practical without sacrificing task quality. Systems-level metrics (routing accuracy, end-to-end latency, and federated efficiency) are the subject of ongoing measurement.
The inference landscape for large language models (LLMs) has bifurcated. On one side, sub-billion and few-billion parameter models now run acceptably on phones and laptops [25, 26, 27], aided by quantization [12, 13, 14, 15] and memory-aware execution [28]. On the other, frontier-scale capability remains concentrated in remote services accessed through third-party APIs. Between these extremes sit local-network deployments: a workstation or home server hosting a mid-sized model behind an efficient serving stack [8, 9].
This fragmentation imposes two costs. First, capability fragmentation: no single backend is best for all requests. A short factual lookup is wasted on a remote frontier model; a multi-step reasoning task overwhelms a pocket model. Routing and cascade systems [20, 21, 22] show that selecting among models per request improves the cost–quality frontier, but existing routers assume a homogeneous trust domain — typically a set of cloud endpoints. Second, privacy cost: cloud-only inference exports raw user data by default. Federated learning demonstrated that model improvement need not centralize raw data [23, 24], yet inference-time orchestration has largely ignored the analogous principle: that execution placement is itself a privacy decision.
We argue for a sovereign orchestration layer: a single control plane, owned by the user, that mediates all inference requests and decides — per request and under explicit policy — whether execution occurs on-device, on the local network, on a self-hosted remote model, or via an external API. Sovereignty here means the default is local, escalation is policy-gated, and data residency is a first-class routing constraint rather than an afterthought.
This paper describes Molly OS, an implementation of this design. Our contributions, ordered as they execute at runtime — specialists serve first, and heterogeneous escalation follows only when needed — are:
Parameter-efficient fine-tuning. Adapter modules [3], prefix-tuning [4], prompt tuning [5], and low-rank adaptation (LoRA) [1] showed that task specialization requires updating only a small fraction of parameters. QLoRA [2] extended this to quantized base models, making specialization feasible on commodity hardware. Molly OS adopts LoRA-style adapters as its unit of specialization because they are cheap to train, store, transmit, and swap.
Multi-adapter serving. S-LoRA [6] and Punica [7] demonstrated that thousands of LoRA adapters can be served concurrently against a shared base model using unified paging and custom batched kernels. Molly OS adapts these ideas to resource-constrained, single-tenant settings, where the challenge is not multi-tenant throughput but tight memory budgets and adapter lifecycle management.
Efficient serving. PagedAttention [8], iteration-level scheduling in Orca [9], IO-aware attention kernels [10], and offloading-based throughput systems [11] form the substrate on which any orchestration layer rests. Molly OS treats these as backend-internal mechanisms and focuses on the layer above them.
Quantization. Post-training quantization methods [12, 13, 15] and mixed-precision decomposition [14] enable 4–8-bit inference with limited quality loss, and are prerequisites for the on-device and LAN tiers of our design.
Mixture-of-Experts and conditional computation. Sparsely gated experts [16], Switch Transformers [17], GShard [18], and Mixtral [19] activate subsets of parameters per token within a model. Molly OS performs conditional computation across models and adapters at the request level; the two are complementary, and MoE models can serve as backends.
Routing, cascades, and model selection. FrugalGPT [20] introduced cost-aware LLM cascades; RouteLLM [21] learns routers from preference data; LLM-Blender [22] ensembles model outputs via pairwise ranking. These works optimize cost and quality over cloud endpoints. Molly OS generalizes the target set to heterogeneous trust domains and adds sovereignty constraints to the routing objective.
Federated learning. Federated averaging [23] and the broader cross-device federated literature [24] established model improvement without centralizing data. Molly OS applies the same principle to adapter-level updates and extends it to inference-time placement decisions.
On-device and small language models. MobileLLM [25], Phi-3 [26], TinyLlama [27], and data-quality-driven small models [29] show that compact models handle a meaningful fraction of real workloads; flash-based weight streaming [28] relaxes memory limits further. These models populate the lowest, most-private tier of our hierarchy.
Speculative decoding. Speculative sampling [30, 31], blockwise parallel decoding [32], and Medusa [33] accelerate large-model decoding using cheaper draft computation. In Molly OS, on-device models can act as draft models for LAN-tier targets, aligning the acceleration hierarchy with the placement hierarchy.
Distillation. Knowledge distillation [34, 35, 36, 37] underlies our continuous specialization loop: traces validated against stronger targets become supervision for compact specialists.
Retrieval and tools. Retrieval-augmented generation [38, 39, 40, 41, 42] and tool-use frameworks [43, 44, 45, 46, 47] are capabilities exposed through the orchestration layer; in particular, HuggingGPT-style task decomposition [47] is a precedent for treating models as routable resources.
Positioning. Prior work optimizes serving efficiency, routing quality, or federated training in isolation. Molly OS combines serving (multi-adapter, quantized), routing (capability- and policy-aware cascades), and sovereignty (residency-constrained placement, federated adapter improvement) in a single layer.
Molly coordinates a concrete operating surface. The compute substrate is a heterogeneous-OS cluster — Linux, macOS, and Windows machines — paired with local storage and with encrypted remote/cloud storage, and Molly treats each machine according to its strengths. Above this substrate it orchestrates the working surfaces of an organization as governed tools: team-member access and API keys, scoped per member; digital payments; trading; an invoicing service; and customer service. These are not separate products bolted on after the fact but functions Molly drives directly, so that a single sovereign system spans both the hardware it runs on and the operations it runs.
Molly OS sits between applications and a heterogeneous pool of execution targets. Every request enters through a unified interface, is annotated with a capability profile (task type, expected difficulty, modality, context needs) and a sovereignty profile (data-sensitivity class, residency constraints), and is dispatched by the router to one of four target tiers:
Supporting subsystems include the adapter registry (Section 7), the policy engine (Section 8), the trace store and specialization pipeline (Section 9), and multimodal backends (Section 10). Retrieval [38, 40] and tool execution [43, 44] are mediated by the same layer so that retrieval corpora and tool I/O obey the same residency rules as model inputs.
flowchart TD
APP[Applications / Clients] --> GW[Unified Inference Interface]
GW --> CLS[Capability + Sensitivity Classifier]
CLS --> RT[Router]
POL[Sovereignty Policy Engine] --> RT
REG[Adapter Registry] --> RT
RT --> T0[T0: On-Device Model + Adapters]
RT --> T1[T1: LAN Model Server]
RT --> T2[T2: Self-Hosted Remote Model]
RT --> T3[T3: External API - policy gated]
T0 --> AGG[Response Aggregator / Verifier]
T1 --> AGG
T2 --> AGG
T3 --> AGG
AGG --> GW
AGG --> TRC[Trace Store]
TRC --> SPC[Specialization Pipeline]
SPC --> REG
RAGS[Retrieval Store] --- RT
TOOLS[Tool Executor] --- RT
Figure 1. Molly OS architecture. All requests pass through a single interface; the router selects among four target tiers under sovereignty policy; traces feed a specialization pipeline that produces new adapters.
Target selection spans on-device execution, LAN machines, and cloud endpoints within one address space. The router chooses among them by live price/performance comparison, weighing latency, cost, and capability for each request rather than binding to any fixed provider.
The router solves, per request, a constrained selection problem: choose the target (and adapter, if applicable) that maximizes expected quality subject to latency, cost, and sovereignty constraints. This generalizes cost–quality routing [20, 21] in two ways: the candidate set spans trust domains, and sovereignty constraints are hard rather than soft.
Capability estimation. A lightweight classifier — itself a T0 model — predicts task category and difficulty. The router maintains per-target, per-category quality estimates calibrated from historical traces, analogous to learned routing from preference data [21]. Adapter availability shifts these estimates: a T0 model with a strong domain adapter may outrank an unadapted T1 model for that domain.
Cascades and fallback. Following the cascade pattern [20], the router may attempt a cheap target first and escalate on low confidence. Confidence is computed from generation-time signals (e.g., self-reported uncertainty, verifier scores) and, where multiple candidates respond, output ranking in the spirit of LLM-Blender [22]. Escalation respects the sovereignty lattice: a request pinned to local execution may escalate T0 → T1 but never to T3. Fallback handles target unavailability (e.g., LAN host offline) by re-routing within the permitted tier set.
Speculative cooperation. When a request lands on T1, the T0 model can serve as a draft model for speculative decoding [30, 31, 33], so the placement hierarchy doubles as an acceleration hierarchy.
flowchart TD
REQ[Incoming Request] --> SENS{Sensitivity class?}
SENS -->|Private| LOCK[Tier set = T0, T1]
SENS -->|Standard| OPEN[Tier set = T0..T3]
LOCK --> CAP[Capability + Difficulty Estimate]
OPEN --> CAP
CAP --> AD{Specialist adapter available?}
AD -->|Yes| LOCAL[Attempt T0 with adapter]
AD -->|No| EST[Score permitted targets]
EST --> PICK[Select max expected quality s.t. latency and cost]
LOCAL --> CONF{Confidence above threshold?}
PICK --> EXEC[Execute on selected target]
EXEC --> CONF
CONF -->|Yes| OUT[Return response]
CONF -->|No| ESC{Higher tier permitted?}
ESC -->|Yes| UP[Escalate to next tier]
UP --> EXEC
ESC -->|No| BEST[Return best local response with caveat]
Figure 2. Routing and cascade flow. Sensitivity classification restricts the permitted tier set before capability-based selection; low-confidence outputs escalate only within the permitted set.
For requests whose predicted domain distribution is not sharply concentrated, the router need not commit to a single target. Instead, it may dispatch a top-k weighted mixture of specialist adapters, where k and the mixture weights are derived from the calibrated domain posterior. The resulting candidate outputs are fused by confidence-weighted ranking, following the output-ensembling approach of [22]: each candidate is scored by the product of its routing weight and a per-target quality estimate, and the highest-ranked output (or a merged composition, where outputs are complementary) is returned. Mixture dispatch is gated by the same tier and budget constraints as single-target routing, with the additional cost of parallel decoding.
A central design choice is that specialization is cheaper than scale at the edge. Rather than hosting many specialized models, each tier hosts one quantized base model [2, 12, 13] and a library of LoRA adapters [1], so that a single base exposes many domain experts.
Concurrent execution. Following S-LoRA [6] and Punica [7], adapter computation is batched: base-model weights are shared across all in-flight requests, and per-request low-rank deltas are applied via batched matrix operations keyed by adapter identity. KV-cache and adapter weights share a unified paged memory pool, extending PagedAttention-style management [8] to adapter pages. Iteration-level scheduling [9] allows requests using different adapters to join and leave batches independently.
Hot/cold management. Edge memory budgets do not permit residency for all adapters. The registry tracks per-adapter access recency and frequency; hot adapters stay pinned in accelerator memory, warm adapters reside in host memory, and cold adapters live on storage. On-demand loading promotes adapters at request time; because adapters are small relative to the base model, promotion latency is bounded by the time to stream a small low-rank adapter from storage, far below base-model load time in our measurements. Eviction is cost-aware: adapters with high reload probability are demoted last.
Adapter portability. Adapters are versioned against base-model checkpoints and quantization configurations, so an adapter trained on a T1 host can be redistributed to T0 devices sharing the same base — this portability underpins the federated mechanism of Section 8.
flowchart LR
subgraph SRV[Adapter-Augmented Serving Engine]
BASE[Shared Quantized Base Model]
SCHED[Iteration-Level Scheduler]
POOL[Unified Paged Memory: KV cache + adapter pages]
K[Batched LoRA Kernels]
SCHED --> BASE
BASE --> K
POOL --- BASE
POOL --- K
end
R1[Request A: legal adapter] --> SCHED
R2[Request B: medical adapter] --> SCHED
R3[Request C: code adapter] --> SCHED
subgraph REG[Adapter Registry]
HOT[Hot: device memory]
WARM[Warm: host memory]
COLD[Cold: storage]
COLD -->|on-demand load| WARM
WARM -->|promote| HOT
HOT -->|evict| WARM
end
HOT --> POOL
Figure 3. Concurrent specialist serving. One shared base model serves heterogeneous adapter requests in the same batch; adapters migrate between hot, warm, and cold states under a cost-aware policy.
A CEO-style orchestrator classifies each request and delegates it to the appropriate specialist agents. Through this harness it exposes the organization's operational functions — payments, trading, invoicing, customer service, and team-member access — as governed tools, so delegation reaches real actions under explicit policy.
Many requests presented to the orchestration layer are not single-shot completions but composite tasks that benefit from explicit decomposition: a query may span multiple domains, require intermediate tool invocations, or demand verification of internally inconsistent draft outputs. We therefore extend the serving path of Section 5 with an agent orchestration mode that activates when triage classifies a request as composite.
Controller. A controller agent performs triage and emits a delegation plan: a typed graph of subtasks, each annotated with a target specialist, a tier constraint, and a per-subtask budget. The plan is admitted only after passing a policy/budget gate that enforces the same sovereignty and tier constraints applied to single requests (Section 4); in particular, every specialist invocation is bound to the least-exposed tier permitted for the data classification of its subtask. This design follows the reasoning–acting paradigm [44] and treats specialists analogously to tools [43, 45, 46, 47], including the model-as-tool composition view of [47], but constrains all delegation through the deployment-wide policy engine rather than leaving routing to free-form agent decisions, in contrast to open-ended conversational frameworks [49, 50, 51].
Specialists. Each specialist is a base model paired with a domain adapter from the continuous-specialization pipeline (Section 5). Subtasks without inter-dependencies in the delegation plan execute in parallel; dependent subtasks follow least-to-most style sequencing [60]. Specialists may internally employ chain-of-thought prompting [57] or program-aided execution for computational subtasks [48], and bootstrapped reasoning traces [55] are retained as candidate training signal for the specialization pipeline.
Fusion. A higher-order fusion step integrates specialist outputs. Where specialists return alternative candidates for the same subtask, fusion applies confidence-weighted ranking in the spirit of output ensembling [22] and self-consistency selection [58]; where outputs are complementary, fusion composes them under the plan's typed schema. Structured multi-candidate integration relates to deliberate search over thoughts [59, 61], although here the branching structure is fixed by the delegation plan rather than expanded dynamically.
Meta-cognition. Before responding, a meta-cognition step checks the fused output for internal consistency, coverage of the original request, and policy compliance. On failure, it triggers bounded refinement—re-invoking specific specialists with critique feedback—following self-reflection and iterative-refinement approaches [53, 54] and tool-interactive critiquing [56]. Disagreement between specialists can additionally be surfaced as a debate-style adjudication round, which has been shown to improve factuality [52]. Refinement depth is capped by the controller's residual budget; exhaustion yields the best-ranked fused output with an attached uncertainty annotation.
We evaluate orchestration on standard agent benchmarks and harnesses, including general agentic evaluation [62], realistic web environments [63], repository-level software tasks [64], and tool-augmented API use [65], quantifying the quality gain over the single-specialist baseline and the added latency overhead is part of ongoing measurement.
flowchart TD
R[Request] --> C[Controller: triage + delegation plan]
G[Policy / Budget Gate] --> C
C --> S1[Specialist A: base + domain adapter]
C --> S2[Specialist B: base + domain adapter]
C --> S3[Specialist C: base + domain adapter]
S1 --> F[Fusion: confidence-weighted ranking]
S2 --> F
S3 --> F
F --> M[Meta-cognition: consistency check]
M -- refine --> C
M -- accept --> O[Response]
Figure 5. Agent orchestration. The controller emits a delegation plan under an explicit policy/budget gate; domain specialists execute in parallel on the least-exposed permitted tier; fusion ranks and integrates outputs; meta-cognition validates consistency and may trigger bounded refinement.
Self-custody holds across the entire heterogeneous-OS cluster: data, adapters, and embeddings remain on the user's own Linux, macOS, and Windows machines, and only adapter deltas — never raw data — leave the boundary during federated exchange.
On-device-first policy. The policy engine assigns each request a sensitivity class derived from content signals and user-declared rules. The default class confines execution to T0/T1. Escalation to T2 requires that the remote infrastructure be user-controlled; escalation to T3 requires explicit policy permission and applies redaction transforms to remove identified sensitive spans before transmission. Retrieval is local-first: personal corpora are indexed and queried on-device or on the LAN [38, 40], never shipped to T3.
Data residency as a routing constraint. Residency is enforced structurally — the router cannot emit a dispatch violating the tier set — rather than by post-hoc filtering. This makes the privacy property auditable at the orchestration layer.
Federated improvement. Devices improve collectively without centralizing raw data, following federated principles [23, 24]. The unit of exchange is the adapter delta: a participant trains or refines a specialist adapter locally (Section 9), and only the low-rank parameters — optionally with privacy-preserving noise consistent with established federated practice [24] — are shared with an aggregation point, which may itself be a LAN host. Because adapters are orders of magnitude smaller than base models, communication cost is modest, echoing the communication-efficiency motivation of federated averaging [23]. Aggregated adapters are redistributed through the registry with version pinning.
flowchart TD
subgraph DEV[On-Device Tier T0]
P1[Phone: SLM + adapters]
P2[Laptop: SLM + adapters]
DATA[(Raw user data - never leaves tier)]
P1 --- DATA
P2 --- DATA
end
subgraph LAN[Local Network Tier T1]
HUB[LAN Model Server + Adapter Aggregator]
end
subgraph REM[Remote Tiers]
T2N[T2: Self-Hosted Model]
T3N[T3: External API]
end
P1 -->|adapter deltas only| HUB
P2 -->|adapter deltas only| HUB
HUB -->|aggregated adapters| P1
HUB -->|aggregated adapters| P2
P1 -.->|policy-gated, redacted requests| T3N
HUB -->|escalated inference| T2N
HUB -.->|policy-gated, redacted| T3N
Figure 4. Federated topology. Raw data remains in the on-device tier; only adapter deltas cross tiers for improvement, and only redacted, policy-gated requests reach external APIs.
The specialization layer runs continuously, assembling high-quality material distilled from frontier models and feeding it into the specialist adapters — turning everyday usage into new capability without surrendering control of the underlying data. The layer is OS-agnostic by design: it is built to exploit heterogeneous operating systems for their respective strengths, training and serving across Linux, macOS, and Windows hosts and drawing the best from whatever compute is present. It combines mixed techniques under one loop — distillation, preference optimization, on-device fine-tuning on Apple Silicon via MLX, self-play, and CUDA-accelerated simulation, including headless robotics training in Isaac Lab and trading/strategy simulation. The specific scheduling and placement across hosts are internal implementation details; what matters at the architectural level is that any available operating system can be enrolled and used for the capability it serves best.
Molly OS treats usage as a supervision source. The loop has four stages, described abstractly:
The economic consequence is a capability gradient: domains a user exercises frequently migrate downward in the tier hierarchy, increasing the locally served fraction over time and reducing both latency and external exposure.
Evaluation results from each specialization cycle additionally update per-domain capability priors, maintained as an exponentially-weighted moving average over held-out task scores for every (base, adapter, tier) target. These priors feed directly into the router's per-target quality estimates, so that usage, evaluation, capability priors, and routing form a closed loop: traffic surfaces domain demand, evaluation measures the resulting adapters, and updated priors shift subsequent dispatch decisions. In particular, a newly promoted adapter whose evaluation exceeds the incumbent's prior immediately redirects routing toward the local tier without manual reconfiguration. This realizes a learned-routing feedback mechanism in the sense of [21], grounded in measured rather than predicted capability.
Multimodal capability is exposed through the same interface and routed by the same policy machinery. Image and audio generation backends are registered as targets with modality-typed capability profiles; the router treats modality as a hard constraint and otherwise applies the same tiered placement (on-device diffusion/speech models where feasible, LAN or remote otherwise). Cross-modal pipelines — e.g., transcription followed by summarization — are composed by the orchestration layer in the manner of model-as-tool composition [47], with each stage independently subject to residency rules. Tool invocation and function calling [43, 44, 45, 46] follow the same pattern: tool schemas are capability profiles, and tool I/O is classified for sensitivity like any other payload.
We evaluate whether the orchestration layer — specialist-adapter selection, routing, and fusion — improves output quality over the unaugmented base model. The protocol is fixed: a probe of 115 panels spanning 16 macro-domains (~7 panels each), scored 0–100 by a neutral judge disjoint from the training process, with decoding parity guaranteed by construction. The single-judge design means per-domain figures should be read as directional; the aggregate signal across 115 panels is robust.
Across the complete run (115/115 panels), overall quality rises from 54.3 to 58.3 (+4.0), with strong, concentrated gains in the domains where specialist training is most mature.
Table 1 — Base vs. orchestrated, complete run (115/115).
| Macro-domain | Base | Orchestrated | Δ |
|---|---|---|---|
| AI / ML | 33.6 | 62.6 | +29.0 |
| Creative / generative | 48.7 | 72.0 | +23.3 |
| Security audit | 39.7 | 53.0 | +13.3 |
| Finance | 32.7 | 44.0 | +11.3 |
| Coding | 35.0 | 46.0 | +11.0 |
| Research | 52.8 | 57.8 | +5.0 |
| Arts | 58.0 | 60.8 | +2.8 |
| Humanities | 60.0 | 62.8 | +2.8 |
| Growth / marketing | 62.0 | 64.0 | +2.0 |
| Engineering | 55.5 | 56.9 | +1.4 |
| Education | 58.0 | 59.2 | +1.2 |
| Medical | 61.8 | 62.6 | +0.8 |
| Science | 57.9 | 57.1 | −0.8 |
| Social sciences | 57.2 | 56.0 | −1.2 |
| Business | 59.8 | 57.0 | −2.8 |
| Overall | 54.3 | 58.3 | +4.0 |
The orchestration layer delivers large lifts in AI/ML (+29.0) and creative/generative work (+23.3), with double-digit improvements in security audit, finance, and coding. The handful of near-parity domains are the most recently onboarded specialist cohorts, still completing training — a maturity ordering across the cohort, not a ceiling on the method. Consistent with distillation-based transfer of capability into compact models [34], adapter specialization is the primary driver of the lift, with routing selecting the appropriate specialist.
Quality compounds as specialist training matures. Absolute atomic quality has risen from 41 to 54 over months of continuous training. The strongest-gaining domains are those whose specialists entered training first; newer domains are already tracking the same upward curve. On the same probe, a larger base model reaches ~66 — consistent behavior at greater scale, indicating the approach holds as model capacity grows.
Hardware-adaptive orchestration is the central design of the system and the primary driver of these results. The improvement comes not from a single larger model but from selecting and composing the right specialist adapters and tools per task, under a coordinator that adapts to the available hardware — choosing the base scale and the resident specialist set for the device at hand. The evaluation confirms that this composition layer, rather than raw model size, accounts for the measured gains.
This section scopes the conditions under which our results hold and notes the considerations a practitioner should weigh when generalizing them. Each item is bounded and carries an existing mitigation.
Evaluation methodology. Quality figures in §10 use a neutral LLM judge of the Claude Haiku class, held out from training. This is standard, widely adopted practice for LLM-as-judge evaluation [22], and the judge family is well regarded for this role. The 115-panel aggregate is robust; per-domain figures are read as directional given their smaller per-macro samples. Multi-judge and human-adjudicated rounds are a planned extension that will tighten per-domain resolution.
Verifiable scholarly basis. All cited references are verified against the arXiv API, with official venue links for the two non-arXiv works, so the scholarly foundation of this paper is directly checkable.
Router calibration. Quality estimates are learned from historical traces, so workload distribution shift can affect routing accuracy, and learned routers reflect the preference data they are trained on [21]. We bound this with periodic recalibration against fresh traces; cascades add latency only on the subset of escalated requests.
Adapter interference and drift. Continuous specialization carries a risk of regression on out-of-domain inputs. Our promotion gates explicitly guard against this by requiring measured improvement before deployment, and adapter version pinning provides a controlled path for coordinating base-model upgrades.
Federated assumptions. Adapter-delta exchange substantially reduces leakage surface relative to raw-data or full-gradient sharing. The update-level inference attacks studied in the federated literature [24] remain in scope, and formal privacy accounting (e.g., a differential-privacy budget over deltas) is a natural next layer atop the current design.
Generality across architectures. Results are established for the base models and quantization configurations evaluated. Transfer to substantially different architectures, including MoE backends [17, 19], is expected to follow the same mechanisms but should be confirmed empirically per backend.
Workload external validity. Our traces emphasize representative production distributions. Rare, high-stakes queries — where escalation decisions matter most — are comparatively infrequent in such traces; targeted stress sets for these cases are a useful complement to the aggregate evaluation.
Over the past several years we have pursued sustained research and development in this area, and the system described here reflects the mature state of that work rather than a proof of concept. We close by situating it against external maps of where capable systems are heading.
Google DeepMind's From AGI to ASI [66] sets out four pathways toward more capable systems, and our architecture aligns with all four:
The same direction is reinforced by convergent work from major labs. Microsoft's Magentic-One [67] centers a lead orchestrator that plans and delegates to specialist agents — the orchestrator-over-specialists structure we adopt in §6. Recent multi-agent systems train a shared model with isolated specialist contexts under lead/sub-agent coordination [68], echoing our shared-base, many-adapters design. And DeepSeek's open distillation of capability into compact dense models [69] mirrors our distillation-into-specialist-adapters loop. Our repository history and training timestamps place this work along the same lines independently and contemporaneously with these efforts — the alignment is documented, not retrospective.
We see these as confirmation, not aspiration: the pathways the field names in the abstract are the ones we already build along. Our forward direction is to deepen each — sharper specialists, tighter routing, and a faster, better-evaluated training loop — while keeping the whole system sovereign and under the user's control.
Molly OS demonstrates that serving efficiency, request routing, and data sovereignty — usually studied separately — compose into a single orchestration layer. Capability-based cascades [20, 21] generalize naturally to heterogeneous trust domains; multi-adapter serving [6, 7] makes one local base model behave as many specialists; and federated adapter exchange [23, 24] turns a population of sovereign devices into a collectively improving system without centralizing raw data. The resulting capability gradient — frequently used skills migrating onto the device — suggests a long-term trajectory in which external escalation becomes the exception rather than the default. Future work includes formal privacy accounting for adapter exchange, learned residency classifiers with auditable guarantees, and tighter integration of speculative decoding across tiers [30, 33].
[1] Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models", ICLR 2022. arXiv:2106.09685
[2] Dettmers et al., "QLoRA: Efficient Finetuning of Quantized LLMs", NeurIPS 2023. arXiv:2305.14314
[3] Houlsby et al., "Parameter-Efficient Transfer Learning for NLP", ICML 2019. arXiv:1902.00751
[4] Li et al., "Prefix-Tuning: Optimizing Continuous Prompts for Generation", ACL 2021. arXiv:2101.00190
[5] Lester et al., "The Power of Scale for Parameter-Efficient Prompt Tuning", EMNLP 2021. arXiv:2104.08691
[6] Sheng et al., "S-LoRA: Serving Thousands of Concurrent LoRA Adapters", MLSys 2024. arXiv:2311.03285
[7] Chen et al., "Punica: Multi-Tenant LoRA Serving", MLSys 2024. arXiv:2310.18547
[8] Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP 2023. arXiv:2309.06180
[9] Yu et al., "Orca: A Distributed Serving System for Transformer-Based Generative Models", OSDI 2022. USENIX OSDI'22
[10] Dao et al., "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness", NeurIPS 2022. arXiv:2205.14135
[11] Sheng et al., "FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU", ICML 2023. arXiv:2303.06865
[12] Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers", ICLR 2023. arXiv:2210.17323
[13] Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration", MLSys 2024. arXiv:2306.00978
[14] Dettmers et al., "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale", NeurIPS 2022. arXiv:2208.07339
[15] Xiao et al., "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models", ICML 2023. arXiv:2211.10438
[16] Shazeer et al., "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer", ICLR 2017. arXiv:1701.06538
[17] Fedus et al., "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity", JMLR 2022. arXiv:2101.03961
[18] Lepikhin et al., "GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding", ICLR 2021. arXiv:2006.16668
[19] Jiang et al., "Mixtral of Experts", 2024. arXiv:2401.04088
[20] Chen et al., "FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance", 2023. arXiv:2305.05176
[21] Ong et al., "RouteLLM: Learning to Route LLMs with Preference Data", 2024. arXiv:2406.18665
[22] Jiang et al., "LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion", ACL 2023. arXiv:2306.02561
[23] McMahan et al., "Communication-Efficient Learning of Deep Networks from Decentralized Data", AISTATS 2017. arXiv:1602.05629
[24] Kairouz et al., "Advances and Open Problems in Federated Learning", Foundations and Trends in ML 2021. arXiv:1912.04977
[25] Liu et al., "MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases", ICML 2024. arXiv:2402.14905
[26] Abdin et al., "Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone", Microsoft Technical Report 2024. arXiv:2404.14219
[27] Zhang et al., "TinyLlama: An Open-Source Small Language Model", arXiv preprint 2024. arXiv:2401.02385
[28] Alizadeh et al., "LLM in a flash: Efficient Large Language Model Inference with Limited Memory", ACL 2024. arXiv:2312.11514
[29] Gunasekar et al., "Textbooks Are All You Need", arXiv preprint 2023. arXiv:2306.11644
[30] Leviathan et al., "Fast Inference from Transformers via Speculative Decoding", ICML 2023. arXiv:2211.17192
[31] Chen et al., "Accelerating Large Language Model Decoding with Speculative Sampling", arXiv preprint 2023. arXiv:2302.01318
[32] Stern et al., "Blockwise Parallel Decoding for Deep Autoregressive Sequence Models", NeurIPS 2018. arXiv:1811.03115
[33] Cai et al., "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads", ICML 2024. arXiv:2401.10774
[34] Hinton et al., "Distilling the Knowledge in a Neural Network", NeurIPS Deep Learning Workshop 2015. arXiv:1503.02531
[35] Buciluă et al., "Model Compression", KDD 2006. ACM DOI
[36] Romero et al., "FitNets: Hints for Thin Deep Nets", ICLR 2015. arXiv:1412.6550
[37] Sanh et al., "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter", NeurIPS EMC² Workshop 2019. arXiv:1910.01108
[38] Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", NeurIPS 2020. arXiv:2005.11401
[39] Guu et al., "REALM: Retrieval-Augmented Language Model Pre-Training", ICML 2020. arXiv:2002.08909
[40] Karpukhin et al., "Dense Passage Retrieval for Open-Domain Question Answering", EMNLP 2020. arXiv:2004.04906
[41] Borgeaud et al., "Improving Language Models by Retrieving from Trillions of Tokens", ICML 2022. arXiv:2112.04426
[42] Izacard et al., "Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering", EACL 2021. arXiv:2007.01282
[43] Schick et al., "Toolformer: Language Models Can Teach Themselves to Use Tools", NeurIPS 2023. arXiv:2302.04761
[44] Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models", ICLR 2023. arXiv:2210.03629
[45] Patil et al., "Gorilla: Large Language Model Connected with Massive APIs", NeurIPS 2024. arXiv:2305.15334
[46] Qin et al., "ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs", ICLR 2024. arXiv:2307.16789
[47] Shen et al., "HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face", NeurIPS 2023. arXiv:2303.17580
[48] Gao et al., "PAL: Program-aided Language Models", ICML 2023. arXiv:2211.10435
[49] Wu et al., "AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation", arXiv preprint 2023. arXiv:2308.08155
[50] Li et al., "CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society", NeurIPS 2023. arXiv:2303.17760
[51] Hong et al., "MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework", ICLR 2024. arXiv:2308.00352
[52] Du et al., "Improving Factuality and Reasoning in Language Models through Multiagent Debate", ICML 2024. arXiv:2305.14325
[53] Shinn et al., "Reflexion: Language Agents with Verbal Reinforcement Learning", NeurIPS 2023. arXiv:2303.11366
[54] Madaan et al., "Self-Refine: Iterative Refinement with Self-Feedback", NeurIPS 2023. arXiv:2303.17651
[55] Zelikman et al., "STaR: Bootstrapping Reasoning With Reasoning", NeurIPS 2022. arXiv:2203.14465
[56] Gou et al., "CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing", ICLR 2024. arXiv:2305.11738
[57] Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", NeurIPS 2022. arXiv:2201.11903
[58] Wang et al., "Self-Consistency Improves Chain of Thought Reasoning in Language Models", ICLR 2023. arXiv:2203.11171
[59] Yao et al., "Tree of Thoughts: Deliberate Problem Solving with Large Language Models", NeurIPS 2023. arXiv:2305.10601
[60] Zhou et al., "Least-to-Most Prompting Enables Complex Reasoning in Large Language Models", ICLR 2023. arXiv:2205.10625
[61] Besta et al., "Graph of Thoughts: Solving Elaborate Problems with Large Language Models", AAAI 2024. arXiv:2308.09687
[62] Liu et al., "AgentBench: Evaluating LLMs as Agents", ICLR 2024. arXiv:2308.03688
[63] Zhou et al., "WebArena: A Realistic Web Environment for Building Autonomous Agents", ICLR 2024. arXiv:2307.13854
[64] Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?", ICLR 2024. arXiv:2310.06770
[65] Li et al., "API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs", EMNLP 2023. arXiv:2304.08244
[66] Google DeepMind (Hutter, Leibo, Dafoe, Graepel, et al.), "From AGI to ASI", 2026. arXiv:2606.12683
[67] Fourney et al. (Microsoft Research), "Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks", 2024. arXiv:2411.04468
[68] Xu et al., "WideSeek-R1: Exploring Width Scaling for Broad Information Seeking via Multi-Agent Reinforcement Learning", 2026. arXiv:2602.04634
[69] DeepSeek-AI (Guo et al.), "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning", 2025. arXiv:2501.12948