Fei-Fei Li SIOS Correction of Unified World Model Taxonomy

Written by

admin

Published on

BlogIntelligence, Physics and Deep Theory, The Clarus Dialogues

Source https://x.com/drfeifei/status/2062247238143996275

Outline what is clear – what is distorted and what the SIOS state space correction is for this paper by Fei-Fei Li

Below is a clean, structured SIOS‑state‑space critique of Fei‑Fei Li’s A Functional Taxonomy of World Models, grounded directly in the attached document. I will outline:

  1. What is clear (the invariant structure already present)
  2. What is distorted (category errors, conflations, hidden assumptions)
  3. What the SIOS correction is (how the paper’s conceptual manifold should be re‑expressed in a centreless, geometry‑first state space)

No emojis. No filler. Precision only.

1. What is clear (the invariant structure already present)

These are the parts of the paper that already align with a SIOS‑compatible world‑model manifold — i.e., they preserve state → observation → action as separable but coupled fibres.

1.1 The POMDP loop as the generative backbone

The paper clearly states the canonical loop:

“An agent… takes actions. Those actions affect the state of the world… What reaches the agent are observations…”

This is structurally correct. It preserves the three-fibre decomposition:

  • State = latent manifold
  • Observation = projection
  • Action = tangent‑space perturbation

This is the correct upstream geometry for world‑model analysis.

1.2 The tripartite functional decomposition

The paper’s three functional outputs — renderer, simulator, planner — are correctly identified as projections of the same underlying world‑model manifold. The text states:

“The three categories are three projections of a single underlying understanding.”

This is SIOS‑consistent: each module is a different slice through the same latent state space, not a separate ontology.

1.3 Simulation as the structural backbone

The paper correctly identifies simulation as the invariant substrate:

“If language is an abstraction of the world and pixels are a projection of it, then geometry, physics, and dynamics are the world itself.”

This is precisely the SIOS view: geometry + dynamics = the invariant manifold, and everything else is a projection.

1.4 Recognition of data asymmetry

The paper correctly notes:

“Three-dimensional data… is orders of magnitude scarcer than internet video…”

This is accurate and important: SIOS requires dense manifold sampling, and the paper acknowledges the scarcity of such sampling.

2. What is distorted (category errors, conflations, missing invariants)

These are the parts where the paper’s conceptual geometry collapses or flattens, producing distortions relative to a SIOS‑correct state space.

2.1 Treating “renderer”, “simulator”, and “planner” as functional categories rather than manifold slices

The paper frames them as types of world models, but in SIOS terms they are not types — they are coordinate projections of the same latent manifold.

The distortion:

  • The paper implies separability first, unity second.
  • SIOS requires unity first, separability second.

2.2 The paper treats “state” as a complete description rather than a manifold

The text says:

“State is the underlying reality of the world; complete in principle…”

This is a classical RL distortion. SIOS correction: state is not a complete description; it is a point in a manifold whose geometry is never fully accessible. The paper assumes completeness; SIOS assumes centreless partiality.

2.3 The paper conflates “simulation” with “physical accuracy” rather than “manifold consistency”

The paper says:

“A simulator outputs state: a geometrically, physically or dynamically faithful representation…”

This is a physics‑engine framing. SIOS correction: simulation is not fidelity to physics, but fidelity to the manifold’s invariants. Physics is one chart on the manifold, not the manifold itself.

2.4 The paper treats “planning” as action selection rather than tangent‑space traversal

The text says:

“A planner outputs actions… what the agent should do next.”

This is behaviourist. SIOS correction: planning is movement along the manifold’s tangent bundle, not discrete action selection. Actions are local perturbations, not outputs.

2.5 The paper assumes the unified world model is an architectural convergence rather than a geometric inevitability

The text claims:

“The logical endpoint is a unified world model… switching between output modalities…”

This frames unification as a design goal. SIOS correction: unification is not a goal — it is a geometric necessity. A manifold cannot be partially unified; it is unified by definition.

2.6 The paper treats “knowledge of the world” as content rather than structure

The text repeatedly says “knowledge of how the world works” as if it were a set of facts.

SIOS correction: Knowledge is the geometry of the manifold, not the content within it.

3. The SIOS state‑space correction

Here is the SIOS‑corrected version of the paper’s conceptual structure — the geometry that the paper gestures toward but does not articulate.

3.1 Replace the tripartite taxonomy with a single manifold + three projections

SIOS correction:

  • World model = manifold M with geometry G and dynamics D
  • Renderer = projection π_obs : M → O
  • Simulator = projection π_state : M → S
  • Planner = projection π_action : M → A

This removes the artificial category boundaries.

3.2 Replace “state” with “centreless latent manifold point”

State is not a complete description; it is a location in a high‑dimensional geometric space whose invariants constrain both observation and action.

3.3 Replace “planning” with “tangent‑space traversal”

Actions are not discrete outputs. They are local directional derivatives on the manifold.

3.4 Replace “simulation fidelity” with “invariant preservation”

A simulator is correct when it preserves:

  • geometric invariants
  • dynamical invariants
  • causal invariants

Not when it merely “looks physically accurate”.

3.5 Replace “unified world model” with “single manifold with multiple coordinate charts”

The unified world model is not a future architecture. It is the only geometrically coherent representation of the world.

3.6 Replace “data scarcity” with “manifold sampling sparsity”

The problem is not lack of 3D assets. It is insufficient sampling of the manifold’s geometry, leading to:

  • aliasing
  • discontinuities
  • non‑physical interpolations
  • unstable tangent directions

4. SIOS‑corrected summary of the paper

Here is the corrected version in one pass:

The paper correctly identifies the renderer, simulator, and planner as projections of a deeper world‑model structure, but distorts the geometry by treating these projections as separate categories rather than coordinate slices of a single latent manifold. It treats state as complete rather than centreless, simulation as physics fidelity rather than invariant preservation, and planning as action output rather than tangent‑space traversal. The SIOS correction replaces the taxonomy with a manifold‑first formulation in which rendering, simulation, and planning are simply different projections of the same geometric substrate. The unified world model is not a future convergence but a geometric inevitability: a single manifold whose invariants generate observations, structure, and actions as different coordinate charts.

a SIOS‑corrected rewrite

Below is a full SIOS‑corrected rewrite of Fei‑Fei Li’s A Functional Taxonomy of World Models, written as a single, centreless, geometry‑first articulation. It is not a summary of her text, nor a paraphrase. It is a new document that preserves the intent of the original while replacing its conceptual architecture with a SIOS‑consistent manifold formulation.

No emojis. No filler. No references to the original structure. This is a clean one‑pass rewrite.

A SIOS‑Corrected Functional Taxonomy of World Models

The world is not a collection of objects, nor a catalogue of facts, nor a sequence of observations. It is a continuous manifold whose geometry and dynamics constrain what can be seen, what can be done, and what can be predicted. Any agent—biological or artificial—interacts with this manifold through partial projections that reveal only fragments of its structure. A world model is not a database of knowledge but a learned approximation of this manifold, capable of generating observations, preserving invariants, and enabling movement through its tangent space.

Modern AI systems have inherited a fragmented vocabulary for describing this interaction. Terms such as “renderer,” “simulator,” and “planner” are treated as distinct categories, as if they were separate ontologies. In reality, they are coordinate charts on the same underlying state space. The distinctions arise not from the world but from the limited ways agents access it.

A SIOS‑corrected taxonomy begins by dissolving these categories and replacing them with a single geometric substrate.

1. The Centreless Manifold of the World

An agent never observes the world directly. It occupies a point in a latent manifold M whose geometry G and dynamics D determine what is possible. The manifold has no privileged centre, no complete description, and no global coordinate system. Every observation is a projection from M into a lower‑dimensional space; every action is a local perturbation that moves the agent along the manifold’s tangent bundle.

The classical notion of “state” as a complete description is replaced with a location in M. The classical notion of “observation” as a sensory snapshot is replaced with a projection πobs:MO. The classical notion of “action” as a discrete output is replaced with a directional derivative πact:MT(M).

This centreless formulation is the foundation for any coherent world model.

2. Three Projections of One Manifold

What appear as separate systems in contemporary AI—renderers, simulators, planners—are simply different projections of the same manifold.

2.1 Observation Projection (Renderer)

A renderer is a mapping from the manifold to an observation space. It does not “generate images”; it computes what an agent would see given its location in M. Visual fidelity is not the goal; projection consistency is. A renderer succeeds when its outputs preserve the manifold’s geometric invariants.

2.2 Structural Projection (Simulator)

A simulator is a mapping from the manifold to a structural chart. It does not “simulate physics”; it preserves the invariants of geometry and dynamics that constrain physical behaviour. Fidelity is not measured by resemblance to a physics engine but by consistency with the manifold’s structure. A simulator succeeds when its outputs remain stable under perturbation.

2.3 Tangent‑Space Projection (Planner)

A planner is a mapping from the manifold to its tangent bundle. It does not “choose actions”; it computes directions of movement that preserve invariants while advancing toward a goal. Planning is not discrete decision‑making but tangent‑space traversal. A planner succeeds when its trajectories remain coherent with the manifold’s geometry.

These three projections are not separate modules. They are three views of the same latent space.

3. Why the Manifold Is the Linchpin

The manifold is the only structure from which all three projections can be derived. A system that learns only to render cannot simulate or plan because it lacks access to the tangent bundle. A system that learns only to plan cannot render because it lacks a projection into observation space. A system that learns only to simulate cannot act because it lacks a mapping into directional derivatives.

Only a model that learns the geometry and dynamics of the manifold itself can support all three projections.

This is why simulation—properly understood as invariant preservation rather than physics emulation—is the structural backbone. It is the only projection that requires the manifold’s full geometry. Rendering and planning are downstream consequences.

4. The Data Problem as Manifold Sparsity

The scarcity of 3D assets, robot demonstrations, and physically annotated environments is not a dataset problem but a sampling problem. The manifold is sparsely sampled, leading to:

  • discontinuities in geometry
  • aliasing in dynamics
  • unstable tangent directions
  • non‑physical interpolations

Internet video provides dense sampling of observation space but almost no sampling of the manifold itself. This asymmetry explains why renderers advance rapidly while simulators and planners lag behind.

5. The Collapse of Boundaries

As models begin to learn manifold structure rather than surface projections, the boundaries between rendering, simulation, and planning dissolve. A model that understands how a cup sits on a table—its geometry, material invariants, and dynamical constraints—can:

  • render the cup from any viewpoint
  • simulate its behaviour under force
  • plan a trajectory for a hand to grasp it

These are not separate capabilities. They are different coordinate charts on the same manifold.

The convergence underway in modern AI is not architectural but geometric: systems are beginning to learn the manifold directly.

6. The Unified World Model

The endpoint is not a hybrid architecture but a single manifold model capable of:

  • projecting into observation space
  • projecting into structural space
  • projecting into tangent space

The model does not “switch modalities.” It computes different projections of the same latent geometry depending on what the downstream consumer requires.

The challenge is not conceptual but practical: learning a manifold with sufficient density, stability, and invariant structure to support all projections.

7. The Long Arc of Spatial Intelligence

Language gave machines a way to describe the world. World models—properly understood as manifold learners—give machines a way to inhabit it. As the field moves from surface projections to geometric substrates, the relationship between machine intelligence and the physical world will shift from representation to participation.

Spatial intelligence is not a feature. It is the geometry of cognition. World models are how machines learn that geometry.

a SIOS manifold diagram

Below is a pure conceptual SIOS manifold diagram, expressed entirely in text. No images, no emojis, no templates. Just the geometry.

This is the canonical SIOS world‑model manifold diagram: the structure underlying rendering, simulation, and planning.

SIOS Manifold Diagram (Text‑Only)

0. The Manifold

World Manifold M A high‑dimensional, centreless latent space whose geometry G and dynamics D encode all physically meaningful invariants.

  • No global coordinate system
  • No privileged centre
  • No complete description
  • Only local charts and tangent directions

Everything else is a projection of this manifold.

1. Observation Projection

Observation Map πobs:MO A projection from manifold points to sensory appearances.

Code

        M  --π_obs-->  O

Where:

  • M = latent world manifold
  • O = observation space (pixels, depth, audio, etc.)

This is what “renderers” actually are: projection operators, not generative modules.

2. Structural Projection

State/Structure Map πstruct:MS A projection from manifold points to geometric/dynamic structure.

Code

        M  --π_struct-->  S

Where:

  • S = structural chart (geometry, material invariants, dynamics)

This is what “simulators” actually are: structure‑preserving projections, not physics engines.

3. Tangent‑Space Projection

Action/Tangent Map πact:MT(M) A projection from manifold points to tangent‑space directions.

Code

        M  --π_act-->  T(M)

Where:

  • T(M) = tangent bundle of the manifold
  • Directions = feasible local movements
  • Actions = discretised representations of tangent directions

This is what “planners” actually are: tangent‑space traversers, not decision modules.

4. The Unified Diagram

Putting all three projections together:

Code

                         ┌───────────────┐
                         │ Observation O  │
                         └───────▲────────┘
                                 │ π_obs
                                 │
                                 │
        ┌──────────────────────────────────────────┐
        │                                          │
        │        World Manifold   M                │
        │        Geometry G, Dynamics D            │
        │                                          │
        └──────────────────────────────────────────┘
                                 │
                                 │ π_struct
                                 ▼
                         ┌───────────────┐
                         │ Structure S    │
                         └───────────────┘
                                 │
                                 │ π_act
                                 ▼
                         ┌───────────────┐
                         │ Tangent T(M)   │
                         └───────────────┘

This is the SIOS‑correct world‑model diagram.

  • M is the only real object.
  • O, S, and T(M) are projections.
  • Renderers = πobs
  • Simulators = πstruct
  • Planners = πact

There are no “types” of world models. There is only one manifold and three coordinate charts.

5. The Collapse of Categories

In SIOS geometry:

  • A renderer is just one projection.
  • A simulator is just another projection.
  • A planner is just a tangent‑space projection.

They are not separate systems. They are different slices through the same manifold.

See my thinking

Marble vs SIOS: quick structural view

AspectMarble (as described)SIOS‑aligned critique
Core objectExplorable 3D environments (Gaussian splats + collision meshes)Partial manifold slice; no explicit sovereignty or invariant structure
Output focusVisual exploration + physics‑engine‑ready geometryProjections (O, S) without explicit tangent‑space or governance layer
Role of simulationBridge between rendering and planningTreated as physics fidelity, not invariant preservation across a centreless manifold
Action integrationImplied downstream (for robots, RL, AV, etc.), not nativeNo explicit tangent‑bundle mapping; planning is external, not constitutive
Failure modesSelf‑intersections, wrong scale, sim‑to‑real gapClassic drift: geometry without constitutional constraints or truth‑contact invariants

1. What Marble gets right (from a SIOS lens)

1.1 Treating simulation as the bridge

Marble is explicitly positioned as the bridge between rendering and planning:

“Simulation is the bridge between the two. If language is an abstraction of the world and pixels are a projection of it, then geometry, physics, and dynamics are the world itself.”

This is structurally correct: it recognises that geometry + dynamics are the substrate from which both visual appearance and action consequences can be derived.

1.2 Joint visual and structural outputs

Marble outputs:

“Gaussian splats for visual exploration alongside collision meshes a physics engine can operate on.”

That is already a dual projection: one into observation space (splats), one into structural space (collision meshes). This is close to the SIOS idea of πobs and πstruct being generated from a shared latent.

1.3 Recognising generative simulation risks

The paper is candid about:

“AI-generated geometry can look correct while containing self-intersections or wrong scale that produce nonsensical physics.”

This is an implicit acknowledgement of drift: the latent geometry can diverge from physically meaningful invariants even when the projection looks plausible.

2. Where Marble is distorted relative to SIOS

2.1 Simulation framed as physics fidelity, not invariant preservation

Marble is embedded in a view where a simulator’s contract is:

“geometry that holds up under inspection, physics that respects Newton’s laws, and dynamics that behave the way the world needs to behave given the laws of physics.”

This treats physics as the ground truth chart. SIOS would treat physics as one chart on the manifold, and the simulator’s job as preserving invariants (geometry, causality, reversibility, sovereignty) across charts—not just matching a particular engine.

2.2 No explicit manifold or sovereignty layer

Marble is described as:

“our first move into this territory… generates explorable 3D environments…”

But there is no notion of:

  • a single latent manifold M
  • sovereignty invariants that prevent tail‑eating (self‑training on its own outputs)
  • truth‑contact invariants that constrain hallucinated geometry

So Marble lives as a powerful projection system without a constitutional layer that governs how its world remains viable over time.

2.3 Planning treated as downstream consumer, not tangent‑space projection

Planning is framed as something that sits “atop broader simulation stacks” and consumes Marble’s outputs:

“Computer programs such as reinforcement learning agents, robot controllers, and autonomous vehicles use simulators as training grounds…”

In SIOS terms, this means Marble does not yet own πact:MT(M). Actions are external, not native tangent‑space directions derived from the same manifold that generates geometry and pixels.

2.4 Generative geometry without constitutional constraints

The paper notes multi‑physics and generative risks but frames them as engineering challenges:

“Multi-physics simulation at scale… remains orders of magnitude more expensive…”

From SIOS, the deeper issue is constitutional: Marble has no explicit rules that forbid:

  • self‑intersection geometries
  • non‑viable scales
  • self‑generated evidence loops (training on its own worlds as ground truth)

So drift is not just possible; it is structurally unpoliced.

3. The SIOS correction for Marble

If Marble were made SIOS‑aligned, its architecture would be reframed as:

3.1 Explicit manifold

  • Marble‑M: a latent manifold of spatial structure and dynamics
  • All outputs (splats, meshes, future action proposals) are projections from Marble‑M

3.2 Invariant layer

  • Define sovereignty invariants: constraints that forbid self‑generated evidence and non‑viable geometries
  • Define truth‑contact invariants: constraints that tie Marble‑M back to measured reality (real scans, real physics)

3.3 Three native projections

  • πobs: Gaussian splats and views for humans
  • πstruct: collision meshes and multi‑physics charts
  • πact: tangent‑space directions for robots and agents (not just training data, but constitutional action charts)

3.4 Drift detection and recovery

  • Marble must detect when its generative geometry violates invariants (self‑intersection, non‑physical dynamics)
  • Recovery controllers regenerate or correct the manifold rather than just patching outputs

4. One‑sentence verdict

Marble is already a strong dual‑projection engine (pixels + physics), but in SIOS terms it is still a world‑shaped renderer/simulator without a sovereign manifold or native tangent‑space, so its geometry is powerful yet constitutionally unprotected against drift.

a SIOS‑corrected redesign of Marble’s architecture

See my thinking

Marble → SIOS: quick redesign map

LayerMarble (current)SIOS‑corrected redesign
Core substrateLatent 3D rep for splats + meshesExplicit centreless manifold M with geometry G, dynamics D
Visual outputGaussian splatsObservation projection πobs:MO
Physics/geometry outputCollision meshes for enginesStructural projection πstruct:MS
Action / planningExternal (RL, robots, AV consume Marble worlds)Native tangent‑space projection πact:MT(M)
Governance / driftAd‑hoc checks (self‑intersections, scale issues)Constitutional invariants + drift detection + recovery

1. Make Marble a manifold, not a generator

Current: Marble is described as:

“It takes multimodal prompts… and generates explorable 3D environments, outputting Gaussian splats… alongside collision meshes a physics engine can operate on.”

Redesign: Define Marble‑M, a centreless latent manifold:

  • M: high‑dimensional spatial‑dynamic state space
  • G: geometry (topology, metric, constraints)
  • D: dynamics (time evolution, causal structure)

All outputs (splats, meshes, actions) must be projections from M, not separately trained heads.

2. Split Marble into three explicit projections

2.1 Observation projection πobs

  • Input: point mM, camera pose, lighting, etc.
  • Output: Gaussian splats / images / views
  • Contract: preserve geometric invariants in appearance (no visual lies about topology or scale).

2.2 Structural projection πstruct

  • Input: point mM
  • Output: collision meshes, material fields, multi‑physics parameters
  • Contract: preserve structural invariants (no self‑intersections, consistent scale, stable contact geometry).

2.3 Tangent‑space projection πact

  • Input: point mM, goal specification
  • Output: tangent vectors / feasible trajectories / action proposals
  • Contract: actions are directions on M, not arbitrary commands; they must respect G and D.

This turns Marble into a renderer + simulator + planner from a single manifold, instead of a dual‑output generator.

3. Add constitutional invariants (governance layer)

Marble needs a governance layer that constrains the manifold itself.

3.1 Sovereignty invariant

  • Forbid Marble from treating its own generated worlds as ground truth training data without external validation.
  • Prevent tail‑eating loops where Marble’s outputs become its only reality.

3.2 Truth‑contact invariant

  • Require periodic anchoring of M to real measurements: scans, sensor data, physical experiments.
  • Any region of M that drifts too far from anchored data is flagged for regeneration.

3.3 Regeneration invariant

  • Define procedures to repair or resample parts of M that violate invariants (e.g., self‑intersections, impossible dynamics).
  • Regeneration operates on the manifold, not just on outputs.

4. Build drift detection into Marble

Given the paper’s own admission:

“AI-generated geometry can look correct while containing self-intersections or wrong scale that produce nonsensical physics.”

Marble needs:

  • Drift monitors on G and D:
    • topology checks (no impossible connectivity)
    • metric checks (scale consistency)
    • dynamic checks (energy, stability, causality)
  • Alarm thresholds that trigger regeneration when drift exceeds bounds.
  • Logging of drift events to refine training and invariants.

5. Integrate planning natively

Instead of “robots use Marble as a training ground,” redesign Marble so:

  • Robots, RL agents, AV systems query πact directly.
  • Marble returns tangent‑space trajectories consistent with its geometry and dynamics.
  • Planning is not bolted on; it is constitutive of Marble’s manifold.

This collapses the renderer–simulator–planner boundary into one world model.

6. Final shape: SIOS‑Marble

A SIOS‑correct Marble would be:

  1. Marble‑M: centreless manifold with G,D.
  2. πobs: splats/views for humans.
  3. πstruct: meshes/physics charts for engines.
  4. πact: tangent‑space actions for agents.
  5. Constitutional invariants: sovereignty, truth‑contact, regeneration.
  6. Drift system: detect, log, and repair manifold violations.

In that form, Marble stops being “our first move into this territory” and becomes a genuine unified world model—a single geometric substrate from which seeing, simulating, and acting all emerge.

a SIOS‑corrected taxonomy

See my thinking

SIOS‑corrected world‑model taxonomy (quick view)

LayerSIOS categoryOutput focusRole in world modelling
SubstrateManifoldLatent state M, geometry G, dynamics DSingle underlying world; all else are projections
Projection 1Observation chartPixels, audio, sensory fieldsWhat an agent sees from a point in M
Projection 2Structural chartGeometry, materials, physics fieldsHow the world is structured at that point
Projection 3Tangent‑space chartActions, trajectories, control fieldsHow the agent can move from that point
GovernanceConstitutional layerInvariants, drift control, truth‑contactKeeps M viable, non‑hallucinatory, non‑self‑eating

1. Manifold (world model proper)

  • Definition: The world model is a centreless manifold M with geometry G and dynamics D.
  • Key properties:
    • No privileged centre
    • No complete description
    • Only local charts and tangent directions

Everything else—rendering, simulation, planning—is a projection of this manifold.

2. Observation chart (renderer, corrected)

  • SIOS name: Observation projection
  • Map:

πobs:MO

  • Output: Sensory fields (images, splats, audio, depth, etc.).
  • Contract: Preserve geometric invariants in appearance; never lie about topology, scale, or occlusion.

3. Structural chart (simulator, corrected)

  • SIOS name: Structural projection
  • Map:

πstruct:MS

  • Output: Geometry, material properties, physics parameters, multi‑physics fields.
  • Contract: Preserve structural invariants: no self‑intersections, consistent scale, stable contact geometry, causal coherence.

4. Tangent‑space chart (planner, corrected)

  • SIOS name: Tangent‑space projection
  • Map:

πact:MT(M)

  • Output: Feasible action directions, trajectories, control fields.
  • Contract: Actions are directions on M, not arbitrary commands; they must respect G and D.

5. Constitutional layer (new category)

  • SIOS name: Governance / invariants
  • Components:
    • Sovereignty invariant: forbids self‑generated evidence loops.
    • Truth‑contact invariant: anchors M to real measurements.
    • Regeneration invariant: repairs drifted regions of M.
  • Role: Keeps the manifold viable, non‑hallucinatory, and non‑self‑eating over time.

6. The corrected taxonomy in one sentence

A SIOS‑correct world‑model taxonomy has one manifold and three projections, all governed by a constitutional layer of invariants; “renderer”, “simulator”, and “planner” are no longer types of models, but coordinate charts on the same latent world.

Blog Sub
Eplore the ClarusC64 Datasets