Three layers, signal to application

Mainstream multimodal AI uses Late Fusion: each modality is compressed to text first, then concatenated. Temporal relations, intensity contrasts, "heart rate rose 3 seconds after they said that" — all lost. Conclusions are coarse, safe, identical for everyone.

Late Fusion (mainstream)
  audio  ──→  ASR        ──→ "boss called me over"     ─┐
  HR     ──→  numeric    ──→ "HR rose from 75 to 110"  ─┼──→  LLM concat:
  vision ──→  caption    ──→ "person in a suit"        ─┘  "you seem to have
                                                            something at work"

Early Fusion (Yunjue)
  audio  ┐
  HR     ┤
  IMU    ┼──→  aligned at raw signal layer → multimodal LLM
  vision ┤      "you got nervous the moment your boss called you over"
  chat   ┘

The model builds a causal chain at the raw-signal layer — "HR rose 3 seconds before that sentence was spoken." Conclusions become precise to the event, the moment, the individual.

Mainstream models cover: video → text (VL), audio → text (ASR / gpt-audio), image → text (most have it natively).

But the combination of HR (dense numeric) + IMU (dense numeric) + HRV (sparse) + audio (continuous waveform) + image (sparse frames) + personal profile (text) + knowledge graph — there is no public-internet training data at scale, because only people wearing always-on multimodal capture devices 24/7 generate it.

This is Yunjue's flank: a new training arena that bypasses the frontal battle with mainstream models. Today we use the alignment capabilities of mainstream multimodal models to collect dense "unusual modalities → human behavior outcome" labels; long-term, we train our own Human-Centric World Model.

All three capabilities below require heart rate, vision, voiceprint, and dialogue to be precisely aligned at the raw-signal layer. Compress each modality to text first, and this information is gone.

These are Early Fusion's current capabilities — and the entry point for accumulating "human behavior outcome" labels that will form the training ground truth of the future Human-Centric World Model.

• In-Situ Self-Evolving paradigm: traditional agents draw a hard line between offline training and online deployment. We propose "inference IS evolution": every inference mutates the system's configuration and immediately feeds the next one.
• Tabula Rasa experiment: the agent starts with an empty toolkit and builds tools entirely through inference-time generate / verify / induce.
• Tool-library convergence: the tools the agent authors converge to a reusable set — only 97 tools synthesized across 2,500 HLE queries — evidence that "general problem-solving" is a learnable, finite, distillable pattern.
• Warm-start transfer: an HLE-evolved toolset bootstraps the other benchmarks; new-tool growth drops to zero on xSciQA / xDS, showing the skills transfer cross-domain.

Code, benchmark scripts, and the versioned tool-generation / modification / merge traces are all CC BY 4.0 open source — auditable and reproducible. Full work in the Yunjue Agent post.

Every night · end-of-day data assembled
   │
   ▼
For each user, run one self-evolution sandbox
   │
   ├─ Audit today        Analyze the event narrative and product-side signals,
   │                     re-examine suspect events (mislabel / composite / boundary)
   │
   ├─ Incremental model  profile / glossary / personal KG / relationship graph /
   │                     tracked items — appended as mode statements, never overwritten
   │
   ├─ Find the gaps      Identify capabilities the user's current Skill library
   │                     does not yet cover
   │
   └─ Author tools       Emit Python tool skeletons and design briefs, validate,
                         and promote verified ones into the shared library
   │
   ▼
Every decision is audited and replayable, node by node

Self-evolution comes in two stages: private evolution (fully automatic, inside a sandbox) + shared extraction (high-value generic tools are de-identified and promoted into the shared library). Skills and tools are private by default; sharing only happens during the "non-personal generic tool" extraction step.

Mainstream "AI assistant" products preload features and ask the user to pick. Result: every user sees the same product, varying only in usage.

Yunjue starts with Zero Skills:

• Day-one feed is empty — the system knows nothing about you, so shows nothing
• The system observes throughout the day and slices activity into event narratives
• After a few days, it identifies "what you do at which times" and writes Skills
• By around two weeks, no two users share the same card library

Slow cold start is a disadvantage — but it's the price of the moat. Each card is a standalone HTML mini-program, which means the same feed shell can host tools, retrospectives, companion dialogue, even bespoke games. The more user-specific Skills accumulate, the higher the switching cost.

Yunjue's self-evolution sandbox runs once a night — reading event narratives, voiceprint profiles, and behavioral patterns to identify capability gaps and author new Skills. There is no fixed feature list; what each user ends up with diverges over time. Typical directions:

All of these directions emerge from a Zero-Skill start — they are not preloaded features and not user-picked from a list.

The commercial impossible-triangle of openness / controllability / economics — traditional approaches can only hit two corners. We believe a self-evolving agent is the viable path across all three. Full argument in Why dynamic self-evolution is the right path for consumer services.

Listening, watching, and measuring heart rate 24/7 looks, from the outside, like always-on capture hardware. This is Yunjue's biggest product burden — and a question engineering has to answer head-on.

Consumer edition (Yunjue's in-house multimodal hardware + iPhone, mass-market user)
  ├─ Option 1: edge-side small model compresses to text summary before upload
  └─ Option 2: edge-side fusion adapter — raw frames / raw data never leave;
              what's uploaded is a fusion vector

Geek edition (developers / power users)
  ├─ Private-server bundle: raw data uploaded, but to YOUR own server, token billing
  └─ Lightweight backend open-sourced: deploy your own IP, hardware-only billing

• Today (deep internal trial): raw data is processed in the cloud under per-device encryption, strict isolation, and no cross-user mixing — the stage where we and our early peers chose to harden the main loop first.
• Mid-term roadmap (after in-house multimodal hardware + edge adapter ship): the consumer edition defaults to an end / edge path — raw data never leaves the end / edge, the cloud only receives a fusion vector. "Whether raw data leaves your device" becomes an explicit user choice, no longer the default.
• Geek-edition options (shipped in parallel): private deployment or lightweight backend open source — raw data fully under your control.
• Tools / Skills are private by default: publishing to the community is an explicit user action. Pre-publish de-identification removes user-specific terms, names, and locations.

The three layers above all serve three long-horizon tracks Yunjue invests in. Remove any one and the remaining two no longer hold.

Early Fusion → Self-Evolving Agent → Human-Centric World Model: all three tracks have corresponding delivery milestones on the product page.