From Floats to Film — A Zero-Assumption Guide to ComfyUI, Diffusion & AI Video

Lee

Published at Oct 8, 2026 · 25 mins to read

• #AI
• #ComfyUI
• #Tutorial
• #Deep Learning

Contents

• Part 0 — The Three-Minute Movie
• Part 1 — Everything Is Numbers
  • 1.1 Floats: what a number looks like
  • 1.2 Vectors are points
  • 1.3 Weights: the model’s knobs
  • 1.4 The forward pass: read-only arithmetic
  • 1.5 Softmax and temperature (brief)
• Part 2 — Getting Language In
  • 2.1 Tokenizers and vocabularies
  • 2.2 Embeddings: meaning as location
  • 2.3 CLIP: one space for text and images
• Part 3 — The Generation Trick
  • 3.1 The VAE: the compression translator
  • 3.2 Latent space: where the model actually works
  • 3.3 Diffusion: kneading noise into structure
  • 3.4 The sampler chain: seed, guider, schedule, loop
  • 3.5 Distillation: how 8 strokes replaced 20
  • 3.6 LoRA: a small box of knobs beside the big one
  • 3.7 Quantization: fewer bytes per knob
• Part 4 — Building Workflows
  • 4.1 The ComfyUI mental model
  • 4.2 MVP 1: text-to-image, seven nodes
  • 4.3 MVP 2: upgrades across the generations
  • 4.4 MVP 3: text-to-video
  • 4.5 MVP 4: audio-visual generation with reference images
  • 4.6 How to read any workflow
  • 4.7 Your first week: roadmap and pitfalls
• Closing

Part 0 — The Three-Minute Movie

You are going to turn a block of random numbers into a short movie — an actual file with pictures and sound, produced on your own machine.

Here is the entire trick in one paragraph:

Start with a huge list of random numbers.
Hand that list, plus a text script, to a trained “sculptor” that repeatedly pushes the numbers toward “what the script describes.”
After a fixed number of strokes, hand the result to a “translator” that converts the numbers back into pixels and sound.

Everything else in this article — floats, weights, tokens, latents, samplers, LoRA, quantization — is just making that one paragraph precise.

The cast you will meet in ComfyUI:

The full pipeline at a glance:

Reading contract: every bold term is defined in the paragraph where it first appears, or was defined earlier. Nothing is assumed.

Part 1 — Everything Is Numbers

Everything you will do in ComfyUI — loading a model, writing a prompt, generating an image or video — is ultimately about lists of numbers.
This part explains what those numbers are, how they are organized, and why that organization lets a model turn noise into pictures.

By the end of Part 1 you will understand the three building blocks that every later workflow is made of: floats, vectors (points), and weights.

1.1 Floats: what a number looks like

A float is simply a number with limited precision. Computers cannot store every possible real number exactly. Instead, they store an approximation using a fixed number of bits.

You can see this limitation immediately. Open any browser console and type:

0.1 + 0.2

You will get 0.30000000000000004, not 0.3. This is not a bug. 0.1 and 0.2 cannot be represented exactly in the floating-point format, so the computer stores the closest values it can, and the tiny errors remain after addition.

Two practical facts matter for everything that follows:

1. Floats come in different sizes.
   A 32-bit float (fp32) keeps about seven significant decimal digits.
   A 16-bit float (fp16) keeps about three.
   Smaller formats use less memory but introduce more rounding error. You will meet fp16, fp8, and even 4-bit formats later when we talk about quantization.

2. A float by itself has no meaning.
   The number 0.8 is not “warm” or “cat-like.” Meaning comes only from the role the system assigns to it — its position in a larger list and how that list is used.

Here is the precision ladder you will see in model filenames:

The last column is the one you can use immediately: model size ≈ parameter count × bytes per number. When you see fp16 or fp8 in a filename, you are looking at a row of this table.

Rounding errors accumulate. Long chains of arithmetic on low-precision floats can produce visible artifacts. Production systems therefore keep sensitive intermediate values in higher precision and only reduce precision for storage or speed.

Every weight, every latent value, and every pixel you will meet later is one of these approximate numbers.

1.2 Vectors are points

One float is a single reading. Stack many floats together and something more useful appears.

• One float is a point on a number line.
• Two floats (latitude, longitude) pin a location on a map.
• Three floats locate a position in a room.

The same idea extends without change: a list of N floats is a point in N-dimensional space. “An array of 4,096 floats” and “a point in 4,096-dimensional space” are two descriptions of the exact same object. The word “dimension” here simply means the number of independent coordinates.

Important clarification: the dimensions of these spaces have nothing to do with the pixels of an image. A 512×512 picture may be represented by a latent containing 16,384 floats. That is a 16,384-dimensional point. None of those axes corresponds to “the pixel at (200, 300).” The axes are abstract directions of variation that the model has found useful.

The single most useful mental rule in this entire field is:

Near means similar.

Two points that sit close together represent similar things. Two words used in similar contexts land close together. Two pictures of the same face land close together. This is the fundamental reason text can steer image generation.

Two simple operations on points power many later techniques:

• Averaging two points lands on their midpoint. A 60/40 blend of two images is, at the latent level, just a weighted average of two points.
• Distance between two points measures how different they are.

You will never calculate these distances by hand across thousands of coordinates. What matters is that the operations are uniform: whole points in, whole points out, the same rule applied to every coordinate.

In ComfyUI, the latent you see flowing between nodes is exactly this kind of point.

1.3 Weights: the model’s knobs

Data is points. A model is a function that takes a point and produces a new point. What makes one model different from another is the large collection of numbers stored inside it — the weights.

Think of an audio mixing desk with thousands of faders. An incoming signal is split into channels; each channel is scaled up or down by a fader; the channels are summed and passed to the next row of faders. A large positive weight means “this channel matters.” A weight near zero means “ignore this.” A negative weight means “this channel should reduce the output.”

A neural network is the same idea pushed to extreme scale: instead of audio channels, arbitrary lists of floats; instead of smooth faders, simple multiply-and-add; instead of thousands of knobs, billions.

Each individual weight is called a parameter. When people say “a 6-billion-parameter model,” they mean a file that contains six billion floating-point numbers arranged as knobs. There is no code and no explicit rules inside the file — only the carefully tuned values of those knobs.

These knob positions are not set by hand. They come from training: show the model many examples, measure how wrong its outputs are, nudge every knob a tiny amount in the direction that reduces the error, and repeat this process an enormous number of times. Once training finishes, the knobs are frozen into a file.

The crucial property for users is this: when you use the model, the knobs are read-only. You never turn them. You only read their current positions and perform arithmetic. This is why model files are large and why the first generation in ComfyUI is often slow — the entire pile of floats must be loaded from disk into GPU memory. Subsequent runs are faster because the model stays resident.

In ComfyUI terms: every time you load a Checkpoint, you are loading one of these read-only piles of knobs into VRAM.

1.4 The forward pass: read-only arithmetic

Using a model is called inference. The computation it performs is the forward pass: data enters, flows in one direction through layers of multiply-and-add, and exits. Weights are read, never written.

A real diffusion model is this structure repeated at massive scale. Each layer takes an input point, multiplies it by weights, adds biases, and produces a new point.

A modern image or video model performs an enormous number of these multiply-add operations for every sampler step. That arithmetic volume is the main reason generation takes time. Every speed technique you will meet later (fewer steps, distillation, quantization, smaller models) is ultimately a way of doing less of this arithmetic.

In ComfyUI, each step inside a KSampler node is one or more forward passes of this kind.

1.5 Softmax and temperature (brief)

After the final layer of many networks, the raw numbers are turned into a probability distribution by the softmax function. Softmax emphasizes the largest values and suppresses the smaller ones, producing numbers that sum to 1.

Temperature controls how sharp that distribution is:

• Low temperature → the largest value dominates (more deterministic)
• High temperature → the distribution becomes flatter (more random)

You will meet temperature mainly when sampling text or when certain advanced samplers expose it. For the core image and video workflows in Part 4 it stays in the background.

This completes the numerical foundation.
Every concept that follows — embeddings, latents, conditioning, samplers — is a list of floats organized as points, processed by read-only weights through forward passes.

You now have the three building blocks. Part 2 will show how language is turned into points that the model can understand.

Part 2 — Getting Language In

Your prompt is ordinary text. The model’s world, from Part 1, is points made of floats.
This part shows the two-stage translation that turns text into points the model can use, and then shows how text and images are forced to live in the same space.

By the end of Part 2 you will understand three ideas that every later workflow depends on: tokens, embeddings, and shared multimodal space.

2.1 Tokenizers and vocabularies

A model cannot read letters or characters. Before any model sees your prompt, a program called a tokenizer cuts the text into pieces and maps each piece to an integer ID.

Each piece is a token.
Each token has a fixed ID taken from a large lookup list called the vocabulary.

Think of the vocabulary as a giant apartment directory: every apartment (token) has a permanent door-plate number (ID). The directory is built once, together with the model, and never changes.

Real numbers to make the scale concrete:

English averages about 0.75 words per token. A 100-word English prompt is therefore around 130 tokens.

Two practical consequences matter immediately:

1. Frequent is short, rare is long. Common words cost one token. Rare names, unusual spellings, numbers, and emoji usually cost several.
2. Nothing is un-encodable. The tokenizer always falls back to bytes. There is no “unknown word” failure — only higher token cost.

The standard method for building the vocabulary is byte-pair encoding (BPE): start from individual bytes, repeatedly merge the most frequent adjacent pairs, and stop after tens of thousands of merges.

Token-craft rules you can use at once:

• Budgets are real. Older CLIP-style encoders truncate at 77 tokens. Modern encoders accept thousands. Front-load what matters.
• Cost is uneven. A long non-English phrase inside an English prompt can quietly consume a large share of a limited budget.
• The same text always produces the same token IDs. If results differ between runs, the cause is elsewhere (usually the seed).

The tokenizer itself understands nothing. It is only a deterministic cutting-and-numbering machine. All meaning appears in the next stage.

In ComfyUI you rarely see the tokenizer directly, but every text box you fill is first turned into a list of these integer IDs.

2.2 Embeddings: meaning as location

Token IDs are only door-plate numbers. ID 104167 has no geometric relationship to ID 104166.

The fix is a lookup table: one row of floats for every ID.
Each row is called an embedding of its token. The whole table is the embedding table.

From Part 1 you already know what a row of 1,536 floats is: a point in 1,536-dimensional space. Looking up a token’s embedding is simply finding where that token lives.

These point positions are not assigned by hand. They begin as random floats and are shaped by training pressure. The governing idea is the distributional hypothesis: you shall know a word by the company it keeps.

The training task is ordinary: given surrounding words, predict the next (or a hidden) word. To succeed, the model must place words that appear in similar contexts close together.

Three truths about embeddings that remain useful:

1. For the model, the embedding is the meaning. Everything the model “knows” about a word enters through the location of its point and the locations of its neighbors.
2. Similarity measures usage, not logical truth. “Buy” and “sell” sit close together because they appear in almost identical contexts, even though they are opposites. Phrase what you want, not what you reject.
3. Absolute coordinates are arbitrary; relative geometry is real. Only the relationships matter.

Modern encoders add one further stage: attention. Before a point leaves the encoder, neighboring token points are allowed to nudge it. Each neighbor pulls with a strength determined by relevance, and softmax turns those relevance scores into proportions. The same word therefore receives a slightly different final point depending on the words around it.

An encoder that turns a stream of tokens into points is called a text encoder. You will wire one in every ComfyUI workflow that accepts a prompt.

2.3 CLIP: one space for text and images

So far, text and images still live in separate worlds. Generation requires them to touch: a prompt must be able to pull on the machinery that forms images.

The construction that makes this possible is CLIP. Its core idea is simple.

Train two encoders side by side:

• a text encoder that turns a caption into a point,
• an image encoder that turns a picture into a point.

Apply one training pressure: for every matched (picture, caption) pair, pull the two points together; for mismatched pairs, push them apart. This is called contrastive training.

After training, the shared space has a striking property: the point for the phrase “a red apple” sits near the points of actual photographs of red apples. Distance across modalities has become meaningful.

Why this lets text command pixels: a prompt is now a target region in the same space where images live. The diffusion process is trained to push its output toward the region indicated by the prompt.

When you later place a CLIPTextEncode (or equivalent) node in ComfyUI, its output is exactly this: a point (or set of points) that the sampler will keep steering the latent toward.

Three practical consequences for prompting:

1. Write natural sentences on modern encoders. These models were trained on captions, not keyword lists.
2. Prompt engineering is partly distribution matching. Saying things the way the training data said them still helps.
3. Know what the space cannot represent reliably. Counting, precise left/right layout, and rendered text are weak. When a prompt demands them, change tools rather than escalating the wording.

The output of the text encoder on your prompt is called conditioning — the script turned into numbers that the generation process is conditioned on.

You now hold the three language-side pillars: tokens, embeddings, and a shared space in which text can point at images.

This completes the language layer.
Part 3 will show the special trick that turns noise into pictures under the command of those language points.

Part 3 — The Generation Trick

Parts 1 and 2 built foundations shared by almost all of modern AI.
This part covers the special mechanism behind image and video generators — diffusion — plus the three practical techniques that make it fast and controllable: distillation, LoRA, and quantization.

By the end of Part 3 you will know exactly what every major slider and node in a ComfyUI sampler is doing.

3.1 The VAE: the compression translator

Raw pixels make terrible working material for a diffusion model. A single 1080p frame already contains roughly 6 million floats, and most of that information is redundant.

The industry solution is the VAE (variational autoencoder). It is a two-way translator:

• The encoder compresses pixels into a much smaller representation (a latent).
• The decoder expands that representation back into pixels.

For a typical 512×512 image the compression looks like this:

pixels (what you see)          compact floats (what the model works on)
512×512×3  ──[VAE encode]──►  64×64×4  ──[VAE decode]──►  512×512×3

The reduction is roughly 48×. The VAE learns which directions of variation actually matter (structure, lighting, texture) and keeps those.

One sentence that orients every later workflow:
The model thinks in latents; humans see pixels; the VAE translates at the border.

In ComfyUI you will almost always decode only at the very end.

3.2 Latent space: where the model actually works

The compressed representation produced by the VAE is called a latent.
The coordinate system it lives in is latent space.

A latent for a 512×512 image is a 64×64×4 grid of floats — 16,384 numbers. That list is simply one point in a 16,384-dimensional space. A video latent is the same idea with an added time axis; it is still one point, just in a larger space.

Three clarifications that prevent common confusion:

1. A latent is data, not a place. “Latent space” is the coordinate system. A latent is one concrete point inside it — the current working draft.
2. “Latent” does not mean hidden memory or cache. It is just a float array that the VAE knows how to decode.
3. In ComfyUI a LATENT wire carries one (or a batch of) complete points.

Working in latent space is not only smaller; it is better. The VAE discards the high-frequency noise that is hard to generate and keeps the structural information the diffusion model is good at shaping.

3.3 Diffusion: kneading noise into structure

Diffusion has two directions. Only the reverse direction is used at generation time.

• Forward (training only): take a clean latent and gradually add noise until it becomes pure noise.
• Reverse (generation): start from pure noise and repeatedly ask a trained model “which way is cleaner, given the prompt?” Each answer slightly reshapes the latent.

The trained model does not memorize images. It acquires a skill: at every noise level it can point toward plausible structure that matches the conditioning.

A useful metaphor: generation is kneading clay, not carving a statue.
You begin with a featureless lump (pure noise). Each stroke gently reduces whatever disagrees with the target and strengthens whatever agrees. Nothing is removed; the deviations simply become smaller until a coherent image remains.

That is why early preview frames look like fog resolving into blobs, then into objects.

The model’s guess on every stroke contains a small error. The rest of Part 3 is largely about managing or reducing that error.

3.4 The sampler chain: seed, guider, schedule, loop

The diffusion loop is broken into four clear roles in ComfyUI:

Seed
Selects which random starting lump you begin with. Same seed + same everything else = identical result. Professionals usually fix the prompt and sweep seeds, then lock a good seed and refine the prompt.

Guider (CFG)
Asks the model twice — once with the prompt and once without — then steps in the amplified difference direction. Too high and the image “burns.”

Schedule
The list of σ values is the pressure plan. The shift parameter bends the middle of that plan. High shift keeps heavy pressure longer (useful for layout and motion).

The loop
The actual blending rule. All of them drive the same σ schedule; the differences are secondary.

When you look at a KSampler node, every major widget now maps directly to one of these four roles.

3.5 Distillation: how 8 strokes replaced 20

Classic diffusion models need many strokes because each stroke still contains noticeable model error.

Distillation trains a student model to match, in far fewer strokes, the result a teacher model reaches after many strokes. The student is specialized for large steps.

Marketing names: turbo, LCM, lightning, hyper, DMD.

Practical rule:

• With a classic model, more steps usually improve quality.
• With a distilled model, the step count is a correct setting, not a quality dial.

3.6 LoRA: a small box of knobs beside the big one

Full fine-tuning of a multi-billion-parameter model is expensive. Most of the change a fine-tune actually needs is small and low-rank.

LoRA freezes the original weights and learns only a thin correction path: two small matrices A and B whose product is added to the original computation.

y = Wx + B(Ax)

The width of the narrow corridor between A and B is called the rank. Rank 8–64 is typical for style and character LoRAs.

Key user-facing facts:

• Strength is a simple multiplier (1.0 = full trained effect).
• LoRAs can be stacked; their corrections add.
• A LoRA for the wrong base model will fail or behave wrongly.
• Decision order of cost: prompt first, seed second, LoRA third, new checkpoint last.

3.7 Quantization: fewer bytes per knob

Quantization stores the same weights in fewer bytes by mapping them onto a coarser set of representable values.

What usually breaks first is not the average weight but the outliers. If quality looks wrong, step one format up. If everything looks fine and VRAM is the constraint, stay quantized.

You now hold the complete generation toolkit.
Part 4 moves from machinery to the factory floor.

Part 4 — Building Workflows

Everything so far has been machinery.
This part is the factory floor: how ComfyUI turns those parts into a visual program you assemble with boxes and wires, and how to read any workflow you find online.

We will build four minimum-viable pipelines (MVPs) in historical order. Each new generation answers one question: what breaks without this node?

4.1 The ComfyUI mental model

A workflow is a directed graph.

• Each node is one factory station.
• Each wire is a conveyor that carries one typed parcel. The color of the wire is its data type.

The main types:

Three trunks appear in almost every graph: MODEL, CONDITIONING, and LATENT.
The VAE sits at the border between latent and pixels.

Basic Workflow Skeleton:

Text-to-Image (7 Nodes):

Nothing runs while you wire. You press Queue. ComfyUI then resolves the execution order and runs the graph once.

Useful habits:

• Double-click empty canvas to search by exact class name.
• Drag from a socket into empty space to see only compatible nodes.
• Ctrl-B bypasses a node. Prefer bypass over mute.
• Read graphs from the output node backwards.

4.2 MVP 1: text-to-image, seven nodes

This is the 2022-era Stable Diffusion 1.5 pipeline.

1. CheckpointLoaderSimple — unpacks MODEL + CLIP + VAE from one file.
2. CLIPTextEncode (×2) — positive and negative prompts → CONDITIONING.
3. EmptyLatentImage — blank latent of pure noise.
4. KSampler — the entire diffusion loop.
5. VAEDecode — latent → pixels.
6. SaveImage — writes the PNG.

Pattern: loaders on the left → conditioning in the middle → sampler as the convergence point → decode and save at the exit.

4.3 MVP 2: upgrades across the generations

Everything added since 2022 answers “what breaks without it?”

• LoraLoader — inserts the low-rank side-path.
• Image-to-image — start from an encoded picture; denoise controls how much original clay is kept.
• Inpainting — same idea plus a MASK.
• ControlNet-style conditioning — targets can be drawn as well as written.
• The loader split (Flux era and later) — UNETLoader + CLIPLoader + VAELoader. You now assemble the bundle yourself.

Adding a LoRA:

Modern Workflow Pattern:

The structural pattern remains identical.

4.4 MVP 3: text-to-video

The latent gains a time axis.

An image latent is (width × height × channels).
A video latent is (width × height × channels × frames).
Video VAEs also compress time, which is why legal frame counts land on particular grids.

Substitutions:

• Loaders become the modern trio.
• Blank canvas becomes a video latent node that also asks for length in frames.
• Exit becomes a video combiner.

Text-to-Video:

Video usually wants a longer heavy-hand phase (higher shift). Cost scales with pixels × frames. Prompt for camera behavior explicitly. Length is measured in frames, not seconds.

4.5 MVP 4: audio-visual generation with reference images

Two extensions:

1. The assembler node — one node accepts raw ingredients (CLIP, VAEs, prompt, reference images, resolution, length) and emits complete CONDITIONING plus a ready-made blank audio-visual latent. Reference images implement identity locking: they preserve who, not pose.
2. Dual-clock scheduling — sound and picture denoise at different tempos, so the schedule carries two shift values inside one loop.

A turbo LoRA can still be chained on the MODEL trunk. The trunk pattern is unchanged.

4.6 How to read any workflow

Four passes on any stranger’s graph:

1. Find the loaders. Note bundled vs split, and any LoRA chains.
2. Follow the three trunks by color: MODEL, CONDITIONING (how many authors?), LATENT (empty or encoded?).
3. Locate the convergence — KSampler or the advanced quartet. Read its widgets with the vocabulary of Section 3.4.
4. Walk the exits — decode(s) and save(s).

Prefer the JSON over screenshots. When adapting a downloaded graph, change only one variable per queue.

4.7 Your first week: roadmap and pitfalls

Install ComfyUI Desktop is the lowest-friction route. Install ComfyUI Manager so you can add the custom-node packs required by later MVPs.

Model folders (under the ComfyUI install):

Expectations
MVP 1 runs on modest hardware. Video MVPs want more VRAM and time. When speed or memory is the problem, lower resolution and length first.

Three classic day-one errors

1. Wrong folder → widget list empty.
2. Precision mismatch → black frames or NaNs. Step one format up the ladder of Section 3.7.
3. Resolution / length too high for VRAM → crash or extreme slowdown. Reduce size first.

When something fails, read the server console first. The traceback almost always names the failing node class.

Closing

You now have the complete path:

• Part 1 — the numerical atoms
• Part 2 — language turned into points
• Part 3 — the generation machinery
• Part 4 — the factory floor and the four progressive pipelines

Every workflow you will ever open is a variation of the same sentence:

loaders → optional patches → conditioning → sampler → decode → save.

The rest is literacy and single-variable experiments.

From floats to film: you now know every station on the line. Open the canvas, wire the seven nodes of MVP 1 from memory, and let the first image render.