programdemos/pipelines/diffusion.xtl
Diffusion sampling four ways, a second Rosetta Stone: the two
equations of a latent diffusion sampler as a paper prints them, read
by X_eTaL-libraries' Equations; the sampler as a pipeline (Flow);
and in plain X_eTaL, as this repository's diffusion demo runs it.
The pipeline's expansion, the fourth way, is what xetal expand
diffusion.xtl shows. All give the same pictures.
The paper's two forms (bold z a latent, D the decoder, eps_theta the denoiser conditioned on the text embedding of the prompt): X0 = D(zT + sum_{t=T}^{1} [a_t z_t + b_t eps_theta(z_t, t, E_text(prompt)) + sigma_t eps_rand] - z_t) X0 = D(z0) = D(zT + sum_{t=1}^{T} dz_t) The first sum is a recurrence (its term at t uses the z_t the term before made), so it is one step as the paper's summand, applied T times (Equations' scan takes one seed for every element, not a latent per element: ask E13); the second is a plain sum of the steps' differences, and telescopes to the same z0. The networks are called inside the equations by name (eps_theta and D), drawn as the paper draws them.
Data
N : Int
A batch of N latents of K numbers, T steps; the schedule a_t, b_t and sigma_t (the last step takes no noise, as DDIM), and the noise eps_rand of every step, drawn once (a formula stands in for a Gaussian draw, so the run is the same everywhere).
N ← 2
c : Float
The networks, stand-ins with made-up weights (the diffusion demo trains real ones): the text encoder gives the prompt's embedding c; the denoiser is a dense layer over each latent joined with its level and c; the decoder a dense layer over the latent.
c ← 0.5 -0.5
ᵘe̲psTheta : Num a => (Float, a) -> Float
ᵘe̲psTheta ← { (Z, t) → (Z c̲at₂ (N c̲at 3) r̲eshape (f̲loat t) c̲at c) ⁿⁿd̲ense We }
ᵘp̲aper : Char -> Char
The paper's symbols for the networks in the equations' LaTeX.
ᵘp̲aper ← { tex → r#"\operatorname{d_ecode}"# r#"\mathcal{D}"# ˢr̲eplace r#"\operatorname{e_psTheta}"# r#"\epsilon_\theta"# ˢr̲eplace tex }
As a paper prints it
ᵘs̲tep : (Float, Int) -> (Float, Int)
One step of the first sum, the paper's summand at level t: a_t z_t + b_t eps_theta(z_t, t, c) + sigma_t eps_rand, read by Equations over the batch (n) and the latent (k), the denoiser called inside the equation (its argument, Z over the whole grid, is the latents); the level goes down one.
ᵘs̲tep ← { (Z, t) → Zn ← ᵉe̲q< Zn_{n,k} = a_{t-1} \, Z_{n,k} + b_{t-1} \, \operatorname{e_psTheta}(Z_{n,k}, t) + sigma_{t-1} \, noise_{t-1,n,k} (Zn, t − 1) }
ᵉe̲q< expands to
(({ @ →
g3:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g2:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g1:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g4:grid ← N c̲at K
g5:g̲rid ← { k → g4:grid r̲eshape (o̲ffsets k s̲elect g4:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g4:grid) r̲eshape 0 }
g4:grid r̲eshape (({ @ →
g6:grid ← N c̲at K
g7:g̲rid ← { k → g6:grid r̲eshape (o̲ffsets k s̲elect g6:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g6:grid) r̲eshape 0 }
g6:grid r̲eshape ((((
((g1:r̲ank (a, "a", 1, "a[t-1]")) g3:a̲t (
((a, "a", 1, "t-1") g2:c̲heck (t − 1)))) ×
((g1:r̲ank (Z, "Z", 2, "Z[n,k]")) g3:a̲t ((
((Z, "Z", 2, "k") g2:c̲heck (0 + g7:g̲rid 2)) + (2 s̲elect s̲hape Z) ×
((Z, "Z", 1, "n") g2:c̲heck (0 + g7:g̲rid 1)))))) + (
((g1:r̲ank (b, "b", 1, "b[t-1]")) g3:a̲t (
((b, "b", 1, "t-1") g2:c̲heck (t − 1)))) × (ᵘe̲psTheta(
((g1:r̲ank (Z, "Z", 2, "Z[n,k]")) g3:a̲t ((
((Z, "Z", 2, "k") g2:c̲heck (0 + g7:g̲rid 2)) + (2 s̲elect s̲hape Z) ×
((Z, "Z", 1, "n") g2:c̲heck (0 + g7:g̲rid 1))))), t)))) + (
((g1:r̲ank (sigma, "sigma", 1, "sigma[t-1]")) g3:a̲t (
((sigma, "sigma", 1, "t-1") g2:c̲heck (t − 1)))) ×
((g1:r̲ank (noise, "noise", 3, "noise[t-1,n,k]")) g3:a̲t ((
((noise, "noise", 3, "k") g2:c̲heck (0 + g7:g̲rid 2)) + (3 s̲elect s̲hape noise) × (
((noise, "noise", 2, "n") g2:c̲heck (0 + g7:g̲rid 1)) + (2 s̲elect s̲hape noise) ×
((noise, "noise", 1, "t-1") g2:c̲heck (t − 1)))))))))
} @))
} @))ᵘc̲ollect : (Float, Int, Float) -> (Float, Int, Float)
The steps' differences dz_t = z_{t-1} - z_t, a row of N x K per step, collected as the T steps run; then the second form, the sum of them, and the decoder.
ᵘc̲ollect ← { (Z, t, dz) → (Zn, tn) ← ᵘs̲tep (Z, t) (Zn, tn, dz c̲at (1 c̲at N c̲at K) r̲eshape Zn − Z) }
eq : Float
eq ← ᵉe̲q< X0_{n,k} = \operatorname{d_ecode}(zT_{n,k} + \left(\sum_{t=1}^{T} dz_{t-1,n,k}\right))
ᵉe̲q< expands to
(({ @ →
g10:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g9:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g8:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g11:grid ← N c̲at K
g12:g̲rid ← { k → g11:grid r̲eshape (o̲ffsets k s̲elect g11:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g11:grid) r̲eshape 0 }
g11:grid r̲eshape ((ᵘd̲ecode((
((g8:r̲ank (zT, "zT", 2, "zT[n,k]")) g10:a̲t ((
((zT, "zT", 2, "k") g9:c̲heck (0 + g12:g̲rid 2)) + (2 s̲elect s̲hape zT) ×
((zT, "zT", 1, "n") g9:c̲heck (0 + g12:g̲rid 1))))) + ({ @ →
g13:grid ← N c̲at K c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 1..T" } (1 + T − 1))
g14:g̲rid ← { k → g13:grid r̲eshape (o̲ffsets k s̲elect g13:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g13:grid) r̲eshape 0 }
'+ r̲/₃ g13:grid r̲eshape (
((g8:r̲ank (dz, "dz", 3, "dz[t-1,n,k]")) g10:a̲t ((
((dz, "dz", 3, "k") g9:c̲heck (0 + g14:g̲rid 2)) + (3 s̲elect s̲hape dz) × (
((dz, "dz", 2, "n") g9:c̲heck (0 + g14:g̲rid 1)) + (2 s̲elect s̲hape dz) ×
((dz, "dz", 1, "t-1") g9:c̲heck ((1 + g14:g̲rid 3) − 1)))))))
} @)))))
} @))eqpipe : Float
The second form as a pipeline of equations: the latents of noise, the steps' differences added, the decoder (a dense layer, its bias Wd's last row); drawn as math a row a stage.
eqpipe ← @ ᶠˡᵒʷp̲ipe< c#"
ᶠˡᵒʷp̲ipe< expands to
({ @ →
g15:stage ← zT
z0 ← { @ →
it ← g15:stage
"Z0[n,k] | N K" ᵉe̲q< "it[n,k] + (sum t:=1..T | dz[t-1,n,k])"
} @
g16:stage ← "X0[n,j] | N K" ᵉe̲q< "(sum k:=0..K-1 | z0[n,k] * Wd[k,j]) + Wd[K,j]"
g16:stage
} @)ᵉe̲q< expands to
({ @ →
g19:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g18:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g17:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g20:grid ← N c̲at K
g21:g̲rid ← { k → g20:grid r̲eshape (o̲ffsets k s̲elect g20:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g20:grid) r̲eshape 0 }
g20:grid r̲eshape ((
((g17:r̲ank (it, "it", 2, "it[n,k]")) g19:a̲t ((
((it, "it", 2, "k") g18:c̲heck (0 + g21:g̲rid 2)) + (2 s̲elect s̲hape it) ×
((it, "it", 1, "n") g18:c̲heck (0 + g21:g̲rid 1))))) + ({ @ →
g22:grid ← N c̲at K c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 1..T" } (1 + T − 1))
g23:g̲rid ← { k → g22:grid r̲eshape (o̲ffsets k s̲elect g22:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g22:grid) r̲eshape 0 }
'+ r̲/₃ g22:grid r̲eshape (
((g17:r̲ank (dz, "dz", 3, "dz[t-1,n,k]")) g19:a̲t ((
((dz, "dz", 3, "k") g18:c̲heck (0 + g23:g̲rid 2)) + (3 s̲elect s̲hape dz) × (
((dz, "dz", 2, "n") g18:c̲heck (0 + g23:g̲rid 1)) + (2 s̲elect s̲hape dz) ×
((dz, "dz", 1, "t-1") g18:c̲heck ((1 + g23:g̲rid 3) − 1)))))))
} @)))
} @)ᵉe̲q< expands to
(({ @ →
g26:a̲t ← { A i → (1 + i) s̲elect r̲avel A }
g25:c̲heck ← { (A, name, k, from) e →
n ← k s̲elect s̲hape A
bad ← w̲here n̲ot r̲avel (0 ≤ e) ∧ e < n
0 = t̲ally bad ? e
"index" ⎕S̲IGNAL name c̲at " axis " c̲at (f̲ormat k) c̲at ": index " c̲at (f̲ormat (f̲irst bad) s̲elect r̲avel e) c̲at " is outside 0.." c̲at (f̲ormat n − 1) c̲at " (from " c̲at from c̲at ")"
}
g24:r̲ank ← { (A, name, q, ref) →
(t̲ally s̲hape A) = q ? A
"rank" ⎕S̲IGNAL ref c̲at " indexes " c̲at (f̲ormat q) c̲at " axes, but " c̲at name c̲at " has rank " c̲at f̲ormat t̲ally s̲hape A
}
g27:grid ← N c̲at K
g28:g̲rid ← { k → g27:grid r̲eshape (o̲ffsets k s̲elect g27:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g27:grid) r̲eshape 0 }
g27:grid r̲eshape ((({ @ →
g29:grid ← N c̲at K c̲at ({ v → (0 ≤ v) ? v◆ "bad-equation" ⎕S̲IGNAL "sum's range runs low..high: 0..K-1" } (1 + (K − 1) − 0))
g30:g̲rid ← { k → g29:grid r̲eshape (o̲ffsets k s̲elect g29:grid) 'l̲eft t̲able ('× r̲/ k d̲rop g29:grid) r̲eshape 0 }
'+ r̲/₃ g29:grid r̲eshape ((
((g24:r̲ank (z0, "z0", 2, "z0[n,k]")) g26:a̲t ((
((z0, "z0", 2, "k") g25:c̲heck (0 + g30:g̲rid 3)) + (2 s̲elect s̲hape z0) ×
((z0, "z0", 1, "n") g25:c̲heck (0 + g30:g̲rid 1))))) ×
((g24:r̲ank (Wd, "Wd", 2, "Wd[k,j]")) g26:a̲t ((
((Wd, "Wd", 2, "j") g25:c̲heck (0 + g30:g̲rid 2)) + (2 s̲elect s̲hape Wd) ×
((Wd, "Wd", 1, "k") g25:c̲heck (0 + g30:g̲rid 3)))))))
} @) +
((g24:r̲ank (Wd, "Wd", 2, "Wd[K,j]")) g26:a̲t ((
((Wd, "Wd", 2, "j") g25:c̲heck (0 + g28:g̲rid 2)) + (2 s̲elect s̲hape Wd) ×
((Wd, "Wd", 1, "K") g25:c̲heck K))))))
} @))drawn : Char
drawn ← @ ᶠˡᵒʷm̲ath< c#"
ᶠˡᵒʷm̲ath< expands to
((r#"\begin{aligned} "# c̲at r#"& "# c̲at r#"zT"# c̲at r#" \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("Z0[n,k] | N K" ᵉt̲ex< "zT[n,k] + (sum t:=1..T | dz[t-1,n,k])") c̲at r#" \qquad \triangleright \, z0 \\ "# c̲at r#"\longrightarrow \; & "# c̲at ("X0[n,j] | N K" ᵉt̲ex< "(sum k:=0..K-1 | z0[n,k] * Wd[k,j]) + Wd[K,j]") c̲at r#""# c̲at r#" \end{aligned}"#))
ᵉt̲ex< expands to
("Z0_{n,k} = zT_{n,k} + \\left(\\sum_{t=1}^{T} dz_{t-1,n,k}\\right)")ᵉt̲ex< expands to
("X0_{n,j} = \\left(\\sum_{k=0}^{K - 1} z0_{n,k} \\, Wd_{k,j}\\right) + Wd_{K,j}")As a pipeline
pipe : Float
The latents of noise, the T steps (p_ower applies the step T times to the latents and the level), the final latents labeled, the decoder.
pipe ← @ ᶠˡᵒʷp̲ipe< c"
Plain X_eTaL
ᵘp̲lain : (Float, Int) -> (Float, Int)
The step without the macro: the schedule's values by level, the noise's slab; then the sampler as the diffusion demo writes it.
ᵘp̲lain ← { (Z, t) → (((t s̲elect a) × Z) + ((t s̲elect b) × ᵘe̲psTheta (Z, t)) + (t s̲elect sigma) × t s̲elect noise, t − 1) }
The same
r0 : Char
r0 ← "the equations and the pipeline of equations" ᵏt̲est eq ᵏn̲ear eqpipe