Training Dynamics of Transformer Attention Heads
Published:
A time-dependent study of W_QK statistics across training checkpoints in the Pythia model suite: how spectral structure, stable rank, and head diversity evolve during pretraining.
Writing
Technical explorations at the intersection of physics and artificial intelligence
Recent Posts
Singular Value Structure of Transformer Attention Heads
Published:
An empirical study of the singular value spectra of W_QK matrices across transformer architectures: spectral distributions, participation ratios, and what they reveal about learned attention geometry.
Inverse Scattering via Physics-Informed Neural Networks
Published:
PINN solving the KdV equation with boundary conditions specified via inverse scattering transform.
Weight Distributions in Transformer Attention Heads
Published:
An empirical study of the statistical distributions of W_Q, W_K, and W_QK weight matrices across transformer architectures.