check findings catalogue¶
Every finding ormas-monitor check can report, what it means, and how strong the evidence is. Each finding in the JSON output has severity, kind, title and detail, and usually layer, batchnorm and channels.
Severities¶
| Severity | Meaning | Default exit code |
|---|---|---|
critical |
The file is broken: the model cannot produce finite output. | 1 |
warning |
Capacity is dead or effectively dead, or something died since the baseline. | 0 (see --fail-on) |
info |
Worth knowing; often harmless. | 0 |
Finding kinds¶
| Kind | Severity | What it means | Evidence |
|---|---|---|---|
nonfinite |
critical | One or more tensors contain NaN or Inf. | Exact. |
missing_tensors |
critical | A sharded checkpoint's index names tensors that are not in its shards. | Exact. |
arch_partial_match |
warning | With --arch: the architecture matches the file only in part (some BatchNorm layers are in one but not the other). Under half matching on either side is an error instead (exit 2). |
Exact count of matching names. |
newly_dead |
warning | With --against: channels that are certified dead now and were not in the baseline. |
Exact, under the same conditions as dead_channels. |
dead_channels |
warning | Incoming filter (with zero or no bias) or BatchNorm scale exactly zero, BatchNorm shift ≤ 0: the BatchNorm outputs a constant ≤ 0, so a ReLU after it is closed for every input and passes no gradient (Theorem 2). | Exact for the weights. Without --arch, the title says "if a ReLU directly follows". Whether the channels stay dead in further training also depends on what a weights file does not hold: the optimizer and its momentum, BatchNorm in training mode, and the ReLU being the BatchNorm's only consumer (--arch tracing confirms that a ReLU receives the unmodified output, not that nothing else reads it). The live monitor checks all of these; see Dead channels. |
decayed_channels |
warning | Filter or BatchNorm scale below 10⁻⁵ × the layer's median (or a zero filter behind a nonzero bias), shift ≤ 0: they output about 0 for every input. What a dead channel looks like after weight decay has shrunk it. | From weights. Not exact, so not certified. |
zero_tensor |
warning | A weight tensor is entirely zero (biases, running means and norm scales excluded). | Exact. |
norm_scale_zero |
warning | LayerNorm/RMSNorm/GroupNorm scale entries at ~0: that feature leaves the norm as a constant for every input. Skipped for families that store 1 + weight (e.g. Gemma), recognised from config.json. |
From weights. Not proven permanent: gradient can still reach the scale. |
identical_weights |
warning | With --against: every tensor is bit-for-bit unchanged; no training step changed the model, or both paths point to the same weights. |
Exact. |
flat_channels |
info | Filter or scale exactly zero but shift > 0, or the BatchNorm output joins a skip connection before any ReLU: the channel outputs a constant and carries no information. | Exact. |
dormant_channels |
info | BatchNorm shift ≤ −3 × |scale|: if a ReLU follows, these fire on under ~0.13% of positions when normalised inputs are roughly normal. | Estimate. Labelled as one. |
collapsed_variance |
info | A BatchNorm's running variance is ~0 (below 10⁻¹²) for some channels: whatever reaches them is the same for every input. Usually a dead filter upstream. | From running statistics. |
zero_filters |
info | A Conv/Linear without a BatchNorm has output units whose weights are all zero; each outputs only its bias. | Exact. |
frozen_tensors |
info | With --against: some tensors are exactly unchanged. Expected for frozen layers (a frozen backbone, adapter-only training); if you meant to train them, they received no updates. |
Exact. |
embedding_rows_unchanged |
info | With --against: rows of an embedding that never moved, typically tokens absent from the fine-tuning data. Usually harmless. |
Exact. |
How the check finds the wiring¶
A state dict holds tensors, not the graph. The check pairs each BatchNorm with the Conv/Linear that feeds it by name (bn1 ↔ conv1, layer.1 ↔ layer.0) and, failing that, by file order and matching size. Without --arch it assumes a BatchNorm feeds a ReLU unless its name contains downsample, shortcut, skip or residual, and qualifies its findings accordingly. With --arch torchvision:<name>, or --arch hf for a Hugging Face image model (built from config.json with random weights, no download; text models fall back to the heuristic and say why), it builds the real network once (on PyTorch's meta device where it can: shapes only, no weights; otherwise on the CPU), runs one forward pass, and uses the exact set of BatchNorms whose output a ReLU module receives unmodified. It does not check that the ReLU is the only consumer of that output. If under half of the file's BatchNorm layers are in the model (or of the model's in the file), the check stops with an error naming the mismatch (exit 2); a smaller mismatch adds an arch_partial_match warning. The terminal summary and the wiring key of the result say which method was used.
The diff (--against)¶
With --against, the result also contains diff:
| Key | Meaning |
|---|---|
changed |
Every floating-point tensor with its relative change ‖new − old‖ / ‖old‖ and absolute change, largest first. |
unchanged |
Tensors that did not move at all. A layer you meant to train and that never moved is worth a look. |
added, removed, reshaped |
Tensors present in only one file, or with a different shape. |
newly_dead |
Per layer, how many channels are certified dead now and were not before. |
dtype_changed, embedding_rows_unchanged |
Tensors whose dtype changed, and per embedding the rows that never moved. |