MonitorConfig
Every threshold the live monitor uses lives in one frozen, validated dataclass. The defaults are the values ormas-monitor 0.1 used, so Monitor(model, optimizer) behaves as it did, and the published demo numbers were measured with them.
from ormas_monitor import Monitor, MonitorConfig
cfg = MonitorConfig(check_every=100, sigma=3.0)
mon = Monitor(model, optimizer, config=cfg)
mon = Monitor(model, optimizer, config=cfg, check_every=20) # keyword shortcuts override the config
The whole configuration is written to the record in run_start.config, so a reader can see which thresholds produced each alarm.
Fields
Sampling and baselines
| Field |
Default |
Meaning |
check_every |
50 |
Check every N-th optimizer step. Hooks do nothing on the others. |
warmup |
8 |
Checks a layer's baseline needs before it can raise an alarm. |
baseline_rate_floor |
0.1 |
Smallest update weight of the running mean and variance of log₁₀(gradient norm). 0.1 means the baseline remembers roughly the last 10 checks. |
Gradient alarms
| Field |
Default |
Meaning |
sigma |
2.5 |
An alarm needs the log₁₀ gradient norm this many standard deviations off its own baseline ... |
alarm_min_decades |
1.0 |
... and at least this many decades away (1.0 = a factor of 10). |
alarm_confirmations |
2 |
Consecutive checks showing the same condition before an alarm is raised. One alarm per episode; alarm_cleared ends it. |
Loss
| Field |
Default |
Meaning |
loss_spike_factor |
1.5 |
A loss spike also needs the loss above this multiple of its baseline mean. |
Channels and repair
| Field |
Default |
Meaning |
patience |
4 |
Checks a channel must stay stuck before it counts as silent, or is repaired when it is not proven dead. |
certified_repair_streak |
2 |
Checks a certified-dead channel must stay stuck before it is repaired. A proven-dead channel cannot recover, so waiting longer gains nothing. |
decayed_rel_tol |
1e-5 |
A filter or BatchNorm scale below this multiple of the layer median (and not exactly zero), with shift ≤ 0, is flagged as decayed. |
repair_seed |
None |
None: repaired filters come from torch's global RNG, as in 0.1. An int: they come from a generator seeded from (repair_seed, step, layer), so the repair is identical in every process. Under torch.distributed, None is treated as 0. |
Certificate
| Field |
Default |
Meaning |
certify_optimizers |
() |
Names of extra optimizer types (class name, or module.QualName) the certificate may trust. Built in: SGD, Adam, AdamW, NAdam, RAdam, Adamax. For an opted-in optimizer, every state tensor shaped like a parameter must be exactly zero for the channel, and only decoupled weight decay is accepted. Opting in asserts that, with zero gradient and zero state, its step leaves the parameter alone apart from decay. See Dead channels. |
Record size
| Field |
Default |
Meaning |
list_limit |
64 |
Channel lists in channels events are truncated to this length. |
channel_list_limit |
256 |
Channel lists in certificate and repair events. |
layer_list_limit |
32 |
Layer lists in nonfinite_gradient and skipped_step events. |
Validation
Every field is validated on construction and a bad value raises ValueError naming the field. For example, check_every, patience and alarm_confirmations must be integers ≥ 1, sigma must be > 0, and loss_spike_factor must be ≥ 1.
Methods
| Method |
Returns |
cfg.replace(**changes) |
A validated copy with changes applied. Unknown names raise TypeError. |
cfg.to_dict() |
A plain dict of every field, as stored in run_start.config. |
Tuning
- Too many alarms on noisy training: raise
sigma or warmup, or alarm_confirmations.
- Overhead too high: raise
check_every, or restrict the monitor with layers= (see Large models).
- Repairs too eager: raise
patience and certified_repair_streak.
None of these defaults has been tuned on a benchmark. Status: DESIGNED.