Python API¶
The public API is small. Everything listed here is stable within a minor version; anything with a leading underscore is not.
Monitor¶
Monitor(
model, optimizer=None, *,
out="ormas_run", # run directory
repair=False, # False | True | "all"
check_every=None, patience=None, sigma=None, warmup=None, # shortcuts; override config
name=None, # label for the reports; default: the run directory's name
sinks=(), # callables sink(kind, payload)
verbose=True, # print one line per event
config=None, # MonitorConfig
scaler=None, # torch.amp.GradScaler, for mixed precision
layers=None, # fnmatch pattern(s): restrict which layers are watched
overwrite=False, # True: replace a record already in `out`
)
| Argument | Meaning |
|---|---|
model |
Any torch.nn.Module; torch.compile and DDP wrappers are unwrapped. The monitor registers forward hooks on its Conv/Linear/BatchNorm and ReLU modules and removes them on close(). |
optimizer |
The optimizer; Lightning and Accelerate wrappers are unwrapped. Needed for the certificate's optimizer conditions, for repair and for scaler. Without it, or with an object that is not a torch.optim.Optimizer (e.g. DeepSpeed's), alarms and loss checks still run and no certificate is issued. |
repair |
False: never change the model. True: repair certified-dead, constant and decayed channels. "all": also channels observed silent. Any other value raises ValueError; repair without an optimizer raises ValueError. |
check_every, patience, sigma, warmup |
Shortcuts for the MonitorConfig fields of the same name. When given, they override config. |
config |
A MonitorConfig with every threshold. Default MonitorConfig(). Anything else raises TypeError. |
scaler |
A torch.amp.GradScaler. Skipped AMP steps become skipped_step events rather than non-finite alarms, and norms are unscaled if unscale_ was not called. Needs the optimizer. See Mixed precision. |
layers |
A pattern string or list of fnmatch patterns on module names; only matching layers are watched. Matching nothing raises ValueError. See Large models. |
sinks |
See Sinks. |
overwrite |
False (default): if out already holds a record (diary.jsonl), raise FileExistsError instead of destroying it. True: replace it. |
Methods and attributes¶
| Name | Meaning |
|---|---|
mon.step(loss=None) |
Call once per optimizer step, after backward() (and after scaler.unscale_ if used), before optimizer.step(). loss may be a tensor, a float or None; an unreduced loss is recorded as its mean (one loss_reduced event). On unsampled steps it only increments a counter. Raises RuntimeError after close(). |
mon.note_loss(loss) |
Record the loss of the current step, for code that cannot pass it to step(). The next step() called without a loss uses it; a loss passed to step() wins. The callbacks use it. |
mon.close(report=True) |
Remove the hooks, write run_end, close the files and, with report=True, write the reports. Returns the path of the engineer report, or None when report=False, the monitor was already closed, or this is not rank 0. Safe to call twice. |
with Monitor(...) as mon: |
Calls close() on exit, including on an exception. A failure while closing never replaces your training exception: it is printed to stderr and your exception propagates. |
mon.step_count |
Optimizer steps seen so far. |
mon.checks |
Checks run so far. |
mon.counts |
{"alarms", "stuck_layers", "repairs", "channels_repaired", "loss_spikes", "nonfinite", "skipped_steps", "checks_without_gradients"} totals. |
mon.config |
The effective MonitorConfig. |
mon.out |
The run directory, as a Path. |
Checkpoint check¶
from ormas_monitor.checkpoint import run, CheckpointError
result = run("model.pt", against=None, trust=False, arch=None, config=None)
Returns a dict with file, tensors, parameters, batchnorm_layers, findings (a list of {severity, kind, title, detail, ...}, see the findings catalogue) and layers (a list of {layer, batchnorm, channels, certified, flat, decayed, dormant, collapsed_var}). With against, also against and diff. Further keys: format, files, wrapper, streamed, dtypes, model_type, wiring, notes, norm_layers, severity_counts. path may be a file or a Hugging Face folder; arch is "torchvision:<name>" or "hf"; config is a CheckConfig. Raises CheckpointError when the file is missing, unreadable, or holds pickled objects and trust=False, and when arch matches under half of the file's BatchNorm layers (a partial match adds an arch_partial_match finding). The --fail-on threshold is applied by the CLI, not by run.
Record¶
| Function | Returns |
|---|---|
verify_run(run_dir_or_diary, public_key=None) |
A VerifyResult: .ok (no check failed), .complete (every check passed), .checks (a list of Check(name, status, message)), .to_dict(). |
seal(run_dir) |
Writes record.json; returns its path. Raises RecordError if the run does not verify. |
sign(run_dir, private_key_path) |
Seals with the signer's fingerprint and writes record.sig; returns its path. Needs the signing extra. seal and sign raise RecordError rather than write through a symbolic link. |
keygen(private_path, public_path) |
Writes an Ed25519 key pair as PEM; returns KeyInfo(private_path, public_path, fingerprint). Refuses to overwrite. |
diff_runs(run_a, run_b) |
A dict: a, b ({path, name, model, architecture_sha256, parameters, steps, intact, complete, sealed, signed, metrics, failed_checks, counts, final_loss}), same_architecture, config_changes, environment_changes, events, layers, worse, better, mixed, more_silent, fewer_silent, verdict (the headline) and verdict_detail (the supporting sentences). |
ormas_monitor.diary remains importable for 0.1 code (Diary, verify, read).
Reports¶
from ormas_monitor.report import write_all, write_run_report, write_summary, write_check_report, write_diff_report
| Function | Writes |
|---|---|
write_all(run_dir, gpu_hour_cost=None, gpus=None, config=None, public_key=None) |
Every report for a run. Returns {"report": engineer HTML, "summary": executive HTML, "summary_md": executive Markdown}. A dollar ESTIMATE appears only when gpu_hour_cost is given. |
write_summary(run_dir, ...) |
The executive summary only. |
write_run_report(run_dir) |
The engineer report only. |
write_check_report(check_result, path) |
The HTML report for a checkpoint.run result. |
write_diff_report(diff_result, path) |
The HTML report for a diff_runs result. |
Integrations¶
| Import | Needs |
|---|---|
from ormas_monitor.integrations.lightning import OrmasMonitorCallback |
[lightning] |
from ormas_monitor.integrations.huggingface import OrmasMonitorCallback |
[huggingface] |
from ormas_monitor.integrations import TensorBoardSink, WandbSink, MLflowSink, WebhookSink, LoggingSink, ConsoleSink |
each sink's extra, only when constructed |
ormas_monitor.integrations.LightningCallback and HuggingFaceCallback are lazy aliases for the two callbacks. See the Lightning, Hugging Face and Sinks guides.
Low-level¶
ormas_monitor.structure holds the weights-only channel test (scan_channels) and the wiring probe (relu_followers(model, example_input)). ormas_monitor.schema holds the event schema (EVENT_FIELDS, validate_event). These are importable but may change between minor versions.