Sending events to TensorBoard, W&B, MLflow or a webhook¶
By default nothing leaves your machine. A sink forwards the monitor's events and metrics to another tool. A sink that sends data over the network (W&B, MLflow with a remote tracking server, a webhook) is the only way data leaves the machine, and only if you add one.
Built-in sinks¶
from ormas_monitor import Monitor
from ormas_monitor.integrations import TensorBoardSink, LoggingSink # or ormas_monitor.integrations.sinks
mon = Monitor(model, optimizer, sinks=[TensorBoardSink("runs/exp1"), LoggingSink()])
| Sink | Install | What it writes | Data off the machine? |
|---|---|---|---|
TensorBoardSink(log_dir=None, *, writer=None, layers=None, prefix="ormas") |
[tensorboard] |
Scalars ormas/loss, ormas/grad_norm/<layer>, ormas/stuck_channels/<layer>, ormas/stuck_channels_total; each event as text under ormas/events and a marker scalar ormas/events/<event>. |
No. Local event files. |
WandbSink(run=None, *, layers=None, prefix="ormas", alert_severity=None) |
[wandb] |
The same scalars, with ormas/step as their x-axis so it never fights your own step counter. With alert_severity="proof", matching events also raise a W&B alert. |
Yes, unless W&B runs offline or disabled. |
MLflowSink(*, layers=None, prefix="ormas") |
[mlflow] |
The same scalars as MLflow metrics on the active run; events as marker metrics. | Yes with a remote tracking URI; no with a local mlruns/ folder. |
WebhookSink(url, *, min_severity="warning", timeout=2.0, headers=None, blocking=False) |
none | POSTs {"source": "ormas-monitor", "text": ..., "event": <diary line>} as JSON for events at or above min_severity. Never sends metrics. Posts on a background thread so a slow endpoint never stalls training. |
Yes, to the URL you give. |
LoggingSink(logger="ormas_monitor", *, min_severity="info", metrics_level=None) |
none | Events to a Python logging logger: critical โ ERROR, warning/action/proof โ WARNING, others โ INFO. |
No. |
ConsoleSink(*, min_severity="warning") |
none | Like LoggingSink, to stderr. Useful with Monitor(verbose=False). |
No. |
layers= takes fnmatch patterns (for example ["encoder.*"]) and limits which layers get per-layer scalars.
Severities, lowest to highest: info, warning, action (a repair), proof (a certificate), critical (non-finite values).
Write your own¶
A sink is any callable sink(kind: str, payload: dict) -> None.
kind |
payload |
|---|---|
"event" |
The full diary line: seq, time, event, prev, hash, and the event's fields. See Record format. |
"metrics" |
One row per check: {"step": int, "loss": float or None, "grad": {layer: float}, "stuck": {layer: [certified, constant, silent, decayed]}}. |
def print_alarms(kind, payload):
if kind == "event" and payload["event"] == "alarm":
print("ALARM", payload["layer"], payload["diagnosis"])
mon = Monitor(model, optimizer, sinks=[print_alarms])
A sink that raises is disabled for the rest of the run. The monitor writes a sink_error event to the record and training continues. A dashboard outage never stops a training run.
What a sink receives¶
Event payloads contain layer names, step numbers, gradient norms and channel indices. They do not contain weights, activations or any training data. The run_start event contains the run name, the model class name, parameter count, optimizer settings, Python and PyTorch versions and the platform string. Review that before you send it to a third-party service.