Field deployment runbook

Practical guidance for running BirdNet-Behavior unattended on a single piece of hardware, in the field, for months at a time, with no operator on site to push a button when something goes wrong.

This is the document you read once before deploying and once again the month before the next field season. It is intentionally opinionated — the defaults pick the safe option at every fork.

Contents

1. Hardware checklist
2. Power and thermals
3. Storage planning
4. Networking
5. System hardening
6. Time synchronisation
7. Watchdog and process supervision
8. Database and backup policy
9. Remote diagnostics and monitoring
10. Update strategy
11. Pre-flight checklist
12. Recovery runbook

1. Hardware checklist

Recommended baseline for unattended deployments:

The bare-metal installer’s --doctor pass will tell you if any of the sanity checks fail (CPU cores, disk space, audio device, etc.) before you ever start the daemon.

2. Power and thermals

• UPS or battery + charge controller. A clean shutdown beats a corrupted database. The Pi 5 PSU draws ~5 W at idle; a 10000 mAh battery rides through a 15-minute mains glitch easily.
• Thermal headroom. Stick a passive heatsink on the Pi 5 SoC; in sealed enclosures add a 40 mm 5 V fan with a temperature switch. The Prometheus /api/v2/metrics endpoint exports birdnet_cpu_count and you can scrape vcgencmd measure_temp in parallel.
• Solar deployments. Size for 3× average daily watt-hours to ride out cloudy weeks. Use a battery monitor (e.g. INA219 over I²C) and stop the service via systemctl stop birdnet-behavior when battery drops below 30 %.

3. Storage planning

How big birds.db actually gets. Measured on a synthetic three-year station — 3 285 000 rows (3 000 detections/day), 180 species, the shipped schema and all sixteen shipped indexes, ANALYZE run:

Two things follow. Budget from the detection rate, not from calendar time — a station hearing 8 000/day reaches the same size in a bit over a year. And note that most of the file, and most of the write amplification on your SD card, is index rather than data; the backup ring multiplies whatever that number is by up to fourteen.

Set the disk policy explicitly:

# /etc/birdnet/birdnet.conf
RECS_DIR=/data/recordings
MAX_FILES_SPECIES=100         # cap clips per species
DISK_PURGE_THRESHOLD=85       # autopurge starts at 85 %

The bundled disk manager supervises two directories, once a minute, with the retention each one needs:

• the recordings directory, holding your extracted clips beside birds.db. These are your data, so they are never removed by age. Only the disk-full backstop touches them: once usage exceeds DISK_PURGE_THRESHOLD it removes a tenth of them, taken from the most-recorded species first (lowest confidence, then oldest, within a species) and never taking a species below PURGE_SPECIES_FLOOR clips (5 unless you set it; 0 removes the floor), so the one clip of a rare bird outlives the thousandth of a common one. It skips any file the database has marked locked (the 🔒 Lock action on Recordings, or on a Today row). The locked set is re-read every cycle, so locking a clip takes effect immediately — no restart needed.
• the raw capture directory (--watch-dir, typically the RAM-backed /tmp/birdnet-stream), which the detector reads and never needs again. It is drained continuously by age and by a total-size ceiling (STREAM_RETENTION_SECS, STREAM_MAX_MB) so the tmpfs self-empties. Set RAW_AUDIO_KEEP_EVERY=N to copy one aged segment in N (in capture order) into <recordings>/raw before the drain removes it: that is the audio a season can be re-analysed from under a new model, and without it only the audio that already triggered a detection survives. Budget it — a 15 s segment at 48 kHz mono is about 1.4 MB, so 1 is roughly 8 GB a day per source and 10 about 0.8 GB — and know that the disk-full purge takes kept raw audio before it takes any clip.

When inference falls behind. The daemon measures its own queue — the raw segments waiting for analysis — and publishes it as birdnet_analysis_queue_depth and on /api/v2/health as analysis_queue_depth. While more than 40 are waiting (ten minutes of 15 s segments from one source), or while the board is at or above 80 °C or reports throttling, it analyses one segment in two, in capture order, and counts the rest in birdnet_segments_shed_total by reason. The queue then drains at twice the rate, the skipped audio is counted rather than lost, and the backlog station-health condition tells you it happened. Before this, the same situation resolved to the stream drain deleting the oldest unanalysed audio with nothing to show for it. If the condition keeps returning, the board cannot keep up with what it is asked: fewer sources, a longer segment, or a faster board.

Two further limits are enforced from the database on the daily maintenance tick, and both leave the detection rows intact — only the audio is reclaimed, so your counts, species lists and analytics are unaffected:

• MAX_FILES_SPECIES — keep the newest N clips per species.
• CLIP_RETENTION_DAYS — reclaim audio older than N days. Off by default (0 = keep forever); set it in Settings → General → System & Display (“Keep Clip Audio”) or the config file if you want a rolling window.

Locked clips are exempt from both. A reclaimed clip keeps its filename and gains the date its audio was removed, so the record of what was captured survives even though the file does not.

All of these can be set from the web UI — you never have to edit this file to change them. A command-line flag or BIRDNET_* variable wins over the UI, which wins over the config file.

Maintenance runs on wall-clock time, not uptime. The daily jobs (integrity check, session prune, species cap) and the weekly backup + reclaim record their completion in the database, so a station that reboots often still runs them — an overdue job fires shortly after the next boot rather than restarting its timer.

Use endurance-class SD cards. A standard A1 card will write out ~3000 P/E cycles per cell; WAL on a busy station can wear that out in a year. SanDisk High Endurance and Samsung PRO Endurance are tested choices. SSD-on-USB is dramatically better if your power budget allows it.

4. Networking

The daemon assumes the network may be down or expensive, and no network failure can slow or block the detection pipeline — every integration is dispatched off the detection path. Verified behaviours:

• BirdWeather uploads — store-and-forward. A post that fails after its in-flight retries is parked in the local database (outbound_queue) and replayed oldest-first when the network returns: batches of 25 with 200 ms spacing so a returning uplink isn’t slammed, per-entry backoff 1 min → 1 h, bounded to 5 000 entries / 48 attempts so a months-long outage can’t fill the disk. Nothing is lost to a flaky link; see §9 for the queue-depth gauge and the self-hosted-ingest override.
• Apprise notifications — best-effort, dispatched off the detection path; non-delivery is logged but never blocks detection. Not queued by design — a look-now alert replayed hours later is noise.
• MQTT publisher — fire-and-forget per detection, opening a fresh bounded-timeout connection; dispatched off the detection path so an offline broker never stalls detection. Not queued by design (live telemetry).
• Wikipedia image cache — populated lazily; works fine with no network at all.
• Auto-update check — daily, non-blocking; failure logged at debug. The check reads bounded response bodies (it cannot be made to exhaust memory by a hostile or misbehaving endpoint).
• Heartbeat URL — fire-and-forget; no retry, by design (the monitoring side is the one that should care if it didn’t arrive).

For cellular deployments, consider scheduling cron pppd up/down windows so the modem only powers up for the daily upload batch.

5. System hardening

The installer’s systemd unit (install.sh) ships hardened by default:

• Type=notify + WatchdogSec=120 — systemd restarts the daemon if it stops calling sd_notify(WATCHDOG=1) for 2 min.
• ProtectSystem=strict, ProtectHome=read-only, explicit ReadWritePaths — the daemon can only write where it needs to.
• PrivateTmp=yes, NoNewPrivileges=yes, LockPersonality=yes, MemoryDenyWriteExecute=yes, RestrictRealtime=yes, RestrictNamespaces=yes.
• SystemCallFilter=@system-service minus the @privileged @resources @mount @debug @cpu-emulation @obsolete @reboot @swap @raw-io @clock @module groups, with sched_setaffinity allowed back: ONNX Runtime pins its threads with it, and a denied call kills the service rather than failing.
• MemoryHigh=768M, MemoryMax=1G, TasksMax=512, LimitNOFILE=65536, LimitNPROC=256 — bounded resource ceilings; runaway processes can’t take down the host. The FP32 model is loaded into anonymous memory, which a limit cannot reclaim — measured at about 575 MB idle and 650 MB while analysing, with the bundled DuckDB analytics on — so the 1 GiB ceiling is headroom for analytics queries, not slack. The Docker compose files use the same limit. On a 512 MB board physical RAM plus zram binds first.
• OOMPolicy=stop — under memory pressure the unit stops cleanly instead of being killed mid-write.

If you customise the unit, run systemd-analyze security birdnet-behavior.service to check the score (the shipped unit is graded around 1.5–2.0, “OK” on the systemd scale).

6. Time synchronisation

Every detection row carries a wall-clock timestamp. If the clock jumps backwards, analytics break.

• Enable systemd-timesyncd (default on Raspberry Pi OS) or chrony (preferred for cellular deployments — handles long offline windows better).
• The unit is ordered after time-sync.target, so on a host running systemd-timesyncd or chrony the sync normally lands first. It is ordering only — nothing Wants= that target, so the service starts even if the clock never syncs. That is why the daemon fails open on clock-dependent gating (see below) rather than assuming a good clock.
• For deployments with no network at all, fit a battery-backed RTC module (DS3231 on I²C) and load rtc-ds1307 at boot.

Daylight saving, and the hour that happens twice

Detections are stored the way BirdNET-Pi stored them: a local Date and a local Time, with no offset recorded alongside. That is fine for 8 759 hours of the year and needs one paragraph of thought for the other one.

Autumn, when the clocks go back. One local hour repeats. A station on Europe/Berlin records 02:00–02:59 twice on the last Sunday in October; one on America/New_York records 01:00–01:59 twice on the first Sunday in November. In the database those two hours are indistinguishable — they carry the same local Date and Time — so:

• Anything that reads the history as a timeline sees that hour’s detections interleaved rather than in order. A dawn-chorus curve is unaffected (the transition is at night); a session or gap analysis spanning that hour is not, and will report a gap or a burst that did not happen.
• Two detections of the same species, at the same local second, at the same rounded confidence, in both passes, collide on the detections table’s identity. The second is refused and lost. It is counted — see birdnet_detection_write_failures_total under Remote diagnostics — so the station can tell you it happened rather than leaving it to the journal. The clip is kept either way, under a -2 suffix.

Spring, when the clocks go forward. One local hour does not exist. Nothing is lost; the history simply has an hour-shaped hole in it, on one night, and any “quietest hour” or “longest gap” statistic computed across that night will find it.

If any of that matters to your analysis — a phenology study, a year-over-year comparison across the transition, published data — run the station on UTC (sudo timedatectl set-timezone UTC) and accept that every hour-of-day chart is then in UTC rather than in your local morning. That is the only configuration in which no local hour ever repeats or vanishes. --doctor reports the station’s offset and how the recording window is interpreted, so you can check which convention a station is on without guessing.

7. Watchdog and process supervision

This is the workhorse of unattended operation:

1. The daemon calls sd_notify(READY=1) once the HTTP server is listening.
2. A background tokio task calls sd_notify(WATCHDOG=1) every WATCHDOG_USEC / 2 (usually 60 s).
3. If the daemon hangs (livelock, GPU deadlock, ML thread stuck), the pings stop. systemd kills it after WatchdogSec and restarts via Restart=always.
4. StartLimitIntervalSec=0 — the rate limit is off and the unit never gives up. A tight restart loop is prevented by backing off instead (RestartSec=10, RestartSteps=10, RestartMaxDelaySec=300), so a permanently-broken install retries quietly every five minutes for ever and recovers by itself the moment its cause is fixed. An unattended box never parks itself in failed waiting for a visit.
5. ExecStartPre runs birdnet-behavior --doctor-gate (or plain --doctor for a binary too old to have the gate, such as a rolled-back .prev): the full doctor report, in the journal on every start. It exits 2, blocking startup, only for a failure the station cannot run past (an unreadable configuration file, an invalid listen address, an unwritable database directory, an unusable HTTPS setup), so the journal shows what is broken instead of just “service kept restarting”. Any other failure is reported and the service starts; run birdnet-behavior --doctor for the plain exit codes.

Smoke-test the watchdog after installation:

# Pause the daemon to simulate a hang. systemd should kill it after 2 min.
sudo kill -STOP "$(pidof birdnet-behavior)"
journalctl -u birdnet-behavior -f
# Expect: "Watchdog timeout" then "Restarting" within 120 s.

Per-source gain and quiet window (manual hardware checks)

Each audio source (Settings → Capture, /station/capture; the old /admin/audio redirects there) carries an optional quiet window and, in the audio_sources table, a software gain (gain_db). Like every other per-source setting (device, sample rate, RTSP transport), these are read when the capture subsystem starts, so restart the service after editing them for the change to take effect:

sudo systemctl restart birdnet-behavior

Gain (gain_db). At unity gain (0 dB) a local microphone records through arecord (the lightest path). A non-zero gain routes that source through ffmpeg instead so the gain can actually be applied. There is currently no control for gain_db in the web UI — the per-source edit form exposes the label, device, quiet window, pipeline toggles and equaliser only, and the value is merely shown in the source’s detail line — so write the column directly to exercise this path, and verify on real hardware:

# Set a source's gain to e.g. +12 dB (no UI control yet — write the column), restart, then:
sqlite3 ~/BirdNet-Behavior/birds.db "UPDATE audio_sources SET gain_db = 12 WHERE label = '<label>';"
ps -ww -C ffmpeg -o args= | grep -- '-af volume'
# USB mic with gain → ffmpeg: "-f alsa -i <device> … -af volume=12.00dB"
# Set the gain back to 0, restart, and confirm the mic is back on arecord:
pgrep -a arecord            # present again at unity gain
# RTSP / PipeWire sources are always ffmpeg; the filter appears only with gain.

A captured clip from the gained source should be audibly louder (or quieter for a negative dB cut) than at unity gain.

Quiet window (schedule_quiet). Set a window that currently includes the time of day (each end is a clock time such as 22:00 or a solar anchor such as sunset+30 / sunrise-15), in the station’s local time — a quiet window means the operator’s night, so it is evaluated against the local clock, as is a fixed:HH:MM-HH:MM recording window. (A solar schedule is not: sunrise and sunset are absolute instants, so that gate is evaluated in UTC and needs no timezone thought at all.) Within ~2 reconcile ticks (≈ a few seconds) the source’s capture subprocess stops:

journalctl -u birdnet-behavior -f
# Expect: "recording paused (outside schedule or in quiet window)" for the source,
# and the birdnet_audio_source_up{source=<id>} gauge drops to 0 on /api/v2/metrics.
# Move the window so "now" is outside it, restart, and confirm the subprocess
# is respawned and the gauge returns to 1.

While the system clock looks unsynced (no RTC yet at boot, NTP not ready) the quiet window is not enforced — capture fails open exactly as the schedule does, so a bogus boot-time date can never silence a source.

Multi-source resilience (USB + several RTSP at once)

You can run one or more local microphones (USB/ALSA or PipeWire) and any number of RTSP streams simultaneously — set them up in Settings → Capture, or seed them from the config file with ALSA_CARDS (;-separated) and RTSP_URLS (,-separated). Each source is captured by its own subprocess and recordings carry a per-source tag (local/MIC_n and RTSP_n, or the source row’s id), so detections, recordings, and the birdnet_audio_source_up{source=…} gauge stay distinct per source.

Every source is supervised independently — the central property for an unattended field station with flaky cameras:

• One source failing never disturbs the others. A dead subprocess (camera rebooted, USB mic unplugged, network blip) is restarted with capped exponential backoff (2 s → 4 s → … → 60 s, then every 60 s forever — a source down for an hour is still recording when it comes back on hour two). The other sources keep recording and the detection pipeline keeps running throughout.
• Silent stalls are caught, not just crashes. An RTSP camera (or a USB mic wedged after a re-enumeration) whose process stays alive but stops delivering audio is detected by watching each source’s newest segment: no fresh segment for several segment-durations (floor 2 min) and the source is restarted, exactly like a crash. Stall detection fails open while the clock is unsynced (segment mtimes aren’t trustworthy before NTP).
• A network outage never blocks detection. BirdWeather, Apprise, MQTT, email, and heartbeat are all dispatched off the detection path, so a dead broker or an offline uplink for days slows none of them down — detections keep landing in the local database and you reconcile from there.

Verify per-source isolation on real hardware before sealing the unit:

journalctl -u birdnet-behavior -f
# Unplug ONE RTSP camera (or one USB mic). Expect, for that source only:
#   "audio source DOWN … still trying to restart"  and the
#   birdnet_audio_source_up{source=<id>} gauge for THAT id drops to 0,
# while every other source's gauge stays 1 and detections keep flowing.
# Plug it back in: "audio source up" and the gauge returns to 1 on its own.

8. Database and backup policy

The daemon runs a scheduled maintenance task in the background (no operator action needed):

• Daily PRAGMA integrity_check. Failure is logged at ERROR and appears in the /api/v2/health endpoint.
• Weekly WAL checkpoint + PRAGMA incremental_vacuum, a MiB at a time, to return the pages that deletes freed. Not VACUUM: that rewrote the whole file through the temp directory under one lock — three times the file size written, staged in the unit’s memory-charged /tmp, and a detection lost to database is locked while it ran. A database created before this mode existed is converted once, at the next start, with the copy staged beside the file; the log says so.
• Weekly backup snapshot taken just before the reclaim, written to ${DATA_DIR}/backups/. The 14 most recent are kept; older ones are pruned automatically.
• Manual snapshot any time: birdnet-behavior --backup-db.

Every one of those lives on the station’s own storage. For a box that is expected to run a season unattended, set OFFSITE_BACKUP so each weekly snapshot is encrypted and uploaded to an object store or an SSH host as well — it rides the same weekly job, keeps the newest OFFSITE_KEEP (default 8), and never touches the local rotation. Restore with birdnet-behavior --decrypt-backup <file> --out <path>. See Backups & Recovery.

Restore from the most recent good backup (only do this after the daemon is stopped):

sudo systemctl stop birdnet-behavior
ls -lt ~/BirdNet-Behavior/backups/ | head -3

# Delete the write-ahead log FIRST. This step is not optional — see below.
rm -f ~/BirdNet-Behavior/birds.db-wal ~/BirdNet-Behavior/birds.db-shm

cp ~/BirdNet-Behavior/backups/birds.db.backup.<timestamp> ~/BirdNet-Behavior/birds.db

# Confirm you got what you asked for before starting the daemon again.
sqlite3 ~/BirdNet-Behavior/birds.db 'PRAGMA integrity_check; SELECT COUNT(*) FROM detections;'
sudo systemctl start birdnet-behavior

Why the rm -f comes first. The database runs in WAL mode, so birds.db-wal holds committed transactions that are not yet in birds.db. Copying a backup over the main file while that log is still beside it does not replace the database: on the next open SQLite replays the log and you are back on the data you were trying to discard.

This was reproduced rather than reasoned about. A backup taken after a VACUUM (1 000 rows) was copied over a database that had since grown to 9 000 rows plus a table the backup did not contain, with the log left uncheckpointed. Reopening reported integrity_check: ok and 9 000 rows — the restore silently did nothing, and said so in green. A second run that happened to leave birds.db-shm in place restored correctly, so it fails intermittently, which is worse than failing every time. Deleting both sidecars removes the ambiguity.

Restoring from the web UI (Settings → Data → Backups & recovery) removes them for you and runs the integrity check itself; this is the manual equivalent.

9. Remote diagnostics and monitoring

Once the unit is sealed and shipped, the loop is:

1. Heartbeat URL (HEARTBEAT_URL=) — set to a free https://healthchecks.io check. The daemon pings it every five minutes, whether or not it has heard a bird, and pings once immediately at startup so a station clears its alarm within seconds of coming back from a power cut. You get an email if the pings stop for longer than your grace period (recommend: 15 min — three missed pings, so one dropped packet on a 4G bearer does not page you).

   This signal answers exactly one question: is the box still there? It deliberately says nothing about whether the station is hearing anything — a December night at 55° N is sixteen hours long, and a heartbeat that only fired on a detection could not tell a quiet night from a dead Pi. “Has it stopped detecting?” is birdnet_detection_silence_seconds and the detection deadman (DEADMAN_HOURS); “is it degrading?” is the station-health alerts. Three signals, three meanings; wire up all three.

2. /api/v2/health — pulled by your monitoring (Uptime Kuma, Healthchecks remote endpoints, custom cron) over the LAN or via a port-forward.

3. /api/v2/metrics — Prometheus text format. Scrape with Prometheus, Grafana Agent, or VictoriaMetrics. Key series: birdnet_uptime_seconds, birdnet_detections_stored, birdnet_process_resident_memory_bytes, birdnet_species_distinct, and the two field-health gauges below.

   Also worth watching for an unattended station: birdnet_noise_floor_dbfs and birdnet_noise_floor_drift_db, per capture source. A microphone that fails outright is caught by birdnet_audio_source_up and by the deadman; one that merely goes deaf — water in the capsule, a web across the port, a connector loosened by a year of thermal cycling — keeps both of those green and shows up only as fewer detections, which is what autumn looks like too. The background does not stop when the birds do, so a sustained drop in one source’s noise floor against its own history points at the equipment. The same figures are on the Settings Health tab. No threshold ships with them on purpose — set one from what your own site does over a few weeks.

   Worth an alert of its own: birdnet_detection_write_failures_total counts detections the model produced and the database refused. It should stay at zero. A non-zero value means the station heard something and could not keep it — a full or read-only disk, a locked database, or the repeated daylight-saving hour described in Time synchronisation. Alert on any increase; the journal line beside it names the species and timestamp.

4. Detection deadman — the end-to-end “is it actually detecting?” check that no per-component gauge can answer. The station measures how long ago the last detection landed and exports it as birdnet_detection_silence_seconds (also detection_silence_secs on /api/v2/health, and the “last detection” figure on Settings → Health, /station — the old /system redirects there). After DEADMAN_HOURS of silence (default 24; 0 disables; also --deadman-hours / BIRDNET_DEADMAN_HOURS) it logs a loud warning and — when Apprise is configured — pushes one alert per quiet episode, with a recovery notice when detections resume. Stations in sparse habitats should raise the threshold; a Grafana alert on the gauge gives finer control.

5. Store-and-forward uploads — BirdWeather posts that fail while the uplink is down are parked in the local database and replayed automatically when connectivity returns (oldest first — the upstream record’s sequence matches what happened in the field — in capped batches, exponential backoff up to 1 h, bounded queue). Watch birdnet_outbound_queue_depth{kind="birdweather"} — a depth that only grows means the uplink (or token) has been broken for a while. Settings → Health (/station) shows a queued-uploads figure whenever the queue is non-empty. MQTT and Apprise/email are deliberately NOT queued: they are live telemetry and look-now alerts, and replaying them hours later is worse than dropping them — the local database is always the ground truth.

   Self-hosted ingest (sensitive species). Research programmes that must keep observation locations under their own governance — rare or endangered species where a public community map is a poaching risk — can redirect the same upload pipeline (including the offline queue and ordered replay) at their own endpoint that implements the BirdWeather station API shape:

   # /etc/birdnet/birdnet.conf
   BIRDWEATHER_URL=https://ingest.example.org/api/v1
   

   (Env equivalent: BIRDNET_BIRDWEATHER_URL; only the host changes — the /stations/<token>/detections path shape is preserved.) The active endpoint is logged at startup so a misdirected station is visible in the first journal lines.

6. birdnet-behavior --doctor-json — for monitoring scripts that speak JSON (Home Assistant command sensor, Nagios, Zabbix). Same exit codes as the human-readable mode. The same report is served over HTTP behind the admin login — /admin/doctor (rendered), /admin/doctor.json (the --doctor-json document) and /admin/support-bundle (the redacted archive --support-bundle writes; it shows the station’s coordinates to one decimal place, about 11 km) — so a station can be diagnosed from a browser without SSH.

7. SSH tunnel via Tailscale / ZeroTier / Cloudflare Tunnel — gives you the web UI from anywhere without exposing a port to the open internet. Recommended over plain port-forward.

Station-health alerts

The detection deadman answers “is the station detecting at all?”. Station health answers the questions that leave it quiet — the faults a station keeps detecting straight through:

Each alerts once per episode, with a recovery notice when it clears, through the same Apprise notifier as the deadman. Every condition must persist for three consecutive five-minute polls before it fires, so a mic that re-enumerates or a disk that spikes during clip extraction stays silent.

Once per delivered alert. The loud journal line is written when the condition is first observed; the push is then retried at every five-minute poll until it actually reaches a destination, so a fault raised while the uplink was down is still announced when the uplink comes back. A recovery that happens before the onset alert got out replaces it, so you are never handed a fault that has already cleared. Alerts about the station are also exempt from NOTIFY_RATE_PER_MINUTE, which is sized for detections — a deadman crossing its threshold during a dawn chorus used to lose that race silently. Whatever still cannot be delivered is counted in birdnet_notifications_dropped_total.

The clock check is the one that catches a silent loss rather than a visible one. A station whose NTP has been unreachable for months keeps recording, keeps detecting, and keeps every gauge green — while filing an entire season under the wrong hours. Nothing else notices, because a plausible-looking date passes every other check the station makes.

It is deliberately silent where it cannot know: in Docker there is no systemd to ask, and the container’s clock is the host’s, so the condition never fires and birdnet_clock_synced is absent rather than 0.

A single-source station that goes fully down is deliberately not reported here — the deadman covers that, with better wording, and two notifications for one fault is how a channel gets muted. This check exists for what the deadman structurally cannot see: some sources up, some down.

On by default. Disable with --station-health-alerts false, or STATION_HEALTH_ALERTS=false in /etc/birdnet/birdnet.conf.

# Confirm it started.
journalctl -u birdnet-behavior | grep 'station-health notifier started'

10. Update strategy

Field-deployment philosophy: don’t auto-update.

• The daemon checks for updates daily (logged at INFO when one is available) but never applies them automatically. The admin panel’s Check for Updates button tells you when one exists; apply it with sudo bash install.sh update.
• Test a new release on a bench unit first. Run it for at least 72 h before pushing it to field units.
• When you do push, do it during the species’ low-activity window (e.g. local 14:00 for songbirds) so a downtime blip costs the fewest detections.
• install.sh keeps the outgoing binary at /usr/local/bin/birdnet-behavior.prev, so a one-line mv rollback is available if the new build misbehaves. The update prints the exact command. The unit it writes asks for --doctor-gate only from a binary that has it, so the rolled-back binary starts under it. A unit written by 0.17.0 did not do that: after rolling back from 0.17.0, run sudo bash install.sh repair with the current installer to rewrite it. When the update added the geomodel to a config that had none, it prints a second rollback command that also takes those lines back out; use that one, because a release from before the geomodel cannot use it.
• Rolling back leaves the database at the newer schema. The previous binary logs a warning about it at every start and keeps recording; every release is tested for this (below). Its analytics pages may stop updating, because the analytics copy carries columns it does not know; the detections are unaffected.
• Every release is gated on this path before it is published: the newest previous releases are installed under systemd, given detections, updated in place, and rolled back with the printed command (installer/test/upgrade-e2e.sh).

11. Pre-flight checklist

Run through this the day you take the unit to the field:

# 1. Hardware and audio.
birdnet-behavior --doctor               # ALL green (or warnings only).

# 2. Time sync working.
timedatectl status | grep "System clock synchronized: yes"

# 3. Disk has > 5 GiB free.
df -h /data

# 4. Backup directory exists and is writable.
ls -la ~/BirdNet-Behavior/backups/

# 5. Watchdog smoke test (see § 7).
sudo kill -STOP "$(pidof birdnet-behavior)" && sleep 130 && \
  systemctl is-active birdnet-behavior

# 6. Reboot test — comes back clean?
sudo reboot
# After reboot, wait 2 min, then:
systemctl is-active birdnet-behavior   # expect "active"
curl -sf http://localhost:8502/api/v2/health | jq .status

# 7. Network failure tolerance.
sudo ip link set wlan0 down
sleep 60
journalctl -u birdnet-behavior --since "1 minute ago" | grep -i error
sudo ip link set wlan0 up

# 8. Heartbeat reaches your dead-man switch.
# Configure HEARTBEAT_URL, restart, then confirm in the dashboard at
# https://healthchecks.io/.

If any of those eight steps fails, fix it on the bench, not in the field.

12. Recovery runbook

Symptom-driven, oldest-known-cause first:

See also: Troubleshooting for general diagnostics not specific to field deployments.

See also

• Troubleshooting — symptom-organised problem solving (the repository’s TROUBLESHOOTING.md is only a pointer to it)
• SECURITY.md — disclosure policy
• docs/architecture/12-risks.md — risk matrix
• docs/architecture/14-diagnostics.md — design of the --doctor system