Field documentation · technical preview

Take the watch.

Everything needed to install, operate, administer, and eventually tune Nightwatch—without crawling behind the rack.

00 / Read this first

Documentation map

Nightwatch has three explanatory rooms. After Hours explains the product, Behind the Rack explains the machinery and cost, and At the Chalkboard exposes the reasoning order. This field manual covers installation, current deployment, operation, extension, and safety.

Available now

User manual →

The current board, adaptive case briefings, service health, research, decisions, exports, and Handled memory.

Available now

Administrator guide →

Deployment, identities, targets, secrets, models, notifications, retention, and safe writes.

Code-configured

Rule guide →

What deterministic hunts and statistical signals do today, plus the intended rule-authoring boundary.

Design target

Skills →

The contract for future bounded investigation methods—not controls that silently exist today.

Product contract

Voice & reasoning →

How Nightwatch explains a conclusion and how wire evidence becomes a governed, growing case.

Status labels are part of the contract. “Available now” means the current release implements it. “Code-configured” means changing it requires source changes. “Design target” describes a proposed capability and is not a promise of present behavior.
01 / Installation guide

From empty host to first round

Nightwatch is a Python 3.10+ process with no pip dependencies. It needs at least one compatible evidence connector and a tool-calling model endpoint. The current distribution can use its wire-data adapter; the core is not tied to that source.

1. Prepare the host

  • Use a dedicated, non-root service account.
  • Allow outbound HTTPS to source system and the configured model provider.
  • Install qrencode for locally generated authenticator QR codes; manual TOTP secret entry remains available without it.
  • Keep the application, database, configuration, secret files, and evidence directory on persistent storage.
  • Expose the UI through TLS and authentication; the default direct bind is loopback-only.

2. Choose an authorized installation track

Packaged release (recommended for operators): obtain the platform archive and checksum, verify the checksum, unpack it, and use the bundled service template. A connector may be packaged separately or included by the distributor. Releases never include credentials, customer data, endpoints, or a working site configuration.

sha256sum -c SHA256SUMS
tar -xzf nightwatch-VERSION-PLATFORM.tar.gz
cd nightwatch-VERSION-PLATFORM
cp nightwatch.example.json nightwatch.json
# Add secret-file references and environment-specific settings.
python3 nightwatch.py --doctor --config nightwatch.json
python3 nightwatch.py --config nightwatch.json

Source installation: clone the repository, copy the credential-free example configuration, install a compatible connector, and run the read-only preflight before initializing the service.

gh auth login
git clone <nightwatch-repository>
cd nightwatch
cp nightwatch.example.json nightwatch.json
AI-assisted installation boundary. The public release does not expose an installer URL that an AI agent can execute. A secret, authenticated installation endpoint may be operated for a particular hosted deployment, but it is deployment-specific, disabled by default, excluded from release bundles, and must never be published in documentation or source. Until that endpoint exists and is separately reviewed, give an AI the packaged archive plus this installation guide.

Open http://127.0.0.1:8321 after startup. Put a maintained TLS reverse proxy in front of Nightwatch before allowing remote access.

3. Pass the preflight

--doctor is read-only. It checks the executable and checksum, database, target authentication, record and metric coverage, lamp groups, telemetry freshness, model tool calling and pricing, bind/auth posture, secret-file modes, and disk capacity. Treat a failed required check as an installation failure.

4. Verify the first round

  1. Confirm the header shows the configured target and a successful checkpoint.
  2. Run Sweep now and Check now.
  3. Open Cases and inspect All Cases, Needs You, On Watch, Services, and Connected.
  4. Confirm model use and deterministic work in the technical telemetry.
  5. Trigger a test webhook if notifications are configured.
Upgrade rule: back up nightwatch.json, nightwatch.db, and the evidence directory; deploy the new release as a unit; run --doctor; then restart. Never replace the database with an empty release artifact.
02 / Deployment guide

Small by default, separable when needed

The supported small deployment is one dedicated application host with local persistent SQLite and evidence storage, fronted by a maintained TLS reverse proxy. It is inexpensive, inspectable, and appropriate for a handful of concurrent analysts.

PUBLIC EDGEDNS, TLS, rate limits, bot defense, and request-size controls.
REVERSE PROXYHTTPS termination, security headers, access logs, and loopback forwarding.
NIGHTWATCHWeb/API, scheduler, workers, connector broker, reasoning harness, and reports under a non-root account.
PERSISTENCESQLite, evidence receipts, exports, protected configuration, backups, and restore checks.

Production baseline

  • Run the committed release as a dedicated non-root service; application files are read-only to the runtime account.
  • Bind the application to loopback. Only the reverse proxy accepts remote traffic.
  • Keep database, evidence, exports, configuration, and secret references on persistent encrypted storage with restrictive modes.
  • Allow outbound traffic only to configured connectors, model routes, mail relay, notifications, and explicitly enabled public context sources.
  • Back up the database and evidence directory together. Test restore to an isolated path and record the result.
  • Monitor process health, scheduler progress, connector health/backoff, model routing, disk, database integrity, mail queue, and certificate expiry.

Growth path

Do not begin with a cluster merely because one may be needed later. First move the production runtime away from the build host. Then increase CPU, memory, and disk. When real concurrency or retention requires it, separate web/API from background workers and move durable state to a transactional database and object storage behind the same repository interfaces. Case identity, audit, connector receipts, and workflow semantics must not change during that migration.

Migration rule. Freeze writes or drain workers, take a consistent database and evidence backup, deploy the exact intended commit, migrate configuration and secrets outside source control, run doctor and integrity checks, start one scheduler, verify authentication and a read-only connector query, then move traffic. Never run two schedulers against one environment unless leader fencing is proven.
03 / User manual

Follow the case, not the product boundary

Current product model. Nightwatch manages cases. Cases may contain security campaigns, service episodes, and linked investigations. The main navigation is Overview / Cases / Investigations / Handled. Investigations holds durable hunts, questions, troubleshooting, and claim reviews that may never need to become cases.

Nightwatch is one operating desk built from shared evidence. It does not require separate SOC and NOC roles. One person may investigate an account, restore a service, decide whether the two are connected, and verify the result.

COLLECTRounds read new detections, records, metrics, identity, and health.
JUDGEDeterministic screening and bounded reasoning build findings.
CONNECTRelated findings become the smallest useful piece of work.
ACTA person investigates, restores, contains, dismisses, or keeps watch.

The cohesive work model

Shape of workThe question it answersTypical action
CaseWhat condition deserves ownership, what supports it, and what happens next?Investigate, restore, contain, confirm, dismiss, or keep watch.
InvestigationWhat question are we researching, and what has the evidence established?Continue, pause, archive, link to a case, or propose a case.
Security campaignDo several security findings form one connected threat story?Preserve stages and provenance inside the case.
Service episodeWhat changed in a measured service during this bounded period?Diagnose, restore, verify, keep measuring, or support a case.

These are aspects of work, not organizational lanes, permissions, or personas. A familiar account doing something new can matter to a case; a machine where several symptoms converge can anchor an incident; a failing dependency can anchor a service. Nightwatch should name the thread and explain why it matters rather than present an inventory of accounts, hosts, addresses, and framework codes.

Overview

Overview is the morning briefing: what needs attention, what Nightwatch is doing, what it handled, and whether apparently separate symptoms may be connected. Ask Nightwatch can answer a quick wire-data question, start a durable investigation, accept a pasted brief, or independently test claims. A quick question does not become a case unless the evidence supports managed work.

A case card on Overview is deliberately a glance: urgency, state, one-sentence thesis, the thread that connects it when useful, compact scope, and the next action. A changed or connected marker may tell you why to reopen it. Evidence, framework identifiers, investigator history, and the full card spread belong in the Case Room.

Investigations Available now

Investigations is the file cabinet for work that is not necessarily a case: a threat hunt, service diagnosis, identity review, architectural question, pasted scenario, or claim challenge. Search active or archived work, see who started it and whether Nightwatch is working, waiting, paused, incomplete, or finished, then resume it without losing the transcript or evidence cutoff. Rename and archive keep work manageable; purge is a separate explicit destructive action.

An investigation can remain research, link to one or more cases, or propose a new case when its findings support a condition that deserves ownership or action. Linking does not copy the transcript or manufacture telemetry. Case-bound research and standalone research use the same durable workspace.

The durable clock

Every scheduled round, evidence-triggered follow-up, and human Run now or Update case now request is recorded before execution with a stable identity, target, reason, due time, prerequisite versions, priority, attempts, and fenced worker lease. Recurring work is anchored to UTC slots, so a slow or failed run does not shift the whole schedule. A human-triggered review wakes immediately but does not erase or postpone the normal cadence.

Collection wakes correlation; correlation and explicit case review wake Case Manager; the exact resulting case version wakes Daylight; a current Daylight result can then wake notification delivery. Failed work retains its error and deterministic retry time. Expired leases are recovered after restart, duplicate triggers are idempotent, and stale workers cannot publish. The activity display summarizes working, ready, retrying, blocked, and completed work without exposing internal tool chatter.

Cases Available now

The Cases desk holds durable cases and the service evidence that may support them. Its All Cases, Needs You, On Watch, Active, Waiting, Resolved, Services, and Connected lenses are different views of the same operating desk. Needs You means a person owns the next decision. On Watch means Nightwatch owns the next bounded check. Active, Waiting, and Resolved make human progress and verification visible. Services can remain measured conditions without becoming cases; Connected shows meaningful case/service overlap without claiming that one caused the other.

Cases grow; Nightwatch does not wait

A case appears as soon as the evidence supports a coherent proposition worth working. It can then gain findings, change confidence, acquire a different presentation, or receive another deterministic assessment. When exact evidence later proves that two visible cases are one story, Nightwatch collapses them into the older case, preserves which assessments joined, and explains why under Reasoning. It does not hide useful work while trying to build an ultimate campaign.

Nightwatch keeps two clocks. A one-off lead on Still on watch expires after a short complete clean window if it never develops. An established case stays active on a much longer case clock. Expiry and quiet resolution archive attention; they do not delete evidence. When new work shares exact evidence—or the same strong subject and activity family—an earlier deck can return as clearly dated historical context.

The Case Room Operational folio

Selecting a case opens its first-class working room rather than stretching the Overview card into a report. Every entry point—Overview, Cases, a connected service, or a direct link—refers to the same durable case. The left side is the case folio: title, phase, owner, priority, confidence, lifecycle, current judgment, and the next owned move. The prominent action follows the phase: accept, record progress, resume, confirm resolution, or reopen. Nightwatch drafts routine progress from the existing case so the analyst can accept or edit instead of completing a ticket form.

Details and Investigator remain independent right-hand rollouts. Either can open beside the folio; when both are open they split the right workspace vertically, with horizontal and vertical resizing. Details holds proof and provenance. Investigator is the case-aware research surface. Neither creates a second case or a second source of truth.

Case workflow

The operational path is On Watch → Needs You → Active → Waiting → Resolved/verification → Handled, with Reopened when new evidence changes a closed judgment. Evidence state and workflow state are deliberately separate. Every transition records actor, time, previous and next phase, summary, owner, next action, and review or resolution context in append-only history.

How Nightwatch lays out a case Available, expanding

Nightwatch presents an argument, not an evidence dump. The case composition adapts to the evidence: two sentences may be enough for a familiar recurrence; an ordered sequence may need a timeline; shared identity or device activity may need a relationship sketch; and a possible service connection may need aligned lanes. The card metaphor means “lay the useful facts on the table.” It is not a fixed deck, a game, or a requirement that every case have equal cards.

The investigation is navigable in layers: case → line of argument → contributing deck → source evidence. The case opens with Nightwatch’s plain-language conclusion, evidence spine, impact, uncertainty, and next move. “How this case came together” then exposes each narrower assessment as a deck and each observed finding beneath it as an evidence card. Nearby cases remain visibly outside the boundary until shared evidence earns a merge.

Every explanation follows the same disclosure order:

  1. Meaning: what Nightwatch thinks happened, why it matters here, and the next useful action.
  2. Reasoning: how the facts connect, what argues against the conclusion, what remains unknown, and what would change the call.
  3. Source: records, detections, measurements, timestamps, queries, and identifiers such as ATT&CK technique IDs.
Speak analyst to analyst. Lead with “The same unfamiliar account reached two file servers six minutes after the login.” Do not lead with a string of ATT&CK identifiers. The codes and links remain available in Source for verification.

The first view contains only what is needed to understand and act. On desktop, the Case Room coordinates an adaptive canvas, an inspector for the selected card or relationship, and one persistent Case investigator. Nightwatch recreates the investigator from the current case file on every turn, carries a bounded handoff from older conversation, retains recent turns verbatim, and keeps the complete transcript for audit and export. The case can therefore grow for days without depending on an opaque provider thread or sending its entire history on every question.

Likely follow-ups that Nightwatch can already answer stay within reach—for example, “Why is this account unusual here?”, “Have we seen this before?”, or “Did the slowdown begin first?” Anything else belongs in the Case investigator. Opening stored evidence should not spend model tokens.

Mobile is a focus stack Available, expanding

Nightwatch does not squeeze a spatial desktop canvas onto a phone. Mobile leads with case identity, thesis, and next action; then it presents one active visual or card and the suggested-next cards in a deliberate vertical order. Inspector and investigator controls open focused, accessible surfaces. Mobile Investigate means selecting, comparing, pinning, and asking—not dragging tiny cards around.

Notes, conversation, and decisions

RecordUse it forEffect
Analyst noteAttributed local context, interpretation, correction, or a person-supplied fact.May influence the next case composition, but Nightwatch must cite it as a human statement. It is not telemetry.
Investigator chatExploring a question with the case-owned investigator.Changes nothing by itself. Explicitly save a useful answer as a note when it should inform the case.
Formal decisionRecording disposition or workflow action with a reason.Changes case state and attention handling and retains author, time, and scope for audit.

Nightwatch keeps human context separate from detections, records, and measurements. It may say “Aaron noted that this was approved maintenance”; it may not turn that note into “telemetry confirms approval.” If later evidence conflicts with a note, the case should show both rather than quietly choosing one.

Planned version visibility. Durable cases can grow and recompose today, but a complete “what changed since you last looked” experience is not yet implemented. Planned work includes visible case versions, evidence-graph hashes, thesis/severity/anchor diffs, the reason for recomposition or merge, note-influence attribution, prior-composition inspection, and conflict-safe updates while an analyst is reading.
DecisionUse it whenEffect
Escalate / confirmEvidence supports a real incident or needs response.Records the analyst outcome and preserves the handoff.
Keep watchingThe case is plausible but the discriminating evidence is not available yet.Keeps the case On Watch with a bounded next check.
Dismiss / benignThe evidence supports an ordinary explanation.Moves attention away while retaining audit history.

Services in Cases

Each lamp corresponds to configured source activity groups and a metric pack. The selected service leads with a plain-English conclusion, why it matters, Nightwatch’s leading explanation, what changed, and what happens next. A service episode is a bounded measurement period; it becomes part of a case only when the condition needs ownership, continued observation, or action. Measurements, named hosts and clients, counter-signals, resolved checks, packet evidence, and exports remain supporting material. History answers whether a condition is new, recurring, or worsening.

Service reasoning runs only on a material episode change. The deterministic measurement remains authoritative; one bounded read-tool turn may settle a precise unanswered wire question before Nightwatch publishes. It must not hand an available evidence-source lookup back to the operator as homework. Missing endpoint, change, ownership, or human context stays explicitly missing. Services do not create a parallel investigator or duplicate the Case Room.

Connected work and causality Available now

When case evidence and service impact overlap, the Connected lens places the case and the measured service condition side by side and draws the exact shared subject and UTC window between them. A separate Nightwatch analysis explains how they connect, why it matters, and whether the relationship is direct wire confirmation, supported contribution, coincidence, or still unproven. Supporting evidence, counter-evidence, resolved checks, missing evidence, impact, and the deciding next step stay expandable. The case and service remain independently navigable; the view does not repeat one summary twice or silently turn overlap into cause. Full durable incident workflow remains in development.

Handled

Handled is durable attention control, not deletion. Repeated benign patterns are grouped with scope and history. A new host, account, or destination is judged again. Reopen a pattern when the business context changes; disable a suppression when it is too broad.

Research, evidence, and exports

  • Chat is scoped to the open work and uses read-only tools.
  • Case notes are analyst-selected; conversation exports do not automatically become case evidence.
  • Markdown and JSON are portable handoffs. HTML is print-friendly. CSV flattens evidence.
  • Packet capture requires a bounded window, purpose, BPF, size limit, and typed human approval. Scheduled work and the model cannot approve it.
03 / Administrator guide

Own the boundaries

Administrators control connectivity, identity, cost, persistence, write permissions, and the definition of “inside.” Those choices change what every analyst sees.

The administration workspace

The gear opens the authenticated control plane. People & access supports search, per-account changes, visible-row selection, and atomic bulk role or enablement updates. Disabling an account preserves authored decisions and audit history while revoking its active sessions. An administrator cannot remove their own access, and Nightwatch always retains at least one active administrator.

Configuration workflow

System configuration exposes eight validated categories: Environment & targets, Service lamps, Schedules & cadence, Investigation context, Notifications, Email & recovery, Authentication security, and Audience & capacity. Search locates a category; each editor saves the complete safe configuration while preserving fields outside its schema. It never resolves or returns secret values, writes a mode-600 backup, and replaces the configuration atomically. Restart Nightwatch after a saved configuration change.

Enrollment, recovery, MFA, sessions, and bot verification

Administrators can generate, replace, enable, and disable a hashed invitation code; the login page exposes account creation only while invitations are enabled. With SMTP configured, signup creates a pending account, sends a single-use 30-minute verification link, and activates a viewer only after verification. Password recovery uses an indistinguishable response for known and unknown addresses, a single-use 20-minute link, bounded request rates, and session revocation after reset.

Each user can review active sessions, revoke an individual sign-in, sign out everywhere, and enroll a standard TOTP authenticator. Nightwatch shows recovery codes once and stores only their hashes. Administrators can require MFA by role and configure session lifetime, secure-cookie behavior, login-attempt windows, lockout duration, and the accepted legal-policy version. The server-console password-reset command is the break-glass path: it re-enables the named local account, revokes its sessions, and clears MFA so the administrator can enroll again. External identity providers remain a future connector boundary.

Optional Cloudflare Turnstile can protect login, signup, and recovery. It is disabled by default and makes no Cloudflare request while disabled. When enabled, Nightwatch requires a public sitekey, secret-file or environment reference, approved hostnames, and selected forms; the backend verifies every token’s success, action, and hostname before authentication logic runs. Turnstile supplements rate limits, lockout, verified email, and MFA—it does not replace them.

Audience and shared capacity

Administrators can bound interactive requests per user and source network, manual actions per user, FIFO interactive concurrency, queue wait time, and deployment interactive cost per day and month. Quiet mode pauses analyst-triggered model work and forced rounds while deterministic scheduled measurement continues. Demo reset and viewer-export policy are disabled by default; when explicitly enabled, reset removes chats and saved notes while preserving cases, evidence, accounts, configuration, and audit history. These limits protect a shared lab and complement—not replace—upstream provider quotas and authorization roles.

Administrative checklist

  • Targets: verify names, hosts, credentials, internal networks/domains, and reconciliation scopes.
  • Identity: require sign-in, use TLS, set secure cookies, assign least privilege, and remove dormant users.
  • Models: qualify tool calling, set primary/fallback budgets, and understand provider data policy.
  • Writes: leave tuning disabled until governance and rollback ownership are established.
  • Storage: monitor database, evidence, backup, and free-disk growth.
  • Notifications: sign webhooks and route links to the public Nightwatch URL.

Enrichment and privacy boundary Available, expanding

Useful context comes from deterministic sources before another reasoning call. Nightwatch resolves exact typed public IPs through prefix, BGP origin, RIR registration, and RPKI, cached for seven days. An opt-in Spamhaus DROP source checks exact typed public IPs and ASNs against a 24-hour cached snapshot. A listing is supporting third-party context; no listing is not evidence of safety. Private, documentation, internal-marked, and prose-derived addresses never leave the system. ATT&CK, KEV, and EPSS context is also available. Observed DNS, configured resolvers, DHCP/IPAM, maintenance, CMDB, IdP, provider ranges, and additional governed sources remain next layers.

  • Prefer customer-local truth to public inference.
  • Forward-confirm reverse-DNS names and retain resolver, TTL, source, and observation time. A name is context, not proof of identity.
  • Never send private addresses, internal names, accounts, or case contents to a public lookup service.
  • Cache public facts, respect provider terms and limits, and show freshness and provenance.
  • “Hosted by this provider” is context—not a safety verdict. “Not on this list” is not a clean bill of health.

The EDR configuration is a provider-neutral stub. It performs no endpoint query, loads no endpoint credential, and enables no response write. Any endpoint adapter must satisfy the same evidence, health, privacy, and failure contract before Nightwatch calls it available.

Backup and recovery

Back up the configuration, SQLite database (using a SQLite-safe snapshot while running), evidence directory, and secret references. The database contains cursors, campaigns, feedback, outcomes, suppression memory, audit records, and session state. Test restore on an isolated bind before declaring recovery complete.

Change control

Record who changed a target scope, detector threshold, model route, retention window, or write capability; why; the expected effect; validation result; and rollback point. Run doctor after infrastructure or credential changes and after every release.

04 / Configuration reference

What the system reads today

The example configuration is the canonical portable template. The table below groups controls by operational consequence; defaults are template defaults, not universal recommendations.

Two configuration surfaces. The administration editor currently supports targets, lamps, intervals, investigation context, notifications, email, authentication security, and audience capacity. Model routes and budgets, retention windows, tuning, evidence storage, and the executable path remain file-configured. The editor preserves those file-only fields but does not expose them.
SectionControlsOperational meaning
llm.primary / fallbackprovider, base_url, secret reference, model, provider optionsSelects native Anthropic or OpenAI-compatible/OpenRouter routes. Use separate keys when budgets and failure domains must be independent.
llm.limitsdaily requests, request spacing, fallback daily/monthly/per-call USD, cooldown, failure fallbackCost and quota circuit breakers. They govern route eligibility, not detector collection.
llmreasoning_queue_limit, tool rounds, temperature, retained tool resultsBounds concurrency, agent depth, sampling, and context size. Raise only after measuring quality and cost.
targets[]name, host, auth, TLS, internal networks/domains, reconciliation scope, direct destinationsDefines evidence sources and identity boundaries. Wrong “inside” scope causes wrong attribution.
lamps[]id, label, connector-declared groups, metric packMaps business-facing service areas to measurable activity.
context_sourcesopt-in reputation provider; disabled EDR provider placeholderDeclares which external facts can actually be collected. The EDR stub never claims endpoint visibility or enables writes.
sweepstriage interval/window/caps, hunt interval/window/overlap/limit, watch baseline/forget/limitsControls collection cadence and volume. Caps create an explicit coverage boundary.
sweeps signalssignal window and per-record limits, Kerberos window and spray thresholdControls deterministic statistical detectors and their input ceilings.
sweeps campaignswatch_quiet_hours, case_quiet_hours, case_context_days, case_context_limit, correlation debounce, Daylight, online intelSeparates one-off lead expiry from established-case lifecycle and bounds how far—and how many—relevant historical decks may be brought forward. The strongest matches are shown; the complete archive remains intact.
sweeps healthboard interval/windows, episode cooldown/tool roundsControls service-health sampling and episode reasoning.
sweeps retentionrun, trace, and handled retention daysBalances forensic history and storage. Confirm policy before reducing.
notificationswebhook URL, signing secret reference, public Nightwatch URL, timeoutSends signed state transitions and correct investigation links.
emailSMTP relay, credential references, sender/reply-to, public URL, signup domainsEnables durable queued delivery for verified signup, password recovery, and account notices.
authsession lifetime, secure cookie, login window/lockout, required-MFA roles, policy version, optional TurnstileControls local authentication security, legal-policy enforcement, and deployment-specific bot verification.
accessper-user/network limits, manual-action rate, concurrency, queue wait, quiet/demo/export policyBounds shared interactive capacity and viewer capabilities.
tuningenabled, minimum occurrences, expiration daysGates connector-declared source-native tuning candidates. Apply and rollback remain typed human actions.
evidence_captureprivate directoryStores approved PCAPs; provision and retain it as sensitive evidence.
uibind, port, minutes-per-detectionControls exposure and workload projection. Loopback is the safe default.
identity recordsusers, roles, enabled state, TOTP, recovery codes, active sessionsDatabase-backed access state managed through People & access and each user’s security panel.
connector_pathabsolute or application-relative pathPins the evidence-source tool surface Nightwatch is allowed to invoke.
Secret reference rules

Prefer *_file or *_env fields. Secret files should be readable only by the service account. The browser editor accepts references but does not resolve or return secret values. Do not store literal credentials in source control, release bundles, support archives, or screenshots.

Multiple targets and reconciliation scope

Each target sweeps independently. Give systems in the same real environment the same reconciliation_scope so cross-system references are possible; isolate unrelated customers or environments with different values. Configure internal CIDRs and domain suffixes explicitly.

Safe interval changes

Shorter intervals increase API work and can outpace downstream reasoning. Change cadence and queue/record caps together, observe a complete peak period, and check coverage indicators. The administration editor enforces bounded values, including a minimum scheduler poll of 10 seconds.

06 / Evidence connectors

Many witnesses, one evidence contract

Nightwatch can begin with one source without treating that source as the product. Connectors publish bounded observations with provenance, freshness, scope, completeness, sensitivity, and native identifiers. The reasoning and case layers consume that common envelope.

Source familyStatus in this distributionBest authority
Wire / NDRAvailable connectorObserved communications, protocol operations, timing, direction, volume, service behavior, and retained payload evidence.
Public network contextAvailable / opt-inOwnership, route validity, provider ranges, and exact typed reputation context; never a local verdict.
Identity from network protocolsAvailable where visibleObserved protocol identities and authentication behavior, not full directory or IdP audit state.
Endpoint / EDRContract onlyProcesses, files, registry, memory, local users, and containment state.
Directory / IdPContract onlyAuthentication decisions, authorization, roles, groups, sessions, and identity risk.
Asset / CMDB / IPAMContract onlyDeclared owner, role, criticality, service membership, address, and lifecycle.
Cloud, vulnerability, change, and ticketingContract onlyControl-plane events, exposure, approved intent, maintenance, ownership, and incidents.
Human contextAttributed notesIntent, local knowledge, decisions, and approvals; never relabeled as observed telemetry.

Required connector behavior

MANIFESTVersion, family, schemas, capabilities, limits, and read/write declarations.
HEALTHAuthentication, reachability, retention, clock, schema, and backoff state.
READIncremental collect, bounded query, entity resolution, and completeness.
RECEIPTScope, time, results, duration, redaction, gaps, and failure without credentials.

A new connector is complete only when contract fixtures prove cursor behavior, pagination, timeouts, rate limits, partial results, staleness, schema drift, permission loss, and secret redaction. Product-specific language and credentials belong in the optional connector package, not in the core case experience.

Current boundary: scheduled wire collection uses the executable at connector_path. It advertises JSON tool schemas and receives target credentials through NIGHTWATCH_SOURCE_* environment variables. Optional Python adapters currently serve the evidence planner and context registry; installing one does not replace scheduled wire collection. Both paths are being converged on the same manifest, envelope, health, receipt, and failure contract.

Source authority is claim-specific. Endpoint is stronger for a process start, an identity provider for an authentication decision, a CMDB for declared ownership, and observed traffic for a completed protocol exchange. Conflicts remain visible; sources do not vote.
05 / Rule guide

Deterministic first, explainable always

Code-configured today

Nightwatch’s hunt rules and statistical signals run before model reasoning. Rules are currently shipped as reviewed source, not uploaded from the browser. Configuration exposes a small set of safe thresholds and resource ceilings.

Current rule contract

  • A stable ID and version.
  • A supported connector record type and query.
  • Required fields and a coverage condition.
  • Actor/entity extraction and an explicit count or threshold.
  • Evidence text that can be cited without inventing telemetry.
  • Tests using sanitized record contracts, including missing/truncated data.

When a new rule belongs

Add a deterministic rule when the evidence condition is precise, repeatable, cheap enough for every round, and independently testable. Use reasoning only to interpret fired evidence in context. Do not encode a vague “suspiciousness” prompt as a rule.

Proposed supported rule packs

Design target

A future rule-pack format should allow signed, versioned packs with metadata, record contracts, queries, thresholds, entity mappings, severity guidance, fixtures, and resource budgets. Packs should pass doctor validation and a dry run before activation. Arbitrary Python, shell, network calls, model prompts, and write tools should remain outside the pack format.

06 / Context & expectations

Know what belongs before judging what changed

Network origin + opt-in reputation available

Installed connectors supply observed network and service evidence. Nightwatch’s context layer adds the local identity, asset, history, ownership, and expectation facts needed to explain what that evidence means here. Deterministic code computes the facts and mismatches; a model explains the smallest defensible case from them.

Current boundary. Nightwatch provides ATT&CK, KEV, EPSS, cached RIR ownership, and a bounded public-IP path from prefix and BGP origin to registered operator and RPKI status. Optional Spamhaus DROP checks add time-bounded exact-indicator context. Nightwatch retains a typed source-status registry with refreshed cases, so a disabled or unavailable connector cannot masquerade as collected evidence. DNS/PTR confirmation, certificate/domain history, provider ranges, and authoritative asset and identity adapters remain planned. EDR is contract-only until a test system exists.

Do not collapse the evidence labels

LabelWhat it means
MismatchObserved and expected values can be compared and differ. That is a fact to explain, not proof of attack.
UnknownData is absent, stale, ambiguous, capped, or unsafe to compare. Unknown is neither match nor mismatch.
Shared infrastructureThe provider, ASN, address, certificate, resolver, CDN, or SaaS serves many tenants. It does not identify the actor or purpose.
ReputationA time-bounded third-party report or score. Supporting context, not local observation or a verdict.
ProofOnly the narrow claim an authoritative source or observed mechanism supports. A valid RPKI origin does not prove benign traffic.

Priority order

  1. Customer-local truth: telemetry, device identity, DNS, DHCP/IPAM, directory/IdP, CMDB/EDR/cloud inventory, maintenance, approved services, and dependency maps.
  2. Locally computed history: novelty, peers, direction, volume, periodicity, account/device relationships, changes, and prior cases.
  3. Cached public facts: registration and routing ownership, RPKI, official provider ranges, domain and certificate facts, ATT&CK, KEV, EPSS, and vendor advisories.
  4. Optional passive DNS or exact-indicator reputation, disabled by default and queried only with an observed public key.
  5. Model reasoning over the typed package—not a model doing CIDR math, date arithmetic, version comparison, or broad web searches.

Every context fact carries provenance, observation/fetch time, expiry, confidence, scope, and limitations. Failed public lookup means unknown. Learned history means “usual in the retained baseline,” never “approved.” Human notes remain attributed human context, not telemetry.

CONTEXT_REASONING_GUIDE.md contains the detailed source matrix and privacy rules. REASONING_ENGINE.md is the complete source-to-case reasoning contract.

07 / Skill guide

Teach a method, not a personality transplant

Placeholder · not implemented

A Nightwatch skill should define a bounded investigation method: what evidence it accepts, which read-only tools it may use, what it must publish, and how success is tested. Skills should not be unscoped prompt fragments.

Proposed skill manifest

id: org.example.identity-triage
version: 1.0.0
accepts: [finding.kerberos, finding.ldap]
tools: [records.search, metrics.query]   # read-only allowlist
publisher: publish_investigation
max_tool_rounds: 6
evidence_contract: complete-or-declare-gap
tests: [fixtures/spray.json, fixtures/ordinary-sso.json]

Design requirements

  • Signed/versioned origin and a compatibility range.
  • Explicit input/output schemas and tool allowlist.
  • Hard token, tool-round, time, and record budgets.
  • Prompt-injection handling and untrusted-data boundaries.
  • Positive, negative, incomplete-evidence, and cost regression tests.
  • Dry-run, staged rollout, audit trail, and one-step rollback.
Not available yet: there is no supported skill upload, registry, hot reload, or third-party execution sandbox in the current release. Existing investigation methods are reviewed application code.
08 / Product voice

The Watch Partner

Product-wide contract

Nightwatch sounds like an experienced analyst working beside the operator: calm, candid, specific, and ready to show the evidence. That voice governs interface copy, generated analysis, deterministic fallbacks, notifications, errors, exports, and documentation.

Say what matters in this order

  1. Lead with the answer.
  2. Explain the effect on people, systems, risk, or the decision.
  3. Show the few facts that make the answer defensible.
  4. Name the strongest ordinary explanation or the missing fact that could change the judgment.
  5. State what Nightwatch already checked and the one remaining move, if there is one.
  6. Keep raw records, codes, identifiers, and calculations available in detail.
Evidence stays labeled. “The wire shows,” “Aaron noted,” “Spamhaus listed,” and “Nightwatch assesses” are different claims. The product must not blur telemetry, human context, third-party context, assumptions, and judgment into one confident paragraph.

Personality without theater

The voice is human because it uses natural words, makes a clear judgment, and respects the reader’s time. It does not use swagger, snark, synthetic empathy, product praise, corporate filler, or dramatic metaphors. It does not soften harm or manufacture urgency. After Hours and Behind the Rack may keep their atmosphere, but a metaphor never replaces a fact or hides a limitation.

NIGHTWATCH_VOICE.md is the complete Watch Partner editorial guide for vocabulary, before-and-after patterns, uncertainty rules, interface conventions, accessibility, localization, and the release checklist.

09 / Reasoning guide

How Nightwatch reasons

Core product contract

Nightwatch is an observable reasoning pipeline. The model is one replaceable specialist near the end, not the pipeline itself. Collection, provenance, normalization, memory, arithmetic, correlation, routing, Daylight, workflow, and audit belong to the harness.

COLLECTBounded connector windows, cursors, receipts, health, and explicit completeness.
UNDERSTANDNormalize entities, preserve conflicts, remember the environment, and compute the facts.
JUDGECorrelate episodes and reason only on a new or materially changed decision package.
CHALLENGEDaylight tries to disprove the first conclusion before the case reaches a person.

Autonomous means doing the homework Available, expanding

Nightwatch resolves, computes, or fetches every relevant fact it is already allowed to obtain before asking the analyst. It reuses fresh case evidence, performs deterministic joins and comparisons, then makes the narrowest useful read-only connector or approved public lookup. Only genuinely inaccessible evidence, human intent, business approval, off-source events, or protected action authority should come back as a question.

Evidence, Harness, Model

Evidence is the versioned fact package from connectors, case memory, local history, public facts, and attributed human knowledge. Harness is Nightwatch: source availability, collection, normalization, caches, permissions, budgets, memory, validation, Daylight, workflow, approvals, and audit. Model is a replaceable specialist that interprets the ambiguity left in the bounded package. A model can explain evidence; it cannot grant itself a source or a write.

The dashboard’s At the Chalkboard room teaches this visually. The repository’s REASONING_ENGINE.md is the normative reasoning and Daylight contract; EVIDENCE_CONNECTORS.md defines source authority and the evidence envelope.

A proper question names the exact missing fact, why Nightwatch could not get it, which source or person can answer, and what each answer changes. “Was change CHG-1842 meant to authorize Aaron’s account on BUILD-04 at 02:10 UTC?” is useful. “Is this expected?” simply hands the investigation back to the human.

Implementation boundary. Scheduled methods collect deterministic evidence with narrow tool sets. The case-owned investigator receives the case, bounded memory, attributed notes, routed read-only evidence-source tools, and an audited query trace. Public network-origin and configured reputation context resolve before reasoning. A typed registry now records available, disabled, planned, unsupported, and unavailable sources with refreshed cases. The universal executor that decomposes every unknown, runs every safe targeted route, records stop conditions, and proves exhaustion is still incomplete.

The cumulative loop

  1. Frame the proposition and decision.
  2. Inventory current facts, coverage, freshness, history, and human context.
  3. Name precise unknowns and classify each as stored, locally computable, safely fetchable, human-only, unavailable, or prohibited.
  4. Resolve local and cached facts, then run the narrowest permitted read-only queries.
  5. Reconcile conflicts, test the ordinary explanation, and stop when the decision is supported or declared budgets and sources are exhausted.
  6. Reason over the typed result and ask only for the inaccessible remainder.
  7. Persist the trace, attribution, gaps, and material case change.
CapabilityStatusImportant limit
Case package, memory, notes, and routed evidence-source queriesCurrentTool use is model-selected within bounded rounds; universal proof of exhaustion is still incomplete.
Deterministic methods, correlation, material-change gate, and model accountingCurrentCoverage is only as complete as installed sources and method contracts.
Separate Daylight pass with fallback and cooldownCurrentUnavailable Daylight leaves the first conclusion authoritative and the gap visible.
Public IP/prefix/origin/owner/RPKI and opt-in reputationCurrentOwnership, route validity, and reputation do not prove traffic purpose.
Endpoint, directory, IdP, CMDB, cloud, vulnerability, and change connectorsContract onlyNo installed connector means Nightwatch must name the gap or ask the right owner.
Universal autonomous evidence plannerRegistry foundationThe safe executor and proof-of-exhaustion trace remain required.

What should become configurable

ControlSafe shapeRequired proof
Method routingChoose a qualified model profile per method and severity band.Tool-call, schema, latency, cost, and labeled-quality evaluation.
Evidence budgetRecord/tool/token ceilings with explicit incomplete coverage.No hidden truncation; stable behavior at each cap.
Decision policyConfidence thresholds for watch, escalate, or request more evidence.Precision/recall and reopening analysis on labeled cases.
Daylight policyEnablement and method-specific challenge strength.Ordinary-explanation capture without suppressing real incidents.
Context policyWhich prior judgments and tool results may be reused.Freshness, tenant isolation, and stale-context tests.
VoiceSmall reviewed presentation profiles, separate from evidence policy.No change to severity, uncertainty, citations, or safety gates.

What should not be a free-form knob

Tool permissions, write authority, publisher schemas, evidence citation rules, secret access, tenant reconciliation, and human approval gates must remain enforced contracts. A custom prompt must never widen them.

Evaluation loop

  1. Freeze a sanitized labeled set, including misses and ordinary activity.
  2. Record the baseline model, method version, cost, latency, and outputs.
  3. Change one policy dimension.
  4. Run schema, safety, quality, and cost gates.
  5. Shadow the profile without affecting the board.
  6. Promote gradually with automatic rollback on contract failure.

Sealed exercise evaluation Library foundation

Lab exercises can test whether Nightwatch helps a person solve the case without teaching the system the answer. Nightwatch investigates blind; an evaluator freezes the case snapshot; only then does an authorized person reveal a separately stored exercise key. The key records expected campaigns, stages, entities, distractors, known limitations, and the useful action.

The answer key must remain outside the ordinary Nightwatch database and unavailable to schedulers, correlators, reasoning routes, tools, prompts, and the operator board. Saying “this is one campaign” before the snapshot would contaminate the test. After reveal, the evaluator may score evidence and stage coverage, grouping and fragmentation, contamination, story quality, prioritization, and action quality.

Implementation status. A tested offline library can validate keys, freeze tamper-evident snapshots, compare them after reveal, and render a debrief. There is not yet an authenticated exercise-key store, upload flow, scheduler integration, or production evaluation screen.
10 / Security & operations

Keep the night shift on a short leash

Hardening baseline

  • Bind Nightwatch to loopback and terminate TLS at a maintained reverse proxy.
  • Require authentication; use secure cookies; rotate credentials and signing secrets.
  • Run as a dedicated non-root account with read-only application files.
  • Restrict secret and evidence files to that account; exclude both from source and support bundles.
  • Allow only required outbound destinations. Keep evidence-source tools read-only for model use.
  • Leave native tuning writes disabled unless reviewed; keep typed approvals and rollback audit enabled.

Operational symptoms

SymptomCheck first
No first roundRun doctor; inspect service logs, target auth, scheduler state, and time.
Empty boardCheck telemetry freshness, target scope, record contracts, and coverage/cap messages.
Reasoning backlogCheck primary quota, request spacing, queue limit, fallback eligibility, and repeated unchanged evidence.
Wrong entity relationshipsAudit internal networks/domains and reconciliation scopes before changing prompts.
Webhook missingVerify URL, secret reference, timeout, receiver signature validation, and transition eligibility.
Configuration button deniedConfirm the account has the Admin role and an active authenticated session.
11 / Configuration design

The control-plane map

The administration workspace exposes implemented controls and reserves clearly labeled locations for future safe controls, while keeping evidence contracts and authority fixed. This is the proposed order of work.

LayerConfiguration directionStatus
EnvironmentTargets, identity boundaries, lamps, credentials, storage, notifications.Current
ResourcesCadence, windows, caps, retention, model routes, budgets.Current
AttentionSuppression scope, tuning eligibility/expiry, severity and notification policy.Partial
DetectionValidated versioned rule packs and safe threshold overrides.Planned
InvestigationSigned bounded skills with schemas, tool allowlists, and tests.Planned
Enrollment & identityInvitation lifecycle, verified SMTP signup, recovery, sessions, TOTP MFA, role policy, and durable mail queue; external identity providers later.Current
ReasoningEvaluated per-method profiles, evidence budgets, and rollout controls.Planned
PresentationAdaptive Meaning / Reasoning / Source case composition, work anchors, and validated visuals.Partial
EnrichmentLocal identity and asset history first; cached public ownership and vulnerability facts second; optional reputation last.Partial
Lab evaluationBlind snapshot, isolated sealed answer key, post-reveal scoring, and analyst debrief.Offline library
AuthorityTool permissions, tenant isolation, publisher contracts, typed approvals.Fixed contract

Documentation version v2026.08.12. When product behavior and this site disagree, treat that as a defect: verify against the example configuration and release tests, then update both in the same change.