Build an Evidence-First AI Security Triage Harness

Why evidence must come before inference

Security alerts are claims that require investigation. A detector can report that a request matched a rule, a process exhibited suspicious behavior, or an identity performed an unusual action. That report is useful, but it is not automatically proof that an intrusion succeeded. An evidence-first harness preserves this distinction by treating the original detection as one input among several rather than as the final conclusion.

The central design rule is simple: deterministic application code establishes investigative scope and collects evidence before any model is asked to interpret the case. The model receives a bounded snapshot instead of direct, open-ended access to every security tool. This makes it easier to identify unsupported conclusions, repeat an evaluation, inspect missing data, and determine exactly which records influenced a recommendation.

A single general-purpose agent often combines too many responsibilities. If the same model chooses the account, selects a time window, queries tools, interprets results, assigns severity, and initiates a response, a prompt becomes the primary control boundary. Prompts can guide behavior, but they should not be the only mechanism preventing cross-tenant access, unrestricted time ranges, unapproved tool use, or destructive actions.

Keep three kinds of truth separate

A useful harness distinguishes observations, interpretations, and decisions. Observations are records such as authentication events, process launches, network requests, enforcement outcomes, or configuration state. Interpretations are reasoned findings derived from those observations. Decisions are organizational actions such as closing an alert, escalating a case, disabling an account, or isolating a host.

Mixing these layers makes audits difficult. A model-generated statement should never become indistinguishable from collected telemetry. Likewise, a recommendation should not be represented as an executed action. Store each layer independently and connect them using immutable identifiers.

A self-hosted security triage harness architecture

The harness should be divided into components with narrow responsibilities. The public-facing coordinator accepts an alert or case identifier, authenticates the caller, validates the request, and starts a workflow. It should not expose direct database access, generic command execution, collector credentials, or an unrestricted proxy to internal tools.

Behind the coordinator, a workflow engine invokes deterministic collectors. Each collector has access only to the source and operations it needs. A normalizer converts returned records into a common evidence envelope. An evidence store retains immutable snapshots or immutable snapshot versions. Specialist analyzers receive selected evidence views, and a synthesis stage combines their findings within a controlled output vocabulary.

The approval service is separate from analysis. It records who reviewed the recommendation and whether the proposed action was approved, rejected, or modified. An execution adapter may perform an approved response, but it should accept a narrowly defined action object rather than arbitrary text from a model.

Recommended trust boundaries

This separation limits the impact of both software defects and model mistakes. If an analyzer produces an unsupported recommendation, it still lacks the credentials and interface needed to execute that recommendation. If the coordinator becomes reachable from the internet, the exposed surface remains a narrow API rather than the telemetry store or administrative console.

Build deterministic evidence snapshots with provenance

A snapshot is the complete, bounded set of evidence admitted for one analysis run. It should identify the subject, tenant or environment, alert, collection window, collectors invoked, records returned, sources that failed, and normalization version. Once analysis starts, the snapshot should not change in place. If new evidence arrives, create a new snapshot version and preserve the earlier one.

Determinism does not mean that every upstream source will always return identical data. It means the harness records exactly what it requested, what it received, and what failed. Given a stored snapshot, the inference portion can be replayed without contacting live sources again.

Define scope in application code

Scope should be derived from validated case data and policy, not from model-generated prose. The coordinator can resolve an alert to an approved tenant identifier, asset set, and collection window. Collectors then receive a typed scope object. They should reject unknown fields, excessive time ranges, unsupported source names, and identifiers that do not belong to the authorized environment.

Keep the collection plan explicit. For example, a plan might request identity activity for the affected principal, recent detector history for the same asset, relevant network events in a bounded interval, and the enforcement result associated with the original alert. The exact sources depend on the tools deployed in the reader's environment. This architecture does not imply access to global telemetry or proprietary threat data.

Use an evidence envelope

A normalized envelope gives downstream code a consistent way to identify records without pretending that all sources mean the same thing. Preserve the original payload or an integrity-protected reference to it, then add normalized metadata around it. Avoid flattening source data so aggressively that source-specific semantics disappear.

{
  "evidence_id": "ev_example_001",
  "case_id": "case_example_001",
  "snapshot_id": "snap_example_001",
  "source": {
    "system": "identity_provider",
    "collector": "identity_activity",
    "collector_version": "reviewed-version"
  },
  "scope": {
    "environment_id": "approved-environment",
    "subject_id": "approved-subject",
    "window_start": "recorded-timestamp",
    "window_end": "recorded-timestamp"
  },
  "retrieval": {
    "status": "complete",
    "retrieved_at": "recorded-timestamp"
  },
  "observation": {
    "type": "authentication_event",
    "original_record_ref": "immutable-record-reference",
    "normalized_fields": {}
  }
}

The example deliberately uses placeholders instead of prescribing database technology, timestamp formats, or vendor-specific fields. Your implementation should adopt a canonical schema and validate it at every boundary. Stable evidence identifiers are especially important because model findings should cite identifiers, not merely quote fragments of telemetry.

Represent absence and failure precisely

“No event found” is not equivalent to “source was not checked.” A source can be unavailable, a query can time out, credentials can be denied, a retention window can exclude the requested period, or a collector can return a partial page. Record those outcomes as first-class evidence about collection quality.

Protect snapshot integrity

Evidence references should be immutable within a completed snapshot. If records need correction, preserve the prior version and create a new one. Access to raw payloads should follow least privilege because snapshots may contain security-sensitive information, personal data, internal addresses, or operational details. The analyst API should return only the fields required for review rather than offering unrestricted evidence-store browsing.

Retention periods, redaction rules, encryption requirements, and access-review procedures vary by organization and jurisdiction. Define them explicitly before collecting production telemetry. This guide does not prescribe a universal retention duration or compliance configuration.

Constrain AI analysis with typed inputs and outputs

Once collection is complete, the harness can create evidence views for narrowly scoped analyzers. One analyzer might review identity behavior, another might compare the current alert with prior dispositions, and another might inspect network observations. Specialization is valuable when each analyzer receives only the evidence relevant to its task and cannot retrieve additional data on its own.

A synthesis stage combines typed findings. It should not fetch new evidence, silently change scope, or invent a disposition outside the approved vocabulary. If it believes additional telemetry is needed, it can request another collection cycle through a structured gap field. Application policy then decides whether that request is permitted.

Require evidence-linked findings

Every factual finding should include one or more evidence identifiers. A statement without a valid reference is either an unsupported hypothesis or a general analytical note. Validate references programmatically by checking that each identifier belongs to the current snapshot and is visible to that analyzer.

{
  "analysis_id": "analysis_example_001",
  "snapshot_id": "snap_example_001",
  "classification": "needs_review",
  "confidence": {
    "label": "limited",
    "reason": "Network collection was partial"
  },
  "findings": [
    {
      "statement": "The admitted records show repeated authentication attempts in the reviewed interval.",
      "evidence_ids": ["ev_example_001", "ev_example_002"],
      "status": "supported"
    }
  ],
  "gaps": [
    {
      "source": "network_activity",
      "reason": "partial_collection",
      "effect": "The destination pattern cannot be fully assessed."
    }
  ],
  "recommendation": {
    "action": "analyst_review",
    "reason": "Available evidence is insufficient for automated disposition."
  }
}

Confidence should describe the strength and completeness of the analysis, not act as permission to execute a response. A high model confidence score is still model output. It cannot replace an authorization policy, and it does not correct missing evidence.

Use a controlled vocabulary

Define the classifications and recommendations the harness is allowed to produce. A minimal workflow might distinguish likely benign activity, suspicious activity requiring review, confirmed policy violation, insufficient evidence, and collection failure. The actual vocabulary should match the organization's incident process and be versioned.

Unknown values should fail validation instead of being accepted as creative model output. Store the schema version, prompt or analysis policy version, model identifier where permitted by organizational policy, snapshot identifier, and validation result. This allows reviewers to compare runs without confusing retrieval changes with interpretation changes.

Separate recommendations from response authorization

The safest default is advisory operation. The harness gathers evidence, generates findings, lists gaps, and proposes a next step. An authorized analyst reviews the case and decides whether to approve an action. This is especially important for actions that can interrupt users, isolate systems, modify firewall policy, revoke credentials, delete data, or suppress future alerts.

Human review should not be a cosmetic button placed after an opaque model result. The reviewer needs the original alert, normalized findings, direct evidence references, collection failures, confidence limits, proposed target, expected action, and rollback considerations. The approval record should identify the reviewer and bind approval to the exact action object that was reviewed.

Prevent approval drift

An approval must become invalid if the action target, action type, material parameters, snapshot, or policy version changes. Otherwise, a benign approval could be reused for a more consequential operation. Generate a stable representation of the proposed action and require the execution adapter to verify that it matches the approved object.

Give execution adapters narrow capabilities

Avoid an execution endpoint that accepts arbitrary natural-language instructions. Define operations such as “disable a specified account,” “isolate a specified host,” or “add a specified indicator to a reviewed block list” only if those operations fit your environment and policy. Each operation should have strict parameter validation, least-privilege credentials, idempotency behavior, a timeout, and an auditable result.

Not every recommendation needs an automated adapter. The harness can create a ticket or present a manual runbook instead. In environments with limited maturity, this may be safer than connecting response credentials to the workflow.

Implementation blueprint for a self-hosted harness

The implementation can use any stack that supports strict request validation, authenticated APIs, durable workflow state, and controlled access to local security systems. The following sequence is intentionally technology-neutral. It defines responsibilities and verification points without guessing a framework, database, model provider, or deployment platform.

A minimal public API surface

The public surface should support the smallest useful analyst workflow. For example, the application may provide operations to submit a case reference, read a sanitized case status, retrieve an analyst-oriented report, and record an authenticated review. Administrative functions, raw telemetry search, credential management, collector configuration, prompt editing, and direct action execution should remain on a separate private interface.

Endpoint names and authentication mechanisms depend on the selected application stack, so this guide does not prescribe exact routes or headers. Whatever interface is chosen, use strict schemas, reject unknown fields, limit response detail, and avoid exposing internal exception traces. A health response should reveal only what an authorized operator needs and should not enumerate internal services.

Local test checklist

• Verify the coordinator listens only on the intended local interface and port.
• Confirm a request without valid application credentials is rejected.
• Confirm one tenant or environment cannot request another tenant's evidence.
• Submit an oversized, malformed, or unexpected request and verify fail-closed validation.
• Simulate an unavailable collector and ensure the report says “not checked” or its equivalent, not “not found.”
• Insert instruction-like text into synthetic telemetry and confirm it remains data rather than controlling the analyzer.
• Return an invalid classification or nonexistent evidence identifier from a test analyzer and confirm schema validation rejects it.
• Attempt execution without approval and verify that no external action occurs.
• Replay a completed snapshot and confirm that collectors are not contacted during the replay.
• Review logs to ensure credentials and sensitive raw payloads are not written unnecessarily.

Expose only the coordinator API with Localtonet

After the harness works locally, a Localtonet HTTP tunnel can provide a public address for the selected local web endpoint. Our client application runs on the device that can reach the coordinator and establishes an outbound connection to a Localtonet relay server. This avoids requiring inbound router port forwarding, firewall changes, VPN setup, or a public IP address.

The tunnel should point to the coordinator or analyst API, not to collectors, security consoles, telemetry databases, model administration interfaces, or action adapters. The tunnel is available only while the selected client or device is connected and the tunnel is running. Creating a tunnel does not by itself start it.

Current relay server choices, regions, account availability, and dashboard options can vary. Obtain available values from the current Localtonet dashboard rather than copying a server code from an example. Likewise, use the actual local address and port configured for your coordinator rather than assuming a default.

For the current interface and configuration sequence, consult the Localtonet HTTP tunnel documentation. The documentation supplements the architecture in this guide, but it does not replace application-level access control.

What should and should not be exposed

Verify the complete public data path

A successful response from the public URL proves only that some route is working. Verification must cover identity, authorization, scope, evidence handling, analysis behavior, and tunnel shutdown. Test with synthetic cases before using production alerts.

Test from outside the local network

Use a separate network to verify that the assigned public URL reaches the coordinator. Begin with an unauthenticated request and confirm rejection. Then use an authorized test identity to submit a synthetic case. Check that the public response does not reveal internal addresses, collector credentials, stack traces, raw model prompts, or unrelated evidence.

Trace one case end to end

Follow the synthetic case through request validation, scope resolution, collection, normalization, snapshot completion, specialist analysis, synthesis, and review. Every stage should have a correlation or case identifier, but logs should avoid unnecessary sensitive payloads. Confirm that every finding references evidence in the same snapshot.

Exercise negative cases

Attempt to request an unauthorized environment, extend a time range beyond policy, inject an unknown collector name, reference a nonexistent case, and submit malformed JSON. The coordinator should reject these requests before they reach collectors or models. Also test concurrent submissions and replayed requests so that retries do not create duplicate actions.

Verify tunnel lifecycle behavior

Stop the Localtonet tunnel and confirm that the assigned public route no longer provides access to the coordinator. Restart it only when required and verify the intended configuration is active. If the Localtonet client disconnects, the tunnel is not available even if its configuration still exists in the dashboard.

Operate with measurable review points

Useful operational measures include collection completion rate, partial-source rate, schema validation failures, unsupported evidence references, analyst agreement with recommendations, approval rejection rate, and time spent waiting for human review. These measures should be interpreted carefully. A lower escalation rate is not automatically better if true incidents are being missed.

Review collector permissions and public API authorization regularly. Rotate application credentials according to organizational policy. Re-evaluate model prompts and schemas when the evidence format changes. Preserve representative snapshots for controlled regression testing only when retention and privacy rules permit it.

Troubleshooting common failure modes

The public URL does not reach the coordinator

First verify the coordinator locally using the exact target address and port configured in the tunnel. Then confirm the Localtonet client is connected, the correct device token was selected, and the tunnel was started. A saved tunnel configuration is not necessarily running. Also verify that the client device itself can reach the local target.

The tunnel works, but every request is unauthorized

This usually points to the harness authentication layer rather than the tunnel. Check the application's expected credential format, identity mapping, clock assumptions if time-bound credentials are used, and authorization policy. Do not disable authentication merely to make the public test pass.

The model claims that missing activity proves safety

Inspect collection status. An unavailable, denied, partial, or out-of-retention source must not be interpreted as an empty result. Strengthen the output schema and validation rules so the model has to list gaps and explain their effect. Consider preventing a benign classification when mandatory evidence sources are incomplete.

Two analysis runs produce different results

Confirm both runs used the same frozen snapshot, schema version, analysis policy, and model configuration. If the agents fetched live evidence independently, the runs may have received different inputs. Move retrieval out of the inference stage and replay the stored snapshot when comparing analysis behavior.

A specialist cites evidence from another case

Reject the response. Evidence identifiers must be validated against the current snapshot and the analyzer's admitted evidence view. Do not rely on the model to enforce this boundary. Investigate cache keys, session reuse, retrieval filters, and tenant-scoping code.

The model follows instructions found inside a log record

Treat all telemetry text as untrusted content. Delimit evidence from system instructions, minimize unnecessary free-form fields, and require a typed response. More importantly, ensure the model has no direct execution authority. Prompt hardening is useful, but application-enforced tool boundaries and human approval provide stronger controls.

An approved action changed before execution

The execution adapter should stop. Bind approval to an immutable action representation that includes the action type, target, material parameters, snapshot, and policy version. Any change requires a new review and approval.

The coordinator exposes too much internal detail

Add an analyst-facing response model instead of serializing internal workflow objects. Remove raw exception text, internal service addresses, unrestricted evidence payloads, model prompts, and credential-related fields. Return opaque case identifiers and the minimum information needed for the caller's role.

Frequently asked questions