Publish a local bot control API without treating network access as permanent trust
A bot may authenticate correctly and still behave badly because of a bug, an unsafe job definition, or a failure to react to a website's instructions. In this guide, we design a continuously accountable gateway around a locally hosted bot API, then expose that gateway through a Localtonet HTTP tunnel. The tunnel supplies public reachability to the selected local service, while authentication, scoped authorization, rate enforcement, robots.txt checks, audit logging, suspension, and reactivation remain responsibilities of the application or its local reverse proxy. The result is an ingress design in which access can be evaluated throughout a job instead of being trusted forever after login.
📋 What's in this guide
Identity is the beginning of bot accountability, not the end
An authenticated request answers a limited question: did the caller present credentials that our gateway accepts? It does not prove that the requested action is appropriate, that the bot will remain within its assigned scope, or that a long-running job will react correctly when an upstream website asks it to slow down. A continuously accountable bot gateway separates these questions and evaluates them at the appropriate points in a job's lifecycle.
This distinction matters for both security and responsible automation. A valid identity can submit an overly broad crawl target. A previously approved job can encounter a new robots.txt policy. Several individually permitted workers can collectively exceed a host's reasonable request rate. An upstream server can return HTTP 429 after conditions change. None of these situations is resolved merely because the original caller passed authentication.
Continuous authorization means that authorization is not a single decision made when a session starts. The gateway checks whether each requested action is still allowed, whether the job remains inside its declared limits, whether upstream feedback requires a pause, and whether an operator has suspended the caller or job. The system records those decisions so that an operator can explain why work was accepted, delayed, denied, suspended, or reactivated.
Treat trust as something supported by continuing evidence. A single malformed request may be an accident, while repeated disregard for limits is a pattern. Your response can therefore be graduated: reject one request, delay a job, lower its effective rate, suspend it, or revoke the caller's access according to your policy.
Separate public ingress from application-level policy enforcement
The cleanest design uses two distinct layers. The Localtonet client runs on a device that can reach the local gateway and establishes an outbound connection to our relay. A Localtonet HTTP tunnel then gives the gateway a public HTTPS address. Behind that ingress, the local application or reverse proxy authenticates requests, authorizes operations, validates job specifications, applies budgets, and records decisions.
With Localtonet, you do not need inbound router port forwarding, a public IP address, firewall changes, or a VPN setup for this HTTP publishing workflow. The tunnel is available only while the selected client or device is connected and the tunnel is running. Creating a tunnel is not the same as starting it, so operational checks must cover both the local gateway and tunnel state.
This boundary is essential: our HTTP tunnel forwards traffic to the local IP address and port you configure. It does not determine whether a bot is behaving responsibly, inspect robots.txt on behalf of your worker, assign application scopes, calculate crawl budgets, or decide whether a job should be suspended. Those controls belong in the service receiving the tunneled traffic and in the worker responsible for outbound retrieval.
Recommended request path
- A control client connects to the public HTTPS address assigned to the Localtonet HTTP tunnel.
- The Localtonet tunnel carries the request to the configured local gateway address and port.
- The gateway authenticates the request and resolves the caller's current permissions.
- The gateway validates the requested action, target, budget, and job parameters.
- If accepted, the gateway stores the job and an audit event before making it available to workers.
- A worker claims the job only if it is active and has remaining concurrency and request budget.
- Before automated retrieval, the worker evaluates the applicable robots.txt rules and any stricter local policy.
- After each upstream response, the worker updates rate, retry, and suspension state.
- Operators use separate privileged actions to inspect, suspend, cancel, or reactivate jobs.
| Layer | Primary responsibility | What it should not be assumed to do |
|---|---|---|
| Localtonet HTTP tunnel | Provide public HTTP or HTTPS reachability to the selected local IP address and port through an outbound client connection. | Evaluate bot conduct, application identities, job scopes, robots.txt permissions, or outbound request budgets. |
| Local gateway or reverse proxy | Authenticate control requests, enforce endpoint access, reject oversized requests, and attach reliable request identifiers. | Assume that possession of a credential grants every operation or target. |
| Bot control application | Validate job intent, apply target and budget policy, persist state, and expose controlled operational actions. | Trust worker-supplied status without reconciling it against stored policy and events. |
| Worker and scheduler | Enforce dispatch timing, concurrency, robots.txt decisions, upstream Retry-After instructions, and suspension state. | Continue work merely because the job was valid when first created. |
| Audit and monitoring system | Preserve decision records, produce alerts, and support incident review. | Become a storage location for credentials, authorization headers, or unrestricted retrieved content. |
| Operator | Review anomalies, correct configuration, rotate compromised credentials, and approve reactivation. | Reactivate a job without understanding whether its unsafe condition has been removed. |
Publish a deliberately designed control gateway rather than a debug server, development console, or general-purpose command executor. The exposed service should contain only the routes and actions remote clients need. Keep destructive operations separately scoped, validate every input, and never place device tokens, API credentials, cookies, or private upstream endpoints in public responses.
Controls for a continuously accountable gateway
Authenticate every request
Require authentication on every non-public gateway route, not only on the job creation endpoint. Status reads can reveal target names, schedules, error messages, and operational details. Cancellation and reactivation routes are more sensitive because they change execution state. If a health endpoint must be unauthenticated for connectivity checks, keep its response minimal and separate it from readiness information that could expose dependencies or configuration.
Validate credentials at request time so revocation takes effect on the next request. Avoid relying only on a long-lived application session that continues after its underlying account or key has been disabled. Never include authentication tokens in URL query strings, logs, examples, or job payloads. Store only what your chosen authentication system requires and protect it according to that system's security guidance.
Authorize actions and targets narrowly
A useful authorization model covers both verbs and resources. Example permissions might distinguish job submission, job inspection, cancellation, suspension, and reactivation. Resource constraints can limit target hosts, URL path prefixes, schedules, request methods, maximum depth, response size, or total requests. These are design examples rather than Localtonet settings.
Resolve authorization against current policy each time a privileged action occurs. A caller permitted to submit work for one owned domain should not be able to alter a payload and target an arbitrary internal address or unrelated public host. Normalize and validate destination names before policy matching, and repeat destination checks after redirects so a permitted starting URL cannot silently become an unapproved destination.
Apply layered request budgets
One global requests-per-second value is rarely sufficient. An accountable scheduler should be able to consider at least the caller, job, destination host, and total worker pool. Concurrency limits cap simultaneous work, while rate limits control starts over time. A total request budget places an upper bound on the job even when its instantaneous rate is low.
Keep these limits enforceable at dispatch time. Checking a rate only when the job is created does not protect a destination if multiple workers later run the same job concurrently. If the system is distributed, the budget state must be coordinated strongly enough that workers cannot each believe they own the same remaining allowance.
| Control | Question answered | Typical enforcement point |
|---|---|---|
| Caller rate limit | How frequently may this identity invoke the control API? | Gateway before request processing |
| Job request budget | How many outbound retrievals may this job attempt in total? | Scheduler before every dispatch |
| Per-host pacing | How quickly may all workers collectively contact one destination? | Shared scheduler or host-specific queue |
| Concurrency ceiling | How many requests or workers may be active at once? | Worker lease or semaphore acquisition |
| Payload and response bounds | How much data may a control request or retrieval consume? | Gateway and retrieval client |
| Failure threshold | When do repeated denials or errors require suspension? | Policy evaluator after recorded outcomes |
Make suspension a real execution state
A suspended job must be ineligible for new dispatch. Workers should check state when acquiring work and again before an outbound request if substantial time has passed. Queued retries should not bypass the suspension decision. In-flight requests may need a defined cancellation policy, but the safe minimum is to stop issuing new work immediately.
Record a machine-readable reason such as repeated rate-limit disregard, robots policy denial, exhausted request budget, invalid target transition, operator action, or credential revocation. Pair that reason with a readable explanation, the triggering event identifiers, the policy version, and the time. This gives an operator enough context to investigate without reconstructing state from scattered log lines.
Require deliberate reactivation
Avoid automatic reactivation after a serious policy suspension. A timer can make sense for an ordinary scheduled pause, particularly one derived from a valid Retry-After instruction, but it should not erase the distinction between backoff and suspension. Reactivation should be a separately authorized operation that records who approved it, what was changed, and whether counters or budgets were preserved.
Preserving history is important. Reusing the same job identifier can be reasonable if the event stream clearly shows the suspension and reactivation. Creating a replacement job can be cleaner when policy or target scope changes materially. In either case, do not delete the evidence that led to the intervention.
Honor robots.txt, HTTP 429, and direct upstream feedback
Check robots.txt before automated retrieval
The robots exclusion protocol gives a website a standard way to publish crawler preferences. Before retrieving automated content, the worker should obtain and evaluate the robots.txt rules applicable to its declared user agent and requested path. The parser should follow RFC 9309 semantics rather than using substring checks or an improvised pattern matcher.
A robots.txt decision belongs in the outbound worker, not in the Localtonet ingress path. The tunnel publishes your control API. It does not observe or govern the separate requests your bot later makes to destination websites. Your scheduler should prevent dispatch when the applicable rules disallow a path and record the policy outcome without repeatedly requesting the disallowed resource.
Cache handling needs care. Caching can avoid fetching robots.txt before every URL, but indefinite caching can preserve an obsolete permission after a site owner changes policy. Define a refresh and failure strategy appropriate to your use case, based on standards and legal or organizational requirements. If the current policy cannot be determined safely, a conservative system pauses retrieval rather than interpreting uncertainty as permission.
An allowed path does not cancel authentication requirements, terms, copyright restrictions, privacy obligations, explicit cease requests, or technical feedback from the destination. Configure the worker to honor stricter signals and your own approved-use policy even when a path is not disallowed by robots.txt.
Interpret HTTP 429 as an instruction to reduce pressure
HTTP 429 means the destination is reporting too many requests. The responsible response is to slow down or pause, not to rotate workers, change identifiers, or immediately retry through another route. If the response supplies a valid Retry-After header, schedule the next eligible attempt no earlier than the indicated time. Retry-After can represent a delay in seconds or an HTTP date, so use a standards-aware parser and account for invalid values without creating a rapid retry loop.
If no usable Retry-After value is present, apply your own bounded backoff policy with jitter. Jitter prevents many paused workers from resuming at exactly the same moment. Cap both retries and elapsed retry time so a permanently rejected job does not remain alive forever. The precise delays and thresholds are operational policy choices and should be selected for the destination, workload, and applicable rules rather than copied blindly from an example.
Backoff must be coordinated per destination where appropriate. If worker A receives a 429 but workers B through Z continue at full speed, the bot has not meaningfully responded to feedback. Store a host-level next-eligible time or equivalent shared limiter that all workers consult before dispatch.
Distinguish control-plane throttling from upstream throttling
Your own gateway can also return HTTP 429 when a client exceeds its control API allowance. Include a meaningful response body that identifies the applicable policy category without revealing sensitive internals. Add Retry-After when the gateway can state a safe retry time. The caller should not interpret a successful tunnel connection as permission to ignore the gateway's response.
An upstream 429 is different. It comes from the website the worker is visiting and should update outbound scheduling state. Keep both event types in the audit model, but label their origins so operators know whether the control client overloaded your gateway or the worker overloaded a destination.
Use a clear outcome model
| Observed condition | Immediate action | Accountability record |
|---|---|---|
| robots.txt disallows the requested path | Do not retrieve the path; deny or skip according to job policy. | Target, declared user agent, policy result, policy retrieval time, and decision reason |
| Upstream returns 429 with valid Retry-After | Pause that destination until the indicated time. | Status, parsed retry time, destination scope, attempt number, and scheduler update |
| Upstream returns 429 without usable Retry-After | Apply bounded backoff with jitter and a retry limit. | Status, parsing outcome, selected delay, policy version, and remaining retries |
| Job exhausts its request budget | Stop new dispatches and move to a terminal or review state. | Configured budget, consumed count, final request identifier, and resulting state |
| Repeated policy violations occur | Suspend the job or identity according to the configured threshold. | Linked violation events, threshold, suspension reason, and actor or automated rule |
| Operator approves corrected work | Reactivate only within the newly confirmed scope and limits. | Approver, review note, changed policy, preserved history, and reactivation time |
Implement the local bot gateway
The exact framework, route names, storage engine, and listening port depend on your application. The examples below describe required behavior rather than prescribing an unverified software package or a Localtonet-specific API. Replace placeholders with values from your own deployment, and keep the gateway local until every control has been verified.
Define identities, scopes, and target boundaries
List every client that may call the gateway and the minimum actions it needs. Separate read-only inspection from job creation, cancellation, suspension, and reactivation. Define permitted destination hosts and any path, method, schedule, concurrency, response-size, or request-count constraints. Set a deny-by-default result for actions and destinations that do not match policy.
Build the narrow control API
Expose only the operations needed to submit and manage jobs. Authenticate each request, authorize the exact operation, validate content type and payload size, normalize destination data, and reject unknown fields when practical. Generate a request identifier at the gateway and return generic client errors that do not expose stack traces, filesystem paths, credentials, or internal network details.
Persist jobs as explicit state machines
Represent pending, active, delayed, suspended, completed, failed, and cancelled states explicitly if those states fit your workflow. Define which transitions are valid and who may request them. Store the current state transactionally with the event that caused it so a crash cannot leave the audit record and runnable state contradicting each other.
Enforce budgets in the scheduler
Before each dispatch, check current job state, total budget, caller allowance, destination pacing, and concurrency. Use leases or another ownership mechanism so two workers do not claim the same unit of work. Recheck critical restrictions before making the outbound request, especially after a delayed queue wait or redirect.
Add robots.txt and upstream feedback handling
Identify the bot accurately, evaluate applicable robots.txt rules before retrieval, and deny disallowed paths. Parse upstream 429 and Retry-After responses, update shared destination scheduling, and apply bounded backoff when no usable instruction is available. Treat direct blocking or cease signals according to your policy rather than searching for ways around them.
Record decisions and protect the audit trail
Write structured events for authentication results, authorization decisions, job creation, dispatch, robots policy outcomes, upstream feedback, budget changes, suspension, cancellation, and reactivation. Restrict audit access, define retention, and avoid credentials or unnecessary content. Make event identifiers linkable so an operator can follow one request through job and worker activity.
Create operational suspension and reactivation paths
Ensure suspension immediately prevents new work and that queued retries consult the new state. Require a separately scoped, authenticated action for reactivation. Capture the operator's reason, any policy changes, and the evidence reviewed. Never make reactivation a side effect of reading or editing an unrelated job field.
A practical audit event shape
A structured event should capture enough information to reproduce a decision without storing secrets. The following is an illustrative schema, not a required Localtonet format. Field names and identifiers should be adapted to your application.
{
"event_id": "generated-event-id",
"occurred_at": "timestamp",
"request_id": "generated-request-id",
"actor_id": "internal-identity-reference",
"job_id": "internal-job-reference",
"action": "job.dispatch",
"resource": "approved-target-reference",
"decision": "delayed",
"reason_code": "upstream_retry_after",
"policy_version": "current-policy-reference",
"next_eligible_at": "timestamp",
"http_status": 429
}Do not log authorization headers, Localtonet device tokens, session secrets, upstream cookies, or full sensitive response bodies. If operators need evidence that a payload was evaluated, a safely generated digest or a reference to access-controlled storage may be more appropriate than duplicating the complete payload in a broad log system.
Verify security and behavior before publishing the gateway
Start the gateway on the local device and verify it through its loopback or local network address before adding public ingress. No universal port can be given because the gateway's listening port is defined by your application. Substitute the actual value you configured.
curl -i http://127.0.0.1:YOUR_PORT/healthA successful health response confirms only that an HTTP process is listening and that this route works. It does not prove authentication, authorization, scheduler enforcement, or upstream compliance. Test each control independently and preserve the resulting audit events.
Minimum local test matrix
| Test | Expected result | Evidence to inspect |
|---|---|---|
| Missing or invalid application credential | The protected route rejects the request and performs no state-changing action. | Authentication failure event without credential contents |
| Valid identity without the required scope | The action is denied even though authentication succeeds. | Actor, requested action, policy version, and authorization reason |
| Unapproved destination or redirect | The job or dispatch is rejected before contacting the unapproved target. | Normalized destination and target-policy decision |
| Request budget exhausted | No additional outbound work is dispatched. | Budget counters and terminal or review-state transition |
| robots.txt denies a test path | The worker does not retrieve that path. | User-agent match, policy result, and skipped dispatch |
| Controlled test server returns 429 | The worker delays future attempts and honors a valid Retry-After value. | Parsed delay, host limiter update, and next eligible time |
| Job is suspended while queued | Workers do not claim or dispatch new work for it. | Suspension transition and rejected lease attempts |
| Unauthorized reactivation attempt | The job remains suspended. | Denied privileged action linked to the actor |
Test robots policies, 429 responses, malformed Retry-After values, redirects, timeouts, and repeated failures against a controlled test origin. Do not generate unwanted traffic against an unrelated public website merely to prove that throttling works.
Also test restart behavior. A process restart must not reset total request counts, discard a destination pause, reactivate suspended jobs, or replay completed work unless that behavior is explicitly intended. Verify clock handling if Retry-After dates are used, and monitor for large time differences between cooperating systems.
Expose the verified gateway with a Localtonet HTTP tunnel
Add remote access only after the local gateway behaves correctly. The Localtonet client must run on a device that can reach the gateway's local address and port. If the gateway runs in a container, virtual machine, or another host, confirm that the client device can connect to the exact target address rather than assuming loopback will cross that boundary.
HTTP tunnels support a Process Type of Random Sub Domain, Custom Sub Domain, or Custom Domain. All three serve the configured content at a public HTTPS address. Availability and related requirements can vary, so use the options currently shown in your dashboard. Exact custom-domain DNS instructions should be taken from current Localtonet documentation rather than guessed.
Install and run the Localtonet client
Install the Localtonet client for the operating system on the device that can reach the local gateway, then run it. Do not copy an authentication token into source code, logs, screenshots, or this article's example commands.
Authenticate or select the client device
Use the device-specific authentication token through the supported product workflow and select the device that will run the tunnel. Tokens identify client devices and must be kept private.
Select an available relay server
Choose a server or region from the values currently available in the Localtonet dashboard. Server codes and availability should be read from the current product rather than hardcoded into deployment documentation.
Create the HTTP tunnel configuration
Create an HTTP tunnel, select the desired Process Type, and point it to the local IP address and port where the verified gateway listens. Use the address reachable from the Localtonet client device. Do not target a development server that lacks the authentication and policy controls described in this guide.
Start the tunnel
Press Start after reviewing the target. Creating the tunnel does not make it run automatically. The public endpoint remains available only while the selected client is connected and the tunnel is running.
Use and verify the assigned public URL
Open the public HTTPS address assigned by Localtonet and repeat the relevant connectivity and authorization tests. Confirm that unauthenticated protected requests still fail, scoped requests still obey policy, and request identifiers reach your application logs. Stop or delete the tunnel when remote access is no longer required.
The current product workflow and available configuration should be confirmed in the Localtonet HTTP tunnel documentation. That page is a supplement to the application controls in this guide, not a replacement for them.
Assume that an internet-facing endpoint will receive unexpected requests. Keep authentication mandatory, reject unsupported methods, constrain request bodies, remove debug routes, protect administrative actions, and monitor failed access attempts. The tunnel provides reachability, not automatic application authorization.
Operate, monitor, and troubleshoot the gateway
Monitor decisions, not only uptime
An uptime check can show that the public URL reaches the local process, but accountability requires behavioral signals. Track accepted and denied control requests, authorization failures, jobs by state, outbound dispatches by destination, robots policy denials, 429 responses, parsed Retry-After values, budget exhaustion, suspensions, and reactivations. Alert on meaningful trends rather than every isolated failure.
Useful investigations ask whether one identity is repeatedly requesting prohibited targets, whether a destination is returning increasing numbers of 429 responses, whether paused jobs are dispatching unexpectedly, and whether operators are reactivating work without a documented correction. Keep metrics and audit events correlated through identifiers, but avoid high-cardinality labels that include full URLs or private data unless your monitoring system is explicitly designed for them.
Use a controlled incident sequence
- Stop new dispatches for the affected job, identity, or destination.
- Preserve audit events and relevant configuration versions.
- Determine whether the cause was a caller, policy error, worker bug, stale robots data, scheduler race, or compromised credential.
- Correct the root cause and rotate credentials if exposure is suspected.
- Test the corrected behavior against a controlled origin.
- Reactivate with an authorized action and a recorded explanation.
- Observe the first resumed requests closely and suspend again if the unsafe pattern returns.
Troubleshoot by layer
| Symptom | Likely layer | Checks |
|---|---|---|
| The local health URL does not respond | Gateway process or local networking | Confirm that the process is running, the configured port is correct, and the address is reachable from the client device. |
| The local URL works but the public URL does not | Localtonet client or tunnel lifecycle | Confirm that the selected device is connected, the HTTP tunnel points to the right local address and port, and the tunnel was started. |
| The public health route works but protected calls fail | Application authentication or authorization | Inspect sanitized decision logs, credential validity, current scopes, route policy, and request formatting. |
| Workers continue after suspension | Scheduler or worker state enforcement | Check queue leases, cached job state, retry queues, transaction ordering, and whether state is rechecked before dispatch. |
| Requests continue during Retry-After | Shared destination limiter | Verify header parsing, time calculations, storage of next-eligible time, and whether every worker uses the same destination state. |
| A disallowed URL is retrieved | Robots or redirect policy | Inspect user-agent matching, parser behavior, cached policy age, URL normalization, and post-redirect authorization checks. |
| Audit events exist but cannot explain a decision | Audit model | Add policy version, reason code, actor, job, request correlation, relevant counters, and state transition details without adding secrets. |
Know when to stop the tunnel
Stop the Localtonet tunnel when remote control access is not needed, during an application security incident, or while replacing an unsafe gateway configuration. Stopping ingress is useful containment, but it does not automatically stop workers that already have queued jobs. Suspend or shut down the execution layer separately when outbound activity must cease.
Deleting a tunnel and revoking an application credential solve different problems. The former removes that configured public path. The latter prevents the credential from authorizing application requests wherever the application remains reachable. Incident procedures should identify which actions are necessary rather than assuming one control substitutes for all others.
Frequently asked questions
Does a Localtonet HTTP tunnel authenticate users of my bot API?
The HTTP tunnel provides public reachability to the local IP address and port you configure. Your local application or reverse proxy must enforce its own request authentication and authorization. Do not interpret possession of the public URL as proof that a caller is trusted.
Does Localtonet evaluate whether my bot honors robots.txt or HTTP 429?
No such behavior should be assumed. The tunnel carries inbound traffic to your control gateway. The outbound worker must evaluate robots.txt, respond to HTTP 429, honor valid Retry-After instructions, and apply your suspension policy. Record those decisions in the application audit trail.
Should a bot retry immediately when Retry-After is absent?
No. Apply a bounded backoff policy with jitter, coordinate it across workers contacting the same destination, and limit both retry count and total elapsed time. Exact delays are application policy choices. Repeated immediate retries can intensify the condition that caused the 429 response.
Is an allowed robots.txt path permission to retrieve and use anything there?
No. robots.txt communicates crawler preferences, but an allowed path does not override authentication, access restrictions, terms, privacy requirements, copyright obligations, or direct instructions to stop. Your approved-use policy may also be stricter than the published robots rules.
What is the difference between pausing and suspending a bot job?
A pause can be a normal scheduling condition with a known next-eligible time, such as waiting for Retry-After. Suspension represents a policy or safety intervention that blocks new dispatches until an authorized review and reactivation. Keep these states separate so a timer cannot accidentally undo a serious intervention.
What should an audit log contain?
Record event and request identifiers, time, actor reference, job reference, requested action, decision, reason code, relevant counters, policy version, state transition, and upstream status where applicable. Do not store Localtonet device tokens, application credentials, authorization headers, session secrets, or unnecessary retrieved content.
Does creating a Localtonet tunnel make the gateway immediately available?
No. Creating a tunnel does not mean it is running. Start it through the documented workflow. The public endpoint is available only while the selected Localtonet client is connected and the tunnel is running.
Can I use a custom domain for the gateway?
HTTP tunnels support Random Sub Domain, Custom Sub Domain, and Custom Domain process types. Use the choices currently available in your dashboard. Because DNS and availability details can change or vary, confirm exact custom-domain requirements in current Localtonet documentation before configuring records.
Publish your accountable gateway with Localtonet
Verify your bot gateway locally, enforce authentication and runtime policy at the application layer, then use a Localtonet HTTP tunnel to provide controlled public reachability without inbound router port forwarding.
Get Started Free →
