Real-Time Bot Detection Platform

Project Overview

A large share of the traffic that arrives at a commercial website is not human. Some of it is benign — search crawlers, uptime monitors, preview fetchers. The rest is not: credential stuffers, scrapers, checkout bots, and automated clicks that quietly consume advertising budgets and poison every analytics number a business makes decisions on. The problem is that all of it arrives through the same front door as real customers, and by the time it shows up in a weekly report the money is already spent.

Our client set out to close that gap with a platform that answers one question — is this visitor a person or a machine? — while the visit is still happening. Not a nightly batch job, not a report: a verdict returned in the same moment the page loads, so that the customer's own systems can act on it immediately.

We spent eight months building that product end to end, from the browser-side signal collector through the scoring services to the data platform underneath them. The engagement ran through 2022 and wrapped up in 2023.

The Challenge

Three constraints defined the engineering work, and each one pulled against the others.

It had to be real time. A verdict that arrives after the visitor has already converted — or already drained a budget — is an audit trail, not a defence. Scoring had to complete inside a request, in a latency budget small enough that a customer would willingly put it in the critical path of their own page load.

It had to be high load. The platform ran at true high-load volume, processing tens of thousands of requests per minute across the customer base, with traffic that spiked without warning — a campaign launch on the customer's side and an attack from the other side look identical to a capacity planner. Peak was never the average, and the system had to be sized for the peak while remaining affordable at the average.

It had to be evidential. Every verdict had to be explainable and re-examinable after the fact. Customers do not act on an unexplained score, and neither the detection logic nor the models behind it can improve without the raw signals that produced yesterday's decisions still being on hand. That meant keeping everything — a retention and analytics problem sitting directly behind a latency problem.

Real time, high volume, and total retention are three requirements that no single database satisfies. Most of the architecture below follows from refusing to compromise on any of them.

Signal Collection in Pure JavaScript

The client side of the platform is a collector that runs inside the customer's page and gathers the evidence a verdict is made from: browser and environment characteristics, rendering and timing behaviour, interaction patterns, and the many small inconsistencies that separate a real browser driven by a person from an automated one wearing a real browser's user agent.

It is written in plain JavaScript — no framework, no build-time runtime, no dependencies. That was a deliberate constraint rather than a preference:

• It runs in someone else's page. The collector is embedded in thousands of sites we do not control, alongside whatever else those sites load. It cannot assume a module system, a polyfill set, or a particular browser baseline, and it must not collide with anything already on the page.
• It has to be small and fast. Anything shipped to every visitor of every customer is measured against the customer's own performance budget. A framework runtime would have been larger than the entire collector.
• It operates in a hostile environment. The subjects of the measurement actively try to defeat it. Signal collection has to be defensive about a page that may be instrumented, patched, or emulated around it — and that argues for code with no layers between it and the browser APIs it reads.

The collector's job ends at gathering and shipping signals. All judgment happens server-side, where the logic is not visible to the thing being judged.

Go on the Hot Path

Every service on the request path is written in Go. The workload is close to the language's ideal shape: enormous numbers of small, independent, mostly I/O-bound requests that have to be handled concurrently at predictable latency.

Two properties mattered most in practice. Goroutines made per-request concurrency cheap enough that ingest, enrichment, and scoring could each fan out internally without a thread pool to tune. And Go's runtime gave us the tail-latency behaviour the product was sold on — the number that matters in a real-time system is not the average response but the slowest few percent, and that is exactly where a heavier runtime spends its budget.

The path itself is deliberately short: ingest and validate, resolve context, evaluate the detection rules and models against the incoming signals, return the verdict. Anything that does not have to happen before the response — persistence, aggregation, enrichment, model feedback — is pushed off the hot path and onto the asynchronous pipeline behind it.

From a Single Database to a Data Platform

The platform launched on PostgreSQL, and that was the right call at the start. One database held the events, the accounts, the configuration, and the aggregates; one query language answered every question; the whole product could be reasoned about in one place while it was still finding its shape.

It did not survive contact with production volume, and the reason is worth stating precisely: the failure was not "PostgreSQL is too slow", it was that a single store was being asked to be four different things at once. A high-rate append-only event sink, a low-latency lookup store on the request path, an analytical warehouse for aggregates over billions of rows, and a search index for investigating individual sessions are four workloads with four contradictory sets of requirements. Tuning for any one of them made the others worse.

So we split them apart, and moved the platform onto Google Cloud:

Redis — the hot path's state. Redis holds everything a scoring decision needs to consult in single-digit milliseconds: recent per-visitor and per-session state, rate and frequency counters, resolved configuration, and reputation lookups. It is what makes an in-request verdict possible at all, and it absorbs the read volume that would otherwise have landed on the primary database.

Pub/Sub — the seam between synchronous and asynchronous. Every scored request is published as an event and the response returns immediately; everything downstream consumes from there. This is the decoupling that lets the request path stay short and lets the rest of the platform be scaled, redeployed, or temporarily slowed without the scoring endpoint noticing. It also means a traffic spike lands in a queue rather than in a database.

Object storage — raw events, kept indefinitely. Every signal payload is archived to S3-compatible object storage in its original form. It is the cheapest durable place to put data whose future use is not yet known, and it is the reason detection logic can be re-run retrospectively over real historical traffic instead of over a summary of it.

BigQuery — the analytical layer. Aggregations across the whole event history, traffic-quality reporting, cohort and campaign analysis, and the exploratory work behind new detection heuristics all run here, on a warehouse built for scans of that size, and entirely off the operational path.

Elasticsearch — investigation. When a customer disputes a verdict or an analyst chases a new pattern, the question is "show me these particular sessions, filtered six ways" — an interactive search problem rather than an aggregation one. Elasticsearch serves that, so exploratory queries never touch the systems that serve traffic.

PostgreSQL stayed, with a much narrower remit: accounts, customer configuration, and the relational data that genuinely wants transactions and constraints. Removing the workloads it was never suited for is what made it a good fit again.

Kubernetes on Google Cloud

The platform runs as containerised services on a Kubernetes cluster in Google Cloud. Given the traffic profile, the alternative was never seriously on the table: load that swings by an order of magnitude within minutes needs infrastructure that adds capacity by itself, and services that are individually cheap to replicate.

The split into small Go services pays off here. Ingest, scoring, and the asynchronous consumers have very different scaling curves, and as separate deployments each one scales on its own signal — the ingest tier follows request rate, the consumers follow queue depth. Rolling deployments meant the scoring endpoint could be updated during business hours without a maintenance window, which for a service sitting in a customer's page-load path was a hard requirement rather than a convenience.

Results

The client ended the engagement with a production platform that scores live traffic in real time, sustains tens of thousands of requests per minute with headroom for spikes several times that, and retains every raw signal it has ever seen — so a detection improvement made today can be validated against years of real traffic rather than against a synthetic benchmark.

Just as importantly, the architecture separates concerns along the lines that actually matter to the business: the hot path stays small, fast, and boring, while analysis, investigation, and model work happen behind a queue where they can grow arbitrarily without ever threatening the response time a customer sees.

This is the kind of system we take on end to end: high-throughput Go services, an event-driven data platform, and the Kubernetes infrastructure to run both. If you have a workload shaped like this one, tell us about it.