Engineering5 minute read
When a client asks whether the AI systems we build retain their data, a model provider's zero data retention (ZDR) commitment forms part of our answer, weighed alongside what the client can inspect directly.
Published
When a client asks whether the AI systems we build retain their data, a model provider's zero data retention (ZDR) commitment forms part of our answer, weighed alongside what the client can inspect directly.
Most ZDR reviews close on a data processing agreement, a SOC 2 report and a contract clause stating that prompts and outputs are not stored. Those documents describe the provider's infrastructure, which the client cannot inspect. They do not reach the client's own logging pipeline, tracing agents or observability tools, all of which commonly store prompts.
For each system we build, we decide which parts of the retention guarantee rest on architecture the client can verify and which can rest on contract. The split depends on how sensitive the data is, which models the task needs and what the client is prepared to operate.
A prompt passes through several systems on its way to a model and back, and most of them can store a copy by default.
Under ZDR terms, providers often still reserve the right to hold some data for abuse monitoring or legal obligations, and those terms vary by provider, product and contract. Application performance tools such as Datadog, New Relic and Sentry can attach request bodies to traces and error reports. LLM observability tools such as Langfuse and LangSmith store prompts and completions by design, since engineers use them to debug model behavior. Reverse proxies, API gateways and load balancers keep access logs, and a crashed inference process can write a core dump holding whatever was in memory.
Engineers leave most of these settings on because they help with debugging. A client can therefore hold ZDR terms with its provider while its own log index keeps every prompt for as long as that index's retention policy allows. A retention guarantee has to cover both ends of the API call.
We use four architectures, listed by how much of the guarantee the client can verify directly. Most systems combine two or more.
An open-weights model runs on an inference server such as vLLM, Text Generation Inference or TensorRT-LLM, in the client's cloud account or data center. Outbound internet access is blocked at the network layer and requests arrive over private endpoints, so no third party receives the data.
The client's own stack still needs configuring. Request logging stays off in the inference server (recent vLLM releases leave it off unless --enable-log-requests is set; older releases needed --disable-log-requests). Proxies and gateways log status codes and latency and drop request bodies. Core dumps are disabled on inference hosts.
The client can audit all of this by reading the infrastructure code and network rules. The costs are GPU capacity, the engineering time to operate it and a quality ceiling set by the best open-weights model for the task.
Confidential computing lets a client use public cloud capacity while keeping data unreadable to the cloud operator during processing. The hardware encrypts memory and isolates the workload from the host: AMD SEV-SNP and Intel TDX on CPUs, and NVIDIA's confidential computing mode on Hopper and later GPUs.
Before sending data, the client's application requests a hardware-signed attestation report identifying the code running in the protected environment. If that measurement matches a build the client has reviewed, the application encrypts its data to that environment. The data is decrypted only in protected memory and the results are encrypted before they leave, so host administrators and host software cannot read it while it is processed.
Researchers have published side-channel and fault attacks against several generations of these technologies. Attestation identifies the running code but cannot show that the code is free of bugs that write data elsewhere. The tooling is also less mature than for standard deployments, which adds engineering time.
When a task needs a closed frontier model, a privacy proxy inside the client's network detects names, account numbers, client and project identifiers and similar values, using Microsoft Presidio, custom entity recognition and pattern rules. It replaces each with a consistent placeholder, so "Acme Corp lost $4.2M through a flaw in auth_v2.py" is sent as "Client_A lost Amount_1 through a flaw in Component_3." The mapping is held in short-lived encrypted memory inside the client's network, and the proxy restores the original values in the response.
Masking handles structured identifiers reliably. Free text is harder. A missed entity goes out unmasked, and context can identify a subject with every name removed, as a description of a regional hospital chain that switched EHR vendors last spring would. We measure detection rates on the client's own data and report masking as reduced exposure.
For lower-sensitivity workloads, we rely on a provider's ZDR or limited-retention terms, directly or through a cloud platform such as Amazon Bedrock or Azure, and configure the client's side so it keeps nothing:
store parameter are set explicitly on every request.The provider's side of this arrangement rests on contract. The client can verify that its own side stores nothing and sends data to one destination.
We classify each data flow before any discussion of models, since sensitivity usually rules some architectures out.
| Approach | What the client can verify | Models available | Engineering effort | Cost shape |
|---|---|---|---|---|
| Model in client network | Full path, from infrastructure code and network rules | Open-weights only | High | Mostly fixed GPU capacity |
| Confidential computing | Running code, by hardware attestation | Open-weights, in supported environments | High | GPU capacity at a premium |
| Masking proxy | What leaves the network, by sampling outbound traffic | Any, including closed frontier models | Moderate, mostly tuning detection | Per token, plus proxy hosting |
| Provider ZDR with clean perimeter | Client side only; provider side by contract | Any covered by the terms | Low to moderate | Per token |
A behavioral health operator's clinical documentation workflow could keep patient notes on a self-hosted model and send masked text to a frontier model for summarization. A sales operations tool working with pipeline data could rely on provider ZDR terms and the client-side controls in approach four. We record the decision for each data flow so the client's security team can review it.
A client's security and legal teams receive the following at handover, and can check each item without relying on us.
The seeded-data test in item four is the most direct check, because it measures what the system stored during a real run.
ZDR in the inference path does not govern data the client chooses to keep, such as conversation history, documents indexed for retrieval or audit logs a regulator requires. Those need their own access controls and deletion policies. It also does not stop a compromised endpoint or an authorized user from copying output.
The evidence describes the system at delivery. A new monitoring integration or a changed logging setting can reintroduce retention, so we build the egress and seeded-data tests into the client's release checks.
disable_log_requests deprecated in favor of enable_log_requests, which defaults to off.Tell us what is holding your operations together. thinkbridge will show you what it looks like built right.