The evidence vault, a year in: attestations, tests, audits

The evidence vault is the part of PursuitAgent that stores the artifacts a DDQ cites. SOC 2 reports, ISO 27001 certificates, penetration-test summary letters, DPA templates, sub-processor lists, insurance certificates. A year in, here’s what it actually holds and how the alerting keeps it from rotting.

What lives in the vault

After the block schema v3 merge, attestations are a kind of KB block. The vault is the attestation-kind subset, filtered by tenant and by category.

A typical mid-market tenant has between 40 and 120 attestations in the vault. The categories:

• Security certifications — SOC 2 Type 1/2, ISO 27001, ISO 27701, FedRAMP authorization letters. Each has an expiry tied to the audit period.
• Pen-test and red-team summaries — issued quarterly or annually, expire on the schedule of the issuing firm’s retest cycle.
• Data-protection agreements — the current signed DPA, sub-processor lists, TIA templates, SCCs. Expire when a new version is published.
• Insurance certificates — cyber, E&O, general liability. Expire on the policy renewal date.
• Internal policies — incident response plan, business continuity plan, data retention schedule. Expire on whatever the policy owner set as the review cadence.
• Scope-of-work letters and compliance attestations — HIPAA BAA templates, PCI-DSS AoCs, GDPR compliance letters. Expire when the issuing entity changes.

A handful of tenants also store agency-specific artifacts — CMMC status letters, ATO memos, FISMA compliance documents — that have their own expiry logic.

The expiry model

Every attestation has three dates in its metadata:

• Issued. When the underlying artifact was signed or the audit period began.
• Expires. When the issuing authority considers the attestation stale.
• Review by. An internal deadline, typically 30–90 days before expiry, that triggers the renewal workflow.

Most attestations have all three. A few don’t have a hard expiry — internal policies, for example — in which case the review-by date is the forcing function.

How the alerting works

Three alert tiers, keyed off the review-by date and the expires date:

1. Yellow — 90 days to expiry. A notification to the attestation’s owner that renewal should start. No action required; informational.
2. Orange — 30 days to expiry. A second notification, plus the attestation is flagged in retrieval. If a DDQ answer cites it, the reviewer sees a “renewal in progress” badge in the draft UI.
3. Red — expired. The attestation is suppressed from retrieval. A DDQ answer that would have cited it surfaces an error to the writer: “No current SOC 2 attestation available.” The writer can override with an explicit acknowledgment, but the default is block.

The red-level suppression is the feature that took the longest to agree on internally. Earlier versions of the product warned but didn’t block; the warning got clicked past and expired attestations made it into submitted DDQs anyway. Safe Security wrote about this specifically — vendors recycle stale answers and buyers collect reassuring-but-false attestations. We’d rather force an explicit override.

What’s hard about expiry tracking

Three things that broke or are still brittle:

Attestations with ambiguous expiry. An ISO 27001 certificate has a clear three-year validity. A pen-test report from a boutique firm might not have an issuer-stated expiry — we infer one from typical retest cadence, which is wrong some of the time. Fix: every attestation-kind block requires an explicit expires date at ingest, even if the source artifact doesn’t state one. The attestation owner has to decide.

Scope changes that don’t trigger re-attestation. A SOC 2 might still be current in date but the scope has changed — the company added a major product surface after the audit and that surface isn’t covered. The system doesn’t catch this, and it won’t catch it without structured scope metadata that auditors typically don’t provide cleanly. We surface scope text to the reviewer on every cite, but automation can’t close this gap.

Tenant-level vs. subsidiary-level attestations. A parent company’s attestation doesn’t always cover a subsidiary’s RFP response. We have a tenant-hierarchy flag on attestations, but it’s been reported wrong enough times that we don’t trust it at retrieval time — the reviewer still has to confirm on cite.

What’s working

The renewal calendar. Every Monday morning, every attestation owner gets a digest of what’s at yellow or orange status in their area. The 90-day yellow alert has been the single highest-leverage part of the system — renewal work starts early, artifacts don’t expire, and DDQ responses don’t get stuck at submission time waiting for an auditor to sign something.

The second working thing: the draft-time badge. Reviewers see the status of every cited attestation inline with the answer. This is a small UI touch but it has measurably reduced the “ship a DDQ with a stale attestation” failure mode. The earlier version of this post would have said something about our own metric here; we track the count but don’t publish it per tenant.

What’s next

A better story on scope metadata — possibly a structured scope DSL that attestation owners fill out at ingest. And a notification channel for buyers: when we update a SOC 2, the tenants we’ve DDQ’d for this year get a push to the buyer-facing customer-trust portal. That second piece is designed but not built.

See the DDQ evidence attachment API post for the integration surface if you want to pull evidence into your own responses programmatically.