The recycled-DDQ-answers audit

Pull last year’s shipped questionnaires. Pick 20 at random across the first three quarters. Read every answer. You will find roughly 40 recycled answers that were true when they shipped and are not true now.

This post is the short audit protocol. One afternoon, one analyst, one list of changes to make. It is the most productive use of idle January time a security-questionnaire team can find.

What to look for

Six categories of drift cover almost all of what goes wrong:

Expired evidence. The answer cited a SOC 2 report or pentest report that is now past the one-year mark. The answer text is still right; the evidence it points to is stale. Re-cite the current report.

Deprecated controls. A control that was deprecated during the year is still answered as active. The MFA product that was swapped in Q3; the backup vendor that changed in Q1. Any question that names a specific product by brand is a candidate.

Numeric drift. An RTO of four hours that is now three hours (improvement). An employee count of 400 that is now 600 (hiring). A 90-day key rotation that is now 60 days. These are not lies; they are stale. A buyer checking against the current SOC 2 will catch the mismatch.

Subprocessor list changes. New subprocessor added mid-year, not reflected in the Q1 DDQ answers. Old subprocessor removed, still listed in the December answers. This is the single most common category of drift we see in audits.

Policy version mismatches. The ISP (information security policy) was updated in June; answers from April still reference the prior version’s wording. The policy is fine; the answer’s cited version is wrong.

Regulatory additions. A new state privacy law or framework (SIG 2024 content update, CAIQ v4 update) that added controls the KB has not been updated to cover. Answers to “do you comply with X” may still be technically true but structurally incomplete.

How to run the audit

Pull 20 shipped questionnaires across Q1–Q3. Open each in parallel with the current KB. For every answer in each questionnaire, ask three questions:

1. Does the answer cite an artifact with a current effective date?
2. Does every numeric or named fact in the answer match the current KB block?
3. Is the cited KB block flagged as fresh, or has its freshness score dropped below the review threshold since the answer shipped?

A “no” on any question puts the answer on the rewrite list.

A reasonable analyst can run this protocol against 20 questionnaires — call it 4,000 answers — in one long afternoon. The reason it is tractable is that most of the 4,000 answers will have shipped from a small number of KB blocks. A drifted block shows up in 40 answers; fixing the block fixes all 40.

What to do with the list

Three buckets.

Immediate rewrite. Any answer that is factually wrong today — deprecated control, retired product name, expired attestation. Rewrite the KB block, mark the prior version retired, and notify any buyer who received the wrong version where the correction is material.

Next-cycle rewrite. Any answer that is stale but not yet wrong. The RTO that is 3.5 hours but still answered as 4. The policy that was updated but whose practical effect has not changed. Update the KB block; the next questionnaire picks up the new version automatically.

Retire. Any answer that was shipped once and describes a control or feature the vendor no longer has. Pull it out of the KB. Do not leave retired blocks sitting around waiting to be accidentally re-cited.

We have written before about the KB block versioning discipline and KB schema evolution that makes this kind of audit cheap. The short version: if your KB blocks do not carry version numbers and effective dates, this audit is expensive. If they do, the audit is a SQL query plus an afternoon of reading. An hour of block maintenance in January saves a day of questionnaire clean-up in March.