Retrieval over Slack history: what works, what's too sharp
An experiment with RAG over customer-Slack channel history. Three useful retrieval patterns, two failure modes that led us to gate the feature behind explicit capture flags, and the operational guardrails.
A customer asked whether PursuitAgent could retrieve against their internal Slack history when drafting. The pitch was straightforward: Slack is where the subject-matter experts actually talk, the answer to many proposal questions lives in a Slack thread from six months ago, and pulling that content into the retrieval scope would close a real gap.
We spent about a month on the experiment. This post is what we shipped, what we gated behind flags, and why.
What works — three patterns
Pattern one: SME threads about specific product behaviors. A Slack thread where an engineer answers a question from a sales rep about how a feature works, with follow-ups, is often the most concrete explanation of the feature in the company’s written record. Better than the docs, because it is in response to a specific ambiguity. Better than the proposal template, because it has not been through a marketing-smoothing pass. For retrieval purposes, these threads are gold.
Pattern two: incident retrospectives. Slack threads that capture a security or reliability incident — detection, response, remediation, follow-up — are the source of truth for answering “has this ever happened” DDQ questions. A DDQ asks “describe your most recent significant security incident.” The proposal team’s KB might have a polished incident-response policy. The Slack thread has the actual story. The polished answer cites the policy; the actual story grounds the polish.
Pattern three: customer-specific conversations. For renewal proposals, a Slack channel that captures the account-team conversations about a specific customer (their pain points, the usage data, the escalations) is the single best source for the exec-summary-level framing. These channels have the current-state information that the CRM does not capture in writing.
All three patterns produced better drafts in side-by-side testing against the same questions answered without Slack content.
What’s too sharp — two failure modes
Failure mode one: unsolicited information disclosure. Slack channels contain a lot of content that should not make it into a proposal even when it is technically relevant. Customer names that are under NDA. Internal commentary about competitors. Informal estimates that the company does not want to be quoted on. Personal opinions that are not company positions.
The retrieval system does not know any of this. It will surface a chunk based on relevance and authority, and the drafter will cite it, and unless the proposal writer catches it the content goes into the draft. In one test we ran, a draft cited a Slack thread in which an engineer referred to a named competitor in language the company would never use publicly. The quote was accurate; using it would have been a mistake.
Failure mode two: stale context promoted to current truth. Slack channels are a running conversation. A message from eight months ago might have been true when it was written and incorrect now. The retrieval system cannot easily tell which messages are provisional and which are settled. A thread in which an SME answered a customer question provisionally, and was later corrected in a separate thread, will surface the original answer without the correction unless the corrections are also indexed and ranked appropriately.
Both failure modes are preventable with enough content pre-processing. Neither is prevented by default, and both were frequent enough in our testing that we could not ship the feature as an on-by-default capability.
What we shipped
The feature is live, gated behind explicit capture flags at the channel level. A customer who wants Slack retrieval configures which channels are in scope, and each channel has three flags: retrievable (the default — content can be retrieved), citation-eligible (whether drafts can cite the content publicly, which is a stricter bar), and time-boxed (the maximum age of content indexed from the channel).
Channels default to not in scope. Opting a channel in requires an admin action and a documented reason. The opt-in UI surfaces examples of the failure modes above so the admin can decide with eyes open. An audit log records every opt-in and opt-out.
On the retrieval side, the system filters Slack chunks by the citation-eligible flag before they can be cited in a draft that will be submitted externally. Slack content flagged as retrievable-only (not citation-eligible) can still appear in the drafting context but will not be cited and the draft will be labeled as draft-informed-by-Slack until a human clears it.
The operational guardrail that mattered most
Two weeks into the rollout, we added a second-layer check: every Slack chunk that gets cited in a draft goes through a PII and confidential-information scan before the draft is presented to the user. The scan is a pattern-matching pass, not an LLM pass. It catches obvious cases (customer names not in the approved list, competitor names, financial figures that look provisional) and flags the draft for review.
The scan catches maybe 1 in 40 drafts. The false-positive rate is noticeable — the scan flags content that is fine about a third of the time it flags — but the cost of a false positive (a short review prompt) is much lower than the cost of a false negative (a leak in a shipped proposal).
What the feature is good for
Customers who have mature Slack archives, named SME channels with high-signal content, and a culture of using Slack for work (not just coordination) get real value from the feature. Customers with messier Slack archives or more casual Slack cultures get less value from it, because the signal-to-noise ratio in the retrieval is worse.
The rough rule of thumb we suggest: if you can point at three or four specific Slack channels where SMEs regularly write paragraphs-not-sentences about the kinds of things proposal questions ask, the feature will work for you. If your Slack is mostly “LGTM” and emoji reactions, it will not.
What is still open
We have not shipped retrieval against Microsoft Teams, which most enterprise customers use instead of Slack. The architecture generalizes but the integration is a separate project. The technical work is scheduled for Q2.
We have also not shipped a good answer for Slack content that is time-sensitive in a way the retention flag does not capture. An “use this for Q4 only” flag on a chunk is the direction we are thinking, but the UX for it is open.