Selma vs human research

Source attribution has been removed from this analysis

An earlier version of this page attributed outcomes to the Data Source field — reporting that “Direct Member Referral” produced 82% of wins and 81.5% of revenue, and that its share of a group’s mix predicted revenue per record at r = +0.65. Those claims were wrong and have been withdrawn.

The field is not fixed at record creation. Its “Direct Member Referral” share climbs monotonically with funnel depth — 10.5% of records at Research Submitted, 42.7% at GD Confirmed, 64.0% at a win — and peaks at 69.0% on No Response From Member, which is a loss. That last figure is what settles it: the label is not marking success, it is marking how far a record travelled. Comparing outcomes across it compares records that progressed against records that did not, which is circular.

What still holds. Everything comparing Selma against human sourcing — Selma is machine-assigned at creation (99% owned by the system account Referral Dev), so it cannot drift with stage. Also unaffected: the name-use permission-request reclassification, the stale-record and duplicate-pursuit findings, every group-level funnel rate, and the $22.9M opportunity outcome join itself. Sections 3 to 7 are unchanged.

Recovering the real provenance needs the value of Data Source at creation, which means Salesforce field history (Referral_Request__History) — or a source field captured once, at intake, and never rewritten.

First, a correction: this is three pipelines, not one funnel

The export contains three structurally different processes that use different status vocabularies. My earlier YTD report pooled them, which produced blended rates that describe none of them. That is corrected here, and it changes the headline numbers.

The tell is that Data Source and Referral Type are blank in almost exactly the same records — 41.0% and 40.8% at one status, 86.2% and 86.2% at another. Two independent fields do not go missing together by accident; they mark a different intake path.

• Researched — 10,907 records. A researcher sources a prospect, writes evidence, and the request runs the full course: research → GD → member → outcome.
• Name-use permission requests — 7,404 records with both fields blank. This pipeline is not referral sourcing. Roughly half ask a Group Director for permission to name a member on an upcoming prospect call — “is it ok to mention Anne as a General Counsel 50 participant?” — and the rest are other administrative notes on member conversations. Explicit permission language appears in 47.9% of them against 1.9% of every other record, a 25× concentration. Structurally the giveaway is cleaner still: they enter at Request Proposed, skip research entirely, and 99.1% sit in the propose/approve/decline band, because a granted permission has no member stage or won state to move to. Their 29 recorded wins are not a failure rate — winning is not what these records do. Counting them in any funnel denominator is a mistake, and my earlier report made it.
• Selma — 4,263 records, 99.5% owned by a single system account (Referral Dev). A bot pipeline, first appearing July 2026.

The end result: what actually closed

Referral outcomes joined to Salesforce opportunity records, pulled via Workbench and matched on the referral item number — 3,387 of 3,387 linked referrals joined, with every 15-character id agreeing with its 18-character Salesforce counterpart. 3,257 distinct opportunities, deduplicated so revenue is never counted twice.

The split below is Selma against human sourcing, and only that. Breaking the human side down by Data Source is what the correction above withdraws.

Selma vs research, age-matched

Selma launched in July, so it has no mature cohort and cannot be compared on lifetime win rate. The fair comparison is records created in the same window, all at least 30 days old, reviewed by the same people under the same conditions.

The one caveat that could soften this: only 10% of Selma records have been reviewed, and if researchers worked the most promising ones first, Selma's true quality is worse than 60.7% suggests, not better. If they worked the queue in order, 60.7% is representative. Either way the direction holds. Stage timestamps would settle it outright.

Why: Selma matches co-occurrence, researchers match relationships

The Comments field carries the evidence behind each ask, and reading it explains the kill rate completely. These are verbatim, from the export.

Selma writes a comment every time and writes a readable one. Length is not the problem — what it is evidence of is the problem.

Selma did not add capacity — it displaced the best source

Volume is normalised per business day, so a partial month and a short month compare honestly.

Selma's scale flattens everything else on that axis, so the same three human sources are shown again below on their own scale. This is where the displacement is legible.

The binding constraint: review capacity is flat

Only two pipelines use the research stage. Counting how many of their records ever leave Research Submitted gives a floor on how much the team can process per month.

Withdrawn: source quality by data source

This section previously ranked each Data Source value on GD approval, follow-through and end-to-end win rate, and named Direct Member Referral the strongest source at 41.5% end-to-end. It has been removed rather than restated, for the reason set out in the correction at the top: the field is rewritten as a record advances, so a ranking of outcomes by source value is substantially a ranking of how far those records got.

The same objection applies to Referral Type, which was also shown here. It goes blank in exactly the same records as Data Source — the two fields track each other to within a tenth of a point at every status — so it is maintained on the same schedule and carries the same contamination.

By group

All 37 communities with at least 150 sourcing records. An earlier version of this section ranked groups by their “Direct Member Referral” share and reported a correlation with revenue per record. That has been withdrawn — per the correction at the top, that share is largely a measure of how far a group’s records travelled, so correlating it with revenue was close to correlating revenue with itself.

What remains are the group measures that never touched Data Source: funnel conversion, Selma exposure, and the end result. Revenue columns are thin — 408 won opportunities across 37 groups — so single deals move a row.

Scroll sideways on a narrow screen.

Carried forward, recomputed on Jan 2025–Sep 2026

These findings from the YTD report survive the correction and the larger window.

• 6,641 referrals are stale — in a live status, untouched for 60+ days. 1,914 of those are GD-approved asks that were never put to a member.
• 6,405 open rows are duplicate pursuit — 2,035 prospects with two or more simultaneously open requests, 54% of the open pipeline.
• Opportunity outcomes are now joined — see section 2. What remains missing is stage-level timestamps on the referral itself, and the prospect's target group, which still blocks any same-group analysis.

What follows from this

1. Put a relationship-strength threshold in front of Selma. Not a volume cap — a bar. Score its output on the kind of evidence section 3 shows researchers accept, and only queue what clears it. The labelled training data already exists.
2. Stop measuring sourcing by volume. Review capacity is the scarce resource at roughly 741/month. The metric that matters is referrals per record reviewed, where researched names beat Selma 10×.
3. Protect Direct Member Referral. It is down 35% per business day and it is the highest-converting source you have. Whatever attention moved away from it should move back.
4. Work the shelf and dedupe — 1,914 approved-but-unsent asks and 6,405 duplicate open rows, both mechanical.
5. Do not read this as “switch Selma off.” It is two and a half months old, its output has never been filtered, and 10% of it has been looked at. The finding is that raw co-occurrence is the wrong signal — which is a fixable specification, not a dead end.