Why survey quotas fail under concurrent traffic (and how to fix it)
Most survey platforms silently over-quota under heavy concurrent traffic. Here's the race condition that causes it, why 'soft counting' can't fix it, and what an actually-holding quota system looks like.
You set your cell target for males 25–34 to n=300. Fieldwork opens Monday, closes Friday. You export the data.
There are 347 in the cell.
Most survey programmers have seen some version of this. The stakeholder asks why the numbers don’t match the plan. You explain about “soft counts” and “close-out lag.” You over-recruit next time. Nobody’s happy.
The mechanism behind that 47-person over-quota is worth understanding, because it isn’t a bug in your platform’s UI — it’s a race condition baked into how most survey platforms count. And it gets worse, not better, the more concurrent respondents you have.
The race condition
Imagine your quota logic in the simplest possible form:
On completion:
1. Read current count for this cell
2. Compare against target
3. Increment count
4. If count > target after increment, close cell
That looks fine. But now imagine two respondents finishing the survey at the same millisecond, both eligible for the same cell that’s at 299 out of 300.
- Respondent A: read count → 299. Under target. Continue.
- Respondent B: read count → 299. Under target. Continue.
- Respondent A: increment → 300.
- Respondent B: increment → 301.
Both complete. Both count. You are now at 301, not 300.
That’s a race condition — two operations that would produce a correct result in sequence produce an incorrect one when interleaved. And “the same millisecond” isn’t rare when the checks happen inside a request handler that spends time on network I/O, database reads, and logic evaluation. Two respondents arriving within a few hundred milliseconds of each other is easily enough time for their read-modify-write cycles to interleave.
For a low-traffic study, this rarely fires — you probably get one respondent every few seconds. For a study where recruitment goes wide (broad email blast, social panel, incentive-linked panel), you can have dozens of concurrent completions during peak minutes. The race fires constantly. Every fire adds one extra respondent to a cell that was supposed to be full.
What “soft counting” actually is (and why it doesn’t help)
The industry-standard band-aid is what platforms sometimes call “soft counts” or “in-flight tracking.” The idea: don’t just count completes — also count respondents currently inside the survey who are on track for a given cell. When your quota is 300, close the cell when soft-count hits 300 even if hard-count is only, say, 275.
This helps a little. It doesn’t fix the underlying problem, because:
1. It just moves the race condition earlier. Instead of racing on the completion increment, you’re now racing on the soft-count increment. Two respondents can still read a soft-count of 299, both decide to enter, and both push it to 301.
2. It creates a new problem: false closes. Respondents in-flight might drop out (browser closed, screening question flunked, session timeout). If your soft-count treats them as “will complete,” you close cells too early and finish under target.
3. The buffer is a guess. Platforms compensate for #2 by adding a fudge factor — “close when soft-count is 5% above target.” That’s operating on hope, not correctness. You still end up over on some cells and under on others.
Soft counting is not a fix. It’s a mitigation for a design flaw.
What an actually-holding quota system looks like
The clean fix is to make the read-and-increment atomic. In database terms: acquire a row-level lock on the cell counter, read + increment + close-if-full in a single transaction, release the lock. If two respondents hit simultaneously, one waits for the other’s transaction to commit before it reads. There is no interleave. The cell hits exactly the target, then closes. No over-quota, no soft counts, no fudge factor.
The tradeoff most platforms cite for not doing this: performance. Row-level locking on a hot counter serializes all completions for that cell, which sounds like it would tank throughput. In practice, quota counters are small integer updates — they hold the lock for microseconds. Even at high concurrent completion rates, the lock contention is negligible compared to any other database operation the survey does on completion.
The reason most platforms don’t do it isn’t performance. It’s that a lot of survey platforms grew out of PHP-era architectures where atomic transactional writes were harder to reason about, and by the time the tech stack could support it cleanly, the quota model was already sedimented into the product. Rewriting the counting layer is a lot of work for a bug that’s hard to see (you only notice the discrepancy after fieldwork ends) and easy to defend (“all platforms have this”).
How to tell if your platform holds quotas
Three tests, from cheapest to most rigorous:
Test 1: read the docs. Search their documentation for “concurrent,” “race,” “atomic,” or “over-quota.” Silence is a signal. Platforms that hold quotas tend to talk about it because it’s a differentiator; platforms that don’t tend not to bring it up.
Test 2: simulate the peak. Use a load-test tool (k6, Locust, even a small Node script) to fire 50 concurrent completions at a cell with target n=10. Check the final count. If it’s exactly 10, you’re fine. If it’s 11–15, you have the race condition and the platform’s mitigation partially works. If it’s 20+, the platform has effectively no protection.
Test 3: audit your past studies. Pull the final n per cell for your last five studies against the quota plan. If most cells are within ±1 of target, you’re on a platform that holds. If cells routinely miss by ±5% or more, you’re either seeing the race condition, soft-count fudge factor, or manual close-out lag.
Why we built Fieldbase this way
Fieldbase’s quota engine locks on every completion. The transaction is: acquire the row lock for the quota cell, check target, increment, close if full, release. Two respondents arriving at the same millisecond serialize through one at a time. The cell hits the target, exactly, and closes. No over-quota. No soft counts. No fudge factor.
You can test the design before fielding using the survey simulator, which fires fake respondents against the real quota engine (same code path as production) and tells you whether a plan hits its cell targets under concurrent load.
This isn’t a novel invention — it’s standard transactional database practice applied to a domain that historically hasn’t. The point isn’t that it’s clever; the point is that it’s how the numbers you set become the numbers you get, even when your recruitment goes wide.
If you’re programming studies where quota precision matters — regulated research, syndicated panels, anything where the discussion guide has a hard n per cell — this is worth testing on whatever platform you’re currently using. The 5-minute load test above tells you the truth.
Fieldbase is AI-native survey software for research programmers. Quotas hold under concurrent traffic; conjoint and MaxDiff are built in; exports drop straight into SPSS, Stata, and Excel. Currently in early access — get in touch.