Why survey quotas fail under concurrent traffic (and how to fix it)

Fire fifty concurrent completions into one cell and on many platforms the quota counter no longer matches the data. Why survey quotas over-fill under concurrent traffic, the race condition at fault, why 'soft counting' doesn't fix it, and what does: an exact count and a screen-out at the door.

You set your cell target for males 25–34 to n=300. Fieldwork opens Monday, closes Friday. You export the data.

There are 347 in the cell.

Most survey programmers have seen some version of this. The stakeholder asks why the numbers don’t match the plan. You explain about “soft counts” and “close-out lag.” You over-recruit next time. Nobody’s happy.

The mechanism behind that 47-person over-quota is worth understanding, because it isn’t a bug in your platform’s UI — it’s a race condition baked into how most survey platforms count. And it gets worse, not better, the more concurrent respondents you have.

The race condition

Imagine your quota logic in the simplest possible form:

On completion:
  1. Read current count for this cell
  2. Add one
  3. Write the new count back

That looks fine. But now imagine two respondents finishing the survey at the same millisecond, both in the same cell, which stands at 298 out of 300.

  • Respondent A: read count → 298.
  • Respondent B: read count → 298.
  • Respondent A: write 299.
  • Respondent B: write 299.

Two people completed. The counter went up by one. The platform now thinks the cell is one complete behind reality.

That’s a race condition — two operations that would produce a correct result in sequence produce an incorrect one when interleaved. And “the same millisecond” isn’t rare when the checks happen inside a request handler that spends time on network I/O, database reads, and logic evaluation. Two respondents arriving within a few hundred milliseconds of each other is easily enough time for their read-modify-write cycles to interleave.

For a low-traffic study, this rarely fires — you probably get one respondent every few seconds. For a study where recruitment goes wide (broad email blast, social panel, incentive-linked panel), you can have dozens of concurrent completions during peak minutes. The race fires constantly. Every fire leaves the counter one more complete behind, and a counter that is behind keeps the cell open. The screener keeps letting people into a cell that is already full, and 300 becomes 347.

What “soft counting” actually is (and why it doesn’t help)

The industry-standard band-aid is what platforms sometimes call “soft counts” or “in-flight tracking.” The idea: don’t just count completes — also count respondents currently inside the survey who are on track for a given cell. When your quota is 300, close the cell when soft-count hits 300 even if hard-count is only, say, 275.

This helps a little. It doesn’t fix the underlying problem, because:

1. It just moves the race condition earlier. Instead of racing on the completion increment, you’re now racing on the soft-count increment. Two respondents can still read a soft-count of 299, both decide to enter, and both push it to 301.

2. It creates a new problem: false closes. Respondents in-flight might drop out (browser closed, screening question flunked, session timeout). If your soft-count treats them as “will complete,” you close cells too early and finish under target.

3. The buffer is a guess. Platforms compensate for #2 by adding a fudge factor — “close when soft-count is 5% above target.” That’s operating on hope, not correctness. You still end up over on some cells and under on others.

Soft counting is not a fix. It’s a mitigation for a design flaw.

What a quota system that holds up looks like

It takes two things, and most platforms only talk about the first.

1. An exact count. Make the read-and-increment atomic. In database terms: acquire a row-level lock on the cell counters, read and increment in a single transaction, commit. If two respondents finish simultaneously, one waits for the other’s transaction to commit before it reads. There is no interleave. The counter always equals the number of completes, and a retry or reload is never counted twice. No fudge factor.

The tradeoff most platforms cite for not doing this: performance. Row-level locking on a hot counter serializes all completions for that cell, which sounds like it would tank throughput. In practice, quota counters are small integer updates — they hold the lock for microseconds. Even at high concurrent completion rates, the lock contention is negligible compared to any other database operation the survey does on completion.

The reason most platforms don’t do it isn’t performance. It’s that a lot of survey platforms grew out of PHP-era architectures where atomic transactional writes were harder to reason about, and by the time the tech stack could support it cleanly, the quota model was already sedimented into the product. Rewriting the counting layer is a lot of work for a bug that’s hard to see (you only notice the discrepancy after fieldwork ends) and easy to defend (“all platforms have this”).

2. A door at the screener, not at the last page. An exact count tells you when a cell is full. What matters next is where you act on it. The fair place is the screener: check the cell there, and anyone who would land in a full cell is screened out in the first minute, before they have invested anything.

That leaves the respondents who were already past the screener when the cell filled. A platform can cut them off on the last page with a quota-full, or let them finish. Cutting them off throws away an interview you have paid for, after the respondent has given you 25 minutes. Letting them finish means a cell can land slightly over target, by at most the number of matching respondents who were mid-survey at that moment, and the count shows you exactly how many. We think that is the right trade.

How to tell if your platform holds quotas

Three tests, from cheapest to most rigorous:

Test 1: read the docs. Search their documentation for “concurrent,” “race,” “atomic,” or “over-quota.” Silence is a signal. Platforms that hold quotas tend to talk about it because it’s a differentiator; platforms that don’t tend not to bring it up.

Test 2: simulate the peak. Use a load-test tool (k6, Locust, even a small Node script) to fire 50 concurrent completions into one cell. Then compare the platform’s quota counter with the number of completes in the exported data. They should be identical: 50 and 50. If the counter says 41, it is losing updates under load, and every lost update is a respondent your screener will let in when it shouldn’t. Then send one more respondent who qualifies for a cell already at target: they should be screened out at the screener, not on the last page.

Test 3: audit your past studies. Pull the quota report and the exported data for your last five studies. If the per-cell counts disagree, the counter is drifting. If cells routinely land well over target, look at where the quota check sits and how long the platform takes to notice a full cell.

Why we built Fieldbase this way

Fieldbase’s quota engine records every completion inside one locked transaction: lock the survey’s quota cells, match the respondent to their cells, increment, commit. Two respondents arriving at the same millisecond serialize through one at a time. The counter always equals the completes, and a reload or retry is never counted twice.

The hard stop sits in the screener. One line of script, quota("qtAgeGender", 1).isFull, reads the exact count and sends anyone who would land in a full cell to the quota-full exit before the survey proper starts. Anyone already past the screener when a cell fills finishes and is counted. We don’t throw away a 25-minute interview on the last page.

You can test the design before fielding using the survey simulator, which fires fake respondents against the real quota engine (same code path as production) and shows how your cells fill and where the screen-outs land.

This isn’t a novel invention — it’s standard transactional database practice applied to a domain that historically hasn’t. The point isn’t that it’s clever; the point is that the numbers on your quota screen match the numbers in your data file, even when your recruitment goes wide.

If you’re programming studies where quota precision matters — regulated research, syndicated panels, anything where the discussion guide has a hard n per cell — this is worth testing on whatever platform you’re currently using. The 5-minute load test above tells you the truth.


Fieldbase is AI-native survey software for research programmers. Quota counts stay exact under concurrent traffic; conjoint and MaxDiff are built in; exports drop straight into SPSS, Stata, and Excel. Currently in early access — get in touch.