AI-assisted survey programming: what the AI is actually good at (and what it isn't)

AI-assisted survey programming is the loudest claim in research software right now. Every platform has an “AI” button. Very few of them will tell you where it breaks.

We spent the last few months building one — an AI that reads a questionnaire document and writes working survey code, and a chat that edits that code in place. It works, and we ship it. But the honest version of this post isn’t “AI writes your surveys now.” It’s a list of the things the model turned out to be genuinely good at, the things it was confidently bad at, and — the part nobody writes about — the fact that almost every fix was structural rather than a better prompt.

What it’s actually good at

1. First-draft translation from a spec document

This is the real win, and it’s bigger than it sounds.

A questionnaire spec — a Word doc from a client, a PDF from a previous wave, an internal brief — is semi-structured prose. A human programmer reads it and performs a mechanical translation: Q1 becomes a single-select with these six options, S2 is a screener with a terminate, the grid in section 4 has these rows and that scale. It’s not intellectually hard. It’s just hours of careful typing, and it’s where a huge fraction of a research programmer’s week goes.

Models are excellent at this. Give one a real authoring reference and a 60-question document, and you get a complete, valid survey draft in a couple of minutes. Not a sketch — question types, option lists, routing, terminates, the lot.

The constraint isn’t quality, it’s output length. Our first version silently truncated: long input chopped at a character limit, long output stopped at the model’s per-response cap. A 300-page questionnaire produced a survey that just… ended. The fix was plumbing, not intelligence — stream the full document in, and when the model hits its output ceiling, ask it to continue from where it stopped and stitch the pieces together. Unglamorous, and it’s the difference between a demo and a tool.

2. Impact analysis — if you hand it a map

Ask a human programmer “if I delete S1, what breaks?” and they’ll trace it: S1 is read by two functions, piped into Q7’s text, and gates Q12’s if: condition. That’s the mental model that makes someone trustworthy on a codebase.

A language model does not have this model of your survey. It has the text of your survey, which is not the same thing — ask it the same question and it will give you a plausible, partially-wrong answer, because it’s pattern-matching rather than resolving references.

So we built the map. The engine already parses the survey; we had been throwing away the parsed model and keeping only the error list. Now we keep it, walk the already-extracted expression fields (conditions, validations, option sources, piping, function bodies) against the set of known symbols, and build a forward and reverse reference graph. That graph goes to the model as context.

With it, “if I delete S1 what breaks” names every dependent correctly. And “rename S1 to COUNTRY” updates the declaration, both function reads, the piped text, and the downstream condition — with zero orphan references left behind.

The lesson generalises: the model’s reasoning is only as good as the structure you give it. The capability wasn’t in the prompt. It was in deciding to compute a symbol table and ship it as context.

3. Answering questions, when you let it

Early on, every message in the chat was treated as an edit request. Ask “why does Q7 only show for half the sample?” and you’d get back a no-op code proposal — a diff with nothing in it — instead of an answer.

That’s a product bug dressed as an AI limitation. The fix was to let the model declare which mode it’s in: prose answer or code change. Answers render as answers, with no Accept button. Edits render as a diff you review.

Worth saying plainly, because it’s the fork in the road for this whole category: an AI that can explain a survey is often more useful day-to-day than one that rewrites it.

What it’s bad at

1. Staying in scope

This is the big one, and it is not fixable with prompt wording.

Ask the model to fix one warning on line 247 of a 900-line survey, send it the whole file, and ask nicely — very nicely, in capitals, with the word MINIMAL in the instructions — and it will frequently hand back the entire survey, “helpfully” normalised. Renamed question IDs. Reformatted option lists. Invented lists that weren’t there. The requested fix is in there somewhere, buried in a 900-line diff nobody can review.

We tried stronger prompting. It reduced the frequency and did not eliminate it, which is the worst possible outcome: a failure mode rare enough to stop testing for and common enough to ship a broken survey.

The fix was structural. A line reference now routes to a targeted path: find the enclosing top-level block, send only that block, splice only that block back. The rest of the file is unchanged by construction — not because the model was asked to behave, but because the other 880 lines were never in the model’s output in the first place. Same approach for question-scoped edits (“make Q7 required”).

If you’re evaluating an AI survey tool, this is the question to ask: when I ask for a one-line change, what guarantees the rest of my survey is untouched? “We tell the model to be careful” is not a guarantee.

2. Seeing things you never showed it

A user typed “fix the warning at line 247.” The model confidently discussed an unrelated block.

The reason was embarrassing once we found it. The validator runs and produces diagnostics — line number, code, message — and those are drawn as markers in the editor gutter. The human could see the warning. The model was sent the survey source and nothing else. It had no idea what warning was being referred to, so it guessed.

This is the most common class of AI bug we hit, and it’s never an AI bug. It’s a context bug. The model only knows what’s in the request. Anything your UI computes and renders for the human — validation state, selection, what’s on screen, what just changed — is invisible unless you deliberately put it in the prompt.

3. Knowing when it isn’t running

Before we had an API key wired into the development environment, the system fell back to a canned template — a keyword heuristic that produced generic output ignoring the input entirely.

For a week, the feature “didn’t work very well.” It wasn’t working at all. The fallback was plausible enough to hide the fact that no model had been called.

Silent degradation is the worst design choice available in AI features, and it’s everywhere, because a plausible fallback demos better than an error. We removed it: if the model isn’t available, the button is disabled and says so. A feature that is honestly off is more trustworthy than one that is quietly pretending.

4. Being cheap by default

A question we had to answer internally: how did one project burn 3.4 million tokens on a handful of small edits?

Because the full authoring reference — tens of thousands of tokens of language documentation — was being shipped on every single request. Roughly 85% of each call was re-reading the same manual to change one line. Worse, the cache that was supposed to absorb this expired faster than people actually work, so a user making a fix every fifteen minutes missed the cache every time and paid full price for the privilege of re-sending an identical document.

Two fixes. Edits now get a condensed digest — the hard rules, types, piping, scripting subset, common mistakes — about a quarter of the size, which is safe because the validator catches anything the shorter reference misses. And the cache window was extended to match how people actually use the tool. Cached context dropped by roughly three quarters with no measurable quality loss on real edits.

The general shape: the expensive part of AI authoring isn’t the thinking, it’s the re-reading. If a vendor charges you per AI action, it’s worth asking what’s actually in each request.

The thing that makes it safe

None of the above is what makes AI-assisted survey programming usable in production. This is:

Every AI change is a proposal, not an edit. It arrives as a reviewable diff with Accept, Refine, and Reject. Nothing touches the survey until a human says so. “Refine” iterates on the proposal rather than the live file, so you can argue with a draft without risking the original.

The validator is the backstop, not the model. Every proposal is parsed and validated before you see it. If it doesn’t compile, it doesn’t get offered. This is the single biggest reason a code-first survey platform is a better home for AI than a drag-and-drop builder: there’s a deterministic correctness check the AI has to pass. You cannot validate a GUI click.

Impact is shown, not assumed. A proposal that removes a symbol used elsewhere flags it: S1 was removed and was used by Q2, Q3, and this function — confirm those were updated. The AI doesn’t get to quietly orphan a reference.

What this means if you’re evaluating a platform

Four questions that separate real AI-assisted survey programming from an AI-shaped button:

Can it read a real spec document — your actual client Word doc, not a cleaned-up sample — and produce a complete survey?
What stops a small edit from rewriting the whole survey? Look for a structural answer, not a prompting answer.
Does every change arrive as a reviewable diff you can reject, with validation already run?
What happens when the AI is wrong or unavailable? Does it fail honestly, or hand you something plausible?

The honest summary of our experience: AI removes most of the typing and almost none of the judgment. It drafts a 60-question survey from a spec in minutes, and it still needs a programmer to read the diff, because it will occasionally do something confidently wrong in a way that looks exactly like something confidently right. That’s not a reason to avoid it. It’s a reason to insist your platform shows you the diff.

Fieldbase is AI-native survey software for research programmers. Surveys are authored as readable code, the AI drafts from your spec document and proposes reviewable diffs, and the validator runs before anything ships. See the scripting page for the authoring language, or how we think about exports for what comes out the other end. Currently in early access — get in touch.