The AI invented a field that doesn't exist: why AI form builders hallucinate, and what actually stops it

What happened: the field you never asked for

You asked for a job application form. Six questions: name, email, role, years of experience, portfolio link, availability. What came back has nine. There is a LinkedIn URL field you never mentioned. There is a Preferred pronouns question you deliberately did not want on this particular form. And the role dropdown offers Senior Engineer, Staff Engineer and Engineering Manager, none of which are the roles you are hiring for.

That is not one bug but two, with different causes and very different odds of ever being fixed by the builder rather than by you.

The first is invented content — questions, options and labels the model produced because they were plausible, not because you asked. The second is wrong structure — a real field type applied to the wrong question. On screen they are indistinguishable: a form that renders fine and is wrong.

The third row is the one engineering can eliminate by construction. The first two are the ones that end up on your desk. Any tool that implies otherwise is selling you something.

Why structured output makes hallucination worse, not better

The intuition is that asking a model for JSON makes it more reliable than asking it for prose. A schema is constraints, constraints are good, therefore fewer mistakes. That is half right, and the wrong half is the expensive one.

Models are rewarded for guessing

Kalai, Nachum, Vempala and Zhang state the mechanism directly in Why Language Models Hallucinate: training and evaluation procedures reward guessing over acknowledging uncertainty. Their analogy is a student sitting a multiple-choice exam. One point for a correct answer, zero for a wrong one, zero for a blank. Under that scoring, never leaving a question blank is not a character flaw. It is the optimal strategy.

That is not one vendor's defect; it is how nearly every benchmark scores. A model that admits it does not know which options belong in your dropdown scores the same as one that gets them wrong, and worse than one that guesses right.

A schema is a shape, and a shape wants filling

Now hand that model a schema. A select field with options: string[]. The schema does not say leave this empty when the user did not name any options. It says: here is an array, produce one.

An empty array is a legal value and almost never the most probable one. The model has been shaped by every dropdown it has ever seen, and every one of them had options in it. So it writes options. Plausible ones.

That is the forcing function, and it gets stronger the more rigid the schema is. A required field in your output type is a required guess: every slot you did not specify gets filled from the training distribution rather than from your request.

Writing a better prompt is not a fix for a structural problem

You can move the rate. Tell the model to leave options empty unless the user names them and it will comply more often. More often is doing all the work in that sentence.

A prompt is a soft constraint on a probabilistic system. It reshapes the distribution; it does not truncate it. When the failure actually matters — a published form collecting real responses with an invented question on it — you need the bad output to be unrepresentable, not merely unlikely to be sampled. Those are different categories of engineering, and only one of them is a guarantee.

The two engineering answers: constrain the output, then verify the write

Constrain the output space until the bad answer has no representation

Vellform's agent does not write a form. It calls typed tools against a form's schema, and the argument types are a closed set: 18 field types and a typed editing agent, 4 working page kinds (Page, Cover, Review, Ending), and 10 conditional operators.

Closed is the load-bearing word. There is no nineteenth field type for a model to invent, because there is no argument value that would express one. A call that reaches for one becomes a rejected call rather than a broken form: the failure surfaces at the tool boundary instead of in front of a respondent.

// A free-text type is an invitation. A union is a wall.
type FieldType =
  | "short_text"
  | "date"
  | "single_select"
  | "multi_select"
  // ...18 in total. There is no 19th, so a model cannot name one.

addField({ type: "linkedin_profile", label: "LinkedIn URL" });
//              ^^^^^^^^^^^^^^^^^^ not assignable to FieldType.
//              The call fails. The form stays intact.

This is the OWASP prompt-injection cheat sheet's guidance for tool-using LLMs — least privilege, tool-specific parameter validation, a bounded action space — applied to a builder instead of a security perimeter. The security argument and the correctness argument land on the same design.

It edits data, never code

No tool in the agent accepts CSS. None accepts a script. There is no argument anywhere in its interface whose value is markup.

That closes a failure class free-generation builders live with: a model emitting a stylesheet that quietly breaks the layout at 375px, or a snippet you have to read before you trust the page. Styling here is a theme — roughly 30 CSS custom properties, and the presets a model may pick from are a fixed list it does not get to extend. A model can pick a preset or set a property's value. It cannot write a rule, and it cannot give you a bespoke effect the property set does not express. A closed vocabulary is exactly as limited as it is safe.

Verify the write by re-reading it

Constraint governs what is representable. It says nothing about whether the change you asked for is the change that landed.

So the loop has three steps, not two. The agent reads the form's schema. It writes a typed change. Then it reads the schema again to confirm the change is actually there. That third step is the one most tooling skips, and it is the difference between the model reporting that it added a field and the field being in the schema.

That is a hallucination check running in the product rather than a paragraph telling you to be careful. What the agent can and cannot change is written down, because a bounded tool is only useful when the bounds are legible.

The failure this does not catch

The same holds for invented content. Preferred pronouns is a perfectly legal label string. There is no type system in which a question you did not want fails to compile. Which is why what follows is a checklist for you and not a feature list for us.

What to check on any AI-generated form, in the order that catches most

Order matters: the first step deletes work the later ones would otherwise cover.

1. The question set — subtract before you edit

Read the list of questions first, ignoring types, labels and logic. Ask one thing of each: did I ask for this? Delete every no. Do not reword an invented question into a useful one — you will keep it because it is already sitting there, which is the sunk-cost fallacy with a text cursor in it.

Nielsen Norman Group's form guidance pushes the same way: cutting unnecessary fields takes time, and the reduced effort is worth it. A generated form hands you fields nobody argued for, so the cut is cheaper here than anywhere.

2. Field type, per question

This is where the wrong-but-valid failures live, so go one question at a time. Anything that should be a choice, a date, or a file upload and arrived as short text will collect free text you then clean by hand, on every response, forever. Ten seconds now against every export from here on.

3. Options inside choice fields

The single most common content hallucination. A model handed a string[] produces plausible strings. Read every option in every dropdown, radio group and checkbox set against the thing it describes: your actual price tiers, your actual departments, your actual shipping regions. Plausible and correct are unrelated properties.

4. Required flags

Models over-require. A boolean called required gets true more often than your form needs, and every unnecessary required field is a place a respondent stops. Walk the list and demote anything you would accept blank.

5. Logic — check the condition, not the existence of the branch

If you asked for a branch, a branch will be there. Whether it splits on the right answer is the separate and more interesting question. Read the operator and the value: is not where you meant is, greater than where you meant greater than or equal to, an option value that stopped matching the moment you renamed it in step 3. That last one is the most common way a working form quietly stops branching. Nothing errors. Everyone just takes the default path.

6. Labels and help text, last

A schema guarantees a label exists. It guarantees nothing about whether the label describes the purpose of the control, which is what the W3C's labelling tutorial actually asks for. Details is a label. So is Additional information (optional). Only one of them tells a respondent what to type.

Prompt for changes, not for masterpieces

The instinct with anything that feels expensive to run is to write one long careful prompt and hope. Wrong shape for this tool, and for a structural reason rather than a stylistic one.

Every AI edit lands in the same undo history as a manual edit. There is no separate pane of AI changes to approve, and there need not be: a wrong AI change is undone exactly the way a wrong drag is undone.

So the cost of a wrong small change is one undo. The cost of a wrong large change is reading forty fields to find the three that moved. Ask for one thing. Look at it. Ask for the next. Ten prompts you verified beat one prompt you admired.

One phrasing note that saves a round trip: name the target, because a rule's action comes from a closed list and the target is what picks it. When they answer X on page 2, send them to page 5 routes. Hide this field when X switches one field off. Make this required when X changes what has to be answered before the page will advance. All three are real actions the tool can take. What it cannot act on is a rule with no target in it — "only show the relevant bits" names nothing to point at, so you will get an approximation.

The trust question underneath all of this

Nothing in this post claimed an accuracy rate, and that is deliberate. The hallucination-rate figures circulating for AI builders come with no methodology, no corpus, and no definition of what counted as a hallucination. Repeating one would be its own small hallucination, performed by a human.

The claim that can honestly be made is narrower and more useful: the failure is bounded. A closed argument space means a class of wrong output has no representation. A re-read after every write means a change either landed or the agent knows it did not. A shared undo stack means the worst case is one keystroke of cleanup. And the residue — a real field type that is wrong for the question — is stated up front rather than found in a response export three weeks later.

NIST's AI Risk Management Framework takes the same posture on purpose: voluntary guidance built around governing, mapping, measuring and managing AI risk, not around eliminating it.

Our model is accurate is not a claim that survives contact with a model that will sometimes be wrong. Here is exactly what it cannot do, and here is what it costs you when it errs is. If you want to test that rather than take it: the free plan is 25 credits a month, no card, and a published form keeps collecting responses at zero credits.