The Grammar Has Two Authors

What breaks when a language model's program stops being scaffolding and becomes the thing the user keeps. Notes from eight weeks of building CalPal, a calculator that writes functions.

A budget that computed nothing

On the 27th of August a user dictated a request into our calculator: create a monthly budget planner for a salary of 220,000 rupees. The language model wrote the obvious thing, and every line after the first came back blank.

as it appeared, production

# Monthly income in INR
monthly_salary          = 220000                220,000

# 50/30/20 rule allocation
needs                   = 50% of monthly_salary
wants                   = 30% of monthly_salary
savings_and_investments = 20% of monthly_salary

weekly_wants_allowance  = wants / 4
daily_wants_allowance   = wants / 30

The report we got was that the calculations were not showing, meaning the whole sheet rather than one line of it. That reading was reasonable — and wrong in a way worth following. Only three lines were genuinely unsupported. The other two were blank because wants had never come into existence — and a name that does not exist cannot be divided by four.

The unsupported thing was 50% of monthly_salary. Our percentage rules matched digits on both sides of the sign, so 50% of 200 answered and 50% of monthly_salary did not. The model had written the most ordinary line in a budget — and our language could not read it.

The model was right and the grammar was wrong. 50% of monthly_salary is a line of code that reads like a sentence — and a parser built for arithmetic has no reason to accept it. Whether that counts as a defect at all depends on what happens to the program after it runs.

When the program is the product

CalPal is a notepad calculator, the interface Soulver established in 2005, where you type in the left column and answers appear in the right. Ours adds a language model, and the model's instruction is not to answer the question but to write a named function that answers it, and then call the function. The model is not the thing that computes — it is the thing that writes what computes, and what it writes stays in the document.

the same request, after

# Delivery date, skipping weekends
start_date     = today                  2026-08-27
lead_time_days = 10                     10
delivery       = business_days_after(start_date, lead_time_days)
                                        2026-09-10

Change lead_time_days to fifteen and every dependent line recomputes without a second call to the model. The function can be saved to a library, shared as a link that carries its inputs as parameters, and read six months later by someone deciding whether to trust it. A number does none of that — if you doubt it, you start again.

That is a small change in architecture — and a large one in what counts as a defect. When the program is scaffolding, a parse failure is an engineering event, and the response is to catch it and retry. When the program is the artifact, the same failure appears on screen, in the user's document, in the middle of their reasoning.

Once the program is the product, its failures stop being engineering failures and become interface failures, which obey different rules.

What the prior work optimises

The literature this sits next to solves a different problem, and solves it well. Program-Aided Language Models [1] and Program of Thoughts [2] have the model emit a program and an interpreter execute it; PAL reports an absolute forty-point gain over chain-of-thought on GSM-hard. The model does the reading and the runtime does the computing, which is a sensible division of labour.

Downstream of that sits a large body of work on making generated output conform: grammar-constrained decoding, grammar masking for modelling tasks, structured-output reliability, guardrail layers that validate before delivery. The engineering is good, and the framing across all of it is consistent. The grammar is fixed and correct. The model is the variable to be constrained.

What all of that work optimises is execution correctness. A generated program is judged on whether running it produces the right answer, and against that objective the program is free to be unreadable, because nobody is going to read it. It is written, executed and discarded — what reaches the user is the number.

Our setting adds a second objective the first does not contain. The program is not discarded. It is the thing the user keeps, edits, and sends to somebody else, so it is judged on whether it runs and also on whether a person can read it, change it, and decide whether to believe it. That is not a disagreement with the prior work — it is a different problem, and most of what we got wrong came from not noticing which of the two we were in.

The grammar has two authors, and they write differently

A transient program has one author. A persistent one has two — the model that drafts it, and the person who edits it afterwards. They do not write alike, and a grammar tuned to either will fail the other.

Our percentage rules were written for the human. A person typing quickly writes 1250 + 18% or 20% off 500, both literal, both digits, and the rules accepted exactly that. Then a model arrived and wrote 50% of monthly_salary — which no reasonable reader would call an error — and the grammar had nothing to say.

That divergence looks systematic rather than accidental, and the reason is in the two authors' situations. A person edits a sheet that already exists, a line at a time, reaching for the shorthand they have typed before and can type fastest. A model writes the whole sheet at once from a sentence somebody said out loud, with no history to imitate, so it reaches for the phrasing nearest the meaning of the request. Shorthand and paraphrase are both legitimate ways of writing the same calculation — and a grammar built from one author's habits will keep meeting the other's.

When a model writes idiomatic output your language rejects, the hypothesis that the language is too small is the one nobody checks. Sometimes the model really is wrong. The tooling only makes that the easier answer to reach for.

Something practical follows from this. Our test suite already checked that the shapes the prompt promises can be evaluated, running worked examples through the real engine so that prompt and engine cannot drift apart. That suite passed throughout, and it could not have caught this, because the prompt never mentions percentages of variables. The model wrote it because that is the natural way to say the thing, not because we asked.

So the suite gained a second section, held deliberately apart — shapes the model produces that the prompt never named, seeded only from output that real requests produced. Imagination cannot populate that category, since by construction it holds whatever nobody thought to specify.

One gap looks like total failure

In a transient pipeline a failed program fails alone. It is retried, or it returns an error, and the blast radius is one request. In a document of dependent lines, a single unsupported idiom takes down everything downstream of it.

Three unparsed lines produced five blank ones — which from outside is indistinguishable from a broken product, and was reported as one. We had been reasoning about coverage as though each gap cost a line. It costs that line and everything the line was going to feed.

That changes what a coverage count is worth. The marginal value of closing any given gap is higher than the count suggests, and silent gaps cost far more than loud ones, because an error message localises the blame to one line and a blank leaves the reader to work it out.

Constrained decoding is the standard answer to output that will not parse, and it would have worked here in the sense that matters least. Masking tokens against our grammar would have stopped 50% of monthly_salary from being written at all, and the model would have reached for something the grammar already had — monthly_salary * 0.5 or its equivalent — which parses, computes, and is worse to read than the line it replaced. The gap does not close. It moves out of the failure log and into the artifact, where the only person who can see it is the reader — who has no idea what was masked.

You cannot tell a note from a question

Because the artifact is a document as well as a program, people write things in it that are not calculations — headings, reminders, half-finished thoughts. Our engine stays quiet on those. A line it cannot parse returns nothing rather than an error, so that a notepad does not shout at someone taking notes.

That silence is also how an unsupported request looks. We set out to fix this by marking the lines that failed, and before building it we measured whether the difference was there to be found.

blank output, by intent

  prose notes       10 / 11 blank    "Hotel 3 nights"
                                     "Team offsite 2026"
                                     "call the bank tomorrow"

  unmet requests     8 / 9  blank    "10 working days from today"
                                     "50 mph in km/h"

It is not. Hotel 3 nights and 10 working days from today are both a few words with a number and a time word in them, and any classifier that caught the second would also scatter warnings across the first. We abandoned the feature.

What replaced it depends on a distinction that had not occurred to us until the marker failed. A warning makes a claim about the line — that it is wrong. An offer makes no claim at all. So the line offers to ask the model about itself. The control is a sparkle and the words ask AI, set in small letterspaced caps in the muted grey, sitting out to the right where a result would have gone. It stays invisible until the line is hovered or given keyboard focus, and stays permanently visible on touch, where there is no hover to reveal anything. Screen readers are told Ask AI about line 3. What the control never does is say something about the line — no question mark, no did you mean, no red or amber. A note can ignore it without having been accused of anything.

When you cannot classify intent, stop trying to classify it and change the speech act instead. An offer survives being wrong. A warning does not.

Measuring it also exposed something we had shipped without noticing. Our failure telemetry counted only results beginning with Error, so the failure the engine produces by design — the blank — had never been reported at all. We had been studying the loud failures and drawing conclusions about the quiet ones.

Capability the model cannot see is capability unshipped

We spent a week giving the engine a date type, with calendar-aware arithmetic, working-day functions and month boundaries. It all worked. Then we asked the deployed system to plan a project.

"a project timeline starting today with a three week design phase"

# before the prompt was told
design_weeks  = 3
calendar_days = design_weeks * 7
business_days = design_weeks * 5

# after
start_date = today
design_end = start_date + 3 weeks        2026-09-17
build_end  = design_end + 2 weeks        2026-10-01

The first version is a true statement about the number three — and it is not a schedule. Nobody asking when a phase ends wants to be told that it lasts twenty-one days.

In a conventional system a capability exists once the code exists. Here the language has two consumers and only one of them can read the source. The other knows whatever a paragraph in a prompt told it. So a feature the model cannot see is, from outside, unshipped — reachable only by the users who guess the syntax unaided.

That was the mild version. The same gap appeared again months later in a form that was much harder to notice, because the second time nothing came back blank.

Asked to plan a meeting between sid in IST and joe in EST, the model wrote this, and every line of it computed:

# Time difference between EST (UTC-5) and IST (UTC+5:30)
time_difference = 10 hours 30 min                  10.5 hours
joe_time_est    = 9:00 am                          9:00 am
sid_time_ist    = joe_time_est + time_difference   7:30 pm

The engine, from real timezone data, says 9am in New York is 6:30 PM in Kolkata. The sheet is an hour out, and it stays an hour out for roughly eight months of the year, because New York is on daylight time in September and a hardcoded offset cannot know that. Asked the same question about London and Sydney, the model wrote a comment observing that the offset "varies from 9 to 11" — and then hardcoded ten anyway.

It had no better option. The engine converts zones properly, using the same data every operating system uses, and it has a feature that converts a whole team on one line. The prompt mentioned neither. What it did do was list "timezones" among the things in scope, which is worse than silence: it promised a capability and taught no vocabulary for reaching it, so the only tool left was arithmetic on a constant the model invented.

The date type failed loudly — a schedule that answered in days rather than dates is visibly not a schedule. This failed quietly, and quiet is the expensive kind. A blank gets reported. A confident wrong answer that recomputes correctly from a wrong constant gets believed, and then sent to the person it is about.

Ambiguity is surfaced, not resolved

Our units library, like most, is US customary throughout. 1 gallon in litres answered 3.78541 whoever asked. Someone in Britain means 4.54609 — a twenty per cent error — and until recently there was no way to say which they meant, because neither US gallon nor imperial gallon parsed.

The obvious fix is to guess better and read the locale. We rejected it — for a reason that has to do with the artifact rather than with the units.

A calculation written in London and opened in New York must produce the same number. A shared artifact cannot depend on who is looking at it.

Personalisation is a virtue when the output is a view and a defect when the output is a document somebody will send to someone else. So the ambiguity is reported instead.

as shipped

1 gallon in litres              3.78541 litres
  assumed  US gallon — write "imperial gallon" for the UK one

1 imperial gallon in litres     4.54609 litres
  (no note. the reader was explicit)

The mechanism generalised past its origin. The word calorie on a food label means a thousand of the word in a physics textbook, and only one reading can be the default, so it dropped into the same list unmodified. So did the fluid ounce, where the trap is that the US one is larger than the imperial while the US gallon is smaller. The two errors run in opposite directions, so anyone reasoning by analogy from gallons arrives confidently at the wrong answer, which is why the assumption is printed rather than left to be inferred.

Which machine should own what

With two engines available, one deterministic and one fluent, the recurring question is which should answer. We wrote out 110 expressions spread across the domains we expected a calculator to meet and ran them through both. The list was ours rather than sampled from usage, which limits what it can show. The boundary that fell out of it had nothing to do with difficulty.

The dividing line is authority. Correctness for a conversion rests on a convention somebody outside the system maintains, and a hallucinated conversion factor is a wrong number that looks entirely right. Correctness for a method rests on constructing something useful, which is what a language model is for. Ask a model how many litres are in a gallon and it will tell you fluently, and you have no way to check short of looking it up, at which point you did not need the model. Ask it for an amortisation schedule and it will write you something you can read and drive.

The line is not as clean as that — and the previous section is why. Currency conversion depends on when you ask. Timezone conversion depends on historical rules that governments change and that somebody has to keep current. Which gallon you meant is a convention with two answers. What the engine owns is not the set of questions with one true answer. It is the set where the answer belongs to an authority outside the system — and part of the engine's job there is saying which authority it used.

Under that rule about half of what had looked like coverage gaps were not gaps. They were questions correctly routed away from the engine.

Showing the working

A calculator that prints only the answer is asking to be trusted. That is a lot to ask from a system with a language model in it, and it turns out to be a lot to ask from the deterministic half too, because a great deal happens between what is typed and what is computed. 50 mph in km/h becomes 50 mi/h to km/h. $10M becomes USD * (10 * 1000000). 50 percent of 200 becomes 200 * (50/100).

All of that rewriting was invisible. When it was right nobody could confirm it, and when it was wrong nobody could see why. 30 m is read as thirty million rather than thirty metres — and short of disbelieving the answer, there was no way to find that out.

every conversion now carries its derivation

100 Mbps in MB/s                12.5 MB / s
  read as  100 Mb/s → MB/s
  1 Mb/s   = 0.125 MB / s

1250 + 18%                      1,475
  18% of 1,250  = 225
  1,250 + 225   = 1,475

2 + 2                           4
  (nothing. an explanation nobody needs is how explanations stop being read)

This has a near neighbour in the literature. Grounded abstraction matching [3] translates generated code back into predictable natural language so that end-user programmers learn what phrasings the model can act on, and a study of twenty-four participants found it improved their grasp of the model's scope. That runs from code toward language, to teach input. Ours runs from input toward the expression that was evaluated, to justify output. Both were arrived at independently, which is probably some evidence that the instinct behind them is sound.

We had not set out to build somewhere to put ambiguity notices. The machinery for explaining conversions turned out to be exactly where the gallon assumption belonged — and we only realised that after writing it.

What we do not know

This essay reports what building the system taught us. It is not a report that the system works, and the distinction is worth being blunt about.

Eleven people used the production deployment in the ten days we examined. When we read the telemetry properly, most events in the project came from our own preview deployments and development machines — sixty-three of a hundred and seventy-eight — and every parse failure recorded on production traced back to a single session by the developer.

There is no user study, no comparison condition and no task-performance data. One system, one developer, no external validity. The design claims here are arguments from construction rather than findings.

The measurements we do stand behind are small and specific, and worth stating with their method attached. The note-against-question rates come from twenty examples we wrote and labelled ourselves before building the marker, not from user data. The cascade of three unsupported lines into five blank ones was one real sheet from one real request. The 110 expressions were our own list. The telemetry covers a ten-day window, and development traffic was separated from production by hostname, which is how the sixty-three were identified. Parse failure throughout means an engine result beginning with the word Error, which is the definition that missed every blank. These are facts about a system rather than about the people using it.

The study that would test the central claim is not hard to describe. Give people repeated-parameter tasks — the same calculation with changing inputs, which is the case the whole design is built for. Compare a condition where the assistant returns an answer against one where it returns a drivable function, and measure re-use, error detection, and whether stated confidence tracks actual correctness. Our prediction is that the answer condition wins the first task and loses every one after it, and that the function condition produces better-calibrated trust because the working is inspectable. We have not run it.

A good deal of this may also be an artifact of one domain. Calculation is unusually forgiving. The target language is small, the semantics are total and correctness is decidable. Whether the two-author problem looks the same where the generated program has side effects is open, and probably harder — there the cost of widening a grammar is not just a parse.

The inversion

The prevailing engineering posture toward generated code is defensive. Assume the model will produce something malformed and constrain it until it cannot. That posture is correct when the program is scaffolding, because then nothing much is lost by narrowing what the model may say. The answer either comes out right or it does not.

When the program is what the user keeps, the calculus reverses. The language now has to be legible to a person and editable by one, it has to mean the same thing wherever it is opened, and it has to accept what both of its authors naturally write. Narrowing it to whatever the parser already handles optimises for the wrong thing. It produces artifacts that parse and read worse, and it hides the places where the language is too small behind the very mechanism installed to keep things safe.

A parse failure is a disagreement between two authors about what the language should contain. Reading it first as evidence that the model erred, rather than as evidence of a mismatch, discards half the signal before anyone has looked at it.

Every design position in this essay descends from the same source. The offer instead of the warning, the reported assumption instead of the locale guess, the visible derivation, the prompt treated as part of the public interface: all of it follows from the program being something somebody keeps, rather than from anything about how language models behave. Stated once: when a model writes a program a person is going to keep, the language stops being an implementation detail and becomes the interface the two of them share — and it has to be designed for reading and editing as well as for running.

There is a less comfortable claim underneath that, and the evidence for it is that I kept missing the same thing.

Three times in six weeks the engine grew a capability the prompt was never told about, and three times the model filled the gap with something plausible instead of leaving it blank. Dates, and it counted weeks into days. Times of day, and a clock became a duration since midnight. Timezones, and it invented an offset that is wrong for most of the year. Each was found by accident — a user's report, a probe written for something else, a question asked in passing. None was found by the tests, which passed throughout, because in every case the sheet computed.

I was looking for this. I had written the earlier sections of this essay before the last two happened, and it still took a stranger's question to surface the third. That is what makes me think it is structural rather than careless: a language with two authors has a seam down the middle of it, and only one author can read the documentation. The seam does not announce itself. It shows up as a confident answer that is quietly an hour out.

So the practical form of everything above is smaller than the argument that produced it. When you extend the language, the extension is not shipped until the model has been told — and the way you find out you forgot is not a failing test. It is somebody asking a question you did not think to ask yourself.

References

1. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., Neubig, G. PAL: Program-aided Language Models. arXiv:2211.10435, 2022.

2. Chen, W., Ma, X., Wang, X., Cohen, W. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. 2022.

3. Liu, M. X., Sarkar, A., Negreanu, C., Zorn, B., Williams, J., Toronto, N., Gordon, A. D. "What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models. CHI 2023. arXiv:2304.06597.

4. Soulver. SoulverCore and the notepad-calculator interface, 2005 onwards. The answer-column interface this system builds on, and prior art with no academic literature attached.