GSoC 2026: LLM Based Template Logic Executor

Author: Devanshi Chhatbar, a contributor to Google Summer of Code 2026
Project: LLM Based Template Logic Executor
Org: Accord Project · Repo: template-engine · Mentors: Diana Lease, Daniel Selman

If you’d rather watch than read, here’s a walkthrough demo video.

The problem I set out to solve

An Accord Project template pairs legal text with a Concerto data model and, optionally, compiled TypeScript logic for its init and trigger operations. That logic lives in logic/logic.ts inside the template archive, and TemplateArchiveProcessor compiles and runs it whenever a request comes in.

“Optionally” is the key word there. Plenty of templates in the Cicero Template Library, and plenty that a user drafts while just exploring an idea, have no logic file at all. Before this project, that meant one thing:

trigger() throws: “No executable logic found.”

The GSoC idea (see the Ideas List 2026) framed the fix simply: build a generic executor in TypeScript that delegates to a reasoning LLM whenever a template has no explicit logic, wired into init and trigger on TemplateArchiveProcessor, with configuration for which provider to call and which keys to use.

The hard part wasn’t “call an LLM.” It was making the LLM’s output trustworthy enough to slot into a pipeline that otherwise runs compiled, statically typed TypeScript:

• The model can’t see the whole ModelManager. A template’s model is just one namespace among many that could be loaded into the same manager, so handing over everything would be wasteful at best and a context-window problem at worst.
• Whatever the model returns has to satisfy the exact runtime shape the template declares (its request, response, state, and event types), not just something that looks plausible.
• Templates don’t name these types consistently. A response might be called PayOut, Payout, or something with no obvious relation to “response” at all. State might extend the runtime State base, or it might just be a bare concept FooState. Matching by name was never going to work.

The goal: run init and trigger through an LLM as a drop-in alternative to compiled logic, constrained tightly enough by the template’s own model that the output ends up exactly as trustworthy as the type system Concerto already enforces everywhere else in the pipeline.

How I approached it

The core idea

The executor really comes down to one idea, worked all the way through: give the model only this template’s types, as a strict JSON Schema that the provider itself enforces. Everything else (detection, tree-shaking, schema construction, post-processing) exists to make that one sentence true.

Finding the right types without guessing names

The executor resolves a template’s runtime types from the model itself, never by matching type names:

All four come back as lists, since a template can declare more than one of each. A template with no state type found is treated as stateless: init returns {} and trigger simply omits state. Stateful templates, on the other hand, require prior state to be passed in so the executor can maintain state history.

Trimming the model down to size

Every type is a vertex, and every field, supertype, relationship, map key or value, and decorator reference is an edge. Tree-shaking keeps only what’s reachable from the root types:

includeDerivedTypes: true adds reverse supertype → subtype edges, so keeping an abstract base also keeps its concrete subtypes. The reduced .cto files get collected alongside the schema and sent as prompt context, so the model never sees the whole model, only the slice it actually needs. The graph-and-filter approach here follows the same pattern @accordproject/concerto-codegen already uses internally.

Turning that into a schema providers can actually use

JSONSchemaVisitor output uses $refs and carries some Concerto-specific keywords that strict structured-output APIs reject outright. Three passes turn each root type into something a provider’s strict mode will accept:

When a template declares multiple responses or events, the types combine with anyOf (resolveUnionSchema), so the model can return whichever one actually fits the incoming request.

What init and trigger actually return

Both schemas are built once in the constructor and kept on the instance, so the object reference stays stable across calls. That matters for providers, Anthropic in particular, that cache the grammar derived from a schema.

How a call actually flows through the executor

1. buildSharedContext() pulls together the template’s name and version, the contract text, the template model’s fully qualified name, the declared type names, and, on the schema-less path, the resolved definitions.
2. ask() calls reasoner.complete(messages, schema), retrying on failure.
3. extractJson parses the reply, tolerating a ```json fence around it.
4. assertInitShape and assertTriggerShape do structural validation on the parsed reply.
5. injectRuntimeMetadata stamps the fields the model should never be inventing itself: $timestamp on the result and each event, and $identifier on state. This mirrors exactly what the compiled TypeScript path produces, so nothing downstream can tell which executor actually ran.

Which providers it talks to

createReasoner(config) switches on config.provider and lazily loads each SDK the first time it’s needed:

When a provider supports strict structured output, the full resolved schema goes out on the wire and the provider enforces it directly. When it doesn’t, the wire schema becomes an open envelope, and the resolved definitions are instead handed to the model as context.schema inside the prompt. That’s a best-effort fallback, but it still works reasonably well in practice.

Getting started

mode decides the routing. disabled never touches the LLM at all, and throws if the template has no logic. fallback prefers compiled TypeScript and only reaches for the LLM when template.hasLogic() is false. force always uses the LLM, which is also the mode the test suite uses to run A/B comparisons against compiled logic. Either way, outputs are validated by Concerto (Serializer.fromJSON) and checked against the runtime class hierarchy afterward, regardless of which path produced them. The LLM doesn’t get a free pass on correctness.

Cleaning up: obligations and the contract back reference

Once the executor was actually live, real templates surfaced a gap: any event extending org.accordproject.runtime.Obligation (payment obligations, penalty clauses, and so on) requires a contract relationship back reference, and neither compiled template logic nor the LLM should really have to set that by hand. TemplateArchiveProcessor.populateObligationBackReferences now fills it in after execution: for each returned event that extends Obligation and doesn’t already carry a contract, it resolves the identifier off the template’s own data instance (which must itself extend Contract) and assigns the relationship string. If something’s wrong, say there’s no resolvable identifier, or the data model doesn’t extend Contract, it throws a specific, actionable error instead of silently producing an invalid event.

The same change also renamed trigger’s state parameter to priorState throughout the executor and the archive processor, and made the requirement explicit: stateful templates must always be seeded with the state returned by a previous init() or trigger() call. There’s no implicit empty state for a template that declares custom state fields.

How I tested LLM output against compiled logic

I added a Cucumber.js suite (test/llm_executor/) that runs a template through both executors, the TypeScript one (mode: "disabled") and the LLM one (mode: "force"), and asks an LLM judge whether the two outputs are semantically equivalent.

There are two flavors, matching the two operations:

• Stateless: draft() followed by one trigger().
• Stateful: init() followed by an ordered sequence of trigger() calls, replayed independently against each executor so the two never share state.

The step definitions themselves are generic. Everything template-specific (which template, which fixtures, which provider) lives in .feature files and config.js. Execution providers (EXEC_PROVIDERS) and judge providers (JUDGE_PROVIDERS) are configured once and keyed by name (anthropic, google, mistral, openai). Feature files pick a provider by name but never hardcode a model. A scenario names its template by label in the Given step, and that label gets joined onto TEMPLATE_DIR with no fallbacks, so a single run can cover templates living anywhere on disk.

One deliberate quirk: the stateless and stateful judge steps are worded differently (“should find the two outputs equivalent” versus “output sequences equivalent”), because Cucumber loads both step files in the same run, and identical step text would be ambiguous between them.

Each run writes both executors’ outputs under the template’s own directory (responses/disabled/logic_output.json, responses/force/<provider>-output.json, with a suffix when a scenario uses non-default fixtures). The judge’s verdict, whether the outputs are equivalent, how confident it is, its reasoning, and any differences found, gets attached to the generated reports/cucumber-report.html. Worked examples ship for copyright-license (stateless) and perishable-goods (stateful).

Live in the Template Playground

The executor isn’t just a library API. It’s wired into the Accord Project Template Playground itself, so anyone can point a template at an LLM and watch init and trigger run without touching any code.

The Request panel gets a mode toggle: Disabled, Fallback, or Force. Init Contract and Send Request drive the same init()/trigger() lifecycle described above. For stateful templates, an Execution chain view lists every call made in the current session, so a multi-step, stateful sequence stays inspectable as it grows instead of only showing the latest result.

You can try it yourself with any of the shipped Cicero Template Library samples (Perishable Goods is a good one) at playground.accordproject.org.

The pull requests

On accordproject/template-engine, by devanshi00:

PR #123 landed the executor itself: type detection, tree-shaking, schema construction, the executor lifecycle, and multi-provider support. PR #171 fixed the obligation back-reference gap described above and tightened the priorState contract for stateful templates. PR #174 added the Cucumber-based test framework (test/llm_executor/), with feature files covering stateful and stateless templates, including a judge harness for comparing LLM output against compiled TypeScript logic on templates like copyright-license, full-payment-upon-demand, perishable-goods, and safte.

On accordproject/cicero-template-library, also by devanshi00, bringing the shared template fixtures in line with what the new executor and its test harness needed:

PR #520 bumped the library onto cicero-core@2.0.0, added init/trigger tests across templates, and fixed the fallout, including rewriting enum-like status objects (InspectionStatus) as const maps typed off the generated interface instead of importing a type the new codegen no longer emits. PR #524 fixed validator errors that only surfaced once templates started running through both the compiled and LLM paths for real: correcting a stale $class version on a request fixture, filling in a description field the rental-deposit logic had been leaving blank, and switching servicelevelagreement’s serviceCredit1/serviceCredit2 and monthlyCredit from bare numbers to proper MonetaryAmounts (with matching test updates). PR #527 is the fixture-side counterpart to PR #171 on template-engine: templates emitting payment or penalty-style events now extend Obligation in their model.cto, their template models extend Contract rather than Clause, and sample data was updated to match (clauseId became contractId), so the obligation back-reference logic actually has something to resolve against.

What I learned

• Building something that sits between an LLM and a strongly typed pipeline meant spending far more time on the tree-shaking and schema-strictness design than on the prompt itself. The constraints do more of the work than the wording ever could.
• Real edge cases only showed up once actual cicero-template-library templates were run through both code paths side by side. It was genuinely interesting to find bugs like this in real template code, not just synthetic test cases.
• Working on the Template Playground UI mattered just as much as the library itself, since it’s what lets other Accord Project users actually reach for this capability.

Thanks

Thank you to my mentors, Diana Lease and Daniel Selman, for their guidance throughout the program, and to the wider Accord Project community for reviewing PRs and answering questions along the way. And thank you to Google for running Google Summer of Code and making this possible.