DSLs Allow Dependable Use of LLMs

The Limits of Upfront Specification

Constructing giant programs includes an excellent many small design choices, and these can not all
be identified upfront or pushed completely from a high-level spec. A specification is at finest a
beginning speculation: the actual constraints, trade-offs, and edge circumstances are found
iteratively, as we proceed with the implementation. We mentioned this at size in an
earlier article, the place we known as it Upfront Specification Impossibility.
The purpose will not be that specs are nugatory, however that the primary one is a speculation to be
revised, by no means a completed blueprint.

The pure response is to iterate: refine the spec, generate code, overview what comes
again,
and feed what we study into the subsequent spherical. That loop works effectively when every spherical produces a
small, reviewable change.

Design Is Found By Implementation

Reviewing code, significantly whereas we’re nonetheless discovering the design,
will not be the identical as writing it.
Whereas reviewing the generated code, we overview via the chunks validating
if it maps to our intent and in search of doable pitfalls.
However reviewing hardly ever forces us to wrestle with the design choices.
Writing code, against this, forces us to assume via concrete choices—equivalent to the place a duty belongs
or what boundaries ought to be uncovered so the design could be prolonged additional.
It’s in making these choices {that a} design most absolutely reveals itself.

The programming language and
paradigm we code in shapes the design perception we get. A purposeful design strategy or an
object-oriented design strategy reveals completely different facets of the design, together with idioms
and patterns which might be pure to the paradigm.

So the place do LLMs slot in?
I see LLMs enjoying two roles.
They’re an excellent assist whereas we form the design and its vocabulary, appearing as brainstorming
companions to assist us discover the design area and uncover the proper abstractions.
As soon as the vocabulary is established, LLMs
work as a wonderful pure language interface to it.

Area Abstractions and DSLs

A helpful method to body that is via Area Pushed Design. Its
core perception is constructing a shared conceptual mannequin of the area in code, after which utilizing that
mannequin – which DDD calls a Ubiquitous Language – each to evolve
the codebase and to provide the group a vocabulary to assume and talk in. Typically, it’s
extremely efficient to construct a website particular language on prime of that mannequin: a constrained syntax for
expressing the area’s ideas and operations. Seen this fashion, most growth is the
technique of constructing a website mannequin and utilizing it to evolve the system. The LLM performs two
distinct roles relying on whether or not the area mannequin already exists.
On this article, I’ll give attention to how Area-Particular Languages (DSLs) work with LLMs.

Why DSLs work so effectively with LLMs

It’s a widespread expertise that DSLs work effectively with LLMs.
PlantUML, Mermaid, and Graphviz are area particular languages for
visible modeling; SQL is a DSL for querying databases; Kubernetes YAML is a DSL for
describing cloud infrastructure.
These should not general-purpose programming languages — they’re
intentionally constrained, designed to specific a slender set of ideas in a single area.
And it’s no shock that LLMs are remarkably good at producing
Mermaid diagrams, SQL queries, or Kubernetes manifests
from a plain English description.

My remark is that DSLs make LLMs extra dependable as a result of they reply so effectively to a
few in-context examples. A general-purpose language like Java presents a lot of legitimate methods
to specific the identical intent. A DSL strips the variation away.
Giving the mannequin a number of examples is sufficient to reliably generate
the right syntax. It is value noting that frontline fashions
are already closely uncovered to PlantUML or Java fluent interfaces
throughout coaching, so they are not ranging from scratch.
It is going to be curious to see how smaller, extra constrained fashions carry out
when tasked with a really novel DSL.

For an agent — an LLM operating in an autonomous generate-and-check loop fairly than
a single shot technology — there’s another profit. A DSL nearly at all times ships with a deterministic
validator: a parser, a JSON schema, a sort checker, or a compiler. The agent can generate a
candidate, run it previous the validator, and restore it from the error, all with out a human in
the loop. Crucially, the errors are phrased on the degree of the area — “you can’t choose
an motion earlier than selecting a consumer” — fairly than as a stack hint buried deep in generated
code. DSL’s toolset itself acts as a wonderful harness. We’ll see it concretely within the
Tickloom examples under, the place the DSL’s grammar is enforced by the host language’s compiler
and the ensuing runs are checked mechanically.

It is very important observe that this isn’t a one-size-fits-all answer.
The benefit holds whereas the DSL stays small and constrained
sufficient that a number of in-context examples can convey its utilization.
There may be additionally an actual upfront value in designing and sustaining the
language and its semantic mannequin. The payoff is subsequently concentrated in well-factored,
genuinely constrained DSLs backed by a validator.

Instance: Utilizing LLMs to generate diagram wealthy powerpoint displays

LLMs make it very easy to construct customized instruments. Whereas instructing
distributed programs, I ceaselessly must create displays which principally have diagrams
explaining distributed operations in a cluster. UML Sequence diagrams have been nice for
that however exhibiting a full sequence diagram whereas explaining the move of messages via the
cluster will not be very helpful. I wanted a software to point out a sequence diagram step-by-step in a
powerpoint presentation. With the assistance of LLMs I used to be in a position to construct a software which processes a
YAML describing the presentation construction with references to the PlantUML diagrams and
generates a powerpoint presentation. The PlantUML diagrams are marked with steps, and the
software generates a separate slide for every step. This made it very easy to create
diagram-rich displays.

Generate a PlantUML sequence diagram exhibiting a cluster of three nodes athens, byzantium
and cyrene. Put a field to mark the cluster. Actor Alice sends a message “title”, “After Daybreak”
to athens, Athens sends message to itself. Put a observe to point out state ‘title: After Daybreak’.
Athens sends message to byzantium which fails. Athens sends to cyrene. Put a observe positioned to
the proper of cyrene. athens then checks with isQuorumReached and returning a synchronous
Success arrow again to Alice.
Put a ‘[step] marker after every message.

This immediate generates the next PlantUML code with step markers:

@startuml
actor Alice

field "Cluster" #lightblue
participant athens
participant byzantium
participant cyrene
finish field

'[step]
Alice -> athens: "title", "After Daybreak"

'[step]
athens -> athens: save()

observe proper of athens
  state:
  title: After Daybreak
finish observe

'[step]
athens -[#red]x byzantium: "title", "After Daybreak"

'[step]
athens -> cyrene: "title", "After Daybreak"

observe proper of cyrene
  state:
  title: After Daybreak
finish observe

'[step]
athens -> athens: isQuorumReached()

'[step]
athens --> Alice: Success

@enduml
          

I used it to create a sequence of slides in a powerpoint presentation.
For doing that, I developed a small YAML specification to explain
the presentation construction and the diagrams for use in every slide.
This allowed me to make use of LLMs to create displays describing complicated
distributed programs ideas, with out having to manually create
animations on the slides. An instance immediate to generate
a slide YAML spec is so simple as following.

Create a slide YAML referring to the diagram ‘quorum-write’ with a title ‘Quorum Write
Instance’

This generates a slide spec YAML like following:

          - slide:
              title: "Quorum Write Instance"
              diagram: "quorum-write"
        

It is essential to notice that even when the immediate is saying create a slide YAML, it isn’t any random YAML spec. As a result of the software
to generate the powerpoint presentation and the YAML specification understood by the software is
used as a context within the immediate, the LLM is ready to generate the right YAML spec which might
be instantly utilized by the software to generate the powerpoint presentation.

The complete YAML spec
could be considered at this
Github repo

Discover that the LLM performed two completely different components on this single instance. First it was a
co-designer — serving to form the step-marked PlantUML extension and the slide YAML on prime of
current PlantUML tooling. Then, as soon as that small DSL existed, it grew to become the natural-language
interface that turns an English request into a legitimate spec. We’ll come again to this
division of labour on the finish of the article.

Constructing the Semantic Mannequin

The instance we lined within the earlier part was simple.
The YAML was used as a provider syntax, and
I course of its parsed syntax tree instantly,
successfully utilizing the syntax tree itself
as my Semantic Mannequin (although this {couples} the syntax to the execution semantics).
However in additional complicated domains, like distributed programs, we want extra complicated semantic fashions
to signify the ideas
within the area and the design choices we have now made within the codebase.
Let’s take a look at an instance based mostly on a small framework I constructed to shortly construct and take a look at
distributed programs.

Instance: Tickloom — a semantic mannequin for distributed programs

Implementing distributed programs equivalent to quorum-based key-value shops or consensus
protocols like Raft and Paxos is a frightening activity.
Even when implementation is guided incrementally via prompts, specs, or fastidiously
constructed .md ability information, the asynchronous runtimes nonetheless expose an awesome
area of doable implementation choices. Threading fashions, networking patterns, storage
coordination, retry habits, and timing semantics all stay entangled inside the generated
code.
The issue will not be merely code technology complexity, however verification complexity.
The ensuing state area created by all doable interleavings throughout thread scheduling,
community delays, course of pauses, and clock skew turns into so giant that systematically
reviewing and validating correctness throughout all interacting behaviors is sort of inconceivable.
That is the rationale we see that Jepsen exams discover bugs even in probably the most battle-tested
distributed programs.

That is precisely the place a semantic mannequin is helpful. Tickloom is a small framework I constructed
to assemble and take a look at distributed algorithms. Its abstractions should not a generic runtime;
they’re a set of design choices about how a distributed course of behaves. Each
node runs in a single-threaded tick loop: every name to tick() advances a
logical clock by one and processes pending work in a set, deterministic order (community,
then message bus, then course of, then storage). Time is measured in ticks, not milliseconds.
Messages are plain Java data. Coordination throughout replicas is expressed via a
Reproduction base class that already is aware of about friends, broadcasts, and quorums.

Threading, timing, community supply are not open inquiries to be
re-decided in each immediate.
What stays for the algorithm writer is the precise protocol logic. A quorum duplicate, for
occasion, is only a set of message handlers expressed within the framework’s vocabulary.

As a result of the framework provides the vocabulary — Reproduction, quorumRequest
,
countResponseIf, MessageType, Handler — a
immediate can keep on the degree of the protocol fairly than the plumbing:

Utilizing the Tickloom Reproduction abstraction, implement a quorum-based key-value retailer.
A consumer GET ought to acquire values from a majority and return the one with the very best timestamp
(last-writer-wins). Apply a write regionally provided that its timestamp is newer than the
saved one.

This excessive degree description produces code like following:

@Override
protected Map<MessageType, Handler> initialiseHandlers() {
    return Map.of(
            LWWMessageType.CLIENT_SET_REQUEST, this::handleClientSetRequest,
            LWWMessageType.CLIENT_GET_REQUEST, this::handleClientGetRequest,
            LWWMessageType.INTERNAL_SET_REQUEST, this::handleInternalSetRequest,
            LWWMessageType.INTERNAL_GET_REQUEST, this::handleInternalGetRequest,
            LWWMessageType.INTERNAL_SET_RESPONSE, this::handleInternalSetResponse,
            LWWMessageType.INTERNAL_GET_RESPONSE, this::handleInternalGetResponse
    );
}

personal void handleClientGetRequest(Message message) {
    var req = deserializePayload(message.payload(), ClientGetRequest.class);
    var internalReq = new InternalGetRequest(req.key());

    this.<InternalGetResponse>quorumRequest(LWWMessageType.INTERNAL_GET_REQUEST, internalReq)
            .countResponseIf(r -> true) // any response is okay, simply want a majority
            .ship()
            .whenComplete((responses, error) -> {
                if (error != null) {
                    ship(createMessage(message.supply(), 
                                       message.correlationId(), 
                                       new ClientGetResponse(req.key(), null, false),
                                       LWWMessageType.CLIENT_GET_RESPONSE));
                    return;
                }

                byte[] highestValue = null;
                lengthy highestTimestamp = -1;

                for (InternalGetResponse r : responses.values()) {
                    if (r.worth() != null && r.timestamp() > highestTimestamp) {
                        highestTimestamp = r.timestamp();
                        highestValue = r.worth();
                    }
                }

                boolean discovered = highestValue != null;
                ship(createMessage(message.supply(), 
                                   message.correlationId(), 
                                   new ClientGetResponse(req.key(), highestValue, discovered),
                                   LWWMessageType.CLIENT_GET_RESPONSE));
            });
}

supply

The semantic mannequin itself acts as a context. The immediate names
ideas that exist as concrete sorts within the codebase, so the LLM will not be inventing a
threading mannequin or a networking layer — it’s filling in protocol logic in opposition to a set,
well-understood substrate.

Even good abstractions assist — with out a DSL

A DSL is one finish of the spectrum, and it isn’t simple to
construct. Earlier than reaching for a language of your personal it’s value noticing {that a} clear set of
abstractions is already a lighter model of the identical thought — and, simply because the framework
provided the vocabulary for the quorum retailer above, a library’s named sorts and strategies are
themselves a vocabulary the mannequin could be grounded in. Tickloom’s semantic mannequin is de facto
simply 4 such seams — Course of/Reproduction for compute and message
dealing with, Community for communication, Storage for persistence, and
a logical Clock for time — and that decomposition does many of the work with no
new syntax in any respect.

For this reason abstractions, not simply DSLs, pair effectively with LLMs.
A immediate to “implement Raft as a Tickloom Reproduction” has restricted state area to discover.
The prevailing QuorumReplica can be utilized in context as a labored instance.

Instance: Constructing a DSL for testing distributed system eventualities

Implementing the algorithm is one factor; exercising it’s one other.
The refined bugs in distributed programs dwell in particular orderings: a write that
replicates to at least one node earlier than a reader’s quorum shifts, a partition that heals at simply the
mistaken second, two coordinators whose clocks have drifted aside. Writing such a state of affairs
instantly in opposition to the testkit includes juggling futures, and
handbook tick() loops. Here’s a clock-skew state of affairs written that means:

Cluster cluster = new Cluster()
        .withProcessIds(Arrays.asList(ATHENS, BYZANTIUM, CYRENE))
        .useSimulatedNetwork()
        .construct(QuorumReplica::new);

cluster.begin();

attempt {
    cluster.tickUntil(cluster::areAllNodesInitialized);

    cluster.setTimeForProcess(ATHENS, 1000L);
    cluster.setTimeForProcess(BYZANTIUM, 2000L);

    QuorumReplicaClient alice = cluster.newClientConnectedTo(ALICE, ATHENS, QuorumReplicaClient::new);
    QuorumReplicaClient bob = cluster.newClientConnectedTo(BOB, BYZANTIUM, QuorumReplicaClient::new);
    QuorumReplicaClient reader = cluster.newClientConnectedTo(READER, ATHENS, QuorumReplicaClient::new);

    TickCompletableFuture<SetResponse> bobWrite = bob.set(KEY.getBytes(StandardCharsets.UTF_8), "B".getBytes(StandardCharsets.UTF_8));
    cluster.tickUntilComplete(bobWrite);

    TickCompletableFuture<SetResponse> aliceWrite = alice.set(KEY.getBytes(StandardCharsets.UTF_8), "A".getBytes(StandardCharsets.UTF_8));
    cluster.tickUntilComplete(aliceWrite);

    TickCompletableFuture<GetResponse> learn = reader.get(KEY.getBytes(StandardCharsets.UTF_8));
    cluster.tickUntilComplete(learn);

    assertEquals("B", new String(learn.getResult().worth(), StandardCharsets.UTF_8));
} lastly {
    cluster.shut();
}

supply

The intent — “Bob writes via Byzantium, Alice writes via Athens, a reader sees Bob’s
worth as a result of Byzantium’s clock was forward” — is buried beneath mechanics.
That is additionally a tough to confirm code: there are dozens of incidental choices (when to tick, methods to encode
bytes, which manufacturing facility overload to name) for an LLM to get subtly mistaken,
and a reviewer should examine each one in all them.

So on prime of the semantic mannequin I constructed an inner DSL whose vocabulary is the
vocabulary of the state of affairs itself — servers, shoppers, who’s related to whom, what every
consumer does, and which faults are in impact whereas it does it. The identical state of affairs turns into:

        State of affairs<QuorumReplicaClient> state of affairs =                        
                        QuorumStepBuilder.state of affairs("LWW misplaced replace by way of server clock skew")
                        .servers(ATHENS, BYZANTIUM, CYRENE)
                        .shoppers(ALICE, BOB, READER)
                        .consumer(ALICE).connectedTo(ATHENS)
                        .consumer(BOB).connectedTo(BYZANTIUM)
                        .given(g -> g.serverTimeAt(ATHENS, 1_000L)
                                     .serverTimeAt(BYZANTIUM, 2_000L))
                        .steps(s -> {
                            s.consumer(BOB).writes(KEY, "B").expectSuccess();
                            s.consumer(ALICE).writes(KEY, "A").expectSuccess();

                            s.consumer(READER).reads(KEY)
                                    .expectResponse(v -> "B".equals(v));
                        });

supply

The DSL is a skinny, declarative floor that compiles all the way down to a pure intermediate
illustration — a State of affairs fabricated from Steps, the place every step carries
an Motion (a learn or write) and elective ClusterEvents (faults
like partitions and message delays). Faults learn as English too:
partition(BYZANTIUM).from(CYRENE), reconnect(BYZANTIUM), delay(INTERNAL_SET_REQUEST).from(ATHENS).to(BYZANTIUM,
CYRENE).byTicks(100). The grammar is enforced by the kind system via progressive
interfaces — you can’t declare a step earlier than the topology, or an motion earlier than choosing a
consumer — so complete courses of malformed eventualities merely don’t compile.
As a result of the DSL is an inner DSL inbuilt Java, the host compiler validates the grammar free of charge, and a malformed
technology comes again as a compile error pinned to precisely the unlawful step fairly than a
runtime shock.

As soon as the DSL exists, a pure
language description of a failure state of affairs maps nearly instantly onto it. A immediate like:

Utilizing the Tickloom state of affairs DSL, write a state of affairs reproducing the DDIA §10.6
non-linearizable quorum learn. A author related to Athens units the important thing, then updates
it whereas replication from Athens to the opposite replicas is delayed. Alice, studying via
Byzantium, is compelled onto a quorum that features Athens and sees the brand new worth; Bob, studying
later via a quorum of Byzantium and Cyrene, nonetheless sees the outdated worth.

yields a state of affairs that stays completely inside the DSL’s constrained vocabulary:

State of affairs<QuorumReplicaClient> state of affairs =
        QuorumStepBuilder.state of affairs("Non-linearizable quorum learn")
                .servers(ATHENS, BYZANTIUM, CYRENE)
                .shoppers(WRITER, ALICE, BOB)
                .consumer(WRITER).connectedTo(ATHENS)
                .consumer(ALICE).connectedTo(BYZANTIUM)
                .consumer(BOB).connectedTo(BYZANTIUM)
                .steps(s -> {
                    // Author units the important thing to VOLD initially and it replicates absolutely
                    s.consumer(WRITER).writes(KEY, VOLD).expectSuccess();

                    // Author updates to VNEW, however replication from Athens is delayed
                    s.consumer(WRITER).writes(KEY, VNEW)
                            .whileClusterEvent(delay(QuorumMessageTypes.INTERNAL_SET_REQUEST)
                                    .from(ATHENS).to(BYZANTIUM, CYRENE).byTicks(100))
                            .expectSuccess();

                    // Alice reads via Byzantium. Pressure quorum to incorporate Athens by partitioning Cyrene.
                    // She is going to learn VNEW from Athens.
                    s.consumer(ALICE).reads(KEY)
                            .whileClusterEvent(partition(BYZANTIUM).from(CYRENE))
                            .expectResponse(v -> VNEW.equals(v));

                    // Bob reads later via Byzantium. Pressure quorum to incorporate Cyrene (and exclude Athens).
                    // The delayed VNEW replication hasn't arrived at Cyrene, so Bob reads VOLD.
                    s.consumer(BOB).reads(KEY)
                            .whileClusterEvent(reconnect(BYZANTIUM))
                            .whileClusterEvent(partition(BYZANTIUM).from(ATHENS))
                            .expectResponse(v -> VOLD.equals(v));
                });

ScenarioResult outcome = state of affairs.run();

supply

As a result of the floor is so small — and the area of legitimate code it could actually generate is a lot
smaller than the area of legitimate Java applications — the LLM has little or no room to hallucinate,
and a reviewer can learn the outcome as an outline of an experiment fairly than as code to
be audited line by line. Even when LLM hallucinates, the interior DSL
will fail to compile permitting LLM to appropriate the errors made.

Two phases working with LLMs

A sample emerges from the examples mentioned above: in each one in all them the LLM was helpful
in two fairly other ways.

The primary section is designing the abstraction or DSL itself. Right here the LLM is finest
handled as a brainstorming companion fairly than a code generator. As argued at the beginning of
this text, the design choices that make up a semantic mannequin can not all be specified
upfront — we uncover the constraints, trade-offs, and edge circumstances as we implement them. So this
section is inherently iterative and feedback-driven: you intend a construction, attempt it in opposition to a
actual case, see the place it seems awkward, and feed what you discovered again into the subsequent
spherical. The LLM speeds that loop up — it sketches alternate options, critiques a design, ports an
thought from one language to a different — however you keep firmly within the driver’s seat, as a result of these
are precisely the selections it’s worthwhile to perceive and personal. The constructions that make a DSL
nice to make use of, like progressive interfaces that make an unlawful state of affairs fail to compile
or a semantic mannequin stored separate from the builder that produces it, are the form of factor
you converge on by iterating, not by writing a specification and producing code.

The second section begins as soon as the abstraction or DSL is in place. Now what the LLM does
adjustments: it turns into a natural-language interface to what you’ve constructed.
The prompts on this article are examples — “implement a quorum retailer as a Tickloom
Reproduction”, “write a state of affairs
reproducing the DDIA §10.6 learn”, “create a slide YAML for this diagram”. In every case the
English description maps nearly instantly onto the vocabulary you outlined, and the LLM is a
reliable generator exactly as a result of the abstraction provides each the context that
grounds the immediate and the harness that checks the outcome.

The DSL because the Supply of Reality

There’s a rising development of treating prompts as the first supply of reality. A well-designed
DSL basically adjustments this dynamic. One of many key benefits I observe of working with DSLs is
that the generated program itself typically turns into the artifact that people preserve. As a result of a
DSL is dense, expressive, and largely freed from incidental boilerplate, it captures the
important intent of the answer in a kind that is still readable lengthy after technology. If
an LLM generates a Tickloom failure state of affairs from a pure language request, the ensuing
state of affairs is already expressed within the vocabulary of the area. If the state of affairs must
change subsequent month, there isn’t any must get well the unique immediate and regenerate
the whole lot. The DSL has sufficient context for LLM to know the intent and work with it. The
enduring asset will not be the immediate, however the DSL and the semantic mannequin.