Snapshot captures — overusing snapshot testing for realistic tests and joyful experiences

The following post reflects on a personal pattern for testing I have been using for quite a while, which I call snapshot captures. It takes roots on snapshot testing (a technique I won’t cover here — I’d advise reading first matklad‘s essay on it: Snapshot Testing For the Masses if you’re not familiar with it) and combines it with a REPL-fashioned development workflow, as the one seen on My Kind of REPL.

In a nutshell: snapshot captures caches computations that occur within monadic contexts as plain data; thus ossifying the values in the codebase, free of their original context.

Consider an HTTP API, improperly documented, requiring authentication in the form of an API key, and lacking a native client for our language of choice. This is a common scenario, indeed: our ventures into uncharted digital territory rapidly earn us a fair amount of insight into how the service works — and with enough time we might end up implementing a decent client abstraction in our language.

So, let’s assume that we designed the client for the service Spirhalite¹ as follows:

type SalmonidId = int

class Salmonid(pydantic.BaseModel): ...

type Salmonids = Sequence[Salmonid]

class Endpoints:
    salmonids: Final[UrlPath] = UrlPath("/v3/salmonids")

@dataclass
class SpirhaliteClient:
    _client: HttpClient # httpx(2), requests, whatever works
    api_key: SecretStr
    base_url: HttpUrl

    def salmonids(self) -> Result[Salmonids, HttpError]: ... 
    def salmonid(self, id: SalmonidId) -> Result[Salmonid, HttpError]: ...        

Once I have such an abstraction, I usually slap a quick test anywhere that constructs an element of that client with a hardcoded key² and I simply play around with it. The test is called def test_repl() and it runs on save. Every time I make a change, I immediately observe the results. «Oh, so the identifier for the salmonids is a UUID, not an integer, that’s why it is crashing» I find out; hence I change the type alias SalmonidId from an integer to a UUID, throw a pprint(salmonids) in there, and study how those lil’ troublemaker small fry are behaving.

This —making a hypothesis, running an experiment and getting back its results in the blink of a keystroke— is a powerful concept. Creators need an immediate connection to what they are creating. Endless hours of careful poring over outdated documentation won’t teach us the fascinating (and frustrating!) nuances of the real world, and so this playful experimentation ends up becoming our path to success rather quickly — putting bread on the table and joy in our minds.

We’re satisfied, yes: «immaculate implementation», we think, «this thing slaps, ain’t no way those mischief-makers break my deserializer — oh, I know, why don’t we throw in some tests to show precisely that?»

Writing those examples is a good call, indeed: the most dangerous systems are unverified ones that appear correct at first glance, and these small programs nudge us closer to correctness³. It is also incredibly helpful for our colleagues⁴ since we are bringing them up to speed on how the service behaves, and what we can expect from it — the test suite effectively becomes a literate program on Spirhalite. In general, programming is theory building, as Peter Naur states in Programming as Theory Building: insight is both the motivating factor and the artifact of software development.

The question is, then: how do we test our client?

We cannot rely on querying the API during testing in most scenarios for a plethora of reasons —restricted quotas, environments with poor or no connectivity at all; secret sharing across team members and services alike— and the infrastructure required to tackle those problems at their root is remarkable overkill for a mere hey, does my type constructor behave well for data coming out of the API? Nonetheless we are certain those tests are worth having.

The first thought is: we mock it. I will confess: I don’t like mocks! They cannot effectively reflect idiosyncrasies of real services; maintaining those mocks forces us to partially reimplement their inner workings. What we’re truly looking for is the shape of the answer — we went to Spirhalite because we wanted to fish those salmonids from out there. Managing keys, dealing with the fallible nature of network connections — all that jazz is incidental to our intentions.

It follows, then, that the one thing worth counterfeiting is the value contained within the body of the HTTP response:

{
  "data": [
    {
      "id": "314dd98d-f586-4804-aa58-593cebe07146",
      "name": "Chumaco",
      // ...
    },
    // ...
  ]
}

In operational terms this isn’t far off from the classic conception of a mock — we’re intercepting a request to a server and responding with predefined data. The key differentiation is that the data is real; a measurement of information which we’re certain has been observed in the system.

In essence, what we have done during the repl session is to crystallize the result of an effectful computation —that is, non-deterministic⁵ and with possible side effects⁶—, thus making our tests pure.

Looking back at our workflow we observe that:

1. We set up the context required to achieve something (API keys, instantiating our client, making sure we reach the server), a procedure that had to be babysat.
2. We saw both the outputs of the computation and its side effects, transforming our type system and assumptions along the way.
3. With the data stripped off its context, we were left with the crux of the system — and so our tests became pure functions of data.

This pattern feels compelling. Would we be able to distill it further, thereby removing any sort of manual intervention?

For the following implementation I will use inline-snapshot, but any framework suits the abstraction as long as it has sufficient expressivity (in this case, the ability to read a snapshot value, and to rewrite the source code when it is missing or wrong).

Let’s consider the following function:

from inline_snapshot import Snapshot, get_snapshot_value, snapshot
from collections.abc import Callable
from types import EllipsisType

def capture[U](
    f: Callable[[], U], 
    record: Snapshot = snapshot()
) -> U:
    value = get_snapshot_value(record)
    if type(value) is not EllipsisType:
        return value

    v = f()
    assert v == record
    return v

where

• f is a computation that produces a value of type U, possibly (obviously, given our task at hand) effectful. Mise en place style, f does not need any additional argument whatsoever to be run.
• snapshot() is a function that can rewrite its call site in the source code whenever something is asserted or compared to it⁷.
• EllipsisType, a type representing a missing value.

The new function capture will, on first run:

1. Run the effectful operation f⁸.
2. Store the return value in the variable record.
3. Return the pure value.

On subsequent runs of capture, bypass f entirely and instead use the cached value.

In our example, we would use the sphiralite client to query the server and capture the response like:

salmonids: Result[Salmonids, HttpError] = capture(client.salmonids)

If we ever want to rerun f we can delete the value contained within the snapshot, since that forces recomputation on the next run.

           ╭──────────────────────────────╮                     
           │                              │                     
           │       External service       │                     
           │                              │                     
           ╰────▲────────────────────┬────╯                     
                │                    │                          
                │                    │                          
            f() │                    │ {"data:" [...]}          
                │                    │                          
                │                    │                          
           ╭────┼────────────────────▼────╮                     
           │░░░░│░░░░░░░░░░░░░░░░░░░░░░░░░│                     
           │░░░░│░░░░░░░░░░░░░░░░░░░░░░░░░│                     
           │░░░░│░Capturing░function░░░░░░│                     
           │░░░░╞════════════════════╗░░░░│                     
           │░░░░│░░░░░░░░░░░░░░░░░░░░║░░░░│                     
           ╰────┼────────────────────╫────╯                     
                │                    ║                          
                │                    ║                          
f::Unit -> JSON │                    ║  captured = {"data": ...}
                │                    ║                          
                │                    ║                          
           ╭────┴────────────────────▽─────╮                    
           │                               │                    
           │          Our program          │                    
           │                               │                    
           ╰───────────────────────────────╯                    

The idea isn’t precisely revolutionary: it is just a cache! This is a cache! What enamors me about this approach is that

The idea isn’t precisely revolutionary — it is just a cache! Despite that fact, by basing off the underlying mechanism on snapshot testing, a series of patterns and consequences begin riddling the codebase — a mold-like infection that enamors me:

1. The pure data will be represented in the source code. We can see it, modify it, edit it, search it — all of our wonderful plaintext abilities, at our disposal. Nothing is hidden from us.
2. Instead of throwing our mundane, scrappy test_repl workflow out of the window, we augmented its Effective expressiveness by infusing it with correctness — thus turning the scaffolding into a proper test.
3. The real server responds to those queries, and its answers are recorded for posterity. If, for some reason, the API specification changes without prior notice, we could force a recapture locally — then our test suite would exercise our deserializers again and catch any drifts from the original schema.
4. Assuming a sufficiently robust language server and a good name for the capture functions, cached data can be navigated to from any point of our program. Each capture function represents, in a nutshell, some sort of query to the API we had to in the past; those actions hold semantic meaning in our heads even in the present, and naming them is an exasperatingly fantastic way of introspecting our understanding of the world.

Additionally, the approach greatly benefits from some kind of global switch that forces recomputations independently of the state of the snapshot. In my own framework I ended up introducing a recapture parameter to the function, whose value is read from the CLI as a flag and exposed to pytest as a fixture:

def capture[U](
    f: Callable[[], U],
    recapture: bool,
    snapshot: Snapshot = snapshot(),
) -> U:
    value = get_snapshot_value(snapshot)
    if type(value) is not EllipsisType and not recapture:
        return value

    v = f()
    assert v == snapshot
    return v

Then I run pytest --recapture path/to/file.py --inline-snapshot=fix whenever I want to update snapshots. Had I skipped the --inline-snapshot=fix parameter, the assertion would detect changes in the snapshot.

I haven’t had the need to expand it further. It has been just enough to work almost universally during the few weeks where I have been employing it, which is what I’m looking for in a pattern like this. It’s highly likely that the concept already exists in practice and I didn’t know about it — nonetheless, even if that were the case —coming up with such abstraction and writing about it, only to find out that I’ve converged into a well-established and known result—, its existence won’t deprive me of the unparallel joy of learning about it; on the contrary, it would augment it.