How Deep Should You Dig?

When all you have is a shovel...

There are some universal questions that you should ask yourself in just about every situation. Like “Should you strike while the iron is hot, or look before you leap?” or “Should you trust your logic, or your intuition?” or in this case, “How deep should you dig?”

Suppose someone comes to you with a problem. Do they want your help, or do they just want to vent? If they do want help, how much detail do you need? Does it matter why Alice is mad at Bob, or is that a sensitive subject that you should avoid asking about?

Or suppose you’re a manager. Can you trust your team to make good decisions, or do they need stronger, more direct guidance? What about the new hires, do you need to teach them yourself or can you delegate?

These are all various forms of that question: How deep should you dig?

Creating JSON from a Template

I promise this is related.

Suppose you want to upload things via an API which takes JSON like this:

{
    "type"     : "html",
    "date"     : "2021-12-21",
    "title"    : "Foo Bar Baz",
    "slug"     : "foo+bar+baz",
    "author"   : {"full_name": "Allison Bob"},
    "fancy_bs" : {"unavoidable": {"useless": {"legacy": ["stuff"]}}},
    "data"     : "<html>\n<p>Lorem Ipsum..."
}

If you haven’t done this before, you might think to create a JSON template, and populate it with string interpolation.

You could have a Python function like this:

def create_json(template, **kwargs):
    return template % kwargs

And a template like this:

template.json

{
    "type"     : "html",
    "date"     : "%(date)s",
    "title"    : "%(title)s",
    "slug"     : "%(slug)s",
    "author"   : {"full_name": "%(author)s"},
    "fancy_bs" : {"unavoidable": {"useless": {"legacy": ["stuff"]}}},
    "data"     : "%(data)s"
}

And invoke the function with appropriate kwargs like so:

result = create_json(
    template,
    date   = my_date,
    title  = my_title,
    slug   = mkslug(my_title),
    author = my_author,
    data   = my_data
)

Now there’s the small matter of escaping the strings to make them JSON-compatible. Replace \ with \\, and " with \", and newlines with \n, and…

Wait a minute, wait a minute. That doesn’t sound right. We shouldn’t make our own logic to escape our strings like this, it’s messy and easy to do wrong.

The fundamental problem is we’re digging too deep into JSON itself. We’re forcing ourselves into a situation where we need to know the minutae of the JSON standard, at least as far as string-literals go. And this is completely avoidable.

Using the Library

Python already has a JSON library, so let’s leverage that.

We can modify create_json() like so:

def create_json(template, **kwargs):

    # JSON-encode every value
    json_kwargs = {
        k: json.dumps(kwargs[k])
        for k in kwargs
    }

    return template % json_kwargs

And update the template to accept wholesale JSON objects (i.e. remove the quotes around templated values):

template.json

{
    "type"     : "html",
    "date"     : %(date)s,
    "title"    : %(title)s,
    "slug"     : %(slug)s,
    "author"   : {"full_name": %(author)s},
    "fancy_bs" : {"unavoidable": {"useless": {"legacy": ["stuff"]}}},
    "data"     : %(data)s
}

Invocation is the same as before.

This isn’t perfect, but it stops us from mucking with JSON minutae. json.dumps() encodes each value as a JSON object, and Python string formatting does the rest.

Perhaps the ugliest consequence is that template.json is no longer valid JSON itself. This hints at an underlying truth: we’re still digging into JSON a little bit. Not very much – the approach is basically “start with a template, insert valid JSON objects in these places, and the result will be valid JSON as well” – but this may still be more invasive than we want to be.

Eliminating the JSON Template

Here’s an idea: skip writing your own JSON entirely. No hand-crafting the template, no custom scripts. Instead we can offload it all to json.dumps().

This is more in line with what real API clients do.

Here, create_json() contains the entire structure.

def create_json(**kwargs):

    # create structure, populated with `kwargs` as appropriate
    upload_dict = {
        'type'     : 'html',
        'date'     : kwargs['date'],
        'title'    : kwargs['title'],
        'slug'     : kwargs['slug'],
        'author'   : {'full_name': kwargs['author']},
        'fancy_bs' : {'unavoidable': {'useless': {'legacy': ['stuff']}}},
        'data'     : kwargs['data']
    }

    # convert to JSON and return
    return json.dumps(upload_dict)

There is no template.json. The entire data structure is incorporated into create_json(), and json.dumps() creates the desired JSON directly from that.

Eliminating any kind of Template

Unless the API is excessively rigid, create_json() can be optimized further:

def create_json(**kwargs):

    # create structure, populated with `kwargs` as appropriate
    upload_dict = {
        'type'     : 'html',
        'fancy_bs' : {'unavoidable': {'useless': {'legacy': ['stuff']}}},
        **kwargs,
    }

    # author needs to be nested
    if 'author' in upload_dict:
        upload_dict['author'] = {'full_name': upload_dict['author']}

    return json.dumps(upload_dict)

The original template structure largely vanished. In a sense, we’re not digging as deep into the API. The caller of create_json() is now free to add extra kwargs, override the type from html to something else, or even muck with fancy_bs if they really feel like it. They can also omit any kwargs that the template required but the API considers optional.

This new version behaves more like a specialized encoder than a templating system: it takes in a dict, performs minimal modifications to deal with API idiosyncrasy, and converts it to JSON.

There are some downsides here too. By not digging as deep into the API, we may push failures downstream. If the new create_json() is called without a title or data, it will happily create JSON anyway, which the server will reject. Perhaps it would be better to catch such errors client-side rather than sending out bad data.

This can be mitigated by checking the kwargs for missing or bad values – and yes, this involves digging a little deeper into the API again.

What if You Need to Dig Deeper?

We’ve mostly discussed the problems of digging too deep, but that’s not always the situation. Often you do need to dig a little deeper.

Suppose the original create_json() and template.json were in use somewhere, and then later updated to the last example shown. One of your users complains the new code is failing with a TypeError. Turns out they like to set date = datetime.date.today(), which worked just fine with the original code, but json.dumps() doesn’t know what to do with it.

There are a few options here, each involves digging a little deeper into something.

1. Dig Deeper into Python (and dig in your heels)

You can explain that create_json() was only ever meant to handle strings. Other types of objects may occasionally work, but they are not officially supported and never have been.

Plus, one shoud not depend on implicit type conversion to produce the right date format anyway. You helpfully suggest they instead use datetime.date.today().strftime('%Y-%m-%d'), which will always produce a string of the correct date format.

Depending on the situation, such an attitude may not go over well. People don’t like when their workflow breaks, even if it never should have worked in the first place.

2. Dig Deeper into the Past

The original code used Python string formatting, which implicitly converts all kwargs into strings. This is easy enough to replicate in the new code:

def create_json(**kwargs):

    # convert kwargs to strings
    kwargs = {
        k: str(kwargs[k])
        for k in kwargs
    }

    ...

The rest of the function is the same as before.

Note that replicating old behavior may also replicate old bugs. (What happens if data is a bytes-like object instead of a string?)

3. Dig Deeper into the Situation

Let’s be realistic here: this is only a problem with date, and only when it’s a certain type of object. We don’t need to mess with the other kwargs. Instead we can do this:

def create_json(**kwargs):

    # make kwargs['date'] json-friendly
    if hasattr(kwargs['date'], 'strftime'):
        kwargs['date'] = kwargs['date'].strftime('%Y-%m-%d')

    ...

The downside of Situation Digging is that the code becomes more specialized for this one use-case. This is probably okay if you never plan to reuse the code for another project.

Final Thoughts

Digging deeper can mean many things, but it generally boils down to getting involved in the details of something. This post isn’t really about Python or JSON, it’s about asking yourself those important questions:

Do I need to re-invent this, or can I work with what someone else made?
Should I get involved in this, or is it none of my business?
Should we really spend this much time designing the bike shed?

And so on.

Too little digging, and you miss important details. Too much, and you become a micromanager, or a nosy gossiper, or you re-invent the wheel because you just don’t trust the original inventor to have done it right, dang it.

Abstraction enables complex programs to be built without any one person having to dig too deep, or too wide. Similar concepts exist for other complex systems (e.g. cars), large organizations, etc. It is an incredibly useful concept but should not be overused. Sometimes, you do need to dig.

How Deep Should You Dig?

Created:	2021-06-06
Updated:	2025-10-06
Tags:	philosophy, python, json