What Is Plain Text Anyway?

Introduction

If you've spent enough time with software developers, you've inevitably heard someone talk about the virtues of Plan Text. If you're a little more buzzword-compliant, it may not have been phrased as that but rather as x As Code e.g. Infrastructure As Code. Technically these are different, but there's enough overlap that I feel comfortable talking with them in one post. So what are they? What makes something plain text? If this plain text is so great, why isn't every format a plain text format? If you only want the shortest answer I can give: plain text is a spectrum which hinges on how effectively tools which are not designed for the format can manipulate it. We're going to have plenty of examples all over.

What Is And Isn't Plain Text

Much of the time if you ask someone about plain text and as a format and why, you'll get references to things like pipes, the UNIX Philosophy, format preservation, all of which are relevant but none of which I feel capture the full spectrum. I actually asked in a Philadelphia Linux Group and received an answer which I think is worth repeating here not to make fun of him for being wrong or anything, but because I think it is a really good rule of thumb answer. But first, we have to determine what a file is.

A file, for our purposes here, is a series of bytes. A byte is simply a number from 0 to 255. We as humans chose this to be the basis of how our computers work, but it wasn't always like that. There are some computers where the number is from 0 to about 64,000. There are some where the number is about from 0 to 1 million. Additionally we can put more meanings on it: we can say that instead of from 0 to 255, that byte is from -128 to 127, and there's a clever way to do this so that you get the right answer with these negative numbers by doing the same math you would with that 0 to 255 range. The important part here is that a file is made up of these numbers and we impose meaning on them.

When we first made computers we only needed them to work with numbers, so this was fine. As we used computers for more things we found ourself having to represent more complicated stuff, like text. In order to do this, we had to agree on how some body of text was going to be represented in terms of bytes. There were a lot of solutions but for a long time the one which won out in the English speaking world was ASCII, the American Standard Code for Information Interchange. ASCII defined meaning assigned to numbers from 0 to 127, so it fit in a byte. It gave us all of the digits, uppercase letters, lowercase letters, most of the punctuation which you would need, but it included control points too. These control points included things which mattered to teletypes like carriage return, which sent the paper back to the start of the line, and some specifically for computers, such as end of file. The definition I was given earlier, by the way, was that a plain text file included only the characters that you can type with a keyboard. This isn't exactly a bad working definition, it mostly lines up with the ASCII printable characters, plus space, newline, and tab.

There are two places where that definition has problems, though less so now than before. The first is that some of those control codes were used in different ways by different computers. Line endings are the easiest to pick on: if you've ever opened a .txt file on Windows and seen that the whole thing is on 1 line, it was probably saved on something which isn't Windows. Windows uses a carriage feed then new line to represent the end of a line, Linux and current Macs use just new line, old Macs (until the 2000's) used carriage return by itself. Now pretty much everything can work with just new line, though. The second is that there are languages other than English and ASCII doesn't cover those languages, so they had to come up with different ways of representing their own languages. Many of them were based on ASCII, not all of them were exactly compatible. If you've ever seen a Japanese Windows PC and seen that folders were separated by the yen symbol instead of \, that's why: in the old Japanese encoding, the yen symbol used the same number which was shown as \. This has mostly been solved by Unicode, namely UTF-8 which is near-ubiquitous now. UTF-8 is a mapping of characters to numbers much like ASCII (in fact, every ASCII character is the same number in UTF-8,) except UTF-8 can also represent all of those other languages.

Those two break how we defined plain text before though, even with the fixes: what I can type on my US keyboard isn't the same as what you can type on your German or French keyboard. We can also represent any arbitrary data as a series of printable characters. One such example is base64 encoding. Beyond those doesn't tell us why plain text is useful. In my opinion a plain text format is defined by how well it behaves with tools that don't know anything about the format, just that a file is a series of bytes. In understandable terms: if you can open the file in Notepad, make changes, and save it by hand, it's probably plain text. If you tried to do it with, say, a .docx file, all you would see is garbage.

Why Plain Text Is Good

So why should you care about whether or not you can open a file in notepad, and where does X As Code come in? The answer is that software developers have gotten very good at writing tools which deal with these files. To pick on the Linux or Mac toolbox, I have tools like wc which will give me a word count of a file, sed which is like a super-powered search and replace tool, awk which is more powerful than even that. These will work on any file you pick.

Important to your engineers are tools like diff which will let you see what the differences between two files and git which stores files and all of their versions. That is why you want your infrastructure as code: you can see what changed, when, why, how often it's changed, all without specialized tools. You can create patches to share those changes with others, let them review the changes and let them change their version to match yours. It would be impractical to create specialized tools for every file format do these things, and even if we did it's unlikely that they would interoperate correctly. Plain text is best thought of as an engineering constraint: by limiting how my file format works to what these tools are good at doing, I gain access to all of these tools for free just like you can copy/paste, rename, and move any file, text or not. If you're just here because you want to know why the software developers in your life are obsessed with doing things As Code, you can stop reading now if you like. From now on we're going into format design.

Why Plain Text Is Bad

So if plain text is so great, why do we even have anything else? There are actually quite a few reasons. The first, and I think is most obvious, is size and space. You can define an image in text. We could say that there's an image format which only supports black and white pixels in plain text like this: it starts with a line with the image width and height in ascii separate by a space, then below that line are height lines of length width consisting of a 1 for white and a 0 for black. A smiley face might look like this:

But that's a lot of text for such a small smiley face. Even worse, what happens if we add color? Even worse, how would we define a sound in text? Some kinds of data are simply awkward to represent in text in a way that's easy to manipulate, which is the entire point of representing them in text. Another example here is a zip file: the whole point of compression is to make the file as small as possible, so compression into plain text is counterproductive outside of a specific set of circumstances (mostly e-mail, these days.)

Another reason to not use plain text files is because of the parsing involved. Because the point of a plain text file is that it can be manipulated by hand, if you're using plain text then you inevitably need to be a little more forgiving of what you accept. Some text editors add an extra blank line at the end of the file for example, and you need to deal with that. Maybe your program would always write out a decimal number to 8 places, but the person editing by hand got lazy and stopped at 5.3. Maybe for performance reasons you can't afford any transformations on the data at all and the file you need to read is exactly how it's represented internally in the program. By letting any tool manipulate the data, you have to accept that it may not be exactly how you left it.

The final reason is, quite frankly, lock-in. The whole point of a plain text format is that it's easy to understand and manipulate. If your product uses a binary format and your competitor wants compatible with Contiki Office Suite as a checkbox on their next release then they have to put a lot more work into reverse engineering Contiki's file format. That format might even be intentionally obfuscated to make this harder and buy you more time.

Like any choice there are a few tradeoffs. While a lot of file formats can sensibly fit into the constraints needed, when you can't meet those constraints at all it's luckily fairly obvious.

The Spectrum Of Plain Text

When I was first defining plain text I referred to it as a spectrum, what do I mean by that? I'm going to pick on a few famous file formats which I feel go from most to least plain text-ey and explain why in order for you to get a feeling for it.

Text

The good old .txt file. It has no structure at all, it's just as series of whatever bytes you turn it into. In some sense this is the most useful format with these tools, but it can also be a little harder to work with because there's no structure at all. A poorly planned search+replace could end up replacing much more than you expect. I can't think of what a specialized tool to view a plain text document would even be.

CSV

The venerable CSV file is probably the oldest database format you'll encounter in the wild. It is a series of lines, each of which consists of a series of columns, each of which is separated by a comma. Very easy to work with, the only real gotcha is the need to properly handle escaping (we'll get to this in the next section,) you can easily and freely reorder lines, and tons of tools either support it explicitly or implicitly. The only reason I reach for a specialized tool (I recommend csvq, for what it's worth,) when dealing with CSV is usually because the file is in the thousands of lines or bigger and is hard to manipulate in my head.

JSON

JSON is a good middle ground format to me. While it's relatively easily manipulated and written by hand in many aspects, there are a couple papercuts: mostly that commas are an explicit object separator and the nested nature mean that more often than not when dealing with JSON I reach for a dedicated tool. The best I've found there is jq or I write a short script using a Python or Perl JSON library. Sometimes I use a generic tool, but when using those I often find it easier to get the solution almost there and correct it by hand. In my opinion the slight loosening of requirements in JSON6 move that particular format close to the plan text end of the spectrum.

YAML

To keep it short and sweet: YAML combines most of the problems with JSON with a bunch more features which make it very hard to manipulate without dedicated tooling, and the YAML spec itself is so complicated that even that dedicated tooling often fails to consistently and correctly work with YAML between implementations.

ISO9660

You can consider this a lot of binary files which can contain plain text data. You probably won't manipulate it with plain text tools, but you can still search for text inside of them and maybe manipulate that specific text. It may be hard to add to what's there due to file offets or size limitations, but seeing if a substring is inside of the file or changing the contents of that string without changing the length is on the table.

Zip

The worst possible case for generic tools. Not only can you not manipulate it, it's often addressed by offsets, and because the contents are compressed you can't even meaningfully search for some text within the file itself. You really can't do anything to this (or any similar compressed file,) without specialized tooling.

Designing Plain Text Formats

Now that we have a good understanding of these formats, a few tips if you're designing one yourself.

First, consider escaping. If it's possible to represent anything meaningful to the format in the data itself, you need a good way to escape it. The traditional way to do this is with a backslash \, so try to follow the principle of least surprise here. Of particular note of how poorly HTML does it: < is, in my opinion, an abomination. I'm not sure why it couldn't have just been \<, I'm sure there's some good historical reason though. Second, think very carefully about your delimiters. CSV is very convenient because it's well delimited, and also has 2 very strong and distinct delimiters (newline and comma.)

Don't rely on non-newline whitespace. It's hard to hand-write and hard to debug. It's very easy to accidentally insert or delete a space somewhere and not notice it, especially on large screens.

Make it easy to re-order within a conceptual block. The fact that a comma on the last key/value pair in JSON is a syntax error makes it annoying to script around. I often just always generate with the trailing comma and then fix it up manually after. Avoid nesting if possible: often you'd like to work with regular languages, which make it impossible to do things like paren/brace matching in a guaranteed way. If you can get away with this.key.nested.name instead of {this { key: { nested: { name: "value" }}}} your users will have an easier time. Finally: try to be line oriented. Most generic tooling does rely on the concept of lines as a separator. Use this to your advantage and try to avoid making programs working on your format look beyond the current line for context to do what they need to do.

Conclusion

Alright, it's 2 AM now and I've been thinking about this for a few hours. Hopefully this gives you a better grasp on some file format design, history, and pros/cons.