On File Formats in Operating Systems

On the one hand, experiments have shown that documents in a compressed XML file are usually significantly smaller than the Microsoft Word’s native file format, a binary format that one might imagine would take less space. The reason relates to a fundamental of the Unix philosophy: Do one thing well. Creating a single tool to do the compression job well is more effective than adhoc compression on parts of the file, because the tool can look across all the data and exploit all repetition in the information.

The Art of Unix Programming (The Pros and Cons of File Compression), Eric Raymond

In fact, the Microsoft version of CSV is a textbook example of how not to design a textual file format. Its problems begin with the case in which the separator character (in this case, a comma) is found inside a field. The Unix way would be to simply escape the separator with a backslash, and have a double escape represent a literal backslash. This design gives us a single special case (the escape character) to check for when parsing the file, and only a single action when the escape is found (treat the following character as a literal). The latter conveniently not only handles the separator character, but gives us a way to handle the escape character and newlines for free. CSV, on the other hand, encloses the entire field in double quotes if it contains the separator. If the field contains double quotes, it must also be enclosed in double quotes, and the individual double quotes in the field must themselves be repeated twice to indicate that they don’t end the field.

The Art of Unix Programming (DSV Style), Eric Raymond