Some years ago I worked for a company making an analytical SQL Database system. At some point I was asked to think about possible new storage method that would be optimized for high speed SSD storage. In the end, this project wasn't further pursued as other projects took precedence, but ever since I have had this weird idea of log structured btree storage in my head.

A few years later, when I got more serious about actually implementing this wild distributed file system idea of mine, I needed a suitable key/value store library that could use an append only object store as underlying storage. Unfortunately, none of the existing golang key/value store libraries offered this capability. After some careful thinking I realized that the Log Structured BTree idea, although originally coming from an SSD flash storage context, is in fact perfectly suited for this particular use case. So I started writing a Log Structured BTree key/value store library, LSB.

It took some time, but I got something working. But then, when I started thinking about how exactly I would use it to implement the logical equivalent of the inode table and directories in my file system, I realized I was going to need some kind of transaction support in the library to avoid excessive complexity in the file system code. This meant that I now had to implement some kind of transaction support. That took a bit of time, and when I had it implemented, I found …

more ...