Re-measuring simpleconf After a Round of Performance Work

In early August I wrote an introduction to simpleconf, the small Go config service I've used since 2017. That post had a benchmark section and a list of caveats, and the caveat I spent the most words on was durability: a 200 OK meant the write was in memory and queued for disk, nothing more.

From September 20 to 22 I spent three evenings on performance in shaunlee/simpleconf, about twenty commits in total. The README numbers moved a lot. This post puts them next to the ones from the August post and tries to work out how much of the difference is actually the code.

The catch is that the two columns come from different computers.

Two machines

The August numbers were measured on an AMD Ryzen 9 5900HX running Linux. The new ones are from an Apple M6. Anything involving a socket now runs inside a Linux VM (OrbStack): etcd, Consul and Valkey only run there, and loopback on macOS is several times slower than on Linux. The in-memory benchmarks run natively on macOS.

So every ratio in the table above is part code and part hardware, and a new laptop alone would have moved some of them. I had two ways to pull those apart.

The first is that most of the changes were benchmarked on the Ryzen at the time they landed, before and after, and the numbers are in the commit messages. Those comparisons are on one machine.

The second is to find benchmarks whose code didn't change. Whatever speedup they show is the hardware.

Whole-document reads as a hardware baseline

Get("") just returns a snapshot string that's already built. That code path is the same before and after, and it went from 5.71 ns to 3.07 ns. So for in-memory work the M6 is about 1.9x faster than the Ryzen. Dividing that out of the Ryzen numbers from just before the last perf commit:

That's a correction from a single reference point, so treat the last column as rough. The allocation counts are not rough, though, and a CPU doesn't change them: Get went from 2 to 0, Set from 7 to 2.

The other servers as a baseline

The README has a chart comparing simpleconf with etcd, Consul and Valkey on the same host, and I have that chart from both machines. Those three servers didn't change between runs, so their speedup tells you what the environment did.

Over HTTP simpleconf picked up more than Consul or etcd did. On the native protocol it picked up less than Valkey. Whatever the M6 VM does, it helps Valkey more than it helped simpleconf.

That also kills something I said in the last round. On the Ryzen, simpleconf was ahead of Valkey at every depth I measured. On the M6, Valkey is faster at unpipelined GET and SET, and at pipelined SET. simpleconf is still clearly ahead on pipelined reads (16.5M GET/s vs 6.66M), and that's the only place. The README has been updated to say this.

What changed in the code

These are the changes that mattered, each with numbers from a single machine.

Flushing only when the next read would block

The TCP loop used to flush its write buffer after every command. If a client pipelined 64 commands, the server made 64 write syscalls. Profiling showed syscalls at about 76% of CPU on the TCP path.

Now it only flushes when the read buffer has no complete command left:

func readlineFlush(reader *bufio.Reader, writer *bufio.Writer) ([]byte, error) {
	if !hasBufferedLine(reader) {
		if err := writer.Flush(); err != nil {
			return nil, err
		}
	}
	return readline(reader)
}

My first version checked Buffered() > 0, which deadlocks. If the buffer holds half a command, the server goes to read the rest from the socket while holding an unflushed reply, and the client is waiting for that reply before sending more. hasBufferedLine looks for a newline in the buffered bytes instead.

Ryzen, 50 connections, GET:

Depth 1 barely changes, as expected, since a ping-pong client never has a second command waiting. The 16.5M GET/s figure comes from this change. The August post didn't have a pipelined number at all.

Not rebuilding the document on every write

In August I wrote that the document was a single JSON string, that every Set rewrote the whole thing, and that this was the right trade-off for a config store. The profile didn't support that once documents got into the tens of kilobytes.

The fix happened in two commits.

The first kept the string representation but made it mutable. The master copy became a []byte edited in place with sjson's ReplaceInPlace, and readers got a separate immutable snapshot that's rebuilt lazily after a write. On a 23 KB document, on the Ryzen:

The second replaced the string with a tree of ordered objects, arrays and raw scalar leaves. A keyed read or write now costs O(depth) rather than O(byte offset). The clearest way to see the difference is to compare the fastest key with the slowest one in a 21.8 KB document:

This fixed a problem I hadn't mentioned in August. With the string, adding a key near the front of the document made every key after it slower to read and write. Performance depended on key order, which nobody manages on purpose. With the tree, a 60 B document and a 60 KB one cost about the same per keyed operation (11.5 ns vs 11.75 ns for Get).

I didn't rewrite gjson's query syntax for the tree. Paths containing #, * or ? (things like friends.#(age>45)#.first) still go to gjson, run against the snapshot that already exists for whole-document reads. So queries behave exactly as they did before.

Plain paths without allocation

The tree made small documents slower at first. On the Ryzen, Get went from 48 ns and 0 allocations with the string to 69 ns and 2 allocations with the tree, because it split the path and copied the leaf value. The commit message for the tree says as much: 16 to 32% slower for heavily pipelined operations on a 16-byte document.

The last perf commit walks plain paths in place, stores scalar leaves as strings, and parses the client's JSON only once on a set. Get and Del don't allocate anymore. This is where the ~3.5x from the first baseline comes from.

fsync

Not a speed change, but it affects how to read the write numbers in the August post. Back then nothing on the write path ever called fsync. I also wrote that the window for losing a write was "microseconds", which was wrong. Records still waiting in the channel were lost on kill -9 as well, not only on power loss.

There's now a db.fsync option, modelled on Redis:

The writer also collects records into one write syscall per batch. I wrote a test that sends SIGKILL in the middle of a write load and then checks every acknowledged key after restart. It found no losses under any of the three settings.

always is expensive, and how expensive depends mostly on the disk. On ext4/NVMe SET dropped from 237K/s to 19.5K/s. On native APFS it went from 186K/s to 24.6K/s. That's why the default is everysec.

HTTP didn't get faster

HTTP GET at 200 connections went from about 263K/s to 1.02M/s between the two READMEs, which looks like the biggest improvement of all. On the Ryzen, though, it was 263K/s before this work and 237K/s after it, with fsync turned on. That's noise. The HTTP gain appears only on the new machine, and Consul's HTTP throughput doubled on the same machine.

It makes sense when you look at what HTTP costs here. There's no pipelining, so the flush change doesn't apply. A response carries around 105 bytes of status line and headers, compared with 10 bytes for the TCP reply $6\n"mark"\n. And the in-memory operation was never the slow part of an HTTP request. If you only talk to simpleconf over HTTP, this round made it more durable and fixed some bugs, but didn't make it faster.

Costs

• Memory. The tree takes about 10x the size of the document. The string plus its snapshot took about 2x. A 60 KB document now uses around 660 KB. That's fine for config and would not be fine for a 50 MB document. TestFootprint in the repo prints these numbers. I added it after getting the figure wrong in the README twice.
• Snapshot rebuilds. Whole-document reads use a snapshot that has to be rebuilt after each write. When full reads and writes are interleaved, every read pays for a rebuild. In the in-place commit, whole-document reads under concurrent writes dropped from 74.9K/s to 53.9K/s.
• Valkey. On the M6 I can't claim simpleconf is faster than Valkey in general anymore.

Bugs fixed on the way

• sjson's in-place code path won't stringify a value that needs escaping. It returns the document unchanged and doesn't report an error. So when I turned on ReplaceInPlace, any string write containing a quote, a backslash or a non-ASCII byte (all Chinese text, for example) was silently dropped. An equivalence probe that compares every reply byte by byte against the previous build caught it, and the fix went in six minutes after the commit that caused it.
• On the Raft path, integers were decoded through float64, so large IDs could lose precision. Request decoding and log replay now keep json.Number.
• An arr.-1 append wrote an empty value to the AOF instead of the value that was appended.
• The TCP listener only bound IPv4. On machines where localhost resolves to ::1 first, the TCP examples in the README couldn't connect.
• Two deadlocks: one in BenchmarkClone, one on re-Init that hung 4 out of 12 test runs.

Before switching to the tree, I kept the old string implementation in the test suite as a reference. A 90-step script (escapes, non-ASCII, numeric edge cases, array padding, deletes that shift indexes, escaped dots, clone, queries) has to produce byte-identical results from both. End to end, 98 TCP probes and 48 HTTP probes also matched the previous build exactly.

What the August post got wrong

• "There is no fsync anywhere." There is now: db.fsync, defaulting to everysec.
• "The document is a string, not a map." It's a tree now, and write cost doesn't grow with document size.
• "BenchmarkGet reports 0 allocs at ~32 ns." Still 0 allocs, now ~10 ns on the M6. For a few days in between it was 69 ns with 2 allocations.
• The etcd/Consul comparison mixed my laptop numbers with published figures from multi-node cloud setups measured years apart. The README now runs all of them on one host with the same Docker setup and network. HTTP GET: simpleconf 1.02M, Consul 162K, etcd 44K. The same caveats apply: etcd and Consul fsync every write through Raft before replying, and etcd is measured through its HTTP gateway.
• go run ./cmd/bin/main.go is now go run ./cmd/simpleconf, since the packages moved under internal/.

Notes for next time

If I hadn't recorded before/after numbers on the same machine as each change went in, I'd have no way to tell which improvements came from the code and which came from the new laptop. The final README numbers alone can't answer that.

Keeping the old implementation around as a test reference was cheap and caught the worst bug of the round, the one that dropped non-ASCII writes without any error.

And I'm keeping the numbers that got worse in the README: the memory cost, the Valkey results, the flat HTTP. They're as much a part of the result as the improvements.

simpleconf is still a small single-binary config store meant for private networks, with no auth and no TLS. The August post still covers setup and usage. What changed is that keyed operations no longer slow down as the document grows, pipelined TCP is much faster, and you can choose how durable a write is.

• GitHub: shaunlee/simpleconf
• Benchmark details and methodology: README
• Other open-source projects: Projects