Hetzner S3 Is Not Good

So.. I'm a die-hard Hetzner fan. Their VPS and bare metal server prices are awesome, and our past experience has been nothing but positive! Our main website runs on a Hetzner server, and the performance you get is nothing short of amazing - like, an 8-core AMD bare metal server with 64GB RAM, for like.. 60€ / month or so? Even after the recent RAM-driven price increases, the value is insane.

So when Hetzner announced it was going to offer an S3-compatible object storage, I was similarly enthusiastic. This was quite literally one of the few missing pieces to almost go "full Hetzner" when deploying your web apps! You see, the prior options for S3 storage in Europe were rather mediocre:

So I was happy to hear that Hetzner was offering S3 storage now! I went ahead migrated many of our buckets to Hetzner.

Now, around one year later, I'm migrating us away to Cloudflare R2.

While I could spend lots of time rerunning tests and showing you detailed logs, the TLDR was that the performance (in the sense of "it doesn't break") wasn't good - the most frustrating symptom of this was that uploads would fail sometimes, which is not easy to handle application-side.

During our R2 migration, I also noticed that some files which were listed in Hetzner S3 turned out to be unavailable (?) when actually trying to copy them to R2. I'm not sure what the root cause of this may be (failed uploads in the past? problematic API results? etc.), but regardless of its underlying cause, it's not something you want to experience in an S3 storage.

Looking at the Hetzner status page, entries like these have sadly become an almost daily occurrence.

One can only speculate about the reasons. One plausible hypothesis I read was that Hetzner chose ceph as the underlying software "stack" when building out their S3 service, and people on the internet (tm) write that configuring and maintaining a ceph setup is tricky, to say the least.

That sounds plausible.

Criticism aside, I commend Hetzner on trying to enter this space.

But yeah.. ~one year after using it, it's not something I can recommend right now.

I do have some respect for the challenges they might be facing right now - given that this product has launched (obviously), they can't simply clap their hands and say "yep, ceph was a bad idea, let's rewrite this from scratch" - implementing far-reaching tech stack changes now while you're running an active S3 service with many reads and writes per second is probably a challenge in itself.

I wonder how they'll handle it. In the meantime, I'll be using R2!