Rails 8: Setting up Litestream for SQLite on Hetzner S3 (on Dokku)

Writing this up because I just tripped over it and spent an hour debugging what ended up being an obvious problem.

So here’s the situation: You’re running Rails 8 with SQLite as its main database, and you want to replicate / backup that database to Hetzner S3. Bonus points if you’re running Rails on your Hetzner bare metal server on dokku, but that’s probably a story for another time..

Steps:

1. Install litestream-ruby gem
2. Set up Hetzner S3 bucket, get S3 credentials (yes, please set up a separate bucket for your SQLite backups, don’t throw those into your ActiveStorage bucket (shudder))
3. It doesn’t work?

TLDR: Show Me The Config Files:

Here are my config files which worked:

litestream.yml

dbs:
  - path: storage/production.sqlite3
    replicas:
      - type: s3
        bucket: $LITESTREAM_REPLICA_BUCKET
        path: storage/production.sqlite3
        access-key-id: $LITESTREAM_ACCESS_KEY_ID
        secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY
        endpoint: $LITESTREAM_REPLICA_ENDPOINT # new!
        sync-interval: 1m # much cheaper than 1s
        retention: 672h # 4 weeks
  - path: storage/production_cable.sqlite3
    replicas:
      - type: s3
        bucket: $LITESTREAM_REPLICA_BUCKET
        path: storage/production_cable.sqlite3
        access-key-id: $LITESTREAM_ACCESS_KEY_ID
        secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY
        endpoint: $LITESTREAM_REPLICA_ENDPOINT # new!
        sync-interval: 1m # much cheaper than 1s
        retention: 672h # 4 weeks
  - path: storage/production_cache.sqlite3
    replicas:
      - type: s3
        bucket: $LITESTREAM_REPLICA_BUCKET
        path: storage/production_cache.sqlite3
        access-key-id: $LITESTREAM_ACCESS_KEY_ID
        secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY
        endpoint: $LITESTREAM_REPLICA_ENDPOINT # new!
        sync-interval: 1m # much cheaper than 1s
        retention: 672h # 4 weeks
  - path: storage/production_queue.sqlite3
    replicas:
      - type: s3
        bucket: $LITESTREAM_REPLICA_BUCKET
        path: storage/production_queue.sqlite3
        access-key-id: $LITESTREAM_ACCESS_KEY_ID
        secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY
        endpoint: $LITESTREAM_REPLICA_ENDPOINT # new!
        sync-interval: 1m # much cheaper than 1s
        retention: 672h # 4 weeks

So there are a few interesting changes here:

• If you’re using a non-AWS provider (e.g. Hetzner S3), you have to actually set the endpoint parameter here. This tripped me up and cost me ~1hr. It’s confusing because in your litestream.rb file (further below), you’re setting this parameter, so you’re sort of assuming that this parameter is actually being passed to litestream somewhere (spoiler: no).
  Digging through the ruby-litestream source code made me understand that it’s mostly just a thin wrapper around litestream. It sets some env variables for you, but you still have to ensure that your litestream.yml file is actually complete and makes use of these env variables. Damn.
• Setting a higher sync-interval reduces S3 costs (significantly). Copy-pasting from the well-written litestream docs here:

The sync-interval setting primarily affects PUT request costs, not storage costs. Litestream only uploads frames when the database changes, but frequent intervals mean more frequent requests when changes occur.

PUT Request Cost Examples (AWS S3 pricing: $0.000005 per request):

• sync-interval: 1s with constant writes: ~2,592,000 requests/month = $12.96/month
• sync-interval: 10s with constant writes: ~259,200 requests/month = $1.30/month
• sync-interval: 1m with constant writes: ~43,200 requests/month = $0.22/month

Note: Actual costs depend on your database write patterns. Litestream batches writes by time interval, so costs scale with write frequency.

• Finally, setting a longer retention parameter, will, in theory, enable us to “time travel” further back in time in case we need to retrieve old backups. The default value of 24h seems a bit low here, but then again it depends on whether you’re using litestream for a) replication or b) backups. Two different use cases really, but for backups it might be useful to have a longer retention period. Also, S3 storage is cheap. That being said, I’m still learning about litestream, so I’m not 100% sure about this.

So the main learning here was that we need to set the damn endpoint when using non-AWS S3. Okay. What else?

litestream.rb

Rails.application.configure do
  # You'll be using your Rails.application.credentials here,
  # but I'll add some dummy values to explain some stuff:

  # Caution: Contrary to the default comment in this file which tells you to enter
  # a bucket name like "my_bucket.fsn1.your-objectstorage.com",
  # you must only add the actual bucket name, not the
  # full URL. Another 20mins spent debugging, damn.
  config.litestream.replica_bucket "my_bucket"

  # Your Hetzner S3 key ID (the shorter string)
  config.litestream.replica_key_id = "ABC123"

  # Your Hetzner S3 secret key (the longer string)
  config.litestream.replica_access_key = "XYZ123123"

  # Don't set this when using non-AWS S3! Make sure
  # it's commented out.
  # config.litestream.replica_region = "us-east-1"

  # You must set this when using non-AWS S3. Even more so,
  # you have to reference its value in litestream.yml file because otherwise, nothing
  # happens with this value (haha)
  config.litestream.replica_endpoint = "fsn1.your-objectstorage.com"

  # Actually read the yml config
  config.config_path = Rails.root.join("config", "litestream.yml")
end

So first off, replace fsn1 with the Hetzner S3 location you’ve chosen. Besides that, the learnings here are:

• Contrary to what the litestream-ruby comments in the config file state, the bucket_name should only be (you guessed it) your bucket name, not the full URL. In other words, use something like “my_bucket”, and not “my_bucket.fsn1.your-objectstorage.com”. This will throw a 404 error.
• You must comment out the repliace_region! Only use this on AWS.
• You must set the replica_endpoint. Further, setting is it not enough – you actually have to use this new env variable in the litestream.yml file, see above. This tripped me up big time, as I assumed that setting the value here would do something. Turns out that no, setting it here doesn’t do anything besides setting an env variable which you yourself have to use in the litestream.yml file.

And that’s it! Litestream should be working in your Rails application, paired with super affordable Hetzner S3 and, best case a lightning fast bare metal server running dokku. Great success!