Rails: How To Migrate S3 Providers Via ActiveStorage Mirroring

So you want to migrate your S3 object storage provider in your Rails application. While that might sound scary at first, the good news is that you can use the built-in ActiveStorage mirroring feature of Rails to do it easily!

Random context: If you’re building companies in Europe like me, it seems to be a common thing to switch cloud providers – they’re mostly not as good as AWS, and with new offerings coming out regularly, it often makes sense to switch. That being said, recently, a few new object storage providers with really good pricing came out, e.g. Hetzner.

Alright, onwards..

Migrating Rails S3 Providers: storage.yml

First off, in your storage.yml, define the cloud provider you’d like to migrate to and add the mirror service below it. Here’s an example – here, we’re assuming that Scaleway is our old provider and Hetzner shall be our new one:

# This is our old provider. This entry was here already.
scaleway:
  service: S3
  endpoint: "https://s3.fr-par.scw.cloud"
  access_key_id: <%= Rails.application.credentials.dig(:scaleway, :access_key_id) %>
  secret_access_key: <%= Rails.application.credentials.dig(:scaleway, :secret_access_key) %>
  region: fr-par
  bucket: your_bucket

# This is the new provider we want to migrate to.
hetzner:
  service: S3
  endpoint: "https://nbg1.your-objectstorage.com"
  access_key_id: <%= Rails.application.credentials.dig(:hetzner, :access_key_id) %>
  secret_access_key: <%= Rails.application.credentials.dig(:hetzner, :secret_access_key) %>
  region: nbg1
  bucket: your_bucket

# This is a new mirror service we create. Initially, it points to the old provider (scaleway) because all our current data is there.
# We'll be using this to tranfer the data later.
mirror:
  service: Mirror
  primary: scaleway
  mirrors: [ hetzner ]

Cool! With that set up, let’s head over to your config/production.rb file to actually use this fancy new mirror provider:

# [...] other production config stuff
# The old entry was:
# config.active_storage.service = :scaleway
# We're now changing it to use the mirror service.
config.active_storage.service = :mirror

And deploy your Rails app with your new changes.

Okay, so you’re probably wondering “I thought we’re migrating to Hetzner, not mirroring?” – yes, good point, but we’re using the fancy Rails mirroring provider to actually copy the data. The rough plan is:

1. Set the mirror provider
2. Copy data to the “secondary” provider defined in your mirror provider (= Hetzner)
3. Remove the mirror provider and simply point to the new provider directly (= Hetzner).

Okay. After the deployment from above has succeeded, we now need to log in to the server via rails c and modify the ActiveStorage::Blob entries:

ActiveStorage.:Blob.update_all(service_name: "mirror")
ActiveStorage::Blob.find_each { |blob| blob.mirror_later }

If you’re now thinking “what?”, here’s the explanation: We want to call mirror_later on each blob, because it’s a cool method which ensures that the blobs are uploaded to all secondary providers defined in your mirror provider in your storage.yml.

However! The method mirror_later only works if the provider us actually the “mirror” provider, not your old provider (scaleway, in our case). This is a bit confusing, but makes perfect sense: Rails didn’t touch our existing blobs when we changed the config in the storage.yml, so it still assumes those are on scaleway and not on the fancy mirror provider. That’s why we’re manually changing it.

After setting the mirror provider above, we can call mirror_later, which uploads all blobs to the secondary provider (Hetzner, in our case).

This kicks off background jobs. So track the progress in whatever background job processing dashboard you have. Very satisfying.

After that, we need to update our config/production.rb to now point to our new provider and no longer the mirror provider:

# [...] other production config stuff
# The old entry was:
# config.active_storage.service = :mirror
# We're now finally changing it to use the new service.
config.active_storage.service = :hetzner

And deploy again.

There’s only one remaining thing to fix: All our ActiveStorage::Blob instances still have “service_name” pointing to “mirror”. Let’s fix that.

Once the deployment is done, enter via rails c again and run:

# Update service_name to your new service name
# defined in storage.yml
ActiveStorage::Blob.update_all(service_name: "hetzner")

.. and that’s it!

All your S3 attachments have been moved over to another provider.

Finally, you only need to clean up your storage.yml as the “mirror” and “scaleway” providers are no longer needed – there’s no longer any Blob which references them.

And that’s how you migrate S3 providers in Rails via ActiveStorage mirroring.
Good luck with your migration!