Setting up Rails Active Storage with Scaleway Object Storage

In this article I will cover everything needed to set up the object storage provided by Scaleway for use with Active Storage in Ruby on Rails. I'll also shed some light on where things didn't go as well as I thought they would.

To be honest, I thought this would be one of those "ok, this will be done in an hour" tasks. Looking at the Rails guide "Active Storage Overview" it looks pretty straightforward. It turned out to take much longer. That's why I want to share my experience.

First things first

To use object storage, we need object storage ... kind of obvious.

As I am developing a medical application for the German market, I cannot use object storage from AWS or DigitalOcean. So I turned to a French provider called Scaleway. The main issues I had should be the same for all providers.

I assume you already have an account set up if you want to follow this article.

First, I wanted to click through the Scaleway interface to create an object store and a bucket to store my uploads in. I still used the default project I was given. As I have different environments of my application (nonprod and prod) I want to have different sets of credentials for my environments and ensure that one does not have access to the other.

Using multiple buckets in a project with different permissions requires bucket policies that cannot be applied from the interface. Hello Terraform ...

Using terraform to set up object storage

To bring in named policies, I went to my terraform project, which I have anyway to set up my infrastructure in Hetzer Cloud. More on that in another post.

I started by defining a bucket and a bucket policy

# Object storage configuration
resource "scaleway_object_bucket" "storage_bucket" {
  name       = "med1-${var.tenant_name}-01"
  region     = var.region
  project_id = scaleway_account_project.project.id

  cors_rule {
    allowed_headers = ["*"]
    allowed_methods = ["GET", "PUT", "POST", "DELETE"]
    # allowed_origins = var.cors_allowed_origins
    allowed_origins = ["https://your.app.tld"]
    expose_headers  = ["ETag"]
    max_age_seconds = 3600
  }
}

# Attention: this is not working!
# Not being used in final solution!
resource "scaleway_object_bucket_policy" "policy" {
  bucket     = scaleway_object_bucket.storage_bucket.id
  region     = scaleway_object_bucket.storage_bucket.region
  project_id = scaleway_object_bucket.storage_bucket.project_id

  policy = jsonencode(
    {
      Version = "2023-04-17",
      Statement = [
        {
          Sid    = "Read and write access",
          Effect = "Allow",
          Principal = {
            SCW = [
              "user_id:${data.scaleway_iam_user.owner.id}",
              "application_id:${scaleway_iam_application.storage_app.id}",
            ]
          },
          Action = [
            "s3:*",
          ]
          Resource = [
            "${scaleway_object_bucket.storage_bucket.name}",
            "${scaleway_object_bucket.storage_bucket.name}/*"
          ]
        }
      ]
    }
  )
}

**Note: The scaleway_object_bucket_policy is not part of my final solution, as I was unable to work around a permission denied error.

You also need an API key attached to an IAM application and an IAM policy for the IAM application to ensure that the API key has the correct permissions.

In other words

  • Define an IAM application
  • Create an API key and bind it to the IAM application.
  • Then define policies on the IAM application.

This is the code for the Scaleway IAM resources.

resource "scaleway_iam_application" "storage_app" {
  name = "med1-${var.tenant_name}-storage-app"
}

resource "scaleway_iam_api_key" "storage_api_key" {
  application_id     = scaleway_iam_application.storage_app.id
  description        = "API key for using object storage application"
  default_project_id = scaleway_account_project.project.id
}

resource "scaleway_iam_policy" "storage_policy" {
  name           = "med1-${var.tenant_name}-storage-policy"
  application_id = scaleway_iam_application.storage_app.id
  rule {
    project_ids = [scaleway_account_project.project.id]
    permission_set_names = [
      "ObjectStorageObjectsRead",
      "ObjectStorageObjectsWrite",
      "ObjectStorageObjectsDelete",
    ]
  }
}

Read my tip for fetching permission sets

Here I decided to create a project for each environment I wanted to have anyway. I thought it'd be better to set it up now than to have to migrate resources later.

I later discovered a documentation article about migrating from one bucket to another, but I don't know how well that works, and it takes time anyway.

After adding the code to create a project

resource "scaleway_account_project" "project" {
  name = var.tenant_name
}

and scoping the IAM api to that project by adding

default_project_id = scaleway_account_project.project.id

to the scaleway_iam_api_key resource, I ran terraform apply to set up everything I needed.

Read more about the terraform scaleway provider.

Wait, where's my secret key?

After setting up the object storage, I wanted to store the access and secret key of the created api key in 1Password, to realise that I don't have access to it from the terraform output.

Among other outputs, I defined one for the api key

output "scaleway_iam_access_key" {
  value = scaleway_iam_api_key.storage_api_key.access_key
}

output "scaleway_iam_secret_key" {
  value = scaleway_iam_api_key.storage_api_key.secret_key
  sensitive = true
}

Notice the sensitive = true?

I had to set the output to be sensitive after I got the following error message

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Output refers to sensitive values
│
│   on outputs.tf line 70:
│   70: output "scaleway_iam_secret_key" {
│
│ To reduce the risk of accidentally exporting sensitive data that was
│ intended to be only internal, Terraform requires that any root module
│ output containing sensitive data be explicitly marked as sensitive, to
│ confirm your intent.
│
│ If you do intend to export this data, annotate the output value as
│ sensitive by adding the following argument:
│     sensitive = true

In general, this totally makes perfect sense when running in any CI like GitHub Actions or GitLab CI, but in my case I needed the information for my deployments ...

Fortunately, there's a solution. You can run terraform output -json to get defined output in JSON format.

After having the infrastructure in place and ensuring the credentials are working, I moved on to configuring my Rails application.

Rails Active Storage configuration

I highly recommend reading through the excellent Rails Guide for Active Storage to get a good overview of what's involved.

My Active Storage for Scaleway looks like this

# config/storage.yml
scaleway:
  service: S3
  access_key_id: <%= ENV.fetch("SCALEWAY_S3_ACCESS_KEY_ID") %>
  secret_access_key: <%= ENV.fetch("SCALEWAY_S3_ACCESS_KEY_SECRET") %>
  region: <%= ENV.fetch("SCALEWAY_S3_REGION") %>
  bucket: <%= ENV.fetch("SCALEWAY_S3_BUCKET_NAME") %>
  endpoint: <%= ENV.fetch("SCALEWAY_S3_ENDPOINT") %>

You will also need to tell ActiveStorage to use the scaleway service in your config/environments/* files. Personally, I only the object storage in production only by configuring it with

# config/environments/production.rb
config.active_storage.service = :scaleway

Don't forget to add the aws-sdk-s3 gem to your gemfile!

# active storage using scaleway s3
gem "aws-sdk-s3", require: false

After some local testing and fixing configuration issues, I got it to work.

Note: I still have remaining issues

Testing locally

To test it locally, set the development config to use the scaleway service.

# config/environments/development.rb
config.active_storage.service = :scaleway

Remember to add http://localhost:3000 to the allowed_origins of the CORS rule of the scaleway_object_bucket resource. Otherwise you'll get an AWS S3 Permission denied error

You may want to click through your application to test everything or use your rails console.

The very first problem I encountered, was Seahorse::Client::NetworkingError - certificate verify failed, because I had the bucket name set incorrectly. See Seahorse Client NetworkingError - certificate verify failed.

Then I got another Permission Denied error when trying to upload attachments. After reading through the bucket policies overview I decided to remove the bucket policy.

Not using IAM and bucket policies at the same time

Initially I used the bucket policy because I wanted to have all my buckets in one project and ensure the correct permissions.

After deciding to use separate projects, I could have done without the policy.

So I removed the scaleway_object_bucket_policy resource, re-ran terraform apply and ... tada ... uploads work 🚀

Wrapping it up

Although it looked like a trivial task, it took several hours to set up the object storage properly and get everything working.

My main problem in the end was the storage bucket policy, which I couldn't get to work. I wanted to use it because I planned to have my buckets in one project in Scaleway and needed to ensure that one application tenant could not inadvertently modify another bucket.

I should have created different projects from the start.

It would also have been great to have rethought everything after deciding on separate projects. This would have saved me hours of debugging.

Anyway, I have learnt a lot again, and although I still have some issues, I am happy to have a proper setup using terraform.

I hope you found this article useful and that you learned something new.

If you have any questions or feedback, didn't understand something, or found a mistake, please send me an email or drop me a note on twitter / x. I look forward to hearing from you.

Please subscribe to my blog if you'd like to receive future articles directly in your email. If you're already a subscriber, thank you.

I will see you next time! 👋

Remaining issues

Aws::S3::Errors::NoSuchKey for variant when deleting an attachment

When deleting an attachment

 INFO -- :   S3 Storage (116.8ms) Deleted file from key: s9jppxubel7hc96a6dulvcd0hy1g
 INFO -- :   S3 Storage (126.2ms) Deleted file from key: 9ye6w1rfcxufsbjkh58xdq79jugm
 INFO -- :   S3 Storage (62.7ms) Deleted files by key prefix: variants/9ye6w1rfcxufsbjkh58xdq79jugm/
ERROR -- : Error performing ActiveStorage::PurgeJob (Job ID: 158ef8d0-0d3b-4438-b5f9-cfbbeed36561) from Sidekiq(default) in 199.24ms: Aws::S3::Errors::NoSuchKey (The specified key does not exist.):

I asked on Reddit and on StackOverflow.

Objects are being created in subfolders of my bucket

Any uploaded object gets placed in a subfolder named after the bucket nam. In my case an upload resulted in

med1-app-nonprod-med1-io-01.s3.nl-ams.scw.cloud/vmyjvl17v175ocfm48g20dthszo2

I guess this is not the case with AWS S3 itself but I am actually not sure 🤷‍♂️

I'd love to have the files being placed in the root of my bucket instead of a subfolder. Drop me a message on Twitter / X in case you how to solve this.

Caveats

Available Scaleway permission sets

Use scw cli to fetch available permission sets

scw iam permission-set list

See official scaleway-cli docs for installation and setup instructions

Ensure credentials are working using awscli

Install awscli and awscli-plugin-endpoint

  1. install python 3.9+ with homebrew: brew install [email protected]
  2. Install awscli and awscli-plugin-endpoint using pip: pip3 install awscli awscli-plugin-endpoint

List buckets

aws s3 ls

Upload a file to your bucket

aws s3api put-object --bucket med1-app-nonprod-med1-io-01 --key avatar_dummy.png --body path/to/avatar_dummy.png

List all objects in a bucket

aws s3api list-objects --bucket med1-app-nonprod-med1-io-01

Seahorse::Client::NetworkingError - certificate verify failed

Seahorse::Client::NetworkingError (SSL_connect returned=1 errno=0 peeraddr=[2001:bc8:1401::8]:443 state=error: certificate verify failed (hostname mismatch)):

When getting the error above while uploading a file check your active storage configuration. The bucket name needs to be the same as the endpoint but without the protocol.

This is relevant when using S3-compatible object storage. It shouldn't be required when using S3 directly.

SCALEWAY_S3_ENDPOINT=https://foobar-01.s3.nl-ams.scw.cloud
SCALEWAY_S3_BUCKET_NAME=foobar-01.s3.nl-ams.scw.cloud

Source: https://www.digitalocean.com/community/questions/seahorse-client-networkingerror-while-uploading-file-to-spaces

Aws::S3::Errors::AccessDenied - Permission denied

Aws::S3::Errors::AccessDenied (Permission denied.):

If you're getting a permission denied error, ensure CORS settings for the bucket have been set correctly.

See Rails Guide for Active Storage

Example CORS configuration for S3

[
  {
    "AllowedHeaders": [
      "Content-Type",
      "Content-MD5",
      "Content-Disposition"
    ],
    "AllowedMethods": [
      "PUT"
    ],
    "AllowedOrigins": [
      "https://www.example.com"
    ],
    "MaxAgeSeconds": 3600
  }
]

Also see Scaleway docs for CORS on an Object Storage bucket

Verify CORS configuration of a bucket:

curl -X OPTIONS -H 'Origin: http://MY_DOMAIN_NAME' http://BUCKETNAME.s3.nl-ams.scw.cloud/index.html -H "Access-Control-Request-Method: GET"

Uploading a file from the rails console

Assuming you have the following class

class User
  has_one_attached :avatar
end

you can use rails console to test an avatar upload without having to click through your application

User.first.avatar.attach(
  io: File.open("#{Rails.root}/app/assets/images/my_file.png"),
  filename: 'my_file.png',
  content_type: 'image/png'
)