Concourse: Configure S3 Resource to work with Alibaba Cloud OSS

Header Graphic for article Concourse: Configure S3 Resource to work with Alibaba Cloud OSS | anynines

Hi everyone, today I struggled with the configuration of the Concourse S3 resource to work with an Alibaba Cloud OSS bucket.

Therefore, I decided to document the issues I faced and how I solved them so maybe someone else can benefit from my pain ;-)

Let’s start!

On first view it looks very straight forward, add the configuration needed and voilà it should work.
The configuration I used looked like the following:


access_key_id:
bucket: my-a9s-test-bucket
endpoint: oss-eu-central-1.aliyuncs.com
region_name: eu-central-1
secret_access_key:

NOTE: The endpoint is crucial if you want to configure the resource for s3 compatible (non aws) resources.

After uploading the pipeline I ended up with the following error :
Concourse: Virtual hosted style

Afterwards, I decided to adjust the endpoint value to `my-test-bucket.oss-eu-central-1.aliyuncs.com`. That resolved the original issue but afterwards I saw the following error message:
Concourse: The specified key does not exist
My first thought was that the URL built up with the provided configuration somehow got messed up and is checking the wrong place. To debug this issue I enabled the access logs for the OSS buckets. Now I could see that I received requests with the following URI:

‘/my-a9s-test-bucket/?marker=&prefix= ‘

So it seems like the resource attaches the bucket name to the endpoint, which might be okay for AWS S3 bucket URLs, but not for Alicloud OSS.

So one more adjustment was missing to get the resource to work, replace the bucket name with ‘/’.
Like this, the resource will check the root directory of your bucket.
So my final configuration looks like:


access_key_id:
bucket: ‘/’
endpoint: oss-eu-central-1.aliyuncs.com
region_name: eu-central-1
secret_access_key:

After I uploaded the pipeline with the updated configuration I was finally able to see the files of my OSS bucket.

I hope this quick guide helped some of you that were also struggling with the S3 resource configuration for OSS buckets.

One quick additional node, as of now it seems like versioning is not supported for s3 buckets, that means the usage of `versioned_file` property will end up in an arrow looking something like this:

Concourse: bucket is not versioned

Leave a Reply

Your email address will not be published. Required fields are marked *