Therefore, I decided to document the issues I faced and how I solved them so maybe someone else can benefit from my pain ;-)
Table of Contents
On first view it looks very straight forward, add the configuration needed and voilà it should work.
The configuration I used looked like the following:
access_key_id: bucket: my-a9s-test-bucket endpoint: oss-eu-central-1.aliyuncs.com region_name: eu-central-1 secret_access_key:
NOTE: The endpoint is crucial if you want to configure the resource for s3 compatible (non aws) resources.
Afterwards, I decided to adjust the endpoint value to `my-test-bucket.oss-eu-central-1.aliyuncs.com`. That resolved the original issue but afterwards I saw the following error message:
My first thought was that the URL built up with the provided configuration somehow got messed up and is checking the wrong place. To debug this issue I enabled the access logs for the OSS buckets. Now I could see that I received requests with the following URI:
So it seems like the resource attaches the bucket name to the endpoint, which might be okay for AWS S3 bucket URLs, but not for Alicloud OSS.
So one more adjustment was missing to get the resource to work, replace the bucket name with ‘/’.
Like this, the resource will check the root directory of your bucket.
So my final configuration looks like:
access_key_id: bucket: ‘/’ endpoint: oss-eu-central-1.aliyuncs.com region_name: eu-central-1 secret_access_key:
After I uploaded the pipeline with the updated configuration I was finally able to see the files of my OSS bucket.
I hope this quick guide helped some of you that were also struggling with the S3 resource configuration for OSS buckets.
One quick additional node, as of now it seems like versioning is not supported for s3 buckets, that means the usage of `versioned_file` property will end up in an arrow looking something like this: