Proposal: Add first-class S3 provider support

Foundation
#1

For many years I’ve wanted to use Sia but have been unable due to technical constraints.

Here are my use-cases:

  • I need a public dropbox/pastebin for file sharing (siasky addressed this!)
  • I use software that needs to put data on an S3-compatible CDN. I’m currently using Cloudflare and a blob storage provider, but I’d rather be using Sia.
  • I’d like to back-up family machines using Sia. Clearly this is out-of-scope for Sia, but could be done using Sia as an S3 provider. See the pattern here?

I understand the Foundation is undergoing growth/restructuring. I’m anxiously awaiting the Utreexo and file chunking/compression improvements.

In the past, Sia paid a bounty to a developer to create Minio (S3) integration. This has since been abandoned and deprecated.

To succeed, Sia needs adoption. To be adopted, it must not be painful. I’ve wanted to use Sia for years, but couldn’t and still can’t because it’s too painful (and I’m a power-user!).

After the foundation has stabilized, I urge that you implement S3 compatibility as a first-class citizen in Sia. Like it or not, most apps with a cloud storage need have support for S3 providers.

There are a number of companies with Reddit accounts who post here that provide this (S3 provider) using Sia as a backbone. This should be a big hint to the Foundation as to the demand for this feature.

tl;dr: implement Sia as an S3 provider, let me add it (siad/sias3) to my docker-compose.yml and move on with my day.

Thank you for coming to my TED talk. Long live Sia, the decentralized internet, and not losing all of my data to an earthquake or flood.

Link to my original post on Reddit: https://old.reddit.com/r/siacoin/comments/mj9gus/my_proposal_to_the_foundation_add_firstclass_s3/

1 Like
#2

I agree, no support for the industry standard is missing out on a ton of opportunity.

1 Like
#3

I’ve made some initial strides towards a key-value API in us: https://github.com/lukechampine/us/blob/kv/renter/renterutil/kv.go

I’m wary of pursuing a first-class S3 API, though. It’s quite a large API, with all sorts of quirks. A proper implementation would require a substantial investment of time and energy, and all of that work would essentially be undercutting or duplicating the work of existing built-on-Sia companies.

I do think that it’s reasonable for a program like user to provide a barebones S3 API, built on the key-value us API. But this would only be an option for power users who are willing to manage their own contracts.

1 Like
#4

In case it wasn’t clear, I’m not saying that Sia needs to support the full S3 API, rather just that it is easier to implement a Minio backend or simple HTTP implementation. https://github.com/lukechampine/us/issues/67 looks like big steps in the right direction, along with the entire WIP kv branch.

all of that work would essentially be undercutting or duplicating the work of existing built-on-Sia companies

The conflict of interest outlined above is just that - a conflict of interest. I have no need or desire to introduce Storwise or Filebase as middlemen for my data. For enterprise, I’m sure that the contract selection, redundancy on additional non-Sia platforms, archival, etc. are all useful. I am not enterprise, and until Sia is adopted by individuals who work for enterprise (hello, that is me) it is unlikely to see wide enterprise adoption.

I know some of my statements are rather poignant, but I’m not rocking the apple cart just for the sake of doing so. I just want to make use of the great technology underpinning Sia.

#5

The conflict of interest outlined above is just that - a conflict of interest. I have no need or desire to introduce Storwise or Filebase as middlemen for my data.

While that’s fair, it would be kinda a jerk move to use community resources to compete with Filebase and Storewise when there are so many other things that need to be addressed before normal users would even realistically want to use Sia. For example, the lack of node scaling, small sector support, and utreexo all make it far more impractical to run a siad node on a NAS(in order to expand the storage) than the lack of an S3 API.

#6

For example, the lack of node scaling, small sector support, and utreexo all make it far more impractical to run a siad node on a NAS(in order to expand the storage) than the lack of an S3 API.

Not disagreeing that these are good features, I mentioned in the linked Reddit post that both utreexo and small sector support (file compression, chunking, call it what whatever) are two of the three features on my wishlist. S3 support is the third.

While that’s fair, it would be kinda a jerk move to use community resources to compete with Filebase and Storewise

As to the elephant in the room, I harbor no ill will. I don’t see this as a “jerk move”. Put simply, I want S3 support in Sia. The current situation is akin to a US consumer goods manufacturer refusing to provide European power plug support, but you can pay some other company to make you an adapter. Why doesn’t the company just provide what customers want? May not be the best analogy, but that’s how it feels. And that doesn’t even touch on the data custody issue.

I’m sure Storewise and Filebase are great companies and as pioneers of this S3 adapter, demonstrate the demand for this functionality. I have no specific knowledge of the offerings of these service-based companies, but I’m sure that they offer their customers other good features (automagic archival, geographic storage constraints, logging, contract management, CDN, multi-provider storage, etc.). My feature request is in no way meant to diminish their businesses. It’s just a sane feature request, acknowledging the conflict of interest as I see it.

1 Like