Grant Proposal: S5 Network and Apps

Project Name: S5 Network

Developer: redsolver

Introduction

S5 is a content-addressed storage network designed and built from the ground up to enable new decentralized apps and websites which have significant improvements compared to their centralized counterparts.

So to understand why S5 is needed, let’s start with some issues that existing services have:

Cloud storage providers - for example Google Drive, Dropbox, …

  • Your data is not encrypted, which grants companies full access to all of your files and in case they get hacked all of your data is stolen too

  • They can arbitrarily block access to your files or restrict sharing for no reason, like Google did early last year (for example Google Drive flags file only containing “1” for copyright infringement | Hacker News)

  • If you’re only using one provider, you temporarily lose access to all of your data if they experience downtime

  • Specific features you want might not be available in their clients

The cloud storage use case will be covered by the Vup Cloud Storage app, which solves all of these issues.

Social media platforms - for example YouTube, Twitter, Instagram, Twitch, TikTok, …

  • You’re locked into their platform, it’s not possible to switch to another platform and keep your connections and content

  • They have full control over your identity and can decide to ban you at any time. Giving a single centralized entity this kind of power is always dangerous, as seen on Twitter some months ago

  • A centralized setup also always carries the risk of censorship, like YouTube did with some infosec content. This also applies to specific rules made up by advertisers, who have large control over the creator economy

  • Your privacy is at risk, because these platforms store a lot of highly personal information about you. They can both use it for malicious purposes like selling it to advertisers, but they can also get hacked/breached and suddenly all of your personal data is being sold by criminals

Tube5 is a YouTube alternative powered by S5, right now it’s still in alpha and read-only: https://tube5.app

Messenger and chat platforms

  • Similar issues like social media platforms for identity

  • Many are not end-to-end-encrypted, which is a serious privacy risk

  • Platforms like Discord make it hard to export chat history and have a strong lock-in effect

S5 supports realtime subscriptions, allowing both public and private chats to be built on it in the future.

Other use cases

  • Decentralized App Stores (SkyDroid)

  • Archival

… and all other projects that need easy-to-use secure decentralized storage :)

So what is S5?

S5 is similar to IPFS, but faster, more flexible and with more commonly needed features built-in, like accounts.

Similar to Sia Skynet it makes Sia storage available on the web and easy to use on all platforms without needing special software, while still being as secure as possible. It solves most usability challenges, like needing Siacoin to get started, syncing the full blockchain and uploading 3x the total data for redundancy. S5 runs on all devices, including web and mobile.

The p2p network does not handle file transfers directly, instead it primarily focuses on efficiently mapping file hashes to storage locations (for example HTTP(S) download URLs). This makes it scalable and trivial to support multiple storage backends.

Flexible storage backends

S5 supports multiple storage backends, in addition to Sia. I believe that the content-addressed storage layer that powers the Internet of tomorrow doesn’t have to depend on a single storage solution, and it’s better to just support all of them! This also makes it possible to utilize the specific strengths of different storage networks, by merging and combining them. As an example, a S5 node can be configured to store small files locally and upload medium to large files directly to Sia. They can still be uploaded to Sia later when enough small files are stored to pack one large file, but they are already available on the network during that time, and the CID is always the same!

There’s already a fully working S5 storage service powered by Sia running on https://sia5.net, but it doesn’t allow anonymous uploads. Here’s an example file stored on it using Sia: S5 CID Explorer

In addition to Sia, files can also be stored on the user’s home NAS, Arweave, S3-compatible buckets, Pixeldrain and some others with exactly the same API and CIDs!

Other improvements

A significant improvement compared to IPFS is no maximum block size, which results in a lot less requests on the p2p network for downloading/streaming files larger than 1 MiB. Verified streaming is still possible, thanks to the BLAKE3 hash function being a Merkle tree. Another improvement is the use of the BLAKE3 hash function by default, which speeds up many operations like adding files to the network. A different project called Iroh that is trying to reimagine IPFS is using the same cryptographic primitives and the S5 file layer will likely be fully compatible with it.

You can read more about the S5 technical details in the (still work-in-progress) docs: https://docs.sfive.net

When using the Vup Cloud Storage app powered by S5, most users will likely choose to use one or multiple remote storage services with monthly payments, similar to the Skynet portal concept. Even when using a remote service, S5 is more secure than Skynet because all API endpoints are already fully trustless (both upload and download validate integrity client-side). Using BLAKE3 also opens up the possibility for S5-native lightweight storage proofs in the future. But it’s also a lot easier to just self-host renterd somewhere instead or even run it locally on the same device as Vup.

Interoperability and compatibility

All S5 metadata formats are designed to be as flexible as possible. For example, both simple blog posts and really complex videos with thumbnails and multiple codecs+resolutions have the same base metadata format.

S5 also supports importing files from a variety of different sources, for example any HTTP(S) endpoint on the internet. It’s easy for users to run a lightweight script that imports an entire Linux package mirror to their S5 node, by simply passing the existing HTTP(S) URLs.

Another interesting concept are bridges, these are an optional feature which can convert any content from centralized services to the S5 metadata format on-the-fly, fully decentralized. A S5 node could simply ask for the hash :yt:video:aqz-KE-bpKQ on the network, and a node that supports the bridge feature will use the API or scrape the relevant data, convert it to the S5 metadata format and return it. This feature will likely be restricted by most nodes to prevent abuse, but it still provides a lot of exciting opportunities.

There are already some projects building on top of S5, for example the Fabstir media player and some others that are still trying it out.

Lume Web will be using S5 components/tech as well to ensure interoperability with the ecosystem.

With the Sia store, all projects using S5 get easy access to storage on Sia.

The grant

This grant will be used for development of the S5 network, related libraries, devtools and documentation, the Vup Cloud Storage app, the Tube5 video streaming app and some other apps that showcase S5 features for both developers and users.

My previous projects can be found on redsolver · GitHub, the Skynet-ones being the most relevant, because many parts of S5 are inspired by Skynet tech. As such, most of my previous projects don’t work anymore because they depend on Skynet. So that’s exactly where S5 comes in.

  • Vup Cloud Storage (https://vup.app)

  • SkyDroid (https://skydroid.app, decentralized Android app store)

  • SkyFeed (decentralized social network)

  • SkySend (temporary file-sharing tool)

  • SkyOS (web-based workspace and window manager for Skynet apps)

  • Skynet Dart SDK

  • A lot of DACs (Data-Access-Controllers for MySky), developer tooling (for example skydeploy)

Grant Specifics

The S5 project is requesting the following budget for one year:

  • 75k EUR for the full-time salary of redsolver, 12 months, paid monthly

  • 5k EUR for infrastructure costs, mostly renting cloud servers for testing and onboarding large amounts of data to the network

  • 4k EUR for logo, website and UI designs

  • 2k EUR for Sia storage costs (paid in SC)

Total: 86,000 EUR

Timeline

This project has a lot of moving parts and multiple of them are developed in parallel, so it’s hard to set a reliable timeline, but there should be at least one big announcement / finished part every month.

  • Publish a full alpha version of the Vup Cloud Storage app powered by S5 (April)

  • Improve Sia (renterd) store, test large files and add support for using a local renterd instance to Vup (April-May)

  • Publish web app to securely download and stream shared end-to-end encrypted files and directories (April)

  • Publish the s5_web_proxy used for the just mentioned web app and s5.cx (s5.cx - S5 Network Docs) as a standalone repository to make integrating trustless streaming of any files directly from the S5 network as easy as using the URL /s5/blob/CID in your image, audio or video elements.

  • Build a simple browser extension for S5 identity management in all major browsers including Chrome (May)

  • Extend and improve the documentation

  • Create high-level bindings for TypeScript/JavaScript (May)

  • Add transcoding+publish support of Tube5 videos to Vup (June)

  • Add interactive elements like followers and comments to Tube5 (June)

  • Migrate the SkyDroid Android App Store to the S5 network

  • Migrate S5 Chat (previously SkyChat) to S5 (self-hosted alternative to Discord with fully decentralized identities and storage). Also has excellent bridging support

These are a lot of projects, but many of them re-use code from my old Skynet projects which makes things easier. So I’m confident that at least 80% of them can be accomplished in the next 12 months.

Risks

I think the biggest risk is that I’m the only core developer of S5, so if something happened to me the project would likely have issues continuing. But everything is fully open-source, so it’s always possible to pick up development where it left off.

Another risk could be the identity browser extension not being accepted by the different extension stores, but I already verified that all APIs I need work with Manifest V3 and adhere to the guidelines, so it’s pretty unlikely that this happens.

Everything is FOSS (free and open-source)

All S5 source code is already fully open-source on s5-dev · GitHub

The (old) Vup Source Code for the Skynet-based version is still on GitHub - vup-app/vup, it will be moved to the s5-dev organization with the first S5-powered alpha release in the next weeks.

New repositories and code will be added there too.

All libraries and the S5 node are MIT-licensed to make adoption as easy as possible.

User-facing applications like Vup and Tube5 are licensed with the EUPL-1.2 (very similar to the GPL), because they are intended to stay open-source and this prevents a malicious actor from taking the project closed-source by forking and investing a lot of developers to move faster, because that would ultimately hurt users.

All docs are licensed CC-BY-4.0, data like JSON schemas for all metadata formats is CC0.

Progress reports

Yes, I agree to submit monthly progress reports. In addition to them, I will also be posting regular updates (every week or so) on the S5 Discord Server (invite link available on https://docs.sfive.net/).

Conclusion

Thanks for your continued support of my projects over the past 3 years. I believe that we can build a better version of today’s web, and after Skynet didn’t turn out the way we hoped for, I’m confident that S5 can be the next step in this journey.

12 Likes

As a fellow builder, I definitely support this proposal.

S5 is solving separate problems from Lume Web (and thus different directions), but both are needed long term and will end up complementing each other.

We are all here to work together, and S5 is a needed part of that. :sunglasses:

2 Likes

I am in full support of this project. It will be paramount in superseding where Skynet left off.

2 Likes

While the timeline is aggressive, I think this is a great project for the SF to back. Has obvious benefits to the ecosystem, and is generally useful tech.

I just hope that Red won’t be on the review board for his own project getting funded :p

1 Like

Based on what @mike76 has said, he gets excluded from the review process.

Finally a proposal from you :slight_smile:

How does it work? In the browser, like with Skynet? Or do you have a backend?

I also support this proposal. But this seems a lot of work for one person. What will happen with all the apps after they are build ? Will you keep supporting/updating them all or will they be left as is ?

Remote storage services (similar to Skynet portals) or users need to run a S5 node, which provides APIs for uploading files and S5-specific features like the registry. Downloads are fetched directly from the storage location, when using Sia they are proxied with HTTP(S) by the renterd instance storing them. This is the same approach the Sia Foundation is planning to use for their open datasets initiative. In the future, S5 clients could support downloading data from Sia hosts directly, using ephemeral accounts.

This one is a no-brainer:) +1

1 Like

I will continue supporting/updating all apps that reach a stable (1.0) milestone, and most of the other ones depending on usage. The S5 core software and the Vup Cloud Storage app have the highest priority. S5 Chat will be used as the primary communication platform for the project when it’s ready, so maintaining it will also be a top priority. SkyDroid was one of the earliest Skynet projects and I’ve been keeping it and many other projects mostly up-to-date in my free time, so with even more time at my disposal I expect everything to work out well.

I’m not really worried about it, would just like it publicly posted by someone from the committee so it isn’t a trust issue in the future.

Committee members are required to recuse themselves from their grant approval process and monthly progress reviews.

I support this proposal. Witnessed what redsolver created and achieved last couple of years, with his capabilities, skills and perseverance, I’m fully confident this grant will be in good hands.

1 Like

+1 :rocket: Very excited to see this project funded.

1 Like

Yes and yes to this proposal!

I fully support the S5 project and look forward to collaborating with @redsolver

Hello Redsolver,

The committee has reviewed your proposal, however, we need additional time to discuss a few details. Unfortunately, this means that we will not have an answer for you by the end of today however we will have an answer prepared for you by tomorrow afternoon.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Hello again Redsolver,

Congratulations, the committee has approved your proposal!

The committee was very excited to see this long-time community project turn into a grant proposal, and are even more excited to see what you’ll be able to do with complete funding.

The committee kindly reminds each team about the importance of gauging progress against milestones. Please note that continued funding is subject to reassessment if sufficient progress is not made.

The first payment will be disbursed May 15, so feel free to adjust your timeline accordingly. Please provide an email address that we can use to reach your team and begin the grants onboarding process!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

2 Likes

Thanks for the approval! My email address is info@redsolver.net.

1 Like

Progress Report (April)

An alpha version of Vup powered by S5 has been published for Linux, Windows and Android. There has been a lot of Vup development progress since that, so a new beta release will be published early May, for all platforms (including iOS and macOS).

Linux: flatpak install flathub app.vup.Vup (Vup Cloud Storage | Flathub)
Windows (winget): winget install Vup.Vup
Android and Windows (direct download): Release Version 0.13.3 BETA · vup-app/vup · GitHub

Vup and S5 already work with Sia (using renterd), I’ve been testing the integration with the Sia-powered S5 node running on https://sia5.net/. There are however still issues with uploading and downloading large files, so I’m currently in the process of debugging that.

One of the goals for April was end-to-end-encrypted streaming of large files directly in the web browser, that’s finished and working well: Encrypted file (This video file has been uploaded with Vup).

Another fun demo is this map of the entire world, streamed directly from the S5 network with full integrity verification: https://map-demo.sfive.cloud/ (I tried uploading the 110 GB map data file with Sia first, but unfortunately there are still some issues with large files as mentioned earlier)

1 Like