Project Name: S5 Network
Developer: redsolver
Introduction
S5 is a content-addressed storage network designed and built from the ground up to enable new decentralized apps and websites which have significant improvements compared to their centralized counterparts.
So to understand why S5 is needed, let’s start with some issues that existing services have:
Cloud storage providers - for example Google Drive, Dropbox, …
-
Your data is not encrypted, which grants companies full access to all of your files and in case they get hacked all of your data is stolen too
-
They can arbitrarily block access to your files or restrict sharing for no reason, like Google did early last year (for example Google Drive flags file only containing “1” for copyright infringement | Hacker News)
-
If you’re only using one provider, you temporarily lose access to all of your data if they experience downtime
-
Specific features you want might not be available in their clients
The cloud storage use case will be covered by the Vup Cloud Storage app, which solves all of these issues.
Social media platforms - for example YouTube, Twitter, Instagram, Twitch, TikTok, …
-
You’re locked into their platform, it’s not possible to switch to another platform and keep your connections and content
-
They have full control over your identity and can decide to ban you at any time. Giving a single centralized entity this kind of power is always dangerous, as seen on Twitter some months ago
-
A centralized setup also always carries the risk of censorship, like YouTube did with some infosec content. This also applies to specific rules made up by advertisers, who have large control over the creator economy
-
Your privacy is at risk, because these platforms store a lot of highly personal information about you. They can both use it for malicious purposes like selling it to advertisers, but they can also get hacked/breached and suddenly all of your personal data is being sold by criminals
Tube5 is a YouTube alternative powered by S5, right now it’s still in alpha and read-only: https://tube5.app
Messenger and chat platforms
-
Similar issues like social media platforms for identity
-
Many are not end-to-end-encrypted, which is a serious privacy risk
-
Platforms like Discord make it hard to export chat history and have a strong lock-in effect
S5 supports realtime subscriptions, allowing both public and private chats to be built on it in the future.
Other use cases
-
Decentralized App Stores (SkyDroid)
-
Archival
… and all other projects that need easy-to-use secure decentralized storage :)
So what is S5?
S5 is similar to IPFS, but faster, more flexible and with more commonly needed features built-in, like accounts.
Similar to Sia Skynet it makes Sia storage available on the web and easy to use on all platforms without needing special software, while still being as secure as possible. It solves most usability challenges, like needing Siacoin to get started, syncing the full blockchain and uploading 3x the total data for redundancy. S5 runs on all devices, including web and mobile.
The p2p network does not handle file transfers directly, instead it primarily focuses on efficiently mapping file hashes to storage locations (for example HTTP(S) download URLs). This makes it scalable and trivial to support multiple storage backends.
Flexible storage backends
S5 supports multiple storage backends, in addition to Sia. I believe that the content-addressed storage layer that powers the Internet of tomorrow doesn’t have to depend on a single storage solution, and it’s better to just support all of them! This also makes it possible to utilize the specific strengths of different storage networks, by merging and combining them. As an example, a S5 node can be configured to store small files locally and upload medium to large files directly to Sia. They can still be uploaded to Sia later when enough small files are stored to pack one large file, but they are already available on the network during that time, and the CID is always the same!
There’s already a fully working S5 storage service powered by Sia running on https://sia5.net, but it doesn’t allow anonymous uploads. Here’s an example file stored on it using Sia: S5 CID Explorer
In addition to Sia, files can also be stored on the user’s home NAS, Arweave, S3-compatible buckets, Pixeldrain and some others with exactly the same API and CIDs!
Other improvements
A significant improvement compared to IPFS is no maximum block size, which results in a lot less requests on the p2p network for downloading/streaming files larger than 1 MiB. Verified streaming is still possible, thanks to the BLAKE3 hash function being a Merkle tree. Another improvement is the use of the BLAKE3 hash function by default, which speeds up many operations like adding files to the network. A different project called Iroh that is trying to reimagine IPFS is using the same cryptographic primitives and the S5 file layer will likely be fully compatible with it.
You can read more about the S5 technical details in the (still work-in-progress) docs: https://docs.sfive.net
When using the Vup Cloud Storage app powered by S5, most users will likely choose to use one or multiple remote storage services with monthly payments, similar to the Skynet portal concept. Even when using a remote service, S5 is more secure than Skynet because all API endpoints are already fully trustless (both upload and download validate integrity client-side). Using BLAKE3 also opens up the possibility for S5-native lightweight storage proofs in the future. But it’s also a lot easier to just self-host renterd somewhere instead or even run it locally on the same device as Vup.
Interoperability and compatibility
All S5 metadata formats are designed to be as flexible as possible. For example, both simple blog posts and really complex videos with thumbnails and multiple codecs+resolutions have the same base metadata format.
S5 also supports importing files from a variety of different sources, for example any HTTP(S) endpoint on the internet. It’s easy for users to run a lightweight script that imports an entire Linux package mirror to their S5 node, by simply passing the existing HTTP(S) URLs.
Another interesting concept are bridges, these are an optional feature which can convert any content from centralized services to the S5 metadata format on-the-fly, fully decentralized. A S5 node could simply ask for the hash :yt:video:aqz-KE-bpKQ
on the network, and a node that supports the bridge feature will use the API or scrape the relevant data, convert it to the S5 metadata format and return it. This feature will likely be restricted by most nodes to prevent abuse, but it still provides a lot of exciting opportunities.
There are already some projects building on top of S5, for example the Fabstir media player and some others that are still trying it out.
Lume Web will be using S5 components/tech as well to ensure interoperability with the ecosystem.
With the Sia store, all projects using S5 get easy access to storage on Sia.
The grant
This grant will be used for development of the S5 network, related libraries, devtools and documentation, the Vup Cloud Storage app, the Tube5 video streaming app and some other apps that showcase S5 features for both developers and users.
My previous projects can be found on redsolver · GitHub, the Skynet-ones being the most relevant, because many parts of S5 are inspired by Skynet tech. As such, most of my previous projects don’t work anymore because they depend on Skynet. So that’s exactly where S5 comes in.
-
Vup Cloud Storage (https://vup.app)
-
SkyDroid (https://skydroid.app, decentralized Android app store)
-
SkyFeed (decentralized social network)
-
SkySend (temporary file-sharing tool)
-
SkyOS (web-based workspace and window manager for Skynet apps)
-
Skynet Dart SDK
-
A lot of DACs (Data-Access-Controllers for MySky), developer tooling (for example skydeploy)
Grant Specifics
The S5 project is requesting the following budget for one year:
-
75k EUR for the full-time salary of redsolver, 12 months, paid monthly
-
5k EUR for infrastructure costs, mostly renting cloud servers for testing and onboarding large amounts of data to the network
-
4k EUR for logo, website and UI designs
-
2k EUR for Sia storage costs (paid in SC)
Total: 86,000 EUR
Timeline
This project has a lot of moving parts and multiple of them are developed in parallel, so it’s hard to set a reliable timeline, but there should be at least one big announcement / finished part every month.
-
Publish a full alpha version of the Vup Cloud Storage app powered by S5 (April)
-
Improve Sia (renterd) store, test large files and add support for using a local renterd instance to Vup (April-May)
-
Publish web app to securely download and stream shared end-to-end encrypted files and directories (April)
-
Publish the s5_web_proxy used for the just mentioned web app and s5.cx (s5.cx - S5 Network Docs) as a standalone repository to make integrating trustless streaming of any files directly from the S5 network as easy as using the URL
/s5/blob/CID
in your image, audio or video elements. -
Build a simple browser extension for S5 identity management in all major browsers including Chrome (May)
-
Extend and improve the documentation
-
Create high-level bindings for TypeScript/JavaScript (May)
-
Add transcoding+publish support of Tube5 videos to Vup (June)
-
Add interactive elements like followers and comments to Tube5 (June)
-
Migrate the SkyDroid Android App Store to the S5 network
-
Migrate S5 Chat (previously SkyChat) to S5 (self-hosted alternative to Discord with fully decentralized identities and storage). Also has excellent bridging support
These are a lot of projects, but many of them re-use code from my old Skynet projects which makes things easier. So I’m confident that at least 80% of them can be accomplished in the next 12 months.
Risks
I think the biggest risk is that I’m the only core developer of S5, so if something happened to me the project would likely have issues continuing. But everything is fully open-source, so it’s always possible to pick up development where it left off.
Another risk could be the identity browser extension not being accepted by the different extension stores, but I already verified that all APIs I need work with Manifest V3 and adhere to the guidelines, so it’s pretty unlikely that this happens.
Everything is FOSS (free and open-source)
All S5 source code is already fully open-source on s5-dev · GitHub
The (old) Vup Source Code for the Skynet-based version is still on GitHub - vup-app/vup, it will be moved to the s5-dev organization with the first S5-powered alpha release in the next weeks.
New repositories and code will be added there too.
All libraries and the S5 node are MIT-licensed to make adoption as easy as possible.
User-facing applications like Vup and Tube5 are licensed with the EUPL-1.2 (very similar to the GPL), because they are intended to stay open-source and this prevents a malicious actor from taking the project closed-source by forking and investing a lot of developers to move faster, because that would ultimately hurt users.
All docs are licensed CC-BY-4.0, data like JSON schemas for all metadata formats is CC0.
Progress reports
Yes, I agree to submit monthly progress reports. In addition to them, I will also be posting regular updates (every week or so) on the S5 Discord Server (invite link available on https://docs.sfive.net/).
Conclusion
Thanks for your continued support of my projects over the past 3 years. I believe that we can build a better version of today’s web, and after Skynet didn’t turn out the way we hoped for, I’m confident that S5 can be the next step in this journey.