The current State of S5 (May 2024)
This will be my second grant proposal for the S5 Network, you can check out the first one here: Grant Proposal: S5 Network and Apps
This first post serves as an overview and “final progress report” for the progress made during the entire first grant. If you only care about the new grant, skip to the “Milestones and Timeline” section!
If you don’t know what S5 is yet, check out this guest blog post: Community Spotlight: S5. A Content-Addressed Data Routing… | by Skunk_Ink | The Sia Blog
Or watch the S5 segment in the recent Sia Community Town Hall: https://www.youtube.com/watch?v=JEtFG3yLjWI&t=580
S5 Node
The S5 Node is written in Dart+Rust and can be easily deployed on most systems using a Docker container. It provides an account system and APIs for uploading and managing small and big blobs. The node connects to the public S5 network (configurable) and stores data in one or multiple blob stores. Currently supported are Sia renterd, S3 (incl. renterd and many others), the local file system, IPFS and Pixeldrain. It can easily handle blobs with a size of 110 GiB each (that’s the largest I tested for now, bigger ones should work too with enough storage space). For renterd you can either use the native or the S3 store. The S3 one is slightly more reliable if you’re hosting the S5 Node on a different server than renterd and is easier to set up. The native renterd store is more efficient and slightly faster, but a bit more complex to set up.
The S5 Node also supports an experimental “File System Provider”. This is a “virtual” store which efficiently indexes any directory on your local FS and then makes it available in the S5-native directory metadata format directly on the network. It can for example be used for mirroring Linux package registries or any other use case where there’s already a good directory structure with a lot of large (or small) files. Some other fun new features include being able to import files from the internet by just passing their HTTP(s) URLs. The S5 Node also supports DNSLink (https://dnslink.dev/) to host websites and web apps on the S5 Network.
There’s also a second implementation of the S5 Node written in Golang by @pcfreak30 which I’m very excited about! I helped debug it and fixed some bugs in the S5 specification in the process.
GitHub Repository: GitHub - s5-dev/S5: Decentralized content-addressed storage network
Admin Web App
The Admin Web App (GitHub - s5-dev/node-admin-web-app: Simple Web App to manage accounts on a S5 Node using the Admin API. Your S5 Node serves this at /s5/admin/app) is a simple web app integrated in the S5 Node to manage accounts and their storage usage with a nice UI. (available on /s5/admin/app
)
FS5
The S5 file system (FS5) is a mutable data structure built on top of S5 primitives which supports both public and private (end-to-end-encrypted) directories. It’s for example being used by Vup and will also be used by some other apps pretty soon (after the spec is stable). The FS is pretty efficient for most operations and only needs two of the S5 primitives (blobs and the registry). One of the more recent new features is Snapshots, which makes it possible to create immutable snapshots of whole directory trees instantly (if cached).
S5 Nodes have a public HTTP endpoint for browsing FS5 directories (downloads redirect, the S5 Node does NOT proxy file content!): Index of archlinux/
Vup Cloud Storage App
Vup is the flagship app of the S5 protocol and network. It’s a decentralized end-to-end-encrypted cloud storage app with support for many storage backends (including Sia) and apps for Android, Linux, Windows, macOS and iOS.
During my first grant, I published a massive new release which fully migrated Vup from the old Skynet backend and data structures to the S5 Network. I also added a video player, image gallery view, mass-pinning tool, new more efficient streaming implementation, video publishing dialog, fixed a bug which prevented browsing public directories (like the archlinux mirror), uploaded some TBs of small to medium-sized files to test scalability and implemented some UI improvements and bug fixes. Vup also runs a local S5 Node and supports configuring it with a self-hosted Sia renterd instance, but due to that abstracting some important parts away, users are advised to instead host a S5 Node alongside their renterd installation in the future (see Deploy S5 Node with Sia renterd - S5 Network Docs). With renterd getting more lightweight in the future (especially with reduced local state requirements), I could see shipping renterd as part of Vup and making setup as easy as possible.
Vup also has rclone and restic support, can do media streaming with an integrated Jellyfin-compatible API server, integrated sync, scripts and a lot more features! (check out the website)
Website: https://vup.app/
Source Code: GitHub - vup-app/vup
Available on Flathub: Vup Cloud Storage auf Linux installieren | Flathub
lib5 Dart
GitHub Repo: GitHub - s5-dev/lib5: This library is used by Dart-based code for the S5 network.
Packages: lib5 | Dart package and lib5_crypto_implementation_dart | Dart package
lib5 contains most of the S5 data structures and code. It’s designed to be modular and different parts like the API implementation or crypto provider (for hashing, signing, encryption) can be swapped out. It also runs on any platform supported by Dart, like Linux, Android, iOS, Windows, macOS and the Web!
It has classes to work with CIDs, Multihashes and S5 file metadata, registry entries, contains a full S5 Node implementation which can run anywhere, some utils and more!
In March, I wrote a new implementation of the S5 file system for lib5. It has improved support for public directory trees, comes with a new, more consistent URI format, integrates nicely with accounts and other S5 APIs and most importantly, does not contain any legacy code from the Skynet days anymore!
s5-dart
The s5
Dart package is intended for developers to easily add all S5 features to their Flutter or Dart app with minimal effort (compared to lib5, which is more low-level). It wraps the lib5
package and pre-configures almost everything for you, so it’s easier to integrate S5 with your app.
-
Runs a local S5 Node and connects to default nodes (can be configured)
-
Provides all S5 APIs (file, registry, streams) with full integrity verification
-
Supports creating and restoring user identities and uploading to third-part storage services (nodes)
Package: s5 | Dart package
Source Code: GitHub - s5-dev/s5-dart: Use the S5 Network in your Flutter and Dart apps
There’s also a Flutter-based demo web app showcasing some basic features for devs (still WIP):
https://demo.sfive.net/ (GitHub - s5-dev/flutter_demo_app: Flutter app which showcases some S5 features using the s5 package)
Web Proxy
The S5 Web Proxy can securely stream files of any size from the S5 network, directly in your web browser. It should work in all major web browsers, including Chrome, Brave, Firefox and Safari.
It works by registering and running a service worker which intercepts specific requests, looks up download URLs for them and then verifies their integrity with BLAKE3/bao running in Rust via WASM.
After adding the service worker to your web app, streaming files from the S5 network is as easy as using the URL /s5/blob/YOUR_CID_HERE?mediaType=image%2Fpng
in img tags, for your video player or just fetching it directly! Encrypted files are supported too.
Source Code: GitHub - s5-dev/web-proxy: Client-side gateway with verified streaming, running in the web browser
Some example apps using this proxy:
-
s5.cx for very basic file sharing, for example encrypted ones: Encrypted file
-
Tube5 Alpha Simple decentralized video platform, uses the s5 web proxy for streaming video, audio, images and text.
-
a web-based map view, powered by the PMTiles format. (no longer online, previously on https://map-demo.sfive.cloud/)
CID Explorer
Web App for exploring and debugging S5 CIDs
Source Code: GitHub - s5-dev/cid-explorer: Web App for exploring and debugging S5 CIDs
Example: S5 CID Explorer
More examples: cid.one - S5 Network Docs
Documentation
-
Started working on specifications for a “stable v1” version of the S5 data structures and formats: Blobs - S5 Network Docs
-
Full Sia Setup Guide: Deploy S5 Node with Sia renterd - S5 Network Docs
-
Explaining basic concepts like content-addressed data: Content-addressed data and CIDs - S5 Network Docs
-
Documentation for running a S5 Node and configuring different stores
-
Uses Markdown with mdbook: GitHub - s5-dev/docs
Archiver
Archiving makes sense for personal archive use cases, for preserving history and satisfying your data hoarder needs. I’m also planning to support Twitch VODs due to them needing a ton of space and Twitch deleting past broadcasts after 7 days for free users and 60 days for paid ones by default, so streamers might be interested in having a good alternative.
I started working on an archiving tool to archive websites and other content available on the Internet on the content-addressed S5 Network. It downloads website files and stores them in an optimized FS5 directory structure (public or private). Can then be browsed like a personal web archive or used for a more coordinated global content-addressed web archiving effort. (Still WIP: GitHub - s5-dev/archiver: Tool to archive websites and other content available on the Internet on the content-addressed S5 Network)
It currently only properly supports web archives, but you can check out this video for an early demo of archiving YouTube videos: https://www.youtube.com/watch?v=JEtFG3yLjWI&t=580
AT Protocol and SkyFeed
The original timeline for the S5 project included a full public identity system with a browser extension for secure client-side signatures, social network elements for the Tube5 video streaming app and a Discord-like decentralized chat platform.
Instead of continuing to build all the primitives and core tech for this myself, I decided to switch to https://atproto.com/. The AT Protocol offers almost all features needed to build truly decentralized applications, and it is already a lot more mature compared to what I could build from scratch. With ATProto, you own your identity, can easily switch between different services and provider without lock-in and there’s both composable curation (via custom feeds) and moderation.
The most relevant app on ATProto right now is Bluesky, which just hit 5 million users. It’s a microblogging app similar to Twitter.
Similar to how S5 works, the AT Protocol can support multiple different transport layers for social data, identity and media blobs. S5 can be one of them where it makes sense, like for a decentralized CDN or external backups for account recovery.
I built the SkyFeed client for Bluesky (https://skyfeed.app) which will make it easier to test and deploy new S5-powered features on the network in the future. The most used feature of SkyFeed is building custom feeds, of which there are already over 39000 ones: https://stats.skyfeed.me/. It also got featured in the bsky blog: Featured Community Project: Skyfeed | Bluesky
The query engine and other components of the Feed Builder can be self-hosted: GitHub - skyfeed-dev/query-engine: Query Engine used by custom feeds created using the SkyFeed Builder
I also started working on a bridge proxy to bridge content from other networks over (client-side): GitHub - skyfeed-dev/bridge-proxy
Ideas from the initial grant proposal which weren’t realized (yet) and why
Among the extensive quantity of things that I did implement and publish, there are also some ideas from my initial grant proposal which I didn’t work on yet for various reasons. So I would like to list them and briefly explain why that’s the case and how their future looks like:
Vup Chat (previously S5 Chat)
I tried the new OpenMLS standard for encrypted communication (https://openmls.tech/) using S5 as the transport and identity layer, but I simply didn’t have enough time to properly build it yet. So @Covalent will be working on this as part of his already approved grant: https://forum.sia.tech/t/small-grant-s5-ecosystem-vup-chat
SkyDroid (decentralized App Store for Android)
Since I created SkyDroid, a new decentralized App Store for Android appeared: Obtainium (GitHub - ImranR98/Obtainium: Get Android app updates straight from the source.). It’s written in Flutter+Dart and already decentralized by downloading updates directly from the developer’s source (for example GitHub releases or F-Droid). So there’s no longer a need for me to build yet another App Store, but I’m looking into extending Obtainium to support S5 as a decentralized CDN layer.
Build a simple browser extension for S5 identity management in all major browsers including Chrome
My work on bsky-related software made me realize that most users don’t care about this (yet), so it’s more important to focus on portable identity and truly secure (native) software first. A browser extension to make everything fully trustless on the web (in addition to native) can then still be added later when needed, when the network grows more and users want to migrate over.
Create high-level bindings for TypeScript/JavaScript
Due to the S5 specifications not being stable yet and changing quickly, writing libraries for TypeScript/JS didn’t make sense for me yet. However, there are already some third-party ones from @pcfreak30 for Lume’s use of S5, but I will write an official one after the v1 specs for S5 are done (see Milestones in my next grant).