S5 Network 2024 (Standard Grant Proposal)

Grants Approved
1

The current State of S5 (May 2024)

This will be my second grant proposal for the S5 Network, you can check out the first one here: Grant Proposal: S5 Network and Apps

This first post serves as an overview and “final progress report” for the progress made during the entire first grant. If you only care about the new grant, skip to the “Milestones and Timeline” section!

If you don’t know what S5 is yet, check out this guest blog post: Community Spotlight: S5. A Content-Addressed Data Routing… | by Skunk_Ink | The Sia Blog

Or watch the S5 segment in the recent Sia Community Town Hall: https://www.youtube.com/watch?v=JEtFG3yLjWI&t=580

S5 Node

The S5 Node is written in Dart+Rust and can be easily deployed on most systems using a Docker container. It provides an account system and APIs for uploading and managing small and big blobs. The node connects to the public S5 network (configurable) and stores data in one or multiple blob stores. Currently supported are Sia renterd, S3 (incl. renterd and many others), the local file system, IPFS and Pixeldrain. It can easily handle blobs with a size of 110 GiB each (that’s the largest I tested for now, bigger ones should work too with enough storage space). For renterd you can either use the native or the S3 store. The S3 one is slightly more reliable if you’re hosting the S5 Node on a different server than renterd and is easier to set up. The native renterd store is more efficient and slightly faster, but a bit more complex to set up.

The S5 Node also supports an experimental “File System Provider”. This is a “virtual” store which efficiently indexes any directory on your local FS and then makes it available in the S5-native directory metadata format directly on the network. It can for example be used for mirroring Linux package registries or any other use case where there’s already a good directory structure with a lot of large (or small) files. Some other fun new features include being able to import files from the internet by just passing their HTTP(s) URLs. The S5 Node also supports DNSLink (https://dnslink.dev/) to host websites and web apps on the S5 Network.

There’s also a second implementation of the S5 Node written in Golang by @pcfreak30 which I’m very excited about! I helped debug it and fixed some bugs in the S5 specification in the process.

GitHub Repository: GitHub - s5-dev/S5: Decentralized content-addressed storage network

Admin Web App

The Admin Web App (GitHub - s5-dev/node-admin-web-app: Simple Web App to manage accounts on a S5 Node using the Admin API. Your S5 Node serves this at /s5/admin/app) is a simple web app integrated in the S5 Node to manage accounts and their storage usage with a nice UI. (available on /s5/admin/app)

FS5

The S5 file system (FS5) is a mutable data structure built on top of S5 primitives which supports both public and private (end-to-end-encrypted) directories. It’s for example being used by Vup and will also be used by some other apps pretty soon (after the spec is stable). The FS is pretty efficient for most operations and only needs two of the S5 primitives (blobs and the registry). One of the more recent new features is Snapshots, which makes it possible to create immutable snapshots of whole directory trees instantly (if cached).

S5 Nodes have a public HTTP endpoint for browsing FS5 directories (downloads redirect, the S5 Node does NOT proxy file content!): Index of archlinux/

Vup Cloud Storage App

Vup is the flagship app of the S5 protocol and network. It’s a decentralized end-to-end-encrypted cloud storage app with support for many storage backends (including Sia) and apps for Android, Linux, Windows, macOS and iOS.

During my first grant, I published a massive new release which fully migrated Vup from the old Skynet backend and data structures to the S5 Network. I also added a video player, image gallery view, mass-pinning tool, new more efficient streaming implementation, video publishing dialog, fixed a bug which prevented browsing public directories (like the archlinux mirror), uploaded some TBs of small to medium-sized files to test scalability and implemented some UI improvements and bug fixes. Vup also runs a local S5 Node and supports configuring it with a self-hosted Sia renterd instance, but due to that abstracting some important parts away, users are advised to instead host a S5 Node alongside their renterd installation in the future (see Deploy S5 Node with Sia renterd - S5 Network Docs). With renterd getting more lightweight in the future (especially with reduced local state requirements), I could see shipping renterd as part of Vup and making setup as easy as possible.

Vup also has rclone and restic support, can do media streaming with an integrated Jellyfin-compatible API server, integrated sync, scripts and a lot more features! (check out the website)

Website: https://vup.app/

Source Code: GitHub - vup-app/vup

Available on Flathub: Vup Cloud Storage auf Linux installieren | Flathub

lib5 Dart

GitHub Repo: GitHub - s5-dev/lib5: This library is used by Dart-based code for the S5 network.

Packages: lib5 | Dart package and lib5_crypto_implementation_dart | Dart package

lib5 contains most of the S5 data structures and code. It’s designed to be modular and different parts like the API implementation or crypto provider (for hashing, signing, encryption) can be swapped out. It also runs on any platform supported by Dart, like Linux, Android, iOS, Windows, macOS and the Web!

It has classes to work with CIDs, Multihashes and S5 file metadata, registry entries, contains a full S5 Node implementation which can run anywhere, some utils and more!

In March, I wrote a new implementation of the S5 file system for lib5. It has improved support for public directory trees, comes with a new, more consistent URI format, integrates nicely with accounts and other S5 APIs and most importantly, does not contain any legacy code from the Skynet days anymore!

s5-dart

The s5 Dart package is intended for developers to easily add all S5 features to their Flutter or Dart app with minimal effort (compared to lib5, which is more low-level). It wraps the lib5 package and pre-configures almost everything for you, so it’s easier to integrate S5 with your app.

  • Runs a local S5 Node and connects to default nodes (can be configured)

  • Provides all S5 APIs (file, registry, streams) with full integrity verification

  • Supports creating and restoring user identities and uploading to third-part storage services (nodes)

Package: s5 | Dart package

Source Code: GitHub - s5-dev/s5-dart: Use the S5 Network in your Flutter and Dart apps

There’s also a Flutter-based demo web app showcasing some basic features for devs (still WIP):

https://demo.sfive.net/ (GitHub - s5-dev/flutter_demo_app: Flutter app which showcases some S5 features using the s5 package)

Web Proxy

The S5 Web Proxy can securely stream files of any size from the S5 network, directly in your web browser. It should work in all major web browsers, including Chrome, Brave, Firefox and Safari.

It works by registering and running a service worker which intercepts specific requests, looks up download URLs for them and then verifies their integrity with BLAKE3/bao running in Rust via WASM.

After adding the service worker to your web app, streaming files from the S5 network is as easy as using the URL /s5/blob/YOUR_CID_HERE?mediaType=image%2Fpng in img tags, for your video player or just fetching it directly! Encrypted files are supported too.

Source Code: GitHub - s5-dev/web-proxy: Client-side gateway with verified streaming, running in the web browser

Some example apps using this proxy:

  • s5.cx for very basic file sharing, for example encrypted ones: Encrypted file

  • Tube5 Alpha Simple decentralized video platform, uses the s5 web proxy for streaming video, audio, images and text.

  • a web-based map view, powered by the PMTiles format. (no longer online, previously on https://map-demo.sfive.cloud/)

CID Explorer

Web App for exploring and debugging S5 CIDs

Source Code: GitHub - s5-dev/cid-explorer: Web App for exploring and debugging S5 CIDs

Example: S5 CID Explorer

More examples: cid.one - S5 Network Docs

Documentation

Archiver

Archiving makes sense for personal archive use cases, for preserving history and satisfying your data hoarder needs. I’m also planning to support Twitch VODs due to them needing a ton of space and Twitch deleting past broadcasts after 7 days for free users and 60 days for paid ones by default, so streamers might be interested in having a good alternative.

I started working on an archiving tool to archive websites and other content available on the Internet on the content-addressed S5 Network. It downloads website files and stores them in an optimized FS5 directory structure (public or private). Can then be browsed like a personal web archive or used for a more coordinated global content-addressed web archiving effort. (Still WIP: GitHub - s5-dev/archiver: Tool to archive websites and other content available on the Internet on the content-addressed S5 Network)

It currently only properly supports web archives, but you can check out this video for an early demo of archiving YouTube videos: https://www.youtube.com/watch?v=JEtFG3yLjWI&t=580

AT Protocol and SkyFeed

The original timeline for the S5 project included a full public identity system with a browser extension for secure client-side signatures, social network elements for the Tube5 video streaming app and a Discord-like decentralized chat platform.

Instead of continuing to build all the primitives and core tech for this myself, I decided to switch to https://atproto.com/. The AT Protocol offers almost all features needed to build truly decentralized applications, and it is already a lot more mature compared to what I could build from scratch. With ATProto, you own your identity, can easily switch between different services and provider without lock-in and there’s both composable curation (via custom feeds) and moderation.

The most relevant app on ATProto right now is Bluesky, which just hit 5 million users. It’s a microblogging app similar to Twitter.

Similar to how S5 works, the AT Protocol can support multiple different transport layers for social data, identity and media blobs. S5 can be one of them where it makes sense, like for a decentralized CDN or external backups for account recovery.

I built the SkyFeed client for Bluesky (https://skyfeed.app) which will make it easier to test and deploy new S5-powered features on the network in the future. The most used feature of SkyFeed is building custom feeds, of which there are already over 39000 ones: https://stats.skyfeed.me/. It also got featured in the bsky blog: Featured Community Project: Skyfeed | Bluesky

The query engine and other components of the Feed Builder can be self-hosted: GitHub - skyfeed-dev/query-engine: Query Engine used by custom feeds created using the SkyFeed Builder

I also started working on a bridge proxy to bridge content from other networks over (client-side): GitHub - skyfeed-dev/bridge-proxy

Ideas from the initial grant proposal which weren’t realized (yet) and why

Among the extensive quantity of things that I did implement and publish, there are also some ideas from my initial grant proposal which I didn’t work on yet for various reasons. So I would like to list them and briefly explain why that’s the case and how their future looks like:

Vup Chat (previously S5 Chat)

I tried the new OpenMLS standard for encrypted communication (https://openmls.tech/) using S5 as the transport and identity layer, but I simply didn’t have enough time to properly build it yet. So @Covalent will be working on this as part of his already approved grant: https://forum.sia.tech/t/small-grant-s5-ecosystem-vup-chat

SkyDroid (decentralized App Store for Android)

Since I created SkyDroid, a new decentralized App Store for Android appeared: Obtainium (GitHub - ImranR98/Obtainium: Get Android app updates straight from the source.). It’s written in Flutter+Dart and already decentralized by downloading updates directly from the developer’s source (for example GitHub releases or F-Droid). So there’s no longer a need for me to build yet another App Store, but I’m looking into extending Obtainium to support S5 as a decentralized CDN layer.

Build a simple browser extension for S5 identity management in all major browsers including Chrome

My work on bsky-related software made me realize that most users don’t care about this (yet), so it’s more important to focus on portable identity and truly secure (native) software first. A browser extension to make everything fully trustless on the web (in addition to native) can then still be added later when needed, when the network grows more and users want to migrate over.

Create high-level bindings for TypeScript/JavaScript

Due to the S5 specifications not being stable yet and changing quickly, writing libraries for TypeScript/JS didn’t make sense for me yet. However, there are already some third-party ones from @pcfreak30 for Lume’s use of S5, but I will write an official one after the v1 specs for S5 are done (see Milestones in my next grant).

2

Grant Proposal: S5 Network 2024 - Finalize v1 specifications, update libraries, publish Vup v1, support third-party apps and archivers

Project Name: S5 Network

Developer: redsolver

Duration: 12 months

Introduction

As seen in the above “State of S5” section, there has been a ton of progress, but instead of making one app truly production-ready, it has been spread across a wide variety of projects. So while that has been very important to gain many valuable insights into ideal requirements and concepts for S5, I would like to switch to a more narrow, focused approach this year and really focus on only the important parts most users and devs care about first:

  • Stabilize the S5 specifications to accelerate adoption and improve performance+scalability

  • Make the Dart and TypeScript libraries production-ready and help devs re-implementing them in other programming languages like Golang

  • Make the flagship Vup App production-ready with a more professional user-friendly design, a web app for easy onboarding and support for collaborative productivity apps like document editing

  • Extend the archiver with many more supported sources and deploy a web archiving service powered by Sia

  • Some other nice-to-have features, like direct p2p connectivity between multiple Vup devices and native RHP4 support for streaming files directly from Sia hosts

Purpose of the grant; who benefits and how the project will serve the Foundation’s mission of user-owned data

S5 adds a content-addressed data routing layer and many other features on top of the Sia Network to make it easier for developers to use and implement decentralized storage in a wide variety of apps and use cases. The Vup Cloud Storage App is all about user-owned data and already provides a massive amount of features for users to efficiently manage large amounts of media files and other data securely on many devices. This grant will make the Vup app more stable and easier to use for an even larger user audience. S5 also aims to enrich and extend the wider ecosystem of content-addressed storage networks like IPFS and Iroh, and be interoperable with them where it makes sense.

Milestones and Timeline

Finish the S5 v1 specifications (June and July)

Design, define and write v1 specifications for the different S5 data structures and features:

  • Blobs, already done: Blobs - S5 Network Docs

  • Registry

  • Encryption

  • File System (including metadata formats)

  • Identity System

  • Streams

In July, I will attend IPFS Camp 2024 (https://2024.ipfs.camp/) for exchanging ideas with the wider IPFS ecosystem and getting feedback on the final S5 specs, to make them the best they can be.

Adapt the S5 Dart and Flutter libraries to the new v1 specifications (August)

In August and likely already earlier while working on the specs, I will adapt the existing Dart/Flutter libraries for S5 to use the new specifications. This will be very important for validating if everything works as expected and adding support for S5 to different apps like GitHub - LinwoodDev/Butterfly: 🎨 Powerful, minimalistic, cross-platform, opensource note-taking app (in collaboration with the developers), to see if the APIs are well-designed and the data structures efficient.

TypeScript Web Library (September)

Build a TypeScript/JS library based on spec v1 in collaboration with @pcfreak30 to ensure the spec is sound and make all S5 features available for the most popular development ecosystem (the web and native JS). The libraries will use Rust via WASM for performance-critical things like hashing, signing and encryption (just like in the Dart libraries).

The TypeScript library will be designed to “just work” (like the s5 dart package), so devs can easily add it to their app to get features like a decentralized end-to-end-encrypted file system (compatible with Vup) and real-time communication between multiple users.

Build Vup Web (October and November)

Using the new TypeScript libraries, build a lightweight web version of the Vup Cloud Storage app (fully compatible with the native version of course). This will also be the first implementation of the new user-friendly Vup UI designed by a professional designer.

Features

  • View and download shared directories and files (both public and encrypted)

  • Sign in with a S5 Identity

  • Upload files and manage the encrypted file system

  • Media Thumbnails

RHP4 support for S5 web apps (when available)

From https://sia.tech/roadmap:

RHP4 will also enable storage consumers to download and upload data directly in the browser without installing additional software.

When this feature lands in Sia, I will add it to the web-based and native S5 libraries (like TypeScript and Dart), to support more efficient direct downloads from the Sia Network.

Improvements to the S5 Node needed for Vup v1 (November)

  • Improved Pinning API (for example background mass-pinning)

  • More powerful Account and Admin APIs, to make managing larger nodes with many users easier

  • Performance and scalability improvements

Get Vup to version v1 (native) (December and January)

  • Update the native Vup App to match the new professional UI design of the web app

  • Migrate all data structures to the S5 specification v1 and the new libraries

  • Advanced two-way sync with file deletion and multi-device support

  • Advanced file sharing permission systems (for example read-only/write per file)

  • Add built-in audio player

  • Release on Linux, Windows, macOS, Android and iOS

Vup Collaborative Productivity apps (February)

Webxdc (https://webxdc.org/) is a standard to bring privacy-friendly web apps to messenger chats in a simple file format containing HTML5, CSS, JavaScript and other asset files. But apart from that use case, this is also the ideal format to provide many different collaborative web apps for Vup (Web and Native) and other S5-powered apps. S5 can both be used as the transport layer (instead of a chat room) for real-time collaboration like drawing on a whiteboard or editing text, and it can also store files you edited permanently in the S5 file system. There are already many Webxdc apps made for https://delta.chat and other messengers and most of them should just work with Vup+S5.

From the Webxdc specification:

All authentication, identity management, social discovery and message transport is outsourced to the host messenger which runs a webxdc app container file and relays application update messages between app users, letting each app inherit offline-first and end-to-end encryption capabilities implemented by the hosting messenger.

In this case, the Vup app is the “hosting messenger”!

Some use cases possible with this:

  • Select a text file in your Vup App and share it for end-to-end-encrypted collaborative editing with other users by simply sharing a link with them. They don’t need to install anything due to the app being web-based and just need to open the link in their browser and start editing with you! In regular intervals, your local Vup App will persist the changes to your encrypted file system.

  • Same with task list editing, whiteboard apps, calendars, board games and more! Any productivity app that runs on the web and could use a file system.

The biggest advantage of this approach over just implementing collaborative productivity apps/features myself in Vup is that it saves me a ton of work and also makes the whole platform easily extensible by third-party developers with new and exciting apps.

Vup P2P (March)

Add Iroh support (https://iroh.computer/) to Vup for keeping true p2p connectivity between multiple devices and users. (only trusted devices by default)

Use Cases

  • You want to stream a video file and one of your other devices already has it cached locally. Instead of needing to reach out to the Internet, your local Vup app will just connect directly to your other device and stream the file from there

  • If enabled, you can connect directly with other users you are editing a text file (or something else, see Webxdc above) with for improved latency and privacy (no server involved)

Archiver (April and May)

Archiving makes sense for personal archive use cases, for preserving history and satisfying your data hoarder needs. It’s also a fantastic way to onboard massive amounts of useful data to the network.

The existing S5 Archiver will be extended to support all the following data sources natively:

  • Websites

  • YouTube Videos

  • AT Protocol data repositories and blobs (for example the entirety of Bluesky data)

  • GIT repositories

  • Twitch VODs and media.ccc.de

After that’s done, I will deploy a large Sia storage cluster with multiple S5 Nodes. This archival infrastructure will then be used to archive massive amounts of Creative-Commons-licensed YouTube Videos, different websites and CommonCrawl archives. It will serve as both a scalability test of the software and network, and as a useful archive.

All parts of the archive with a public license that permits it (like Creative Commons), will be publicly browsable using a S5-powered web app.

Potential Risks

  • Milestones taking longer than expected to complete

  • Technical risks are pretty limited due to all milestones already being validated by a lot of research and found to be feasible. Nevertheless, there are always unexpected issues that could arise and result in new technical challenges causing some code and ideas to not work out as expected.

Budget

I’m requesting the following budget for 12 months of work:

  • 80k EUR for the full-time salary of redsolver for 12 months (inflation-adjusted)

  • 5k EUR for UI design work (especially for UI design in the Vup v1 native and web apps and developer documentation)

  • 5k EUR for infrastructure costs. This includes public S5 nodes (for bootstrapping) and a free-only S5 Node with a bit of storage for onboarding users and developers. It will also fund a big Sia cluster for the archival infrastructure coming later in the grant (up to 100 TB).

Total: 90,000 EUR

Development Information

All specifications, source code and documentation are hosted on s5-dev · GitHub and vup-app · GitHub

Monthly progress reports will be submitted in the forum.

3 Likes
3

I am in full support of this proposal and the protocol work is needed for me to get the newer S5 changes and revisions into Lume.

Additionally, Vup will be the best demonstration of private, decentralized cloud storage (a dropbox) for end users and to represent the Sia ecosystem.

1 Like
4

I remember the early versions of Vup and can’t really wait for it getting production ready and with a new professional design. As Derrick said, it’s the best possible demonstration we will have. Not only I am definitely in the support of this proposal, but if you are looking for some feedback to collect ideas for the new design or some QoL improvements, just send me a DM (on discord) anytime:)

1 Like
5

Excited to see the refined Vup V1 design and to see some integration with Iroh, +1 to this grant!

1 Like
6

I think having a dedicated budget to hire a designer is a great idea that should be done more in the future. These kind of open source developer spaces tend to attract great coders, but very little in terms of design work. Hence why apps like gimp… well look like they do.

In terms of the grant I’m totally in support of it, though I would question the viability of “productivity apps” in vup. The point of this grant is to be more focused and direct, but building a productivity app is a hell of a lot of work. Like a good spreadsheet app would take a year of work at minimum to build by itself, let alone all the other stuff you’ll be doing. So I’m wondering if this will be more of an integration with another app using vup as the carrier? Or something else?

But overall, yeah, good proposal.

1 Like
7

The whole point of supporting the Webxdc standard (instead of building custom productivity apps directly in Vup) is actually that I won’t need to build them myself! There’s already a long list of collaborative webxdc apps created by third-party developers and most of them should “just work” in Vup after I added the basic APIs: webxdc apps

1 Like
8

The push towards Web3 with the infrastructure, tools and maturity that S5 provides over Sia can’t come soon enough for my ecosystem. I am in full support of this proposal.

9

Thanks for your new proposal to your ongoing project with the Sia Foundation. The original S5 grant was well done and important for the Sia ecosystem, and the committee is excited to see the work continue. The objectives outlined for this year’s work are interesting and will complement the existing network components well.

We’ll need more clear dates on your milestones, as payments are now linked to reasonable progress towards them but we’ll work that out with you during onboarding for the grant.

We’ll reach out to you to get set up!

1 Like
10

S5 Progress Report (June)

What progress was made on your grant this month?

  • Continued researching and refining the final specs for S5 v1

  • Wrote an initial draft of the registry spec: docs/src/spec/registry.md at main · s5-dev/docs · GitHub

  • Actively worked with @Covalent to improve S5 libraries for the s5_deploy tool and fix bugs in lib5 (Dart), discussed and planned details of the S5 technology used in Vup Chat

  • Helped some other developers with questions about S5 and discussed plans for new features and data structures in the S5 v1 specs

Links to repos worked on this month:

What will you be working on next month?

  • Research, design, define and write v1 specifications for the different S5 data structures and features (Encryption, File System, Streams, Identity, Accounts)

  • Continue to stay in frequent exchange with developers building on S5

  • Attend IPFS Camp 2024 (https://2024.ipfs.camp/) for exchanging ideas with the wider IPFS ecosystem and getting feedback on the final S5 specs, to make sure they are the best they can be