Standard Grant: S5 Network 2024

S5 Progress Report (September)

What progress was made on your grant this month?

  • Published the S5 v1 specifications on web (hosted on S5 via DNSLink): Blobs - S5 Network Docs
  • Updated some parts of the Dart library to match the S5 v1 specs
    • S5APIProvider base class (API Interface)
    • CryptoImplementation base class (API Interface)
    • New BlobCID (Blobs)
    • util/derive_hash.dart (Key Derivation)

Links to repos worked on this month:

What will you be working on next?

  • Build the S5 TypeScript/JS library based on the v1 specs, to enable developers building apps for the most popular development ecosystem (the web and native JS) to fully utilize all S5 features (also needed for the Vup Web app in November of this grant)
  • In parallel to building the TS lib, finish migrating the Dart lib to v1
  • Get feedback from other developers on the S5 specifications (including from other content-addressed network projects) and based on that make small changes to the specs if needed
  • Assist developers in adding the new S5 v1 Dart and Flutter libraries to their app, and eliminate remaining developer experience pain points in the process
2 Likes

S5 Progress Report (October)

What progress was made on your grant this month?

Started building the S5 TypeScript/JS library based on the v1 specs. Adapted some of the APIs to better match the web platform, like using Blob objects for large file uploads. Progress has been slower than expected, nevertheless the lib is a requirement for the Vup Web app, so it will be finished in a few weeks at most.

Links to repos worked on this month:

What will you be working on next?

  • Hire the professional designer for the Vup V1 UI, so it can already be worked on in parallel while I finish the parts of the S5 TS/JS library required for Vup Web
  • After that’s done, I’ll start working on Vup Web with the features S5 identity creation and recovery, storage service account management and using the decentralized file system (FS5)
  • Get developers to use the S5 v1 libs in their projects and collect feedback

Hello @redsolver

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

S5 Progress Report (November)

What progress was made on your grant this month?

  • Found and hired a professional designer for the new Vup web app and UI design. He’s currently busy working out a consistent design language and will soon start with implementing different components for the app directly in HTML+CSS using https://storybook.js.org/ Storybook makes it easy to collaborate because after the components are ready, I can directly use them in the web app together with the S5 TypeScript library
  • Got a new domain for the S5 project! s5.pro for S5 Protocol. The docs are already there: https://docs.s5.pro/ and a landing page on the main domain will be added soon. It replaces the previous sfive.net domain
  • Created a new blobsum command line utility for easily and efficiently calculating S5 Blob Identifiers (CIDs) of files on your system. It’s a fork of b3sum

Links to repos worked on this month:

What will you be working on next?

  • Continue working together with the designer on the new UI and components
  • Continue working on the S5 TS/JS library
  • After the first components are ready, start implementing all the different S5 file system features and the overall Vup web app (S5 identity creation and recovery, storage service accounts, …)
1 Like

Hello,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

S5 Progress Report (December)

What progress was made on your grant this month?

  • The designer has been busy working on different components, concepts/views and utilities for the Vup Web UI, while I gave some feedback. Vup Web is slightly behind schedule now, but should finally get a nice MVP ready this month with all basic file management features working

  • Bluesky now has over 26M users which also resulted in significantly more load on my feed infrastructure and kept me quite busy putting out fires. So this month I built a new powerful open-source indexer in Rust to keep up with the network growth and have less issues in the future. The new indexer will also play a crucial role in bridging the AT Protocol and Bluesky data and media over to the S5 protocol as part of the archiver milestone planned later in 2025 for this grant

  • Attended the 38th Chaos Communication Congress and had some interesting discussions there around S5, the AT Protocol and other projects building decentralized protocols and networks

Links to repos worked on this month:

What will you be working on next?

  • Use the first finished design components to build the Vup Web UI with all basic features implemented and working

  • Continue working on the S5 TS/JS library and implementing all v1 specs

  • Design a landing page for the S5 protocol which gives a quick overview over which features are possible with S5 v1 and onboards developers to use it in their project (to be deployed on https://s5.pro)

Hello,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Timeline Correction

The timeline listed in the second post of this thread is off by one month due to the grant taking a few weeks to be approved and onboarded, which resulted in the official timeline in the grant agreement being shifted by one month. Unfortunately I can’t edit the timeline in this thread anymore due to it being the second post (instead of the first), so instead I’m just posting the correct timeline here for reference:

Milestones and Timeline

Finish the S5 v1 specifications (2024-09-02)

Design, define and write v1 specifications for the different S5 data structures and features:

  • Blobs, already done: Blobs - S5 Network Docs
  • Registry
  • Encryption
  • File System (including metadata formats)
  • Identity System
  • Streams

In July, I will attend IPFS Camp 2024 (https://2024.ipfs.camp/) for exchanging ideas with the wider IPFS ecosystem and getting feedback on the final S5 specs, to make them the best they can be.

Adapt the S5 Dart and Flutter libraries to the new v1 specifications (2024-10-02)

In August and likely already earlier while working on the specs, I will adapt the existing Dart/Flutter libraries for S5 to use the new specifications. This will be very important for validating if everything works as expected and adding support for S5 to different apps like GitHub - LinwoodDev/Butterfly: 🎨 Powerful, minimalistic, cross-platform, opensource note-taking app (in collaboration with the developers), to see if the APIs are well-designed and the data structures efficient.

TypeScript Web Library (2024-11-02)

Build a TypeScript/JS library based on spec v1 in collaboration with @pcfreak to ensure the spec is sound and make all S5 features available for the most popular development ecosystem (the web and native JS). The libraries will use Rust via WASM for performance-critical things like hashing, signing and encryption (just like in the Dart libraries).

The TypeScript library will be designed to “just work” (like the s5 dart package), so devs can easily add it to their app to get features like a decentralized end-to-end-encrypted file system (compatible with Vup) and real-time communication between multiple users.

Build Vup Web (2025-01-02)

Using the new TypeScript libraries, build a lightweight web version of the Vup Cloud Storage app (fully compatible with the native version of course). This will also be the first implementation of the new user-friendly Vup UI designed by a professional designer.

Features

  • View and download shared directories and files (both public and encrypted)
  • Sign in with a S5 Identity
  • Upload files and manage the encrypted file system
  • Media Thumbnails

RHP4 support for S5 web apps (when available)

From https://sia.tech/roadmap:

RHP4 will also enable storage consumers to download and upload data directly in the browser without installing additional software.

When this feature lands in Sia, I will add it to the web-based and native S5 libraries (like TypeScript and Dart), to support more efficient direct downloads from the Sia Network.

Improvements to the S5 Node needed for Vup v1 (2025-01-02)

  • Improved Pinning API (for example background mass-pinning)
  • More powerful Account and Admin APIs, to make managing larger nodes with many users easier
  • Performance and scalability improvements

Get Vup to version v1 (native) (2025-03-02)

  • Update the native Vup App to match the new professional UI design of the web app
  • Migrate all data structures to the S5 specification v1 and the new libraries
  • Advanced two-way sync with file deletion and multi-device support
  • Advanced file sharing permission systems (for example read-only/write per file)
  • Add built-in audio player
  • Release on Linux, Windows, macOS, Android and iOS

Vup Collaborative Productivity apps (2025-04-02)

Webxdc (https://webxdc.org/) is a standard to bring privacy-friendly web apps to messenger chats in a simple file format containing HTML5, CSS, JavaScript and other asset files. But apart from that use case, this is also the ideal format to provide many different collaborative web apps for Vup (Web and Native) and other S5-powered apps. S5 can both be used as the transport layer (instead of a chat room) for real-time collaboration like drawing on a whiteboard or editing text, and it can also store files you edited permanently in the S5 file system. There are already many Webxdc apps made for https://delta.chat and other messengers and most of them should just work with Vup+S5.

From the Webxdc specification:

All authentication, identity management, social discovery and message transport is outsourced to the host messenger which runs a webxdc app container file and relays application update messages between app users, letting each app inherit offline-first and end-to-end encryption capabilities implemented by the hosting messenger.

In this case, the Vup app is the “hosting messenger”!

Some use cases possible with this:

  • Select a text file in your Vup App and share it for end-to-end-encrypted collaborative editing with other users by simply sharing a link with them. They don’t need to install anything due to the app being web-based and just need to open the link in their browser and start editing with you! In regular intervals, your local Vup App will persist the changes to your encrypted file system.
  • Same with task list editing, whiteboard apps, calendars, board games and more! Any productivity app that runs on the web and could use a file system.

The biggest advantage of this approach over just implementing collaborative productivity apps/features myself in Vup is that it saves me a ton of work and also makes the whole platform easily extensible by third-party developers with new and exciting apps.

Vup P2P (2025-05-02)

Add Iroh support (https://iroh.computer/) to Vup for keeping true p2p connectivity between multiple devices and users. (only trusted devices by default)

Use Cases

  • You want to stream a video file and one of your other devices already has it cached locally. Instead of needing to reach out to the Internet, your local Vup app will just connect directly to your other device and stream the file from there
  • If enabled, you can connect directly with other users you are editing a text file (or something else, see Webxdc above) with for improved latency and privacy (no server involved)

Archiver (2025-08-02)

Archiving makes sense for personal archive use cases, for preserving history and satisfying your data hoarder needs. It’s also a fantastic way to onboard massive amounts of useful data to the network.

The existing S5 Archiver will be extended to support all the following data sources natively:

  • Websites
  • YouTube Videos
  • AT Protocol data repositories and blobs (for example the entirety of Bluesky data)
  • GIT repositories
  • Twitch VODs and media.ccc.de

After that’s done, I will deploy a large Sia storage cluster with multiple S5 Nodes. This archival infrastructure will then be used to archive massive amounts of Creative-Commons-licensed YouTube Videos, different websites and CommonCrawl archives. It will serve as both a scalability test of the software and network, and as a useful archive.

All parts of the archive with a public license that permits it (like Creative Commons), will be publicly browsable using a S5-powered web app.

1 Like

S5 Progress Report (January)

What progress was made on your grant this month?

Links to repos worked on this month:

What will you be working on next?

  • Finish implementing full identity and account support in the S5 TypeScript library (currently ~70% done)

  • Continue working on Vup Web after the next components from the designer are available

  • Design a landing page for the S5 protocol which gives a quick overview over which features are possible with S5 v1 and onboards developers to use it in their project (to be deployed on https://s5.pro)

S5 Progress Report (February)

What progress was made on your grant this month?

  • Implemented pretty much all missing parts in the S5 TypeScript library
    • full identity implementation, with seed phrase generation, verification and recovery, key derivation
    • added identity-backed api, including upload, portal account registration and login
    • file system: added full identity support, plaintext upload, directory and file creation, home file system
    • registry write support
    • hidden db implementation for legacy compat with Vup portal accounts
    • added example script which mirrors all apps from the webxdc app store to a new directory in the s5 file system
  • My designer made solid progress on the Vup Web design components, quite a lot of new ones were added to the repository
  • Started sketching some ideas on how the S5 Protocol landing page could look like and speaking with @Julian who will work on it (I’ve been already working together with him in 2020 and 2021 for different Skynet apps (like for example the https://vup.app/ website), so really looking forward to doing it again).

Links to repos worked on this month:

What will you be working on next?

  • Add tests to the S5 TypeScript library in order to ensure everything is implemented correctly
  • Get the first proper release of Vup Web v1 out. The TypeScript library now already includes everything needed so that shouldn’t cause any delays, and all of the different components and designs are also finally coming together
  • In Vup Web, implement initial support for browsing and using webxdc apps (this is the first step for collaborative productivity apps)
  • Initiate work on the S5 landing page
  • Get the existing native Flutter-based Vup to version 1 and add features like advanced two-way sync (this milestone is now delayed due to Vup Web taking longer than expected)
1 Like

Due to poor time management on my part, I am unfortunately unable to publish my progress report for March on time today. I would like to apologize for this and will post it as soon as possible (no later than April 5th)

Hi @redsolver, thanks for the update. What’s the current status of your report?

S5 Progress Report (March)

What progress was made on your grant this month?

Links to repos worked on this month:

What will you be working on next?

  • Implement sharing in Vup Web, polish the user experience and add more features

  • In Vup Web, implement initial support for browsing and using webxdc apps (this milestone is now delayed)

  • Publish the new S5 landing page

  • Get the existing native Flutter-based Vup to version 1 and add features like p2p connectivity and advanced two-way sync

Hello,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

For reference, here are the current estimated completion dates for my milestones:

  • Collaborative Productivity apps (delayed to 2025-05-02)
  • Get Vup (Flutter) to version 1 (delayed to 2025-06-02)
  • Vup P2P (delayed to 2025-06-02)
  • Archiver (2025-08-02, still as planned)

This is not my progress report, the April report will be posted on 2025-05-02 as usual

S5 Progress Report (April)

What progress was made on your grant this month?

  • Published an initial version of the new S5 landing page on https://s5.pro

  • I attended the first ever AT Proto / Bluesky conference in Europe and gave a talk (redsolver at AHOY! European Social Web Day) about SkyFeed and its early beginnings in the Sia Skynet ecosystem. I also presented and discussed a new content-addressed architecture for the SkyFeed Indexer and AppView which can scale beyond the current (already massive) Bluesky network size by storing time-sharded snapshots of records on S5-compatible storage networks like Sia.

Links to repos worked on this month:

What will you be working on next?

Hello

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

@redsolver Just a quick note that we’re sending to all current grantees.

When submitting progress reports, especially ones that contain work towards your milestones, development work must be easy to access and review. Please directly link to work done, and if that report says “support added for feature x”, please link either to a branch or to a commit range showing the diff for that feature.

Moving forward, we will be assuming that progress that is not linked to directly was not made.

S5 Progress Report (May)

What progress was made on your grant this month?

My main priority this month was preparing the 1.0 release for all S5 libraries, because developers building on S5 need a stable release (in adddition to the v1 spec) to properly start integrating S5 in their production apps and my other delayed milestones (collaborative productivity apps, Vup Native v1 and Vup P2P) all depend on these libraries supporting v1 features like stream messages, new identifier formats, improved data structures, and latest dependencies.

What will you be working on next?

  • Publish 1.0 versions for all S5 libraries and help developers integrate S5 into their apps

  • Implement sharing in Vup Web, polish the user experience and add more features

  • In Vup Web, implement initial support for browsing and using webxdc apps

  • Use the now stable v1 streams API to implement the collavorative web apps feature

  • Get the existing native Flutter-based Vup to version 1 and add features like p2p connectivity and advanced two-way sync

1 Like

Final Progress Report

This grant now comes to an end, but the S5 development velocity is not slowing down and I expect a complete stable v1 release of the new rust library, s5.js and s5-dart/flutter in about 2 months, including all the nice long-awaited RHP4 features!

Unfortunately one of my “potential risks” came true, and some milestones took longer than expected to complete, especially Vup Web with the typescript library. But first, let’s quickly cover what I did the last month (June):

Rust Rewrite started

For more details about this, please check out my new grant which I’ll submit shortly after this final progress report.

This month, I started implementing all S5 specs in Rust. There are many good reasons for this:

  • Rust is very easy to integrate in both native (ffi) and web-based applications via WASM

  • Many other content-addressed data projects like Iroh use Rust, so integrating them is trivial. For example, p2p quic connections between s5 nodes or content discovery will be handled by Iroh’s well-tested and optimized implementation, improving the overall reliability and saving me time by being able to re-use their very nice local blake3 blob store implementation and blake3 bao tree verification

  • There is excellent tooling for bridging Rust code with Flutter/Dart available, so long-term I won’t need to maintain that additional full s5 implementation

  • Rust is performant, safe and great for building distributed applications.

As an example, I implemented direct RHP4 streaming from Sia hosts in Rust, and could simply compile it to WASM and use it in any web browser: https://rhp4-webtransport-demo.sia5.net/

Uploading a file to Sia is also pretty easy (and efficient!) with s5-rs:

let store = SiaBlobStore::new(
  "s5-blobs", // bucket name
  "http://localhost:9980/api/worker",
  "http://localhost:9980/api/bus",
  "password",
);
let path = "path/to/my/file.mp4";
let (hash, size) = store.import_file(path.into()).await?;
let bid = BlobId::new(hash, size);

And I published a little command line tool to quickly calculate S5 blob identifiers for local files: https://crates.io/crates/blobsum

Milestones Review of the S5 2024 Grant

Now that it’s over, let’s go through all milestones I initially planned for the 2024 grant and see how they went.

RHP4 support for S5 web apps (completed)

From https://sia.tech/roadmap:

RHP4 will also enable storage consumers to download and upload data directly in the browser without installing additional software.

Here we go: https://rhp4-webtransport-demo.sia5.net/

source code: https://github.com/s5-dev/s5-rs/tree/main/s5_experiments/sia_rhp4_webtransport_streamer and GitHub - s5-dev/s5-rs

Finish the S5 v1 specifications (mostly completed)

diff: Comparing 2994b10848e3500e26108b473be3020be7d06fb7..main · s5-dev/docs · GitHub

The following specs are final:

The following specs are still subject to minor changes:

The file system spec will be finalized after the Rust implementation is done

The media and web app metadata formats did not get a spec, because they are now part of the new file system directory format.

Adapt the S5 Dart and Flutter libraries to the new v1 specifications (completed)

link: GitHub - s5-dev/lib5: This library is used by Dart-based code for the S5 network.

diff: Comparing 994ab52855ae1fd7894f2d2aa04824caa68ab32d..main · s5-dev/lib5 · GitHub

TypeScript Web Library (completed)

link: GitHub - s5-dev/s5.js: TypeScript/JS Library for S5

Build Vup Web (MVP released)

initial release available on https://web-preview.vup.app/

source code: GitHub - vup-app/vup-web

Vup Collaborative Productivity apps (work in progress)

  • full mirror of the webxdc app store to the s5 file system, in Rust and TypeScript

  • finalized low-level s5 streams specification used for p2p real-time messages

  • serve webxdc apps from the s5_web_proxy service worker client side (currently being re-written in Rust, almost done)

  • integrate with the vup web app UI and usere file system

Improvements to the S5 Node needed for Vup v1 (delayed)

I did a few updates for the lib5 node implementation and the full s5 node, and it supports pretty much all APIs needed by Vup right now.

However, the new pinning API was not implemented yet and is now being specific in the S5 Rust implementation I started working on.

Get Vup to version v1 (native) and Vup P2P (delayed)

Due to other milestones being delayed, I couldn’t start working on this one.

Vup Native already uses Rust bindings for performance-critical functions like encryption and hashing,

so the new full S5 implementation in Rust should simplify updating Vup and result in significant performance improvements along the way.

With the new Rust lib, Vup will also get Iroh p2p connectivity support automatically due to the S5 Rust lib being interleaved with Iroh.

Archiver (delayed)

Due to other milestones being delayed, I couldn’t finish this one either.