Standard Grant: S5 Network 2024

Grants Complete
35

Final Progress Report

This grant now comes to an end, but the S5 development velocity is not slowing down and I expect a complete stable v1 release of the new rust library, s5.js and s5-dart/flutter in about 2 months, including all the nice long-awaited RHP4 features!

Unfortunately one of my “potential risks” came true, and some milestones took longer than expected to complete, especially Vup Web with the typescript library. But first, let’s quickly cover what I did the last month (June):

Rust Rewrite started

For more details about this, please check out my new grant which I’ll submit shortly after this final progress report.

This month, I started implementing all S5 specs in Rust. There are many good reasons for this:

  • Rust is very easy to integrate in both native (ffi) and web-based applications via WASM

  • Many other content-addressed data projects like Iroh use Rust, so integrating them is trivial. For example, p2p quic connections between s5 nodes or content discovery will be handled by Iroh’s well-tested and optimized implementation, improving the overall reliability and saving me time by being able to re-use their very nice local blake3 blob store implementation and blake3 bao tree verification

  • There is excellent tooling for bridging Rust code with Flutter/Dart available, so long-term I won’t need to maintain that additional full s5 implementation

  • Rust is performant, safe and great for building distributed applications.

As an example, I implemented direct RHP4 streaming from Sia hosts in Rust, and could simply compile it to WASM and use it in any web browser: https://rhp4-webtransport-demo.sia5.net/

Uploading a file to Sia is also pretty easy (and efficient!) with s5-rs:

let store = SiaBlobStore::new(
  "s5-blobs", // bucket name
  "http://localhost:9980/api/worker",
  "http://localhost:9980/api/bus",
  "password",
);
let path = "path/to/my/file.mp4";
let (hash, size) = store.import_file(path.into()).await?;
let bid = BlobId::new(hash, size);

And I published a little command line tool to quickly calculate S5 blob identifiers for local files: crates.io: Rust Package Registry

Milestones Review of the S5 2024 Grant

Now that it’s over, let’s go through all milestones I initially planned for the 2024 grant and see how they went.

RHP4 support for S5 web apps (completed)

From https://sia.tech/roadmap:

RHP4 will also enable storage consumers to download and upload data directly in the browser without installing additional software.

Here we go: https://rhp4-webtransport-demo.sia5.net/

source code: s5-rs/s5_experiments/sia_rhp4_webtransport_streamer at main · s5-dev/s5-rs · GitHub and GitHub - s5-dev/s5-rs · GitHub

Finish the S5 v1 specifications (mostly completed)

diff: Comparing 2994b10848e3500e26108b473be3020be7d06fb7..main · s5-dev/docs · GitHub

The following specs are final:

The following specs are still subject to minor changes:

The file system spec will be finalized after the Rust implementation is done

The media and web app metadata formats did not get a spec, because they are now part of the new file system directory format.

Adapt the S5 Dart and Flutter libraries to the new v1 specifications (completed)

link: GitHub - s5-dev/lib5: This library is used by Dart-based code for the S5 network. · GitHub

diff: Comparing 994ab52855ae1fd7894f2d2aa04824caa68ab32d..main · s5-dev/lib5 · GitHub

TypeScript Web Library (completed)

link: GitHub - s5-dev/s5.js: TypeScript/JS Library for S5 · GitHub

Build Vup Web (MVP released)

initial release available on https://web-preview.vup.app/

source code: GitHub - vup-app/vup-web · GitHub

Vup Collaborative Productivity apps (work in progress)

  • full mirror of the webxdc app store to the s5 file system, in Rust and TypeScript

  • finalized low-level s5 streams specification used for p2p real-time messages

  • serve webxdc apps from the s5_web_proxy service worker client side (currently being re-written in Rust, almost done)

  • integrate with the vup web app UI and usere file system

Improvements to the S5 Node needed for Vup v1 (delayed)

I did a few updates for the lib5 node implementation and the full s5 node, and it supports pretty much all APIs needed by Vup right now.

However, the new pinning API was not implemented yet and is now being specific in the S5 Rust implementation I started working on.

Get Vup to version v1 (native) and Vup P2P (delayed)

Due to other milestones being delayed, I couldn’t start working on this one.

Vup Native already uses Rust bindings for performance-critical functions like encryption and hashing,

so the new full S5 implementation in Rust should simplify updating Vup and result in significant performance improvements along the way.

With the new Rust lib, Vup will also get Iroh p2p connectivity support automatically due to the S5 Rust lib being interleaved with Iroh.

Archiver (delayed)

Due to other milestones being delayed, I couldn’t finish this one either.

1 Like