S5 v1: Rewrite it in Rust (Large Grant Proposal)

[redsolver]

Progress Report (August)

What progress was made on your grant this month?

• Added a guide for compiling the CLI, configuring a local node with a renterd instance and importing files from a http mirror: https://github.com/s5-dev/s5-rs/blob/main/guides/sia-web-cdn.md (docs: add sia-web-cdn guide · s5-dev/s5-rs@bf672e1 · GitHub)

• Created s5_cli for importing data to S5 stores and serving it using a unified config file format (feat: add s5_cli · s5-dev/s5-rs@1cc4154 · GitHub)

• Added s5_node crate (feat: add s5_node crate · s5-dev/s5-rs@a5647c1 · GitHub)

• Completely refactored the store type system to be easier to use and not require generic type parameters anymore. All stores are now dyn-compatible. Added local store implementation and started working on s5 store. Improved the API for interacting with a FS5 directory to make it more ergonomic to modify directories (feat!: refactor blob store types, make it dyn-compatible, add local s… · s5-dev/s5-rs@bc2cb3b · GitHub)

• Added the http_importer for efficiently importing any data hosted over http(s) somewhere to any s5 store like for example renterd (feat: add http_importer tool · s5-dev/s5-rs@a00f093 · GitHub)

• Improvements to the Sia store, like using the pinned api object endpoint (feat(blob_stores/sia): migrate to pinned object api endpoint · s5-dev/s5-rs@22fbd47 · GitHub), checking renter settings to be compatible with S5, detecting the zen network and dynamically fetching quic addresses of hosts

• Added basic README (docs: add README file · s5-dev/s5-rs@45bd612 · GitHub)

• Added encrypted p2p networking using Iroh and implemented the s5_blobs protocol for serving files from a blob store like Sia (feat: add s5_blobs protocol · s5-dev/s5-rs@f344ae5 · GitHub)

• Quite a few fixes and performance improvements, see Commits · s5-dev/s5-rs · GitHub for details

Delayed Milestones

• The RHP4 proxy is not “production-ready” yet, but all server-side parts are now fully finished and the only thing missing is adding the s5_blobs protocol client to it and making it easier to add the proxy to a custom web app

• s5-rs is missing complete implementations of the registry, streams, accounts and identity specs. This delay is due to them depending on the Iroh protocol API and networking which I just recently added for the s5_blobs protocol

• The s3 store is not finished, but due to there being a completely custom one purpose-built for Sia, this is less relevant than initially planned

Adding tests to all crates is planned for the 2025-10-02 milestone, as written in the original grant proposal

What will you be working on next?

• RHP4 web proxy: add s5 networking using Iroh for dynamic blob location requests, add http streaming support in addition to rhp4 as a fallback for older browsers (and Safari). Make it easy for devs to configure and add to their app. write documentation

• Implement the missing specs in s5-rs

• Implement end-to-end-encryption for the S5 file system

• Add tests to all S5 rust crates

• Make sure that all parts of the s5.js TypeScript are properly migrated to be compatible with s5-rs and S5 v1