S5 v1: Rewrite it in Rust (Large Grant Proposal)

Grants Complete
6

Progress Report (July)

What progress was made on your grant this month?

Continued implementing all the basic S5 specs in Rust:

  • Monorepo for all s5-rs crates: GitHub - s5-dev/s5-rs

  • Added Sia Blob Store, using the new PinnedObject API endpoint to extract the file encryption key (feat: add sia blob store · s5-dev/s5-rs@8903243 · GitHub)

  • Decided to use Iroh for all p2p communication between S5 Nodes, including web browsers, to simplify implementations (see iroh 0.33.0 - Browsers and Discovery and 0-RTT, oh my! - Iroh for Iroh browser support). As such, there’s no need to have a custom WebSocket-based protocol for S5 anymore. S5 will run some public Iroh relay nodes (as mentioned in the grant proposal and budget), to improve connectivity for all users.

  • Tested importing files to the S5 Sia Blob Store from HTTP mirrors using rclone

Prepare to make the RHP4 WASM-based blob proxy ready for production use:

What will you be working on next?

  • Highest priority: Add networking using Iroh for both native and web

  • Implement S5 Registry and Streams protocols, I delayed this until Iroh is added because they will be implemented as Iroh protocols (one Iroh p2p connection can transport any number of protocols in parallel)

  • RHP4 web proxy: add s5 networking using Iroh for dynamic blob location requests, add http streaming support in addition to rhp4 as a fallback for older browsers (and Safari)

  • Make the FS APIs more ergonomic, so developers can just call something like “fs.write(path, bytes)” and having it take care of blob store upload and pinning, instead of having to work with the FS5 directory data structure directly

1 Like
7

Hello,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

8

Progress Report (August)

What progress was made on your grant this month?

Delayed Milestones

  • The RHP4 proxy is not “production-ready” yet, but all server-side parts are now fully finished and the only thing missing is adding the s5_blobs protocol client to it and making it easier to add the proxy to a custom web app

  • s5-rs is missing complete implementations of the registry, streams, accounts and identity specs. This delay is due to them depending on the Iroh protocol API and networking which I just recently added for the s5_blobs protocol

  • The s3 store is not finished, but due to there being a completely custom one purpose-built for Sia, this is less relevant than initially planned

Adding tests to all crates is planned for the 2025-10-02 milestone, as written in the original grant proposal

What will you be working on next?

  • RHP4 web proxy: add s5 networking using Iroh for dynamic blob location requests, add http streaming support in addition to rhp4 as a fallback for older browsers (and Safari). Make it easy for devs to configure and add to their app. write documentation

  • Implement the missing specs in s5-rs

  • Implement end-to-end-encryption for the S5 file system

  • Add tests to all S5 rust crates

  • Make sure that all parts of the s5.js TypeScript are properly migrated to be compatible with s5-rs and S5 v1

1 Like
9

Hello

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

10

Timeline Update

The “Make the RHP4 WASM-based blob proxy production ready” component has been moved from Milestone #1 to Milestone #4, due on 2025-11-02. The reason for this is that the current implementation based on renterd is in a working MVP state, but the upcoming indexd architecture fits S5 (and many other use cases) significantly better. So it makes more sense to wait until indexd is available for testing, instead of polishing up the current (soon-to-be-deprecated) implementation.

1 Like
11

Progress Report (September)

What progress was made on your grant this month?

Milestone Tasks Progress

Can be validated by running cargo test in the s5_fs subdirectory

Can be validated by running cargo test in the s5_core subdirectory

Other process

Remaining Tasks

  • Migrate the TypeScript library to the new file system

Jules has been busy with migrating most of s5.js to the new CBOR-based directory metadata structures, in October we will work together on upstreaming his changes to the official s5.js repository (see GitHub - julesl23/s5.js: TypeScript/JS Library for S5)

  • specs impl: accounts, identity

instead of implementing a completely custom identity solution, s5 now simply uses iroh ed25519 public keys for node identity and will likely use atproto’s did:plc for public identities covering multiple nodes.

What will you be working on next?

  • Work with Jules on upstreaming his changes to s5.js so they are properly migrated to be compatible with s5-rs and S5 v1

  • Extending the config format to allow configuring trusted third-party nodes and upload+pinning targets

  • Implement Sia RHP4 uploads and downloads in S5 using indexd

  • Make the RHP4 WASM-based blob proxy production ready, so devs can easily use files streamed directly from Sia hosts in their web apps

  • Release a stable v1 version of all S5 libraries and code

1 Like
12

Hello @redsolver,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

13

Grant Update Proposal

Reasons

  • Remove the indexd-dependent milestone (and one monthly payment) from this grant due to it not being available yet, blocking RHP4 uploads/downloads and the productionization of the WASM blob proxy. I will submit a small, focused grant for indexd integration only once it’s available (target November).

  • Align reporting with the new deadline: progress updates by the 25th each month

  • Clarify validation requirements for milestone completion

Revised milestones

2025-10-25

  • Allow configuring trusted nodes by their ed25519 iroh pubkey and syncing private s5 file systems with them

  • Add a config option for adding s5 nodes who are allowed to upload and pin data to specific blob stores and using that for remote blob upload and sync functionaility

Validation Requirements for Milestone Completion

  • Comprehensive and well-documented Rust test which sets up three S5 nodes (two trusted and one untrusted), configures the untrusted one to allow the other two nodes to upload and pin blobs, creates a directory and a few files on one of the trusted nodes and then correctly syncs it to the second trusted node via the untrusted node (end-to-end-encrypted). This test validates the common use case of users syncing data securely between their devices even if they are not both online at the same time

2025-11-25

  • Stable release of the s5 rust crates

  • Well-documented JavaScript and Dart bindings via UniFFI

  • Vup Web Migration to s5-rust

Validation Requirements for Milestone Completion

  • All s5 rust crates published on crates.io with documentation for all public API methods

  • JavaScript bindings to s5 rust published on npm

  • Dart bindings to s5 rust published on pub.dev

  • Vup Web migrated to s5-rust with the following features working: account creation/login, creating directories, uploading files to remote s5 nodes, downloading files (all end-to-end-encrypted)

1 Like
14

Hi @redsolver - thank you for detailing the changes above. Confirming this means you are okay with receiving your final payment on December 15th with your final report submitted November 25th now instead of October 25th? This will be due to the fact that the final payment for the grant will be issued after the final successful technical review.

I imagine you’ve detailed a slightly longer duration for Milestone #4 (your new final milestone) because the additional time is needed, and there are no issues with that on our side.

15

Yes, I’m okay with receiving my final payment on December 15th and no payment on November 15th.

1 Like
16

Sorry for the delayed progress report, I’m trying my best to get it out today.

1 Like
17

Hello @redsolver,

Please have your progress report submitted be EOD today.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

18

Hi @redsolver - this is the final notice for your October progress report.

If your full and complete report is not posted by 9am ET tomorrow (November 4th), you will have to miss this payment cycle.

19

Progress Report (October)

I would like to apologize for the delayed report and thank you for your understanding.

Milestone 2025-10-25 completed

Validation: Clone latest repo and run cargo test -p s5_node --test fs_sync_complete

Commits implementing the milestone: Comparing d5566096102f797fe201ffc4df19a3c5814afaf7...857e6e38e617c50f67037945b8cc17b0f41b65e4 · s5-dev/s5-rs · GitHub

Next Up: Working on the 2025-11-25 milestone

20

Hi @redsolver, this report is greatly lacking in detail and does not follow the Development Guide format.

Please revise the report to match the correct format by 5pm ET.

1 Like
21

Grant Milestone Progress Report (October) [revised]

What progress was made on your grant this month? Summarize your progress into a few sentences or bullet points.

  • Implemented automatic sharding of the S5 file system to more efficiently handle large directory trees
  • Added an option for configuring trusted nodes by their ed25519 iroh pubkey and syncing private s5 file systems with them
  • Added a config option for adding s5 nodes who are allowed to upload and pin data to specific blob stores and using that for remote blob upload and sync
  • For the FS, added snapshot export and merge functionality for syncing plaintext and encrypted directories
  • Implemented a remote blob store client which uses a remote s5 node over iroh as a blob store, but exposes the same API as local blob stores
  • Implemented a comprehensive and well-documented Rust test which sets up three S5 nodes (two trusted and one untrusted), configures the untrusted one to allow the other two nodes to upload and pin blobs, creates a directory and a few files on one of the trusted nodes and then correctly syncs it to the second trusted node via the untrusted node (end-to-end-encrypted). This test validates the common use case of users syncing data securely between their devices even if they are not both online at the same time
Milestone Task Commits Additional Notes
Milestone 2025-10-25: E2EE FS sync over untrusted node Implementation d5566…857e6 Validation: Checkout commit 857e6e3 (as of now same as main) and run cargo test -p s5_node --test fs_sync_complete

What will you be working on next month?

  • Stable release of the s5 rust crates
  • Well-documented JavaScript and Dart bindings via UniFFI
  • Vup Web Migration to s5-rust

Link to an easy to test version or a demo video.

Run the test (which is the validation requirement for this milestone, see the section above):

  1. git clone https://github.com/s5-dev/s5-rs
  2. cd s5-rs
  3. cargo test -p s5_node --test fs_sync_complete
1 Like
22

This month’s review was successful and there is no actionable feedback. The team is looking forward to seeing what you do over the next month!

2 Likes
23

Final Grant Progress Report: S5 Network

What progress was made on your grant this month?

This month I focused on polishing and releasing s5-rs crates for S5 v1.0, ensuring comprehensive documentation and robust testing.

  • Shipped v1.0.0-beta.1: Published the full suite of S5 crates to crates.io.

  • FUSE Integration: Implemented a functional FUSE driver (s5_fuse), allowing users to mount S5 file systems directly on their OS.

  • FS Internals: Added automatic sharding for large directories and Garbage Collection (GC) to s5_fs.

  • Core Refactor: Locked down the s5_core APIs (blob stores, importers, registry) for the v1 release.

  • Bindings & Vup Web: The JS/Dart bindings and Vup Web migration are now done!

Detail tasks worked on this month per milestone

Milestone Task Status Commits Notes
2025-11-25 Stable release of s5 rust crates Completed 857e6e3…3d4515d All crates published to crates.io.
2025-11-25 JS & Dart bindings via UniFFI Completed 3d4515d…079b4ca Published on npm and pub.dev. Using wasm-bindgen and flutter_rust_bridge directly instead of UniFFI because they are more reliable and mature
2025-11-25 Vup Web Migration to s5-rust Completed 515b3c3 Needs a Remote S5 Node to work, see README for instructions

Link to an easy to test version or a demo video.

(Includes s5_core, s5_fs, s5_node, s5_cli, s5_blobs, s5_registry, s5_fuse, and store backends)


cargo install --git https://github.com/s5-dev/s5-rs s5_cli

s5 --help
  • Testing: The workspace includes 22 integration tests and unit tests across all crates. Validate with:

cargo test --workspace

Provide an overall summary of everything achieved during this grant.

I completely re-architected and rewrote the S5 Network in Rust, moving from a prototype to a stable foundation.

  • Rust Rewrite: Delivered a modular implementation of the S5 protocol (s5_core, s5_node).

  • Iroh Integration: Replaced the legacy WebSocket protocol with Iroh for robust P2P networking and NAT traversal.

  • Advanced File System: Built s5_fs, a secure, end-to-end encrypted filesystem with snapshots and sharding.

  • CLI & Tooling: Built s5_cli, featuring the s5 tree command for inspecting data structures and s5_fuse for mounting.

  • Sia Integration: Shipped a renterd-backed s5_store_sia backend that exposes Sia as a first-class Store, including direct BlobLocation::SiaFile links so S5 clients or CDNs can download directly from Sia hosts.

  • Documentation: Wrote READMEs for all 14 crates, 4 end-to-end guides (workflows, friend/community storage, decentralized build cache), plus a full CLI and configuration reference.

See docs/guides/s5-workflows.md for detailed workflows.

If there were challenges you experienced and/or any work you weren’t able to complete, list it here.

  • WASM Integration: Generating bindings for the web (WASM) proved more challenging than expected due to tokio dependencies in s5_core and s5_blobs that don’t compile for wasm32-unknown-unknown. While iroh and irpc work in WASM, core crates depend on tokio, redb, and fs4 which are not WASM-compatible. I am currently refactoring these into a s5_client_core to isolate WASM-compatible logic, which caused a slight delay in the final bindings and Vup Web migration.

What lessons have you learned as a result of this grant?

  • Rust Reliability: Rust was the perfect choice for S5. Its type system and safety guarantees allowed me to build complex concurrent logic with confidence, resulting in a highly stable and performant system.

  • P2P: Leveraging Iroh was a better choice than maintaining a custom networking stack.

  • FFI: API design needs to account for FFI boundaries early on to avoid friction.

What are you most proud of about your work on this grant?

I’m really happy with how s5_fs turned out. It supports end-to-end encryption, efficient sync, and automatic sharding, but still feels fast and usable. Seeing it work with the FUSE driver and s5 tree command is a highlight.

1 Like
24

Hi @redsolver - since certain items are still in progress, I’ll check back in a couple of days. Be sure to link to the specific task’s code in the detail for the past month.

Please note given the American Thanksgiving holiday the deadline for reports in order to allow adequate time for review is Dec. 1 at 10am ET.

1 Like
25

Hi, I updated the progress report to link to the specific task’s code (commits) in the task detail table.

I’m aware that the deadline for reports is now, but unfortunately I’m still busy with the final git commit, this will be done in about 2 hours. I will update the progress report to be final with all in-progress items finished immediatly after that.