S5 v1: Rewrite it in Rust (Large Grant Proposal)

redsolver · November 4, 2025, 7:27pm

Grant Milestone Progress Report (October) [revised]

What progress was made on your grant this month? Summarize your progress into a few sentences or bullet points.

Implemented automatic sharding of the S5 file system to more efficiently handle large directory trees
Added an option for configuring trusted nodes by their ed25519 iroh pubkey and syncing private s5 file systems with them
Added a config option for adding s5 nodes who are allowed to upload and pin data to specific blob stores and using that for remote blob upload and sync
For the FS, added snapshot export and merge functionality for syncing plaintext and encrypted directories
Implemented a remote blob store client which uses a remote s5 node over iroh as a blob store, but exposes the same API as local blob stores
Implemented a comprehensive and well-documented Rust test which sets up three S5 nodes (two trusted and one untrusted), configures the untrusted one to allow the other two nodes to upload and pin blobs, creates a directory and a few files on one of the trusted nodes and then correctly syncs it to the second trusted node via the untrusted node (end-to-end-encrypted). This test validates the common use case of users syncing data securely between their devices even if they are not both online at the same time

Milestone	Task	Commits	Additional Notes
Milestone 2025-10-25: E2EE FS sync over untrusted node	Implementation	d5566…857e6	Validation: Checkout commit `857e6e3` (as of now same as `main`) and run `cargo test -p s5_node --test fs_sync_complete`

What will you be working on next month?

Stable release of the s5 rust crates
Well-documented JavaScript and Dart bindings via UniFFI
Vup Web Migration to s5-rust

Link to an easy to test version or a demo video.

Run the test (which is the validation requirement for this milestone, see the section above):

git clone https://github.com/s5-dev/s5-rs
cd s5-rs
cargo test -p s5_node --test fs_sync_complete

mecsbecs · November 10, 2025, 4:10pm

This month’s review was successful and there is no actionable feedback. The team is looking forward to seeing what you do over the next month!

redsolver · November 26, 2025, 5:42am

Final Grant Progress Report: S5 Network

What progress was made on your grant this month?

This month I focused on polishing and releasing s5-rs crates for S5 v1.0, ensuring comprehensive documentation and robust testing.

Shipped v1.0.0-beta.1: Published the full suite of S5 crates to crates.io.
FUSE Integration: Implemented a functional FUSE driver (s5_fuse), allowing users to mount S5 file systems directly on their OS.
FS Internals: Added automatic sharding for large directories and Garbage Collection (GC) to s5_fs.
Core Refactor: Locked down the s5_core APIs (blob stores, importers, registry) for the v1 release.
Bindings & Vup Web: The JS/Dart bindings and Vup Web migration are now done!

Detail tasks worked on this month per milestone

Milestone	Task	Status	Commits	Notes
2025-11-25	Stable release of s5 rust crates	Completed	857e6e3…3d4515d	All crates published to crates.io.
2025-11-25	JS & Dart bindings via UniFFI	Completed	3d4515d…079b4ca	Published on npm and pub.dev. Using wasm-bindgen and flutter_rust_bridge directly instead of UniFFI because they are more reliable and mature
2025-11-25	Vup Web Migration to s5-rust	Completed	515b3c3	Needs a Remote S5 Node to work, see README for instructions

Link to an easy to test version or a demo video.

Rust Crates: https://crates.io/users/redsolver

(Includes s5_core, s5_fs, s5_node, s5_cli, s5_blobs, s5_registry, s5_fuse, and store backends)

Source Code: https://github.com/s5-dev/s5-rs
Quick Start: Install the CLI to try it out immediately:


cargo install --git https://github.com/s5-dev/s5-rs s5_cli

s5 --help

Testing: The workspace includes 22 integration tests and unit tests across all crates. Validate with:


cargo test --workspace

Provide an overall summary of everything achieved during this grant.

I completely re-architected and rewrote the S5 Network in Rust, moving from a prototype to a stable foundation.

Rust Rewrite: Delivered a modular implementation of the S5 protocol (s5_core, s5_node).
Iroh Integration: Replaced the legacy WebSocket protocol with Iroh for robust P2P networking and NAT traversal.
Advanced File System: Built s5_fs, a secure, end-to-end encrypted filesystem with snapshots and sharding.
CLI & Tooling: Built s5_cli, featuring the s5 tree command for inspecting data structures and s5_fuse for mounting.
Sia Integration: Shipped a renterd-backed s5_store_sia backend that exposes Sia as a first-class Store, including direct BlobLocation::SiaFile links so S5 clients or CDNs can download directly from Sia hosts.
Documentation: Wrote READMEs for all 14 crates, 4 end-to-end guides (workflows, friend/community storage, decentralized build cache), plus a full CLI and configuration reference.

See docs/guides/s5-workflows.md for detailed workflows.

If there were challenges you experienced and/or any work you weren’t able to complete, list it here.

WASM Integration: Generating bindings for the web (WASM) proved more challenging than expected due to tokio dependencies in s5_core and s5_blobs that don’t compile for wasm32-unknown-unknown. While iroh and irpc work in WASM, core crates depend on tokio, redb, and fs4 which are not WASM-compatible. I am currently refactoring these into a s5_client_core to isolate WASM-compatible logic, which caused a slight delay in the final bindings and Vup Web migration.

What lessons have you learned as a result of this grant?

Rust Reliability: Rust was the perfect choice for S5. Its type system and safety guarantees allowed me to build complex concurrent logic with confidence, resulting in a highly stable and performant system.
P2P: Leveraging Iroh was a better choice than maintaining a custom networking stack.
FFI: API design needs to account for FFI boundaries early on to avoid friction.

What are you most proud of about your work on this grant?

I’m really happy with how s5_fs turned out. It supports end-to-end encryption, efficient sync, and automatic sharding, but still feels fast and usable. Seeing it work with the FUSE driver and s5 tree command is a highlight.

mecsbecs · November 26, 2025, 2:43pm

Hi @redsolver - since certain items are still in progress, I’ll check back in a couple of days. Be sure to link to the specific task’s code in the detail for the past month.

Please note given the American Thanksgiving holiday the deadline for reports in order to allow adequate time for review is Dec. 1 at 10am ET.

redsolver · December 1, 2025, 3:00pm

Hi, I updated the progress report to link to the specific task’s code (commits) in the task detail table.

I’m aware that the deadline for reports is now, but unfortunately I’m still busy with the final git commit, this will be done in about 2 hours. I will update the progress report to be final with all in-progress items finished immediatly after that.

mecsbecs · December 1, 2025, 3:02pm

Please tag me when you do so we can get this review into the team’s queue.

redsolver · December 2, 2025, 3:45am

@mecsbecs Sorry for the delay, all milestone tasks are now finally completed and I updated the final progress report to be complete.

mecsbecs · December 4, 2025, 2:36pm

Hello @redsolver - congratulations, your final technical review was successful!

This grant will be marked as ‘Complete’ here on the Forum and you will be sent an offboarding email soon.

Thank you for continuing to build on Sia.