I would like to see a commitment from S5 on no more BC (backwards compatibility) breaking spec changes, at the application layer, or the networking layer by the end of this grant.
That’s the primary intention of this grant; by the end of this year there will be a complete S5 1.0.0 stable release with no further breaking changes. So all releases after 1.0.0 will be fully backwards compatible with 1.0.0, especially in regard to identifier formats for blobs and directories, as well as all data structures.
And if you are going to go and make changes at the scale discussed, I don’t think there is a point in keeping the websocket interface, as it is tech debt that will be a liability in a year or so.
Unfortunately the WebSocket interface will likely stay around (although using the new p2p protocol) because not all web browsers support WebTransport yet. But it’s only used as a generic transport layer, so it shouldn’t really cause much trouble.
As I have been working with the current version of s5 for the past couple months, I can say this would be a welcomed update for me. Both having a stable api and better node performance. My app is mobile only so reducing load on the system is rather important.
@redsolver Thanks for your proposal to The Sia Foundation Grants Program.
After review, the committee has decided to approve your proposal. Congratulations! They’re excited to see what you can accomplish with this grant.
We’ll reach out to your provided email address for onboarding. This shouldn’t take long unless your info has changed from last time, but you may still need to adjust your timelines.
Decided to use Iroh for all p2p communication between S5 Nodes, including web browsers, to simplify implementations (see iroh 0.33.0 - Browsers and Discovery and 0-RTT, oh my! - Iroh for Iroh browser support). As such, there’s no need to have a custom WebSocket-based protocol for S5 anymore. S5 will run some public Iroh relay nodes (as mentioned in the grant proposal and budget), to improve connectivity for all users.
Tested importing files to the S5 Sia Blob Store from HTTP mirrors using rclone
Prepare to make the RHP4 WASM-based blob proxy ready for production use:
Highest priority: Add networking using Iroh for both native and web
Implement S5 Registry and Streams protocols, I delayed this until Iroh is added because they will be implemented as Iroh protocols (one Iroh p2p connection can transport any number of protocols in parallel)
RHP4 web proxy: add s5 networking using Iroh for dynamic blob location requests, add http streaming support in addition to rhp4 as a fallback for older browsers (and Safari)
Make the FS APIs more ergonomic, so developers can just call something like “fs.write(path, bytes)” and having it take care of blob store upload and pinning, instead of having to work with the FS5 directory data structure directly
Completely refactored the store type system to be easier to use and not require generic type parameters anymore. All stores are now dyn-compatible. Added local store implementation and started working on s5 store. Improved the API for interacting with a FS5 directory to make it more ergonomic to modify directories (feat!: refactor blob store types, make it dyn-compatible, add local s… · s5-dev/s5-rs@bc2cb3b · GitHub)
The RHP4 proxy is not “production-ready” yet, but all server-side parts are now fully finished and the only thing missing is adding the s5_blobs protocol client to it and making it easier to add the proxy to a custom web app
s5-rs is missing complete implementations of the registry, streams, accounts and identity specs. This delay is due to them depending on the Iroh protocol API and networking which I just recently added for the s5_blobs protocol
The s3 store is not finished, but due to there being a completely custom one purpose-built for Sia, this is less relevant than initially planned
Adding tests to all crates is planned for the 2025-10-02 milestone, as written in the original grant proposal
What will you be working on next?
RHP4 web proxy: add s5 networking using Iroh for dynamic blob location requests, add http streaming support in addition to rhp4 as a fallback for older browsers (and Safari). Make it easy for devs to configure and add to their app. write documentation
Implement the missing specs in s5-rs
Implement end-to-end-encryption for the S5 file system
Add tests to all S5 rust crates
Make sure that all parts of the s5.js TypeScript are properly migrated to be compatible with s5-rs and S5 v1
The “Make the RHP4 WASM-based blob proxy production ready” component has been moved from Milestone #1 to Milestone #4, due on 2025-11-02. The reason for this is that the current implementation based on renterd is in a working MVP state, but the upcoming indexd architecture fits S5 (and many other use cases) significantly better. So it makes more sense to wait until indexd is available for testing, instead of polishing up the current (soon-to-be-deprecated) implementation.
Migrate the TypeScript library to the new file system
Jules has been busy with migrating most of s5.js to the new CBOR-based directory metadata structures, in October we will work together on upstreaming his changes to the official s5.js repository (see GitHub - julesl23/s5.js: TypeScript/JS Library for S5)
specs impl: accounts, identity
instead of implementing a completely custom identity solution, s5 now simply uses iroh ed25519 public keys for node identity and will likely use atproto’s did:plc for public identities covering multiple nodes.
What will you be working on next?
Work with Jules on upstreaming his changes to s5.js so they are properly migrated to be compatible with s5-rs and S5 v1
Extending the config format to allow configuring trusted third-party nodes and upload+pinning targets
Implement Sia RHP4 uploads and downloads in S5 using indexd
Make the RHP4 WASM-based blob proxy production ready, so devs can easily use files streamed directly from Sia hosts in their web apps
Release a stable v1 version of all S5 libraries and code
Remove the indexd-dependent milestone (and one monthly payment) from this grant due to it not being available yet, blocking RHP4 uploads/downloads and the productionization of the WASM blob proxy. I will submit a small, focused grant for indexd integration only once it’s available (target November).
Align reporting with the new deadline: progress updates by the 25th each month
Clarify validation requirements for milestone completion
Revised milestones
2025-10-25
Allow configuring trusted nodes by their ed25519 iroh pubkey and syncing private s5 file systems with them
Add a config option for adding s5 nodes who are allowed to upload and pin data to specific blob stores and using that for remote blob upload and sync functionaility
Validation Requirements for Milestone Completion
Comprehensive and well-documented Rust test which sets up three S5 nodes (two trusted and one untrusted), configures the untrusted one to allow the other two nodes to upload and pin blobs, creates a directory and a few files on one of the trusted nodes and then correctly syncs it to the second trusted node via the untrusted node (end-to-end-encrypted). This test validates the common use case of users syncing data securely between their devices even if they are not both online at the same time
2025-11-25
Stable release of the s5 rust crates
Well-documented JavaScript and Dart bindings via UniFFI
Vup Web Migration to s5-rust
Validation Requirements for Milestone Completion
All s5 rust crates published on crates.io with documentation for all public API methods
JavaScript bindings to s5 rust published on npm
Dart bindings to s5 rust published on pub.dev
Vup Web migrated to s5-rust with the following features working: account creation/login, creating directories, uploading files to remote s5 nodes, downloading files (all end-to-end-encrypted)
Hi @redsolver - thank you for detailing the changes above. Confirming this means you are okay with receiving your final payment on December 15th with your final report submitted November 25th now instead of October 25th? This will be due to the fact that the final payment for the grant will be issued after the final successful technical review.
I imagine you’ve detailed a slightly longer duration for Milestone #4 (your new final milestone) because the additional time is needed, and there are no issues with that on our side.
Grant Milestone Progress Report (October) [revised]
What progress was made on your grant this month? Summarize your progress into a few sentences or bullet points.
Implemented automatic sharding of the S5 file system to more efficiently handle large directory trees
Added an option for configuring trusted nodes by their ed25519 iroh pubkey and syncing private s5 file systems with them
Added a config option for adding s5 nodes who are allowed to upload and pin data to specific blob stores and using that for remote blob upload and sync
For the FS, added snapshot export and merge functionality for syncing plaintext and encrypted directories
Implemented a remote blob store client which uses a remote s5 node over iroh as a blob store, but exposes the same API as local blob stores
Implemented a comprehensive and well-documented Rust test which sets up three S5 nodes (two trusted and one untrusted), configures the untrusted one to allow the other two nodes to upload and pin blobs, creates a directory and a few files on one of the trusted nodes and then correctly syncs it to the second trusted node via the untrusted node (end-to-end-encrypted). This test validates the common use case of users syncing data securely between their devices even if they are not both online at the same time
Milestone
Task
Commits
Additional Notes
Milestone 2025-10-25: E2EE FS sync over untrusted node