S5 v1: Rewrite it in Rust (Large Grant Proposal)

Project Name: S5 Network
Name of the organization or individual submitting the proposal: redsolver
Duration: 5 months

Introduction

Describe your project

S5 adds a content-addressed data routing layer and many other features on top of the Sia Network to make it easier for developers to use and implement decentralized storage in a wide variety of apps and use cases. S5 aims to enrich and extend the wider ecosystem of content-addressed storage networks like IPFS and Iroh, and be interoperable with them where it makes sense.

In my previous grant I noticed that reliably planning 12 months ahead for a complex project like S5 is pretty much impossible, so this new grant is now reduced to 5 months and hyper-focused on the goal of a full stable S5 v1 release, so the core for everyone building on S5 is the best it can be.

The major pivot here is that my main focus is now implementing all S5 specs in Rust, for the following reasons:

  • Rust is very easy to integrate in both native (ffi) and web-based applications via WASM

  • Many other content-addressed data projects like Iroh use Rust, so integrating them is trivial. For example, p2p quic connections between s5 nodes or content discovery will be handled by Iroh’s well-tested and optimized implementation, improving the overall reliability and saving me time by being able to re-use their very nice local blake3 blob store implementation and blake3 bao tree verification

  • There is excellent tooling for bridging Rust code with Flutter/Dart available, so long-term I won’t need to maintain that additional full s5 implementation

  • Rust is performant, safe and great for building distributed applications.

As an example, I implemented direct RHP4 streaming from Sia hosts in Rust, and could simply compile it to WASM and use it in any web browser: https://rhp4-webtransport-demo.sia5.net/

TLDR: Rust is the best choice for a project like S5 and the excellent tooling for integrating with other frameworks and languages even reduces my development work in the future

Who benefits from your project?

  • Primarily developers building all kinds over content-addressed applications on top of Sia

  • Sia community members using applications built on S5 (both by myself and others)

How does the project serve the Foundation’s mission of user-owned data?

Making it as easy as possible for users to truly own their data and access it everywhere they go is the core of what S5 stands for.

We cannot provide grants to residents of jurisdictions under increased FATF monitoring, those that have active OFAC sanctions, or those that fail our bank compliance tests. We also cannot provide grants if your payment bank account is located in those same locations. Please review the following list.

  • Are you a resident of any jurisdiction on that list? No

  • Will your payment bank account be located in any jurisdiction on that list? No

Grant Specifics

Amount of money requested and justification with a comprehensive breakdown of expenses:

Salary is exactly the same per month in EUR as my previous grant, but due to the current exchange rates for USD it’s now suddenly a Large Grant

I’m requesting the following budget for 5 months of work:

  • 40k USD (~34k EUR) for the full-time salary of redsolver for 5 months

  • 5k USD (~4.2k EUR) for UI design work

    • 4,000 USD web-based developer tooling (like http://cid.one/)

    • 1,000 USD proper logo for the S5 protocol

  • 3k USD (~2.5k EUR) for infrastructure costs

    • 2,000 USD public S5 nodes (for bootstrapping and registry storage) and a free-only S5 Node with a bit of storage for onboarding users and developers

    • 1,000 USD Running Iroh relays for improved p2p connectivity (2x Hetzner dedis with 10G uplink).

Total: 48,000 USD (~41k EUR)

Timeline with measurable objectives and goals.

REQUIRED: Milestones with which to judge your progress. Milestones should be easy for the Grants Committee to understand and evaluate as your project moves through its term. The Committee reserves the right to accept, modify, or reject proposed milestones to ensure they represent thoughtful and reasonable project evaluation checkpoints. Further payments may be withheld for missed milestones.

Milestone 1: 90% of the Rust implementation 2025-08-02

  • Continue implementing all the basic S5 specs in Rust

    • already finished: blobs, blob store (sia, s3, local), file system (no encryption), http importer

    • todo: registry, streams, accounts, identity

  • Make the RHP4 WASM-based blob proxy production ready, so devs can easily use files streamed directly from Sia hosts in their web apps (moved to milestone #4 due to waiting for indexd)

Milestone 2: Iroh and Networking 2025-09-02

  • Implement networking in Rust

    • the existing S5 WebSocket-based protocol

    • but also the new one powered by and compatible with Iroh which uses direct encrypted p2p QUIC connections on native and WebTransport for web

Milestone 3: Final File System 2025-10-02

  • Implement end-to-end-encryption for the S5 file system

  • Add tests to all S5 rust crates

  • Migrate the TypeScript library to the new file system (most other core components of s5.js are already using s5 v1 structures)

Milestone 4: 2025-11-02

  • Release a stable v1 version of all S5 libraries and code

  • Implement Sia RHP4 uploads in S5, both on web and native

  • Make the RHP4 WASM-based blob proxy production ready, so devs can easily use files streamed directly from Sia hosts in their web apps (moved here from milestone #1 due to waiting for indexd)

Milestone 5: 2025-12-02

  • Update Vup Web to use the S5 v1 file system and Rust streaming code, including all the RHP4 features

  • Release Vup Web in Public Beta

Bonus

My top priority is completing all milestones listed above, but if there’s enough time remaining I will also work on the following sub-projects which are closely related to and depend on features implemented in s5-rs:

  • FS5 fuse driver written in rust (I already have a read-only MVP working, used it for prototyping and testing the final S5 file system spec)

  • Serve web apps from the s5_web_proxy service worker and integrate webxdc

Potential Risks

  • Milestones taking longer than expected to complete

  • Technical risks are pretty limited due to all milestones already being validated by a lot of research and found to be feasible. Nevertheless, there are always unexpected issues that could arise and result in new technical challenges causing some code and ideas to not work out as expected.

Development Information

Will all of your project’s code be open-source?

Yes.

Leave a link where code will be accessible for review.

Do you agree to submit monthly progress reports?

Yes.

Do you agree to designate a point of contact for committee questions and concerns?

Yes, myself (redsolver). Contact info is at the end of the grant proposal.

Provide links to previous work or code from all team members.

Have you developed a proof of concept for this idea already? If not, you can develop this as part of another grant before submitting this grant.

Yes, see GitHub - s5-dev/s5-rs and all other repos linked above

Do you agree to participate in a demo at our monthly community call at significant milestones or after the grant’s completion?

Yes.

Contact info

I will support this, but I would like to see a commitment from S5 on no more BC (backwards compatibility) breaking spec changes, at the application layer, or the networking layer by the end of this grant.

Follow semver, and while spec revisions will be welcomed for improvements down the road, S5 must ossify some, to use a BTC culture term for the next wave of Sia builders to be using it.

As it stands right now, Lume will not be touching S5 until it is clear that things will not shift around again.

And if you are going to go and make changes at the scale discussed, I don’t think there is a point in keeping the websocket interface, as it is tech debt that will be a liability in a year or so.

Kudos.

1 Like

I would like to see a commitment from S5 on no more BC (backwards compatibility) breaking spec changes, at the application layer, or the networking layer by the end of this grant.

That’s the primary intention of this grant; by the end of this year there will be a complete S5 1.0.0 stable release with no further breaking changes. So all releases after 1.0.0 will be fully backwards compatible with 1.0.0, especially in regard to identifier formats for blobs and directories, as well as all data structures.

And if you are going to go and make changes at the scale discussed, I don’t think there is a point in keeping the websocket interface, as it is tech debt that will be a liability in a year or so.

Unfortunately the WebSocket interface will likely stay around (although using the new p2p protocol) because not all web browsers support WebTransport yet. But it’s only used as a generic transport layer, so it shouldn’t really cause much trouble.

1 Like

As I have been working with the current version of s5 for the past couple months, I can say this would be a welcomed update for me. Both having a stable api and better node performance. My app is mobile only so reducing load on the system is rather important.

@redsolver Thanks for your proposal to The Sia Foundation Grants Program.

After review, the committee has decided to approve your proposal. Congratulations! They’re excited to see what you can accomplish with this grant.

We’ll reach out to your provided email address for onboarding. This shouldn’t take long unless your info has changed from last time, but you may still need to adjust your timelines.

2 Likes

Progress Report (July)

What progress was made on your grant this month?

Continued implementing all the basic S5 specs in Rust:

  • Monorepo for all s5-rs crates: GitHub - s5-dev/s5-rs

  • Added Sia Blob Store, using the new PinnedObject API endpoint to extract the file encryption key (feat: add sia blob store · s5-dev/s5-rs@8903243 · GitHub)

  • Decided to use Iroh for all p2p communication between S5 Nodes, including web browsers, to simplify implementations (see iroh 0.33.0 - Browsers and Discovery and 0-RTT, oh my! - Iroh for Iroh browser support). As such, there’s no need to have a custom WebSocket-based protocol for S5 anymore. S5 will run some public Iroh relay nodes (as mentioned in the grant proposal and budget), to improve connectivity for all users.

  • Tested importing files to the S5 Sia Blob Store from HTTP mirrors using rclone

Prepare to make the RHP4 WASM-based blob proxy ready for production use:

What will you be working on next?

  • Highest priority: Add networking using Iroh for both native and web

  • Implement S5 Registry and Streams protocols, I delayed this until Iroh is added because they will be implemented as Iroh protocols (one Iroh p2p connection can transport any number of protocols in parallel)

  • RHP4 web proxy: add s5 networking using Iroh for dynamic blob location requests, add http streaming support in addition to rhp4 as a fallback for older browsers (and Safari)

  • Make the FS APIs more ergonomic, so developers can just call something like “fs.write(path, bytes)” and having it take care of blob store upload and pinning, instead of having to work with the FS5 directory data structure directly

1 Like

Hello,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Progress Report (August)

What progress was made on your grant this month?

Delayed Milestones

  • The RHP4 proxy is not “production-ready” yet, but all server-side parts are now fully finished and the only thing missing is adding the s5_blobs protocol client to it and making it easier to add the proxy to a custom web app

  • s5-rs is missing complete implementations of the registry, streams, accounts and identity specs. This delay is due to them depending on the Iroh protocol API and networking which I just recently added for the s5_blobs protocol

  • The s3 store is not finished, but due to there being a completely custom one purpose-built for Sia, this is less relevant than initially planned

Adding tests to all crates is planned for the 2025-10-02 milestone, as written in the original grant proposal

What will you be working on next?

  • RHP4 web proxy: add s5 networking using Iroh for dynamic blob location requests, add http streaming support in addition to rhp4 as a fallback for older browsers (and Safari). Make it easy for devs to configure and add to their app. write documentation

  • Implement the missing specs in s5-rs

  • Implement end-to-end-encryption for the S5 file system

  • Add tests to all S5 rust crates

  • Make sure that all parts of the s5.js TypeScript are properly migrated to be compatible with s5-rs and S5 v1

1 Like

Hello

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Timeline Update

The “Make the RHP4 WASM-based blob proxy production ready” component has been moved from Milestone #1 to Milestone #4, due on 2025-11-02. The reason for this is that the current implementation based on renterd is in a working MVP state, but the upcoming indexd architecture fits S5 (and many other use cases) significantly better. So it makes more sense to wait until indexd is available for testing, instead of polishing up the current (soon-to-be-deprecated) implementation.

1 Like

Progress Report (September)

What progress was made on your grant this month?

Milestone Tasks Progress

Can be validated by running cargo test in the s5_fs subdirectory

Can be validated by running cargo test in the s5_core subdirectory

Other process

Remaining Tasks

  • Migrate the TypeScript library to the new file system

Jules has been busy with migrating most of s5.js to the new CBOR-based directory metadata structures, in October we will work together on upstreaming his changes to the official s5.js repository (see GitHub - julesl23/s5.js: TypeScript/JS Library for S5)

  • specs impl: accounts, identity

instead of implementing a completely custom identity solution, s5 now simply uses iroh ed25519 public keys for node identity and will likely use atproto’s did:plc for public identities covering multiple nodes.

What will you be working on next?

  • Work with Jules on upstreaming his changes to s5.js so they are properly migrated to be compatible with s5-rs and S5 v1

  • Extending the config format to allow configuring trusted third-party nodes and upload+pinning targets

  • Implement Sia RHP4 uploads and downloads in S5 using indexd

  • Make the RHP4 WASM-based blob proxy production ready, so devs can easily use files streamed directly from Sia hosts in their web apps

  • Release a stable v1 version of all S5 libraries and code

1 Like

Hello @redsolver,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Grant Update Proposal

Reasons

  • Remove the indexd-dependent milestone (and one monthly payment) from this grant due to it not being available yet, blocking RHP4 uploads/downloads and the productionization of the WASM blob proxy. I will submit a small, focused grant for indexd integration only once it’s available (target November).

  • Align reporting with the new deadline: progress updates by the 25th each month

  • Clarify validation requirements for milestone completion

Revised milestones

2025-10-25

  • Allow configuring trusted nodes by their ed25519 iroh pubkey and syncing private s5 file systems with them

  • Add a config option for adding s5 nodes who are allowed to upload and pin data to specific blob stores and using that for remote blob upload and sync functionaility

Validation Requirements for Milestone Completion

  • Comprehensive and well-documented Rust test which sets up three S5 nodes (two trusted and one untrusted), configures the untrusted one to allow the other two nodes to upload and pin blobs, creates a directory and a few files on one of the trusted nodes and then correctly syncs it to the second trusted node via the untrusted node (end-to-end-encrypted). This test validates the common use case of users syncing data securely between their devices even if they are not both online at the same time

2025-11-25

  • Stable release of the s5 rust crates

  • Well-documented JavaScript and Dart bindings via UniFFI

  • Vup Web Migration to s5-rust

Validation Requirements for Milestone Completion

  • All s5 rust crates published on crates.io with documentation for all public API methods

  • JavaScript bindings to s5 rust published on npm

  • Dart bindings to s5 rust published on pub.dev

  • Vup Web migrated to s5-rust with the following features working: account creation/login, creating directories, uploading files to remote s5 nodes, downloading files (all end-to-end-encrypted)

1 Like

Hi @redsolver - thank you for detailing the changes above. Confirming this means you are okay with receiving your final payment on December 15th with your final report submitted November 25th now instead of October 25th? This will be due to the fact that the final payment for the grant will be issued after the final successful technical review.

I imagine you’ve detailed a slightly longer duration for Milestone #4 (your new final milestone) because the additional time is needed, and there are no issues with that on our side.

Yes, I’m okay with receiving my final payment on December 15th and no payment on November 15th.

1 Like

Sorry for the delayed progress report, I’m trying my best to get it out today.

1 Like

Hello @redsolver,

Please have your progress report submitted be EOD today.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Hi @redsolver - this is the final notice for your October progress report.

If your full and complete report is not posted by 9am ET tomorrow (November 4th), you will have to miss this payment cycle.

Progress Report (October)

I would like to apologize for the delayed report and thank you for your understanding.

Milestone 2025-10-25 completed

Validation: Clone latest repo and run cargo test -p s5_node --test fs_sync_complete

Commits implementing the milestone: Comparing d5566096102f797fe201ffc4df19a3c5814afaf7...857e6e38e617c50f67037945b8cc17b0f41b65e4 · s5-dev/s5-rs · GitHub

Next Up: Working on the 2025-11-25 milestone

Hi @redsolver, this report is greatly lacking in detail and does not follow the Development Guide format.

Please revise the report to match the correct format by 5pm ET.

1 Like