S5 v1: Rewrite it in Rust (Large Grant Proposal)

Grants Approved
1

Project Name: S5 Network
Name of the organization or individual submitting the proposal: redsolver
Duration: 5 months

Introduction

Describe your project

S5 adds a content-addressed data routing layer and many other features on top of the Sia Network to make it easier for developers to use and implement decentralized storage in a wide variety of apps and use cases. S5 aims to enrich and extend the wider ecosystem of content-addressed storage networks like IPFS and Iroh, and be interoperable with them where it makes sense.

In my previous grant I noticed that reliably planning 12 months ahead for a complex project like S5 is pretty much impossible, so this new grant is now reduced to 5 months and hyper-focused on the goal of a full stable S5 v1 release, so the core for everyone building on S5 is the best it can be.

The major pivot here is that my main focus is now implementing all S5 specs in Rust, for the following reasons:

  • Rust is very easy to integrate in both native (ffi) and web-based applications via WASM

  • Many other content-addressed data projects like Iroh use Rust, so integrating them is trivial. For example, p2p quic connections between s5 nodes or content discovery will be handled by Iroh’s well-tested and optimized implementation, improving the overall reliability and saving me time by being able to re-use their very nice local blake3 blob store implementation and blake3 bao tree verification

  • There is excellent tooling for bridging Rust code with Flutter/Dart available, so long-term I won’t need to maintain that additional full s5 implementation

  • Rust is performant, safe and great for building distributed applications.

As an example, I implemented direct RHP4 streaming from Sia hosts in Rust, and could simply compile it to WASM and use it in any web browser: https://rhp4-webtransport-demo.sia5.net/

TLDR: Rust is the best choice for a project like S5 and the excellent tooling for integrating with other frameworks and languages even reduces my development work in the future

Who benefits from your project?

  • Primarily developers building all kinds over content-addressed applications on top of Sia

  • Sia community members using applications built on S5 (both by myself and others)

How does the project serve the Foundation’s mission of user-owned data?

Making it as easy as possible for users to truly own their data and access it everywhere they go is the core of what S5 stands for.

We cannot provide grants to residents of jurisdictions under increased FATF monitoring, those that have active OFAC sanctions, or those that fail our bank compliance tests. We also cannot provide grants if your payment bank account is located in those same locations. Please review the following list.

  • Are you a resident of any jurisdiction on that list? No

  • Will your payment bank account be located in any jurisdiction on that list? No

Grant Specifics

Amount of money requested and justification with a comprehensive breakdown of expenses:

Salary is exactly the same per month in EUR as my previous grant, but due to the current exchange rates for USD it’s now suddenly a Large Grant

I’m requesting the following budget for 5 months of work:

  • 40k USD (~34k EUR) for the full-time salary of redsolver for 5 months

  • 5k USD (~4.2k EUR) for UI design work

    • 4,000 USD web-based developer tooling (like http://cid.one/)

    • 1,000 USD proper logo for the S5 protocol

  • 3k USD (~2.5k EUR) for infrastructure costs

    • 2,000 USD public S5 nodes (for bootstrapping and registry storage) and a free-only S5 Node with a bit of storage for onboarding users and developers

    • 1,000 USD Running Iroh relays for improved p2p connectivity (2x Hetzner dedis with 10G uplink).

Total: 48,000 USD (~41k EUR)

Timeline with measurable objectives and goals.

REQUIRED: Milestones with which to judge your progress. Milestones should be easy for the Grants Committee to understand and evaluate as your project moves through its term. The Committee reserves the right to accept, modify, or reject proposed milestones to ensure they represent thoughtful and reasonable project evaluation checkpoints. Further payments may be withheld for missed milestones.

Milestone 1: 90% of the Rust implementation 2025-08-02

  • Continue implementing all the basic S5 specs in Rust

    • already finished: blobs, blob store (sia, s3, local), file system (no encryption), http importer

    • todo: registry, streams, accounts, identity

  • Make the RHP4 WASM-based blob proxy production ready, so devs can easily use files streamed directly from Sia hosts in their web apps

Milestone 2: Iroh and Networking 2025-09-02

  • Implement networking in Rust

    • the existing S5 WebSocket-based protocol

    • but also the new one powered by and compatible with Iroh which uses direct encrypted p2p QUIC connections on native and WebTransport for web

Milestone 3: Final File System 2025-10-02

  • Implement end-to-end-encryption for the S5 file system

  • Add tests to all S5 rust crates

  • Migrate the TypeScript library to the new file system (most other core components of s5.js are already using s5 v1 structures)

Milestone 4: 2025-11-02

  • Release a stable v1 version of all S5 libraries and code

  • Implement Sia RHP4 uploads in S5, both on web and native

Milestone 5: 2025-12-02

  • Update Vup Web to use the S5 v1 file system and Rust streaming code, including all the RHP4 features

  • Release Vup Web in Public Beta

Bonus

My top priority is completing all milestones listed above, but if there’s enough time remaining I will also work on the following sub-projects which are closely related to and depend on features implemented in s5-rs:

  • FS5 fuse driver written in rust (I already have a read-only MVP working, used it for prototyping and testing the final S5 file system spec)

  • Serve web apps from the s5_web_proxy service worker and integrate webxdc

Potential Risks

  • Milestones taking longer than expected to complete

  • Technical risks are pretty limited due to all milestones already being validated by a lot of research and found to be feasible. Nevertheless, there are always unexpected issues that could arise and result in new technical challenges causing some code and ideas to not work out as expected.

Development Information

Will all of your project’s code be open-source?

Yes.

Leave a link where code will be accessible for review.

Do you agree to submit monthly progress reports?

Yes.

Do you agree to designate a point of contact for committee questions and concerns?

Yes, myself (redsolver). Contact info is at the end of the grant proposal.

Provide links to previous work or code from all team members.

Have you developed a proof of concept for this idea already? If not, you can develop this as part of another grant before submitting this grant.

Yes, see GitHub - s5-dev/s5-rs and all other repos linked above

Do you agree to participate in a demo at our monthly community call at significant milestones or after the grant’s completion?

Yes.

Contact info

2

I will support this, but I would like to see a commitment from S5 on no more BC (backwards compatibility) breaking spec changes, at the application layer, or the networking layer by the end of this grant.

Follow semver, and while spec revisions will be welcomed for improvements down the road, S5 must ossify some, to use a BTC culture term for the next wave of Sia builders to be using it.

As it stands right now, Lume will not be touching S5 until it is clear that things will not shift around again.

And if you are going to go and make changes at the scale discussed, I don’t think there is a point in keeping the websocket interface, as it is tech debt that will be a liability in a year or so.

Kudos.

1 Like
3

I would like to see a commitment from S5 on no more BC (backwards compatibility) breaking spec changes, at the application layer, or the networking layer by the end of this grant.

That’s the primary intention of this grant; by the end of this year there will be a complete S5 1.0.0 stable release with no further breaking changes. So all releases after 1.0.0 will be fully backwards compatible with 1.0.0, especially in regard to identifier formats for blobs and directories, as well as all data structures.

And if you are going to go and make changes at the scale discussed, I don’t think there is a point in keeping the websocket interface, as it is tech debt that will be a liability in a year or so.

Unfortunately the WebSocket interface will likely stay around (although using the new p2p protocol) because not all web browsers support WebTransport yet. But it’s only used as a generic transport layer, so it shouldn’t really cause much trouble.

1 Like
4

As I have been working with the current version of s5 for the past couple months, I can say this would be a welcomed update for me. Both having a stable api and better node performance. My app is mobile only so reducing load on the system is rather important.

5

@redsolver Thanks for your proposal to The Sia Foundation Grants Program.

After review, the committee has decided to approve your proposal. Congratulations! They’re excited to see what you can accomplish with this grant.

We’ll reach out to your provided email address for onboarding. This shouldn’t take long unless your info has changed from last time, but you may still need to adjust your timelines.

2 Likes
6

Progress Report (July)

What progress was made on your grant this month?

Continued implementing all the basic S5 specs in Rust:

  • Monorepo for all s5-rs crates: GitHub - s5-dev/s5-rs

  • Added Sia Blob Store, using the new PinnedObject API endpoint to extract the file encryption key (feat: add sia blob store · s5-dev/s5-rs@8903243 · GitHub)

  • Decided to use Iroh for all p2p communication between S5 Nodes, including web browsers, to simplify implementations (see iroh 0.33.0 - Browsers and Discovery and 0-RTT, oh my! - Iroh for Iroh browser support). As such, there’s no need to have a custom WebSocket-based protocol for S5 anymore. S5 will run some public Iroh relay nodes (as mentioned in the grant proposal and budget), to improve connectivity for all users.

  • Tested importing files to the S5 Sia Blob Store from HTTP mirrors using rclone

Prepare to make the RHP4 WASM-based blob proxy ready for production use:

What will you be working on next?

  • Highest priority: Add networking using Iroh for both native and web

  • Implement S5 Registry and Streams protocols, I delayed this until Iroh is added because they will be implemented as Iroh protocols (one Iroh p2p connection can transport any number of protocols in parallel)

  • RHP4 web proxy: add s5 networking using Iroh for dynamic blob location requests, add http streaming support in addition to rhp4 as a fallback for older browsers (and Safari)

  • Make the FS APIs more ergonomic, so developers can just call something like “fs.write(path, bytes)” and having it take care of blob store upload and pinning, instead of having to work with the FS5 directory data structure directly

1 Like