Standard Grant: S5 Gateway & TypeScript Client

Grants Approved
2

Hey Jules,

Reviewed the grant proposal. Good ideas tackling a lightweight gateway and SDK on S5 – definitely valuable for developer ergonomics and providing a usable tool.

But I see some significant technical points that need realignment with how S5 is actually developing right now. Based on my experience with the protocol and past efforts, these are critical for the project’s success and relevance.

Here’s the main feedback:

  1. Replicating the “Skynet Client API” is Building on Legacy, Not the Future: The proposal aims to mirror the original Skynet Client API (uploadFile, uploadDirectory, getJSON, setRegistryEntry, etc.).

    • Problem: While S5 was inspired by Skynet, it has its own architecture and is moving forward with new, standardized specifications. Crucially, the FS5 (File System S5) spec is the intended way to handle directories and metadata going forward, replacing all previous manifest formats (Skynet’s old approach, S5’s earlier ones).
    • Building your SDK/gateway around the legacy Skynet Client API means it targets an interface and set of concepts that are being superseded by FS5 and the evolving S5 primitives. This won’t be compatible with the direction S5 is heading.

  2. s5client-js is the Wrong Dependency: You listed s5client-js as a base library. You must drop this. It’s based on old, experimental code and is not the foundation for building modern S5 tooling that works with FS5.

    • Solution: You need to build on Redsolver’s official s5.js library instead. It’s the correct (though still WIP) base, and contributing to it is the way forward for the community. Coordinate with Redsolver; using s5client-js will make the project incompatible long-term.

  3. Implement FS5, Don’t Create a New Manifest: The proposal talks about gaining “an interoperable manifest format.”

    • Problem: The interoperable format for S5 directories is the FS5 specification currently being developed. Don’t create another one within this project. https://xkcd.com/927/ :upside_down_face:
    • Action: Your work on directories/manifests must focus entirely on correctly implementing and testing against the FS5 spec.

  4. A Better SDK Goal: Cross-Protocol Abstraction: Instead of strictly mirroring a legacy API, a more impactful SDK (that the gateway could then use) would abstract across multiple decentralized storage protocols (like S5 and IPFS) using common concepts (Multiformats, Multibase). This would provide broader utility than just replicating a single, outdated interface.

Summary: Avoid replicating the old “Skynet Client API”; build for the actual, evolving S5 standard (especially FS5) and its primitives, using the correct base library (s5.js). Focus your manifest/interop efforts on implementing/supporting the FS5 spec. Consider the SDK’s design goal as potentially broader than just S5 legacy replication.

Getting the details right on interfacing with S5’s current direction is crucial for this project to be useful and relevant. Happy to give more input.

Kudos.

4 Likes
3

Hi pcfreak30,

Thanks for the detailed review!

Totally agree—aligning with FS5 and the official s5.js library is the right way to go. I’ll revise the proposal to:
• drop s5client-js in favour of Redsolver’s s5.js;
• implement directory handling against the draft FS5 spec (so no new manifest format);
• keep high-level helper names (uploadFile, etc.) that are easy to use, but map them to FS5 under the hood.

I’ll check with Redsolver how draft the FS5 specs are and the plans for s5.js as it’s missing some file handling that Skynet API had, that I could add in.

I’ll design the gateway and SDK to be CID-first. The code will parse any CIDv1, inspect the multicodec prefix, and handle the two S5 codecs (blob and registry).
For all other codecs it will return “unsupported protocol” until future extensions are added. This keeps the architecture open to IPFS or other networks later, without expanding the current proposal timeline.

3 Likes
4

To be clear a JS impl of FS5 exists, you just need to contribute s5.js/src/fs at 0b415bc5365855f10c0a28ec4564e8bf9060e1ad · s5-dev/s5.js · GitHub.

5

Cheers — Good to know that FS5 already handles the core “what files are in this directory” task that s5client-js did :)

What it doesn’t do is let you fetch a single file straight from the root CID, e.g.
/CID/path/to/myFile1.txt.
To achieve that the S5 Gateway will download the FS5 manifest, look up the child file’s CID and fetch the file’s content.

FS5 simply stores the list of files. It does not treat index.html (or any file) as a default page for HTTP requests, fall back to index.html when a path ends in /, set MIME-types, or handle HTTP Range requests. I’ll add those behaviours to the S5 Gateway.

6

Something I feel might be a good opportunity that fits in somewhere.

These have come to my mind repeatedly that S5 and probably at a higher abstraction should have this as a tool. helia-verified-fetch basically creates a fetch like api that abstracts out IPFS, and service-worker-gateway uses some of the principles earlier Lume work did as well in loading data with client-based proxying.

Not clear where this falls in your plans, but its DevX tooling that makes sense to be created at some point.

Just giving you some more ideas :P.

7

Thanks for the links!
The ipfs work is out of scope for the proposal but will certainly look at the architecture and ensure that helia-verified-fetch and a service-worker gateway can be plugged in later. Cheers.

8

Hey @juleslai, thanks for submitting this grant proposal!

I fully agree with @pcfreak30’s feedback on not replicating the old Skynet Client API, and also that using s5.js as the base for your project makes the most sense.

Many of the features you mention (like uploading files and DNSLink) are already supported by S5 nodes, though not by the TypeScript implementation yet. So a full S5 node implementation in TypeScript (based on s5.js) with these features could be useful, and your grant proposal kinda looks like that’s what you actually want to build :)

Some other features you mentioned (updating mutable registry entries, subscribing to changes over WebSocket) are possible with both the TypeScript and Dart libraries, and I’m not sure if I see the benefit of adding new HTTP endpoints for these (like Skynet had). The issue here is that data like signed registry entries returned from these endpoints should still be verified by clients, so they need to use (atleast some parts) of the S5 library anyways, and could simply use the full lib with all operations over a websocket connection using the s5 p2p protocol between nodes directly.

Regarding the FS5 implementation in s5.js, you’re correct in that it’s missing some of the more advanced features like thumbnails for media files and could also benefit from some more quality-of-life utils for working with directory structures. So if you would like to work on adding these, they should likely be just implemented in s5.js directly.

On the topic of ipfs’s verified-fetch and their service-worker-gateway, S5 actually already has a fully working variant of that for both plaintext and encrypted files: GitHub - s5-dev/web-proxy: Client-side gateway with verified streaming, running in the web browser
It does however not support resolving and browsing FS5 directory structures yet, so that could be a useful addition.

9

Hi @Redsolver,

Thanks for the reply. Good news that you have implemented a lot of the basic operations that I don’t have to code now :)

Yep, can add to s5.js:
Thumbnail + media metadata pipeline – small JPEG/WebP previews, basic EXIF/size detection.
Directory utilities – walk large FS5 trees, filter by media type, cache manifests.

I believe it would be great to add more general key/value store operations. What do you think?
I plan to extend s5.js with:

  1. Path-based API

    • Add getByPath(rootKey, path) and putByPath(rootKey, path, data) helpers that transparently fetch, traverse, and update nested FS5 manifests.

  2. LWW-Register CRDT support

    • Wrap registry updates in a Last-Writer-Wins register (using the entry’s revision counter as the timestamp) to handle concurrent writes at a per-field or per-object level ([Wikipedia][1]).

  3. Manifest caching & sharding

    • Integrate an in-memory (or IndexedDB) cache for manifest CIDs,
    • And support HAMT/B-Tree sharding metadata in the FS5DirectoryHeader to efficiently handle large directories ([S5 Network Docs][2]).

My goal is to give JS/TS developers OrbitDB-like get/put semantics on S5, with built-in conflict resolution, low latency, and scalable directories. I noticed the FS5 spec still marks sharding as “TODO” in the header—has any sharding support been implemented yet, or is that still on the roadmap?

10

The issue I was trying to highlight was that the S5.js library maybe a bit too low-level detailed for some Web3 devs. The “<cid>/path” system is not as easy to use as a general “path” system where “a/b” gives back content from e.g. paths “a/b/c”, “a/b/d” and “a/b/c/e”. With optional parameters such as max depth level, another option or function to just retrieve all the child CIDs rather than content etc. The implementation I have would have only one registrySet call per file update. So a change in the lowest level “folder” would propagate manifest blob changes up to the root and then I call one registrySet for the new CID.

Now another question is, would it better to put this in another repo e.g. s5ext.js ?

11

I have limited access to when I can sit down on the internet for next few days. I will redo the grant proposal with changes from the discussions after that :)

12

imo any work that ties into s5 core directly should be a PR to the library itself. This is all a community effort after all.

It can be spun out later if its architecturally decided that a separate high level lib is needed and to keep high/low boundaries.

13

Makes sense. Cheers.

14

Here is the updated grant proposal:

15

Introduction

Project Name:
Enhanced s5.js – Advanced Directory Handling and Media Support

Applicant:
Jules Lai (sole developer, founder of Fabstir)

Describe your project

The official s5.js library provides fundamental S5 operations, but developers need higher-level features for efficient application development:

  • Simple path-based storage: “Store this data at /profile/settings”
  • Media handling: “Generate thumbnails for all images in /gallery”
  • Reliable updates: “Handle concurrent writes safely”

This grant adds these developer-friendly features directly into s5.js, making it easier to build robust applications on S5.

Planned enhancements

  1. Path-based API — getByPath(rootKey, path) / putByPath(rootKey, path, data).
    Traverses nested FS5 manifests, fetches or updates the target blob, and bubbles changed CIDs up to the root with a single registrySet.

  2. LWW-register wrapper — optional Last-Writer-Wins conflict-resolution around registry updates using the revision counter as timestamp.

  3. Thumbnail & media metadata pipeline — JPEG /WebP previews, basic EXIF/size detection and helper that returns { thumbnailCID, width, height, mime }.

  4. Directory utilities — Directory walker with optional limit / cursor pagination; integrates in-memory or IndexedDB manifest cache.

  5. FS5 sharding groundwork — implement the HAMT/B-Tree header fields marked TODO and add split / merge helpers so very large directories can be handled once the spec finalises.

These enhancements complement the existing s5-dev/web-proxy project, which provides client-side gateway functionality with verified streaming. While the web-proxy handles content delivery, my additions to s5.js will enable better directory browsing and media handling within that environment. The thumbnail pipeline and directory utilities are particularly valuable for web-proxy users, as they enable efficient browsing of large media collections directly in the browser. This maintains the web-proxy’s commitment to client-side verification whilst adding rich media capabilities that users expect from modern web applications.

All code will be submitted as pull requests to the upstream s5.js repository; a temporary fork lives under Fabstir until each PR is merged.

Who benefits?

Audience Benefit
JS/TS Web-3 developers Simple get/put path semantics and thumbnails instead of hand-crafting FS5 blobs.
S5 core library Gains high-level utilities without fragmenting into third-party forks.
Fabstir ecosystem Can store app bundles and user media with one call, paving the way for a decentralised app store.
Sia community Higher-level tooling encourages broader adoption of S5 + FS5.

Mission alignment

By making S5 easier to build on, more users will self-host data on Sia while retaining cryptographic control of their CIDs and registry keys—directly serving the Foundation’s goal of user-owned data.

Compliance

Resident of prohibited jurisdiction? No
Payment bank in prohibited jurisdiction? No

Grant Specifics

Amount requested

49 600 USD — paid in equal 6 200 USD monthly tranches (standard Foundation schedule).

Timeline & milestones (eight months)

Month Target date Milestone & deliverables Budget (USD)
1 7/2/25 Project set-up & design Repos, branching strategy, FS5 test fixtures; code-contribution guidelines & project board; tool-chain investigation; one-off business overhead 6 200
2 8/2/25 Path helpers v0.1 — basic get/put for single-level directories, unit tests 6 200
3 9/2/25 Path-cascade optimisation — multi-level update with single registrySet; docs & examples 6 200
4 10/2/25 WASM foundation & basic media — pipeline setup, code-splitting, image metadata extraction, baseline performance recorded 6 200
5 11/2/25 Advanced media processing — building on Month 4: JPEG / PNG / WebP thumbnails (≤ 64 kB average), progressive rendering, browser test matrix; bundle ≤ 700 kB compressed 6 200
6 12/2/25 Directory utilities & caching — walker with limit/cursor pagination, IndexedDB/in-memory cache, filtered listings, performance benchmarks 6 200
7 1/2/26 Sharding groundwork — HAMT header fields, split/merge helpers, integration tests 6 200
8 2/2/26 Documentation site update, demo scripts, screencast, forum feedback buffer, PRs merged upstream 6 200
Total 49 600 USD

The Foundation disburses 6 200 USD per month; internal line items show expected effort but may vary slightly while the total remains capped at 49 600 USD.

Technical Implementation Details

Media Processing Strategy

  • Primary Implementation (Month 4):

    • WASM module setup with code splitting
    • Lazy loading for optimal bundle size
    • Basic image metadata extraction
    • Browser compatibility testing suite
    • Performance baseline establishment

  • Advanced Features (Month 5):

    • Thumbnail generation with quality/size options
    • Support for JPEG, PNG, WebP formats
    • Progressive loading for large images
    • Fallback = skip thumbnail if local WASM fails
    • Comprehensive browser testing

Caching Implementation

  • Tiered Storage Approach:

    • In-memory LRU cache for frequent access
    • IndexedDB for persistent storage
    • Configurable cache size limits
    • Automatic cache invalidation
    • Browser storage quota management

  • Performance Targets:

    • Sub-100ms access for cached items
    • Efficient bulk operations
    • Minimal memory footprint
    • Graceful degradation strategy

Success Indicators

  • putByPath updates a deep file and results in exactly one registrySet.
  • Concurrent writes resolve correctly under the LWW wrapper (demonstrated in tests).
  • Thumbnail helper generates a preview ≤ 64 kB average size (95 th percentile) and correct MIME metadata for PNG, JPEG, WebP.
  • Directory walker lists 10 000 cached entries in ≤ 2 s on a typical laptop.
  • Thumbnail generation completes in ≤ 500 ms for a 1 MP JPEG on a mid-range laptop; larger images scale linearly.

Risks & Mitigation

Risk Mitigation
WASM bundle size exceeds targets Keep the thumbnail module ≤ 700 kB compressed
Browser compatibility issues Comprehensive testing matrix, graceful degradation paths, polyfills where needed
Thumbnail processing performance Progressive quality settings, worker thread implementation, cancel stale operations
Cache storage limitations Implement LRU eviction, respect browser quotas, clear warning when approaching limits
Fallback service availability Local processing first, clear error handling

*Fallback = If local WASM fails, thumbnail generation is skipped; no external upload is attempted.

Development Information

  • All code MIT-licensed.
  • Upstream PR target: https://github.com/s5-dev/s5.js (temporary fork at https://github.com/Fabstir/s5.js).
  • Existing helper libs: s5-encryptWasm, multiformats, sharp-wasm.
  • Monthly progress reports in the Sia Forum.

Community engagement & maintenance

Area Commitment
Progress updates Monthly forum posts.
Code reviews PRs opened early; iterate with maintainers.
Issue handling Best-effort review; no formal SLA.
CI GitHub Actions: lint, unit + browser tests.

Identity & Billing

Storage payments remain the S5-portal operator’s responsibility; no billing layer added.

References

Declaration

I certify that the information above is accurate and that all deliverables will be released under the MIT licence.

Contact info

1 Like
16

Technical Deep Dive: Why Enhanced s5.js Can Outperform OrbitDB

Following up on my grant proposal discussion - wanted to share the specific performance advantages that enhanced s5.js would bring compared to OrbitDB’s approach with IPFS.

Core Performance Gains:

1. Bounded Directory Traversal (vs OrbitDB’s full tree walks)

  • Problem: OrbitDB often needs to traverse entire directory structures to find relevant data

  • Our solution: Depth-limited queries that stop at specified levels

  • Result: getByPath("media/gallery", {depth: 2}) only fetches 2 levels deep, not the entire tree

2. Size-Aware Pagination (vs OrbitDB’s count-based only)

  • Problem: OrbitDB pagination by count can return unpredictable data sizes (20 tiny files vs 20 huge videos)

  • Our solution: Hybrid limits - {limit: 20, maxBytes: 5MB} whichever comes first

  • Result: Predictable bandwidth usage, especially crucial for mobile users

  • Performance: Prevents accidental 100MB+ page loads

3. Single Registry Updates (vs OrbitDB’s multi-step consensus)

  • Problem: OrbitDB requires multiple network round-trips for writes due to its consensus mechanism

  • Our solution: Path-based updates that cascade changes up to root in one registrySet call

  • Result: putByPath("docs/file.txt", data) = one network operation, not several

4. Native Content-Addressed Caching (vs OrbitDB’s external caching)

  • Built-in IndexedDB/memory cache that understands S5’s CID structure

  • Automatic cache invalidation when content changes

  • Thumbnail cache integration - no duplicate storage

Real-World Impact Examples:

Media Gallery Loading:

So for a scenario where 1000+ images are stored and client is requesting 12 thumbnail images per page.


OrbitDB: Fetch entire manifest + 12 full-size images for thumbnails

Enhanced s5.js: Fetch page 1 manifest + 12 pre-generated thumbnails

Result: ~90% bandwidth reduction

As well as implementing, I’d be making S5 inherently more efficient for these common operations.

1 Like
17

Thanks for your proposal to The Sia Foundation Grants Program.

After review, the committee has decided to approve your proposal. Congratulations! They’re excited to see what you can accomplish with this grant.

We’ll reach out to your provided email address for onboarding. This shouldn’t take long unless your info has changed from last time, but you may still need to adjust your timelines.

18

Thank you! I’m delighted that the committee has approved the proposal. Looking forward to contributing the technical and performance enhancements directly to the core s5.js library.

Here’s to making S5 even more accessible for developers! :rocket:

Cheers,
Jules

1 Like
19

Monthly Progress Report - Month 1 (Enhanced s5.js Grant)

What progress was made on your grant this month?

• Completed comprehensive architectural design for Enhanced s5.js, documenting path-based APIs, DAG-CBOR serialisation, and HAMT sharding implementation
• Consulted with Redsolver (s5 author) who provided valuable architectural guidance and reviewed the initial design approach
• Established repository structure with CI/CD pipeline configuration and development toolchain
• Created detailed DirV1 type definitions that match the Rust S5 v1 implementation, ensuring cross-implementation compatibility
• Designed deterministic CBOR encoding configuration to guarantee consistent content addressing across platforms
• Developed comprehensive media processing architecture with WASM foundation and browser compatibility layers

List repos worked on this month with links to PRs and relevant commits.

• Link to Enhanced S5_js - Revised Code Design.md
• Link to Enhanced S5_js - Revised Code Design - part II.md

What will you be working on next month?

• Implement DirV1 serialisation/deserialisation with DAG-CBOR, including proper integer key mappings matching Rust’s #[n(X)] attributes
• Develop basic path-based get/put operations for single-level directories with automatic directory creation
• Create comprehensive unit tests for deterministic CBOR encoding to ensure consistent hashing
• Begin integration with existing S5 API infrastructure for the new v1 specification

20

Hello, @juleslai!

My name is John and I’m an engineer for the Sia Foundation. I’m reviewing this month’s progress and had a couple of questions. In your updated grant plan (Standard Grant: S5 Gateway & TypeScript Client - #15 by juleslai), your first milestone is as follows:

Project set-up & design Repos, branching strategy, FS5 test fixtures; code-contribution guidelines & project board; tool-chain investigation; one-off business overhead

I’ve read through Part I of your Revised Code Design and skimmed the second part (will read this more thoroughly as we get around that fourth month). I recognize that this is significant work towards the grant. You’ve provided starting code/design for many of the most important pieces of it. It is thoughtful work and a great way to start any project, imo.

With that said, though, how much of the above milestone items did we accomplish? I want to be clear that my and any other Sia reviewers’ primary task here is to check for those things you stated you would accomplish after the month you stated you would accomplish them. We look for tight alignment between the report you submit to us and the milestone items you’ve scheduled to be done at this time.

Is there a repo that’s not in your report that I need to see? I’ve looked through your github account’s repos but could have missed it. Is it perhaps private? Put another way: in what repo did you accomplish “Established repository structure with CI/CD pipeline configuration and development toolchain” ? That may get us on the same page here. I don’t think you’re referring to this linked design repo, right?

21

Hi John,

Thank you for reviewing my progress report. I’ll clarify what was accomplished and the current approach:

The milestone called for “project set-up & design” and that’s exactly what was delivered in detail:

Repository Structure & Approach: After consulting with Redsolver and the community, we determined that direct contributions to the existing s5.js repository will be more effective than creating separate wrapper repositories. This represents better technical alignment with the S5 ecosystem.
Toolchain & Development Approach: Established TypeScript development environment with TDD methodology. For the WASM components (months 4-5), I’ll create a separate Rust repository with appropriate CI/CD for WASM compilation.
Code Contribution Guidelines: Working directly with Redsolver to ensure contributions align with s5.js project standards and S5 v1 specification compliance.

Current Month’s Focus:
Month 2 will deliver the first working code: DirV1 serialisation/deserialisation and basic path-based operations, as outlined in my technical design.
The design work completed last month was to ensure the implementation will be architecturally sound and compatible with the broader S5 ecosystem.

Repository Links:
Implementation work will be submitted as pull requests to s5-dev/s5.js. The Rust WASM components will have their own repository when we reach months 4-5.

Hope that clarifies,
Jules

1 Like