Standard Grant: S5 Gateway & TypeScript Client

Introduction

Project Name:
S5 Gateway & TypeScript Client – a Skynet-style API for modern Sia storage

Name of the organisation or individual submitting the proposal:
Jules Lai (sole developer, founder of Fabstir)


Describe your project

This project builds upon my previous grants for developing the Fabstir Media Player and work on s5client-js. It revives the ease-of-use once offered by Skynet portals, but on the modern S5 + Sia stack.

  • @fabstir/s5-gateway – a lightweight Node/TypeScript service that fronts any S5 portal.
    It provides simple REST routes for uploading files, downloading via /<CID>/<path>, updating mutable registry entries and subscribing to changes over WebSocket.
    An optional DNSLink resolver lets site owners map a normal domain to an S5 CID by publishing dnslink=/s5/<cid> in their DNS.

  • @s5-client/core – a browser/Node SDK mirroring the original Skynet Client (uploadFile, uploadDirectory, getJSON, setRegistryEntry, subscribeRegistry).
    Developers can integrate decentralised storage in minutes without touching renterd or contract logic.

Both components will be MIT-licensed, fully open-source and ship with a Docker-compose demo stack for local testing.


Who benefits from your project?

  • Web-3 developers – regain the “one-call” upload workflow that disappeared with Skynet.
  • Existing Sia users – obtain a thin, language-agnostic gateway that can be deployed in front of any portal.
  • End Users: Faster access to content through intelligent caching
  • dApp communities – can serve static sites or app bundles from Sia using ordinary HTTPS links or DNSLink hostnames.
  • The Sia ecosystem – gains a public OpenAPI specification, shared test vectors and an interoperable manifest format, making it easier for others to build compatible gateways.
  • S5 Portal Operators: Enhanced functionality without additional complexity

How does the project serve the Foundation’s mission of user-owned data?

By reducing the barrier to entry for S5, the gateway and SDK allow users to self-host their files and web-apps on Sia while retaining cryptographic ownership (CIDs and Ed25519-signed registry entries).
No central account system or pay-walled portal is required; any user can run the gateway locally or point the SDK at a community instance, keeping data control firmly in the user’s hands. Open-source codebase and standardised interfaces allow users to verify how their data is handled.


Are you a resident of any jurisdiction on that list? No
Will your payment bank account be located in any jurisdiction on that list? No


Development Information

Will all of your project’s code be open-source?
Yes. All new code will be released under the MIT licence. The project may depend on existing open-source components (e.g. s5client-js, S5) but will not develop or embed closed-source code.

Links where code will be accessible for review

Do you agree to submit monthly progress reports?
Yes.


Grant Specifics

Motivation & Goals

Goal Outcome
Developer ergonomics One SDK call or REST request to upload & fetch content.
DNS-friendly naming Resolve any hostname publishing dnslink=/s5/<cid> TXT records.
Open specification Single OpenAPI 3.1 spec, Mermaid/C4 diagrams, copy-paste docs.
Interoperability Public test vectors & JSON schema so other gateways can match behaviour.
Separation of concerns Gateway is generic; Fabstir back-end merely uses it.
Sustainability MIT licence, automated CI tests, monthly progress updates.

Technical Approach

Components

Component Tasks
Low-level tools Existing s5client-js, s5-encryptWasm.
Gateway – @fabstir/s5-gateway • Express server.
• Routes: /upload, /download/:cid/*, /metadata, /registry, /subscribe, /resolve/:hostname.
• Internally uses s5client-js (upload, path-resolution, encryption, dedup).
• Configurable LRU + on-disk cache; relies on the S5 portal’s own pinning mechanism—no extra cron jobs.
SDK – @s5-client/core • Upload helpers (uploadFile, uploadDirectory).
• Manifest cache & resolvePath.
• Registry CAS helpers; WebSocket push.
• No dependency on Fabstir login—callers inject an Ed25519 key if needed.
Documentation OpenAPI 3.1 → Redoc; Mermaid C4 diagrams; Quick-start guide.
Interoperability tests s5-interop-tests repo (blob & directory CIDs) – gateway must serve byte-identical data.
Demo stack Docker compose (local only): S5 portal, s5-gateway, sample uploader. Requires S5_AUTH_TOKEN; a review token will be supplied to the Foundation.
Load testing & tuning k6 scripts establish baseline; later sprint meets or beats baseline.

DNSLink resolution

/resolve/:hostname fetches the _dnslink TXT record and returns the referenced CID. The feature can be disabled; raw CID paths always work.


Deliverables, Milestones & Budget

(eight-month schedule — ≈ 32 weeks)

Month Milestone (end-of-month deliverable) Budget (USD) Key checkpoint
1 Project set-up & design
• GitHub repos created
• OpenAPI skeleton & issue board
• Initial Docker-compose scaffold
• One-off business-overhead allowance
3 500 Spec repo visible; empty gateway container starts
2 SDK core v0.1
@s5-client/core with uploadFile, uploadDirectory, manifest cache & path resolver
4 500 SDK demo uploads a file & fetches it by CID
3 Mutable data & push
• Gateway /registry, /subscribe routes
• SDK v0.2 with CAS + WebSocket
4 500 Registry update & push demo
4 Core gateway REST routes
/upload, /download/:cid/*, /metadata fully functional
• Interop test CIDs served identically
4 000 File upload & path fetch via gateway
5 DNSLink resolver integration
/resolve/:hostname route, caching, tests
3 000 Domain with dnslink=/s5/<cid> resolves & loads
6 Cache & dedup optimisation
• k6 baseline captured
• LRU cache + content dedup added
• Report shows measurable improvement
4 500 Improved k6 report published
7 Docs & demo stack
• MkDocs + Redoc site
• Docker compose: S5 portal, gateway, sample uploader
• Tuning sprint meets or beats baseline
4 500 Docker Compose setup verified, and “Quick-start guide” successfully followed during internal testing.
8 Packaging, outreach & feedback buffer
• v1.0 tags, screencast, blog post
• Forum feedback addressed; minor doc updates
3 500 Final artefacts & forum announcement
Total 32 000 USD

Monthly disbursement of 4 000 USD; continuing payments are contingent on timely submission of the standard progress report.


Success Indicators

  • End-to-end upload → path fetch demo in the Docker stack.
  • Registry CAS conflict tests pass with automatic retries.
  • DNSLink resolution; demonstrate that resolving and fetching a domain via DNSLink adds only minimal overhead compared with a direct CID fetch, and document the observed latency in the performance report.
  • Gateway serves byte-identical data on public test vectors.
  • Successfully tested the Quick-start guide internally to ensure it is easy to follow and can be completed quickly without external assistance.

Performance data and k6 scripts will be published.


Community Engagement & Maintenance

Area Commitment
Progress updates Written update monthly in Sia Forum “Grants”.
Interoperability Maintain s5-interop-tests repo; accept PRs.
Issue handling Repos are active Fabstir projects; issues addressed as part of ongoing work.
Licensing & CI MIT licence; GitHub Actions build, test and publish Docker images.

Identity & Billing

The gateway accepts any Ed25519 key supplied by the caller; derivation from Fabstir’s SEA credentials is documented but optional. Storage payments remain the responsibility of the portal operator; the gateway does not add a second billing layer.


Risks & Mitigation

Risk Mitigation
July 2025 Sia protocol upgrade Track S5 portal release notes and upgrade guidance; timeline includes buffer for any required API changes.
S5 spec changes Adjust gateway during the optimisation sprint if required.
DNS outages Resolver caches TXT records; users can fall back to raw CIDs.

References

Contact info


Hey Jules,

Reviewed the grant proposal. Good ideas tackling a lightweight gateway and SDK on S5 – definitely valuable for developer ergonomics and providing a usable tool.

But I see some significant technical points that need realignment with how S5 is actually developing right now. Based on my experience with the protocol and past efforts, these are critical for the project’s success and relevance.

Here’s the main feedback:

  1. Replicating the “Skynet Client API” is Building on Legacy, Not the Future: The proposal aims to mirror the original Skynet Client API (uploadFile, uploadDirectory, getJSON, setRegistryEntry, etc.).

    • Problem: While S5 was inspired by Skynet, it has its own architecture and is moving forward with new, standardized specifications. Crucially, the FS5 (File System S5) spec is the intended way to handle directories and metadata going forward, replacing all previous manifest formats (Skynet’s old approach, S5’s earlier ones).
    • Building your SDK/gateway around the legacy Skynet Client API means it targets an interface and set of concepts that are being superseded by FS5 and the evolving S5 primitives. This won’t be compatible with the direction S5 is heading.
  2. s5client-js is the Wrong Dependency: You listed s5client-js as a base library. You must drop this. It’s based on old, experimental code and is not the foundation for building modern S5 tooling that works with FS5.

    • Solution: You need to build on Redsolver’s official s5.js library instead. It’s the correct (though still WIP) base, and contributing to it is the way forward for the community. Coordinate with Redsolver; using s5client-js will make the project incompatible long-term.
  3. Implement FS5, Don’t Create a New Manifest: The proposal talks about gaining “an interoperable manifest format.”

    • Problem: The interoperable format for S5 directories is the FS5 specification currently being developed. Don’t create another one within this project. https://xkcd.com/927/ :upside_down_face:
    • Action: Your work on directories/manifests must focus entirely on correctly implementing and testing against the FS5 spec.
  4. A Better SDK Goal: Cross-Protocol Abstraction: Instead of strictly mirroring a legacy API, a more impactful SDK (that the gateway could then use) would abstract across multiple decentralized storage protocols (like S5 and IPFS) using common concepts (Multiformats, Multibase). This would provide broader utility than just replicating a single, outdated interface.

Summary: Avoid replicating the old “Skynet Client API”; build for the actual, evolving S5 standard (especially FS5) and its primitives, using the correct base library (s5.js). Focus your manifest/interop efforts on implementing/supporting the FS5 spec. Consider the SDK’s design goal as potentially broader than just S5 legacy replication.

Getting the details right on interfacing with S5’s current direction is crucial for this project to be useful and relevant. Happy to give more input.

Kudos.

4 Likes

Hi pcfreak30,

Thanks for the detailed review!

Totally agree—aligning with FS5 and the official s5.js library is the right way to go. I’ll revise the proposal to:
• drop s5client-js in favour of Redsolver’s s5.js;
• implement directory handling against the draft FS5 spec (so no new manifest format);
• keep high-level helper names (uploadFile, etc.) that are easy to use, but map them to FS5 under the hood.

I’ll check with Redsolver how draft the FS5 specs are and the plans for s5.js as it’s missing some file handling that Skynet API had, that I could add in.

I’ll design the gateway and SDK to be CID-first. The code will parse any CIDv1, inspect the multicodec prefix, and handle the two S5 codecs (blob and registry).
For all other codecs it will return “unsupported protocol” until future extensions are added. This keeps the architecture open to IPFS or other networks later, without expanding the current proposal timeline.

3 Likes

To be clear a JS impl of FS5 exists, you just need to contribute s5.js/src/fs at 0b415bc5365855f10c0a28ec4564e8bf9060e1ad · s5-dev/s5.js · GitHub.

Cheers — Good to know that FS5 already handles the core “what files are in this directory” task that s5client-js did :)

What it doesn’t do is let you fetch a single file straight from the root CID, e.g.
/CID/path/to/myFile1.txt.
To achieve that the S5 Gateway will download the FS5 manifest, look up the child file’s CID and fetch the file’s content.

FS5 simply stores the list of files. It does not treat index.html (or any file) as a default page for HTTP requests, fall back to index.html when a path ends in /, set MIME-types, or handle HTTP Range requests. I’ll add those behaviours to the S5 Gateway.

Something I feel might be a good opportunity that fits in somewhere.

These have come to my mind repeatedly that S5 and probably at a higher abstraction should have this as a tool. helia-verified-fetch basically creates a fetch like api that abstracts out IPFS, and service-worker-gateway uses some of the principles earlier Lume work did as well in loading data with client-based proxying.

Not clear where this falls in your plans, but its DevX tooling that makes sense to be created at some point.

Just giving you some more ideas :P.

Thanks for the links!
The ipfs work is out of scope for the proposal but will certainly look at the architecture and ensure that helia-verified-fetch and a service-worker gateway can be plugged in later. Cheers.

Hey @juleslai, thanks for submitting this grant proposal!

I fully agree with @pcfreak30’s feedback on not replicating the old Skynet Client API, and also that using s5.js as the base for your project makes the most sense.

Many of the features you mention (like uploading files and DNSLink) are already supported by S5 nodes, though not by the TypeScript implementation yet. So a full S5 node implementation in TypeScript (based on s5.js) with these features could be useful, and your grant proposal kinda looks like that’s what you actually want to build :)

Some other features you mentioned (updating mutable registry entries, subscribing to changes over WebSocket) are possible with both the TypeScript and Dart libraries, and I’m not sure if I see the benefit of adding new HTTP endpoints for these (like Skynet had). The issue here is that data like signed registry entries returned from these endpoints should still be verified by clients, so they need to use (atleast some parts) of the S5 library anyways, and could simply use the full lib with all operations over a websocket connection using the s5 p2p protocol between nodes directly.

Regarding the FS5 implementation in s5.js, you’re correct in that it’s missing some of the more advanced features like thumbnails for media files and could also benefit from some more quality-of-life utils for working with directory structures. So if you would like to work on adding these, they should likely be just implemented in s5.js directly.

On the topic of ipfs’s verified-fetch and their service-worker-gateway, S5 actually already has a fully working variant of that for both plaintext and encrypted files: GitHub - s5-dev/web-proxy: Client-side gateway with verified streaming, running in the web browser
It does however not support resolving and browsing FS5 directory structures yet, so that could be a useful addition.

Hi @Redsolver,

Thanks for the reply. Good news that you have implemented a lot of the basic operations that I don’t have to code now :)

Yep, can add to s5.js:
Thumbnail + media metadata pipeline – small JPEG/WebP previews, basic EXIF/size detection.
Directory utilities – walk large FS5 trees, filter by media type, cache manifests.

I believe it would be great to add more general key/value store operations. What do you think?
I plan to extend s5.js with:

  1. Path-based API

    • Add getByPath(rootKey, path) and putByPath(rootKey, path, data) helpers that transparently fetch, traverse, and update nested FS5 manifests.
  2. LWW-Register CRDT support

    • Wrap registry updates in a Last-Writer-Wins register (using the entry’s revision counter as the timestamp) to handle concurrent writes at a per-field or per-object level ([Wikipedia][1]).
  3. Manifest caching & sharding

    • Integrate an in-memory (or IndexedDB) cache for manifest CIDs,
    • And support HAMT/B-Tree sharding metadata in the FS5DirectoryHeader to efficiently handle large directories ([S5 Network Docs][2]).

My goal is to give JS/TS developers OrbitDB-like get/put semantics on S5, with built-in conflict resolution, low latency, and scalable directories. I noticed the FS5 spec still marks sharding as “TODO” in the header—has any sharding support been implemented yet, or is that still on the roadmap?

The issue I was trying to highlight was that the S5.js library maybe a bit too low-level detailed for some Web3 devs. The “<cid>/path” system is not as easy to use as a general “path” system where “a/b” gives back content from e.g. paths “a/b/c”, “a/b/d” and “a/b/c/e”. With optional parameters such as max depth level, another option or function to just retrieve all the child CIDs rather than content etc. The implementation I have would have only one registrySet call per file update. So a change in the lowest level “folder” would propagate manifest blob changes up to the root and then I call one registrySet for the new CID.

Now another question is, would it better to put this in another repo e.g. s5ext.js ?

I have limited access to when I can sit down on the internet for next few days. I will redo the grant proposal with changes from the discussions after that :)

imo any work that ties into s5 core directly should be a PR to the library itself. This is all a community effort after all.

It can be spun out later if its architecturally decided that a separate high level lib is needed and to keep high/low boundaries.

Makes sense. Cheers.

Here is the updated grant proposal:

Introduction

Project Name:
Enhanced s5.js – Advanced Directory Handling and Media Support

Applicant:
Jules Lai (sole developer, founder of Fabstir)

Describe your project

The official s5.js library provides fundamental S5 operations, but developers need higher-level features for efficient application development:

  • Simple path-based storage: “Store this data at /profile/settings”
  • Media handling: “Generate thumbnails for all images in /gallery”
  • Reliable updates: “Handle concurrent writes safely”

This grant adds these developer-friendly features directly into s5.js, making it easier to build robust applications on S5.

Planned enhancements

  1. Path-based APIgetByPath(rootKey, path) / putByPath(rootKey, path, data).
    Traverses nested FS5 manifests, fetches or updates the target blob, and bubbles changed CIDs up to the root with a single registrySet.

  2. LWW-register wrapper — optional Last-Writer-Wins conflict-resolution around registry updates using the revision counter as timestamp.

  3. Thumbnail & media metadata pipeline — JPEG /WebP previews, basic EXIF/size detection and helper that returns { thumbnailCID, width, height, mime }.

  4. Directory utilities — Directory walker with optional limit / cursor pagination; integrates in-memory or IndexedDB manifest cache.

  5. FS5 sharding groundwork — implement the HAMT/B-Tree header fields marked TODO and add split / merge helpers so very large directories can be handled once the spec finalises.

These enhancements complement the existing s5-dev/web-proxy project, which provides client-side gateway functionality with verified streaming. While the web-proxy handles content delivery, my additions to s5.js will enable better directory browsing and media handling within that environment. The thumbnail pipeline and directory utilities are particularly valuable for web-proxy users, as they enable efficient browsing of large media collections directly in the browser. This maintains the web-proxy’s commitment to client-side verification whilst adding rich media capabilities that users expect from modern web applications.

All code will be submitted as pull requests to the upstream s5.js repository; a temporary fork lives under Fabstir until each PR is merged.

Who benefits?

Audience Benefit
JS/TS Web-3 developers Simple get/put path semantics and thumbnails instead of hand-crafting FS5 blobs.
S5 core library Gains high-level utilities without fragmenting into third-party forks.
Fabstir ecosystem Can store app bundles and user media with one call, paving the way for a decentralised app store.
Sia community Higher-level tooling encourages broader adoption of S5 + FS5.

Mission alignment

By making S5 easier to build on, more users will self-host data on Sia while retaining cryptographic control of their CIDs and registry keys—directly serving the Foundation’s goal of user-owned data.

Compliance

Resident of prohibited jurisdiction? No
Payment bank in prohibited jurisdiction? No


Grant Specifics

Amount requested

49 600 USD — paid in equal 6 200 USD monthly tranches (standard Foundation schedule).

Timeline & milestones (eight months)

Month Target date Milestone & deliverables Budget (USD)
1 7/2/25 Project set-up & design Repos, branching strategy, FS5 test fixtures; code-contribution guidelines & project board; tool-chain investigation; one-off business overhead 6 200
2 8/2/25 Path helpers v0.1 — basic get/put for single-level directories, unit tests 6 200
3 9/2/25 Path-cascade optimisation — multi-level update with single registrySet; docs & examples 6 200
4 10/2/25 WASM foundation & basic media — pipeline setup, code-splitting, image metadata extraction, baseline performance recorded 6 200
5 11/2/25 Advanced media processing — building on Month 4: JPEG / PNG / WebP thumbnails (≤ 64 kB average), progressive rendering, browser test matrix; bundle ≤ 700 kB compressed 6 200
6 12/2/25 Directory utilities & caching — walker with limit/cursor pagination, IndexedDB/in-memory cache, filtered listings, performance benchmarks 6 200
7 1/2/26 Sharding groundwork — HAMT header fields, split/merge helpers, integration tests 6 200
8 2/2/26 Documentation site update, demo scripts, screencast, forum feedback buffer, PRs merged upstream 6 200
Total 49 600 USD

The Foundation disburses 6 200 USD per month; internal line items show expected effort but may vary slightly while the total remains capped at 49 600 USD.


Technical Implementation Details

Media Processing Strategy

  • Primary Implementation (Month 4):

    • WASM module setup with code splitting
    • Lazy loading for optimal bundle size
    • Basic image metadata extraction
    • Browser compatibility testing suite
    • Performance baseline establishment
  • Advanced Features (Month 5):

    • Thumbnail generation with quality/size options
    • Support for JPEG, PNG, WebP formats
    • Progressive loading for large images
    • Fallback = skip thumbnail if local WASM fails
    • Comprehensive browser testing

Caching Implementation

  • Tiered Storage Approach:

    • In-memory LRU cache for frequent access
    • IndexedDB for persistent storage
    • Configurable cache size limits
    • Automatic cache invalidation
    • Browser storage quota management
  • Performance Targets:

    • Sub-100ms access for cached items
    • Efficient bulk operations
    • Minimal memory footprint
    • Graceful degradation strategy

Success Indicators

  • putByPath updates a deep file and results in exactly one registrySet.
  • Concurrent writes resolve correctly under the LWW wrapper (demonstrated in tests).
  • Thumbnail helper generates a preview ≤ 64 kB average size (95 th percentile) and correct MIME metadata for PNG, JPEG, WebP.
  • Directory walker lists 10 000 cached entries in ≤ 2 s on a typical laptop.
  • Thumbnail generation completes in ≤ 500 ms for a 1 MP JPEG on a mid-range laptop; larger images scale linearly.

Risks & Mitigation

Risk Mitigation
WASM bundle size exceeds targets Keep the thumbnail module ≤ 700 kB compressed
Browser compatibility issues Comprehensive testing matrix, graceful degradation paths, polyfills where needed
Thumbnail processing performance Progressive quality settings, worker thread implementation, cancel stale operations
Cache storage limitations Implement LRU eviction, respect browser quotas, clear warning when approaching limits
Fallback service availability Local processing first, clear error handling

*Fallback = If local WASM fails, thumbnail generation is skipped; no external upload is attempted.


Development Information

  • All code MIT-licensed.
  • Upstream PR target: https://github.com/s5-dev/s5.js (temporary fork at https://github.com/Fabstir/s5.js).
  • Existing helper libs: s5-encryptWasm, multiformats, sharp-wasm.
  • Monthly progress reports in the Sia Forum.

Community engagement & maintenance

Area Commitment
Progress updates Monthly forum posts.
Code reviews PRs opened early; iterate with maintainers.
Issue handling Best-effort review; no formal SLA.
CI GitHub Actions: lint, unit + browser tests.

Identity & Billing

Storage payments remain the S5-portal operator’s responsibility; no billing layer added.


References


Declaration

I certify that the information above is accurate and that all deliverables will be released under the MIT licence.


Contact info


1 Like

Technical Deep Dive: Why Enhanced s5.js Can Outperform OrbitDB

Following up on my grant proposal discussion - wanted to share the specific performance advantages that enhanced s5.js would bring compared to OrbitDB’s approach with IPFS.

Core Performance Gains:

1. Bounded Directory Traversal (vs OrbitDB’s full tree walks)

  • Problem: OrbitDB often needs to traverse entire directory structures to find relevant data

  • Our solution: Depth-limited queries that stop at specified levels

  • Result: getByPath("media/gallery", {depth: 2}) only fetches 2 levels deep, not the entire tree

2. Size-Aware Pagination (vs OrbitDB’s count-based only)

  • Problem: OrbitDB pagination by count can return unpredictable data sizes (20 tiny files vs 20 huge videos)

  • Our solution: Hybrid limits - {limit: 20, maxBytes: 5MB} whichever comes first

  • Result: Predictable bandwidth usage, especially crucial for mobile users

  • Performance: Prevents accidental 100MB+ page loads

3. Single Registry Updates (vs OrbitDB’s multi-step consensus)

  • Problem: OrbitDB requires multiple network round-trips for writes due to its consensus mechanism

  • Our solution: Path-based updates that cascade changes up to root in one registrySet call

  • Result: putByPath("docs/file.txt", data) = one network operation, not several

4. Native Content-Addressed Caching (vs OrbitDB’s external caching)

  • Built-in IndexedDB/memory cache that understands S5’s CID structure

  • Automatic cache invalidation when content changes

  • Thumbnail cache integration - no duplicate storage

Real-World Impact Examples:

Media Gallery Loading:

So for a scenario where 1000+ images are stored and client is requesting 12 thumbnail images per page.


OrbitDB: Fetch entire manifest + 12 full-size images for thumbnails

Enhanced s5.js: Fetch page 1 manifest + 12 pre-generated thumbnails

Result: ~90% bandwidth reduction

As well as implementing, I’d be making S5 inherently more efficient for these common operations.

1 Like

Thanks for your proposal to The Sia Foundation Grants Program.

After review, the committee has decided to approve your proposal. Congratulations! They’re excited to see what you can accomplish with this grant.

We’ll reach out to your provided email address for onboarding. This shouldn’t take long unless your info has changed from last time, but you may still need to adjust your timelines.

Thank you! I’m delighted that the committee has approved the proposal. Looking forward to contributing the technical and performance enhancements directly to the core s5.js library.

Here’s to making S5 even more accessible for developers! :rocket:

Cheers,
Jules

1 Like