Standard Grant: S5 Gateway & TypeScript Client

Grants Proposed
1

Introduction

Project Name:
S5 Gateway & TypeScript Client – a Skynet-style API for modern Sia storage

Name of the organisation or individual submitting the proposal:
Jules Lai (sole developer, founder of Fabstir)

Describe your project

This project builds upon my previous grants for developing the Fabstir Media Player and work on s5client-js. It revives the ease-of-use once offered by Skynet portals, but on the modern S5 + Sia stack.

  • @fabstir/s5-gateway – a lightweight Node/TypeScript service that fronts any S5 portal.
    It provides simple REST routes for uploading files, downloading via /<CID>/<path>, updating mutable registry entries and subscribing to changes over WebSocket.
    An optional DNSLink resolver lets site owners map a normal domain to an S5 CID by publishing dnslink=/s5/<cid> in their DNS.

  • @s5-client/core – a browser/Node SDK mirroring the original Skynet Client (uploadFile, uploadDirectory, getJSON, setRegistryEntry, subscribeRegistry).
    Developers can integrate decentralised storage in minutes without touching renterd or contract logic.

Both components will be MIT-licensed, fully open-source and ship with a Docker-compose demo stack for local testing.

Who benefits from your project?

  • Web-3 developers – regain the “one-call” upload workflow that disappeared with Skynet.
  • Existing Sia users – obtain a thin, language-agnostic gateway that can be deployed in front of any portal.
  • End Users: Faster access to content through intelligent caching
  • dApp communities – can serve static sites or app bundles from Sia using ordinary HTTPS links or DNSLink hostnames.
  • The Sia ecosystem – gains a public OpenAPI specification, shared test vectors and an interoperable manifest format, making it easier for others to build compatible gateways.
  • S5 Portal Operators: Enhanced functionality without additional complexity

How does the project serve the Foundation’s mission of user-owned data?

By reducing the barrier to entry for S5, the gateway and SDK allow users to self-host their files and web-apps on Sia while retaining cryptographic ownership (CIDs and Ed25519-signed registry entries).
No central account system or pay-walled portal is required; any user can run the gateway locally or point the SDK at a community instance, keeping data control firmly in the user’s hands. Open-source codebase and standardised interfaces allow users to verify how their data is handled.

Are you a resident of any jurisdiction on that list? No
Will your payment bank account be located in any jurisdiction on that list? No

Development Information

Will all of your project’s code be open-source?
Yes. All new code will be released under the MIT licence. The project may depend on existing open-source components (e.g. s5client-js, S5) but will not develop or embed closed-source code.

Links where code will be accessible for review

Do you agree to submit monthly progress reports?
Yes.

Grant Specifics

Motivation & Goals

Goal Outcome
Developer ergonomics One SDK call or REST request to upload & fetch content.
DNS-friendly naming Resolve any hostname publishing dnslink=/s5/<cid> TXT records.
Open specification Single OpenAPI 3.1 spec, Mermaid/C4 diagrams, copy-paste docs.
Interoperability Public test vectors & JSON schema so other gateways can match behaviour.
Separation of concerns Gateway is generic; Fabstir back-end merely uses it.
Sustainability MIT licence, automated CI tests, monthly progress updates.

Technical Approach

Components

Component Tasks
Low-level tools Existing s5client-js, s5-encryptWasm.
Gateway – @fabstir/s5-gateway • Express server.
• Routes: /upload, /download/:cid/*, /metadata, /registry, /subscribe, /resolve/:hostname.
• Internally uses s5client-js (upload, path-resolution, encryption, dedup).
• Configurable LRU + on-disk cache; relies on the S5 portal’s own pinning mechanism—no extra cron jobs.
SDK – @s5-client/core • Upload helpers (uploadFile, uploadDirectory).
• Manifest cache & resolvePath.
• Registry CAS helpers; WebSocket push.
• No dependency on Fabstir login—callers inject an Ed25519 key if needed.
Documentation OpenAPI 3.1 → Redoc; Mermaid C4 diagrams; Quick-start guide.
Interoperability tests s5-interop-tests repo (blob & directory CIDs) – gateway must serve byte-identical data.
Demo stack Docker compose (local only): S5 portal, s5-gateway, sample uploader. Requires S5_AUTH_TOKEN; a review token will be supplied to the Foundation.
Load testing & tuning k6 scripts establish baseline; later sprint meets or beats baseline.

DNSLink resolution

/resolve/:hostname fetches the _dnslink TXT record and returns the referenced CID. The feature can be disabled; raw CID paths always work.

Deliverables, Milestones & Budget

(eight-month schedule — ≈ 32 weeks)

Month Milestone (end-of-month deliverable) Budget (USD) Key checkpoint
1 Project set-up & design
• GitHub repos created
• OpenAPI skeleton & issue board
• Initial Docker-compose scaffold
• One-off business-overhead allowance		 3 500 Spec repo visible; empty gateway container starts
2 SDK core v0.1
• @s5-client/core with uploadFile, uploadDirectory, manifest cache & path resolver		 4 500 SDK demo uploads a file & fetches it by CID
3 Mutable data & push
• Gateway /registry, /subscribe routes
• SDK v0.2 with CAS + WebSocket		 4 500 Registry update & push demo
4 Core gateway REST routes
• /upload, /download/:cid/*, /metadata fully functional
• Interop test CIDs served identically		 4 000 File upload & path fetch via gateway
5 DNSLink resolver integration
• /resolve/:hostname route, caching, tests		 3 000 Domain with dnslink=/s5/<cid> resolves & loads
6 Cache & dedup optimisation
• k6 baseline captured
• LRU cache + content dedup added
• Report shows measurable improvement		 4 500 Improved k6 report published
7 Docs & demo stack
• MkDocs + Redoc site
• Docker compose: S5 portal, gateway, sample uploader
• Tuning sprint meets or beats baseline		 4 500 Docker Compose setup verified, and “Quick-start guide” successfully followed during internal testing.
8 Packaging, outreach & feedback buffer
• v1.0 tags, screencast, blog post
• Forum feedback addressed; minor doc updates		 3 500 Final artefacts & forum announcement
Total 32 000 USD

Monthly disbursement of 4 000 USD; continuing payments are contingent on timely submission of the standard progress report.

Success Indicators

  • End-to-end upload → path fetch demo in the Docker stack.
  • Registry CAS conflict tests pass with automatic retries.
  • DNSLink resolution; demonstrate that resolving and fetching a domain via DNSLink adds only minimal overhead compared with a direct CID fetch, and document the observed latency in the performance report.
  • Gateway serves byte-identical data on public test vectors.
  • Successfully tested the Quick-start guide internally to ensure it is easy to follow and can be completed quickly without external assistance.

Performance data and k6 scripts will be published.

Community Engagement & Maintenance

Area Commitment
Progress updates Written update monthly in Sia Forum “Grants”.
Interoperability Maintain s5-interop-tests repo; accept PRs.
Issue handling Repos are active Fabstir projects; issues addressed as part of ongoing work.
Licensing & CI MIT licence; GitHub Actions build, test and publish Docker images.

Identity & Billing

The gateway accepts any Ed25519 key supplied by the caller; derivation from Fabstir’s SEA credentials is documented but optional. Storage payments remain the responsibility of the portal operator; the gateway does not add a second billing layer.

Risks & Mitigation

Risk Mitigation
July 2025 Sia protocol upgrade Track S5 portal release notes and upgrade guidance; timeline includes buffer for any required API changes.
S5 spec changes Adjust gateway during the optimisation sprint if required.
DNS outages Resolver caches TXT records; users can fall back to raw CIDs.

References

Contact info

2

Hey Jules,

Reviewed the grant proposal. Good ideas tackling a lightweight gateway and SDK on S5 – definitely valuable for developer ergonomics and providing a usable tool.

But I see some significant technical points that need realignment with how S5 is actually developing right now. Based on my experience with the protocol and past efforts, these are critical for the project’s success and relevance.

Here’s the main feedback:

  1. Replicating the “Skynet Client API” is Building on Legacy, Not the Future: The proposal aims to mirror the original Skynet Client API (uploadFile, uploadDirectory, getJSON, setRegistryEntry, etc.).

    • Problem: While S5 was inspired by Skynet, it has its own architecture and is moving forward with new, standardized specifications. Crucially, the FS5 (File System S5) spec is the intended way to handle directories and metadata going forward, replacing all previous manifest formats (Skynet’s old approach, S5’s earlier ones).
    • Building your SDK/gateway around the legacy Skynet Client API means it targets an interface and set of concepts that are being superseded by FS5 and the evolving S5 primitives. This won’t be compatible with the direction S5 is heading.

  2. s5client-js is the Wrong Dependency: You listed s5client-js as a base library. You must drop this. It’s based on old, experimental code and is not the foundation for building modern S5 tooling that works with FS5.

    • Solution: You need to build on Redsolver’s official s5.js library instead. It’s the correct (though still WIP) base, and contributing to it is the way forward for the community. Coordinate with Redsolver; using s5client-js will make the project incompatible long-term.

  3. Implement FS5, Don’t Create a New Manifest: The proposal talks about gaining “an interoperable manifest format.”

    • Problem: The interoperable format for S5 directories is the FS5 specification currently being developed. Don’t create another one within this project. https://xkcd.com/927/ :upside_down_face:
    • Action: Your work on directories/manifests must focus entirely on correctly implementing and testing against the FS5 spec.

  4. A Better SDK Goal: Cross-Protocol Abstraction: Instead of strictly mirroring a legacy API, a more impactful SDK (that the gateway could then use) would abstract across multiple decentralized storage protocols (like S5 and IPFS) using common concepts (Multiformats, Multibase). This would provide broader utility than just replicating a single, outdated interface.

Summary: Avoid replicating the old “Skynet Client API”; build for the actual, evolving S5 standard (especially FS5) and its primitives, using the correct base library (s5.js). Focus your manifest/interop efforts on implementing/supporting the FS5 spec. Consider the SDK’s design goal as potentially broader than just S5 legacy replication.

Getting the details right on interfacing with S5’s current direction is crucial for this project to be useful and relevant. Happy to give more input.

Kudos.

4 Likes
3

Hi pcfreak30,

Thanks for the detailed review!

Totally agree—aligning with FS5 and the official s5.js library is the right way to go. I’ll revise the proposal to:
• drop s5client-js in favour of Redsolver’s s5.js;
• implement directory handling against the draft FS5 spec (so no new manifest format);
• keep high-level helper names (uploadFile, etc.) that are easy to use, but map them to FS5 under the hood.

I’ll check with Redsolver how draft the FS5 specs are and the plans for s5.js as it’s missing some file handling that Skynet API had, that I could add in.

I’ll design the gateway and SDK to be CID-first. The code will parse any CIDv1, inspect the multicodec prefix, and handle the two S5 codecs (blob and registry).
For all other codecs it will return “unsupported protocol” until future extensions are added. This keeps the architecture open to IPFS or other networks later, without expanding the current proposal timeline.

3 Likes
4

To be clear a JS impl of FS5 exists, you just need to contribute s5.js/src/fs at 0b415bc5365855f10c0a28ec4564e8bf9060e1ad · s5-dev/s5.js · GitHub.

5

Cheers — Good to know that FS5 already handles the core “what files are in this directory” task that s5client-js did :)

What it doesn’t do is let you fetch a single file straight from the root CID, e.g.
/CID/path/to/myFile1.txt.
To achieve that the S5 Gateway will download the FS5 manifest, look up the child file’s CID and fetch the file’s content.

FS5 simply stores the list of files. It does not treat index.html (or any file) as a default page for HTTP requests, fall back to index.html when a path ends in /, set MIME-types, or handle HTTP Range requests. I’ll add those behaviours to the S5 Gateway.

6

Something I feel might be a good opportunity that fits in somewhere.

These have come to my mind repeatedly that S5 and probably at a higher abstraction should have this as a tool. helia-verified-fetch basically creates a fetch like api that abstracts out IPFS, and service-worker-gateway uses some of the principles earlier Lume work did as well in loading data with client-based proxying.

Not clear where this falls in your plans, but its DevX tooling that makes sense to be created at some point.

Just giving you some more ideas :P.

7

Thanks for the links!
The ipfs work is out of scope for the proposal but will certainly look at the architecture and ensure that helia-verified-fetch and a service-worker gateway can be plugged in later. Cheers.

8

Hey @juleslai, thanks for submitting this grant proposal!

I fully agree with @pcfreak30’s feedback on not replicating the old Skynet Client API, and also that using s5.js as the base for your project makes the most sense.

Many of the features you mention (like uploading files and DNSLink) are already supported by S5 nodes, though not by the TypeScript implementation yet. So a full S5 node implementation in TypeScript (based on s5.js) with these features could be useful, and your grant proposal kinda looks like that’s what you actually want to build :)

Some other features you mentioned (updating mutable registry entries, subscribing to changes over WebSocket) are possible with both the TypeScript and Dart libraries, and I’m not sure if I see the benefit of adding new HTTP endpoints for these (like Skynet had). The issue here is that data like signed registry entries returned from these endpoints should still be verified by clients, so they need to use (atleast some parts) of the S5 library anyways, and could simply use the full lib with all operations over a websocket connection using the s5 p2p protocol between nodes directly.

Regarding the FS5 implementation in s5.js, you’re correct in that it’s missing some of the more advanced features like thumbnails for media files and could also benefit from some more quality-of-life utils for working with directory structures. So if you would like to work on adding these, they should likely be just implemented in s5.js directly.

On the topic of ipfs’s verified-fetch and their service-worker-gateway, S5 actually already has a fully working variant of that for both plaintext and encrypted files: GitHub - s5-dev/web-proxy: Client-side gateway with verified streaming, running in the web browser
It does however not support resolving and browsing FS5 directory structures yet, so that could be a useful addition.

9

Hi @Redsolver,

Thanks for the reply. Good news that you have implemented a lot of the basic operations that I don’t have to code now :)

Yep, can add to s5.js:
Thumbnail + media metadata pipeline – small JPEG/WebP previews, basic EXIF/size detection.
Directory utilities – walk large FS5 trees, filter by media type, cache manifests.

I believe it would be great to add more general key/value store operations. What do you think?
I plan to extend s5.js with:

  1. Path-based API

    • Add getByPath(rootKey, path) and putByPath(rootKey, path, data) helpers that transparently fetch, traverse, and update nested FS5 manifests.

  2. LWW-Register CRDT support

    • Wrap registry updates in a Last-Writer-Wins register (using the entry’s revision counter as the timestamp) to handle concurrent writes at a per-field or per-object level ([Wikipedia][1]).

  3. Manifest caching & sharding

    • Integrate an in-memory (or IndexedDB) cache for manifest CIDs,
    • And support HAMT/B-Tree sharding metadata in the FS5DirectoryHeader to efficiently handle large directories ([S5 Network Docs][2]).

My goal is to give JS/TS developers OrbitDB-like get/put semantics on S5, with built-in conflict resolution, low latency, and scalable directories. I noticed the FS5 spec still marks sharding as “TODO” in the header—has any sharding support been implemented yet, or is that still on the roadmap?

10

The issue I was trying to highlight was that the S5.js library maybe a bit too low-level detailed for some Web3 devs. The “<cid>/path” system is not as easy to use as a general “path” system where “a/b” gives back content from e.g. paths “a/b/c”, “a/b/d” and “a/b/c/e”. With optional parameters such as max depth level, another option or function to just retrieve all the child CIDs rather than content etc. The implementation I have would have only one registrySet call per file update. So a change in the lowest level “folder” would propagate manifest blob changes up to the root and then I call one registrySet for the new CID.

Now another question is, would it better to put this in another repo e.g. s5ext.js ?

11

I have limited access to when I can sit down on the internet for next few days. I will redo the grant proposal with changes from the discussions after that :)

12

imo any work that ties into s5 core directly should be a PR to the library itself. This is all a community effort after all.

It can be spun out later if its architecturally decided that a separate high level lib is needed and to keep high/low boundaries.

13

Makes sense. Cheers.