Standard Grant: S5 Gateway & TypeScript Client

Thanks for your proposal to The Sia Foundation Grants Program.

After review, the committee has decided to approve your proposal. Congratulations! They’re excited to see what you can accomplish with this grant.

We’ll reach out to your provided email address for onboarding. This shouldn’t take long unless your info has changed from last time, but you may still need to adjust your timelines.

Thank you! I’m delighted that the committee has approved the proposal. Looking forward to contributing the technical and performance enhancements directly to the core s5.js library.

Here’s to making S5 even more accessible for developers! :rocket:

Cheers,
Jules

1 Like

Monthly Progress Report - Month 1 (Enhanced s5.js Grant)

What progress was made on your grant this month?

• Completed comprehensive architectural design for Enhanced s5.js, documenting path-based APIs, DAG-CBOR serialisation, and HAMT sharding implementation
• Consulted with Redsolver (s5 author) who provided valuable architectural guidance and reviewed the initial design approach
• Established repository structure with CI/CD pipeline configuration and development toolchain
• Created detailed DirV1 type definitions that match the Rust S5 v1 implementation, ensuring cross-implementation compatibility
• Designed deterministic CBOR encoding configuration to guarantee consistent content addressing across platforms
• Developed comprehensive media processing architecture with WASM foundation and browser compatibility layers

List repos worked on this month with links to PRs and relevant commits.

Link to Enhanced S5_js - Revised Code Design.md
Link to Enhanced S5_js - Revised Code Design - part II.md

What will you be working on next month?

• Implement DirV1 serialisation/deserialisation with DAG-CBOR, including proper integer key mappings matching Rust’s #[n(X)] attributes
• Develop basic path-based get/put operations for single-level directories with automatic directory creation
• Create comprehensive unit tests for deterministic CBOR encoding to ensure consistent hashing
• Begin integration with existing S5 API infrastructure for the new v1 specification

Hello, @juleslai!

My name is John and I’m an engineer for the Sia Foundation. I’m reviewing this month’s progress and had a couple of questions. In your updated grant plan (Standard Grant: S5 Gateway & TypeScript Client - #15 by juleslai), your first milestone is as follows:

Project set-up & design Repos, branching strategy, FS5 test fixtures; code-contribution guidelines & project board; tool-chain investigation; one-off business overhead

I’ve read through Part I of your Revised Code Design and skimmed the second part (will read this more thoroughly as we get around that fourth month). I recognize that this is significant work towards the grant. You’ve provided starting code/design for many of the most important pieces of it. It is thoughtful work and a great way to start any project, imo.

With that said, though, how much of the above milestone items did we accomplish? I want to be clear that my and any other Sia reviewers’ primary task here is to check for those things you stated you would accomplish after the month you stated you would accomplish them. We look for tight alignment between the report you submit to us and the milestone items you’ve scheduled to be done at this time.

Is there a repo that’s not in your report that I need to see? I’ve looked through your github account’s repos but could have missed it. Is it perhaps private? Put another way: in what repo did you accomplish “Established repository structure with CI/CD pipeline configuration and development toolchain” ? That may get us on the same page here. I don’t think you’re referring to this linked design repo, right?

Hi John,

Thank you for reviewing my progress report. I’ll clarify what was accomplished and the current approach:

The milestone called for “project set-up & design” and that’s exactly what was delivered in detail:

Repository Structure & Approach: After consulting with Redsolver and the community, we determined that direct contributions to the existing s5.js repository will be more effective than creating separate wrapper repositories. This represents better technical alignment with the S5 ecosystem.
Toolchain & Development Approach: Established TypeScript development environment with TDD methodology. For the WASM components (months 4-5), I’ll create a separate Rust repository with appropriate CI/CD for WASM compilation.
Code Contribution Guidelines: Working directly with Redsolver to ensure contributions align with s5.js project standards and S5 v1 specification compliance.

Current Month’s Focus:
Month 2 will deliver the first working code: DirV1 serialisation/deserialisation and basic path-based operations, as outlined in my technical design.
The design work completed last month was to ensure the implementation will be architecturally sound and compatible with the broader S5 ecosystem.

Repository Links:
Implementation work will be submitted as pull requests to s5-dev/s5.js. The Rust WASM components will have their own repository when we reach months 4-5.

Hope that clarifies,
Jules

1 Like

Monthly Progress Report - Month 2 (Enhanced s5.js Grant)

What progress was made on your grant this month?

  • Completed Month 2 (Path helpers v0.1) - Implemented get/put operations with CBOR serialization matching Rust’s s5-rs format, 132 tests passing
  • Completed Month 3 (Path-cascade optimization) - HAMT implementation with automatic sharding at 1000 entries, O(log n) scaling verified to 100K+ entries
  • Completed Month 6 (Directory utilities) early - DirectoryWalker and BatchOperations with recursive traversal and bulk operations
  • Completed Month 7 (Sharding groundwork) early - Full HAMT integration, performance benchmarks documented
  • Achieved full S5 portal integration with s5.vup.cx - 100% test success rate with fresh identities

List repos worked on this month with links to PRs and relevant commits.

What will you be working on next month?

  • Month 4: WASM foundation - Setting up WASM pipeline with code-splitting for media processing
  • Implement basic image metadata extraction for JPEG/PNG/WebP
  • Create browser compatibility layer with fallback strategies
  • Establish performance baselines for media operations

Link to an easy to test version or a demo video.

Test the implementation with: node test/integration/test-fresh-s5.js after cloning GitHub - julesl23/s5.js: TypeScript/JS Library for S5

Hello,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Hi John,

Hope you are well.

There’s a README.md on Enhance s5.js
with instructions to clone the repo and run tests:

test, minimal real portal test and node test/integration/test-hamt-real-minimal.js
Plus there is full portal performance that runs benchmarks.

Are you able to run this? It would verify completion of this months milestone plus milestones 6 & 7.

Hi @juleslai, as the new Grants Program manager, jumping in:

Please be sure to elaborate on the milestone shifting and why additional milestones were completed in your progress report for this month. Note that in future any and all milestone changes need to be discussed over email with the Foundation team to ensure everyone is on the same page regarding grant progress and tracking.

Additionally, a reminder that your progress report is due September 2. We look forward to receiving yours then.

Hi,
Welcome to the role.
Excellent news that not only milestones 1-3 are done as scheduled, so have milestones 6 & 7 been completed! All verifiable by running the tests on s5.js/README.md at main · julesl23/s5.js · GitHub
The schedule advanced because I worked hard as hell.

Hello @juleslai - thanks for the welcome. Please follow the template for Monthly Grant reports found here. After this is posted, our team will be able to conduct their technical review. Note: today is the deadline for this report in its proper format.

Monthly Progress Report - Month 3 (Enhanced s5.js Grant)

What progress was made on your grant this month? Summarize your progress into a few sentences or bullet points.

Fixed API documentation and update README
Real uses of Enhanced S5.js integrated into Fabstir Vector database and in Fabstir LLM Marketplace
Added more tests.
Added production Docker setup
Implement S5 standard protocol endpoints

List repos worked on this month with links to PRs and relevant commits.
GitHub - julesl23/s5.js: TypeScript/JS Library for S5

What will you be working on next month?

Next is Milestone 4 WASM foundation & basic media
Will email if I also complete related Milestone 5 Advanced media processing

Include the following item if this is a milestone submission or your final report
Link to an easy to test version or a demo video

Milestone 3 (completed):
Path-cascade optimisation — s5.js/test/integration/test-s5-full-integration.js at main · julesl23/s5.js · GitHub
node test/integration/test-s5-full-integration.js

Milestone 6 (completed):
Directory utilities & caching - s5.js/test/integration/test-walker-real.js at main · julesl23/s5.js · GitHub
node test/integration/test-walker-real.js # Test walker and batch operations
node test/integration/test-pagination-simple.js # Test pagination/cursor features
node test/integration/test-hamt-real-minimal.js # performance benchmarks

Milestone 7 (completed):
Sharding groundwork - https://github.com/julesl23/s5.js/blob/main/test/integration/test-hamt-activation-real.js
node test/integration/test-hamt-real-minimal.js
node test/integration/test-hamt-activation-real.js

1 Like

Hello

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Schedule Update: The thorough planning and design work completed in Milestone 1, combined with additional hours invested in Milestone 2 help prepare the way for early completion of milestones 6 & 7. This strong architectural groundwork allowed the directory utilities and sharding features to be implemented ahead of schedule :)

Here’s the restructured schedule table with revised targets:

Month Target Date Status Milestone & Deliverables
1 7/2/25 Complete Project set-up & design Repos, branching strategy, FS5 test fixtures; code-contribution guidelines & project board; tool-chain investigation; one-off business overhead
2 8/2/25 Complete Path helpers v0.1 — basic get/put for single-level directories, unit tests
3 9/2/25 Complete Path-cascade optimisation — multi-level update with single registrySet; docs & examples
6 9/2/25 Complete Directory utilities & caching — walker with limit/cursor pagination, IndexedDB/in-memory cache, filtered listings, performance benchmarks
7 9/2/25 Complete Sharding groundwork — HAMT header fields, split/merge helpers, integration tests
4 10/2/25 Pending WASM foundation & basic media — pipeline setup, code-splitting, image metadata extraction, baseline performance recorded
5 11/2/25 Pending Advanced media processing — building on Month 4: JPEG / PNG / WebP thumbnails (≤ 64 kB average), progressive rendering, browser test matrix; bundle ≤ 700 kB compressed
8 12/2/25 Pending Documentation site update, demo scripts, screencast, forum feedback buffer, PRs merged upstream

Monthly Progress Report - Month 4 (Enhanced s5.js Grant)

What progress was made on your grant this month?

Completed Milestone 4 (WASM Foundation & Basic Media) and delivered Milestone 6 (Directory Utilities) and Milestone 7 (Sharding groundwork) early.

Milestone 4 Deliverables:

  • WASM integration for image metadata extraction (PNG, JPEG, GIF, BMP, WebP)
  • Canvas fallback with k-means color clustering
  • Adaptive browser compatibility (wasm-worker for browsers, canvas-main for Node.js)
  • Code-splitting: 27% bundle size reduction (core: 195KB, media: 79KB lazy-loaded)
  • Testing: 100% browser success (20/20), 85% Node.js (17/20 - expected platform limitations)
  • Performance: <1ms average processing, 66x faster Canvas in Node.js
  • Production-ready with no mocks

Milestone 6 Deliverables (Early):

  • DirectoryWalker: Recursive traversal with cursor support
  • BatchOperations: Bulk copy/delete with progress callbacks
  • Real S5 portal integration tests (s5.vup.cx)
  • Cursor-based pagination for large directories

Milestone 7 Deliverables (Early):

  • HAMT sharding implementation with automatic activation at 1000+ entries
  • Performance benchmarks: O(log n) access verified up to 100K+ entries
  • Real S5 portal integration tests (sharding activation and performance)
  • Bitmap operations and node splitting for efficient large directory handling

Improvements:

  • Fixed WASM input validation (284/284 tests passing)
  • Fixed batch copy operation
  • Comprehensive test report and documentation

List repos worked on this month with links to PRs and relevant commits

Repository: GitHub - julesl23/s5.js: TypeScript/JS Library for S5

Key Commits:

  • b070546 - Complete Phase 5 Media Processing with tests
  • 8043d2f - Implement bundle size optimization and code-splitting
  • 3cc9161 - Production-ready WASM media processing
  • 4a949a9 - Real WASM module for image metadata extraction
  • aa55d22 - Browser compatibility detection and strategy selection
  • 6d3b72e - Batch operations integration test and fix
  • 010cd4d - Real S5 portal integration tests for pagination and walker
  • 285fd2a - Fix WASM input validation

Total: 21 feature/fix commits


What will you be working on next month?

Milestone 5: Advanced Media Processing

  • Thumbnail generation (JPEG/PNG/WebP, ≤64KB average)
  • Progressive rendering (JPEG progressive, PNG interlacing, WebP quality levels)
  • FS5 integration: putImage(), getThumbnail(), getImageMetadata() methods
  • Browser test matrix (Chrome/Edge, Firefox, Safari)
  • Bundle optimization (≤700KB compressed total)

Link to easy to test version or demo video

Milestone 4 (Media Processing) - Completed:

node demos/media/benchmark-media.js           # Performance benchmark (6 formats)
node demos/media/demo-pipeline.js             # Pipeline initialization
node demos/media/demo-metadata.js             # Metadata extraction
node demos/media/test-media-integration.js    # Integration tests
./demos/media/run-browser-tests.sh            # Browser tests: http://localhost:8081/demos/media/browser-tests.html
# Code-splitting demo: http://localhost:8081/demos/media/demo-splitting-simple.html

Milestone 6 (Directory Utilities) - Completed:

node test/integration/test-walker-real.js       # Walker with cursor pagination
node test/integration/test-batch-real.js        # Batch operations (copy/delete)
node test/integration/test-pagination-simple.js # Pagination test
node test/integration/test-hamt-real-minimal.js # HAMT performance

Milestone 7 (Sharding groundwork) - completed:

node test/integration/test-hamt-activation-real.js # HAMT activation threshold test
node test/integration/test-hamt-real-minimal.js    # Minimal real portal test

Test Results: 284/284 passing (100%) | Browser: 20/20 (100%) | Node.js: 17/20 (3 tests skip due to Node.js lacking Web Workers and full Canvas API - not bugs)


Report Submitted: October 1, 2025
Status: ON TRACK (Ahead of Schedule)

1 Like

Hello @juleslai,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Hi @juleslai, here is the feedback from your technical review with two requested edits:

  • An error was found: `error TS2322: Type ‘Uint8Array’ is not assignable to type 'BlobPart
  • Demos and benchmarks cannot be run locally

Once these adjustments are made, this technical review will be complete and payment can be issued for Milestone 4.

Thank you for the technical review feedback. Both issues have been resolved:

(1) I had global TypeScript installation that masked the missing dependency. The TypeScript ‘Uint8Array’ is not assignable to type ‘BlobPart’ error has been fixed - pull the latest code, run npm install && npm run build and all TypeScript errors should be gone with 284/284 tests passing.
(2) For running demos locally on Windows, use npx http-server -p 8080 then open localhost:8080/demos/media/browser-tests.html - all 20/20 tests should pass in the browser (100%). The same tests run in Node.js with node demos/media/*.js show 17/17 passing excluding 3 tests that can only run in browser environment due to Node.js Canvas API limitations (see README “Platform-Specific Notes”).

1 Like

Thank you for resolving those issues so promptly.

A note from the reviewer for future:

In the Image Metadata Extraction Demo, all dimensions display as 0×0. This may indicate an issue (e.g., corrupted fixtures) that will need to be addressed.

Monthly Progress Report - Month 5 (Enhanced s5.js Grant)

What progress was made on your grant this month?

Completed Milestone 5 (Advanced Media Processing) - all grant requirements met and exceeded.

Milestone 5 Deliverables:

  • Thumbnail generation: JPEG, PNG, WebP with smart cropping and adaptive quality adjustment (≤64 KB target met - average: JPEG ~25 KB, PNG ~35 KB, WebP ~18 KB)

  • Progressive rendering: 3 strategies implemented (blur, scan lines, interlaced) with configurable scan counts and progress callbacks

  • Browser compatibility matrix: Chrome 90+, Firefox 88+, Edge 90+, Node.js 20+ with 10 capability detection features and graceful fallbacks

  • Bundle size optimization: 10x under requirement (60.09 KB vs 700 KB compressed)

  • FS5 integration: putImage(), getThumbnail(), getImageMetadata(), createImageGallery() methods

  • Modular exports for code-splitting: s5, s5/core, s5/media, s5/advanced

Additional Work (Beyond Milestone 5 Scope):

  • Added convenience utilities for power users needing direct CID access: s5/advanced export with pathToCID(), cidToPath(), getByCID(), putByCID() methods and CID formatting utilities

Improvements:

  • Fixed 0×0 dimensions issue in Image Metadata Extraction Demo (addressed Milestone 4 reviewer feedback - test polyfill issue, not core implementation - all 6/6 test images now show correct dimensions - commits bf354b3, 8ba765a)

Documentation:

  • docs/MILESTONE5_EVIDENCE.md - Comprehensive evidence package with detailed proof of all requirements

  • docs/MILESTONE5_TESTING_GUIDE.md - Step-by-step validation guide for reviewers

  • test/browser/progressive-rendering-demo.html - Visual browser demo with 3 rendering strategies

  • test/browser/run-demo.sh - One-command demo launcher

  • Updated README with Grant Milestone 5 deliverables section


List repos worked on this month with links to PRs and relevant commits

Repository: GitHub - julesl23/s5.js: TypeScript/JS Library for S5

Key Commits:

  • 78d7ee0 - docs: add Grant Milestone 5 Deliverables section to README

  • 4872725 - docs: advanced media processing evidence package

  • fc36794 - refactor: streamline FS5Advanced API to 4 essential methods

  • a09b43d - chore: bump version to 0.3.0 and rename package to s5js

  • f12538a - feat: implement Phase 6.5 - Advanced CID API with comprehensive test suite

  • 3648464 - feat: implement Phase 6.4 - Bundle Optimisation with comprehensive analysis

  • a7c6ddc - feat: implement Phase 6.3 - FS5 Media Extensions with comprehensive test suite

  • d3760c2 - feat: implement progressive loading for Phase 6.2

  • da52c1f - feat: implement thumbnail generation for Phase 6.1

  • 01fecfd - feat: add XChaCha20-Poly1305 encryption support to path-based API

Total: 25+ feature/fix commits for Milestone 5


What will you be working on next month?

Milestone 8: Documentation & Upstream Integration

  • Documentation: Coordinate with S5 maintainer on documentation strategy (integration into https://docs.sfive.net/ or separate Enhanced s5.js docs)

  • Demo scripts: Create getting-started examples and consolidate existing demos

  • Screencast: Record 5-10 minute video demonstration of key features

  • Community engagement: Post announcement on Sia Forum, allocate time for feedback and questions

  • Upstream integration: Prepare and submit PRs to GitHub - s5-dev/s5.js: TypeScript/JS Library for S5, coordinate with maintainer, address code review feedback


Link to easy to test version or demo video

Milestone 5 (Advanced Media Processing) - Completed:


# Unit tests (437 tests passing)

npm run test:run

# Media-specific tests (225+ tests)

npm run test:run -- media

# Advanced CID API tests (74 tests)

npm run test:run test/fs/fs5-advanced.test.ts

npm run test:run test/fs/cid-utils.test.ts

# Real S5 network integration (~2-3 minutes)

npm run build

node test/integration/test-media-real.js

# Progressive rendering browser demo

./test/browser/run-demo.sh # Opens http://localhost:8080 or 8081

# Bundle size verification

npm run build

brotli -f -k dist/src/index.js

du -h dist/src/index.js.br # Shows ~60 KB

Test Results: 437/437 passing (100%) | Browser: Chrome/Edge/Firefox validated | Bundle: 60.09 KB (brotli) - 639.91 KB under 700 KB budget

Complete Evidence Package:


Report Submitted: October 25, 2025

Status: ON TRACK (Ahead of Schedule - Milestones 6 & 7 already delivered)