Standard Grant: Sluby - Developer-First Video Infrastructure on Sia [Resubmission of SiaStream]

Introduction

Project Name: Sluby

Name of the organization or individual submitting the proposal: Mert Köklü

Software engineer with 4+ years in blockchain tooling and infrastructure, former AI Video Analytics Team Lead at an NVIDIA partner company. Previous grant work delivered across 10+ protocols including Sia, Arbitrum, Starknet, dYdX, and ICP. Long-term Sia ecosystem contributor. Full background and project links in Development Information below.

This is a resubmission of a previously rejected Large Grant - SiaStream, revised to address all three points of committee feedback: the project name has been changed to comply with brand guidelines, and the budget has been restructured as a Standard Grant.

Describe your project.

Building video infrastructure today means assembling expensive, vendor-locked pipelines across multiple cloud services, or accepting managed platforms that hold plaintext access to all uploaded content.Sluby provides an alternative: a self-hosted, open-source video infrastructure platform built natively on Sia. Developers deploy the full stack on their own infrastructure via Docker Compose, connected to their own indexd instance. It delivers a complete pipeline for uploading, transcoding, storing, and streaming video content, comparable in developer experience to Mux or Cloudflare Stream, but backed by decentralized, encrypted, contract-enforced storage.

Developers interact with Sluby through a REST API, a TypeScript client SDK, and a drop-in React player component. Behind these interfaces, the platform handles large file uploads via the TUS protocol, an open standard for reliable, resumable file transfers over HTTP. It performs adaptive bitrate transcoding through FFmpeg, producing multiple quality levels that players switch between based on network conditions. Each output artifact is stored on Sia, with dynamic HLS manifest rewriting and low-latency delivery through a byte-range-aware HTTP gateway with a caching layer. A management Studio provides a web dashboard for uploading, monitoring processing, browsing assets, configuring API keys, and previewing playback.

Every video artifact is encrypted client-side by the Sia SDK before the local environment, then erasure-coded and distributed across independent storage providers so that the original data can be reconstructed from a subset of hosts. The erasure coding parameters are configurable, allowing operators to tune the balance between redundancy and storage cost. The platform treats video as a durable, content-addressed data asset under the developer’s cryptographic control.

A working reference implementation has been developed and validated on the Sia Zen testnet. It is currently live at https://siastream.com (migrating to sluby.app) and available for reviewer evaluation alongside a video walkthrough of the platform at https://youtu.be/-NnKGXTSVjw.

Who benefits from your project?

Application developers building video-enabled products gain a production-ready API that eliminates months of pipeline engineering. Instead of assembling separate upload, transcoding, storage, and delivery systems, a developer can upload a video, receive a playback URL, and embed the React player, all through a few API calls. Sia’s storage costs are roughly 5–10x lower than AWS S3, making a material difference for video workloads where storage and bandwidth dominate expenses.

The Sia network and its storage providers benefit from sustained, high-volume demand. Video is the largest category of internet traffic, and each video stored on Sluby generates ongoing storage contracts and bandwidth utilization. A modest platform hosting 10,000 hours of content at three quality tiers would store approximately 36 TB on Sia, directly increasing network utilization and Siacoin demand.

The Sia Foundation gains a high-visibility demonstration that Sia can support bandwidth-intensive, latency-sensitive workloads beyond archival storage, serving as a tangible showcase for the v2 architecture.

Content creators and organizations using applications built on Sluby inherit Sia’s structural privacy guarantees: all content is encrypted locally before upload, and storage providers hold only unintelligible ciphertext. This makes Sluby suitable for use cases requiring censorship resistance, data privacy compliance, or long-term integrity, from creator platforms and educational institutions to journalistic archives and enterprise tooling.

How does the project serve the Foundation’s mission of user-owned data? What problem does your project solve?

This project directly addresses the Grants Committee’s stated interest in funding tools for “sharing/viewing large single files (videos, data sets, high-res photos),” listed as a priority project idea in the April 2026 funding guidelines. It satisfies both required grant themes simultaneously: Sluby is built with the official Sia SDKs for all storage operations and built on indexd for object tracking, health monitoring, and repair coordination.

Despite proven capability for video streaming, the Sia ecosystem offers zero production-ready video infrastructure. Previous attempts have not delivered a sustainable solution: the original SiaStream was lost with Skynet Labs’ shutdown in 2022, DTube abandoned decentralization entirely in December 2025 due to IPFS storage unreliability, and community proposals like StreamWeave and Sia Live Archive did not reach completion. The gap persists because no project has solved storage, transcoding, delivery, and developer experience together.

Sluby solves this by providing a complete, abstracted video pipeline where all content is stored on Sia. The SDK encrypts data client-side before upload and distributes encrypted shards across independent providers via erasure coding, so no single entity can access plaintext video. Content ownership remains cryptographically with the application developer, a structural guarantee no centralized platform can offer.

The platform also makes Sia accessible to a broader developer audience. Web developers accustomed to S3 or Cloudflare Stream face a steep learning curve with Sia’s SDK and indexer workflows. Sluby abstracts this behind standard REST APIs, TUS uploads, and HLS delivery, so every application built on Sluby becomes an application built on Sia.

Are you a resident of any jurisdiction on that list? No

Will your payment bank account be located in any jurisdiction on that list? No


Grant Specifics

Amount of money requested and justification with a reasonable breakdown of expenses:

Total requested: $40,000 USD over 5 months ($8,000/month)

This budget covers a single full-time developer working across the entire stack. The reduced scope reflects lessons from prior committee feedback, focusing on delivering a core platform that external developers can adopt.

Category Monthly Cost 5-Month Total Description
Developer compensation $7,200 $36,000 Full-time development across backend, API, SDK, player, Studio, Sia integration, and documentation
Infrastructure (development) $800 $4,000 Cloud Server for development and testing, Sia storage contracts for mainnet operation, domain registration
Total $8,000 $40,000

No marketing, community engagement, or promotional expenses are included.


What is the high-level architecture overview for the grant, i.e. how does your project build on Sia? What security best practices are you following?

Architecture Overview

Sluby is a fully self-hosted platform deployed via Docker Compose on the developer’s own infrastructure. No third-party server is involved — the developer runs the entire stack connected to their own indexd instance. The architecture separates a control plane from a data plane. The control plane handles uploads, transcoding, metadata, and API keys. The data plane uses the indexd SDK on the client application for content retrieval, with an optional local caching layer for performance. The following sections describe each layer of the pipeline from ingestion through delivery.

Ingestion. Clients upload video files through the TUS v1.0.0 resumable upload protocol. TUS guarantees resumability via its offset-tracking mechanism: if a multi-gigabyte upload is interrupted, the client queries the server for the current byte offset and resumes from that point. The embedded TUS server validates the API key, creates an asset record in PostgreSQL, and writes chunks to local staging disk. On completion, it enqueues a transcoding job.

Processing. A BullMQ worker, a Redis-backed job queue for Node.js, executes FFmpeg to produce HTTP Live Streaming output. HLS is the dominant adaptive streaming protocol, and Sluby packages it in fMP4 containers, a segment-friendly variant of the MP4 format. The pipeline defaults to four renditions at 1080p, 720p, 540p, and 360p, customizable up to 4K. FFmpeg is configured with forced keyframe alignment, disabled scene-change detection, and the independent_segments flag to ensure clean quality switching between renditions.

A key optimization is the use of single_file output mode. Instead of producing hundreds of individual small segment files per rendition, FFmpeg writes each rendition as a single continuous .m4s data file. The HLS playlist then references specific byte ranges within that file rather than separate segment files. This reduces the per-video Sia upload count from hundreds of operations to just N + 2 for an N-rendition ladder: one upload per rendition data file, one packed batch for playlists and thumbnails, and one master manifest upload. Thumbnails are extracted at 25%, 50%, and 75% of duration.

Storage. After transcoding, each output artifact is stored on the Sia network using the official Sia Foundation TypeScript SDKs. In Sia’s architecture, upload and pin are separate operations. Uploading distributes encrypted, erasure-coded shards across storage providers. Pinning then registers the object with indexd so the indexer can monitor its health and trigger automatic repair if redundancy drops. Each stored object is identified by a 32-byte Object ID, a content-addressed hash that uniquely identifies the object on the network.

Large rendition data files are uploaded individually via uploadAndPin, as they naturally fill their own slabs. Sia stores data in 4 MiB sectors grouped into slabs, where total slab size depends on the configured number of data shards. Small artifacts like playlists, thumbnails, and the master manifest are grouped into a single uploadAndPinPacked call, sharing slabs to avoid wasting a full slab per small file. Every returned Object ID is persisted to PostgreSQL, mapping it to the parent asset, rendition, and role.

Sluby uses two official Sia Foundation TypeScript SDKs. The sia-storage SDK handles control-plane operations such as connecting, pinning, deleting, and sharing. The @siafoundation/sia SDK handles data-plane operations such as uploading, downloading, and packed uploads. Object references are translated between the two SDKs using their built-in seal/open serialization, a round-trip encoding that converts one SDK’s object handle into a portable byte string and back.

Manifest rewriting. HLS players locate video data through playlist files called manifests, which list the available renditions and their segment locations. After artifacts are uploaded, the worker rewrites each rendition playlist to replace local file references with gateway URLs mapped to Sia Object IDs, preserving all byte-range offsets. The master playlist is rewritten to reference each rendition’s Sia object. Rewritten manifests are uploaded to Sia and the master Object ID becomes the asset’s playback entry point.

Delivery. The client application retrieves video content directly from Sia using the indexd SDK. When the React player requests a byte range from a rendition file, the SDK’s download(object, {offset, length}) call fetches only the overlapping sectors from the relevant slabs and returns decrypted content directly to the player. No intermediary handles or inspects the playback data, preserving viewer privacy by design.

For performance, Sluby includes an optional local Nginx reverse proxy cache with aggressive time-to-live values for immutable segment data. Cache warming after transcoding can prefetch artifacts, ensuring low-latency playback for repeated requests. This caching layer runs entirely within the developer’s own deployment and is not required for the delivery path to function.

Why this works on Sia. Sia’s storage architecture is uniquely suited to video delivery. Every slab is content-addressed, and each host serves its sectors over a range-request protocol. When the SDK receives a download(object, {offset, length}) call, it pulls only the sectors that overlap the requested byte range. No full-object reconstruction is required.

Sluby turns this into a first-class delivery primitive. When the client application requests a byte range from a data.m4s file, the indexd SDK passes that range directly into a byte-range download, and Sia returns exactly those bytes from the relevant slabs. This is the same streaming contract that Mux, Cloudflare Stream, and every S3-backed video platform depend on. A viewer watching one minute of a two-hour video pulls only that minute’s bytes from Sia, not the whole object. First-frame latency is a single slab fetch rather than a full-file download.

Paired with single_file HLS output, per-video Sia operations drop from hundreds to N + 2, and viewer-facing bandwidth cost scales with what is actually watched. This is the core reason Sia is a viable video storage primitive and not just archival storage, and it is what lets Sluby match the developer experience of centralized platforms without giving up decentralization, encryption, or content addressing.

State management. The Sia indexer does not provide directory hierarchies or metadata search because all object metadata is encrypted and opaque to the indexer. Sluby therefore maintains a PostgreSQL database as the authoritative mapping between assets, renditions, artifacts, and Sia Object IDs. A background reconciliation worker periodically verifies database state against the indexer by paginating through tracked objects using cursor-based listing, flagging orphaned objects for cleanup.

Sluby MVP

Technology Stack

  • Backend: Node.js with Express, TypeScript
  • Database: PostgreSQL via Drizzle ORM
  • Job Queue: BullMQ backed by Redis with three queues for transcoding, upload, and finalization
  • Upload Protocol: @tus/server embedded in the Express application
  • Transcoding: FFmpeg invoked as a subprocess, producing fMP4 HLS with single_file output mode
  • Sia Integration: Official Sia Foundation TypeScript SDKs: sia-storage for the control plane and @siafoundation/sia for the data plane, bridged via seal/open serialization
  • Caching: In-memory LRU for manifests and thumbnails; Nginx reverse proxy cache for segment delivery
  • Frontend (Studio): Vite + React + Tailwind CSS + shadcn/ui, TanStack Query
  • React Player: hls.js wrapper with Safari native fallback, published to npm
  • Client SDK: TypeScript client wrapping the REST API with tus-js-client, published to npm
  • Container: Docker with Docker Compose

Security Best Practices

  • Client-side encryption. The Sia SDK encrypts all data client-side with per-slab stream ciphers before transmission. Storage providers and the indexer never see plaintext content.
  • API key security. Keys are SHA-256 hashed before persistence; the raw key is shown only once at creation. Granular scopes of upload, read, and manage are enforced per key.
  • Sia key isolation. The Ed25519 application key, Sia’s digital signature key, is loaded only in the backend process from environment configuration and never exposed to clients.
  • Input validation. Upload size limits at the TUS layer, environment schema validation via Zod, and API input sanitization.
  • License-safe FFmpeg. Invoked as a subprocess rather than linked as a library, preserving Apache 2.0 compatibility.

Timeline with measurable objectives and goals.

The project spans 5 months with 5 milestones. Each produces independently verifiable deliverables.

Milestone #1: Core Backend and Sia Storage Pipeline - due June 25

Objective: Deliver the backend service, video processing pipeline, and Sia storage integration, validating the complete upload-to-store path.

Deliverables:

  • Backend service with embedded TUS server handling resumable uploads.
  • FFmpeg transcoding pipeline producing fMP4 HLS output in single_file mode with default renditions at 1080p/5,000 kbps, 720p/2,800 kbps, 540p/1,400 kbps, and 360p/800 kbps, customizable up to 4K.
  • Sia storage integration via both official TypeScript SDKs: uploadAndPin for large rendition files, uploadAndPinPacked for playlists and thumbnails, with object handle bridging via seal/open.
  • HLS manifest rewriting engine replacing local paths with gateway URLs mapped to Sia Object IDs, preserving byte-range offsets.
  • PostgreSQL schema via Drizzle ORM for assets, renditions, artifacts, upload sessions, and processing jobs. BullMQ job queue with Redis for pipeline orchestration.
  • Thumbnail extraction and Docker Compose development stack with backend, PostgreSQL, and Redis.

Verification: Upload a video via TUS, observe transcoding to four renditions, verify all artifacts are uploaded and pinned on the Sia Zen testnet, and confirm PostgreSQL correctly maps each Sia Object ID to its parent asset and rendition.


Milestone #2: Delivery Gateway and REST API - due July 25

Objective: Deliver the HTTP gateway for Sia-backed playback and the complete developer-facing REST API.

Deliverables:

  • Client-side delivery integration using the indexd SDK for direct byte-range content retrieval from Sia. Optional local Nginx caching layer for performance optimization. Cross-Origin Resource Sharing configuration for embedded playback across domains.
  • In-memory LRU cache for manifests and thumbnails, optional Nginx reverse proxy cache for segments with X-Cache-Status headers, and cache warming on transcoding completion.
  • REST API designed spec-first with OpenAPI 3.1, a machine-readable API specification standard. Endpoints cover asset CRUD with cursor-based pagination and filters, upload session creation, processing status, playback URL generation, playback ID management, and per-asset Sia storage breakdown.
  • Scoped API key system with SHA-256 hashing and scope enforcement for upload, read, and manage permissions.
  • Background reconciliation worker comparing PostgreSQL state against indexer inventory via cursor-based sdk.objects().

Verification: Stream a video end-to-end via HLS, verify adaptive quality switching and byte-range requests, confirm cache hits via Nginx headers. Exercise the full API lifecycle: create an API key, upload a video, poll for completion, and stream playback.


Milestone #3: TypeScript SDK and React Player - due August 25

Objective: Deliver a typed client SDK and a production-ready React player component.

Deliverables:

  • TypeScript SDK (@sluby/sdk on npm) wrapping the REST API with typed methods for asset management, upload creation, and playback URL retrieval. Integrates tus-js-client for browser-side resumable uploads with progress events, pause/resume, and retry.
  • React player component (@sluby/player on npm) wrapping hls.js with integrated indexd SDK for direct Sia content retrieval, adaptive quality selection, loading states, error recovery with automatic retry, and Safari native HLS fallback.
  • Built with tsup for ESM and CJS module output, with a React 18+ peer dependency.
  • Asset deletion pipeline: soft-delete in PostgreSQL, asynchronous Sia unpin and prune, garbage collection for partial deletes.

Verification: Build a minimal web application using the TypeScript SDK and React player. Upload a video with progress tracking, poll for readiness, embed the player, and verify adaptive streaming and error recovery across Chrome, Firefox, and Safari. Delete an asset and confirm Sia objects are unpinned.


Milestone #4: Studio Web Application - due September 25

Objective: Deliver the management Studio for visual control over the video lifecycle.

Deliverables:

  • Studio dashboard built with Vite, React, and shadcn/ui: asset totals, storage usage, latest uploads, cache statistics, and indexer status.
  • Upload page with TUS progress visualization, drag-and-drop, and processing status indicator.
  • Asset library with search, filtering, and sorting.
  • Asset detail view: embedded player, rendition info, Sia storage breakdown showing Object IDs, slab and sector counts, host distribution, and redundancy ratio.
  • Developer page with ready-to-copy integration snippets in JavaScript and curl.
  • Settings: API key creation and rotation, theme toggle.

Verification: Upload a video through Studio, observe processing status, browse the asset library, inspect per-asset Sia storage details, preview playback with quality switching, and manage API keys.


Milestone #5 (Final): Documentation, Testing, and Release - due October 25

Objective: Deliver documentation, test suite, production deployment configuration, and public release.

Deliverables:

  • Quickstart guide achieving first video playback within 5 minutes from Docker Compose.
  • API reference auto-generated from the OpenAPI 3.1 spec with interactive endpoint explorer.
  • Integration guides for React and plain JavaScript.
  • Architecture documentation with data flow diagrams and SDK integration patterns.
  • Test suites covering the backend (route handlers, FFmpeg arguments, manifest rewriting, API key hashing), the SDK (client construction, error mapping, TUS integration), and the React player (hls.js wiring).
  • Docker Compose configurations for development and production deployment with named volumes, restart policies, and TLS termination.
  • Monorepo under Apache 2.0 with contributing guide.

Verification: Clone the repository, run docker compose up, follow the quickstart to upload and stream a video within 5 minutes. All test suites pass. Production guide covers TLS and Sia indexer connection for testnet and mainnet.

End-of-Grant Deliverables

At grant conclusion, the following components will be delivered under Apache 2.0:

  • Backend service handling TUS resumable uploads, FFmpeg transcoding to four-rendition HLS, per-artifact Sia storage via the official SDKs, and manifest rewriting.
  • Client-side delivery via indexd SDK with direct byte-range retrieval from Sia, and optional local Nginx caching layer.
  • REST API with OpenAPI 3.1 specification, scoped API keys, and asset lifecycle management.
  • TypeScript SDK (@sluby/sdk on npm) for asset management and browser uploads.
  • React player component (@sluby/player on npm) with adaptive quality switching, error recovery, and Safari fallback.
  • Studio web application for visual asset management, upload monitoring, Sia storage inspection, and API key management.
  • Documentation including quickstart guide, API reference, and integration guides.
  • Test suite covering backend, SDK, and player.
  • Docker Compose configurations for development and production deployment.

Who is the target user for your project?

Sluby targets application developers and engineering teams who need video capabilities without building custom video infrastructure. No knowledge of Sia’s storage internals, erasure coding, or indexer workflows is required to integrate or use the platform. This includes teams building creator platforms, educational technology, media archives, enterprise internal tooling, and consumer applications requiring video upload, processing, and playback.

The secondary audience is organizations with data sovereignty, censorship resistance, or content integrity requirements. These include human rights organizations, journalistic outlets, and enterprises under strict privacy regulations that need video infrastructure where no third party can access or remove their content.


What are your plans for this project following the grant?

Sluby is not a grant-scoped project. All core components produced during the grant (backend, API, SDK, React player, and Studio) are open-source under Apache 2.0 and will remain so. Post-grant development will continue with features beyond this scope, including a Python SDK, webhook event delivery, and access control via signed playback tokens.

Community engagement will be sustained through GitHub Discussions for technical issues, a Telegram channel for developer interaction, an X account for updates and ecosystem visibility, and monthly progress reports on the Sia Forum.

I am committed to this project because the opportunity is clear: video infrastructure on Sia has validated demand, no existing solution, and a clear path to continued development beyond the grant.


Potential risks that will affect the outcome of the project:

Sia SDK stability. The TypeScript SDKs are pre-1.0 and may introduce breaking changes. Mitigation: all SDK interactions are isolated behind a bridging abstraction layer. Versions are pinned in package.json, and upgrades require passing a full upload, pin, download, and unpin integration test suite. The seal/open bridging between the two SDKs provides an additional buffer against changes in either SDK independently.

Cold retrieval latency. Uncached Sia downloads may take 1–2 seconds to first byte, since the SDK must contact multiple storage hosts and reconstruct erasure-coded slabs before serving data. Mitigation: the optional local cache combining Nginx reverse proxy and in-memory LRU absorbs repeated requests. The single_file HLS mode reduces per-playback Sia operations. Cache warming after transcoding ensures the first viewer hits cache. A CDN can sit in front of Nginx for production deployments.

Concurrent stream memory. Each cold Sia download requires RAM for slab reconstruction, the process of reassembling erasure-coded data from multiple hosts. Mitigation: the optional Nginx cache absorbs repeated requests, reducing concurrent cold downloads. The client-side SDK handles retrieval independently per viewer.


Development Information

Will all of your project’s code be open-source?

Yes. All code will be released under the Apache 2.0 license. All third-party dependencies are MIT or Apache 2.0 licensed. FFmpeg (LGPL 2.1+) is invoked as a subprocess only, preserving Apache 2.0 compatibility.

Leave a link where code will be accessible for review.

https://github.com/justmert/sluby (currently private; will be made public upon grant approval — reviewers can contact me for access)

Do you agree to submit monthly progress reports?

Yes. Monthly reports will be posted to the Sia Forum aligned with each milestone, detailing completed work, technical decisions, deviations, and next steps.

Provide links to previous work or code from all team members.

Mert Köklü

Mert is an experienced software engineer with over four years of blockchain development experience, specializing in tooling, integrations, and backend infrastructure. He holds Bachelor’s degree in Computer Science and has led engineering teams focused on both AI and blockchain technologies.

He previously worked with ApeWorX, where he developed StarkNet plugins and compiler tools for the Cairo language. Before that, he served as AI Video Analytics Team Lead at an NVIDIA partner company, managing large-scale intelligent vision projects.

He is also a long-term Sia ecosystem contributor, having developed multiple grant-funded projects for the network.

Recent Grantee Projects:

  • SoroReveal: Decompile Soroban smart contracts back to readable source code (web)
  • Alerts dYdX: Monitor positions, alerts, and account status on dYdX in real time (web)
  • Kurtosis-Orbit: Tool for spinning up complete Arbitrum Orbit rollup environments (repo)
  • Arbitrum Python SDK: Python library for interacting with Arbitrum (repo)
  • ICP Agent Kit: Enables natural language interaction with the Internet Computer (repo)
  • Cosmic: Local Starknet block explorer (repo)

Have you developed a proof of concept for this idea already?

Yes. A working reference implementation is live at https://siastream.com (migrating to sluby.app), with a video walkthrough of the platform at https://youtu.be/-NnKGXTSVjw. The MVP demonstrates the complete pipeline end-to-end against the Sia network: TUS uploads, four-rendition fMP4 HLS transcoding with single_file mode, per-artifact Sia storage via both official TypeScript SDKs, byte-range delivery through a caching gateway, a REST API, a TypeScript SDK, a React player component, and a management Studio. On the Zen testnet, a single test video produces approximately 10–12 pinned Sia objects across tens of slabs, all independently verifiable on Siascan, with first-frame playback achieved within seconds via the range-passthrough gateway.


Contact info

Email: [email protected]

Any other preferred contact methods:

Hi @mert - thank you for this revised proposal. We’ve reached capacity for next week’s Grants Committee meeting, so this proposal will be slotted for review during the next meeting on May 26th.

Thank you for your patience.

Thanks for your proposal to The Sia Foundation Grants Program.

After review, the Committee has decided to approve your proposal. Congratulations! They’re excited to see what you can accomplish with this grant.

However, the Committee requests the following change to the proposal:

  • Use the indexd SDK on the client app instead of outlining tasks for work to be done on the server in order to better preserve privacy.

We’ll reach out to your provided email address for onboarding. This shouldn’t take long unless your info has changed from last time, but you may still need to adjust your timelines.

Please indicate over email when the proposal has been edited to reflect the requested change(s).

Thanks for giving this opportunity.

Just to clarify: Sluby is an entirely self-hosted, open-source platform. There is no centralized server, the developer runs the full stack on their own infrastructure via Docker Compose, connected to their own indexd instance with their own application key. No third party ever touches the content. I think the proposal language around “server,” “gateway,” and “backend” may have given the impression of a hosted intermediary, which isn’t the case in this proposal.

That said, I have revised the proposal to make the self-hosted nature explicit throughout and reframe the architecture to emphasize client-side SDK usage for content retrieval. Thank you @mecsbecs

Hi @mert,

This is a reminder that your June progress report is now late.

Progress reports are expected to be posted by the 25th of the month in order to allow adequate time for the monthly technical review. Reports posted later than the 2nd run the risk of nonpayment on the 15th of the following month.

Please be sure to follow the Progress Report Template and submit your progress report so we may continue funding your grant.

Sluby, Progress Report 1 (June, Milestone 1)

Repo: GitHub - justmert/Sluby · GitHub

Milestone 1, Core Backend and Sia Storage Pipeline (due June 25): done.

What progress was made this month?

  • Milestone 1 is finished: I upload a video over TUS, FFmpeg transcodes it into four HLS renditions, every output is stored and pinned on Sia, and the object IDs are tracked in Postgres. Ran the full path on the Zen testnet, where one test video gives about 10 to 12 pinned objects, all visible on Siascan.

  • HLS uses FFmpeg single_file mode, so each rendition is one .m4s the playlist reads by byte range. That drops per-video Sia uploads from a few hundred to about N+2, and playback pulls only the bytes it needs.

  • Every Sia object ID maps back to its asset, rendition, and role (renditions and artifacts tables), which the storage breakdown and later cleanup build on.

  • The whole stack runs from docker compose up (backend, Postgres, Redis, Nginx).

Tasks this month

Solo repo, so work lands as commits on main.

Milestone Task Commit(s)
1 Backend with embedded TUS resumable uploads f83414a
1 FFmpeg fMP4 HLS, single_file, four renditions (1080p/720p/540p/360p) f83414a, 72fe226
1 Sia storage via the sia-storage SDK, packed uploads for small files f83414a, eeafc59
1 HLS manifest rewriting to Sia object IDs, byte ranges preserved 72fe226
1 Postgres schema (assets, renditions, artifacts, sessions, jobs) and BullMQ f83414a, 4397f35
1 Thumbnails and the Docker Compose dev stack f83414a

Problems and how I’m handling them

  • I kept a single SDK (sia-storage) instead of the two-SDK plus seal/open plan from the proposal. It handled both upload and download cleanly, so I dropped the extra bridge. Flagging it as a deviation; all Sia calls sit behind one module if I need the second SDK later.

  • The Zen testnet’s small host pool means too much upload parallelism hits “no more hosts available.” I capped concurrency and packed the small files, which should relax on mainnet.

  • Cold Sia reads take a second or two to first byte. The byte-range gateway and a local cache cover repeats for now; client-side retrieval and cache warming come in later milestones.

What’s next

Milestone 2, delivery gateway and REST API: write the OpenAPI 3.1 spec and build the endpoints against it, add the worker that reconciles the database against the indexer, and finish caching (cache-status headers and warming after a transcode).


And sorry for the late progress report @mecsbecs, thank you for your efforts.

1 Like

Hi @mert - your technical review is complete and your first milestone is approved.

Your reviewer wanted me to pass on the following notes for future:

  • Put changes in a PR with a good description first, then merge it
  • Shouldn’t CACHE_MAX_SIZE in blob-manager.ts default to 10240 like how it is in env.ts and .env.example?
  • Reduce duplication of blocks like this:

```

return {

id: asset.id,

title: asset.title,

description: asset.description,

manifestObjectId: asset.manifestObjectId,

thumbnailObjectIds: asset.thumbnailObjectIds,

durationMs: asset.durationMs ?? 0,

resolution: asset.resolution ?? ‘’,

status: asset.status,

accessTier: asset.accessTier,

creatorAddress: asset.creatorAddress,

segmentCount: asset.segmentCount,

totalStorageBytes: asset.totalStorageBytes,

createdAt: asset.createdAt,

updatedAt: asset.updatedAt,

};

```

We look forward to seeing what you accomplish over the next moth!

Thanks so much, really glad to hear M1 is approved, and please pass my thanks along to the reviewer for the thorough read.

Quick notes on the feedback:

  • CACHE_MAX_SIZE: good catch, that was a real inconsistency. It now reads from the same validated config as env.ts and .env.example, so there is a single source of truth (10240) instead of two defaults that could drift apart.
  • The repeated asset-mapping block is now pulled into one helper, so that field list lives in a single place.

Both of those are already in the Milestone 2 work. I have also noted the PR-first workflow and will use it for the larger changes going forward. Thanks again.

1 Like

Sluby, Progress Report 2 (July, Milestone 2)

Repo: GitHub - justmert/Sluby · GitHub

Milestone 2, Delivery Gateway and REST API (due July 25): done.

What progress was made on your grant this month?

  • Milestone 2 is finished. Playback now comes straight from Sia: the backend serves any stored object over a byte-range gateway, so a viewer only pulls the bytes they actually watch, and there is a full REST API in front of it.

  • The API is spec-first, described by an OpenAPI 3.1 document that the backend serves and renders. It covers asset listing with cursor pagination, upload sessions, processing status, playback URL generation, playback ID management, and a per-asset Sia storage breakdown. API keys are scoped into upload, read, and manage, and stored as SHA-256 hashes.

  • Manifests and thumbnails are cached in memory with X-Cache-Status headers and warmed right after a transcode, with an optional nginx cache in front of the segment path.

  • I added a background worker that reconciles the database against the indexer’s pinned inventory and flags anything that has drifted, so orphaned or missing objects surface instead of going unnoticed.

Tasks this month

Milestone Task Commit(s) Additional Notes
2 Delivery gateway: byte-range playback from Sia, cache-status headers, signed URLs for private assets 77337b3, 366c357
2 Caching: in-memory LRU for manifests/thumbnails, X-Cache-Status headers, warm on transcode 77337b3, 1442c45
2 REST API, spec-first with OpenAPI 3.1 (asset CRUD + cursor pagination, upload sessions, status, playback URLs, playback IDs, storage breakdown) cee3f3d, 7033ba9, 8c3b913
2 Scoped API keys and per-owner access control (upload/read/manage, SHA-256) c8d17ed
2 Background reconciliation worker (database against indexer inventory) 9ab454a, 8aba334 Built on the SDK’s objectEvents() feed

Summarize any problems that you ran into this month and how you’ll be solving them.

  • My proposal described the reconciliation worker using sdk.objects(), but that call does not exist in the current sia-storage SDK. The real cursor feed is objectEvents(), so I built the worker on that; it does the same job of replaying the indexer’s inventory.

What will you be working on next?

Milestone 3: the TypeScript SDK (@sluby/sdk) and the React player. The player wraps hls.js with direct byte-range retrieval from Sia, adaptive quality, error recovery, and a Safari native-HLS fallback, and the SDK wraps the REST API with resumable uploads. I will also build the asset deletion pipeline: soft-delete in the database, then an asynchronous unpin and prune on Sia.


Thank you @mecsbecs, and please pass my thanks along to the reviewers as well.

1 Like