Small Grant: MirrorLine — APT Package Mirror Toolkit Using the Official Sia SDK + indexd
Project Name
MirrorLine
Applicant
Dapps over Apps
Grant Type
Small Grant
Amount Requested
$8,900 USD
Estimated Timeline
8 weeks
Summary
MirrorLine is an open-source, self-hostable APT package mirror toolkit built on the official Sia SDK and indexd.
It allows a maintainer to take a small existing APT/deb repository directory, upload and pin the repository files through Sia, and serve those files through a standard HTTP mirror gateway that unmodified APT clients can use.
The goal is intentionally narrow:
Demonstrate that an APT/deb repository can be mirrored through a normal HTTP repository URL while the underlying repository files are uploaded, pinned, retrieved, cached, and verified through the Sia SDK + indexd stack.
MirrorLine is not a package registry, CDN, S3 gateway, renterd tool, filesystem, static website host, sync engine, or dashboard. It is a focused developer infrastructure tool for one package-mirror workflow.
Problem
Package mirrors are important developer infrastructure. They are used by CI pipelines, server provisioning, reproducible build environments, developer machines, and production systems to retrieve package metadata and artifacts over standard HTTP.
Today, most package mirrors depend on centralized object storage, cloud CDNs, university mirrors, company-operated servers, or traditional hosting providers.
Sia has a strong opportunity to support software artifact distribution, but the ecosystem still needs focused application-layer examples that show how existing developer workflows can use Sia-backed storage without requiring users to run Sia-specific tools or change their package clients.
The problem is not that developers need a new package manager.
The problem is that Sia needs practical developer tooling that connects current SDK/indexd storage flows to existing software distribution workflows.
MirrorLine solves this for one narrow and testable target: APT/deb package repositories.
What MirrorLine Will Build
MirrorLine will build two core components.
1. mirrorline ingest
A CLI that scans an existing APT/deb repository directory, computes file hashes, uploads changed files through the official Sia SDK, pins uploaded objects through indexd, and writes a local manifest mapping repository paths to Sia object references.
Example:
mirrorline ingest ./testdata/aptrepo \
--manifest ./mirrorline.db \
--indexer "$MIRRORLINE_INDEXER_URL"
The manifest will store:
repository path
Sia object reference
file size
SHA-256 hash
content type
upload timestamp
pin status
The ingest command will skip unchanged files by hash on repeat runs so that maintainers do not re-upload identical files unnecessarily.
2. mirrorline serve
A lightweight HTTP gateway that serves the same path structure expected by APT.
Example:
mirrorline serve \
--manifest ./mirrorline.db \
--cache-dir ./cache \
--listen 127.0.0.1:8080
When an APT client requests paths such as:
/dists/stable/InRelease
/dists/stable/main/binary-amd64/Packages
/pool/main/h/hello-mirrorline/hello-mirrorline_0.1.0_amd64.deb
MirrorLine will:
- Resolve the request path through the local manifest.
- Retrieve the matching object through the Sia SDK on cache miss.
- Verify the downloaded bytes against the stored SHA-256 hash.
- Store verified files in a bounded local disk cache.
- Serve the response through normal HTTP.
- Reuse the disk cache for repeated requests.
APT clients continue to use a normal HTTP mirror URL. The Sia-specific storage logic stays behind the gateway.
How This Uses Sia
MirrorLine uses Sia as the storage and retrieval layer for repository files.
The project will use the Sia SDK and indexd flow for:
uploading repository files
pinning uploaded objects
retrieving objects
tracking object references needed by the application manifest
MirrorLine does not modify indexd and does not require changes to the Sia SDK. It adds the application-layer behavior needed for an APT mirror:
APT path mapping
local manifest management
HTTP gateway behavior
cache behavior
hash verification
package-client conformance testing
The boundary is clear:
Sia SDK + indexd:
object upload, object retrieval, object pinning, object tracking
MirrorLine:
APT repository path mapping, local manifest, HTTP serving, cache, hash verification, conformance tests
Why This Is Useful to the Sia Ecosystem
MirrorLine gives the Sia ecosystem a concrete reference implementation for software package mirror infrastructure.
The value is not simply “hosting files on Sia.” The value is showing that Sia can back an existing developer workflow while users keep using normal package tooling.
This creates a practical foundation for future software artifact distribution use cases. Future package formats such as RPM, APK, Homebrew, npm, PyPI, Cargo, Maven, and Docker registries are not part of this grant.
Who Benefits
Package Maintainers
Maintainers get a small reference tool for mirroring APT repository files to Sia-backed storage while keeping a normal HTTP mirror interface.
Developers and Infrastructure Teams
Developers can test a Sia-backed package mirror without installing a new package manager or learning Sia internals.
Sia Builders
Sia builders get a focused example of how to combine SDK upload/download, indexd pinning, application-layer manifests, HTTP serving, cache behavior, and third-party client compatibility.
The Sia Ecosystem
Sia gets a developer infrastructure use case that is more concrete than a demo app and more focused than generic file hosting, sync, static hosting, or media delivery.
Scope Included in This Grant
This grant includes:
- A CLI for ingesting an existing APT/deb repository directory.
- Uploading repository files through the official Sia SDK.
- Pinning uploaded objects through indexd.
- A local SQLite manifest mapping repository paths to Sia object references.
- Re-upload skipping for unchanged files by hash.
- A lightweight HTTP gateway for APT-compatible paths.
- Bounded disk cache with configurable maximum size.
- Hash verification before serving downloaded or cached files.
- A small fixture APT repository in
testdata/. - Docker Compose setup for local review.
- End-to-end conformance test using an unmodified Debian/Ubuntu APT client.
- CI tests for manifest parsing, path safety, hash verification, cache behavior, and gateway responses.
- README, architecture document, reviewer walkthrough, and tagged
v0.1.0release.
Scope Excluded From This Grant
To keep the grant focused and deliverable, MirrorLine will not include:
No RPM support
No APK support
No Homebrew support
No npm support
No PyPI support
No Cargo support
No Maven support
No Docker registry support
No full Debian or Ubuntu archive mirror
No automatic upstream mirror crawler
No package signing service
No package registry UI
No hosted SaaS
No CDN
No analytics dashboard
No S3 compatibility
No renterd dependency
No filesystem/NFS/FUSE layer
No browser local storage
No user account system
No payment system
These exclusions are intentional. The goal is to deliver a working APT/deb package mirror toolkit within 8 weeks, not to overpromise a full software distribution platform.
Technical Architecture
Existing APT/deb repository directory
↓
mirrorline ingest
↓
hash files and detect changes
↓
Sia SDK upload
↓
indexd pinning
↓
local SQLite manifest
↓
mirrorline serve
↓
HTTP request path resolver
↓
SDK download on cache miss
↓
hash verification
↓
bounded disk cache
↓
normal HTTP response
↓
unmodified APT client
Manifest
MirrorLine will use SQLite for the local manifest because it is simple, inspectable, portable, and sufficient for the MVP.
Example manifest fields:
id
repo_path
sia_object_ref
sha256
size_bytes
content_type
uploaded_at
pin_status
last_verified_at
Cache
The gateway will use a bounded local disk cache. Operators can configure the maximum cache size. Cache hits will serve verified files directly from disk. Cache misses will retrieve from Sia through the SDK, verify the bytes, then populate the cache.
Safety
MirrorLine will not log recovery phrases, app keys, or secrets. The README will document secret handling and .env usage clearly.
The gateway will include path traversal protection so HTTP request paths cannot escape the manifest or local cache directory.
Milestones
Milestone 1 — Ingest CLI, SDK Upload, indexd Pinning, and Manifest
Timeline: Weeks 1–4
Payment: $4,300
Deliverables
- Public GitHub repository.
- Open-source license.
- Project scaffold.
mirrorline ingestCLI.- Scans an existing APT/deb repository directory.
- Computes SHA-256 hash for each file.
- Uploads new or changed files through the official Sia SDK.
- Pins uploaded objects through indexd.
- Stores path-to-object mappings in SQLite.
- Skips unchanged files on repeat ingest.
- Includes a small fixture APT repository in
testdata/aptrepo. - Includes
.env.example. - Includes CI for formatting, linting, and unit tests.
- Includes
ARCHITECTURE.md.
Acceptance Criteria
A reviewer can run:
cp .env.example .env
# add reviewer indexer/app-key values
docker compose run --rm mirrorline ingest ./testdata/aptrepo
Expected result:
All fixture repository files are uploaded.
All uploaded objects are pinned.
mirrorline.db contains one row per repository file.
Re-running ingest skips unchanged files by hash.
No recovery phrase, app key, or secret is printed in logs.
Milestone 2 — HTTP Gateway, Cache, APT Conformance Test, and Release
Timeline: Weeks 5–8
Payment: $4,600
Deliverables
mirrorline serveHTTP gateway.- Path-safe resolver from HTTP request path to manifest row.
- SDK download path for cache misses.
- Bounded disk cache with configurable maximum size.
- SHA-256 verification before serving cached or downloaded files.
- HTTP response headers suitable for APT clients.
- Basic resume/range behavior only where cleanly supported by the SDK and needed for the fixture test.
- Docker Compose quickstart.
- End-to-end APT conformance test.
docs/reviewer-walkthrough.md.- Tagged
v0.1.0release.
Acceptance Criteria
A reviewer can run:
docker compose up
Then, in a clean Debian/Ubuntu container, the reviewer can add the local MirrorLine gateway as an APT source and run:
apt update
apt install hello-mirrorline
Expected result:
APT fetches repository metadata through MirrorLine.
APT fetches the test .deb package through MirrorLine.
MirrorLine serves bytes retrieved from Sia-backed objects.
The installed package hash matches the original fixture package.
A second apt update/install flow uses the local disk cache.
No browser local storage is used.
No renterd dependency is required.
No recovery phrase, app key, or secret appears in logs.
Budget
Total requested: $8,900
| Category | Amount |
|---|---|
| Ingest CLI, hashing, SDK upload, indexd pinning, manifest | $2,700 |
| HTTP gateway, path resolver, cache, hash verification | $2,500 |
| Fixture APT repo and end-to-end conformance test | $1,400 |
| Docker Compose, CI, release packaging | $900 |
| Documentation, reviewer walkthrough, basic security hardening | $900 |
| Final review fixes and tagged release | $500 |
| Total | $8,900 |
Risks and Mitigations
Risk 1: Cold retrieval latency
APT clients expect reliable HTTP responses. Cold retrieval from distributed storage can be slower than local disk or CDN-backed object storage.
Mitigation: MirrorLine includes a bounded disk cache, hash verification, and repeat-request cache hits as core deliverables. The MVP is not positioned as a CDN or production-scale distro mirror. The goal is package-mirror conformance and a working Sia-backed reference path.
Risk 2: Indexer setup confusion
Reviewers and operators need clear setup instructions.
Mitigation: MirrorLine will document the expected indexer/app-key configuration in .env.example and the reviewer walkthrough. The grant does not include building or operating a new indexer.
Risk 3: Being mistaken for static hosting
MirrorLine serves files over HTTP, but it is not a static website host.
Mitigation: MirrorLine has no accounts, no hosted deployment platform, no subdomains, no website routing, no UI, no analytics, and no CDN layer. It serves only APT repository paths from a local manifest.
Risk 4: Being mistaken for filesystem work
MirrorLine is not a filesystem gateway.
Mitigation: It does not expose POSIX behavior, NFS, FUSE, directories as a general storage abstraction, or arbitrary file browsing. It maps APT repository paths to pinned Sia objects for one package-mirror workflow.
Risk 5: Over-scoping
Software distribution is a broad category.
Mitigation: The grant is limited to a small APT/deb fixture repository, ingest CLI, HTTP gateway, cache, and conformance test. RPM, APK, Homebrew, npm, PyPI, Cargo, Maven, Docker registries, and full OS mirrors are excluded from this grant.
Success Criteria
The project is successful when an external reviewer can:
- Clone the repository.
- Configure the required indexer/app-key connection.
- Run
mirrorline ingestagainst the fixture APT repository. - Confirm repository files are uploaded and pinned through Sia.
- Start the MirrorLine HTTP gateway.
- Add the gateway as an APT source in a clean Debian/Ubuntu container.
- Run
apt update. - Install the fixture package.
- Confirm the downloaded package hash matches the original file.
- Repeat the install/update flow and observe cache usage.
- Review the code, docs, and tests without needing private infrastructure beyond the documented Sia/indexer credentials.
Maintenance Plan
After the grant is completed, we will provide a short post-release maintenance period for bug fixes related to the delivered MVP.
This includes:
fixing reproducible reviewer issues
clarifying setup documentation
patching issues in the path resolver or cache layer
responding to GitHub issues related to the delivered APT fixture flow
Future package formats such as RPM, APK, Homebrew, npm, PyPI, Cargo, Maven, and Docker registries are outside the grant scope and would only be considered after the APT/deb conformance MVP is complete and reviewed.
Team
Team
MirrorLine will be delivered by Dapps Over Apps, a developer tooling team with prior experience building Sia-focused infrastructure.
The team previously built HostaProbe, a Sia preflight QA tool for validating SDK upload/download flows, indexd pinning, API key setup, byte verification, and CI-ready diagnostic reporting.
This prior Sia-specific work gives the team practical familiarity with the current SDK/indexd stack and reduces delivery risk for MirrorLine.
For this grant, the team will focus only on the scoped deliverables: APT/deb repository ingestion, Sia SDK integration, indexd pinning, local manifest management, HTTP gateway behavior, cache handling, conformance testing, and documentation.
Technical Delivery
Implementation will be handled by a small developer tooling team with backend, CLI, testing, and infrastructure experience. The team will keep the delivery intentionally narrow to match the requested budget and 8-week timeline.
The project is scoped to avoid research-heavy or protocol-level work. The main delivery work is application-layer engineering: CLI ingestion, SDK integration, manifest management, HTTP gateway behavior, cache handling, tests, and documentation.
Why MirrorLine Should Be Funded
MirrorLine is a strong fit for the Sia Grants Program because it is narrow, practical, developer-facing, and built around the current SDK/indexd direction.
It does not ask the Sia core team to build new protocol functionality. It does not duplicate renterd, indexd, SDKs, static hosting, NFS, sync tooling, or media infrastructure.
It uses the Sia stack as the storage layer and adds one missing application-layer workflow: APT package mirror compatibility.
The final deliverable is easy to judge:
Can a reviewer run an unmodified APT client against a MirrorLine gateway and install a test package whose bytes are stored and retrieved through Sia?
If yes, the project delivered.
That makes MirrorLine a focused, measurable, and realistic small-grant project.