Project Name: sia-sync-core — Real-Time File Sync Engine for Sia Storage Applications
Applicant: ILE Labs
Duration: 3 months
License: MIT / Apache-2.0
Introduction
Describe your project
sia-sync-core is an open-source Rust library built on the Sia Rust SDK (sia_storage) that gives developers a production-ready sync engine for applications where small files change constantly — config files, notes, project assets, chat attachments, collaborative documents, and device settings.
Today, developers building sync-capable applications on Sia face a specific, well-documented problem: Sia erasure-codes every upload into a fixed number of shards (10 data + 20 parity = 30 shards by default). Each shard occupies a full sector on a storage provider. This means a 1-byte file consumes the same storage footprint as a file that fills an entire slab. When small files change frequently, the naive approach — re-upload the entire file on every edit — results in massive slab waste, redundant network transfers, and unpredictable sync latency.
sia-sync-core solves this by providing:
A local change-detection engine that watches a directory or file set, computes content hashes, and identifies which objects have actually changed since the last sync cycle — only changed objects trigger network operations.
An intelligent packed-upload pipeline built on the SDK’s upload_packed API — changed files are batched into shared slabs before erasure coding, so ten small edits share one slab instead of wasting ten.
A metadata-driven sync state tracker using the SDK’s encrypted object metadata to store sync version vectors, content hashes, and timestamps — the indexer becomes the source of truth, no external database required.
A conflict resolution layer for multi-device scenarios — when the same file is modified on two devices before either syncs, sia-sync-core detects the conflict via version vectors and applies a configurable resolution strategy (last-writer-wins, keep-both, or application-defined merge).
$ sia-sync-core watch ./my-project --interval 5s
> Watching 47 files in ./my-project
> Sync cycle 1: 3 files changed, 2 new
> Packing 5 objects into 1 slab (saved 4 slab allocations)
> Uploaded: 12.4 KB across 5 objects in 1.8s
> Metadata updated: version vectors committed to indexer
> Sync cycle 2: 1 file changed (config.json)
> Delta detected: 340 bytes changed of 2.1 KB file
> Packed upload: 1 object, 1 shared slab
> Synced in 0.9s
$ sia-sync-core pull ./my-project --from device-b
> Fetching remote state from indexer...
> 2 files updated remotely
> 1 conflict detected: notes.md (modified on both devices)
> Resolution: keep-both → notes.md, notes.md.conflict
> Pull complete: 3 objects downloaded in 2.1s
Who benefits from your project?
Sia SDK developers — any team building an application where users create, edit, and sync small files (notes apps, password managers, config sync tools, project collaboration) gains a ready-made sync layer instead of solving repacking and conflict resolution from scratch.
End users of Sia-based applications — faster sync times, lower storage costs (shared slabs instead of wasted individual slabs), and reliable conflict handling when syncing across devices.
The Sia ecosystem — sia-sync-core directly addresses the “small files with constant changes” gap identified in the Foundation’s top priority projects list, making Sia competitive with centralized sync platforms (Dropbox, iCloud, Syncthing) for a category of applications that currently cannot be built efficiently on the network.
How does this serve the Foundation’s mission of user-owned data?
sia-sync-core makes it practical for developers to build applications where users own and sync their own data on Sia — without relying on centralized sync servers controlled by third parties. When a notes app or project tool uses sia-sync-core, the user’s files are encrypted client-side, erasure-coded across independent storage providers, and synchronized through the indexer — no corporation holds or controls the data at any point. This is user-owned data applied to the most common daily computing pattern: editing and syncing files across devices.
Grant Specifics
Amount requested and justification
Total: $25,000 USD
| Category | Amount | Description |
|---|---|---|
| M1: Change detection and packed sync engine | $8,500 | Rust library: file watcher, content hashing, batched upload_packed pipeline using Sia Rust SDK |
| M2: Metadata sync state and conflict resolution | $9,000 | Version vector tracking via SDK object metadata, multi-device conflict detection and resolution strategies |
| M3: CLI, demo application, and documentation | $7,500 | Developer CLI (sia-sync-core watch/pull/push), working demo app, full docs and examples |
All payments are milestone-gated. No payment is released without delivery review.
Timeline with measurable objectives
Total duration: 3 months
Milestone 1 — Change Detection and Packed Sync Engine (Weeks 1–4) — $8,500
| Task | Deliverable |
|---|---|
| Implement local file watcher with content-hash change detection | Rust library that monitors a directory, computes Blake3 hashes, and identifies changed/new/deleted files per sync cycle |
Build batched upload pipeline using sia_storage::upload_packed |
Changed files are grouped into packed uploads, sharing slab overhead instead of each consuming a full slab |
Implement download pipeline with sia_storage::download |
Pull changed objects from the network and write to local filesystem |
| Build object-to-file mapping registry | Persistent local index mapping filesystem paths to Sia Object IDs |
| Automated test suite | Unit tests for change detection, integration tests for packed upload/download against SDK |
Validation: Sync cycle completes for a directory of 50 small files (< 10 KB each) in under 10 seconds. Only changed files trigger network operations — unchanged files produce zero network traffic. Packed uploads reduce slab usage by 80%+ compared to individual uploads for typical small-file workloads. All tests pass in CI.
Milestone 2 — Metadata Sync State and Conflict Resolution (Weeks 5–8) — $9,000
| Task | Deliverable |
|---|---|
| Implement version vector sync state using SDK object metadata | Each object’s encrypted metadata stores a version vector (device ID + sequence number) and content hash, updated via sdk.update_object_metadata() |
| Multi-device state comparison engine | On pull, compare local version vectors against remote metadata from indexer to determine: no-change, fast-forward update, or conflict |
| Conflict resolution strategies | Three built-in strategies: last-writer-wins (by timestamp), keep-both (rename conflicting copy), application-defined callback |
| Deletion propagation | Track deleted files via tombstone metadata entries; propagate deletions across devices during sync |
| Integration test suite for multi-device scenarios | Simulated two-device sync with conflict injection and resolution verification |
Validation: Two simulated devices sync a shared directory. Concurrent edits to the same file are detected as conflicts. All three resolution strategies produce correct results. Deletion on device A propagates to device B on next pull. Version vectors are stored exclusively in SDK object metadata — no external database.
Milestone 3 — CLI, Demo Application, and Documentation (Weeks 9–12) — $7,500
| Task | Deliverable |
|---|---|
| Developer CLI | sia-sync-core init, watch, push, pull, status, resolve commands |
| Working demo application | A minimal Rust notes application that syncs a folder of .md files across two devices using sia-sync-core, demonstrating the full create → edit → sync → conflict-resolve lifecycle |
| Complete documentation | Quickstart guide, API reference, architecture overview, 10+ example usage scenarios |
| Open-source release | Published to GitHub under MIT/Apache-2.0 with crates.io release |
Validation: External developer clones the repo, runs the demo notes app, and completes a full sync cycle (create file → edit → push → pull on second device) in under 5 minutes following the README. CLI works without modification. Crate compiles on stable Rust 1.75+.
Post-grant roadmap and sustainability
sia-sync-core will be maintained as a community resource for 12 months post-delivery. This includes:
- SDK compatibility updates when
sia_storagereleases breaking changes - Community contribution acceptance and review
- Phase 2 scope: Selective sync (sync only specific directories or file types), real-time push notifications via indexer webhooks, indexd-native sync state storage for tighter ecosystem integration
The indexd compatibility layer is explicitly targeted for Phase 2. As the Foundation moves further into indexd-based development, sia-sync-core’s sync state management will migrate from object metadata to indexd-native data structures, ensuring the library evolves with the ecosystem.
Technical Architecture
Three components, each independently usable:
1. Change Detection and Packed Upload Engine (Rust)
Watches a local directory using notify (cross-platform filesystem events). On each sync cycle:
- Computes Blake3 content hashes for all watched files
- Compares against the local object registry to identify changed/new/deleted files
- Groups changed files into batches using
sia_storage::upload_packed - Each batch shares a single slab, reducing storage overhead for small files by 80%+
- Updates the local registry with new Object IDs
2. Metadata-Driven Sync State Engine
Every Sia object managed by sia-sync-core stores encrypted sync metadata via the SDK’s update_object_metadata API:
{
"sync_version": {"device-a": 4, "device-b": 3},
"content_hash": "blake3:a1b2c3...",
"modified_at": "2026-05-14T18:30:00Z",
"deleted": false
}
On pull, the engine fetches object metadata from the indexer and compares version vectors to determine sync actions. No external database or server is required — the indexer is the single source of truth.
3. CLI and Demo Application
Developer CLI for environment control and sync operations. A working demo application (Sia-backed notes sync) demonstrates the library in a real-world scenario.
Security considerations:
- All object data and metadata are encrypted client-side by the Sia SDK before leaving the device
- Sync state (version vectors, content hashes) is stored in encrypted object metadata — the indexer cannot read it
- No credentials stored — the CLI uses the SDK’s standard authentication flow
- No external servers — sync coordination happens entirely through the Sia SDK and indexer
- Rust memory safety throughout — no unsafe blocks in core sync logic
- Open-source: full codebase auditable by community
Development Information
Will all code be open-source? Yes. MIT / Apache-2.0 dual license.
Code location: github.com/ILE-Labs/sia-sync-core
Relevant prior work:
| Repo | Relevance |
|---|---|
| spark-note-poc | Rust cryptographic library (commitments, nullifiers, key derivation) — demonstrates the exact Rust + cryptographic data management skills sia-sync-core requires |
| stylus-debug-suite | Production Rust developer toolkit, live on crates.io — proves the team ships publishable Rust crates |
| mx-tx-simulator | Fixed 4 bugs in MultiversX official SDK — demonstrates protocol-level depth and upstream contribution culture |
| solana-cpi-lens | Transaction call tree reconstruction — demonstrates complex data-flow analysis capability |
Proof of concept: spark-note-poc is the proof of concept for Rust cryptographic capability. stylus-debug-suite is the proof of concept for shipping production Rust crates. Together, they demonstrate ILE Labs can design, implement, and publish a Rust library that handles cryptographic data operations — the core requirement for sia-sync-core.
Monthly progress reports: Yes, following Foundation format with PR links per task.
Demo availability: Yes, available for demo at monthly community calls at Milestone 2 and Milestone 3 completion.
ILE Labs — github.com/ILE-Labs | [email protected]