Project Name: CoreSync-rs
Applicant Information:
ILE Labs — ILE-Labs · GitHub
We are a Rust-focused blockchain infrastructure team specializing in low-level execution environments, cryptographic tooling, and high-performance systems. Open-source work includes stylus-debug-suite (Arbitrum Rust toolkit, live on crates.io), mx-tx-simulator (MultiversX transaction simulator, fixed 4 upstream SDK bugs), and foc-devkit (Filecoin FVM testing framework).
Prior Sia experience:
Before writing a single line of this proposal, we audited the sia_storage 0.9.1 source to understand what the SDK actually supports — not what we assumed. That audit revealed three constraints that directly shaped the current architecture:
- Writes are append-only. There is no in-place patch or slab-splice API — slab mutators are crate-private.
- Per-slab encryption keys are randomised on every upload. Client-side sector-level dedup is not possible at the SDK layer.
- The minimum on-network write is approximately 120 MiB per slab, regardless of payload size.
We also ran a full local Sia stack — PostgreSQL, indexd, and sia_storage SDK — and resolved two concrete integration blockers: the MaxMind GeoIP database requirement for indexd startup, and a hostname signature mismatch in sia_storage 0.9.1 where binding to 127.0.0.1 rather than localhost is required for signature verification to pass.
Full runbook: core-sync-rs/docs/LIVE_DEMO.md at main · ILE-Labs/core-sync-rs · GitHub
Note on the prior submission:
CoreSync-rs was reviewed and rejected by the Committee in June 2026. The cited reasons were: the live demo referenced renterd (which builders are not encouraged to use), the milestones lacked detail on repacking, and the deliverables appeared to be integration work around an existing crate rather than original engineering.
All three issues have been addressed. The repacking challenge is now the central engineering problem of the proposal, addressed with a specific architectural solution. The original work above FastCDC — the manifest schema, sync engine, delta assembly, and indexd integration — is explicitly framed and scoped.
Describe your project.
CoreSync-rs is an open-source Rust library that implements mutable-file sync on Sia’s append-only, erasure-coded storage substrate. It uses a log-structured, content-addressed design: FastCDC chunks (~KB, the dedup unit) are packed into ~40 MiB blocks (the write unit, sized to amortize the SDK’s slab floor) via the block upload path. A manifest layer maps chunk hashes to block and offset. On edit, only new chunks are uploaded — batched into one new block — and a new manifest points at unchanged old blocks plus the new one. No re-upload of unchanged data. No in-place patch required.
The library is built on two official Sia integration points: the sia_storage SDK for all storage operations, and indexd for manifest persistence.
How does the projected outcome serve the Foundation’s mission of user-owned data? What problem does your project solve?
Mutable data is the dominant use case for storage: documents, databases, application state. Every one of these changes constantly. Without differential sync, building Dropbox-style applications, backup tools, or collaborative editors on Sia means re-uploading the entire file on every edit — making continuous sync impractical and expensive.
CoreSync-rs gives developers a correct sync primitive for Sia’s actual storage model. The key word is correct: the prior submission assumed Sia behaved like a mutable block store. The SDK audit confirmed it does not. The current design works with the append-only, slab-based reality rather than around it, and delivers a provable storage bound as a result.
Are you a resident of any jurisdiction on that list? No
Will your payment bank account be located in any jurisdiction on that list? No
Grant Specifics
Amount requested: $8,000 USD
| Milestone | Item | Amount |
|---|---|---|
| Month 1 | Log-structured content-addressed store: block packer, manifest layer, reassembler, reconstruction integrity tests | $5,000 |
| Month 2 | SDK integration verification, benchmark, developer documentation, integration example | $3,000 |
The higher allocation in Month 1 reflects where the original engineering lives: the block packer, manifest spill logic, and reassembler are new systems work. Month 2 covers the integration surface, benchmark, and documentation needed for the release.
What is the high-level architecture overview? How does this project build on Sia?
CoreSync-rs builds on Sia through two official integration points only:
sia_storage SDK — all storage operations:
- Block upload path for packing CDC delta chunks into ~40 MiB blocks, amortizing the 120 MiB slab floor
- Ranged download for reconstruction: fetches only the slabs intersecting needed chunks, never full objects
- Delete and prune operations for Phase 2 compaction (out of scope for this grant)
indexd — manifest persistence:
- Each file’s chunk manifest (chunk_hash → block_id, offset, length) stored and retrieved per sync cycle
- Manifests exceeding the indexd metadata limit spill to a dedicated manifest object; only the root pointer stays in indexd metadata
Data flow:
Local file
→ FastCDC chunking (existing, tested)
→ SHA-256 chunk hashes
→ diff against indexd-stored manifest
→ pack new chunks into ~40 MiB block via sia_storage
→ write updated manifest to indexd
Security: Written entirely in safe Rust, no unsafe blocks. No network access during chunking or diff computation. SHA-256 chunk hashing with integrity verification before any delta is accepted. Credentials loaded from environment variables only, never hardcoded. Crash consistency: new block written and verified before manifest commit — a crash leaves uncollected garbage, never a dangling manifest reference.
Goals and timeline:
Goal: Deliver a Rust library that correctly implements mutable-file sync on Sia’s immutable substrate, with verified SDK integration and a live demo showing real bandwidth savings.
Month 1 — $5,000
src/block.rs— block packer accumulating CDC chunks into ~40 MiB staging buffers, flushed via SDK block uploadsrc/manifest.rsextension — chunk_hash → (block_id, offset, len) mapping; manifest spill to dedicated object when indexd metadata limit is exceededsrc/reassemble.rs— file reconstruction via ranged SDK download per chunktests/store_integration.rs— four edit scenarios (append, middle-insert, in-place, identical file) with SHA-256 equality assertion and confirmation that only new chunks were uploaded
Deliverable: all four edit scenarios reconstruct correctly with SHA-256 equality; test output confirms only changed chunks uploaded.
Month 2 — $3,000
- End-to-end demo against real Sia services (indexd + sia_storage SDK) with documented output showing bandwidth savings on first and second sync
benches/bandwidth.rs— N=50 mixed edits, reported upload bytes vs file size, reuse percentage- Developer documentation: integration guide, API reference, example showing CoreSync-rs used in a simple sync application
- GitHub release tagged v0.1.0
Deliverable: live demo output published in LIVE_DEMO.md with real numbers; codebase released at v0.1.0.
Who is the target user?
Developers building continuous sync, backup, or collaborative editing tools on Sia who need mutable-file semantics without re-uploading unchanged data or building their own chunking and manifest layer from scratch.
Plans following the grant:
Phase 2 (separate proposal after delivery): compactor with bounded on-network storage guarantee, CLI and background watch daemon, streaming I/O for large files, crates.io publication.
Maintenance: repository kept under MIT license, monthly forum updates beyond the grant period, SDK version monitoring when sia_storage releases updates.
Potential risks:
| Risk | Mitigation |
|---|---|
| sia_storage SDK API changes | SDK version pinned in Cargo.toml. Block packer and ranged download isolated in dedicated modules — SDK changes require updating one adapter layer. |
| indexd metadata limit varies | Manifest spill to a dedicated object implemented from Month 1 — root pointer in indexd is always a fixed-size hash. |
| ~40 MiB block size suboptimal for very small files | Disclosed in README: for very small files the slab floor means plain re-upload can win. This boundary is stated honestly rather than hidden. |
Development Information
Will all code be open-source? Yes. MIT license. No closed-source components. Third-party dependencies (fastcdc, sha2, serde, reqwest, sia_storage) are all appropriately open-source licensed.
Current state: 33/33 tests passing. Mock demo shows 81.7% upload savings on append edits and 38.3% savings on middle-insert edits. Both sia-live and sia-sdk feature flags compile cleanly with actionable error messages when credentials are absent.
Do you agree to submit monthly progress reports? Yes.
Contact
Email: [email protected]
Other: Telegram @charlesCode