Standard Grant: Vup Vault - Personal Backup, Sync & Archive with indexd

Grants Complete
15

Grant Milestone Progress Report (May)

What progress was made on your grant this month?

  • Completed Milestone 4 end-to-end. The upload → read-back → share-URL cycle runs against the live Foundation indexer at https://sia.storage. With the AppKey and a stored SealedObject cached client-side, S5 apps using Sia storage no longer need renterd running and can fetch blobs direct from Sia hosts via Sdk::download.

  • Built s5_store_packing to bundle many small writes into larger packs before they reach a slow or per-operation-expensive backend. Generic over three sub-stores (blobs, manifests, staging) and reusable with any Store implementation, not just IndexdStore. The first-fit + slab-waste heuristic is ported from Sia Foundation’s s3d.

  • Bumped iroh 0.97 → 1.0.0-rc.0 to clear two RUSTSEC advisories that surfaced in hickory-proto 0.25 mid-month.

Architecture and design choices

IndexdStore<P, M> adapts sia_storage::Sdk to S5’s existing Store trait. P is any Store holding a 72-byte fixed-format pointer per caller path (object_id + metadata_hash). M is any BlobsReadWrite caching SealedObject bytes by their BLAKE3 hash; misses fall through to Sdk::object and write back. PackingStore<B, M, S> is an independent decorator that bundles small writes into ≤ 256 MiB packs before they hit a slow backend, with three pluggable sub-stores. Both crates speak only existing S5 traits and compose with TieredStore, CachedBlobsRead, LocalStore, MemoryStore, and the rest of the existing ecosystem.

Design choices:

  1. No BlobLocation::IndexdObject variant. Store::provide() returns the SDK’s signed share URL wrapped as BlobLocation::Url. Backend-specific location surfaces don’t belong on the key-value Store trait, which couples a storage concern (path → bytes) to a network-distribution concern (where on the network to fetch). The dev branch carries a deprecation notice on Store::provide() to formalise this.

  2. SealedObject persisted client-side. It is the trustless capability: with the AppKey it lets a client fetch from Sia hosts even if the indexd provider disappears. The current implementation caches SealedObjects in a content-addressed BlobsReadWrite keyed by their BLAKE3 hash.

  3. No fork or patch of sia_storage. Upstream ships SiaDecodable for SealedObject but not SiaEncodable. The new crate carries a local encoder mirroring the decoder field-for-field, with a TODO to upstream.

  4. Two encryption layers, intentional. L1 is FS5’s existing per-vault deterministic encryption (HKDF(plaintext_hash, vault_key, …)), which gives cross-device dedup. L3 is the SDK’s per-slab encryption applied at upload. No L2 inside PackingStore; if you need another layer, drop in an EncryptedStore<Inner> decorator.

  5. Three pluggable backends in PackingStore. blobs holds pack bodies (content-addressed). manifests holds per-pack PackManifest records (list() required at startup). staging is a durable write-ahead buffer for pending writes (LocalStore for crash survival, MemoryStore when re-ingest is cheap).

  6. Self-identifying manifests. 48-byte header with {magic, pack_hash, blob_count} followed by sorted 16-byte members {prefix[12], length: u32}. Reading a manifest does not depend on its filename, so the manifest store can use S5’s existing paths::path_for_hash convention (base64url / base32_fs / sharded per the backing store’s features()).

  7. In-memory index layout. HashMap<prefix, pack_index> for dispatch, binary search in the pack’s sorted members for the entry, cumulative sum of prior lengths for the offset. Avoids carrying length and pack_hash per entry; trades a small read-side CPU cost for the memory savings.

Status against the original M4 task table

Grant deliverable Status Notes
s5_store_indexd crate implementing all Store trait methods ✓ Every Store method implemented. 8 offline tests cover wire formats and AppKey vault. The interactive examples/end_to_end.rs exercises the live cycle.
Map S5 blob paths to indexd Object keys (consistent key derivation from content hash) ✓ Pointers are keyed by the caller’s path; the SDK assigns object IDs internally (Blake2b256 of slab metadata). IndexdStore records {object_id, BLAKE3-of-SealedObject} per caller path, so any path scheme works including FS5’s blob3/<base32(hash)> form. The pointer Store handles the indirection; no separate path-to-object-key translation step.
App registration flow: browser-based OAuth, store credentials ✓ (with deviation) Browser-OAuth via Builder::request_connection / wait_for_approval. The AppKey persists at ~/.config/s5/keys/indexd_appkey.bin, age-encrypted when a recipient is configured, 0600 plaintext otherwise. I went with age-encrypted files instead of the proposal’s keyring crate suggestion, matching the existing s5_node::identity_vault pattern for the master signing key. This keeps the security model unified across S5 and sidesteps platform-specific keychain failure modes (locked keychain, no daemon, etc.).
BlobLocation::IndexdObject variant for direct RHP4 downloads ✗ (covered differently) Direct RHP4 downloads still happen; Sdk::download issues them through internal SDK paths. I skipped the new variant per design choice 1. The dev branch’s deprecation notice on Store::provide() documents the plan: location sharing moves out of Store into a dedicated higher-level API.

Merged / open this month

Milestone Task PR Notes
M4 s5_store_indexd crate #8 IndexdStore<P, M> over sia_storage::Sdk. Browser-OAuth + age-encrypted AppKey caching. 72-byte fixed pointer wire format. Local SiaEncodable for SealedObject. put_bytes overwrites and reclaims the old Sia object best-effort. 8 offline tests + interactive E2E example.
M4 s5_store_packing crate #7 First-fit bin-packing with a slab-waste heuristic, self-identifying manifests, three pluggable backends, and crash-safe append-only writes that recover on restart. 15 unit + integration tests.
M4 iroh 0.97 → 1.0.0-rc.0, irpc 0.13 → 0.15 bundled in #7 Brings hickory-proto to 0.26.1, clears RUSTSEC-2026-0118 and -0119.

Validation

After git checkout stores-indexd:

# offline test suite for both new crates
cargo test -p s5_store_packing -p s5_store_indexd
# interactive E2E against the live Foundation indexer
# one-shot OAuth on first run, AppKey cached at ~/.config/s5/keys/indexd_appkey.bin
cargo run --example end_to_end -p s5_store_indexd

What will you be working on next month?

  • Land BlobsDelete and the Store::provide() deprecation on main.

  • Performance: parallel blob uploads in IndexdStore and intense testing

  • Vup Web refresh: mobile-responsive layout, modal-based dialogs replacing browser prompts, first-run onboarding wizard for indexd.

  • Documentation pass: user guide for vup, architecture doc for the pointer-and-metadata-cache layering inside IndexdStore.

  • Publish to crates.io: s5_store_indexd, s5_store_packing, updated s5_fs.

1 Like