Small Grant: indexd Adapter Recovery Harness

Grants Rejected
1

Introduction

Project Name:
indexd Adapter Recovery Harness

Our working MVP:

https://github.com/CreativesOnchain/indexd-Adapter-Recovery-Harnes

Name of the organization or individual submitting the proposal:
Creatives Onchain

Describe your project

indexd Adapter Recovery Harness is a repo-local CLI and GitHub Action for filesystem and sync-style integrations built on indexd.

It solves one specific reliability problem which is when an integration keeps local metadata or query state on top of indexd, can it recover correctly after interruption, resume from a saved cursor, replay packed small-object updates, and rebuild the same normalized local state after delete and prune-related changes?

This project focuses on that recovery layer because it is where real integration risk appears. indexd provides an object and event model, while application-level features such as paths, directories, metadata queries, and filesystem-style views are handled by the integration itself. When an integration maintains that additional local state, recovery after interruption is not a convenience feature. It is part of whether the integration can be trusted in normal use.

This grant does not build another integration, metadata store, VFS, sync engine, dashboard, browser tool, or SDK. It provides a shared recovery and rebuild harness that integrations can run in CI before release.

Demand and target audience

The target users are developers building filesystem, sync, backup, and adapter-style software on top of indexd, especially teams that maintain a local catalog, cursor, or query layer above the object and event model.

This demand is visible in the type of indexd integration work already being built on Sia. Approved and active integration work already depends on local-state handling above the raw object layer. That includes designs where state is reconstructed after restart, updated from event streams, and combined with packed uploads for small-object workloads. In that environment, the question is no longer whether local recovery matters. The question is whether integrations can verify that recovery before shipping changes.

The immediate users of this harness are integration developers. The end users who depend on those integrations benefit indirectly because failures in this layer can surface as stale listings, broken resume behavior, missing updates after restart, or cleanup mismatches.

How does the project serve the Foundation’s mission of user-owned data?

User-owned data is only practical when software built on top of Sia can recover, resync, and present the correct state after both normal operations and failure scenarios.

This project improves the reliability of third-party indexd integrations without moving more application logic into Foundation-maintained infrastructure. It supports the growth of useful Sia software by making recovery and rebuild behavior measurable before release.

Are you a resident of any jurisdiction on the sanctioned or monitored list?

No.

Will your payment bank account be located in any jurisdiction on that list?

No.

Grant Specifics

Amount requested and budget breakdown

Total requested: USD $9,000

Budget breakdown

  • Core runner, fixture engine, and local harness: $1,600

  • Adapter contract and normalized-state snapshot format: $900

  • Packed small-object update scenario set: $1,400

  • Recovery and rebuild scenario set: $1,700

  • Delete and prune recovery scenario set plus JSON and JUnit reporting: $1,400

  • GitHub Action, reference adapter, documentation, release, and 30-day maintenance window: $2,000

High-level architecture overview

The harness has four parts.

1. Scenario runner

A CLI written in Go that executes repeatable scenarios against either:

  • a user-supplied indexd test environment, or

  • a local reference environment included in the repository

The runner creates fixtures, applies updates, interrupts and resumes replay, runs delete and prune flows, and records expected outcomes.

2. Adapter contract

Each integration under test implements a small adapter contract with four commands:

  • apply-fixture

  • export-live-state

  • resume-from-cursor

  • rebuild-state

This keeps the harness reusable without forcing a specific storage model on integrations.

3. Normalized-state comparator

The harness compares:

  • the integration’s live local state after normal operations, and

  • the integration’s rebuilt local state after replay from indexd

Both are exported as normalized JSON snapshots with stable ordering so the harness can detect divergence clearly and repeatably.

4. GitHub Action

A GitHub Action wraps the CLI and produces:

  • console summary output

  • JSON artifacts

  • JUnit output for CI systems

Initial scope

Version 1 includes only three scenario families:

  • packed small-object updates

  • interruption plus cursor resume

  • delete and prune recovery

These three scenarios cover the core recovery question the project is designed to answer:

after realistic update, interruption, replay, and cleanup flows, does rebuilt local state still match live local state?

What is out of scope

This grant will not:

  • build a metadata store

  • ship a VFS

  • ship a sync engine

  • build a dashboard

  • modify indexd, hostd, or official SDKs

  • deliver browser tooling

  • build a consumer-facing application

  • implement a protocol gateway such as NFS, SMB, or WebDAV

The output is a reusable adapter recovery harness for integrations that already maintain app-layer state on top of indexd.

Security and engineering practices

  • Test-only credentials will be supplied through environment variables or CI secrets and will never be committed.

  • Sensitive values will be redacted from logs and artifacts.

  • The harness will use synthetic fixtures only.

  • Adapter commands will not print credentials or other sensitive values.

  • Dependencies will be pinned for reproducibility.

  • Releases will clearly state the supported indexd version range used in testing.

  • The repository will be fully open source under the MIT license.

Timeline with measurable objectives

Milestone 1 — Core harness and first scenario family

Timeline: 2 weeks( Month 1)
Budget: $4,200

Deliverables

  • CLI scaffold and local harness

  • adapter contract and normalized JSON snapshot schema

  • packed small-object update scenario set

  • reference adapter

  • public repository with setup documentation

Success criteria

  • the reference adapter runs the packed-update scenario in CI

  • live-state and rebuilt-state snapshots match

  • an intentionally broken fixture produces a repeatable failure report

Milestone 2 — Recovery, prune, and release

Timeline: 2 additional weeks (Month 1)
Budget: $4,800

Deliverables

  • interruption and cursor-resume scenario set

  • delete and prune recovery scenario set

  • JSON and JUnit reporting

  • GitHub Action

  • example workflow file

  • v1.0 release

  • 30-day maintenance window after release

Success criteria

  • all three scenario families run in CI

  • the harness detects divergence between live-state and rebuilt-state snapshots

  • the documentation shows how an external integration adds the adapter contract and runs the harness in its own repository

Potential risks and mitigation

Risk 1: indexd behavior changes during development

If supported behavior changes during the build window, fixture expectations may need adjustment.

Mitigation:
Version 1 will pin a supported version range and keep fixture expectations versioned.

Risk 2: Differences between integration models

Different integrations may model local state differently.

Mitigation:
Version 1 targets filesystem and sync-style integrations that can export a normalized local snapshot through the adapter contract. That keeps the harness reusable while avoiding unnecessary abstraction in the first release.

Risk 3: Local and remote environments behave differently

A harness that only works in one environment is not useful.

Mitigation:
The runner will support both a local reference environment and a user-supplied test environment so integrations can validate against their own setup.

Plans beyond the duration of the grant

The repository will remain public after the grant.

We will:

  • keep issues enabled

  • publish patch releases during the maintenance window

  • keep the harness focused on the initial recovery scenarios until at least one external integration validates the adapter contract in practice

Development Information

Will all of your project’s code be open source?

Yes, 100%.

Leave a link where code will be accessible for review

GitHub - CreativesOnchain/indexd-Adapter-Recovery-Harnes: A CI-ready testing harness and CLI tool designed to guarantee that indexd integrations can safely recover from catastrophic system interruptions without suffering localized data drift. · GitHub

Do you agree to submit monthly progress reports?

Yes.

Do you agree to participate in a demo at a community call at a significant milestone or after completion?

Yes.

Contact info

Email: [email protected]
Creatives Onchain is a builder collective focused on developer tooling, protocol infrastructure, and operator-facing systems for blockchain ecosystems.

A recent example of our work is Stylus Trace Studio in the Arbitrum ecosystem. It is a profiling and trace inspection tool for Arbitrum Stylus that helps developers inspect execution traces, analyze gas and ink usage, generate flamegraphs, and catch performance regressions during development and CI. This is relevant to this proposal because it reflects direct experience building low-level developer tooling with structured outputs for real engineering workflows.

Links:
https://crates.io/crates/stylus-trace-studio/0.1.11
https://github.com/CreativesOnchain/Stylus-Trace

Why this project is relevant now

This project addresses a concrete integration problem in the part of the Sia ecosystem where active work is already happening: software built on top of indexd that keeps local state above the object and event model.

As more integrations build file views, local catalogs, and replay-based update logic on top of indexd, recovery correctness becomes a release requirement, not a later improvement. This harness gives those integrations a practical way to verify that local state can survive interruption, replay, and cleanup without drift.

That makes it useful immediately to the kind of indexd integrations already emerging on Sia, while staying separate from product development, protocol gateway work, and core Foundation roadmap work.

Prove of Work:

Creatives Onchain is a builder collective focused on developer tooling, protocol infrastructure, and operator-facing systems for blockchain ecosystems.

A recent example of our work is Stylus Trace Studio in the Arbitrum ecosystem. It is a profiling and trace inspection tool for Arbitrum Stylus that helps developers inspect execution traces, analyze gas and ink usage, generate flamegraphs, and catch performance regressions during development and CI. This is relevant to this proposal because it reflects direct experience building low-level developer tooling with structured outputs for real engineering workflows.

Links:
https://crates.io/crates/stylus-trace-studio/0.1.11
https://github.com/CreativesOnchain/Stylus-Trace

2

Hi @CreativeOnchain - thank you for your proposal. Please edit your timeline to reflect our monthly timeline structure.

In order for this proposal to be reviewed by the Committee at next week’s meeting, the above edit will be needed by this Wed. April 22 by 5pm ET.

3

Hi @mecsbecs , we have updated the timeline , it is a one month project but we will constantly keep maintaining after the grant for 3 month and more. Let us knwo if you need us to do anything else. Thanks.

4

Hi @CreativeOnchain - thank you for clarifying it is a one-month project. This will be reviewed by the Committee next week.

5

Thank you @mecsbecs . We also wanted to mention that we have built a working MVP for this project and we would like it to be reviewed as well: https://github.com/CreativesOnchain/indexd-Adapter-Recovery-Harnes

6

Hello @CreativeOnchain,

The Committee was unfortunately unable to get to your proposal during today’s meeting. Your proposal has now been slotted for review during the next meeting on May 12th.

Thank you for your patience.

7

Thanks for your proposal to The Sia Foundation Grants Program.

After review, the Committee has decided to reject your proposal citing the following reasons:

  • The Committee believes this would be more useful as an application-specific concept, rather than as a general solution.

We’ll be moving this to the Rejected section of the Forum. Thanks again for your proposal, and you’re always welcome to submit new requests if you feel you can address the Committee’s concerns.