Introduction
Project Name: ObsidianLog
Name of the organization or individual submitting the proposal: Emmanuel Glory Praise
Software engineer with 5+ years of experience building full-stack and blockchain systems, specializing in TypeScript and Rust. At INTMAX (L2 privacy protocol), engineered and maintained SDKs powering the INTMAX Wallet integration layer, enabling developers to build privacy-preserving applications using client-side proving. Experienced in designing end-to-end systems across backend services, data models, and APIs, with a focus on scalability and developer usability. Built and implemented modular smart contract architectures leveraging standards like EIP-2535 (Diamond Standard). Contributed to developer ecosystem growth at Web3Bridge and 0xShip HQ through workshops, conference talks, and technical content, onboarding developers into Web3. Hands-on experience with SDKs and smart contract development across blockchains, including EVM chains (Solidity), Polkadot (ink!), Stellar (Soroban), Arbitrum (Stylus), etc.
Github: http://github.com/emmaglorypraise/
Describe your project.
ObsidianLog is a long-term log archival system built natively on Sia using the official developer SDKs.
It sits alongside existing observability stacks—Datadog, Grafana, and self-hosted ELK—as a cold-tier destination. Logs flow into hot tools for active monitoring, then automatically archive to Sia: client-side encrypted, zstd compressed, hash-chained for tamper-evidence, and queryable at a fraction of the cost.
The problem it solves is structural: many engineering teams are expected to retain operational logs for long periods due to internal audit requirements, incident investigation needs, and organizational retention policies. Existing cloud log storage solutions become prohibitively expensive at scale, causing teams to aggressively prune logs after short retention windows.
ObsidianLog makes long-term, tamper-evident archival economically viable by combining:
-
client-side encryption
-
compression append-only storage
-
cryptographic verification
on top of decentralized infrastructure.
This MVP grant funds the foundational layer:
-
a core Sia SDK storage library in Rust
-
a Vector-compatible HTTP ingest server
-
a CLI query tool, an interactive
obsidianlog initsetup wizard that reduces onboarding to under 15 minutes -
a live GitHub Actions demo any developer can fork immediately.
ObsidianLog is designed specifically for long-term decentralized operational log archival and observability workflows on Sia, with a strong focus on developer usability and self-hosted ownership.
By leveraging indexd, ObsidianLog also reduces the operational complexity traditionally required to interact with Sia, enabling developers to archive logs without managing low-level blockchain concerns such as contract formation or storage coordination.
How does the projected outcome serve the Foundation’s mission of user-owned data? What problem does your project solve?
Today, every engineering team’s operational logs containing sensitive system behaviour, security events, user actions, and infrastructure telemetry are owned and controlled by AWS, Datadog, or Elastic.
Teams:
-
cannot verify tamper-evidence
-
cannot guarantee retention
-
have no meaningful access controls independent of the vendor.
When you archive logs to AWS Glacier, Amazon controls the keys, the retention, and the access.
ObsidianLog returns that ownership entirely to the team that generated the data.
Encryption keys are generated locally and never transmitted; they exist only on the user’s infrastructure. Storage contracts are formed directly between the user’s Siacoin wallet and Sia hosts, with no intermediary. The builder is entirely out of the payment flow.
Every log chunk is append-only and hash-chained: SHA-256(previous chunk) is embedded in each new chunk, creating a cryptographic tamper-evident chain that any auditor can independently verify. If a chunk is altered or deleted, the chain breaks, detectably, provably, without relying on any third party’s word.
ObsidianLog provides the technical primitives teams need for verifiable long-term archival: client-side encryption, append-only storage, and cryptographic tamper-evidence. Whether these primitives satisfy a specific compliance or audit requirement depends on the team’s own implementation, operational policies, and auditor interpretation.
The project does not make compliance guarantees on behalf of storage providers or the Sia network. Instead, it gives teams full ownership over their archival pipeline and the ability to independently verify the integrity and retention of stored operational data.
ObsidianLog makes the Sia mission concrete for engineering teams that need verifiable long-term operational archival.
Are you a resident of any jurisdiction on that list? No
Will your payment bank account be located in any jurisdiction on that list? No
Grant Specifics
Amount of money requested and justification with a reasonable breakdown of expenses:
Total requested: $10,000 for a 3-month MVP development period.
| Line Item | Detail | Amount |
|---|---|---|
| Developer fees | 3 months work | $10,000 |
Grant payments are expected monthly via ACH/wire in USD.
Sia storage costs excluded: ObsidianLog is architecturally designed so that each user provisions and funds their own indexd deployment. The builder incurs zero ongoing storage or hosting costs. The grant funds engineering time only.
What is the high-level architecture overview for the grant?
What security best practices are you following?
ObsidianLog is a pipeline with four layers. Every log batch passes through all four before and after it touches Sia.
The system is designed to align with Sia’s modular architecture, leveraging indexd to simplify interaction with the network while maintaining full user control over data.
1. Ingestion Layer
The primary ingestion integration for the MVP is a local HTTP ingest endpoint that Vector routes to using its built-in HTTP sink, the fastest-growing modern log agent, and the de facto standard for new infrastructure builds. Developers add a single destination block to their vector.toml :
[sinks.obsidian]
type = "http"
inputs = ["your_source"]
uri = "<http://localhost:7080/ingest>"
encoding.codec = "json"
batch.timeout_secs = 300
batch.max_bytes = 10485760
ObsidianLog runs a lightweight local ingest server that receives log batches from Vector at this endpoint. No application code changes are required. Vector handles buffering, backpressure, and retry. If Sia is temporarily unavailable, logs buffer locally and flush when connectivity resumes; no data is lost. A native compiled Vector sink plugin is scoped into Phase 2 once the ingestion pipeline and integration interfaces stabilize.
2. Processing Layer
Each log batch passes through a deterministic, ordered pipeline before any data leaves the user’s infrastructure:
-
Parsing: Structured fields (timestamp, service, level, host, and trace_id) are extracted into a lightweight metadata index. The index is stored separately and is less than 1% of the full log size. All queries hit the index first; full log chunks are fetched only when needed.
-
Compression: zstd compression is applied to the full log batch. Log text is among the most compressible data types - repetitive structure, repeated field names, predictable patterns. In practice, zstd achieves 90–97% reduction on log data. 100 GB/day of raw logs becomes approximately 3–10 GB stored.
-
Encryption: The compressed batch is encrypted with AES-256-GCM using a user-generated key. Keys are created during
obsidianlog init, stored locally in the user’s keychain or a local secrets file, and never transmitted to any external system. The encryption is authenticated: the GCM tag ensures ciphertext integrity in addition to confidentiality. -
Hash chaining: Each encrypted chunk is assigned a SHA-256 hash. That hash is embedded in the next chunk’s header:
chunk_n.prev_hash = SHA-256(chunk_{n-1}). This creates an append-only tamper-evident chain. Any deletion, reordering, or modification of a chunk breaks the chain at a detectable position. Theobsidianlog verifycommand traverses the full chain and reports any integrity violations. -
Chunking: Logs are grouped into configurable time windows (default: 1 hour) and written as discrete chunk files. Chunks are named by time window for efficient range retrieval.
3. Sia Storage & Indexing Layer (via indexd)
During this process, metadata references are registered and resolved via indexd, which acts as both an indexing layer and a gateway for interacting with the Sia network. This enables efficient discovery and retrieval of stored log chunks without scanning full archives, while also abstracting away lower-level complexities such as contract management and storage coordination.
This aligns naturally with the system’s architecture, where lightweight metadata indexes are queried first before fetching encrypted log data.
The system separates responsibilities clearly:
-
Sia network: durable, decentralized storage of encrypted log chunks
-
indexd: structured indexing and retrieval coordination
-
ObsidianLog: ingestion, processing, encryption, and query logic
The on-Sia storage layout is:
/<bucket>/
index/
<service>/<YYYY-MM-DD-HH>.idx ← metadata index (lightweight, always fetched first)
chunks/
<service>/<YYYY-MM-DD-HH>.bin ← encrypted + compressed log data
manifest.json ← root manifest linking all chunks with hash chain values
All writes are append-only. Chunks are never modified or deleted post-write. The manifest tracks the current chain head and all chunk references.
4. Query Layer
The CLI queries the metadata index to identify relevant chunks (fast, no large downloads), then fetches and decrypts only the matching chunks.
Filters available in the MVP:
-
time range
-
service name
-
log level
-
host
-
keyword
Output formats:
-
human-readable terminal
-
JSON
-
raw (for piping to jq).
The obsidianlog verify subcommand traverses and validates the full hash chain.
Security practices followed:
-
Client-side encryption only: No plaintext log data ever leaves the user’s infrastructure. Encryption occurs before data is written to the Sia network and registered via indexd.
-
User-controlled keys: Key generation happens locally during
obsidianlog init. Keys are stored in the user’s OS keychain (via the keyring crate on Linux/macOS/Windows) or an explicit local secrets file with 0600 permissions. No key escrow, no key transmission. -
Authenticated encryption: AES-256-GCM provides both confidentiality and ciphertext integrity. Tampered ciphertext is rejected at decryption time.
-
Append-only storage model: Chunks are write-once. The storage model is designed to prevent in-place modification, relying on append-only writes combined with indexd-coordinated retrieval and hash chaining for tamper evidence. This provides strong tamper-evidence without requiring a blockchain.
-
No intermediary in the storage path: ObsidianLog connects to the user’s Sia node with indexd support for indexing and retrieval coordination, without any centralized intermediary. There is no ObsidianLog-operated proxy, gateway, or relay for the self-hosted MVP. The builder has zero access to user data.
-
Dependency audit: The Rust dependency tree will be audited with cargo audit in CI. Critical dependencies (zstd, aes-gcm, sha2) are maintained by the RustCrypto and zstd-rs communities with strong security track records.
Compliance Note
ObsidianLog is a self-hosted, user-controlled archival system, not a managed compliance platform.
The project provides technical primitives commonly useful in retention and audit workflows:
-
client-side encryption
-
append-only archival
-
cryptographic tamper-evidence
-
independent verification of stored data integrity
Compliance obligations remain entirely with the user, including host selection, operational controls, retention policies, and auditor interpretation. ObsidianLog does not certify or guarantee that storage providers on the Sia network satisfy any specific regulatory framework.
Teams with retention or audit requirements may use ObsidianLog as part of their broader operational and compliance processes.
What are the goals of this small grant? Please provide a general timeline for completion.
The MVP answers one core question: can real logs flow from a production-grade log agent into Sia, encrypted, compressed, and tamper-proof, and be retrieved accurately through a query interface? Every deliverable in the MVP exists to answer that question with evidence.
Everything else is explicitly deferred to Phase 2.
Month 1 - Core Storage & Ingestion
Goals:
Build the production core storage library and production-ready HTTP ingest server.
Deliverables:
-
obsidianlog-storecrate: storage library with compression, encryption, chunking, and hash-chain verification -
obsidianlog-ingestcrate: HTTP ingest service compatible with Vector -
integration test suite with CI
-
Architecture Decision Record documenting finalized storage decisions
-
Month 1 progress report submitted to the Sia Foundation forum
Month 2 - Query Tooling & Developer Experience
Goals:
Build retrieval tooling, query interface, and onboarding experience.
Deliverables:
-
obsidianlog CLI query interface
-
hash-chain verification tooling
-
obsidianlog initsetup wizard -
cross-platform binaries for Linux, macOS, and Windows
-
Docker Compose quickstart
-
Month 2 progress report submitted to the Sia Foundation forum
Month 3 - Launch & Ecosystem Integration
Goals:
Ship public developer tooling, integrations, and launch materials.
Deliverables:
-
GitHub Actions reusable workflow
-
documentation site
-
live end-to-end demo
-
example integrations with Grafana and SIEM export workflows
-
final MVP report with usage metrics and developer feedback
-
public launch
Success criteria
By the end of Month 3:
-
Logs flow from Vector to Sia end-to-end, typically within 60 seconds from setup completion under normal conditions
-
Archived logs are retrievable via CLI with correct filtering and intact content
-
obsidianlog verifyhash-chain check passes on all stored chunks -
obsidianlog initcompletes in under 15 minutes on a clean machine -
GitHub Actions demo is publicly available and forkable
-
At least 10 external developers have tested the tool and provided feedback. Outreach will target the Sia Discord, developer communities, and the Vector community Slack.
Who is the target user for your project?
The MVP is targeted at technically capable developers and small engineering teams comfortable with:
-
CLIs
-
config files
-
self-hosted infrastructure
-
self-hosted Sia infrastructure using indexd
Primary audience (MVP):
| Segment | Why They Adopt |
|---|---|
| Solo developers and indie hackers | Zero existing log retention. ObsidianLog provides audit-supporting archival at effectively zero cost. |
| Web3 / crypto-native teams | Already comfortable with wallet infrastructure. Philosophically aligned with user-owned data. Will trial early. |
| Sia ecosystem developers | Already using Sia. ObsidianLog is a natural addition to their existing workflow. |
| Privacy-first startups | Legal tech, health tech adjacent. Want “we don’t store your logs with Big Tech” as a genuine claim, not marketing. |
| Open source project maintainers | Long-term redundant archival for CI logs, release artifacts, and telemetry. |
Secondary audience (Phase 2, post-grant):
Mid-size engineering teams at growth-stage companies.
What are your plans for this project following the grant?
ObsidianLog is planned as a three-phase project.
Phase 2 - Full Product
Phase 2 expands ObsidianLog from a decentralized archival pipeline into a broader observability and operational intelligence platform for long-term infrastructure data.
The long-term vision is not only preserving logs cheaply, but enabling teams to query, analyze, verify, and re-integrate historical operational data into existing observability workflows.
The broader goal is to establish Sia as viable infrastructure for long-term operational and observability data, not only traditional file archival.
Search & Analysis
-
full-text search across archived logs
-
fuzzy and approximate matching
-
anomaly and spike detection for historical patterns
-
query relevance ranking
Workflow Integrations
-
Grafana datasource plugin
-
SIEM export integrations
-
Fluentd and Logstash plugins
-
batch import tooling for S3 and CloudWatch migrations
Audit & Verification
-
cryptographically signed export bundles
-
retention policy engine
-
append-only audit logs for query activity
-
integrity verification tooling
User Experience
-
web UI for search and visualization
-
timeline exploration and filtering
-
operational health dashboard
Phase 3 - Commercial
Phase 3 introduces the commercial tier that makes ObsidianLog accessible to non-technical teams and enterprises while remaining self-sustaining:
Managed hosted gateway:
Developers pay in USD via card while Siacoin complexity is fully abstracted. The gateway operator handles SC conversion and absorbs price exposure
Enterprise tier:
Features include SLA guarantees, SSO integration, dedicated compliance support, volume pricing
Stable USD pricing
Stable USD pricing to eliminate Siacoin volatility for budget-sensitive enterprise teams
Dashboard:
Operational tooling includes usage analytics, cost projections, storage health monitoring
SIEM integrations:
Enterprise integrations include: Splunk and IBM QRadar forwarding from archived logs
Sustainability model:
The open-source core (storage library, CLI, ingestion plugins) will remain MIT-licensed permanently. Community contributions keep the project alive independent of commercial outcomes.
Revenue from the managed gateway and enterprise tier funds ongoing development. Because the builder incurs zero infrastructure costs in the self-hosted model, the project is sustainable even at zero commercial revenue during the post-grant open-source phase.
Potential risks that will affect the outcome of the project:
- Cross-platform static binary shipping
Distributing a single Rust binary that works reliably on Linux, macOS, and Windows, including ARM variants, requires careful toolchain setup, especially around OpenSSL and system library linking.
Mitigation:
Static linking via
-
musl (Linux)
-
x86_64-apple-darwin
-
aarch64-apple-darwin (macOS)
-
x86_64-pc-windows-gnu (Windows)
will be established in CI during Month 2, not left to Month 3. The release pipeline is built before it is needed.
- Onboarding friction
The self-hosted Sia + indexd setup requires more steps than traditional SaaS tools like Datadog. This is the biggest adoption risk for the initial cohort.
Mitigation: The obsidianlog init wizard (Month 3 deliverable) is designed to reduce setup from a multi-hour process to under 15 minutes on a clean machine. A Docker Compose quickstart that spins up indexd alongside ObsidianLog will be published as an alternative path for developers who prefer containerized workflows.
- Sia network availability at query time
Users querying archived logs depend on Sia hosts being available.
Mitigation: ObsidianLog uses 3x redundancy via indexd’s default replication. The obsidianlog verify command provides a health check that confirms archive integrity and host availability on demand. Long-term retention and audit-support workflows - the primary driver - require that archived evidence is retrievable at audit time, not necessarily in real-time. Sia’s durability model (redundant storage across independent hosts) is well-suited to this access pattern.
- Indexd maturity and integration risk
Indexd is an evolving component of the Sia ecosystem, and integration patterns may continue to change as the ecosystem matures.
Mitigation: ObsidianLog is designed with modular storage abstractions to minimize coupling to any single integration layer. The storage pipeline and retrieval interfaces are intentionally isolated to allow future adaptation if indexd APIs or operational patterns evolve.
Development Information
Will all of your project’s code be open-source?
Yes. All code produced under this grant will be released under the MIT license.
Code review link
GitHub - http://github.com/emmaglorypraise/ObsidianLog
Do you agree to submit monthly progress reports?
Yes. Monthly progress reports will be submitted to the Sia Foundation forum at the end of each month
Contact
Email: [email protected]
Linkedin: https://linkedin.com/in/emmaglorypraise