Small Grant: Orbitdb Sia Plugin

Project Name: Darkola

Name of the organization or individual submitting the proposal: Adrien Dalt

Project Description :

Darkola is an innovative tool that extends the capabilities of OrbitDB by allowing it to use the Sia decentralized storage network as its primary backend. This plugin provides OrbitDB users with a highly available, secure, and resilient storage solution, while preserving the decentralized and peer-to-peer nature of OrbitDB.

Key Features:

  • Decentralized Storage: The plugin allows OrbitDB data to be stored on the Sia network, distributing data across many nodes and reducing the risk of data loss in the event of a single node failure.
  • High Availability: The Sia network is designed to provide high availability, meaning that stored data is accessible at all times, even when individual servers fail.
  • Enhanced Security: Sia uses end-to-end encryption to protect data, and the network is designed to be resilient to attacks.
  • Cost-Effective: Storing on Sia is generally less expensive than traditional cloud storage solutions, especially for large volumes of data.
  • Seamless Integration: The plugin seamlessly integrates into OrbitDB, allowing developers to use Sia as a storage backend without having to significantly modify their code.

Current Project Status:

The plugin development is complete and has been successfully tested in a development environment. The code is fully functional and ready to be integrated into OrbitDB applications. The project need a fully functional documentation and some examples of use.

Next Steps:

  • Production Testing: We will encourage users to test the plugin in production environments and provide feedback.
  • User Documentation: We will develop comprehensive and easy-to-understand user documentation to help developers use the plugin.
  • Plugin publishing : We will publish the plugin on the node packet manager website www.npmjs.com

How does the projected outcome serve the foundation’s mission of user-owned data?

This plugin will enable OrbitDB users to store their data in a more secure and decentralized manner. By using Sia, users will no longer be dependent on a single centralized storage provider, significantly reducing the risks of data loss and censorship. Additionally, by making data storage more affordable and accessible, this plugin will encourage broader adoption of decentralized technologies.

Grant Specifics

Amount of grant requested: $3,500 For one month of work

Justification with a reasonable breakdown of expenses:

• Development | $2,500: Developer fees for plugin optimization and documentation writing.

• Testing | $500: Fees for conducting all unit tests, integration tests and performance tests.

• General expenses | $500: Fees for communication in developer groups and forums, hosting and cloud services fees.

Objectives of this small grant:

• Develop a functional plugin to use Sia as a storage backend for OrbitDB.

• Integrate the plugin into the OrbitDB ecosystem.

• Document the plugin to facilitate its use by the community.

Potential risks that will affect the outcome of the project:

• Technical complexity: Integrating Sia into OrbitDB can be complex due to the architectural differences between the two systems.

• Performance: The plugin performance can be affected by the latency and bandwidth of the Sia network.

• Adoption: Adoption of the plugin by the OrbitDB community can be slow if the benefits are not clearly demonstrated.

Development Information

Will all code in your project be open-source?

Yes, all plugin code will be released under a permissive open-source license (e.g., MIT or Apache 2.0).

Leave a link where code will be accessible for review.

Do you agree to submit monthly progress reports?

Yes we agree

Email Adress :

[email protected]

1 Like

Thanks for your recent proposal to the Sia Foundation Grants Program. After review, the committee has voted to approve your proposal.

We’ll reach out to the email address provided in your proposal. Onboarding to the Grants Program can take up to a couple weeks, please adjust your timelines as appropriate and be ready to provide those to us during the process.

Congratulations!

Hello Steve,
Thanks to the committee for this grant.
We have already done the documentation part of the project and it’s available here :

We hope to finish publishing the package to NPMJS this week as well.

UPDATE : I have setting up the pipeline and publishing the package on npmjs.com registry.
darkolats - npm

We have also carried out functional tests and we can confirm that the package is perfectly functional and usable for any Web3 project.

Hello Lauke,

I’m currently evaluating darkolats as a potential replacement for OrbitDB in my graph database in order to store to Sia instead of ipfs. I have a question about its capabilities:

Does darkolats support prefix path key matching in the get function? Specifically, if I store data with a hierarchical key (e.g., a/b/c), can I retrieve it with a partial key match like a/b, like how OrbitDB would do?

Hello Jules and sorry for the late answer.
The weekend was a little complicated on my side but it’s now that I see your requests. As it stands, I have only tested the plugin with full paths and I have not taken the time to test it with partial keys. I will take the time tomorrow to check and if possible integrate it into the plugin.

What progress was made on your grant this month?

Here are the different points I worked on during the previous month:

- We worked on the technical documentation of the plugin. We wrote detailed documentation to allow users to easily integrate the plugin into their project.

- We also worked during this month on setting up a pipeline to facilitate the publication of the plugin on the NPMJS marketplace

- We did a set of functional tests to verify the robustness of the plugin

Link to repos worked on this month

What will you be working on next month?

  • During the next month we will try to integrate the prefix path key matching in the get function

Hello @Lauke

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Hi Darkola,

Could be that my eyes are deceiving but seems you have to go back to the drawing board on this!

Your current implementation doesn’t store OrbitDB’s key-value store anywhere. This is a basic requirement. In OrbitDB, users can store data using any arbitrary key and retrieve data using that same key.

Your implementation takes an arbitrary key in the put operation then throws that away as your get operation has to use the returned CID from put to retrieve the same data. So its left to the user to manage the S5 CIDs to get their data. This makes no sense!

Because of this, it’s not possible to support prefix key queries to retrieve multiple matching results. You will need to maintain an index to do this as well as the key-value store.

Also without maintaining this mapping between user keys and S5 CIDs, it’s impossible to implement the other OrbitDB functions that you didn’t implement; del, iterator, merge and clear.

When you have this key-value mapping you will also have to maintain its sync between your OrbitDB node instances and deal with conflict resolution etc.

It seems you have a whole lot of code outstanding to do what OrbitDB does. You seem to have only done an extremely thin wrapper to S5 whilst offering confusing behaviour for put and get operation that makes no sense to me.

I can’t even begin to use this. What is going on? This has no OrbitDB functionality.