Grant Proposal: IPFS/Sia Renterd/IPFSR

IPFS Integration

Introduction

Project Name: IPFSR

Name of the organization or individual submitting the proposal:

Alvin Reyes

Describe your project.

By implementing a rented storage backend, the IPFS gateway will have the capability to securely store its data on Sia. This integration will provide a reliable and efficient solution for storing and retrieving data, enhancing the overall functionality and performance of the IPFS gateway.

Approach

The approach is to create a configuration around a Light IPFS or IPFS node to use Sia (via renterd) as a backend. This can either be done thru KUBO (GitHub - ipfs/kubo: An IPFS implementation in Go) - the generally accepted complete implementation of IPFS or a Light IPFS node - a lighter stripped down version of Kubo (whypfs-core).

There are multiple approach to achieve this.

Approach 1: Middleware between IPFS and SIA/Renterd - Replication

Approach is to create a middleware service between IPFS and Sia/Renterd. This service will act as a bridge, allowing any Content Identifier (CID) or content to be uploaded to Sia via renterd. When a user adds a file to an IPFS node, the middleware will listen or list all CIDs and then asynchronously call renterd API to upload the objects, essentially cloning all the data on an IPFS blockstore to SIA via rentered.

By utilizing this approach, we have the capability to successfully restore the data from Sia/Renterd and transfer it to a new IPFS node, ensuring the seamless continuity of information and preserving data integrity throughout the process.

This will be a daemon process that will connect an IPFS node and a SIA Renterd Node. It will list all ROOT CIDS from IPFS, pull it to the Renderd objects API for replication.

Approach 2: SIA/Renterd datastore implementation.

One approach is to create a data store implementation that uses Sia to store the data in batches. This implementation will persist all the data coming from IPFS to Sia as objects through renterd. However, it is worth noting that this approach may have performance limitations.

This will be a library that IPFS can use to replace the data store implementation interface with go-data-store GitHub - ipfs/go-datastore: key-value datastore interfaces. We will need the IPFS and Renterd to be hosted on the same machine sharing the same file system for a performant experience.

A few more approaches:

  • Approach 3: Renterd back up to IPFS (reverse)
  • Approach 4: IPFS Blockstore periodic backup (replication)

These approach are my recommendation based on my experience as a solution provider for IPFS users/businesses and FC Storage Providers. I’d like to get the feedback and recommendation of the core team to ensure that the solution fits with Sia/Renterd mission/vision, core values and goals.

Who benefits from your project?

In any of these approach, Sia Renterd will benefit in any of them since IPFS usage has been increasing year after year.

  • Users who uses IPFS or a variation of IPFS that uses it to serve content. With the rented storage backend, users will have the peace of mind knowing that their data is securely stored on Sia, aligning with the principles of user-owned data.
  • Any project/businesses who uses IPFS as their cache/storage layer and wanted to use Sia as a replication layer for additional content addressable immutability. IPFS has a huge following from developers to enterprise businesses and a Sia backed integration will potentially immensely help the network usage.

I am using Sia for my ventures and I expect to maintain this tool in the long run.

How does the project serve the Foundation’s mission of user-owned data?

Promoting user control and ownership of their data.

Grant Specifics

Amount of money requested and justification with a reasonable breakdown of expenses:

Consider the following when submitting your budget.

  • The Foundation can only pay out grant funds in $USD via ACH or wire.
  • For grants of a period of one year or more, payments will be made quarterly.
  • For grants with a period of less than one year, payments will be made monthly.

REQUIRED: Milestones with which to judge your progress.

Milestones should be easy for the Grants Committee to understand and evaluate as your project moves through its term. The Committee reserves the right to accept, modify, or reject proposed milestones to ensure they represent thoughtful and reasonable project evaluation checkpoints. Further payments may be withheld for missed milestones.

Team

Alvin Reyes (Software Engineer)

Linkedin: https://www.linkedin.com/in/alvinpreyes

Github: https://github.com/alvin-reyes

Timeline with measurable objectives and goals.

Total Price of implementation:

  • Development: 20,000.00 USD
  • Website Development: 3,000.00 USD for the website (part of go-to-market strategy).

The estimate time is based on the fact that I’ll be doing this as a side project.

Milestone 1 - Conceptualization and Prototype

  • Experiment with renterd and identify the integration points between IPFS and renterd in order to explore the potential synergies between the two systems.
  • Create an initial integration prototype by extending the data store implementation to use renterd, which will allow for a seamless integration and enhanced functionality (for approach 1).
  • Create an initial integration prototype by creating a simple blockstore event reader and lister to identify all the Content Identifiers (CIDs) stored on IPFS, which will provide valuable insights and data for further analysis and optimization.

Duration: 2 weeks

Output: Open source prototype and a conceptual design

Milestone 2 - Implementation

  • Build/Test the middleware components
    • Set up Sia/Renterd, which involves creating contracts and establishing the necessary infrastructure.
    • Implement IPFS and integrate an event listener to track all Content IDs (CIDs) across the system.
    • Develop a robust database system to keep track of the mapping between CIDs and Renterd instances, ensuring efficient management and retrieval of data.
    • Establish a comprehensive monitoring mechanism to ensure the seamless functioning of the Sia/Renterd and IPFS setup, enabling real-time tracking and analysis of system performance.
    • Implement automated backup and recovery mechanisms to safeguard data integrity and availability in case of any unforeseen events.
  • Build/Test the data store implementation
  • Build/Test storing and retrieval to and from Sia via renterd.
  • Build/Test rehydration in case IPFS blockstore gets corrupted or deleted.

Duration: 2 month(s) or less

Output: Definition of done - open source code, test documentation, technical design, documentation

Milestone 3 - QA / Soft and Hard Launch

  • Quality assurance testing.
  • Data integrity testing.
  • Beta testing with IPFS backed projects to use Sia/Renterd.
  • go-to-market

Duration: 3 weeks

Output: Open source code with functional test documentation and client-facing/dev design documentation. Go-To-Market strategy (website will be a separate cost if the team agrees)

Potential risks that will affect the outcome of the project:

    • Unexpected changes or limitations in the Renterd API.
  • Prioritization i.e I’m doing this on the side and I will probably have days where I can only spend an hour or less for it.
  • QA - optional but I’d like to have a good quality assurance testing and that may or may not push the timeline.

Development Information

Will all of your project’s code be open-source?

Yes. Source code will be open as soon as it’s done.

Leave a link where code will be accessible for review.

Links.

Do you agree to submit monthly progress reports?

Yes

Contact info

Email: areyesonl@gmail.com

Any other preferred contact methods: me@areyesonl.com

Hi @alvinpreyes,

Thank you for putting together this grant proposal, I am very excited for IPFS to be integrated into the ecosystem. These approaches look logical and I look forward to what you figure out and decide during conceptualization and prototyping.

I wanted to share that Nate and myself have also been discussing the best way to integrate IPFS and have recently put together an experimental implementation. The goal has been to figure out an architecture that can provide efficient and fairly fast “hot” storage while fully supporting IPFS/IPLD behaviours, APIs, bitswap, etc.

The approach we went with is most similar to Approach 2, but we opted to build a fully custom node from the IPFS boxo libraries so that we could really customize the implementation and build around the differences between the two protocols - we think this approach will allow us to built past any performance limitations. The node stores incoming block data in a renterd bucket with a few novel optimizations and a database to track certain IPFS/IPLD metadata. We are still learning about IPFS, making sure we do not break any core IPFS behaviours, and working on some missing features, but it already seems to work well.

This work was done very very recent so we have not shared it yet, but I wanted to mention it here as it will be valuable to get your thoughts on our approach, discuss implementation details, and collaborate in the future, especially if you decide to shoot for that more warm/hot storage approach with this grant. I am very excited to have a few people (hopefully even more in the future) figuring out how Sia and IPFS can best fit together!

1 Like

What I actually have for delta.storage is the second one which includes a middleware but slightly different since the CIDs will come from a frontend (resource server) of https://app.delta.storage and a custom light version of IPFS (GitHub - application-research/delta-edge-urid: Edge Gateway <> Aggregator for Delta Import Deals) - the edge kubo node which can peer with Kubo (it uses boxo) nodes. I use that as a client-facing gateway for users to upload and retrieve files. The middleware then pulls all the CIDs from that gateway and make replications to Sia via Renterd (it was smooth btw).

I can adjust my current approach 1 (sorry, not 2) and make it a generic service for integrators.

Thanks for your proposal! Due to the budget requested for this grant, this qualifies as a Standard Grant for the committee and has a review turn around time of 4-6 weeks. The committee should first be able to review this grant at their meeting on October 31st. We’ll keep you updated!

1 Like

It sounds like a benefit of your approach(s) is that the resulting components can probably be fit into a variety of existing IPFS setup/deployment, whereas our approach is aiming to achieve slightly hotter/fast access with the tradeoff that you must run our new IPFS node (while still being able to participate in the network). I like the idea of having projects exploring both of these options and tradeoffs.

Yes exactly. IPFS has a broad scope of use. I’m actually using it for some compute-over-data projects for sourcing and importing large datasets. The benefits of plug-and-play is it can be adjusted to whatever as long as the underlying storage uses IPFS + Sia.

Hello @alvinpreyes

Thanks for your submission! The committee would like to see the following changes to your proposal:

  • Please decide whether you are asking for the budget for the website. The committee is open to considering it, but not to deciding whether or not it is part of your request.
  • Please write your own risk assessment for your project. Even if the risks are similar, currently you have copied and pasted it from the RFP Telegram Alert Bot: https://forum.sia.tech/t/rfp-renterd-telegram-alert-bot/

Pending these changes, the committee has thoroughly reviewed your proposal and should be able to make a determination without waiting another two weeks for the next meeting.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

1 Like

For the first point - I have added the details on the Potential risks that will affect the outcome of the project:. I honestly thought I added this but turns out I didn’t.

For the second point - I would like to include the website.

Thanks @alvinpreyes, the committee has approved this proposal!

We’ll reach out via email to set up grant program onboarding. Onboarding typically takes a week or two, so free to adjust your internal timelines as appropriate.