Grant Proposal: IPFS/Sia Renterd/IPFSR

Grants Complete
1

IPFS Integration

Introduction

Project Name: IPFSR

Name of the organization or individual submitting the proposal:

Alvin Reyes

Describe your project.

By implementing a rented storage backend, the IPFS gateway will have the capability to securely store its data on Sia. This integration will provide a reliable and efficient solution for storing and retrieving data, enhancing the overall functionality and performance of the IPFS gateway.

Approach

The approach is to create a configuration around a Light IPFS or IPFS node to use Sia (via renterd) as a backend. This can either be done thru KUBO (GitHub - ipfs/kubo: An IPFS implementation in Go) - the generally accepted complete implementation of IPFS or a Light IPFS node - a lighter stripped down version of Kubo (whypfs-core).

There are multiple approach to achieve this.

Approach 1: Middleware between IPFS and SIA/Renterd - Replication

Approach is to create a middleware service between IPFS and Sia/Renterd. This service will act as a bridge, allowing any Content Identifier (CID) or content to be uploaded to Sia via renterd. When a user adds a file to an IPFS node, the middleware will listen or list all CIDs and then asynchronously call renterd API to upload the objects, essentially cloning all the data on an IPFS blockstore to SIA via rentered.

image
image2000Ɨ1374 211 KB

By utilizing this approach, we have the capability to successfully restore the data from Sia/Renterd and transfer it to a new IPFS node, ensuring the seamless continuity of information and preserving data integrity throughout the process.

image
image2000Ɨ1374 210 KB

This will be a daemon process that will connect an IPFS node and a SIA Renterd Node. It will list all ROOT CIDS from IPFS, pull it to the Renderd objects API for replication.

Approach 2: SIA/Renterd datastore implementation.

One approach is to create a data store implementation that uses Sia to store the data in batches. This implementation will persist all the data coming from IPFS to Sia as objects through renterd. However, it is worth noting that this approach may have performance limitations.

image
image2000Ɨ1201 134 KB

This will be a library that IPFS can use to replace the data store implementation interface with go-data-store GitHub - ipfs/go-datastore: key-value datastore interfaces. We will need the IPFS and Renterd to be hosted on the same machine sharing the same file system for a performant experience.

A few more approaches:

  • Approach 3: Renterd back up to IPFS (reverse)
  • Approach 4: IPFS Blockstore periodic backup (replication)

These approach are my recommendation based on my experience as a solution provider for IPFS users/businesses and FC Storage Providers. Iā€™d like to get the feedback and recommendation of the core team to ensure that the solution fits with Sia/Renterd mission/vision, core values and goals.

Who benefits from your project?

In any of these approach, Sia Renterd will benefit in any of them since IPFS usage has been increasing year after year.

  • Users who uses IPFS or a variation of IPFS that uses it to serve content. With the rented storage backend, users will have the peace of mind knowing that their data is securely stored on Sia, aligning with the principles of user-owned data.
  • Any project/businesses who uses IPFS as their cache/storage layer and wanted to use Sia as a replication layer for additional content addressable immutability. IPFS has a huge following from developers to enterprise businesses and a Sia backed integration will potentially immensely help the network usage.

I am using Sia for my ventures and I expect to maintain this tool in the long run.

How does the project serve the Foundationā€™s mission of user-owned data?

Promoting user control and ownership of their data.

Grant Specifics

Amount of money requested and justification with a reasonable breakdown of expenses:

Consider the following when submitting your budget.

  • The Foundation can only pay out grant funds in $USD via ACH or wire.
  • For grants of a period of one year or more, payments will be made quarterly.
  • For grants with a period of less than one year, payments will be made monthly.

REQUIRED: Milestones with which to judge your progress.

Milestones should be easy for the Grants Committee to understand and evaluate as your project moves through its term. The Committee reserves the right to accept, modify, or reject proposed milestones to ensure they represent thoughtful and reasonable project evaluation checkpoints. Further payments may be withheld for missed milestones.

Team

Alvin Reyes (Software Engineer)

Linkedin: https://www.linkedin.com/in/alvinpreyes

Github: https://github.com/alvin-reyes

Timeline with measurable objectives and goals.

Total Price of implementation:

  • Development: 20,000.00 USD
  • Website Development: 3,000.00 USD for the website (part of go-to-market strategy).

The estimate time is based on the fact that Iā€™ll be doing this as a side project.

Milestone 1 - Conceptualization and Prototype

  • Experiment with renterd and identify the integration points between IPFS and renterd in order to explore the potential synergies between the two systems.
  • Create an initial integration prototype by extending the data store implementation to use renterd, which will allow for a seamless integration and enhanced functionality (for approach 1).
  • Create an initial integration prototype by creating a simple blockstore event reader and lister to identify all the Content Identifiers (CIDs) stored on IPFS, which will provide valuable insights and data for further analysis and optimization.

Duration: 2 weeks

Output: Open source prototype and a conceptual design

Milestone 2 - Implementation

  • Build/Test the middleware components
    • Set up Sia/Renterd, which involves creating contracts and establishing the necessary infrastructure.
    • Implement IPFS and integrate an event listener to track all Content IDs (CIDs) across the system.
    • Develop a robust database system to keep track of the mapping between CIDs and Renterd instances, ensuring efficient management and retrieval of data.
    • Establish a comprehensive monitoring mechanism to ensure the seamless functioning of the Sia/Renterd and IPFS setup, enabling real-time tracking and analysis of system performance.
    • Implement automated backup and recovery mechanisms to safeguard data integrity and availability in case of any unforeseen events.
  • Build/Test the data store implementation
  • Build/Test storing and retrieval to and from Sia via renterd.
  • Build/Test rehydration in case IPFS blockstore gets corrupted or deleted.

Duration: 2 month(s) or less

Output: Definition of done - open source code, test documentation, technical design, documentation

Milestone 3 - QA / Soft and Hard Launch

  • Quality assurance testing.
  • Data integrity testing.
  • Beta testing with IPFS backed projects to use Sia/Renterd.
  • go-to-market

Duration: 3 weeks

Output: Open source code with functional test documentation and client-facing/dev design documentation. Go-To-Market strategy (website will be a separate cost if the team agrees)

Potential risks that will affect the outcome of the project:

    • Unexpected changes or limitations in the Renterd API.
  • Prioritization i.e Iā€™m doing this on the side and I will probably have days where I can only spend an hour or less for it.
  • QA - optional but Iā€™d like to have a good quality assurance testing and that may or may not push the timeline.

Development Information

Will all of your projectā€™s code be open-source?

Yes. Source code will be open as soon as itā€™s done.

Leave a link where code will be accessible for review.

Links.

Do you agree to submit monthly progress reports?

Yes

Contact info

Email: [email protected]

Any other preferred contact methods: [email protected]

2 Likes
2

Hi @alvinpreyes,

Thank you for putting together this grant proposal, I am very excited for IPFS to be integrated into the ecosystem. These approaches look logical and I look forward to what you figure out and decide during conceptualization and prototyping.

I wanted to share that Nate and myself have also been discussing the best way to integrate IPFS and have recently put together an experimental implementation. The goal has been to figure out an architecture that can provide efficient and fairly fast ā€œhotā€ storage while fully supporting IPFS/IPLD behaviours, APIs, bitswap, etc.

The approach we went with is most similar to Approach 2, but we opted to build a fully custom node from the IPFS boxo libraries so that we could really customize the implementation and build around the differences between the two protocols - we think this approach will allow us to built past any performance limitations. The node stores incoming block data in a renterd bucket with a few novel optimizations and a database to track certain IPFS/IPLD metadata. We are still learning about IPFS, making sure we do not break any core IPFS behaviours, and working on some missing features, but it already seems to work well.

This work was done very very recent so we have not shared it yet, but I wanted to mention it here as it will be valuable to get your thoughts on our approach, discuss implementation details, and collaborate in the future, especially if you decide to shoot for that more warm/hot storage approach with this grant. I am very excited to have a few people (hopefully even more in the future) figuring out how Sia and IPFS can best fit together!

3 Likes
3

What I actually have for delta.storage is the second one which includes a middleware but slightly different since the CIDs will come from a frontend (resource server) of https://app.delta.storage and a custom light version of IPFS (GitHub - application-research/delta-edge-urid: Edge Gateway <> Aggregator for Delta Import Deals) - the edge kubo node which can peer with Kubo (it uses boxo) nodes. I use that as a client-facing gateway for users to upload and retrieve files. The middleware then pulls all the CIDs from that gateway and make replications to Sia via Renterd (it was smooth btw).

I can adjust my current approach 1 (sorry, not 2) and make it a generic service for integrators.

1 Like
4

Thanks for your proposal! Due to the budget requested for this grant, this qualifies as a Standard Grant for the committee and has a review turn around time of 4-6 weeks. The committee should first be able to review this grant at their meeting on October 31st. Weā€™ll keep you updated!

3 Likes
5

It sounds like a benefit of your approach(s) is that the resulting components can probably be fit into a variety of existing IPFS setup/deployment, whereas our approach is aiming to achieve slightly hotter/fast access with the tradeoff that you must run our new IPFS node (while still being able to participate in the network). I like the idea of having projects exploring both of these options and tradeoffs.

1 Like
6

Yes exactly. IPFS has a broad scope of use. Iā€™m actually using it for some compute-over-data projects for sourcing and importing large datasets. The benefits of plug-and-play is it can be adjusted to whatever as long as the underlying storage uses IPFS + Sia.

2 Likes
7

Hello @alvinpreyes

Thanks for your submission! The committee would like to see the following changes to your proposal:

  • Please decide whether you are asking for the budget for the website. The committee is open to considering it, but not to deciding whether or not it is part of your request.
  • Please write your own risk assessment for your project. Even if the risks are similar, currently you have copied and pasted it from the RFP Telegram Alert Bot: https://forum.sia.tech/t/rfp-renterd-telegram-alert-bot/

Pending these changes, the committee has thoroughly reviewed your proposal and should be able to make a determination without waiting another two weeks for the next meeting.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

3 Likes
8

For the first point - I have added the details on the Potential risks that will affect the outcome of the project:. I honestly thought I added this but turns out I didnā€™t.

For the second point - I would like to include the website.

1 Like
9

Thanks @alvinpreyes, the committee has approved this proposal!

Weā€™ll reach out via email to set up grant program onboarding. Onboarding typically takes a week or two, so free to adjust your internal timelines as appropriate.

1 Like
10

Hi @alvinpreyes, weā€™ll need to see your first monthly progress report in order to send your next grant payment. You can find the format here. Reminder that these are due by the 2nd of each month. Thanks!

Apologies! You can disregard this. Your first progress report would have only been due two weeks after your first funding, and we donā€™t require them for that short a time period. Your first progress report will be due February 2. Thanks!

4 Likes
11

Yes. Iā€™ll be sending my progress report by end of January to Feb 2. Iā€™m looking to build as much integration as possible with IPFS (for my own work as well) and contribute that back to Sia. Iā€™m hoping itā€™ll be appreciated by the foundation. Thank you!

6 Likes
12

Progress Report Form - Jan 29, 2024

What progress was made on your grant this month?

Approach 1 - Build the IPFS and Renterd Integration - (still in progress - repos are private)

image

Approach 2 - Built the IPFS and Renterd Integration

1504Ɨ749 55.2 KB

IPFS Renterd Integration with the following features:

Fork of go-ds-flatfs for Sia/Renterd Integration

  • Achievement: The successful forking and modification of go-ds-flatfs to integrate with Sia and Renterd is a significant technical accomplishment. This custom version allows seamless interaction between an IPFS (KUBO) instance and Renterd, a decentralized storage system.
  • Impact: This integration ensures that data stored on the IPFS network can be automatically and efficiently duplicated onto Renterd. It not only provides redundancy but also leverages the decentralized nature of Renterd for enhanced data security and availability.

Automatic Data Replication to Renterd

  • Functionality Realized: With this feature, any content uploaded to IPFS is instantly replicated to Renterd. This creates a dual-storage mechanism where data resides both on the IPFS node and on Renterd.
  • Advantages and Use Cases: This automatic replication offers a robust backup solution, significantly reducing the risk of data loss. Itā€™s particularly useful in scenarios where data integrity and availability are critical. For instance, in a distributed application relying on IPFS for storage, this feature ensures that data remains accessible even if some IPFS nodes go offline.

Restoration of Data from Renterd

  • Implemented Mechanism: The ability to restore deleted files from Renterd upon a ā€˜getā€™ request is a key feature. When data is removed from the local flatfs storage to save space or for other reasons, it remains accessible through Renterd.
  • Practical Implications: This feature effectively creates a fail-safe for data stored in IPFS. Users donā€™t have to worry about permanently losing data due to local deletions, as Renterd acts as a secondary storage layer. Itā€™s especially valuable in decentralized applications where data persistence is crucial, as it provides a way to recover data without relying on a single point of failure.

Overall Significance of the Integration

  • Enhanced Data Reliability and Efficiency: The integration significantly boosts the reliability of data storage and retrieval within the IPFS ecosystem. It aligns with the core principles of IPFS and decentralized storage ā€” ensuring data is not just stored but is also redundantly available and persistently accessible.
  • Future Prospects and Potential Expansion: This integration could pave the way for further advancements in decentralized data storage solutions. The model could be replicated or adapted for integration with other decentralized storage networks, enhancing the robustness and capabilities of IPFS as a distributed file system.

Links to repos worked on this month:

A few more updates

What will you be working on this month?

  • Continue working on approach 1 and open source the project by end of February
  • Conduct testing and continuous enhancement of the protocol to ensure robust performance.
  • Explore and implement various methods for integrating with Renterd, build how-to guides, modules, libraries etc.
  • More integrations with IPFS tools to use Sia/Renterd as permanent storage.
  • Launch a more polished dedicated website for IPFSR to provide information, updates, and support resources.

Long term plan

  • Continuously maintain the project and add more tools, libraries and modules
  • Add more integration tooling, support for variations of IPFS (light, embedded etc).
13

Progress Report Form - March 5, 2024

What progress was made on your grant this month?

Added the following component: GitHub - IPFSR/go-ds-leveldb-sia: An implementation of go-datastore using leveldb with each key value pair backed by Sia Renterd node

The above extends the current implementation of approach 2 to also back up the leveldb of IPFS to Sia.

Links to repos worked on this month:

What will you be working on this month?

  • Continue working on approach 1 and open source the project by end of March
  • Conduct testing and continuous enhancement of the protocol to ensure robust performance.
  • Explore and implement various methods for integrating with Renterd, build how-to guides, modules, libraries etc.
  • More integrations with IPFS tools to use Sia/Renterd as permanent storage.
  • Launch a more polished dedicated website for IPFSR to provide information, updates, and support resources.

Long term plan

  • Continuously maintain the project and add more tools, libraries and modules
  • Add more integration tooling, support for variations of IPFS (light, embedded etc).
14

Hello @alvinpreyes,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

15

Hello @alvinpreyes,

You never posted a March progress report (Due on April 2nd detailing the progress of your work during March), your previous progress report was for February (though you called it the March report)

Seeing as how it is the 21st, it would be a bit redundant to ask you to post two reports within the span of 10 days, so please make sure you include your March and April progress in your April Progress Report (Due May 2nd).

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

16

Progress Report Form - May 1, 2024

What progress was made on your grant this month?

Added the following component:

Aside from the level db datastore, we also modified the ipfs-desktop app that hooks to IPFSR.

What will you be working on this month?

  • We are still working on the Approach 1. We will open source the version when ready.
  • Conduct testing and continuous enhancement of the protocol to ensure robust performance.
  • Explore and implement various methods for integrating with Renterd, build how-to guides, modules, libraries etc.
  • More integrations with IPFS tools to use Sia/Renterd as permanent storage.
  • Launch a more polished dedicated website for IPFSR to provide information, updates, and support resources.

Long term plan

  • Continuously maintain the project and add more tools, libraries and modules
  • Add more integration tooling, support for variations of IPFS (light, embedded etc)
17

Hello @alvinpreyes

Thank you for your progress report!

Congratulations on completing your grant! The team will reach out to you with information about the wind-down process and your next steps.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee