IPFS Integration
Introduction
Project Name: IPFSR
Name of the organization or individual submitting the proposal:
Alvin Reyes
Describe your project.
By implementing a rented storage backend, the IPFS gateway will have the capability to securely store its data on Sia. This integration will provide a reliable and efficient solution for storing and retrieving data, enhancing the overall functionality and performance of the IPFS gateway.
Approach
The approach is to create a configuration around a Light IPFS or IPFS node to use Sia (via renterd) as a backend. This can either be done thru KUBO (GitHub - ipfs/kubo: An IPFS implementation in Go) - the generally accepted complete implementation of IPFS or a Light IPFS node - a lighter stripped down version of Kubo (whypfs-core).
There are multiple approach to achieve this.
Approach 1: Middleware between IPFS and SIA/Renterd - Replication
Approach is to create a middleware service between IPFS and Sia/Renterd. This service will act as a bridge, allowing any Content Identifier (CID) or content to be uploaded to Sia via renterd. When a user adds a file to an IPFS node, the middleware will listen or list all CIDs and then asynchronously call renterd API to upload the objects, essentially cloning all the data on an IPFS blockstore to SIA via rentered.
By utilizing this approach, we have the capability to successfully restore the data from Sia/Renterd and transfer it to a new IPFS node, ensuring the seamless continuity of information and preserving data integrity throughout the process.
This will be a daemon process that will connect an IPFS node and a SIA Renterd Node. It will list all ROOT CIDS from IPFS, pull it to the Renderd objects API for replication.
Approach 2: SIA/Renterd datastore implementation.
One approach is to create a data store implementation that uses Sia to store the data in batches. This implementation will persist all the data coming from IPFS to Sia as objects through renterd. However, it is worth noting that this approach may have performance limitations.
This will be a library that IPFS can use to replace the data store implementation interface with go-data-store GitHub - ipfs/go-datastore: key-value datastore interfaces. We will need the IPFS and Renterd to be hosted on the same machine sharing the same file system for a performant experience.
A few more approaches:
- Approach 3: Renterd back up to IPFS (reverse)
- Approach 4: IPFS Blockstore periodic backup (replication)
These approach are my recommendation based on my experience as a solution provider for IPFS users/businesses and FC Storage Providers. I’d like to get the feedback and recommendation of the core team to ensure that the solution fits with Sia/Renterd mission/vision, core values and goals.
Who benefits from your project?
In any of these approach, Sia Renterd will benefit in any of them since IPFS usage has been increasing year after year.
- Users who uses IPFS or a variation of IPFS that uses it to serve content. With the rented storage backend, users will have the peace of mind knowing that their data is securely stored on Sia, aligning with the principles of user-owned data.
- Any project/businesses who uses IPFS as their cache/storage layer and wanted to use Sia as a replication layer for additional content addressable immutability. IPFS has a huge following from developers to enterprise businesses and a Sia backed integration will potentially immensely help the network usage.
I am using Sia for my ventures and I expect to maintain this tool in the long run.
How does the project serve the Foundation’s mission of user-owned data?
Promoting user control and ownership of their data.
Grant Specifics
Amount of money requested and justification with a reasonable breakdown of expenses:
Consider the following when submitting your budget.
- The Foundation can only pay out grant funds in $USD via ACH or wire.
- For grants of a period of one year or more, payments will be made quarterly.
- For grants with a period of less than one year, payments will be made monthly.
REQUIRED: Milestones with which to judge your progress.
Milestones should be easy for the Grants Committee to understand and evaluate as your project moves through its term. The Committee reserves the right to accept, modify, or reject proposed milestones to ensure they represent thoughtful and reasonable project evaluation checkpoints. Further payments may be withheld for missed milestones.
Team
Alvin Reyes (Software Engineer)
Linkedin: https://www.linkedin.com/in/alvinpreyes
Github: https://github.com/alvin-reyes
- I’m the Founder of Delta.Storage (https://delta.storage) - Data Preservation Platform that uses Sia as one of the DSN/Replication system.
- I’m the Founder and main host of Blockchain for Executives Podcast (https://www.youtube.com/@BlockchainforExecutives).
- I’m a Former Software Architect and Engineer for Protocol Labs. I’m the author of delta storage and the Lead Architect of Filecoin Data Tools - an platform for onboarding data to the Filecoin network. (https://filecoin.io/blog/posts/introducing-filecoin-data-tools-fdt/#:~:text=FDT (Filecoin Data Tools) is,revenue and incentivisation built in.). I’m a contributor to several PL tech stack such as: Lotus, IPFS, Boost, IPLD, Lurk, Estuary.
- I’m an advocate of Web3/Blockchain Tech, and my focus has been building use cases on top of Decentralized Storage Networks.
Timeline with measurable objectives and goals.
Total Price of implementation:
- Development: 20,000.00 USD
- Website Development: 3,000.00 USD for the website (part of go-to-market strategy).
The estimate time is based on the fact that I’ll be doing this as a side project.
Milestone 1 - Conceptualization and Prototype
- Experiment with renterd and identify the integration points between IPFS and renterd in order to explore the potential synergies between the two systems.
- Create an initial integration prototype by extending the data store implementation to use renterd, which will allow for a seamless integration and enhanced functionality (for approach 1).
- Create an initial integration prototype by creating a simple blockstore event reader and lister to identify all the Content Identifiers (CIDs) stored on IPFS, which will provide valuable insights and data for further analysis and optimization.
Duration: 2 weeks
Output: Open source prototype and a conceptual design
Milestone 2 - Implementation
- Build/Test the middleware components
- Set up Sia/Renterd, which involves creating contracts and establishing the necessary infrastructure.
- Implement IPFS and integrate an event listener to track all Content IDs (CIDs) across the system.
- Develop a robust database system to keep track of the mapping between CIDs and Renterd instances, ensuring efficient management and retrieval of data.
- Establish a comprehensive monitoring mechanism to ensure the seamless functioning of the Sia/Renterd and IPFS setup, enabling real-time tracking and analysis of system performance.
- Implement automated backup and recovery mechanisms to safeguard data integrity and availability in case of any unforeseen events.
- Build/Test the data store implementation
- Build/Test storing and retrieval to and from Sia via renterd.
- Build/Test rehydration in case IPFS blockstore gets corrupted or deleted.
Duration: 2 month(s) or less
Output: Definition of done - open source code, test documentation, technical design, documentation
Milestone 3 - QA / Soft and Hard Launch
- Quality assurance testing.
- Data integrity testing.
- Beta testing with IPFS backed projects to use Sia/Renterd.
- go-to-market
Duration: 3 weeks
Output: Open source code with functional test documentation and client-facing/dev design documentation. Go-To-Market strategy (website will be a separate cost if the team agrees)
Potential risks that will affect the outcome of the project:
-
- Unexpected changes or limitations in the Renterd API.
- Prioritization i.e I’m doing this on the side and I will probably have days where I can only spend an hour or less for it.
- QA - optional but I’d like to have a good quality assurance testing and that may or may not push the timeline.
Development Information
Will all of your project’s code be open-source?
Yes. Source code will be open as soon as it’s done.
Leave a link where code will be accessible for review.
Links.
Do you agree to submit monthly progress reports?
Yes
Contact info
Email: areyesonl@gmail.com
Any other preferred contact methods: me@areyesonl.com