Small Grant: Lume Web - IPFS Portal

State of Lume

Our Journey in a Nutshell

Our path to today has been a multi-year effort, broken down into three main phases:

  • Phase 1: Research & Reality Check (Grant 1)
    We started with broad experiments, trying to build a new “web3” browsing experience. This was a valuable reality check that showed us our initial vision was a long way off and needed a more focused approach.

  • Phase 2: The Pivot (Grant 2)
    Based on direct community feedback, we shifted gears to build practical, foundational tools for the ecosystem. This led to the first versions of S5 and IPFS support. While successful, the platform grew complex and we accumulated some tech debt.

  • Phase 3: Hardening the Core (This Year)
    This year was all about the “boring but necessary” work. We rewrote backend systems, stabilized our infrastructure, and built crucial tools like an abuse prevention plugin. We got our house in order so we could build on a rock-solid foundation.

Where We Are Now

That foundational work is done. The risky refactoring is behind us, the core portal is stable, and the plugin system works. With the engineering challenges solved, we’re finally ready to focus on what matters most: shipping a polished, user-friendly experience.

Introduction

Project Name: Lume Web - IPFS Portal

Name of the organization or individual submitting the proposal: Hammer Technologies LLC

Describe your project.

This project has the goal of getting a polished, working service for IPFS hosting.

Background

The Lume Portal has been around since 2024 and Lume as a whole received three grants already. The portal consists of several plugins (like IPFS, S5, abuse, billing, admin, dashboard), and the core system. These plugins are in various stages of maturity due to both time constraints and several past re-architecture efforts being required.

Effort has already been spent to rewrite the IPFS portal plugin (the golang side). The efforts of this grant request is a mix of frontend efforts, and working on any remaining issues with the go plugin. As it stands, the plugin could be used via swagger/API and the basics would work.

This grant proposal aims to get the dashboard and IPFS plugins ready for a zen testnet beta.

Current State of the App

The image above illustrates the current state and starting point for this grant’s work. It shows the core Lume application shell after a user has successfully logged in. The foundational backend elements are in place: a working Go backend, user account management (registration, login, 2FA).

As annotated in the image, the main content area is intentionally blank. The primary goal of this grant is to build out the two key feature plugins—the user Dashboard and the IPFS Support—which will populate this space and create a complete, user-facing product ready for beta testing.

Some may also ask the question Did you not already do IPFS?. Yes, I did in 2024 (Standard Grant: Lume Web 2024 - #55 by pcfreak30). But, as major internal changes to the portal happened, I rewrote the IPFS plugin very recently as stated before, and I completely re-did how the webapps work… Effort is required to finish the IPFS integration in the new version of the portal, as well as the user account area, and bring this to something polished thats more than a MVP that the community has been vocal about.

How does the projected outcome serve the Foundation’s mission of user-owned data?

This will enable IPFS support for uploading users data to the Sia network. For users, this will mean they can use any app in the IPFS ecosystem, or any app in the Sia ecosystem that uses IPFS as a datarail, and have a reliable pinning provider. They of course, can also use the portal’s webapp as well.

Two Sia grants are planning to integrate with Lume when viable, Small Grant: DecaNotes - Private decentralized notes and Standard Grant: SecureSphere Decentralized Vault. Both are expected to via the IPFS Pinner API (helia).

We cannot provide grants to residents of jurisdictions under increased FATF monitoring, those that have active OFAC sanctions, or those that fail our bank compliance tests. We also cannot provide grants if your payment bank account is located in those same locations. Please review the following list.

Are you a resident of any jurisdiction on that list? No

Will your payment bank account be located in any jurisdiction on that list? No

Grant Specifics

Amount of money requested and justification with a reasonable breakdown of expenses:

$10,000

  • $9,330 - 2 months Salary for Derrick Hammer
  • $670 - R&D Infrastructure

What are the goals of this small grant? Please provide a general timeline for completion.

  • Month 1: Rebuild the portal dashboard
    • The shell for all apps is in place, but the dashboard experience needs to be re-created and old react code from the legacy app ported where it makes sense to.
    • Tasks:
      • Ensuring all account functions work correctly
      • Ensuring a smooth UX
      • Remove any serious remaining bugs found in the process before we get to the upload components.
      • Get community feedback on the UI.
    • Validations for this milestone will include:
      • Can you register?
      • Can you login?
      • Can you reset your password?
      • Can you setup 2FA?
      • Can you update your profile info?
      • Can you upload an avatar?
      • Can you change your email?
      • Can you reset your password?
      • Can you manage API keys?
      • Is the app responsive on mobile and tablet devices?
        • A best effort basis will be made to catch all responsive design issues, but as this can be a very large matrix, it will be only reasonable to test the most popular devices and platforms. It is likely the community will continue to find small issues over time, but the goal is to get a polished UI.
  • Month 2:

The end goal of this timeline is to get a polished service, in a zen beta status (not fully launched), ready for the Sia community to be able to use. The portal will talk to a renterd instance in the Zen network until it is time for live demand.

Lume will also submit itself to pinning-service-compliance | This repo checks the compliance of IPFS Pinning Services against the pinning spec as a compliant software/service and to the IPFS desktop this year, but that will not be considered as a milestone under this grant.

Potential risks that will affect the outcome of the project:

  • Any unexpected issues with frontend development. I have done my best to proactively de-risk any frontend problems, but experience has shown me already you can’t predict unknown unknowns.
  • Any issues with the IPFS protocol that I may not easily control. Over half of the IPFS protocol code base is a direct port of fsd (thanks foundation :wink:), and so I expect any issues I run into, fsd would as well.
  • Any changes to renterd itself, however this is the least of my concerns since renterd is abstracted to a single class, uses the rest API (and has been pretty early on) and should be pretty easy to update.

Development Information

Will all of your project’s code be open-source?

Yes

Leave a link where code will be accessible for review.

Do you agree to submit monthly progress reports?

Yes

Contact info

Email:
[email protected]

Any other preferred contact methods:

1 Like

Thanks @pcfreak30 for the proposal I really like this idea IPFS support would open up a lot of possibilities. I can already see ways this could be useful for projects like SecureSphere where users need to interact with IPFS data while keeping Sia as the backend. Excited to see it progress!

What does it actually mean to make an “IPFS Portal” in this case? Are you creating an IPFS pinning service that stores the data on Sia and has a different API & Web-UI? Like how does this differ from just using an fsd node?

The FSD node actually DOESN’T implement the pinner API. That was something I was planning to bring up as an issue in the near future on GitHub, as that poses a barrier for anyone wanting something simpler and more basic then what Lume’s portal software is…

Also, do note that FSD is designed solely as a laser focused L2 of sorts, with no user account system of its own. So there are higher level product differences… even if I did copy much of the effort from the foundation…

To answer the end user differences, the biggest thing is the fact there is a webapp system in place to manage your uploaded data (pins). That is frankly a component I spent a lot more time on then I wanted to, and what will now enable the portal as a platform to support anything required on the frontend in the future.

This request is probably 70-80% effort on the webapp side, to rebuild things, and test, for it to be end user friendly. My prior efforts earlier this month were redirected from IPFS to resolving issues the foundation brought up with the abuse effort so I didn’t finish and verify the compliance, nor start on the dashboard effort. Given the fact I already had IPFS solved from 2024… most of the backend efforts was actually in the data modeling design and backend plumbing…

As is stands, 1 month here is mainly focused on the dashboard (a webapp plugin), and the other is on the upload aspects of the dashboard and IPFS as webapp plugin.

The pinner API is the ecosystem standard that the IPFS shipyard team, and before that… Protocol Labs themselves developed. It was for external providers, and helia, kubo, and IPFS desktop app all support it as a spec, IPFS Pinning Service API.

Technically speaking, and as stated before, about half of the IPFS protocol code was copied from fsd, and the medium term goal of Lume is still to get a live, paid, managed service (which I would like this year if I can).

So, now, in abstract, regardless of the next integration (and thus optimistically… demand) Lume brings to Sia, these two tasks are prerequisites to be completed for any integration next for Lume, and I am requesting the time I am so I can give them a dedicated focus.

I will be clear that I have plans for Lumes roadmap after IPFS as a grant, but as it has been requested, I am going in much smaller steps in terms of project management and community governance of grants.

Kudos.

Alright that makes sense. A couple more clarifiers.

How exactly does the Lume API work then? For every integration do you have to host an entirely new API on the same node? Like IPFS gets it’s pinning API, S5 gets it’s API, etc, etc? Then all of it in the end gets pointed at Sia on the backend?

How do you plan on dealing with IPFS leeching? I talked to this a good deal with Zach from filebase a while ago, the vast majority of IPFS bandwidth is from peering. As it is designed as a completely free content addressable network on the backend, bandwidth is burned real fast on anything remotely popular. Will the plugin make this viewable to the user? Will that be part of the billing model?

How exactly does the Lume API work then?

This is where it gets fun.

I know I have gotten a lot of heat for the direction my project has taken (most criticism is warranted)… But overall I am trying NOT TO define my own API’s. I stopped doing that after 2023. I think the best term is aggregator. I am building an aggregator for the ecosystem. I defined custom API’s for the account, admin, abuse, b/c I had to, but any time a HTTP proto exists for an ecosystem, I will just use that.

Every integration is a plugin that has a API and Protocol entity/class (in short… an interface). More recently there was an APIExtension entity added (so abuse could sainly tag onto the admin API).

All plugins are compiled into the portal as side effect imports technically _ "portal-plugin" which then register themselves (akin to a ES6 TS import that runs global code) . This was inspired by caddy. So where possible, the plugin embeds the protocol, though its certainly possible, and required for some in the future, like bluesky or iroh, to run a second server due to language barriers (no go impl).

Hosting wise, you must use wildcard DNS as the HTTP logic uses vhosts effectively to route everything. Its all an integrated framework to serve API’s and operate protocols based around user pins.

Also know from a technical POV, every protocol gets a dedicated bucket in renterd. That part is actually the more simpler aspects of the system.

Upfront, no integration will give a free open gateway over HTTP to users (social media is going to need to be handled carefully). Basically I am not repeating Skynet and that has been a real hard line ive taken since I started on the portal in 2024…

In the near future, a quota plugin (with a new simpler billing plugin with it), will enable the bean counting, and direct IPFS leeching will be supported, but the block availability is actually subject to a ready column in the db for a given block, which the blockstore checks… So the node side of the plugin will fail when bitswap WANT requests come in. That itself will trigger in the future when the quota on a block/pin runs out…

Additionally, the ipfs. subdomain is JWT protected. You either need to login, or get an API key JWT. There will be no freeloading off the API unless thats added as a setting/logic to the API code.

So, the HTTP is more restricted b/c the internet police are more OCD about rouge content on HTTP then the P2P ports, and because the internet is full of trolls, and a quota plugin, idea this year, will enforce limits.

Every integration will do similar things, with public access or gateways being heavily critical against internet freeloading.

I also have no plans to run a kubo or take the liability of running a Lume public gateway any time in the near future. That is a ops, business, and legal headache I don’t need.

A bit unrelated but… I actually don’t have DNSLink support back into the system yet, b/c that requires to have an approval process in place for domains, and because there is no ready protocol plugin yet… But, due to the lesson the community learned around you.hns.siasky.net or siasky.net/X …, and I seeing how Lume’s friends at eth.limo successfully handle subdomain requests, I am being very cautious about non-authorized access…

As for the billing model, I did this in the 1st billing plugin I made, and I think it still makes sense: the usage of each pin will get the cost socialized across all paying users of that pin at the time of access. But the billing is a discussion for another day :upside_down_face: .

So the high level goals become more about asking what web3 (crypto or pure P2P) or web2 protocols are supported, and less about What is Lumes API. Granted, in IPFS I do add some custom endpoints for uploading (POST and TUS), but I comply with the IPFS ecosystem spec as well.

Kudos.

Thanks for your proposal to The Sia Foundation Grants Program.

After review, the committee has decided to approve your proposal. Congratulations! They’re excited to see what you can accomplish with this grant.

We’ll reach out to your provided email address for onboarding. This shouldn’t take long unless your info has changed from last time, but you may still need to adjust your timelines.

Per request of the foundation, this is the grants effective timeline:

Milestone Due Date
Portal Dashboard 9/2/25
Upload UI, Queue/Operations UI, & File Manager UI 10/2/25

Hi @pcfreak30, this is a reminder that your progress report deadline is coming up on September 2nd.

We look forward to receiving it then!

August Progress Report

What progress was made on your grant this month?

Please summarize your progress in 3-5 sentences or bullet points:

  • Implemented account dashboard per milestones
  • Did work on UI libraries
  • Changes to various go libraries
  • Updated frontend plugin to work with current portal version
  • Deployed pinner.xyz. Milestone work can be reviewed and tested.

Summarize any problems that you ran into this month and how you’ll be solving them.

N/A

List repos worked on this month with links to PRs and relevant commits:

What will you be working on next?

Please summarize your development goals into a few sentences or bullet points:

  • Finish IPFS service and verify compliance
  • Build IPFS file manager, uploader UI, and queue UI.
  • Plan next priority for grant request
1 Like

Hello

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

I am posting a ~mid-month update to demostrate the current progress and that the grant is on track given I have made an overlapping grant request to meet the next grant meeting.

Video Demo:

Completed:

  • IPFS Compliance
  • REST API for querying ongoing operations
  • Upload Manager UI
  • Efforts with UI framework code to enable the upload UI

TODO:

  • Operations UI (a ui mock is ready)
  • File Manager (powered by IPFS pinner API which is now compliant, a ui mock is ready)
  • Pinning UI flow
  • UX sanity checks
  • Verifying IPFS content is accessible

It is expected that the rest of the UI efforts will completed on schedule given the hardest parts were IPFS compliance, getting a wizard form supported in the UI, and ensuring car processing + uploading worked correctly.

For anyone interested, The next grant request is at: Small Grant: Lume Web - LBRY Integration

September Progress Report

What progress was made on your grant this month?

Please summarize your progress in 3-5 sentences or bullet points:

  • Created operations UI and required API for it
  • Created upload manager UI that support both files and folders (it uses car uploading internally)
  • Created file manager ui and a dedicated API for it
  • Created pinning ui accessible at the file manager
  • Files can be downloaded, and unpinned
  • Files can be accessed after 5-10 min roughly (possibly less) at public gateways such as https://dweb.link/ipfs/X
  • IPFS API service is IPFS pinner compliant

**Detail tasks worked on this month per milestone with the appropriate Pull Request(s) links

I will provide full commit ranges for completeness for changed repos as well as selective PR’s per the table that holds some of the work, As I have followed a development process prior to the new grants guide. These PR’es cross-cut a lot of concerns so there is a lot of overlap in what PR has what.

Milestone Task Pull Request(s) Additional Notes
Upload UI Create file upload UI #499 #502 #506
Operations UI Create UI that shows the current background operations for an account #507 #508 #206
File Manager UI Create file manager ui to support small and large uploads #510 2242306
IPFS pinning compliance Ensure IPFS pinner API interface is compliant #108

Summarize any problems that you ran into this month and how you’ll be solving them.

Due to complexity, I had to shift to having a trust-based file manager backend thats built ontop of all existing data query code infra. That means the file manager has its own API and a SQL table that has user file listings extracted. I would like to have this be trustless, but that is a very client heavy task with performance and UX considerations. I have not removed any code infra preventing it (the previous iteration from 2024 had a blocks meta api serving unixfs data), so this will just be a future effort to evaluate.

Nothing to solve for this now… Just a technical shift had to be made out of practicality…

What will you be working on next month?

Taking in feedback and fixing any issues that come up, as well as any UI changes that may be needed per feedback.

Link to an easy to test version or a demo video.

https://pinner.xyz/

Developer Instructions for testing compliance

Reference GitHub - ipfs-shipyard/pinning-service-compliance: This repo checks the compliance of IPFS Pinning Services against the pinning spec

  • Go to https://account.pinner.xyz/
  • Get an API key
  • run npx @ipfs-shipyard/pinning-service-compliance -s "https://ipfs.pinner.xyz/api" <auth_token> Replace auth token with JWT.

Provide an overall summary of everything achieved during this grant.

  • Did significant work around UI components per the needs of the milestones.
  • Implemented a dashboard UI with basic account controls and auth.
  • Created a file manager UI with support for filenames in folders (IPFS protocol design limitations). Supporting naming on individual files is a todo.
  • Created an upload system supporting folders and files, and large data.
  • Enabled pinning of data and accessing from a known gateway.
  • Created an API that is IPFS compliant.

What are you most proud of about your work on this grant?

  1. This is the 1st platform (that im aware of) that is anything close to major pinning services, and is FOSS.

  2. That there is a launched service for testing and an overall system design that I am proud of (from all accumulated work). The community can finally see and get an answer to the question asked over the years: What is Lume :upside_down_face:.

1 Like

Hello @pcfreak30,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Hi @pcfreak30 - here is the feedback from your technical review with a requested edit before we can mark this grant as complete:

  • There isn’t the ability to download larger files (~90MB, ~140MB). When attempted, instead of the expected file the reviewer received a 6 or so KB file containing random data.

We’ve determined this inability to download a larger file falls a short of providing a polished service, which was the end goal, and so would like this resolved before this grant can be marked ‘complete.’

Hello,

First, sorry for this bug. I got download approaches a bit mixed up and was trying to treat the server as who would serve the whole file, but I never copied the unixfs gateway logic fsd has for that. I was also trying to prevent any download buffering on the client side.

Here are the two primary PR’es that fixes this.

Right now, be aware that the 1st time download for a large file can take 10-30 seconds to trigger the file dialog. After, it is stored in browser (via helia indexddb block storage).

A future improvement could be a dialog progress bar for this, but it functions correctly now.

Thanks.

1 Like

To keep a breadcrumb trail for anyone following along with Lume, the next step of the journey is at Small Grant: Lume Web - LBRY Integration (Revised), and has been approved at the time of writing.

Kudos.