Standard Grant: Lume Web 2025

Quick Update: Abuse System MVP & Framework Progress

This is not my monthly report for June

Here’s a short video demoing a very rough MVP for the admin side of the abuse system.

The good news is that I’ve gotten unit tests passing for both the DB and API code, all running on the testing framework I built. The core web framework also has its unit tests from May.

On the tech side, two key things:

  1. DB Testing: I learned a hard lesson from a previous side quest: Don’t try to mock the DB. The testing framework now uses temporary SQLite DBs, runs all migrations, and tests against a real structure. It just works.
  2. UI Rewrite: In the process of building this, I rewrote the layout for the admin/dashboard UI (the shared GeneralLayout component). This includes a new menu and profile menu UI.

All major webapp framework challenges are now largely solved. The tooling will likely evolve to a rolldown plugin for module federation, but the path forward is clear.

While there’s a lot more that could be done here, and the TS codebase could use some love, these are low-priority tasks against everything else required this year.

I will be focusing next on:

  • Doing more internal refactoring on subsystems like events, config, cron, db, and possibly more. I want to get to something I will be more happy with before I view it as as final design for a while (as many plugins will depend on it)
  • Creating an abstraction so I can more rapidly create plugins and handle the most common actions (upload, pin, dl). This may include adding to the helper libraries.
  • Get the IPFS plugin using this abstraction and get full unit testing on it, ideally also get it in compliance with the IPFS pinner spec test suite.
  • Get the portal dashboard ported over to the new framework.

I have also found that for the atproto integration, community tooling has been made in several languages and there is both a golang tool at indigo/cmd/goat at fa0e9d0066c10c010f28053a4bd0fa49337a6562 · bluesky-social/indigo · GitHub and a rust tool at GitHub - NorthskySocial/pds-migration: Backend Application for user PDS migration. So leveraging these will largely solve the required backend code for migrating user accounts over, and that then becomes more of a question of the a webapp plugin for a wizard, etc.

I may or may not get to any significant progress on atproto this month. I will be largely focusing on getting IPFS back online and usable again.

That said, I do expect to get my milestones back on track based on the effort done already and what I plan on doing to improve the speed of upcoming integrations.

If the committee has any questions or concerns about my milestones, please let me know.

Progress Report for June

What progress was made on your grant this month?

  • Created a MVP of the admin side of the abuse.
    • Please note that I am considering the abuse work done for now and will revisit it once priorities deem it important to do so.
  • Redesigned and rewrote the config manager subsystem
  • Redesigned and rewrote the event bus subsystem
  • Moved from dbmate to goose for DB migrations
  • Added full unit testing and integration testing to core portal services. Majority of tests are passing, with low priority issues to be handled at a later date.
  • Started on re-design of IPFS plugin, especially with the data model.

Summarize any problems that you ran into this month and how you’ll be solving them.

Please summarize your issues into a few sentences or bullet points:

N/A

List repos worked on this month with links to PRs and relevant commits:

What will you be working on next?

Please summarize your development goals into a few sentences or bullet points:

  • IPFS plugin
  • User dashboard webapp
  • Getting lumeportal.com ready for test users
  • Nostr integration

Grant Timeline & Q3 Focus

This section details significant shifts in our development timeline and priorities for Q3, driven by recent challenges and strategic considerations.

Current State & Adjustments:

  • Delays: Grant progress has been heavily impacted by the foundational rebuild of the webapp portal, which required creating a new architecture.
  • Abuse Plugin: To prevent further derailing the grant and keep core priorities on track, the abuse plugin MVP is being considered completed and will be evolved when required.
  • atproto (Bluesky) Punted: Due to significant project and engineering risks, coupled with high complexity and unknowns, atproto will be the last integration tackled in Q3.
  • Billing: While not part of my milestones, I am not expecting to be doing any sales this year as that requires a few weeks of work that I likely no longer have a buffer for.

Improving Development Velocity:

Despite the initial delays, we’re building momentum:

  • Stable Portal Framework: I now have a robust and stable Golang-side portal framework, including a powerful testing library that has already delivered massive gains in efficiency. This means portal-side development should take less time moving forward.
  • Helper Library: I’ll be creating a helper Go library during the IPFS plugin rework, which will significantly speed up the development of subsequent integrations.

General Development Observations:

Through this process, I’ve consistently observed three key patterns:

  • Webapp Development Challenges: Webapp development often takes longer due to immature and evolving tooling that lacks features like Hot Module Reload (HMR), slowing down iteration cycles.
  • Protocol Integration Core: The majority of integration effort lies in enabling the portal to communicate with a new protocol and completing any necessary database modeling. Once that’s done, the rest of the work is far more standardized.
  • DevOps Overhead: DevOps efforts requiring new infrastructure components can consume up to 10x the time of typical R&D. Improving this area requires its own dedicated research.

Revised Q3 Integration Plan:

My broad aim is to complete each integration within 1-2 weeks, though some may extend to 3. atproto is currently estimated to require the most time. The current priority order is:

  • IPFS (Target: Mid-July):
    • Will include the dashboard, file manager, and API keys.
    • Will be compliant with the pinner spec, allowing use with IPFS desktop via an API key.
  • Nostr (Estimate: 1-2 weeks):
    • Based on NIP-96 and NIP-98 (if thats deemed needed), this is the simplest integration as it’s fully HTTP-based.
  • LBRY (Estimate: 1-3 weeks):
    • Research indicates a need to extract and adapt code from the reflector.go project to create a new library for network communication. After that, development should be similar to IPFS/S5.
  • atproto (Bluesky) (Estimate: 3-5 weeks):
    • Highest Risk: This integration currently requires running a separate server, as a feasible Go port isn’t possible within a reasonable timeframe (even if the grant timeline were on track :cry:).
    • Operational Complexities: The TypeScript reference implementation uses SQLite databases per account, which poses significant operational challenges in a cluster environment.
    • rsky Option: As previously mentioned, rsky is a potential Rust-based alternative, but it’s still in active development.
    • Research Focus: This quarter will require substantial research to determine the optimal design and infrastructure decisions for this plugin.

Q3 Strategic Shift: Delivering Tangible Results

Responding to community feedback and looking ahead to 2025 grant obligations, I’m restructuring Q3 to focus on tangible progress that the Sia community can actively use. While the grant to date has centered on compliance and foundational efforts, Lume will be delivering concrete, usable results this quarter.

S5 Integration on Hold:

The one exception to delivering “most” integrations is S5, which is being put on hold. This was not part of the milestones. However, Due to significant, ongoing changes to the S5 specification, I cannot risk spending 1-2+ weeks (or more) porting s5.js into a Go version only to have that effort rendered useless by a “rug pull” from a rapidly evolving spec. While supporting Vup is a very large, long-term priority for Lume, it’s currently too risky to build on until the specification stabilizes and provides a clear, dependable foundation.

1 Like

hey @pcfreak30 I’m trying to learn more about your LumeWeb. On my adventures I have ran into a few things I’m hoping you can help address:


1) When I click “Download Extension” on the landing page of the website https://lumeweb.com

It takes me to this unrelated website:
https://drivelatino.com/2020/12/nissan-armada-2021-ahora-con-mayor-potencia-seguridad-y-tecnologia.html

I assumed this may have been a hosting provider issue, but it seems to have been like this for a few months now.

Could you share a valid link to download the extension?


2) I can’t seem to upload any files on pinner.xyz, even under my storage limit of 999b.
Do I need to perform any additional configuration on my side to start using it?

Screenshot 2025-07-05 085053

The link to connect on discord https://discord.lumeweb.com/ is also broken.


3) How do I increase my monthly storage limit of 999b or pay for any of the services until web3portal.com becomes available?


4) I enabled 2FA on my account, but when I go to sign in it says I need to enter my code, yet there’s nowhere to enter it. After closing the tab and reopening in a new window, I was eventually prompted for the OTP.
I think I’ve tracked the issue down to the login URL param /login?to=%2Fuploads being present. When it’s not present, I’m correctly redirected and prompted for OTP.


5) How do I access the Lume web browser like in this video?
https://www.youtube.com/watch?v=WkRJ4o1MyYI


6) Are the core portal features mentioned in the 2025 proposal available now on pinner.xyz?
(RBAC, storage/renter management, configuration management, etc.)


7) I was an early adopter of Skynet and built apps in that ecosystem, but often ran into data consistency issues (ie writes disappearing and suddenly reappearing).
Which parts of Skynet have been forked or reused for LumeWeb specifically, and has anything been done to address data consistency? Have you experienced any data consistency issues with LumeWeb?


8) I’ve noticed a concerning pattern over the last several years:

Mid–2022:
“Lume had to be nearly completely rebuilt over the last six months”
– Oct 2022 Grant Proposal

Late 2023:
“I see an opportunity to rewrite many internals while keeping the same high-level API”
– 2024 Grant Discussion

Late 2023:
“The portal: Needs a large refactor/rewrite to support many protocols”
– 2024 Grant Discussion

Late 2023:
“The relay: It needs a rewrite… ideally use a better JS runtime”
– 2024 Grant Discussion

November 2024:
“But the current JavaScript P2P code needs to be rewritten in Golang“
– Precursor to 2025 Grant Proposal

February 2024:
“Completely redesigned our protocol and storage architecture”
– 2025 Grant Proposal

February 2025:
“I am intending to do another refactor of the webapp code base and create a frontend framework/foundation with module federation”
– 2025 Grant Proposal

March 2025:
“Needed to recreate all infra”
– 2025 Grant Proposal

This recurring pattern of architectural changes, redo’s, and rewrites each year raises a few questions for me:

  • What lessons were learned from the past reworks that necessitated another overhaul in 2025?

  • Is there a stable, long-term technical direction the project is working toward, or are the rewrites symptomatic of shifting priorities or unclear planning?

  • Could this pattern of reworking indicate foundational design issues, and how is this being addressed going forward?


9) You mentioned the frontend was built using Refine.dev, Tailwind CSS, and React.
However, from my experience on the site (as shared above), I encountered multiple UX bugs (2FA loop, broken upload functionality, and download redirects).
Is there a QA plan in place for frontend usability and accessibility, especially with plans to launch a paid service?


10) A core part of the proposal discusses the “Sia Layer 2” functionality and integration with renter management, billing, etc.
Could you clarify which parts of that are currently live and usable today vs. still in development?


11) You mentioned:

“The current JavaScript P2P code needs to be rewritten in Golang”

Has any of that rewrite started? If not, do you anticipate any blockers or shifts in approach? What is the timeline and resource allocation for that effort? Why must it be rewritten?


12) The $244,000 in cumulative funding to date is substantial and reflects a multi-year commitment from the Foundation.
Could you share a breakdown or retrospective on what the community has received in terms of:

  • Stable production grade tools
  • Actual usage or adoption stats
  • Open source repos with tagged releases

13) Are there any internal metrics or KPIs you’re using to measure success and progress?
Something like upload volume, daily/monthly active users, or network bandwidth consumed?


14) Despite clear feedback to narrow the focus of this project

“Lume’s first year of funding was largely you working with dozens of different technology tools and networks and very little direct interaction with Sia software and APIs, that’s totally fine, but now I would like to see any follow up Lume grant present:”

  • “A drastically narrower scope”
  • “A drastically clearer focus on Sia”
    @parox, November 2023, Lume Web 2024 Grant Discussion

The 2024 plan still spanned a broad set of infrastructure layers and technologies, many of which are not Sia exclusive:

  • Portal rewrite
  • Relay rewrite
  • P2P redesign in Golang
  • Kernel refactor
  • web3.news
  • Cloud hosting
  • Billing system
  • Dashboard
  • Terraform + DevOps
  • Bun.js migration
  • Module federation

All in all, my main concern is there’s this repeated narratives like:

  • “All software is MVP that needs more iterations.”
  • “Much work had to be rewritten or rebuilt.”
  • “Portal is in early beta.”
  • “Billing plugin not deployed.”
  • “Admin tools incomplete.”
  • “S5 plugin needs rewrite.”

And after three years and nearly a quarter million dollars, most components are still in alpha/beta or pending rewrite. The rate of rewrites vs production delivery appears disproportionate to the funding spent. You are one of the most active community members in Discord and on these forums and I can’t help but feel like you should be spending less time with the community and more time on completing this project.


I realize this is a lot to digest but so is the body of work, rewrites, and discussion around LumeWeb over the past three years. My intention here isnt to attack, but seek clarity and accountability as someone who cares deeply about the success of the Sia ecosystem

This is a highly ambitious project and you have been highly communicative but at this stage I think its reasonable for the community to ask for a more stable path forward, clearer deliverables, and better alignment between funding and outcomes. I am personally exhausted from reviewing your material and wading through all the technical jargon while trying to keep a focus on the meat and potatoes of LumeWeb

Looking forward to your responses and hoping these questions help sharpen focus and direction moving ahead. Also I recommending pinning this video to your grants https://www.youtube.com/watch?v=qMxoM1fQMJ4

Best regards
Jacob

2 Likes

Abuse Milestone Update

Per the request of the foundation I have fixed outstanding issues with the abuse plugin.

Based on discussions with the foundation, this yearly Lume grant has been closed and efforts for Lume moving forward are being transitioned to smaller grant requests. I intend to do feature/outcome focused grant requests moving forward that are more tangible for both the foundation and community to understand and see value from. Many important things were accomplished in this grant, and we are moving forward.

We will be starting with one for IPFS, Small Grant: Lume Web - IPFS Portal.

Kudos.