Standard Grant: Lume Web 2024

Progress Report for March

What progress was made on your grant this month?

  • Did some preparation to enable scaling with the portal database and S5 KV databases
  • Added a price tracker module for sia to track the last X days from siacentral’s API and update the renterd pricing config, as well as run daily.
  • Internal refactoring with HTTP routing
  • Implemented cookie support for logins, mainly for the dashboard
  • Various misc bug fixes/refactoring
  • Implemented native bao verified streaming support support thanks to Add support for Bao chunk groups by lukechampine · Pull Request #19 · lukechampine/blake3 · GitHub.
  • Fixed seeking issues causing Vup streaming to not function correctly.
  • Helped redsolver identify bugs during testing with S5/Vup.
  • Created MVP demo, and launched https://account.testing.pinner.xyz

Summarize any problems that you ran into this month and how you’ll be solving them.

Parallel TUS uploads turned out to have some quirks to them. The behavior of CLI vs JS TUS showed some issues and required me to temporarily restrict the dashboard to upload in serial.

This will be worked on more in Q2 with some possible design/strategy changes for TUS uploads, but is potentially complex enough to need to be punted a quarter.

Links to repos worked on this month:

What will you be working on next?

  • CI/CD devops, workflow revamp and a migration to monorepo(s)
    • There are a lot of issues with the current workflows I use to develop, and a lot of friction, causing slow downs. Based on research, and looking at the Foundation for some inspiration, I will be changing the dev tools and workflows used (but avoiding SaaS’s), and ideally enabling more collaboration in the future.
  • Refactor the cron (task) system and how it is used, implement persistent task handling. This will be a critical system in the long term.
  • Handle any community feedback for the webapp.
  • Begin research on Q2 priorities and the best way to design and implement.

Q1 Milestones Report

Per the grant proposal here is the video demo demonstrating all but one milestone: https://youtube.com/watch?v=qMxoM1fQMJ4

Please be aware the demo voice over is AI generated

Any testing can be performed at https://account.testing.pinner.xyz

One slight issue hit: One of the validations is the registry. Due to existing known issues with compatibility between the Dart and Golang implementations of S5, where Vup has not yet been updated, it was not possible to demonstrate a visual usage of the registry. The core dart libraries have been updated though and a dart test case (sample script) demonstrating the network function can be provided to the foundation upon request.

1 Like

Great to see you launch the initial version of the S5 node and portal, looking forward to see you iterate on the features!

Seeing that you are revamping your workflow, I would like to suggest you use GitHub as your primary repository and mirror this to your Lume git server for backup. Although it is very cool and more decentralized to work on your own git server, GitHub is where the dev ecosystem lives and imo activity and development on GitHub is necessary for discovery and contribution.

With that, the subtitles in French are very great !

Thank googles AI then? :upside_down_face:

Thanks. I am researching and will consider what you have said.

Progress Report for April

What progress was made on your grant this month?

  • Migrated development back to GitHub for the short-medium term
  • Implemented initial version of a persistent cron system
  • Did research and planning for the quarter
  • Kicked off design priorities with team
  • Major progress made on portal sync system using hypercore P2P as a means of distributing metadata.

Summarize any problems that you ran into this month and how you’ll be solving them.

No problems this month

Links to repos worked on this month:

What will you be working on next?

  • Finish the sync system. This will require bundling the nodejs server into the golang portal, and require handling keeping renterd meta in sync with the portal.
  • Start on Q2 design → dev work which includes mobile support/design, better colors, and a theme system. More will come as needed this quarter.
  • Work on portal scaling (cluster) code and any other needed quality of life tasks.
  • Handle community feedback as needed.
  • Work on frontend tasks if time permits this month.
2 Likes

Hello @pcfreak30,

Thank you for your perfect progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Progress Report for May

What progress was made on your grant this month?

  • Mostly finished the sync system. More testing will be needed.
  • Finished Q2 design work with my team.
  • Implemented much of the needed cluster functions, though more work is still needed.
  • Started on rewrites of the S5 JS code base as new packages, one for encoding and one for metadata.
  • Started on a large refactor of the portal to modularize it. Inspiration has been taken from caddy and building with plugins.

Summarize any problems that you ran into this month and how you’ll be solving them.

No problems this month

Links to repos worked on this month:

What will you be working on next?

  • Finish portal refactor and get the build process working in the new design.
  • Finish cluster support, which includes on S5.
  • Implement the S5 filesystem in TS to use with the dashboard.
  • Implement the mobile dashboard UI with team.
  • Implement the theming system with team.
  • Add the bones of an admin panel.
  • Add the ability to manage certain settings on-the-fly in the UI.
  • Finish sync system, create a demo, and video of the portal sync per milestone requirements.
  • Implement improvements to S5 management inside the dashboard if time permits.
1 Like

Hello @pcfreak30

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Progress Report for June

What progress was made on your grant this month?

  • Finished MVP of sync plugin and quarterly video demo
  • Finished portal architecture re-design
    • xportal a fork of xcaddy can now be used to build any kind of portal
  • Implemented the mobile UI
  • Implemented a new theme color
  • The starting point of a portal settings editor for the admin UI has been created

Summarize any problems that you ran into this month and how you’ll be solving them.

  • Logistics limitations with the team did not allow as much app development to get done as hoped. Several tasks, including the admin, a navigation menu, and a theme switcher are still waiting to be completed. These are expected to be done early Q3.
  • S5 has been a blocking problem and so no work around it has been completed further. I am awaiting the specs for S5 (of which has been given another grant) to ensure the go port is up to date and improve on the S5 dashboard support.

Links to repos worked on this month:

The testing portal is online to view these updates: https://account.testing.pinner.xyz

What will you be working on next?

  • More cluster support, specifically around sharing configuration via etcd.
  • Ability to live edit settings and the admin ui to work with it.
  • Refactoring the dashboard webapp to prepare it for the tasks to come.
  • Researching IPFS, S5, billing, and support systems as needed.
  • Depending on priorities, start on one or more of the major Q3 focuses.

Q2 Milestones Report

Per the grant proposal here is the video demo for the quarter: https://www.youtube.com/watch?v=ec80A_QrmZI

Here is the GitHub repo with all guides and demo code: GitHub - LumeWeb/portal-sync-demo

And this is the fork of simple-seeder GitHub - LumeWeb/portal-sync-simple-seeder: Dead simple seeder with zero bugs

Due to the more abstract nature of this quarters priority, if anyone wishes to experiment on this, they should reach out since the infra on running these demos is more complex with multiple servers, and the portals had to be reset at-least 1 time to show different scenarios.

Hello @pcfreak30

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Hello @pcfreak30

This is a reminder that your progress report is due.

Please be sure to follow the Progress Report Template and submit your progress report by the 2nd of each month. Thanks!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Progress Report for July

What progress was made on your grant this month?

  • Started on a refactor of the dashboard webapp.
  • Made progress on some UX updates with team.
  • Theme support, including a theme switcher has been added to the webapp.
  • Started on work for the admin ui, including a very bare bones support for cron task management, with an admin portal plugin.
  • Active further internal refactoring on architecture of the portal while planning on the needs of both S5 and IPFS.
  • Started and got over half way done on the IPFS support.

Summarize any problems that you ran into this month and how you’ll be solving them.

N/A

Links to repos worked on this month:

(please note that much effort is in temporary branches right now and has not been put in develop yet)

What will you be working on next?

  • As the web refactor includes major app changes, continue that work as needed.
  • Finish the IPFS support and test.
  • Work on the admin UI and settings management.
  • Start on S5 updates if things are mature enough to do so.
  • Start on the billing support, though I would prefer to do so after all protocols are added in.
  • Handle other misc internal tasks.

Progress Report for August

What progress was made on your grant this month?

  • Did significant work on the webapp
    • Added an upload management system, powered by Uppy, and updated the UI
    • Got an initial support for IPFS working in both small files and TUS
    • Added the basics of an uploads queue manager
    • Added the UI for subscription management
  • Got the majority of IPFS completed. Testing pinning, serving on the network, and downloading data remain.
  • Got the majority of billing support completed. More testing will need to be done. Work is also needed on restricting uploads/downloads based on the quota system.
  • Refactored the backend to use a new data model for managing upload requests, and have every upload now always go to a queue.

Summarize any problems that you ran into this month and how you’ll be solving them.

N/A

Links to repos worked on this month:

What will you be working on next?

  • Finish billing support
  • Finish IPFS support
  • Get a support system integrated, probably with oauth
  • Re-create the file manager
  • Implement and/or update various account qualify-of-life functions
  • Getting the UI improved as needed
  • Implement API Keys
  • Create Q3 video demo per milestones

The admin UI/plugin and some other backend functions, and/or abstractions, including a few scaling/cluster related things will be punted to Q4. Updating and re-adding S5 support will be punted to Q4 as well based on the current timelines of that project, but there is a chance that could take longer.

1 Like

Hello @pcfreak30

Thank you for your perfect Progress Report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Progress Report for September

What progress was made on your grant this month?

  • Completed IPFS support, including tracking unixfs metadata provided via API for the client
  • Completed billing support. Includes:
    • Tracking quota usage of users
    • Limiting access if over usage
      • Every protocol plugin will need to be checking the users usage on its endpoints, this is not centralized in the core outside possible helpers
    • A very basic plans support (a db table that currently needs direct manual entry). This is used for the quota side, not the billing side.
    • Support for a free plan via config that requires no external systems
    • Support for paid plans that require external systems
    • Paid plans rely on Killbill.io for subscriptions (java based) and hyperswitch.io (rust/js) for payments. These systems are rather complex and documentation will be created over time as issues are found for running them in prod.
      • A lot of research went into finding FOSS billing systems to avoid vendor lockin and future proof merchant control for political and business risk. There are very very few options that are suitable and maintained/have funding. The rest are SasS.
      • Stripe is supported now via hyperswitch, as you can treat hyperswitch as a payment gateway in itself.
    • Got the webapp in an alpha state with:
      • IPFS service
      • Account management and security
      • Uploads management
      • 2FA
      • API keys
      • Dashboard statistics, removing the mock UI’s
    • The webapp will likely heavily evolve over time in architecture, as plugins will need to be supported in a dynamic way. For now, things are activated in a feature flag approach and detected in the webapp as a monolith from the /api/meta endpoint accessible from any domain of the portal server.
      • In the future I will be looking at using tech called module federation and creating the infra needed for this to work. However, the JS ecosystem is heavily in R&D mode and these things are just not ready yet :(.
      • So for now things are done the most pragmatic way until it can be more sophisticated.
    • The support plugin has been created and uses oAuth2 to SSO into freescout.net, a laravel-based PHP support system. This required buying a plugin that provides an end-user support portal, but as all plugins are AGPL, the plugin has been forked and is on GitHub now.
    • Lots of internal refactoring and internal changes have been made this month and more will be made over time in the future.

Summarize any problems that you ran into this month and how you’ll be solving them.

  • Only persistent issue has been IPFS DHT/discovery. After fixing the configuration logic for announcement addresses (the original mistake made), this worked earlier this quarter, but it has been flaky lately. Direct peer connections via multiaddresses work regardless (thus pinning works). This was also tested with fsd to compare against a known working node besides kubo, with no success (fsd didn’t work either). It is possible a mis-configuration exists somewhere, but time will need to be spent to find the cause.
    • Outside that IPFS is working as expected.

Links to repos worked on this month:

What will you be working on next?

  • Webapp Changes:
    • Lots of small bugs exist in the webapp that need to be fixed
    • UX fixes are needed to be done by my team
    • Community feedback is needed to polish the webapp UI further
  • Solve IPFS network discovery problems
  • Start on work with S5 support again and help @redsolver as needed on the typescript side
  • Start on planning & design for the portal website frontend
  • Work on further internal changes to the portal, if time permits this month, such as:
    • Restructuring how config files are stored
    • Live config editing system
    • Updating portal price tracker to use explored
    • Working on more cluster-related support

Q3 Milestones Report

Per the grant proposal here is the video demo for the quarter: https://youtu.be/iqm7K_m8FeU

The testing node is live at: https://account.testing.pinner.xyz/

A small update:

I have been able to figure out the problems with the IPFS DHT. Discovery of the portal was working properly, however the provider function of the IPFS node code had a bug in it around broadcasting new content that was recently uploaded.

With some other misc fixes/changes I was able to verify a file is accessible via a public gateway. Pinning data from a local IPFS desktop client though may still be fickle since that might have to use a relay node due to NAT or other network quirks. Edge cases around this will be worked on more as they come up, if possible.

Given this, focus will be to continue testing, and fixing webapp bugs, along with the other priorities I have already provided for October.

Thanks.

Progress Report for October

What progress was made on your grant this month?

  • Did bug fixes on dashboard webapp.
  • Started designing for the portal frontend.
  • Refactored the config system to create a more modular config file scheme.
  • Made progress on live config editing and the admin plugin.
  • Added RBAC permissions system in core via casbin library.
  • Replaced using etcd with redis for tus in cluster mode.
  • Worked on CI automations/workflows.
  • Did research on devops, akash, and looking for infra options.

Summarize any problems that you ran into this month and how you’ll be solving them.

N/A

Links to repos worked on this month:

What will you be working on next?

  • Devops code (terraform/tofu)
  • Replace price tracking subsystem with host tracker that uses hostscore to find the best hosts
  • Work on cluster support further while building out devops
  • More webapp bug fixes based on feedback
  • Start on S5 efforts (depends on redsolvers progress)

The admin plugin will be de-prioritized and efforts on it continue after other Q4 priorities are completed.

Hello @pcfreak30

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Progress Report for November

What progress was made on your grant this month?

  • Did heavy research on akash technology to figure out project needs.
    • Spent a significant amount of time onboarding and figuring out the legos of the system
    • Based on both technical requirements and bizdev discussion, ended up launching a provider on the network to operate Lume’s workloads.
    • Had to fork the terraform akash provider and will maintain that until changes can be upstreamed.
    • Forked the akash code code to work on some SSL support, but will be punting that until sometime in Q1 2025 likely.
    • Forked akash code code as well to enable some env vars to be available inside container.
  • Developed several docker images that will need continued work.
  • Worked on some more cluster related support in the portal.
  • Experimented with multiple “TACO” systems (CI systems for terraform/tofu) and will be creating a minimal setup for a soft launch.
  • Finished most design work for front of the portal.

Summarize any problems that you ran into this month and how you’ll be solving them.

N/A

Links to repos worked on this month:

What will you be working on next?

  • Get a test cluster running.
  • Get a minimal prod deployed.
  • Get the portal frontend site built.
  • Get the billing ready and deployed.
    • (There are potential compliance things to be tackled here :/)
  • Add further cluster support to the portal, specifically IPFS needs to have some cluster support for its databases added.
  • Create the monitoring configurations.
  • Create docs for running a private IPFS portal.
  • Complete a soft launch and find the bugs that will come :P

Important: Based on discussions with @redsolver, S5 support will be punted to Q1 2025. I do not feel things are ready yet to do another integration attempt. Additionally I have enough to do as-is and that is likely a 1-2 week effort easily.