Standard Grant: Lume Web 2024

Grants Approved
43

Thank googles AI then? :upside_down_face:

44

Thanks. I am researching and will consider what you have said.

45

Progress Report for April

What progress was made on your grant this month?

  • Migrated development back to GitHub for the short-medium term
  • Implemented initial version of a persistent cron system
  • Did research and planning for the quarter
  • Kicked off design priorities with team
  • Major progress made on portal sync system using hypercore P2P as a means of distributing metadata.

Summarize any problems that you ran into this month and how you’ll be solving them.

No problems this month

Links to repos worked on this month:

What will you be working on next?

  • Finish the sync system. This will require bundling the nodejs server into the golang portal, and require handling keeping renterd meta in sync with the portal.
  • Start on Q2 design → dev work which includes mobile support/design, better colors, and a theme system. More will come as needed this quarter.
  • Work on portal scaling (cluster) code and any other needed quality of life tasks.
  • Handle community feedback as needed.
  • Work on frontend tasks if time permits this month.
2 Likes
46

Hello @pcfreak30,

Thank you for your perfect progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

47

Progress Report for May

What progress was made on your grant this month?

  • Mostly finished the sync system. More testing will be needed.
  • Finished Q2 design work with my team.
  • Implemented much of the needed cluster functions, though more work is still needed.
  • Started on rewrites of the S5 JS code base as new packages, one for encoding and one for metadata.
  • Started on a large refactor of the portal to modularize it. Inspiration has been taken from caddy and building with plugins.

Summarize any problems that you ran into this month and how you’ll be solving them.

No problems this month

Links to repos worked on this month:

What will you be working on next?

  • Finish portal refactor and get the build process working in the new design.
  • Finish cluster support, which includes on S5.
  • Implement the S5 filesystem in TS to use with the dashboard.
  • Implement the mobile dashboard UI with team.
  • Implement the theming system with team.
  • Add the bones of an admin panel.
  • Add the ability to manage certain settings on-the-fly in the UI.
  • Finish sync system, create a demo, and video of the portal sync per milestone requirements.
  • Implement improvements to S5 management inside the dashboard if time permits.
1 Like
48

Hello @pcfreak30

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

49

Progress Report for June

What progress was made on your grant this month?

  • Finished MVP of sync plugin and quarterly video demo
  • Finished portal architecture re-design
    • xportal a fork of xcaddy can now be used to build any kind of portal
  • Implemented the mobile UI
  • Implemented a new theme color
  • The starting point of a portal settings editor for the admin UI has been created

Summarize any problems that you ran into this month and how you’ll be solving them.

  • Logistics limitations with the team did not allow as much app development to get done as hoped. Several tasks, including the admin, a navigation menu, and a theme switcher are still waiting to be completed. These are expected to be done early Q3.
  • S5 has been a blocking problem and so no work around it has been completed further. I am awaiting the specs for S5 (of which has been given another grant) to ensure the go port is up to date and improve on the S5 dashboard support.

Links to repos worked on this month:

The testing portal is online to view these updates: https://account.testing.pinner.xyz

What will you be working on next?

  • More cluster support, specifically around sharing configuration via etcd.
  • Ability to live edit settings and the admin ui to work with it.
  • Refactoring the dashboard webapp to prepare it for the tasks to come.
  • Researching IPFS, S5, billing, and support systems as needed.
  • Depending on priorities, start on one or more of the major Q3 focuses.

Q2 Milestones Report

Per the grant proposal here is the video demo for the quarter: https://www.youtube.com/watch?v=ec80A_QrmZI

Here is the GitHub repo with all guides and demo code: GitHub - LumeWeb/portal-sync-demo

And this is the fork of simple-seeder GitHub - LumeWeb/portal-sync-simple-seeder: Dead simple seeder with zero bugs

Due to the more abstract nature of this quarters priority, if anyone wishes to experiment on this, they should reach out since the infra on running these demos is more complex with multiple servers, and the portals had to be reset at-least 1 time to show different scenarios.

50

Hello @pcfreak30

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

51

Hello @pcfreak30

This is a reminder that your progress report is due.

Please be sure to follow the Progress Report Template and submit your progress report by the 2nd of each month. Thanks!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

52

Progress Report for July

What progress was made on your grant this month?

  • Started on a refactor of the dashboard webapp.
  • Made progress on some UX updates with team.
  • Theme support, including a theme switcher has been added to the webapp.
  • Started on work for the admin ui, including a very bare bones support for cron task management, with an admin portal plugin.
  • Active further internal refactoring on architecture of the portal while planning on the needs of both S5 and IPFS.
  • Started and got over half way done on the IPFS support.

Summarize any problems that you ran into this month and how you’ll be solving them.

N/A

Links to repos worked on this month:

(please note that much effort is in temporary branches right now and has not been put in develop yet)

What will you be working on next?

  • As the web refactor includes major app changes, continue that work as needed.
  • Finish the IPFS support and test.
  • Work on the admin UI and settings management.
  • Start on S5 updates if things are mature enough to do so.
  • Start on the billing support, though I would prefer to do so after all protocols are added in.
  • Handle other misc internal tasks.
53

Progress Report for August

What progress was made on your grant this month?

  • Did significant work on the webapp
    • Added an upload management system, powered by Uppy, and updated the UI
    • Got an initial support for IPFS working in both small files and TUS
    • Added the basics of an uploads queue manager
    • Added the UI for subscription management
  • Got the majority of IPFS completed. Testing pinning, serving on the network, and downloading data remain.
  • Got the majority of billing support completed. More testing will need to be done. Work is also needed on restricting uploads/downloads based on the quota system.
  • Refactored the backend to use a new data model for managing upload requests, and have every upload now always go to a queue.

Summarize any problems that you ran into this month and how you’ll be solving them.

N/A

Links to repos worked on this month:

What will you be working on next?

  • Finish billing support
  • Finish IPFS support
  • Get a support system integrated, probably with oauth
  • Re-create the file manager
  • Implement and/or update various account qualify-of-life functions
  • Getting the UI improved as needed
  • Implement API Keys
  • Create Q3 video demo per milestones

The admin UI/plugin and some other backend functions, and/or abstractions, including a few scaling/cluster related things will be punted to Q4. Updating and re-adding S5 support will be punted to Q4 as well based on the current timelines of that project, but there is a chance that could take longer.

1 Like
54

Hello @pcfreak30

Thank you for your perfect Progress Report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

55

Progress Report for September

What progress was made on your grant this month?

  • Completed IPFS support, including tracking unixfs metadata provided via API for the client
  • Completed billing support. Includes:
    • Tracking quota usage of users
    • Limiting access if over usage
      • Every protocol plugin will need to be checking the users usage on its endpoints, this is not centralized in the core outside possible helpers
    • A very basic plans support (a db table that currently needs direct manual entry). This is used for the quota side, not the billing side.
    • Support for a free plan via config that requires no external systems
    • Support for paid plans that require external systems
    • Paid plans rely on Killbill.io for subscriptions (java based) and hyperswitch.io (rust/js) for payments. These systems are rather complex and documentation will be created over time as issues are found for running them in prod.
      • A lot of research went into finding FOSS billing systems to avoid vendor lockin and future proof merchant control for political and business risk. There are very very few options that are suitable and maintained/have funding. The rest are SasS.
      • Stripe is supported now via hyperswitch, as you can treat hyperswitch as a payment gateway in itself.
    • Got the webapp in an alpha state with:
      • IPFS service
      • Account management and security
      • Uploads management
      • 2FA
      • API keys
      • Dashboard statistics, removing the mock UI’s
    • The webapp will likely heavily evolve over time in architecture, as plugins will need to be supported in a dynamic way. For now, things are activated in a feature flag approach and detected in the webapp as a monolith from the /api/meta endpoint accessible from any domain of the portal server.
      • In the future I will be looking at using tech called module federation and creating the infra needed for this to work. However, the JS ecosystem is heavily in R&D mode and these things are just not ready yet :(.
      • So for now things are done the most pragmatic way until it can be more sophisticated.
    • The support plugin has been created and uses oAuth2 to SSO into freescout.net, a laravel-based PHP support system. This required buying a plugin that provides an end-user support portal, but as all plugins are AGPL, the plugin has been forked and is on GitHub now.
    • Lots of internal refactoring and internal changes have been made this month and more will be made over time in the future.

Summarize any problems that you ran into this month and how you’ll be solving them.

  • Only persistent issue has been IPFS DHT/discovery. After fixing the configuration logic for announcement addresses (the original mistake made), this worked earlier this quarter, but it has been flaky lately. Direct peer connections via multiaddresses work regardless (thus pinning works). This was also tested with fsd to compare against a known working node besides kubo, with no success (fsd didn’t work either). It is possible a mis-configuration exists somewhere, but time will need to be spent to find the cause.
    • Outside that IPFS is working as expected.

Links to repos worked on this month:

What will you be working on next?

  • Webapp Changes:
    • Lots of small bugs exist in the webapp that need to be fixed
    • UX fixes are needed to be done by my team
    • Community feedback is needed to polish the webapp UI further
  • Solve IPFS network discovery problems
  • Start on work with S5 support again and help @redsolver as needed on the typescript side
  • Start on planning & design for the portal website frontend
  • Work on further internal changes to the portal, if time permits this month, such as:
    • Restructuring how config files are stored
    • Live config editing system
    • Updating portal price tracker to use explored
    • Working on more cluster-related support

Q3 Milestones Report

Per the grant proposal here is the video demo for the quarter: https://youtu.be/iqm7K_m8FeU

The testing node is live at: https://account.testing.pinner.xyz/

56

A small update:

I have been able to figure out the problems with the IPFS DHT. Discovery of the portal was working properly, however the provider function of the IPFS node code had a bug in it around broadcasting new content that was recently uploaded.

With some other misc fixes/changes I was able to verify a file is accessible via a public gateway. Pinning data from a local IPFS desktop client though may still be fickle since that might have to use a relay node due to NAT or other network quirks. Edge cases around this will be worked on more as they come up, if possible.

Given this, focus will be to continue testing, and fixing webapp bugs, along with the other priorities I have already provided for October.

Thanks.