Hello @pcfreak30,
Thank you for your perfect progress report!
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
Hello @pcfreak30
Thank you for your progress report!
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
Progress Report for June
What progress was made on your grant this month?
- Finished MVP of sync plugin and quarterly video demo
- Finished portal architecture re-design
xportal a fork of xcaddy can now be used to build any kind of portal
- Implemented the mobile UI
- Implemented a new theme color
- The starting point of a portal settings editor for the admin UI has been created
Summarize any problems that you ran into this month and how you’ll be solving them.
- Logistics limitations with the team did not allow as much app development to get done as hoped. Several tasks, including the admin, a navigation menu, and a theme switcher are still waiting to be completed. These are expected to be done early Q3.
- S5 has been a blocking problem and so no work around it has been completed further. I am awaiting the specs for S5 (of which has been given another grant) to ensure the go port is up to date and improve on the S5 dashboard support.
Links to repos worked on this month:
The testing portal is online to view these updates: https://account.testing.pinner.xyz
What will you be working on next?
- More cluster support, specifically around sharing configuration via etcd.
- Ability to live edit settings and the admin ui to work with it.
- Refactoring the dashboard webapp to prepare it for the tasks to come.
- Researching IPFS, S5, billing, and support systems as needed.
- Depending on priorities, start on one or more of the major Q3 focuses.
Q2 Milestones Report
Per the grant proposal here is the video demo for the quarter: https://www.youtube.com/watch?v=ec80A_QrmZI
Here is the GitHub repo with all guides and demo code: GitHub - LumeWeb/portal-sync-demo
And this is the fork of simple-seeder GitHub - LumeWeb/portal-sync-simple-seeder: Dead simple seeder with zero bugs
Due to the more abstract nature of this quarters priority, if anyone wishes to experiment on this, they should reach out since the infra on running these demos is more complex with multiple servers, and the portals had to be reset at-least 1 time to show different scenarios.
Hello @pcfreak30
Thank you for your progress report!
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
Hello @pcfreak30
This is a reminder that your progress report is due.
Please be sure to follow the Progress Report Template and submit your progress report by the 2nd of each month. Thanks!
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
Progress Report for July
What progress was made on your grant this month?
- Started on a refactor of the dashboard webapp.
- Made progress on some UX updates with team.
- Theme support, including a theme switcher has been added to the webapp.
- Started on work for the admin ui, including a very bare bones support for cron task management, with an admin portal plugin.
- Active further internal refactoring on architecture of the portal while planning on the needs of both S5 and IPFS.
- Started and got over half way done on the IPFS support.
Summarize any problems that you ran into this month and how you’ll be solving them.
N/A
Links to repos worked on this month:
(please note that much effort is in temporary branches right now and has not been put in develop yet)
What will you be working on next?
- As the web refactor includes major app changes, continue that work as needed.
- Finish the IPFS support and test.
- Work on the admin UI and settings management.
- Start on S5 updates if things are mature enough to do so.
- Start on the billing support, though I would prefer to do so after all protocols are added in.
- Handle other misc internal tasks.
Progress Report for August
What progress was made on your grant this month?
- Did significant work on the webapp
- Added an upload management system, powered by
Uppy, and updated the UI
- Got an initial support for IPFS working in both small files and TUS
- Added the basics of an uploads queue manager
- Added the UI for subscription management
- Got the majority of IPFS completed. Testing pinning, serving on the network, and downloading data remain.
- Got the majority of billing support completed. More testing will need to be done. Work is also needed on restricting uploads/downloads based on the quota system.
- Refactored the backend to use a new data model for managing upload requests, and have every upload now always go to a queue.
Summarize any problems that you ran into this month and how you’ll be solving them.
N/A
Links to repos worked on this month:
What will you be working on next?
- Finish billing support
- Finish IPFS support
- Get a support system integrated, probably with oauth
- Re-create the file manager
- Implement and/or update various account qualify-of-life functions
- Getting the UI improved as needed
- Implement API Keys
- Create Q3 video demo per milestones
The admin UI/plugin and some other backend functions, and/or abstractions, including a few scaling/cluster related things will be punted to Q4. Updating and re-adding S5 support will be punted to Q4 as well based on the current timelines of that project, but there is a chance that could take longer.
1 Like
Hello @pcfreak30
Thank you for your perfect Progress Report!
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
Progress Report for September
What progress was made on your grant this month?
- Completed IPFS support, including tracking unixfs metadata provided via API for the client
- Completed billing support. Includes:
- Tracking quota usage of users
- Limiting access if over usage
- Every protocol plugin will need to be checking the users usage on its endpoints, this is not centralized in the core outside possible helpers
- A very basic plans support (a db table that currently needs direct manual entry). This is used for the quota side, not the billing side.
- Support for a free plan via config that requires no external systems
- Support for paid plans that require external systems
- Paid plans rely on Killbill.io for subscriptions (java based) and hyperswitch.io (rust/js) for payments. These systems are rather complex and documentation will be created over time as issues are found for running them in prod.
- A lot of research went into finding FOSS billing systems to avoid vendor lockin and future proof merchant control for political and business risk. There are very very few options that are suitable and maintained/have funding. The rest are SasS.
- Stripe is supported now via hyperswitch, as you can treat hyperswitch as a payment gateway in itself.
- Got the webapp in an alpha state with:
- IPFS service
- Account management and security
- Uploads management
- 2FA
- API keys
- Dashboard statistics, removing the mock UI’s
- The webapp will likely heavily evolve over time in architecture, as plugins will need to be supported in a dynamic way. For now, things are activated in a
feature flag approach and detected in the webapp as a monolith from the /api/meta endpoint accessible from any domain of the portal server.
- In the future I will be looking at using tech called
module federation and creating the infra needed for this to work. However, the JS ecosystem is heavily in R&D mode and these things are just not ready yet :(.
- So for now things are done the most pragmatic way until it can be more sophisticated.
- The support plugin has been created and uses oAuth2 to SSO into freescout.net, a laravel-based PHP support system. This required buying a plugin that provides an end-user support portal, but as all plugins are AGPL, the plugin has been forked and is on GitHub now.
- Lots of internal refactoring and internal changes have been made this month and more will be made over time in the future.
Summarize any problems that you ran into this month and how you’ll be solving them.
- Only persistent issue has been IPFS DHT/discovery. After fixing the configuration logic for announcement addresses (the original mistake made), this worked earlier this quarter, but it has been flaky lately. Direct peer connections via multiaddresses work regardless (thus pinning works). This was also tested with
fsd to compare against a known working node besides kubo, with no success (fsd didn’t work either). It is possible a mis-configuration exists somewhere, but time will need to be spent to find the cause.
- Outside that IPFS is working as expected.
Links to repos worked on this month:
What will you be working on next?
- Webapp Changes:
- Lots of small bugs exist in the webapp that need to be fixed
- UX fixes are needed to be done by my team
- Community feedback is needed to polish the webapp UI further
- Solve IPFS network discovery problems
- Start on work with S5 support again and help @redsolver as needed on the typescript side
- Start on planning & design for the portal website frontend
- Work on further internal changes to the portal, if time permits this month, such as:
- Restructuring how config files are stored
- Live config editing system
- Updating portal price tracker to use explored
- Working on more cluster-related support
Q3 Milestones Report
Per the grant proposal here is the video demo for the quarter: https://youtu.be/iqm7K_m8FeU
The testing node is live at: https://account.testing.pinner.xyz/
A small update:
I have been able to figure out the problems with the IPFS DHT. Discovery of the portal was working properly, however the provider function of the IPFS node code had a bug in it around broadcasting new content that was recently uploaded.
With some other misc fixes/changes I was able to verify a file is accessible via a public gateway. Pinning data from a local IPFS desktop client though may still be fickle since that might have to use a relay node due to NAT or other network quirks. Edge cases around this will be worked on more as they come up, if possible.
Given this, focus will be to continue testing, and fixing webapp bugs, along with the other priorities I have already provided for October.
Thanks.
Progress Report for October
What progress was made on your grant this month?
- Did bug fixes on dashboard webapp.
- Started designing for the portal frontend.
- Refactored the config system to create a more modular config file scheme.
- Made progress on live config editing and the admin plugin.
- Added RBAC permissions system in core via casbin library.
- Replaced using etcd with redis for tus in cluster mode.
- Worked on CI automations/workflows.
- Did research on devops, akash, and looking for infra options.
Summarize any problems that you ran into this month and how you’ll be solving them.
N/A
Links to repos worked on this month:
What will you be working on next?
- Devops code (terraform/tofu)
- Replace price tracking subsystem with host tracker that uses hostscore to find the best hosts
- Work on cluster support further while building out devops
- More webapp bug fixes based on feedback
- Start on S5 efforts (depends on redsolvers progress)
The admin plugin will be de-prioritized and efforts on it continue after other Q4 priorities are completed.
Hello @pcfreak30
Thank you for your progress report!
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
Progress Report for November
What progress was made on your grant this month?
- Did heavy research on akash technology to figure out project needs.
- Spent a significant amount of time onboarding and figuring out the legos of the system
- Based on both technical requirements and bizdev discussion, ended up launching a provider on the network to operate Lume’s workloads.
- Had to fork the terraform akash provider and will maintain that until changes can be upstreamed.
- Forked the akash code code to work on some SSL support, but will be punting that until sometime in Q1 2025 likely.
- Forked akash code code as well to enable some env vars to be available inside container.
- Developed several docker images that will need continued work.
- Worked on some more cluster related support in the portal.
- Experimented with multiple “TACO” systems (CI systems for terraform/tofu) and will be creating a minimal setup for a soft launch.
- Finished most design work for front of the portal.
Summarize any problems that you ran into this month and how you’ll be solving them.
N/A
Links to repos worked on this month:
What will you be working on next?
- Get a test cluster running.
- Get a minimal prod deployed.
- Get the portal frontend site built.
- Get the billing ready and deployed.
- (There are potential compliance things to be tackled here :/)
- Add further cluster support to the portal, specifically IPFS needs to have some cluster support for its databases added.
- Create the monitoring configurations.
- Create docs for running a private IPFS portal.
- Complete a soft launch and find the bugs that will come :P
Important: Based on discussions with @redsolver, S5 support will be punted to Q1 2025. I do not feel things are ready yet to do another integration attempt. Additionally I have enough to do as-is and that is likely a 1-2 week effort easily.
Hello,
Thank you for your progress report!
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
Progress Report for December
What progress was made on your grant this month?
- Spent significant time with akash, testing components, and developing terraform code. This includes handling a lot of gotchas in various parts of the systems
- Worked further on docker images
- Got website design converted and implemented plugin
portal-plugin-frontend. Also created a WIP means to have private portals more easily customize the frontend site by forking a GitHub template repo.
- Implemented monitoring systems for basic system resources, mysql, and redis. This includes grafana dashboards.
- Did further work around CI/deployment (TACO) with digger.
- Deployed a production system to pinner.xyz
- Launched a new docs site at https://docs.lumeweb.com with a portal setup guide.
Summarize any problems that you ran into this month and how you’ll be solving them.
So, unfortunately, I have hit a few problems/catches:
- I estimate I burnt 1-2 weeks R&D time on trying to create custom database cluster system for MySQL, which was powered by ProxySQL and etcd. While I kept getting what seemed close, things kept breaking, and the code was getting unwieldy, as it was in bash. This is something I plan to continue based on growth of the service (and will start from 0 with what I learned). However, I took an executive decision to completely punt using a cluster for the DB mid-December based on a gut feeling.
The domino effect is that I am having to also punt the billing and work on it next this month (In January), as it effectively ate the time I would have used to get the devops for that developed…
I am honestly not happy about this delay as I did want to have everything ready as expected for EOY, but I will roll with the punches so-to-speak and get everything done.
- I had more logistics issues with my freelancers and I had to hire a backup developer to port the frontend site to stay ontrack. I will post the design file for that in Lumes design git repo soon.
- I found out renterd’s UI does not yet support cluster mode for basic operations, and have had to temporarily cheat in the terraform code so it actually treats as a solo (normal) renter. I will follow core development to enable full cluster support when thats ready.
- The renter instances in prod system are consensus syncing rather slowly? so it might be an extra day or two before the portal can be used. At the time of writing they are at roughly 30 and 40% within half a day.
- For security reasons in operations, I require SSL to be enabled on the mysql (percona) databases. However as they are self signed, renterd does not yet support that. I opened an issue at MySQL DB SSL support · Issue #1752 · SiaFoundation/renterd · GitHub, but until thats supported, I am having to maintain a fork of renterd with a 1 line patch as part of the docker image GitHub - LumeWeb/akash-renterd.
- While renterd supports prometheus data in the
dev branch, it has been put on the v2 milestone. As such I have not gone and tried to develop any monitoring code for it as I assume things are not ready. I will go back and implement this once its clear renterd v2 can be used in prod.
Overall I have made the best effort I could to ensure that the portal system and related tasks are online per my milestones and will be playing catch up to get everything handled in January.
Links to repos worked on this month:
What will you be working on next?
- Working out any kinks in the systems of
pinner.xyz
- Get billing systems online for
pinner.xyz
- Deploy community portal at
web3portal.com
- Do further akash/monitoring/devops development as needed as there are rough patches that will need to be solved in the akash ecosystem.
- Focus back on frontend development with the website and dashboard. The website could get some improvements with talking to the portal API’s, the dashboard has a backlog now of UX problems. Additionally, I am going to have to use
module federation in the dashboard and eventually the admin plugin (which got sent to bottom priority) to keep the webapp aspect of the system sane.
- See where the state of S5 is at and hopefully start researching/planning another integration attempt.
Lume Web 2024 Grant Final Report
Overall Summary
For the 2024 grant period, we focused on building core Sia integration following community feedback. We transformed the initial MVP with basic S5 integration into a working portal system with plugin support.
Work Completed
Core System
- Implemented a plugin-based architecture similar to Caddy
- Built in Go, aligning with Sia ecosystem tools
- Added cluster coordination using etcd
- Created a unified request model for operations
- Set up database support for MySQL/SQLite using gorm
Plugins Developed
-
Dashboard Plugin
- User authentication system
- API key management
- Two-factor authentication
- OAuth2 social login
-
S5 Plugin
- Basic implementation
- Planned rewrite to match current specs
-
IPFS Plugin
- Upload and pinning support
- UnixFS metadata tracking
-
Billing Plugin
- Free tier with usage limits
- Paid plan framework
- Usage tracking
-
Admin Plugin
- Cron job monitoring
- System settings interface
- Partial settings editor
-
Support Plugin
- Freescout integration
- OAuth2 SSO support
Technical Implementation
- Used TUS protocol for large file uploads
- Set up monitoring for system resources and MySQL
- Implemented Grafana dashboards
- Created docker images for deployment
- Developed terraform code for Akash deployment
Note: Renterd monitoring implementation is pending until renterd v2 adds proper Prometheus support (currently marked for v2 milestone).
Current Status
We’ve deployed to pinner.xyz and launched documentation at docs.lumeweb.com. The system includes basic resource monitoring and MySQL tracking.
Incomplete Work
- S5 plugin needs rewrite for current specifications
- Billing system deployment delayed due to database clustering issues
- Frontend needs stability improvements
- Admin tools require additional development
Strategic Decisions
- Built working MVP of sync plugin with JavaScript P2P implementation
- Current hybrid JavaScript approach proved too complex to maintain
- Future development will revisit this with a Golang implementation when appropriate
- Will reassess timing for P2P features based on project priorities
Next Steps
See our approved 2025 grant proposal for continuation of this work.
Repository Links
Core Components
Infrastructure
Hello,
Thank you for your progress report!
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
The Pinner.xyz portal is now working for uploads and is using renterd for the s3 buffer storage as well.
If the foundation needs an account to test anything, they can DM over established channels for a login as I do not wish to have an account abuse on the system before I am able to handle it 
. However registration simple, and no billing plugin is currently active.
I will be focusing next on getting billing services online.
Thanks.