Summarize any problems that you ran into this month and how you’ll be solving them.
Parallel TUS uploads turned out to have some quirks to them. The behavior of CLI vs JS TUS showed some issues and required me to temporarily restrict the dashboard to upload in serial.
This will be worked on more in Q2 with some possible design/strategy changes for TUS uploads, but is potentially complex enough to need to be punted a quarter.
CI/CD devops, workflow revamp and a migration to monorepo(s)
There are a lot of issues with the current workflows I use to develop, and a lot of friction, causing slow downs. Based on research, and looking at the Foundation for some inspiration, I will be changing the dev tools and workflows used (but avoiding SaaS’s), and ideally enabling more collaboration in the future.
Refactor the cron (task) system and how it is used, implement persistent task handling. This will be a critical system in the long term.
Handle any community feedback for the webapp.
Begin research on Q2 priorities and the best way to design and implement.
One slight issue hit: One of the validations is the registry. Due to existing known issues with compatibility between the Dart and Golang implementations of S5, where Vup has not yet been updated, it was not possible to demonstrate a visual usage of the registry. The core dart libraries have been updated though and a dart test case (sample script) demonstrating the network function can be provided to the foundation upon request.
Great to see you launch the initial version of the S5 node and portal, looking forward to see you iterate on the features!
Seeing that you are revamping your workflow, I would like to suggest you use GitHub as your primary repository and mirror this to your Lume git server for backup. Although it is very cool and more decentralized to work on your own git server, GitHub is where the dev ecosystem lives and imo activity and development on GitHub is necessary for discovery and contribution.
Finish the sync system. This will require bundling the nodejs server into the golang portal, and require handling keeping renterd meta in sync with the portal.
Start on Q2 design → dev work which includes mobile support/design, better colors, and a theme system. More will come as needed this quarter.
Work on portal scaling (cluster) code and any other needed quality of life tasks.
Handle community feedback as needed.
Work on frontend tasks if time permits this month.
Finished MVP of sync plugin and quarterly video demo
Finished portal architecture re-design
xportal a fork of xcaddy can now be used to build any kind of portal
Implemented the mobile UI
Implemented a new theme color
The starting point of a portal settings editor for the admin UI has been created
Summarize any problems that you ran into this month and how you’ll be solving them.
Logistics limitations with the team did not allow as much app development to get done as hoped. Several tasks, including the admin, a navigation menu, and a theme switcher are still waiting to be completed. These are expected to be done early Q3.
S5 has been a blocking problem and so no work around it has been completed further. I am awaiting the specs for S5 (of which has been given another grant) to ensure the go port is up to date and improve on the S5 dashboard support.
Due to the more abstract nature of this quarters priority, if anyone wishes to experiment on this, they should reach out since the infra on running these demos is more complex with multiple servers, and the portals had to be reset at-least 1 time to show different scenarios.
Added an upload management system, powered by Uppy, and updated the UI
Got an initial support for IPFS working in both small files and TUS
Added the basics of an uploads queue manager
Added the UI for subscription management
Got the majority of IPFS completed. Testing pinning, serving on the network, and downloading data remain.
Got the majority of billing support completed. More testing will need to be done. Work is also needed on restricting uploads/downloads based on the quota system.
Refactored the backend to use a new data model for managing upload requests, and have every upload now always go to a queue.
Summarize any problems that you ran into this month and how you’ll be solving them.
Get a support system integrated, probably with oauth
Re-create the file manager
Implement and/or update various account qualify-of-life functions
Getting the UI improved as needed
Implement API Keys
Create Q3 video demo per milestones
The admin UI/plugin and some other backend functions, and/or abstractions, including a few scaling/cluster related things will be punted to Q4. Updating and re-adding S5 support will be punted to Q4 as well based on the current timelines of that project, but there is a chance that could take longer.
Completed IPFS support, including tracking unixfs metadata provided via API for the client
Completed billing support. Includes:
Tracking quota usage of users
Limiting access if over usage
Every protocol plugin will need to be checking the users usage on its endpoints, this is not centralized in the core outside possible helpers
A very basic plans support (a db table that currently needs direct manual entry). This is used for the quota side, not the billing side.
Support for a free plan via config that requires no external systems
Support for paid plans that require external systems
Paid plans rely on Killbill.io for subscriptions (java based) and hyperswitch.io (rust/js) for payments. These systems are rather complex and documentation will be created over time as issues are found for running them in prod.
A lot of research went into finding FOSS billing systems to avoid vendor lockin and future proof merchant control for political and business risk. There are very very few options that are suitable and maintained/have funding. The rest are SasS.
Stripe is supported now via hyperswitch, as you can treat hyperswitch as a payment gateway in itself.
Got the webapp in an alpha state with:
IPFS service
Account management and security
Uploads management
2FA
API keys
Dashboard statistics, removing the mock UI’s
The webapp will likely heavily evolve over time in architecture, as plugins will need to be supported in a dynamic way. For now, things are activated in a feature flag approach and detected in the webapp as a monolith from the /api/meta endpoint accessible from any domain of the portal server.
In the future I will be looking at using tech called module federation and creating the infra needed for this to work. However, the JS ecosystem is heavily in R&D mode and these things are just not ready yet :(.
So for now things are done the most pragmatic way until it can be more sophisticated.
The support plugin has been created and uses oAuth2 to SSO into freescout.net, a laravel-based PHP support system. This required buying a plugin that provides an end-user support portal, but as all plugins are AGPL, the plugin has been forked and is on GitHub now.
Lots of internal refactoring and internal changes have been made this month and more will be made over time in the future.
Summarize any problems that you ran into this month and how you’ll be solving them.
Only persistent issue has been IPFS DHT/discovery. After fixing the configuration logic for announcement addresses (the original mistake made), this worked earlier this quarter, but it has been flaky lately. Direct peer connections via multiaddresses work regardless (thus pinning works). This was also tested with fsd to compare against a known working node besides kubo, with no success (fsd didn’t work either). It is possible a mis-configuration exists somewhere, but time will need to be spent to find the cause.
I have been able to figure out the problems with the IPFS DHT. Discovery of the portal was working properly, however the provider function of the IPFS node code had a bug in it around broadcasting new content that was recently uploaded.
With some other misc fixes/changes I was able to verify a file is accessible via a public gateway. Pinning data from a local IPFS desktop client though may still be fickle since that might have to use a relay node due to NAT or other network quirks. Edge cases around this will be worked on more as they come up, if possible.
Given this, focus will be to continue testing, and fixing webapp bugs, along with the other priorities I have already provided for October.
(There are potential compliance things to be tackled here :/)
Add further cluster support to the portal, specifically IPFS needs to have some cluster support for its databases added.
Create the monitoring configurations.
Create docs for running a private IPFS portal.
Complete a soft launch and find the bugs that will come :P
Important: Based on discussions with @redsolver, S5 support will be punted to Q1 2025. I do not feel things are ready yet to do another integration attempt. Additionally I have enough to do as-is and that is likely a 1-2 week effort easily.