Standard Grant Proposal: Lume Web 2024

Grants Approved
27

I wouldn’t classify a frontend for a public web app as marketing. That’s required for basic access. My primary concern is with the secondary websites alluded to in your proposal and comments.

28

Ok, will it be an issue for the front end to have a pricing UI or marketing-related content (assuming I am not hiring for the content)?

Asking to be sure based on what has been requested.

To clarify, the intention is to have 2 web apps from a technical POV:

  • Frontend of portal
  • Backend dashboard

Lastly, would it be an issue to leave the design & development budget the same and consider anything I might deduct as a buffer? I will state that while I intended to have a second site, it was not guaranteed, and I still don’t have a guarantee it would have been built anyway, as I’m still trying to secure the asset.

29

Why cannot you keep the scope at what you are sure you can achieve within the project timeframe? You can always expand if you feel you have resources, and if you run out of budget, well, the Committee is generally open to considering such requests.

30

Sorry, can you please clarify cannot you keep the scope at what you are sure you can achieve within the project timeframe?

Scope-wise, the only thing guaranteed now that I’m not doing is a second marketing site, and no content/seo, etc. nothing else has changed. I am only trying to understand what is not wanted given marketing is being asked to be removed. The rest it seems is mostly communication issues on definitions.

31

To clarify, based on the committee’s request, the scope of the development work budget only removes a single marketing site that was not 100% guaranteed (I don’t have the asset yet). And that I estimate to be a fairly small portion relative to everything else, assuming I executed on it.

From the start, my definition of marketing sites was portal web app frontend(s) that also served to do marketing via its content and pricing.

Nate has stated that would generally not fall to marketing, thus the miscommunication of definitions, but to remove the second site.

So, my questions ensure the committee understands what I mean and what apps I’m developing. I’m also trying to ask, since marketing is being asked to be removed, what are things on the frontend app that would not be allowed?

The question about keeping the budget the same is due to the second site being a minor part of the budget, which I was not 100% sure was guaranteed.

So this is about me ensuring I don’t break rules on what’s being allowed and ensuring the committee understands what I actually meant by marketing sites.

Thanks.

32

To make it plain: please remove this section entirely. There is little point in keeping this discussion. As was mentioned, there are legal issues involved, and you surely wouldn’t want that.

33

Yes, That I am not disputing.

I am trying to understand what limitations, if any, are on the portal front web app design or content (where I’m not paying for the content) based on the no-marketing limitation. I do not want to violate the rules unknowingly due to that request.

34

I think we can summarize the changes here:

  • Remove the $9,000 marketing line item(s) from the budget. These items are either strictly marketing (the sites and and the social media) which we don’t currently fund, or too early to matter (the video) for something that isn’t quite built yet
  • A front end essential to using the app is not considered marketing
  • The front-end can certainly talk about your project and what you offer. Think back to the old Skynet portals. They told you what Skynet was, how to use it, and how it worked.
35

Introduction

Project Name: Lume Web

Name of the organization or individual submitting the proposal: Hammer Technologies LLC

Describe your project:

Lume Web is our way of stitching together the vast web3 landscape. We’re building a hub for everyone to access decentralized content easily without relying on any central power.

Our aim with Lume is to be that one-stop gateway for all decentralized content in a genuinely decentralized way.

Our goals for 2024:

  • Multi-protocol Sia Portal Software Development
    • Rewrite MVP software to support multiple decentralized protocols.
    • Begin with S5 protocol implementation in Golang; collaborate with @redsolver.
    • Use subdomains for each protocol to serve as API servers.
    • Add support for IPFS, with plans to include more protocols.
    • Aim for portal software enabling paid hosting and competition with large closed-source firms, maintaining democratic/decentralized access.
    • Introduce a single subscription service for diverse data management.
  • Billing and Subscription Integration
    • Add billing support, initially incorporating Stripe, avoiding crypto transactions for users.
  • Web Dashboard and Portal Frontend
    • Develop a web dashboard for web portal.
    • Develop a front end webapp for web portal.
  • File Sharing Functionality between Portals
    • Implement hypercore technology for a decentralized file metadata-sharing system.
    • Focus on on-demand contract creation for file imports and redundancy vs. pre-creating contracts like skyd did.
    • Keep P2P logs/blockchains public to ensure data persistence and resilience.
    • This approach intends to expose the sector data to the community for redundancy and ensure the portal can track who is accessing the data. S5’s network takes a different approach to this, which may make tracking user usage more complicated and does not currently share sector data.
  • OAuth Integration with Ticket System
    • Enable linking accounts to a 3rd party ticket support system with SSO.
  • API Documentation
    • Create full documentation of the supported portal API’s.
  • Community Hosting Service Management
    • Set up a robust hosting service utilizing Sia’s decentralized storage, offering both paid tiers of storage services and a free but limited storage plan.

As the dashboard will be the most significant frontend effort, we are looking to get the following designed and implemented:

  • Account Management
    • Login Screen (username/email, password fields, two-factor authentication)
    • Account Dashboard (profile and security settings, notification settings, account status)
    • Password Recovery Process
    • Account Creation/Sign-Up Process
    • Account Deletion/Closure
    • Design and User Experience Considerations (user-friendly interface, responsive design, security, accessibility)
  • Subscription Process with Stripe Integration
    • Plan Selection Screen (overview of subscription plans, plan details)
    • Stripe Redirection for Payment (Stripe checkout integration, pre-filled information)
    • Confirmation Before Redirection (review and confirm plan selection, terms of service)
    • Post-Payment Processing (success and failure handling, email confirmation)
    • Subscription Management (dashboard for subscription status, modify/cancel subscription)
    • Design and User Experience Considerations (simplicity in plan selection, seamless Stripe integration, responsive design)
  • IPFS File Management (powered by Sia)
    • File Explorer Interface (navigation pane, main viewing area, breadcrumb trail)
    • File and Folder Operations (upload, create, delete, rename, move)
    • IPFS-Specific Features (pin files/folders, retrieve IPFS hashes)
    • Search and Filter Functions (search by name, type, metadata; filter by size, date, etc.)
    • File Previews and Details (previews for specific file types, detailed file metadata display)
    • Design and User Experience Considerations (intuitive layout, responsive design, performance optimization)
  • S5 File Management (powered by Sia)
  • File List Interface (displays file name, size, type, last modified date, flat file listing without traditional folders)
  • Manifest-Based Navigation (virtual folder structure navigation for directories/apps)
  • File Operations (upload, download, delete, rename files, manage manifests for virtual directories)
  • Search and Filtering (search by name, type, metadata; filter by size, date, etc.)
  • File Previews and Details (previews for specific file types, detailed file metadata display)
  • Public Key Management
    • Key Registration and Setup (key upload interface, key verification process)
    • Key Management Dashboard (list of registered keys, add/delete key options)
  • Admin Panel
    • Dashboard (overview and statistics, quick access panel)
    • User Account Management (subscriptions, billing)

For more information, you can visit https://docs.lumeweb.com/ or check out a snapshot of the documentation at the time of writing at https://git.lumeweb.com/LumeWeb/docs.lumeweb.com/src/commit/e5520d24023895d14d3f950f9f3f45b7d34b265a.

Additionally, as the Lume project is slowly maturing, we are attempting to be more structured and organized in project management.

Who benefits from your project?

The project benefits:

  • Sia community/users by:
    • Enabling an L2 portal software that gives access to multiple decentralized protocols and stores data on the Sia network.
    • Starting a hosting service for users who do not want to run a node themselves.
    • Evangelize the Sia network and spread awareness that it is the data backbone of web3.
  • Data hosters by:
    • Providing cheap, reliable, decentralized storage via multiple distribution protocols to the broader community.
    • Democratizing access to FOSS file portal software that will enable pinning on multiple protocols, ensuring censorship resistance.
  • DApps/Web Apps by:
    • Enabling web apps to be hosted on the Sia network through several distribution protocols, which should ensure censorship resistance and grow network demand.

How does the project serve the Foundation’s mission of user-owned data?

For 2024’s focus, it provides a reliable platform for Sia users to host their data for both S5 and IPFS-based protocols. It also creates a community service based on the software built.

The web needs paid data storage to ensure things stay online and censorship-resistant and users need private data storage that retains their privacy and self-sovereignty.

Grant Specifics

Amount of money requested and justification with a reasonable breakdown of expenses:

Total Budget Request: $79,000

Upfront Budget Request: $19750

The initial funding needed includes:

  • Derrick Hammer’s Q1 Salary: $13,750
  • Project Budget for Q1: $6,000

Detailed Expenses Breakdown

1. Salary Expenses: $55,000

  • For Derrick Hammer
  • Justification: This represents an inflation-adjusted increase from the previous year.

2. Annual Project Budget: $24,000

  • Design and Development Work: $19,000
    • Portal dashboard - Design and development of a portal dashboard with both user and admin access.
    • Frontend portal app - Design and development of a frontend webapp for the portal.
  • Server Infrastructure Costs: $4,000
    • Note: This is subject to change based on testing and scaling needs.
  • Sia Network Subsidy for Storage: $1,000
    • Contingency: Additional funds may be necessary if demand exceeds expectations.

Notes:

  • The progression and timing of milestones may be adjusted or reordered in light of emerging risks, unforeseeable events, or community input.
  • The dashboard will be built gradually throughout the year as milestones are hit.
  • A hosting subsidy for running the community service will be requested in a grant for 2025.

Timeline with measurable objectives and goals

Q1: Build core portal: $19,750 (Paid as part of up-front payment)

Goal: Complete architectural redesign with support for multiple protocols. Start with S5 support and DNSLink

Validations:

  • Should be able to upload and download small files
  • Should be able to upload and download large files over TUS
  • Should be able to login/register with S5’s account API’s
  • Should be able to import a file from another peer and store in the portal’s contracts
  • Should fully support all S5 HTTP node APIs. Admin APIs may be skipped or mocked until they can be routed internally for the dashboard functions.
  • Should fully support the S5 registry
  • Should have API documentation, in collaboration with redsolver as needed
  • Should support serving a webapp over DNSLink

Milestone Progress: Will create a video showing uploading/downloading, login/register, S5 registry, DNSLink, and file import in the monthly report

Q2: Integrate hypercore for file syncing: $19,750

Goal: Integrate hypercore for metadata sharing

Validations:

  • Portal should be able to import metadata from a dead portal
  • Portal should be able to import a file from another live portal
  • Multiple peers should be able to seed the logs of portals

Milestone Progress: Will create a video showing the validations in the monthly report

Q3: Add IPFS protocol and billing: $19,750

Goal: Integrate IPFS into the portal

Validations:

  • Should be able to browse Sia-stored IPFS content from public gateways
  • Should be able to upload files to the portal
  • Should be able to download files from the portal
  • Undecided: Implement hypercore sharing of IPFS files. This will be determined when implementing if it makes sense to do so.

Goal: Complete integration of Stripe billing for subscriptions

Validations:

  • Ensure basic billing and subscription functions work
  • Ensure support SSO works with ticketing system

Milestone Progress: Will create a video showing in the monthly report:

  • Browsing IPFS content pinned on the portal
  • Upload and download from the portal
  • Subscription sign up
  • Logging into ticket system

Q4: Community Hosting: $19,750

Goal: Set up hosting environment and monitoring systems

Validations:

  • Should have monitoring setup of the portal usage metrics based on existing grants
  • Should have any possible alerts setup, storage, or money, based on existing grants, and implement if needed, assuming the required webhooks exist
  • Should have devops infrastructure created and documented for others to setup a portal
  • Should be successfully stressed tested that large data can be both uploaded and downloaded

Milestone Progress: Will have a portal service ready for the community to test and use

Potential risks that will affect the project’s outcome:

  • The S5 development might encounter technical hurdles in the transition to Golang, causing project delays and increased complexity.
  • Operational challenges with Stripe for payment processing might force a shift to alternative payment processors, affecting transaction handling.
  • Unanticipated complications with renterd could arise, which have yet to be specified but could affect the project’s storage or networking functions.
  • The ABAO support might not be ported over from Rust to the required platform by @nemo within the necessary timeframe, as discussed in Bao Tree Support · Issue #17 · lukechampine/blake3 · GitHub, potentially affecting cryptographic or hashing functionalities.
  • Developmental difficulties with IPFS on the portal could hinder incorporating support for its protocol, leading to potential problems.
  • If data is served anonymously over web2 from the portal directly, there might be a need to implement anti-abuse measures to ensure data integrity and compliance with regulatory standards.

Development Information

Will all of your project’s code be open-source? Yes

Leave a link where the code will be accessible for review.

Do you agree to submit monthly progress reports? Yes

Contact info

Email: The email the Foundation already has on file.

Any other preferred contact methods: Discord @pcfreak30

Early submission

We are aware that we are submitting this about 1-1.5 months before our current grant is completed. Our justification is that we need to ensure continued funding in 2024 (Jan) and cannot afford to have a large gap, as we have no other significant active income at this time. Lume is our full-time mission.

Additionally, our plan is for the end of our 2023 grant to finish with the web3.news MVP as a deliverable.

36

Hello @pcfreak30,

Thanks for providing us with the requested updates to your proposal. The removal of marketing items, change to the budget disbursement, and additional milestone validations were very helpful during our assessment. After review, the committee has approved this grant!

We’ll reach out via email to get you set up. Onboarding typically takes a week or two, so feel free to adjust your internal timelines as appropriate.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

1 Like
37

Progress Report for January

What progress was made on your grant this month?

  • Got an S5 node implemented with most major protocol features. One recent new feature, message streaming, has not been added yet.
  • Rewrote the portal using Jape and have a bare-bones framework structure support for protocol and APIs
  • Implemented most S5 HTTP API’s
  • Basic account support
  • Basic pinning support
  • Basic upload and TUS support. Uploads go to an S3 buffer when they cannot fit memory (tus vs. post) and then are processed as a scheduled task and pushed via multipart upload.
  • Basic download support
  • The following goals for this quarter have been met but are subject to more improvements and testing:
    • Should be able to upload and download small files
    • Should be able to upload and download large files over TUS
    • Should be able to login/register with S5’s account APIs
  • The following goals for this quarter have been partially met:
    • Should fully support all S5 HTTP node APIs
      • At least one has not been added yet, and most not well-tested
    • Should fully support the S5 registry
      • It has been implemented at the protocol and HTTP API level but is not well-tested yet.

Summarize any problems that you ran into this month and how you’ll be solving them.

No problems big enough to need to document.

Links to repos worked on this month:

Note: view develop branch

What will you be working on next?

  • Getting the initial portal design finished
  • Porting the design into a webapp and getting it fully built out or large progress made
  • Further evolving the backend upload and download support. Work on parallel TUS uploads
  • Adding DNSLink support
  • Building the S5 JS SDK further (@lumeweb/s5-js) and implementing Swagger potentially
  • Work on blake3 BAO support
1 Like
38

Progress Report for February

What progress was made on your grant this month?

  • I refactored the bones of the portal architecture several times to shape it.
    • The portal is now positioned to act as a generic layer 2 for sia to support X protocols, enabling it to support web3 services and beyond.
    • It is subject to more design changes as needed in the long term.
    • Configuration management is now managed via go structs and is much more structured and organized.
    • Swagger can be supported for every API.
  • Most account APIs have been created. This includes:
    • Login
    • Register
    • Verify Email
    • 2FA
    • Password Reset
  • A basic mailer module has been added for SMTP email support.
  • Network pinning has been implemented and supports media, webapp, and directory manifests. In practical terms, this means:
    • If you pin a file the portal does not have, it will look for it, verify it, and create a background task to queue it.
    • If you pin a file that is a metadata file aka a manifest, it will build a tree (list) of all child CID’s and pin them all.
  • Refactoring and protocol bug fixes to libs5
  • Added swagger support to S5
  • Added initial bao verification
    • Several approaches were attempted, and I was more successful than last year. I ended up using the go-plugin with a rust GRPC child server. However, this seems to have poor performance and will not scale well potentially… based on how long each request takes.
    • It will work for the short/medium term, but the BAO support in nemo’s blake3 library will need to be completed and I switch over to that.
  • The @lumeweb/s5-js js library got refactored to use https://github.com/anymaniax/orval which itself use axios, the same library s5-js used before, so it minimizes the design changes.
  • DNSLink support has been partially implemented but needs testing, and some lower-level problems need to be solved.
  • Have tentively decided to outsource billing efforts to https://github.com/juspay/hyperswitch, an Apache2 rust-based billing system that can be integrated later this year, and eventually extended as needed.
    • This will significantly reduce complexity on billing while enabling a lot more powerful control over the commerce side.
  • I have made design plans to enable the billing to be optional medium to long term, however the priority efforts will be on ensuring a commercial-focused system per the projects goals.
  • I have also started enabling (WIP) the portal to work in a single-node mode and in a clustered mode such this can work solo without billing or needing a complex setup. This will evolve through out the year, and functions supported as the community requests it for private use vs commercial.

Summarize any problems you ran into this month and how you’ll solve them.

  • The project has had a ~ two week scheduling delay with the team on the dashboard webapp. As a result, this month was spent on backend engineering (my role) and preparing as much as possible for creating an MVP demo for the quarter.
  • While BAO/blake3/verified streaming was able to be solved crudely, it will be critical in the long term to get the Golang port completed by Nemo. A rough estimate is that a 100 GB file could take half a day to process.

Links to repos worked on this month:

What will you be working on next?

  • Get an MVP Webapp Demo out with all milestones demonstrated
  • Work on parallel tus support
  • Creating videos for the end of the quarter
  • Do more work with Vup & redsolver to ensure compatibility if time permits. The last known issue with it was a streaming encryption problem.
39

Hello @pcfreak30,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

40

Progress Report for March

What progress was made on your grant this month?

  • Did some preparation to enable scaling with the portal database and S5 KV databases
  • Added a price tracker module for sia to track the last X days from siacentral’s API and update the renterd pricing config, as well as run daily.
  • Internal refactoring with HTTP routing
  • Implemented cookie support for logins, mainly for the dashboard
  • Various misc bug fixes/refactoring
  • Implemented native bao verified streaming support support thanks to Add support for Bao chunk groups by lukechampine · Pull Request #19 · lukechampine/blake3 · GitHub.
  • Fixed seeking issues causing Vup streaming to not function correctly.
  • Helped redsolver identify bugs during testing with S5/Vup.
  • Created MVP demo, and launched https://account.testing.pinner.xyz

Summarize any problems that you ran into this month and how you’ll be solving them.

Parallel TUS uploads turned out to have some quirks to them. The behavior of CLI vs JS TUS showed some issues and required me to temporarily restrict the dashboard to upload in serial.

This will be worked on more in Q2 with some possible design/strategy changes for TUS uploads, but is potentially complex enough to need to be punted a quarter.

Links to repos worked on this month:

What will you be working on next?

  • CI/CD devops, workflow revamp and a migration to monorepo(s)
    • There are a lot of issues with the current workflows I use to develop, and a lot of friction, causing slow downs. Based on research, and looking at the Foundation for some inspiration, I will be changing the dev tools and workflows used (but avoiding SaaS’s), and ideally enabling more collaboration in the future.
  • Refactor the cron (task) system and how it is used, implement persistent task handling. This will be a critical system in the long term.
  • Handle any community feedback for the webapp.
  • Begin research on Q2 priorities and the best way to design and implement.

Q1 Milestones Report

Per the grant proposal here is the video demo demonstrating all but one milestone: https://youtube.com/watch?v=qMxoM1fQMJ4

Please be aware the demo voice over is AI generated

Any testing can be performed at https://account.testing.pinner.xyz

One slight issue hit: One of the validations is the registry. Due to existing known issues with compatibility between the Dart and Golang implementations of S5, where Vup has not yet been updated, it was not possible to demonstrate a visual usage of the registry. The core dart libraries have been updated though and a dart test case (sample script) demonstrating the network function can be provided to the foundation upon request.

41

Great to see you launch the initial version of the S5 node and portal, looking forward to see you iterate on the features!

Seeing that you are revamping your workflow, I would like to suggest you use GitHub as your primary repository and mirror this to your Lume git server for backup. Although it is very cool and more decentralized to work on your own git server, GitHub is where the dev ecosystem lives and imo activity and development on GitHub is necessary for discovery and contribution.

42

With that, the subtitles in French are very great !

43

Thank googles AI then? :upside_down_face:

44

Thanks. I am researching and will consider what you have said.

45

Progress Report for April

What progress was made on your grant this month?

  • Migrated development back to GitHub for the short-medium term
  • Implemented initial version of a persistent cron system
  • Did research and planning for the quarter
  • Kicked off design priorities with team
  • Major progress made on portal sync system using hypercore P2P as a means of distributing metadata.

Summarize any problems that you ran into this month and how you’ll be solving them.

No problems this month

Links to repos worked on this month:

What will you be working on next?

  • Finish the sync system. This will require bundling the nodejs server into the golang portal, and require handling keeping renterd meta in sync with the portal.
  • Start on Q2 design → dev work which includes mobile support/design, better colors, and a theme system. More will come as needed this quarter.
  • Work on portal scaling (cluster) code and any other needed quality of life tasks.
  • Handle community feedback as needed.
  • Work on frontend tasks if time permits this month.
1 Like
46

Hello @pcfreak30,

Thank you for your perfect progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee