Standard Grant Proposal: Proxmox Backup Cloud Extension (Continuation)

#Project Name: Proxmox Backup Cloud Extension (Continuation)
Project Lead: Rodney Wild
Company: A-Z Computer Solutions

Purpose:
The goal of the continued phase is to further integrate Proxmox Backup Server (PBS) with the Sia decentralized cloud storage network, enabling seamless and redundant cloud-based backup solutions. Building upon the progress made in the previous months, this phase will focus on refining the existing cloud backup module, enhancing the functionality of the Sia renterd plugin, and ensuring robust testing to prepare for a public beta release.

The primary objectives remain:
Increasing the amount of data stored on the Sia network by leveraging Proxmox backups.
Offering a reliable off-site backup solution that adds redundancy for disaster recovery scenarios.
Finalizing a stable integration that can be widely adopted by businesses, helping them securely store backups on the Sia network.
Open-Source Commitment:
The project’s source code will remain open-source, hosted on GitHub, with full transparency for the community.

Project Timeline (2.5 Months Extension):

Month 1 (Weeks 1-4):

Complete the final development of the Sia renterd plugin and resolve any remaining technical challenges.
Conduct internal testing of the renterd plugin in various real-world scenarios to ensure reliability and efficiency.
Prepare documentation for both developers and users, covering setup, configuration, and usage of the Sia cloud backup module.

Month 2 (Weeks 5-8):

Implement feedback from internal testing and address any critical bugs or issues.
Begin beta testing with a select group of users and community members to gather feedback on performance and usability.
Refine the integration based on beta testing, focusing on improving user experience, reliability, and scalability.

Half Month (Weeks 9-10):

Finalize any remaining tasks based on beta feedback.
Prepare the module for a public release, ensuring that all key features are working as expected.
Draft a public release announcement, including the project’s accomplishments and future outlook.

Potential Risks:
Delays in Testing Feedback:
The beta testing phase could uncover unexpected issues, potentially delaying the public release.

Mitigation: Prioritize frequent communication with testers to ensure early identification and resolution of issues.

Loss of Development Resources:
Similar to previous concerns, losing key development personnel could impact project delivery.
Mitigation: Maintain an active pipeline of backup developers who are familiar with the project’s codebase.

Projected project completion cost and funds requested:

480 hours * 35$ per hour = $16,800 USD

The above hours include no extra money for the project manager and any funds not use will go back to the Sia Foundation. The above estimate (other than the added contingency) comes directly from our developer.

1 Like

Thanks @rdwild. Quick question: Your timeline cites 10 weeks of work but your budget request appears to be for 12 (480/40). What’s going on there?

1 Like

As you know, development is not always an exact science. There is a 20% contingency added on to the quote I get from the developer. The developer came back with an a slight adjustment (400 hours and I added the 20% contingency (80 hours) which will be returned if not used by the developer.

Rodney Wild
A-Z Computer Solutions

1 Like

Thanks for your latest proposal to The Sia Foundation Grants Program.

After review, the committee has decided to approve your proposal. They’re definitely excited and hopeful to see the completion of your work so far.

We’ll reach out to your provided email address for onboarding. Obviously this shouldn’t take too long as this is a continuation of your previous grant.

1 Like

Thanx Steve! The details should be the same.

1 Like

Progress Report Form Oct 2024

What progress was made on your grant this month?

 Middleware Development for Sia Integration:

 Cloud Backup API Enhancements:

 Integration Testing and Code Maintenance:

 User Interface (UI) Improvements:

Phase 8: Beta testing and bug cleanup:

Documentation and Repository Management:

We updated and merged our work on GitHub, including all major middleware and API connection changes, code maintenance tasks, and ExtJS UI updates.

Cloud Backup Middleware Development

Objective: Build middleware to enable interaction between Proxmox Backup Server’s cloud backup APIs and Sia’s decentralized storage API, addressing connectivity issues.

Changes:

Developed the middleware layer to route API calls effectively and provide consistent data handling across platforms.

Added error-handling routines to manage connectivity errors and log middleware actions for debugging.

What was done in the code:

src/middleware/siaIntegrationMiddleware.rs: Main middleware file for API requests.

src/middleware/siaIntegrationTest.js: Middleware test script for Sia API functionality checks.

API Connectivity Enhancements

Objective: Improve cloud backup API’s performance and security by updating dependencies and fixing broken functions.

Changes:Replaced outdated Rust dependencies with optimized libraries for faster and more stable performance.

Fixed broken functions within cloud backup API to ensure proper backend/frontend communication.

pbs/src/api2/cloud/backup.rs: Main cloud backup API file.

pbs/src/api2/cloud/mod.rs: Module configuration for cloud backup.

pbs/src/api2/cloud/restore.rs: Restoration logic for cloud backups.

Encryption & Decryption for Data Security

Objective: Secure data interactions by implementing hash encryption and decryption routines.

Changes:Added encryption methods to secure sensitive data during transmission between cloud backup UI and Sia storage.

www/cloud/window/Hashcrypt.js: Contains hash encryption and decryption methods.

Cloud Backup UI Development

Objective: Improve the cloud backup user experience through API integrations and UI styling enhancements.

Changes:Integrated backup and restore functionality directly into the cloud backup UI.

Enhanced UI and styling for improved user experience and accessibility.

www/cloud/window/CloudBackupJob.js: UI logic for backup jobs.

www/css/ext6-pbscloud.css: Stylesheet for cloud backup UI.

www/cloud/backupworks/backupschedule.js: UI for backup scheduling options.

Access Control and Backup Group Management

Objective: Enhance user access control settings and allow management of backup group ownership.

Changes:Introduced features for editing access control within the UI.

Added functionality to change backup group owners directly from the interface.

www/cloud/window/ACLEdit.js: Access control settings.

www/cloud/window/GroupManager.js: Management of backup groups.

ExtJS Frontend Development and API Testing

Objective: Ensure compatibility and test the UI for Sia integration using ExtJS and API testing tools.

Changes:Continued refining the ExtJS frontend for seamless user interactions with the Sia API.

Conducted API connection tests to validate functionality for cloud backups.

src/cloud/test/alloc_writable_media.rs: Test file for API connections.

www/cloud/window/UIComponents.js: Various UI components.

Links to repos worked on this month:

Merged branch ‘develop’ of https://github.com/AZComputerSolutions/PBSCloudBackupModule

What will you be working on this month?

Beta Testing and Bug Fixes:

 Test Server Setup and Deployment:

 Middleware Enhancements:

 UI Refinements and User Experience Improvements:

 Documentation and GitHub Management

 Please summarize your development goals into a few sentences or bullet points:

Fix bugs and code dependencies changes in rust that make our code no longer compile . Get a test server setup and working. Document setting up a test server. So that anyone can do it would the code in our github site. I would also like to note that because of the pending approval by the Sia foundation (thank-you for your continued support) we have been running at full development speed this last month.

We were ask by the Sia foundation to shorten our monthly reports but as always, the full version of this document is available at https://www.pbscloudbackup.com/docs/sia/sia-foundation-monthly-report-oct-2024/

1 Like

PBSCloud Backup Progress Report Nov 2024

What progress was made on your grant this month?

  • Sia Middleware Proxmox
  • Cloud Backup API Refinements:
  • Advanced Security and Compliance Measures:
  • UI Enhancements and New Features: Integration Testing and Pre-release Documentation:
  • Repository Optimization and Code Consolidation:
  • Sia Middleware Proxmox
  • Cloud Backup API Refinements:
  • Advanced Security and Compliance Measures:
  • UI Enhancements and New Features:
  • Integration Testing and Pre-release Documentation:
  • Repository Optimization and Code Consolidation:

Phase 8: Beta testing and bug cleanup:

Documentation and Repository Management: We updated and merged our work on GitHub, including all major middleware and API connection changes, code maintenance tasks, and ExtJS UI updates.

Cloud Backup Middleware Development Objective: Build middleware to enable interaction between Proxmox Backup Server’s cloud backup APIs and Sia’s decentralized storage API, addressing connectivity issues.

Changes:

Developed the middleware layer to route API calls effectively and provide consistent data handling across platforms.

Added error-handling routines to manage connectivity errors and log middleware actions for debugging.

What was done in the code:

src/middleware/siaIntegrationMiddleware.rs: Modularized middleware and added caching logic. src/middleware/loadBalancer.rs: Implemented dynamic load balancing for middleware operations.

pbs/src/api2/cloud/backup.rs: Integrated asynchronous processing and analytics reporting. pbs/src/api2/cloud/auth.rs: Added token-based authentication.

www/cloud/security/AuditLogger.js: Implemented logging for critical actions. www/cloud/security/EncryptionHandler.js: Enhanced encryption algorithms.

www/cloud/window/StorageDashboard.js: Created Sia storage monitoring dashboard. www/css/ext6-pbscloud.css: Updated styles for localization and accessibility. www/cloud/backupworks/dependencyMap.js: Added dependency visualization features.

Links to repos worked on this month:

Merged branch ‘develop’ of https://github.com/AZComputerSolutions/PBSCloudBackupModule

What will you be working on this month?

Release Candidate Preparation:

Resolve final bugs and ensure all features meet performance benchmarks.

Conduct a soft launch with select beta testers.

Long-term Maintenance Planning:

Develop a roadmap for supporting additional storage backends.

Document middleware architecture for handoff and future scalability.

Please summarize your development goals into a few sentences or bullet points:

Fix bugs and code dependencies changes in rust that make our code no longer compile . Get a test server setup and working. Document setting up a test server. So that anyone can do it would the code in our github site.

As always, this document has been shortened greatly, please see the full version of this document at https://www.pbscloudbackup.com/docs/sia/sia-foundation-monthly-report-nov-2024/

1 Like

Hello,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

1 Like

Sia Foundation Monthly Report – Dec 2024

What progress was made on your grant this month?

 **Middleware Completion and Advanced Sia Integration**

 **Expanded Cloud Backup API Capabilities**

 **Sia API Enhancements and Optimization**

 **Security and Compliance Advancements**

 **User Interface (UI) Improvements**

 **Phase 8: Beta testing and bug cleanup:**

 **Middleware Scalability**

Challenge: The middleware faced delays under high loads due to inadequate request handling.

Resolution: Implemented asynchronous request processing and dynamic load balancing to ensure consistent performance under heavy data throughput.

Sia Network Constraints

Challenge: Node unavailability and bandwidth limitations impacted decentralized storage performance.

Resolution: Developed robust fallback mechanisms and optimized Sia API interactions for redundancy and reliability.

UI Synchronization Issues

Challenge: Real-time status updates and dependency visualizations caused inconsistencies in the user interface.

Resolution: Integrated advanced caching and synchronous update mechanisms to maintain data consistency across the UI.

What was done in the code:

src/middleware/siaIntegrationMiddleware.rs: Maintenance mode adaptation and improved error-handling logic.

Cloud Backup API Enhancements

Updated API types to standardize interactions and improve compatibility across modules.

Made significant updates to the cloud backup API, optimizing data handling and request/response cycles.

Enhanced access control functionalities with updates to the acl.rs file, introducing more granular permissions.

Files Updated:

pbs/src/api2/cloud/backup.rs: Cloud backup API updates to improve efficiency and scalability.

File Modified:

pbs-api-types/src/acl.rs: Access control updates for better security and role-based permissions.

pbs-api-types/src/libs.rs: Basic API types used by most of the cloud backup code.

pbs-api-types/src/matintenance.rs: Checks the current maintenance mode against an attempted operation.

pbs-api-types/src/network.rs: Configuration is done manually using other tools and network interface type.

Links to repos worked on this month:

Merged branch ‘develop’ of https://github.com/AZComputerSolutions/PBSCloudBackupModule

What will you be working on this month?

Release Candidate Preparation:

Final Testing and Stable Release

Conduct final beta testing and resolve any remaining bugs or performance bottlenecks. Prepare for the stable release of the Proxmox Backup Server Cloud Backup Plugin with full Sia integration. Enhanced Documentation and Community Collaboration and Publish comprehensive user and developer documentation, including setup guides and troubleshooting resources. Release the plugin on GitHub and collaborate with the open-source community to gather feedback for future improvements.

Long-Term Maintenance and Scalability

Plan for the integration of additional storage backends and further middleware scalability. Initiate discussions for partnerships with other decentralized storage solutions to expand functionality.

UI/UX Enhancements

Finalize multi-language localization and add contextual help features to improve user experience. Continue refining the user interface, including advanced visualization tools for backup job dependencies and performance metrics.

Please summarize your development goals into a few sentences or bullet points:

Fix bugs and code dependencies changes in rust that make our code no longer compile . Get a test server setup and working. Document setting up a test server. So that anyone can do it would the code in our github site.

As always, this document has been shortened greatly, please see the full version of this document is available at https://www.pbscloudbackup.com/docs/sia/pbscloud-backup-progress-report-dec-2024/

1 Like

Hello,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

1 Like

Hey @rdwild, our dev team was having some trouble evaluating your progress so far and in particular up to your current milestone, which is “Complete the final development of the Sia renterd plugin”.

We’ll need either a setup guide, working demo, or video demo to evaluate. They mentioned they quickly scanned the repo for our code or API routes and didn’t see anything. The files mentioned in some recent reports were either too simple or not updated in a while.

Once we get that, they’ll re-evaluate and be able to send this month’s payment.

1 Like

You guys asked for short reports, the long report link has been at the bottom of every report. I talked with my developer and the API stuff was not committed. Please see the following link:

If there is anything else that is needed please let me know. Thanx.

1 Like

They were happy to see the API code get committed. They’d still like to see a setup guide, video demo, or working demo to verify progress. Thanks.

1 Like

Thanx Steve, I have already talked to my dev on that very subject and also passed along your comments. We should have that very soon.

1 Like

There was a slight delay in getting this month’s report; it will be in by the end of the day.

2 Likes

Sia Foundation Monthly Report – Jan 2025

What progress was made on your grant this month?

Phase 8: Beta testing and bug cleanup:

Sia API Integration

Full Sia API Integration: Completed the integration of the Sia API within the Proxmox Backup Server Cloud Backup Plugin. This allows seamless interaction with the Sia decentralized storage network for reliable and secure cloud backups.

File Redundancy and Bandwidth Monitoring: Added advanced Sia API functionalities, including file redundancy checks and bandwidth monitoring, improving overall system reliability.

Fallback Mechanisms: Implemented fallback systems to ensure continued operation during Sia node unavailability or network limitations, guaranteeing uninterrupted backup and restore processes.

Cloud Backup API Enhancements

API Standardization: Finalized the standardization of API interactions, ensuring improved compatibility and a streamlined data handling process across modules.

Performance Optimizations: Continued refining the API’s scalability and performance, ensuring smooth operations even with large backup datasets.

Role-Based Access Control (RBAC): Strengthened security by further enhancing role-based permissions in the cloud backup API, enabling more granular access control.

Security and Compliance Advancements

Strengthened Encryption: Continued to enhance data encryption at rest and in transit, ensuring compliance with GDPR and industry standards.

Token-Based Authentication: Improved security by replacing session-based authentication with token-based authentication for all API interactions.

Penetration Testing & Security Audits: Performed extensive penetration testing to detect and resolve vulnerabilities. Completed a security audit to ensure all components are secure and compliant.

UI/UX Enhancements

Enhanced User Interface: Implemented new UI elements, including real-time backup status indicators, progress bars, and error notifications, providing a more intuitive user experience.

Dashboard Improvements: Refined the dashboard to display enhanced metrics, including Sia storage usage, capacity estimates, and backup job dependencies.

Multi-Language Support: Expanded the plugin’s accessibility by adding support for multiple languages, broadening the user base.

Testing and Documentation

System Stress Testing: Conducted extensive stress testing to ensure system stability, successfully handling even higher-than-expected data throughput.

Updated Documentation: Continued work on detailed user and developer documentation, focusing on the new Sia API integration and cloud backup API features.

Codebase Refinement: Refined the codebase for clarity and performance, performing static code analysis and enhancing the CI/CD pipeline.

  1. Key Technical Challenges and Resolutions

Sia API Integration Challenges

Challenge: Integrating the complex Sia API with the Proxmox Backup Server required handling asynchronous operations and network instability.

Resolution: Developed robust asynchronous handling techniques and fallback strategies to mitigate node unavailability and network latency.

UI Synchronization with Real-Time Data

Challenge: The real-time synchronization of backup statuses and capacity metrics led to occasional UI inconsistencies.

Resolution: Introduced improved caching mechanisms and refined the synchronization logic to ensure smooth UI updates.

Scalability and Load Balancing in Middleware

Challenge: Ensuring the middleware could handle the increased data throughput due to full Sia API integration.

Resolution: Optimized the middleware for enhanced load balancing and asynchronous processing, ensuring it can scale efficiently with increasing user demand.

  1. Upcoming Goals

Final Testing & Stable Release Preparation

Complete final rounds of beta testing with focus on real-world performance under large-scale conditions.

Resolve any outstanding bugs and performance issues, aiming for a stable release in early February.

Expanded Documentation and Community Engagement

Finalize and release comprehensive documentation covering the new Sia API integration and enhanced API features.

Publish the plugin on GitHub, inviting open-source contributions and feedback from the broader community.

Long-Term Maintenance & Future Integrations

Plan the integration of additional storage backends and work towards optimizing middleware scalability.

Explore potential partnerships with other decentralized storage providers to further enhance functionality.

UI/UX Finalization

Complete multi-language localization and continue refining the user interface, adding advanced visualizations for backup job dependencies and storage utilization.

Add contextual help and tutorials within the UI to improve user experience for both beginners and advanced users.

What was done in the code:

src/middleware/siaIntegrationMiddleware.rs: Enhanced error-handling logic and optimized integration with Sia API.

pbs/src/api2/cloud/backup.rs: Final adjustments to API to improve scalability and performance.

pbs-api-types/src/acl.rs: Updates to access control logic for more granular role-based permissions.

pbs-api-types/src/libs.rs: Optimizations to API types for better compatibility and data handling.

pbs-api-types/src/maintenance.rs: New logic for maintenance mode checks during cloud backup operations.

pbs-api-types/src/network.rs: Updated network configuration to better manage decentralized storage communications.

Links to repos worked on this month:

Merged branch ‘develop’ of https://github.com/AZComputerSolutions/PBSCloudBackupModule

What will you be working on this month?

Release Candidate Preparation:

Final Testing and Stable Release

Conduct final beta testing and resolve any remaining bugs or performance bottlenecks. Prepare for the stable release of the Proxmox Backup Server Cloud Backup Plugin with full Sia integration. Enhanced Documentation and Community Collaboration and Publish comprehensive user and developer documentation, including setup guides and troubleshooting resources. Release the plugin on GitHub and collaborate with the open-source community to gather feedback for future improvements.

UI/UX Enhancements

Finalize multi-language localization and add contextual help features to improve user experience. Continue refining the user interface, including advanced visualization tools for backup job dependencies and performance metrics.

Please summarize your development goals into a few sentences or bullet points:

We have documented setting up a test server. So that anyone can do it with the code in our github site. I would like to note that the Proxmox group has been less than helpful in helping us set up things so we can do testing before incorporating our code into their code base. We have setup a work around where we use an overlay system. I have been able to setup a test version of the product but there are still lots of bugs in it. We are currently working to remove bugs we are finding as we move towards an beta release and external test of our software.

  1. PBS Plus Package Structure (Debian)

Copy

pbs-plus_1.0_all.deb

├── etc/

│ ├── pbs-plus/

│ │ ├── overlay/ # Custom files to overlay

│ │ │ ├── usr/

│ │ │ │ └── share/

│ │ │ │ └── proxmox-backup-ui/

│ │ │ │ └── pbs-ui.js # Modified Ember components

│ │ ├── ssl/

│ │ │ └── pbs-plus.pem # Self-signed certificate

│ ├── systemd/system/

│ │ └── pbs-plus.service

├── DEBIAN/

│ ├── control

│ ├── postinst

│ └── prerm

  1. Overlay Mount Systemd Service

  2. Certificate Management

Copy

Post-install script (DEBIAN/postinst)

#!/bin/sh

update-ca-certificates --fresh

systemctl restart nginx

  1. Ember.js UI Customization

Since PBS uses Ember.js (v3.28+), modify these core files:

Target Files:

/usr/share/proxmox-backup-ui/js/pbs.js # Main entry point

/usr/share/proxmox-backup-ui/templates/*.hbs # Handlebar templates

/usr/share/proxmox-backup-ui/public/assets/ # CSS/Images

As always, the full version of this document is available at

https://www.pbscloudbackup.com/docs/sia/sia-foundation-monthly-report-jan-2025/

Screen Shots (more to come)