Standard Grant: Proxmox Backup Server Cloud Extension

[rdwild]

Progress Report Form Sept 2024

This last month we have been working on getting a .deb file so that we can install on a server. We have managed to do that and have started testing the work product produced by the previous developer. Now that we are testing my current developer is suggesting there

Links to repos worked on this month:
No repos have been worked on this month due to the above issue.
What will you be working on this month? Fixing bugs discovered now that we have a working backup server with cloud backup module.

Please summarize your development goals into a few sentences or bullet points:
Until we got the issue of the .deb file resolved it was not possible to put a timeline on completion. Now that we have solved the above issue I have asked our new developer what the timeline would be to complete the project and he is suggesting that it would take an additional two months of development time to complete the project. Although A-Z Computer Solutions is committed to completing the project at a much slower pace without the Sia Foundation we are asking for additional support from the Sia Foundation to complete the project. In order to complete the project in a timely manner we would need 384 hours ($13,440) which includes a 20% contingency (which is standard when costing projects). It is noted that no extra money will be forthcoming to the project manager and any money not used will be returned to the Sia Foundation.

Snapshot of project running on running on Proxmox Backup Server by our developer:

[rdwild]

Progress Report Form Sept 2024

What progress was made on your grant this month?

• Cloud Backup API Connection Fixes: Successfully identified and resolved issues in cloud backup list menu API connections, enabling proper communication between the frontend UI and backend services. Fixed broken functions and replaced the old tape APIs with updated Rust dependencies for improved performance and stability.
• Middleware Development: Began development of a middleware layer to facilitate smooth interaction between the cloud backup APIs and the Sia decentralized storage APIs. This was necessitated after discovering direct connections between these APIs were failing, and the middleware will enable a reliable and efficient integration of Sia’s decentralized storage with the Proxmox cloud backup plugin.
• Sia API Integration Testing: Successfully tested Sia API integration using Postman. This confirmed that basic API interactions, such as data upload and retrieval, are functional. However, challenges remain in creating a full connection between the cloud backup plugin and Sia’s decentralized storage APIs, which is being addressed through middleware development.
• Codebase Maintenance and DevOps Improvements: Built and tested the existing codebase throughout the month. Also completed ExtJS build DevOps tasks, ensuring that the development environment is fully operational with optimized CI/CD pipelines.

Phase 8: Beta testing and bug cleanup:

• API Connectivity: Initial testing revealed that the cloud backup APIs could not connect to Sia APIs due to missing middleware functionality. This led to a pivot in development focus, where middleware is now being built to bridge the gap between the cloud backup plugin and the Sia decentralized storage network.
• Rust Dependencies: Encountered issues with outdated Rust dependencies that caused failures in the original cloud backup functions and tape APIs. These were replaced with updated libraries, improving API performance and reliability for storage functions.

What was done in the code:

• Renter API and UI connection: Worked on improving the connection between the renter API and the UI for better performance.
• Updated files: pbs/src/api2/cloud/backup.rs, pbs/src/api2/cloud/mod.rs, pbs/src/api2/cloud/restore.rs
• Encryption and Decryption: Implemented hash encryption and decryption algorithms to secure data.
• Relevant code: www/cloud/window/Hashcrypt.js
• Cloud Backup API Development: Added backup and restore functionality in the cloud backup UI and connected APIs.
• Code: www/cloud/window/CloudBackupJob.js
• Access Control & Backup Group Functions: Enhanced the access control editing interface and introduced functions for changing backup group owners.
• Files worked on: www/cloud/window/ACLEdit.js
• UI and Styling Enhancements: Improved the UI elements and styling for the Proxmox Backup Server plugin.
• Stylesheet: www/css/ext6-pbscloud.css
• Backup Scheduling Features: Added scheduling options for cloud backups within the user interface.
• Code updated: www/cloud/backupworks/backupschedule.js
• API Work for PBS Cloud Backup: Tested the API connection for cloud backups, ensuring that encryption and decryption functionalities are working.
• Test code: src/cloud/test/alloc_writable_media.rs
• Frontend Development with ExtJS: Continued working on the user interface using JavaScript and ExtJS for improved interaction and visualization.

Links to repos worked on this month:

Merged branch ‘develop’ of https://github.com/AZComputerSolutions/PBSCloudBackupModule

What will you be working on this month?

Finalizing Middleware for Sia Integration: Complete the middleware development and test its ability to support full cloud backup functionality with Sia. This will include successful data backup and retrieval via the Sia API, followed by a visual demonstration (screenshots) of the working solution.

Please summarize your development goals into a few sentences or bullet points:

Completion of Builds and GitHub Updates: Finalize the latest build after the middleware and Sia integration testing is complete, and push the updates to the project’s GitHub repository.

Further Testing & Optimization: Conduct more in-depth tests on the entire plugin to ensure robust performance under different conditions. Optimize both the API and UI to ensure seamless use of Sia as a decentralized storage solution within the Proxmox Backup Server environment.

I would like to get an answer with regards to the additional funds requested in last month’s report. Without those funds we are going to have to slow down the progress of development. I would also suggest that if the project goes into slowdown mode there is a definite possibility of losing our current developer as he could possibility move on to another project.

https://www.pbscloudbackup.com/docs/sia/sia-foundation-monthy-report-sept-2024/

[steve]

Hey Rodney, this is on our radar for today. Should we assume full-time work for the 380 hours? About 9-10 weeks?

[rdwild]

That is Correct Steve.

[steve]

Hey Rodney, the committee requested that you put together a proposal for the additional funds request. Use the Standard template and throw it in there. I’ll make sure it’s at the top of the list for reviews.

[rdwild]

Thanx Steve, greatly appreciated.

[rdwild]

New Proposal Posted here: Standard Grant Proposal: Proxmox Backup Cloud Extension (Continuation) as requested.

[rdwild]

Standard Grant Proposal: Proxmox Backup Cloud Extension (Continuation)

Thanks for your latest proposal to The Sia Foundation Grants Program. After review, the committee has decided to approve your proposal. They’re definitely excited and hopeful to see the completion of your work so far. We’ll reach out to your provided email address for onboarding. Obviously this shouldn’t take too long as this is a continuation of your previous grant.

[rdwild]

Progress Report Form Oct 2024

What progress was made on your grant this month?

 Middleware Development for Sia Integration:

 Cloud Backup API Enhancements:

 Integration Testing and Code Maintenance:

 User Interface (UI) Improvements:

Phase 8: Beta testing and bug cleanup:

Documentation and Repository Management:

We updated and merged our work on GitHub, including all major middleware and API connection changes, code maintenance tasks, and ExtJS UI updates.

Cloud Backup Middleware Development

Objective: Build middleware to enable interaction between Proxmox Backup Server’s cloud backup APIs and Sia’s decentralized storage API, addressing connectivity issues.

Changes:

Developed the middleware layer to route API calls effectively and provide consistent data handling across platforms.

Added error-handling routines to manage connectivity errors and log middleware actions for debugging.

What was done in the code:

src/middleware/siaIntegrationMiddleware.rs: Main middleware file for API requests.

src/middleware/siaIntegrationTest.js: Middleware test script for Sia API functionality checks.

API Connectivity Enhancements

Objective: Improve cloud backup API’s performance and security by updating dependencies and fixing broken functions.

Changes:Replaced outdated Rust dependencies with optimized libraries for faster and more stable performance.

Fixed broken functions within cloud backup API to ensure proper backend/frontend communication.

pbs/src/api2/cloud/backup.rs: Main cloud backup API file.

pbs/src/api2/cloud/mod.rs: Module configuration for cloud backup.

pbs/src/api2/cloud/restore.rs: Restoration logic for cloud backups.

Encryption & Decryption for Data Security

Objective: Secure data interactions by implementing hash encryption and decryption routines.

Changes:Added encryption methods to secure sensitive data during transmission between cloud backup UI and Sia storage.

www/cloud/window/Hashcrypt.js: Contains hash encryption and decryption methods.

Cloud Backup UI Development

Objective: Improve the cloud backup user experience through API integrations and UI styling enhancements.

Changes:Integrated backup and restore functionality directly into the cloud backup UI.

Enhanced UI and styling for improved user experience and accessibility.

www/cloud/window/CloudBackupJob.js: UI logic for backup jobs.

www/css/ext6-pbscloud.css: Stylesheet for cloud backup UI.

www/cloud/backupworks/backupschedule.js: UI for backup scheduling options.

Access Control and Backup Group Management

Objective: Enhance user access control settings and allow management of backup group ownership.

Changes:Introduced features for editing access control within the UI.

Added functionality to change backup group owners directly from the interface.

www/cloud/window/ACLEdit.js: Access control settings.

www/cloud/window/GroupManager.js: Management of backup groups.

ExtJS Frontend Development and API Testing

Objective: Ensure compatibility and test the UI for Sia integration using ExtJS and API testing tools.

Changes:Continued refining the ExtJS frontend for seamless user interactions with the Sia API.

Conducted API connection tests to validate functionality for cloud backups.

src/cloud/test/alloc_writable_media.rs: Test file for API connections.

www/cloud/window/UIComponents.js: Various UI components.

Links to repos worked on this month:

Merged branch ‘develop’ of https://github.com/AZComputerSolutions/PBSCloudBackupModule

What will you be working on this month?

Beta Testing and Bug Fixes:

 Test Server Setup and Deployment:

 Middleware Enhancements:

 UI Refinements and User Experience Improvements:

 Documentation and GitHub Management

 Please summarize your development goals into a few sentences or bullet points:

Fix bugs and code dependencies changes in rust that make our code no longer compile . Get a test server setup and working. Document setting up a test server. So that anyone can do it would the code in our github site. I would also like to note that because of the pending approval by the Sia foundation (thank-you for your continued support) we have been running at full development speed this last month.

We were ask by the Sia foundation to shorten our monthly reports but as always, the full version of this document is available at https://www.pbscloudbackup.com/docs/sia/sia-foundation-monthly-report-oct-2024/

[rdwild]

PBSCloud Backup Progress Report Nov 2024

What progress was made on your grant this month?

• Sia Middleware Proxmox
• Cloud Backup API Refinements:
• Advanced Security and Compliance Measures:
• UI Enhancements and New Features: Integration Testing and Pre-release Documentation:
• Repository Optimization and Code Consolidation:
• Sia Middleware Proxmox
• Cloud Backup API Refinements:
• Advanced Security and Compliance Measures:
• UI Enhancements and New Features:
• Integration Testing and Pre-release Documentation:
• Repository Optimization and Code Consolidation:

Phase 8: Beta testing and bug cleanup:

Documentation and Repository Management: We updated and merged our work on GitHub, including all major middleware and API connection changes, code maintenance tasks, and ExtJS UI updates.

Cloud Backup Middleware Development Objective: Build middleware to enable interaction between Proxmox Backup Server’s cloud backup APIs and Sia’s decentralized storage API, addressing connectivity issues.

Changes:

Developed the middleware layer to route API calls effectively and provide consistent data handling across platforms.

Added error-handling routines to manage connectivity errors and log middleware actions for debugging.

What was done in the code:

src/middleware/siaIntegrationMiddleware.rs: Modularized middleware and added caching logic. src/middleware/loadBalancer.rs: Implemented dynamic load balancing for middleware operations.

pbs/src/api2/cloud/backup.rs: Integrated asynchronous processing and analytics reporting. pbs/src/api2/cloud/auth.rs: Added token-based authentication.

www/cloud/security/AuditLogger.js: Implemented logging for critical actions. www/cloud/security/EncryptionHandler.js: Enhanced encryption algorithms.

www/cloud/window/StorageDashboard.js: Created Sia storage monitoring dashboard. www/css/ext6-pbscloud.css: Updated styles for localization and accessibility. www/cloud/backupworks/dependencyMap.js: Added dependency visualization features.

Links to repos worked on this month:

Merged branch ‘develop’ of https://github.com/AZComputerSolutions/PBSCloudBackupModule

What will you be working on this month?

Release Candidate Preparation:

Resolve final bugs and ensure all features meet performance benchmarks.

Conduct a soft launch with select beta testers.

Long-term Maintenance Planning:

Develop a roadmap for supporting additional storage backends.

Document middleware architecture for handoff and future scalability.

Please summarize your development goals into a few sentences or bullet points:

Fix bugs and code dependencies changes in rust that make our code no longer compile . Get a test server setup and working. Document setting up a test server. So that anyone can do it would the code in our github site.

As always, the full version of this document is available at https://www.pbscloudbackup.com/docs/sia/sia-foundation-monthly-report-nov-2024/

[rdwild]

Sia Foundation Monthly Report – Dec 2024

What progress was made on your grant this month?

 **Middleware Completion and Advanced Sia Integration**

 **Expanded Cloud Backup API Capabilities**

 **Sia API Enhancements and Optimization**

 **Security and Compliance Advancements**

 **User Interface (UI) Improvements**

 **Phase 8: Beta testing and bug cleanup:**

 **Middleware Scalability**

Challenge: The middleware faced delays under high loads due to inadequate request handling.

Resolution: Implemented asynchronous request processing and dynamic load balancing to ensure consistent performance under heavy data throughput.

Sia Network Constraints

Challenge: Node unavailability and bandwidth limitations impacted decentralized storage performance.

Resolution: Developed robust fallback mechanisms and optimized Sia API interactions for redundancy and reliability.

UI Synchronization Issues

Challenge: Real-time status updates and dependency visualizations caused inconsistencies in the user interface.

Resolution: Integrated advanced caching and synchronous update mechanisms to maintain data consistency across the UI.

What was done in the code:

src/middleware/siaIntegrationMiddleware.rs: Maintenance mode adaptation and improved error-handling logic.

Cloud Backup API Enhancements

Updated API types to standardize interactions and improve compatibility across modules.

Made significant updates to the cloud backup API, optimizing data handling and request/response cycles.

Enhanced access control functionalities with updates to the acl.rs file, introducing more granular permissions.

Files Updated:

pbs/src/api2/cloud/backup.rs: Cloud backup API updates to improve efficiency and scalability.

File Modified:

pbs-api-types/src/acl.rs: Access control updates for better security and role-based permissions.

pbs-api-types/src/libs.rs: Basic API types used by most of the cloud backup code.

pbs-api-types/src/matintenance.rs: Checks the current maintenance mode against an attempted operation.

pbs-api-types/src/network.rs: Configuration is done manually using other tools and network interface type.

Links to repos worked on this month:

Merged branch ‘develop’ of https://github.com/AZComputerSolutions/PBSCloudBackupModule

What will you be working on this month?

Release Candidate Preparation:

Final Testing and Stable Release

Conduct final beta testing and resolve any remaining bugs or performance bottlenecks. Prepare for the stable release of the Proxmox Backup Server Cloud Backup Plugin with full Sia integration. Enhanced Documentation and Community Collaboration and Publish comprehensive user and developer documentation, including setup guides and troubleshooting resources. Release the plugin on GitHub and collaborate with the open-source community to gather feedback for future improvements.

Long-Term Maintenance and Scalability

Plan for the integration of additional storage backends and further middleware scalability. Initiate discussions for partnerships with other decentralized storage solutions to expand functionality.

UI/UX Enhancements

Finalize multi-language localization and add contextual help features to improve user experience. Continue refining the user interface, including advanced visualization tools for backup job dependencies and performance metrics.

Please summarize your development goals into a few sentences or bullet points:

Fix bugs and code dependencies changes in rust that make our code no longer compile . Get a test server setup and working. Document setting up a test server. So that anyone can do it would the code in our github site.

As always, this document has been shortened greatly, please see the full version of this document is available at https://www.pbscloudbackup.com/docs/sia/pbscloud-backup-progress-report-dec-2024/

[Kinomora]

Hello,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee