Standard Grant: Proxmox Backup Server Cloud Extension

Progress Report Form May 2024

What progress was made on your grant this month?

Last Month, We’ve done Javascript UI and apis. Now 90% completed

Phase 7: Build the cloud backup S3 Plugin:

  • Worked on the renterd API and UI connection API
  • Tested Encryption and decryption apis and working.
  • Worked on the UI development with javascript
  • Connection UI and apis

What was done in the code:

  • Worked on the renterd API and UI connection API

    • pbs/src/api2/cloud/backup.rs
    • pbs/src/api2/cloud/mod.rs
    • pbs/src/api2/cloud/restore.rs
  • CryptoUsers_code www/dashboard/cryptousers.js

  • Encrypt and decrypt hash algorithm implement www/cloud/window/Hashcrypt.js

  • CloudBackupJob, CloudBackup and restore api added www/cloud/window/CloudBackupJob.js

  • ACLEdit and directory , backupgroupchangeowner functions www/cloud/window/ACLEdit.js

  • EXtJS frontend pbscloud css www/css/ext6-pbscloud.css

  • Backup schedule and works ui www/cloud/backupworks/backupschedule.js

  • api works for pbs cloud backup src/cloud/test/alloc_writable_media.rs

  • Tested Encryption and decryption apis and working.

  • Worked on the UI development with javascript

  • Connection UI and apis

Links to repos worked on this month:

Merged branch ‘develop’ of https://github.com/AZComputerSolutions/PBSCloudBackupModule

What will you be working on this month?

  • 1. Continue Cloud backup S3 plugin and test module
    • Test Parameters and Functions
    • Test all apis and UI parts if any isses on it.
    • APi for backup data is deduplicated and encrypted
    • API for backup storage can be local or cloud storages, including clod storages
    • Backup data is deduplicated and encrypted.
    • Backup storage can be local or cloud storages, including cold storages.
    • Allows multiple clients to concurrently access a backup repository using lock-free operations.
    • Backups by default are append-only on the repository.
    • The operations are robustly designed and can be safely aborted and efficiently resumed.
    • Snapshot organization is possible by hostname, backup paths, label and tags. Also a rich set of metadata is saved with each snapshot.
    • Retention policies and cleaning of old backups can be highly customized.
    • Follow-up backups only process changed files, but still create a complete backup snapshot.
    • In-place restore only modifies files which are changed.

Please summarize your development goals into a few sentences or bullet points:

After talking with my development group a refactor of the Proxmox backup server code by Proxmox has necessitated additional work causing us to be behind this last month by about two weeks. This has pushed out the release of screen shots that were planned for the end of May into mid June. I believe we are still on budget, but slightly off-track time wise for this reason.

For more detail, please see the following link: https://www.pbscloudbackup.com/docs/sia/sia-foundation-update-may-2024/

Hello @rdwild

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Update: I should get an alpha iso copy of the software for testing early next week.

We are currently having issues compiling our iso due to an issue with proxmox code. We have reached out to them and hope to have a resolution soon. I will update here once that is the case.

Any updates on this? Thanks.

Sorry Steve, yes. I wanted to wait until I had the full facts (this will become evident why when you read on) before updating. Our primary developer ghosted us for the last three weeks to a month. I finally put a deadline on him returning to the project as of this past Monday which passed with no response from our primary developer. Feeling that something was going on and that we were behind on the project I brought in an additional developer about a month and a half ago just before we hit the issue of not being able to generate an ISO image of our project. My lead developer sent a note off to the proxmox team but did not get a response from them. About a week and a half later I reached out to the proxmox team to see if I could get an answer and got one. Which was to create a .deb file instead of an iso. I needed to give our developer that has been working on the project and did the costing research project some time to come back to the table but as of this last Monday and he did not. I reach out to our secondary developer yesterday after the deadline to see if he can finish the project. I won’t know for sure for a couple of days if that is the case or I need to bring in someone else. I will report back in a couple of days with an update. I did ask our backup dev if he can create an install .deb file for our project and he said yes. We are currently in a state of not knowing 100% where we are in the project timeline due to issues during the last period with our lead developer. I have asked our backup dev to update the project timeline once he has a better idea of what additional work needs to be done if any.

As soon as I have the .deb file I will be doing an install using it and will have screen shots available here once that is the case. Once we have done our base testing the .deb file and install instructions will be posted on our website and a link will be posted here to beta test the software.

Regards
Rodney Wild
A-Z Computer Solutions

Hello @rdwild

This is a reminder that your progress report is due.

Please be sure to follow the Progress Report Template and submit your progress report by the 2nd of each month. Thanks!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Our report is going to be delayed a couple of weeks due to the above situation. There is no point in just re-iterating what I have already posted above. I am waiting to hear back from my new dev on what he thinks is needed to get to the finish line. As soon as I know more, it will be posted here along with a project progress report.

Progress Report - August 2024

Cloud Backup Plugin Development for Proxmox Backup Server (Sia Integration)

  1. .deb Package Development

Task: Build the project into a Debian package.

Actions:

Executed the build process using the make .deb command within the project’s root directory.

Code Involved: Modified the Makefile to ensure all dependencies were correctly linked, specifically addressing issues related to the libcloudbackup.so library.

Outcome: Successfully generated a .deb package, ready for deployment on Debian-based systems, ensuring compatibility with Proxmox Backup Server environments.

  1. Test Server Configuration

Task: Setup a secure environment for testing the plugin.

Actions:

Configured SSH access to the test server by generating SSH keys and adding the public key to the ~/.ssh/authorized_keys on the remote server.

Code Involved: Modified the ssh_config file to restrict access only to specific IP addresses and ensure the use of SSH version 2 for enhanced security.

Outcome: Established a secure and stable connection to the test server, allowing for remote deployment and testing of the Cloud Backup plugin.

  1. UI Overhaul with ExtJS

Task: Redesign the user interface using the ExtJS framework.

Actions:

Rebuilt the UI components by creating new Ext.panel.Panel instances for the backup configuration screens, replacing the old hardcoded UI with a more modular approach.

Code Involved: Refactored UI logic in ui/main.js and ui/backupSettings.js to integrate with the new ExtJS components, enhancing maintainability and allowing for future updates without requiring a complete rebuild.

Outcome: Successfully migrated all UI elements to the new ExtJS framework, improving the UI’s responsiveness and reducing build dependencies.

  1. Cloud Backup Functionality Verification

Task: Test and verify the backup functionalities with Sia integration.

Actions:

Conducted end-to-end tests on the backupJob.js module to verify the proper creation, execution, and restoration of backups using the Sia decentralized storage.

Code Involved: Enhanced test scripts in tests/backupTest.js to include scenarios for failure handling and recovery, ensuring robust performance under various conditions.

Outcome: Verified that all cloud backup operations are functioning as expected, with successful uploads, retrievals, and error handling when interacting with the Sia network.

  1. DevOps Environment Setup

Task: Establish a DevOps environment for continuous integration and deployment.

Actions:

Set up CI/CD pipelines in the .gitlab-ci.yml file to automate the build, test, and deployment processes for Sia API integration and S3 bucket management.

Code Involved: Integrated the siaDeploy.sh script to automate the deployment of Sia-related services within Docker containers, ensuring consistent and reproducible environments.

Outcome: Achieved a fully automated DevOps pipeline, reducing manual intervention and increasing the speed and reliability of deployments.

  1. Proxmox Backup Server (PBS) Update Implementation Task: Implement and test the latest Proxmox Backup Server updates. Actions: Integrated the latest PBS updates by merging the upstream changes from the PBS-master branch into the project. Code Involved: Resolved merge conflicts in critical files like pbs-restore.js and pbs-api.js, ensuring that the Cloud Backup plugin remains fully compatible with the latest PBS features and APIs. Outcome: Successfully incorporated the latest PBS updates, enhancing the plugin’s compatibility and feature set.

Links to repos worked on this month:

What will you be working on this month?
This coming month we will be working on testing the .deb file and working on any bugs.
We will also be creating documentation for the project

Cloud backup development is ongoing and because of issues with our previous developer, we are behind by about a month. The previous developer had issues with generating an ISO which we were expecting to use for testing and the delay that it caused has caused us to run over budget. I have talked with our current developer who was helping our previous developer and he has now become our primary developer. He suggests that it is going to take another 80 hours to complete the project. A-Z Computer Solutions is committed to completing the project but at a slower pace due to the funding issues.

https://www.pbscloudbackup.com/docs/sia/sia-foundation-update-aug-2024/

Hey @rdwild, thanks for providing your August report. Please also provide your September report as that is also now late.

It’s ok if there isn’t much to write about since your last, but sticking to the schedule is important for us.

Ok sorry about the delay. I was still having communications issues with my current developer which I think I have worked out. I will have an updated project report Sunday Sept 22. End of day.

Progress Report Form Sept 2024

This last month we have been working on getting a .deb file so that we can install on a server. We have managed to do that and have started testing the work product produced by the previous developer. Now that we are testing my current developer is suggesting there

Links to repos worked on this month:
No repos have been worked on this month due to the above issue.
What will you be working on this month? Fixing bugs discovered now that we have a working backup server with cloud backup module.

Please summarize your development goals into a few sentences or bullet points:
Until we got the issue of the .deb file resolved it was not possible to put a timeline on completion. Now that we have solved the above issue I have asked our new developer what the timeline would be to complete the project and he is suggesting that it would take an additional two months of development time to complete the project. Although A-Z Computer Solutions is committed to completing the project at a much slower pace without the Sia Foundation we are asking for additional support from the Sia Foundation to complete the project. In order to complete the project in a timely manner we would need 384 hours ($13,440) which includes a 20% contingency (which is standard when costing projects). It is noted that no extra money will be forthcoming to the project manager and any money not used will be returned to the Sia Foundation.

Snapshot of project running on running on Proxmox Backup Server by our developer:

Progress Report Form Sept 2024

What progress was made on your grant this month?

  • Cloud Backup API Connection Fixes: Successfully identified and resolved issues in cloud backup list menu API connections, enabling proper communication between the frontend UI and backend services. Fixed broken functions and replaced the old tape APIs with updated Rust dependencies for improved performance and stability.
  • Middleware Development: Began development of a middleware layer to facilitate smooth interaction between the cloud backup APIs and the Sia decentralized storage APIs. This was necessitated after discovering direct connections between these APIs were failing, and the middleware will enable a reliable and efficient integration of Sia’s decentralized storage with the Proxmox cloud backup plugin.
  • Sia API Integration Testing: Successfully tested Sia API integration using Postman. This confirmed that basic API interactions, such as data upload and retrieval, are functional. However, challenges remain in creating a full connection between the cloud backup plugin and Sia’s decentralized storage APIs, which is being addressed through middleware development.
  • Codebase Maintenance and DevOps Improvements: Built and tested the existing codebase throughout the month. Also completed ExtJS build DevOps tasks, ensuring that the development environment is fully operational with optimized CI/CD pipelines.

Phase 8: Beta testing and bug cleanup:

  • API Connectivity: Initial testing revealed that the cloud backup APIs could not connect to Sia APIs due to missing middleware functionality. This led to a pivot in development focus, where middleware is now being built to bridge the gap between the cloud backup plugin and the Sia decentralized storage network.
  • Rust Dependencies: Encountered issues with outdated Rust dependencies that caused failures in the original cloud backup functions and tape APIs. These were replaced with updated libraries, improving API performance and reliability for storage functions.

What was done in the code:

  • Renter API and UI connection: Worked on improving the connection between the renter API and the UI for better performance.
  • Updated files: pbs/src/api2/cloud/backup.rs, pbs/src/api2/cloud/mod.rs, pbs/src/api2/cloud/restore.rs
  • Encryption and Decryption: Implemented hash encryption and decryption algorithms to secure data.
  • Relevant code: www/cloud/window/Hashcrypt.js
  • Cloud Backup API Development: Added backup and restore functionality in the cloud backup UI and connected APIs.
  • Code: www/cloud/window/CloudBackupJob.js
  • Access Control & Backup Group Functions: Enhanced the access control editing interface and introduced functions for changing backup group owners.
  • Files worked on: www/cloud/window/ACLEdit.js
  • UI and Styling Enhancements: Improved the UI elements and styling for the Proxmox Backup Server plugin.
  • Stylesheet: www/css/ext6-pbscloud.css
  • Backup Scheduling Features: Added scheduling options for cloud backups within the user interface.
  • Code updated: www/cloud/backupworks/backupschedule.js
  • API Work for PBS Cloud Backup: Tested the API connection for cloud backups, ensuring that encryption and decryption functionalities are working.
  • Test code: src/cloud/test/alloc_writable_media.rs
  • Frontend Development with ExtJS: Continued working on the user interface using JavaScript and ExtJS for improved interaction and visualization.

Links to repos worked on this month:

Merged branch ‘develop’ of https://github.com/AZComputerSolutions/PBSCloudBackupModule

What will you be working on this month?

Finalizing Middleware for Sia Integration: Complete the middleware development and test its ability to support full cloud backup functionality with Sia. This will include successful data backup and retrieval via the Sia API, followed by a visual demonstration (screenshots) of the working solution.

Please summarize your development goals into a few sentences or bullet points:

Completion of Builds and GitHub Updates: Finalize the latest build after the middleware and Sia integration testing is complete, and push the updates to the project’s GitHub repository.

Further Testing & Optimization: Conduct more in-depth tests on the entire plugin to ensure robust performance under different conditions. Optimize both the API and UI to ensure seamless use of Sia as a decentralized storage solution within the Proxmox Backup Server environment.

I would like to get an answer with regards to the additional funds requested in last month’s report. Without those funds we are going to have to slow down the progress of development. I would also suggest that if the project goes into slowdown mode there is a definite possibility of losing our current developer as he could possibility move on to another project.

https://www.pbscloudbackup.com/docs/sia/sia-foundation-monthy-report-sept-2024/

Hey Rodney, this is on our radar for today. Should we assume full-time work for the 380 hours? About 9-10 weeks?

That is Correct Steve.

Hey Rodney, the committee requested that you put together a proposal for the additional funds request. Use the Standard template and throw it in there. I’ll make sure it’s at the top of the list for reviews.

Thanx Steve, greatly appreciated.

New Proposal Posted here: Standard Grant Proposal: Proxmox Backup Cloud Extension (Continuation) as requested.

Progress Report Form Oct 2024

What progress was made on your grant this month?

 Middleware Development for Sia Integration:

 Cloud Backup API Enhancements:

 Integration Testing and Code Maintenance:

 User Interface (UI) Improvements:

Phase 8: Beta testing and bug cleanup:

Documentation and Repository Management:

We updated and merged our work on GitHub, including all major middleware and API connection changes, code maintenance tasks, and ExtJS UI updates.

Cloud Backup Middleware Development

Objective: Build middleware to enable interaction between Proxmox Backup Server’s cloud backup APIs and Sia’s decentralized storage API, addressing connectivity issues.

Changes:

Developed the middleware layer to route API calls effectively and provide consistent data handling across platforms.

Added error-handling routines to manage connectivity errors and log middleware actions for debugging.

What was done in the code:

src/middleware/siaIntegrationMiddleware.rs: Main middleware file for API requests.

src/middleware/siaIntegrationTest.js: Middleware test script for Sia API functionality checks.

API Connectivity Enhancements

Objective: Improve cloud backup API’s performance and security by updating dependencies and fixing broken functions.

Changes:Replaced outdated Rust dependencies with optimized libraries for faster and more stable performance.

Fixed broken functions within cloud backup API to ensure proper backend/frontend communication.

pbs/src/api2/cloud/backup.rs: Main cloud backup API file.

pbs/src/api2/cloud/mod.rs: Module configuration for cloud backup.

pbs/src/api2/cloud/restore.rs: Restoration logic for cloud backups.

Encryption & Decryption for Data Security

Objective: Secure data interactions by implementing hash encryption and decryption routines.

Changes:Added encryption methods to secure sensitive data during transmission between cloud backup UI and Sia storage.

www/cloud/window/Hashcrypt.js: Contains hash encryption and decryption methods.

Cloud Backup UI Development

Objective: Improve the cloud backup user experience through API integrations and UI styling enhancements.

Changes:Integrated backup and restore functionality directly into the cloud backup UI.

Enhanced UI and styling for improved user experience and accessibility.

www/cloud/window/CloudBackupJob.js: UI logic for backup jobs.

www/css/ext6-pbscloud.css: Stylesheet for cloud backup UI.

www/cloud/backupworks/backupschedule.js: UI for backup scheduling options.

Access Control and Backup Group Management

Objective: Enhance user access control settings and allow management of backup group ownership.

Changes:Introduced features for editing access control within the UI.

Added functionality to change backup group owners directly from the interface.

www/cloud/window/ACLEdit.js: Access control settings.

www/cloud/window/GroupManager.js: Management of backup groups.

ExtJS Frontend Development and API Testing

Objective: Ensure compatibility and test the UI for Sia integration using ExtJS and API testing tools.

Changes:Continued refining the ExtJS frontend for seamless user interactions with the Sia API.

Conducted API connection tests to validate functionality for cloud backups.

src/cloud/test/alloc_writable_media.rs: Test file for API connections.

www/cloud/window/UIComponents.js: Various UI components.

Links to repos worked on this month:

Merged branch ‘develop’ of https://github.com/AZComputerSolutions/PBSCloudBackupModule

What will you be working on this month?

Beta Testing and Bug Fixes:

 Test Server Setup and Deployment:

 Middleware Enhancements:

 UI Refinements and User Experience Improvements:

 Documentation and GitHub Management

 Please summarize your development goals into a few sentences or bullet points:

Fix bugs and code dependencies changes in rust that make our code no longer compile . Get a test server setup and working. Document setting up a test server. So that anyone can do it would the code in our github site. I would also like to note that because of the pending approval by the Sia foundation (thank-you for your continued support) we have been running at full development speed this last month.

We were ask by the Sia foundation to shorten our monthly reports but as always, the full version of this document is available at https://www.pbscloudbackup.com/docs/sia/sia-foundation-monthly-report-oct-2024/