Standard Grant: Proxmox Backup Server Cloud Extension

Progress Report for the PBS Cloud Backup module - Jan 2024

What progress was made on your grant this month?

Phase 4. Study the PBS tape feature:
• Acknowledging the significant role of layers in distinguishing and differentiating between various reading sources.
• Required understanding how different layers contribute to the processing and interpretation of information from different inputs. So that the same could be applied in cloud backup.

Phase 5. Build the cloud backup module:
• Setup renterd configuration
• Accessed the renterd portal
• Worked on the renterd API with client
• Understanding the underlying protocols, encryption methods, and data transfer processes utilized by Proxmox VE to facilitate efficient and secure backup operations
• Started making code changes for the cloud backup module.
• Created a cloud backup api stub endpoint
• Assessing Proxmox for supporting multiple cloud providers via plugins

Accomplishments for the Month:
• Moved towards building the cloud backup feature, started the initial phase of the development
The Following tasks were closed this month in openproject (project management tool) :
• Dive deep into tape_read, read, and block_reader traits
• api.rs
• backup.rs
• restore.rs
• Understand how the implementation is reading chunks
• Understand the backup flow of tape feature
• Grasp the abstraction layer functionality
• Acknowledge the layer’s role in differentiating between reading sources
• Study the write feature for trait implementation
• Study the restore feature for trait implementation
• Study the backup schedule and triggering functionality
• Study the feature for understanding incremental file saving approach
• Study the proxmox backup protocol
• Develop a Plugin-Enabled Module for Cloud Storage
• Architectural Consideration
• Implement the Strategy Pattern
• Understand its role in algorithm encapsulation and interchangeability
• Implement Restore Functionality
• Enable file-level and snapshot-level restores
• Allow selective restores based on timestamps, file types, etc
• Implement Backup Functionality
• Support full and incremental backups
• Write incremental, fully deduplicated backup files
• Manage versioning and ensure data integrity during backups

Links to repos worked on this month:
GitHub - AZComputerSolutions/PBSCloudBackupModule

What will you be working on this month?

During this month the backup module’s development would be continued, which will include:
Implementation of Backup functionality
• Adding of the cloud backup module
• Implement the cloud backup api under the path (/api2/json/cloud/backup)
• Add the following functionality in relation to the cloud backup module:
1. Modify the PoolWriter, CatalogSet, NewChunksIterator for the cloud module.
2. Implement the releveant functionality and removing existing tape functionality, like MediaPool, used_tapes, etc.
• Following structs/traits need to be implemented in relation to the cloud backup functionality in api2/cloud/backup.rs:
1. TapeBackupJobStatus → CloudBackupJobStatus
2. TapeBackupJobConfig → CloudBackupJobConfig
3. TapeBackupJobSetup → CloudBackupJobSetup
4. TapeBackupJobStatus → CloudBackupJobStatus
5. TapeBackupJobSummary → CloudBackupJobSummary
6. use::crate::tape → change to relevant cloud trait and implement the functionality
7. TAPE_BACKUP_JOB_ROUTER → CLOUD_BACKUP_JOB_ROUTER
8. tape-backup-job → cloud-backup-job
9. do_tape_backup_job → do_cloud_bakcup_job
10. backup_job.setup → implement relevant cloud functionality, to accept cloud jobs
11. let drive_lock = lock_tape_device(&drive_config, &setup.drive)?; → tape locking and device locking functionality needs to be removed
12. let _ = set_tape_device_state(&setup.drive, “”); → tape status needs to be removed
13. TAPE_STATUS_DIR → needs to be removed
14. pool_writer.set_media_status_full(&uuid)?; → This would be done along with the CatalogSet and NewChunksIterator as mentioned above
15. bail!(“Tape backup finished with some errors. Please check the task log.”); → Update the notifications according to the notifications from S3 network
16. summary.used_tapes = match pool_writer.get_used_media_labels() → after the classes, mentioned above are modified, this would be implemented according to the cloud

Currently the development is being done in an isolated repository. As mentioned in the risks initially, there are chances that this S3 cloud backup feature does not get accepted into the code base of proxmox so in the short term we have elected to isolate the code until it does get accepted.

Development of the proxmox backup feature has started, although slow, the cloud api endpoint stub has been implemented using the same code structure as the tape backup feature.

Sia Foundation monthly update Feb 2024
Progress Report for the PBS Cloud Backup module

What progress was made on your grant this month?

Progress has been slow due to some health issues with our developer and some problems faced in code understanding.

Phase 5. Build the cloud backup module:

  1. What was done in the code:
    • added /api2/json/cloud/backup endpoint
    • copied tape/backup method inside cloud/backup without changes
    • added src\cloud\pool_writer folder
    • added src/api2/cloud/backup.rs and added ancillary structs
    • added method bakcup and backup_worker in src/api2/cloud/backup.rs
    • added pbs-api-types/src/cloud/mod.rs
    • Added pbs-config/cloud_job.rs
    • Implement the relevant cloud functionality and removed existing tape functionality, like MediaPool, used_tapes, etc.

  2. What was done in DevOps:
    • Setup the renterd service again
    • Worked on the renterd client to upload file through renterd API: renterd

Accomplishments for the Month:

• Not much has been achieved accept for fixing the error on adding cloud backup structs, similar to tape backup structs.
• One of the milestones which were discussed were to figure out how will the rust-S3 plugin be used to interact with the sia S3 cloud storage network:

sia backup diagram 1

Architectural discussion was done to ensure the right architecture is being implemented, the following is what will be implemented:

sia backup diagram 2

The Following tasks were closed this month in openproject (project management tool) :

• Construct Traits for Client Functionality
• Implement Backup Functionality
• Implement List Functionality
• Reading and Management of Snapshots
Links to repos worked on this month:
GitHub - AZComputerSolutions/PBSCloudBackupModule
What will you be working on this month?

During this month the backup module’s development would be continued, which will include:
During the next month(March) the backup module’s development would be continued, which will include, the unfinished tasks from last month:

Implementation of Backup functionality

• Following structs/traits need to be implemented in relation to the cloud backup functionality in api2/cloud/backup.rs:

i. TapeBackupJobStatus → CloudBackupJobStatus
ii. TapeBackupJobConfig → CloudBackupJobConfig
iii. TapeBackupJobSetup → CloudBackupJobSetup
iv. TapeBackupJobStatus → CloudBackupJobStatus
v. TapeBackupJobSummary → CloudBackupJobSummary
vi. use::crate::tape → change to relevant cloud trait and implement the functionality
vii. TAPE_BACKUP_JOB_ROUTER → CLOUD_BACKUP_JOB_ROUTER
viii. tape-backup-job → cloud-backup-job
ix. do_tape_backup_job → do_cloud_bakcup_job
x. backup_job.setup → implement relevant cloud functionality, to accept cloud jobs
xi. let drive_lock = lock_tape_device(&drive_config, &setup.drive)?; → tape locking and device locking functionality needs to be removed
xii. let _ = set_tape_device_state(&setup.drive, “”); → tape status needs to be removed
xiii. TAPE_STATUS_DIR → needs to be removed
xiv. pool_writer.set_media_status_full(&uuid)?; → This would be done along with the CatalogSet and NewChunksIterator as mentioned above
xv. bail!(“Tape backup finished with some errors. Please check the task log.”); → Update the notifications according to the notifications from S3 network
xvi. summary.used_tapes = match pool_writer.get_used_media_labels() → after the classes, mentioned above are modified, this would be implemented according to the cloud

Currently the development is being done in an isolated repository. As mentioned in the risks initially, there are chances that this S3 cloud backup feature does not get accepted into the code base of proxmox, so in the short term we have elected to isolate the code until it does get accepted. There was also an issue where the developer was using renterd api which needed to be corrected to use the rust-s3 create instead. This was caught and corrected early (due to weekly reporting) and the developer is now back on track.

Progress Report for the PBS Cloud Backup module - Feb 2024
What progress was made on your grant this month?
Progress has been slow due to some health issues with our developer and some problems faced in code understanding.
Phase 5. Build the cloud backup module:

  1. What was done in the code:
    • added /api2/json/cloud/backup endpoint
    • copied tape/backup method inside cloud/backup without changes
    • added src\cloud\pool_writer folder
    • added src/api2/cloud/backup.rs and added ancillary structs
    • added method bakcup and backup_worker in src/api2/cloud/backup.rs
    • added pbs-api-types/src/cloud/mod.rs
    • Added pbs-config/cloud_job.rs
    • Implement the relevant cloud functionality and removed existing tape functionality, like MediaPool, used_tapes, etc.
  2. What was done in DevOps:
    • Setup the renterd service again
    • Worked on the renterd client to upload file through renterd API: renterd

Accomplishments for the Month:
• Not much has been achieved accept for fixing the error on adding cloud backup structs, similar to tape backup structs.
• One of the milestones which were discussed were to figure out how will the rust-S3 plugin be used to interact with the sia S3 cloud storage network:

Architectural discussion was done to ensure the right architecture is being implemented, the following is what will be implemented:

The Following tasks were closed this month in openproject (project management tool) :

• Construct Traits for Client Functionality
• Implement Backup Functionality
• Implement List Functionality
• Reading and Management of Snapshots
Links to repos worked on this month:
GitHub - AZComputerSolutions/PBSCloudBackupModule
What will you be working on this month?

During this month the backup module’s development would be continued, which will include:
During the next month(March) the backup module’s development would be continued, which will include, the unfinished tasks from last month:
Implementation of Backup functionality
• Following structs/traits need to be implemented in relation to the cloud backup functionality in api2/cloud/backup.rs:
i. TapeBackupJobStatus → CloudBackupJobStatus
ii. TapeBackupJobConfig → CloudBackupJobConfig
iii. TapeBackupJobSetup → CloudBackupJobSetup
iv. TapeBackupJobStatus → CloudBackupJobStatus
v. TapeBackupJobSummary → CloudBackupJobSummary
vi. use::crate::tape → change to relevant cloud trait and implement the functionality
vii. TAPE_BACKUP_JOB_ROUTER → CLOUD_BACKUP_JOB_ROUTER
viii. tape-backup-job → cloud-backup-job
ix. do_tape_backup_job → do_cloud_bakcup_job
x. backup_job.setup → implement relevant cloud functionality, to accept cloud jobs
xi. let drive_lock = lock_tape_device(&drive_config, &setup.drive)?; → tape locking and device locking functionality needs to be removed
xii. let _ = set_tape_device_state(&setup.drive, “”); → tape status needs to be removed
xiii. TAPE_STATUS_DIR → needs to be removed
xiv. pool_writer.set_media_status_full(&uuid)?; → This would be done along with the CatalogSet and NewChunksIterator as mentioned above
xv. bail!(“Tape backup finished with some errors. Please check the task log.”); → Update the notifications according to the notifications from S3 network
xvi. summary.used_tapes = match pool_writer.get_used_media_labels() → after the classes, mentioned above are modified, this would be implemented according to the cloud

Currently the development is being done in an isolated repository. As mentioned in the risks initially, there are chances that this S3 cloud backup feature does not get accepted into the code base of proxmox, so in the short term we have elected to isolate the code until it does get accepted. There was also an issue where the developer was using renterd api which needed to be corrected to use the rust-s3 create instead. This was caught and corrected early (due to weekly reporting) and the developer is now back on track.

https://www.pbscloudbackup.com/docs/sia/sia-foundation-update-feb-2024/

Hello @rdwild,

Thank you for your monthly Progress Report, however, you have strayed a bit from what we ask for when it comes to the reports.

These reports are useful for the team to get a quick at-a-glance summary of work done in the last month, and helps us and the committee be informed on the progress of the work.

We do not require grantees to write long, in-depth progress reports because it spends unnecessary time away from your own project, as well as the committee’s time who reviews these every month.

We appreciate the dedication to in-depth details, however, we do ask that in the future progress reports are kept to 3-5 brief bullet pointed statements per section (with 3 main sections- progress this month, issues encountered this month, goals next month, as well as simple links to updated repos). You can find the progress report template here: About the Grants category - #8

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

I’ll provide less detail in future reports with the ability to click on a link below for more details if needed.

Progress Report Form March 2024

What progress was made on your grant this month?

Phase 6. Continue Build the cloud backup module:

What was done in the code:

  • added /pbs-api-types/src/cloud/*
  • modified /examples/completion.rs
  • modified examples/dynamic-files.rs
  • Implement the relevant cloud functionality and removed existing tape functionality, like media , media_location , drive , etc.

Links to repos worked on this month:

Merged branch ‘develop’ of [**GitHub - AZComputerSolutions/PBSCloudBackupModule)

What will you be working on this month?

During the next month(April) the backup module’s development would be continued, which will include, the unfinished tasks from last month and Testing and Debugging completed functions:

Implementation of Backup functionality

modify backup_readers for cloudbackup

modify backup_repo for cloudbackup

modify backup_specification for cloudbackup

modify backup_writers for cloudbackup

add cloudbackup_info for pbs-database

add cloudbackup_key for proxmox-backup_client

complete cloud_writer for cloud_backup_written datas

Test cloud_backup functions with api

Debug cloud_backup functions with api

Please summarize your development goals into a few sentences or bullet points:

Cloud backup development is on-going and on track.

For more detail please see the following link: Sia Foundation Update – Mar 2024 – PBS Cloud Backup

Hey @rdwild, looks like that linked repo is private. Mind opening that up for us?

Progress Report Form April 2024

What progress was made on your grant this month?

Apis and UI for Proxmox backup server cloud part done.
Tested apis

Phase 6. Continue Build the cloud backup module:

What was done in the code:

  • www/cloud/form
    AllocationSelector , ChangerSelector , CloudDevicePathSelector
    DriveSelector , KeySelector , PoolSelector and RetentionSelector is done
  • www/cloud/window
    ChangerEdit , DriveEdit , EncryptionEdit , Erase , LabelMedia, MediaRemoveWindow , PoolEdit window is done
    BackupJobs , BackupOverview, ChangerConfig, ChangeStatus , Cloudinventory and CloudManagement ui is done.Links to repos worked on this month:

Merged branch ‘develop’ of GitHub - AZComputerSolutions/PBSCloudBackupModule

What will you be working on this month?

Finishing remain ui and apis , testing all apis for matching ui for cloudbackup

Implementation of Backup functionality

  • modify backupjobs for cloudbackup
  • modify backup_restore for cloudbackup
  • testing backup cryption apis for cloudbackup

Please summarize your development goals into a few sentences or bullet points:

Cloud backup development is on-going and on track. Based on conversations with my developer we will be starting plugin development end of May and are on schedule. Unless there is any issues that pop up, we should have some screenshots end of may. They will be posted here and on the pbscloudbackup website.

For more detail please see the following link: https://www.pbscloudbackup.com/docs/sia/sia-foundation-update-april-2024/

Hello @rdwild,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

There is going to be a slight delay (a couple of days) in this month’s report.

Progress Report Form May 2024

What progress was made on your grant this month?

Last Month, We’ve done Javascript UI and apis. Now 90% completed

Phase 7: Build the cloud backup S3 Plugin:

  • Worked on the renterd API and UI connection API
  • Tested Encryption and decryption apis and working.
  • Worked on the UI development with javascript
  • Connection UI and apis

What was done in the code:

  • Worked on the renterd API and UI connection API

    • pbs/src/api2/cloud/backup.rs
    • pbs/src/api2/cloud/mod.rs
    • pbs/src/api2/cloud/restore.rs
  • CryptoUsers_code www/dashboard/cryptousers.js

  • Encrypt and decrypt hash algorithm implement www/cloud/window/Hashcrypt.js

  • CloudBackupJob, CloudBackup and restore api added www/cloud/window/CloudBackupJob.js

  • ACLEdit and directory , backupgroupchangeowner functions www/cloud/window/ACLEdit.js

  • EXtJS frontend pbscloud css www/css/ext6-pbscloud.css

  • Backup schedule and works ui www/cloud/backupworks/backupschedule.js

  • api works for pbs cloud backup src/cloud/test/alloc_writable_media.rs

  • Tested Encryption and decryption apis and working.

  • Worked on the UI development with javascript

  • Connection UI and apis

Links to repos worked on this month:

Merged branch ‘develop’ of https://github.com/AZComputerSolutions/PBSCloudBackupModule

What will you be working on this month?

  • 1. Continue Cloud backup S3 plugin and test module
    • Test Parameters and Functions
    • Test all apis and UI parts if any isses on it.
    • APi for backup data is deduplicated and encrypted
    • API for backup storage can be local or cloud storages, including clod storages
    • Backup data is deduplicated and encrypted.
    • Backup storage can be local or cloud storages, including cold storages.
    • Allows multiple clients to concurrently access a backup repository using lock-free operations.
    • Backups by default are append-only on the repository.
    • The operations are robustly designed and can be safely aborted and efficiently resumed.
    • Snapshot organization is possible by hostname, backup paths, label and tags. Also a rich set of metadata is saved with each snapshot.
    • Retention policies and cleaning of old backups can be highly customized.
    • Follow-up backups only process changed files, but still create a complete backup snapshot.
    • In-place restore only modifies files which are changed.

Please summarize your development goals into a few sentences or bullet points:

After talking with my development group a refactor of the Proxmox backup server code by Proxmox has necessitated additional work causing us to be behind this last month by about two weeks. This has pushed out the release of screen shots that were planned for the end of May into mid June. I believe we are still on budget, but slightly off-track time wise for this reason.

For more detail, please see the following link: https://www.pbscloudbackup.com/docs/sia/sia-foundation-update-may-2024/

Hello @rdwild

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Update: I should get an alpha iso copy of the software for testing early next week.

We are currently having issues compiling our iso due to an issue with proxmox code. We have reached out to them and hope to have a resolution soon. I will update here once that is the case.

Any updates on this? Thanks.

Sorry Steve, yes. I wanted to wait until I had the full facts (this will become evident why when you read on) before updating. Our primary developer ghosted us for the last three weeks to a month. I finally put a deadline on him returning to the project as of this past Monday which passed with no response from our primary developer. Feeling that something was going on and that we were behind on the project I brought in an additional developer about a month and a half ago just before we hit the issue of not being able to generate an ISO image of our project. My lead developer sent a note off to the proxmox team but did not get a response from them. About a week and a half later I reached out to the proxmox team to see if I could get an answer and got one. Which was to create a .deb file instead of an iso. I needed to give our developer that has been working on the project and did the costing research project some time to come back to the table but as of this last Monday and he did not. I reach out to our secondary developer yesterday after the deadline to see if he can finish the project. I won’t know for sure for a couple of days if that is the case or I need to bring in someone else. I will report back in a couple of days with an update. I did ask our backup dev if he can create an install .deb file for our project and he said yes. We are currently in a state of not knowing 100% where we are in the project timeline due to issues during the last period with our lead developer. I have asked our backup dev to update the project timeline once he has a better idea of what additional work needs to be done if any.

As soon as I have the .deb file I will be doing an install using it and will have screen shots available here once that is the case. Once we have done our base testing the .deb file and install instructions will be posted on our website and a link will be posted here to beta test the software.

Regards
Rodney Wild
A-Z Computer Solutions

Hello @rdwild

This is a reminder that your progress report is due.

Please be sure to follow the Progress Report Template and submit your progress report by the 2nd of each month. Thanks!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Our report is going to be delayed a couple of weeks due to the above situation. There is no point in just re-iterating what I have already posted above. I am waiting to hear back from my new dev on what he thinks is needed to get to the finish line. As soon as I know more, it will be posted here along with a project progress report.

Progress Report - August 2024

Cloud Backup Plugin Development for Proxmox Backup Server (Sia Integration)

  1. .deb Package Development

Task: Build the project into a Debian package.

Actions:

Executed the build process using the make .deb command within the project’s root directory.

Code Involved: Modified the Makefile to ensure all dependencies were correctly linked, specifically addressing issues related to the libcloudbackup.so library.

Outcome: Successfully generated a .deb package, ready for deployment on Debian-based systems, ensuring compatibility with Proxmox Backup Server environments.

  1. Test Server Configuration

Task: Setup a secure environment for testing the plugin.

Actions:

Configured SSH access to the test server by generating SSH keys and adding the public key to the ~/.ssh/authorized_keys on the remote server.

Code Involved: Modified the ssh_config file to restrict access only to specific IP addresses and ensure the use of SSH version 2 for enhanced security.

Outcome: Established a secure and stable connection to the test server, allowing for remote deployment and testing of the Cloud Backup plugin.

  1. UI Overhaul with ExtJS

Task: Redesign the user interface using the ExtJS framework.

Actions:

Rebuilt the UI components by creating new Ext.panel.Panel instances for the backup configuration screens, replacing the old hardcoded UI with a more modular approach.

Code Involved: Refactored UI logic in ui/main.js and ui/backupSettings.js to integrate with the new ExtJS components, enhancing maintainability and allowing for future updates without requiring a complete rebuild.

Outcome: Successfully migrated all UI elements to the new ExtJS framework, improving the UI’s responsiveness and reducing build dependencies.

  1. Cloud Backup Functionality Verification

Task: Test and verify the backup functionalities with Sia integration.

Actions:

Conducted end-to-end tests on the backupJob.js module to verify the proper creation, execution, and restoration of backups using the Sia decentralized storage.

Code Involved: Enhanced test scripts in tests/backupTest.js to include scenarios for failure handling and recovery, ensuring robust performance under various conditions.

Outcome: Verified that all cloud backup operations are functioning as expected, with successful uploads, retrievals, and error handling when interacting with the Sia network.

  1. DevOps Environment Setup

Task: Establish a DevOps environment for continuous integration and deployment.

Actions:

Set up CI/CD pipelines in the .gitlab-ci.yml file to automate the build, test, and deployment processes for Sia API integration and S3 bucket management.

Code Involved: Integrated the siaDeploy.sh script to automate the deployment of Sia-related services within Docker containers, ensuring consistent and reproducible environments.

Outcome: Achieved a fully automated DevOps pipeline, reducing manual intervention and increasing the speed and reliability of deployments.

  1. Proxmox Backup Server (PBS) Update Implementation Task: Implement and test the latest Proxmox Backup Server updates. Actions: Integrated the latest PBS updates by merging the upstream changes from the PBS-master branch into the project. Code Involved: Resolved merge conflicts in critical files like pbs-restore.js and pbs-api.js, ensuring that the Cloud Backup plugin remains fully compatible with the latest PBS features and APIs. Outcome: Successfully incorporated the latest PBS updates, enhancing the plugin’s compatibility and feature set.

Links to repos worked on this month:

What will you be working on this month?
This coming month we will be working on testing the .deb file and working on any bugs.
We will also be creating documentation for the project

Cloud backup development is ongoing and because of issues with our previous developer, we are behind by about a month. The previous developer had issues with generating an ISO which we were expecting to use for testing and the delay that it caused has caused us to run over budget. I have talked with our current developer who was helping our previous developer and he has now become our primary developer. He suggests that it is going to take another 80 hours to complete the project. A-Z Computer Solutions is committed to completing the project but at a slower pace due to the funding issues.

https://www.pbscloudbackup.com/docs/sia/sia-foundation-update-aug-2024/

Hey @rdwild, thanks for providing your August report. Please also provide your September report as that is also now late.

It’s ok if there isn’t much to write about since your last, but sticking to the schedule is important for us.

Ok sorry about the delay. I was still having communications issues with my current developer which I think I have worked out. I will have an updated project report Sunday Sept 22. End of day.