Please post an updated proposal in this thread.
2 Likes
Just waiting on the amount of time to develop the S3 plugin from the developer. The only thing we did not do was budget it. Hoping to have it before the Tuesday cut off for the next meeting.
Project Name: Proxmox Backup Cloud Extension
Project Lead: Rodney Wild
Company: A-Z Computer Solutions
Purpose
We purpose to increase the amount of stored data on the Sia network. At the moment there are a low number of big renters uploading data to the sia network. We purpose to develop a cloud backup module for proxmox backup that would take proxmox backup data and hold a replication of it at an external service provider. This can either be done directly via a local sia host or via a satellite when that service is complete. Proxmox backup server is a great candidate for using sia because of its large data sets. Even a basic proxmox backup can be TBs in size and if freely adopted could put many petabytes of data onto the sia network. Backups out in the sia network add an extra level of redundancy to local backups should a disaster recovery scenario happen and allow businesses that use the PBC extension to have a greater level of recoverability from disaster situations. For example, if the business or data center where their primary backup data is housed burns down to the ground or gets flooded. When it comes to backups, we think its good idea to have the original running host, an on site backup and fairly recent off site backup just in case something happens for business continuity purposes.
Open-Source Commitment
The project source code shall be maintained in a public repository located on github.
Project Timeline
The project shall have the following milestones:
1 Month – 1, 1.1 & 1.2, 1.3: Setup a Proxmox VE Enviournment, Build and deploy a test instance of PBS, Study the PBS contribution guidelines
2 Month – 1.4, 1.5: Study the PBS tape feature, Start Starting Build the cloud backup feature
3 Month – 1.5: Continue Building the cloud backup feature
4 Month – 1.5 Finish Building the cloud backup feature
5, 6, 7 Month – 1.6: Building the S3 plugin
8 Month – Finish up development of the plugin and start Beta Testing of Cloud Backup System and website development
9 Month – Release
Please note that the above milestones are based on times provided by the developer and adjusted by the project manager.
Potential Risks:
Non-adherence to Proxmox Backup Server’s (PBS) development guidelines could lead to non-acceptance of the cloud backup feature in PBS’s code repository, making the project maintenance intensive.
The other rise would be the loss of a developer once the project is started and not being able to find another developer.
I am not a developer.
Impact of Identified Risks:
The first risk could substantially increase the project’s dependency on maintenance.
Mitigation Strategies:
The code could be cloned and modified in a new repository and needs to be systematically updated to align with the ongoing changes in the original PBS code.
Having backup developers or redundant resources is essential to maintain project continuity in case of any resource unavailability. As I started the hiring process with an agency, I think that the risk is low. I also already have another developer That was identified via our testing evaluation process that was very close to the current developer that I can call on should this be required.
Monitoring and Managing Risks:
Continuous monitoring and managing of the identified risks are crucial.
Risk mitigation strategies should be revised as more information becomes available and project conditions evolve.
Budget
The project is requesting 82,000 USD, for the following 8 months. The spendings can be itemized as follows:
72,000 USD shall constitute the salary of the developer, paid monthly or quarterly, at the discretion of the Foundation, at the beginning of each period. This person will be a hired position.
1,000 USD shall be used to pay for the project infrastructure (test servers and environment, subscriptions, etc.) for the duration of the project, paid upon completion of Phase 1.
2,000 USD shall be used for testing of the product (we plan to have several people do testing of the modules and will cover a small subsidy to cover cost of people testing the product while it is in beta similar to the way the foundation has incentivized participation in the testnet).
7,000 USD paid to the project manager of the project to cover administrative and support costs.
Reporting
The progress of the development shall be reported on a monthly basis in the community Discord.
Conclusion
The project lead is kindly asking the Foundation to review this proposal. He is also encouraging the community to ask as many questions as needed to improve the clarity of the project scope and the expected outcome. Our goal is to increase the amount of data on the sia network and increase the level of redundancy of backups in the proxmox virtual environment and provide for disaster recovery.
Development Information
Will all of your project’s code be open-source?
[Projects can use closed-source components, but can’t develop closed-source code. If any of your project’s code is closed-source, please describe what code and why.]
There is a possibility that there will be extra functionality developed via additional plugins for other cloud platforms. but these will be closed source and a premium feature. Sia will be the only open-source plugin developed for the cloud back project.
Leave a link where the code will be accessible for review.
Do you agree to submit monthly progress reports?
[Progress reports must be submitted monthly here in the forum.]
I agree to share monthly updates in the forum and on discord.
There is an attached document with quite a bit more detail than his provided here. Although the research project looked at the possibility of an S5 and Satellite it has been determined that S5 is not advanced enough.
Link to updated detailed supporting docs: Nextcloud
1 Like
Wow it feels like a book 
Hello @rdwild
Congratulations, the committee has approved this proposal!
Because this grant is considered a Standard Grant and thus requires milestones, after reviewing your Timeline and Goals the committee has determined these to the be milestones they will track:
- Month 1: Hire dev, setup test instance, study guidelines
- Month 4: Finish building cloud backup integration
- Month 7: Finish S3 plugin
- Month 9: Release
We’ll reach out via email to set up grant program onboarding. Onboarding typically takes a week or two, so free to adjust your internal timelines as appropriate.
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
1 Like
That is Awesome Thanx.
1 Like
Project Update: We have contracts signed and our developer is now hired. The developer environment is about 50% setup (there was a slight delay waiting on storage for the dev server). We purchased a website domain for the project which we will be updating you on soon. And our github repository for the project will be setup in the next week. You can see it in the message above. More updates to come.
1 Like
The Proxmox group has asked us to repost the project in the PBS dev email group. We had already done that but have done that as requested.
There is going to be a small delay in reporting this month as I am waiting on some notes from our developer.
2 Likes
Progress Report for the PBS Cloud Backup module - Dec 2023
What progress was made on your grant this month?
The first three phases of Proxmox VE Environment have been successfully completed.
Phase 1. Proxmox VE Server Setup:
• Successfully installed Proxmox VE Server.
• Configured a Debian 12.4 VM for PBS development.
• Updated the kernel to version 6.5 for compatibility with PBS.
• Configured initial test environment parameters for PBS.
Phase 2. PBS Build and Deployment:
• Downloaded PBS source code and built it on the Debian 12.4 VM.
• Installed ZFS 2.2 to meet PBS requirements.
• Addressed issues with the “backup user” and “group backup.”
• Encountered an “OS directory not found” error during PBS proxy service execution.
• Mapped PBS ISO with the built code for future compatibility.
• Initiated the mapping of dependencies between the built code and the ISO.
• Built latest source code and deployed with dependencies
• Ran the deployed source code
• Defining and Running Tests for all use cases still needs to be done
Phase 3. PBS Contribution Guidelines:
• Made sample rust code with the guidelines mentioned
• Studied rust-s3, with examples for studying the library to use it to hook it to S3 network.
Accomplishments for the Month:
• Completed the setup of the Proxmox VE Server and Debian 12.4 VM.
• Successfully built and deployed PBS, resolving initial compilation errors.
• Ran the deployed source code alongside with the ISO instance.
• Explored additional repositories for pbtest and developer for additional packages.
• Investigated and addressed issues related to the “backup user” and “group backup.”
The Following tasks were closed this month in openproject (project management tool) :
• Setup a Proxmox VE Server
• Installed the dev machine virtual machine
• Install Virtual Machines and Containers
• Installed and Setup a testnet instance of renterd with S3 for development
• Configure Networking and Storage
• Backup and Restore Testing within Proxmox VE
• Setup Development Environment
• Configure test environment parameters for PBS
• Build VE Server
• Clone the PBS repository for source-level access
• Install necessary dependencies and tools for building PBS
• Build PBS from Source
• Review PBS’s build documentation to understand the build process
• Handle dependency management; ensure all required libraries and modules are available
• Compile the PBS codebase, addressing any compilation errors or warnings
• Package the built binaries and assets for deployment
• Deploy the Built PBS Instance
• Determine the deployment configuration based on the test environment setup
• Deploy the PBS instance in the test environment
• Study the PBS contribution guidelines
• Review the documentation on PBS’s contribution process
• Understand the code style and standards required by PBS
• Familiarize with the pull request and code review process
• Learn about the testing procedures for contributions
• Determine any necessary steps for feature proposal and acceptance
• Create a new branch for development purposes
• Push the forked repository to personal or project GitHub account
• Ensure that the forked repository is private if needed
• Configure upstream settings for future sync with the original repository
Links to repos worked on this month:
No repos in the last milestone, but there will be repos in the next monthly report as we move into the development milestone in January.
What will you be working on this month?
- Troubleshoot PBS Proxy Service:
• Dedicate time to resolve the persistent “OS directory not found” error in the PBS proxy service.
• Explore new strategies and tricks to overcome the proxy service issue. - Configuration and Dependencies:
• Continue mapping dependencies between the PBS built code and ISO.
• Finalize test environment parameters for PBS, ensuring compatibility. - Documentation and Reporting:
• Document challenges and solutions encountered during the proxy service troubleshooting.
• Prepare detailed documentation on the configured test environment parameters. - Start the understanding of the existing PBS backup features
• Study the existing backup and restore operations - Move towards building the cloud backup feature
• Start the initial phase of the development
I would like to note that because of a small delay in starting and the Christmas period we are about a week behind currently but we should be able to get caught up by the next reporting at the end of January.
2 Likes
Coding has started!
2 Likes
Rdwild reached out to let me know that their progress report is on its way, but will be late due to a medical issue on their end.
Progress Report for the PBS Cloud Backup module - Jan 2024
What progress was made on your grant this month?
Phase 4. Study the PBS tape feature:
• Acknowledging the significant role of layers in distinguishing and differentiating between various reading sources.
• Required understanding how different layers contribute to the processing and interpretation of information from different inputs. So that the same could be applied in cloud backup.
Phase 5. Build the cloud backup module:
• Setup renterd configuration
• Accessed the renterd portal
• Worked on the renterd API with client
• Understanding the underlying protocols, encryption methods, and data transfer processes utilized by Proxmox VE to facilitate efficient and secure backup operations
• Started making code changes for the cloud backup module.
• Created a cloud backup api stub endpoint
• Assessing Proxmox for supporting multiple cloud providers via plugins
Accomplishments for the Month:
• Moved towards building the cloud backup feature, started the initial phase of the development
The Following tasks were closed this month in openproject (project management tool) :
• Dive deep into tape_read, read, and block_reader traits
• api.rs
• backup.rs
• restore.rs
• Understand how the implementation is reading chunks
• Understand the backup flow of tape feature
• Grasp the abstraction layer functionality
• Acknowledge the layer’s role in differentiating between reading sources
• Study the write feature for trait implementation
• Study the restore feature for trait implementation
• Study the backup schedule and triggering functionality
• Study the feature for understanding incremental file saving approach
• Study the proxmox backup protocol
• Develop a Plugin-Enabled Module for Cloud Storage
• Architectural Consideration
• Implement the Strategy Pattern
• Understand its role in algorithm encapsulation and interchangeability
• Implement Restore Functionality
• Enable file-level and snapshot-level restores
• Allow selective restores based on timestamps, file types, etc
• Implement Backup Functionality
• Support full and incremental backups
• Write incremental, fully deduplicated backup files
• Manage versioning and ensure data integrity during backups
Links to repos worked on this month:
GitHub - AZComputerSolutions/PBSCloudBackupModule
What will you be working on this month?
During this month the backup module’s development would be continued, which will include:
Implementation of Backup functionality
• Adding of the cloud backup module
• Implement the cloud backup api under the path (/api2/json/cloud/backup)
• Add the following functionality in relation to the cloud backup module:
1. Modify the PoolWriter, CatalogSet, NewChunksIterator for the cloud module.
2. Implement the releveant functionality and removing existing tape functionality, like MediaPool, used_tapes, etc.
• Following structs/traits need to be implemented in relation to the cloud backup functionality in api2/cloud/backup.rs:
1. TapeBackupJobStatus → CloudBackupJobStatus
2. TapeBackupJobConfig → CloudBackupJobConfig
3. TapeBackupJobSetup → CloudBackupJobSetup
4. TapeBackupJobStatus → CloudBackupJobStatus
5. TapeBackupJobSummary → CloudBackupJobSummary
6. use::crate::tape → change to relevant cloud trait and implement the functionality
7. TAPE_BACKUP_JOB_ROUTER → CLOUD_BACKUP_JOB_ROUTER
8. tape-backup-job → cloud-backup-job
9. do_tape_backup_job → do_cloud_bakcup_job
10. backup_job.setup → implement relevant cloud functionality, to accept cloud jobs
11. let drive_lock = lock_tape_device(&drive_config, &setup.drive)?; → tape locking and device locking functionality needs to be removed
12. let _ = set_tape_device_state(&setup.drive, “”); → tape status needs to be removed
13. TAPE_STATUS_DIR → needs to be removed
14. pool_writer.set_media_status_full(&uuid)?; → This would be done along with the CatalogSet and NewChunksIterator as mentioned above
15. bail!(“Tape backup finished with some errors. Please check the task log.”); → Update the notifications according to the notifications from S3 network
16. summary.used_tapes = match pool_writer.get_used_media_labels() → after the classes, mentioned above are modified, this would be implemented according to the cloud
Currently the development is being done in an isolated repository. As mentioned in the risks initially, there are chances that this S3 cloud backup feature does not get accepted into the code base of proxmox so in the short term we have elected to isolate the code until it does get accepted.
Development of the proxmox backup feature has started, although slow, the cloud api endpoint stub has been implemented using the same code structure as the tape backup feature.
Sia Foundation monthly update Feb 2024
Progress Report for the PBS Cloud Backup module
What progress was made on your grant this month?
Progress has been slow due to some health issues with our developer and some problems faced in code understanding.
Phase 5. Build the cloud backup module:
-
What was done in the code:
• added /api2/json/cloud/backup endpoint
• copied tape/backup method inside cloud/backup without changes
• added src\cloud\pool_writer folder
• added src/api2/cloud/backup.rs and added ancillary structs
• added method bakcup and backup_worker in src/api2/cloud/backup.rs
• added pbs-api-types/src/cloud/mod.rs
• Added pbs-config/cloud_job.rs
• Implement the relevant cloud functionality and removed existing tape functionality, like MediaPool, used_tapes, etc. -
What was done in DevOps:
• Setup the renterd service again
• Worked on the renterd client to upload file through renterd API: renterd
Accomplishments for the Month:
• Not much has been achieved accept for fixing the error on adding cloud backup structs, similar to tape backup structs.
• One of the milestones which were discussed were to figure out how will the rust-S3 plugin be used to interact with the sia S3 cloud storage network:
Architectural discussion was done to ensure the right architecture is being implemented, the following is what will be implemented:
The Following tasks were closed this month in openproject (project management tool) :
• Construct Traits for Client Functionality
• Implement Backup Functionality
• Implement List Functionality
• Reading and Management of Snapshots
Links to repos worked on this month:
GitHub - AZComputerSolutions/PBSCloudBackupModule
What will you be working on this month?
During this month the backup module’s development would be continued, which will include:
During the next month(March) the backup module’s development would be continued, which will include, the unfinished tasks from last month:
Implementation of Backup functionality
• Following structs/traits need to be implemented in relation to the cloud backup functionality in api2/cloud/backup.rs:
i. TapeBackupJobStatus → CloudBackupJobStatus
ii. TapeBackupJobConfig → CloudBackupJobConfig
iii. TapeBackupJobSetup → CloudBackupJobSetup
iv. TapeBackupJobStatus → CloudBackupJobStatus
v. TapeBackupJobSummary → CloudBackupJobSummary
vi. use::crate::tape → change to relevant cloud trait and implement the functionality
vii. TAPE_BACKUP_JOB_ROUTER → CLOUD_BACKUP_JOB_ROUTER
viii. tape-backup-job → cloud-backup-job
ix. do_tape_backup_job → do_cloud_bakcup_job
x. backup_job.setup → implement relevant cloud functionality, to accept cloud jobs
xi. let drive_lock = lock_tape_device(&drive_config, &setup.drive)?; → tape locking and device locking functionality needs to be removed
xii. let _ = set_tape_device_state(&setup.drive, “”); → tape status needs to be removed
xiii. TAPE_STATUS_DIR → needs to be removed
xiv. pool_writer.set_media_status_full(&uuid)?; → This would be done along with the CatalogSet and NewChunksIterator as mentioned above
xv. bail!(“Tape backup finished with some errors. Please check the task log.”); → Update the notifications according to the notifications from S3 network
xvi. summary.used_tapes = match pool_writer.get_used_media_labels() → after the classes, mentioned above are modified, this would be implemented according to the cloud
Currently the development is being done in an isolated repository. As mentioned in the risks initially, there are chances that this S3 cloud backup feature does not get accepted into the code base of proxmox, so in the short term we have elected to isolate the code until it does get accepted. There was also an issue where the developer was using renterd api which needed to be corrected to use the rust-s3 create instead. This was caught and corrected early (due to weekly reporting) and the developer is now back on track.
Progress Report for the PBS Cloud Backup module - Feb 2024
What progress was made on your grant this month?
Progress has been slow due to some health issues with our developer and some problems faced in code understanding.
Phase 5. Build the cloud backup module:
- What was done in the code:
• added /api2/json/cloud/backup endpoint
• copied tape/backup method inside cloud/backup without changes
• added src\cloud\pool_writer folder
• added src/api2/cloud/backup.rs and added ancillary structs
• added method bakcup and backup_worker in src/api2/cloud/backup.rs
• added pbs-api-types/src/cloud/mod.rs
• Added pbs-config/cloud_job.rs
• Implement the relevant cloud functionality and removed existing tape functionality, like MediaPool, used_tapes, etc.
• - What was done in DevOps:
• Setup the renterd service again
• Worked on the renterd client to upload file through renterd API: renterd
Accomplishments for the Month:
• Not much has been achieved accept for fixing the error on adding cloud backup structs, similar to tape backup structs.
• One of the milestones which were discussed were to figure out how will the rust-S3 plugin be used to interact with the sia S3 cloud storage network:
Architectural discussion was done to ensure the right architecture is being implemented, the following is what will be implemented:
The Following tasks were closed this month in openproject (project management tool) :
• Construct Traits for Client Functionality
• Implement Backup Functionality
• Implement List Functionality
• Reading and Management of Snapshots
Links to repos worked on this month:
GitHub - AZComputerSolutions/PBSCloudBackupModule
What will you be working on this month?
During this month the backup module’s development would be continued, which will include:
During the next month(March) the backup module’s development would be continued, which will include, the unfinished tasks from last month:
Implementation of Backup functionality
• Following structs/traits need to be implemented in relation to the cloud backup functionality in api2/cloud/backup.rs:
i. TapeBackupJobStatus → CloudBackupJobStatus
ii. TapeBackupJobConfig → CloudBackupJobConfig
iii. TapeBackupJobSetup → CloudBackupJobSetup
iv. TapeBackupJobStatus → CloudBackupJobStatus
v. TapeBackupJobSummary → CloudBackupJobSummary
vi. use::crate::tape → change to relevant cloud trait and implement the functionality
vii. TAPE_BACKUP_JOB_ROUTER → CLOUD_BACKUP_JOB_ROUTER
viii. tape-backup-job → cloud-backup-job
ix. do_tape_backup_job → do_cloud_bakcup_job
x. backup_job.setup → implement relevant cloud functionality, to accept cloud jobs
xi. let drive_lock = lock_tape_device(&drive_config, &setup.drive)?; → tape locking and device locking functionality needs to be removed
xii. let _ = set_tape_device_state(&setup.drive, “”); → tape status needs to be removed
xiii. TAPE_STATUS_DIR → needs to be removed
xiv. pool_writer.set_media_status_full(&uuid)?; → This would be done along with the CatalogSet and NewChunksIterator as mentioned above
xv. bail!(“Tape backup finished with some errors. Please check the task log.”); → Update the notifications according to the notifications from S3 network
xvi. summary.used_tapes = match pool_writer.get_used_media_labels() → after the classes, mentioned above are modified, this would be implemented according to the cloud
Currently the development is being done in an isolated repository. As mentioned in the risks initially, there are chances that this S3 cloud backup feature does not get accepted into the code base of proxmox, so in the short term we have elected to isolate the code until it does get accepted. There was also an issue where the developer was using renterd api which needed to be corrected to use the rust-s3 create instead. This was caught and corrected early (due to weekly reporting) and the developer is now back on track.
https://www.pbscloudbackup.com/docs/sia/sia-foundation-update-feb-2024/
Hello @rdwild,
Thank you for your monthly Progress Report, however, you have strayed a bit from what we ask for when it comes to the reports.
These reports are useful for the team to get a quick at-a-glance summary of work done in the last month, and helps us and the committee be informed on the progress of the work.
We do not require grantees to write long, in-depth progress reports because it spends unnecessary time away from your own project, as well as the committee’s time who reviews these every month.
We appreciate the dedication to in-depth details, however, we do ask that in the future progress reports are kept to 3-5 brief bullet pointed statements per section (with 3 main sections- progress this month, issues encountered this month, goals next month, as well as simple links to updated repos). You can find the progress report template here: About the Grants category - #8
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
I’ll provide less detail in future reports with the ability to click on a link below for more details if needed.
Progress Report Form March 2024
What progress was made on your grant this month?
Phase 6. Continue Build the cloud backup module:
What was done in the code:
- added /pbs-api-types/src/cloud/*
- modified /examples/completion.rs
- modified examples/dynamic-files.rs
- Implement the relevant cloud functionality and removed existing tape functionality, like media , media_location , drive , etc.
Links to repos worked on this month:
Merged branch ‘develop’ of [**GitHub - AZComputerSolutions/PBSCloudBackupModule)
What will you be working on this month?
During the next month(April) the backup module’s development would be continued, which will include, the unfinished tasks from last month and Testing and Debugging completed functions:
Implementation of Backup functionality
modify backup_readers for cloudbackup
modify backup_repo for cloudbackup
modify backup_specification for cloudbackup
modify backup_writers for cloudbackup
add cloudbackup_info for pbs-database
add cloudbackup_key for proxmox-backup_client
complete cloud_writer for cloud_backup_written datas
Test cloud_backup functions with api
Debug cloud_backup functions with api
Please summarize your development goals into a few sentences or bullet points:
Cloud backup development is on-going and on track.
For more detail please see the following link: Sia Foundation Update – Mar 2024 – PBS Cloud Backup