Proxmox Backup Server Cloud Extension

Project Name: Proxmox Backup Cloud Extension

Project Lead: Rodney Wild

Company: A-Z Computer Solutions

Purpose

We propose to increase the amount of stored data on the Sia network. At the moment there are a low number of big renters uploading data to the sia network. We propose to develop a cloud backup module for proxmox backup that would take proxmox backup data and hold a replication of it at an external service provider. This can be done directly via a local sia host or a satellite when that service is complete. Proxmox backup server is a great candidate for using sia because of its large data sets. Even a basic proxmox backup can be TBs in size and if freely adopted could put many petabytes of data onto the sia network. Backups out in the sia network add an extra level of redundancy to local backups should a disaster recovery scenario happen and allow businesses that use the PBC extension to have a greater level of recoverability from disaster situations. For example, if the business or data center where their primary backup data is housed burns down to the ground or gets flooded. When it comes to backups, we think it’s a good idea to have the original running host, an on-site backup, and a fairly recent off-site backup just in case something happens for business continuity purposes.

Open-Source Commitment

The project source code shall be maintained in a public repository located on Git Hub.

Project Timeline

The project shall have the following milestones:

1 Month – 1, 1.1 & 1.2, 1.3: Setup a Proxmox VE environment, Build and deploy a test instance of PBS, Study the PBS contribution guidelines

2 Month – 1.4, 1.5: Study the PBS tape feature, Start Starting to Build the cloud backup feature

3 Month – 1.5: Continue Building the cloud backup feature

4 Month – 1.5 Finish Building the cloud backup feature

5 Month – 1.6: Start Building the sia renterd plugin

6 Month – 1.6: Finish Start Building the sia renterd plugin

7 Month – Beta Testing of Cloud Backup System and website development

8 Month – Release

Please note that the above milestones are based on times provided by the developer and adjusted by the project manager.

Potential Risks:

Non-adherence to Proxmox Backup Server’s (PBS) development guidelines could lead to non-acceptance of the cloud backup feature in PBS’s code repository, making the project maintenance-intensive.

The other rise would be the loss of a developer once the project is started and not being able to find another developer.

I am not a developer.

Impact of Identified Risks:

The first risk could substantially increase the project’s dependency on maintenance.

Mitigation Strategies:

The code could be cloned and modified in a new repository and needs to be systematically updated to align with the ongoing changes in the original PBS code.

Having backup developers or redundant resources is essential to maintain project continuity in case of any resource unavailability. As I started the hiring process with an agency, I think that the risk is low. I also already have another developer That was identified via our testing evaluation process that was very close to the current developer that I can call on should this be required.

Monitoring and Managing Risks:

Continuous monitoring and managing of the identified risks are crucial.

Risk mitigation strategies should be revised as more information becomes available and project conditions evolve.

Budget

The project is requesting 80,000 USD, for the following 8 months. The spending can be itemized as follows:

70,000 USD shall constitute the developer’s salary, paid monthly or quarterly, at the discretion of the Foundation, at the beginning of each period. This person will be a hired position.

1,000 USD shall be used to pay for the project infrastructure (test servers and environment, subscriptions, etc.) for the duration of the project, paid upon completion of Phase 1.

2,000 USD shall be used for testing of the product (we plan to have a number of people do testing of the modules and will cover a small subsidy to cover the cost of people testing the product while it is in beta similar to the way the foundation has incentivized participation in the testnet).

7,000 USD paid to the project manager of the project to cover administrative and support costs

Reporting

The progress of the development shall be reported on a monthly basis in the community Discord.

Conclusion

The project lead is kindly asking the Foundation to review this proposal. He is also encouraging the community to ask as many questions as needed to improve the clarity of the project scope and the expected outcome. Our goal is to increase the amount of data on the sia network and increase the level of redundancy of backups in the proxmox virtual environment and provide for disaster recovery.

Development Information
Will all of your project’s code be open-source?
[Projects can use closed-source components, but can’t develop closed-source code. If any of your project’s code is closed-source, please describe what code and why.]

There is a possibility that there will be extra functionality developed via additional plugins for other cloud platforms. but these will be closed source and a premium feature. Sia will be the only open-source plugin developed for the cloud backup project.

Leave a link where the code will be accessible for review.

Do you agree to submit monthly progress reports?
[Progress reports must be submitted monthly here in the forum.]
I agree to share monthly updates in the forum and on Discord.

There is an attached document with quite a bit more detail than his provided here. Although the research project looked at the possibility of an S5 and Satellite it has been determined that S5 is not advanced enough at this time and Satellite could be handled with the current renterd plugin code. Documentation will however be included for the setup of a Satellite renterd.

Link to supporting documentation: Removed, see updated submission.

1 Like

Emal from the proxmox backup server dev team:

In generally we plan to add support for S3 off-site syncs, i.e., a
slightly specialized version of sync jobs (a bit closer to tape ones) so
that admins can configure off-site copies to any S3 capable storage.

S3 is widely poplar and while surely it has some shortcomings, like any
tech has, it is proven to work and there are many open source
implementations for server and client, like e.g., a Ceph setup can be
used as S3 target. Iow., S3 is established and won’t go away anytime
soon, if Sia Storage network has a bridge for S3 it would then be
already set.

If you have general contributions for improving/fixing things in one of
our repos feel free to send, but please ensure to follow our developer
documentation’s rules and guidelines:

https://pbs.proxmox.com/wiki/index.php/Developer_Documentation#Example

But please note that we cannot give you an upfront guarantee of accepting
any code, as an initial implementation is only a small fraction of the
work, continuous maintenance (which we only can bet on us doing, external
devs often tend to vanish after their changes got in) is much more work.

regards
Thomas

1 Like

Instead of working on a Sia-specific plugin. Would you be open to working on the S3 off-site sync? Since renterd now comes with S3 support and Proxmox already intends to build an S3-compatible cloud backup in the future, that might be simpler and more palatable than adding and maintaining a plugin system + plugin for each backend.

1 Like

Yes, it was already planned at some point. What ever gets more data into the sia network.

1 Like

Hello @rdwild,

Thanks for this proposal!

We appreciate the thorough breakdown but the Committee members felt the goals are a bit ambitious for the timeline. We ask to please remove the renterd plugin and re-submit your proposal focused on S3 plugin as the main Sia integration point.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Do I resubmit here or create a new proposal? It will take me a bit to remove the rendered and add in S3. I would like to note that it does not change the timeline as the developer suggested that plugins would take about the same amount of time to create. ie: Renderd and S3.

I would also note that creating an S3 plugin first would allow proxmox users backing up to the cloud to choose other venders other than Sia. One that comes to mind is Amazon, this is the reason why it was not originally considered as a first plugin and renderd was.

1 Like

Please post an updated proposal in this thread.

2 Likes

Just waiting on the amount of time to develop the S3 plugin from the developer. The only thing we did not do was budget it. Hoping to have it before the Tuesday cut off for the next meeting.

Project Name: Proxmox Backup Cloud Extension

Project Lead: Rodney Wild

Company: A-Z Computer Solutions

Purpose

We purpose to increase the amount of stored data on the Sia network. At the moment there are a low number of big renters uploading data to the sia network. We purpose to develop a cloud backup module for proxmox backup that would take proxmox backup data and hold a replication of it at an external service provider. This can either be done directly via a local sia host or via a satellite when that service is complete. Proxmox backup server is a great candidate for using sia because of its large data sets. Even a basic proxmox backup can be TBs in size and if freely adopted could put many petabytes of data onto the sia network. Backups out in the sia network add an extra level of redundancy to local backups should a disaster recovery scenario happen and allow businesses that use the PBC extension to have a greater level of recoverability from disaster situations. For example, if the business or data center where their primary backup data is housed burns down to the ground or gets flooded. When it comes to backups, we think its good idea to have the original running host, an on site backup and fairly recent off site backup just in case something happens for business continuity purposes.

Open-Source Commitment

The project source code shall be maintained in a public repository located on github.

Project Timeline

The project shall have the following milestones:

1 Month – 1, 1.1 & 1.2, 1.3: Setup a Proxmox VE Enviournment, Build and deploy a test instance of PBS, Study the PBS contribution guidelines

2 Month – 1.4, 1.5: Study the PBS tape feature, Start Starting Build the cloud backup feature

3 Month – 1.5: Continue Building the cloud backup feature

4 Month – 1.5 Finish Building the cloud backup feature

5, 6, 7 Month – 1.6: Building the S3 plugin

8 Month – Finish up development of the plugin and start Beta Testing of Cloud Backup System and website development

9 Month – Release

Please note that the above milestones are based on times provided by the developer and adjusted by the project manager.

Potential Risks:

Non-adherence to Proxmox Backup Server’s (PBS) development guidelines could lead to non-acceptance of the cloud backup feature in PBS’s code repository, making the project maintenance intensive.

The other rise would be the loss of a developer once the project is started and not being able to find another developer.

I am not a developer.

Impact of Identified Risks:

The first risk could substantially increase the project’s dependency on maintenance.

Mitigation Strategies:

The code could be cloned and modified in a new repository and needs to be systematically updated to align with the ongoing changes in the original PBS code.

Having backup developers or redundant resources is essential to maintain project continuity in case of any resource unavailability. As I started the hiring process with an agency, I think that the risk is low. I also already have another developer That was identified via our testing evaluation process that was very close to the current developer that I can call on should this be required.

Monitoring and Managing Risks:

Continuous monitoring and managing of the identified risks are crucial.

Risk mitigation strategies should be revised as more information becomes available and project conditions evolve.

Budget

The project is requesting 82,000 USD, for the following 8 months. The spendings can be itemized as follows:

72,000 USD shall constitute the salary of the developer, paid monthly or quarterly, at the discretion of the Foundation, at the beginning of each period. This person will be a hired position.

1,000 USD shall be used to pay for the project infrastructure (test servers and environment, subscriptions, etc.) for the duration of the project, paid upon completion of Phase 1.

2,000 USD shall be used for testing of the product (we plan to have several people do testing of the modules and will cover a small subsidy to cover cost of people testing the product while it is in beta similar to the way the foundation has incentivized participation in the testnet).

7,000 USD paid to the project manager of the project to cover administrative and support costs.

Reporting

The progress of the development shall be reported on a monthly basis in the community Discord.

Conclusion

The project lead is kindly asking the Foundation to review this proposal. He is also encouraging the community to ask as many questions as needed to improve the clarity of the project scope and the expected outcome. Our goal is to increase the amount of data on the sia network and increase the level of redundancy of backups in the proxmox virtual environment and provide for disaster recovery.

Development Information
Will all of your project’s code be open-source?
[Projects can use closed-source components, but can’t develop closed-source code. If any of your project’s code is closed-source, please describe what code and why.]

There is a possibility that there will be extra functionality developed via additional plugins for other cloud platforms. but these will be closed source and a premium feature. Sia will be the only open-source plugin developed for the cloud back project.

Leave a link where the code will be accessible for review.

Do you agree to submit monthly progress reports?
[Progress reports must be submitted monthly here in the forum.]
I agree to share monthly updates in the forum and on discord.

There is an attached document with quite a bit more detail than his provided here. Although the research project looked at the possibility of an S5 and Satellite it has been determined that S5 is not advanced enough.

Link to updated detailed supporting docs: Nextcloud

1 Like

Wow it feels like a book :slight_smile:

Hello @rdwild

Congratulations, the committee has approved this proposal!

Because this grant is considered a Standard Grant and thus requires milestones, after reviewing your Timeline and Goals the committee has determined these to the be milestones they will track:

  • Month 1: Hire dev, setup test instance, study guidelines
  • Month 4: Finish building cloud backup integration
  • Month 7: Finish S3 plugin
  • Month 9: Release

We’ll reach out via email to set up grant program onboarding. Onboarding typically takes a week or two, so free to adjust your internal timelines as appropriate.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

1 Like

That is Awesome Thanx.

1 Like

There is nothing there yet, but here is a git repository: AZComputerSolutions · GitHub

Project Update: We have contracts signed and our developer is now hired. The developer environment is about 50% setup (there was a slight delay waiting on storage for the dev server). We purchased a website domain for the project which we will be updating you on soon. And our github repository for the project will be setup in the next week. You can see it in the message above. More updates to come.

1 Like

Project announcement on proxmox forums: external datastore | Proxmox Support Forum

2 Likes

The Proxmox group has asked us to repost the project in the PBS dev email group. We had already done that but have done that as requested.

There is going to be a small delay in reporting this month as I am waiting on some notes from our developer.

2 Likes

Progress Report for the PBS Cloud Backup module - Dec 2023

What progress was made on your grant this month?
The first three phases of Proxmox VE Environment have been successfully completed.

Phase 1. Proxmox VE Server Setup:
• Successfully installed Proxmox VE Server.
• Configured a Debian 12.4 VM for PBS development.
• Updated the kernel to version 6.5 for compatibility with PBS.
• Configured initial test environment parameters for PBS.

Phase 2. PBS Build and Deployment:
• Downloaded PBS source code and built it on the Debian 12.4 VM.
• Installed ZFS 2.2 to meet PBS requirements.
• Addressed issues with the “backup user” and “group backup.”
• Encountered an “OS directory not found” error during PBS proxy service execution.
• Mapped PBS ISO with the built code for future compatibility.
• Initiated the mapping of dependencies between the built code and the ISO.
• Built latest source code and deployed with dependencies
• Ran the deployed source code
• Defining and Running Tests for all use cases still needs to be done

Phase 3. PBS Contribution Guidelines:
• Made sample rust code with the guidelines mentioned
• Studied rust-s3, with examples for studying the library to use it to hook it to S3 network.

Accomplishments for the Month:
• Completed the setup of the Proxmox VE Server and Debian 12.4 VM.
• Successfully built and deployed PBS, resolving initial compilation errors.
• Ran the deployed source code alongside with the ISO instance.
• Explored additional repositories for pbtest and developer for additional packages.
• Investigated and addressed issues related to the “backup user” and “group backup.”

The Following tasks were closed this month in openproject (project management tool) :

• Setup a Proxmox VE Server
• Installed the dev machine virtual machine
• Install Virtual Machines and Containers
• Installed and Setup a testnet instance of renterd with S3 for development
• Configure Networking and Storage
• Backup and Restore Testing within Proxmox VE
• Setup Development Environment
• Configure test environment parameters for PBS
• Build VE Server
• Clone the PBS repository for source-level access
• Install necessary dependencies and tools for building PBS
• Build PBS from Source
• Review PBS’s build documentation to understand the build process
• Handle dependency management; ensure all required libraries and modules are available
• Compile the PBS codebase, addressing any compilation errors or warnings
• Package the built binaries and assets for deployment
• Deploy the Built PBS Instance
• Determine the deployment configuration based on the test environment setup
• Deploy the PBS instance in the test environment
• Study the PBS contribution guidelines
• Review the documentation on PBS’s contribution process
• Understand the code style and standards required by PBS
• Familiarize with the pull request and code review process
• Learn about the testing procedures for contributions
• Determine any necessary steps for feature proposal and acceptance
• Create a new branch for development purposes
• Push the forked repository to personal or project GitHub account
• Ensure that the forked repository is private if needed
• Configure upstream settings for future sync with the original repository

Links to repos worked on this month:

No repos in the last milestone, but there will be repos in the next monthly report as we move into the development milestone in January.

What will you be working on this month?

  1. Troubleshoot PBS Proxy Service:
    • Dedicate time to resolve the persistent “OS directory not found” error in the PBS proxy service.
    • Explore new strategies and tricks to overcome the proxy service issue.
  2. Configuration and Dependencies:
    • Continue mapping dependencies between the PBS built code and ISO.
    • Finalize test environment parameters for PBS, ensuring compatibility.
  3. Documentation and Reporting:
    • Document challenges and solutions encountered during the proxy service troubleshooting.
    • Prepare detailed documentation on the configured test environment parameters.
  4. Start the understanding of the existing PBS backup features
    • Study the existing backup and restore operations
  5. Move towards building the cloud backup feature
    • Start the initial phase of the development
    I would like to note that because of a small delay in starting and the Christmas period we are about a week behind currently but we should be able to get caught up by the next reporting at the end of January.
2 Likes

Coding has started!

2 Likes

Rdwild reached out to let me know that their progress report is on its way, but will be late due to a medical issue on their end.