Thanks @rdwild. Quick question: Your timeline cites 10 weeks of work but your budget request appears to be for 12 (480/40). What’s going on there?
As you know, development is not always an exact science. There is a 20% contingency added on to the quote I get from the developer. The developer came back with an a slight adjustment (400 hours and I added the 20% contingency (80 hours) which will be returned if not used by the developer.
Rodney Wild
A-Z Computer Solutions
Thanks for your latest proposal to The Sia Foundation Grants Program.
After review, the committee has decided to approve your proposal. They’re definitely excited and hopeful to see the completion of your work so far.
We’ll reach out to your provided email address for onboarding. Obviously this shouldn’t take too long as this is a continuation of your previous grant.
Thanx Steve! The details should be the same.
Progress Report Form Oct 2024
What progress was made on your grant this month?
Middleware Development for Sia Integration:
Cloud Backup API Enhancements:
Integration Testing and Code Maintenance:
User Interface (UI) Improvements:
Phase 8: Beta testing and bug cleanup:
Documentation and Repository Management:
We updated and merged our work on GitHub, including all major middleware and API connection changes, code maintenance tasks, and ExtJS UI updates.
Cloud Backup Middleware Development
Objective: Build middleware to enable interaction between Proxmox Backup Server’s cloud backup APIs and Sia’s decentralized storage API, addressing connectivity issues.
Changes:
Developed the middleware layer to route API calls effectively and provide consistent data handling across platforms.
Added error-handling routines to manage connectivity errors and log middleware actions for debugging.
What was done in the code:
src/middleware/siaIntegrationMiddleware.rs: Main middleware file for API requests.
src/middleware/siaIntegrationTest.js: Middleware test script for Sia API functionality checks.
API Connectivity Enhancements
Objective: Improve cloud backup API’s performance and security by updating dependencies and fixing broken functions.
Changes:Replaced outdated Rust dependencies with optimized libraries for faster and more stable performance.
Fixed broken functions within cloud backup API to ensure proper backend/frontend communication.
pbs/src/api2/cloud/backup.rs: Main cloud backup API file.
pbs/src/api2/cloud/mod.rs: Module configuration for cloud backup.
pbs/src/api2/cloud/restore.rs: Restoration logic for cloud backups.
Encryption & Decryption for Data Security
Objective: Secure data interactions by implementing hash encryption and decryption routines.
Changes:Added encryption methods to secure sensitive data during transmission between cloud backup UI and Sia storage.
www/cloud/window/Hashcrypt.js: Contains hash encryption and decryption methods.
Cloud Backup UI Development
Objective: Improve the cloud backup user experience through API integrations and UI styling enhancements.
Changes:Integrated backup and restore functionality directly into the cloud backup UI.
Enhanced UI and styling for improved user experience and accessibility.
www/cloud/window/CloudBackupJob.js: UI logic for backup jobs.
www/css/ext6-pbscloud.css: Stylesheet for cloud backup UI.
www/cloud/backupworks/backupschedule.js: UI for backup scheduling options.
Access Control and Backup Group Management
Objective: Enhance user access control settings and allow management of backup group ownership.
Changes:Introduced features for editing access control within the UI.
Added functionality to change backup group owners directly from the interface.
www/cloud/window/ACLEdit.js: Access control settings.
www/cloud/window/GroupManager.js: Management of backup groups.
ExtJS Frontend Development and API Testing
Objective: Ensure compatibility and test the UI for Sia integration using ExtJS and API testing tools.
Changes:Continued refining the ExtJS frontend for seamless user interactions with the Sia API.
Conducted API connection tests to validate functionality for cloud backups.
src/cloud/test/alloc_writable_media.rs: Test file for API connections.
www/cloud/window/UIComponents.js: Various UI components.
Links to repos worked on this month:
Merged branch ‘develop’ of https://github.com/AZComputerSolutions/PBSCloudBackupModule
What will you be working on this month?
Beta Testing and Bug Fixes:
Test Server Setup and Deployment:
Middleware Enhancements:
UI Refinements and User Experience Improvements:
Documentation and GitHub Management
Please summarize your development goals into a few sentences or bullet points:
Fix bugs and code dependencies changes in rust that make our code no longer compile . Get a test server setup and working. Document setting up a test server. So that anyone can do it would the code in our github site. I would also like to note that because of the pending approval by the Sia foundation (thank-you for your continued support) we have been running at full development speed this last month.
We were ask by the Sia foundation to shorten our monthly reports but as always, the full version of this document is available at https://www.pbscloudbackup.com/docs/sia/sia-foundation-monthly-report-oct-2024/
PBSCloud Backup Progress Report Nov 2024
What progress was made on your grant this month?
- Sia Middleware Proxmox
- Cloud Backup API Refinements:
- Advanced Security and Compliance Measures:
- UI Enhancements and New Features: Integration Testing and Pre-release Documentation:
- Repository Optimization and Code Consolidation:
- Sia Middleware Proxmox
- Cloud Backup API Refinements:
- Advanced Security and Compliance Measures:
- UI Enhancements and New Features:
- Integration Testing and Pre-release Documentation:
- Repository Optimization and Code Consolidation:
Phase 8: Beta testing and bug cleanup:
Documentation and Repository Management: We updated and merged our work on GitHub, including all major middleware and API connection changes, code maintenance tasks, and ExtJS UI updates.
Cloud Backup Middleware Development Objective: Build middleware to enable interaction between Proxmox Backup Server’s cloud backup APIs and Sia’s decentralized storage API, addressing connectivity issues.
Changes:
Developed the middleware layer to route API calls effectively and provide consistent data handling across platforms.
Added error-handling routines to manage connectivity errors and log middleware actions for debugging.
What was done in the code:
src/middleware/siaIntegrationMiddleware.rs: Modularized middleware and added caching logic. src/middleware/loadBalancer.rs: Implemented dynamic load balancing for middleware operations.
pbs/src/api2/cloud/backup.rs: Integrated asynchronous processing and analytics reporting. pbs/src/api2/cloud/auth.rs: Added token-based authentication.
www/cloud/security/AuditLogger.js: Implemented logging for critical actions. www/cloud/security/EncryptionHandler.js: Enhanced encryption algorithms.
www/cloud/window/StorageDashboard.js: Created Sia storage monitoring dashboard. www/css/ext6-pbscloud.css: Updated styles for localization and accessibility. www/cloud/backupworks/dependencyMap.js: Added dependency visualization features.
Links to repos worked on this month:
Merged branch ‘develop’ of https://github.com/AZComputerSolutions/PBSCloudBackupModule
What will you be working on this month?
Release Candidate Preparation:
Resolve final bugs and ensure all features meet performance benchmarks.
Conduct a soft launch with select beta testers.
Long-term Maintenance Planning:
Develop a roadmap for supporting additional storage backends.
Document middleware architecture for handoff and future scalability.
Please summarize your development goals into a few sentences or bullet points:
Fix bugs and code dependencies changes in rust that make our code no longer compile . Get a test server setup and working. Document setting up a test server. So that anyone can do it would the code in our github site.
As always, this document has been shortened greatly, please see the full version of this document at https://www.pbscloudbackup.com/docs/sia/sia-foundation-monthly-report-nov-2024/
Hello,
Thank you for your progress report!
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
Sia Foundation Monthly Report – Dec 2024
What progress was made on your grant this month?
**Middleware Completion and Advanced Sia Integration**
**Expanded Cloud Backup API Capabilities**
**Sia API Enhancements and Optimization**
**Security and Compliance Advancements**
**User Interface (UI) Improvements**
**Phase 8: Beta testing and bug cleanup:**
**Middleware Scalability**
Challenge: The middleware faced delays under high loads due to inadequate request handling.
Resolution: Implemented asynchronous request processing and dynamic load balancing to ensure consistent performance under heavy data throughput.
Sia Network Constraints
Challenge: Node unavailability and bandwidth limitations impacted decentralized storage performance.
Resolution: Developed robust fallback mechanisms and optimized Sia API interactions for redundancy and reliability.
UI Synchronization Issues
Challenge: Real-time status updates and dependency visualizations caused inconsistencies in the user interface.
Resolution: Integrated advanced caching and synchronous update mechanisms to maintain data consistency across the UI.
What was done in the code:
src/middleware/siaIntegrationMiddleware.rs: Maintenance mode adaptation and improved error-handling logic.
Cloud Backup API Enhancements
Updated API types to standardize interactions and improve compatibility across modules.
Made significant updates to the cloud backup API, optimizing data handling and request/response cycles.
Enhanced access control functionalities with updates to the acl.rs file, introducing more granular permissions.
Files Updated:
pbs/src/api2/cloud/backup.rs: Cloud backup API updates to improve efficiency and scalability.
File Modified:
pbs-api-types/src/acl.rs: Access control updates for better security and role-based permissions.
pbs-api-types/src/libs.rs: Basic API types used by most of the cloud backup code.
pbs-api-types/src/matintenance.rs: Checks the current maintenance mode against an attempted operation.
pbs-api-types/src/network.rs: Configuration is done manually using other tools and network interface type.
Links to repos worked on this month:
Merged branch ‘develop’ of https://github.com/AZComputerSolutions/PBSCloudBackupModule
What will you be working on this month?
Release Candidate Preparation:
Final Testing and Stable Release
Conduct final beta testing and resolve any remaining bugs or performance bottlenecks. Prepare for the stable release of the Proxmox Backup Server Cloud Backup Plugin with full Sia integration. Enhanced Documentation and Community Collaboration and Publish comprehensive user and developer documentation, including setup guides and troubleshooting resources. Release the plugin on GitHub and collaborate with the open-source community to gather feedback for future improvements.
Long-Term Maintenance and Scalability
Plan for the integration of additional storage backends and further middleware scalability. Initiate discussions for partnerships with other decentralized storage solutions to expand functionality.
UI/UX Enhancements
Finalize multi-language localization and add contextual help features to improve user experience. Continue refining the user interface, including advanced visualization tools for backup job dependencies and performance metrics.
Please summarize your development goals into a few sentences or bullet points:
Fix bugs and code dependencies changes in rust that make our code no longer compile . Get a test server setup and working. Document setting up a test server. So that anyone can do it would the code in our github site.
As always, this document has been shortened greatly, please see the full version of this document is available at https://www.pbscloudbackup.com/docs/sia/pbscloud-backup-progress-report-dec-2024/
Hello,
Thank you for your progress report!
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
Hey @rdwild, our dev team was having some trouble evaluating your progress so far and in particular up to your current milestone, which is “Complete the final development of the Sia renterd plugin”.
We’ll need either a setup guide, working demo, or video demo to evaluate. They mentioned they quickly scanned the repo for our code or API routes and didn’t see anything. The files mentioned in some recent reports were either too simple or not updated in a while.
Once we get that, they’ll re-evaluate and be able to send this month’s payment.
You guys asked for short reports, the long report link has been at the bottom of every report. I talked with my developer and the API stuff was not committed. Please see the following link:
If there is anything else that is needed please let me know. Thanx.
They were happy to see the API code get committed. They’d still like to see a setup guide, video demo, or working demo to verify progress. Thanks.
1 Like
Thanx Steve, I have already talked to my dev on that very subject and also passed along your comments. We should have that very soon.
There was a slight delay in getting this month’s report; it will be in by the end of the day.
1 Like
Sia Foundation Monthly Report – Jan 2025
What progress was made on your grant this month?
Phase 8: Beta testing and bug cleanup:
Sia API Integration
Full Sia API Integration: Completed the integration of the Sia API within the Proxmox Backup Server Cloud Backup Plugin. This allows seamless interaction with the Sia decentralized storage network for reliable and secure cloud backups.
File Redundancy and Bandwidth Monitoring: Added advanced Sia API functionalities, including file redundancy checks and bandwidth monitoring, improving overall system reliability.
Fallback Mechanisms: Implemented fallback systems to ensure continued operation during Sia node unavailability or network limitations, guaranteeing uninterrupted backup and restore processes.
Cloud Backup API Enhancements
API Standardization: Finalized the standardization of API interactions, ensuring improved compatibility and a streamlined data handling process across modules.
Performance Optimizations: Continued refining the API’s scalability and performance, ensuring smooth operations even with large backup datasets.
Role-Based Access Control (RBAC): Strengthened security by further enhancing role-based permissions in the cloud backup API, enabling more granular access control.
Security and Compliance Advancements
Strengthened Encryption: Continued to enhance data encryption at rest and in transit, ensuring compliance with GDPR and industry standards.
Token-Based Authentication: Improved security by replacing session-based authentication with token-based authentication for all API interactions.
Penetration Testing & Security Audits: Performed extensive penetration testing to detect and resolve vulnerabilities. Completed a security audit to ensure all components are secure and compliant.
UI/UX Enhancements
Enhanced User Interface: Implemented new UI elements, including real-time backup status indicators, progress bars, and error notifications, providing a more intuitive user experience.
Dashboard Improvements: Refined the dashboard to display enhanced metrics, including Sia storage usage, capacity estimates, and backup job dependencies.
Multi-Language Support: Expanded the plugin’s accessibility by adding support for multiple languages, broadening the user base.
Testing and Documentation
System Stress Testing: Conducted extensive stress testing to ensure system stability, successfully handling even higher-than-expected data throughput.
Updated Documentation: Continued work on detailed user and developer documentation, focusing on the new Sia API integration and cloud backup API features.
Codebase Refinement: Refined the codebase for clarity and performance, performing static code analysis and enhancing the CI/CD pipeline.
- Key Technical Challenges and Resolutions
Sia API Integration Challenges
Challenge: Integrating the complex Sia API with the Proxmox Backup Server required handling asynchronous operations and network instability.
Resolution: Developed robust asynchronous handling techniques and fallback strategies to mitigate node unavailability and network latency.
UI Synchronization with Real-Time Data
Challenge: The real-time synchronization of backup statuses and capacity metrics led to occasional UI inconsistencies.
Resolution: Introduced improved caching mechanisms and refined the synchronization logic to ensure smooth UI updates.
Scalability and Load Balancing in Middleware
Challenge: Ensuring the middleware could handle the increased data throughput due to full Sia API integration.
Resolution: Optimized the middleware for enhanced load balancing and asynchronous processing, ensuring it can scale efficiently with increasing user demand.
- Upcoming Goals
Final Testing & Stable Release Preparation
Complete final rounds of beta testing with focus on real-world performance under large-scale conditions.
Resolve any outstanding bugs and performance issues, aiming for a stable release in early February.
Expanded Documentation and Community Engagement
Finalize and release comprehensive documentation covering the new Sia API integration and enhanced API features.
Publish the plugin on GitHub, inviting open-source contributions and feedback from the broader community.
Long-Term Maintenance & Future Integrations
Plan the integration of additional storage backends and work towards optimizing middleware scalability.
Explore potential partnerships with other decentralized storage providers to further enhance functionality.
UI/UX Finalization
Complete multi-language localization and continue refining the user interface, adding advanced visualizations for backup job dependencies and storage utilization.
Add contextual help and tutorials within the UI to improve user experience for both beginners and advanced users.
What was done in the code:
src/middleware/siaIntegrationMiddleware.rs: Enhanced error-handling logic and optimized integration with Sia API.
pbs/src/api2/cloud/backup.rs: Final adjustments to API to improve scalability and performance.
pbs-api-types/src/acl.rs: Updates to access control logic for more granular role-based permissions.
pbs-api-types/src/libs.rs: Optimizations to API types for better compatibility and data handling.
pbs-api-types/src/maintenance.rs: New logic for maintenance mode checks during cloud backup operations.
pbs-api-types/src/network.rs: Updated network configuration to better manage decentralized storage communications.
Links to repos worked on this month:
Merged branch ‘develop’ of https://github.com/AZComputerSolutions/PBSCloudBackupModule
What will you be working on this month?
Release Candidate Preparation:
Final Testing and Stable Release
Conduct final beta testing and resolve any remaining bugs or performance bottlenecks. Prepare for the stable release of the Proxmox Backup Server Cloud Backup Plugin with full Sia integration. Enhanced Documentation and Community Collaboration and Publish comprehensive user and developer documentation, including setup guides and troubleshooting resources. Release the plugin on GitHub and collaborate with the open-source community to gather feedback for future improvements.
UI/UX Enhancements
Finalize multi-language localization and add contextual help features to improve user experience. Continue refining the user interface, including advanced visualization tools for backup job dependencies and performance metrics.
Please summarize your development goals into a few sentences or bullet points:
We have documented setting up a test server. So that anyone can do it with the code in our github site. I would like to note that the Proxmox group has been less than helpful in helping us set up things so we can do testing before incorporating our code into their code base. We have setup a work around where we use an overlay system. I have been able to setup a test version of the product but there are still lots of bugs in it. We are currently working to remove bugs we are finding as we move towards an beta release and external test of our software.
- PBS Plus Package Structure (Debian)
Copy
pbs-plus_1.0_all.deb
├── etc/
│ ├── pbs-plus/
│ │ ├── overlay/ # Custom files to overlay
│ │ │ ├── usr/
│ │ │ │ └── share/
│ │ │ │ └── proxmox-backup-ui/
│ │ │ │ └── pbs-ui.js # Modified Ember components
│ │ ├── ssl/
│ │ │ └── pbs-plus.pem # Self-signed certificate
│ ├── systemd/system/
│ │ └── pbs-plus.service
├── DEBIAN/
│ ├── control
│ ├── postinst
│ └── prerm
-
Overlay Mount Systemd Service
-
Certificate Management
Copy
Post-install script (DEBIAN/postinst)
#!/bin/sh
update-ca-certificates --fresh
systemctl restart nginx
- Ember.js UI Customization
Since PBS uses Ember.js (v3.28+), modify these core files:
Target Files:
/usr/share/proxmox-backup-ui/js/pbs.js # Main entry point
/usr/share/proxmox-backup-ui/templates/*.hbs # Handlebar templates
/usr/share/proxmox-backup-ui/public/assets/ # CSS/Images
As always, the full version of this document is available at
https://www.pbscloudbackup.com/docs/sia/sia-foundation-monthly-report-jan-2025/
Screen Shots (more to come)
Hey @rdwild we need your February report in the next couple of days to have our review done on time.
2 Likes
Thanx for the update Steve, I was out of the office last few days due to my birthday. I have the report from our dev and just need to reformat it so it should be available before the deadline. Thanx for letting me know.
1 Like
Really excited about this. But is there an estimated release Date? And will this be directly integrated into pve?
Morning OttoBoom. I see this is the first time you have posted. Welcome! Not yet, We are currently finishing internal beta testing (our developer is currently in hospital, so there will be an update as soon as he is available) to get the product to the point where it can be externally tested. Keep an eye out here and in the grants-program Sia Discord group for updates. We will also post an update when its ready on the project website.
https://www.pbscloudbackup.com/
Rodney Wild
A-Z Computer Solutions