Sia Foundation Monthly Report – Jan 2025
What progress was made on your grant this month?
Phase 8: Beta testing and bug cleanup:
Sia API Integration
Full Sia API Integration: Completed the integration of the Sia API within the Proxmox Backup Server Cloud Backup Plugin. This allows seamless interaction with the Sia decentralized storage network for reliable and secure cloud backups.
File Redundancy and Bandwidth Monitoring: Added advanced Sia API functionalities, including file redundancy checks and bandwidth monitoring, improving overall system reliability.
Fallback Mechanisms: Implemented fallback systems to ensure continued operation during Sia node unavailability or network limitations, guaranteeing uninterrupted backup and restore processes.
Cloud Backup API Enhancements
API Standardization: Finalized the standardization of API interactions, ensuring improved compatibility and a streamlined data handling process across modules.
Performance Optimizations: Continued refining the API’s scalability and performance, ensuring smooth operations even with large backup datasets.
Role-Based Access Control (RBAC): Strengthened security by further enhancing role-based permissions in the cloud backup API, enabling more granular access control.
Security and Compliance Advancements
Strengthened Encryption: Continued to enhance data encryption at rest and in transit, ensuring compliance with GDPR and industry standards.
Token-Based Authentication: Improved security by replacing session-based authentication with token-based authentication for all API interactions.
Penetration Testing & Security Audits: Performed extensive penetration testing to detect and resolve vulnerabilities. Completed a security audit to ensure all components are secure and compliant.
UI/UX Enhancements
Enhanced User Interface: Implemented new UI elements, including real-time backup status indicators, progress bars, and error notifications, providing a more intuitive user experience.
Dashboard Improvements: Refined the dashboard to display enhanced metrics, including Sia storage usage, capacity estimates, and backup job dependencies.
Multi-Language Support: Expanded the plugin’s accessibility by adding support for multiple languages, broadening the user base.
Testing and Documentation
System Stress Testing: Conducted extensive stress testing to ensure system stability, successfully handling even higher-than-expected data throughput.
Updated Documentation: Continued work on detailed user and developer documentation, focusing on the new Sia API integration and cloud backup API features.
Codebase Refinement: Refined the codebase for clarity and performance, performing static code analysis and enhancing the CI/CD pipeline.
- Key Technical Challenges and Resolutions
Sia API Integration Challenges
Challenge: Integrating the complex Sia API with the Proxmox Backup Server required handling asynchronous operations and network instability.
Resolution: Developed robust asynchronous handling techniques and fallback strategies to mitigate node unavailability and network latency.
UI Synchronization with Real-Time Data
Challenge: The real-time synchronization of backup statuses and capacity metrics led to occasional UI inconsistencies.
Resolution: Introduced improved caching mechanisms and refined the synchronization logic to ensure smooth UI updates.
Scalability and Load Balancing in Middleware
Challenge: Ensuring the middleware could handle the increased data throughput due to full Sia API integration.
Resolution: Optimized the middleware for enhanced load balancing and asynchronous processing, ensuring it can scale efficiently with increasing user demand.
- Upcoming Goals
Final Testing & Stable Release Preparation
Complete final rounds of beta testing with focus on real-world performance under large-scale conditions.
Resolve any outstanding bugs and performance issues, aiming for a stable release in early February.
Expanded Documentation and Community Engagement
Finalize and release comprehensive documentation covering the new Sia API integration and enhanced API features.
Publish the plugin on GitHub, inviting open-source contributions and feedback from the broader community.
Long-Term Maintenance & Future Integrations
Plan the integration of additional storage backends and work towards optimizing middleware scalability.
Explore potential partnerships with other decentralized storage providers to further enhance functionality.
UI/UX Finalization
Complete multi-language localization and continue refining the user interface, adding advanced visualizations for backup job dependencies and storage utilization.
Add contextual help and tutorials within the UI to improve user experience for both beginners and advanced users.
What was done in the code:
src/middleware/siaIntegrationMiddleware.rs: Enhanced error-handling logic and optimized integration with Sia API.
pbs/src/api2/cloud/backup.rs: Final adjustments to API to improve scalability and performance.
pbs-api-types/src/acl.rs: Updates to access control logic for more granular role-based permissions.
pbs-api-types/src/libs.rs: Optimizations to API types for better compatibility and data handling.
pbs-api-types/src/maintenance.rs: New logic for maintenance mode checks during cloud backup operations.
pbs-api-types/src/network.rs: Updated network configuration to better manage decentralized storage communications.
Links to repos worked on this month:
Merged branch ‘develop’ of https://github.com/AZComputerSolutions/PBSCloudBackupModule
What will you be working on this month?
Release Candidate Preparation:
Final Testing and Stable Release
Conduct final beta testing and resolve any remaining bugs or performance bottlenecks. Prepare for the stable release of the Proxmox Backup Server Cloud Backup Plugin with full Sia integration. Enhanced Documentation and Community Collaboration and Publish comprehensive user and developer documentation, including setup guides and troubleshooting resources. Release the plugin on GitHub and collaborate with the open-source community to gather feedback for future improvements.
UI/UX Enhancements
Finalize multi-language localization and add contextual help features to improve user experience. Continue refining the user interface, including advanced visualization tools for backup job dependencies and performance metrics.
Please summarize your development goals into a few sentences or bullet points:
We have documented setting up a test server. So that anyone can do it with the code in our github site. I would like to note that the Proxmox group has been less than helpful in helping us set up things so we can do testing before incorporating our code into their code base. We have setup a work around where we use an overlay system. I have been able to setup a test version of the product but there are still lots of bugs in it. We are currently working to remove bugs we are finding as we move towards an beta release and external test of our software.
- PBS Plus Package Structure (Debian)
Copy
pbs-plus_1.0_all.deb
├── etc/
│ ├── pbs-plus/
│ │ ├── overlay/ # Custom files to overlay
│ │ │ ├── usr/
│ │ │ │ └── share/
│ │ │ │ └── proxmox-backup-ui/
│ │ │ │ └── pbs-ui.js # Modified Ember components
│ │ ├── ssl/
│ │ │ └── pbs-plus.pem # Self-signed certificate
│ ├── systemd/system/
│ │ └── pbs-plus.service
├── DEBIAN/
│ ├── control
│ ├── postinst
│ └── prerm
-
Overlay Mount Systemd Service
-
Certificate Management
Copy
Post-install script (DEBIAN/postinst)
#!/bin/sh
update-ca-certificates --fresh
systemctl restart nginx
- Ember.js UI Customization
Since PBS uses Ember.js (v3.28+), modify these core files:
Target Files:
/usr/share/proxmox-backup-ui/js/pbs.js # Main entry point
/usr/share/proxmox-backup-ui/templates/*.hbs # Handlebar templates
/usr/share/proxmox-backup-ui/public/assets/ # CSS/Images
As always, the full version of this document is available at
https://www.pbscloudbackup.com/docs/sia/sia-foundation-monthly-report-jan-2025/
Screen Shots (more to come)
