Thank you for the report, everything looks good so far!
Progress Report (May)
S5 Network
Version 0.11.3 of the S5 Node has been released (GitHub - s5-dev/S5: Decentralized content-addressed storage network). Highlights include a new Admin Web UI, multi-arch Docker image, WebSocket connections for the p2p network, a new file streaming implementation and some improvements to the Sia renterd object store.
S5 Dart packages have been published on pub.dev, this makes it easy for Dart and Flutter developers to install and use them. (Packages of publisher sfive.net)
Vup Cloud Storage App
A lot of new features and improvements have been implemented since the last Beta release, but a new one is not quite ready yet. The next release will happen this month and be finally available on all platforms.
ATProto and Bluesky
The original timeline for the S5 project included a full public identity system with a browser extension for secure client-side signatures, social network elements for the Tube5 video streaming app and a Discord-like decentralized chat platform.
Instead of continuing to build all of the primitives and core tech for this myself, I decided to switch to https://atproto.com/. The AT Protocol offers almost all features needed to build truly decentralized applications and it is already a lot more mature compared to what I could build from scratch.
The most relevant app on ATProto right now is Bluesky, which just hit 100k users. It’s a microblogging app similar to Twitter.
This are some examples where S5 could be used with ATProto and Bluesky:
- Host large media files, for example videos or big images
- Store content-addressed backups of ATProto repos to make account recovery even easier
- Publish web apps securely
- Archive content
After custom lexicon support lands on the official instance, videos in the S5 metadata format (GitHub - s5-dev/json-schemas) can be published directly on ATProto with only some minor modifications. Tube5 will switch to publishing video metadata on ATProto, but the video files will still be streamed from S5 by default. If everything goes well, the ATProto blob format will be fully compatible with S5 (both using the BLAKE3 hash function) and large media files can be hosted on both S5 and ATProto servers using the same hash. I’m also working on some custom lexicons for building a Reddit-like social platform on top of ATProto: GitHub - redsolver/atproto
Apart from custom lexicons, I built a lot fun stuff for the Bluesky ecosystem this month. The first one is SkyFeed (https://skyfeed.app), a third-party client with some unique features like real-time updates. SkyFeed was the first Bluesky client to support custom feeds, you can read more about that and other cool new Bluesky features in their recent blog post (Private Beta Update & Roadmap - Bluesky). I also built an indexer (GitHub - skyfeed-dev/indexer: ATProto/Bluesky Indexer powered by SurrealDB.) which powers all of my custom feeds. They serve up to 400 feed requests per minute (~ 6.7 reqs/second).
Thanks for the monthly report redsolver!
Progress Report (June)
S5 Network
The S5 Node now supports deleting files on Sia and fetching stats about the storage usage. There’s also a new experimental IPFS Object Store which stores files on an IPFS node and has the same hash for files smaller than 1 MiB, so they can be fetched from both networks (S5 and IPFS) (Commits · s5-dev/S5 · GitHub)
Vup Cloud Storage App
The first public Vup Beta powered by S5 was released for all platforms! Version 0.14.3 BETA is available for Windows, Linux, macOS, iOS and Android. It can be used with a local instance of renterd, a self-hosted S5 Node or a third-party node. All data is fully end-to-end-encrypted and Vup can be configured to store metadata and/or files on multiple remotes for redundancy and even higher reliability. Installation instructions are available on Install Vup
ATProto and Bluesky
I created a lexicon/schema for publishing videos, audio and other media content on ATProto. There’s a new federated sandbox network and I published a test video there: Tube5 Alpha The metadata is stored on ATProto and integrates with Bluesky social features, the media blobs are compatible with ATProto CIDs and streamed from the S5 Network.
Bluesky continues to grow and is now at 217k registered users. I’m hosting some custom feeds and they peaked at ~ 1000 requests/minute this week. I’m not counting individual users for privacy reasons.
The SkyFeed app (https://skyfeed.app) got a major update this month, adding the “Feed Builder”. It allows anyone to build a custom feed using a visual editor and then publish it for anyone to use, even in the official app. At the moment, there are already 369 custom feeds using it published on the network.
Progress Report (July)
S5 Network
The S5 Node got a minor new release which changes the default Sia object store prefix to /s5/
. I’ve been focusing on use cases for S5 beyond the Vup app, for example mirroring videos from YouTube and VODs from Twitch to S5/Sia with as much metadata as possible. For Twitch I designed a new, efficient format for archiving the chat log with all messages. Archiving Twitch streams is a great use case for Sia, because they take up a lot of space and need to be available for streaming with not too much latency. Also, Twitch deletes past broadcasts after 7 days for free users and 60 days for paid ones. The next step is finding some Twitch streamers who are interested in trying out the Sia archival feature.
The YouTube mirror is working well, but not automated yet. Here’s the July Sia Community Town Hall streamed directly from Sia: Tube5 Alpha
ATProto and Bluesky
I got video/audio upload to S5/Sia with the SkyFeed app working, which will make it possible to easily publish videos on Bluesky. It only works in the sandbox network, thumbnails and transcoding are not supported yet, but as soon as the production bsky.social instance allows custom lexicons, the feature should be production-ready in 2-3 weeks.
Bluesky continues to grow and is now at 457k registered users. Out of the 5889 custom feeds on Bluesky, 5048 are hosted on SkyFeed infrastructure. It’s awesome seeing so many users building cool feeds using my tool, but with the 2x increase in network usage the old infrastructure could no longer keep up and caused downtimes of 20-50 minutes per day, so I had to focus on a major rewrite last week. Of course hosting ~ 86% of all Bluesky feeds is also a big centralization risk, so all of the infra can be self-hosted (GitHub - skyfeed-dev/skyfeed-me: Feed Generator for feeds created using the SkyFeed builder, running at skyfeed.me) and users can move their published feeds to another provider with a minor change to the metadata. Unfortunately there are no other providers yet, so getting some third-party ones online is a priority.
Next up
- Major S5 Node release with new features and improved account system
- New Vup release (focus on sharing features)
- Mirroring some Twitch channels to S5/Sia (Metadata on ATProto)
- Automate YouTube mirror for new Creative Commons licensed videos (Metadata on ATProto)
- Improve the SkyFeed feed builder and get third-party providers online
- Open-source and release the SkyFeed app for Android and iOS
Hello @redsolver,
Thank you for your July progress report! It looks great.
The Committee has voted to create a progress report template which should make it easier for both you, the Grantee, to provide monthly updates as well as to enable the Committee to quickly review progress without spending too much time on minute details.
For August’s progress report (which should be provided in September), please follow the template outlined in this post: About the Grants category - #8
Thank you for your help and cooperation.
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
Progress Report (August)
What progress was made on your grant this month?
- S5 Node: New release with major improvements to account management and the Admin API + web app
- Vup: Add video player, image gallery view, mass-pinning tool, new more efficient streaming implementation, local config for S3 providers and more
- SkyFeed: Updated query engine and indexer with new features and performance improvements
- S5 web proxy: Fixed some bugs when streaming larger video files
- S3.wiki: Simple website to compare storage providers (https://s3.wiki)
- Got featured on the AT Protocol blog (Featured Community Project: SkyFeed | AT Protocol) and Bluesky blog (Algorithmic Choice with Custom Feeds - Bluesky)
Links to repos worked on this month:
- GitHub - s5-dev/S5: Decentralized content-addressed storage network
- GitHub - vup-app/vup
- GitHub - skyfeed-dev/query-engine: Query Engine used by custom feeds created using the SkyFeed Builder
- GitHub - skyfeed-dev/indexer: ATProto/Bluesky Indexer powered by SurrealDB.
- GitHub - s5-dev/web-proxy: Client-side gateway with verified streaming, running in the web browser
- GitHub - redsolver/s3-wiki: Compare S3 storage providers
What will you be working on this month?
- Release the new Vup Beta version for all platforms
- Get some more third-party S5 nodes and SkyFeed query engine providers online
- Finish the new logo for the SkyFeed app and release it on Android and iOS
- Make tools for publishing public videos on the S5 network more user-friendly
- Automate Twitch and YouTube S5 media mirrors
Progress Report (September)
What progress was made on your grant this month?
- Released a new Vup Beta version for all platforms (see Release Version 0.15.0 BETA · vup-app/vup · GitHub for full changelog with new features)
- Released a new S5 Node version (Release Version 0.13.3 · s5-dev/S5 · GitHub)
- Fixed some video streaming bugs in the S5 web proxy
- Finished the logo for SkyFeed and added a lot of new features (https://skyfeed.app/)
- Updated SkyFeed Stats website: https://stats.skyfeed.me/
Links to repos worked on this month:
- GitHub - vup-app/vup
- GitHub - s5-dev/S5: Decentralized content-addressed storage network
- GitHub - s5-dev/web-proxy: Client-side gateway with verified streaming, running in the web browser
- GitHub - skyfeed-dev/indexer: ATProto/Bluesky Indexer powered by SurrealDB.
- GitHub - skyfeed-dev/query-engine: Query Engine used by custom feeds created using the SkyFeed Builder
What will you be working on this month?
- The biggest challenge for S5 is the lack of public third-party nodes/services, so that’s still the highest priority
- Release the SkyFeed app for Android and iOS, migrate to SurrealDB 1.0 and prepare support for streaming media from the S5 network
- New features for the Vup app, for example a real-time collaborative text editor
Progress Report (October)
What progress was made on your grant this month?
- Added renterd bucket support to the S5 Node and mirrored ~ 1 TB of creative-commons-licensed video files from YouTube
- Created a video tutorial for installing and setting up Vup: https://www.youtube.com/watch?v=B21Z__xgssU The goal is to significantly increase the Vup user count (with easy-to-setup storage services like Pixeldrain) and use that as leverage to get native third-party S5 Node providers online
- Updated https://s3.wiki to make comparing storage providers easier
- Published a major new SkyFeed update, including the first release for Android (Release Version 0.9.2 · skyfeed-dev/app · GitHub)
- Updated SkyFeed Stats website with graphs: https://stats.skyfeed.me/
- Migrated to SurrealDB 1.0 and fixed a lot of SkyFeed Builder performance issues
Links to repos worked on this month:
- GitHub - s5-dev/S5: Decentralized content-addressed storage network
- GitHub - skyfeed-dev/app
- GitHub - skyfeed-dev/indexer: ATProto/Bluesky Indexer powered by SurrealDB.
- GitHub - skyfeed-dev/query-engine: Query Engine used by custom feeds created using the SkyFeed Builder
- GitHub - redsolver/s3-wiki: Compare S3 storage providers
- GitHub - flathub/app.vup.Vup
What will you be working on this month?
- Create a CLI tool to transcode and publish videos directly to the S5 network (using ffmpeg). It will be written in Dart and also shipped as part of the next Vup release
- Getting third-party S5 nodes/services online
- Use S5 to create regular snapshots/archives of the Bluesky network, this will make it easier for third-party providers to self-host the query infrastructure
- Write a spec for the S5 Identity system and use it to sync SkyFeed account data
- Self-hosted bridge proxy for the AT Protocol which bridges other networks
- SkyFeed: Push all source code to GitHub
- SkyFeed: Add List Builder and other new features
SkyFeed App Changelog for October
- The source code of any feed published using the Builder can now be viewed in the app, and forked if it has an open-source license
- The sidebar is now collapsible and shows a list of trending hashtags
- You can now create multiple workspaces/decks and quickly switch between them
- Added Gallery View for all feeds
- Feed Builder: Highlight RegEx matches in preview feed
- Feed Builder: You can now paste post and list links directly instead of needing to convert them to at:// URIs manually
- Feed Builder: New “Limit” block
- Feed Builder: New “Embed” target for the “Remove” block (for example remove all quote posts or only keep posts which link to a custom feed)
- The RegEx text field is now up to 24x bigger! (only if needed)
- “Single User” input blocks now optionally include like, reply and repost counts
- Improved mobile layout
- Create Post: Set Post Language
- Create Post: Embed (quote) posts and feeds
- Create Post: Hashtags are now extracted automatically
- Create Post: Images you post now add aspect ratio metadata
- New Theme: Light Default
- New Theme: Black (OLED)
- Virtual Feeds for all hashtags on the network (sorted by New, Hot, Rising, Top or Random)
- Trending hashtags in sidebar (10min, 1h, 4h, 10h, 24h, 3d, 7d)
- New UI for replies
- Settings: Default Column Width
- Settings: Font Size
- Settings: Font Family
- Settings: New button to reset all open columns to default width
- New more efficient WebSocket stream for real-time updates
- Re-order columns with drag n’ drop
- Other Bug fixes and UI improvements
Progress Report (November)
What progress was made on your grant this month?
- Added experimental FileSystemProvider to the S5 Node. This is a “virtual” store which efficiently indexes any directory on your local FS and then makes it available in the S5-native directory metadata format directly on the network. I set it up to serve regular snapshot archives of the entire Bluesky network on S5 and mirror them to my Sia node. These snapshots will make it very easy for new third-party SkyFeed query engine providers to bootstrap their local database. The FS Provider can also be used for mirroring Linux package registries or any other use case where there’s already a good directory structure with a lot of large files.
- Published first version of the AT Proto Bridge Proxy - This piece of software makes it possible to view Mastodon feeds, YouTube channels, RSS feeds and Hacker News in any Bluesky client you want (in addition to all of the native Bluesky content) and efficiently organize all of them in lists and feeds.
- Researched some more serialization formats to decide on a final option for S5 metadata formats (for directories, web apps and media) - I will likely continue using msgpack because of the excellent ecosystem and extremely good performance+efficiency
- SkyFeed Query Engine: Added trending tags, new powerful list mode and improved performance
- SkyFeed App: Added List Architect UI (makes it possible to create giant lists based on RegEx-searching usernames, domain handles and profile descriptions) and fixed some bugs
Links to repos worked on this month:
- GitHub - s5-dev/S5: Decentralized content-addressed storage network
- GitHub - skyfeed-dev/bridge-proxy
- GitHub - skyfeed-dev/query-engine: Query Engine used by custom feeds created using the SkyFeed Builder
- GitHub - skyfeed-dev/indexer: ATProto/Bluesky Indexer powered by SurrealDB.
What will you be working on this month?
- Make the FS Provider production-ready and publicly serve some datasets like the Bluesky snapshot archives (can be downloaded using the Vup app)
- Write a spec for the S5 Identity system and docs for the different metadata formats
- Create a CLI tool to transcode and publish videos directly to the S5 network (using ffmpeg). It will be written in Dart and also be shipped as part of the next Vup release
- SkyFeed: Push all source code to GitHub and use S5 Identity system to sync account data
- Getting third-party S5 nodes/services online
Progress Report (December)
What progress was made on your grant this month?
- Improved S5 Node FileSystemProvider, a full mirror of the archlinux package repository is now hosted on S5:
skyfs://7R7fFlGQrZH2BK0qZ7a7U6GrwUyAGqqnXM1ETtARkieR@shared-readonly
(can be browsed with the next Vup version) - Published S5 Node v0.13.5
- Created a new library for publishing media content on S5 with correct metadata
- Vup App: Added video publishing dialog, fixed a bug which prevented browsing public directories (like the archlinux mirror), uploaded one TB of small to medium-sized files to test scalability, UI improvements and bug fixes
- Wrote a new guide for setting up a S5 Node with Sia-backed storage (needs v0.13.5 or later): Setup With Sia - S5 Network Docs
- SkyFeed: Improved infrastructure and stats service: https://stats.skyfeed.me
- SkyFeed: Some new features, see @skyfeed.app on Bluesky
Links to repos worked on this month:
- GitHub - s5-dev/S5: Decentralized content-addressed storage network
- GitHub - s5-dev/docs
- GitHub - s5-dev/media_uploader
- GitHub - vup-app/vup
- GitHub - redsolver/SkyFS
What will you be working on this month?
- Write a spec for the S5 Identity system and metadata formats
- Ask projects using S5 which features they need/what’s missing, aggregate the results and use them to determine new milestones for the S5 project
- Publish a new Vup release
- SkyFeed: Push all source code to GitHub, use S5 Identity system to sync account data and publish new Android version
- Getting third-party S5 nodes/services online
Progress Report (January)
What progress was made on your grant this month?
- I did a lot of research on p2p technologies and data structures to validate the design decisions made in S5 and designed a new “Message Stream” primitive. It can be used to efficiently transfer a stream of messages in real-time over the network. I made a demo app where everyone trying it can chat with each other: https://demo.sfive.net
- Using the new “Message Stream” primitive, I started working on a decentralized chat app using OpenMLS (https://openmls.tech/) for end-to-end encrypted and scalable group chats. It’s written in Flutter+Rust, runs on all platforms (including web) and basic chat features like creating/joining groups and sending messages are already working well!
- Created a new “s5” Dart package intended for developers to easily add all S5 features to their Flutter or Dart app with minimal effort (compared to lib5, which is more low-level). lib5 now also includes a full lightweight S5 node implementation (works on web!) and other improvements like new APIs (identity, accounts, message streams) and better routing
- Added support for directory metadata to cid.one, for example: S5 CID Explorer
- Wrote S5 blog post for the Sia Blog
- SkyFeed: New Android release and improved Feed Builder
Links to repos worked on this month:
- GitHub - s5-dev/lib5: This library is used by Dart-based code for the S5 network.
- GitHub - s5-dev/s5-dart: Use the S5 Network in your Flutter and Dart apps
- GitHub - s5-dev/lib5_crypto_implementation_dart: Provides cryptographic primitives for S5 using Dart libraries
What will you be working on this month?
- Publish the new Dart packages on pub.dev to make them easy for developers to add to their projects and use
- Add more features to the Flutter demo app (for example accounts and the virtual filesystem) and publish its source code, so developers can learn how to use S5 by example
- Write a spec for the S5 Identity system and metadata formats
- Improve and extend S5 documentation
- Maybe publish a new Vup version, after upgrading to the new Dart libraries
- SkyFeed: Push all source code to GitHub and use S5 Identity system to sync account data
Hello @redsolver,
Thank you for your progress report!
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
Progress Report (February)
What progress was made on your grant this month?
- Wrote (and tested) a complete guide for deploying a production-ready S5 Node backed by renterd on a new machine: Deploy S5 Node with Sia renterd - S5 Network Docs
- Published the brand-new
s5
package on pub.dev: s5 | Dart package This package is designed to make it as easy as possible for developers to add S5 features to their Dart and Flutter apps, servers and websites! - Published the Flutter/Dart S5 Demo App which uses the new package: GitHub - s5-dev/docs Made for developers to understand how to use
s5
and discover all the available features - Updated lower-level S5 Dart packages (see Packages of publisher sfive.net)
- Migrated cid.one to the new
s5
package (so it now connects directly to the network and runs a full node in the browser), published source code and added syntax highlighting (example page: S5 CID Explorer) - The AT Protocol (which Bluesky is based on) added support for custom lexicons (record types) this month, so I published a custom video lexicon which stores the metadata on ATP and streams all media blobs directly from the S5 Network: @red.pds.skyfeed.dev on Bluesky (tube5.app is the video player)
Links to repos worked on this month:
- GitHub - s5-dev/docs
- GitHub - s5-dev/s5-dart: Use the S5 Network in your Flutter and Dart apps
- GitHub - s5-dev/flutter_demo_app: Flutter app which showcases some S5 features using the s5 package
- GitHub - s5-dev/cid-explorer: Web App for exploring and debugging S5 CIDs
- GitHub - s5-dev/lib5: This library is used by Dart-based code for the S5 network.
- GitHub - s5-dev/lib5_crypto_implementation_dart: Provides cryptographic primitives for S5 using Dart libraries
What will you be working on this month?
- Add more features to the Flutter demo app (for example accounts and the virtual filesystem)
- Write a spec for the S5 Identity system and metadata formats
- Improve and extend S5 documentation
- Design some new custom AT Proto lexicons with big S5-powered media blobs (for example music and podcasts)
- SkyFeed: Add S5 Identity System and connect it with S5 storage services for storing backups of personal user data (in case a PDS goes offline)
Hello @redsolver,
Thank you for your progress report!
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
Progress Report (March)
What progress was made on your grant this month?
- The S5 file system (FS5) is a mutable data structure built on top of S5 primitives. This month, I wrote a new implementation of it for lib5. It has improved support for public directory trees, comes with a new, more consistent URI format, integrates nicely with accounts and other S5 APIs and most importantly, does not contain any legacy code from the Skynet days anymore!
- I also added a completely new “Snapshot” feature to FS5. With Snapshots, users and developers can create short immutable CIDs for deep directory trees (both public and private) pretty much instantly. They are great for keeping a personal archived copy of a public directory, or just private backups
- There’s now a second S5 Node implementation written in Golang using renterd! It will also power an independent third-party node (pinner.xyz), which I’m really excited about. I spent some time testing it with @pcfreak30 and fixed some bugs found in S5 apps during that
- The Admin Web App for managing accounts on S5 Nodes using the Admin API is now published on GitHub
- I spent a week following the DMA (Digital Markets Act) workshops organized by the EU Commission. Different Gatekeepers (Apple, Meta, Google, Amazon, Microsoft, ByteDance) explained how they intend to comply with the DMA. The workshops gave some interesting insights on new features like data portability, which could for example enable Instagram or Google users to set up automatic backups of all their personal data directly to a S5/Sia-powered storage service. The DMA also comes with interoperability requirements, so for example Vup Chat users could directly message WhatsApp users in the future! There were also a lot more fun topics covered, so if you’re interested in chatting about them, feel free to ping me somewhere
- Fixed storage locations not showing up on the CID Utils web app (cid.one)
- Small Improvements to the Sia Setup Guide in the S5 Docs
Links to repos worked on this month:
- GitHub - s5-dev/lib5: This library is used by Dart-based code for the S5 network.
- GitHub - s5-dev/node-admin-web-app: Simple Web App to manage accounts on a S5 Node using the Admin API. Your S5 Node serves this at /s5/admin/app
- GitHub - s5-dev/cid-explorer: Web App for exploring and debugging S5 CIDs
- GitHub - s5-dev/docs
What will you be working on this month?
- Add FS5 and Account features to the S5 Demo App (so users and devs can try them)
- Add a simple real-time collaborative text editor to the demo app
- Add a HTTP redirect API endpoint for S5 file systems to the S5 Node, this will for example make it possible to use it as a Linux Distro package mirror
- With the new FS, archives of data can now be better organized. So I will extend the existing S5 archiving tools for YouTube (and Twitch) videos to store data in a proper directory structure which should help a lot with discovery and working together on giant archives
- Instead of developing a completely new App Store powered by S5 (see initial grant proposal), I will try to add content-addressed storage features to the very good open-source Obtainium App Store (GitHub - ImranR98/Obtainium: Get Android App Updates Directly From the Source.) It’s written in Flutter/Dart, so all S5 libraries should be easy to integrate!
- I wanted to write specifications for the S5 Identity system for a while, but due to the core data structures still changing fast, I didn’t do it yet. Now that the FS is in a more or less stable state, I feel comfortable writing a proper spec
Progress Report (April)
What progress was made on your grant this month?
- Added
/s5/fs/browse
API to the S5 Node. This makes it possible to easily browse both encrypted and plaintext directories like this: Index of archlinux/ When trying to download a file it redirects to the S5-native storage location, which makes this a complete HTTP file server. For example, the/browse/CID
URL can just be used as an archlinux package mirror with the pacman package manager as-is! - Published S5 archiver (GitHub - s5-dev/archiver: Tool to archive websites and other content available on the Internet on the content-addressed S5 Network), starting with web support. Downloads website files and stores them in an optimized FS5 directory structure (public or private). Can then be browsed like a personal web archive or used for a more coordinated global content-addressed web archiving effort. More archiving types (like YT videos or git repos) will be added soon
- Added File System demo to https://demo.sfive.net/
- Added File System wrapper to s5-dart
- lib5: Added FS5 methods
createOrUpdateFile
,listDirectoryRecursive
,getFileReference
andgetDirectoryCID
+ some other improvements to lib5 - docs: Fixed some outdated details
Links to repos worked on this month:
- GitHub - s5-dev/lib5: This library is used by Dart-based code for the S5 network.
- GitHub - s5-dev/s5-dart: Use the S5 Network in your Flutter and Dart apps
- GitHub - s5-dev/archiver: Tool to archive websites and other content available on the Internet on the content-addressed S5 Network
- GitHub - s5-dev/docs
- GitHub - s5-dev/flutter_demo_app: Flutter app which showcases some S5 features using the s5 package
What will you be working on this month?
- Wrap up this grant with a final progress report and submit a new S5 grant proposal for the next 12 months
- Add account utils and a simple collaborative plain text editor to the S5 Demo App
- Deploy a permanent instance of the web archiver using Sia and start preserving some important websites
- Extend the archiver to support GIT repositories, AT Protocol personal data archives and YouTube videos
- Write specifications for the S5 Identity system, file system and metadata formats
- Update the
skydeploy
tool to support publishing web apps on S5 using a simple CLI interface
Hello @redsolver,
Thank you for your progress report!
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
Hello @redsolver
I understand that your proposal for S5 Network 2024 (Standard Grant Proposal) is intended to act as your final progress report, however, for documentation purposes we ask that you do a write-up on this thread detailing the progress your grant has achieved.
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
Final Progress Report for the “S5 Network and Apps” Grant
This is a write-up detailing the progress the entire first “S5 Network and Apps” Grant has achieved in the last 12 months. It acts as the 14. and final progress report for my first 12-month grant and is also used as an introduction for my new grant proposal: S5 Network 2024 (Standard Grant Proposal)
This blog post gives a good overview over high-level achievements of the S5 Network during the first year: Community Spotlight: S5. A Content-Addressed Data Routing… | by Skunk_Ink | The Sia Blog
And there’s also a S5 segment in this Sia Community Town Hall with some more recent updates: https://www.youtube.com/watch?v=JEtFG3yLjWI&t=580
S5 Node
The S5 Node is written in Dart+Rust and can be easily deployed on most systems using a Docker container. It provides an account system and APIs for uploading and managing small and big blobs. The node connects to the public S5 network (configurable) and stores data in one or multiple blob stores. Currently supported are Sia renterd, S3 (incl. renterd and many others), the local file system, IPFS and Pixeldrain. It can easily handle blobs with a size of 110 GiB each (that’s the largest I tested for now, bigger ones should work too with enough storage space). For renterd you can either use the native or the S3 store. The S3 one is slightly more reliable if you’re hosting the S5 Node on a different server than renterd and is easier to set up. The native renterd store is more efficient and slightly faster, but a bit more complex to set up.
The S5 Node also supports an experimental “File System Provider”. This is a “virtual” store which efficiently indexes any directory on your local FS and then makes it available in the S5-native directory metadata format directly on the network. It can for example be used for mirroring Linux package registries or any other use case where there’s already a good directory structure with a lot of large (or small) files. Some other fun new features include being able to import files from the internet by just passing their HTTP(s) URLs. The S5 Node also supports DNSLink (https://dnslink.dev/) to host websites and web apps on the S5 Network.
There’s also a second implementation of the S5 Node written in Golang by @pcfreak30 which I’m very excited about! I helped debug it and fixed some bugs in the S5 specification in the process.
GitHub Repository: GitHub - s5-dev/S5: Decentralized content-addressed storage network
Admin Web App
The Admin Web App (GitHub - s5-dev/node-admin-web-app: Simple Web App to manage accounts on a S5 Node using the Admin API. Your S5 Node serves this at /s5/admin/app) is a simple web app integrated in the S5 Node to manage accounts and their storage usage with a nice UI. (available on /s5/admin/app
)
FS5
The S5 file system (FS5) is a mutable data structure built on top of S5 primitives which supports both public and private (end-to-end-encrypted) directories. It’s for example being used by Vup and will also be used by some other apps pretty soon (after the spec is stable). The FS is pretty efficient for most operations and only needs two of the S5 primitives (blobs and the registry). One of the more recent new features is Snapshots, which makes it possible to create immutable snapshots of whole directory trees instantly (if cached).
S5 Nodes have a public HTTP endpoint for browsing FS5 directories (downloads redirect, the S5 Node does NOT proxy file content!): https://sia5.net/s5/fs/browse/z389tnQFiQFAHUPbtWSQynBBcnXzk61BLPJYoB3WrFjiY6Ka/archlinux/
Vup Cloud Storage App
Vup is the flagship app of the S5 protocol and network. It’s a decentralized end-to-end-encrypted cloud storage app with support for many storage backends (including Sia) and apps for Android, Linux, Windows, macOS and iOS.
During my first grant, I published a massive new release which fully migrated Vup from the old Skynet backend and data structures to the S5 Network. I also added a video player, image gallery view, mass-pinning tool, new more efficient streaming implementation, video publishing dialog, fixed a bug which prevented browsing public directories (like the archlinux mirror), uploaded some TBs of small to medium-sized files to test scalability and implemented some UI improvements and bug fixes. Vup also runs a local S5 Node and supports configuring it with a self-hosted Sia renterd instance, but due to that abstracting some important parts away, users are advised to instead host a S5 Node alongside their renterd installation in the future (see https://docs.sfive.net/guide/deploy-renterd.html). With renterd getting more lightweight in the future (especially with reduced local state requirements), I could see shipping renterd as part of Vup and making setup as easy as possible.
Vup also has rclone and restic support, can do media streaming with an integrated Jellyfin-compatible API server, integrated sync, scripts and a lot more features! (check out the website)
Website: https://vup.app/
Source Code: GitHub - vup-app/vup
Available on Flathub: Vup Cloud Storage auf Linux installieren | Flathub
lib5 Dart
GitHub Repo: GitHub - s5-dev/lib5: This library is used by Dart-based code for the S5 network.
Packages: lib5 | Dart package and lib5_crypto_implementation_dart | Dart package
lib5 contains most of the S5 data structures and code. It’s designed to be modular and different parts like the API implementation or crypto provider (for hashing, signing, encryption) can be swapped out. It also runs on any platform supported by Dart, like Linux, Android, iOS, Windows, macOS and the Web!
It has classes to work with CIDs, Multihashes and S5 file metadata, registry entries, contains a full S5 Node implementation which can run anywhere, some utils and more!
In March, I wrote a new implementation of the S5 file system for lib5. It has improved support for public directory trees, comes with a new, more consistent URI format, integrates nicely with accounts and other S5 APIs and most importantly, does not contain any legacy code from the Skynet days anymore!
s5-dart
The s5
Dart package is intended for developers to easily add all S5 features to their Flutter or Dart app with minimal effort (compared to lib5, which is more low-level). It wraps the lib5
package and pre-configures almost everything for you, so it’s easier to integrate S5 with your app.
-
Runs a local S5 Node and connects to default nodes (can be configured)
-
Provides all S5 APIs (file, registry, streams) with full integrity verification
-
Supports creating and restoring user identities and uploading to third-part storage services (nodes)
Package: s5 | Dart package
Source Code: GitHub - s5-dev/s5-dart: Use the S5 Network in your Flutter and Dart apps
There’s also a Flutter-based demo web app showcasing some basic features for devs (still WIP):
https://demo.sfive.net/ (GitHub - s5-dev/flutter_demo_app: Flutter app which showcases some S5 features using the s5 package)
Web Proxy
The S5 Web Proxy can securely stream files of any size from the S5 network, directly in your web browser. It should work in all major web browsers, including Chrome, Brave, Firefox and Safari.
It works by registering and running a service worker which intercepts specific requests, looks up download URLs for them and then verifies their integrity with BLAKE3/bao running in Rust via WASM.
After adding the service worker to your web app, streaming files from the S5 network is as easy as using the URL /s5/blob/YOUR_CID_HERE?mediaType=image%2Fpng
in img tags, for your video player or just fetching it directly! Encrypted files are supported too.
Source Code: GitHub - s5-dev/web-proxy: Client-side gateway with verified streaming, running in the web browser
Some example apps using this proxy:
-
s5.cx for very basic file sharing, for example encrypted ones: Encrypted file
-
Tube5 Alpha Simple decentralized video platform, uses the s5 web proxy for streaming video, audio, images and text.
-
a web-based map view, powered by the PMTiles format. (no longer online, previously on https://map-demo.sfive.cloud/)
CID Explorer
Web App for exploring and debugging S5 CIDs
Source Code: GitHub - s5-dev/cid-explorer: Web App for exploring and debugging S5 CIDs
Example: S5 CID Explorer
More examples: cid.one - S5 Network Docs
Documentation
-
Started working on specifications for a “stable v1” version of the S5 data structures and formats: Blobs - S5 Network Docs
-
Full Sia Setup Guide: https://docs.sfive.net/guide/deploy-renterd.html
-
Explaining basic concepts like content-addressed data: Content-addressed data and CIDs - S5 Network Docs
-
Documentation for running a S5 Node and configuring different stores
-
Uses Markdown with mdbook: GitHub - s5-dev/docs
Archiver
Archiving makes sense for personal archive use cases, for preserving history and satisfying your data hoarder needs. I’m also planning to support Twitch VODs due to them needing a ton of space and Twitch deleting past broadcasts after 7 days for free users and 60 days for paid ones by default, so streamers might be interested in having a good alternative.
I started working on an archiving tool to archive websites and other content available on the Internet on the content-addressed S5 Network. It downloads website files and stores them in an optimized FS5 directory structure (public or private). Can then be browsed like a personal web archive or used for a more coordinated global content-addressed web archiving effort. (Still WIP: GitHub - s5-dev/archiver: Tool to archive websites and other content available on the Internet on the content-addressed S5 Network)
It currently only properly supports web archives, but you can check out this video for an early demo of archiving YouTube videos: https://www.youtube.com/watch?v=JEtFG3yLjWI&t=580
AT Protocol and SkyFeed
The original timeline for the S5 project included a full public identity system with a browser extension for secure client-side signatures, social network elements for the Tube5 video streaming app and a Discord-like decentralized chat platform.
Instead of continuing to build all the primitives and core tech for this myself, I decided to switch to https://atproto.com/. The AT Protocol offers almost all features needed to build truly decentralized applications, and it is already a lot more mature compared to what I could build from scratch. With ATProto, you own your identity, can easily switch between different services and provider without lock-in and there’s both composable curation (via custom feeds) and moderation.
The most relevant app on ATProto right now is Bluesky, which just hit 5 million users. It’s a microblogging app similar to Twitter.
Similar to how S5 works, the AT Protocol can support multiple different transport layers for social data, identity and media blobs. S5 can be one of them where it makes sense, like for a decentralized CDN or external backups for account recovery.
I built the SkyFeed client for Bluesky (https://skyfeed.app) which will make it easier to test and deploy new S5-powered features on the network in the future. The most used feature of SkyFeed is building custom feeds, of which there are already over 39000 ones: https://stats.skyfeed.me/. It also got featured in the bsky blog: Featured Community Project: Skyfeed | Bluesky
The query engine and other components of the Feed Builder can be self-hosted: GitHub - skyfeed-dev/query-engine: Query Engine used by custom feeds created using the SkyFeed Builder
I also started working on a bridge proxy to bridge content from other networks over (client-side): GitHub - skyfeed-dev/bridge-proxy
Ideas from the initial grant proposal which weren’t realized (yet) and why
Among the extensive quantity of things that I did implement and publish, there are also some ideas from my initial grant proposal which I didn’t work on yet for various reasons. So I would like to list them and briefly explain why that’s the case and how their future looks like:
Vup Chat (previously S5 Chat)
I tried the new OpenMLS standard for encrypted communication (https://openmls.tech/) using S5 as the transport and identity layer, but I simply didn’t have enough time to properly build it yet. So @Covalent will be working on this as part of his already approved grant: https://forum.sia.tech/t/small-grant-s5-ecosystem-vup-chat
SkyDroid (decentralized App Store for Android)
Since I created SkyDroid, a new decentralized App Store for Android appeared: Obtainium (GitHub - ImranR98/Obtainium: Get Android app updates straight from the source.). It’s written in Flutter+Dart and already decentralized by downloading updates directly from the developer’s source (for example GitHub releases or F-Droid). So there’s no longer a need for me to build yet another App Store, but I’m looking into extending Obtainium to support S5 as a decentralized CDN layer.
Build a simple browser extension for S5 identity management in all major browsers including Chrome
My work on bsky-related software made me realize that most users don’t care about this (yet), so it’s more important to focus on portable identity and truly secure (native) software first. A browser extension to make everything fully trustless on the web (in addition to native) can then still be added later when needed, when the network grows more and users want to migrate over.
Create high-level bindings for TypeScript/JavaScript
Due to the S5 specifications not being stable yet and changing quickly, writing libraries for TypeScript/JS didn’t make sense for me yet. However, there are already some third-party ones from @pcfreak30 for Lume’s use of S5, but I will write an official one after the v1 specs for S5 are done (see Milestones in my next grant).