Grant Proposal: S5 Network and Apps

Grants Approved
21

Thank you for the report, everything looks good so far!

22

Progress Report (May)

S5 Network

Version 0.11.3 of the S5 Node has been released (GitHub - s5-dev/S5: Decentralized content-addressed storage network). Highlights include a new Admin Web UI, multi-arch Docker image, WebSocket connections for the p2p network, a new file streaming implementation and some improvements to the Sia renterd object store.

S5 Dart packages have been published on pub.dev, this makes it easy for Dart and Flutter developers to install and use them. (Packages of publisher sfive.net)

Vup Cloud Storage App

A lot of new features and improvements have been implemented since the last Beta release, but a new one is not quite ready yet. The next release will happen this month and be finally available on all platforms.

ATProto and Bluesky

The original timeline for the S5 project included a full public identity system with a browser extension for secure client-side signatures, social network elements for the Tube5 video streaming app and a Discord-like decentralized chat platform.

Instead of continuing to build all of the primitives and core tech for this myself, I decided to switch to https://atproto.com/. The AT Protocol offers almost all features needed to build truly decentralized applications and it is already a lot more mature compared to what I could build from scratch.

The most relevant app on ATProto right now is Bluesky, which just hit 100k users. Itā€™s a microblogging app similar to Twitter.

This are some examples where S5 could be used with ATProto and Bluesky:

  • Host large media files, for example videos or big images
  • Store content-addressed backups of ATProto repos to make account recovery even easier
  • Publish web apps securely
  • Archive content

After custom lexicon support lands on the official instance, videos in the S5 metadata format (GitHub - s5-dev/json-schemas) can be published directly on ATProto with only some minor modifications. Tube5 will switch to publishing video metadata on ATProto, but the video files will still be streamed from S5 by default. If everything goes well, the ATProto blob format will be fully compatible with S5 (both using the BLAKE3 hash function) and large media files can be hosted on both S5 and ATProto servers using the same hash. Iā€™m also working on some custom lexicons for building a Reddit-like social platform on top of ATProto: GitHub - redsolver/atproto

Apart from custom lexicons, I built a lot fun stuff for the Bluesky ecosystem this month. The first one is SkyFeed (https://skyfeed.app), a third-party client with some unique features like real-time updates. SkyFeed was the first Bluesky client to support custom feeds, you can read more about that and other cool new Bluesky features in their recent blog post (Private Beta Update & Roadmap - Bluesky). I also built an indexer (GitHub - skyfeed-dev/indexer: ATProto/Bluesky Indexer powered by SurrealDB.) which powers all of my custom feeds. They serve up to 400 feed requests per minute (~ 6.7 reqs/second).

1 Like
23

Thanks for the monthly report redsolver!

24

Progress Report (June)

S5 Network

The S5 Node now supports deleting files on Sia and fetching stats about the storage usage. Thereā€™s also a new experimental IPFS Object Store which stores files on an IPFS node and has the same hash for files smaller than 1 MiB, so they can be fetched from both networks (S5 and IPFS) (Commits Ā· s5-dev/S5 Ā· GitHub)

Vup Cloud Storage App

The first public Vup Beta powered by S5 was released for all platforms! Version 0.14.3 BETA is available for Windows, Linux, macOS, iOS and Android. It can be used with a local instance of renterd, a self-hosted S5 Node or a third-party node. All data is fully end-to-end-encrypted and Vup can be configured to store metadata and/or files on multiple remotes for redundancy and even higher reliability. Installation instructions are available on Install Vup

ATProto and Bluesky

I created a lexicon/schema for publishing videos, audio and other media content on ATProto. Thereā€™s a new federated sandbox network and I published a test video there: Tube5 Alpha The metadata is stored on ATProto and integrates with Bluesky social features, the media blobs are compatible with ATProto CIDs and streamed from the S5 Network.

Bluesky continues to grow and is now at 217k registered users. Iā€™m hosting some custom feeds and they peaked at ~ 1000 requests/minute this week. Iā€™m not counting individual users for privacy reasons.

The SkyFeed app (https://skyfeed.app) got a major update this month, adding the ā€œFeed Builderā€. It allows anyone to build a custom feed using a visual editor and then publish it for anyone to use, even in the official app. At the moment, there are already 369 custom feeds using it published on the network.

image
image1920Ɨ1505 183 KB

26

Progress Report (July)

S5 Network

The S5 Node got a minor new release which changes the default Sia object store prefix to /s5/. Iā€™ve been focusing on use cases for S5 beyond the Vup app, for example mirroring videos from YouTube and VODs from Twitch to S5/Sia with as much metadata as possible. For Twitch I designed a new, efficient format for archiving the chat log with all messages. Archiving Twitch streams is a great use case for Sia, because they take up a lot of space and need to be available for streaming with not too much latency. Also, Twitch deletes past broadcasts after 7 days for free users and 60 days for paid ones. The next step is finding some Twitch streamers who are interested in trying out the Sia archival feature.

The YouTube mirror is working well, but not automated yet. Hereā€™s the July Sia Community Town Hall streamed directly from Sia: Tube5 Alpha

ATProto and Bluesky

I got video/audio upload to S5/Sia with the SkyFeed app working, which will make it possible to easily publish videos on Bluesky. It only works in the sandbox network, thumbnails and transcoding are not supported yet, but as soon as the production bsky.social instance allows custom lexicons, the feature should be production-ready in 2-3 weeks.

Bluesky continues to grow and is now at 457k registered users. Out of the 5889 custom feeds on Bluesky, 5048 are hosted on SkyFeed infrastructure. Itā€™s awesome seeing so many users building cool feeds using my tool, but with the 2x increase in network usage the old infrastructure could no longer keep up and caused downtimes of 20-50 minutes per day, so I had to focus on a major rewrite last week. Of course hosting ~ 86% of all Bluesky feeds is also a big centralization risk, so all of the infra can be self-hosted (GitHub - skyfeed-dev/skyfeed-me: Feed Generator for feeds created using the SkyFeed builder, running at skyfeed.me) and users can move their published feeds to another provider with a minor change to the metadata. Unfortunately there are no other providers yet, so getting some third-party ones online is a priority.

Next up

  • Major S5 Node release with new features and improved account system
  • New Vup release (focus on sharing features)
  • Mirroring some Twitch channels to S5/Sia (Metadata on ATProto)
  • Automate YouTube mirror for new Creative Commons licensed videos (Metadata on ATProto)
  • Improve the SkyFeed feed builder and get third-party providers online
  • Open-source and release the SkyFeed app for Android and iOS
27

Hello @redsolver,

Thank you for your July progress report! It looks great.

The Committee has voted to create a progress report template which should make it easier for both you, the Grantee, to provide monthly updates as well as to enable the Committee to quickly review progress without spending too much time on minute details.

For Augustā€™s progress report (which should be provided in September), please follow the template outlined in this post: About the Grants category - #8

Thank you for your help and cooperation.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

28

Progress Report (August)

What progress was made on your grant this month?

  • S5 Node: New release with major improvements to account management and the Admin API + web app
  • Vup: Add video player, image gallery view, mass-pinning tool, new more efficient streaming implementation, local config for S3 providers and more
  • SkyFeed: Updated query engine and indexer with new features and performance improvements
  • S5 web proxy: Fixed some bugs when streaming larger video files
  • S3.wiki: Simple website to compare storage providers (https://s3.wiki)
  • Got featured on the AT Protocol blog (Featured Community Project: SkyFeed | AT Protocol) and Bluesky blog (Algorithmic Choice with Custom Feeds - Bluesky)

Links to repos worked on this month:

What will you be working on this month?

  • Release the new Vup Beta version for all platforms
  • Get some more third-party S5 nodes and SkyFeed query engine providers online
  • Finish the new logo for the SkyFeed app and release it on Android and iOS
  • Make tools for publishing public videos on the S5 network more user-friendly
  • Automate Twitch and YouTube S5 media mirrors
29

Progress Report (September)

What progress was made on your grant this month?

Links to repos worked on this month:

What will you be working on this month?

  • The biggest challenge for S5 is the lack of public third-party nodes/services, so thatā€™s still the highest priority
  • Release the SkyFeed app for Android and iOS, migrate to SurrealDB 1.0 and prepare support for streaming media from the S5 network
  • New features for the Vup app, for example a real-time collaborative text editor
1 Like
30

Progress Report (October)

What progress was made on your grant this month?

  • Added renterd bucket support to the S5 Node and mirrored ~ 1 TB of creative-commons-licensed video files from YouTube
  • Created a video tutorial for installing and setting up Vup: https://www.youtube.com/watch?v=B21Z__xgssU The goal is to significantly increase the Vup user count (with easy-to-setup storage services like Pixeldrain) and use that as leverage to get native third-party S5 Node providers online
  • Updated https://s3.wiki to make comparing storage providers easier
  • Published a major new SkyFeed update, including the first release for Android (Release Version 0.9.2 Ā· skyfeed-dev/app Ā· GitHub)
  • Updated SkyFeed Stats website with graphs: https://stats.skyfeed.me/
  • Migrated to SurrealDB 1.0 and fixed a lot of SkyFeed Builder performance issues

Links to repos worked on this month:

What will you be working on this month?

  • Create a CLI tool to transcode and publish videos directly to the S5 network (using ffmpeg). It will be written in Dart and also shipped as part of the next Vup release
  • Getting third-party S5 nodes/services online
  • Use S5 to create regular snapshots/archives of the Bluesky network, this will make it easier for third-party providers to self-host the query infrastructure
  • Write a spec for the S5 Identity system and use it to sync SkyFeed account data
  • Self-hosted bridge proxy for the AT Protocol which bridges other networks
  • SkyFeed: Push all source code to GitHub
  • SkyFeed: Add List Builder and other new features

SkyFeed App Changelog for October

  • The source code of any feed published using the Builder can now be viewed in the app, and forked if it has an open-source license
  • The sidebar is now collapsible and shows a list of trending hashtags
  • You can now create multiple workspaces/decks and quickly switch between them
  • Added Gallery View for all feeds
  • Feed Builder: Highlight RegEx matches in preview feed
  • Feed Builder: You can now paste post and list links directly instead of needing to convert them to at:// URIs manually
  • Feed Builder: New ā€œLimitā€ block
  • Feed Builder: New ā€œEmbedā€ target for the ā€œRemoveā€ block (for example remove all quote posts or only keep posts which link to a custom feed)
  • The RegEx text field is now up to 24x bigger! (only if needed)
  • ā€œSingle Userā€ input blocks now optionally include like, reply and repost counts
  • Improved mobile layout
  • Create Post: Set Post Language
  • Create Post: Embed (quote) posts and feeds
  • Create Post: Hashtags are now extracted automatically
  • Create Post: Images you post now add aspect ratio metadata
  • New Theme: Light Default
  • New Theme: Black (OLED)
  • Virtual Feeds for all hashtags on the network (sorted by New, Hot, Rising, Top or Random)
  • Trending hashtags in sidebar (10min, 1h, 4h, 10h, 24h, 3d, 7d)
  • New UI for replies
  • Settings: Default Column Width
  • Settings: Font Size
  • Settings: Font Family
  • Settings: New button to reset all open columns to default width
  • New more efficient WebSocket stream for real-time updates
  • Re-order columns with drag nā€™ drop
  • Other Bug fixes and UI improvements
31

Progress Report (November)

What progress was made on your grant this month?

  • Added experimental FileSystemProvider to the S5 Node. This is a ā€œvirtualā€ store which efficiently indexes any directory on your local FS and then makes it available in the S5-native directory metadata format directly on the network. I set it up to serve regular snapshot archives of the entire Bluesky network on S5 and mirror them to my Sia node. These snapshots will make it very easy for new third-party SkyFeed query engine providers to bootstrap their local database. The FS Provider can also be used for mirroring Linux package registries or any other use case where thereā€™s already a good directory structure with a lot of large files.
  • Published first version of the AT Proto Bridge Proxy - This piece of software makes it possible to view Mastodon feeds, YouTube channels, RSS feeds and Hacker News in any Bluesky client you want (in addition to all of the native Bluesky content) and efficiently organize all of them in lists and feeds.
  • Researched some more serialization formats to decide on a final option for S5 metadata formats (for directories, web apps and media) - I will likely continue using msgpack because of the excellent ecosystem and extremely good performance+efficiency
  • SkyFeed Query Engine: Added trending tags, new powerful list mode and improved performance
  • SkyFeed App: Added List Architect UI (makes it possible to create giant lists based on RegEx-searching usernames, domain handles and profile descriptions) and fixed some bugs

Links to repos worked on this month:

What will you be working on this month?

  • Make the FS Provider production-ready and publicly serve some datasets like the Bluesky snapshot archives (can be downloaded using the Vup app)
  • Write a spec for the S5 Identity system and docs for the different metadata formats
  • Create a CLI tool to transcode and publish videos directly to the S5 network (using ffmpeg). It will be written in Dart and also be shipped as part of the next Vup release
  • SkyFeed: Push all source code to GitHub and use S5 Identity system to sync account data
  • Getting third-party S5 nodes/services online
32

Progress Report (December)

What progress was made on your grant this month?

  • Improved S5 Node FileSystemProvider, a full mirror of the archlinux package repository is now hosted on S5: skyfs://7R7fFlGQrZH2BK0qZ7a7U6GrwUyAGqqnXM1ETtARkieR@shared-readonly (can be browsed with the next Vup version)
  • Published S5 Node v0.13.5
  • Created a new library for publishing media content on S5 with correct metadata
  • Vup App: Added video publishing dialog, fixed a bug which prevented browsing public directories (like the archlinux mirror), uploaded one TB of small to medium-sized files to test scalability, UI improvements and bug fixes
  • Wrote a new guide for setting up a S5 Node with Sia-backed storage (needs v0.13.5 or later): Setup With Sia - S5 Network Docs
  • SkyFeed: Improved infrastructure and stats service: https://stats.skyfeed.me
  • SkyFeed: Some new features, see @skyfeed.app on Bluesky

Links to repos worked on this month:

What will you be working on this month?

  • Write a spec for the S5 Identity system and metadata formats
  • Ask projects using S5 which features they need/whatā€™s missing, aggregate the results and use them to determine new milestones for the S5 project
  • Publish a new Vup release
  • SkyFeed: Push all source code to GitHub, use S5 Identity system to sync account data and publish new Android version
  • Getting third-party S5 nodes/services online
34

Progress Report (January)

What progress was made on your grant this month?

  • I did a lot of research on p2p technologies and data structures to validate the design decisions made in S5 and designed a new ā€œMessage Streamā€ primitive. It can be used to efficiently transfer a stream of messages in real-time over the network. I made a demo app where everyone trying it can chat with each other: https://demo.sfive.net
  • Using the new ā€œMessage Streamā€ primitive, I started working on a decentralized chat app using OpenMLS (https://openmls.tech/) for end-to-end encrypted and scalable group chats. Itā€™s written in Flutter+Rust, runs on all platforms (including web) and basic chat features like creating/joining groups and sending messages are already working well!
  • Created a new ā€œs5ā€ Dart package intended for developers to easily add all S5 features to their Flutter or Dart app with minimal effort (compared to lib5, which is more low-level). lib5 now also includes a full lightweight S5 node implementation (works on web!) and other improvements like new APIs (identity, accounts, message streams) and better routing
  • Added support for directory metadata to cid.one, for example: S5 CID Explorer
  • Wrote S5 blog post for the Sia Blog
  • SkyFeed: New Android release and improved Feed Builder

Links to repos worked on this month:

What will you be working on this month?

  • Publish the new Dart packages on pub.dev to make them easy for developers to add to their projects and use
  • Add more features to the Flutter demo app (for example accounts and the virtual filesystem) and publish its source code, so developers can learn how to use S5 by example
  • Write a spec for the S5 Identity system and metadata formats
  • Improve and extend S5 documentation
  • Maybe publish a new Vup version, after upgrading to the new Dart libraries
  • SkyFeed: Push all source code to GitHub and use S5 Identity system to sync account data
35

Hello @redsolver,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

36

Progress Report (February)

What progress was made on your grant this month?

  • Wrote (and tested) a complete guide for deploying a production-ready S5 Node backed by renterd on a new machine: Deploy S5 Node with Sia renterd - S5 Network Docs
  • Published the brand-new s5 package on pub.dev: s5 | Dart package This package is designed to make it as easy as possible for developers to add S5 features to their Dart and Flutter apps, servers and websites!
  • Published the Flutter/Dart S5 Demo App which uses the new package: GitHub - s5-dev/docs Made for developers to understand how to use s5 and discover all the available features
  • Updated lower-level S5 Dart packages (see Packages of publisher sfive.net)
  • Migrated cid.one to the new s5 package (so it now connects directly to the network and runs a full node in the browser), published source code and added syntax highlighting (example page: S5 CID Explorer)
  • The AT Protocol (which Bluesky is based on) added support for custom lexicons (record types) this month, so I published a custom video lexicon which stores the metadata on ATP and streams all media blobs directly from the S5 Network: @red.pds.skyfeed.dev on Bluesky (tube5.app is the video player)

Links to repos worked on this month:

What will you be working on this month?

  • Add more features to the Flutter demo app (for example accounts and the virtual filesystem)
  • Write a spec for the S5 Identity system and metadata formats
  • Improve and extend S5 documentation
  • Design some new custom AT Proto lexicons with big S5-powered media blobs (for example music and podcasts)
  • SkyFeed: Add S5 Identity System and connect it with S5 storage services for storing backups of personal user data (in case a PDS goes offline)
37

Hello @redsolver,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

38

Progress Report (March)

What progress was made on your grant this month?

  • The S5 file system (FS5) is a mutable data structure built on top of S5 primitives. This month, I wrote a new implementation of it for lib5. It has improved support for public directory trees, comes with a new, more consistent URI format, integrates nicely with accounts and other S5 APIs and most importantly, does not contain any legacy code from the Skynet days anymore!
  • I also added a completely new ā€œSnapshotā€ feature to FS5. With Snapshots, users and developers can create short immutable CIDs for deep directory trees (both public and private) pretty much instantly. They are great for keeping a personal archived copy of a public directory, or just private backups
  • Thereā€™s now a second S5 Node implementation written in Golang using renterd! It will also power an independent third-party node (pinner.xyz), which Iā€™m really excited about. I spent some time testing it with @pcfreak30 and fixed some bugs found in S5 apps during that
  • The Admin Web App for managing accounts on S5 Nodes using the Admin API is now published on GitHub
  • I spent a week following the DMA (Digital Markets Act) workshops organized by the EU Commission. Different Gatekeepers (Apple, Meta, Google, Amazon, Microsoft, ByteDance) explained how they intend to comply with the DMA. The workshops gave some interesting insights on new features like data portability, which could for example enable Instagram or Google users to set up automatic backups of all their personal data directly to a S5/Sia-powered storage service. The DMA also comes with interoperability requirements, so for example Vup Chat users could directly message WhatsApp users in the future! There were also a lot more fun topics covered, so if youā€™re interested in chatting about them, feel free to ping me somewhere
  • Fixed storage locations not showing up on the CID Utils web app (cid.one)
  • Small Improvements to the Sia Setup Guide in the S5 Docs

Links to repos worked on this month:

What will you be working on this month?

  • Add FS5 and Account features to the S5 Demo App (so users and devs can try them)
  • Add a simple real-time collaborative text editor to the demo app
  • Add a HTTP redirect API endpoint for S5 file systems to the S5 Node, this will for example make it possible to use it as a Linux Distro package mirror
  • With the new FS, archives of data can now be better organized. So I will extend the existing S5 archiving tools for YouTube (and Twitch) videos to store data in a proper directory structure which should help a lot with discovery and working together on giant archives
  • Instead of developing a completely new App Store powered by S5 (see initial grant proposal), I will try to add content-addressed storage features to the very good open-source Obtainium App Store (GitHub - ImranR98/Obtainium: Get Android App Updates Directly From the Source.) Itā€™s written in Flutter/Dart, so all S5 libraries should be easy to integrate!
  • I wanted to write specifications for the S5 Identity system for a while, but due to the core data structures still changing fast, I didnā€™t do it yet. Now that the FS is in a more or less stable state, I feel comfortable writing a proper spec