From f8173aec6182145a74653afeb960e16d1f18ea7c Mon Sep 17 00:00:00 2001 From: Patrik Fodor Date: Tue, 14 Oct 2025 21:08:49 +0200 Subject: [PATCH 1/8] add initial goals documentation Signed-off-by: Patrik Fodor --- goals.md | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 goals.md diff --git a/goals.md b/goals.md new file mode 100644 index 0000000..360e34d --- /dev/null +++ b/goals.md @@ -0,0 +1,62 @@ +## Core Features + +- **Media Playback:** Support on-demand playback for media files in various formats. + - **Caching & Performance:** + - Use cached segments for seamless seeking and fast resume. + - If another user starts playback with the same parameters, reuse existing cached data to minimize transcoding and bandwidth usage. + - Includes CDN capabilities and P2P federation by Blint. +- **Metadata Management:** + - Automatically identify and fetch metadata from databases like IMDb, TVDB, etc. + - Allow users to edit and correct metadata manually. +- **Transcoding Options:** + - Pre-encode media into multiple formats for reduced compute load (at the expense of storage). + - Or use dynamic, on-the-fly transcoding for flexibility and storage efficiency. +- **Clients** + - Web Client + - Native Clients: Desktop (Windows, Linux, MacOS), Mobile (Android, iOS), TV (Android TV, Apple TV, WebOS, Roku, Someone could create one for Xbox) + - Multiuser Support in clients: You are able to login with multiple users and change between them. +- **Offline Access:** + - Clients can batch-download content for offline playback, respecting user permissions and device limits. +- **Scalability:** + - Support distributed and remote ffmpeg-based transcoding pipelines for high availability and load balancing. +- **User Management:** + - Authentication, profiles, access control, and session tracking. +- **Library & Personalization:** + - Organize and manage libraries by media type (movies, shows, etc.). + - Create and manage playlists and collections. + - Track user history, progress, and play counts. + - Allow users to group media into watchlists, favorites, or custom categories. + - Provide statistics and usage insights (e.g., most watched, time spent, active users). + - Show statics for users about themselves + - Resume from last position across devices. + +## Nice to Have / Later + +- **Sync Play:** Watch movies or shows together in real time across multiple clients. +- **Integration with ARR Stack:** Compatibility with Radarr, Sonarr, Lidarr, and similar automation tools. +- **Parental controls** +- **Jellyfin-Compatible API** and **Plex-Compatible API** +- **Plugin Support:** Extend functionality with custom or third-party plugins. +- **Smart Playback Features:** + - Intro, outro, recap skipper. +- **Recommendations:** Personalized movie and series suggestions based on user history and preferences. +- **External Sync:** Sync watch history, ratings, and metadata with services like Trakt.tv. +- **Subtitle Management:** Automatic subtitle fetching, editing, and syncing with playback. - Probably bazarr is enough for that. +- **Multi-Format Detection:** Detect multiple versions (e.g., 1080p, 4K, HDR) of the same media and show them in a unified view. +- **IPTV / Live TV:** Stream and manage live channels alongside on-demand content. + +## Architecture + +- **Frontend/Backend Separation:** Clearly separated user interface and backend services. +- **Microservice-Ready:** Modular, loosely coupled components that can scale independently. +- **Extensible Protocol Support:** Easily add or replace streaming and media delivery protocols. +- **Telemetry & Observability:** Built-in tracing, logging, and metrics collection. +- **Plugin System:** Enable community and custom extensions. +- **Robust API Design:** Clean, well-documented REST and gRPC interfaces for interoperability. + +## Patrik's Dreams (Not Necessarily Realistic Goals) + +- On-demand streaming over QUIC. *(WebRTC was suggested but may not fit our use case.)* +- **Segmented Formats:** Support CMAF alongside traditional MPEG-TS for more efficient adaptive streaming. +- **Streaming Standards:** Full compatibility with both HLS and MPEG-DASH. +- **Processing Backends:** Flexible support for ffmpeg, GStreamer, or alternative media frameworks. -- 2.50.1 From c6a972fba3b4e6a4fd172471bdf0a30b82a18530 Mon Sep 17 00:00:00 2001 From: Patrik Fodor Date: Tue, 14 Oct 2025 22:09:52 +0200 Subject: [PATCH 2/8] expand goals documentation Signed-off-by: Patrik Fodor --- goals.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/goals.md b/goals.md index 360e34d..c3a60bb 100644 --- a/goals.md +++ b/goals.md @@ -4,7 +4,8 @@ - **Caching & Performance:** - Use cached segments for seamless seeking and fast resume. - If another user starts playback with the same parameters, reuse existing cached data to minimize transcoding and bandwidth usage. - - Includes CDN capabilities and P2P federation by Blint. +- **CDN capabilities and P2P federation** + - TODO: Define this feature in detail. - **Metadata Management:** - Automatically identify and fetch metadata from databases like IMDb, TVDB, etc. - Allow users to edit and correct metadata manually. @@ -16,7 +17,7 @@ - Native Clients: Desktop (Windows, Linux, MacOS), Mobile (Android, iOS), TV (Android TV, Apple TV, WebOS, Roku, Someone could create one for Xbox) - Multiuser Support in clients: You are able to login with multiple users and change between them. - **Offline Access:** - - Clients can batch-download content for offline playback, respecting user permissions and device limits. + - Clients can batch-download content for offline playback. - **Scalability:** - Support distributed and remote ffmpeg-based transcoding pipelines for high availability and load balancing. - **User Management:** @@ -49,6 +50,7 @@ - **Frontend/Backend Separation:** Clearly separated user interface and backend services. - **Microservice-Ready:** Modular, loosely coupled components that can scale independently. +- **Resiliency**: Temporary outages in one service should not disrupt the viewing experience. - **Extensible Protocol Support:** Easily add or replace streaming and media delivery protocols. - **Telemetry & Observability:** Built-in tracing, logging, and metrics collection. - **Plugin System:** Enable community and custom extensions. -- 2.50.1 From 41fb1e5b55201b566def69e57c5b82cc89b5d6a1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?B=C3=A1lint=20R=C3=A9thelyi=20=28Blint=29?= Date: Thu, 16 Oct 2025 09:15:21 +0200 Subject: [PATCH 3/8] docs: define CDN capabilities and P2P federation in goals.md Added a detailed specification for CDN functionality, including: - Lightweight CDN node concept (simple executable for users) - Secure connectivity using pre-shared keys - Dynamic registration with main metadata server and DNS assignment - Content awareness, replication, and streaming fallback - Peer-to-peer (P2P) sharing for distributed content transfer - Federation and collaboration between CDNs - Future idea: predictive caching based on user activity This formalizes the CDN architecture for Walros, supporting distributed caching, federation, and smarter prefetching mechanisms." --- goals.md | 40 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2 deletions(-) diff --git a/goals.md b/goals.md index c3a60bb..a7703e0 100644 --- a/goals.md +++ b/goals.md @@ -4,8 +4,44 @@ - **Caching & Performance:** - Use cached segments for seamless seeking and fast resume. - If another user starts playback with the same parameters, reuse existing cached data to minimize transcoding and bandwidth usage. -- **CDN capabilities and P2P federation** - - TODO: Define this feature in detail. +- **CDN Capabilities and P2P Federation** + - **Lightweight CDN Node:** + - Each CDN node runs as a standalone executable binary that can be easily deployed by both technical and non-technical users. + - Minimal configuration required: + - **Data Directory:** Defines where cached media files are stored. + - **Optional Name:** Human-readable CDN identifier (used for UI labeling and dynamic DNS registration). + - **Mode:** Can operate as a **shared CDN** (available to multiple users) or a **local-only CDN**. + + - **Secure Connectivity:** + - Each CDN connects to the **main metadata server** using a **pre-shared key** for authentication and trust establishment. + - This ensures only authorized CDNs can register and exchange data with the main network. + + - **Dynamic Registration & Discovery:** + - Upon startup, the CDN connects to the **main metadata server** and automatically registers itself. + - The main server assigns a **dynamic DNS name** to each CDN for seamless connectivity. + - CDNs periodically update their metadata to report available content, storage capacity, and health status. + + - **Content Awareness & Distribution:** + - The main server maintains a global index of all available CDNs and the content cached on each. + - Users can choose which CDN should serve their content in the client UI. + - If a requested content is missing from the selected CDN, users have two options: + 1. **Request content replication:** Copy or prefetch the content from another CDN or the main server. + 2. **Stream fallback:** Stream the content directly from the main server or the nearest available CDN. + + - **Peer-to-Peer (P2P) Sharing:** + - CDNs and the main server interconnect via **P2P protocols** to simplify file transfer and reduce reliance on centralized infrastructure. + - This enables faster, bandwidth-efficient distribution of popular content between geographically distributed CDNs. + + - **Federation & Collaboration:** + - Multiple CDNs can share content and metadata within a trusted federation network. + - The system can automatically balance or replicate popular content across CDNs to optimize bandwidth usage and ensure availability. + + - **Future Idea - Predictive Caching:** + - CDNs may automatically pre-cache upcoming content based on user activity patterns. + - If a user watches the first movie in a series, the CDN pre-fetches sequels (e.g., parts 2, 3, etc.). + - If a user starts a new TV show, the CDN preloads subsequent episodes or seasons. + - This predictive caching minimizes latency and improves the user experience by ensuring likely-next content is available instantly. + - **Metadata Management:** - Automatically identify and fetch metadata from databases like IMDb, TVDB, etc. - Allow users to edit and correct metadata manually. -- 2.50.1 From 079a1fb3ee13d193e442a4e5a32774cc57dda1a4 Mon Sep 17 00:00:00 2001 From: Patrik Fodor Date: Thu, 16 Oct 2025 09:54:27 +0200 Subject: [PATCH 4/8] docs: improve docs in goals.md - Change "series" to "TV show" to make it clearer - Move CDN future goal to the Later section Signed-off-by: Patrik Fodor --- goals.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/goals.md b/goals.md index a7703e0..4404ac8 100644 --- a/goals.md +++ b/goals.md @@ -36,12 +36,6 @@ - Multiple CDNs can share content and metadata within a trusted federation network. - The system can automatically balance or replicate popular content across CDNs to optimize bandwidth usage and ensure availability. - - **Future Idea - Predictive Caching:** - - CDNs may automatically pre-cache upcoming content based on user activity patterns. - - If a user watches the first movie in a series, the CDN pre-fetches sequels (e.g., parts 2, 3, etc.). - - If a user starts a new TV show, the CDN preloads subsequent episodes or seasons. - - This predictive caching minimizes latency and improves the user experience by ensuring likely-next content is available instantly. - - **Metadata Management:** - Automatically identify and fetch metadata from databases like IMDb, TVDB, etc. - Allow users to edit and correct metadata manually. @@ -59,7 +53,7 @@ - **User Management:** - Authentication, profiles, access control, and session tracking. - **Library & Personalization:** - - Organize and manage libraries by media type (movies, shows, etc.). + - Organize and manage libraries by media type (movies, TV shows, etc.). - Create and manage playlists and collections. - Track user history, progress, and play counts. - Allow users to group media into watchlists, favorites, or custom categories. @@ -69,18 +63,23 @@ ## Nice to Have / Later -- **Sync Play:** Watch movies or shows together in real time across multiple clients. +- **Sync Play:** Watch movies or TV shows together in real time across multiple clients. - **Integration with ARR Stack:** Compatibility with Radarr, Sonarr, Lidarr, and similar automation tools. - **Parental controls** - **Jellyfin-Compatible API** and **Plex-Compatible API** - **Plugin Support:** Extend functionality with custom or third-party plugins. - **Smart Playback Features:** - Intro, outro, recap skipper. -- **Recommendations:** Personalized movie and series suggestions based on user history and preferences. +- **Recommendations:** Personalized movie and TV show suggestions based on user history and preferences. - **External Sync:** Sync watch history, ratings, and metadata with services like Trakt.tv. - **Subtitle Management:** Automatic subtitle fetching, editing, and syncing with playback. - Probably bazarr is enough for that. - **Multi-Format Detection:** Detect multiple versions (e.g., 1080p, 4K, HDR) of the same media and show them in a unified view. - **IPTV / Live TV:** Stream and manage live channels alongside on-demand content. +- **CDN - Predictive Caching:** + - CDNs may automatically pre-cache upcoming content based on user activity patterns. + - If a user watches the first movie in a movie series, the CDN pre-fetches sequels (e.g., parts 2, 3, etc.). + - If a user starts a new TV show, the CDN preloads subsequent episodes or seasons. + - This predictive caching minimizes latency and improves the user experience by ensuring likely-next content is available instantly. ## Architecture -- 2.50.1 From beecae7c09814d6b191a1a8b4ad2a4d010d24be0 Mon Sep 17 00:00:00 2001 From: Patrik Fodor Date: Thu, 16 Oct 2025 10:00:50 +0200 Subject: [PATCH 5/8] docs: remove extra white spaces in goals.md Signed-off-by: Patrik Fodor --- goals.md | 46 +++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/goals.md b/goals.md index 4404ac8..3bdbe43 100644 --- a/goals.md +++ b/goals.md @@ -5,36 +5,36 @@ - Use cached segments for seamless seeking and fast resume. - If another user starts playback with the same parameters, reuse existing cached data to minimize transcoding and bandwidth usage. - **CDN Capabilities and P2P Federation** - - **Lightweight CDN Node:** - - Each CDN node runs as a standalone executable binary that can be easily deployed by both technical and non-technical users. + - **Lightweight CDN Node:** + - Each CDN node runs as a standalone executable binary that can be easily deployed by both technical and non-technical users. - Minimal configuration required: - - **Data Directory:** Defines where cached media files are stored. - - **Optional Name:** Human-readable CDN identifier (used for UI labeling and dynamic DNS registration). - - **Mode:** Can operate as a **shared CDN** (available to multiple users) or a **local-only CDN**. + - **Data Directory:** Defines where cached media files are stored. + - **Optional Name:** Human-readable CDN identifier (used for UI labeling and dynamic DNS registration). + - **Mode:** Can operate as a **shared CDN** (available to multiple users) or a **local-only CDN**. - - **Secure Connectivity:** - - Each CDN connects to the **main metadata server** using a **pre-shared key** for authentication and trust establishment. - - This ensures only authorized CDNs can register and exchange data with the main network. + - **Secure Connectivity:** + - Each CDN connects to the **main metadata server** using a **pre-shared key** for authentication and trust establishment. + - This ensures only authorized CDNs can register and exchange data with the main network. - - **Dynamic Registration & Discovery:** - - Upon startup, the CDN connects to the **main metadata server** and automatically registers itself. - - The main server assigns a **dynamic DNS name** to each CDN for seamless connectivity. - - CDNs periodically update their metadata to report available content, storage capacity, and health status. + - **Dynamic Registration & Discovery:** + - Upon startup, the CDN connects to the **main metadata server** and automatically registers itself. + - The main server assigns a **dynamic DNS name** to each CDN for seamless connectivity. + - CDNs periodically update their metadata to report available content, storage capacity, and health status. - - **Content Awareness & Distribution:** - - The main server maintains a global index of all available CDNs and the content cached on each. - - Users can choose which CDN should serve their content in the client UI. + - **Content Awareness & Distribution:** + - The main server maintains a global index of all available CDNs and the content cached on each. + - Users can choose which CDN should serve their content in the client UI. - If a requested content is missing from the selected CDN, users have two options: - 1. **Request content replication:** Copy or prefetch the content from another CDN or the main server. - 2. **Stream fallback:** Stream the content directly from the main server or the nearest available CDN. + 1. **Request content replication:** Copy or prefetch the content from another CDN or the main server. + 2. **Stream fallback:** Stream the content directly from the main server or the nearest available CDN. - - **Peer-to-Peer (P2P) Sharing:** - - CDNs and the main server interconnect via **P2P protocols** to simplify file transfer and reduce reliance on centralized infrastructure. - - This enables faster, bandwidth-efficient distribution of popular content between geographically distributed CDNs. + - **Peer-to-Peer (P2P) Sharing:** + - CDNs and the main server interconnect via **P2P protocols** to simplify file transfer and reduce reliance on centralized infrastructure. + - This enables faster, bandwidth-efficient distribution of popular content between geographically distributed CDNs. - - **Federation & Collaboration:** - - Multiple CDNs can share content and metadata within a trusted federation network. - - The system can automatically balance or replicate popular content across CDNs to optimize bandwidth usage and ensure availability. + - **Federation & Collaboration:** + - Multiple CDNs can share content and metadata within a trusted federation network. + - The system can automatically balance or replicate popular content across CDNs to optimize bandwidth usage and ensure availability. - **Metadata Management:** - Automatically identify and fetch metadata from databases like IMDb, TVDB, etc. -- 2.50.1 From 406b052de2daef86ea002c5b677f10fbb0b73129 Mon Sep 17 00:00:00 2001 From: marcsello Date: Fri, 17 Oct 2025 12:03:57 +0200 Subject: [PATCH 6/8] Add Marcsello's dreams --- goals.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/goals.md b/goals.md index 3bdbe43..c8a3f60 100644 --- a/goals.md +++ b/goals.md @@ -97,3 +97,10 @@ - **Segmented Formats:** Support CMAF alongside traditional MPEG-TS for more efficient adaptive streaming. - **Streaming Standards:** Full compatibility with both HLS and MPEG-DASH. - **Processing Backends:** Flexible support for ffmpeg, GStreamer, or alternative media frameworks. + +## Marcsello's Dreams (Even Less Realistic Goals) + +- **Segment based media distribution across CDNs:** Instead of replicating full media files everywhere, allow replicating only parts of the media files, for even more granular disk usage and better load balancing. +- **CDN appliance:** A pre-built hardware (contains some disks and hardware transcoder) which can be easily deployed and registered by anyone. + - **Custom built Linux based OS:** Buildroot, Nix, or something similar that allows building a compact OS image that can be installed and updated on the target hardware. Built for hosting the CDN software described above. + - **Fail-safe unattended software update for the CDN appliance:** If a software update goes wrong, the appliance should be able to recover without any human intervention. -- 2.50.1 From 8f6850877f948f66672df78375da92389c3d3478 Mon Sep 17 00:00:00 2001 From: marcsello Date: Sun, 19 Oct 2025 22:36:03 +0200 Subject: [PATCH 7/8] More Marcsello dreams --- goals.md | 1 + 1 file changed, 1 insertion(+) diff --git a/goals.md b/goals.md index c8a3f60..318c2ca 100644 --- a/goals.md +++ b/goals.md @@ -104,3 +104,4 @@ - **CDN appliance:** A pre-built hardware (contains some disks and hardware transcoder) which can be easily deployed and registered by anyone. - **Custom built Linux based OS:** Buildroot, Nix, or something similar that allows building a compact OS image that can be installed and updated on the target hardware. Built for hosting the CDN software described above. - **Fail-safe unattended software update for the CDN appliance:** If a software update goes wrong, the appliance should be able to recover without any human intervention. +- **Dynamic encoding profile change based on load:** On live transcode, change the encoding profile from slower to faster ones as demand increases to serve more users more smoothly, when there are less users use slower profiles for better quality. \ No newline at end of file -- 2.50.1 From 06c73db0570ae14c43e66eb2908db7d32facdf9f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?B=C3=A1lint=20Blint=20R=C3=A9thelyi?= Date: Thu, 27 Nov 2025 16:16:58 +0100 Subject: [PATCH 8/8] update goals, answer, fine-tune CDN section --- goals.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/goals.md b/goals.md index 318c2ca..5aa9681 100644 --- a/goals.md +++ b/goals.md @@ -11,10 +11,10 @@ - **Data Directory:** Defines where cached media files are stored. - **Optional Name:** Human-readable CDN identifier (used for UI labeling and dynamic DNS registration). - **Mode:** Can operate as a **shared CDN** (available to multiple users) or a **local-only CDN**. + - What content does a CDN node deliver: + - media (for starter) - - **Secure Connectivity:** - - Each CDN connects to the **main metadata server** using a **pre-shared key** for authentication and trust establishment. - - This ensures only authorized CDNs can register and exchange data with the main network. + - **Secure Connectivity** - **Dynamic Registration & Discovery:** - Upon startup, the CDN connects to the **main metadata server** and automatically registers itself. @@ -28,12 +28,10 @@ 1. **Request content replication:** Copy or prefetch the content from another CDN or the main server. 2. **Stream fallback:** Stream the content directly from the main server or the nearest available CDN. - - **Peer-to-Peer (P2P) Sharing:** - - CDNs and the main server interconnect via **P2P protocols** to simplify file transfer and reduce reliance on centralized infrastructure. - - This enables faster, bandwidth-efficient distribution of popular content between geographically distributed CDNs. + - **Peer-to-Peer (P2P) Sharing** - **Federation & Collaboration:** - - Multiple CDNs can share content and metadata within a trusted federation network. + - Multiple CDNs can share content and metadata within a connection between the instances via a secure tunnel. - The system can automatically balance or replicate popular content across CDNs to optimize bandwidth usage and ensure availability. - **Metadata Management:** -- 2.50.1