Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Plex Cluster: Beta Testers Needed #21

Open
Fmstrat opened this issue Feb 4, 2020 · 69 comments
Open

Plex Cluster: Beta Testers Needed #21

Fmstrat opened this issue Feb 4, 2020 · 69 comments

Comments

@Fmstrat
Copy link
Owner

Fmstrat commented Feb 4, 2020

Hi everyone,

You've been mentioned because at some point you have posted an issue here or participated via PR. I am migrating away from plex-db-sync to a new "instant sync" software platform I've developed called Plex Cluster.

I am looking for beta testers for the below platform. Reply here if you are interested!

Features

  • Syncs watched status on-demand, right away
  • Uses standard Plex APIs without accessing the database
  • Can work across multiple Plex servers
  • Scheduled "full update" can completely sync user's watched status for any user that has been active on each Plex server since running Plex Cluster
  • Syncs media that is contained on both (or multiple) servers without erroring when media does not exist
  • Works for Plex.tv or Managed users

Architecture

Plex Cluster contains two applications:

  • Plex Cluster Proxy, which is installed alongside every Plex server and acts as a proxy (using nginx) between Plex Media Server and the internet. This application's job is to pass any requests that come to it along to the Plex server while catching any requests that mark watched status and also forwarding them on the Plex Cluster Manager. There is one instance of Plex Cluster Proxy for each Plex server.
  • Plex Cluster Manager is then used to synchronize the status between Plex servers. There is only one instance of Plex Cluster Manager which is used by all instances of Plex Cluster Proxy.

An example use case:

  1. Plex client requests to watch a show with GIUD (unique identifier) ABC from the TV Shows library on Plex Server 1
  2. The request goes through Plex Cluster Proxy 1 and is passed along to Plex Server 1
  3. The user stops the show midway.
  4. The request goes through Plex Cluster Proxy 1 and is passed along to Plex Server 1, but is also passed along to Plex Cluster Manager
  5. Plex Cluster Manager takes the request and checks for any other Plex server with a library named TV Shows and GUID ABC
  6. For any other servers it finds, such as Plex Server 2, it forwards the request on to them and the show is instantly marked as watched up to that midway point.
  7. User switches over to Plex Server 2 and sees that they are midway through show ABC

@ChaosZero112 @daxxi13 @Malcolmest @seertenedos @skinnyzaz @sbcrumb @goofballtech @onezero1010101 @ddurdle @dpeet
@Fmstrat
Copy link
Owner Author

Fmstrat commented Feb 4, 2020

I should note, while this solution can be run "raw,", it is designed to be a true cluster setup. This means supporting more than 2 servers, but also means it focuses on containerization (Docker or K8 if you really wanted to).

If you desire to beta test, but do not wish to run a container platform such as Docker, I would still be interested in your participation and assistance in updating the setup README.

Thank you.

@seertenedos
Copy link

I am interested in testing it and i am actually in the process of trying to sync 2 servers as the minute all running in docker

@sbcrumb
Copy link

sbcrumb commented Feb 4, 2020

Count me in.

@lusky3
Copy link

lusky3 commented Feb 4, 2020

I could assist. I have a 3-instance container setup ready to go.

@RXWatcher
Copy link

I have multiple servers on different continents and would love this. Please count me in. -RX

@goofballtech
Copy link

goofballtech commented Feb 4, 2020 via email

@seertenedos
Copy link

If something is watched on both servers will it update both servers to the oldest watch date? That is one issue I currently have that has caused old watched items on one server to show as newly watched on my second server when it synced.

@fredtbrown
Copy link

I would love to test however I am running on windows and no containers.

@zerodayz1
Copy link

Running Plex on docker on two servers. This is lit, as the kids say. I'm in.

@ddurdle
Copy link

ddurdle commented Feb 4, 2020 via email

@Fmstrat
Copy link
Owner Author

Fmstrat commented Feb 4, 2020

@seertenedos @sbcrumb @ChaosZero112 @RXWatcher @goofballtech @zerodayz1 @ddurdle At some point in the near future you should expect to get an invite to participate.

@fredtbrown As stated you could run this natively, but you would either need to do things like install bash and nginx. If you are interested, I would recommend installing Docker for Windows, and running the Plex Cluster services in containers, and Plex as a system application (like you do now). Let me know if you would like to try, but during the beta I'll really be looking for issues, and won't have a lot of time for support.

@seertenedos As for "oldest watched" that was part of the reason I created plex-db-sync. The "instant" sync feature of Plex Cluster should remove that issue, as the marked times will be seconds apart unless something goes down. The "full sync" feature is really just a catch all, or a way to sync up users at the very beginning. My suggestion would actually be to sync everything with plex-db-sync, the switch to Plex Cluster. That way the times are all lined up.

@dpeet
Copy link

dpeet commented Feb 4, 2020

@Fmstrat I'm in as well!

@seertenedos
Copy link

@Fmstrat you have content on both servers covered when you watch them and http call come in which is the main one but there is 2 issues missed I think.

  1. I watched something on server 1 but server 2 does not have that show at the time. Later on server 2 gets the show or episode. (Being able to trigger full update would be fine but ideally being able to set older watch dates on it would be ideal.) I am getting this as I slowly back fill new servers. (I also have a third server I have offline and just want to keep the watch status and sync them in as other servers add the shows if they add them)

  2. Will you capture watch status change via sync and other non direct stream over http protocols?

  3. More an very cool to have would be for the central server to have the watch status from all servers stored so if a server does or gets corrupted that is the only one with a show that watch status is not lost. I have lost watched status from Plex a few times over the years and even sometimes have some of my older watch status disappear for some unknown reason.

Your idea of interception is fantastic and using the API to set watched status is also smart. I assume for full sync you may read the database file to speed it up but write will likely be via API. I am guessing you can't set the date you watched something via API but maybe that is a special option to sync watch dates to oldest date but actually updates to database files and restarts Plex. It would not write new watch status but instead just update the ones in the database.

If you can't do all that it is fine just some ideas as the interception was a smart idea.

@daxxi13
Copy link
Contributor

daxxi13 commented Feb 5, 2020

@Fmstrat I'd like to help too

@onezero1010101
Copy link

@Fmstrat I'm interesting in testing as well.

@hergonoway
Copy link

@Fmstrat : count me in too. Plex db sync is working great between my 2 servers, but a non-async solution is definitly what I seek for a long time :)

quick question : does the Plex Cluster Manager cache requests for a certain amount of time in case of non availibility of one or multiple node ?

@simssj
Copy link

simssj commented Feb 14, 2020 •
edited
Loading

Sign me up. I have a cluster of three plex instances, all remote from one another. One is on MacOS and the other two are on Raspberry Pis. All are running "bare metal" (without any containerization) but Docker is installed and running on each node for other housekeeping tasks.
Our family uses these nodes as a distributed media center and family movie library; keeping tabs on what's been watched, where, is a real challenge.
I came here to PR a modification into the current plex-db-sync script (to add a small delay while stopping the Plex servers, to allow for some settle time) but I'm prefer to get on the new Plex Cluster bandwagon.

@edrock200
Copy link

edrock200 commented Feb 27, 2020 •
edited
Loading

I'm interested! I have one docker instance and 5 standalones. One docker on Ubuntu, 3 centos 7 minimals and 2 centos 8 minimals.

@skinnyzaz
Copy link

I am interested in testing

@onezero1010101
Copy link

@Fmstrat Have you closed beta testing? Or are you going to release a repo that anyone willing can test? Would be nice to let everyone know if you are going to be accepting any new testers or if we just need to wait until its made public. I know I'm holding off on setting up a new cluster for this, rather then move my existing sync over. Thanks

@Deadly-Bytes
Copy link

I would like to test if possible also

@TheImpossible21
Copy link

Is it still possible to join the beta testing?

@seertenedos
Copy link

seertenedos commented Apr 1, 2020 via email

@sbcrumb
Copy link

sbcrumb commented Apr 3, 2020

@seertenedos @sbcrumb @ChaosZero112 @RXWatcher @goofballtech @zerodayz1 @ddurdle At some point in the near future you should expect to get an invite to participate.

@fredtbrown As stated you could run this natively, but you would either need to do things like install bash and nginx. If you are interested, I would recommend installing Docker for Windows, and running the Plex Cluster services in containers, and Plex as a system application (like you do now). Let me know if you would like to try, but during the beta I'll really be looking for issues, and won't have a lot of time for support.

@seertenedos As for "oldest watched" that was part of the reason I created plex-db-sync. The "instant" sync feature of Plex Cluster should remove that issue, as the marked times will be seconds apart unless something goes down. The "full sync" feature is really just a catch all, or a way to sync up users at the very beginning. My suggestion would actually be to sync everything with plex-db-sync, the switch to Plex Cluster. That way the times are all lined up.

Any status on this? Are we still getting an invite? or has this been put on the back burner ?

@Fmstrat
Copy link
Owner Author

Fmstrat commented Apr 3, 2020 •
edited
Loading

@sbcrumb Yes, an invite will come. I've been working through a launch of a government contract I work on in the midst of my company having some changes now, so timing has been tight. I haven't been able to work on the final two issues yet (Roku session variables and reconnection attempts).

@ninthwalker
Copy link

@Fmstrat Add me to your list to test when you get around to finishing things up.
Very familiar with containers and clustering in general. Thanks.

@notlimahrelyt
Copy link

Very interested. I have 4 servers running docker.

@rjblake
Copy link

rjblake commented Apr 14, 2020

I'd like to participate - have 3x Plex Server (1x Synology, 1x Linux, 1x Mac OS)

@geek-baba
Copy link

I just requested similar feature, I would love to test this, I have bare metal primary plex, kube-plex on k8s cluster and 3rd one one as plain docker on OMV NAS.

@pmow
Copy link

pmow commented Apr 17, 2020

Interested. Local/Cloud dockers with local/cloud storage. What a fantastic idea.

@AllofTheThangz
Copy link

Due to the complexity of my media - I have 6 servers all pointing to the same media. I would love to cluster these and call it a day. I have searched high and low, and this is the most promising option. Please either expose your current work and allow us to assist in making it a reality, or tell us it's off the table so we don't sit in limbo. I would love to beta and contribute towards this.

@bdschuster
Copy link

Hi! I would love an invite! I have 3 servers running docker pointing all to the same media. Thanks in advance.

@michael-robbins
Copy link

Happy to assist, I have 4 plex servers all running the same library setup/layout. This would be better than 'chaining' plex-db-sync across them all!

@tempaone
Copy link

Love to join too, i am building 5 servers, 4 for 4k and one central for everything else

@pyrometheous
Copy link

I'm interested in this as well. I'm currently trying to provision a few pgblitz servers. One main one that's a more powerful rig, and a couple Intel NUCs for local streaming at my house and my apartment. I've noticed that remote streaming is unreliable, so instead of hauling a server to each place, I'd just have the NUC plugged into the router in each location.

I'm currently playing around with Syncthing to sync metadata across two servers, and so far I'm seeing some success. I haven't tried with users or playback progress yet. But this project looks like exactly what I'm wanting to do.

I'm just doing 1080p content right now, but I'll probably be switching to 4k within the next couple of years.

@n0rbertg
Copy link

Hi,
I'd like an invite as well if possible. Just in the process of syncing two servers, really could use this feature.

@freetheword
Copy link

freetheword commented Jun 1, 2020 •
edited
Loading

I'm interested in testing as well.

@MishkaFur
Copy link

Happy to test, 2 Windows based Plex servers but looking to shift them over to linux/docker etc

@geek-baba
Copy link

Not sure the status of this but anyone able to use the current plex-db-sync to sync a bare metal install and a backup docker install? Asking here because dint see any response in the original post... #24

@tempaone
Copy link

tempaone commented Jul 4, 2020

Is this project still happening ?
I've been waiting for this project to expand away from just one server.

@tempaone
Copy link

tempaone commented Jul 4, 2020

@daxxi13 would you happen to know what is happening with this project?
mentioned you as your are a contributor

@stroodle96
Copy link

Very interested. I have 3 docker servers all with identical configuration and media libraries.

@chrisjohns03
Copy link

@Fmstrat I'm interested in testing this. I currently have multiple plex servers I'm trying to keep in sync

@Falconx1000
Copy link

@Fmstrat I'm interested in this as well. Currently have 2 plex servers running in kubernetes

@sharkbit
Copy link

sharkbit commented Sep 5, 2020

Is this still a thing? is Plex Cluster going to come out of the shadows? I am currently running windows Plex servers with Trax to sync but since plugin support is no longer supported looking for a better option. I would convert my servers to Linux if this really does exist.

@derekcentrico
Copy link

Happy to test. Haven't been big on containers up to this point. But, can work to transition to containers if necessary.

@AllofTheThangz
Copy link

AllofTheThangz commented Sep 8, 2020 via email

@sbcrumb
Copy link

sbcrumb commented Sep 8, 2020 via email

@PharQ
Copy link

PharQ commented Sep 14, 2020

I have a server on a Synology Nas and on windows that I have people access inside and out of my house. I also have access to friends' servers that I would like to have this work for.

@bishopdynamics
Copy link

I have several PMS instances running on debian, I would love to test and potentially contribute

@jbishop129
Copy link

+1 for this. With Plex still running on SQLite db, it makes any type of db-native syncing very challenging. I have two Plex servers in different locations, and the simple ask of, having my watched/unwatched status for TV shows is really only viable with Trakt.tv service. Using a third-party cloud service to share this data with really urks my self-hosted mindset. @Fmstrat , I very much look forward to your progress on this project.

@Jomack16
Copy link

I would also be interested in testing this, I run 2 plex servers. One in a container on Unraid and one installed on a bare metal box.

@AllofTheThangz
Copy link

AllofTheThangz commented Dec 12, 2020 via email

@BEisem
Copy link

BEisem commented Apr 2, 2021

Is this no longer happening? I would be very much interested, but it seems like maybe the project is dead.

@Fmstrat
Copy link
Owner Author

Fmstrat commented Apr 2, 2021

Damned if it didn't fall off my radar. I'll just open the repo later today.

@geek-baba
Copy link

Damned if it didn't fall off my radar. I'll just open the repo later today.

Thanks! What is the roadmap and if any progress was made...

@chazlarson
Copy link

I'd love to test this; I run several servers as well.

@AllofTheThangz
Copy link

AllofTheThangz commented Apr 2, 2021 via email

@Fmstrat
Copy link
Owner Author

Fmstrat commented Apr 2, 2021

And it's up: https://github.com/Fmstrat/plex-cluster

Fair warning to all, I'll watch the issues list, but this will likely need to be community driven. I haven't had a lot of spare time (as if you couldn't tell from how long this has taken).

@AllofTheThangz
Copy link

AllofTheThangz commented Apr 2, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests