Recommendations for Pekko Connectors Deployment

[Susmit07]

Hi Developers

Whats the ideal way of deploying pekko connectors in production?

Use case we will watch different hdfs and s3 directories and pull the data to push to s3 object store via pekko s3 sink connectors

Is it a Cluster Singleton?

https://pekko.apache.org/docs/pekko/current/typed/cluster-singleton.html

[raboof]

AFAIK we don't have any particular recommendations for deploying Pekko-based applications.

The simplest solution is to deploy as a single node: that way you don't have to worry about clustering and making sure each node in the cluster processes a unique subset of the incoming files.

From #814 (reply in thread) I understand you're running on Kubernetes, that you really want to run on multiple nodes for scalability and parallelism, and that you're prepared to handle the additional complexity that comes with that. We have some documentation on how to deploy a Pekko cluster to K8s at https://pekko.apache.org/docs/pekko-management/current/kubernetes-deployment/index.html . You probably also want to configure the Split Brain Resolver based on Kubernetes Lease documented at https://pekko.apache.org/docs/pekko/current/split-brain-resolver.html#lease.

For the application side you could indeed either use Cluster Singleton of Cluster Sharding.

[Susmit07]

ok

[Susmit07]

@raboof do you see any concerns if we deploy our pekko connectors directly as docker image within k8s cluster that we dont have to deal with the additional complexity of cluster and let k8s handle the failover

[raboof]

That sounds like a good idea to me.