[Feature] Support equals()&hashcode() in FlinkTableSinkBase

[xiangyuf]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

When using partial-update merge-engine, we want to reuse sink node across multiple datastreams. To achieve this, we need to override equals() for Paimon DynamicTableSink.

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[JingsongLi]

The idea is good, we plan to do this but always no time... Feel free to open PR

[xintongsong]

@xiangyuf
Reusing sink node for partial-update is a good idea. We have an internally solution for this. Contributing it to the community requires opening a FLIP in the Flink community, which as Jingsong said we just couldn't find time for. If you're interested, we can share that solution with you and see if we can cooperate on that.

As far as I can see, one thing different from your plan is that, our solution does not rely on equals() && hashcode() of FlinkTableSinkBase, but would use some public fields of the sink nodes to identify those that can be reused. The benefit is that, this feature can be enabled for any connector, without relying on the connector codes to implement equals() && hashcode().

[xiangyuf]

@xiangyuf Reusing sink node for partial-update is a good idea. We have an internally solution for this. Contributing it to the community requires opening a FLIP in the Flink community, which as Jingsong said we just couldn't find time for. If you're interested, we can share that solution with you and see if we can cooperate on that.

As far as I can see, one thing different from your plan is that, our solution does not rely on equals() && hashcode() of FlinkTableSinkBase, but would use some public fields of the sink nodes to identify those that can be reused. The benefit is that, this feature can be enabled for any connector, without relying on the connector codes to implement equals() && hashcode().

@xintongsong Thx for ur reply. I'd like to hear your internal solution.