Update Key spanning multiple columns

[cody-scott]

Feature Description

In one of our databases our update key is stored across multiple columns; one for date and one for time.

Our current replication process joins these columns prior to replication into a single delta column and uses that as the incremental field to filter new/changed records. How would you go about achieving this with Sling?

One idea was sql replication where the where filter is performing the combining/filtering of the new data. Unclear if this would work for anything but a full refresh, which at that stage might as well just perform the comparison on the primary key.

Another option is to add the column as part of the load processes, but this breaks the column typing as the calls to information_schema won't discover the column as it doesn't exist in the source set. A possible solution would be to allow additional columns to be defined in the replication file (with column type), such as {name: combined_date_column, type: datetime}

[flarco]

In your case, I would use a custom SQL. Combine the date/time columns into one, name it the same as the existing target
datetime column, and use that as the update_key. Sling handles incremental mode with custom SQL just fine. Just make sure to put in the {incremental_where_cond} or {incremental_value} as shown in the docs.

[cody-scott]

So in effect the date only part becomes the combined version. The incremental will check against the sql statement for changes then?