Decoding Issues During Data Transfer from Oracle to SQL Server

[GoranDervisevic]

Hi!

I am having a decoding issue with Sling both in version Version 1.2.15 and 1.2.18. I am transferring data from an Oracle database to an SQL Server database. Here is a sample:

sling run --src-conn ORACLE --src-stream "ORACLE.TABLE" --tgt-conn SQL_SERVER --tgt-object "SQLS.TABLE"

In the source Oracle table I have a column called “description” which contains Swedish symbols such as ä or ö however when transferred to SQL server table the symbols are lost. Here are some word example:

Oracle: särskild
SQL Server: särskild

Oracle: Eftersök
SQL Server: Eftersök

Oracle: Inför
SQL Server: Inför

I have tried all of the decoding transformations after “decode_latin1” and “decode_utf8” did not work, and the Sling command I believe is correct:

sling run --src-conn ORACLE --src-stream "ORACLE.TABLE" --tgt-conn SQL_SERVER --tgt-object "SQLS.TABLE" --transforms "{ description: [ decode_latin1 ] }"

Just for testing reasons I tried all of the decoding and encoding transformations and some of them did transform but not the one I want. Also I did try "replace_accents" but symbols like ä or ö are lost and they just become a or o.

I did also extract the Oracle table as a csv file. If I open it in Excel the symbols are correct. If I import the csv file with the SQL Server Import Wizard it imports the symbols correctly in the “description” column. But I would prefer to use Database to Database extraction.

Has anyone else experienced this issue or found a solution for preserving special characters when transferring data between these databases using Sling?

Any advice or solutions would be appreciated. Thanks!

[flarco]

Hi, are you using BCP to insert into SQL Server?
If so, can you try using --tgt-options '{ use_bulk: false }' to see how it works without BCP?
Also, you can run with -d for debug mode.

[GoranDervisevic]

Hi! I am using BCP, version 15.

I disabled it and it works! The symbols in the "description" column are correct now, thank you.

I looked into why it works without BCP, one of the reasons being that by disabling BCP Sling would then switch to standard row-by-row inserts which does reduce the insert speed but it handles Unicode and special characters better.

I'm also just thinking if maybe it is the BCP version in question that is the issue? Something to look into more.

But thank you @flarco! Thank you for the help.

[flarco]

There was an outstanding issue for this: #340

I just pushed a fix which I think works.
Can you try it with the latest dev build and report back?
See links here: https://github.com/slingdata-io/sling-cli?tab=readme-ov-file#automated-dev-builds

[flarco]

Try with BCP enabled.

[GoranDervisevic]

It works!

Version 1.2.19.dev (2024-09-09)

Did not have to use any transformations, the command was:

sling run --src-conn ORACLE --src-stream "ORACLE.TABLE" --tgt-conn SQL_SERVER --tgt-object "SQLS.TABLE"

The symbols in "description" column are correct.

[GoranDervisevic]

Hi @flarco,

Will the fix be merged into the main branch soon?