Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug when replicating NUMERIC columns from postgres to parquet #405

Open
joconnor-ecaa opened this issue Oct 16, 2024 · 1 comment
Open
Labels
bug Something isn't working

Comments

@joconnor-ecaa
Copy link

Issue Description

  • Description of the issue:

Were replicating postgres tables to parquet files in Google Cloud Storage using Dagster's sling integration. When replicating a table using NUMERIC types, the outputs are occasionally wrong come out wrong: specifically, their decimal place moves, e.g. 0.123456... becomes 12345.6...

After some experimentation we noticed that this happens only when the NUMERIC has a large number of digits after the decimal point. Casting to float or rounding to

  • Sling version (sling --version):

1.2.20

  • Operating System (linux, mac, windows):

linux

  • Replication Configuration:
source: POSTGRES_DB
target: GCS_BUCKET
streams:
  gs://bucket/file.pq:
    object: public.some_table
  • Log Output (please run command with -d):

I don't have this right now, will follow up when possible.

@flarco
Copy link
Collaborator

flarco commented Oct 16, 2024

Interesting, thanks for raising. Some example values would be helpful.

@flarco flarco added the bug Something isn't working label Oct 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants