CSV vs parquet

[amardeep]

I was looking into using sling to sync data from snowflake to bigquery.

• As far as i could tell, sling exports data from source in csv format. Is there any plan to support parquet instead? Any pros/cons of using csv.
• Snowflake supports exporting (unloading) directly to s3/gcs. Is there any plans to support that.

[flarco]

Hi, yes CSV enables streaming (don't need the whole file in memory). Parquet needs to fit into memory for sling. There are complex ways to do this without streaming, but sling doesn't support that.

For your second point, true, but not supported ATM. There were plans to support bypassing sling (Snowflake -> GCS -> BigQuery instead of Snowflake -> Internal Staging -> Sling -> GCS -> BigQuery) but it's more of an edge case and can bring complications since we can't validate row count and value. Sling does this automatically as data passes though it.

Maybe write a python script? to export all your tables into GCS as parquet and import into BigQuery. The tedious part is writing all the DDLs in BQ (which is why sling exists :). So you'll have to map the column types over. Feel free to use types_native_to_general.tsv and types_general_to_native.tsv which sling uses.

[dridk]

A good approach would be to partitioned data. For instance, read N line from the stream and stored it as part1.parquet, then part2.parquet ....