Skip to content

Commit

Permalink
[docs] Add document for nested column evolution (#4619)
Browse files Browse the repository at this point in the history
  • Loading branch information
tsreaper authored Dec 2, 2024
1 parent 39b82ec commit 271de7d
Show file tree
Hide file tree
Showing 2 changed files with 100 additions and 0 deletions.
16 changes: 16 additions & 0 deletions docs/content/flink/sql-alter.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,10 @@ If you use object storage, such as S3 or OSS, please use this syntax carefully,

The following SQL adds two columns `c1` and `c2` to table `my_table`.

{{< hint info >}}
To add a column in a row type, see [Changing Column Type](#changing-column-type).
{{< /hint >}}

```sql
ALTER TABLE my_table ADD (c1 INT, c2 STRING);
```
Expand All @@ -99,6 +103,10 @@ otherwise this operation may fail, throws an exception like `The following colum
ALTER TABLE my_table DROP (c1, c2);
```

{{< hint info >}}
To drop a column in a row type, see [Changing Column Type](#changing-column-type).
{{< /hint >}}

## Dropping Partitions

The following SQL drops the partitions of the paimon table.
Expand Down Expand Up @@ -185,6 +193,14 @@ The following SQL changes type of column `col_a` to `DOUBLE`.
ALTER TABLE my_table MODIFY col_a DOUBLE;
```

Paimon also supports changing columns of row type, array type, and map type.

```sql
-- col_a previously has type ARRAY<MAP<INT, ROW(f1 INT, f2 STRING)>>
-- the following SQL changes f1 to BIGINT, drops f2, and adds f3
ALTER TABLE my_table MODIFY col_a ARRAY<MAP<INT, ROW(f1 BIGINT, f3 DOUBLE)>>;
```

## Adding watermark

The following SQL adds a computed column `ts` from existing column `log_ts`, and a watermark with strategy `ts - INTERVAL '1' HOUR` on column `ts` which is marked as event time attribute of table `my_table`.
Expand Down
84 changes: 84 additions & 0 deletions docs/content/spark/sql-alter.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,27 @@ ALTER TABLE my_table ADD COLUMNS (
);
```

The following SQL adds a nested column `f3` to a struct type.

```sql
-- column v previously has type STRUCT<f1: STRING, f2: INT>
ALTER TABLE my_table ADD COLUMN v.f3 STRING;
```

The following SQL adds a nested column `f3` to a struct type, which is the element type of an array type.

```sql
-- column v previously has type ARRAY<STRUCT<f1: STRING, f2: INT>>
ALTER TABLE my_table ADD COLUMN v.element.f3 STRING;
```

The following SQL adds a nested column `f3` to a struct type, which is the value type of a map type.

```sql
-- column v previously has type MAP<INT, STRUCT<f1: STRING, f2: INT>>
ALTER TABLE my_table ADD COLUMN v.value.f3 STRING;
```

## Renaming Column Name

The following SQL renames column `c0` in table `my_table` to `c1`.
Expand All @@ -103,6 +124,27 @@ The following SQL renames column `c0` in table `my_table` to `c1`.
ALTER TABLE my_table RENAME COLUMN c0 TO c1;
```

The following SQL renames a nested column `f1` to `f100` in a struct type.

```sql
-- column v previously has type STRUCT<f1: STRING, f2: INT>
ALTER TABLE my_table RENAME COLUMN v.f1 to f100;
```

The following SQL renames a nested column `f1` to `f100` in a struct type, which is the element type of an array type.

```sql
-- column v previously has type ARRAY<STRUCT<f1: STRING, f2: INT>>
ALTER TABLE my_table RENAME COLUMN v.element.f1 to f100;
```

The following SQL renames a nested column `f1` to `f100` in a struct type, which is the value type of a map type.

```sql
-- column v previously has type MAP<INT, STRUCT<f1: STRING, f2: INT>>
ALTER TABLE my_table RENAME COLUMN v.value.f1 to f100;
```

## Dropping Columns

The following SQL drops two columns `c1` and `c2` from table `my_table`.
Expand All @@ -111,6 +153,27 @@ The following SQL drops two columns `c1` and `c2` from table `my_table`.
ALTER TABLE my_table DROP COLUMNS (c1, c2);
```

The following SQL drops a nested column `f2` from a struct type.

```sql
-- column v previously has type STRUCT<f1: STRING, f2: INT>
ALTER TABLE my_table DROP COLUMN v.f2;
```

The following SQL drops a nested column `f2` from a struct type, which is the element type of an array type.

```sql
-- column v previously has type ARRAY<STRUCT<f1: STRING, f2: INT>>
ALTER TABLE my_table DROP COLUMN v.element.f2;
```

The following SQL drops a nested column `f2` from a struct type, which is the value type of a map type.

```sql
-- column v previously has type MAP<INT, STRUCT<f1: STRING, f2: INT>>
ALTER TABLE my_table DROP COLUMN v.value.f2;
```

## Dropping Partitions

The following SQL drops the partitions of the paimon table. For spark sql, you need to specify all the partition columns.
Expand Down Expand Up @@ -156,3 +219,24 @@ ALTER TABLE my_table ALTER COLUMN col_a AFTER col_b;
```sql
ALTER TABLE my_table ALTER COLUMN col_a TYPE DOUBLE;
```

The following SQL changes the type of a nested column `f2` to `BIGINT` in a struct type.

```sql
-- column v previously has type STRUCT<f1: STRING, f2: INT>
ALTER TABLE my_table ALTER COLUMN v.f2 TYPE BIGINT;
```

The following SQL changes the type of a nested column `f2` to `BIGINT` in a struct type, which is the element type of an array type.

```sql
-- column v previously has type ARRAY<STRUCT<f1: STRING, f2: INT>>
ALTER TABLE my_table ALTER COLUMN v.element.f2 TYPE BIGINT;
```

The following SQL changes the type of a nested column `f2` to `BIGINT` in a struct type, which is the value type of a map type.

```sql
-- column v previously has type MAP<INT, STRUCT<f1: STRING, f2: INT>>
ALTER TABLE my_table ALTER COLUMN v.value.f2 TYPE BIGINT;
```

0 comments on commit 271de7d

Please sign in to comment.