Direct memory leaks when reading parquet files containing interleaving plain/dictionary pages

[CodingJun]

Apache Iceberg version

1.5.2

Query engine

Spark

Please describe the bug 🐞

When I used spark to continuously rewrite the data file, I found a direct memory leak. The reason is that if parquet file containing interleaving plain/dictionary encoding in different RowGroups, the old Vector cannot be released normally.

I found a pull request that mentioned this issue, but it has not been merged.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[Fokko]

@CodingJun Are you able to check if the problem persists in 1.7.0?

[CodingJun]

@CodingJun Are you able to check if the problem persists in 1.7.0?

Yes, I have checked, and there is still a problem in 1.7.0.