Skip to content

[AMORO-4044] Fix partition mapping bug in TableEntriesScan for tables with evolved PartitionSpec#4085

Open
j1wonpark wants to merge 1 commit intoapache:masterfrom
j1wonpark:fix/AMORO-4044-partition-mapping-bug
Open

[AMORO-4044] Fix partition mapping bug in TableEntriesScan for tables with evolved PartitionSpec#4085
j1wonpark wants to merge 1 commit intoapache:masterfrom
j1wonpark:fix/AMORO-4044-partition-mapping-bug

Conversation

@j1wonpark
Copy link
Contributor

@j1wonpark j1wonpark commented Feb 13, 2026

Why are the changes needed?

The entries metadata table returns a unified super-struct partition containing fields from all PartitionSpecs. In TableEntriesScan, both buildDataFile() and buildDeleteFile() passed this unified partition directly to withPartition(), which causes a partition struct mismatch for files written under a non-latest PartitionSpec. This is the root cause of #4044.

This PR fixes the issue at the TableEntriesScan level by projecting the unified partition to the spec-specific partition type using StructProjection.createAllowMissing(), so that all callers of TableEntriesScan get correct partition values regardless of spec evolution.

Close #4044.

Brief change log

  • Add projectPartition() method that uses StructProjection.createAllowMissing() to project the unified partition struct to the file's spec-specific partition type
  • Apply the projection in both buildDataFile() and buildDeleteFile()
  • Cache the unified partition type via Partitioning.partitionType(table) for reuse

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible
  • Add screenshots for manual tests if appropriate
  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
    no
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
    not applicable

… with evolved PartitionSpec

The entries metadata table returns a unified super-struct partition
containing fields from all PartitionSpecs. buildDataFile() and
buildDeleteFile() passed this directly to withPartition(), causing
partition mismatch after spec evolution. Fix by projecting the unified
partition to the spec-specific type using StructProjection.

Signed-off-by: Jiwon Park <jpark92@outlook.kr>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: For Iceberg tables whose PartitionSpec has been changed, Amoro will throw an error when executing the clean-dangling-delete-files operation.

1 participant