Skip to content

GH-49078: [FS][Azure] Fix lossy pickling of SubTreeFileSystem(base_path, AzureFileSystem(...))#49140

Open
Tom-Newton wants to merge 16 commits intoapache:mainfrom
Tom-Newton:tomnewton/fix_lossy_pickling_of_azure_filesystem/GH-49078
Open

GH-49078: [FS][Azure] Fix lossy pickling of SubTreeFileSystem(base_path, AzureFileSystem(...))#49140
Tom-Newton wants to merge 16 commits intoapache:mainfrom
Tom-Newton:tomnewton/fix_lossy_pickling_of_azure_filesystem/GH-49078

Conversation

@Tom-Newton
Copy link
Contributor

@Tom-Newton Tom-Newton commented Feb 4, 2026

Rationale for this change

Fix #49078

What changes are included in this PR?

  • Implement getters on the C++ side of AzureOptions, for the values that are currently stored only on the python side.
    • This required adding some more member variables
    • I decided to add ClearCredentials , so that it can't get into strange states by configuring one credential type then another. IMO configuring the credentials during initialisation on the AzureOptions would be neater but I don't want to make this PR too big.
  • Update the C++ side AzureOptions::Equals
  • Remove python side self attributes and instead depend on getters from C++ side.

Are these changes tested?

  • Updated tests on the C++ side for the updated Equals and newly added getter methods.
  • Added a fixture pickle_with_and_without_subtree_filesystem, which can be used in place of the pickle_module. This adds test combinations with and without wrapping the filesystem in a SubTreeFilesystem before pickling it.

Are there any user-facing changes?

Only that pickling SubTreeFileSystem(base_path, AzureFileSystem(...)) now works properly.

@github-actions
Copy link

github-actions bot commented Feb 4, 2026

⚠️ GitHub issue #49078 has been automatically assigned in GitHub to PR creator.

@Tom-Newton Tom-Newton force-pushed the tomnewton/fix_lossy_pickling_of_azure_filesystem/GH-49078 branch from 28ad544 to d0a02e3 Compare February 6, 2026 00:25
@Tom-Newton Tom-Newton marked this pull request as ready for review February 6, 2026 09:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Python][FS][Azure] Pickling SubTreeFileSystem(base_path, AzureFileSystem(...)) is lossy

1 participant