Conversation
…ial XarrayBuffer Set up the dataframe interchange protocol scaffolding based on earlier guidance and implemented the initial XarrayBuffer component. Introduced a dedicated xarray_sql.dataframe_protocol subpackage so protocol-specific logic has its own organized namespace rather than crowding existing modules. Open to restructuring this if you prefer the classes to live directly under the main package. Currently placed both the abstract Buffer interface and the concrete XarrayBuffer implementation in core.py; happy to split these into separate files if that aligns better with the project structure. Once the layout is settled, I’ll proceed with implementing the remaining two protocol classes as discussed.
|
I don't have pre-commit set up, but to pass the lint check, I recommend running the following: I'm excited to read this PR! That was fast. |
alxmrs
left a comment
alxmrs
left a comment
There was a problem hiding this comment.
A closer look, this is quite promising! Thanks for your quick, quality work here.
…lumn. In the XarrayColumn, the null_count function currently returns None. Should we compute the actual value here, or keep it as None for now?
alxmrs
left a comment
alxmrs
left a comment
There was a problem hiding this comment.
I have concerns for the current data model of XarrayColumn, but the interfaces are looking sharp.
| can expose protocol chunks via num_chunks()/get_chunks(). | ||
|
|
||
| Notes: | ||
| - For lazy (dask) arrays, materializing buffers requires compute. This can |
There was a problem hiding this comment.
Forgive me if I misunderstand; I don't think we want this assert. The trick we exploit in this library is that it is cheap to flatten ndarrays into vectors (a matter of changing metadata/strides). Doing this while handling chunks will take some care, but I believe it is possible.
There was a problem hiding this comment.
It may be of help to see the "core.py" at the package root to see what I've tried in the past to achieve this effect.
This change removes the 1-D restriction in XarrayColumn.__init__ by flattening multi-dimensional DataArrays to 1-D using .data.ravel(). This mirrors the approach already used in core.py for handling ndarray inputs.
|
Hey @alxmrs — just checking in. |
2 participants
Set up the dataframe interchange protocol scaffolding and implemented the initial XarrayBuffer component.
Introduced a dedicated xarray_sql.dataframe_protocol subpackage so protocol-specific logic has its own organized namespace rather than crowding existing modules. Open to restructuring this if you prefer the classes to live directly under the main package.
Currently placed both the abstract Buffer interface and the concrete XarrayBuffer implementation in core.py; happy to split these into separate files if that aligns better with the project structure.
Once the layout is settled, I’ll proceed with implementing the remaining two protocol classes as discussed.