Feature/snowflake s3 stage v1#19

Closed

Abhishek Patil (abhishek-pattern) wants to merge 167 commits intomainfrom

feature/snowflake-s3-stage-v1

Commits on Feb 5, 2026

Add S3-Snowflake stage operations for large-scale data transfers
Abhishek Patil (abhishek-pattern)
committed
feat: implement S3 stage operations for Snowflake integration and add functional tests
Abhishek Patil (abhishek-pattern)
committed

Commits on Feb 9, 2026

feat: enhance Snowflake to S3 operations with configuration and query generation
Abhishek Patil (abhishek-pattern)
committed
feat: add S3 to Snowflake data transfer functionality
Abhishek Patil (abhishek-pattern)
committed
feat: add S3 data folder constant for Snowflake stage operations
Abhishek Patil (abhishek-pattern)
committed
refactor: remove outdated functional tests for pandas via S3 stage
Abhishek Patil (abhishek-pattern)
committed
feat: add batch inference function for Snowflake integration and infer schema from DataFrame
Abhishek Patil (abhishek-pattern)
committed
fix: add missing import for listing files in S3 for batch inference
Abhishek Patil (abhishek-pattern)
committed
feat: add batch size parameter to Snowflake to S3 copy query and update table creation logic
Abhishek Patil (abhishek-pattern)
committed
feat: enhance batch inference with multiprocessing for file processing
Abhishek Patil (abhishek-pattern)
committed
feat: refactor file processing in batch inference to use a dedicated function for improved readability and maintainability
Abhishek Patil (abhishek-pattern)
committed
feat: refactor file processing in batch inference to inline function for improved clarity and performance
Abhishek Patil (abhishek-pattern)
committed
feat: add function to download all files from S3 folder with URI validation
Abhishek Patil (abhishek-pattern)
committed
feat: replace file listing with direct download of all files from S3 in batch inference
Abhishek Patil (abhishek-pattern)
committed
feat: update input file processing to read parquet format instead of using S3 file retrieval function
Abhishek Patil (abhishek-pattern)
committed
feat: refactor _download_all_files_in_s3_folder to use a direct S3 client assignment for improved clarity
Abhishek Patil (abhishek-pattern)
committed
feat: optimize batch inference file processing using multiprocessing for improved performance
Abhishek Patil (abhishek-pattern)
committed
feat: add picklability check for process_file function before multiprocessing
Abhishek Patil (abhishek-pattern)
committed
feat: refactor process_file function for improved picklability and integrate with multiprocessing
Abhishek Patil (abhishek-pattern)
committed
feat: replace itertools.partial with functools.partial for improved performance
Abhishek Patil (abhishek-pattern)
committed
feat: replace multiprocessing.Pool with metaflow.parallel_map for improved parallel processing
Abhishek Patil (abhishek-pattern)
committed
feat: refactor process_file function for improved integration with parallel_map
Abhishek Patil (abhishek-pattern)
committed
feat: replace parallel_map with concurrent.futures.ProcessPoolExecutor for improved parallel processing
Abhishek Patil (abhishek-pattern)
committed
feat: switch from ProcessPoolExecutor to ThreadPoolExecutor for improved concurrency in batch inference
Abhishek Patil (abhishek-pattern)
committed

Commits on Feb 11, 2026

feat: enhance batch inference and S3 integration with schema inference and temporary file handling
Abhishek Patil (abhishek-pattern)
committed

Commits on Feb 12, 2026

Commits on Feb 16, 2026

Commits on Feb 18, 2026

Commits on Feb 19, 2026

Commits on Feb 23, 2026

Commits on Feb 24, 2026

Commits on Feb 25, 2026

Commits on Feb 26, 2026

fix: enhance S3 DataFrame retrieval by adding type mapping for decimal columns to ensure pandas compatibility
Abhishek Patil (abhishek-pattern)
committed

Commits on Feb 27, 2026