Optimize job processing to handle more than 100k jobs/sec

[uuuchit]

Optimize job processing to handle more than 100k jobs/sec by implementing batch processing, parallel execution, and Redis pipelining.

• pop-queue/queue.js

  • Add configuration options for batch size, parallel execution, and Redis pipelining.
  • Implement batch processing for job execution.
  • Enhance Redis operations to use pipelining and Lua scripts for atomic operations.
  • Modify job runner to handle batch processing and parallel execution.

• pop-queue/load-test.js

  • Add tests for batch processing, parallel execution, and Redis pipelining.
  • Create and enqueue jobs for batch processing, parallel execution, and Redis pipelining tests.

• pop-queue/config.js

  • Add configuration options for batch size, parallel execution, and Redis pipelining.
  • Validate new configuration values.

• .github/wiki/Scaling-and-Performance.md

  • Update documentation to include new performance optimizations and configuration options for batch size, parallel execution, and Redis pipelining.