feat: add processor service with Redis-backed job queue

- Introduced a new `processor` service in the Docker Compose setup to handle face matching jobs.
- Configured Redis as a job queue and state management system for processing searches.
- Updated the backend to enqueue jobs and manage user locks using Redis.
- Added environment variables for Redis configuration and runtime paths.
- Created technical design documentation for the processor service outlining architecture, queue model, and search lifecycle.
- Updated package.json and package-lock.json to include dependencies for BullMQ and ioredis in the processor workspace.
- Added sample PKL files for local testing in the `test_pkl` directory.
This commit is contained in:
MaddoScientisto 2026-04-11 17:53:22 +02:00
commit bbb9c193ce
20 changed files with 1313 additions and 108 deletions

View file

@ -0,0 +1,12 @@
export const config = {
redisUrl: process.env.FACEAI_REDIS_URL || 'redis://redis:6379',
queueName: process.env.FACEAI_QUEUE_NAME || 'faceai-searches',
workerConcurrency: Number(process.env.FACEAI_WORKER_CONCURRENCY || 2),
workerTimeoutMs: Number(process.env.FACEAI_WORKER_TIMEOUT_MS || 5 * 60 * 1000),
runtimeRoot: process.env.FACEAI_RUNTIME_ROOT || '/data/runtime',
pklRoot: process.env.FACEAI_PKL_ROOT || '/data/pkl',
fallbackPklRoot: process.env.FACEAI_TEST_PKL_ROOT || '/data/pkl/test',
matcherBinary: process.env.FACEAI_MATCHER_BINARY || '/opt/face-recognition/face_matcher',
searchTtlSeconds: Number(process.env.FACEAI_SEARCH_TTL_SECONDS || 24 * 60 * 60),
resultTtlSeconds: Number(process.env.FACEAI_RESULT_TTL_SECONDS || 24 * 60 * 60)
};