Refactor FaceAI build process and update Docker configurations to include matcher binary
This commit is contained in:
parent
b13c306883
commit
d36b0f4e46
15 changed files with 177 additions and 30 deletions
6
.dockerignore
Normal file
6
.dockerignore
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
*
|
||||
!faceai/
|
||||
!faceai/**
|
||||
!bin/
|
||||
!bin/Face_Recognition_Unix/
|
||||
!bin/Face_Recognition_Unix/**
|
||||
|
|
@ -6,6 +6,7 @@ on:
|
|||
- master
|
||||
paths:
|
||||
- faceai/**
|
||||
- bin/Face_Recognition_Unix/**
|
||||
- .forgejo/workflows/publish-faceai-container.yml
|
||||
workflow_dispatch:
|
||||
|
||||
|
|
@ -13,7 +14,7 @@ env:
|
|||
REGISTRY: ${{ vars.FORGEJO_REGISTRY }}
|
||||
IMAGE_NAMESPACE: ${{ vars.IMAGE_NAMESPACE }}
|
||||
IMAGE_NAME: ${{ vars.IMAGE_NAME != '' && vars.IMAGE_NAME || 'faceai' }}
|
||||
BUILD_CONTEXT: faceai
|
||||
BUILD_CONTEXT: .
|
||||
DOCKERFILE_PATH: faceai/docker/Dockerfile
|
||||
|
||||
jobs:
|
||||
|
|
@ -33,7 +34,8 @@ jobs:
|
|||
if [ -z "${IMAGE_NAMESPACE}" ]; then echo "vars.IMAGE_NAMESPACE is required"; exit 1; fi
|
||||
if [ -z "${IMAGE_NAME}" ]; then echo "vars.IMAGE_NAME resolved to an empty value"; exit 1; fi
|
||||
if [ ! -f "${DOCKERFILE_PATH}" ]; then echo "Dockerfile not found at ${DOCKERFILE_PATH}"; exit 1; fi
|
||||
if [ ! -f "${BUILD_CONTEXT}/package.json" ]; then echo "Build context is invalid: ${BUILD_CONTEXT}"; exit 1; fi
|
||||
if [ ! -f "faceai/package.json" ]; then echo "faceai/package.json is missing from the repository checkout"; exit 1; fi
|
||||
if [ ! -f "bin/Face_Recognition_Unix/face_matcher" ]; then echo "bin/Face_Recognition_Unix/face_matcher is missing from the repository checkout"; exit 1; fi
|
||||
|
||||
- name: Validate registry secrets
|
||||
run: |
|
||||
|
|
|
|||
|
|
@ -75,12 +75,11 @@ The checked-in `docker-compose.yml` starts:
|
|||
|
||||
The local stack also mounts:
|
||||
|
||||
- `../bin/Face_Recognition_Unix` into the processor container as the matcher binary source
|
||||
- `../test_pkl` into both the public FaceAI container and the processor container as the shared read-only PKL dataset root
|
||||
- `./logs` into both the public FaceAI container and the processor container as the persistent diagnostics directory
|
||||
- `../www` into the PHP container so the real bridge files are used
|
||||
|
||||
The `processor` service is built from `docker/processor.Dockerfile`, which uses a Debian Trixie-based Node 22 image, applies the current package upgrades available during build, and installs `libxcb1` so the bundled Linux `face_matcher` binary can run locally.
|
||||
The `processor` service is built from `docker/processor.Dockerfile` using the repository root as Docker build context. That image copies the checked-in `bin/Face_Recognition_Unix` directory into `/opt/face-recognition` during `docker build`, so the matcher binary is baked into the image instead of being mounted from the host at runtime.
|
||||
|
||||
### Persistent Logs
|
||||
|
||||
|
|
@ -212,9 +211,23 @@ LIVE_SITE_RUN_UPLOAD_FLOW=1
|
|||
When enabled, the live suite also:
|
||||
|
||||
- validates that the legacy Face ID handoff URL includes the race storage metadata expected by FaceAI
|
||||
- opens the real FaceAI app and asserts that the legacy header stylesheets load from the live legacy site
|
||||
- opens the real FaceAI app and asserts that the legacy header stylesheets load from the live legacy site without injecting cross-origin Font Awesome assets
|
||||
- confirms the app does not emit the `MISSING_RACE_STORAGE` invalid-race error on launch
|
||||
- uploads the supplied portrait image and verifies that search creation succeeds
|
||||
- uploads the supplied portrait image, waits for the search to complete, and requires a redirect back to the legacy result page with rendered results
|
||||
|
||||
### Processor Troubleshooting
|
||||
|
||||
If the processor logs show an error like `spawn /opt/face-recognition/face_matcher ENOENT`, the problem is not the upload flow itself. It means the running processor cannot see the matcher binary at the configured `FACEAI_MATCHER_BINARY` path.
|
||||
|
||||
With the current checked-in Dockerfiles, the matcher binary is copied into the image from the repository source tree during `docker build`. The runtime container no longer needs a host bind mount for `/opt/face-recognition`.
|
||||
|
||||
Published images now get that binary because the Forgejo container workflow builds from the repository root, which lets `faceai/docker/Dockerfile` copy:
|
||||
|
||||
```text
|
||||
bin/Face_Recognition_Unix/face_matcher
|
||||
```
|
||||
|
||||
If a running processor still reports `ENOENT`, the deployed image was built before this change or the build did not include the checked-in matcher directory.
|
||||
|
||||
## Optional Backend And Frontend Dev Loop
|
||||
|
||||
|
|
@ -320,7 +333,6 @@ services:
|
|||
- /mnt/storage/data/faceai/runtime:/data/runtime
|
||||
- /mnt/storage/data/faceai/logs:/data/logs
|
||||
- /mnt/nas12/nas2/RUS:/data/pkl:ro
|
||||
- /mnt/storage/data/faceai/bin/Face_Recognition_Unix:/opt/face-recognition:ro
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
|
@ -372,7 +384,7 @@ Processor settings:
|
|||
| Variable | Required | Example | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `FACEAI_PKL_ROOT` | yes | `/data/pkl` | mounted race-to-PKL dataset root |
|
||||
| `FACEAI_MATCHER_BINARY` | yes | `/opt/face-recognition/face_matcher` | matcher executable inside the processor container |
|
||||
| `FACEAI_MATCHER_BINARY` | yes | `/opt/face-recognition/face_matcher` | matcher executable baked into the image |
|
||||
| `FACEAI_WORKER_CONCURRENCY` | optional | `2` | BullMQ worker concurrency |
|
||||
| `FACEAI_WORKER_TIMEOUT_MS` | optional | `300000` | matcher timeout in milliseconds |
|
||||
|
||||
|
|
|
|||
|
|
@ -117,7 +117,10 @@ const emit = defineEmits([
|
|||
|
||||
<div class="faceai-dropzone-inner">
|
||||
<div class="faceai-dropzone-icon">
|
||||
<i class="fa fa-cloud-upload" aria-hidden="true"></i>
|
||||
<svg viewBox="0 0 24 24" aria-hidden="true" focusable="false">
|
||||
<path d="M12.75 3.75a.75.75 0 0 0-1.5 0v8.69L8.78 9.97a.75.75 0 1 0-1.06 1.06l3.75 3.75a.75.75 0 0 0 1.06 0l3.75-3.75a.75.75 0 1 0-1.06-1.06l-2.47 2.47V3.75Z" fill="currentColor" />
|
||||
<path d="M5.25 15a3 3 0 0 0 .28 6h12.94a3.53 3.53 0 0 0 .27-7.04 5.75 5.75 0 0 0-11.17-1.48A3.75 3.75 0 0 0 5.25 15Zm1.5 0c0-.87.35-1.65.92-2.22a.75.75 0 0 0 .18-.77 4.25 4.25 0 0 1 8.29 1.39v.25a.75.75 0 0 0 .75.75h1.58a2.03 2.03 0 0 1 0 4.06H5.53A1.5 1.5 0 1 1 5.53 15h1.22Z" fill="currentColor" />
|
||||
</svg>
|
||||
</div>
|
||||
|
||||
<template v-if="selectedFile">
|
||||
|
|
@ -290,6 +293,11 @@ const emit = defineEmits([
|
|||
box-shadow: inset 0 0 0 1px rgba(191, 158, 117, 0.24);
|
||||
}
|
||||
|
||||
.faceai-dropzone-icon svg {
|
||||
width: 44px;
|
||||
height: 44px;
|
||||
}
|
||||
|
||||
.faceai-dropzone-title {
|
||||
font-size: 1.2rem;
|
||||
font-weight: 700;
|
||||
|
|
|
|||
|
|
@ -64,7 +64,12 @@ function closeMenu() {
|
|||
<ul class="navbar-nav ml-auto">
|
||||
<li class="nav-item">
|
||||
<a class="nav-link active" href="#" @click="closeMenu">
|
||||
<i class="fa fa-user" aria-hidden="true"></i> Il mio account
|
||||
<span class="faceai-inline-icon" aria-hidden="true">
|
||||
<svg viewBox="0 0 24 24" focusable="false">
|
||||
<path d="M12 12.5a4.25 4.25 0 1 0 0-8.5 4.25 4.25 0 0 0 0 8.5Zm0 2.25c-4.48 0-8.25 2.18-8.25 4.75 0 .28.22.5.5.5h15.5a.5.5 0 0 0 .5-.5c0-2.57-3.77-4.75-8.25-4.75Z" fill="currentColor" />
|
||||
</svg>
|
||||
</span>
|
||||
Il mio account
|
||||
</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
|
|
@ -78,3 +83,18 @@ function closeMenu() {
|
|||
</nav>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<style scoped>
|
||||
.faceai-inline-icon {
|
||||
display: inline-flex;
|
||||
width: 1rem;
|
||||
height: 1rem;
|
||||
margin-right: 0.35rem;
|
||||
vertical-align: text-bottom;
|
||||
}
|
||||
|
||||
.faceai-inline-icon svg {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
}
|
||||
</style>
|
||||
|
|
|
|||
|
|
@ -36,7 +36,6 @@ export function legacyAsset(path) {
|
|||
export function injectLegacyStylesheets() {
|
||||
const stylesheets = [
|
||||
legacyAsset('/vendor/bootstrap/css/bootstrap.min.css'),
|
||||
legacyAsset('/css/font-awesome.min.css'),
|
||||
'https://fonts.googleapis.com/css?family=Roboto:300,300i,400,400i,700,700i',
|
||||
legacyAsset('/css/custom-style.css')
|
||||
];
|
||||
|
|
|
|||
|
|
@ -40,6 +40,10 @@ export async function runFaceMatcher({ matcherBinary, selfiePath, pklPath, csvPa
|
|||
|
||||
child.on('error', (error) => {
|
||||
clearTimeout(timer);
|
||||
if (error?.code === 'ENOENT') {
|
||||
reject(new Error(`face_matcher not found at ${matcherBinary}. Check FACEAI_MATCHER_BINARY and the processor bind mount.`));
|
||||
return;
|
||||
}
|
||||
reject(error);
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import { constants as fsConstants } from 'node:fs';
|
||||
import fs from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
import { Worker } from 'bullmq';
|
||||
|
|
@ -15,6 +16,22 @@ import { parseMatcherCsv, resolvePklPath, runFaceMatcher } from './worker-utils.
|
|||
|
||||
const connection = createRedisConnection(config.redisUrl);
|
||||
|
||||
async function ensureMatcherBinaryAvailable() {
|
||||
try {
|
||||
await fs.access(config.matcherBinary, fsConstants.X_OK);
|
||||
} catch (error) {
|
||||
console.error(`FaceAI processor cannot start because the matcher binary is unavailable: ${config.matcherBinary}`);
|
||||
console.error('Ensure FACEAI_MATCHER_BINARY points to a real executable and that the processor bind mount contains face_matcher.');
|
||||
if (error?.code === 'ENOENT') {
|
||||
console.error('The configured matcher path does not exist inside the processor runtime.');
|
||||
} else if (error?.code === 'EACCES') {
|
||||
console.error('The configured matcher path exists but is not executable by the processor runtime.');
|
||||
}
|
||||
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
function formatLogLine(message, details) {
|
||||
const timestamp = new Date().toISOString();
|
||||
if (details === undefined) {
|
||||
|
|
@ -147,6 +164,8 @@ async function processJob(job) {
|
|||
}
|
||||
}
|
||||
|
||||
await ensureMatcherBinaryAvailable();
|
||||
|
||||
const worker = new Worker(config.queueName, processJob, {
|
||||
connection,
|
||||
concurrency: config.workerConcurrency
|
||||
|
|
|
|||
|
|
@ -48,8 +48,8 @@ services:
|
|||
|
||||
processor:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: docker/processor.Dockerfile
|
||||
context: ..
|
||||
dockerfile: faceai/docker/processor.Dockerfile
|
||||
image: regalami-faceai-processor-local
|
||||
container_name: regalami-faceai-processor
|
||||
restart: unless-stopped
|
||||
|
|
@ -74,7 +74,6 @@ services:
|
|||
volumes:
|
||||
- .:/app
|
||||
- ./logs:/data/logs
|
||||
- ../bin/Face_Recognition_Unix:/opt/face-recognition:ro
|
||||
- ../test_pkl:/data/pkl:ro
|
||||
- faceai-runtime:/data/runtime
|
||||
depends_on:
|
||||
|
|
|
|||
|
|
@ -1,25 +1,40 @@
|
|||
FROM node:20-alpine AS build
|
||||
FROM node:22-trixie-slim AS build
|
||||
|
||||
RUN apt-get update \
|
||||
&& apt-get upgrade -y \
|
||||
&& apt-get install -y --no-install-recommends libxcb1 \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY package.json ./
|
||||
COPY apps/frontend/package.json apps/frontend/package.json
|
||||
COPY apps/backend/package.json apps/backend/package.json
|
||||
COPY apps/processor/package.json apps/processor/package.json
|
||||
COPY faceai/package.json ./package.json
|
||||
COPY faceai/apps/frontend/package.json apps/frontend/package.json
|
||||
COPY faceai/apps/backend/package.json apps/backend/package.json
|
||||
COPY faceai/apps/processor/package.json apps/processor/package.json
|
||||
|
||||
RUN npm install
|
||||
|
||||
COPY . .
|
||||
COPY faceai/ .
|
||||
COPY bin/Face_Recognition_Unix /opt/face-recognition
|
||||
|
||||
RUN chmod +x /opt/face-recognition/face_matcher
|
||||
|
||||
RUN npm run build
|
||||
|
||||
FROM node:20-alpine AS runtime
|
||||
FROM node:22-trixie-slim AS runtime
|
||||
|
||||
RUN apt-get update \
|
||||
&& apt-get upgrade -y \
|
||||
&& apt-get install -y --no-install-recommends libxcb1 \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY --from=build /app /app
|
||||
COPY --from=build /opt/face-recognition /opt/face-recognition
|
||||
|
||||
ENV NODE_ENV=production
|
||||
ENV FACEAI_MATCHER_BINARY=/opt/face-recognition/face_matcher
|
||||
EXPOSE 3001
|
||||
|
||||
CMD ["npm", "run", "start"]
|
||||
|
|
@ -6,3 +6,10 @@ RUN apt-get update \
|
|||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY faceai /app
|
||||
COPY bin/Face_Recognition_Unix /opt/face-recognition
|
||||
|
||||
RUN chmod +x /opt/face-recognition/face_matcher
|
||||
|
||||
ENV FACEAI_MATCHER_BINARY=/opt/face-recognition/face_matcher
|
||||
|
|
@ -4,6 +4,7 @@ const {
|
|||
LIVE_SITE_BASE_URL,
|
||||
LIVE_SITE_PORTRAIT_PATH,
|
||||
LIVE_SITE_RACE_URL,
|
||||
LIVE_SITE_RESULT_URL_PATTERN,
|
||||
LIVE_SITE_RUN_UPLOAD_FLOW,
|
||||
ensureLiveAuthenticatedRacePage,
|
||||
requirePortraitFixture
|
||||
|
|
@ -51,6 +52,34 @@ async function openLiveFaceAi(page) {
|
|||
};
|
||||
}
|
||||
|
||||
async function waitForSearchCompletion(page, searchId) {
|
||||
return expect.poll(async () => {
|
||||
return page.evaluate(async (id) => {
|
||||
const response = await fetch(`/api/searches/${id}`, { credentials: 'include' });
|
||||
if (!response.ok) {
|
||||
return {
|
||||
status: 'poll-error',
|
||||
httpStatus: response.status
|
||||
};
|
||||
}
|
||||
|
||||
const payload = await response.json();
|
||||
return {
|
||||
status: payload.status,
|
||||
errorMessage: payload.errorMessage || null,
|
||||
completionCode: payload.completionCode || null,
|
||||
matchCount: payload.matchCount ?? null,
|
||||
resultId: payload.resultId || null
|
||||
};
|
||||
}, searchId);
|
||||
}, {
|
||||
timeout: 3 * 60 * 1000,
|
||||
message: `Expected FaceAI search ${searchId} to complete successfully.`
|
||||
}).toMatchObject({
|
||||
status: 'completed'
|
||||
});
|
||||
}
|
||||
|
||||
test('loads a live race page with an authenticated session', async ({ page }) => {
|
||||
await ensureLiveAuthenticatedRacePage(page);
|
||||
|
||||
|
|
@ -70,6 +99,7 @@ test('launches the live FaceAI app with race storage metadata and a styled heade
|
|||
await expect(page.locator('nav.navbar')).toBeVisible();
|
||||
await expect(page.locator('link[data-legacy-href*="bootstrap.min.css"]')).toHaveCount(1);
|
||||
await expect(page.locator('link[data-legacy-href*="custom-style.css"]')).toHaveCount(1);
|
||||
await expect(page.locator('link[data-legacy-href*="font-awesome"]')).toHaveCount(0);
|
||||
|
||||
const legacyStylesheetHrefs = await page.locator('link[data-legacy-href]').evaluateAll((elements) => {
|
||||
return elements.map((element) => element.getAttribute('href') || '');
|
||||
|
|
@ -116,6 +146,32 @@ test('accepts the supplied portrait image for the live upload flow', async ({ pa
|
|||
expect(searchResponse.ok(), 'Expected the live upload flow to create a FaceAI search successfully.').toBe(true);
|
||||
|
||||
const searchPayload = await searchResponse.json();
|
||||
expect(searchPayload.id || searchPayload.searchId, 'Expected the search creation response to include a search identifier.').toBeTruthy();
|
||||
await expect(page.locator('.faceai-feedback')).not.toContainText(/Impossibile|Unable|Errore|Error/i);
|
||||
const searchId = searchPayload.id || searchPayload.searchId;
|
||||
expect(searchId, 'Expected the search creation response to include a search identifier.').toBeTruthy();
|
||||
|
||||
const completion = await page.evaluate(async (id) => {
|
||||
const response = await fetch(`/api/searches/${id}`, { credentials: 'include' });
|
||||
return response.json();
|
||||
}, searchId).catch(() => null);
|
||||
|
||||
if (completion?.status === 'failed') {
|
||||
throw new Error(`FaceAI search ${searchId} failed immediately: ${completion.errorMessage || 'unknown error'}`);
|
||||
}
|
||||
|
||||
await waitForSearchCompletion(page, searchId);
|
||||
|
||||
await page.waitForURL(new RegExp(`^${escapeRegExp(LIVE_SITE_RESULT_URL_PATTERN)}`), {
|
||||
timeout: 3 * 60 * 1000
|
||||
});
|
||||
|
||||
await expect.poll(async () => page.url(), {
|
||||
timeout: 15 * 1000,
|
||||
message: 'Expected the browser to land on the legacy result page after FaceAI completed.'
|
||||
}).toMatch(new RegExp(`^${escapeRegExp(LIVE_SITE_BASE_URL)}/`));
|
||||
|
||||
await expect(page.locator('body')).toContainText(/Vista filtrata da FaceAI|foto da FaceAI|ID foto:/i);
|
||||
await expect.poll(async () => page.locator('.gallery-card').count(), {
|
||||
timeout: 15 * 1000,
|
||||
message: 'Expected the legacy return page to render at least one FaceAI result card.'
|
||||
}).toBeGreaterThan(0);
|
||||
});
|
||||
|
|
@ -7,6 +7,7 @@ const LIVE_SITE_BASE_URL = process.env.LIVE_SITE_BASE_URL || 'https://www.regala
|
|||
const LIVE_SITE_LOGIN_URL = process.env.LIVE_SITE_LOGIN_URL || `${LIVE_SITE_BASE_URL}/login_clienti-it.html`;
|
||||
const LIVE_SITE_RACE_URL = process.env.LIVE_SITE_RACE_URL || `${LIVE_SITE_BASE_URL}/42%20HALF%20MARATHON%20FIRENZE_gara-1018545---96-1.html`;
|
||||
const LIVE_FACEAI_BASE_URL = process.env.LIVE_FACEAI_BASE_URL || 'https://ai.regalamiunsorriso.it';
|
||||
const LIVE_SITE_RESULT_URL_PATTERN = process.env.LIVE_SITE_RESULT_URL_PATTERN || `${LIVE_SITE_BASE_URL}/faceai_return.php`;
|
||||
const LIVE_SITE_USERNAME = process.env.LIVE_SITE_USERNAME || '';
|
||||
const LIVE_SITE_PASSWORD = process.env.LIVE_SITE_PASSWORD || '';
|
||||
const LIVE_SITE_PORTRAIT_PATH = process.env.LIVE_SITE_PORTRAIT_PATH || path.join(WORKSPACE_ROOT, 'test_pkl', 'live', 'test_portrait_1.png');
|
||||
|
|
@ -109,6 +110,7 @@ module.exports = {
|
|||
LIVE_SITE_PASSWORD,
|
||||
LIVE_SITE_PORTRAIT_PATH,
|
||||
LIVE_SITE_RACE_URL,
|
||||
LIVE_SITE_RESULT_URL_PATTERN,
|
||||
LIVE_SITE_RUN_UPLOAD_FLOW,
|
||||
LIVE_SITE_USERNAME,
|
||||
dismissCookieBanner,
|
||||
|
|
|
|||
|
|
@ -48,7 +48,6 @@ services:
|
|||
- /var/docker/faceai/runtime:/data/runtime
|
||||
- /var/docker/faceai/logs:/data/logs
|
||||
- /mnt/nas12/nas2/RUS:/data/pkl:ro
|
||||
- /var/docker/faceai/bin/Face_Recognition_Unix:/opt/face-recognition:ro
|
||||
depends_on:
|
||||
- redis
|
||||
|
||||
|
|
|
|||
|
|
@ -10,13 +10,12 @@ All files in this rollout are deployed from the current working tree.
|
|||
|
||||
## New Files
|
||||
|
||||
- `www/_inc_faceai_identity.jsp`
|
||||
- `www/_js/lang.js`
|
||||
- None in this rollout.
|
||||
|
||||
## Updated Files
|
||||
|
||||
- `www/fotoCR-en.jsp`
|
||||
- `www/fotoCR.jsp`
|
||||
- `www/_js/rus-ecom-240621.js`
|
||||
- `www/faceai_handoff.php`
|
||||
|
||||
## Excluded Files
|
||||
|
||||
|
|
@ -28,7 +27,7 @@ All files in this rollout are deployed from the current working tree.
|
|||
- Remote host: `marco@83.149.164.4:410`
|
||||
- Remote staging path: `/home/marco/regalamiunsorriso/incoming/www`
|
||||
- Remote live path: `/home/sites/regalamiunsorriso/www`
|
||||
- Total files in this manifest: `4`
|
||||
- Total files in this manifest: `2`
|
||||
|
||||
## Transfer Method
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue