Image embeddings from an exported CLIP image encoder, and the plumbing to score face-verification pairs with them: the preprocessing CLIP was trained with, an encoder that runs an ONNX image tower, and a reader for LFW-style pair lists.
ClipPreprocessor does what Hugging Face's CLIPImageProcessor does: resize the shortest side to 224 with a bicubic filter, centre-crop 224 x 224, read the pixels as RGB, scale to 0..1 and normalise with the CLIP mean (0.48145466, 0.4578275, 0.40821073) and standard deviation (0.26862954, 0.26130258, 0.27577711), as a float tensor in CHW order. The default resize reproduces Pillow's bicubic filter bit for bit (its antialiasing support scaling and its 8-bit fixed-point arithmetic, written in Objeck, so no native code is added). OpenCV's INTER_CUBIC does not antialias when it shrinks an image and uses a different kernel, so it gives pixels that differ by up to a few grey levels; it is available as a mode and measured against the default in docs/AI.md.
ClipEncoder opens a CLIP image ONNX model (input pixel_values, output the embedding), embeds one image or a batch, and optionally L2-normalises. It scores pairs by cosine similarity, embedding every distinct image once. FacePairs reads the standard LFW pair lists (pairs.txt and friends) or a plain CSV.
Models are exported from the Hugging Face checkpoints, for example with torch.onnx.export of CLIPModel.get_image_features; the text tower is not needed. Nothing here ends the program on bad input: a call that cannot do what it was asked returns Nil or false and the object keeps the reason, read with GetError(), which is "" when there is nothing to report. A model that fails to open is not an error at construction (see IsOpen). Not thread-safe: use one object per thread. Compile with -lib ml_clip,ml_verify,opencv,onnx,models,cipher,json,gen_collect.
FacePairs
A list of image pairs, each marked as the same person or different people, as the Labeled Faces in the Wild benchmark ships them. Nothing here ends the program: a list that cannot be read has IsValid() false and GetError() says why.
Operations
GetError #
Why the list could not be read.
method : public : GetError() ~ StringReturn
| Type | Description |
|---|---|
| String | the reason, or "" when it was read |
GetFirst #
The first image of a pair.
method : public : GetFirst(i:Int) ~ StringParameters
| Name | Type | Description |
|---|---|---|
| i | Int | pair index |
Return
| Type | Description |
|---|---|
| String | the path, or Nil when the index is out of range |
GetFold #
The fold of a pair: 0 to 9 in pairs.txt, 0 elsewhere.
method : public : GetFold(i:Int) ~ IntParameters
| Name | Type | Description |
|---|---|---|
| i | Int | pair index |
Return
| Type | Description |
|---|---|
| Int | the fold, or -1 when the index is out of range |
GetImages #
The distinct image paths in the list, in order of first appearance.
method : public : GetImages() ~ String[]Return
| Type | Description |
|---|---|
| String | the paths |
GetLabels #
The labels, in pair order: what Verification->Evaluate takes with the scores.
method : public : GetLabels() ~ Bool[]Return
| Type | Description |
|---|---|
| Bool | true for a same pair |
GetSecond #
The second image of a pair.
method : public : GetSecond(i:Int) ~ StringParameters
| Name | Type | Description |
|---|---|---|
| i | Int | pair index |
Return
| Type | Description |
|---|---|
| String | the path, or Nil when the index is out of range |
IsSame #
Whether a pair is the same person.
method : public : IsSame(i:Int) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| i | Int | pair index |
Return
| Type | Description |
|---|---|
| Bool | true for a same pair (false for a different pair or an index out of range) |
IsValid #
Whether the list was read.
method : public : IsValid() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when it holds pairs |
OfFold #
The pairs of some folds.
method : public : OfFold(fold:Int) ~ FacePairsParameters
| Name | Type | Description |
|---|---|---|
| fold | Int | the fold to keep |
Return
| Type | Description |
|---|---|
| FacePairs | the pairs of that fold |
ReadCsv # function
Reads a plain pair list: one pair per line, "image1,image2,label", label 1 for the same person and 0 for different people. A line starting with # and blank lines are skipped. Paths are joined to images_dir (use "" for paths that are complete) and may not contain a comma.
function : ReadCsv(csv_file:String, images_dir:String) ~ FacePairsParameters
| Name | Type | Description |
|---|---|---|
| csv_file | String | the pair list |
| images_dir | String | the directory the image paths are relative to, or "" |
Return
| Type | Description |
|---|---|
| FacePairs | the pairs; check IsValid() |
ReadLfw # function
Reads an LFW pair list: pairs.txt (10 folds of 300 same and 300 different pairs, the "10_folds" set of scikit-learn), pairsDevTrain.txt (2200 pairs) or pairsDevTest.txt (1000 pairs, scikit-learn's "test" set). The first line is a header and is skipped; a line of three tab-separated fields (name, image number, image number) is a same pair, a line of four (name, number, name, number) a different pair. The image of person Name and number N is images_dir/Name/Name_000N.jpg. The fold of a pair is its 600-line block in pairs.txt and 0 in the other two files.
function : ReadLfw(pairs_file:String, images_dir:String) ~ FacePairsParameters
| Name | Type | Description |
|---|---|---|
| pairs_file | String | the pair list |
| images_dir | String | the directory with one folder per person (lfw_funneled or lfw) |
Return
| Type | Description |
|---|---|
| FacePairs | the pairs; check IsValid() |
Example
pairs := FacePairs->ReadLfw("lfw_home/pairsDevTest.txt", "lfw_home/lfw_funneled");
"{$pairs->Size()} pairs"->PrintLine();Stride #
Every step-th pair, starting with the first: a smaller list that keeps the order and so the mix of same and different pairs.
method : public : Stride(step:Int) ~ FacePairsParameters
| Name | Type | Description |
|---|---|---|
| step | Int | 1 or more |
Return
| Type | Description |
|---|---|
| FacePairs | the smaller list (an invalid one when the step is below 1) |