Image embeddings from an exported CLIP image encoder, and the plumbing to score face-verification pairs with them: the preprocessing CLIP was trained with, an encoder that runs an ONNX image tower, and a reader for LFW-style pair lists.
ClipPreprocessor does what Hugging Face's CLIPImageProcessor does: resize the shortest side to 224 with a bicubic filter, centre-crop 224 x 224, read the pixels as RGB, scale to 0..1 and normalise with the CLIP mean (0.48145466, 0.4578275, 0.40821073) and standard deviation (0.26862954, 0.26130258, 0.27577711), as a float tensor in CHW order. The default resize reproduces Pillow's bicubic filter bit for bit (its antialiasing support scaling and its 8-bit fixed-point arithmetic, written in Objeck, so no native code is added). OpenCV's INTER_CUBIC does not antialias when it shrinks an image and uses a different kernel, so it gives pixels that differ by up to a few grey levels; it is available as a mode and measured against the default in docs/AI.md.
ClipEncoder opens a CLIP image ONNX model (input pixel_values, output the embedding), embeds one image or a batch, and optionally L2-normalises. It scores pairs by cosine similarity, embedding every distinct image once. FacePairs reads the standard LFW pair lists (pairs.txt and friends) or a plain CSV.
Models are exported from the Hugging Face checkpoints, for example with torch.onnx.export of CLIPModel.get_image_features; the text tower is not needed. Nothing here ends the program on bad input: a call that cannot do what it was asked returns Nil or false and the object keeps the reason, read with GetError(), which is "" when there is nothing to report. A model that fails to open is not an error at construction (see IsOpen). Not thread-safe: use one object per thread. Compile with -lib ml_clip,ml_verify,opencv,onnx,models,cipher,json,gen_collect.
ClipEncoder
A CLIP image encoder: an exported ONNX image tower plus the preprocessing, giving one embedding per image. Not thread-safe. Close() releases the model.
Operations
- New
- Close
- Embed
- EmbedBatch
- EmbedImage
- GetDimension
- GetError
- GetPreprocessor
- IsOpen
- ScorePairs
- SetBatchSize
- SetVerbose
Embed #
Embeds one image file.
method : public : Embed(path:String, normalize:Bool) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| path | String | the image file |
| normalize | Bool | true to scale the embedding to length 1 |
Return
| Type | Description |
|---|---|
| Float | the embedding, or Nil (read GetError()) |
EmbedBatch #
Embeds image files, one row per file, in batches.
method : public : EmbedBatch(paths:String[], normalize:Bool) ~ Float[,]Parameters
| Name | Type | Description |
|---|---|---|
| paths | String | the image files |
| normalize | Bool | true to scale each embedding to length 1 |
Return
| Type | Description |
|---|---|
| Float | a (paths x dimension) matrix, or Nil when any image cannot be read or the run fails (GetError() names the image) |
EmbedImage #
Embeds one decoded image.
method : public : EmbedImage(image:API.OpenCV.Image, normalize:Bool) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| image | Image | the decoded image |
| normalize | Bool | true to scale the embedding to length 1 |
Return
| Type | Description |
|---|---|
| Float | the embedding, or Nil (read GetError()) |
GetDimension #
The embedding length (512 for the two base models, 768 for ViT-L/14).
method : public : GetDimension() ~ IntReturn
| Type | Description |
|---|---|
| Int | the length, or -1 when the model does not declare it and nothing has been embedded yet |
GetError #
Why the last call failed or the model did not open.
method : public : GetError() ~ StringReturn
| Type | Description |
|---|---|
| String | the reason, or "" when there is nothing to report |
GetPreprocessor #
The preprocessing in use.
method : public : GetPreprocessor() ~ ClipPreprocessorReturn
| Type | Description |
|---|---|
| ClipPreprocessor | the preprocessor |
IsOpen #
Whether the model opened.
method : public : IsOpen() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when images can be embedded |
New # constructor
Opens an image tower with the default preprocessing (224 pixels, Pillow bicubic).
New(model_path:String)Parameters
| Name | Type | Description |
|---|---|---|
| model_path | String | the ONNX model (input pixel_values [N, 3, 224, 224], output the embedding [N, D]) |
Example
encoder := ClipEncoder->New("clip-vit-base-patch32-image.onnx");
if(<>encoder->IsOpen()) {
encoder->GetError()->ErrorLine();
};New # constructor
Opens an image tower with session options and a chosen preprocessor.
New(model_path:String, config:Map<String,String>, pre:ClipPreprocessor)Parameters
| Name | Type | Description |
|---|---|---|
| model_path | String | the ONNX model |
| config | Map<String,String> | session options for API.Onnx.Session (for example ep = cpu), or Nil |
| pre | ClipPreprocessor | the preprocessing; its size must be the model's input size |
ScorePairs #
Scores every pair by the cosine similarity of the two images' embeddings. Each distinct image is embedded once, however many pairs use it.
method : public : ScorePairs(pairs:FacePairs) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| pairs | FacePairs | the pairs |
Return
| Type | Description |
|---|---|
| Float | one cosine per pair in pair order, which is what Verification->Evaluate takes with FacePairs->GetLabels; Nil when an image cannot be read or the run fails (GetError() says which) |