Bundle ONNX Runtime inference library for running pre-trained models. Supports object detection (YOLO), image classification (ResNet), segmentation (DeepLab), pose estimation (OpenPose), face recognition, and Phi-3 text generation. Compile with -lib onnx.
Phi3Session
Phi-3 / SLM inference session for text generation. Supports autoregressive generation with KV-cache management, temperature-based sampling, and configurable EOS tokens.
Example
session := Phi3Session->New("phi3-mini/model.onnx");
token_ids := [1, 3148, 1001, 29901, 825, 338, 29871, 29906, 718, 29871, 29906, 29973];
eos_tokens := [2, 32007];
result := session->Generate(token_ids, 128, 0.7, eos_tokens);
each(token in result->GetTokens()) {
token->PrintLine();
};
session->Close();Operations
Generate #
Generate text tokens from input token IDs
method : public : Generate(token_ids:Int[], max_tokens:Int, temperature:Float, eos_tokens:Int[]) ~ API.Onnx.Phi3ResultParameters
| Name | Type | Description |
|---|---|---|
| token_ids | Int | input token IDs (prompt) |
| max_tokens | Int | maximum number of tokens to generate |
| temperature | Float | sampling temperature (0.0 for greedy) |
| eos_tokens | Int | array of end-of-sequence token IDs |
Return
| Type | Description |
|---|---|
| Phi3Result | generation result with output token IDs |
Example
session := Phi3Session->New("phi3-mini/model.onnx");
tokenizer := Phi3Tokenizer->New("phi3-mini/tokenizer.json");
ids := tokenizer->Encode("<|user|>\nWhat is 2+2?<|end|>");
eos := [32007, 32000];
result := session->Generate(ids, 128, 0.0, eos);
tokenizer->Decode(result->GetTokens())->PrintLine();
session->Close();