Objeck Documentation

Bundle

Client for HTTP model-inference servers. Sends tensors to a running server and reads tensors back, so a model trained in TensorFlow, PyTorch, ONNX or scikit-learn can be used from Objeck without a native binding for each runtime. Two wire protocols, because no single one covers the field: - OPEN_INFERENCE is the KServe v2 protocol, spoken by NVIDIA Triton, KServe, OpenVINO Model Server, Seldon MLServer and others. Triton alone serves TensorFlow, ONNX, PyTorch, TensorRT and OpenVINO behind it, so this one protocol reaches most servers and most runtimes. It is the default. - TF_SERVING is TensorFlow Serving's own REST API, which does not implement KServe v2 and so needs its own request shape. The difference is confined to Client: a Tensor is the same object either way. Choosing the protocol rather than the server is the whole point -- a client written against one server's API reaches one server. This is a client, not a runtime. The server is a separate process the user runs; nothing here loads a model. For in-process inference without a server, API.Onnx runs models directly. Compile with -lib inference.

Client

A connection to one inference server. Holds the host and the wire protocol; a Tensor is the same object either way, so switching servers is a constructor argument rather than a rewrite.

Example

client := Client->New("http://localhost:8000");
input := Tensor->New("input", shape, pixels);
outputs := client->Predict("resnet50", input);
if(outputs = Nil) {
EndPoint->GetLastError()->PrintLine();
};

BuildTfServingBody #

TF Serving infers a tensor's shape from how deeply its data is nested, so the flat elements have to be rebuilt into nested arrays on the way out and flattened again on the way back. That costs a JsonElement per row rather than per tensor, which is the main reason to prefer v2 for anything image-sized.

method : private : BuildTfServingBody() ~ JsonElement

Parameters

Name	Type	Description

DescribeRequest #

The request body this client would send, without sending it. A rejected request is reported by the server in terms of what it received, which is hard to act on without seeing what was sent -- the two protocols shape the same tensors very differently, and a tensor nested to the wrong depth looks identical in Objeck. Printing this is usually faster than reading the error.

method : public : DescribeRequest(inputs:Vector<Tensor>) ~ String

Parameters

Name	Type	Description
inputs	Vector<Tensor>	input tensors

Return

Type	Description
String	the JSON body, or Nil when the inputs are unusable

IsReady #

Whether the server reports the model loaded and ready to serve. Worth calling once at start-up: a model name the server does not know, or one still loading, is otherwise a rejected inference much later.

method : public : IsReady(model:String) ~ Bool

Parameters

Name	Type	Description
model	String	model name

Return

Type	Description
Bool	true when the server says it is ready

JsonToFlat # function

Flattens a response of any nesting depth into one float array, in row-major order -- the same order Nest wrote it out in.

function : JsonToFlat() ~ Float[]

Parameters

Name	Type	Description

Metadata #

What the server says the model expects and returns. Only implemented for KServe v2, whose metadata response is specified. TensorFlow Serving's /metadata returns a signature definition in a different shape; it returns Nil with the last error set rather than guessing at it.

method : public : Metadata(model:String) ~ ModelMetadata

Parameters

Name	Type	Description
model	String	model name

Return

Type	Description
ModelMetadata	metadata, or Nil with EndPoint->GetLastError set

Nest #

Rebuilds nested arrays from flat data, one level per dimension.

method : private : Nest() ~ JsonElement

Parameters

Name	Type	Description

NestedShape #

The shape implied by how deeply a response is nested.

method : private : NestedShape() ~ Int[]

Parameters

Name	Type	Description

New # constructor

A client speaking KServe v2, which reaches the most servers.

New(host:String)

Parameters

Name	Type	Description
host	String	base URL, e.g. "http://localhost:8000"

New # constructor

A client speaking a chosen protocol.

New(host:String, protocol:Client->Protocol)

Parameters

Name	Type	Description
host	String	base URL, e.g. "http://localhost:8501" for TensorFlow Serving
protocol	Client->Protocol	the server's wire protocol

Predict #

Runs a model on one input tensor, for the common single-input case.

method : public : Predict(model:String, input:Tensor) ~ Vector<Tensor>

Parameters

Name	Type	Description
model	String	model name as the server knows it
input	Tensor	the input tensor

Return

Type	Description
Vector<Tensor>	output tensors, or Nil with EndPoint->GetLastError set

Predict #

Runs a model. Each input is checked against its own shape before anything is sent: a count that disagrees with the shape is rejected by the server with a message that names neither the tensor nor the mismatch, so it is cheaper to catch here.

method : public : Predict(model:String, inputs:Vector<Tensor>) ~ Vector<Tensor>

Parameters

Name	Type	Description
model	String	model name as the server knows it
inputs	Vector<Tensor>	input tensors, named as the model expects

Return

Type	Description
Vector<Tensor>	output tensors, or Nil with EndPoint->GetLastError set

ReadDescriptors #

Metadata tensors carry a name, shape and datatype but no data.

method : private : ReadDescriptors() ~ Vector<Tensor>

Parameters

Name	Type	Description

Trim # function

A trailing slash would produce a double slash in every URL.

function : Trim() ~ String

Parameters

Name	Type	Description