Client for HTTP model-inference servers. Sends tensors to a running server and reads tensors back, so a model trained in TensorFlow, PyTorch, ONNX or scikit-learn can be used from Objeck without a native binding for each runtime. Two wire protocols, because no single one covers the field: - OPEN_INFERENCE is the KServe v2 protocol, spoken by NVIDIA Triton, KServe, OpenVINO Model Server, Seldon MLServer and others. Triton alone serves TensorFlow, ONNX, PyTorch, TensorRT and OpenVINO behind it, so this one protocol reaches most servers and most runtimes. It is the default. - TF_SERVING is TensorFlow Serving's own REST API, which does not implement KServe v2 and so needs its own request shape. The difference is confined to Client: a Tensor is the same object either way. Choosing the protocol rather than the server is the whole point -- a client written against one server's API reaches one server. This is a client, not a runtime. The server is a separate process the user runs; nothing here loads a model. For in-process inference without a server, API.Onnx runs models directly. Compile with -lib inference.
HttpEngine
A model server as an Engine. The remote counterpart of API.Onnx.OnnxEngine: the same interface, a model reached over HTTP rather than loaded in-process. Swapping one for the other changes nothing else in calling code, which is the point -- the backend is a deployment decision. A server is the only GPU path Objeck has today, and the only route for a model that will not convert to ONNX. IsOpen() asks the server whether the model is ready, so unlike an in-process engine it is a network call and can change between calls. It is not cached: a server that was ready a minute ago is not evidence about now.
Example
engine := HttpEngine->New("http://gpu-box:8000", "iris");
outputs := engine->Run(inputs);
if(outputs = Nil) {
EndPoint->GetLastError()->PrintLine();
};Operations
Close #
Nothing to release: there is no handle, only a URL. Present because the interface has it, so calling code can close any engine without knowing which kind it holds.
method : public : Close() ~ NilGetClient #
The underlying client, for what the Engine interface does not carry -- Metadata and DescribeRequest, neither of which an in-process engine has.
method : public : GetClient() ~ ClientReturn
| Type | Description |
|---|---|
| Client | the client |
IsOpen #
Whether the server reports the model ready. A network call, not a cached flag: unlike an in-process engine this can change between calls, and a server that was ready a minute ago is not evidence about now.
method : public : IsOpen() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when the server says the model is ready |
New # constructor
Constructor, using the Open Inference protocol (Triton, KServe).
New(host:String, model:String)Parameters
| Name | Type | Description |
|---|---|---|
| host | String | server base URL, e.g. http://gpu-box:8000 |
| model | String | the model name the server serves it under |