Client for HTTP model-inference servers. Sends tensors to a running server and reads tensors back, so a model trained in TensorFlow, PyTorch, ONNX or scikit-learn can be used from Objeck without a native binding for each runtime.
Two wire protocols, because no single one covers the field:
- OPEN_INFERENCE is the KServe v2 protocol, spoken by NVIDIA Triton, KServe, OpenVINO Model Server, Seldon MLServer and others. Triton alone serves TensorFlow, ONNX, PyTorch, TensorRT and OpenVINO behind it, so this one protocol reaches most servers and most runtimes. It is the default. - TF_SERVING is TensorFlow Serving's own REST API, which does not implement KServe v2 and so needs its own request shape.
The difference is confined to Client: a Tensor is the same object either way. Choosing the protocol rather than the server is the whole point -- a client written against one server's API reaches one server.
This is a client, not a runtime. The server is a separate process the user runs; nothing here loads a model. For in-process inference without a server, API.Onnx runs models directly. Compile with -lib inference.
Client->Protocol
Which wire protocol the server speaks.
Items
- OPEN_INFERENCE — KServe v2, the Open Inference Protocol. Triton, KServe, OpenVINO Model Server and Seldon MLServer. Flat data with an explicit shape and datatype in both directions, which is why it is the default.
- TF_SERVING — TensorFlow Serving's own REST API. It does not implement KServe v2, so it needs its own request shape: the tensor is nested to express its shape rather than carrying one, and the datatype is inferred.