Getting Started with ONNX: Train and Deploy Custom Models
A practical, end-to-end guide to ONNX: what it is, how to export models from PyTorch and TensorFlow, run fast inference with ONNX Runtime, and ship to production.
inference
8 guides tagged “inference”.
Production ML Workflows: How We Serve an ONNX Model with FastAPI
A real, honest production architecture: an ONNX image classifier served by FastAPI on Railway, loaded from object storage at startup, with one shared inference session on CPU — and what we'd improve.
Server-Side vs On-Device ML Inference: How to Choose
The core trade-off in ML deployment: run inference on a central server or push it to the device. Real case: why TrichAi chose the server, what it cost, and when you'd choose differently.
Quantizing ONNX Models: When It's Actually Worth It
A practical guide to INT8 and FP16 quantization for ONNX models — how much you save, what you risk, and a real decision: should an unquantized production model be quantized?
The Silent Accuracy Killer: Preprocessing Mismatch
Your model scores 94% in the notebook and falls apart in production. The cause is usually not the model — it's a preprocessing mismatch between training and inference. Here's how to find and prevent it.
Choosing an ONNX Runtime Execution Provider: CPU, CUDA, TensorRT, CoreML
ONNX Runtime can dispatch the same model to very different hardware backends. A practical guide to execution providers — what each is for, how the fallback chain works, and how to choose.
Choosing a Model Format: ONNX vs TorchScript vs SavedModel
Once a model is trained, how you serialize it shapes everything downstream. A practical comparison of ONNX, TorchScript, and TensorFlow SavedModel — portability, performance, and lock-in.
Batching Inference Requests: Throughput vs Latency
Processing requests one at a time wastes hardware; batching them trades a little latency for a lot of throughput. How dynamic batching works, when it helps, and when a single shared session is enough.