Hugot Project: Technical Feasibility Assessment of Hardware Accelerators for Go-Based Transformer Inference

Forum topic · ✨步子哥 · 2026-04-20

Summary

This report evaluates the hardware accelerator feasibility of Hugot, a Go library built on ONNX that runs Hugging Face Transformer models for inference and fine-tuning without a Python runtime. Hugot offers three pluggable backends: a pure-Go backend (GoMLX-based, lightweight deployment, no CGO), an ONNX Runtime backend (fastest CPU inference, generative pipeline support, accelerator access via execution providers such as CUDA, TensorRT, DirectML, CoreML, OpenVINO, and NNAPI), and an OpenXLA backend (training/fine-tuning and TPU support). Benchmark data cited in the report shows an NVIDIA L4 GPU delivering roughly 12.6x faster inference than a 192-core AMD EPYC CPU, while first-generation TPUs achieved up to 83x CPU and 29x GPU speedups in Google's internal tests. The report also covers performance tuning (batch size, thread settings, graph optimization, planned quantization) and deployment suitability across cloud, edge, and local server environments, concluding that GPU acceleration via ORT is the most mature path, TPU suits Google Cloud high-throughput workloads, and NPUs are indirectly supported through ONNX Runtime execution providers.

Hugot Project: Hardware Accelerator Technical Feasibility Assessment

*English adaptation of a Chinese technical evaluation report (originally published on zhichai.net). Charts from the source have been summarized in text.*

Project Overview

Hugot is an ONNX-based pipeline library, written in Go, for running inference and training of Transformer models (including LLMs) originally trained in Python with Hugging Face — without depending on a Python runtime or external services. It is maintained by Knights Analytics and used in production for AI-driven data-cleaning tasks.

Hugot supports many pipeline types through a unified API: text classification, text generation, feature extraction (embeddings), image classification, object detection, question answering, tabular models, zero-shot classification, cross-encoders, and named entity recognition. It also supports fine-tuning of feature-extraction pipelines and aims for prediction parity with the Python Hugging Face ecosystem once models are exported to ONNX.

Technical Architecture

Hugot uses pluggable backends and modular pipelines. Three backends are supported:

Pure Go backend (SimpleGo) — default, built on GoMLX; fully Go, no C/C++ dependencies, works with CGO disabled, small container images. Optimized for smaller models and batch sizes around 32; larger workloads should use the C backends.
ONNX Runtime (ORT) backend — enabled with -tags ORT; requires libonnxruntime.so (path set via WithOnnxLibraryPath). Inference only, but the fastest CPU backend and the only one supporting generative pipelines. Exposes ONNX Runtime execution providers (CUDA, TensorRT, DirectML, OpenVINO, CoreML, NNAPI, etc.), e.g., via WithCuda.
OpenXLA backend — enabled with -tags XLA; based on Google's XLA compiler and PJRT runtime. Supports training/fine-tuning and is the only backend with TPU support; generative pipeline support is still incomplete. Requires PJRT plugin libraries and the tokenizers.a static library at build time.

Pipelines share a common abstraction (BasePipeline struct and Pipeline interface in pipelines/pipeline.go); new pipeline types are added by embedding BasePipeline, implementing the interface, and registering in NewPipeline. A Session manages multiple pipelines by alias. All models must be converted to ONNX (e.g., via transformers.onnx or Optimum), giving zero-loss migration from Python.

Hardware Accelerator Support and Performance

Officially tested accelerators: CPU, GPU (CUDA), and TPU; other backends (TensorRT, DirectML, CoreML, OpenVINO, NNAPI) are available indirectly through ONNX Runtime execution providers.

GPU (CUDA)

Mature and high-performance. Requirements: NVIDIA driver + CUDA toolkit matching the ORT GPU build (e.g., ORT 1.24.4 needs CUDA 12.x and cuDNN 9.x), the ORT GPU library, and GoMLX's tokenizers.a. Enabled with the WithCuda option. Benchmarks cited: an NVIDIA L4 GPU was ~12.6x faster than a 192-core AMD EPYC CPU for a satellite-image embedding model, retaining ~11x advantage even at full CPU thread utilization. (Source chart: NVIDIA L4 GPU 12.6x vs. CPU baseline 1x.)

TPU

Supported via the OpenXLA backend with the WithTPU option and PJRT plugins. TPUs excel at large-scale matrix computation; Google's internal figures for first-generation TPUs report 83x inference speedup vs. contemporary CPUs and 29x vs. GPUs. (Source chart: CPU 1x baseline, GPU 29x, TPU 83x.) Best suited to large-batch workloads in Google Cloud environments; deployment complexity and operator coverage are considerations.

NPU and Other Accelerators

NPU support depends on ONNX Runtime execution providers: NNAPI (Android NPUs), CoreML (Apple Neural Engine), OpenVINO (Intel VPUs), etc. Compatibility and gains vary by device; operators may be limited. TensorRT EP adds graph-level optimization and kernel fusion on NVIDIA GPUs; DirectML covers DirectX 12 GPUs on Windows.

Performance Analysis

GPU vs CPU: GPUs offer ~an order of magnitude higher inference throughput; CPUs can be competitive for low-latency single requests.
TPU vs GPU: TPUs win on throughput/energy efficiency at large batch sizes, but have higher deployment friction and a more closed ecosystem.
NPU: best energy efficiency for edge scenarios; peak performance and accuracy depend on quantization/pruning constraints.
Text generation is only supported via the ORT backend (ONNX Runtime Generative AI optimizations); a WithGenerativeEngine option adds concurrent request handling and smart batching.

Tuning Best Practices

Tune batch size: ~32 is a good starting point for the pure Go backend; larger batches can fill GPUs/TPUs but must fit memory.
On CPU, use WithInterOpNumThreads / WithIntraOpNumThreads (e.g., setting threads to 1 to reduce contention) and optionally WithCpuMemArena(false) / WithMemPattern(false).
On GPU/TPU, pin sessions to devices (WithCuda device ID), run multiple sessions across GPUs, and keep data resident in device memory. Hugot memory-maps ONNX models to reduce footprint.
Enable graph optimization via WithGraphOptimizationLevel; 4-bit and other quantization support is planned.

Deployment Suitability

Cloud: Hugot packages with Dockerfiles/compose files; no Python runtime needed. Embedding inference directly in Go services avoids HTTP round-trips and serialization overhead versus calling external Python services. GPU instances for low latency, CPU instances with thread tuning for cost-sensitive workloads.
Edge: Feasible on higher-end edge devices with GPUs or NPUs via ORT execution providers (CUDA or NNAPI); low-power devices depend on model size and quantization.
Local servers: The pure Go backend simplifies air-gapped or CGO-free deployments; C backends unlock more performance where libraries can be installed.

Conclusion

Hugot is technically viable across CPU, GPU, and TPU, with indirect NPU support via ONNX Runtime execution providers. GPU acceleration through the ORT backend is the most mature and highest-impact option today; TPU is attractive for high-throughput Google Cloud deployments via the XLA backend; and NPUs extend Hugot to edge/mobile use cases subject to per-device operator compatibility testing.