Intel OpenVINO Model Optimization

An independent, reproducible model optimization and CPU inference study using PyTorch, Intel OpenVINO, and NNCF with a pretrained ResNet18 model.

The project evaluates FP32 and INT8 inference performance, model footprint, numerical consistency, and classification accuracy under a controlled and reproducible benchmark methodology.

Key Result

On the measured CPU workload, NNCF INT8 post-training quantization reduced the OpenVINO model footprint by 3.94× and produced a 7.46× mean-latency ratio improvement versus OpenVINO FP32, while changing Imagenette Top-1 accuracy by only -0.076 percentage points.

Top-5 accuracy remained effectively stable. Its small positive sample variation is not interpreted as an accuracy improvement.

These results are specific to the measured hardware, model, dataset, workload, and software configuration. They are not universal OpenVINO performance claims or official ImageNet-1K accuracy results.

Architecture

flowchart TD
    A[PyTorch ResNet18 FP32] --> B[OpenVINO FP32 IR]
    B --> C[NNCF Post-Training Quantization]
    C --> D[OpenVINO INT8 IR]
    D --> E[Performance + Accuracy Evaluation]

Loading

Research Questions

• RQ1: What CPU inference difference is observed between native PyTorch FP32 and OpenVINO FP32?
• RQ2: How much does INT8 PTQ reduce footprint and latency?
• RQ3: What accuracy degradation results from quantization?
• RQ4: Which configuration provides the strongest measured deployment trade-off?

The complete answers are generated from source result files in benchmark_summary.md.

Experimental Setup

• torchvision pretrained ResNet18 with ImageNet-1K output space
• Static performance input [1, 3, 224, 224], batch size 1
• CPU-only primary benchmark, 10 warm-ups and 1,000 measured iterations
• Direct PyTorch-to-OpenVINO conversion with FP16 compression disabled
• NNCF 2.19 INT8 PTQ; 30 FakeQuantize operations observed
• 128 real calibration images from Imagenette train
• 3,925 labelled validation images from Imagenette val
• Explicit WNID-to-ImageNet-index mapping
• Exact ResNet18_Weights.DEFAULT.transforms() preprocessing

The performance benchmark uses deterministic synthetic tensors because it measures execution, not semantic quality. The final INT8 quality experiment uses real calibration and validation images with strict split separation.

Dataset

Imagenette is a practical ten-class subset of ImageNet that preserves canonical WordNet synsets. Train supplies calibration only; val supplies accuracy evaluation only. The preparation script checks source-file identity to prevent overlap and creates a deterministic near-balanced calibration subset.

These are Imagenette subset results, not official full ImageNet-1K validation accuracy.

data/
├── imagenet_class_index.json
├── calibration/       # selected from Imagenette train only
└── validation/        # complete Imagenette val

Folder labels are never trusted as alphabetical class indices. Every WNID must resolve unambiguously to the pretrained ImageNet-1K output index.

Model Optimization

1. Load the same FP32 pretrained torchvision model used by the baseline.
2. Convert directly with openvino.convert_model().
3. Force static [1,3,224,224] input and save with compress_to_fp16=False.
4. Build an nncf.Dataset from preprocessed real train images.
5. Apply nncf.quantize() and structurally verify FakeQuantize nodes.
6. Compile FP32 and INT8 explicitly for CPU.

Results

Backend	Precision	Size MB	Mean ms	Median ms	P95 ms	Throughput img/s	Top-1 %	Top-5 %
PyTorch FP32	FP32	46.836	13.787	5.002	6.161	72.53	78.166	95.006
OpenVINO FP32	FP32	46.801	7.994	7.883	9.833	125.09	78.166	95.006
OpenVINO INT8	INT8	11.879	1.071	1.038	1.180	933.31	78.089	95.083

Measured values are preserved in results/benchmark_results.csv and results/accuracy_results.csv; derived calculations and prose are generated from those files.

Derived comparison: INT8 vs OpenVINO FP32	Result
Footprint compression	3.94×
Mean-latency ratio	7.46×
Top-1 delta	-0.076 percentage points

Accuracy Preservation

OpenVINO FP32 matched PyTorch FP32 on this subset. INT8 changed Top-1 from 78.166% to 78.089%, a -0.076-percentage-point delta. Top-5 remained effectively stable. Because the validation subset is finite, the small observed positive Top-5 variation is not interpreted as quantization improving accuracy.

Numerical Consistency

• PyTorch FP32 and OpenVINO FP32 produced matching output shapes and the same representative Top-1 class. Maximum absolute logit difference was 2.289e-05; mean absolute difference was 2.454e-06.
• INT8 retained matching output shape and representative Top-1, with the larger expected logit deviation (maximum 1.721, mean 0.155). Dataset-level quality remained effectively preserved.

PyTorch Latency Distribution

PyTorch showed outlier-sensitive timing: mean latency was 13.787 ms, while median was 5.002 ms and p95 was 6.161 ms. The arithmetic-mean comparison yields a 1.72× ratio in favor of OpenVINO FP32, but typical stable latency behavior does not support a simplistic claim that OpenVINO FP32 is universally faster. Mean, median, and p95 must be read together.

OpenVINO benchmark_app separately confirmed that latency mode and multi-request/multi-stream throughput mode answer different questions. Those results are not substituted for the synchronous custom measurements above.

Reproduce the Experiment

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pytest -q

python scripts/run_baseline.py --iterations 1000
python scripts/convert_openvino.py

python scripts/prepare_imagenette.py \
  --source-dir /path/to/imagenette2 \
  --calibration-samples 128

python scripts/quantize_int8.py \
  --calibration-data-dir data/calibration \
  --class-index data/imagenet_class_index.json \
  --calibration-samples 128

python scripts/run_benchmark.py --backend openvino-fp32 --iterations 1000
python scripts/run_benchmark.py --backend openvino-int8 --iterations 1000

python scripts/evaluate_accuracy.py \
  --data-dir data/validation \
  --class-index data/imagenet_class_index.json \
  --batch-size 32

python scripts/capture_environment.py
python scripts/generate_report.py

Project Structure

configs/                 Benchmark configuration
scripts/                 Conversion, PTQ, benchmark, evaluation, reporting
src/                     Shared data, metrics, optimization and report logic
tests/                   Self-contained unit tests
artifacts/               Regenerable OpenVINO IR outputs (ignored)
data/                    Mapping retained; raw/calibration/validation ignored
results/                 Measured data, reports, environment and charts

Methodology

• Same model, ordered deterministic inputs, batch size, warm-ups, iterations, timer, CPU device, throughput formula, and RSS method across custom performance backends
• Combined XML+BIN size for OpenVINO IR
• Accuracy evaluated on identical preprocessed float32 NCHW tensors
• No OpenVINO-side resize, normalization, layout conversion, or uint8 conversion
• Calibration and validation strictly separated
• Measured source files retained without destructive rounding
• Report calculations fail on missing values rather than fabricate fallbacks

Environment

Machine and package information is captured programmatically in environment.json. The recorded experiment used an Intel Core Ultra 9 275HX, Linux, Python 3.12.3, PyTorch 2.13.0, OpenVINO 2026.3, and NNCF 2.19.0. No usernames or absolute home paths are included here.

Limitations

• Imagenette is a ten-class subset, not full ImageNet-1K.
• Results are specific to this CPU, model, workload, software stack, and threading behavior.
• PyTorch benchmark latency contained material outliers.
• Synthetic inputs were used for performance development; final quality used real train-only calibration.
• NNCF emitted a compatibility warning because installed PyTorch is newer than its recommended version.
• Accuracy values must not be generalized to full ImageNet.
• RSS delta is not precise peak model memory.
• No claim is made that these results generalize to every model or CPU.

Future Work

• Full ImageNet-1K validation
• Additional model architectures
• INT4 and weight compression
• Structured pruning and sparsity
• OpenVINO GPU and supported NPU evaluation
• Batch-size scaling and asynchronous inference
• Power and energy measurement
• Controlled CPU affinity and threading experiments

Technologies

Python · PyTorch · torchvision · OpenVINO · NNCF · NumPy · pandas · matplotlib · pytest