Phi-2 (2.7B) - CPU-Optimized Streaming Code Generator

CPU optimizations enabled:

Dynamic INT8 quantization (~2x faster)
KV cache enabled
Low memory usage mode
Optimized generation parameters

For even better CPU performance, consider:

Using a smaller model (Phi-1.5, TinyLlama)
ONNX Runtime conversion
Intel OpenVINO toolkit

Running on: CPU (Quantized)

Code Prompt

Max New Tokens

1 500

Use Sampling (slower but more creative)

Temperature (only if sampling enabled)

0.1 1.5

Streaming Output

CPU Speed Tips:

Fastest: Disable sampling, use 20-30 tokens
Balanced: Disable sampling, use 30-50 tokens
For better speed: Consider switching to Phi-1.5 (1.3B) or TinyLlama (1.1B)

Current optimizations:

✓ INT8 dynamic quantization (~2x speedup)
✓ KV caching enabled
✓ Greedy decoding by default