💫 Local Bench

Validate your inference. Benchmark LLM performance directly on your local hardware to ensure absolute confidence in your agent’s speed and capabilities.

Overview

Local Bench is a benchmarking and profiling tool for local language models running on your Companion Hub. It provides standardized test suites, hardware utilization metrics, and comparative reports that tell you exactly how fast and capable your setup is — so you can make informed decisions about which models to run for which tasks.

Before deploying a new model to your Companion Agents or Spellbook workflows, run it through Local Bench to understand its latency profile, throughput, memory footprint, and quality on standard tasks.

Key Features

• Token throughput (tok/s) — measures prompt and generation throughput across different context lengths
• Time-to-first-token (TTFT) — latency for agent responsiveness evaluation
• Memory footprint — RAM and VRAM usage per model and quantization level
• CPU/GPU utilization — real-time hardware metrics during inference
• Quality benchmarks — runs MMLU, HellaSwag, and custom task suites to evaluate accuracy
• Side-by-side comparison — compare multiple models across all metrics in a single report
• Export reports — exportable JSON/CSV for custom analysis
• Hardware profile — automatically detects and documents your Hub’s hardware capabilities

Use Cases

• “Which quantization of Llama 3.1 8B runs fastest on my hardware?”
• “Does my new GPU significantly improve TTFT for coding tasks?”
• “Can I run Qwen 2.5 72B acceptably on my Core server?”
• Before deploying a new model to production Spellbook workflows
• Document your Hub’s capabilities for sharing or support purposes

Supported Backends

Benchmark Suites

Setup

Usage

Running a Benchmark

1. Navigate to New Benchmark
2. Select the model(s) to test
3. Choose a suite (Quick, Standard, or Full)
4. Click Start and wait for completion

Results are displayed in the Results tab and saved for future comparison.

Comparing Models

On the Compare page, select two or more previous results to see a side-by-side matrix of all metrics.

Scheduling Automated Benchmarks

Configure recurring benchmarks in Settings → Schedule to track performance regressions after system updates.

CLI

# Run a quick benchmark on llama3.2:3b
ci-bench run --model llama3.2:3b --suite quick
 
# Compare two results
ci-bench compare result-001 result-002
 
# Export results as JSON
ci-bench export result-001 --format json > benchmark.json

Understanding Results

Results will vary significantly based on your specific hardware. The Companion Core server  is benchmarked and documented in the Reference section.

Troubleshooting

Backend connection refused Ensure the inference backend is running. For Ollama, check Hub → Apps → Ollama → Status. Verify the endpoint URL in Local Bench settings.

Benchmark stalls at 0% The model may be loading for the first time. Check Ollama logs in the Hub for download/load progress.

VRAM out of memory errors The selected model is too large for your GPU. Try a smaller quantization (Q4_K_M instead of F16) or a smaller model variant.

Inconsistent results between runs Background processes on your Hub can affect measurements. For reproducible results, pause other inference-heavy apps before benchmarking.

Links

• GitHub Repository 
• Companion Core Hardware 
• Tools Cache Mount