VMware Private AI Foundation with NVIDIA is the enterprise platform for running generative AI workloads on your own infrastructure. But what about your local development environment? What if you want to experiment with models, test prompts, or build familiarity with private AI concepts before deploying the full stack?
That is exactly where llama.cpp comes in.
What is llama.cpp?
llama.cpp is an open-source inference engine written in C++ by Georgi Gerganov. It was originally built to run Meta's LLaMA models locally, but has since grown into one of the most versatile and widely used inference runtimes available. It supports dozens of model families in GGUF format, runs on CPU, NVIDIA CUDA, AMD ROCm, and Apple Metal — and it is fast.
On Apple Silicon, llama.cpp is particularly well-suited. The unified memory architecture of the M-series chips means that the CPU and GPU share the same memory pool. On a MacBook Pro M2 Max with 64 GB of unified memory, you can comfortably run quantized 70B parameter models locally — something that would require a high-end discrete GPU on any other platform.
Why This Matters for Private AI
The core idea behind VMware Private AI Foundation with NVIDIA is that your data stays inside your own infrastructure. No public cloud, no external API calls, no data leaving your environment. llama.cpp brings that same principle to your local machine.
This makes it an ideal companion for anyone working with Private AI Foundation:
- Development and prototyping — test prompts, RAG pipelines, and agent logic locally before deploying to VCF
- Disconnected environments — llama.cpp works fully offline, with no dependency on external model registries or APIs
- Understanding model behaviour — build intuition for how models respond before you integrate them into production workloads via Private AI Services
- Cost-efficient experimentation — no GPU cluster required for development work
Setting Up llama.cpp on Apple Silicon
Install via Homebrew
The easiest way to get started. Homebrew automatically compiles llama.cpp with Metal GPU acceleration enabled for Apple Silicon.
brew install llama.cpp
Or Build from Source
If you want the latest features or full control over the build:
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release -j$(sysctl -n hw.logicalcpu)
Downloading a Model
llama.cpp uses the GGUF model format. Models in this format are available on Hugging Face. Install the Hugging Face CLI first:
pip install huggingface-hub
Then download a model. A good starting point:
# Lightweight and fast — good for quick tests
huggingface-cli download bartowski/Llama-3.2-3B-Instruct-GGUF \
--include "*.Q8_0.gguf" \
--local-dir ~/models
# Serious model — fits comfortably in 64 GB unified memory
huggingface-cli download bartowski/Meta-Llama-3.1-70B-Instruct-GGUF \
--include "*Q4_K_M*" \
--local-dir ~/models
The Q4_K_M suffix refers to the quantization level — a good balance between model quality and memory footprint.
Starting the Inference Server
llama.cpp includes a built-in server with an OpenAI-compatible REST API. This means any tool that works with the OpenAI API will work with your local llama.cpp instance — no code changes needed.
llama-server \
--model ~/models/Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf \
--host 0.0.0.0 \
--port 8080 \
--n-gpu-layers 99 \
--ctx-size 8192
The --n-gpu-layers 99 flag offloads all layers to the Metal GPU. On an M2 Max this makes a significant difference in inference speed compared to CPU-only mode.
Testing the API
Once the server is running, test it with a simple curl request:
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "llama",
"messages": [
{"role": "user", "content": "Explain VMware Private AI Foundation in one paragraph."}
]
}'
You now have a fully local, OpenAI-compatible inference endpoint running on your own hardware.
Adding Open WebUI
For a proper chat interface, Open WebUI is the go-to option. It runs in Docker and connects directly to your llama.cpp server.
docker run -d \
--add-host=host.docker.internal:host-gateway \
-p 3000:3000 \
ghcr.io/open-webui/open-webui:main
Open your browser at http://localhost:3000, add your llama.cpp endpoint (http://host.docker.internal:8080) as a custom OpenAI-compatible connection, and you have a full ChatGPT-like interface running entirely on your local machine — no internet connection required.
The Bigger Picture
llama.cpp and Open WebUI give you a lightweight, fully local private AI stack that you can run anywhere — on your MacBook, in a lab VM, or even in an air-gapped environment. It is not a replacement for VMware Private AI Foundation with NVIDIA in production, but it is an excellent way to build familiarity with models, experiment with RAG pipelines, and develop the intuition you need to design production-grade private AI platforms on VCF 9.
For anyone working on VMware Private AI Foundation deployments, having this kind of local stack running is simply good practice.
