How to Deploy a 27B LLM Locally on a Single or Dual RTX 3090

A Practical Guide: Running a 27B Large Language Model on an RTX 3090

Have you ever wondered why the expensive RTX 3090 you bought is only used for Stable Diffusion or mining? 24GB of VRAM is sitting right there, and not running a large model on it feels like a waste. But once you try it yourself, you'll quickly realize that local LLM deployment is full of pitfalls: slow model downloads, out-of-memory (OOM) crashes due to VRAM limits, choosing between vLLM and llama.cpp, Docker image pull failures, and cryptic curl errors after startup... You spend half a day fiddling without even seeing the model load.

This guide solves exactly that. I will take you from zero to a fully functional local LLM service with an OpenAI-compatible API on a Linux machine equipped with a single or dual RTX 3090, using the ready-made configuration from the club-3090 project. By the end, you'll be able to run models like Qwen3.6-27B on your own GPU and know how to choose between vLLM (high throughput) and llama.cpp (large context).

I. Prerequisites

Before we begin, ensure your environment meets the following requirements. These are not optional; missing any will lead to major roadblocks.

Note on NVIDIA Driver Version: This is a frequent stumbling block. If nvidia-smi shows a driver version below 580, the vLLM container may fail to start. Update your driver before proceeding.

Once verified, let's get to work.

II. Clone the Repository & Install Dependencies

## 1. Clone the project
git clone https://github.com/noonghunna/club-3090.git
cd club-3090

## 2. Install PyYAML (required for pre-flight checks)
## Usually pre-installed on Ubuntu/Debian. If not:
python3 -m pip install pyyaml
## Or via apt:
sudo apt install python3-yaml

After cloning, the core tools reside in the scripts/ directory. You'll find setup.sh (download models), launch.sh (start services), switch.sh (switch configurations), etc. The reason this project is so user-friendly is that it packages all the tasks that normally take hours of documentation reading into interactive wizard scripts. You don't need to manually craft Docker Compose files; the scripts automatically select the optimal configuration based on your hardware.

III. Download the Model

## Interactive download (script prompts for model and storage path)
bash scripts/setup.sh

## Or specify the model name directly for a one-step process:
bash scripts/setup.sh qwen3.6-27b

This script handles three things:

1. Hardware Pre-flight Check – Verifies GPU count, driver version, and available disk space.
2. Model Weights Download – Pulls pre-quantized models from HuggingFace by default (AutoRound INT4 version, retaining the BF16 MTP head. Tested for excellent performance and controlled VRAM usage).
3. SHA Verification – Automatically verifies file integrity after download to prevent corrupted files.

On Storage Location: The script will ask for a storage path. If your default disk lacks space, you can specify a mounted external drive. The MODEL_DIR environment variable can bypass interactive prompts.

Download time depends on your network. The INT4 quantized version of a 27B model is about 15-20GB. Using a proxy or a mirror site is highly recommended.

IV. Choose an Inference Engine & Start the Service

Once the model is downloaded, it's time to start the service. This is the most critical decision point in the entire process: which inference engine should you choose?

club-3090 supports three engines, each with a specific use case:

• vLLM (Recommended for Dual GPUs) – High throughput, supports vision, tool calling, and MTP (Multi-Token Prediction). Dual-GPU vLLM can reach 127 TPS in coding scenarios.
• llama.cpp (Recommended for Single GPU) – Ultimate stability. A single GPU can handle 200K context without OOM during long-context prefill. You trade a bit of speed for rock-solid reliability, ideal for Agent workflows.
• ik_llama (Fastest for Single GPU) – Based on advanced GGUF quantization (IQ4_KS), offering the highest single-GPU speed at approximately 69 TPS.

My Recommendation: Start with llama.cpp or ik_llama for a single-GPU experience. If you need production-grade throughput, go for dual-GPU vLLM.

Starting it is straightforward, just one command:

bash scripts/launch.sh

The script interactively asks: model → GPU count → VRAM budget, then starts automatically. To skip the wizard and specify configuration directly:

## Single-GPU vLLM (Light mode, 32K context, ~32-33 TPS)
bash scripts/launch.sh --variant vllm/minimal

## Dual-GPU vLLM (262K context + vision support)
bash scripts/launch.sh --variant vllm/dual

## Single-GPU llama.cpp (Stable route, large context without OOM)
bash scripts/switch.sh llamacpp/default

After startup, the script automatically runs verify-full.sh for a health check (8 checks, taking ~1-2 minutes). This ensures the service is healthy before accepting requests, which is much more reliable than blindly running curl and hoping for the best like many tutorials suggest.

V. In Practice: Making Your First API Call

Once running, the service exposes an OpenAI-compatible API at localhost:8020. This means any code using the OpenAI SDK can switch to it seamlessly.

## Quick verification with curl
curl -sf http://localhost:8020/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.6-27b",
    "messages": [
      {"role": "user", "content": "Explain the core idea of the Transformer architecture in simple terms."}
    ],
    "max_tokens": 500
  }'

If you're using Python, the syntax is identical to calling OpenAI; just change the base_url:

from openai import OpenAI

## Point to the local service
client = OpenAI(
    api_key="local-not-needed",  # No API key needed for local instance
    base_url="http://localhost:8020/v1"
)

response = client.chat.completions.create(
    model="qwen3.6-27b",
    messages=[
        {"role": "system", "content": "You are a senior tech blogger"},
        {"role": "user", "content": "Explain the value of Docker to a beginner backend developer in one paragraph."}
    ],
    max_tokens=300,
    temperature=0.7
)

print(response.choices[0].message.content)

Why does this matter? Your existing code barely needs changes. Just swap base_url from api.openai.com to localhost:8020/v1 to switch to a local model effortlessly. This is incredibly useful for feature integration during development without burning cash on every API request.

VI. Performance Benchmarking

After getting the service running, you're likely wondering: how does it actually perform? The project includes ready-made benchmarking scripts:

## Throughput test (3 warm-up runs + 5 official measurements)
bash scripts/bench.sh

## Behavioral quality test (tool calling accuracy, instruction following, structured output, etc.)
bash scripts/quality-test.sh --medium   # 5 test suites, ~15-25 mins

bench.sh outputs TPS (tokens per second), allowing you to evaluate throughput on your hardware. Based on the project's baseline data, Qwen3.6-27B can reach 89-127 TPS on dual-GPU vLLM (coding scenarios) and 51-60 TPS on single-GPU llama.cpp.

VII. Common Issues & Pitfalls

1. Single-GPU vLLM Long-Context OOM (Cliff Issue)

This is a known limitation. Single 24GB GPU vLLM will OOM during long prompt prefill exceeding 50K tokens. Solution: Switch to dual-GPU vLLM (vllm/dual, TP=2 bypasses this), or revert to llama.cpp on a single GPU.

2. Slow or Failed HuggingFace Downloads

The script defaults to HF. If network access is poor, manually download model weights to MODEL_DIR first, specify the existing path when running setup.sh, or set environment variables to skip the download.

3. launch.sh Fails After Model Download

Check logs first:

docker logs <container_name> 2>&1 | tail -50

The most common cause is insufficient VRAM or a mismatched driver version. If the pre-flight script misjudges your setup, you can bypass it with --force:

bash scripts/switch.sh --force llamacpp/default

4. Windows Users

Native Windows cannot run this project's Docker setup. Install WSL2 first, follow the docs/WSL_SETUP.md guide, and complete the entire workflow inside WSL2.

VIII. What's Next?

With the service up and running, you can expand based on your needs:

1. Switch Configurations On-the-Fly: Use bash scripts/switch.sh vllm/dual to switch engines and parameters instantly without stopping or reinstalling.
2. Add More Models: Currently supports Qwen3.6-27B, Gemma 4 31B, Qwen3.6 35B-A3B (MoE), etc. Just run bash scripts/setup.sh <model_name>.
3. Connect Open WebUI: For a web-based chat interface, pair it with Open WebUI pointing the backend to localhost:8020.
4. Stay Updated: The project updates frequently (engine versions, new compose variants, patches): bash scripts/update.sh.
5. Run the Terminal UI: If you prefer a visual interface, try the project's built-in c3 cockpit—a terminal UI similar to lazydocker:
   bash 复制代码

   uv pip install -e tools/serve-cockpit
   c3

Running a 27B model on a single 3090 isn't a new concept, but having all quantization schemes, engine configurations, context lengths, and pre-training issues validated and packaged into ready-to-use Docker Compose files and one-click scripts is true engineering. I hope this guide saves you from pitfalls so you can spend your time creating value with models rather than fighting OOM errors.