From Base Model Selection to DPO Alignment and QLoRA Fine-Tuning: A Complete LLM Customization Guide

Forum topic · 小凯 · 2026-08-31

Summary

This guide presents a full lifecycle workflow for transforming a raw pretrained base model into an industrial-grade assistant: base model selection under VRAM budget constraints, continual pretraining with 70-85% domain data plus 15-30% general corpora to prevent catastrophic forgetting, supervised fine-tuning (SFT) with loss masking on response tokens only, preference alignment via DPO (which eliminates the reward model and PPO complexity of RLHF), evaluation using MMLU-Pro/GSM8K benchmarks and LLM-as-judge blind tests, and deployment with vLLM's PagedAttention or llama.cpp. It then covers parameter-efficient fine-tuning (PEFT): LoRA's low-rank side-branch updates with zero inference overhead, QLoRA's NF4 quantization, double quantization, and paged optimizer enabling 70B model tuning on a single 24GB GPU, and classic bottleneck adapters. Recommended engineering routes are matched to scenarios such as vertical-domain brains, single-GPU fine-tuning, high-concurrency serving, and offline private deployment. Key references include LoRA (ICLR 2022), QLoRA (NeurIPS 2023), DPO (NeurIPS 2023), and vLLM PagedAttention (SOSP 2023).

Key points

A raw base model is essentially a "wild next-token continuation engine" — knowledgeable but unaligned. Turning it into a useful, safe, low-latency assistant requires six lifecycle stages, followed by parameter-efficient fine-tuning (PEFT) when compute is limited.

Part 1: Full LLM Lifecycle

1. Base Model Selection

Choose by VRAM budget, roughly M_VRAM ≈ 2×Φ GB for FP16 weights plus 0.5×Φ GB KV-cache headroom (Φ = parameter count in billions).

Edge / ≤8GB VRAM: Qwen2.5-1.5B/3B, MiniCPM-3
Enterprise / 16–48GB: Qwen2.5-7B/14B, Llama-3.1-8B
Complex reasoning / agent orchestration: DeepSeek-V3, DeepSeek-R1, Qwen2.5-72B

2. Continual Pretraining & SFT

Continual pretraining data mix: 70–85% domain data + 15–30% general corpora to avoid catastrophic forgetting of general knowledge.
Follow the LIMA principle: ~1,000 curated high-quality multi-turn dialogues beat 100,000 noisy ones for SFT.
Apply loss masking: gradients backpropagate only on assistant response tokens, never on user prompts.

\mathcal{L}_{\text{SFT}}(\theta) = -\sum_{i=1}^{N} \sum_{t \in \text{Response}} \log P_\theta \left( y_t \mid x, y_{<t} \right)

3. Preference Alignment

1. RLHF (PPO): four models in memory simultaneously; fragile and hard to tune. 2. DPO: closed-form solution that removes the reward model and RL entirely — the modern de facto standard. 3. RLAIF / Constitutional AI: replace human labeling with frontier-model scoring, cutting costs by ~99%.

DPO loss, where y_w is the preferred and y_l the rejected response:

\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]

4. Evaluation (three dimensions)

Objective benchmarks: MMLU-Pro (knowledge), GSM8K/MATH (reasoning), SWE-bench (coding)
LLM-as-judge blind tests: Arena-Hard-Auto, MT-Bench
Industrial safety suite: private gold sets for hallucination rate, JSON format compliance, prompt-injection defense

5. Deployment

Cloud, high concurrency: vLLM or SGLang — PagedAttention fragments-free KV-cache management plus continuous batching for multi-fold throughput gains.
Private / edge: llama.cpp or LocalAI — pure C++, GGUF memory mapping, zero Python dependency.

Part 2: Parameter-Efficient Fine-Tuning (PEFT)

Freeze 99%+ of backbone weights and tune only ~0.1% of parameters.

LoRA

Task adaptation updates have very low intrinsic rank. Freeze W_0 and add a low-rank side branch:

W = W_0 + \Delta W = W_0 + \frac{\alpha}{r} (B \cdot A), \quad r \ll \min(d, k)

At deployment, merge weights (W_final = W_0 + (α/r)BA) for zero inference latency overhead.

QLoRA — tuning a 70B model on one consumer GPU

Three pillars: 1. NF4 (NormalFloat4): information-theoretically optimal 4-bit quantization for normally distributed weights 2. Double quantization: quantize the quantization constants too, saving ~0.37 bits/parameter 3. Paged optimizer: spills optimizer state to CPU memory on VRAM spikes, preventing OOM

Forward pass: 4-bit weights → dequantize on the fly to BF16 → matmul → gradients flow to 16-bit LoRA adapters.

Adapter

The PEFT ancestor: serial bottleneck layers (down-projection → up-projection with residual) inserted after Attention/FFN blocks:

\text{Adapter}(h) = \sigma\left( h \cdot W_{\text{down}} \right) \cdot W_{\text{up}} + h

Adds slight serial inference latency, but remains relevant for multi-task routing and hot-swappable plugins.

Decision Matrix

References

1. LoRA: Hu et al., *LoRA: Low-Rank Adaptation of Large Language Models*, ICLR 2022 — arXiv:2106.09685 2. QLoRA: Dettmers et al., *QLoRA: Efficient Finetuning of Quantized LLMs*, NeurIPS 2023 — arXiv:2305.14314 3. DPO: Rafailov et al., *Direct Preference Optimization: Your Language Model is Secretly a Reward Model*, NeurIPS 2023 — arXiv:2305.18290 4. vLLM: Kwon et al., *Efficient Memory Management for Large Language Model Serving with PagedAttention*, SOSP 2023 — arXiv:2309.06180