llama-factoryfine-tuningpeftdposftdeepspeedtutorial

LLaMA-Factory Guide: Unified LLM Fine-Tuning Architecture

October 3, 2026

LLaMA-Factory Architecture: Unified Model and Dataset Adapters, PEFT Kernels, DeepSpeed ZeRO-3, and Multi-Stage Alignment Pipeline

The Problem: The Fractured and Fragile Post-Training Toolchain

In modern enterprise artificial intelligence, automated data extraction systems, and autonomous agent platforms like AiFetchly, foundation models are rarely deployed directly out of pre-training. A raw pre-trained base model simply acts as a next-token predictor over massive web corpora; it lacks the specialized instruction-following capabilities, strict schema compliance, and domain grounding required to parse complex documents or execute deterministic tool calls.

Transforming raw base checkpoints (such as LLaMA 3.3, Qwen 2.5, DeepSeek, Mistral, or Gemma 2) into high-precision production engines requires post-training: Supervised Fine-Tuning (SFT), preference alignment (DPO, ORPO, KTO), and reinforcement learning.

However, until recently, engineering teams attempting to fine-tune open-weights models faced an operational minefield characterized by tokenization drift, fragmented PEFT libraries, and brittle distributed configurations:

The Traditional LLM Fine-Tuning Fragmentation Tax:
┌─────────────────────────────────────────────────────────────────┐
│ Bottleneck 1: Tokenizer & Prompt Template Drift                 │
│ • ChatML, Alpaca, Llama-3, and ShareGPT use clashing templates  │
│ ❌ Manually rewriting regex and special token masks per model   │
│ ❌ Prompt loss leakage: Naive collators train on user prompts!  │
├─────────────────────────────────────────────────────────────────┤
│ Bottleneck 2: Fragmented PEFT & Kernel Ecosystem                │
│ • Separate, uncoordinated codebases for LoRA, QLoRA, and DoRA   │
│ ❌ Un-fused PyTorch loss spikes VRAM by 8GB+ on large vocabularies│
│ ❌ Incompatible CUDA kernels fail across diverse GPU hardware   │
├─────────────────────────────────────────────────────────────────┤
│ Bottleneck 3: Multi-Stage Alignment Pipeline Fractures          │
│ • SFT, Reward Modeling, and DPO require distinct frameworks    │
│ ❌ Writing custom data conversion scripts between training steps │
│ ❌ Model format mismatches between training and serving engines  │
├─────────────────────────────────────────────────────────────────┤
│ Bottleneck 4: Distributed Infrastructure Complexity             │
│ • DeepSpeed ZeRO-2/3, FSDP, and multi-node MPI orchestration    │
│ ❌ Hundreds of lines of boilerplate launch scripts to debug     │
└─────────────────────────────────────────────────────────────────┘
Result: Machine learning engineers spend 70% of their time writing
plumbing code rather than optimizing model performance!

1. Tokenizer and Prompt Template Inconsistencies

Every major AI research lab formats conversation history differently:

Meta LLaMA 3 / 3.3: Uses explicit header tokens (<|start_header_id|>user<|end_header_id|>\n\n...<|eot_id|>).
Alibaba Qwen 2.5: Employs the ChatML convention (<|im_start|>user\n...<|im_end|>).
Google Gemma 2: Structures conversations around <start_of_turn>user\n...<end_of_turn>.
Mistral / Mixtral: Uses [INST] ... [/INST] tags without standard role delimiters.

When engineering teams train different base models or migrate from one architecture to another, custom data-loading scripts frequently suffer from template corruption:

A missing end-of-turn token causes the model to generate runaway conversational loops during inference.
Naive collators fail to apply cross-entropy loss masks to the prompt tokens, forcing the model to memorize the user’s questions rather than learning how to generate the assistant’s structured response.

2. The Memory Wall and Un-Fused Operator Spikes

Modern foundation models feature massive vocabularies to support multilingual understanding and code intelligence:

LLaMA 3.3: $V = 128,256$ tokens
Qwen 2.5: $V = 152,064$ tokens
DeepSeek-V3: $V = 129,280$ tokens

In standard PyTorch training, calculating cross-entropy loss over a sequence length of 4,096 tokens with a large vocabulary requires materializing the full logit tensor in GPU memory:

$$\text{Logit Memory} = \text{Batch Size} \times \text{Sequence Length} \times V \times 2 \text{ bytes}$$

For a modest batch of 16,384 tokens, allocating and backpropagating through un-fused logit matrices consumes over 8.4 GB of High-Bandwidth Memory (HBM) purely for the loss calculation. Without fused Triton kernels, training runs trigger sudden Out-Of-Memory (OOM) crashes even when model weights and optimizer states are compressed.

3. The Multi-Stage Alignment Disconnect

Production alignment is an iterative lifecycle:

Domain Adaptation / Pre-training: Ingesting domain-specific documentation and code repositories.
Supervised Fine-Tuning (SFT): Training on curated input-output demonstrations and tool-calling trajectories.
Preference Alignment (DPO / ORPO / KTO): Aligning responses to human preferences using pair-wise chosen and rejected datasets.

Traditionally, software engineers had to stitch together different libraries for each phase: writing custom scripts for SFT, exporting checkpoints to a second library for Direct Preference Optimization (DPO), and using a third toolchain to merge LoRA weights for inference serving.

LLaMA-Factory Architecture: The Unified Post-Training Engine

To resolve the tooling fragmentation and engineering overhead of LLM adaptation, LLaMA-Factory (created by Yaowei Zheng and the open-source community) provides a unified, highly optimized post-training framework.

Rather than forcing developers to build custom PyTorch pipelines for each model family, LLaMA-Factory introduces a modular architecture that abstracts tokenization, parameter-efficient adaptation, operator acceleration, and distributed execution into a cohesive systems stack:

LLaMA-Factory Unified System Architecture:
┌─────────────────────────────────────────────────────────────────┐
│ 1. Data Layer: Unified Dataset & Prompt Template Adapter        │
│ • Normalizes Alpaca, ShareGPT, and Open-R1 formats              │
│ • Dynamic loss masking (trains only on assistant completions)   │
│ • Multi-sequence packing (eliminates padding token compute waste)│
└─────────────────────────────────┬───────────────────────────────┘
                                  │ Tokenized Batch Stream
                                  ▼
┌─────────────────────────────────────────────────────────────────┐
│ 2. Core Execution Engine: Modular Post-Training Lifecycle       │
│ • Pre-training: Causal language modeling on raw domain text     │
│ • SFT: Supervised fine-tuning on structured instruction pairs   │
│ • Reward Modeling: Pairwise ranking loss computation            │
│ • Direct Alignment: DPO, ORPO, KTO, SimPO, and BPO             │
│ • Reinforcement Learning: PPO and GRPO (vLLM-accelerated)      │
└─────────────────────────────────┬───────────────────────────────┘
                                  │ PEFT & Kernel Optimizations
                                  ▼
┌─────────────────────────────────────────────────────────────────┐
│ 3. Acceleration & Memory Infrastructure Layer                   │
│ • PEFT Suite: LoRA, QLoRA (NF4), DoRA, GaLore, PiSSA, LongLoRA  │
│ • Fused Operators: Liger Kernel (Triton RoPE, RMSNorm, CE Loss) │
│ • Attention Kernels: FlashAttention-2 and SDPA                  │
│ • Distributed Frameworks: DeepSpeed ZeRO-2/3 and PyTorch FSDP   │
└─────────────────────────────────┬───────────────────────────────┘
                                  │ Export & Deployment
                                  ▼
┌─────────────────────────────────────────────────────────────────┐
│ 4. Production Serving & Quantization Engine                     │
│ • Zero-downtime LoRA weight merging into base checkpoints       │
│ • Multi-format quantization: AWQ, GPTQ, and GGUF (llama.cpp)    │
│ • Native high-throughput inference serving via built-in vLLM    │
└─────────────────────────────────────────────────────────────────┘

1. The Unified Dataset and Prompt Template Adapter

The foundation of LLaMA-Factory is its Model and Data Abstraction Layer.

Instead of requiring bespoke data collators for each architecture, LLaMA-Factory decouples raw dataset formats from model-specific tokenizers through a centralized configuration registry (dataset_info.json):

Format Agnostic: Natively ingests datasets in standard formats (Alpaca single-turn dictionaries, ShareGPT multi-turn conversational trees, or DPO chosen/rejected pairs).
Automated Template Resolution: When a user specifies --template llama3 or --template qwen, LLaMA-Factory automatically binds the appropriate system prompts, user tags, assistant prefixes, and end-of-turn tokens without manual string concatenation.
Strict Loss Masking: The collator automatically sets labels = -100 for all prompt tokens, ensuring gradients are computed strictly on assistant generation tokens.
Sequence Packing: Combines multiple short conversation trajectories into a single maximum-length context block (e.g., 4,096 or 8,192 tokens) using attention block-diagonal masks, eliminating up to 60% of GPU compute wasted on zero-padding.

2. Comprehensive Parameter-Efficient Fine-Tuning (PEFT) Suite

LLaMA-Factory supports the entire spectrum of modern parameter-efficient adaptation techniques:

Parameter-Efficient Adaptation Spectrum in LLaMA-Factory:
1. Full Fine-Tuning (16-Bit)
   • Updates all weights W_0.
   • High VRAM; requires multi-GPU DeepSpeed ZeRO-3 for 70B models.
2. Freeze Tuning
   • Freezes base layers; trains only top K layers or embedding heads.
3. LoRA (Low-Rank Adaptation)
   • W = W_0 + (alpha / r) * B * A
   • Freezes base weights; trains low-rank adapter matrices (r = 8 to 64).
4. QLoRA (Quantized LoRA)
   • Base weights quantized to 4-bit NormalFloat (NF4) with Double Quantization.
   • Cuts base model VRAM by 75%; fits a 70B model onto a single 48GB GPU.
5. DoRA (Weight-Decomposed LoRA)
   • Decomposes weights into directional adapters and magnitude vectors.
   • Matches full fine-tuning learning capacity with low-rank parameter counts.
6. GaLore (Gradient Low-Rank Projection)
   • Allows full parameter learning while projecting optimizer states into low-rank subspaces.

By unifying these methods under a single configuration flag (--finetuning_type lora | qlora | dora | galore | full), engineering teams can benchmark adaptation paradigms on identical datasets without altering training code.

3. Fused Operator Acceleration via Liger Kernels

To overcome the cross-entropy logit memory bottleneck, LLaMA-Factory integrates LinkedIn Liger Kernels—a suite of custom OpenAI Triton kernels designed specifically for transformer training:

Fused Linear Cross-Entropy Loss: Computes the linear projection and cross-entropy loss in chunks within GPU SRAM without materializing the complete $[B, L, V]$ logit tensor in high-bandwidth memory.
Fused RMSNorm and RoPE: Performs root-mean-square normalization and rotary positional embeddings in-place, eliminating redundant memory roundtrips.
SwiGLU Activation Fusion: Fuses intermediate gating projections in feed-forward networks.

Enabling Liger Kernels (--enable_liger_kernel true) reduces peak training VRAM consumption by 35% to 45% and increases token-per-second training throughput by 2.2x to 2.8x.

4. Direct Multi-Stage Alignment (SFT, DPO, ORPO, KTO)

LLaMA-Factory bridges the historical chasm between instruction tuning and preference alignment:

Supervised Fine-Tuning (stage: sft): Standard next-token instruction learning with sequence packing.
Direct Preference Optimization (stage: dpo): Optimizes the implicit policy reward directly against chosen and rejected completions:

$$\mathcal{L}{\text{DPO}}(\pi\theta; \pi_{\text{ref}}) = -\mathbb{E}{(x, y_w, y_l)} \left[ \log \sigma \left( \beta \log \frac{\pi\theta(y_w | x)}{\pi_{\text{ref}}(y_w | x)} - \beta \log \frac{\pi_\theta(y_l | x)}{\pi_{\text{ref}}(y_l | x)} \right) \right]$$

Odds Ratio Preference Optimization (stage: orpo): Combines SFT cross-entropy loss and odds-ratio alignment into a single monolithic training step, eliminating the need for a separate reference model in GPU memory.
Kahneman-Tversky Optimization (stage: kto): Enables preference alignment using unpaired binary feedback (thumbs up / thumbs down) rather than expensive paired comparisons.

Hands-On Implementation: Fine-Tuning and Aligning Qwen 2.5 with LLaMA-Factory

To demonstrate the simplicity and power of LLaMA-Factory, let us walk through a complete enterprise workflow: fine-tuning an open-weights model (Qwen-2.5-7B-Instruct) on domain data using QLoRA, accelerating the run with Liger Kernels and FlashAttention-2, and exporting the merged checkpoint for high-throughput serving with vLLM.

Step 1: Installation and Environment Setup

Clone the repository and install LLaMA-Factory with all acceleration dependencies:

git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics,deepspeed,bitsandbytes,liger-kernel]"

Verify that the CLI is correctly registered:

llamafactory-cli version

Step 2: Register a Custom Extraction Dataset

LLaMA-Factory uses data/dataset_info.json to index datasets. Suppose we have harvested thousands of complex technical web documents using AiFetchly Web Scraping and formatted them into instruction-response pairs:

Create a sample dataset file at data/aifetchly_extraction.json:

[
  {
    "instruction": "Extract the software architecture profile, dependencies, and streaming capabilities from the raw HTML.",
    "input": "<div>aiFetchly desktop application built on Astro 6.1, TypeScript 5.9, and Tailwind 4.0. Backend utilizes FastMCP and LanceDB for sub-5ms vector search with SSE token streaming.</div>",
    "output": "{\n  \"product_name\": \"aiFetchly\",\n  \"frontend\": [\"Astro 6.1\", \"TypeScript 5.9\", \"Tailwind 4.0\"],\n  \"backend\": [\"FastMCP\", \"LanceDB\"],\n  \"supports_streaming\": true\n}"
  }
]

Now, register the dataset in data/dataset_info.json:

{
  "aifetchly_extraction": {
    "file_name": "aifetchly_extraction.json",
    "columns": {
      "prompt": "instruction",
      "query": "input",
      "response": "output"
    }
  }
}

Step 3: Configure the Training Recipe (YAML)

LLaMA-Factory allows defining entire training jobs using declarative YAML configuration files, eliminating brittle bash flag strings.

Create examples/train_qwen_qlora.yaml:

### Model Configuration
model_name_or_path: Qwen/Qwen2.5-7B-Instruct
trust_remote_code: true

### Training Stage
stage: sft
do_train: true
finetuning_type: lora
lora_target: all
lora_rank: 16
lora_alpha: 32
lora_dropout: 0.05

### Quantization & Acceleration
quantization_bit: 4
quantization_type: nf4
double_quantization: true
flash_attn: fa2
enable_liger_kernel: true

### Dataset Configuration
dataset: aifetchly_extraction
template: qwen
cutoff_len: 4096
packing: true
overwrite_cache: true
preprocessing_num_workers: 8

### Hyperparameters
output_dir: saves/qwen2.5-7b-aifetchly-lora
logging_steps: 10
save_steps: 100
plot_loss: true
learning_rate: 2.0e-4
num_train_epochs: 3.0
per_device_train_batch_size: 4
gradient_accumulation_steps: 4
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000

Key architectural features enabled in this configuration:

finetuning_type: lora with lora_target: all: Injects low-rank adapters across all linear layers (attention projections and MLP feed-forward gates).
quantization_bit: 4 (nf4): Loads base weights in 4-bit NormalFloat, cutting model memory from 14GB to ~4.5GB.
flash_attn: fa2 & enable_liger_kernel: true: Unlocks fused Triton kernels for cross-entropy and attention calculations.
packing: true: Packs sequences to eliminate padding tokens, dramatically boosting GPU FLOP utilization.

Step 4: Launch Distributed Fine-Tuning

Launch the training job using the unified CLI:

llamafactory-cli train examples/train_qwen_qlora.yaml

For multi-GPU clusters running DeepSpeed ZeRO-3, simply attach the DeepSpeed configuration:

deepspeed --num_gpus=4 src/llamafactory/launcher.py \
    examples/train_qwen_qlora.yaml \
    --deepspeed examples/deepspeed/ds_z3_config.json

During execution, LLaMA-Factory automatically logs training metrics, renders loss curves in real time, and writes adapter checkpoints to saves/qwen2.5-7b-aifetchly-lora/.

Step 5: Merge Adapters and Export for High-Throughput Serving

Once fine-tuning completes, the low-rank adapter weights must be merged into the base model parameters to eliminate runtime latency during deployment.

Create examples/export_qwen.yaml:

### Base Model & Trained Adapter
model_name_or_path: Qwen/Qwen2.5-7B-Instruct
adapter_name_or_path: saves/qwen2.5-7b-aifetchly-lora
template: qwen
finetuning_type: lora

### Export Configuration
export_dir: models/qwen2.5-7b-aifetchly-merged
export_size: 4
export_device: cpu
export_legacy_format: false

Execute the export command:

llamafactory-cli export examples/export_qwen.yaml

The resulting directory models/qwen2.5-7b-aifetchly-merged contains a standalone, fully-fused model checkpoint ready to be served via vLLM with zero inference overhead:

python -m vllm.entrypoints.openai.api_server \
    --model models/qwen2.5-7b-aifetchly-merged \
    --tensor-parallel-size 1 \
    --port 8000 \
    --gpu-memory-utilization 0.90

Architectural Comparison: LLaMA-Factory vs. Post-Training Frameworks

To evaluate where LLaMA-Factory fits within the AI engineering ecosystem, compare its architectural capabilities against alternative post-training frameworks:

Architectural Dimension	Hugging Face TRL	Unsloth	Axolotl	LLaMA-Factory
Model Architecture Support	Broad (~50 model families)	Focused (~10 popular architectures)	Broad (~40 model families)	Extensive (100+ LLMs and VLMs)
Template Normalization	Manual tokenizer application	Manual formatting scripts	YAML-based template mapping	Fully automated template registry (`dataset_info.json`)
PEFT Methods Supported	LoRA, QLoRA	Custom Triton LoRA/QLoRA	LoRA, QLoRA, DoRA, ReFT	Full, Freeze, LoRA, QLoRA, DoRA, GaLore, PiSSA
Fused Operator Integration	Manual Liger Kernel hooks	Custom proprietary Triton kernels	Liger Kernel & FlashAttention	Native Liger Kernel, FlashAttention-2, and Unsloth
Direct Alignment Algorithms	DPO, PPO, KTO, ORPO	DPO, GRPO (reasoning models)	DPO, KTO, IPO	SFT, DPO, ORPO, KTO, PPO, SimPO, BPO, GRPO
Interface & Workspaces	Code-only (Python scripts)	Code-only (Python notebooks)	CLI & YAML	Unified CLI, YAML recipes, and LLaMA Board WebUI
Serving & Export Pipeline	Manual PyTorch checkpointing	GGUF export	Hugging Face export	Direct LoRA merge, AWQ/GGUF export, and vLLM serving

Production Integration with aiFetchly

In production data harvesting and web intelligence platforms, relying entirely on frontier commercial APIs (like GPT-4o or Claude 3.5 Sonnet) for routine extraction tasks introduces heavy token expenses and external network latency.

By combining AiFetchly with LLaMA-Factory, enterprise teams can implement a closed-loop data harvesting, distillation, and deployment engine:

aiFetchly + LLaMA-Factory Closed-Loop Architecture:
┌───────────────────────────┐      ┌───────────────────────────┐
│     Live Public Web       │ ───► │   aiFetchly Scraper       │
│ E-Commerce, Financial SEC │      │   • Headless Chrome CDP   │
└───────────────────────────┘      │   • Clean DOM to Markdown │
                                   └─────────────┬─────────────┘
                                                 │ Raw Web Data
                                                 ▼
┌───────────────────────────┐      ┌───────────────────────────┐
│ Enterprise Applications   │ ◄─── │ Frontier Teacher Model    │
│ Low-latency private APIs  │      │ (Generates synthetic SFT  │
└───────────────────────────┘      │  instruction pairs)       │
              ▲                    └─────────────┬─────────────┘
              │                                  │ Curated JSON
              │ Merged Checkpoint                ▼
┌─────────────┴─────────────┐      ┌───────────────────────────┐
│ vLLM Inference Engine     │ ◄─── │ LLaMA-Factory Engine      │
│ • Sub-5ms token latency   │      │ • QLoRA + Liger Kernels   │
│ • 100% on-premise privacy │      │ • DPO preference tuning   │
│ • Zero cloud API costs    │      │ • Automatic sequence pack │
└───────────────────────────┘      └───────────────────────────┘

High-Fidelity Document Extraction: AiFetchly Data Fetching harvests messy, heterogeneous web documents, strips styling debris, and normalizes tabular data.
Synthetic Data Generation: A frontier teacher model formats complex extraction tasks into high-quality instruction datasets.
Domain Fine-Tuning with LLaMA-Factory: Small, cost-effective models (such as Qwen 2.5 3B or LLaMA 3.2 1B) are fine-tuned using QLoRA and Liger Kernels, matching the teacher’s extraction precision on specific schemas.
Zero-Cost Private Serving: The distilled model is merged and served via vLLM directly within customer infrastructure, delivering sub-5ms extraction latency with zero third-party data leakage.

To discover how autonomous intelligence can transform your data workflows, explore the AiFetchly Documentation or consult our AiFetchly Pricing.