LoRA Fine-Tuning Studio

1. Base Model

Select a model ? Choose a base model to fine-tune. Smaller models train faster and need less GPU memory; larger ones generally produce better results.

2. Training Data

Dataset repo ? A public or private dataset on the Hugging Face Hub, in the form user/dataset_name (e.g. tatsu-lab/alpaca).

Split ? Which dataset split to use for training. Common values: train, validation, test.

HF Token (optional) ? A Hugging Face access token. Required for private/gated datasets and models. Token is sent only to your training backend.

Upload file ? Upload training data as JSONL, CSV, or Parquet. Each record should contain a text/instruction field aligned with your prompt template.

Click to upload or drag a file here

Accepted: .jsonl, .json, .csv, .parquet, .txt (max 5 GB)

Text / prompt column ? The column name in your dataset that holds the text to train on. For instruction datasets this is often text, prompt, or a templated field.

3. Fine-Tuning Approach

Method ? LoRA injects small trainable rank-decomposition matrices into selected layers, leaving the base weights frozen. QLoRA adds 4-bit quantization of the base model for lower memory; DoRA decomposes weight updates into magnitude and direction.

LoRA Configuration

Rank (r) ? The dimension of the low-rank decomposition. Higher r = more trainable parameters and capacity, but more memory and risk of overfitting. Typical: 4–64. Start with 8 or 16.

Alpha (α) ? Scaling factor for the LoRA update. The effective scale is α/r. A common rule of thumb is α = 2 × r. Larger α makes LoRA updates have a bigger effect on the base weights.

Dropout ? Probability of dropping LoRA activations during training. Helps regularize and prevents overfitting on small datasets. Typical range: 0.0–0.1.

Target modules ? Which projection layers receive LoRA adapters. Attention projections (q_proj, v_proj) are the most common and cheapest. Adding MLP projections (gate_proj, up_proj, down_proj) increases capacity but costs memory.

Bias ? none: keep all bias terms frozen (default and most efficient). all: train every bias. lora_only: only train biases in LoRA-adapted layers.

Task type ? Sets the head used by PEFT. CAUSAL_LM for autoregressive language modeling, SEQ_CLS for classification, SEQ_2_SEQ_LM for encoder-decoder, TOKEN_CLS for tagging.

Training Hyperparameters

Learning rate ? Step size for the optimizer. LoRA typically uses higher LRs than full fine-tuning. Typical range: 1e-5 to 5e-4. Start with 2e-4.

Epochs ? Number of complete passes over the training dataset. Too many causes overfitting; too few underfits. Typical: 1–5 for instruction-tuning.

Batch size ? Number of training samples processed per forward/backward pass per device. Limited by GPU memory. Combine with gradient accumulation to simulate larger batches.

Grad accumulation ? Accumulate gradients over N micro-batches before stepping the optimizer. Effective batch = batch_size × accumulation × num_devices. Lets you simulate large batches with limited memory.

Warmup ratio ? Fraction of total training steps used to linearly ramp the learning rate from 0 to its target. Stabilizes early training. Typical: 0.0–0.1.

Weight decay ? L2 regularization strength applied to weights (excluding biases and LayerNorm). Helps prevent overfitting. Typical: 0.0–0.1.

Max sequence length ? Maximum number of tokens per training example; longer ones get truncated. Memory and compute scale roughly quadratically with sequence length.

Optimizer ? adamw_torch is the standard. adamw_8bit (bitsandbytes) cuts optimizer memory ~75% with little quality loss. adafactor uses even less memory but can be slower to converge.

LR scheduler ? How the learning rate changes over training. cosine decays smoothly to zero (most common for LoRA). linear decays linearly. constant keeps it fixed.

Precision ? bf16 is preferred on Ampere/Hopper GPUs (A100/H100). fp16 for older GPUs. fp32 is the safest but uses 2× memory and is slower.

Quantization ? Loads the base model in lower precision to save VRAM. 4-bit (nf4) with double-quantization is the QLoRA default. 8-bit is a milder option. none keeps the model in full precision.

Gradient checkpointing ? Trades compute for memory by recomputing forward activations during the backward pass. Cuts VRAM usage substantially at a ~20–30% speed cost. Recommended for large models on small GPUs.

Seed ? Random seed for reproducible runs. Affects data shuffling, dropout, and weight init. Use the same seed to reproduce a run exactly.

Logging steps ? How often (in optimizer steps) to log training loss and metrics. Lower values give finer-grained curves but more log noise.

Save steps ? How often to write a checkpoint of the LoRA adapter. Lower = safer against crashes but more disk usage.