Decide between fine-tuning and application changes

A textbook classifier confuses two categories, so its team considers fine-tuning. First locate the failure: missing context, ambiguous labels, invalid output, and inadequate task mapping require different repairs. Parameter adaptation and inference deployment cannot automatically fix application constraints.

A synthetic classification task motivates the choices. A million-parameter arithmetic example estimates weight bytes without selecting, training, or deploying a real model.

Establish a comparable baseline

Record the model, prompt, context, validation, and samples. Separate category confusion, unsupported answers, omissions, latency failures, and permission denials. Fixed examples support comparisons; training loss and demonstrations alone do not establish application improvement.

Separate business errors from input quality. Missing information should permit abstention, and overlapping labels need task repair. Define acceptance according to error costs rather than treating every mistake equally.

Keep independent test data out of training selection. Annotators need consistent rules, and disagreement can reveal ambiguity. More training with unreliable labels may reinforce the wrong behavior.

Choose a path from the failure

Prompts and context express rules and examples. Retrieval supplies changing knowledge with provenance and access controls. Fine-tuning can adapt task mappings, style, or formatting under suitable data conditions. It is not a reliably editable factual database.

Stable classification rules with many trustworthy examples may justify a fine-tuning experiment. Missing current policy suggests retrieval repair; missing schema fields suggest output constraints. Methods can be combined while retaining evidence of each contribution.

Inference deployment addresses loading, concurrency, queues, resources, and versions. A serving framework does not change the training objective or guarantee better classification. Report task quality separately from serving performance.

Split data by leakage-prone units

Transformers documentation describes continuing from pretrained weights on task-specific data. Verify model licensing, data rights, task format, and tool compatibility. Hosted fine-tuning and self-managed training have different interfaces and support boundaries.

Split by documents, templates, authors, or time where appropriate. Neighboring passages and rewritten variants across training and test can inflate generalization. Preserve the separate roles of training, validation, and test, and reassess independence after adding data.

Remove duplicates, correct labels, inspect class balance, and version preprocessing. Retain the base model, tokenizer, templates, and training conditions. Rerun abstention, permission, and unsupported-input checks beyond the main classification score.

LoRA reduces trained parameters conditionally

LoRA freezes pretrained weights and trains low-rank updates represented through smaller matrices. This can reduce trainable parameters. Low rank is a choice and assumption, not a universal guarantee of full fine-tuning quality.

Rank, target modules, and data affect results. Paper findings on particular models and tasks do not establish current performance. Adapter merging, quantization, and kernels depend on the implementation and need compatibility checks.

Adapters usually depend on a specific base model. Keeping an adapter without the matching base, tokenizer, and template may not reproduce behavior. Rollback must restore this related artifact set.

Weight storage is only part of memory

Raw weight storage can be estimated from parameter count and bit width. Runtime also includes quantization metadata, activations, KV cache, workspaces, and framework overhead. Context and concurrency change demand, so file size cannot establish serving capacity.

The example rounds raw bits up to bytes and adds an invented overhead amount. It does not size a real KV cache or deployment.

def weight_bytes(parameters, bits):
    if type(parameters) is not int or type(bits) is not int:
        raise ValueError("integer sizes required")
    if parameters < 0 or bits <= 0:
        raise ValueError("invalid size")
    return (parameters*bits + 7)//8
assert weight_bytes(1000, 16) == 2000
assert weight_bytes(1000, 4) == 500
base = weight_bytes(1000000, 16)
overhead = 400000
assert base + overhead > base
print(base, base + overhead)

The actual output is 2000000 2400000. A million sixteen-bit parameters require two million raw bytes, then the teaching overhead adds four hundred thousand. These are decimal bytes, not MiB, and not complete runtime memory.

Quantization may reduce storage while quality and speed depend on hardware and kernels. Lower bit width does not guarantee higher throughput. Training also needs gradients and optimizer state, so this weight-only function cannot estimate training memory.

Verify serving under the target workload

vLLM provides scheduling and memory-management capabilities. Its latest documentation may describe a developer preview. Pin a version and verify model, quantization, and hardware compatibility instead of treating a documentation heading as a guarantee.

Measure intended input and output lengths, concurrency, and request mixtures. Observe queue time, first-token latency, total duration, errors, and resources. Long and short requests can interact, and downstream capacity also limits scaling.

Version the model, adapter, tokenizer, template, server, and configuration. Use a controlled rollout with stopping criteria and previous artifacts. Recheck output shape, refusals, and tool arguments so changed behavior respects the application contract.

Exercise: decide from evidence

Classify ten failures as input, knowledge, rule, model-behavior, or serving problems. Compare the prerequisites, acceptance checks, and costs of prompting, retrieval, and fine-tuning, then choose one narrow experiment.

Accept it when document-level leakage is excluded, samples and configurations are traceable, and the old model remains recoverable. Explain omitted memory components and plan separate workload measurements. Without reliable labels and independent evaluation, gather evidence before claiming fine-tuning will help.