Semi Supervised

How to read this page: This article maps the topic from beginner to expert across six levels � Remembering, Understanding, Applying, Analyzing, Evaluating, and Creating. Scan the headings to see the full scope, then read from wherever your knowledge starts to feel uncertain. Learn more about how BloomWiki works ?

Semi-supervised learning sits between supervised learning (which requires labels for all training data) and unsupervised learning (which uses no labels). It leverages a small amount of labeled data alongside a large amount of unlabeled data to train better models than either approach alone. Since labeled data is expensive and time-consuming to acquire while unlabeled data is often abundantly available, semi-supervised learning is highly practical. Modern variants include pseudo-labeling, consistency regularization, and graph-based methods.

Remembering

• Semi-supervised learning — Learning using a small labeled dataset and a large unlabeled dataset simultaneously.
• Pseudo-labeling — Using a model's predictions on unlabeled data as provisional labels, then retraining on those labels.
• Consistency regularization — Enforcing that model predictions remain consistent under perturbations of unlabeled inputs.
• Mean Teacher — A semi-supervised method where a student model is trained, and the teacher model is an exponential moving average of student weights; teacher provides pseudo-labels.
• FixMatch — A state-of-the-art semi-supervised image classification method using confidence thresholding and weak/strong augmentation consistency.
• MixMatch — A holistic semi-supervised approach combining pseudo-labeling, consistency regularization, and MixUp data augmentation.
• Self-training — Train on labeled data, predict labels for unlabeled data, retrain on the combination; repeat iteratively.
• Co-training — Train two models on different feature views; each provides pseudo-labels for the other.
• Graph-based methods — Propagate labels through a graph where edges represent similarity between examples (label propagation).
• Label propagation — Semi-supervised algorithm that spreads labels from labeled to unlabeled examples through a similarity graph.
• Manifold assumption — The assumption that data lies on a low-dimensional manifold; points on the same manifold should have the same label.
• Smoothness assumption — If two points are close in input space, they should have similar labels.
• Cluster assumption — Decision boundaries should lie in low-density regions between clusters.
• Confidence threshold — In pseudo-labeling, only use predictions where model confidence exceeds a threshold; avoids noisy pseudo-labels.

Understanding

Semi-supervised learning works by exploiting the structure of the unlabeled data distribution to constrain the label function. The key assumptions:

Smoothness: Nearby points → similar labels. If two images of dogs are close in feature space, they should both be labeled "dog."

Cluster: Classes form clusters. The decision boundary should pass through low-density regions between clusters, not through high-density regions.

Manifold: Data lies on lower-dimensional manifolds. Using unlabeled data to learn the manifold structure helps place decision boundaries correctly.

Self-training process:

1. Train on labeled data.
2. Predict labels for unlabeled data.
3. Add high-confidence predictions to training set.
4. Retrain.
5. Repeat. Risk: confident errors propagate (confirmation bias). Mitigated by strict confidence thresholds.

FixMatch: The state-of-the-art simple baseline. For each unlabeled image:

1. Apply weak augmentation (horizontal flip, crop).
2. If prediction confidence > 0.95, use as pseudo-label.
3. Apply strong augmentation (RandAugment).
4. Train student to predict the pseudo-label on the strongly augmented view. This enforces consistency across augmentation strengths while only training on confident pseudo-labels.

When does semi-supervised help most? When labeled data is very scarce (<1000 examples) and unlabeled data shares the same distribution as labeled data. When distributions differ (domain shift between labeled and unlabeled), semi-supervised can hurt — a form of negative transfer.

Applying

FixMatch implementation: <syntaxhighlight lang="python"> import torch import torch.nn.functional as F

def fixmatch_loss(model, labeled_x, labels, unlabeled_x_weak, unlabeled_x_strong,

                 threshold=0.95, lambda_u=1.0):
   # Supervised loss on labeled data
   logits_labeled = model(labeled_x)
   loss_supervised = F.cross_entropy(logits_labeled, labels)

   # Pseudo-label on weakly augmented unlabeled data
   with torch.no_grad():
       logits_weak = model(unlabeled_x_weak)
       probs_weak = F.softmax(logits_weak, dim=-1)
       max_probs, pseudo_labels = probs_weak.max(dim=-1)
       # Mask: only use predictions above confidence threshold
       mask = (max_probs >= threshold).float()

   # Consistency loss: predict pseudo-label on strongly augmented version
   logits_strong = model(unlabeled_x_strong)
   loss_unsupervised = (F.cross_entropy(logits_strong, pseudo_labels, reduction='none') * mask).mean()

   return loss_supervised + lambda_u * loss_unsupervised

</syntaxhighlight>

Semi-supervised method selection

Image classification → FixMatch, FlexMatch, FreeMatch (confidence threshold scheduling)

NLP → UDA (Unsupervised Data Augmentation), pre-train then fine-tune (BERT approach)

Graph data → Label propagation, Graph Convolutional Networks (GCN)

Small labeled set (<100 samples) → Mean Teacher, MixMatch

Production setting → Self-training with pseudo-labels (simple, scalable)

Analyzing

Semi-Supervised Methods Comparison

Method	Labeled Data Needed	Key Idea	Best Domain
Self-training	~10-20%	Confidence filtering	Any
FixMatch	<1%	Consistency + threshold	Vision
Mean Teacher	<5%	EMA teacher labels	Vision
Label Propagation	~5%	Graph diffusion	Low-dim, graph
BERT fine-tuning	<1% (semi)	Large pre-training	NLP

Failure modes: Pseudo-label noise — incorrect confident predictions pollute training. Distribution mismatch — unlabeled data from different distribution hurts performance. Over-fitting on pseudo-labels — model memorizes spurious patterns in pseudo-labels. Confirmation bias — model fails to correct its own early confident errors.

Evaluating

Evaluation must match the practical setting: hold out a labeled test set; train only on the (small labeled) + (large unlabeled) split. Report accuracy as a function of labeled data fraction (1%, 5%, 10%) to show the semi-supervised benefit curve. Compare against:

1. supervised-only (small labeled set),
2. fully supervised (all labeled), and
3. self-supervised pre-training + fine-tuning as competing baselines.

Creating

Designing a semi-supervised pipeline:

1. Start with self-training — simple, effective, easy to implement.
2. Set high confidence threshold (0.95+) to avoid noisy pseudo-labels.
3. Apply curriculum: increase unlabeled data usage as model improves (FlexMatch adaptive threshold).
4. For vision: use FixMatch with RandAugment strong augmentation.
5. For NLP: leverage domain-adaptive pre-training on unlabeled data, then fine-tune on labels.
6. Monitor pseudo-label quality: compute accuracy of pseudo-labels on held-out labeled data as a proxy for noise level.