Master Thesis: Mean Teacher and Adversarial Unlabeled Data for Language Classification
A detailed research narrative from my M.Sc. thesis on adversarial robustness, Mean Teacher training, and resilient BERT fine-tuning for real-world NLP.
How to Build Language Models That Do Not Break
Have you ever wondered how easily a state-of-the-art AI can be tricked by a single word?
During my Master's thesis at the University of Koblenz-Landau, I explored this exact question in the context of adversarial robustness for NLP. The project started with a practical concern: many language models look excellent on clean benchmarks, but their behavior can degrade sharply once inputs are intentionally perturbed.
Thesis PDF: Access the full thesis
Why This Problem Matters
NLP systems now support high-impact tasks such as fake-news detection, content moderation, and risk communication. In these settings, reliability is as important as raw accuracy.
An adversarial attack in text is usually a small and targeted modification that does not meaningfully change the sentence for a human reader, but can push a model toward the wrong label. This mismatch between human interpretation and model prediction reveals a structural weakness: the model may rely on brittle surface cues instead of stable semantic features.
In my thesis framing, the core objective was straightforward: improve robustness under attack while preserving performance on clean data.
Method Overview: Adapting Mean Teacher to NLP
I adapted the Mean Teacher paradigm, a semi-supervised strategy first established in computer vision, to transformer-based text classification.
This direction was also informed by my earlier 2020 technical exploration of Mean Teacher, where the method stood out for its intuitive learning dynamics and strong results in computer vision before adaptation to NLP.
Instead of training only one network, the approach uses two models with identical architecture:
- A Student model, optimized with standard backpropagation.
- A Teacher model, updated by an exponential moving average of student weights.
This structure creates stable training targets over time. The teacher acts as a smoothed temporal ensemble of student states, which helps reduce overreaction to noisy updates.
The teacher update rule is:
where is the student parameter vector at step , is the teacher parameter vector, and controls smoothing strength.
Consistency Training Under Perturbation
The main robustness mechanism is consistency regularization. During training, the model sees perturbed inputs, including adversarially transformed text, and is encouraged to keep predictions consistent between student and teacher.
Intuitively, this penalizes unstable decision boundaries and promotes invariance to superficial perturbations. In other words, the model learns to care more about semantic signal and less about fragile lexical artifacts.
A key practical element is the noise strategy itself. Injecting stochasticity and perturbation during training helps avoid overfitting to narrow token patterns and improves behavior on unseen samples.
Objective Formulation
The total training objective combines supervised task learning and consistency learning.
The classification cost on labeled data is binary cross-entropy:
The consistency cost on perturbed unlabeled inputs is:
The final objective is the weighted sum:
where controls the trade-off between task fidelity and robustness regularization.
Experimental Setup
I evaluated this approach on two datasets: IMDB movie reviews and a COVID-19 fake-news dataset. To test robustness, I used four black-box attack recipes: TextFooler, TextBugger, PWWS, and BAE.
The proposed robust variants, MTBERT and MTDistilBERT, were compared against conventional fine-tuning baselines under both clean and attacked conditions.
Main Findings
The results were consistently encouraging. Under attack, the Mean Teacher variants showed an improvement of roughly 20 to 30 percentage points compared with standard approaches. At the same time, clean-data performance did not collapse; it remained stable and in some runs improved slightly by around 0 to 2 percentage points.
Another important observation was attack effort. The attackers generally needed more queries to find successful adversarial transformations against MTBERT, indicating higher practical resistance.
In line with my earlier fake-news detection experiments, I also observed a repeated training pattern: the teacher model can start outperforming the student only after sufficient optimization steps. In prior runs, this crossover appeared around epoch 15, though the exact point depends on task and setup.
Practical Interpretation
These outcomes suggest that robustness does not have to be achieved by sacrificing normal-task quality. With consistency regularization and EMA-based teacher guidance, it is possible to improve adversarial resilience while retaining competitive clean accuracy.
The trade-off is computational. Mean Teacher fine-tuning is heavier in time and memory because two model states are maintained. However, for trust-sensitive pipelines such as misinformation detection, this overhead can be justified by the reliability gain.
The convergence behavior is sensitive to training configuration. In practice, epoch count, batch size, training-set size, and EMA coefficient jointly shape whether teacher guidance becomes beneficial. Parameter tuning is therefore not optional; it is a core part of making the method work reliably.
Closing Reflection
One lesson from this work is that intelligence and reliability are different properties. A model can be strong on benchmarks and still be weak under adversarial pressure.
Robust NLP therefore requires training objectives that explicitly encode stability. For my thesis, Mean Teacher with adversarial consistency provided a practical and effective step in that direction.
If you want the complete experimental protocol, ablations, and detailed results, please refer to the full thesis PDF.
References
- Saini, B. (2020, October 28). Machine Learning: Understanding Mean Teacher Model. Medium. https://bksaini078.medium.com/machine-learning-understanding-mean-teacher-model-ffb8d07819a4
- Tarvainen, A., & Valpola, H. (2017). Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in Neural Information Processing Systems 30. https://proceedings.neurips.cc/paper/2017/hash/68053af2923e00204c3ca7c6a3150cf7-Abstract.html
- Miyato, T., Maeda, S., Koyama, M., & Ishii, S. (2018). Virtual Adversarial Training: A regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8), 1979-1993. https://doi.org/10.1109/TPAMI.2018.2858821
- Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT 2019. https://doi.org/10.48550/arXiv.1810.04805
- Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. NeurIPS EMC2 Workshop. https://doi.org/10.48550/arXiv.1910.01108
- Jin, D., Jin, Z., Zhou, J. T., & Szolovits, P. (2020). Is BERT Really Robust? Natural Language Attack on Text Classification and Entailment. AAAI 2020. https://doi.org/10.1609/aaai.v34i05.6311
- Li, J., Ji, S., Du, T., Li, B., & Wang, T. (2020). TextBugger: Generating Adversarial Text Against Real-world Applications. NDSS 2020. https://dx.doi.org/10.14722/ndss.2020.23134
- Ren, S., Deng, Y., He, K., & Che, W. (2019). Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency. ACL 2019. https://aclanthology.org/P19-1103/
- Garg, S., & Ramakrishnan, G. (2020). BAE: BERT-based Adversarial Examples for Text Classification. EMNLP 2020. https://doi.org/10.18653/v1/2020.emnlp-main.498
- Curious AI. Mean Teacher project resources. https://github.com/CuriousAI/mean-teacher
- Saini, B. Fake news detection with weakly supervised learning (code repository). https://github.com/bksaini078/fake_news_detection