👀 Views: 2 💬 Answers: 1 📅 Created: 2025-06-07

transformers gpt-2 fine-tuning machine-learning Python

Does anyone know how to I'm relatively new to this, so bear with me. I'm trying to fine-tune a pre-trained GPT-2 model using the Hugging Face Transformers library on a custom dataset of short stories. After setting up my training loop, I'm encountering some unexpected behaviors. While the loss initially decreases, it quickly plateaus and the outputs are nonsensical, which doesn't seem to improve even after multiple epochs. Here's a snippet of my training code: ```python from transformers import GPT2Tokenizer, GPT2LMHeadModel, Trainer, TrainingArguments from datasets import load_dataset # Load the tokenizer and model tokenizer = GPT2Tokenizer.from_pretrained('gpt2') model = GPT2LMHeadModel.from_pretrained('gpt2') # Load my custom dataset dataset = load_dataset('text', data_files='my_stories.txt') # Tokenize the input data def tokenize_function(examples): return tokenizer(examples['text'], padding='max_length', truncation=True) tokenized_datasets = dataset.map(tokenize_function, batched=True) # Set training arguments training_args = TrainingArguments( output_dir='./results', overwrite_output_dir=True, num_train_epochs=3, per_device_train_batch_size=2, save_steps=10_000, save_total_limit=2, logging_steps=200, prediction_loss_only=True, ) # Create Trainer instance trainer = Trainer( model=model, args=training_args, train_dataset=tokenized_datasets['train'], ) # Start training trainer.train() ``` I'm using Transformers version 4.25.1 and my dataset consists of approximately 500 short stories, each averaging around 300 words. I noticed that the output text generated after training doesn't resemble coherent stories anymore; it often consists of random phrases that do not relate to the training data. Furthermore, the loss is stagnating at around 3.5, which feels too high after three epochs. I have also tried adjusting the learning rate from the default 5e-5 to 1e-4 but it didn't help. Additionally, I've ensured that my dataset is formatted correctly and that there's enough variability in the content. Has anyone experienced similar issues with GPT-2 fine-tuning, or can provide insights into potential misconfigurations or best practices for training on custom datasets? This is part of a larger service I'm building. Any ideas what could be causing this? I've been using Python for about a year now. Am I missing something obvious? For context: I'm using Python on Debian. I appreciate any insights!