👀 Views: 62 💬 Answers: 1 📅 Created: 2025-06-07

transformers huggingface gpt-2 generative-ai Python

I'm attempting to set up I've been struggling with this for a few days now and could really use some help. I'm currently using the Hugging Face Transformers library (version 4.21) to generate text using a fine-tuned GPT-2 model. While the generated text is often coherent, I'm experiencing significant variability in the output when using the same input prompt multiple times. For example, when running the following code: ```python from transformers import GPT2LMHeadModel, GPT2Tokenizer import torch model_name = 'gpt2' tokenizer = GPT2Tokenizer.from_pretrained(model_name) model = GPT2LMHeadModel.from_pretrained(model_name) prompt = "Once upon a time in a land far," input_ids = tokenizer.encode(prompt, return_tensors='pt') output = model.generate(input_ids, max_length=50, num_return_sequences=3, do_sample=True, top_k=50, top_p=0.95) for i in range(3): print(tokenizer.decode(output[i], skip_special_tokens=True)) ``` I receive different continuations each time I run this, which is expected since I'm sampling outputs. However, I'm trying to achieve a balance between variability and consistency in the output. I’ve tried fixing the `seed` for the random number generator using `torch.manual_seed(seed_value)` before generating, but it seems to have no effect on the output. Additionally, I’ve experimented with setting `num_return_sequences` to 1 to see if that stabilizes the results, yet the variability persists. Is there a recommended approach to reduce this variability while still using sampling techniques like top-k and top-p? Are there any specific configurations or best practices in Hugging Face that might help? Also, could the issue be related to the model's training data or fine-tuning process? I’m keen to understand how to make the outputs more predictable without sacrificing too much creativity. I'd really appreciate any guidance on this. For reference, this is a production service. Is this even possible?