Unexpected decrease in accuracy after fine-tuning a pretrained model using PyTorch 1.10
I tried several approaches but none seem to work. I'm currently fine-tuning a pretrained ResNet50 model from torchvision on a custom dataset for image classification. Initially, I achieved about 85% validation accuracy after training for 10 epochs. However, after making some adjustments to the learning rate and batch size, I'm now observing a significant drop to around 70%. I've tried several configurations, but nothing seems to help. Here's the code I'm using for the training loop: ```python import torch import torchvision.transforms as transforms from torchvision import datasets, models from torch import nn, optim # Data transformations transform = transforms.Compose([ transforms.Resize((224, 224)), transforms.ToTensor(), ]) # Load datasets train_dataset = datasets.ImageFolder(root='path/to/train', transform=transform) train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=32, shuffle=True) # Load pretrained ResNet50 model model = models.resnet50(pretrained=True) # Modify the final layer for our dataset num_ftrs = model.fc.in_features model.fc = nn.Linear(num_ftrs, len(train_dataset.classes)) criterion = nn.CrossEntropyLoss() optimizer = optim.Adam(model.parameters(), lr=0.001) # Training loop num_epochs = 10 for epoch in range(num_epochs): model.train() running_loss = 0.0 correct = 0 total = 0 for inputs, labels in train_loader: optimizer.zero_grad() outputs = model(inputs) loss = criterion(outputs, labels) loss.backward() optimizer.step() running_loss += loss.item() _, predicted = torch.max(outputs.data, 1) total += labels.size(0) correct += (predicted == labels).sum().item() epoch_loss = running_loss / len(train_loader) epoch_accuracy = correct / total print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {epoch_loss:.4f}, Accuracy: {epoch_accuracy:.4f}') ``` One of the main changes I made was reducing the learning rate to 0.0001 and increasing the batch size to 64. I also implemented early stopping based on validation loss. However, the accuracy continues to drop with each epoch. I suspect that there might be an issue with overfitting or perhaps the dataset's class distribution is affecting the training. I have verified that my dataset is correctly set up and the classes are balanced. I would appreciate any insights into why I'm seeing this decline in accuracy and suggestions for debugging this issue. Are there any specific techniques I can apply to improve the model’s performance after fine-tuning? For context: I'm using Python on macOS.