TensorFlow 2.12 - Custom Loss Function Not Reducing Loss as Expected with Keras Model
I've been banging my head against this for hours. I'm currently working on a classification problem using TensorFlow 2.12 with Keras, and I've implemented a custom loss function. However, I'm noticing that the loss isn't decreasing as I expected during training. My model architecture is relatively straightforward, consisting of a couple of Dense layers followed by a Dropout layer. Here’s my model setup: ```python import tensorflow as tf from tensorflow import keras from tensorflow.keras import layers model = keras.Sequential([ layers.Dense(128, activation='relu', input_shape=(input_shape,)), layers.Dropout(0.5), layers.Dense(num_classes, activation='softmax') ]) ``` For the custom loss function, I've defined it like this: ```python def custom_loss(y_true, y_pred): epsilon = tf.keras.backend.epsilon() y_pred = tf.clip_by_value(y_pred, epsilon, 1. - epsilon) return -tf.reduce_mean(y_true * tf.math.log(y_pred)) ``` I am compiling the model with this loss function and using the Adam optimizer: ```python model.compile(optimizer='adam', loss=custom_loss, metrics=['accuracy']) ``` During training, I’m using the fit method on a dataset that has been preprocessed correctly. However, after several epochs, the loss remains relatively stagnant around 0.6, not showing a notable decline. I've checked my data pipeline and ensured that the labels are one-hot encoded. Here’s how I’m fitting the model: ```python history = model.fit(train_dataset, epochs=30, validation_data=val_dataset) ``` I tried varying the learning rate from the Adam optimizer and included gradient clipping to manage potential exploding gradients, but it didn’t seem to help. Additionally, I verified that the input data is normalized appropriately (scaled between 0 and 1). I’m also using a batch size of 32, which has worked well in the past. Does anyone have insight into why the loss isn’t decreasing? Are there potential pitfalls in using a custom loss function that I might be overlooking? Any suggestions for debugging this would be greatly appreciated. Thanks for any help you can provide!