Unexpected NaN values in Keras model training with custom loss function
I'm dealing with Quick question that's been bugging me - I'm currently working on a Keras model for a regression task and have implemented a custom loss function to better capture the nuances of my data. However, during training, I'm working with unexpected NaN values in the loss output. My model is quite simple, consisting of a few dense layers, and I'm using TensorFlow 2.10.0. Here's the custom loss function I wrote: ```python import tensorflow as tf def custom_loss(y_true, y_pred): loss = tf.reduce_mean(tf.square(y_true - y_pred)) # Mean squared behavior return loss + tf.reduce_mean(tf.maximum(0.0, y_pred - 1.0)) # Penalty for predictions > 1 ``` I compile my model like this: ```python model.compile(optimizer='adam', loss=custom_loss) ``` During training, I receive the following warning: `WARNING:tensorflow:Gradients do not exist for variables [<tf.Variable 'dense/kernel:0' shape=(10, 1) dtype=float32>].` I suspect that this could be related to the NaN values appearing in my predictions. I've checked my dataset, and there are no NaN or infinite values in the input data. I'm training with a batch size of 32 and using early stopping. After a few epochs, loss values fluctuate wildly, and I can see NaNs in the logs. I’ve tried normalizing my input data, but the scenario continues. Also, I added gradient clipping to mitigate any potential exploding gradients without success. Does anyone have insights on why this might be happening, or suggestions on how to debug this? Could there be any specific edge cases in the custom loss function that I might be missing? I'm working on a CLI tool that needs to handle this. Any ideas what could be causing this? Thanks for taking the time to read this!