Unexpected NaN values during training of a TensorFlow model with custom loss function
I tried several approaches but none seem to work. I've been banging my head against this for hours. I'm currently working on a neural network using TensorFlow 2.10 to predict stock prices. I've written a custom loss function that should calculate the mean squared behavior, but I'm getting unexpected NaN values during training, which is causing my model to unexpected result. I've checked my data for NaNs or infinite values, and everything seems fine. Here's my custom loss function: ```python import tensorflow as tf def custom_mse(y_true, y_pred): return tf.reduce_mean(tf.square(y_true - y_pred)) ``` When I compile my model with this loss function: ```python model.compile(optimizer='adam', loss=custom_mse) ``` I initialize my model as follows: ```python model = tf.keras.Sequential([ tf.keras.layers.Dense(64, activation='relu', input_shape=(input_shape,)), tf.keras.layers.Dense(32, activation='relu'), tf.keras.layers.Dense(1) ]) ``` During the training phase, I call: ```python history = model.fit(X_train, y_train, epochs=100, batch_size=32) ``` After a few epochs, I see the following behavior message in my logs: `tensorflow:behavior: behavior while training the model. Details: NaN loss encountered.` I've tried normalizing my data and using a learning rate of 0.001, but it still doesn't work. I also set the Keras `clipnorm` parameter in the optimizer to prevent exploding gradients, yet the question continues. I'm unsure if the scenario is with the loss function or if something else is leading to the NaNs. Any insights would be appreciated! This is part of a larger application I'm building.