Unexpected NaN values in Keras model training with TensorFlow 2.9.1
Hey everyone, I'm running into an issue that's driving me crazy. I'm working with unexpected `NaN` values during the training phase of my model built using Keras and TensorFlow 2.9.1. My model is designed for a regression task, and I'm using the Mean Squared behavior loss function. The data has been preprocessed, and I have confirmed that there are no NaN or infinite values in the input features. Here’s a simplified version of my model: ```python import tensorflow as tf from tensorflow import keras model = keras.Sequential([ keras.layers.Dense(64, activation='relu', input_shape=(input_dim,)), keras.layers.Dense(64, activation='relu'), keras.layers.Dense(1) # Output layer for regression ]) model.compile(optimizer='adam', loss='mean_squared_error', metrics=['mae']) ``` I’ve also tried normalizing my inputs using MinMaxScaler from scikit-learn, but the question continues. Here’s how I’m fitting my model: ```python y_train = ... # Your training labels history = model.fit(X_train, y_train, epochs=100, batch_size=32, validation_split=0.2) ``` During training, the loss starts at a reasonable value but soon becomes `NaN`. I’ve added a custom callback to monitor the loss: ```python class NaNCallback(keras.callbacks.Callback): def on_epoch_end(self, epoch, logs=None): if logs.get('loss') is None or logs.get('loss') != logs.get('loss'): print(f'Epoch {epoch + 1}: Loss is NaN!') ``` I’ve tried changing the learning rate and optimizing algorithms, but nothing seems to work. I also ensured that the input data types are correct and compatible with TensorFlow. I'm running this on a machine with a GTX 1080 GPU and TensorFlow is installed with GPU support. Could the scenario be related to how I'm scaling my inputs, or is there something more fundamental I’m missing? Any insights would be greatly appreciated. My development environment is macOS. Is there a better approach?