Unexpected NaN values in TensorFlow Keras model during training with L2 regularization
I'm working on a project and hit a roadblock. Hey everyone, I'm running into an issue that's driving me crazy. I tried several approaches but none seem to work. Hey everyone, I'm running into an issue that's driving me crazy... I'm currently training a neural network using TensorFlow 2.8 with Keras and I've enabled L2 regularization in my loss function to prevent overfitting. However, during training, I am working with unexpected `NaN` values in my loss metric after a few epochs. I have tried adjusting the learning rate, but the scenario continues. Here's the model architecture I'm using: ```python import tensorflow as tf from tensorflow.keras import layers, models, regularizers model = models.Sequential([ layers.Dense(64, activation='relu', input_shape=(input_dim,), kernel_regularizer=regularizers.l2(0.01)), layers.Dense(64, activation='relu', kernel_regularizer=regularizers.l2(0.01)), layers.Dense(1, activation='sigmoid') ]) model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001), loss='binary_crossentropy', metrics=['accuracy']) ``` The training data is scaled using MinMaxScaler, and I'm using binary cross-entropy as my loss function. Here's how I fit the model: ```python history = model.fit(X_train, y_train, epochs=50, batch_size=32, validation_data=(X_val, y_val)) ``` The training dataset is reasonably sized and I haven't encountered any issues during the preprocessing stage. However, after about 10 epochs, the loss starts to output `NaN` values. The log shows: ``` Epoch 10/50 1/1 [==============================] - 0s 15ms/step - loss: nan - accuracy: 0.5000 - val_loss: 0.6931 - val_accuracy: 0.5000 ``` I've already verified that there are no `NaN` values in the input data. Additionally, I checked the gradients right before the loss calculation, and they seem to be finite. Could this scenario be related to the L2 regularization or perhaps an interaction with the Adam optimizer? Any recommendations on how I can debug this further or prevent `NaN` values from appearing in the training loss would be greatly appreciated. My development environment is Windows. For context: I'm using Python on macOS. Am I missing something obvious? Could someone point me to the right documentation?