Unexpected NaN values during model training with TensorFlow 2.8 and Adam optimizer
I'm attempting to set up This might be a silly question, but I'm working with an scenario where my training process in TensorFlow 2.8 produces NaN values for the loss function when I use the Adam optimizer. My model is a simple feedforward neural network for a regression task, but as soon as I start training, the loss jumps to NaN after a couple of epochs. I've tried normalizing my input data and ensuring that the target values are within a reasonable range, yet the question continues. Here’s a snippet of my model and training loop: ```python import tensorflow as tf from tensorflow.keras import layers, models # Sample data import numpy as np X_train = np.random.rand(1000, 10).astype(np.float32) Y_train = np.random.rand(1000, 1).astype(np.float32) # Build the model model = models.Sequential([ layers.Dense(64, activation='relu', input_shape=(10,)), layers.Dense(64, activation='relu'), layers.Dense(1) ]) # Compile the model model.compile(optimizer='adam', loss='mean_squared_error') # Train the model model.fit(X_train, Y_train, epochs=100, batch_size=32) ``` I've also checked for any potential divisions by zero or logarithm operations on zero, as well as ensuring that there are no exploding gradients in my model, but nothing seems to help. The Adam optimizer is known for its robustness, so I am perplexed as to why this is occurring. Additionally, I tried using a learning rate scheduler and adjusted the learning rate down to 1e-4 and 1e-5, but it didn't resolve the scenario either. I printed out the loss value during training and it starts at a reasonable number but rapidly escalates to NaN. Any insights or troubleshooting steps would be greatly appreciated! I'm on macOS using the latest version of Python. Any advice would be much appreciated. The project is a mobile app built with Python. I'm open to any suggestions.