Unexpected NaN values when training a Keras model with TimeSeries data - best practices for?
I'm testing a new approach and I'm updating my dependencies and I'm currently working on a time series forecasting question using TensorFlow 2.7.0 and Keras... While training my model, I noticed that the loss starts to return NaN values, which halts the training process. I've ensured that my input data is normalized, and I'm using the Adam optimizer with a learning rate of 0.001. Here's a snippet of my model architecture: ```python from tensorflow.keras.models import Sequential from tensorflow.keras.layers import LSTM, Dense, Dropout model = Sequential() model.add(LSTM(50, return_sequences=True, input_shape=(timesteps, features))) model.add(Dropout(0.2)) model.add(LSTM(50)) model.add(Dropout(0.2)) model.add(Dense(1)) model.compile(optimizer='adam', loss='mean_squared_error') ``` I have preprocessed my time series data, ensuring there are no missing values. However, despite this, I keep running into an scenario where the training loss becomes NaN after just a few epochs. Here's how I'm fitting the model: ```python history = model.fit(X_train, y_train, epochs=50, batch_size=32, validation_data=(X_val, y_val)) ``` I've also tried reducing the batch size to 16, but the scenario continues. I suspect it could be due to exploding gradients, but Iām not getting any warnings related to that. I've applied gradient clipping to the optimizer as follows: ```python from tensorflow.keras.optimizers import Adam optimizer = Adam(learning_rate=0.001, clipnorm=1.0) model.compile(optimizer=optimizer, loss='mean_squared_error') ``` In addition, I checked the gradients during training using a callback to log their values, but they seem to be within reasonable ranges. I also tried using `tf.debugging.check_numerics` to catch any NaNs in my input data or model weights, but that hasn't pointed to anything either. Is there something I'm missing in the configuration or preprocessing steps? Any insights on resolving this scenario would be greatly appreciated. I appreciate any insights! My team is using Python for this service.