Unexpected Gradient Explosion in LSTM with TensorFlow 2.9.0
I'm currently training an LSTM model using TensorFlow 2.9.0 for sequence prediction, but I'm working with an scenario where the gradients explode during training... I have a sequence length of 50, and I'm using a batch size of 32 with an embedding dimension of 128. My model architecture consists of one LSTM layer followed by a Dense layer. Whenever I start training, I receive the following warning: ``` WARNING:tensorflow:Gradients explosion detected. ``` I've tried implementing gradient clipping by adding `tf.keras.optimizers.Adam(learning_rate=0.001, clipnorm=1.0)` to the optimizer, but the question continues. Here’s a snippet of my model setup: ```python import tensorflow as tf model = tf.keras.Sequential([ tf.keras.layers.Embedding(input_dim=10000, output_dim=128, input_length=50), tf.keras.layers.LSTM(64, return_sequences=False), tf.keras.layers.Dense(1, activation='sigmoid') ]) optimizer = tf.keras.optimizers.Adam(learning_rate=0.001, clipnorm=1.0) model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy']) ``` Additionally, I checked the input data and ensured it was properly normalized. However, I'm still seeing this explosion in gradients. I’ve also tried different learning rates (0.01 and 0.0001) but with no luck. Could this be related to the LSTM configuration, or should I be looking at my data preprocessing? Are there specific techniques or adjustments in the LSTM layer that can help stabilize the gradients during training? I'm working on a service that needs to handle this. Am I missing something obvious? This is part of a larger service I'm building.