How to implement guide with tensorflow 2.12 mixed precision training: gradients implementation guide as expected
I'm trying to figure out I'm trying to implement mixed precision training using TensorFlow 2.12 with a custom model that leverages the `tf.keras` API, but I'm running into issues where the gradients are not updating as expected... I've set up my model and optimizer like this: ```python import tensorflow as tf # Enable mixed precision training policy = tf.keras.mixed_precision.Policy('mixed_float16') tf.keras.mixed_precision.set_global_policy(policy) # Define a simple model model = tf.keras.Sequential([ tf.keras.layers.Dense(128, activation='relu', input_shape=(784,), dtype='float16'), tf.keras.layers.Dense(10, activation='softmax', dtype='float16') ]) # Use an Adam optimizer with the mixed precision policy optimizer = tf.keras.mixed_precision.LossScaleOptimizer(tf.keras.optimizers.Adam(learning_rate=0.001)) model.compile(loss='sparse_categorical_crossentropy', optimizer=optimizer, metrics=['accuracy']) ``` When I run my training loop, I notice that the loss is not decreasing as expected, and the training accuracy remains stagnant. I also see warnings like: ``` WARNING:tensorflow:Gradients do not match the expected dtype. ``` I've tried manually setting the `dtype` in my layers to `float32`, but the performance doesn't improve. I am also using a dataset created with `tf.data.Dataset` and feeding it into the model. Here’s how I'm training it: ```python model.fit(dataset, epochs=10) ``` Is there something specific that I might be missing in setting up the mixed precision or the optimizer? Could it be related to the loss scaling? Any suggestions on how to debug this further would be appreciated! I'm using Python 3.10 in this project.