Unexpected NaN values during model training in TensorFlow 2.6 with Sparse Categorical Crossentropy
I need some guidance on I've searched everywhere and can't find a clear answer..... This might be a silly question, but I'm working with NaN values in my loss during training when using TensorFlow 2.6 with the Sparse Categorical Crossentropy loss function. I have a classification question with 10 classes, and my training data consists of images that have been preprocessed correctly to a range of [0, 1]. I implemented the following: ```python import tensorflow as tf from tensorflow.keras import layers, models # Load your dataset here # Assuming `train_images` and `train_labels` are defined model = models.Sequential([ layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)), layers.MaxPooling2D((2, 2)), layers.Flatten(), layers.Dense(128, activation='relu'), layers.Dense(10) # 10 classes ]) model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = model.fit(train_images, train_labels, epochs=10) ``` I've tried normalizing my dataset by dividing pixel values by 255, but I still face the same scenario. The NaN values seem to appear right from the first epoch. I also checked for any NaNs in the dataset before training, and there aren't any. Additionally, I attempted using different optimizers like RMSprop and lowering the learning rate to 0.001, but that hasn’t helped either. I receive the following behavior in the logs: ``` WARNING:tensorflow:Model was constructed with shape (None, 28, 28, 1) for input KerasTensor(type_spec=TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name="conv2d_input"), name="conv2d_input", description="created by layer 'conv2d_input'"). ``` Could it be related to the batch size or the dataset size? The batch size I am using is 32, and I have about 10,000 training images. Any suggestions on debugging this scenario or alternative configurations I might have overlooked would be greatly appreciated. My development environment is Ubuntu. How would you solve this? For context: I'm using Python on Ubuntu. Is this even possible? This is for a web app running on Ubuntu 20.04. Could someone point me to the right documentation? Is there a simpler solution I'm overlooking?