GCP Cloud Pub/Sub message ordering guide with multiple subscriptions in Java
I'm performance testing and I'm working with a question with message ordering in GCP Cloud Pub/Sub while using multiple subscriptions for the same topic in a Java application. I've set up a topic with two subscriptions, and I'm publishing messages with a specific ordering key. The scenario arises when I try to process the messages concurrently across different subscriptions. I've noticed that messages that should be processed in order are sometimes handled out of sequence. Here’s a snippet of how I am publishing messages: ```java import com.google.cloud.pubsub.v1.Publisher; import com.google.pubsub.v1.PubsubMessage; String orderKey = "order-1"; // Example order key Publisher publisher = Publisher.newBuilder(topicName).build(); for (int i = 0; i < 10; i++) { PubsubMessage message = PubsubMessage.newBuilder() .setData(ByteString.copyFromUtf8("Message " + i)) .setOrderingKey(orderKey) .build(); publisher.publish(message); } ``` I've ensured that the `enableMessageOrdering` setting is true for both subscriptions. However, when I consume messages using separate threads, the order is not guaranteed as expected. The behavior I receive is related to the ordering key, specifically: `PERMISSION_DENIED: The message does not have a valid ordering key`. I tried to ensure that my subscribers are set up correctly as shown below: ```java import com.google.cloud.pubsub.v1.MessageReceiver; import com.google.cloud.pubsub.v1.SubscriptionAdminClient; import com.google.cloud.pubsub.v1.Subscriber; MessageReceiver receiver = (message, consumer) -> { System.out.printf("Received message: %s%n", message.getData().toStringUtf8()); consumer.ack(); }; Subscriber subscriber = Subscriber.newBuilder(subscriptionName, receiver).build(); subscriber.startAsync().awaitRunning(); ``` The `PERMISSION_DENIED` behavior seems to indicate an scenario with how the messages are being consumed, but I need to see any misconfigurations. I've checked IAM permissions for both topics and subscriptions, and they seem fine. What can I do to ensure the messages are processed in the correct order while using multiple subscriptions? I recently upgraded to Java 3.11. What are your experiences with this?