Thanks for stopping by to let us know something could be better!
Steps to reproduce
ConcurrentModificationException rarely occurs under a mild load (we send ~3-5 pub sub messages per second through the same Publisher instance), each message has an ordering key. The ordering keys' set is fixed, and there are only 3 of them if that matters (i.e there is approximately 1-2 messages with the same ordering key per second, and there is only 3 different keys that are used).
Unfortunately, there are no steps to reproduce as it seems like this is not really possible, because everything is guarded by a reentrant lock. I spent a couple of hours trying to reproduce and/or find a root cause just by staring at the code.
I have a theory, however, that that happens when the (another?) critical section is entered again in a nested call by the same thread, therefore nested section unlocks the lock when it is left, and another thread can enter, causing them both to modify the hash map. As I said, I couldn't find the code path that could lead to that though.
Another thing that there was a log message right before the failure:
Mar 15, 2025 2:59:46 PM com.google.cloud.pubsub.v1.Publisher publishOutstandingBatch
WARNING: Attempted to publish batch with zero messages.
Stack trace
java.util.concurrent.ExecutionException: java.util.ConcurrentModificationException
at java.base/java.util.concurrent.FutureTask.report(Unknown Source)
at java.base/java.util.concurrent.FutureTask.get(Unknown Source)
at x.y.z.FailFastExecutors$FailFastScheduledThreadPoolExecutor.afterExecute(FailFastExecutors.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
caused by: java.util.ConcurrentModificationException: null
at java.base/java.util.HashMap$HashIterator.remove(Unknown Source)
at com.google.cloud.pubsub.v1.Publisher.publishAllWithoutInflight(Publisher.java:449)
at com.google.cloud.pubsub.v1.Publisher.access$1700(Publisher.java:95)
at com.google.cloud.pubsub.v1.Publisher$2.run(Publisher.java:383)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
External references such as API reference guides
Any additional information below
Following these steps guarantees the quickest resolution possible.
Thanks!
Thanks for stopping by to let us know something could be better!
Steps to reproduce
ConcurrentModificationExceptionrarely occurs under a mild load (we send ~3-5 pub sub messages per second through the same Publisher instance), each message has an ordering key. The ordering keys' set is fixed, and there are only 3 of them if that matters (i.e there is approximately 1-2 messages with the same ordering key per second, and there is only 3 different keys that are used).Unfortunately, there are no steps to reproduce as it seems like this is not really possible, because everything is guarded by a reentrant lock. I spent a couple of hours trying to reproduce and/or find a root cause just by staring at the code.
I have a theory, however, that that happens when the (another?) critical section is entered again in a nested call by the same thread, therefore nested section unlocks the lock when it is left, and another thread can enter, causing them both to modify the hash map. As I said, I couldn't find the code path that could lead to that though.
Another thing that there was a log message right before the failure:
Stack trace
External references such as API reference guides
Any additional information below
Following these steps guarantees the quickest resolution possible.
Thanks!