Repository navigation
Deadletter redrive fails on the -native graalvm docker image #1225
Description
Activity
- changed the title
[-]Deadletter redrive fails on the -native graalvm build[/-][+]Deadletter redrive fails on the -native graalvm (and jvm) build[/+]on Jun 16, 2026 I've just tried switching to the jvm image. And while it doesn't crash, it doesn't seem to like the redrive either. Some sample logs:
15:25:26.643 [elasticmq-pekko.actor.default-dispatcher-6] INFO o.a.pekko.actor.RepointableActorRef - Message [org.elasticmq.actor.queue.QueueEvent$Restore] to Actor[pekko://elasticmq/user/$a#1967717404] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://elasticmq/user/$a#1967717404] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 15:29:03.891 [elasticmq-pekko.actor.default-dispatcher-11] INFO o.elasticmq.actor.QueueManagerActor - Creating queue CreateQueueData(admintest-orders-in-dev1-dlq,None,None,None,None,None,None,false,false,None,None,Map()) 15:29:03.903 [elasticmq-pekko.actor.default-dispatcher-10] INFO o.a.pekko.actor.RepointableActorRef - Message [org.elasticmq.actor.queue.QueueEvent$QueueCreated] to Actor[pekko://elasticmq/user/$a#1967717404] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://elasticmq/user/$a#1967717404] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 15:29:03.922 [elasticmq-pekko.actor.default-dispatcher-9] INFO o.elasticmq.actor.QueueManagerActor - Creating queue CreateQueueData(admintest-orders-in-dev1,None,None,None,None,None,Some(DeadLettersQueueData(admintest-orders-in-dev1-dlq,1)),false,false,None,None,Map()) 15:29:03.923 [elasticmq-pekko.actor.default-dispatcher-13] INFO o.a.pekko.actor.RepointableActorRef - Message [org.elasticmq.actor.queue.QueueEvent$QueueCreated] to Actor[pekko://elasticmq/user/$a#1967717404] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://elasticmq/user/$a#1967717404] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 15:29:03.936 [elasticmq-pekko.actor.default-dispatcher-9] INFO o.a.pekko.actor.RepointableActorRef - Message [org.elasticmq.actor.queue.QueueEvent$MessageAdded] to Actor[pekko://elasticmq/user/$a#1967717404] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://elasticmq/user/$a#1967717404] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 15:29:03.936 [elasticmq-pekko.actor.default-dispatcher-11] ERROR o.e.a.q.QueueActorStorage$ResultWithEvents - Failed to notify queue event listener. The state may be inconsistent. org.apache.pekko.pattern.AskTimeoutException: Recipient [Actor[pekko://elasticmq/user/$a#1967717404]] had already been terminated. Message of type [org.elasticmq.actor.queue.QueueEvent$MessageAdded]. 15:29:24.945 [elasticmq-pekko.actor.default-dispatcher-9] INFO o.apache.pekko.actor.ActorSystemImpl - Request timeout encountered for request [POST / Strict(126 bytes)] 15:29:24.953 [elasticmq-pekko.actor.default-dispatcher-11] ERROR o.e.r.s.TheSQSRestServerBuilder$$anon$1 - Exception when running routes org.apache.pekko.pattern.AskTimeoutException: Ask timed out on [Actor[pekko://elasticmq/user/$b/queue-admintest-orders-in-dev1-9a530cf4#-1319960575]] after [21000 ms]. Message of type [org.elasticmq.msg.SendMessage]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. 15:29:26.752 [elasticmq-pekko.actor.default-dispatcher-10] ERROR o.e.a.q.QueueActorStorage$ResultWithEvents - Failed to notify queue event listener. The state may be inconsistent. org.apache.pekko.pattern.AskTimeoutException: Recipient [Actor[pekko://elasticmq/user/$a#1967717404]] had already been terminated. Message of type [org.elasticmq.actor.queue.QueueEvent$MessageAdded]. 15:29:26.752 [elasticmq-pekko.actor.default-dispatcher-14] INFO o.a.pekko.actor.RepointableActorRef - Message [org.elasticmq.actor.queue.QueueEvent$MessageAdded] to Actor[pekko://elasticmq/user/$a#1967717404] was not delivered. [5] dead letters encountered. If this is not an expected behavior then Actor[pekko://elasticmq/user/$a#1967717404] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 15:29:33.823 [elasticmq-pekko.actor.default-dispatcher-14] INFO o.elasticmq.actor.QueueManagerActor - Deleting queue admintest-orders-in-dev1 15:29:33.824 [elasticmq-pekko.actor.default-dispatcher-14] INFO o.a.pekko.actor.RepointableActorRef - Message [org.elasticmq.actor.queue.QueueEvent$QueueDeleted] to Actor[pekko://elasticmq/user/$a#1967717404] was not delivered. [6] dead letters encountered. If this is not an expected behavior then Actor[pekko://elasticmq/user/$a#1967717404] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 15:29:33.829 [elasticmq-pekko.actor.default-dispatcher-11] INFO o.elasticmq.actor.QueueManagerActor - Deleting queue admintest-orders-in-dev1-dlq 15:29:33.830 [elasticmq-pekko.actor.default-dispatcher-11] INFO o.a.pekko.actor.RepointableActorRef - Message [org.elasticmq.actor.queue.QueueEvent$QueueDeleted] to Actor[pekko://elasticmq/user/$a#1967717404] was not delivered. [7] dead letters encountered. If this is not an expected behavior then Actor[pekko://elasticmq/user/$a#1967717404] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 15:29:47.763 [elasticmq-pekko.actor.default-dispatcher-11] INFO o.apache.pekko.actor.ActorSystemImpl - Request timeout encountered for request [POST / Strict(126 bytes)] 15:29:47.772 [elasticmq-pekko.actor.default-dispatcher-11] ERROR o.e.r.s.TheSQSRestServerBuilder$$anon$1 - Exception when running routes org.apache.pekko.pattern.AskTimeoutException: Ask timed out on [Actor[pekko://elasticmq/user/$b/queue-admintest-orders-in-dev1-9a530cf4#-1319960575]] after [21000 ms]. Message of type [org.elasticmq.msg.SendMessage]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. 15:31:38.781 [elasticmq-pekko.actor.default-dispatcher-9] INFO o.elasticmq.actor.QueueManagerActor - Creating queue CreateQueueData(probe-send,None,None,None,None,None,None,false,false,None,None,Map()) 15:31:38.783 [elasticmq-pekko.actor.default-dispatcher-11] INFO o.a.pekko.actor.RepointableActorRef - Message [org.elasticmq.actor.queue.QueueEvent$QueueCreated] to Actor[pekko://elasticmq/user/$a#1967717404] was not delivered. [8] dead letters encountered. If this is not an expected behavior then Actor[pekko://elasticmq/user/$a#1967717404] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 15:31:38.817 [elasticmq-pekko.actor.default-dispatcher-11] ERROR o.e.a.q.QueueActorStorage$ResultWithEvents - Failed to notify queue event listener. The state may be inconsistent. org.apache.pekko.pattern.AskTimeoutException: Recipient [Actor[pekko://elasticmq/user/$a#1967717404]] had already been terminated. Message of type [org.elasticmq.actor.queue.QueueEvent$MessageAdded].It's not a happy bunny.
Looking into it, there's some earlier storage problems - I think I have a different problem with the jvm image that I will continue to look into on my own. The -native image though, that's a definite problem.
- changed the title
[-]Deadletter redrive fails on the -native graalvm (and jvm) build[/-][+]Deadletter redrive fails on the -native graalvm docker image[/+]on Jun 16, 2026 Ok - the jvm version was a permissions error on a mountable data volume mount (running as root cleared it). I confirmed that the DLQ redrive does work as expected on the jvm build, so this is an issue restricted graalvm.
Running the -native redrive, and while it appears to succeed, it crashes immediately afterwards with the following logs:
Claude is pretty certain this is a native graal vm problem - I've not worked with java in... 6(?) years now, so not much of an ability to dive straight into it anymore. I did have it produce this - hopefully it's useful to you. Apologies if the use of an LLM is inappropriate here - I didn't set it trying to find bugs, just as a diagnostic tool when I encountered this - here's its full output:
While I'm here, new UI is great, almost completely replaces some custom tooling I had written already (which stored recently sent messages in browser storage to be able to resend/template messages more quickly) - great job there!