ROS2 Callback Groups and Executor Deadlocks: Why Your Node Freezes in Silence

You add a synchronous service call inside a timer callback. The code compiles. The node launches. Everything looks fine — until it doesn't. The node freezes, no error is thrown, and ros2 node list still shows it alive. This is the ros2 callback groups executor deadlock pattern, and it has wasted days of engineering time across hundreds of ROS2 projects.

How the Executor Actually Schedules Callbacks

ROS2 executors pull work from a queue and dispatch it to threads. The single-threaded executor runs one callback at a time. The multithreaded executor runs several in parallel — but only when the callback group configuration allows it.

Every callback (timer, subscriber, service server, action server) belongs to a callback group. There are two types:

• MutuallyExclusive — at most one callback in the group runs at a time, regardless of how many threads are available
• Reentrant — callbacks in the group can run concurrently on different threads

The critical detail: every callback you create is assigned to the node's default callback group, which is MutuallyExclusive. This is not documented prominently. Most engineers discover it the hard way.

The Deadlock Pattern

Here is the code that causes it:

// Node constructor
timer_ = this->create_wall_timer(
    std::chrono::milliseconds(100),
    std::bind(&MyNode::timer_callback, this)
);
client_ = this->create_client<MySrv>("my_service");

void MyNode::timer_callback() {
    auto request = std::make_shared<MySrv::Request>();
    auto future = client_->async_send_request(request);

    // Synchronous wait inside the callback
    if (rclcpp::spin_until_future_complete(this->get_node_base_interface(), future)
        != rclcpp::FutureReturnCode::SUCCESS) {
        RCLCPP_ERROR(this->get_logger(), "Service call failed");
    }
}

Here is what happens at runtime:

1. The timer fires. The executor picks up timer_callback and starts executing it. 2. Inside the callback, an async request is sent and then immediately waited on with spin_until_future_complete. 3. The service processes the request and sends a response. The response arrives as a new callback — the service client's done-callback. 4. The done-callback needs to execute for the future to resolve. 5. The done-callback is assigned to the same default MutuallyExclusive group as the timer. 6. The executor cannot run the done-callback while timer_callback is still executing and holds the group lock. 7. The wait never completes. The node hangs.

This is documented in ros2/ros2 issue #1400 and confirmed in rclcpp issue #2278, where the same pattern causes a hang in the EventsExecutor as well. It is not limited to one executor type.

The ROS2 docs acknowledge it indirectly: "Setting up callback groups of a node incorrectly can lead to deadlocks" and "synchronous calls to actions or services should not be done in callbacks." They don't show you what incorrect looks like in practice.

Diagnosing a Deadlock vs a Slow Callback

Before changing your callback group assignment, confirm you actually have a deadlock and not just a slow service.

Step 1 — Check executor thread utilization.

# While the node is frozen, inspect its threads
ps -eLf | grep your_node_name

If a MultiThreadedExecutor is running but all threads are idle except one (which is stuck), the executor is blocked waiting on a lock, not on computation.

Step 2 — Check the callback group assignment.

ros2 node info /your_node_name

This lists publishers, subscribers, service clients, and service servers but does not expose callback group names directly. For that, add a temporary log in your node constructor:

RCLCPP_INFO(this->get_logger(), "Timer callback group: %p",
    (void*)timer_->get_callback_group().get());
RCLCPP_INFO(this->get_logger(), "Client callback group: %p",
    (void*)client_->get_callback_group().get());

If both pointers are identical, the timer and client share the same group. That is your deadlock.

Step 3 — Check for the starvation variant.

In February 2025, a ROS Discourse thread documented an active bug in the multithreaded executor where multiple timers in a MutuallyExclusive group cause task starvation — not a hard deadlock, but a situation where one timer repeatedly wins the scheduling race and the others are starved. A fix was filed as rclcpp PR #2702. Research published at RTAS 2024 on end-to-end timing jitter in ROS2 confirms that the multithreaded executor's scheduling can produce priority inversion in chains with mixed-priority callbacks. The executor is not as parallel as its name implies.

The Decision Rule for Callback Group Assignment

Three cases cover most production nodes:

Case 1: Two callbacks that do not interact. They can safely share the default MutuallyExclusive group. The executor serializes them, which is predictable and avoids data races. Use this by default.

Case 2: A callback that calls a service or action and needs the response in the same execution context. Put the client in a separate MutuallyExclusive group so its done-callback can run while the caller is waiting:

// Constructor
auto client_group = this->create_callback_group(
    rclcpp::CallbackGroupType::MutuallyExclusive
);
client_ = this->create_client<MySrv>("my_service", rmw_qos_profile_services_default, client_group);

Now the timer callback and the client's done-callback are in different groups. The executor can dispatch both without a lock conflict. This is the minimum fix for the deadlock.

Case 3: High-frequency callbacks that must not block each other. Use a Reentrant group and async patterns throughout. Be aware that reentrant callbacks require your code to be thread-safe — shared state needs explicit protection.

auto reentrant_group = this->create_callback_group(
    rclcpp::CallbackGroupType::Reentrant
);

The safest general rule: never wait synchronously inside a callback. Use async_send_request and handle the response in a separate callback. Keep the synchronous wait — spin_until_future_complete — only in main() or in dedicated non-callback threads. If you must call a service from inside a callback and cannot refactor, separate the client into its own MutuallyExclusive group as shown in Case 2.

What to Take Away

The default callback group assignment in ROS2 is safe and predictable for simple nodes. It becomes a trap the moment you add a service call inside a callback without thinking about group ownership. The freeze is silent, the node stays alive, and nothing in the log tells you what happened.

Three questions to ask when reviewing any node that calls a service or action:

1. Is the call inside a callback? 2. Do the timer/subscriber and the client share the same callback group? 3. Is there a synchronous wait (spin_until_future_complete, future.get()) inside that callback?

If all three are yes, you have a deadlock waiting to happen. Separate the groups, or move to async response handling.