Walk into any modern robotics lab and you will likely see a social robot performing a flawless one-on-one conversation. It maintains eye contact, responds to a single voice, and executes precise gestures. However, move that same robot into a crowded room with three or four people, and the illusion shatters. The robot becomes a confused observer, unable to discern who is speaking, who is listening, or where to direct its attention. This gap between singular interaction and true social fluency is the primary hurdle for Physical AI, as most current systems are optimized for a vacuum rather than the chaotic, overlapping dynamics of human groups.

The Five-Stage Pipeline from Intent to Action

To bridge this gap, a specialized program hosted by Imperial College London from July 20 to 24, 2026, focused on the architectural requirements of multi-party Human-Robot Interaction (HRI). Funded by UK RAS STEPS and guided by Marie Daniels, the initiative brought together five Research Training Partnership (RTP) members to dismantle the silos between high-level cognitive reasoning and low-level hardware execution. The core of their work centers on a rigorous five-stage control pipeline that ensures a natural language command does not simply remain a string of text but becomes a physical reality.

The process begins with the Large Language Model (LLM), which serves as the cognitive engine. Rather than outputting conversational prose, the LLM is tasked with translating a user's natural language intent into structured commands. These are formalized data formats, such as JSON or specific schemas, that strip away human ambiguity and replace it with parameters that a machine can process. For instance, a request to look at a specific person is converted into a target coordinate and a designated action ID.

Once structured, these commands are mapped to high-level functions. These are pre-designed software modules that represent discrete capabilities of the robot, such as wave, nod, or rotate. This mapping layer ensures that the LLM does not need to understand the physics of the robot, only the available library of actions. The high-level function then defines the target coordinates for the robot's end-effector or head position.

This is where the system enters the realm of Inverse Kinematics (IK). While forward kinematics calculates where a hand will be given certain joint angles, Inverse Kinematics works backward. It takes the desired target coordinate and calculates the exact angles every single joint in the robotic arm or neck must assume to reach that point. This calculation generates a joint trajectory, a time-sequenced map of movement that prevents the robot from colliding with itself or moving in an erratic fashion.

These trajectories are then fed into the control stack. This layer acts as the translator between software logic and hardware electricity. The control stack manages the timing and sequence of signals, converting numerical trajectory data into electrical pulses. Finally, these pulses reach the actuators—the motors and cylinders that provide the physical force. The actuator converts the electrical signal into rotation or linear motion, completing the journey from a thought in an LLM to a movement in the physical world.

The Social Layer: Solving the Multi-User Bottleneck

While the five-stage pipeline handles the physics of movement, multi-party interaction introduces a layer of social complexity that standard pipelines ignore. The real challenge is not just moving the robot, but deciding who to move toward and when. The Imperial College research highlights three critical control elements that transform a mechanical tool into a social agent: active speaker detection, identity tracking, and gaze management.

Active speaker detection is the first line of defense against social confusion. In a group setting, the robot must analyze incoming audio signals to triangulate the source of the sound. This acoustic data is then synchronized in real-time with vision sensor data. By matching the audio source with the visual detection of a human mouth moving or a subtle shift in posture, the robot can identify the current speaker with high precision, drastically reducing response latency in fast-paced conversations.

Once a speaker is identified, the robot employs identity tracking to maintain a persistent relationship with each individual. This involves assigning a unique ID to every person in the room based on visual features and spatial positioning. The system must handle re-identification, meaning if a user turns their head or is momentarily blocked by another person, the robot does not treat them as a stranger when they reappear. This persistence allows the robot to maintain the context of the conversation, ensuring that a response intended for Person A is not accidentally delivered to Person B.

Finally, the robot must communicate its attention through gaze and head rotation. In human psychology, gaze is the primary signal of engagement. To avoid looking like a jerky machine, the researchers focused on the acceleration and deceleration curves of the robot's head movements. By smoothing the start and end points of a rotation, the robot can shift its attention between a questioner and a listener in a way that feels organic. This physical alignment—looking at the person you are speaking to—is often more important for user trust than the actual linguistic accuracy of the LLM's response.

This integration requires a deep fusion of kinematics and dynamics. Kinematics handles the geometry of the movement, but dynamics accounts for mass, force, and torque. For the robot to maintain a steady gaze while moving its base or interacting with an object, it must constantly calculate the torque required to counteract gravity and inertia. This is the same engineering challenge found in quadruped locomotion, where a robot must shift its center of mass in real-time to navigate uneven terrain, or in tactile sensing, where a robot must adjust its grip strength based on the material of an object to prevent slippage.

For AI practitioners moving toward Physical AI, the takeaway is clear: the bottleneck is rarely the LLM's reasoning capability. Instead, the failure point is almost always the interface between the LLM's output and the physical trajectory. The transition from a text-based response to a JSON-structured command, and subsequently to an IK-calculated joint angle, is where the social experience is won or lost. The future of robotics lies not in larger models, but in the seamless integration of these disparate engineering disciplines into a single, fluid control pipeline.

True social intelligence in robotics is achieved only when the precision of inverse kinematics meets the nuance of human social cues.