Extended Reality (XR) systems are often most useful when users are engaged in ongoing physical tasks, yet current interaction techniques still largely assume the hands are available. We present opportunistic body input, an interaction paradigm that redirects continuous XR control to whichever available body region remains free in the moment.
To investigate how users naturally coordinate these spare-body movements, we conducted an elicitation study across six hand-busy scenarios. We found that while users' preferred spare body regions shift dynamically based on physical constraints, the resulting spontaneous movements share a consistent, low-dimensional kinematic structure organized around a dominant principal axis.
Building on these findings, we present DOBI (Dynamic Opportunistic Body Input), a real-time XR technique that uses gaze to target a UI element, a brief trigger gesture to identify the recruited spare joint, and the region's subsequent motion to drive continuous 1D control. A 1D Fitts' law study establishes the baseline motor performance of this paradigm across four distinct body regions, achieving throughputs up to 2.62 bits/s with an overall 5.0% error rate, and a dual-task usability study shows that DOBI supports reliable, low-effort control (SUS = 84.2) while users remain engaged in realistic hand-busy activities.
When our hands are full, we already improvise with other body parts. We nudge a door shut with a shoulder, or point at something with a tilt of the head, without ever deciding to. We wanted to bring that instinct to input, so that beyond the hands, any spare body part and its motion could be recruited at the moment it was needed. The idea follows opportunistic controls in tangible interaction, which repurpose whatever the environment already offers; in the same way, we look for what the body itself leaves free. To make that work, DOBI fixes no body part in advance and calibrates nothing ahead of time.
The user looks at a control target and double-repeats a gesture with any spare joint. The same gesture signals intent and picks the joint.
A mirrored skeleton shows the recruited limb and how far it can move. Moving that joint drives the control target; dwelling discards an accidental trigger.
The user simply stops moving. Holding still briefly commits the value and disengages control, and the user goes back to the task.
Twelve experienced XR users physically enacted six everyday hand-busy scenarios — cooking, cleaning, repairing, washing hands, walking, and sitting — and improvised spare-body movements for continuous control, yielding 437 free-form input designs (426 with valid kinematic recordings).
Even the globally highest-rated region (right wrist, M = 4.78) ranked first in only two of six scenarios, dropping substantially in the others. The top-rated region changed with the task (Friedman tests, all p < .01) — so no single region should be fixed at design time. The user, not the system, must initiate which region to use.
“Wrist rotation feels easier, but when my hands are wet, using my elbow feels more intuitive.” — P3
Although participants were free to devise any movement, their designs clustered into 11 recurring articulations. Participants varied widely in which region they recruited, but not in how they moved a given region once selected — each region's anatomy admits only a small number of natural, controllable motions. The system therefore does not require open-ended motion recognition.
Modelling each joint as a 6D vector (log-map rotation + position delta) and applying PCA over the recruited kinematic subtree, the elicited movements concentrate variance on a single component (η1: median 97.6%, IQR 95.0–98.9%), follow straight trajectories in pose space (median S = 0.95), and maintain proportionality between intended and projected control levels (median R2 = 0.98). Unlike prior PCA-based body control, no dedicated calibration session is needed — the 1D structure emerges spontaneously from the user's self-chosen movement.
Wrist movements produced the largest median PC1 span (1.09), more than double that of head movements (0.45), with substantial variation across users and contexts. Agreement on direction, however, was near-universal (Head 94%, Shoulder 97%, Elbow 95%, Wrist 92%). DOBI therefore calibrates gain dynamically per activation while assigning polarity implicitly.
DOBI turns a brief trigger gesture into a full control mapping. The same motion that signals intent also serves as a “kinematic preview” that supplies the spatial data needed to extract the control axis and calibrate gain — so there is no separate setup phase.
Borrowing the familiar double-click metaphor, the trigger is a quick double-repetition movement — a double wrist flick, a double shoulder shrug — a universal temporal signature that applies to any joint and is distinguishable from incidental task motion. We collected 3,916 samples from 12 participants, with negative samples drawn from participants actually performing the hand-busy tasks rather than prompted non-trigger motions.
Double-repetition gestures averaged 1.06 s (95th percentile 1.52 s), motivating a fixed 1.5 s rolling detection window. A lightweight ST-GCN (0.51M parameters — far smaller than the original 3.1M architecture) with parallel detection and root-joint identification heads achieved 98.5% binary detection accuracy with a 1.6% false positive rate, and 95.9% root joint identification across 8 classes.
Peak analysis isolates the two repetitions within the detected window. The identified root joint defines a kinematic subtree (root plus all distal joints), suppressing background noise from the ongoing physical task. Pose-linearized PCA over those frames yields the 1D motion axis.
Two quality gates then filter the gesture: a linearity gate (η1 ≥ 0.75) requiring the motion be genuinely one-dimensional, and a range gate (ρ ≥ 0.15) filtering out unintentional micro-twitches. The span observed during the trigger sets the input gain; all subsequent motion along the axis maps directly into continuous scalar values, which different activation functions then interpret as proportional, rate-controlled, step-based, or threshold control.
Two complementary studies: a controlled 1D Fitts' law study establishing baseline motor performance, and a dual-task usability study testing whether DOBI holds up while users are genuinely busy.
N = 14 · 4,200 trials
Participants performed an ISO 9241-9 reciprocal tapping task mapped to a normalized virtual slider, crossing four spare-body movements (Shoulder, Arm, Elbow, Wrist) with five indices of difficulty.
All four regions produced linear models (R2 = 0.74–0.97), confirming that DOBI's PCA-based mapping yields proportional 1D control consistent with standard human motor behavior. Throughput averaged 2.23 bits/s at a 5.0% error rate — comparable to prior hands-free XR baselines, while leaving the hands free for the primary task.
| Condition | TP (bits/s) | Error (%) | MT (ms) |
|---|---|---|---|
| Arm | 1.96 (0.32) | 5.8 (4.3) | 1300 (167) |
| Shoulder | 1.85 (0.48) | 5.7 (3.4) | 1435 (296) |
| Elbow | 2.40 (0.70) | 5.5 (3.5) | 1224 (250) |
| Wrist | 2.62 (0.40) | 3.2 (2.7) | 1138 (163) |
N = 6 · 186 trigger attempts
Participants used DOBI on four ambient XR widgets (video, music, weather, maps) while carrying out three hand-busy primary tasks — cooking, carrying boxes, and walking.
Of 186 trigger attempts, 173 (93.0%) succeeded, 2 (1.1%) were false positives, and 11 (6.0%) were true triggers whose mapping was unsatisfactory and required a re-trigger. Participants rated the system highly usable (SUS = 84.2, SD = 5.2), well above the conventional 68 threshold, with Likert ratings highest for interaction responsiveness (M = 6.3), ease of operation (6.2), and task-flow preservation (6.2).
“I instinctively selected the movement with the least torque.” — P6
“I can control without looking, after the trigger.” — P3
If you find this work useful, please cite it as:
@inproceedings{kim2026dobi,
author = {Kim, Rachel and Qian, Xun and Yoon, Sang Ho},
title = {DOBI: Dynamic Opportunistic Body Input via Spare Joint
Recruitment for Hands-Free XR},
booktitle = {The 39th Annual ACM Symposium on User Interface Software
and Technology (UIST '26)},
year = {2026},
location = {Detroit, MI, USA},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
isbn = {979-8-4007-2856-3},
doi = {10.1145/3830398.3830670},
url = {https://doi.org/10.1145/3830398.3830670}
}