Gestural motion and speech interface control method for 3d audio-video-data navigation on handheld devices
First Claim
Patent Images
1. A system for navigation on a handheld device comprising:
- a gesture module that can receive gesture commands from a physical interface on the device or a remote rendering computer server by executing stored first computer instructions on a processor contained in the devicea motion module that produces motion commands by detecting motion of the device in 3-dimensional space by executing second stored computer instructions on said processor that read and process data from motion sensors contained in the device;
a speech module that can decode voice commands spoken through a microphone contained in the device by executing third stored computer instructions on said processor;
a graphics driver adapted to display an image on a display screen attached to the device;
data from each of said gesture module, said motion module and said speech module combined in a prevalence module that executes fourth stored computer instructions on said processor to prioritize a gesture command over a motion command, and prioritize a motion command over a speech command, said prevalence module adapted to issue an action command based on either a gesture, a motion of the device, or a speech command to said graphics driver to cause said graphics driver to modify the image on the display according to said gesture, motion or speech command; and
,wherein said prevalence module transitions smoothly between gesture, motion and speech commands;
and wherein the gesture, motion and speech commands include the following modalities;
M1—
static and displaced gesture commands;
M2—
static and displaced motion commands; and
wherein the gesture, motion and speech commands include at least one of;
M3—
static and displaced speech commands;
M4—
displaced gesture commands and static motion commands;
M5—
static gesture commands and displaced motion commands;
M6—
displaced motion commands and static speech commands;
M7—
displaced gesture commands and static speech commands;
M8—
static gesture commands and displaced speech commands; and
M9—
static motion commands and displaced speech commands.
3 Assignments
0 Petitions
Accused Products
Abstract
A cognizant and adaptive method of informing a multi-modal navigation interface or a user'"'"'s intent. This provides the user with the experience of exploring an immersive representation of the processed multimedia (audio-video-data) sources available that automatically adapts to her/his fruition preference. These results are obtained by first reconciling and aligning the User and the Device'"'"'s frames of reference in tri-dimensional space and then dynamically and adaptively Smoothly Switching and/or combining both Gesture, Motion and Speech modalities. The direct consequence is a user experience that naturally adapts to the user choice of interaction and movement.
-
Citations
17 Claims
-
1. A system for navigation on a handheld device comprising:
-
a gesture module that can receive gesture commands from a physical interface on the device or a remote rendering computer server by executing stored first computer instructions on a processor contained in the device a motion module that produces motion commands by detecting motion of the device in 3-dimensional space by executing second stored computer instructions on said processor that read and process data from motion sensors contained in the device; a speech module that can decode voice commands spoken through a microphone contained in the device by executing third stored computer instructions on said processor; a graphics driver adapted to display an image on a display screen attached to the device; data from each of said gesture module, said motion module and said speech module combined in a prevalence module that executes fourth stored computer instructions on said processor to prioritize a gesture command over a motion command, and prioritize a motion command over a speech command, said prevalence module adapted to issue an action command based on either a gesture, a motion of the device, or a speech command to said graphics driver to cause said graphics driver to modify the image on the display according to said gesture, motion or speech command; and
,wherein said prevalence module transitions smoothly between gesture, motion and speech commands; and wherein the gesture, motion and speech commands include the following modalities;
M1—
static and displaced gesture commands;
M2—
static and displaced motion commands; and
wherein the gesture, motion and speech commands include at least one of;
M3—
static and displaced speech commands;
M4—
displaced gesture commands and static motion commands;
M5—
static gesture commands and displaced motion commands;
M6—
displaced motion commands and static speech commands;
M7—
displaced gesture commands and static speech commands;
M8—
static gesture commands and displaced speech commands; and
M9—
static motion commands and displaced speech commands. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9)
-
-
10. A method for image navigation on a handheld computer device having a touch screen, camera or joystick, a set of motion or geo-location sensors and a microphone driving a speech recognition sub-system, said device also having a 3D graphic engine on board or connected to a remote rendering device and a current device window relating to a displayed image, comprising:
-
receiving a gesture command from said touch screen, camera or joystick, and/or; receiving a motion command from said motion sensors, and/or ; receiving a speech command from said speech recognition sub-system; computing static class data from a static gesture command, if present;
else computing static data from a static motion command, if present;
else computing static data from a static speech command if present;computing displaced class data from a displaced gesture command, if present;
else computing displaced class data from a displaced motion command, if present;
else computing displaced data from a displaced speech command if present;determining locally or remotely an update for said current device window based on said static class data and said displaced class data; commanding the 3D graphic engine locally or remotely to perform said update to the current device window for said displayed image and wherein the gesture, motion and speech commands include the following modalities;
static and displaced speech commands;
displaced gesture commands and static motion commands;
static gesture commands and displaced motion commands;
displaced motion commands and static speech commands;
displaced gesture commands and static speech commands;
static gesture commands and displaced speech commands; and
static motion commands and displaced speech commands. - View Dependent Claims (11)
-
-
12. A system for navigation on a handheld device comprising:
-
a gesture module configured to receive gesture commands from a physical interface on the device; a motion module that produces motion commands by detecting motion of the device in 3-dimensional space configured to read and process data from motion sensors contained in the device; a speech module configured to decode voice commands spoken through a microphone contained in the device; a graphics driver adapted to display an image on a display screen attached to the device; data from each of said gesture module, said motion module and said speech module combined in a prevalence module that executes stored computer instructions on said processor to prioritize a gesture command over a motion command, and prioritize a motion command over a speech command, said prevalence module adapted to issue an action command based on either a gesture, a motion of the device, or a speech command to said graphics driver to cause said graphics driver to modify the image on the display according to said gesture, motion or speech command; and
,wherein said prevalence module transitions smoothly between gesture, motion and speech commands; and wherein the gesture, motion and speech commands include the following modalities;
M1—
static and displaced gesture commands;
M2—
static and displaced motion commands;
M3—
static and displaced speech commands;
M4—
displaced gesture commands and static motion commands;
M5—
static gesture commands and displaced motion commands;
M6—
displaced motion commands and static speech commands ;
M7—
displaced gesture commands and static speech commands;
M8—
static gesture commands and displaced speech commands; and
M9—
static motion commands and displaced speech commands. - View Dependent Claims (13, 14, 15, 16, 17)
-
Specification