Gestural motion and speech interface control method for 3d audio-video-data navigation on handheld devices

• US 9,395,764 B2
• Filed: 04/16/2014
• Issued: 07/19/2016
• Est. Priority Date: 04/25/2013
• Status: Active Grant

• Current Assignee: Flippo Costanzo
• Original Assignee: Hyperreality, Inc.
• Inventors: Buttafarro, Roberto, Costanzo, Filippo, Tornisiello, Fernando, Funto, Fabrizio
• Primary Examiner(s): Zhou, Hong
• Application Number:
  US14/254,055
• Publication Number:
  US 20140320394A1
• Time in Patent Office: 825 Days
• Field of Search: 345/6, 345156-173, 345/419, 345679-680, 715/863, 463 31- 32, 463/37
• US Class Current: 1/1
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

A cognizant and adaptive method of informing a multi-modal navigation interface or a user'"'"'s intent. This provides the user with the experience of exploring an immersive representation of the processed multimedia (audio-video-data) sources available that automatically adapts to her/his fruition preference. These results are obtained by first reconciling and aligning the User and the Device'"'"'s frames of reference in tri-dimensional space and then dynamically and adaptively Smoothly Switching and/or combining both Gesture, Motion and Speech modalities. The direct consequence is a user experience that naturally adapts to the user choice of interaction and movement.

Claims

1. A system for navigation on a handheld device comprising:

a gesture module that can receive gesture commands from a physical interface on the device or a remote rendering computer server by executing stored first computer instructions on a processor contained in the device

a motion module that produces motion commands by detecting motion of the device in 3-dimensional space by executing second stored computer instructions on said processor that read and process data from motion sensors contained in the device;

a speech module that can decode voice commands spoken through a microphone contained in the device by executing third stored computer instructions on said processor;

a graphics driver adapted to display an image on a display screen attached to the device;

data from each of said gesture module, said motion module and said speech module combined in a prevalence module that executes fourth stored computer instructions on said processor to prioritize a gesture command over a motion command, and prioritize a motion command over a speech command, said prevalence module adapted to issue an action command based on either a gesture, a motion of the device, or a speech command to said graphics driver to cause said graphics driver to modify the image on the display according to said gesture, motion or speech command; and

,

wherein said prevalence module transitions smoothly between gesture, motion and speech commands;

and wherein the gesture, motion and speech commands include the following modalities;

M1—

static and displaced gesture commands;

M2—

static and displaced motion commands; and

wherein the gesture, motion and speech commands include at least one of;

M3—

static and displaced speech commands;

M4—

displaced gesture commands and static motion commands;

M5—

static gesture commands and displaced motion commands;

M6—

displaced motion commands and static speech commands;

M7—

displaced gesture commands and static speech commands;

M8—

static gesture commands and displaced speech commands; and

M9—

static motion commands and displaced speech commands.

2. The system for navigation of claim 1 wherein said device is a telephone or tablet computer.

3. The system for navigation of claim 1 wherein said display is a touch screen and gestures are entered on virtual controls on said touch screen.

4. The system for navigation of claim 1 wherein said sensors include at least a motion sensing device.

5. The system for navigation of claim 1 wherein said first, second, third and fourth set of computer instructions are stored in a memory contained is said device or in a remote computer server.

6. The system for navigation of claim 1 wherein said motion module generates a motion command when a device motion parameter exceeds a predetermined value.

7. The system for navigation of claim 1 wherein said gesture commands, said motion commands and said speech commands each belong either to a static class or a displaced class, wherein commands belonging to the static class change view direction of said image with no change in displacement, and commands belonging to the displaced class change displacement of said image with no change in view direction.

8. The system for navigation of claim 7 configured to provide a user with a seamless transition between situations where interaction changes from gesture or speech commands to motion commands in either a static or a displaced class.

9. The system for navigation of claim 1 configured to provide a smooth and adaptive automatic switching among modalities M1, M2, M3, M4, M5, M6, M7, M8 and M9.

10. A method for image navigation on a handheld computer device having a touch screen, camera or joystick, a set of motion or geo-location sensors and a microphone driving a speech recognition sub-system, said device also having a 3D graphic engine on board or connected to a remote rendering device and a current device window relating to a displayed image, comprising:

receiving a gesture command from said touch screen, camera or joystick, and/or;

receiving a motion command from said motion sensors, and/or ;

receiving a speech command from said speech recognition sub-system;

computing static class data from a static gesture command, if present;

else computing static data from a static motion command, if present;

else computing static data from a static speech command if present;

computing displaced class data from a displaced gesture command, if present;

else computing displaced class data from a displaced motion command, if present;

else computing displaced data from a displaced speech command if present;

determining locally or remotely an update for said current device window based on said static class data and said displaced class data;

commanding the 3D graphic engine locally or remotely to perform said update to the current device window for said displayed image

and wherein the gesture, motion and speech commands include the following modalities;

static and displaced speech commands;

displaced gesture commands and static motion commands;

static gesture commands and displaced motion commands;

displaced motion commands and static speech commands;

displaced gesture commands and static speech commands;

static gesture commands and displaced speech commands; and

static motion commands and displaced speech commands.

11. The method of claim 10 further comprising providing a user with a seamless transition between situations where interaction changes from gesture or speech commands to motion commands in either a static or a displaced class.

12. A system for navigation on a handheld device comprising:

a gesture module configured to receive gesture commands from a physical interface on the device;

a motion module that produces motion commands by detecting motion of the device in 3-dimensional space configured to read and process data from motion sensors contained in the device;

a speech module configured to decode voice commands spoken through a microphone contained in the device;

a graphics driver adapted to display an image on a display screen attached to the device;

data from each of said gesture module, said motion module and said speech module combined in a prevalence module that executes stored computer instructions on said processor to prioritize a gesture command over a motion command, and prioritize a motion command over a speech command, said prevalence module adapted to issue an action command based on either a gesture, a motion of the device, or a speech command to said graphics driver to cause said graphics driver to modify the image on the display according to said gesture, motion or speech command; and

,

wherein said prevalence module transitions smoothly between gesture, motion and speech commands;

and wherein the gesture, motion and speech commands include the following modalities;

M1—

static and displaced gesture commands;

M2—

static and displaced motion commands;

M3—

static and displaced speech commands;

M4—

displaced gesture commands and static motion commands;

M5—

static gesture commands and displaced motion commands;

M6—

displaced motion commands and static speech commands ;

M7—

displaced gesture commands and static speech commands;

M8—

static gesture commands and displaced speech commands; and

M9—

static motion commands and displaced speech commands.

13. The system for navigation of claim 12 wherein said device is a telephone or tablet computer.

14. The system for navigation of claim 12 wherein said display is a touch screen and gestures are entered on virtual controls on said touch screen.

15. The system for navigation of claim 12 wherein said sensors include at least a motion sensing device.

16. The system for navigation of claim 12 wherein said gesture commands, said motion commands and said speech commands each belong either to a static class or a displaced class, wherein commands belonging to the static class change view direction of said image with no change in displacement, and commands belonging to the displaced class change displacement of said image with no change in view direction.

17. The system for navigation of claim 12 configured to provide a smooth and adaptive automatic switching among modalities M1, M2, M3, M4, M5, M6, M7, M8 and M9.