Virtual reality system with control command gestures

• US 9,927,870 B2
• Filed: 03/07/2017
• Issued: 03/27/2018
• Est. Priority Date: 06/30/2015
• Status: Active Grant

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Ariadne's Thread (Usa), Inc.
• Inventors: Li, Adam
• Primary Examiner(s): Du, Haixia
• Application Number:
  US15/452,638
• Publication Number:
  US 20170185144A1
• Time in Patent Office: 385 Days
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 7/74 : involving reference images or patches
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/11 : Hand-related biometrics; Hand pose recognition

Abstract

A virtual reality system that uses gestures to obtain commands from a user. Embodiments may use sensors mounted on a virtual reality headset to detect head movements, and may recognize selected head motions as gestures associated with commands. Commands associated with gestures may modify the user'"'"'s virtual reality experience, for example by selecting or modifying a virtual world or by altering the user'"'"'s viewpoint within the virtual world. Embodiments may define specific gestures to place the system into command mode or user input mode, for example to temporarily disable normal head tracking within the virtual environment. Embodiments may also recognize gestures of other body parts, such as wrist movements measured by a smart watch.

Claims

1. A virtual reality system with control command gestures, comprising:

at least one display viewable by a user;

at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user;

a pose analyzer coupled to said at least one sensor, that calculates pose data of said pose of one or more body parts of said user, based on said sensor data generated by said at least one sensor;

a control state comprising a command mode flag that is either true or false;

an enter command mode command that sets said command mode flag to true when executed, wherein said enter command mode command is associated with an enter command mode gesture of one or more of said one or more body parts of said user;

an exit command mode command that sets said command mode flag to false when executed, wherein said exit command mode command is associated with an exit command mode gesture of one or more of said one or more body parts of said user;

one or more control commands, each configured to modify said control state when executed, each associated with one or more gestures of one or more of said one or more body parts of said user;

a gesture recognizer coupled to said pose analyzer and to said control state, wherein said gesture recognizer

receives said pose data from said pose analyzer;

when said command mode flag is true,

determines whether said user has performed a gesture associated with a control command; and

,

executes said control command to modify said control state when said user has performed said gesture associated with said control command; and

,

when said command mode flag is false,

disables execution of any control command of said one or more control commands;

a 3D model of a scene; and

,

a scene renderer coupled to said at least one display, said pose analyzer, said control state, and said 3D model, wherein said scene renderer

optionally modifies or selects said 3D model of a scene based on said control state;

receives said pose data from said pose analyzer;

calculates one or more rendering virtual camera poses, based on said pose data and on said control state;

calculates one or more 2D projections of said 3D model, based on said one or more rendering virtual camera poses and on said control state; and

,

transmits said one or more 2D projections to said at least one display.

2. The system of claim 1 wherein

said gesture recognizer comprises

a gesture motion pattern for each of said one or more gestures;

and,

a pose data time series comprising samples of said pose data received at a sequence of times from said pose analyzer; and

,

said determines whether said user has performed a gesture associated with a control command comprises

compare said pose data time series to the gesture motion pattern for each of said one or more gestures.

3. The system of claim 1 wherein

said one or more body parts of said user comprise a head of said user;

and,

said one or more gestures comprise gesture motions of said head of said user.

4. The system of claim 3 wherein said gesture motions of said head of said user comprise one or more of

a turn of said head right at an angular velocity exceeding a right turn threshold;

a turn of said head left at an angular velocity exceeding a left turn threshold;

a nod of said head upward at an angular velocity exceeding an upward nod threshold;

a nod of said head downward at an angular velocity exceeding a downward nod threshold;

a turn of said head to the left followed by a turn of said head to the right, wherein a time interval between a start of said turn of said head to the left and a finish of said turn of said head to the right is less than a left-right threshold;

a turn of said head to the right followed by a turn of said head to the left wherein a time interval between a start of said turn of said head to the right and a finish of said turn of said head to the left is less than a right-left threshold;

a nod of said head upward followed by a nod of said head downward wherein a time interval between a start of said nod of said head upward and a finish of said nod of said head downward is less than an up-down threshold; and

,

a nod of said head downward followed by a nod of said head upward wherein a time interval between a start of said nod of said head downward and a finish of said nod of said head upward is less than a down-up threshold.

5. The system of claim 1 wherein said one or more control commands comprise one or more of

a command to switch said 3D model of a scene from a first 3D model of a first scene to a second 3D model of a second scene;

a command to pause time modifications of said 3D model;

a command to start said time modifications of said 3D model;

a command to rewind said time modifications of said 3D model;

a command to fast-forward said time modifications of said 3D model;

a command to modify a playback rate of said time modifications of said 3D model; and

,

a command to modify a location of said user relative to said 3D model.

6. The system of claim 1 wherein

said control state comprises

a user input in progress flag that is either true or false;

and,

a user selection value;

said one or more control commands comprise

a start user input command that sets said user input in progress flag to true;

one or more modify user selection commands that change said user selection value when said user input in progress flag is true; and

a complete user input command that sets said user input in progress flag to false; and

,

said scene renderer

overlays a user input control onto one or more of said one or more 2D projections while said user input in progress flag is true;

modifies an appearance of said user input control based on said user selection value.

7. The system of claim 6 wherein

said one or more body parts of said user comprise a head of said user;

said one or more gestures comprise gesture motions of said head of said user; and

,

said scene renderer freezes said one or more rendering virtual camera poses while said user input in progress flag is true.

8. The system of claim 7 wherein

said complete user input command is associated with a gesture motion of said head of said user comprising said head remaining substantially still for a period of time exceeding a complete input time threshold value.

9. The system of claim 1 wherein said one or more body parts of said user comprise

a head of said user;

and

a second body part of said user that differs from said head of said user.

10. The system of claim 9 wherein one or more of said one or more control commands are associated with one or more gestures of said second body part of said user.

11. The system of claim 9 wherein

said scene renderer calculates said one or more rendering virtual camera poses, based on said pose of said head of said user;

and,

said one or more control commands are each associated with one or more gestures of said second body part of said user.

12. The system of claim 9 wherein

said second body part of said user is a wrist of said user;

and,

said at least one sensor comprises one or more wrist sensors that measure one or more aspects of a pose of said wrist of said user.

13. The system of claim 12 wherein said one or more wrist sensors comprise one or more of

a watch;

and

a fitness band.

14. The system of claim 1, further comprising:

an image warper coupled to said at least one display, said scene renderer, and said pose analyzer, wherein said image warper

receives said one or more rendering virtual camera poses from said scene renderer;

receives said pose data from said pose analyzer;

calculates a change in pose between said one or more virtual camera poses and said pose data;

generates a rerendering approximation of said one or more 2D projections of said 3D model on said at least one display based on said change in pose; and

,

modifies one or more pixels of said at least one display based on said rerendering approximation.

15. The system of claim 14, wherein said rerendering approximation comprises calculating a pixel translation vector;

and,

translating one or more pixels of said one or more 2D projections by said pixel translation vector.

16. The system of 15, wherein said calculating said pixel translation vector comprises

approximating said change in pose as a rotation around a unit vector {circumflex over (ω

)} comprising {circumflex over (ω

)}_y and {circumflex over (ω

)}_w by an angle Δ

θ

;

calculating a spatial translation vector ({circumflex over (ω

)}_yΔ

θ

, −

{circumflex over (ω

)}_xΔ

θ

);

calculating a scaling factor to convert spatial distances to pixels based on pixel dimensions and fields of view of said one or more 2D projections; and

,

calculating said pixel translation vector by scaling said spatial translation vector by said scaling factor.