User interface for integrated gestural interaction and multi-user collaboration in immersive virtual reality environments

• US 10,353,532 B1
• Filed: 02/19/2015
• Issued: 07/16/2019
• Est. Priority Date: 12/18/2014
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Holz, David S., Fox, Barrett, Hay, Kyle A., Hare, Gabriel A., Yu, Wilbur Yung Sheng, Edelhart, Dave, Medich, Jody, Plemmons, Daniel
• Primary Examiner(s): McDowell, Jr., Maurice L.
• Application Number:
  US14/626,820
• Time in Patent Office: 1,608 Days
• Field of Search: 345619, 345629-641, 345156, 715764, 715810, 715830, 715833, 715835, 715848, 715863
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

The technology disclosed relates to user interfaces for controlling augmented reality environments. Real and virtual objects can be seamlessly integrated to form an augmented reality by tracking motion of one or more real objects within view of a wearable sensor system using a combination a RGB (red, green, and blue) and IR (infrared) pixels of one or more cameras. It also relates to enabling multi-user collaboration and interaction in an immersive virtual environment. In particular, it relates to capturing different sceneries of a shared real world space from the perspective of multiple users. The technology disclosed further relates to sharing content between wearable sensor systems. In particular, it relates to capturing images and video streams from the perspective of a first user of a wearable sensor system and sending an augmented version of the captured images and video stream to a second user of the wearable sensor system.

Claims

1. A method of tracking motion of a wearable sensor system, the method including:

capturing a video stream of a scene of a real world space using at least one camera electronically coupled to the wearable sensor system;

detecting, from the captured video stream, (i) a hand of a user, (ii) multiple electronic devices included in the scene of the real world space, and (iii) one or more features of the detected hand;

determining at least one point on the hand to which a virtual device including a contextual menu can be affixed, the virtual device providing a virtual interface for interacting with the multiple electronic devices included in the scene of the real world space;

generating for display, a presentation output based on information from the at least one camera, the presentation output including (i) a visual rendering of the hand and (ii) at least one instance of the virtual device affixed to the visual rendering of the hand, wherein the contextual menu included in the virtual device includes menu items facilitating a control interface for changing operational modes of the multiple electronic devices included in the scene of the real world space; and

responsive to a selection of one of the menu items of the contextual menu by a detected finger gesture to select an operation of a particular electronic device of the multiple electronic devices, (i) updating the contextual menu included in the virtual device to include menu items specifically for the particular electronic device and (ii) detecting, from the captured video stream, a finger gesture made freely in 3D space and determining that the finger gesture detected from the captured video stream and made freely in the 3D space indicates a request to interact with the particular electronic device of the multiple electronic devices included in the scene of the real world space.

2. The method of claim 1, wherein the presentation output is generated for display across an interface of the wearable sensor system.

3. The method of claim 2, further including automatically sizing the virtual device to fit the presentation output based on determined motion information of the wearable sensor system with respect to at least one feature of the detected one or more features of the hand.

4. The method of claim 3, wherein the automatic sizing of the virtual device further includes adjusting a size of the virtual device in relation to a size of the hand.

5. The method of claim 1, wherein the one or more features of the hand also includes one or more features of a body portion engaged with the wearable sensor system.

6. The method of claim 1, wherein the contextual menu included in the virtual device corresponds to a unique control feature that controls a specific operation of the particular electronic device.

7. The method of claim 1, wherein the virtual device includes an ornamentation.

8. The method of claim 1, wherein the virtual device includes a scroll bar.

9. The method of claim 1, wherein the virtual device includes a knob.

10. The method of claim 1, wherein the virtual device includes a slider.

11. The method of claim 1, wherein the virtual device includes a button.

12. The method of claim 1, wherein the virtual device includes a virtual object indicating a command input.

13. A method of providing an augmented reality environment, the method including:

capturing a position of a user body portion and multiple electronic devices within a field of view of one or more cameras;

rendering for display, a representation of the user body portion;

determining at least one point on the rendered representation of the user body portion to which a virtual device including a contextual command input can be affixed;

integrating the virtual device including the contextual command input onto the rendered representation of the user body portion; and

generating for display, a presentation output based on information from the one or more cameras, the presentation output including (i) a visual rendering of the user body portion and (ii) at least one instance of the virtual device affixed to the visual rendering of the user body portion, wherein the contextual command input included in the virtual device facilitates a control interface for changing operational modes of the multiple electronic devices; and

responsive to a selection of one item of the contextual command input by a detected finger gesture to select an operation of a particular electronic device of the multiple electronic devices, (i) updating the contextual command input included in the virtual device to include items specifically for the particular electronic device and (ii) detecting, from a captured video stream, a finger gesture made freely in 3D space and determining that the finger gesture detected from the captured video stream and made freely in the 3D space indicates a request to interact with the particular electronic device of the multiple electronic devices included in the field of view.

14. The method of claim 13, wherein the integrating includes:

adjusting a size of the virtual device to correspond to a size of the rendered representation of the user body portion.

15. The method of claim 14, wherein the adjusting includes:

sizing the virtual device to maintain a fixed relationship between the size of the virtual device and the size of the rendered representation of the user body portion.

16. The method of claim 13, wherein the determining of the at least one point includes:

determining, based at least upon an identity of the user body portion, the at least one point to which the virtual device can be affixed.

17. The method of claim 13, wherein the one or more cameras capture the position of the user body portion in at least one of an RGB mode with ambient illumination and an infrared (IR) mode when an IR source is activated.

18. A system, comprising:

a sensory system including one or more optical sensors configured to sense information of a real world space;

a processing system configured to (i) determine, from the information sensed by the sensory system, at least one of a position and a motion of a hand of a user, (ii) determine at least one point on the hand to which a virtual device including a contextual menu item can be affixed for interacting with multiple electronic devices included in a scene of the real world space, (iii) detect multiple electronic devices included in a scene of the real world space and (iv) detect, from a captured video stream of the scene of the real world space, a finger gesture made freely in 3D space; and

a wearable rendering subsystem configured to display a presentation including (i) a visual rendering of the hand and (ii) at least one instance of the virtual device affixed to the visual rendering of the hand, wherein the contextual menu item included in the virtual device includes menu items facilitating a control interface for changing operational modes of the multiple electronic devices included in the scene of the real world space,

wherein, the processing system and the wearable rendering subsystem, responsive to a selection of one of the menu items of the contextual menu by a detected finger gesture to select an operation of a particular electronic device of the multiple electronic devices, (i) updating the contextual menu included in the virtual device to include menu items specifically for the particular electronic device and (ii) detecting, from the captured video stream, a finger gesture made freely in 3D space and determining that the finger gesture detected from the captured video stream and made freely in the 3D space indicates a request to interact with the particular electronic device of the multiple electronic devices included in the scene of the real world space.

19. A non-transitory computer readable storage medium impressed with computer program instructions to track motion of a wearable sensor system, the instructions, when executed on a processor, implement a method comprising:

capturing a video stream of a scene of a real world space using at least one camera electronically coupled to the wearable sensor system;

detecting, from the captured video stream, (i) a hand of a user, (ii) multiple electronic devices included in the scene of the real world space, and (iii) one or more features of the detected hand;

determining at least one point on the hand to which a virtual device including a contextual menu item can be affixed, the virtual device providing a virtual interface for interacting with the multiple electronic devices included in the scene of the real world space;

generating for display, a presentation output based on information from the at least one camera, the presentation output including (i) a visual rendering of the hand and (ii) at least one instance of the virtual device affixed to the visual rendering of the hand, wherein the contextual menu item included in the virtual device is determined so as to facilitate a control interface for changing operational modes of the multiple electronic devices included in the scene of the real world space; and

responsive to a selection of one of the menu items of the contextual menu by a detected finger gesture to select an operation of a particular electronic device of the multiple electronic devices, (i) updating the contextual menu included in the virtual device to include menu items specifically for the particular electronic device and (ii) detecting, from the captured video stream, a finger gesture made freely in 3D space and determining that the finger gesture detected from the captured video stream and made freely in the 3D space indicates a request to interact with the particular electronic device of the multiple electronic devices included in the scene of the real world space.