Integrated gestural interaction and multi-user collaboration in immersive virtual reality environments

• US 10,437,347 B2
• Filed: 06/22/2018
• Issued: 10/08/2019
• Est. Priority Date: 06/26/2014
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Ultrahaptics Ip Two Limited (Ultraleap Holdings Ltd.)
• Inventors: Holz, David S., Perez, Matias, Onofrio, Davide
• Primary Examiner(s): Nguyen, Jennifer T
• Application Number:
  US16/016,463
• Publication Number:
  US 20190056791A1
• Time in Patent Office: 473 Days
• Field of Search: 345 8, 345156, 345157, 345419
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0181 Adaptation to the pilot/driver
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths

Abstract

The technology disclosed relates to tracking motion of a wearable sensor system using a combination a RGB (red, green, and blue) and IR (infrared) pixels of one or more cameras. In particular, it relates to capturing gross features and feature values of a real world space using RGB pixels and capturing fine features and feature values of the real world space using IR pixels. It also relates to enabling multi-user collaboration and interaction in an immersive virtual environment. It also relates to capturing different sceneries of a shared real world space from the perspective of multiple users. It further relates to sharing content between wearable sensor systems. In further relates to capturing images and video streams from the perspective of a first user of a wearable sensor system and sending an augmented version of the captured images and video stream to a second user of the wearable sensor system.

Claims

1. A system for creating a multi-user interactive virtual environment, the system including:

a plurality of wearable sensor systems for displaying a virtual reality to a user wearing the system, including first wearable sensor system engaged by a first user and a second wearable sensor system engaged by a second user;

a plurality of cameras, wherein at least one camera is electronically coupled to each of a wearable sensor system of the plurality of wearable sensor systems; and

one or more processors coupled to a memory storing instructions for creating a multi-user interactive virtual environment, which instructions when executed by one or more processors perform;

capturing a first video stream of a real world space using at least one camera electronically coupled to the first wearable sensor system engaged by a first user;

capturing a second video stream of a real world space using at least one camera electronically coupled to the second wearable sensor system engaged by a second user;

using images from the first and second video streams captured by the camera coupled to the first wearable sensor system and by the camera coupled to the second wearable sensor system, generating respective three-dimensional maps of a real world space by extracting one or more feature values from the real world space from the first and second video streams;

determining motion information of the first and second wearable sensor systems with respect to each other based on comparison between the respective three-dimensional maps of the real world space;

extracting body portion movement information captured by the first and second wearable sensor system with respect to a moving reference frame, including;

repeatedly determining movement information for the wearable sensor system and at least one body portion at successive times to form a sequence of movement information; and

analyzing the sequence of movement information formed to determine a path of the body portion;

tracking movement of the body portion along the path of the body portion over a region of the real world space;

comparing the path tracked to a plurality of path templates and identifying a template that best matches the path; and

using the template that matches the path to obtain control information to control an external system.

2. The system of claim 1, wherein the instructions for generating respective three-dimensional maps further includes instructions that when executed perform determining a graph of features of the real world space based on the one or more feature values extracted.

3. The system of claim 1, further including instructions that when executed perform generating for display, across respective interfaces of the first and second wearable sensor systems, presentation outputs based on information from the images from the first and second video streams captured by the camera coupled to the first wearable sensor system and the camera coupled to the second wearable sensor system.

4. The system of claim 3, further including instructions that when executed perform automatically calibrating responsiveness of the presentation outputs based on the motion information of the first and second wearable sensor systems with respect to each other.

5. The system of claim 4, further including instructions that when executed perform proportionally adjusting respective perceived fields of view of the presentation outputs responsive to the motion information of the first and second wearable sensor systems with respect to each other.

6. The system of claim 1, further including instructions that when executed perform sharing content between wearable sensor systems;

wherein the first video stream of a real world space is captured at a time t0 using at least one camera electronically coupled to a first wearable sensor system engaged by a first user, wherein the first video stream is captured at a field of view of the first user;

wherein the second video stream of the real world space is captured at the time t0 using at least one camera electronically coupled to the first wearable sensor system, wherein the second video stream is captured at a field of view of the camera; and

establishing a communication channel between the first wearable sensor system and a second wearable sensor system and transmitting the second video stream to the second wearable sensor system.

7. The system of claim 6, wherein the field of view of the at least one camera substantially overlaps with the field of view of a user of the at least one camera.

8. The system of claim 6, wherein the field of view of the at least one camera encompasses and exceeds the field of view of a user of the at least one camera.

9. The system of claim 6, wherein the field of view of the at least one camera narrows and deceeds the field of view of a user of the at least one camera.

10. The system of claim 6, wherein the field of view of the at least one camera is separate and additional to the field of view of a user of the at least one camera.

11. The system of claim 6, further including using short-beam illumination elements to capture a narrow-field of view.

12. A method for creating a multi-user interactive virtual environment, the method including:

capturing a first video stream of a real world space using at least one camera electronically coupled to the first wearable sensor system engaged by a first user;

capturing a second video stream of a real world space using at least one camera electronically coupled to the second wearable sensor system engaged by a second user;

using images from the first and second video streams captured by the camera coupled to the first wearable sensor system and by the camera coupled to the second wearable sensor system, generating respective three-dimensional maps of a real world space by extracting one or more feature values from the real world space from the first and second video streams;

determining motion information of the first and second wearable sensor systems with respect to each other based on comparison between the respective three-dimensional maps of the real world space;

extracting body portion movement information captured by the first and second wearable sensor system with respect to a moving reference frame, including;

repeatedly determining movement information for the wearable sensor system and at least one body portion at successive times to form a sequence of movement information; and

analyzing the sequence of movement information formed to determine a path of the body portion;

tracking movement of the body portion along the path of the body portion over a region of the real world space;

comparing the path tracked to a plurality of path templates and identifying a template that best matches the path; and

using the template that matches the path to obtain control information to control an external system.

13. The method of claim 12, further including:

sharing content between wearable sensor systems;

wherein the first video stream of a real world space is captured at a time t0 using at least one camera electronically coupled to a first wearable sensor system engaged by a first user, wherein the first video stream is captured at a field of view of the first user;

wherein the second video stream of the real world space is captured at the time t0 using at least one camera electronically coupled to the first wearable sensor system, wherein the second video stream is captured at a field of view of the camera; and

establishing a communication channel between the first wearable sensor system and a second wearable sensor system and transmitting the second video stream to the second wearable sensor system.

14. A non-transitory computer readable medium storing instructions for creating a multi-user interactive virtual environment, which instructions when executed by one or more processors perform:

capturing a first video stream of a real world space using at least one camera electronically coupled to the first wearable sensor system engaged by a first user;

capturing a second video stream of a real world space using at least one camera electronically coupled to the second wearable sensor system engaged by a second user;

using images from the first and second video streams captured by the camera coupled to the first wearable sensor system and by the camera coupled to the second wearable sensor system, generating respective three-dimensional maps of a real world space by extracting one or more feature values from the real world space from the first and second video streams;

determining motion information of the first and second wearable sensor systems with respect to each other based on comparison between the respective three-dimensional maps of the real world space;

extracting body portion movement information captured by the first and second wearable sensor system with respect to a moving reference frame, including;

repeatedly determining movement information for the wearable sensor system and at least one body portion at successive times to form a sequence of movement information; and

analyzing the sequence of movement information formed to determine a path of the body portion;

tracking movement of the body portion along the path of the body portion over a region of the real world space;

comparing the path tracked to a plurality of path templates and identifying a template that best matches the path; and

using the template that matches the path to obtain control information to control an external system.

15. The non-transitory computer readable medium of claim 14, further including instructions for:

sharing content between wearable sensor systems;

wherein the first video stream of a real world space is captured at a time t0 using at least one camera electronically coupled to a first wearable sensor system engaged by a first user, wherein the first video stream is captured at a field of view of the first user;

wherein the second video stream of the real world space is captured at the time t0 using at least one camera electronically coupled to the first wearable sensor system, wherein the second video stream is captured at a field of view of the camera; and

establishing a communication channel between the first wearable sensor system and a second wearable sensor system and transmitting the second video stream to the second wearable sensor system.