System and method for immersive and interactive multimedia generation

• US 10,256,859 B2
• Filed: 03/03/2016
• Issued: 04/09/2019
• Est. Priority Date: 10/24/2014
• Status: Active Grant

• Current Assignee: Shanghai YVR Technology Company Limited
• Original Assignee: uSens Inc.
• Inventors: Fei, Yue, He, Anli
• Primary Examiner(s): Hoang, Phi
• Application Number:
  US15/060,429
• Publication Number:
  US 20160260260A1
• Time in Patent Office: 1,132 Days
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G01B 11/22 for measuring depth
• CPC Class Codes: G01C 3/08 Use of electric radiation detectors
• CPC Class Codes: G01S 17/08 for measuring distance only indirect measurement G01S17/46; active triangulation systems G01S17/48
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0154 with movable elements
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04B 1/385 Transceivers carried on the body, e.g. in helmets
• CPC Class Codes: H04B 2001/3866 carried on the head
• CPC Class Codes: H04N 5/33 : Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04W 4/026 : using orientation information, e.g. compass
• CPC Class Codes: H04W 4/70 : Services for machine-to-machine communication [M2M] or machine type communication [MTC]

Abstract

A method is disclosed. The method may be implemented by an immersive and interactive multimedia generation system. The method may comprise projecting, by a first component of the system, a plurality of patterns to a physical environment in which a second component of the system is located, acquiring, by the second component of the system, a first image and a second image of at least a part of the physical environment, the first and second images including, respectively, first and second pixel data corresponding to at least some of the plurality of projected patterns, and determining, by the second component of the system, a change in at least one of an orientation or a position of the second component within the physical environment based on a relationship between the first and the second pixel data.

Claims

1. A method, implemented by an immersive and interactive multimedia generation system, comprising:

projecting, by a first component of the system, a plurality of patterns to a physical environment in which a second component of the system is located;

acquiring, by a first detector of the second component of the system, a first image and a second image of at least a part of the physical environment, the first and second images including, respectively, first and second pixel data corresponding to at least some of the plurality of projected patterns; and

determining, by the second component of the system, a change in at least one of an orientation or a position of the first detector relative to the plurality of projected patterns based on a relationship between the first and the second pixel data, the first component and the plurality of projected patterns being static relative to the physical environment and the first detector being movable relative to the physical environment;

acquiring, by a second detector of the second component of the system, a third image and a fourth image of the at least a part of the physical environment; and

aligning, by the second component of the system, the first image with the third image, and the second image with the fourth image to generate a rendering of the at least a part of the physical environment,

wherein aligning the first image with the third image, and the second image with the fourth image comprises reflecting, at a location, the plurality of patterns before they are acquired by the first detector, and letting a visual light pass through, at the same location, before the visual light is acquired by the second detector.

2. The method of claim 1, wherein the physical environment is an indoor environment.

3. The method of claim 1, wherein:

the first component is disposed at a fixed position; and

the second component is attached to a user moving in the physical environment.

4. The method of claim 1, wherein the patterns are infrared patterns and the second component has no prior knowledge of the patterns.

5. The method of claim 1, further comprising:

detecting a hand gesture from the first and second images;

determining a user instruction associated with the hand gesture; and

determining a rendering of the physical environment based on the determined user instruction.

6. The method of claim 1, wherein the patterns comprise a turning angle.

7. The method of claim 6, wherein the patterns are “

L”

shaped.

8. The method of claim 1, further comprising:

monitoring, in real-time, positions of a user of the system relative to objects in the physical environment;

determining, based on the monitored positions, if the user will collide with one of the objects in the physical environment; and

providing instructions to display a warning overlaying a rendering of the physical environment to the user.

9. The method of claim 1, further comprising:

monitoring, in real-time, objects in the physical environment;

generating a 3D model of the physical environment, the 3D model including a position of a real world object; and

providing instructions to display a virtual object at the position of the real world object in a rendering of the physical environment.

10. An immersive and interactive multimedia generation system, comprising:

a first apparatus configured to project a plurality of patterns to a physical environment in which a second component of the system is located; and

a second apparatus configured to;

acquire a first image and a second image of at least a part of the physical environment by a first detector of the second apparatus, the first and second images including, respectively, first and second pixel data corresponding to at least some of the plurality of projected patterns, and

determine a change in at least one of an orientation or a position of the second component relative to the plurality of projected patterns based on a relationship between the first and the second pixel data, the first component and the plurality of projected patterns being static relative to the physical environment and the first detector being movable relative to the physical environment;

acquire a third image and a fourth image of the at least a part of the physical environment by a second detector of the second apparatus; and

align the first image with the third image, and the second image with the fourth image to generate a rendering of the at least a part of the physical environment,

wherein the second apparatus is configured to align the first image with the third image, and the second image with the fourth image by reflecting, at a location, the plurality of patterns before they are acquired by the first detector of the second apparatus, and letting a visual light pass through, at the same location, before the visual light is acquired by the second detector of the second apparatus.

11. The system of claim 10, wherein the physical environment is an indoor environment.

12. The system of claim 10, wherein:

the first apparatus is disposed at a fixed position; and

the second apparatus is attached to a user moving in the physical environment.

13. The system of claim 10, wherein the patterns are infrared patterns and the second apparatus has no prior knowledge of the patterns.

14. The system of claim 10, wherein the second apparatus is further configured to:

detect a hand gesture from the first and second images;

determine a user instruction associated with the hand gesture; and

determine a rendering of the physical environment based on the determined user instruction.

15. The system of claim 10, wherein the patterns comprise a turning angle.

16. The system of claim 15, wherein the patterns are “

L”

shaped.

17. The system of claim 10, wherein the second apparatus is further configured to:

monitor, in real-time, positions of a user of the system relative to objects in the physical environment;

determine, based on the monitored positions, if the user will collide with one of the objects in the physical environment; and

provide instructions to display a warning overlaying a rendering of the physical environment to the user.

18. The system of claim 10, wherein the second apparatus is further configured to:

monitor, in real-time, objects in the physical environment;

generate a 3D model of the physical environment, the 3D model including a position of a real world object; and

provide instructions to display a virtual object at the position of the real world object in a rendering of the physical environment.