System and method for immersive and interactive multimedia generation

• US 10,223,834 B2
• Filed: 12/14/2017
• Issued: 03/05/2019
• Est. Priority Date: 10/24/2014
• Status: Active Grant

• Current Assignee: Shanghai YVR Technology Company Limited
• Original Assignee: uSens Inc.
• Inventors: Fei, Yue, He, Anli
• Primary Examiner(s): Ge, Jin
• Application Number:
  US15/842,140
• Publication Number:
  US 20180108180A1
• Time in Patent Office: 446 Days
• Field of Search: 345419
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 20/693 Acquisition
• CPC Class Codes: G06V 20/695 Preprocessing, e.g. image segmentation
• CPC Class Codes: G06V 20/698 Matching; Classification
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images

Abstract

An apparatus comprising a processor and a non-transitory computer-readable storage medium storing instructions, when executed by the processor, cause the apparatus to perform a method comprising determining a first and second set of coordinates, associated with a first and second feature, respectively, in one or more first and second images, respectively, the first and second images are captured from a first and second position and/or orientation relative to the physical environment, respectively, re-projecting the first set of coordinates to one or more 2D spaces associated with the second images, comparing the re-projected first set of coordinates with the second set of coordinates in at least one of position closeness, feature closeness, or stereo constraints to determine a correspondence between the first and second features and determining a change between the first and second orientations and/or positions with respect to the physical environment based on the determined correspondence.

Claims

1. An apparatus, comprising a processor and a non-transitory computer-readable storage medium storing instructions that, when executed by the processor, cause the apparatus to perform a method, the method comprising:

determining a first set of coordinates associated with a first feature in one or more first images, wherein the first images are captured from a first position or a first orientation relative to a physical environment;

determining a second set of coordinates associated with a second feature in one or more second images, wherein the second images are captured from a second position or a second orientation relative to the physical environment;

re-projecting the first set of coordinates to one or more 2D spaces associated with the second images;

comparing the re-projected first set of coordinates with the second set of coordinates in at least one of position closeness, feature closeness, or stereo constraints to determine a correspondence between the first and second features;

determining a change between the first and second orientations or a change between the first and second positions with respect to the physical environment based on the determined corresponding first and second features;

determining a rendering of the physical environment based on the one or more first and second images and on the determined change in the orientation or position,

providing data related to the rendering of the physical environment to a display system;

wherein comparing the re-projected first set of coordinates with the second set of coordinates in at least one of the position closeness, feature closeness, or stereo constraints comprises determining a distance between locations of the re-projected first set of coordinates and locations of the second set of coordinates in the one or more 2D spaces; and

wherein determining the change between the first and second orientations or the change between the first and second positions with respect to the physical environment based on the determined corresponding first and second features comprises determining the change between the first and second orientations or the change between the first and second positions with respect to the physical environment based on minimizing the distance in the one or more 2D spaces.

2. The apparatus of claim 1, wherein:

the first and second sets of coordinates each comprise 3D coordinates; and

the first and second features each comprise at least one of;

a pre-existing marking in the physical environment or a projected IR (infrared) marking in the physical environment.

3. The apparatus of claim 1, further comprising a first camera and a second camera, wherein:

the one or more first images comprise a first left image and a first right image both captured at a first time;

the first left image is captured by the first camera;

the first right image is captured by the second camera;

the apparatus is in the first position or the first orientation relative to the physical environment at the first time;

the one or more second images comprise a second left image and a second right image both captured at a second time;

the second left image is captured by the first camera;

the second right image is captured by the second camera; and

the apparatus is in the second position or the second orientation relative to the physical environment at the second time.

4. The apparatus of claim 3, wherein:

the first set of coordinates are determined based on a distance between the first feature and the first and second cameras; and

the second set of coordinates are determined based on a distance between the second feature and the first and second cameras.

5. The apparatus of claim 3, wherein:

the first feature is obtained based on location difference minimization between the first left image captured by the first camera at the first time and the first right image captured by the second camera at the first time; and

the second feature is obtained based on location difference minimization between the second left image captured by the first camera at the second time and the second right image captured by the second camera at the second time.

6. The apparatus of claim 3, further comprising a mirror, the first camera being configured to acquire an infra-red (IR) image, the second camera being configured to acquire a red-green-blue (RGB) image, and the mirror being configured to reflect IR light away from the second camera and to align the IR image with the RGB image.

7. The apparatus of claim 1, wherein the method further comprises:

determining a hand gesture from at least one of the first or second images;

determining a user instruction associated with the hand gesture; and

determining a change in the rendering of the object based on the user instruction.

8. The apparatus of claim 1, wherein the method further comprises:

blending an image of a virtual object with an image of a physical object located in the physical environment in the rending of the physical environment, wherein the blending is based on a first set of coordinates associated with the virtual object and a second set of coordinates associated with the physical object.

9. A method comprising:

determining, by a processor, a first set of coordinates associated with a first feature in one or more first images, wherein the first images are captured from a first position or a first orientation relative to a physical environment;

determining, by the processor, a second set of coordinates associated with a second feature in one or more second images, wherein the second images are captured from a second position or a second orientation relative to the physical environment;

re-projecting, by the processor, the first set of coordinates to one or more 2D spaces associated with the second images;

comparing, by the processor, the re-projected first set of coordinates with the second set of coordinates in at least one of position closeness, feature closeness, or stereo constraints to determine a correspondence between the first and second features;

determining, by the processor, a change between the first and second orientations or a change between the first and second positions with respect to the physical environment based on the determined corresponding first and second features;

rendering, by the processor, of the physical environment based on the one or more first and second images and on the determined change in orientation or position; and

providing, by the processor, data related to the rendering of the physical environment to a display system,

wherein the comparing, by the processor, the re-projected first set of coordinates with the second set of coordinates in at least one of position closeness, feature closeness, or stereo constraints to determine a correspondence between the first and second features comprises determining a distance between locations of the re-projected first set of coordinates and locations of the second set of coordinates in the one or more 2D spaces; and

wherein the determining, by the processor, a change between the first and second orientations or a change between the first and second positions with respect to the physical environment based on the determined corresponding first and second features comprises determining the change between the first and second orientations or a change between the first and second positions with respect to the physical environment based on minimizing the distance in the one or more 2D spaces.

10. The method of claim 9, further comprising:

capturing, by a first camera, a first left image;

capturing, by a second camera, a first right image, wherein,

the one or more first images comprise the first left image and the first right image, and

the first left image and the first right image are captured at a first time in the first position or the first orientation relative to the physical environment at the first time;

capturing, by the first camera, a second left image; and

capturing, by the second camera, a second right image, wherein,

the one or more second images comprise the second left image and the second right image, and

the second left image and the second right image are captured at a second time in the second position or the second orientation relative to the physical environment at the second time.

11. The method of claim 10, wherein:

determining, by the processor, the first set of coordinates based on a distance between the first feature and the first and second cameras; and

determining, by the processor, the second set of coordinates based on a distance between the second feature and the first and second cameras.

12. The method of claim 11, wherein:

obtaining, by the processor, the first feature based on location difference between the first left image captured by the first camera at the first time and the first right image captured by the second camera at the first time; and

obtaining, by the processor, the second feature based on location difference between the second left image captured by the first camera at the second time and the second right image captured by the second camera at the second time.

13. The method of claim 11, further comprising:

determining a hand gesture from at least one of the first or second images;

determining a user instruction associated with the hand gesture;

determining that the user instruction is associated with a selection of an object being rendered; and

determining a change in the rendering of the object based on the user instruction.

14. The method of claim 9, further comprising:

blending, by the processor, an image of a virtual object with an image of a physical object located in the physical environment in the rending of the physical environment, wherein the blending is based on a first set of coordinates associated with the virtual object and a second set of coordinates associated with the physical.

15. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform a method, the method comprising:

determining a first set of coordinates associated with a first feature in one or more first images, wherein the first images are captured from a first position or a first orientation relative to a physical environment;

determining a second set of coordinates associated with a second feature in one or more second images, wherein the second images are captured from a second position or a second orientation relative to the physical environment;

re-projecting the first set of coordinates to one or more 2D spaces associated with the second images;

comparing the re-projected first set of coordinates with the second set of coordinates in at least one of position closeness, feature closeness, or stereo constraints to determine a correspondence between the first and second features;

determining a change between the first and second orientations or a change between the first and second positions with respect to the physical environment based on the determined corresponding first and second features;

determining a rendering of the physical environment based on the one or more first and second images and on the determined change in the orientation or position; and

providing data related to the rendering of the physical environment to a display system,

wherein the comparing the re-projected first set of coordinates with the second set of coordinates in at least one of position closeness, feature closeness, or stereo constraints to determine a correspondence between the first and second features comprises determining a distance between locations of the re-projected first set of coordinates and locations of the second set of coordinates in the one or more 2D spaces; and

wherein the determining the change between the first and second orientations or a change between the first and second positions with respect to the physical environment based on the determined corresponding first and second features comprises determining the change between the first and second orientations or a change between the first and second positions with respect to the physical environment based on minimizing the distance in the one or more 2D spaces.