System and method for immersive and interactive multimedia generation

• US 9,858,722 B2
• Filed: 10/26/2015
• Issued: 01/02/2018
• Est. Priority Date: 10/24/2014
• Status: Active Grant

• Current Assignee: Shanghai YVR Technology Company Limited
• Original Assignee: uSens Inc.
• Inventors: Fei, Yue, He, Anli
• Primary Examiner(s): Ge, Jin
• Application Number:
  US14/923,073
• Publication Number:
  US 20160117860A1
• Time in Patent Office: 799 Days
• Field of Search: 345419
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 20/693 Acquisition
• CPC Class Codes: G06V 20/695 Preprocessing, e.g. image segmentation
• CPC Class Codes: G06V 20/698 Matching; Classification
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images

Abstract

An apparatus is disclosed. The apparatus comprises an optical sensing system that comprises at least one camera, the at least one camera being configured to acquire an image of a physical environment. The apparatus further comprises a processing system. The processing system comprises an orientation and position determination module configured to detect salient features from the image, and determine a change in orientation and/or position of the apparatus with respect to the physical environment based on the detected salient features. The processing system also comprises a rendering module configured to determine a rendering of the physical environment based on the image and on the determined change in orientation and/or position of the apparatus, and provide data related to the rendering of the physical environment to a display system.

Claims

1. An apparatus, comprising:

an optical sensing system comprising at least one camera, the at least one camera being configured to;

acquire one or more first images of a physical environment when the apparatus is at a first orientation and/or position in the physical environment; and

acquire one or more second images of the physical environment when the apparatus is at a second orientation and/or position in the physical environment;

a processing system comprising;

an orientation and position determination module configured to;

detect one or more first salient features from the one or more first images,

determine a first set of coordinates associated with the first salient features,

detect one or more second salient features from the one or more second images,

determine a second set of coordinates associated with the second salient features,

re-project the first set of coordinates to one or more 2D spaces associated with the one or more second images,

compare the re-projected first set of coordinates with the second set of coordinates in position closeness, feature closeness, and stereo constraints in the one or more 2D spaces to determine the one or more second salient features that correspond to the one or more first salient features, and

determine a change between the first and second orientations and/or positions of the apparatus with respect to the physical environment based on the determined corresponding first and second detected salient features;

and

a rendering module configured to;

determine a rendering of the physical environment based on the one or more first and second images and on the determined change in orientation and/or position of the apparatus, and

provide data related to the rendering of the physical environment to a display system.

2. The apparatus of claim 1, wherein:

the optical sensing system comprises a first camera and a second camera, the first camera being configured to acquire a first left image and a second left image of the physical environment, and the second camera being configured to acquire a first right image and a second right image of the physical environment, the first images comprising the first left image and the first right image, and the second images comprising the second left image and the second right image;

the orientation and position determination module is configured to;

responsive to a determination that the first and second salient features correspond to each other, determine the change in orientation and/or position of the apparatus with respect to the physical environment based on the first and second set of coordinates;

and

the rendering module is configured to determine a stereoscopic rendering of the physical environment based on the first and second images and on the determined change in orientation and/or position of the apparatus.

3. The apparatus of claim 2, wherein each of the first and second cameras includes red-green-blue-infra-red (RGB-IR) pixel sensors.

4. The apparatus of claim 1, wherein the optical sensing system comprises a first camera, a second camera, and a mirror, the first camera being configured to acquire an infra-red (IR) image, the second camera being configured to acquire a red-green-blue (RGB) image, and the mirror being configured to reflect IR light away from the second camera and to align the IR image with the RGB image.

5. The apparatus of claim 4, wherein the first camera is a time-of-flight camera.

6. The apparatus of claim 4, wherein the first camera is an IR camera.

7. The apparatus of claim 1, wherein the optical sensing system comprises an IR illuminator configured to project one or more patterns into the physical environments;

wherein the detected salient features include the one or more patterns.

8. The apparatus of claim 1, wherein:

the processing system further comprises a hand gesture determination module configured to;

detect a hand gesture from the one or more images, and

determine a user instruction associated with the hand gesture; and

the rendering module is configured to determine the rendering of the physical environment based on the determined user instruction.

9. The apparatus of claim 8, wherein:

the hand gesture determination module is configured to determine that the user instruction is associated with a selection of an object being rendered; and

the rendering module is configured to determine a change in the rendering of the object based on the user instruction.

10. The apparatus of claim 9, wherein the object is a virtual object.

11. The apparatus of claim 9, wherein the object is a physical object present in the physical environment.

12. The apparatus of claim 1, wherein the rendering module is configured to blend an image of a virtual object with an image of a physical object located in the physical environment in the rendering of the physical environment;

wherein the blending is based on a first set of coordinates associated with the virtual object and a second set of coordinates associated with the physical object.

13. The apparatus of claim 12, wherein the blending includes rendering of audio data associated with the virtual object and audio signal originated from the physical environment.

14. The apparatus of claim 1, further comprising a connector configured to connect to a smart phone.

15. The apparatus of claim 14, wherein the smart phone comprises at least part of the

processing system.

16. The apparatus of claim 14, wherein:

the smart phone is configured to provide additional content;

and the rendering module is configured to determine a rendering of the physical environment based on the one or more images and the additional content.

17. A system, comprising:

a processor; and

a non-transitory computer-readable storage medium storing instructions that, when executed by the processor, cause the system to;

acquire one or more first images of a physical environment when the system is at a first orientation and/or position in the physical environment,

detect one or more first salient features from the one or more first images,

determine a first set of coordinates associated with the first salient features,

acquire one or more second images of the physical environment when the apparatus is at a second orientation and/or position in the physical environment,

detect one or more second salient features from the one or more second images,

determine a second set of coordinates associated with the second salient features,

re-project the first set of coordinates to one or more 2D spaces associated with the one or more second images,

compare the re-projected first set of coordinates with the second set of coordinates in position closeness, feature closeness, and stereo constraints in the one or more 2D spaces to determine the one or more second salient features that correspond to the one or more first salient features,

determine a change between the first and second orientations and/or positions of the apparatus with respect to the physical environment based on the determined corresponding first and second detected salient features,

determine a rendering of the physical environment based on the one or more first and second images and on the determined change in orientation and/or position of the apparatus, and

provide data related to the rendering of the physical environment to a display system.

18. The system of claim 17, wherein the instructions further cause the system to:

detect a hand gesture from the one or more images;

determine a user instruction associated with the hand gesture; and

determine the rendering of the physical environment based on the determined user instruction.

19. A method, comprising:

acquiring one or more first images of a physical environment when the system is at a first orientation and/or position in the physical environment;

detecting one or more first salient features from the one or more first images;

determining a first set of coordinates associated with the first salient features;

acquiring one or more second images of the physical environment when the apparatus is at a second orientation and/or position in the physical environment;

detecting one or more second salient features from the one or more second images,

determining a second set of coordinates associated with the second salient features,

re-projecting the first set of coordinates to one or more 2D spaces associated with the one or more second images,

comparing the re-projected first set of coordinates with the second set of coordinates in position closeness, feature closeness, and stereo constraints in the one or more 2D spaces to determine the one or more second salient features that correspond to the one or more first salient features,

determining a change between the first and second orientations and/or positions of the apparatus with respect to the physical environment based on the determined corresponding first and second detected salient features;

determining a rendering of the physical environment based on the one or more first and second images and on the determined change in orientation and/or position of the apparatus; and

providing data related to the rendering of the physical environment to a display system.

20. The method of claim 19, further comprising:

detecting a hand gesture from the one or more images;

determining a user instruction associated with the hand gesture; and

determining the rendering of the physical environment based on the determined user instruction.