SYSTEM AND METHOD FOR IMMERSIVE AND INTERACTIVE MULTIMEDIA GENERATION

• US 20160117860A1
• Filed: 10/26/2015
• Published: 04/28/2016
• Est. Priority Date: 10/24/2014
• Status: Active Grant

• Current Assignee: Shanghai YVR Technology Company Limited
• Original Assignee: uSens Inc.
• Inventors: FEI, Yue, HE, Anli
• Granted Patent: US 9,858,722 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 20/693 Acquisition
• CPC Class Codes: G06V 20/695 Preprocessing, e.g. image segmentation
• CPC Class Codes: G06V 20/698 Matching; Classification
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images

Abstract

An apparatus is disclosed. The apparatus comprises an optical sensing system that comprises at least one camera, the at least one camera being configured to acquire an image of a physical environment. The apparatus further comprises a processing system. The processing system comprises an orientation and position determination module configured to detect salient features from the image, and determine a change in orientation and/or position of the apparatus with respect to the physical environment based on the detected salient features. The processing system also comprises a rendering module configured to determine a rendering of the physical environment based on the image and on the determined change in orientation and/or position of the apparatus, and provide data related to the rendering of the physical environment to a display system.

Claims

1. An apparatus, comprising:

an optical sensing system comprising at least one camera, the at least one camera being configured to acquire an image of a physical environment;

a processing system comprising;

an orientation and position determination module configured to;

detect salient features from the image, and

determine a change in orientation and/or position of the apparatus with respect to the physical environment based on the detected salient features;

and

a rendering module configured to;

determine a rendering of the physical environment based on the image and on the determined change in orientation and/or position of the apparatus, and

provide data related to the rendering of the physical environment to a display system.

2. The apparatus of claim 1, wherein:

the optical sensing system comprises a first camera and a second camera, the first camera being configured to acquire a first image of the physical environment, and the second camera being configured to acquire a second image of the physical environment;

the orientation and position determination module is configured to;

detect first salient features from the first and second images when the apparatus is at a first position in the physical environment,

determine a first set of coordinates associated with the first salient features,

detect second salient features from the first and second images when the apparatus is at a second position in the physical environment,

determine a second set of coordinates associated with the second salient features, and

responsive to a determination that the first and second salient features correspond to each other, determine a change in orientation and/or position of the apparatus with respect to the physical environment based on the first and second set of coordinates;

and

the rendering module is configured to determine a stereoscopic rendering of the physical environment based on the first and second images and on the determined change in orientation and/or position of the apparatus.

3. The apparatus of claim 2, wherein each of the first and second cameras includes red-green-blue-infra-red (RGB-IR) pixel sensors.

4. The apparatus of claim 1, wherein the optical sensing system comprises a first camera, a second camera, and a mirror, the first camera being configured to acquire an infra-red (IR) image, the second camera being configured to acquire a red-green-blue (RGB) image, and the mirror being configured to reflect IR light away from the second camera and to align the IR image with the RGB image.

5. The apparatus of claim 4, wherein the first camera is a time-of-flight camera.

6. The apparatus of claim 4, wherein the first camera is an IR camera.

7. The apparatus of claim 1, wherein the optical sensing system comprises an IR illuminator configured to project one or more patterns into the physical environments;

wherein the detected salient features include the one or more patterns.

8. The apparatus of claim 1, wherein:

the processing system further comprises a hand gesture determination module configured to;

detect a hand gesture from the image, and

determine a user instruction associated with the hand gesture;

and

the rendering module is configured to determine the rendering of the physical environment based on the determined user instruction.

9. The apparatus of claim 8, wherein:

the hand gesture determination module is configured to determine that the user instruction is associated with a selection of an object being rendered; and

the rendering module is configured to determine a change in the rendering of the object based on the user instruction.

10. The apparatus of claim 9, wherein the object is a virtual object.

11. The apparatus of claim 9, wherein the object is a physical object present in the physical environment.

12. The apparatus of claim 1, wherein the rendering module is configured to blend an image of a virtual object with an image of a physical object located in the physical environment in the rendering of the physical environment;

wherein the blending is based on a first set of coordinates associated with the virtual object and a second set of coordinates associated with the physical object.

13. The apparatus of claim 12, wherein the blending includes rendering of audio data associated with the virtual object and audio signal originated from the physical environment.

14. The apparatus of claim 1, further comprising a connector configured to connect to a smart phone.

15. The apparatus of claim 14, wherein the smart phone comprises at least part of the processing system.

16. The apparatus of claim 14, wherein:

the smart phone is configured to provide additional content;

and the rendering module is configured to determine a rendering of the physical environment based on the image and the additional content.