System and method for immersive and interactive multimedia generation

• US 10,320,437 B2
• Filed: 03/03/2016
• Issued: 06/11/2019
• Est. Priority Date: 10/24/2014
• Status: Active Grant

• Current Assignee: Shanghai YVR Technology Company Limited
• Original Assignee: uSens Inc.
• Inventors: Fei, Yue, He, Anli
• Primary Examiner(s): Urban, Edward F
• Assistant Examiner(s): Hu, Rui M
• Application Number:
  US15/060,462
• Publication Number:
  US 20160261300A1
• Time in Patent Office: 1,195 Days
• Field of Search: 455566, 4555752, 345419, 345633
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G01B 11/22 for measuring depth
• CPC Class Codes: G01C 3/08 Use of electric radiation detectors
• CPC Class Codes: G01S 17/08 for measuring distance only indirect measurement G01S17/46; active triangulation systems G01S17/48
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0154 with movable elements
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04B 1/385 Transceivers carried on the body, e.g. in helmets
• CPC Class Codes: H04B 2001/3866 carried on the head
• CPC Class Codes: H04N 23/20 : for generating image signals from infrared radiation only
• CPC Class Codes: H04N 5/33 : Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04W 4/026 : using orientation information, e.g. compass
• CPC Class Codes: H04W 4/70 : Services for machine-to-machine communication [M2M] or machine type communication [MTC]

Abstract

A foldable apparatus is disclosed. The apparatus may comprise at least one camera configured to acquire an image of a physical environment, an orientation and position determination module configured to determine a change in orientation and/or position of the apparatus with respect to the physical environment based on the acquired image, a housing configured to hold the at least one camera and the orientation and position determination module, and a first strap attached to the housing and configured to attach the housing to a head of a user of the apparatus.

Claims

1. A foldable apparatus, comprising:

at least one camera configured to acquire an image of a physical environment;

an orientation and position determination module configured to determine a change in orientation and/or position of the apparatus with respect to the physical environment based on the acquired image, to determine depth information of a virtual object based on a pre-determined location of the virtual object and a location of the at least one camera, and to determine depth information of a physical object based on a pre-determined location of the physical object and the location of the at least one camera, and wherein the depth information of the virtual object is associated with a pixel of a first image of the virtual object and the depth information of the physical object is associated with a pixel of a second image of the physical object;

a graphics and audio rendering module configured to;

determine rendering of a visual image including the virtual object and the physical object in the physical environment based on their depth information, wherein the depth information of the virtual object is compared with the depth information of the physical object in the physical environment to render the pixel of the first image or the pixel of the second image based on the comparison result; and

determine a virtual audio source in the physical environment, and based on a distance between the virtual audio source and the physical object in the physical environment, adjust rendering of audio;

a housing configured to hold the at least one camera and the orientation and position determination module; and

a first strap attached to the housing and configured to attach the housing to a head of a user of the apparatus.

2. The apparatus of claim 1, wherein:

the at least one camera is further configured to monitor, in real-time, positions of the user relative to objects in the physical environment; and

the orientation and position determination module is further configured to;

determine, based on the monitored positions, if the user will collide with one of the objects in the physical environment, and

provide instructions to display a warning overlaying a rendering of the physical environment.

3. The apparatus of claim 1, wherein:

the at least one camera is further configured to monitor, in real-time, the physical object in the physical environment; and

the orientation and position determination module is further configured to;

generate a 3D model of the physical environment, the 3D model including a position of the physical object; and

provide instructions to display a virtual object at the position in the rendering of the physical environment.

4. The apparatus of claim 1, wherein the housing comprises a detachable back plate to enclose the first strap inside the housing, when the apparatus is folded.

5. The apparatus of claim 4, further comprising a second strap attached to the housing and configured to:

attach the housing to a head of a user of the apparatus, when the apparatus is unfolded; and

attach the back plate to the housing to fold the apparatus.

6. The apparatus of claim 4, wherein at least one of the back plate or the first strap comprises:

a battery; and

at least one of a charging contact point or a wireless charging receiving circuit to charge the battery.

7. The apparatus of claim 1, wherein the housing comprises a mobile phone fixture to hold a mobile phone inside the housing.

8. The apparatus of claim 1, wherein the housing comprises:

a foldable face support attached to the housing; and

a foldable face cushion attached to the foldable face support, wherein the foldable face cushion in configured to lean the housing against the user'"'"'s face.

9. The apparatus of claim 8, wherein the foldable face support comprises a spring support.

10. The apparatus of claim 8, wherein the foldable face support is a bendable material.

11. The apparatus of claim 8, wherein:

the foldable face support is inflated by a micro air-pump, when the apparatus is unfolded; and

the foldable face support is deflated by the micro air-pump, when the apparatus is folded.

12. The apparatus of claim 1, further comprising at least one of a gyroscope, an accelerometer, or a magnetometer, held by the housing.

13. The apparatus of claim 1, further comprising a hand gesture determination module configured to detect a hand gesture from the acquired image and held by the housing.

14. The apparatus of claim 13, wherein the graphics and audio rendering module is further configured to adjust rendering of audio based on the detected hand gesture.

15. The apparatus of claim 1, wherein the housing comprises a front plate, the front plate comprising openings.

16. The apparatus of claim 15, further comprising at least two cameras and an infrared emitter held by the housing, the at least two cameras and the infrared emitter monitoring the physical environment through the openings.

17. The apparatus of claim 16, further comprising at least two lenses corresponding to the two cameras.

18. The apparatus of claim 17, further comprising a slider configured to adjust at least one of a distance between the at least two cameras, a distance between the openings, or a distance between the at least two lenses, to match with the user'"'"'s inter-pupil distances.

19. The apparatus of claim 16, further comprising a display screen to display a rendering of the physical environment.

20. The apparatus of claim 19, further comprising a focus adjustment knob configured to adjust a distance between the at least two lenses and the display screen.

21. The apparatus of claim 1, wherein the housing further comprises a decoration plate to cover the openings, when the apparatus is not in use.