MOBILE DEVICE WITH THREE DIMENSIONAL AUGMENTED REALITY

• US 20130121559A1
• Filed: 11/16/2011
• Published: 05/16/2013
• Est. Priority Date: 11/16/2011
• Status: Abandoned Application

• Current Assignee: Sharp Laboratories of America Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Sharp Laboratories of America Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Inventors: HU, Xiaoyan, YUAN, Chang
• Application Number:
  US13/298,228
• Publication Number:
  US 20130121559A1
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

A method for determining an augmented reality scene by a mobile device includes estimating 3D geometry and lighting conditions of the sensed scene based on stereoscopic images captured by a pair of imaging devices. The device accesses intrinsic calibration parameters of a pair of imaging devices of the device independent of a sensed scene of the augmented reality scene. The device determines two dimensional disparity information of a pair of images from the device independent of a sensed scene of the augmented reality scene. The device estimates extrinsic parameters of a sensed scene by the pair of imaging devices, including at least one of rotation and translation. The device calculates a three dimensional image based upon a depth of different parts of the sensed scene based upon a stereo matching technique. The device incorporates a three dimensional virtual object in the three dimensional image to determine the augmented reality scene.

Claims

1. A method for determining an augmented reality scene by a mobile device comprising:

(a) said mobile device accessing intrinsic calibration parameters of a pair of imaging devices of said mobile device in a manner independent of a sensed scene of said augmented reality scene;

(b) said mobile device determining two dimensional disparity information of a pair of images from said mobile device based upon a stereo matching technique;

(c) said mobile device estimating extrinsic parameters of a sensed scene by said pair of imaging devices, including at least one of rotation and translation;

(d) said mobile device calculating a three dimensional image based upon a depth of different parts of a said sensed scene based upon a triangulation technique;

(e) said mobile device incorporating a three dimensional virtual object in said three dimensional image to determine said augmented reality scene.

2. The method of claim 1 wherein said mobile device estimates three dimensional geometry and lighting conditions of the sensed scene based on one or more stereoscopic images sensed by a pair of imaging devices.

3. The method of claim 1 wherein said calibration parameters are based upon sensing at least one calibration image.

4. The method of claim 1 wherein said calibration parameters characterize an image distortion of said pair of imaging devices.

5. The method of claim 1 wherein said calibration parameters characterize a focal length of said imaging devices.

6. The method of claim 1 wherein said calibration parameters characterize a center of an image.

7. The method of claim 1 wherein said calibration parameters are based upon a projective transformation.

8. The method of claim 1 wherein said calibration parameters include distortion.

9. The method of claim 8 wherein said distortion is radial distortion.

10. The method of claim 1 wherein said extrinsic parameters are based upon structure from motion process.

11. The method of claim 10 wherein said structure from motion process includes the use of feature points.

12. The method of claim 11 wherein said structure from motion process includes a bundle adjustment.

13. The method of claim 12 wherein said bundle adjustment is further based upon said intrinsic calibration parameters and an estimation of said extrinsic parameters.

14. The method of claim 1 wherein said stereo matching technique includes block matching of at least one stereoscopic image pair.

15. The method of claim 1 wherein said stereo matching technique includes sweeping a plane across said sensed scene based on multiple stereoscopic images.

16. The method of claim 15 wherein said stereo matching technique includes sweeping said plane in a direction along a principal axis of the reference camera.

17. The method of claim 1 wherein 1 wherein said mobile device provides information to a user of said mobile device in how to modify obtaining said sensed scene.

18. The method of claim 1 wherein said three dimensional virtual object is rendered on non-planar surfaces in the sensed scene.

19. The method of claim 1 wherein said three dimensional virtual object is partially occluded by said three dimensional image in said augmented reality scene.

20. The method of claim 1 wherein said three dimensional image is said augmented reality scene is partially occluded by said three dimensional virtual object.

21. The method of claim 1 wherein lighting included with said augmented reality scene is based upon estimated lighting of said three dimensional image which is used as the basis for said lighting for said three dimensional virtual object.

22. The method of claim 1 wherein said augmented reality scene is based upon said three dimensional virtual object being rendered based upon each of said imaging devices.