Apparatus and method for estimating camera pose

• US 9,916,689 B2
• Filed: 02/02/2015
• Issued: 03/13/2018
• Est. Priority Date: 02/17/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Rhee, Seon Min, Park, Seung In, Lee, Yong Beom, Lee, Hyong Euk
• Primary Examiner(s): Chow, Jeffrey
• Application Number:
  US14/611,824
• Publication Number:
  US 20150235378A1
• Time in Patent Office: 1,135 Days
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

An apparatus for estimating a camera pose includes an image acquisition unit to acquire a photographed image, a motion sensor to acquire motion information of the apparatus for estimating the camera pose, a static area detector to detect a static area of the photographed image based on the photographed image and the motion information, and a pose estimator to estimate a camera pose based on the detected static area.

Claims

1. An apparatus for estimating a camera pose, the apparatus comprising:

an image acquisition unit configured to acquire a first image and a second image;

a motion sensor configured to acquire motion information of the apparatus;

a processor implemented static area detector configured to

detect a position of a first point of the first image and a position of a second point of the second image,

determine a predicted position of the second point based on the detected position of the first point and the motion information, and

determine an area of the second image to be a static area in response to a difference between the predicted position of the second point and the detected position of the second point being less than a threshold; and

a pose estimator configured to estimate the camera pose based on the determined static area.

2. The apparatus of claim 1, wherein the first image is a first color image and the second image is a second color image.

3. The apparatus of claim 2, wherein the pose estimator is configured to:

match corresponding points between the first image and the second image; and

estimate a rotation value from the motion information.

4. The apparatus of claim 3, wherein the pose estimator is configured to calibrate the estimated rotation value to correspond to a disposition in which the camera is considered to be at a position identical to the motion sensor.

5. The apparatus of claim 3, wherein the pose estimator is configured to:

calculate an essential matrix based on a matching relationship between the matched corresponding points and the estimated rotation value; and

estimate a translation value based on the calculated essential matrix.

6. The apparatus of claim 5, wherein the static area detector is configured to determine the static area based on the estimated rotation value and the estimated translation value.

7. The apparatus of claim 6, wherein the static area detector is configured to:

apply the estimated rotation value and the estimated translation value to the first point of the first image; and

predict the position of the second point of the second image.

9. The apparatus of claim 1, wherein the image acquisition unit is configured to acquire a first depth image and a second depth image respectively as the first image and the second image.

10. The apparatus of claim 9, wherein the pose estimator is configured to select a pair of three-dimensional (3D) points from the first depth image and the second depth image that satisfies the motion information acquired from the motion sensor.

11. The apparatus of claim 10, wherein the pose estimator is configured to estimate a translation value based on the selected pair of 3D points.

12. The apparatus of claim 11, wherein the static area detector is configured to:

apply a rotation value and the translation value to a 3D point of the first depth image; and

estimate a predicted position of a 3D point of the second depth image.

13. The apparatus of claim 12, wherein the rotation value is estimated from the motion information.

8. An apparatus for estimating a camera pose, the apparatus comprising:

an image acquisition unit to acquire a photographed image;

a motion sensor to acquire motion information of the apparatus for estimating the camera pose;

a static area detector to detect a static area of the photographed image based on the photographed image and the motion information; and

a pose estimator to estimate the camera pose based on the detected static area,

wherein the image acquisition unit acquires a first color image and a second color image,

wherein the pose estimator matches corresponding points between the first color image and the second color image, estimates a rotation value from the motion information, calibrates the estimated rotation value to correspond to a disposition in which the camera is at a position identical to the motion sensor, calculates an essential matrix based on a matching relationship between the matched corresponding points and the estimated rotation value, estimates a translation value based on the calculated essential matrix,

wherein the static area detector applies the estimated rotation value and the estimated translation value to the corresponding point of the first color image, detects a predicted position of the corresponding point of the second color image, and detects an area in which a difference between the predicted position of the corresponding point of the second color image and an actual position of the corresponding point of the second color image is less than a predetermined threshold value to be the static area.

14. An apparatus for estimating a camera pose, the apparatus comprising:

an image acquisition unit to acquire a photographed image;

a motion sensor to acquire motion information of the apparatus for estimating the camera pose;

a static area detector to detect a static area of the photographed image based on the photographed image and the motion information; and

a pose estimator to estimate the camera pose based on the detected static area,

wherein the image acquisition unit acquires a first depth image and a second depth image,

wherein the pose estimator selects a pair of three-dimensional (3D) points from the first depth image and the second depth image that satisfies the motion information acquired from the motion sensor, and estimates a translation value based on the selected pair of 3D points, and

wherein the static area detector applies a rotation value and the translation value to a 3D point of the first depth image, and estimates a predicted position of a 3D point of the second depth image, and

wherein the static area detector detects an area in which a difference between the predicted position of the 3D point of the second depth image and an actual position of a 3D point of the second depth image is less than a predetermined threshold value to be the static area.

15. The apparatus of claim 14, wherein the rotation value is estimated from the motion information.

16. A processor implemented method of controlling an apparatus for estimating a camera pose, the method comprising:

detecting a position of a first point of an acquired first image and a position of a second point of an acquired second image;

determining a predicted position of the second point based on the detected position of the first point and acquired motion information from a motion sensor;

determining an area of the second image to be a static area in response to a difference between the predicted position of the second point and the detected position of the second point being less than a threshold; and

estimating the camera pose based on the determined static area.

17. The method of claim 16, wherein the

acquired first image and the acquired second image are

an image pair respectively as a first color image and a second color image, or

a depth image pair respectively as a first depth image and a second depth image.

18. The method of claim 17, wherein the estimating of the camera pose is based on:

corresponding points of the first color image and the second color image, or

a pair of three-dimensional (3D) points from the static area based on the first depth image and the second depth image.

19. The method of claim 16, further comprising:

outputting a control signal corresponding to the estimated camera pose.

20. The method of claim 19, further comprising:

executing an application that provides an augmented reality (AR) service,

wherein the outputting of the control signal corresponding to the estimated camera pose comprises;

providing an AR service corresponding to the estimated camera pose.

21. The method of claim 19, further comprising:

displaying a panoramic image having a size greater than a display screen of the apparatus for estimating the camera pose,

wherein the outputting of the control signal corresponding to the estimated camera pose comprises;

translating the displayed panoramic image corresponding to the estimated camera pose, and displaying the translated panoramic image.

22. The method of claim 21, wherein the translating of the displayed panoramic image corresponding to the estimated camera pose comprises:

translating the displayed panoramic image corresponding to translation information of the estimated camera pose.

23. The method of claim 16, further comprising

acquiring the first image and the second image captured by at least one camera of a device;

acquiring the motion information provided by the motion sensor configured to detect motion of the device.