Method and apparatus for estimating three-dimensional position and orientation through sensor fusion

• US 9,208,565 B2
• Filed: 02/29/2012
• Issued: 12/08/2015
• Est. Priority Date: 07/27/2011
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Lee, Hyong Euk, Kim, Sang Hyun, Choi, Chang Kyu
• Primary Examiner(s): Heidemann, Jason
• Application Number:
  US13/408,654
• Publication Number:
  US 20130028469A1
• Time in Patent Office: 1,378 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

An apparatus and method of estimating a three-dimensional (3D) position and orientation based on a sensor fusion process. The method of estimating the 3D position and orientation may include determining a position of a marker in a two-dimensional (2D) image, determining a depth of a position in a depth image corresponding to the position of the marker in the 2D image to be a depth of the marker, estimating a 3D position of the marker calculated based on the depth of the marker as a marker-based position of a remote apparatus, estimating an inertia-based position and an inertia-based orientation by receiving inertial information associated with the remote apparatus, estimating a fused position based on a weighted sum of the marker-based position and the inertia-based position, and outputting the fused position and the inertia-based orientation.

Claims

1. An apparatus for estimating a position and an orientation, the apparatus comprising:

one or more processors configured to

determine a position of a marker in a two-dimensional (2D) image;

determine, in a depth image, a position corresponding to the position of the marker in the 2D image, and to determine a depth of the corresponding position in the depth image to be a depth of the marker; and

estimate, based on the depth of the marker, a marker-based position indicating a three-dimensional (3D) position of the marker,

wherein the one or more processors are further configured to determine a 2D position value excluding the depth of the marker, based on the depth of the marker, a field of view at which the 2D image and the depth image are photographed, and a distance from a predetermined reference position to the marker.

2. The apparatus of claim 1, wherein the one or more processors are further configured to calculate an average depth of a predetermined area set based on the corresponding position in the depth image, and determines the average depth to be the depth of the marker.

3. The apparatus of claim 2, wherein the predetermined area corresponds to an ambient area of the marker, which is an area excluding a central area affected by a light of the marker from a predetermined radius set, based on the corresponding position in the depth image.

4. The apparatus of claim 1, wherein:

the 2D image is a 2D image photographed when the marker is activated; and

the depth image is an image including depth information, of the same area as the 2D image, photographed when the marker is inactivated within a predetermined period of time after the 2D image is photographed.

5. The apparatus of claim 1, wherein:

the marker emits or reflects a light source; and

a photographed light source is discerned within the 2D image.

6. The apparatus of claim 1, wherein:

the marker emits or reflects an infrared light;

the 2D image is an infrared light image photographed, based on an infrared light; and

the depth image is an image including depth information, photographed based on an infrared light.

7. The apparatus of claim 1, wherein the one or more processors are further configured to

estimate an inertia-based position and an inertia-based orientation of a remote apparatus, by receiving inertial information associated with the remote apparatus;

and

estimate a fused position, based on a weighted-sum of the marker-based position and the inertia-based position.

8. The apparatus of claim 7, wherein, when the weighted-sum is calculated, the fusion estimator assigns a relatively higher weight to the inertia-based position than to the marker-based position, as a movement speed of the remote apparatus decreases.

9. The apparatus of claim 7, wherein, when the weighted-sum is calculated, the fusion estimator assigns a relatively higher weight to the inertia-based position than to the marker-based position, as a distance from the remote apparatus increases.

10. The apparatus of claim 7, wherein, when a movement speed of the remote apparatus is lower than a predetermined speed while the weighted-sum is being calculated, the fusion estimator estimates the inertia-based position as the fused position.

11. The apparatus of claim 7, wherein the one or more processors are further configured to estimate the inertia-based position and the inertia-based orientation based on a variance of inertial information, and the variance is determined based on a latest estimated fused position or a latest estimated marker-based position.

12. The apparatus of claim 7, wherein the one or more processors are further configured to:

estimate the marker-based position as the fused position at a point in time when the marker-based position exists; and

estimate the inertia-based position as the fused position at a point in time when the marker-based position is absent.

13. A method of estimating a position and an orientation, the method comprising:

determining a position of a marker in a two-dimensional (2D) image;

determining a position corresponding to the position of the marker, in a depth image;

determining a depth of the corresponding position in the depth image to be a depth of the marker; and

estimating, based on the depth of the marker, a marker-based position of a remote apparatus, which indicates a three-dimensional (3D) position of the marker,

wherein the estimating comprises determining the 3D position of the marker by calculating a 2D position value, excluding the depth of the marker, based on the depth of the marker, a field of view at which the 2D image and the depth image are photographed, and a distance from a predetermined reference position to the marker.

14. The method of claim 13, wherein the determining of the depth of the marker comprises:

calculating an average depth of a predetermined area set, based on the corresponding position in the depth image, and determining the average depth to be the depth of the marker.

15. The method of claim 14, wherein the predetermined area corresponds to an ambient area of the marker, which is an area excluding a central area affected by a light of the marker from a predetermined radius set based on the corresponding position in the depth image.

16. The method of claim 13, wherein:

the 2D image is a 2D image photographed when the marker is activated; and

the depth image is an image including depth information, of the same area as the 2D image, photographed when the marker is inactivated within a predetermined period of time after the 2D image is photographed.

17. The method of claim 13, further comprising:

estimating an inertia-based position and an inertia-based orientation of the remote apparatus, by receiving inertial information associated with the remote apparatus; and

estimating a fused position based on a weighted-sum of the marker-based position and the inertia-based position; and

outputting the fused position and the inertia-based orientation.

18. The method of claim 17, wherein, when the weighted-sum is calculated, assigning a relatively higher weight to the inertia-based position than to the marker-based position, as a movement speed of the remote apparatus decreases.

19. The method of claim 17, wherein, when the weighted-sum is calculated, assigning a relatively higher weight to the inertia-based position than to the marker-based position, as a distance from the remote apparatus increases.

20. A system for estimating a position and an orientation, the system comprising:

at least one marker generator to generate and output a marker;

one more sensors configured to photograph a two-dimensional (2D) image and a depth image;

and

one or more processors configured

to determine a position of the marker in the 2D image, to determine, in the depth image, a position corresponding to the position of the marker in the 2D image, determining a depth of the corresponding position in the depth image to be a depth of the marker, and to estimate, based on the depth of the marker, a marker-based position which indicates a three-dimensional (3D) position of the marker,

wherein the one or more processors are further configured to determine a 2D position value excluding the depth of the marker, based on the depth of the marker, a field of view at which the 2D image and the depth image are photographed, and a distance from a predetermined reference position to the marker.

21. The system of claim 20, wherein the one or more sensors comprises a depth sensor that photographs the 2D image and the depth image.

22. The system of claim 20, wherein the one or more sensors comprises an image sensor to photograph the 2D image and a depth sensor to photograph the depth image.

23. The system of claim 20, wherein the one or more processors are further configured to calculate an average depth of a predetermined area set, based on the corresponding position in the depth image, and determines the average depth to be the depth of the marker.

24. The system of claim 20, wherein:

the 2D image is a 2D image photographed when the marker is activated; and

the depth image is an image including depth information, of the same area as the 2D image, photographed when the marker is inactivated within a predetermined period of time after the 2D image is photographed.

25. The system of claim 20, wherein:

the marker is an infrared light marker that emits or reflects an infrared light;

the 2D image is an infrared light image photographed, based on an infrared light; and

the depth image is an image including depth information, photographed based on an infrared light.

26. The system of claim 20,

further comprising

an inertia sensor to measure inertial information, and

wherein the one or more processors are further configured to estimate an inertia-based position and an inertia-based orientation of the marker, based on the inertial information, estimates a fused position of the marker by fusing the marker-based position and the inertia-based position;

and outputs the fused position and the inertia-based orientation of the marker.