METHOD AND APPARATUS FOR ESTIMATING THREE-DIMENSIONAL POSITION AND ORIENTATION THROUGH SENSOR FUSION

• US 20130028469A1
• Filed: 02/29/2012
• Published: 01/31/2013
• Est. Priority Date: 07/27/2011
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Lee, Hyong Euk, Kim, Sang Hyun, Choi, Chang Kyu
• Granted Patent: US 9,208,565 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

An apparatus and method of estimating a three-dimensional (3D) position and orientation based on a sensor fusion process. The method of estimating the 3D position and orientation may include determining a position of a marker in a two-dimensional (2D) image, determining a depth of a position in a depth image corresponding to the position of the marker in the 2D image to be a depth of the marker, estimating a 3D position of the marker calculated based on the depth of the marker as a marker-based position of a remote apparatus, estimating an inertia-based position and an inertia-based orientation by receiving inertial information associated with the remote apparatus, estimating a fused position based on a weighted sum of the marker-based position and the inertia-based position, and outputting the fused position and the inertia-based orientation.

Claims

1. An apparatus for estimating a position and an orientation, the apparatus comprising:

a marker determining unit to determine a position of a marker in a two-dimensional (2D) image;

a depth determining unit to determine, in a depth image, a position corresponding to the position of the marker in the 2D image, and to determine a depth of the corresponding position in the depth image to be a depth of the marker; and

a marker-based estimator to estimate, based on the depth of the marker, a marker-based position indicating a three-dimensional (3D) position of the marker.

2. The apparatus of claim 1, wherein the marker-based estimator determines a 2D position value excluding the depth of the marker, based on the depth of the marker, a field of view at which the 2D image and the depth image are photographed, and a distance from a predetermined reference position to the marker.

3. The apparatus of claim 1, wherein the depth determining unit calculates an average depth of a predetermined area set based on the corresponding position in the depth image, and determines the average depth to be the depth of the marker.

4. The apparatus of claim 3, wherein the predetermined area corresponds to an ambient area of the marker, which is an area excluding a central area affected by a light of the marker from a predetermined radius set, based on the corresponding position in the depth image.

5. The apparatus of claim 1, wherein:

the 2D image is a 2D image photographed when the marker is activated; and

the depth image is an image including depth information, of the same area as the 2D image, photographed when the marker is inactivated within a predetermined period of time after the 2D image is photographed.

6. The apparatus of claim 1, wherein:

the marker emits or reflects a light source; and

a photographed light source is discerned within the 2D image.

7. The apparatus of claim 1, wherein:

the marker emits or reflects an infrared light;

the 2D image is an infrared light image photographed, based on an infrared light; and

the depth image is an image including depth information, photographed based on an infrared light.

8. The apparatus of claim 1, further comprising:

an inertia-based estimator to estimate an inertia-based position and an inertia-based orientation of a remote apparatus, by receiving inertial information associated with the remote apparatus; and

a fusion estimator to estimate a fused position, based on a weighted-sum of the marker-based position and the inertia-based position.

9. The apparatus of claim 8, wherein, when the weighted-sum is calculated, the fusion estimator assigns a relatively higher weight to the inertia-based position than to the marker-based position, as a movement speed of the remote apparatus decreases.

10. The apparatus of claim 8, wherein, when the weighted-sum is calculated, the fusion estimator assigns a relatively higher weight to the inertia-based position than to the marker-based position, as a distance from the remote apparatus increases.

11. The apparatus of claim 8, wherein, when a movement speed of the remote apparatus is lower than a predetermined speed while the weighted-sum is being calculated, the fusion estimator estimates the inertia-based position as the fused position.

12. The apparatus of claim 8, wherein the inertia-based estimator estimates the inertia-based position and the inertia-based orientation based on a variance of inertial information, and the variance is determined based on a latest estimated fused position or a latest estimated marker-based position.

13. The apparatus of claim 8, wherein the fusion estimator performs:

estimating the marker-based position as the fused position at a point in time when the marker-based position exists; and

estimating the inertia-based position as the fused position at a point in time when the marker-based position is absent.

14. A method of estimating a position and an orientation, the method comprising:

determining a position of a marker in a two-dimensional (2D) image;

determining a position corresponding to the position of the marker, in a depth image;

determining a depth of the corresponding position in the depth image to be a depth of the marker; and

estimating, based on the depth of the marker, a marker-based position of a remote apparatus, which indicates a three-dimensional (3D) position of the marker.

15. The method of claim 14, wherein the estimating comprises:

determining the 3D position of the marker by calculating a 2D position value, excluding the depth of the marker, based on the depth of the marker, a field of view at which the 2D image and the depth image are photographed, and a distance from a predetermined reference position to the marker.

16. The method of claim 14, wherein the determining of the depth of the marker comprises:

calculating an average depth of a predetermined area set, based on the corresponding position in the depth image, and determining the average depth to be the depth of the marker.

17. The method of claim 16, wherein the predetermined area corresponds to an ambient area of the marker, which is an area excluding a central area affected by a light of the marker from a predetermined radius set based on the corresponding position in the depth image.

18. The method of claim 14, wherein:

the 2D image is a 2D image photographed when the marker is activated; and

the depth image is an image including depth information, of the same area as the 2D image, photographed when the marker is inactivated within a predetermined period of time after the 2D image is photographed.

19. The method of claim 14, further comprising:

estimating an inertia-based position and an inertia-based orientation of the remote apparatus, by receiving inertial information associated with the remote apparatus; and

estimating a fused position based on a weighted-sum of the marker-based position and the inertia-based position; and

outputting the fused position and the inertia-based orientation.

20. The method of claim 19, wherein, when the weighted-sum is calculated, assigning a relatively higher weight to the inertia-based position than to the marker-based position, as a movement speed of the remote apparatus decreases.

21. The method of claim 19, wherein, when the weighted-sum is calculated, assigning a relatively higher weight to the inertia-based position than to the marker-based position, as a distance from the remote apparatus increases.

22. A system for estimating a position and an orientation, the system comprising:

a transmitting apparatus comprising at least one marker generator to generate and output a marker;

a receiving apparatus to photograph a two-dimensional (2D) image and a depth image;

a remote apparatus comprising one of the transmitting apparatus and the receiving apparatus; and

an estimating apparatus to determine a position of the marker in the 2D image, to determine, in the depth image, a position corresponding to the position of the marker in the 2D image, determining a depth of the corresponding position in the depth image to be a depth of the marker, and to estimate, based on the depth of the marker, a marker-based position of the remote apparatus, which indicates a three-dimensional (3D) position of the marker.

23. The system of claim 22, wherein the receiving apparatus comprises a depth sensor that photographs the 2D image and the depth image.

24. The system of claim 22, wherein the receiving apparatus comprises an image sensor to photograph the 2D image and a depth sensor to photograph the depth image.

25. The system of claim 22, wherein the estimating apparatus determines a 2D position value excluding the depth of the marker, based on the depth of the marker, a field of view at which the 2D image and the depth image are photographed, and a distance from a predetermined reference position to the marker.

26. The system of claim 22, wherein the estimating apparatus calculates an average depth of a predetermined area set, based on the corresponding position in the depth image, and determines the average depth to be the depth of the marker.

27. The system of claim 22, wherein:

the 2D image is a 2D image photographed when the marker is activated; and

the depth image is an image including depth information, of the same area as the 2D image, photographed when the marker is inactivated within a predetermined period of time after the 2D image is photographed.

28. The system of claim 22, wherein:

the marker is an infrared light marker that emits or reflects an infrared light;

the 2D image is an infrared light image photographed, based on an infrared light; and

the depth image is an image including depth information, photographed based on an infrared light.

29. The system of claim 22, wherein:

the remote apparatus further comprises an inertia sensor unit to measure inertial information, and

the estimating apparatus estimates an inertia-based position and an inertia-based orientation of the remote apparatus, based on the inertial information, estimates a fused position of the remote apparatus by fusing the marker-based position and the inertia-based position; and

outputs the fused position and the inertia-based orientation of the remote apparatus.