IMAGE PROCESSING METHOD, PARTICULARLY USED IN A VISION-BASED LOCALIZATION OF A DEVICE

• US 20150245020A1
• Filed: 06/05/2013
• Published: 08/27/2015
• Est. Priority Date: 09/27/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Meier, Peter, Benhimane, Mohamed Selim
• Granted Patent: US 9,888,235 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01S 19/53 Determining attitude
• CPC Class Codes: G01S 5/0247 Determining attitude
• CPC Class Codes: G01S 5/163 Determination of attitude using inertial means G01C9/00; control of attitude G05D1/49
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/20101 Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/631 Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

An image processing method includes the steps of providing at least one image of at least one object or part of the at least one object, and providing a coordinate system in relation to the image, providing at least one degree of freedom in the coordinate system or at least one sensor data in the coordinate system, and computing image data of the at least one image or at least one part of the at least one image constrained or aligned by the at least one degree of freedom or the at least one sensor data.

Claims

1. An image processing method for vision-based localization of a device, comprising the steps of:

providing at least one image of at least one object or part of the at least one object, and providing a coordinate system in relation to the image;

providing at least one degree of freedom in the coordinate system or at least one sensor data in the coordinate system; and

computing image data of the at least one image or at least one part of the at least one image constrained or aligned by the at least one degree of freedom or the at least one sensor data.

2. The method according to claim 1, wherein the computed image data is a result of image processing including morphological image operations or image filtering.

3. The method according to claim 2, wherein the computing of image data is constrained or aligned according to degrees of freedom with a particular confidence degree or is constrained or aligned according to sensors providing data used in the image processing method in relation to the image or a device capturing the image.

4. The method according to claim 1, wherein the image processing method includes an image gradient computation which comprises filtering of gravity aligned gradients in the image.

5. The method according to claim 1, wherein the image processing method includes an image gradient computation which comprises image filtering with kernels, wherein the kernels are aligned with axes of the image or with a gravity vector.

6. The method according to claim 5, wherein in case the image plane of the image is fronto-parallel to the gravity vector, the same gravity aligned kernel is applied to the whole image.

7. The method according to claim 1, the method further including a process for matching at least one feature of the object in the at least one image with at least one feature of a digital representation of the object.

8. The method according to claim 1, wherein the image processing method is used to generate a distance transform image based on the result of the image processing method.

9. The method according to claim 1, further including running a global registration algorithm, where different degrees of freedom are iteratively refined and a quality of the iteration is measured by a predefined cost-function.

10. The method according to claim 1, wherein the image processing method is operated in relation to a device which is associated with a capturing device for capturing the at least one image, wherein the method further comprises the steps of capturing the at least one image with the capturing device and providing an estimation of intrinsic parameters of the capturing device.

11. The method according to claim 1, wherein the image processing method is operated in relation to a device which is associated with a capturing device for capturing the at least one image, wherein the method further comprises providing position and orientation data, and providing measurements of different degrees of freedom of the capturing device with respect to the coordinate system with respective confidence values.

12. The method according to claim 1, further including the step of generating or filtering in the image vertical or horizontal gradients of planar or piecewise planar objects attached to a gravity aligned coordinate system.

13. The method according to claim 1, wherein the image processing method is operated in relation to a device which is associated with a capturing device for capturing the at least one image, wherein the method further comprises the steps of capturing the at least one image with the capturing device and sharpening the image in the direction of a movement of the capturing device.

14. The method according to claim 1, wherein the image processing method is operated in relation to a device which is associated with a capturing device for capturing the at least one image, wherein the method further comprises the steps of capturing the at least one image with the capturing device and an adaption of a size of a kernel of an image filter dependent on the position of the capturing device with respect to the coordinate system.

15. The method according to claim 1, further providing at least one sensor which provides information on one or multiple degrees of freedom, wherein the at least one sensor includes one or more of a GPS sensor, an inertial sensor, an accelerometer, a gyroscope, a magnetometer, an odometer, or a mechanical sensor, or one or more tracking systems.

16. The method according to claim 1, wherein the method includes at least one of image processing, or filtering of information, or and extraction of information.

17. The method according to claim 1, wherein the method is a method for determining a position and orientation of a device, which is associated with a capturing device for capturing the at least one image, with respect to an object in an augmented reality application or robotic system navigation.

18. The method according to claim 1, wherein the at least one degree of freedom, particularly of a capturing device for capturing the at least one image, is provided with a confidence degree, which is used for the image processing method.

19. The method according to claim 18, wherein the image processing method comprises image filtering with kernels, wherein the kernels are computed depending on a gravity vector and confidence degrees of degrees of freedom comprising the gravity vector.

20. The method according to claim 19, wherein the image processing method includes an edge detection method which comprises a Gaussian kernel for edge detection, which is oriented with respect to a gravity vector, while standard deviations are computed using the confidence degrees of the degrees of freedom comprising the gravity vector.

21. A non-transitory computer readable medium comprising software code sections adapted to perform an image processing method for vision-based localization of a device, comprising the steps of:

providing at least one image of at least one object or part of the at least one object, and providing a coordinate system in relation to the image;

providing at least one degree of freedom in the coordinate system or at least one sensor data in the coordinate system; and

computing image data of the at least one image or at least one part of the at least one image constrained or aligned by the at least one degree of freedom or the at least one sensor data.