Image processing method, particularly used in a vision-based localization of a device

• US 9,888,235 B2
• Filed: 06/05/2013
• Issued: 02/06/2018
• Est. Priority Date: 09/27/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Meier, Peter, Benhimane, Mohamed Selim
• Primary Examiner(s): Kholdebarin, Iman K
• Application Number:
  US14/430,169
• Publication Number:
  US 20150245020A1
• Time in Patent Office: 1,707 Days
• Field of Search: None
• CPC Class Codes: G01S 19/53 Determining attitude
• CPC Class Codes: G01S 5/0247 Determining attitude
• CPC Class Codes: G01S 5/163 Determination of attitude using inertial means G01C9/00; control of attitude G05D1/49
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/20101 Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/631 Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

An image processing method includes the steps of providing at least one image of at least one object or part of the at least one object, and providing a coordinate system in relation to the image, providing at least one degree of freedom in the coordinate system or at least one sensor data in the coordinate system, and computing image data of the at least one image or at least one part of the at least one image constrained or aligned by the at least one degree of freedom or the at least one sensor data.

Claims

1. An image processing method for vision-based localization of a device, comprising the steps of:

obtaining an image depicting at least part of an object, and providing a coordinate system in relation to the image;

providing at least one degree of freedom in the coordinate system or at least one sensor data in the coordinate system;

computing image data of the image or at least part of the image constrained or aligned by the at least one degree of freedom or the at least one sensor data; and

based on the image data, determining a position and orientation of a device with respect to an object in an augmented reality application or robotic system navigation application, the device associated with a capturing device for capturing the image.

2. The method according to claim 1, wherein the computed image data is a result of image processing including morphological image operations or image filtering.

3. The method according to claim 2, wherein the computing of image data is constrained or aligned according to degrees of freedom with a particular confidence degree or is constrained or aligned according to sensors providing data used in the image processing method in relation to the image or a device capturing the image.

4. The method according to claim 1, wherein computing the image data includes an image gradient computation which comprises filtering of gravity aligned gradients in the image.

5. The method according to claim 1, wherein computing the image data includes performing an image gradient computation which comprises image filtering with kernels, wherein the kernels are aligned with axes of the image or with a gravity vector.

6. The method according to claim 5, wherein in response to an image plane of the image being fronto-parallel to the gravity vector, a same gravity aligned kernel is applied to each pixel in the image.

7. The method according to claim 1, the method further including a process for matching at least one feature of the object in the image with at least one feature of a digital representation of the object.

8. The method according to claim 1, wherein the image data corresponds to a distance transform image.

9. The method according to claim 1, further including running a global registration algorithm, where different degrees of freedom are iteratively refined and a quality of each iteration is measured by a predefined cost-function.

10. The method according to claim 1, further comprising capturing the image with a capturing device and obtaining an estimation of intrinsic parameters of the capturing device.

11. The method according to claim 1, further comprising obtaining position and orientation data associated with a capturing device that captured the image, and determining the at least one degree of freedom in the coordinate system and associated confidence values based on the position and orientation data.

12. The method according to claim 1, further including the step of generating or filtering in the image vertical or horizontal gradients of planar or piecewise planar objects attached to a gravity aligned coordinate system.

13. The method according to claim 1, wherein the image data corresponds to a sharpened version of the image, the sharpened version sharpened in a direction of a movement of a capturing device associated with capturing the image.

14. The method according to claim 1, wherein computing the image data includes setting a size of a kernel of an image filter based on the position of a capturing device with respect to the coordinate system, the capturing device associated with capturing the image.

15. The method according to claim 1, further providing at least one sensor which provides information on one or multiple degrees of freedom, wherein the at least one sensor includes one or more of a GPS sensor, an inertial sensor, an accelerometer, a gyroscope, a magnetometer, an odometer, or a mechanical sensor, or one or more tracking systems.

16. The method according to claim 1, wherein the method includes at least one of image processing, filtering of information, and extraction of information.

17. The method according to claim 1, wherein the at least one degree of freedom is associated with a capturing device for capturing the image, the method further comprising obtaining a confidence degree associated with the at least one degree of freedom.

18. The method according to claim 17, wherein computing the image data comprises performing image filtering with a kernel, wherein the at least one degree of freedom corresponds to a gravity vector, and wherein the kernel is computed based on the gravity vector and the confidence degree.

19. The method according to claim 18, wherein the kernel comprises a Gaussian kernel for edge detection, the Gaussian kernel oriented with respect to the gravity vector, and associated with a standard deviation computed using the confidence degree.

20. A non-transitory computer readable medium comprising software code sections adapted to perform an image processing method for vision-based localization of a device, comprising the steps of:

obtaining an image depicting at least part of an object, and providing a coordinate system in relation to the image;

providing at least one degree of freedom in the coordinate system or at least one sensor data in the coordinate system;

computing image data of the image or at least one part of the image constrained or aligned by the at least one degree of freedom or the at least one sensor data; and

based on the image data, determining a position and orientation of a device with respect to an object in an augmented reality application or robotic system navigation application, the device associated with a capturing device for capturing the image.