Method for tracking keypoints in a scene

• US 10,262,207 B2
• Filed: 07/24/2015
• Issued: 04/16/2019
• Est. Priority Date: 07/24/2014
• Status: Active Grant

• Current Assignee: Universität Zürich
• Original Assignee: Universität Zürich
• Inventors: Delbruck, Tobias, Brandli, Christian, Osswald, Marc
• Primary Examiner(s): Akhavannik, Hadi
• Application Number:
  US15/328,084
• Publication Number:
  US 20170206422A1
• Time in Patent Office: 1,362 Days
• Field of Search: None
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/20 Image preprocessing
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/80 Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level multimodal speaker identification or verification G10L17/10
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

The present invention relates to a method for tracking a position of at least one keypoint in an image of an object moving relative to an optical sensor, wherein light reflected from the object (12) is detected so as to generate an image of the object, and wherein a position of at least one keypoint (K^j) in said image is detected and updated using temporal contrast events (Ev) of the optical sensor. The invention also relates to a corresponding computer program and device.

Claims

1. Method for tracking a position of at least one keypoint in an image of an object moving relative to an optical sensor, comprising the steps of:

a) detecting light reflected from the object (12) and generating an image of the object (12) comprising image pixels, wherein a pixel value is assigned to each image pixel, wherein each pixel value is monotonically related to the intensity of the detected light for the respective image pixel,

b) detecting a position of at least one keypoint (K^j) in said image, particularly assigning a descriptor vector to the at least one keypoint (K^j), generating an image patch (I), wherein the image patch (I) assigns to a pre-defined number of image pixels arranged around the at least one keypoint (K^j) the pixel value of the respective image pixel of the image patch (I), and computing at least one gradient patch (dI/dx) along a degree of freedom (x) of the image patch (I), wherein said at least one gradient patch (dI/dx) assigns to each image pixel of the image patch (I) a gradient of the pixel values of the image patch (I) at the respective image pixel along said degree of freedom,

c) detecting light reflected by the object by means of the optical sensor (14) that comprises a plurality of sensor pixels, wherein each sensor pixel generates a photocurrent being proportional to the intensity of the light impinging on the respective sensor pixel, and generates a signal related to said photocurrent, and wherein each sensor pixel outputs a temporal contrast event (Ev) merely when the respective signal due to the light impinging on the respective sensor pixel increases by an amount being larger than a first threshold (Θ

_ON) or decreases by an amount being larger than a second threshold (Θ

_OFF) since the last temporal contrast event (Ev) from the respective sensor pixel, wherein each temporal contrast event (Ev(u,v,t)) carries the coordinates (u,v) of its associated sensor pixel, a time (t) at which the respective temporal contrast event (Ev) occurred, as well as a polarity value indicating whether the respective temporal contrast event (Ev) is an ON event at which said signal increased by an amount larger than said first threshold (Θ

_ON), or an OFF event at which said signal decreased by an amount larger than said second threshold (Θ

_OFF),

d) updating the position of the at least one keypoint (K^j) using said temporal contrast events (Ev), and

e) particularly repeating steps c) to d) until a new image is generated in which case steps a) to e) are conducted.

2. Method according to claim 1, characterized in that step c) further comprises the steps of:

storing a number (N) of the most recent temporal contrast events (B^j) within the extent of the at least one gradient patch (dI/dx) around the latest position of the at least one keypoint (K^j),

computing a temporal contrast patch (S_E) by summing the weighted polarity values of the stored temporal contrast events (B^j) according to their position (x,y) within the temporal contrast patch (S_E),

computing a comparison (C) between the at least one gradient patch (dI/dx) and the temporal contrast patch (S_E), and

computing a motion (Δ

x) of the at least one keypoint (K^j) in the image along said degree of freedom using said computed comparison (C).

3. Method according to claim 2, characterized in that step d) further comprises updating the position of the at least one gradient patch (dI/dx) associated to the at least one keypoint (K^j) using said motion (Δ

x) along said degree of freedom.

4. Method according to claim 2, characterized in that the following most recent temporal contrast events (B^j) are stored per keypoint

B^j={Ev(x,y,t):

(d((x,y,t),K^j)<

θ

)∩

(|B|=N)∩

(∇

t>

∇

{circumflex over (t)}∈

)},

={(Ev({circumflex over (x)},ŷ

,{circumflex over (t)});

d({circumflex over (x)},ŷ

,{circumflex over (t)}),K^j)<

θ

)∩

(Ev({circumflex over (x)},ŷ

,{circumflex over (t)})∉

B^j)}

5. Method according to claim 1, characterized in that said image is also sampled with said optical sensor (14), so that the temporal contrast events (Ev) are aligned with said image.

6. Method according to claim 1, characterized in that said image is sampled by a further optical sensor (13) that is aligned with the optical sensor (14).

7. Method according to claim 2, characterized in that the temporal contrast patch (S_E) is computed according to

8. Method according to claim 2, characterized in that said motion (Δ

x) is computed according to

9. Method according to claim 1, characterized in that a plurality of gradient patches (dI/dx, dI/dy, dI/dz, dI/dα

, dI/dβ

, dI/dγ

) corresponding to several degrees of freedom of the image patch are computed, wherein the respective gradient patch assigns to each image pixel of the image patch (I) a gradient of the pixel values of the image patch (I) at the respective image pixel along the respective degree of freedom.

10. Method according to claim 2, characterized in that said comparison (C) is a cross correlation, particularly a zero-mean cross correlation.

11. Method according to claim 2, characterized in that said comparison (C) and the position of the at least one keypoint are updated on the acquisition of a single or fixed number of temporal contrast events (Ev).

12. A non-transitory computer readable medium configure to track a position of at least one keypoint in an image of an object moving relative to an optical sensor, wherein the computer program comprises program code for conducting the following steps when the computer program is executed on a computer:

a) detecting a position of at least one keypoint (K^j) in said image, particularly assigning a descriptor vector to the at least one keypoint (K^j), generating an image patch (I), wherein the image patch (I) assigns to a pre-defined number of image pixels arranged around the at least one keypoint (K^j) the pixel value of the respective image pixel of the image patch (I), and computing at least one gradient patch (dI/dx) along a degree of freedom (x) of the image patch (I), wherein said at least one gradient patch (dI/dx) assigns to each image pixel of the image patch (I) a gradient of the pixel values of the image patch (I) at the respective image pixel along said degree of freedom, and

b) updating the position of the at least one keypoint (K^j) using temporal contrast events (Ev) output by an optical sensor (14) merely when a signal of the optical sensor (14) related to a photocurrent due to the light impinging on a sensor pixel of the optical sensor (14) increases by an amount being larger than a first threshold (Θ

_ON) or decreases by an amount being larger than a second threshold (Θ

_OFF) since the last temporal contrast event (Ev) from the respective sensor pixel, wherein each temporal contrast event (Ev(u,v,t)) carries the coordinates (u,v) of its associated sensor pixel, a time (t) at which the respective temporal contrast event (Ev) occurred, as well as a polarity value indicating whether the respective temporal contrast event (Ev) is an ON event at which said signal increased by an amount larger than said first threshold (Θ

_ON), or an OFF event at which said signal decreased by an amount larger than said second threshold (Θ

_OFF), and

c) particularly repeating step b) until a new image is provided in which case steps a) to c) are performed.

13. Device for tracking a position of at least one keypoint in an image of an object moving relative to an optical sensor means, comprising

an optical sensor means (14, 13) that is designed to detect light reflected from an object (12) and to generate an image of the object (12) comprising image pixels,

wherein the device is designed to detect a position of at least one keypoint (K^j) in said image, and

wherein said optical sensor means (14, 13) is further designed to detect light reflected by the object (12) by means of a plurality of sensor pixels, wherein each sensor pixel is designed to generate a photocurrent being proportional to the intensity of the light impinging on the respective sensor pixel, and to generate a signal related to said photocurrent, and wherein each sensor pixel is designed to output a temporal contrast event (Ev) merely when the respective signal due to the light impinging on the respective sensor pixel increases by an amount being larger than a first threshold (Θ

_ON) or decreases by an amount being larger than a second threshold (Θ

_OFF) since the last temporal contrast event (Ev) from the respective sensor pixel, wherein each temporal contrast event (Ev(u,v,t)) carries the coordinates (u,v) of its associated sensor pixel, a time (t) at which the respective temporal contrast event (Ev) occurred, as well as a polarity value indicating whether the respective temporal contrast event (Ev) is an ON event at which said signal increased by an amount larger than said first threshold (Θ

_0N), or an OFF event at which said signal decreased by an amount larger than said second threshold (Θ

_OFF), and wherein

the device is designed to update the position of the at least one keypoint (K^j) using said temporal contrast events, and wherein the device is designed to assign a descriptor vector or an identifier to the at least one keypoint (K^j), and wherein the device is designed to generate an image patch that assigns to a pre-defined number of image pixels arranged around the at least one keypoint the pixel value of the respective image pixel of the image patch, and wherein the device is designed to compute at least one gradient patch along a degree of freedom of the image patch, wherein said at least one gradient patch assigns to each image pixel of the image patch a gradient of the pixel values of the image patch at the respective image pixel along said degree of freedom.

14. Device according to claim 13, characterized in that the optical sensor means is formed as a single optical sensor (14) that is designed to sample said image as well as to generate said temporal contrast events (Ev) so that the temporal contrast events (Ev) are aligned with said image, or in that the sensor means comprises an optical sensor (14) designed to generate said temporal contrast events (Ev) as well as a separate further optical sensor (13) designed to sample said image, which further optical sensor (13) is aligned with the optical sensor (14), so that the temporal contrast events (Ev) are aligned with said image.