Method and device for generating binary descriptors in video frames

• US 10,268,929 B2
• Filed: 12/16/2016
• Issued: 04/23/2019
• Est. Priority Date: 12/17/2015
• Status: Active Grant

• Current Assignee: Stmicroelectronics France (STMicroelectronics NV)
• Original Assignee: STMicroelectronics SA (STMicroelectronics NV)
• Inventors: Alibay, Manu, Auberger, Stephane, Stanciulescu, Bogdan-Florin
• Primary Examiner(s): Elahi, Shan E
• Application Number:
  US15/381,919
• Publication Number:
  US 20170177980A1
• Time in Patent Office: 858 Days
• Field of Search: 348 36
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A method generates a binary descriptor associated with a given point in a current frame of a succession of video frames obtained by an apparatus such as an image sensor. The method includes determining a pattern of points pairs around said given point in the current frame, and performing intensity comparison processing between the two points of each pair. The apparatus is likely to move in a rotation between the previous frame and the current frame. The method includes processing the pattern of points of the current frame with tridimensional rotation information representative of the apparatus rotation between the previous frame and the current frame and obtained from inertial measurements provided by at least one inertial sensor.

Claims

1. A method, comprising:

generating a binary descriptor associated with a given point in a current frame of a succession of video frames obtained by an apparatus configured to move in a rotation between a previous frame and the current frame, the generating including determining a pattern of point pairs of points around said given point in the current frame, performing intensity comparison processing between the points of each pair, and processing said pattern of point pairs of said current frame with tridimensional rotation information representative of said apparatus rotation between the previous frame and the current frame, wherein processing said pattern of points of said current frame comprises, for each point of each point pair of said pattern;

determining bidimensional coordinates of said point from bidimensional coordinates of a homologue point in the previous frame and from an in-plane rotation matrix corresponding to an in-plane rotation, in a plane, of an in-plane angle around said given point;

converting said determined bidimensional coordinates of said point into tridimensional coordinates;

determining a second rotation matrix representative of said tridimensional rotation information;

applying said second rotation matrix to said tridimensional coordinates of said point to obtain new tridimensional coordinates; and

projecting said new tridimensional coordinates back onto said plane of said in-plane rotation to obtain a projected point having projected coordinates; and

processing the succession of video frames based on the binary descriptor.

2. The method according to claim 1, wherein said tridimensional rotation information is obtained from inertial measurements.

3. The method according to claim 1 wherein performing intensity comparison processing between the points of each point pair comprises determining intensities of the projected points of said point pair and comparing the intensities.

4. The method according to claim 1, wherein said given point is an origin of each point pair.

5. The method according to claim 1, wherein the points of said pattern of point pairs are included in a zone of L×

M points centered on said given point.

6. The method according to claim 1, wherein said given point is a keypoint of said current frame.

7. The method according to claim 1, wherein said apparatus belongs to a platform.

8. The method of claim 1 wherein the processing the succession of video frames based on the binary descriptor includes at least one of:

video stabilization;

panorama creation;

denoising; and

modifying image dynamics.

9. A device, comprising:

generating means for generating a binary descriptor associated with a given point in a current frame of a succession of video frames obtained by an apparatus configured to move in a rotation between a previous frame and the current frame, said generating means including first means for determining a pattern of point pairs of points around said given point in the current frame, and comparison means for performing intensity comparison processing between the points of each point pair, and processing means for processing said pattern of point pairs of said current frame with tridimensional rotation information representative of said apparatus rotation between the previous frame and the current frame, wherein said processing means for processing said pattern of pair points comprises a processing module configured, for each point of each point pair of said pattern, to;

determine bidimensional coordinates of said point from bidimensional coordinates of a homologue point in the previous frame and from an in-plane rotation matrix corresponding to an in-plane rotation, in a plane, of an in-plane angle around said given point;

convert said determined bidimensional coordinates of said point into tridimensional coordinates;

determine a second rotation matrix representative of said tridimensional rotation information;

apply said second rotation matrix to said tridimensional coordinates of said point to obtain new tridimensional coordinates; and

project said new tridimensional coordinates back onto said plane of said in-plane rotation to obtain a projected point having projected coordinates; and

,

means for processing the succession of video frames based on generated binary descriptors.

10. The device according to claim 9, wherein said processing means are configured to receive said tridimensional rotation information from inertial measurements provided by at least one inertial sensor.

11. The device according to claim 9 wherein said comparison means are configured to determine respective intensities of the projected points of said point pair and to compare the intensities.

12. The device according to claim 9, wherein said given point is an origin of each point pair.

13. The device according to claim 9, wherein said pattern of point pairs is included in a zone of L×

M points centered on said given point.

14. The device according to claim 9, wherein said given point is a keypoint of said current frame.

15. The device of claim 9 wherein processing the succession of video frames based on generated binary descriptors includes at least one of:

video stabilization;

panorama creation;

denoising; and

modifying image dynamics.

16. A tablet platform, comprising:

an apparatus configured to capture a succession of video frames including a current frame and a previous frame and configured to move in a rotation between the previous frame and the current frame; and

a device including;

generating means for generating a binary descriptor associated with a given point in the current frame, said generating means including first means for determining a pattern of point pairs of points around said given point in the current frame, and comparison means for performing intensity comparison processing between the points of each point pair, and processing means for processing said pattern of point pairs of said current frame with tridimensional rotation information representative of said apparatus rotation between the previous frame and the current frame, wherein said processing means for processing said pattern of pair points comprises a processing module configured, for each point of each point pair of said pattern, to;

determine bidimensional coordinates of said point from bidimensional coordinates of a homologue point in the previous frame and from an in-plane rotation matrix corresponding to an in-plane rotation, in a plane, of an in-plane angle around said given point;

convert said determined bidimensional coordinates of said point into tridimensional coordinates;

determine a second rotation matrix representative of said tridimensional rotation information;

apply said second rotation matrix to said tridimensional coordinates of said point to obtain new tridimensional coordinates; and

project said new tridimensional coordinates back onto said plane of said in-plane rotation to obtain a projected point having projected coordinates; and

means for processing the succession of video frames based on generated binary descriptors.

17. The platform according to claim 16, further comprising at least one inertial sensor configured to provide said tridimensional rotation information.

18. The platform according to claim 16 wherein said comparison means are configured to determine respective intensities of the projected points of said point pair and to compare the intensities.

19. The platform according to claim 16, wherein said pattern of point pairs is included in a zone of L×

M points centered on said given point.

20. The platform of claim 16 wherein processing the succession of video frames based on generated binary descriptors includes at least one of:

video stabilization;

panorama creation;

denoising; and

modifying image dynamics.