Barcode: global binary patterns for fast visual inference

• US 10,416,681 B2
• Filed: 07/12/2017
• Issued: 09/17/2019
• Est. Priority Date: 07/12/2017
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Lee, Teng-Yok, Ramalingam, Srikumar, Taguchi, Yuichi, Patil, Sonali
• Primary Examiner(s): Shaawat, Mussa A
• Application Number:
  US15/647,428
• Publication Number:
  US 20190018423A1
• Time in Patent Office: Days
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G06F 17/16 Matrix or vector computation , e.g. matrix-matrix or matrix-vector multiplication, matrix factorization matrix transposition G06F7/78
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06K 7/1434 Barcodes with supplemental or add-on codes
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59

Abstract

Systems and methods for an imaging system having a memory with a historical localization dictionary database having geo-located driving image sequences, such that each reference image is applied to a threshold to produce a binary representation. A sensor to acquire a sequence of input images of a dynamic scene. An encoder to determine, for each input image in the sequence, a histogram of each input image indicating a number of vertical edges at each bin of the input image and to threshold the histogram to produce a binary representation of the input image. A visual odometer to compare the binary representations of each input image and each reference image, by matching an input image against a reference image. Wherein the visual odometer determines a location of the input image based on a match between the input image and the reference image.

Claims

1. An imaging system having a memory with stored thereon executable instructions, and stored reference imaging data including a historical localization dictionary database having geo-located driving image sequences and other features acting as signatures for distinct locations, such that each reference image is applied to a threshold to produce a binary representation, the imaging system comprising:

a sensor to acquire a sequence of input images of a dynamic scene;

an encoder to determine, for each input image in the sequence, a histogram of each input image in the sequence indicating a number of vertical edges at each bin of the input image and to threshold the histogram to produce a binary representation of the input image; and

a visual odometer to compare the binary representations of each input image in the sequence and each reference image in the database, by matching one or more input image against one or more reference image in the database,

wherein the visual odometer determines a location of the input image based on a match between the one or more input image and the one or more reference image.

2. The imaging system of claim 1, wherein the state of the imaging system includes the location and an orientation of the imaging system.

3. The imaging system of claim 2, wherein the orientation of the imaging system includes a turn-classification algorithm that uses the binary representations of consecutive input images in the sequence in a neural network, such that an input to the neural network is a binary matrix obtained by stacking the binary representations of consecutive input images in the sequence from consecutive images.

4. The imaging system of claim 1, wherein the visual odometer tracks the change in the state of the imaging system using a neural network.

5. The imaging system of claim 4, wherein the neural network, for each control step, receives as an input the binary representations of at least some images concatenated in a tensor to produce the change in the state of the imaging system.

6. The imaging system of claim 4, wherein the neural network, for each control step, receives as an input the binary representations of the sequence of images determined for each bin of each binary representation.

7. The imaging system of claim 1, wherein the visual odometer uses a hamming distance approach to compare the binary representations of each input image in the sequence and each image in the database.

8. The imaging system of claim 1, wherein the encoder filters one or more of the vertical edges at each bin of the input image, such that a strength of the vertical images of each pixel is measured as a magnitude of a difference of a gradient along a horizontal and a vertical direction.

9. The imaging system of claim 1, wherein the imaging system is in communication with a vehicle control system, a vehicle parking system, a vehicle navigation system, etc.

10. An imaging system, the imaging system having a memory with stored reference imaging data including a historical localization dictionary database, the database having a sequence of reference images, such that each reference image is applied to a threshold to produce a binary representation, comprising:

a sensor to acquire a sequence of input images of a dynamic scene;

an encoder to determine, for each input image in the sequence, a histogram of each input image in the sequence indicating a number of vertical edges at each bin of the input image and to threshold the histogram to produce a binary representation of the input image;

a visual odometer to compare the binary representations of each input image in the sequence and each reference image in the database, by associating the input image with a corresponding reference image in the database, to identify an associated location of the input image; and

an output interface of the encoder outputs the identified associated location of each input image or stores the identified associated location of each input image in the memory.

11. The imaging system of claim 10, wherein the visual odometer tracks the change in the state of the imaging system using a neural network.

12. The imaging system of claim 11, wherein the visual odometer tracks the change in the state of the imaging system using a neural network, such that the neural network, for each control step, receives as an input the binary representations of at least 100 images concatenated in a tensor to produce the change in the state of the imaging system.

13. The imaging system of claim 11, wherein the neural network, for each control step, receives as an input the binary representations of the sequence of images combined with depth values determined for each bin of each binary representation.

14. The imaging system of claim 10, wherein the visual odometer uses a hamming distance approach to compare the binary representations of each input image in the sequence and each reference image in the database.

15. An imaging system to detect a location of a vehicle while the vehicle is moving, the imaging system having a memory with stored imaging data that includes a historical localization dictionary database having a sequence of images, such that each image is applied to a threshold to produce a binary representation, comprising:

a sensor mounted to the vehicle to acquire a sequence of input images of a dynamic scene;

an encoder to determine, for each input image in the sequence, a histogram of each input image in the sequence indicating a number of vertical edges at each bin of the input image and to threshold the histogram to produce a binary representation of the input image;

a visual odometer to compare the binary representations of each input image in the sequence and each image in the database, by associating the input image with a corresponding image in the database, to identify an associated location of the input image; and

an output interface of the encoder outputs the identified associated location of each input image or stores the identified associated location of each input image in the memory.

16. The imaging system of claim 15, wherein the threshold is based on one or more vertical line segments of each vertical edge being within or out of a vertical tolerance range.

17. The imaging system of claim 15, wherein the state of the imaging system includes the location and an orientation of the imaging system, such that the orientation of the imaging system includes a turn-classification algorithm, and the turn-classification algorithm uses the binary representations of consecutive input images in the sequence in a neural network, such that an input to the neural network is a binary matrix obtained by stacking the binary representations of consecutive input images in the sequence from consecutive images.