Method and system of sparse code based object classification with sensor fusion

• US 8,081,209 B2
• Filed: 06/26/2008
• Issued: 12/20/2011
• Est. Priority Date: 06/26/2008
• Status: Active Grant

• Current Assignee: Toyota Motor Corporation
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Ji, Zhengping, Prokhorov, Danil V.
• Primary Examiner(s): Maung, Zarni
• Application Number:
  US12/147,001
• Publication Number:
  US 20090322871A1
• Time in Patent Office: 1,272 Days
• Field of Search: 348113-118
• US Class Current: 348/115
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 7/412 based on a comparison between measured values and known or stored values
• CPC Class Codes: G06F 18/2136 based on sparsity criteria, e.g. with an overcomplete basis
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

A system and method for object classification based upon the fusion of a radar system and a natural imaging device using sparse code representation. The radar system provides a means of detecting the presence of an object within a predetermined path of a vehicle. Detected objects are then fused with the image gathered by the camera and then isolated in an attention window. The attention window is then transformed into a sparse code representation of the object. The sparse code representation is then compared with known sparse code representation of various objects. Each known sparse code representation is given a predetermined variance and subsequent sparse code represented objects falling within said variance will be classified as such. The system and method also includes an associative learning algorithm wherein classified sparse code representations are stored and used to help classifying subsequent sparse code representation.

Claims

1. A system for object classification wherein the natural image of a radar return is isolated and processed for subsequent classification, said system comprising:

a radar system, said radar system having a predetermined range and providing a return for the detection of an object within said predetermined range;

a natural imaging device, said natural imaging device providing a real-time visual of the environment;

a computer, wherein said computer executing a software program that associates said return of said radar system with the real-time visual of the natural imaging device using a perspective mapping transformation to provide an isolated natural image of said return;

said computer further executing an algorithm for processing the image of each of at least one attention window to generate an orientation-selective filter associated with each of said at least one attention window;

said computer further executing a sparse code generation algorithm, whereby each image of each of said at least one attention window is transformed via orientation-selective filters by said sparse code generation algorithm to provide a sparse code representation of each image; and

an associative learning framework, whereby said associative learning framework classifies each of said sparse code representation, and said associative learning algorithm identifies subsequent sparse code representation by comparing said subsequent sparse code representation with known classifications of sparse code representation.

2. A system as set forth in claim 1 further comprising at least one attention window, each of said at least one attention window having a predetermined width and height from said return, said at least one attention window including the image of said return as captured by said natural imaging device.

3. A system as set forth in claim 1 wherein said algorithm for generating said orientation-selective filter is a lobe component analysis.

4. A method of object classification based upon the fusion of a radar system and a natural imaging device, said method comprising:

performing a perspective mapping transformation of a return of said radar system with a real-time image from said natural imaging device,

creating an attention window for each object detected by said radar, wherein said attention window having a predetermined width and height from an object and the image of said object is displayed in each attention window;

generating orientation-selective filters of the image,

transforming said image via orientation-selective filters into a sparse code representation;

classifying each sparse code representation;

storing each classified sparse code representation in a database; and

using each stored classified sparse code representation to further classify subsequent sparse code representations of subsequent images of objects captured in said attention windows.