Visual search system architectures based on compressed or compact feature descriptors

• US 9,258,564 B2
• Filed: 02/07/2013
• Issued: 02/09/2016
• Est. Priority Date: 02/07/2012
• Status: Active Grant

• Current Assignee: STMicroelectronics SRL (STMicroelectronics NV)
• Original Assignee: STMicroelectronics SRL (STMicroelectronics NV)
• Inventors: Pau, Danilo Pietro
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): WOLDEMARIAM, AKLILU K
• Application Number:
  US13/762,349
• Publication Number:
  US 20130216135A1
• Time in Patent Office: 1,097 Days
• Field of Search: 382/190, 382/197, 382/203, 382/233, 382/236, 382/100, 382/173, 382/250, 382/248, 382/298, 707/728, 707/748, 348/387, 348/390, 348/403, 348/411
• US Class Current: 1/1
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/13 Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
• CPC Class Codes: H04N 19/91 Entropy coding, e.g. variable length coding [VLC] or arithmetic coding

Abstract

An embodiment of a visual search system includes at least one imaging device, each imaging device operable to capture a corresponding image, a feature extraction device coupled to each imaging device, each feature extraction device operable to generate feature descriptors from the image received from the corresponding imaging device. A descriptor encoding device is coupled to each feature extraction device and operable to generate compressed feature descriptors from the received feature descriptors. An application processor is coupled to each descriptor encoding device. The application processor is operable to process the received compressed feature descriptors to generate output information as a function of the processed compressed feature descriptors.

Claims

1. A vehicle visual search system, comprising:

at least four cameras, each camera configured to be positioned on one side of a vehicle and configured to capture corresponding digital images, each of the cameras comprising;

a feature descriptor extraction circuit configured to receive the corresponding captured digital images and configured to generate feature descriptors from the captured digital images; and

a descriptor encoding circuit coupled to the feature descriptor extraction circuit to receive the feature descriptors and configured to generate compact feature descriptors from the feature descriptors wherein the descriptor encoding circuit generates the compact feature descriptors by performing zero-thresholding on the feature descriptors to generate zero-threshold delimited three-dimensional feature descriptors, quantizing the zero-threshold delimited three-dimensional feature descriptors to generate quantized feature descriptors, and coding the quantized feature descriptors to generate the compact feature descriptors; and

an application processor connected through a wired network to each of the at least four cameras, the application processor configured to receive the compact feature descriptors through the wired network and to process the received compact feature descriptors to generate output information that is utilized in controlling the operation of the vehicle.

2. The vehicle visual search system of claim 1 wherein the compact feature descriptors comprise Scale-Invariant Feature Transform (SIFT) descriptors.

3. The Vehicle visual search system of claim 1 wherein the compact feature descriptors comprise Signature of Histograms of orientations (SHOT) descriptors.

4. The vehicle visual search system of claim 1 wherein the cameras collectively provide a 360 degree field of view around the vehicle.

5. The vehicle visual search system of claim 1 wherein the wired network comprises an Ethernet network.

6. The vehicle visual search system of claim 1 wherein, the application processor configured to process the received compact feature descriptors without decoding the compact feature descriptors to perform matching of the captured digital images.

7. The vehicle visual search system of claim 1 wherein the generated output information comprises textual information presented in a video corresponding to the digital images captured by the cameras.

13. The vehicle visual search system of claim 1 wherein the application processor processes the received compact feature descriptors by performing pair wise matching using the received compact feature descriptors from pairs of the cameras and by performing tracking of objects for each camera using the received compact feature descriptors from the camera.

8. A vehicle visual search system, comprising:

a plurality of cameras, each camera positioned on one side of a vehicle and configured to capture corresponding digital images, each of the cameras comprising;

a feature descriptor extraction circuit configured to receive the corresponding captured digital images and configured to generate feature descriptors from the captured digital images;

a descriptor encoding circuit coupled to the feature descriptor extraction circuit to receive the feature descriptors and configured to generate compact feature descriptors from the feature descriptors, and configured to communicate these compact feature descriptors over a first wireless communications network, wherein the descriptor encoding circuit generates the compact feature descriptors by performing zero-thresholding on the feature descriptors to generate zero-threshold delimited three-dimensional feature descriptors, quantizing the zero-threshold delimited three-dimensional feature descriptors to generate quantized feature descriptors, and coding the quantized feature descriptors to generate the compact feature descriptors;

a server system configured to receive over a second wireless network the compact feature descriptors and to process the received compact feature descriptors to generate search result information that is then communicated over the second wireless network; and

a portable electronic device configured to receive the compact feature descriptors over the first wireless network and to communicate these compact feature descriptors over the second wireless network to the server system, and the portable electronic device configured to receive search result information over the second wireless network from the server system and to communicate this search result information over the first wireless network to an application processor, the application processor configured to cause information relating to the search result information to be provided to a person in the vehicle.

9. The vehicle visual search system of claim 8 wherein the compact feature descriptors comprise Scale-Invariant Feature Transform (SIFT) descriptors.

10. The vehicle visual search system of claim 8 wherein the compact feature descriptors comprise Signature of Histograms of Orientations (SHOT) descriptors.

11. The vehicle visual search system of claim 8 wherein the at least one imaging device comprises an application processor that communicates with the imager circuitry to provide search result information to a driver of the vehicle.

12. The vehicle visual search system of claim 8 wherein the portable electronic device comprises a smart phone.

14. The vehicle search system of claim 8 wherein the application processor provides information relating to the search result information to a person in the vehicle by displaying a digital image contained in the search result information.

15. The vehicle search system of claim 14 wherein the application processor further provides additional information to the person in the form of descriptive text that is superimposed on the displayed digital image.