Method and device for detecting objects from scene images by using dynamic knowledge base

• US 10,664,728 B2
• Filed: 02/28/2018
• Issued: 05/26/2020
• Est. Priority Date: 12/30/2017
• Status: Active Grant

• Current Assignee: Wipro Limited
• Original Assignee: Wipro Limited
• Inventors: Govindaraj, Balaji, Zaid, Mohd, Jagannath, Sujatha
• Primary Examiner(s): Ky, Kevin
• Application Number:
  US15/907,678
• Publication Number:
  US 20190205706A1
• Time in Patent Office: 818 Days
• Field of Search: None
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/24147 Distances to closest patterns, e.g. nearest neighbour classification
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/046 Forward inferencing; Production systems
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

A method and device for detecting objects from scene images by using dynamic knowledge base is disclosed. The method includes segmenting a scene image captured by at least one camera into a plurality of image segments. Each of the plurality of image segments include a plurality of pixels. The method further includes detecting at least one object in each of the plurality of image segments. The method includes finding a plurality of similar images from an online database based on the at least one object. The method further includes identifying using a knowledge base engine, at least one similar object in the plurality of similar images. The method includes updating the knowledge base engine based on the at least one similar object identified from the plurality of similar images. The method further includes training a neural network to detect objects from scene images, based on the updated knowledge base engine.

Claims

1. A method for detecting objects from scene images, the method comprising:

segmenting, by an image processing device, a scene image captured by at least one camera into a plurality of image segments, wherein each of the plurality of image segments comprise a plurality of pixels;

detecting, by the image processing device, at least one object in each of the plurality of image segments, wherein detecting the at least one object comprises retrieving object related descriptions associated with the at least one object by searching an online database with each of the plurality of image segments;

finding, by the image processing device, a plurality of similar images from the online database based on the at least one object identified in each of the plurality of image segments, wherein finding the plurality of similar images comprises;

generating a search query that comprises vector features created for an image segment and the object related descriptions associated with the at least one object in the image segment; and

retrieving the plurality of similar images from the online database based on execution of the search query, wherein each of the plurality of similar images are retrieved along with an associated description;

identifying using a knowledge base engine, by the image processing device, at least one similar object in the plurality of similar images based on the associated description and image characteristics;

updating, by the image processing device, the knowledge base engine based on the at least one similar object identified from the plurality of similar images; and

training, by the image processing device, a neural network to detect objects from scene images captured by the at least one camera, based on the updated knowledge base engine.

2. The method of claim 1, wherein segmenting comprises:

grouping pixels in the scene image to generate the plurality of image segments, wherein pixels in each image segment comprises same image characteristics, and wherein the image characteristics comprise at least one of color, intensity, texture, or shape; and

creating vector features for each of the plurality of image segments based on associated image characteristics.

3. The method of claim 2 further comprising assigning a label to each of the plurality of image segments based on the associated image characteristics.

4. The method of claim 2, wherein the at least one object in the image segment is identified based on the vector features created for the image segment.

5. The method of claim 1, wherein identifying the at least one similar object comprises:

assigning confidence scores to terms in the associated description related to the plurality of similar images;

ranking the terms based on the assigned confidence scores; and

selecting at least one term as at least one class for the knowledge base engine, based on the ranking, wherein the at least one term is ranked above a threshold rank.

6. The method of claim 5, wherein updating the knowledge base engine comprises assigning each of the at least one class as a main class or a sub class based on the ranking.

7. The method of claim 1, wherein the neural network is a convolutional neural network that is trained using the updated knowledge base engine.

8. The method of claim 1 further comprising communicating, by the image processing device, information related to the detected objects to a navigation control system in an autonomous vehicle for navigation.

9. An image processing device for detecting objects from scene images, the image processing device comprising:

a processor; and

a memory communicatively coupled to the processor, wherein the memory stores processor instructions, which, on execution, causes the processor to;

segment a scene image captured by at least one camera into a plurality of image segments, wherein each of the plurality of image segments comprise a plurality of pixels;

detect at least one object in each of the plurality of image segments, wherein detecting the at least one object comprises retrieving object related descriptions associated with the at least one object by searching an online database with each of the plurality of image segments;

find a plurality of similar images from the online database based on the at least one object identified in each of the plurality of image segments, wherein finding the plurality of similar images comprises;

generating a search Query that comprises vector features created for an image segment and the object related descriptions associated with the at least one object in the image segment; and

retrieving the plurality of similar images from the online database based on execution of the search query, wherein each of the plurality of similar images are retrieved along with an associated description;

identify using a knowledge base engine, at least one similar object in the plurality of similar images based on the associated description and image characteristics;

update the knowledge base engine based on the at least one similar object identified from the plurality of similar images; and

train a neural network to detect objects from scene images captured by the at least one camera, based on the updated knowledge base engine.

10. The image processing device of claim 9 further comprising the at least one camera to capture the scene image.

11. The image processing device of claim 9, wherein to segment the scene image, the processor instructions cause the processor to:

group pixels in the scene image to generate the plurality of image segments, wherein pixels in each image segment comprises same image characteristics, and wherein the image characteristics comprise at least one of color, intensity, texture, or shape; and

create vector features for each of the plurality of image segments based on associated image characteristics.

12. The image processing device of claim 11, the processor instructions cause the processor to assign a label to each of the plurality of image segments based on the associated image characteristics.

13. The image processing device of claim 11, wherein the at least one object in the image segment is identified based on the vector features created for the image segment.

14. The image processing device of claim 9, wherein to identify the at least one similar object, the processor instructions cause the processor to:

assign confidence scores to terms in the associated description related to the plurality of similar images;

rank the terms based on the assigned confidence scores; and

select at least one term as at least one class for the knowledge base engine, based on the ranking, wherein the at least one term is ranked above a threshold rank.

15. The image processing device of claim 14, wherein to update the knowledge base engine, the processor instructions cause the processor to assign each of the at least one class as a main class or a sub class based on the ranking.

16. The image processing device of claim 9, wherein the neural network is a convolutional neural network that is trained using the updated knowledge base engine.

17. The image processing device of claim 9, wherein the processor instructions cause the processor to communicate information related to the detected objects to a navigation control system in an autonomous vehicle for navigation.

18. A non-transitory computer-readable storage medium having stored thereon, a set of computer-executable instructions causing a computer comprising one or more processors to perform steps comprising:

segmenting a scene image captured by at least one camera into a plurality of image segments, wherein each of the plurality of image segments comprise a plurality of pixels;

detecting at least one object in each of the plurality of image segments, wherein detecting the at least one object comprises retrieving object related descriptions associated with the at least one object by searching an online database with each of the plurality of image segments;

finding a plurality of similar images from the online database based on the at least one object identified in each of the plurality of image segments, wherein finding the plurality of similar images comprises;

generating a search query that comprises vector features created for an image segment and the object related descriptions associated with the at least one object in the same segment; and

retrieving the plurality of similar images from the online database based on execution of the search query, wherein each of the plurality of similar images are retrieved along with an associated description;

identifying using a knowledge base engine, at least one similar object in the plurality of similar images based on the associated description and image characteristics;

updating the knowledge base engine based on the at least one similar object identified from the plurality of similar images; and

training a neural network to detect objects from scene images captured by the at least one camera, based on the updated knowledge base engine.