METHOD AND SYSTEM FOR DESCRIBING AND ORGANIZING IMAGE DATA

• US 20110153617A1
• Filed: 12/18/2009
• Published: 06/23/2011
• Est. Priority Date: 12/18/2009
• Status: Active Grant

• Current Assignee: Toyota Motor Corporation
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Uehara, Yasuo
• Granted Patent: US 8,237,792 B2
• Time in Patent Office: Days
• US Class Current: 707/741
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion

Abstract

The present invention is directed to methods and systems for describing and organizing image data. An analysis system includes a sensor, a processing unit, a machine, a database, and/or a display. The analysis system is connected to an electronic device. The sensor generates sensor data which is imported by the processing unit. The processing unit analyzes the sensor data to determine features of the sensor data such as the subject, the time, the location, the object, the event, and/or the action in the sensor data. The processing unit creates entries for the sensor data in a features index. The processing unit receives a query for an item and uses the features index to determine search results including the item, such as a location and a time of the item. The processing unit can also determine whether a machine has completed its mission using the features in the sensor data.

Claims

1. A method of analyzing a scene comprising:

importing sensor data;

identifying a subject in the sensor data;

identifying a time in the sensor data;

identifying a location in the sensor data;

identifying an object in the sensor data;

identifying an event in the sensor data;

identifying an action in the sensor data; and

creating an entry in the feature index for the sensor data from the subject, the time, the location, the object, the event, and the action.

2. The method of claim 1 wherein the sensor data includes a video clip.

3. The method of claim 2 further comprising using the video clip to identify the subject in the sensor data, the time in the sensor data, the location in the sensor data, the object in the sensor data, the event in the sensor data, or the action in the sensor data.

4. The method of claim 1 wherein the sensor data includes images.

5. The method of claim 1 wherein the sensor data includes global positioning system data.

6. The method of claim 1 further comprising accessing the sensor data using the feature index.

7. The method of claim 1 further comprising inputting a query and retrieving the results of the query using the feature index.

8. The method of claim 7 wherein the query is for finding a subject or an object that was lost.

9. The method of claim 1 further comprising inputting a subject, a time, a location, an object, an event, and an action defining a mission.

10. The method of claim 9 further comprising

comparing the subject, the time, the location, the object, the event, and the action defining the mission to the subject, the time, the location, the object, the event, and the action from the sensor data;

and

determining that the mission is completed when the subject, the time, the location, the object, the event, and the action defining the mission and the time, the location, the object, the event, and the action from the sensor data substantially match.

11. A method for finding an item comprising:

importing sensor data;

identifying a subject in the sensor data;

identifying a time in the sensor data;

identifying a location in the sensor data;

identifying an object in the sensor data;

identifying an event in the sensor data;

identifying an action in the sensor data;

creating an entry in the feature index for the sensor data from the subject, the time, the location, the object, the event, and the action;

receiving a query for an item;

searching the feature index to determine whether the item is contained in the sensor data; and

displaying the result of the search.

12. The method of claim 11 further comprising analyzing the query for the item to determine whether the result of the search will be more specific than a predetermined specificity, and asking for a more detailed query when the result of the search will not be more specific than the predetermined specificity.

13. The method of claim 11 wherein the sensor data includes image data.

14. The method of claim 11 wherein the sensor data includes video data.

15. The method of claim 11 wherein the sensor data includes global positioning system data.

16. The method of claim 11 wherein the result includes a time and location of the item.

17. A method for operating a machine comprising:

receiving a mission state of the machine;

receiving a current state of the machine;

analyzing the mission state of the machine and the current state of the machine;

continuing to operate the machine when the mission state of the machine and the current state of the machine are substantially different; and

ceasing to operate the machine when the mission state of the machine and the current state of the machine are substantially the same.

18. The method of claim 17 wherein the mission state and the current state, each includes a subject, a time, a location, an object, an event, and an action.

19. The method of claim 18 wherein the analysis of the mission state of the machine and the current state of the machine includes comparing the subject, the time, the location, the object, the event, and the action of the mission state of the machine to the subject, the time, the location, the object, the event, and the action of the current state of the machine.

20. The method of claim 17 further comprising importing sensor data, wherein the current state of the machine is based on the sensor data.