Real-time annotation of images in a human assistive environment

• US 8,494,223 B2
• Filed: 04/30/2012
• Issued: 07/23/2013
• Est. Priority Date: 04/23/2009
• Status: Active Grant

• Current Assignee: Hyundai Motor Company (Hyundai Motor Group), Kia Corp.
• Original Assignee: International Business Machines Corporation
• Inventors: Connell, Jonathan, Haas, Norman, Hampapur, Arun, Li, Ying, Olyha, Robert S. Jr., Pankanti, Sharathchandra
• Primary Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US13/459,940
• Publication Number:
  US 20120213413A1
• Time in Patent Office: 449 Days
• Field of Search: 382/104, 382/107, 348/149, 348/154, 348/155, 348/169, 340/425.5, 340/441, 340/436, 340/466, 340/910, 340/917, 340/933, 340/937, 340/960, 340/978, 701/20, 701/74, 701/79, 701/93, 701/96, 701/110, 701/119, 701/121
• US Class Current: 382/104
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G09B 9/052 characterised by provision for recording or measuring trainee's performance devices for psychotechnics, e.g. for vehicle drivers A61B5/16, A61B5/18

Abstract

A method, information processing system, and computer program storage product annotate video images associated with an environmental situation based on detected actions of a human interacting with the environmental situation. A set of real-time video images are received that are captured by at least one video camera associated with an environment presenting one or more environmental situations to a human. One or more user actions made by the human that is associated with the set of real-time video images with respect to the environmental situation are monitored. A determination is made, based on the monitoring, that the human driver has one of performed and failed to perform at least one action associated with one or more images of the set of real-time video images. The one or more images of the set of real-time video images are annotated with a set of annotations.

Claims

1. A method of annotating video images associated with an environmental situation based on detected actions of a human interacting with the environmental situation, the method comprising:

receiving, with an information processing system, a set of real-time video images captured by at least one video camera associated with an environment presenting one or more environmental situations to a human utilizing the environment;

determining, based on one or more images in the set of real-time video images, that the human is encountering a given environmental situation;

storing, based on the determining, the one or more images;

monitoring, with the information processing system and based on the determining, one or more user actions made by the human with respect to the environmental situation;

determining, based on the monitoring, that the human has one of performed and failed to perform at least one action associated with the environmental situation; and

annotating, with the information processing system, the one or more images with a set of annotations based on the at least one action that has been one of performed and failed to be performed by the human.

2. The method of claim 1, further comprising:

determining an association between one or more consecutive images in the set of real-time video images associated with the at least one action that has been one of performed and failed to be performed by the human, wherein the association defines a time interval and an annotation information set associated with that time interval.

3. The method of claim 1, wherein annotating the one or more images comprises:

storing an information set associated with the one or more images in a data store, wherein the information set comprises data associated with the at least one action that has been one of performed and failed to be performed by the human.

4. The method of claim 1, further comprising:

aggregating the one or more images of the set of real-time video images with a plurality of image sets, wherein each image set in the plurality of image sets is associated with a set of real-time video images each associated with another environmental situation of the environment that is substantially similar to the environmental situation.

5. The method of claim 4, wherein each image set in the plurality of image sets is associated with a different human that has one of performed and failed to perform at least one action with respect to the another environmental situation that is substantially similar to the environmental situation.

6. The method of claim 5, wherein each image set in the plurality of image sets is annotated based on the at least action that has been one of performed and failed to be performed by the different human associated therewith.

7. The method of claim 1, wherein the environment is a moving vehicle, and wherein the one or more user actions are at least one vehicle control and maneuver action.

8. An information processing system for annotating video images associated with an environment of a moving vehicle, based on detected human actions of a driver of the moving vehicle, the information processing system comprising:

a memory;

a processor communicatively coupled to the memory;

an environment manager communicatively coupled to the memory and the processor, wherein the environment manager is configured to;

receive, with an information processing system, a set of real-time video images captured by at least one video camera associated with an environment of a moving vehicle, wherein the set of real-time video images are associated specifically with at least one vehicle control and maneuver environmental situation of the moving vehicle;

monitor, with the information processing system, one or more user control input signals corresponding to one or more vehicle control and maneuver actions made by a human driver of the moving vehicle that is associated with the set of real-time video images, with respect to the vehicle control and maneuver environmental situation;

determine, based on the monitoring, that the human driver has performed at least one vehicle control and maneuver action associated with one or more images of the set of real-time video images;

annotate, with the information processing system, the one or more images of the set of real-time video images with a set of annotations based on the at least one vehicle control and maneuver action performed by the human driver; and

aggregate the one or more images of the set of real-time video images with a plurality of image sets, wherein each image set in the plurality of image sets is associated with a set of real-time video images each associated with another vehicle control and maneuver environmental situation of the moving vehicle that is substantially similar to the at least one vehicle control and maneuver environmental situation of the moving vehicle.

9. The information processing system of claim 8, wherein the environment manager is further configured to annotate the one or more images by at least one of:

time-stamping the one or more images; and

appending an information set to the one or more images, wherein the information set is associated with the at least one vehicle control and maneuver action performed by the human driver.

10. The information processing system of claim 8, wherein the environment manager is further configured to annotate the one or more images by at least:

storing an information set associated with the one or more images in a data store, wherein the information set comprises data associated with the at least one vehicle control and maneuver action performed by the human driver.

11. The information processing system of claim 8, wherein a different human driver is associated with one or more image sets in the plurality of image sets, wherein the different human driver has performed at least one vehicle control and maneuver action with respect to the another vehicle control and maneuver environmental situation that is substantially similar to the at least one vehicle control and maneuver environmental situation.

12. The information processing system of claim 11, wherein each image set in the plurality of image sets is annotated based on the at least one vehicle control and maneuver action performed by the different human driver associated therewith.

13. The information processing system of claim 8, wherein the environment manager is further configured to identify by:

comparing the set of annotations of the one or more images with at least one set of annotations associated with a set of images that have been annotated based on a different human driver that has performed at least one vehicle control and maneuver action with respect to another vehicle control and maneuver environmental situation that is substantially similar to the at least one vehicle control and maneuver environmental situation of the human driver of the moving vehicle.

14. A non-transitory computer program storage product having a computer program stored thereon for annotating video images associated with an environment of a moving vehicle, based on detected human actions of a driver of the moving vehicle, the computer program comprising instructions for:

receiving a set of real-time video images captured by at least one video camera associated with an environment of a moving vehicle, wherein the set of real-time video images are associated specifically with at least one vehicle control and maneuver environmental situation of the moving vehicle;

determining, based on one or more images in the set of real-time video images, that a human driver of the moving vehicle is encountering a given environmental situation;

storing, based on the determining, the one or more images;

monitoring, and based on the determining, one or more user control input signals corresponding to one or more vehicle control and maneuver actions made by the human driver of the moving vehicle with respect to the vehicle control and maneuver environmental situation;

determining, based on the monitoring, that the human driver has performed at least one vehicle control and maneuver action associated with the vehicle control and maneuver environmental situation; and

annotating the one or more images with a set of annotations based on the at least one vehicle control and maneuver action performed by the human driver as indicated by the one or more user control input signals.

15. The non-transitory computer program storage product of claim 14, further comprising instructions for:

determining an association between one or more consecutive images in the set of real-time video images associated with the at least one action that has been one of performed and failed to be performed by the human, wherein the association defines a time interval and an annotation information set associated with that time interval.

16. The non-transitory computer program storage product of claim 14, further comprising instructions for:

aggregating the one or more images of the set of real-time video images with a plurality of image sets, wherein each image set in the plurality of image sets is associated with a set of real-time video images each associated with a vehicle control and maneuver environmental situation of the moving vehicle that is substantially similar to the at least one vehicle control and maneuver environmental situation of the moving vehicle.

17. The non-transitory computer program storage product of claim 16, wherein a different driver is associated with one or more images sets in the plurality of image sets, wherein the human driver has performed at least one vehicle control and maneuver action with respect to the vehicle control and maneuver environmental situation that is substantially similar to the at least one vehicle control and maneuver environmental situation.