System and method for generating an activity summary of a person

• US 10,271,017 B2
• Filed: 05/07/2013
• Issued: 04/23/2019
• Est. Priority Date: 09/13/2012
• Status: Active Grant

• Current Assignee: General Electric Company
• Original Assignee: General Electric Company
• Inventors: Tu, Peter Henry, Yu, Ting, Gao, Dashan, Yao, Yi
• Primary Examiner(s): Harold, Jefferey F
• Assistant Examiner(s): Satti, Humam M
• Application Number:
  US13/888,941
• Publication Number:
  US 20140071287A1
• Time in Patent Office: 2,177 Days
• Field of Search: 348143, 348144, 348150, 348152, 348153, 348159
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19602 Image analysis to detect motion of the intruder, e.g. by frame subtraction
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

In accordance with one aspect of the present technique, a method includes receiving one or more videos from one or more image capture devices. The method further includes generating a video-loop of the person from the one or more videos. The video-loop depicts the person in the commercial site. The method also includes generating an action clip from the video-loop. The action clip includes a suspicious action performed by the person in the commercial site. The method further includes generating an activity summary of the person including the video-loop and the action clip.

Claims

1. A method for generating an activity summary of a person in a commercial site, the method comprising:

receiving one or more videos from one or more image capture devices;

determining metadata associated with the person from the one or more videos, the metadata including location data of the person and an appearance descriptor that represents a spatial distribution of color corresponding to the person;

generating a video-loop of the person from the one or more videos by combining sets of images from the one or more videos based on a similarity of the metadata associated with the person, wherein the video-loop includes images of an entire trip, including all activities, of the person in the commercial site;

generating at least one action clip from the video-loop, wherein the at least one action clip includes a suspicious action performed by the person in the commercial site; and

generating the activity summary of the person including the video-loop and the at least one action clip.

2. The method of claim 1, wherein the suspicious action includes at least one of grasping an object, removing a component from the object and hiding the object.

3. The method of claim 1, wherein generating the at least one action clip further comprises:

analyzing one or more images of the video-loop to determine the suspicious action performed by the person;

determining an image analysis score for each of the one or more images based on the analysis of the one or more images;

identifying a suspicious image from the one or more images based on the one or more image analysis scores; and

generating the at least one action clip including the identified suspicious image from the video-loop.

4. The method of claim 3, wherein the at least one action clip begins with the identified suspicious image.

5. The method of claim 1, wherein generating the action dip further comprises:

analyzing one or more sequences of images from the video-loop to determine one or more motion features associated with the person;

identifying a suspicious sequence of images from the one or more sequences of images based on the one or more motion features; and

generating the at least one action clip including the suspicious sequence of images from the video-loop.

6. The method of claim 1, wherein generating the video-loop of the person further comprises:

receiving a first video of the one or more videos from a first image capture device of the one or more image capturing devices

identifying a first set of images including the person from the first video;

receiving a second video of the one or more videos from a second image capture device of the one or more image capturing devices;

identifying a second set of images including the person from the second video; and

generating the video-loop of the person by combining the first set of images and the second set of images.

7. The method of claim 1, further comprising:

determining whether the person is approaching an exit of the commercial site; and

sending the activity summary for display in response to determining that the person is approaching the exit of the commercial site.

18. The method of claim 1, further comprising:

identifying one or more spatiotemporal interest points from the video-loop based on two-dimensional Gaussian smoothing and temporal Gabor filtering.

19. The method of claim 18, further comprising:

analyzing a sequence of images represented by the one or more spatiotemporal interest points to determine shape features and motion features associated with the person.

20. The method of claim 19, wherein a grasping action is determined as motion feature through an Adaboost algorithm or Fisher'"'"'s linear discriminant algorithm.

8. A system for generating an activity summary of a person in a commercial site, the system comprising:

at least one processor;

a tracking module stored in a memory and, executable by the at least one processor, the tracking module for receiving one or more videos from one or more image capture devices, determining metadata associated with the person from the one or more videos, the metadata including location data of the person and an appearance descriptor that represents a spatial distribution of color corresponding to the person, and generating a video-loop of the person from the one or more videos by combining sets of images from the one or more videos based on a similarity of the metadata associated with the person, wherein the video-loop includes images of an entire trip, including all activities, of the person in the commercial site;

an analysis module stored in the memory and executable by the at least one processor, the analysis module communicatively coupled to the tracking module for generating at least one action clip from the video-loop, wherein the at least one action clip includes a suspicious action performed by the person in the commercial site; and

a summary generator stored in the memory and executable by the at least one processor, the summary generator communicatively coupled to the analysis module for generating the activity summary of the person including the video-loop and the at least one action clip.

9. The system of claim 8, wherein the analysis module is further configured to:

analyze one or more images of the video-loop to determine the suspcious action performed by the person;

determine an image analysis score for each of the one or more images based on the analysis of the one or more images;

identify a suspicious image from the one or more images based on the one or more image analysis scores; and

generate the at least one action clip including the identified suspicious image from the video-loop.

10. The system of claim 8, wherein the analysis module is further configured to:

analyze one or more sequences of images from the video-loop to determine one or more motion features associated with the person;

identify a suspicious sequence of images from the one or more sequences of images based on the one or more motion features; and

generate the at least one action clip including the suspicious sequence of images from the video-loop.

11. The system of claim 8, wherein the tracking module is further configured to:

receive a first video of the one or more videos from a first image capture device of the one or more image capturing devices;

identify a first set of images including the person from the first video;

receive a second video of the one or more videos from a second image capture device of the one or more image capturing devices;

identify a second set of images including the person from the second video; and

generate the video-loop of the person by combining the first set of images and the second set of images.

12. The system of claim 8, wherein the summary generator is further configured to determine whether the person is approaching an exit of the commercial site and send the activity summary for display in response to determining that the person is approaching the exit of the commercial site.

13. A computer program product comprising a non-transitory computer readable medium encoding instructions that, in response to execution by at least one processor, cause the processor to perform operations comprising:

receiving one or more videos from one or more image capture devices;

determining metadata associated with the person from the one or more videos, the metadata including location data of the person and an appearance descriptor that represents a spatial distribution of color corresponding to the person;

generating a video-loop of a person from the one or more videos by combining sets of images from the one or more videos based on a similarity of the metadata associated with the person, wherein the video-loop includes images of an entire trip, including all activities, of the person in a commercial site;

generating at least one action clip from the video-loop, wherein the at least one action clip includes a suspicious action performed by the person in the commercial site; and

generating an activity summary of the person including the video-loop and the at least one action clip.

14. The computer program product of claim 13, further causing the processor to perform operations comprising:

analyzing one or more images of the video-loop to determine the suspcious action performed by the person;

determining an image analysis score for each of the one or more images based on the analysis of the one or more images;

identifying a suspicious image from the one or more images based on the one or more image analysis scores; and

generating the action dip including the identified suspicious image from the video-loop.

15. The computer program product of claim 14, wherein the at least one action clip begins with the suspicious image.

16. The computer program product of claim 13, further causing the processor to perform operations comprising:

analyzing one or more sequence of images from the video-loop to determine one or amore motion features associated with the person;

identifying a suspicious sequence of images from the one or more sequence of images based on the one or more motion features; and

generating the at least one action clip including the suspicious sequence of images from the video-loop.

17. The computer program product of claim 13, further causing the processor to perform operations comprising:

determining whether the person is approaching an exit of the commercial site; and

sending the activity summary for display in response to determining that the person is approaching the exit of the commercial site.