Video analysis method and system

• US 10,489,654 B1
• Filed: 08/04/2017
• Issued: 11/26/2019
• Est. Priority Date: 08/04/2017
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Olgiati, Andrea, Singhal, Nitin
• Primary Examiner(s): Couso, Jose L
• Application Number:
  US15/669,659
• Time in Patent Office: 844 Days
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/42 of sport video content
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes

Abstract

A system processes a video stream to identify an activity performed by a subject. The system examines the video segment for the presence or absence of an object that is correlated with the activity. If the object is found, and if the object is positively correlated with the detected activity, a confidence level associated with detecting the activity is increase. If the object is found, and if it is negatively correlated with the activity, the confidence level is decreased. In other examples, absence of an object may be positively or negatively correlated with the activity. In one embodiment, the system may detect a number of alternative activities, and may use the presence or absence of various objects to determine the most likely activity being performed. In another embodiment, the system may use the presence of various objects to narrow the set of activities that are searched for.

Claims

1. A computer-implemented method, comprising:

obtaining, by a computing system, a digital video;

detecting, by the computing system, a person recorded in the video;

determining, by the computing system, a set of a plurality of different possible activities corresponding to a motion of the person recorded in the video; and

using, by the computing system, the identity of an object to select one activity from the set of the plurality of possible activities as the identity of the motion of the person and to exclude the other activities of the plurality of different possible activities as being the identity of the motion of the person.

2. The computer-implemented method of claim 1, further comprising:

detecting an interaction between the person and the object; and

using the interaction to select the activity.

3. The computer-implemented method of claim 1, further comprising:

detecting another object in the video; and

using the identity of the other object to determine the set of activities.

4. The computer-implemented method of claim 1, wherein determining an activity is accomplished at least in part by:

identifying a set of objects that are correlated with the set of activities;

detecting whether each object in a set of objects is present in the video; and

using the detection of each object in the set of objects to select the activity.

5. A computing system, comprising:

one or more non-transitory machine-readable mediums having stored thereon a set of instructions, which if performed by one or more processors, cause the computing system to at least;

obtain a video that includes one or more video segments, the one or more video segments including a segment of video;

detect a first object in the segment of video; and

determine an activity performed by the first object in the segment of video based at least in part on a second object in the segment of video, the activity determined from a set of a plurality of possible activities performed by the first object based at least in part on the identity of the second object, with the other activities of the plurality of possible activities being excluded as the identity of a motion of the first object based at least in part of the identity of the second object.

6. The computing system of claim 5, wherein the instructions further cause the computing system to:

determine a set of objects that are correlated to the activity; and

search the segment of video for presence of each object in the set of objects.

7. The computing system of claim 5, wherein:

the absence of the second object is positively correlated with the activity; and

the instructions cause the computing system to determine the activity at least in part by determining that the second object is not present in the segment of video.

8. The computing system of claim 5, wherein the instructions further cause the computing system to:

determine a confidence level associated with the determining of the activity in the segment of video;

adjust the confidence level based at least in part on having detected presence of the second object in the segment of video; and

wherein presence of the second object is correlated with the activity.

9. The computing system of claim 8, wherein:

the confidence level is decreased as a result of detecting presence of the second object; and

the second object is negatively correlated with the activity.

10. The computing system of claim 5, wherein the instructions further cause the computing system to:

acquire a video stream that includes a plurality of video segments, the plurality of video segments including the segment of video;

identify a set of video segments from the plurality of video segments that contain a confirming object, the confirming object positively correlated with the activity; and

search the set of video segments for presence of the activity.

11. The computing system of claim 5, wherein the activity is detected by at least:

determining a set of possible activities performed by the first object;

determining a set of objects that are correlated to a set of possible activities, the set of objects including the second object, and the set of activities including the activity; and

selecting the activity from the set of possible activities based at least in part on presence or absence of each object in the set of objects in the segment of video.

12. The computing system of claim 5, wherein:

the first object is a human subject; and

the activity is determined at least in part by determining a relative position between the first object and the second object.

13. A non-transitory computer-readable storage medium having stored thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to at least:

acquire a video that includes one or more video segments, the one or more video segments including a segment of video;

detect a first object in the segment of video; and

determine an activity performed by the first object in the segment of video based at least in part on the identity of a second object in the segment of video, the activity determined from a set of a plurality of identified possible movement activities of the first object, with the other activities of the plurality of identified possible movement activities of the first object being excluded as the identity of the activity of the first object based at least in part of the identity of the second object.

14. The non-transitory computer-readable storage medium of claim 13, wherein the instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to:

identify the segment of video from a video stream; and

identify the second object by analyzing an individual frame in the segment of video.

15. The non-transitory computer-readable storage medium of claim 13, wherein:

the activity is an action performed by a human subject; and

the human subject interacts with the second object.

16. The non-transitory computer-readable storage medium of claim 13, wherein the instructions that cause the computer system to:

identify a set of objects that are correlated with the activity, the set of objects including the second object; and

determine whether each object in the set of objects is present in the segment of video, without searching for presence of other objects.

17. The non-transitory computer-readable storage medium of claim 13, wherein the instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to:

determine that the activity is one of a class of activities; and

identify another activity from the class of activities based at least in part on the presence of the second object.

18. The non-transitory computer-readable storage medium of claim 13, wherein the instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to:

determine a confidence level associated with detection of the activity in the segment of video; and

adjust the confidence level based at least in part on having detected the absence of the second object in the segment of video; and

wherein presence of the second object is correlated with the activity.

19. The non-transitory computer-readable storage medium of claim 18, wherein:

the confidence level is decreased as a result of detecting the absence of the second object; and

the absence of the second object is negatively correlated with the activity.

20. The non-transitory computer-readable storage medium of claim 18, wherein:

the confidence level is increased as a result of detecting the absence of the second object; and

the absence of the second object is positively correlated with the activity.