Activity recognition method and system

• US 10,691,950 B2
• Filed: 03/07/2018
• Issued: 06/23/2020
• Est. Priority Date: 03/10/2017
• Status: Active Grant

• Current Assignee: Turing Video
• Original Assignee: Turing Video, Inc.
• Inventors: Cao, Song, Wang, Fang
• Primary Examiner(s): Lu, Zhiyu
• Application Number:
  US15/914,360
• Publication Number:
  US 20180260630A1
• Time in Patent Office: 839 Days
• CPC Class Codes: G06F 18/24317 Piecewise classification, i.e. whereby each classification requires several discriminant rules
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 21/22 responsive to presence or absence of persons
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/2743 Video hosting of uploaded data from client
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

An activity recognition system may comprise a local device and a server. The local device may be configured to: obtain a video; determine whether at least one human is present in one or more frames of the video; in response to determining the at least one human present in the video, determine if the at least one human in each of the frames corresponds to the same person; in response to determining that the at least one human in each of the frames corresponds to the same person, determine if the person is a stranger; and in response to determining that the person is not a stranger, transmit at least a portion of the video to the server. The server may be configured to: predict an activity class associated with the person; and trigger an alarm based on the activity class.

Claims

1. An activity recognition system, comprising a local device and a server, wherein:

the local device is configured to;

obtain a video;

determine whether at least one human is present in one or more frames of the video;

in response to determining the at least one human present in the video, determine if the at least one human in each of the frames corresponds to the same person;

in response to determining that the at least one human in each of the frames corresponds to the same person, determine if the person is a stranger; and

in response to determining that the person is not a stranger, transmit at least a portion of the video to the server; and

the server is configured to;

predict an activity class associated with the person; and

trigger an alarm based on the activity class.

2. The activity recognition system according to claim 1, wherein:

the local device comprises a camera; and

to obtain the video, the local device is configured to cause the camera to stream the video in real time.

3. The activity recognition system according to claim 1, wherein to determine whether the at least one human is present in the one or more frames of the video, the local device is configured to:

execute a human detection algorithm for each of the frames to determine a percentage of the frames comprising the at least one human; and

in response to determining the percentage exceeding a threshold, determine that the at least one human is present in the one or more frames of the video.

4. The activity recognition system according to claim 1, wherein:

the local device comprises a camera configured to obtain the video, the video capturing an environment;

to execute the human detection algorithm for the frame, the local device is configured to;

initialize a background, wherein the background is updated in real time and the update includes at least one of a viewpoint of the camera or an illumination of the environment; and

compare an input associated with the frame with the background to perform human detection for the frame.

5. The activity recognition system according to claim 1, wherein to determine if the at least one human in each of the frames corresponds to the same person, the local device is configured to:

execute a re-identification algorithm to verify one or more features associated with the at least one human across the one or more frames; and

determine if the one or more features are consistent.

6. The activity recognition system according to claim 1, wherein:

to determine if the person is the stranger, the local device is configured to;

execute a stranger detection algorithm to determine an identity of the person; and

in response to determining the identity not in a white list, determine the person as the stranger; and

the local device is further configured to;

in response to determining that the person is a stranger, trigger the alarm.

7. The activity recognition system according to claim 1, wherein to predict the activity class associated with the person, the server is configured to:

extract statistical scene features from the at least one portion of the video;

extract temporal features from the at least one portion of the video;

extract pose features from the at least one portion of the video; and

combine the statistical scene features, the temporal features, and the pose features to predict the activity class of the person.

8. The activity recognition system according to claim 7, wherein the pose feature comprises a plurality of pose patches, the pose patches each associated with full body, upper body, left elbow, and right elbow.

9. The activity recognition system according to claim 7, wherein the server is further configured to:

obtain deformable features from the at least one portion of the video; and

combine the statistical scene features, the temporal features, the pose features, and the deformable features to predict the activity class of the person.

10. An activity recognition method, comprising:

obtaining, by a local device, a video;

determining, by the local device, whether at least one human is present in one or more frames of the video;

in response to determining the at least one human present in the video, determining, by the local device, if the at least one human in each of the frames corresponds to the same person;

in response to determining that the at least one human in each of the frames corresponds to the same person, determining, by the local device, if the person is a stranger;

in response to determining that the person is not a stranger, transmitting, by the local device, at least a portion of the video to a server;

predicting, by the server, an activity class associated with the person; and

triggering, by the server, an alarm based on the activity class.

11. The activity recognition method according to claim 10, wherein determining whether the at least one human is present in the one or more frames of the video comprises:

executing a human detection algorithm for each of the frames to determine a percentage of the frames comprising the at least one human; and

in response to determining the percentage exceeding a threshold, determining that the at least one human is present in the one or more frames of the video.

12. The activity recognition method according to claim 10, wherein:

the video is obtained from a camera, the video capturing an environment;

executing the human detection algorithm for the frame comprises;

initializing a background, wherein the background is updated in real time and the update includes at least one of a viewpoint of the camera or an illumination of the environment; and

comparing an input associated with the frame with the background to perform human detection for the frame.

13. The activity recognition method according to claim 10, wherein determining if the at least one human in each of the frames corresponds to the same person comprises:

executing a re-identification algorithm to verify one or more features associated with the at least one human across the one or more frames; and

determining if the one or more features are consistent.

14. The activity recognition method according to claim 10, wherein:

determining if the person is the stranger comprises;

executing a stranger detection algorithm to determine an identity of the person; and

in response to determining the identity not in a white list, determining the person as the stranger; and

the method further comprises;

in response to determining that the person is a stranger, triggering the alarm.

15. The activity recognition method according to claim 10, wherein predicting the activity class associated with the person comprises:

extracting statistical scene features from the at least one portion of the video;

extracting temporal features from the at least one portion of the video;

extracting pose features from the at least one portion of the video; and

combining the statistical scene features, the temporal features, and the pose features to predict the activity class of the person.

16. The activity recognition method according to claim 15, wherein the pose feature comprises a plurality of pose patches, the pose patches each associated with full body, upper body, left elbow, and right elbow.

17. The activity recognition method according to claim 15, further comprising obtaining deformable features from the at least one portion of the video, wherein predicting the activity class associated with the person comprises:

combining the statistical scene features, the temporal features, the pose features, and the deformable features to predict the activity class of the person.

18. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processor to perform an activity recognition method, the method comprising:

(1) obtaining, by a local device, a video;

(2) determining, by the local device, whether at least one human is present in one or more frames of the video;

(3) in response to determining the at least one human present in the video, determining, by the local device, if the at least one human in each of the frames corresponds to the same person;

(4) in response to determining that the at least one human in each of the frames corresponds to the same person, determining, by the local device, if the person is a stranger;

(5) in response to determining that the person is not a stranger, transmitting, by the local device, at least a portion of the video to a server;

(6) predicting, by the server, an activity class associated with the person; and

(7) triggering, by the server, an alarm based on the activity class.

19. The non-transitory computer-readable storage medium of claim 18, wherein the one or more processors comprise at least a first processor associated with the local device, and at least a second processor associated with the server.

20. The non-transitory computer-readable storage medium of claim 18, wherein:

the first processor is caused to perform (1)-(5);

the second processor is caused to perform (6)-(7); and

performing (6) requires more computing power and more costs than performing (1)-(5).