PERSON COUNTING METHOD AND PERSON COUNTING SYSTEM

• US 20200134875A1
• Filed: 10/17/2019
• Published: 04/30/2020
• Est. Priority Date: 10/24/2018
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: YI, Hong, JIA, Haijing, GONG, Weitao, WANG, Wei
• Granted Patent: US 11,055,878 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30242 Counting objects in image
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A person counting method and a person counting system are provided. The method includes extracting a group of person images to obtain a first image set; dividing the first image set into first and second subsets based on whether a related image exists in a second image set, and reusing a person ID of the related image; estimating posture patterns of images in the first subset, and storing the images in the first subset into an image library based on person IDs and the posture patterns; and selecting a target image whose similarity to an image in the second subset is highest from the image library, reusing a person ID of the target image when the similarity is greater than a threshold, and assigning a new person ID and incrementing a person counter by 1 when the similarity is not greater than the threshold.

Claims

1. A person counting method comprising:

extracting, from a first video frame of a video source, a group of images of persons to obtain a first image set, each image corresponding to one person;

dividing the first image set into a first subset and a second subset based on whether a related image corresponding to an image in the first image set exists in a second image set, and reusing a person ID of the related image in the second image set for the corresponding image in the first subset, the second image set being a group of images of persons extracted from a video frame preceding the first video frame, a prediction position where the related image in the second image set is located in the first video frame matching an actual position where the corresponding image in the first image set is located in the first video frame;

estimating posture patterns of images in the first subset, and storing the images in the first subset into an image library based on person IDs and the posture patterns; and

for each image in the second subset, selecting, from the image library, a target image whose similarity to the image in the second subset is highest, reusing a person ID of the target image for the corresponding image in the second subset when the similarity of the target image is greater than a predetermined threshold, and assigning a new person ID to the corresponding image in the second subset and outputting a value of a counter for counting persons after incrementing the counter by 1 when the similarity of the target image is less than or equal to the predetermined threshold.

2. The person counting method as claimed in claim 1,

wherein the video frames of the video source are panoramic video frames, and

wherein the person counting method further includes

performing panoramic image transform processing on the images in the first subset and the second subset to update the images in the first subset and the second subset, before estimating the posture patterns of the images in the first subset.

3. The person counting method as claimed in claim 2,

wherein performing the panoramic image transform processing on the images in the first subset and the second subset includes

updating, by a perspective projection transformation algorithm, the images in the first subset and the second subset.

4. The person counting method as claimed in claim 1,

wherein estimating the posture patterns of the images in the first subset includes

for each image to be estimated in the first subset, predicting, based on a position change of an image having the same person ID as the image to be estimated in continuous video frames, a moving direction of the image to be estimated;

calculating an included angle between the moving direction and an equator of the first video frame; and

determining, based on a predetermined correspondence between angles and posture patterns, a posture pattern corresponding to the included angle as the posture pattern of the image to be estimated.

5. The person counting method as claimed in claim 1,

wherein storing the images in the first subset into the image library includes

storing each image in the first subset, the person ID and the posture pattern of the image, position information of the image in a video frame to which the image belongs, and timestamp information of the video frame to which the image belongs into a record having the same person ID and the same posture pattern as the image in the image library.

6. The person counting method as claimed in claim 5,

wherein selecting the target image whose similarity to the image in the second subset is highest from the image library includes

recognizing, by a deep learning algorithm, each image to be recognized in the second subset and each image in the image library to obtain top N candidate images in a descending order of scores, N being an integer greater than 1;

and

calculating, based on position information of the image to be recognized and the candidate images, position similarities;

calculating, based on timestamp information of the image to be recognized and the candidate images, time similarities; and

obtaining similarity scores of the candidate images by calculating weighted sums of the respective position similarities and the respective time similarities, and determining the candidate image with the highest similarity score as the target image.

7. A person counting system comprising:

an image extracting unit configured to extract, from a first video frame of a video source, a group of images of persons to obtain a first image set, each image corresponding to one person;

a set dividing unit configured to divide the first image set into a first subset and a second subset based on whether a related image corresponding to an image in the first image set exists in a second image set, and reuse a person ID of the related image in the second image set for the corresponding image in the first subset, the second image set being a group of images of persons extracted from a video frame preceding the first video frame, a prediction position where the related image in the second image set is located in the first video frame matching an actual position where the corresponding image in the first image set is located in the first video frame;

a posture estimating unit configured to estimate posture patterns of images in the first subset, and store the images in the first subset into an image library based on person IDs and the posture patterns; and

a person counting unit configured to select, for each image in the second subset, from the image library, a target image whose similarity to the image in the second subset is highest, reuse a person ID of the target image for the corresponding image in the second subset when the similarity of the target image is greater than a predetermined threshold, and assign a new person ID to the corresponding image in the second subset and outputting a value of a counter for counting persons after incrementing the counter by 1 when the similarity of the target image is less than or equal to the predetermined threshold.

8. The person counting system as claimed in claim 7,

wherein the video frames of the video source are panoramic video frames, and

wherein the set dividing unit further includes

an image transform unit configured to perform panoramic image transform processing on the images in the first subset and the second subset to update the images in the first subset and the second subset.

9. The person counting system as claimed in claim 8,

wherein the image transform unit updates, by a perspective projection transformation algorithm, the images in the first subset and the second subset.

10. The person counting system as claimed in claim 7,

wherein the posture estimating unit for each image to be estimated in the first subset, predicts, based on a position change of an image having the same person ID as the image to be estimated in continuous video frames, a moving direction of the image to be estimated;

calculates an included angle between the moving direction and an equator of the first video frame; and

determines, based on a predetermined correspondence between angles and posture patterns, a posture pattern corresponding to the included angle as the posture pattern of the image to be estimated.

11. The person counting system as claimed in claim 7,

wherein the posture estimating unit stores each image in the first subset, the person ID and the posture pattern of the image, position information of the image in a video frame to which the image belongs, and timestamp information of the video frame to which the image belongs into a record having the same person ID and the same posture pattern as the image in the image library.

12. The person counting system as claimed in claim 11,

wherein the person counting unit

recognizes, by a deep learning algorithm, each image to be recognized in the second subset and each image in the image library to obtain top N candidate images in a descending order of scores, N being an integer greater than 1;

and

calculates, based on position information of the image to be recognized and the candidate images, position similarities;

calculates, based on timestamp information of the image to be recognized and the candidate images, time similarities; and

obtains similarity scores of the candidate images by calculating weighted sums of the respective position similarities and the respective time similarities, and determines the candidate image with the highest similarity score as the target image.

13. A person counting system comprising:

a memory storing computer-readable instructions; and

one or more processors configured to execute the computer-readable instructions such that the one or more processors are configured to

extract, from a first video frame of a video source, a group of images of persons to obtain a first image set, each image corresponding to one person;

divide the first image set into a first subset and a second subset based on whether a related image corresponding to an image in the first image set exists in a second image set, and reuse a person ID of the related image in the second image set for the corresponding image in the first subset, the second image set being a group of images of persons extracted from a video frame preceding the first video frame, a prediction position where the related image in the second image set is located in the first video frame matching an actual position where the corresponding image in the first image set is located in the first video frame;

estimate posture patterns of images in the first subset, and store the images in the first subset into an image library based on person IDs and the posture patterns; and

for each image in the second subset, select, from the image library, a target image whose similarity to the image in the second subset is highest, reuse a person ID of the target image for the corresponding image in the second subset when the similarity of the target image is greater than a predetermined threshold, and assign a new person ID to the corresponding image in the second subset and outputting a value of a counter for counting persons after incrementing the counter by 1 when the similarity of the target image is less than or equal to the predetermined threshold.

14. The person counting system as claimed in claim 13,

wherein the video frames of the video source are panoramic video frames, and

wherein the one or more processors are configured to

perform panoramic image transform processing on the images in the first subset and the second subset to update the images in the first subset and the second subset.

15. The person counting system as claimed in claim 14,

wherein the one or more processors are configured to

update, by a perspective projection transformation algorithm, the images in the first subset and the second subset.

16. The person counting system as claimed in claim 13,

wherein the one or more processors are configured to

for each image to be estimated in the first subset, predict, based on a position change of an image having the same person ID as the image to be estimated in continuous video frames, a moving direction of the image to be estimated;

calculate an included angle between the moving direction and an equator of the first video frame; and

determine, based on a predetermined correspondence between angles and posture patterns, a posture pattern corresponding to the included angle as the posture pattern of the image to be estimated.

17. The person counting system as claimed in claim 13,

wherein the one or more processors are configured to

store each image in the first subset, the person ID and the posture pattern of the image, position information of the image in a video frame to which the image belongs, and timestamp information of the video frame to which the image belongs into a record having the same person ID and the same posture pattern as the image in the image library.

18. The person counting system as claimed in claim 17,

wherein the one or more processors are configured to

recognize, by a deep learning algorithm, each image to be recognized in the second subset and each image in the image library to obtain top N candidate images in a descending order of scores, N being an integer greater than 1;

calculate, based on position information of the image to be recognized and the candidate images, position similarities;

calculate, based on timestamp information of the image to be recognized and the candidate images, time similarities; and

obtain similarity scores of the candidate images by calculating weighted sums of the respective position similarities and the respective time similarities, and determine the candidate image with the highest similarity score as the target image.