Person behavior analysis device, person behavior analysis system, person behavior analysis method, and monitoring device for detecting a part of interest of a person

• US 9,558,398 B2
• Filed: 06/30/2014
• Issued: 01/31/2017
• Est. Priority Date: 07/02/2013
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Inventors: Iwai, Kazuhiko
• Primary Examiner(s): Shah, Utpal
• Application Number:
  US14/319,042
• Publication Number:
  US 20150010204A1
• Time in Patent Office: 946 Days
• US Class Current: 1/1
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A behavior analysis/monitoring device includes: a person detection unit configured to detect a person(s) from image information obtained by capturing images covering an area around an item placement area; a part-of-interest detection unit configured to detect, for each person detected by the person detection unit, a part of interest set in a part of an upper body of the person excluding hands and arms; a position measurement unit configured to measure a position of the part of interest detected by the part-of-interest detection unit; and an item pick-up action determination unit configured to obtain a displacement of the part of interest based on the position of the part of interest obtained by the position measurement unit and to determine whether each person detected by the person detection unit performed an item pick-up action based on the displacement of the part of interest of the person.

Claims

1. A person behavior analysis device for performing analysis of item pick-up actions, which are actions of picking up an item placed in an item placement area, the device comprising:

a person detection processor configured to detect one or more persons from image information obtained by capturing images covering an area around the item placement area;

a part-of-interest detection processor configured to detect, for each person detected by the person detection processor, a part of interest set only in a part of an upper body of the person excluding hands and arms;

a position measurement processor configured to measure a position of the part of interest detected by the part-of-interest detection processor;

an item pick-up action determination processor configured to obtain a displacement of the part of interest based on the position of the part of interest obtained by the position measurement processor and to determine whether each person detected by the person detection processor performed an item pick-up action based on the displacement of the part of interest of the person; and

an output information generation processor configured to generate output information representing a result of analysis based on a result of determination by the item pick-up action determination processor.

2. The person behavior analysis device according to claim 1, wherein the part of interest is a head or shoulder of a person.

3. The person behavior analysis device according to claim 1, wherein the item placement area includes a display shelf having a plurality of shelf boards on which items are placed such that the shelf boards are spaced apart in a vertical direction.

4. The person behavior analysis device according to claim 1, further comprising:

a vicinity area setting processor configured to set, around the item placement area, a vicinity area which is a maximum area in which a person can touch an item in the item placement area by extending a hand; and

an area entry determination processor configured to determine that, of the persons detected by the person detection processor, a person who did not enter the vicinity area is a person who could not perform the item pick-up action.

5. The person behavior analysis device according to claim 4, wherein the vicinity area setting processor sets the vicinity area based on an arm length estimated from a result of measurement by the position measurement processor in accordance with correlation between height and arm length.

6. The person behavior analysis device according to claim 4, further comprising:

a staying time measurement processor configured to measure, for each person entering the vicinity area, a staying time in the vicinity area; and

a staying time assessment processor configured to determine that, of the persons detected by the person detection processor, a person whose staying time does not reach a predetermined threshold value is a person who could not perform the item pick-up action.

7. The person behavior analysis device according to claim 4, wherein the item pick-up action determination processor determines whether each person entering the vicinity area performed the item pick-up action based on a displacement of the part of interest of the person with respect to a position of the part of interest when the person entered the vicinity area in an upright posture.

8. The person behavior analysis device according to claim 1, wherein the item pick-up action determination processor determines whether each person detected by the person detection processor performed the item pick-up action based on a positional relationship between a reference position set in the item placement area and the part of interest of the person.

9. The person behavior analysis device according to claim 2, wherein the position measurement processor measures a central point of a head of each person.

10. The person behavior analysis device according to claim 1, wherein the position measurement processor obtains information relating to an apparent size of the part of interest in captured images from the image information, and measures the position of the part of interest based on a change in the apparent size of the part of interest.

11. The person behavior analysis device according to claim 1, wherein:

the person detection processor detects a plurality of persons who are present in the vicinity of the item placement area at the same time;

the part-of-interest detection processor detects the part of interest of each person detected by the person detection processor;

the position measurement processor measures a position of the part of interest of each person detected by the part-of-interest detection processor; and

the item pick-up action determination processor determines whether each person detected by the person detection processor performed the item pick-up action.

12. The person behavior analysis device according to claim 1, wherein:

a plurality of item placement areas are arranged for respective categories of items; and

the output information generation processor generates output information relating to a result of totaling of at least one of a number of occurrences of item pick-up actions for each item placement areas and staying times of detected persons in an area around each item placement area.

13. The person behavior analysis device according to claim 12, wherein the output information generation processor generates output information relating to a display image for displaying the result of totaling superimposed on an image representing the plurality of item placement areas arranged in accordance with an actual arrangement of the item placement areas.

17. The person behavior analysis device according to claim 1, wherein the part of interest is a head of the person.

14. A person behavior analysis system for performing analysis of item pick-up actions, which are actions of picking up an item placed in an item placement area, the system comprising:

an imaging device configured to capture an image of an area around the item placement area; and

a plurality of information processing devices,

wherein the plurality of information processing devices jointly comprise;

a person detection processor configured to detect one or more persons from image information obtained by capturing images by the imaging device;

a part-of-interest detection processor configured to detect, for each person detected by the person detection processor, a part of interest set only in a part of an upper body of the person excluding hands and arms;

a position measurement processor configured to measure a position of the part of interest detected by the part-of-interest detection processor;

an item pick-up action determination processor configured to obtain a displacement of the part of interest based on the position of the part of interest obtained by the position measurement processor and to determine whether each person detected by the person detection processor performed an item pick-up action based on the displacement of the part of interest of the person; and

an output information generation processor configured to generate output information representing a result of analysis based on a result of determination by the item pick-up action determination processor.

18. The person behavior analysis system for performing analysis according to claim 14, wherein the part of interest is a head of the person.

15. A person behavior analysis method for performing, with an information processing device, analysis of item pick-up actions, which are actions of picking up an item placed in an item placement area, the method comprising:

detecting one or more persons from image information obtained by capturing images covering an area around the item placement area;

detecting, for each person detected by the detecting of the one or more persons, a part of interest set only in a part of an upper body of the person excluding hands and arms;

measuring a position of the part of interest detected by the detecting of the part of interest;

obtaining a displacement of the part of interest from the position of the part of interest obtained by the measuring of the position of the part of interest and determining whether each person detected by the detecting of the one or more persons performed an item pick-up action based on the displacement of the part of interest of the person; and

generating output information representing a result of analysis based on a result of determination by the obtaining of the displacement of the part of interest.

19. The person behavior analysis method according to claim 15, wherein the detecting of the part of interest comprises detecting a head of the person.

16. A monitoring device for notifying a monitoring person of a person performing an item pick-up action which is an action of picking up an item placed in a placement area, comprising:

a person detection processor configured to detect one or more persons from image information obtained by capturing images covering an area around the item placement area;

a part-of-interest detection processor configured to detect, for each person detected by the person detection processor, a part of interest set only in a part of an upper body of the person excluding hands and arms;

a position measurement processor configured to measure a position of the part of interest detected by the part-of-interest detection processor;

an item pick-up action determination processor configured to obtain a displacement of the part of interest based on the position of the part of interest obtained by the position measurement processor and to determine whether each person detected by the person detection processor performed an item pick-up action based on the displacement of the part of interest of the person; and

a notification processor configured to generate notification information relating to a person performing the item pick-up action based on a result of determination by the item pick-up action determination processor.

20. The monitoring device according to claim 16, wherein the part of interest is a head of the person.