Method and apparatus for recognizing actions

• US 9,298,989 B2
• Filed: 11/24/2014
• Issued: 03/29/2016
• Est. Priority Date: 11/29/2013
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Ito, Fumito, Kuwabara, Shohei, Suwa, Sayaka
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US14/551,162
• Publication Number:
  US 20150154449A1
• Time in Patent Office: 491 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

An information processing apparatus obtains a picture containing an image of a person from an imaging device. Also obtained are first data indicating distance of the person and other subjects seen in the picture from the imaging device, second data indicating distance of a wrist of the person from the imaging device, and third data indicating a coordinate position of the wrist in the picture. The information processing apparatus extracts a set of pixels whose coordinates fall within a specified coordinate range defined around the coordinate position of the wrist and whose distances from the imaging device fall within a specified distance range defined around the distance of the wrist. Then based on a ratio representing how many of the extracted pixels have a distinctive color tone, the information processing apparatus determines whether the person has picked up an object.

Claims

1. A method for recognizing actions, comprising:

obtaining, by an information processing apparatus, a picture containing an image of a person from an imaging device, as well as first data indicating distance of the person and other subjects seen in the picture from the imaging device on an individual pixel basis, second data indicating distance of a wrist of the person from the imaging device, and third data indicating a coordinate position of the wrist in the picture;

extracting, by the information processing apparatus, a set of pixels from the obtained picture whose respective coordinates fall within a specified coordinate range defined around the coordinate position of the wrist and whose respective distances fall within a specified distance range defined around the distance of the wrist; and

determining whether the person has picked up an object, based on a ratio representing how many of the extracted pixels have a distinctive color tone.

2. The method according to claim 1, wherein the extracting pixels and the determining whether the person has picked up an object are performed upon detection of movement of the wrist out of a specified area in a view space.

3. The method according to claim 2, wherein the method further comprises detecting presence of the wrist in the specified area in the view space, by testing whether the coordinate position of the wrist falls within an area previously defined as part of pictures to be taken by the imaging device.

4. The method according to claim 1, wherein:

the obtaining a picture, first data, second data, and third data and the extracting pixels are performed at first and second occasions, the first occasion being when the wrist has moved into a specified area in a view space, the second occasion being when the wrist has moved out of the specified area;

the extracting includes extracting first pixels from a first picture obtained on the first occasion, and extracting second pixels from a second picture obtained on the second occasion;

the method further comprises comparing a first ratio representing how many of the first pixels have the distinctive color tone with a second ratio representing how many of the second pixels have the distinctive color tone; and

the determining whether the person has picked up an object is based on a result of the comparing the first ratio with the second ratio.

5. The method according to claim 4, wherein the method further comprises calculating the distinctive color tone from color code values of the first pixels extracted on the first occasion.

6. The method according to claim 1, wherein the method further comprises calculating the specified coordinate range from relative positions between the coordinate position of the wrist and another body part located between a shoulder and finger tips of the person.

7. The method according to claim 1, wherein the distinctive color tone is a range of color code values representing skin colors.

8. An action recognition apparatus comprising:

a processor configured to perform a process including;

obtaining a picture containing an image of a person from an imaging device, as well as first data indicating distance of the person and other subjects seen in the picture from the imaging device on an individual pixel basis, second data indicating distance of a wrist of the person from the imaging device, and third data indicating a coordinate position of the wrist in the picture;

extracting a set of pixels from the obtained picture whose respective coordinates fall within a specified coordinate range defined around the coordinate position of the wrist and whose respective distances fall within a specified distance range defined around the distance of the wrist; and

determining whether the person has picked up an object, based on a ratio representing how many of the extracted pixels have a distinctive color tone.

9. The action recognition apparatus according to claim 8, wherein:

the extracting pixels is performed upon detection of movement of the wrist out of a specified area in a view space; and

the determining whether the person has picked up an object is performed based on the extracted pixels.

10. The action recognition apparatus according to claim 9, wherein the process further includes detecting presence of the wrist in the specified area in the view space, by testing whether the coordinate position of the wrist falls within an area previously defined as part of pictures to be taken by the imaging device.

11. The action recognition apparatus according to claim 8, wherein:

the obtaining a picture, first data, second data, and third data and the extracting pixels are performed at first and second occasions, the first occasion being when the wrist has moved into a specified area in a view space, the second occasion being when the wrist has moved out of the specified area;

the extracting includes extracting first pixels from a first picture obtained on the first occasion, and extracting second pixels from a second picture obtained on the second occasion;

the process further includes comparing a first ratio representing how many of the first pixels have the distinctive color tone with a second ratio representing how many of the second pixels have the distinctive color tone; and

the determining whether the person has picked up an object is based on a result of the comparing the first ratio with the second ratio.

12. The action recognition apparatus according to claim 11, wherein the process further includes calculating the distinctive color tone from color code values of the first pixels extracted on the first occasion.

13. The action recognition apparatus according to claim 8, wherein the process further includes calculating the specified coordinate range from relative positions between the coordinate position of the wrist and another body part located between a shoulder and finger tips of the person.

14. The action recognition apparatus according to claim 8, wherein the distinctive color tone is a range of color code values representing skin colors.

15. A non-transitory computer-readable storage medium storing a program for recognizing actions, wherein the program causes a computer to perform a process comprising:

obtaining a picture containing an image of a person from an imaging device, as well as first data indicating distance of the person and other subjects seen in the picture from the imaging device on an individual pixel basis, second data indicating distance of a wrist of the person from the imaging device, and third data indicating a coordinate position of the wrist in the picture;

extracting a set of pixels from the obtained picture whose respective coordinates fall within a specified coordinate range defined around the coordinate position of the wrist and whose respective distances fall within a specified distance range defined around the distance of the wrist; and

determining whether the person has picked up an object, based on a ratio representing how many of the extracted pixels have a distinctive color tone.