Information processing device and method, program and recording medium for identifying a gesture of a person from captured image data

• US 10,234,957 B2
• Filed: 03/16/2018
• Issued: 03/19/2019
• Est. Priority Date: 01/29/2009
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Yamaoka, Keisuke, Yokono, Jun
• Primary Examiner(s): Nguyen, Luong T
• Application Number:
  US15/923,975
• Publication Number:
  US 20180210556A1
• Time in Patent Office: 368 Days
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

An information processing device includes: an outline extraction unit extracting an outline of a subject from a picked-up image of the subject; a characteristic amount extraction unit extracting a characteristic amount, by extracting sample points from points making up the outline, for each of the sample points; an estimation unit estimating a posture of a high degree of matching as a posture of the subject by calculating a degree of the characteristic amount extracted in the characteristic amount extraction unit being matched with each of a plurality of characteristic amounts that are prepared in advance and represent predetermined postures different from each other; and a determination unit determining accuracy of estimation by the estimation unit using a matching cost when the estimation unit carries out the estimation.

Claims

1. An information processing apparatus, comprising:

circuitry configured to;

obtain an image of at least a part of a human subject, the image being captured by an imaging device;

estimate a position of a joint of the human subject based on the obtained image, the joint being a place at which two parts of the human subject are joined;

output, using a plurality of points indicative of the estimated position of the joint, joint position data indicative of the estimated position of the joint and estimation accuracy data indicative of an accuracy of the estimated position of the joint based on an estimation result of the position of the joint; and

estimate a posture of the human subject based on the output joint position data and the output estimation accuracy data.

2. The information processing apparatus of claim 1, wherein

the estimation accuracy data is generated by a discriminator learned by a machine learning.

3. The information processing apparatus of claim 2, wherein

the discriminator is learned by Boosting.

4. The information processing apparatus of claim 1, wherein

the joint position data includes position information of a coordinate of the image data, and

the position information of the coordinate indicates two-dimensional coordinate of each joint of the human subject.

5. The information processing apparatus of claim 1, wherein

the estimation accuracy data includes a numerical value indicative of a likelihood of the estimation result.

6. The information processing apparatus of claim 1, wherein the circuitry is further configured to:

output a plurality of the joint position data for a plurality of the estimated positions of a plurality of the joints; and

estimate the posture of the human subject based on the output plurality of joint position data and the output estimation accuracy data.

7. The information processing apparatus of claim 1, wherein the circuitry is further configured to:

output the image of the at least a part of the human subject; and

overlap the plurality of points indicative of the estimated position of the joint on the image and a straight line between the plurality of points on the image.

8. The information processing apparatus of claim 1, wherein the circuitry is further configured to:

generate an instruction to control a gaming application in accordance with the estimated posture.

9. The information processing apparatus of claim 1, wherein the circuitry is further configured to:

determine a gesture of the at least the part of the human subject based on the estimated posture.

10. The information processing apparatus of claim 1, wherein the circuitry is further configured to:

determine whether the estimated posture corresponds to one of predetermined postures.

11. The information processing apparatus of claim 1, wherein the circuitry is configured to:

output, using the plurality of points and a plurality of lines indicative of the estimated position of the joint, the joint position data.

12. A method, comprising:

obtaining, using an imaging device, an image of at least a part of a human subject;

estimating, using circuitry, a position of a joint of the human subject based on the obtained image, the joint being a place at which two parts of the human subject are joined;

outputting, using a plurality of points indicative of the estimated position of the joint, joint position data indicative of the estimated position of the joint and estimation accuracy data indicative of an accuracy of the estimated position of the joint based on an estimation result of the position of the joint; and

estimating a posture of the human subject based on the output joint position data and the output estimation accuracy data.

13. The method of claim 12, wherein

the estimation accuracy data is generated by a discriminator learned by a machine learning.

14. The method of claim 13, wherein

the discriminator is learned by Boosting.

15. The method of claim 12, wherein

the joint position data includes position information of a coordinate of the image data, and

the position information of the coordinate indicates two-dimensional coordinate of each joint of the human subject.

16. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform a method, comprising:

obtaining, using an imaging device, an image of at least a part of a human subject;

estimating a position of a joint of the human subject based on the obtained image, the joint being a place at which two parts of the human subject are joined;

outputting, using a plurality of points indicative of the estimated position of the joint, joint position data indicative of the estimated position of the joint and estimation accuracy data indicative of an accuracy of the estimated position of the joint based on an estimation result of the position of the joint; and

estimating a posture of the human subject based on the output joint position data and the output estimation accuracy data.

17. The non-transitory computer-readable medium of claim 16, wherein

the estimation accuracy data is generated by a discriminator learned by a machine learning.

18. The non-transitory computer-readable medium of claim 17, wherein

the discriminator is learned by Boosting.

19. The non-transitory computer-readable medium of claim 16, wherein

the joint position data includes position information of a coordinate of the image data, and

the position information of the coordinate indicates two-dimensional coordinate of each joint of the human subject.