Information processing device and information processing method

• US 8,879,787 B2
• Filed: 10/23/2009
• Issued: 11/04/2014
• Est. Priority Date: 02/19/2009
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.), Sony Corporation (Sony Group Corp.)
• Inventors: Ikenoue, Shoichi
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): YANG, WEI WEN
• Application Number:
  US13/147,979
• Publication Number:
  US 20120039507A1
• Time in Patent Office: 1,838 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user

Abstract

An image acquisition unit of an information processing device acquires data for moving image including an image of a user and captured by an image capturing device. An initial processing unit determines correspondence between an amount of movement of the user and a parameter defining an image to be ultimately output in a conversion information storage unit. A tracking processing unit uses a particle filter to perform visual tracking in the moving image so as to estimate the magnification and translation amount of the user'"'"'s head contour. The input value conversion unit converts the amount of movement of the user into the parameter defining an image using the magnification and the translation amount as parameters. The output data generation unit generates an image based on the parameter. The output control unit controls the generated image so as to be displayed on a display device.

Claims

1. An information processing device comprising:

an image capturing device configured to capture a movement of an object within a moving image, wherein the object is a person;

an input value acquisition unit configured to estimate an amount of movement of the object with respect to the image capturing device from a reference position in a three-dimensional space defined in the vertical, horizontal, and depth directions, in the moving image by referring to the head and shoulder contours of the person which is approximated by a Greek ohm-shaped curve, wherein the image of the person derived from the ohm-shaped curve is subjected to an edge extracting process, wherein one or more edges near the curve are detected so as to obtain parameters that substantially match the contours of the person and an edge image is thereby obtained, wherein the ohm-shaped curve may be translated, expanded, contracted, or rotated by changing the parameters defining the curve, wherein said edge extracting process is repeated for each frame, so that tracking of the person proceeds, wherein movement of the object is determined based, at least in part, on its horizontal and vertical coordinates relative to a reference position in a two-dimensional plane, as viewed from above the image capturing device, the vertical direction representing depth from the perspective of the image capturing device, wherein the vertical distance from the reference position is determined by the object'"'"'s detected change in size relative to its size at the reference position; and

an output data generation unit configured to make a change in a performing process according to the amount of movement in the three-dimensional space that is estimated by the input value acquisition unit and to generate output data accordingly.

2. The information processing device according to claim 1, wherein the input value acquisition unit comprises:

a tracking processing unit configured to represent a typical shape of the object using a curve, to compute a matching score between the curve and an edge image of the image frame forming the moving image captured by the image capturing device while changing at least the magnification and the amount of translation of the curve, and to acquire, for each image frame, data on the magnification and the amount of translation that defines a curve that best matches the object in the image frame; and

an input value conversion unit configured to estimate the amount of movement in the vertical and horizontal directions of the three-dimensional space using the data on the amount of translation acquired by the tracking processing unit, and to estimate an amount of movement in the depth direction using the data on the magnification.

3. The information processing device according to claim 1, wherein the input value acquisition unit comprises:

a tracking processing unit configured to represent a typical shape of the object using a curve, to compute a matching score between the curve and an edge image of the image frame forming the moving image captured by the image capturing device while changing at least the magnification and the amount of translation of the curve, and to acquire, for each image frame, a curve that best matches the object in the image frame; and

an input value conversion unit configured to estimate an amount of movement in the depth direction using the number of pixels in an area surrounded by the curve acquired by the tracking processing unit.

4. The information processing device according to claim 2, wherein the tracking processing unit represents a shape of head contour of a human being as the object using an Greek ohm-shaped curve, and the input value conversion unit estimates the amount of movement of a head of a user in the three-dimensional space.

5. The information processing device according to claim 1,

wherein the output data generation unit performs a process using a variable uniquely determined by the amount of movement,

the information processing device further comprising an initial processing unit configured to identify a range of movement of the object in the respective horizontal, vertical, and depth directions prior to the processing in the output data generation unit so as to define correspondence between the range of movement and the input value of output data generation unit.

6. The information processing device according to claim 1,

wherein the output data generation unit performs a process using a variable uniquely determined by the amount of movement, based on predefined correspondence,

the information processing device further comprising an initial processing unit configured to display on the display device a mirror image of a user captured by the image capturing device prior to the processing by the output data generation unit, display a curve representing a typical shape of the object at a position on the mirror image corresponding to end points of a permitted range of movement of the object in the respective directions and defined by the correspondence, and let the user recognize the permitted range of movement of the object by moving the object on a displayed image so that the curve matches an actual contour line of the object.

7. The information processing device according to claim 1, wherein the input value acquisition unit estimates an actual amount of movement in the vertical and horizontal directions by correcting an apparent amount of movement in the vertical and horizontal directions determined by using the image capturing device, based on an amount of movement of the object in the depth direction.

8. The information processing device according to claim 1, wherein the output data generation unit does not reflect a variation in the amount of movement in the process performed when the amount of movement from the reference position in the three-dimensional space does not exceed a predefined threshold value.

9. The information processing device according to claim 2, wherein the input value conversion unit subjects the data on an amount of translation and the data on magnification acquired in each image frame to noise-removal filtering, using a temporal change in the data for preceding image frames, thereby correcting the data accordingly.

10. The information processing device according to claim 4,

wherein the output data generation unit generates data for an image representing a field of view of a user in a virtual world, change a viewpoint in the field of view in accordance with an amount of head movement of the user in the three-dimensional space,

and the information processing device further comprises a display device configured to display the image generated by the output data generation unit so that the user can see the image as the user moves in the three-dimensional space.

11. The information processing device according to claim 4,

wherein the output data generation unit generates data for an image representing an object indicating a user in a virtual world, move the object in accordance with an amount of head movement of the user in the three-dimensional space,

and the information processing device further comprises a display device configured to display the image generated by the output data generation unit so that the user can see the image as the user moves in the three-dimensional space.

12. An information processing device comprising:

an image capturing device configured to capture a movement of an object within a moving image, wherein said object is a person;

an input value acquisition unit configured to estimate an amount of movement of the object with respect to the image capturing device from a reference position in a two-dimensional space defined in the vertical and horizontal directions, and further estimate a distance in the depth direction from the reference position, from the moving image by referring to the head and shoulder contours of the person which is approximated by a Greek ohm-shaped curve, wherein the image of the person derived from the ohm-shaped curve is subjected to an edge extracting process, wherein one or more edges near the curve are detected so as to obtain parameters that substantially match the contours of the person and an edge image is thereby obtained, wherein the ohm-shaped curve may be translated, expanded, contracted, or rotated by changing the parameters defining the curve, wherein said edge extracting process is repeated for each frame, so that tracking of the person proceeds, wherein movement of the object is determined based, at least in part, on its horizontal and vertical coordinates relative to a reference position in a two-dimensional plane, as viewed from above the image capturing device, the vertical direction representing depth from the perspective of the image capturing device, wherein the vertical distance from the reference position is determined by the object'"'"'s detected change in size relative to its size at the reference position; and

an output data generation unit configured to make a change in a performing process according to the amount of movement in the two-dimensional space estimated by the input value acquisition unit and generate output data accordingly,

wherein the input value acquisition unit estimates an actual amount of movement in the vertical and horizontal directions by correcting an apparent amount of movement in the vertical and horizontal directions determined by using the image capturing device, based on the distance of the object from the reference position in the depth direction.

13. An information processing method comprising:

capturing a movement of an object within a moving image using an image capturing device, wherein said object is a person;

estimating an amount of movement of the object with respect to the image capturing device from a reference position in a three-dimensional space defined in the vertical, horizontal, and depth directions, in the moving image by referring to the head and shoulder contours of the person which is approximated by a Greek ohm-shaped curve, wherein the image of the person derived from the ohm-shaped curve is subjected to an edge extracting process, wherein one or more edges near the curve are detected so as to obtain parameters that substantially match the contours of the person and an edge image is thereby obtained, wherein the ohm-shaped curve may be translated, expanded, contracted, or rotated by changing the parameters defining the curve, wherein said edge extracting process is repeated for each frame, so that tracking of the person proceeds, wherein movement of the object is determined based, at least in part, on its horizontal and vertical coordinates relative to a reference position in a two-dimensional plane, as viewed from above the image capturing device, the vertical direction representing depth from the perspective of the image capturing device, wherein the vertical distance from the reference position is determined by the object'"'"'s detected change in size relative to its size at the reference position; and

making a change in a performing process according to the estimated amount of movement in the three-dimensional space and generating output data accordingly.

14. The information processing method according to claim 13,

wherein the generation of output data comprises generating data for a moving image that varies in accordance with the amount of movement of the object in the three-dimensional space,

the method further comprising displaying the moving image.

15. A computer program embedded in a non-transitory computer-readable recording medium, comprising:

a module configured to capture a movement of an object within a moving image using an image capturing device, wherein said object is a person;

a module configured to estimate an amount of movement of the object with respect to the image capturing device from a reference position in a three-dimensional space defined in the vertical, horizontal, and depth directions, in the moving image by referring to the head and shoulder contours of the person which is approximated by a Greek ohm-shaped curve, wherein the image of the person derived from the ohm-shaped curve is subjected to an edge extracting process, wherein one or more edges near the curve are detected so as to obtain parameters that substantially match the contours of the person and an edge image is thereby obtained, wherein the ohm-shaped curve may be translated, expanded, contracted, or rotated by changing the parameters defining the curve, wherein said edge extracting process is repeated for each frame, so that tracking of the person proceeds, wherein movement of the object is determined based, at least in part, on its horizontal and vertical coordinates relative to a reference position in a two-dimensional plane, as viewed from above the image capturing device, the vertical direction representing depth from the perspective of the image capturing device, wherein the vertical distance from the reference position is determined by the object'"'"'s detected change in size relative to its size at the reference position; and

a module configured to make a change in a performing process according to the estimated amount of movement in the three-dimensional space and to generate output data accordingly.

16. A non-transitory computer-readable recording medium having embodied thereon a computer program comprising:

a module configured to capture a movement of an object within a moving image using an image capturing device, wherein said object is a person;

a module configured to estimate an amount of movement of the object with respect to the image capturing device from a reference position in a three-dimensional space defined in the vertical, horizontal, and depth directions, in the moving image by referring to the head and shoulder contours of the person which is approximated by a Greek ohm-shaped curve, wherein the image of the person derived from the ohm-shaped curve is subjected to an edge extracting process, wherein one or more edges near the curve are detected so as to obtain parameters that substantially match the contours of the person and an edge image is thereby obtained, wherein the ohm-shaped curve may be translated, expanded, contracted, or rotated by changing the parameters defining the curve, wherein said edge extracting process is repeated for each frame, so that tracking of the person proceeds, wherein movement of the object is determined based, at least in part, on its horizontal and vertical coordinates relative to a reference position in a two-dimensional plane, as viewed from above the image capturing device, the vertical direction representing depth from the perspective of the image capturing device, wherein the vertical distance from the reference position is determined by the object'"'"'s detected change in size relative to its size at the reference position; and

a module configured to make a change in a performing process according to the estimated amount of movement in the three-dimensional space and to generate output data accordingly.