Information processing apparatus, method, and program for gesture recognition and control

• US 9,229,533 B2
• Filed: 01/27/2011
• Issued: 01/05/2016
• Est. Priority Date: 03/08/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Shigeta, Osamu, Noda, Takuro
• Primary Examiner(s): To, Jennifer
• Assistant Examiner(s): SUN, LI P
• Application Number:
  US13/015,120
• Publication Number:
  US 20110216075A1
• Time in Patent Office: 1,804 Days
• Field of Search: 715/863, 345/156, 382/168
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]

Abstract

An information processing apparatus includes a detection unit configured to detect a gesture made by a user, a recognition unit configured to recognize a type of the gesture detected by the detection unit, a control unit configured to control operation of a first application and a second application, and an output unit configured to output information of the first application or the second application. If the gesture is recognized by the recognition unit while the control unit is controlling the operation of the first application in the foreground, the control unit controls the operation of the second application operating in the background of the first application on the basis of the type of the gesture recognized by the recognition unit.

Claims

1. An information processing apparatus comprising:

a personal computer connected to a camera separate from the personal computer, the personal computer having;

processing circuitry configured to;

detect a gesture made by a user based on an image of a predetermined portion of the user obtained by the camera;

output for a predetermined period of time, while in a normal mode, a moving image representative of the user'"'"'s actual movement regarding the gesture made by the user;

recognize, in response to output of the moving image representative of the user'"'"'s actual movement regarding the gesture made by the user, a gesture command from the detected gesture;

determine whether the recognized gesture command has associated therewith a previously stored gesture command among a plurality of previously stored gesture commands by comparing the recognized gesture command to each of the plurality of previously stored gesture commands; and

control operation of a first application and a second application,

wherein the circuitry is further configured to output information associated with one or more of the first application and the second application,

when the recognized gesture command is determined, by the circuitry based on the comparison of the recognized gesture command to each of the plurality of previously stored gesture commands, to have associated therewith a previously stored gesture command while the circuitry controls operation of the first application in a foreground, the circuitry controls operation of the second application operating in a background based on a control operation corresponding to the previously stored gesture command, except when the previously stored gesture command is a gesture command to switch modes, in which case the circuitry switches to a feedback mode if currently in the normal mode and switches to the normal mode if currently in the feedback mode,

when the recognized gesture command is determined, by the circuitry based on the comparison of the recognized gesture command to each of the plurality of previously stored gesture commands, not to have associated therewith a previously stored gesture command while the circuitry controls operation of the first application in the foreground, operation information is fed back to the user,

when the recognized gesture command is determined, by the circuitry based on the comparison of the recognized gesture command to each of the plurality of previously stored gesture commands, to be the gesture command to switch modes and the current mode is the normal mode, the circuitry switches to the feedback mode and the operation information is fed back to the user,

the operation information fed back to the user is fed back simultaneously while the circuitry is outputting information associated with the one or more of the first application and the second application and includes;

a list of possible gestures associated with the previously stored gesture commands based on the gesture or another gesture made by the user,

a moving image representative of the user'"'"'s actual movement regarding the gesture or another gesture made by the user, and

a predetermined moving image for learning a gesture command corresponding to one of the plurality of previously stored gesture commands, and

the gesture command to switch modes is the same gesture command to switch from the feedback mode to the normal mode and to switch from the normal mode to the feedback mode.

2. The information processing apparatus according to claim 1, wherein the processing circuitry is further configured to:

infer the previously stored gesture command among the plurality of previously stored gesture commands by analogy when it is determined by the circuitry that the recognized gesture command does not have associated therewith the previously stored gesture command.

3. The information processing apparatus according to claim 1, wherein the moving image representative of the user'"'"'s actual movement regarding the gesture made by the user is output by the circuitry on a screen as a mirror image of the user.

4. The information processing apparatus according to claim 1, wherein, in the normal mode, when operation of the second application is controlled based on the recognized gesture command, the circuitry is further configured to output information corresponding to a control result based on the recognized gesture command for the predetermined period of time.

5. The information processing apparatus according to claim 4, wherein the circuitry is further configured to cause output of audio information as the information corresponding to the control result based on the recognized gesture command.

6. The information processing apparatus according to claim 1, wherein switching to the feedback mode includes the circuitry causing display of an image of the feedback mode semi-transparently superimposed on an image of the first application.

7. The information processing apparatus according to claim 1, wherein switching to the feedback mode includes the circuitry causing display of an image of the feedback mode in a region that is part of an image of the first application.

8. The information processing apparatus according to claim 1, wherein the detection of the gesture by the user includes detection of a predetermined portion of a hand of the user and calculation of a center of gravity of the portion of the hand.

9. The information processing apparatus according to claim 1, wherein the detected gesture includes movement of the user'"'"'s hand.

10. An information processing method implemented using a personal computer connected to a camera separate from the personal computer, the information processing method comprising:

detecting, using processing circuitry of the personal computer, a gesture made by a user based on an image of a predetermined portion of the user obtained by the camera;

outputting for a predetermined period of time, using the processing circuitry, while in a normal mode, a moving image representative of the user'"'"'s actual movement regarding the gesture made by the user;

recognizing, using the processing circuitry, in response to output of the moving image representative of the user'"'"'s actual movement regarding the gesture made by the user, a gesture command from the detected gesture;

determining, using the processing circuitry, whether or not the recognized gesture command has associated therewith a previously stored gesture command among a plurality of previously stored gesture commands by comparing the recognized gesture command to each of the plurality of previously stored gesture commands;

controlling, using the processing circuitry, operation of a first application and a second application; and

outputting information, using the processing circuitry, associated with one or more of the first application and the second application,

wherein, when the recognized gesture command is determined, based on the comparing the recognized gesture command to each of the plurality of previously stored gesture commands, to have associated therewith a previously stored gesture command while operation of the first application is controlled in a foreground, controlling operation of the second application operating in a background based on a control operation corresponding to the previously stored gesture command, except when the previously stored gesture command is a gesture command to switch modes, in which case a current mode is switched to a feedback mode if the current mode is the normal mode and is switched to the normal mode if the current mode is the feedback mode,

when the recognized gesture command is determined, based on the comparing the recognized gesture command to each of the plurality of previously stored gesture commands, not to have associated therewith a previously stored gesture command while operation of the first application is controlled in the foreground, feeding back operation information to the user,

when the recognized gesture command is determined, based on the comparison of the recognized gesture command to each of the plurality of previously stored gesture commands, to be the gesture command to switch modes and the current mode is the normal mode, the current mode is switched to the feedback mode and the operation information is fed back to the user,

the operation information fed back to the user is fed back simultaneously while the circuitry is outputting information associated with the one or more of the first application and the second application and includes;

a list of possible gestures associated with the previously stored gesture commands based on the gesture or another gesture made by the user,

a moving image representative of the user'"'"'s actual movement regarding the gesture or another gesture made by the user, and

a predetermined moving image for learning a gesture command corresponding to one of the plurality of previously stored gesture commands, and

the gesture command to switch modes is the same gesture command to switch from the feedback mode to the normal mode and to switch from the normal mode to the feedback mode.

11. The information processing method according to claim 10, wherein said detecting the gesture made by the user includes detection of a predetermined portion of a hand of the user and calculation of a center of gravity of the portion of the hand.

12. The information processing method according to claim 10, wherein the detected gesture includes movement of the user'"'"'s hand.

13. A non-transitory computer-readable storage medium storing computer-readable instructions thereon that when executed by a personal computer connected to a camera separate from the personal computer cause processing circuitry of the personal computer to perform a method comprising:

detecting a gesture made by a user based on an image of a predetermined portion of the user obtained by the camera;

outputting for a predetermined period of time, while in a normal mode, a moving image representative of the user'"'"'s actual movement regarding the gesture made by the user;

recognizing, in response to output of the moving image representative of the user'"'"'s actual movement regarding the gesture made by the user, a gesture command from the detected gesture;

determining whether or not the recognized gesture command has associated therewith a previously stored gesture command among a plurality of previously stored gesture commands by comparing the recognized gesture command to each of the plurality of previously stored gesture commands;

controlling operation of a first application and a second application; and

outputting information associated with one or more of the first application and the second application,

wherein, when the recognized gesture command is determined, based on the comparing the recognized gesture command to each of the plurality of previously stored gesture commands, to have associated therewith a previously stored gesture command while operation of the first application is controlled in a foreground, controlling operation of the second application operating in a background based on a control operation corresponding to the previously stored gesture command, except when the previously stored gesture command is a gesture command to switch modes, in which case a current mode is switched to a feedback mode if the current mode is the normal mode and is switched to the normal mode if the current mode is the feedback mode,

when the recognized gesture command is determined, based on the comparing the recognized gesture command to each of the plurality of previously stored gesture commands, not to have associated therewith a previously stored gesture command while operation of the first application is controlled in the foreground, feeding back operation information to the user,

when the recognized gesture command is determined, based on the comparison of the recognized gesture command to each of the plurality of previously stored gesture commands, to be the gesture command to switch modes and the current mode is the normal mode, the current mode is switched to the feedback mode and the operation information is fed back to the user,

the operation information fed back to the user is fed back simultaneously while the circuitry is outputting information associated with the one or more of the first application and the second application and includes;

a list of possible gestures associated with the previously stored gesture commands based on the gesture or another gesture made by the user,

a moving image representative of the user'"'"'s actual movement regarding the gesture or another gesture made by the user, and

a predetermined moving image for learning a gesture command corresponding to one of the plurality of previously stored gesture commands, and

the gesture command to switch modes is the same gesture command to switch from the feedback mode to the normal mode and to switch from the normal mode to the feedback mode.

14. The non-transitory computer-readable medium according to claim 13, wherein said detecting the gesture made by the user includes detection of a predetermined portion of a hand of the user and calculation of a center of gravity of the portion of the hand.

15. The non-transitory computer-readable medium according to claim 13, wherein the detected gesture includes movement of the user'"'"'s hand.

16. An information processing system comprising:

a personal computer connected to a camera separate from the personal computer, the personal computer having;

processing circuitry configured to;

detect a gesture made by a user based on an image of a predetermined portion of the user obtained by the camera;

output for a predetermined period of time, while in a normal mode, a moving image representative of the user'"'"'s actual movement regarding the gesture made by the user;

recognize, in response to output of the moving image representative of the user'"'"'s actual movement regarding the gesture made by the user, a gesture command from the detected gesture;

determine whether or not the recognized gesture command has associated therewith a previously stored gesture command among a plurality of previously stored gesture commands by comparing the recognized gesture command to each of the plurality of previously stored gesture commands;

store the plurality of previously stored gesture commands; and

control operation of a first application and a second application,

wherein the circuitry is further configured to output information associated with one or more of the first application and the second application,

when the recognized gesture command is determined, by the circuitry based on the comparison of the recognized gesture command to each of the plurality of previously stored gesture commands, to have associated therewith a previously stored gesture command while the circuitry controls operation of the first application in a foreground, the circuitry controls operation of the second application operating in a background based on a control operation corresponding to the previously stored gesture command, except when the previously stored gesture command is a gesture command to switch modes, in which case the circuitry switches to a feedback mode if currently in the normal mode and switches to the normal mode if currently in the feedback mode,

when the recognized gesture command is determined, by the circuitry based on the comparison of the recognized gesture command to each of the plurality of previously stored gesture commands, not to have associated therewith a previously stored gesture command while the circuitry controls operation of the first application in the foreground, operation information is fed back to the user,

when the recognized gesture command is determined, by the circuitry based on the comparison of the recognized gesture command to each of the plurality of previously stored gesture commands, to be the gesture command to switch modes and the current mode is the normal mode, the current mode is switched to the feedback mode and the operation information is fed back to the user,

the operation information fed back to the user is fed back simultaneously while the circuitry is outputting information associated with the one or more of the first application and the second application and includes;

a list of possible gestures associated with the previously stored gesture commands based on the gesture or another gesture made by the user,

a moving image representative of the user'"'"'s actual movement regarding the gesture or another gesture made by the user, and

a predetermined moving image for learning a gesture command corresponding to one of the plurality of previously stored gesture commands, and

the gesture command to switch modes is the same gesture command to switch from the feedback mode to the normal mode and to switch from the normal mode to the feedback mode.

17. The information processing system according to claim 16, wherein the detection of the gesture by the user includes detection of a predetermined portion of a hand of the user and calculation of a center of gravity of the portion of the hand.