GESTURE AND VOICE RECOGNITION FOR CONTROL OF A DEVICE

• US 20130127712A1
• Filed: 03/20/2012
• Published: 05/23/2013
• Est. Priority Date: 11/18/2011
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Matsubayashi, Koji
• Granted Patent: US 8,847,881 B2
• Time in Patent Office: Days
• US Class Current: 345/158
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/47217 for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks specific graphical features in visual interfaces H04N21/4312
• CPC Class Codes: H04N 21/482 End-user interface for program selection broadcast systems using EPGs H04H60/72
• CPC Class Codes: H04N 21/485 End-user interface for client configuration

Abstract

A user interface allows one or more gestures to be devised by a user and mapped or associated with one or more commands or operations of a TV or other device. The user can select the command/operation that is to be associated with each gesture that he/she devised or created. The user is not limited to the use of pre-set gestures that were previously programmed into a system and is not limited to using pre-set commands/operations that were previously associated with pre-set gestures. In alternative embodiments, voice commands or other audible signals are devised by a user and are mapped or associated with commands/operations of a device.

Claims

1. A method for providing an interface, comprising:

receiving, by a computing device via a camera in communication with the computing device, a first gesture of a user;

entering into the computing device a selection for an operation of the computing device, wherein the operation is one of a plurality of different operations available to the user for the selection;

associating, by the computing device, the operation with the first gesture;

receiving, by the computing device via the camera, a second gesture, wherein the second gesture is substantially the same as the first gesture; and

performing the operation by the computing device in response to the receiving of the second gesture.

2. The method of claim 1 wherein the operation is a starting of an application by the computing device.

3. The method of claim 1 wherein the computing device is a television and wherein the camera is embedded in the television.

4. The method of claim 1 wherein the computing device has a display and wherein the operation is displaying a preselected one of a plurality of channels of content available to the user.

5. The method of claim 1 wherein the computing device is rendering content on a display, and wherein the operation is one of pausing the rendering of the content, fast-forwarding the rendering of the content, recording the content and modifying a volume level of audio of the content.

6. The method of claim 1, wherein the selection for the operation of the computing device is a first selection of a first operation, the method further comprising:

entering, by the user into the computing device, a second selection for a second operation of the computing device, wherein the second operation is a second one of the plurality of different operations available to the user for the second selection,

wherein the associating of the first operation with the first gesture includes associating both the first operation and the second operation with the first gesture, and

wherein the performing of the first operation includes automatically performing the first operation and the second operation by the computing device in response to the receiving of the second gesture.

7. The method of claim 6 wherein the computing device has a display, wherein the first operation is displaying a preselected first channel on the display, wherein the second operation is displaying a preselected second channel on the display after the preselected first channel has been displayed for a predetermined time period, and wherein each of the first channel and the second channel are of a plurality of channels of content available to the user.

8. The method of claim 7 further comprising entering, by the user into the computing device, the predetermined time period which is one of a plurality of time periods available for selection by the user.

9. The method of claim 6 wherein the computing device has a display, wherein the first operation is displaying an item of content on the display, and wherein the second operation is recording the item of content while it is being displayed.

10. A method for providing an interface, comprising:

receiving, by a computing device via a camera in communication with the computing device, a first plurality of gestures of a user;

entering, by the user into the computing device, a plurality of selections for a plurality of operations of the computing device;

associating, by the computing device, the plurality of operations with the first plurality of gestures, wherein each one of the plurality of operations is associated with a different one of the plurality of selections;

receiving, by the computing device via the camera, a second plurality of gestures, wherein each one of the second plurality of gesture is substantially the same as a corresponding one of the first plurality of gestures; and

performing each one of the plurality of operations by the computing device in response to the receiving of each corresponding one of the second plurality of gestures.

11. A method for providing an interface, comprising:

receiving, by a first computing device via a first camera in communication with the first computing device, a first gesture of a user;

entering, by the user into the first computing device, a selection for an operation of a second computing device, wherein the operation is one of a plurality of different operations available to the user for the selection;

associating, by the first computing device, the operation with the first gesture;

transmitting, from the first computing device to the second computing device, data corresponding to the first gesture and corresponding to the selection of the operation of the second computing device;

receiving, by the second computing device via a second camera in communication with the second computing device, a second gesture, wherein the second gesture is substantially the same as the first gesture; and

performing the operation by the second computing device in response to the receiving of the second gesture.

12. The method of claim 11 wherein the first computing device is one of a laptop computer, a desktop computer, a tablet computer, and a handheld device, and wherein the second computing device is a television,

13. A method for providing an interface, comprising:

receiving, by a computing device via a camera in communication with the computing device, a first gesture;

receiving, by the computing device via the camera, an image of a face of a first person;

receiving, by the computing device via the camera, an image of a face of a second person,

receiving, by the computing device via the camera, a second gesture that is substantially the same as the first gesture;

determining whether the second gesture was made by the first person;

performing an operation by the computing device in response to the receiving of the second gesture if the second gesture was determined to have been made by the first person; and

refraining from performing the operation by the computing device in response to the receiving of the second gesture if the second gesture was determined to have been made by other than the first person.

14. The method of claim 13 further comprising:

entering, by the first person into the computing device, a selection for the operation of the computing device, wherein the operation is one of a plurality of different operations available to the first person for the selection; and

associating, by the computing device, the operation with the first gesture.

15. A method for providing an interface, comprising:

receiving, by a computing device via a microphone in communication with the computing device, a first audio signal generated by a user;

entering, by the user into the computing device, a selection for an operation of the computing device, wherein the operation is one of a plurality of different operations available to the user for the selection;

associating, by the computing device, the operation with the first audio signal;

receiving, by the computing device via the microphone, a second audio signal, wherein the second audio signal is substantially the same as the first audio signal; and

performing the operation by the computing device in response to the receiving of the second audio signal.

16. The method of claim 15 wherein the first audio signal is one of a spoken word, a snapping of fingers, a clapping of hands, and a whistling.

17. The method of claim 15 wherein the operation is a starting of an application by the computing device.

18. The method of claim 15 wherein the computing device is a television and wherein the microphone is embedded in the television,

19. The method of claim 15 wherein the computing device has a display and wherein the operation is displaying a preselected one of a plurality of channels of content available to the user.

20. The method of claim 15 wherein the computing device is rendering content on a display, and wherein the operation is one of pausing the rendering of the content, rewinding the rendering of the content, fast-forwarding the rendering of the content, recording the content and modifying a volume level of audio of the content.

21. The method of claim 15, wherein the selection for the operation of the computing device is a first selection of a first operation, the method further comprising:

entering, by the user into the computing device, a second selection for a second operation of the computing device, wherein the second operation is a second one of the plurality of different operations available to the user for the second selection,

wherein the associating of the first operation with the first audio signal includes associating both the first operation and the second operation with the first audio signal, and

wherein the performing of the first operation includes automatically performing the first operation and the second operation by the computing device in response to the receiving of the second audio signal.

22. The method of claim 21 wherein the computing device has a display, wherein the first operation is displaying a preselected first channel on the display, wherein the second operation is displaying a preselected second channel on the display after the preselected first channel has been displayed for a predetermined time period, and wherein each of the first channel and the second channel are of a plurality of channels of content available to the user.

23. The method of claim 22 further comprising entering, by the user into the computing device, the predetermined time period which is one of a plurality of time periods available for selection by the user.

24. A method for providing an interface, comprising:

receiving, by a computing device via a microphone in communication with the computing device, a first voice command generated by a first person;

associating, by the computing device, the first voice command with the first person;

receiving, by the computing device via the microphone, a second voice command that is substantially the same as the first voice command;

determining whether the second voice command was generated by the first person;

performing an operation by the computing device in response to the receiving of the second voice command if it was determined to have been generated by the first person; and

refraining from performing the operation by the computing device in response to the receiving of the second voice command if it was determined to have been generated by another person who is different than the first person.

25. The method of claim 24 further comprising:

entering, by the first person into the computing device, a selection for the operation of the computing device, wherein the operation is one of a plurality of different operations available to the first person for the selection; and

associating, by the computing device, the operation with the first voice command.

26. A device for use by a user, comprising:

a memory;

a camera; and

a processor coupled to the memory and the camera and operable to perform steps comprising;

receiving via the camera a first gesture of the user;

receiving a selection by the user for an operation to be performed by the processor, wherein the operation is one of a plurality of different operations available for the selection by the user;

associating the operation with the first gesture;

receiving via the camera a second gesture, wherein the second gesture is substantially the same as the first gesture; and

performing the operation in response to the receiving of the second gesture.

27. The device of claim 26 wherein the operation is a starting of an application by the processor.

28. The device of claim 26 wherein the device has a display and wherein the operation is displaying a preselected one of a plurality of channels of content available to the user,

29. The device of claim 26 wherein the device has a display and is configured for rendering content on the display, and wherein the operation is one of pausing the rendering of the content, fast-forwarding the rendering of the content, recording the content and modifying a volume level of audio of the content.

30. The device of claim 26, wherein the selection for the operation to be performed by the processor is a first selection of a first operation, and wherein the processor is further operable to perform steps comprising:

receiving a second selection from the user for a second operation to be performed by the processor, wherein the second operation is a second one of the plurality of different operations available for the second selection by the user,

wherein the associating of the first operation with the first gesture includes associating both the first operation and the second operation with the first gesture, and

wherein the performing of the first operation includes automatically performing the first operation and the second operation in response to the receiving of the second gesture.

31. The device of claim 30 wherein the device has a display, wherein the first operation is the displaying of a preselected first channel on the display, wherein the second operation is the displaying of a preselected second channel on the display after the preselected first channel has been displayed for a predetermined time period, and wherein each of the first channel and the second channel are of a plurality of channels of content available to the user.

32. The device of claim 31 wherein the processor is further operable to perform steps comprising receiving from the user the predetermined time period which is one of a plurality of time periods available for selection by the user.

33. The device of claim 30 wherein the device has a display, wherein the first operation is displaying of an item of content on the display, and wherein the second operation is recording the item of content while it is being displayed.

34. A non-transitory, computer-readable storage medium for use by a user, wherein the storage medium contains instructions that, when executed by a processor coupled to a camera, cause the processor to perform a method comprising:

receiving via the camera a first gesture of the user;

receiving a selection by the user for an operation to be performed by the processor, wherein the operation is one of a plurality of different operations available for the selection by the user;

associating the operation with the first gesture;

receiving via the camera a second gesture, wherein the second gesture is substantially the same as the first gesture; and

performing the operation in response to the receiving of the second gesture.

35. The storage medium of claim 34 wherein the operation is a starting of an application.

36. The storage medium of claim 34 wherein the processor is further coupled to a display and wherein the operation is displaying a preselected one of a plurality of channels of content available to the user.

37. The storage medium of claim 34 wherein the processor is further coupled to a display and is configured for rendering content on the display, and wherein the operation is one of pausing the rendering of the content, fast-forwarding the rendering of the content, recording the content and modifying a volume level of audio of the content.