Voice and video control of interactive electronically simulated environment

• US 20070139443A1
• Filed: 12/12/2005
• Published: 06/21/2007
• Est. Priority Date: 12/12/2005
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Marks, Richard, Chen, Ruxin
• Granted Patent: US 8,549,442 B2
• Time in Patent Office: Days
• US Class Current: 345/629
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4331 Caching operations, e.g. of an advertisement for later insertion during playback
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4438 Window management, e.g. event handling following interaction with the user interface
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 21/482 End-user interface for program selection broadcast systems using EPGs H04H60/72

Abstract

A method of moving objects in a graphical user interface, includes obtaining a video image of a user of the interface; displaying the video image on a display such that the video image is superposed with one or more objects displayed on the display; and moving one or more objects displayed on the display based on recognition of motions of the video image of the user. Recognition of motions of the video image may include recognition of motions of an image of the user'"'"'s hand.

Claims

1. A method of moving objects in a graphical user interface, comprising:

obtaining a video image of a user of the interface;

displaying the video image on a display such that the video image is superposed with one or more objects displayed on the display;

moving one or more objects displayed on the display based on recognition of motions of the video image of the user.

2. The method of claim 1 wherein recognition of motions of the video image includes recognition of motions of an image of the user'"'"'s hand.

3. The method of claim 2 wherein recognition of motions of an image of the user'"'"'s hand include recognition of a change in aspect ratio of the image of the user'"'"'s hand.

4. The method of claim 1 wherein recognition of motions of the video image includes recognizing whether a user is using his left or right hand to select or move one or more icons from the contents icon array.

5. The method of claim 4 wherein recognizing whether a user is using his left or right hand includes determining a relative location of an image of the user'"'"'s hand relative to an image of the user'"'"'s head.

6. The method of claim 5 wherein determining a relative location of the user'"'"'s hand relative to an image of the user'"'"'s head includes determining whether a user'"'"'s arm crosses a user'"'"'s chest.

7. The method of claim 1 wherein recognition of motions of the video image includes recognizing a change in aspect ratio of a user'"'"'s hand or an object held within the user'"'"'s hand.

8. The method of claim 1 wherein the one or more objects includes a plurality of contents icons.

9. The method of claim 8, further comprising receiving a voice command to select one or more icons from the plurality of contents icons.

10. The method of claim 9, wherein receiving a voice command to select one or more icons from the contents icon array includes displaying a list of selected content icons based on a keyword in the voice command.

11. The method of claim 10 wherein receiving a voice command includes the use of a device having one or more microphones attached to a video image capture device.

12. The method of claim 8 wherein moving one or more objects includes receiving a command to scroll and/or select one or more contents icons of the plurality of contents icons displayed on display;

and

retrieving an appropriate number of additional contents icons from a contents icon database to provide a scrolled display in the graphical user interface,

wherein the contents icon database is divided into a plurality of entries, each entry of the plurality of entries including at least data for a contents icon of the plurality of contents icons and a link to a contents file, each entry of the database is configured to be fixed in size so that the data for the contents icon can be quickly accessed, and the contents icon can be scrolled in the graphical user interface at a relatively high speed.

13. The method of claim 12, wherein the relatively high speed indicates a scroll speed that is fast enough to scroll and visually smoothly display the contents icons in the graphical user interface.

14. The method of claim 12, wherein configuring each entry to be fixed in size includes configuring every entry of the plurality of entries to be a particular size.

15. The method of claim 14, wherein

the database has a cache for storing data from the database.

16. The method of claim 15, further comprising

retrieving a first plurality of entries from the database and storing the entries in the cache.

17. The method of claim 16, further comprising

retrieving a second plurality of entries from the database;

and

overwriting data in the cache with the second plurality of entries without updating the first plurality of entries in the database unless a change has been made to at least one entry of the first plurality of entries stored in the cache.

18. The method of claim 8, wherein the contents icon includes a thumbnail image representative of data in the contents file.

19. The method of claim 1, wherein the contents icon includes a sequence of thumbnail images representative of data in the contents file.

20. The method of claim 19, wherein the sequence of thumbnail images forms an animation sequence.

21. The method of claim 1, further comprising

configuring the contents icon in every entry of the plurality of entries to be a particular size.

22. The method of claim 1, wherein the graphical user interface includes a menu screen for a multimedia processing system.

23. The method of claim 22, wherein the menu screen is configured as a series of icon arrays.

24. The method of claim 23, wherein configuring a series of icon arrays includes arranging a medium icon array in a horizontal direction.

25. The method of claim 24, wherein the medium icon array includes a plurality of medium icons.

26. The method of claim 25, wherein the plurality of medium icons includes a photograph icon, a music icon, a moving image icon, a broadcast icon, an optical disk icon, a web icon, and a game icon.

27. The method of claim 19, wherein the medium icon array includes an intersection area.

28. The method of claim 27, wherein configuring a series of icon arrays includes arranging a contents icon array in a vertical direction for a particular medium icon that is scrolled into the intersection area.

29. The method of claim 28, wherein configuring a series of icon arrays includes positioning the contents icon array proximate a left or right side of the display based on whether the user'"'"'s left hand or right hand is used to move and/or select the particular medium icon.

30. The method of claim 29, wherein the contents icon array includes a plurality of contents icons and an attention area.

31. The method of claim 30, further comprising

accessing the contents file of a contents icon scrolled into the attention area, wherein the contents file is accessed using the link in a database entry of the contents icon scrolled into the attention area.

32. The method of claim 31, wherein the database is stored in a multimedia processing apparatus.

33. The method of claim 1, wherein the graphical user interface is implemented in a multimedia processing apparatus.

34. The method of claim 33 wherein the multimedia processing apparatus is a game console, television, digital video recorder (DVR), cable set-top-box, home media server or consumer electronic device.

35. A graphical user interface accessible to a user suitable for receiving user-supplied inputs that facilitate selecting a multimedia icon, comprising:

a medium icon array arranged in a first direction, and including a plurality of user selectable medium icons;

a contents icon array arranged in a second direction, and including a plurality of user selectable contents icons, a real-time video image of a user of the graphical user interface, wherein the medium icon array and the contents icon array superposed over the image of the user, wherein the icons from the medium icon array and/or contents icon array are selected and moved based on motions of the image of the user;

wherein the plurality of user selectable contents icons is accessed from a database divided into a plurality of entries, each entry configured to be fixed in size and including at least data for each contents icon of the plurality of user selectable contents icons, such that the contents icons can be quickly accessed and scrolled in the graphical user interface at a relatively high speed.

36. The graphical user interface of claim 35, wherein the medium icon array and the contents icon array are configured in a cross shape with an intersecting area.

37. The graphical user interface of claim 35, further comprising

a cache configured to store data from the database.

38. The graphical user interface of claim 37, wherein the cache is configured to be overwritten without updating to the database unless changes have been made to the cache.

39. The graphical user interface of claim 35, wherein the database is stored in a multimedia processing apparatus.

40. The graphical user interface of claim 35, wherein the graphical user interface is implemented in a multimedia processing apparatus.

41. The graphical user interface of claim 40, wherein the multimedia processing apparatus is a game console, television, digital video recorder (DVR), cable set-top-box, home media server or consumer electronic device.

42. A computer program, stored in a tangible storage medium, for moving objects in a graphical user interface, the program comprising executable instructions that cause a computer to:

obtain a video image of a user of the interface;

display the video image on a display;

based on recognition of motions of the video image of the user, moving one or more objects displayed in the graphical user interface.

43. The method of claim 42, wherein moving one or more objects displayed in the graphical user interface includes scrolling a plurality of contents icons displayed in the graphical user interface.

44. The method of claim 43, further comprising retrieving an appropriate number of additional contents icons from a contents icon database to provide a scrolled display in the graphical user interface,

wherein the contents icon database is divided into a plurality of entries, each entry of the plurality of entries including at least data for a contents icon of the plurality of contents icons and a link to a contents file, each entry of the database is configured to be fixed in size so that the data for the contents icon can be quickly accessed, and the contents icon can be scrolled in the graphical user interface at a relatively high speed.

45. A multimedia processing apparatus, comprising:

a video image capture device adapted to capture a real-time video image of a user of a graphical user interface;

a display adapted to show the medium icon array and the contents icon array superposed over the image of the user; and

a display control that allows the user to move objects shown in the graphical user interface based on motions of the image of the user.

46. The multimedia apparatus of claim 45 wherein the display controller is configured to recognize of motions of an image of the user'"'"'s hand.

47. The multimedia apparatus of claim 46 wherein the display controller is configured to recognize a change in aspect ratio of the image of the user'"'"'s hand.

48. The multimedia apparatus of claim 46 wherein the display controller is configured to recognize whether a user is using his left or right hand to select or move one or more icons from the contents icon array.

49. The multimedia apparatus of claim 48 wherein recognizing whether a user is using his left or right hand includes determining a relative location of an image of the user'"'"'s hand relative to an image of the user'"'"'s head.

50. The method of claim 48 wherein determining a relative location of the user'"'"'s hand relative to an image of the user'"'"'s head includes determining whether a user'"'"'s arm crosses a user'"'"'s chest.

51. The multimedia apparatus of claim 46 wherein the display controller is configured to recognize a change in aspect ratio of a user'"'"'s hand or an object held within the user'"'"'s hand.

52. The apparatus of claim 45 wherein the video image capture device includes a digital camera.

53. The apparatus of claim 45, further comprising one or more microphones coupled to the display control to manipulate the medium icon array and/or the content icon array by voice commands.

54. The apparatus of claim 45, further comprising a memory configured as a contents icon database, including

a plurality of entries including data for a plurality of contents icons and links to multimedia contents,

wherein each entry of said plurality of entries is fixed in size such that the plurality of contents icons can be quickly accessed to allow the contents icons to be scrolled in a graphical user interface at a relatively high speed;

and

a processor configured to load the plurality of contents icons to allow a user to scroll and select a particular contents icon from the plurality of contents icons.

55. The multimedia processing apparatus of claim 45, wherein the graphical user interface includes a menu screen for a multimedia processing apparatus.

56. The multimedia processing apparatus of claim 55, wherein the menu screen is configured as a series of icon arrays.

57. The multimedia processing apparatus of claim 56, wherein the series of icon arrays includes a medium icon array arranged in a horizontal direction.

58. The multimedia processing apparatus of claim 57, wherein the medium icon array includes a plurality of medium icons.

59. The multimedia processing apparatus of claim 58, wherein the plurality of medium icons includes a photograph icon, a music icon, a moving image icon, a broadcast icon, an optical disk icon, a web icon, and a game icon.

60. The multimedia processing apparatus of claim 57, wherein the medium icon array includes a center area.

61. The multimedia processing apparatus of claim 60, wherein the series of icon arrays includes a contents icon array arranged in a vertical direction for a particular medium icon that is scrolled into the center area.

62. The multimedia processing apparatus of claim 61, wherein the contents icon array includes a plurality of contents icons and an attention area.

63. The multimedia processing apparatus of claim 62, wherein the links in the plurality of entries are used to access multimedia contents of a contents icon scrolled into the attention area.

64. The multimedia processing apparatus of claim 45, wherein the multimedia processing apparatus is a game console, television, digital video recorder (DVR), cable set-top-box, home media server or consumer electronic device.

65. A method of scrolling in a graphical user interface, comprising:

receiving a voice command to scroll a plurality of contents icons displayed in the graphical user interface; and

retrieving an appropriate number of additional contents icons from a contents icon database to provide a scrolled display in the graphical user interface,

wherein the contents icon database is divided into a plurality of entries, each entry of the plurality of entries including at least data for a contents icon of the plurality of contents icons and a link to a contents file, each entry of the database is configured to be fixed in size so that the data for the contents icon can be quickly accessed, and the contents icon can be scrolled in the graphical user interface at a relatively high speed.

66. A multimedia processing apparatus, comprising:

an audio signal capture device;

a memory configured as a contents icon database, including

a plurality of entries including data for a plurality of contents icons and links to multimedia contents,

wherein each entry of said plurality of entries is fixed in size such that the plurality of contents icons can be quickly accessed to allow the contents icons to be scrolled in a graphical user interface at a relatively high speed; and

a processor coupled to the audio signal capture device, the processor being configured to load the plurality of contents icons to allow a user to scroll and select a particular contents icon from the plurality of contents icons based on one or more voice commands from a user.

67. The multimedia processing apparatus of claim 66, wherein the multimedia processing apparatus is a game console, television, digital video recorder (DVR), cable set-top-box, home media server or consumer electronic device.