Voice and video control of interactive electronically simulated environment

• US 8,549,442 B2
• Filed: 12/12/2005
• Issued: 10/01/2013
• Est. Priority Date: 12/12/2005
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Marks, Richard L., Chen, Ruxin
• Primary Examiner(s): Abdul-Ali, Omar
• Assistant Examiner(s): KEATON, SHERROD L
• Application Number:
  US11/302,516
• Publication Number:
  US 20070139443A1
• Time in Patent Office: 2,850 Days
• Field of Search: 715/863, 715/853, 715/706, 715/709, 715/730, 715/866, 715/784, 715/786, 715/835, 715/839, 715/838
• US Class Current: 715/863
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4331 Caching operations, e.g. of an advertisement for later insertion during playback
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4438 Window management, e.g. event handling following interaction with the user interface
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 21/482 End-user interface for program selection broadcast systems using EPGs H04H60/72

Abstract

A method of moving objects in a graphical user interface, includes obtaining a video image of a user of the interface; displaying the video image on a display such that the video image is superposed with one or more objects displayed on the display; and moving one or more objects displayed on the display based on recognition of motions of the video image of the user. Recognition of motions of the video image may include recognition of motions of an image of the user'"'"'s hand.

Claims

1. A method of moving objects in a graphical user interface, comprising:

obtaining a video image of a user of the interface with an image capture device that the user faces while facing a display;

displaying a mirror image of the video image on the display such that the video image is superposed with one or more objects displayed on the display; and

moving one or more objects displayed on the display based on recognition of motions of the mirror image of the video image of the user, wherein moving the one or more objects includes, determining whether the user'"'"'s arm crosses the user'"'"'s chest, determining whether the user is using the user'"'"'s left or right hand to select or move the one or more objects, determining a relative location of an image of the user'"'"'s hand relative to an image of the user'"'"'s head, and positioning the one or more objects differently relative to the user'"'"'s head in the mirror image depending on whether the user'"'"'s left hand or right hand is used to move or select the one or more objects and whether the user'"'"'s arm crosses the user'"'"'s chest.

2. The method of claim 1 wherein recognition of motions of the video image includes recognition of motions of an image of the user'"'"'s hand or an object held in the user'"'"'s hand.

3. The method of claim 2 wherein recognition of motions of an image of the user'"'"'s hand or an object held in the user'"'"'s hand include recognition of a change in aspect ratio of the image of the user'"'"'s hand or an object held in the user'"'"'s hand.

4. The method of claim 1 wherein recognition of motions of the video image includes recognizing a change in aspect ratio of a user'"'"'s hand or an object held within the user'"'"'s hand.

5. The method of claim 1 wherein the one or more objects includes a plurality of contents icons.

6. The method of claim 5, further comprising receiving a voice command to select one or more icons from the plurality of contents icons.

7. The method of claim 6, wherein receiving a voice command to select one or more icons from the contents icon array includes displaying a list of selected content icons based on a keyword in the voice command.

8. The method of claim 7 wherein receiving a voice command includes the use of a device having one or more microphones attached to a video image capture device.

9. The method of claim 5 wherein moving one or more objects includes receiving a command to scroll and/or select one or more contents icons of the plurality of contents icons displayed on display;

and

retrieving an appropriate number of additional contents icons from a contents icon database to provide a scrolled display in the graphical user interface,

wherein the contents icon database is divided into a plurality of entries, each entry of the plurality of entries including at least data for a contents icon of the plurality of contents icons and a link to a contents file, each entry of the database is configured to be fixed in size so that the data for the contents icon can be quickly accessed, and the contents icon can be scrolled in the graphical user interface at a relatively high speed.

10. The method of claim 9, wherein the relatively high speed indicates a scroll speed where the contents icons are scrolled without pausing to accommodate loading icon images to scroll and visually smoothly display the contents icons in the graphical user interface.

11. The method of claim 9, wherein configuring each entry to be fixed in size includes configuring every entry of the plurality of entries to be a particular size.

12. The method of claim 11, wherein the database has a cache for storing data from the database.

13. The method of claim 12, further comprising retrieving a first plurality of entries from the database and storing the entries in the cache.

14. The method of claim 13, further comprising

retrieving a second plurality of entries from the database;

and

overwriting data in the cache with the second plurality of entries without updating the first plurality of entries in the database unless a change has been made to at least one entry of the first plurality of entries stored in the cache.

15. The method of claim 5, wherein the contents icon includes a thumbnail image representative of data in the contents file.

16. The method of claim 1, wherein the contents icon includes a sequence of thumbnail images representative of data in the contents file.

17. The method of claim 16, wherein the sequence of thumbnail images forms an animation sequence.

18. The method of claim 1, further comprising configuring the contents icon in every entry of the plurality of entries to be a particular size.

19. The method of claim 1, wherein the graphical user interface includes a menu screen for a multimedia processing system.

20. The method of claim 19, wherein the menu screen is configured as a series of icon arrays.

21. The method of claim 20, wherein configuring a series of icon arrays includes arranging a medium icon array in a horizontal direction.

22. The method of claim 21, wherein the medium icon array includes a plurality of medium icons.

23. The method of claim 22, wherein the plurality of medium icons includes a photograph icon, a music icon, a moving image icon, a broadcast icon, an optical disk icon, a web icon, and a game icon.

24. The method of claim 16, wherein the medium icon array includes an intersection area.

25. The method of claim 24, wherein configuring a series of icon arrays includes arranging a contents icon array in a vertical direction for a particular medium icon that is scrolled into the intersection area.

26. The method of claim 25, wherein the contents icon array includes a plurality of contents icons and an attention area.

27. The method of claim 26, further comprising accessing the contents file of a contents icon scrolled into the attention area, wherein the contents file is accessed using the link in a database entry of the contents icon scrolled into the attention area.

28. The method of claim 27, wherein the database is stored in a multimedia processing apparatus.

29. The method of claim 1, wherein the graphical user interface is implemented in a multimedia processing apparatus.

30. The method of claim 29 wherein the multimedia processing apparatus is a game console, television, digital video recorder (DVR), cable set-top-box, home media server or consumer electronic device.

51. The method of claim 1 wherein the video image is obtained with a single video image capture device.

31. A computer program, stored in a non-transitory storage medium, for

moving objects in a graphical user interface, the program comprising executable instructions that cause a computer to:

obtain a video image of a user of the interface;

display a mirror image of the video image of the user on a display;

determine whether a user'"'"'s arm crosses a user'"'"'s chest; and

based on recognition of motions of the mirror image of the video image of the user, moving one or more objects displayed in the graphical user interface, wherein moving the one or more objects includes determining whether the user is using the user'"'"'s left or right hand to select or move the one or more objects, determining a relative location of an image of the user'"'"'s hand relative to an image of the user'"'"'s head, and positioning the one or more objects differently relative to the user'"'"'s head in the mirror image depending on whether the user'"'"'s left hand or right hand is used to move or select the one or more objects and whether the user'"'"'s arm crosses the user'"'"'s chest.

32. The method of claim 31, wherein moving one or more objects displayed in the graphical user interface includes scrolling a plurality of contents icons displayed in the graphical user interface.

33. The method of claim 32, further comprising retrieving an appropriate number of additional contents icons from a contents icon database to provide a scrolled display in the graphical user interface,

wherein the contents icon database is divided into a plurality of entries, each entry of the plurality of entries including at least data for a contents icon of the plurality of contents icons and a link to a contents file, each entry of the database is configured to be fixed in size so that the data for the contents icon can be quickly accessed, and the contents icon can be scrolled in the graphical user interface at a relatively high speed.

34. A multimedia processing apparatus, comprising:

a video image capture device adapted to capture a real-time video image of a user of a graphical user interface;

a display adapted to show the medium icon array and the contents icon array superposed over a mirror image of the image of the user; and

a display control that allows the user to move one or more objects shown in the graphical user interface based on motions of the mirror image of the image of the user, wherein the display control is configured to determine whether the user'"'"'s arm crosses the user'"'"'s chest, determine whether the user is using the user'"'"'s left or right hand to select or move the one or more objects, determine a relative location of an image of the user'"'"'s hand relative to an image of the user'"'"'s head, and position the one or more objects differently relative to the user'"'"'s head in the mirror image depending on whether the user'"'"'s left hand or right hand is used to move or select the one or more objects and whether the user'"'"'s arm crosses the user'"'"'s chest.

35. The multimedia apparatus of claim 34 wherein the display controller is configured to recognize of motions of an image of the user'"'"'s hand.

36. The multimedia apparatus of claim 35 wherein the display controller is configured to recognize a change in aspect ratio of the image of the user'"'"'s hand.

37. The multimedia apparatus of claim 35 wherein the display controller is configured to recognize a change in aspect ratio of a user'"'"'s hand or an object held within the user'"'"'s hand.

38. The apparatus of claim 34 wherein the video image capture device includes a digital camera.

39. The apparatus of claim 34, further comprising one or more microphones coupled to the display control to manipulate the medium icon array and/or the content icon array by voice commands.

40. The apparatus of claim 34, further comprising a memory configured as a contents icon database, including

a plurality of entries including data for a plurality of contents icons and links to multimedia contents,

wherein each entry of said plurality of entries is fixed in size such that the plurality of contents icons can be quickly accessed to allow the contents icons to be scrolled in a graphical user interface at a relatively high speed;

and

a processor configured to load the plurality of contents icons to allow a user to scroll and select a particular contents icon from the plurality of contents icons.

41. The multimedia processing apparatus of claim 34, wherein the graphical user interface includes a menu screen for a multimedia processing apparatus.

42. The multimedia processing apparatus of claim 41, wherein the menu screen is configured as a series of icon arrays.

43. The multimedia processing apparatus of claim 42, wherein the series of icon arrays includes a medium icon array arranged in a horizontal direction.

44. The multimedia processing apparatus of claim 43, wherein the medium icon array includes a plurality of medium icons.

45. The multimedia processing apparatus of claim 44, wherein the plurality of medium icons includes a photograph icon, a music icon, a moving image icon, a broadcast icon, an optical disk icon, a web icon, and a game icon.

46. The multimedia processing apparatus of claim 43, wherein the medium icon array includes a center area.

47. The multimedia processing apparatus of claim 46, wherein the series of icon arrays includes a contents icon array arranged in a vertical direction for a particular medium icon that is scrolled into the center area.

48. The multimedia processing apparatus of claim 47, wherein the contents icon array includes a plurality of contents icons and an attention area.

49. The multimedia processing apparatus of claim 48, wherein the links in the plurality of entries are used to access multimedia contents of a contents icon scrolled into the attention area.

50. The multimedia processing apparatus of claim 34, wherein the multimedia processing apparatus is a game console, television, digital video recorder (DVR), cable set-top-box, home media server or consumer electronic device.