Using a portable device to interface with a video game rendered on a main display

• US 10,099,147 B2
• Filed: 09/30/2013
• Issued: 10/16/2018
• Est. Priority Date: 08/19/2004
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Mallinson, Dominic S., Marks, Richard L.
• Primary Examiner(s): Deodhar, Omkar
• Assistant Examiner(s): Lee, Wei
• Application Number:
  US14/042,631
• Publication Number:
  US 20140031120A1
• Time in Patent Office: 1,842 Days
• Field of Search: 463 31
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/217 using environment-related information, i.e. information generated otherwise than by the player, e.g. ambient temperature or humidity
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/6081 generating an output signal, e.g. under timing constraints, for spatialization
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1632 External expansion units, e.g. docking stations
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 1/00323 : with a measuring, monitoring or signaling apparatus, e.g. for transmitting measured information to a central location
• CPC Class Codes: H04W 4/33 : for indoor environments, e.g. buildings

Abstract

A method for interfacing with a video game is provided, including: rendering display data on a main display, the display data defining a scene rendered by the video game, the display data being configured to include a visual cue; capturing the display data by an image capture device incorporated into a portable device; analyzing the captured display data to identify the visual cue; in response to identification of the visual cue, determining additional information that is in addition to the scene of the video game that is displayed on the main display, the additional information defining graphics or text to be added to the scene of the video game; presenting the additional information on a personal display incorporated into the portable device, the presentation of the additional information being synchronized to the rendering of the display data on the main display.

Claims

1. A method for interfacing with a video game, comprising:

rendering display data on a main display, the display data defining a scene rendered by the video game, the display data being configured to include a visual cue;

capturing the display data by an image capture device incorporated into a portable device while being directed toward the main display to enable the capturing;

analyzing the captured display data to identify the visual cue as depicted in the scene of the video game as depicted in the display data;

in response to identification of the visual cue, determining additional information that is in addition to the scene of the video game that is displayed on the main display, the additional information defining graphics or text to be added to the scene of the video game;

presenting the additional information on a personal display incorporated into the portable device, the presentation of the additional information being synchronized to the rendering of the display data on the main display, the synchronization being driven by the identification of the visual cue.

2. The method of claim 1, wherein the image capture device is a video capture device configured to continuously capture changing frames defined by the display data as it is rendered on the main display, wherein the additional information presented on the personal display is updated accordingly based on identification of the visual cue or additional visual cues defined in the changing frames by the display data.

3. The method of claim 1, wherein the visual cue is defined by an object of the video game or a marker.

4. The method of claim 1, wherein the additional information defines at least one additional object of the video game.

5. The method of claim 1, wherein determining the additional information includes receiving the additional information over a wireless connection by the portable device.

6. The method of claim 1, wherein determining the additional information includes generating the additional information by the portable device.

7. The method of claim 1, wherein presenting the additional information includes incorporating the graphics into the scene of the video game to provide extra functionality for the video game.

8. The method of claim 1, wherein determining additional information includes accessing a user preference setting defined for a user of the portable device, the additional information being defined based on the user preference setting.

9. A method for interfacing with a video game, comprising:

rendering display data on a main display, the display data defining a scene rendered by the video game, the display data being configured to include a visual cue;

capturing the display data by a first image capture device incorporated into a first portable device while being directed toward the main display to enable the capturing;

analyzing the captured display data of the first image capture device to identify the visual cue as depicted in the scene of the video game as depicted in the display data, and, in response to identification of the visual cue, determining first additional information that is in addition to the scene of the video game that is displayed on the main display, the first additional information defining graphics or text to be added to the scene of the video game;

presenting the first additional information on a first personal display incorporated into the first portable device, the presentation of the first additional information being synchronized to the rendering of the display data on the main display, the synchronization being driven by the identification of the visual cue;

capturing the display data by a second image capture device incorporated into a second portable device while being directed toward the main display to enable the capturing;

analyzing the captured display data of the second image capture device to identify the visual cue as depicted in the scene of the video game as depicted in the display data, and, in response to identification of the visual cue, determining second additional information that is in addition to the scene of the video game that is displayed on the main display, the second additional information defining graphics or text to be added to the scene of the video game, the second additional information being different from the first additional information;

presenting the second additional information on a second personal display incorporated into the second portable device, the presentation of the second additional information being synchronized to the rendering of the display data on the main display, the synchronization being driven by the identification of the visual cue.

10. The method of claim 9,

wherein the first image capture device is a first video capture device configured to continuously capture changing frames defined by the display data as it is rendered on the main display, wherein the first additional information presented on the first personal display is updated accordingly based on identification of the visual cue or additional visual cues defined in the changing frames by the display data;

wherein the second image capture device is a second video capture device configured to continuously capture the changing frames defined by the display data as it is rendered on the main display, wherein the second additional information presented on the second personal display is updated accordingly based on identification of the visual cue or additional visual cues defined in the changing frames by the display data.

11. The method of claim 9, wherein the visual cue is defined by an object of the video game or a marker.

12. The method of claim 9, wherein the first additional information or the second additional information defines at least one additional object of the video game.

13. The method of claim 9, wherein determining the first additional information or the second additional information includes receiving the additional information over a wireless connection by the first portable device or the second portable device, respectively.

14. The method of claim 9, wherein determining the first additional information or the second additional information includes generating the first additional information or the second additional information, respectively, by the first portable device or the second portable device, respectively.

15. The method of claim 9, wherein presenting the first additional information or the second additional information includes incorporating the graphics or text of the first additional information, or the graphics or text of the second additional information, respectively, into the scene of the video game to provide extra functionality for the video game.

16. The method of claim 9, wherein determining the first additional information or the second additional information includes accessing a user preference setting defined for a user of the first portable device or the second portable device, respectively, the first additional information or the second additional information being respectively defined based on the user preference setting.

17. A portable device for interfacing with a video game, comprising:

an image captured device configured to capture display data rendered on a main display while being directed toward the main display to enable the capturing, the display data defining a scene rendered by the video game, the display data being configured to include a visual cue;

image recognition logic configured to analyze the captured display data to identify the visual cue as depicted in the scene of the video game as depicted in the display data;

image generation logic, configured in response to identification of the visual cue, to determine additional information that is in addition to the scene of the video game that is displayed on the main display, the additional information defining graphics or text to be added to the scene of the video game;

a personal display configured to present the additional information, the presentation of the additional information being synchronized to the rendering of the display data on the main display, the synchronization being driven by the identification of the visual cue.

18. The portable device of claim 17, wherein the image capture device is a video capture device configured to continuously capture changing frames defined by the display data as it is rendered on the main display, wherein the additional information presented on the personal display is updated accordingly based on identification of the visual cue or additional visual cues defined in the changing frames by the display data.

19. The portable device of claim 17, wherein the image generation logic is configured to determine the additional information by receiving the additional information over a wireless connection or by generating the additional information at the portable device.

20. The portable device of claim 17, further comprising:

a head mount for enabling the portable device to be worn by a user.

21. A method for interfacing with a video game, comprising:

capturing display data, rendered on a main display, using an image capture device incorporated into a portable device while being directed toward the main display to enable the capturing, the display data defining a scene rendered by the video game, the display data being configured to include a visual cue;

analyzing the captured display data to identify the visual cue as depicted in the scene of the video game as depicted in the display data;

in response to identification of the visual cue, determining additional information that is in addition to the scene of the video game that is displayed on the main display, the additional information including graphics or text related to the scene of the video game;

presenting the additional information on a display of the portable device, the presentation of the additional information being responsive to the identification of the visual cue.

22. The method of claim 21, wherein the image capture device is a video capture device configured to continuously capture changing frames defined by the display data as it is rendered on the main display, wherein the additional information presented on the personal display is updated accordingly based on identification of the visual cue or additional visual cues defined in the changing frames by the display data.

23. The method of claim 21, wherein the visual cue is defined by an object of the video game or a marker.

24. The method of claim 21, wherein the additional information includes at least one additional object of the video game.

25. The method of claim 21, wherein determining the additional information includes receiving the additional information over a wireless connection by the portable device.

26. The method of claim 21, wherein determining the additional information includes generating the additional information by the portable device.

27. The method of claim 21, wherein determining additional information includes accessing a user preference setting defined for a user of the portable device, the additional information being defined based on the user preference setting.