Systems and methods for using reduced hops to generate an augmented virtual reality scene within a head mounted system

• US 10,137,361 B2
• Filed: 12/30/2013
• Issued: 11/27/2018
• Est. Priority Date: 06/07/2013
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Perry, David
• Primary Examiner(s): Ahmed, Masud
• Application Number:
  US14/144,210
• Publication Number:
  US 20140364209A1
• Time in Patent Office: 1,793 Days
• Field of Search: 463 19- 43
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/33 using wide area network [WAN] connections
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/50 Controlling the output signals based on the game progress
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/71 using secure communication between game devices and game servers, e.g. by encrypting game data or authenticating players
• CPC Class Codes: A63F 13/90 Constructional details or arrangements of video game devices not provided for in groups A63F13/20 or A63F13/25, e.g. housing, wiring, connections or cabinets
• CPC Class Codes: A63F 13/98 Accessories, i.e. detachable arrangements optional for the use of the video game device, e.g. grip supports of game controllers
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/30 characterized by output arrangements for receiving control signals generated by the game device
• CPC Class Codes: A63F 2300/303 for displaying additional data, e.g. simulating a Head Up Display
• CPC Class Codes: A63F 2300/401 Secure communication, e.g. using encryption or authentication
• CPC Class Codes: A63F 2300/532 using secure communication, e.g. by encryption, authentication
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 : Head tracking input arrangements
• CPC Class Codes: G06F 3/013 : Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 : Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

Systems and methods for using reduced hops to generate an augmented virtual reality scene in a head mounted display. The head mounted display is used for game play via a game cloud system. The head mounted display includes a communications circuit for sending real-world media associated with a game program via a network. The real-world media is processed by the game cloud system and streamed directly from the communications circuit to the game cloud system. The head mounted display further includes a game processing circuit coupled to the communications circuit. The game processing circuit is used for decoding computer-generated interactive media received from the game cloud system via the network. The game processing circuit drives a portion of interactivity associated with the game program by superimposing the computer-generated interactive media on the real-world media. The computer-generated interactive media is generated based on the real-world media.

Claims

1. A head mounted display for game play via a game cloud system, the head mounted display comprising:

a communications circuit of the head-mounted display for sending real-world media to a game program via the Internet, the game program being executed by the game cloud system and the real-world media is captured via a camera of the head mounted display, the communications circuit of the head-mounted display is configured to communicate with the game cloud system via a router that connects to the Internet without using a computer to communicate with the router and the Internet;

a decoder in the head-mounted display coupled to the communications circuit, wherein the decoder is configured to decode compressed computer-generated interactive media received from a compressor of the game cloud system via the Internet and the router to provide decoded computer-generated interactive media; and

a game processing circuit driving a portion of interactivity associated with the game program by superimposing a portion of the decoded computer-generated interactive media on the real-world media, the compressed computer-generated interactive media generated based on the real-world media.

2. The head mounted display of claim 1,

wherein the game processing circuit is used for rendering images of the real-world media and the portion of decoded computer-generated interactive media, wherein the communications circuit is a wireless access card configured to receive the computer-generated interactive media from the game cloud system without using the computer.

3. The head mounted display of claim 1, further comprising:

a user input circuit coupled to the game processing circuit, the user input circuit for generating an input based on a head action of a user; and

an audio circuit for generating sounds based on a portion of the decoded computer-generated interactive media or based on the input, wherein the communications circuit comprises a wireless access card configured to generate packets having the input for sending via the Internet to the game cloud system for processing of the input by the game cloud system.

4. The head mounted display of claim 1, further comprising:

a user input circuit coupled to the game processing circuit, the user input circuit for generating an input based on a head action of a user, wherein the user input circuit includes one or more inertial sensors for determining a position and motion of the head mounted display, wherein the communications circuit is configured to generate packets having the input for sending via the Internet to the game cloud system for processing of the input by the game cloud system.

5. The head mounted display of claim 4, wherein the inertial sensors include a gyroscope, a magnetometer, and an accelerometer.

6. The head mounted display of claim 1, further comprising a communications circuit to facilitate peer-to-peer multichannel communication with another head mounted display via pairing.

7. The head mounted display of claim 1, further comprising a user input circuit coupled to the game processing circuit, the user input circuit for generating an input based on a head action of a user, wherein the communications circuit, the user input circuit, and the game processing circuit are integrated in one or more individual circuit chips.

8. The head mounted display of claim 1, wherein the game processing circuit includes a microprocessor for rendering the portion of decoded computer-generated interactive media, the head mounted display further comprising a buffer for buffering a stream of the compressed computer-generated interactive media.

9. The head mounted display of claim 1, further comprising:

an additional communications circuit for communicating with a hand-held controller, wherein the additional communications circuit is configured to receive an input from the hand-held controller, wherein the communications circuit is configured to packetize the input and send the input via the Internet to the game cloud system.

10. The head mounted display of claim 1, further comprising a user input circuit for receiving an action from a user to generate an input, wherein the input is a voice of a user or a gaze of the user, wherein the communications circuit is configured to generate packets from the input and send the packets via the Internet to the game cloud system.

11. The head mounted display of claim 10, wherein the input is mapped to an input of a hand-held controller to determine a change in state of the game program.

12. The head mounted display of claim 1, wherein the head mounted display is used with a hand-held controller that communicates directly with the game cloud system, wherein the hand-held controller is configured to packetize a selection of a button to generate one or more packets and send the one or more packets via the Internet to the game cloud system.

13. The head mounted display of claim 1, further comprising:

an additional communications circuit configured to receive a position and motion of a hand-held controller from the hand-held controller; and

a display screen configured to display a single augmented virtual reality scene generated from a position and motion of the head mounted display and the position and motion of the hand-held controller.

14. The head mounted display of claim 1, wherein the head mounted display is used with a hand-held controller that communicates with the computer.

15. The head mounted display of claim 1, further comprising a user input circuit for receiving an action from a user to generate an input, the input including a position and motion detected by the user input circuit, wherein the communications circuit includes a wireless access card configured to communicate the input directly to the game cloud system.

16. The head mounted display of claim 1, further comprising:

a display device configured to display one or more wireless networks for communication with the game cloud system;

a user input circuit configured to receive an indication of a selection of one of the wireless networks,

wherein the communications circuit is configured to couple to the game cloud system via the one of the wireless networks upon the reception of the indication of the selection of the one of the wireless networks by the user input circuit.

17. The head mounted display of claim 1, further comprising:

a microcontroller configured to execute an operating system to further execute a network access application, wherein the network access application is executed to access one or more wireless networks;

a user input device configured to recognize a gesture to facilitate a determination that a selection of one of the wireless networks is made,

wherein the communications circuit is configured to couple to the game cloud system via the one of the wireless networks in response to the determination of the selection of the one of the wireless networks.

18. The head mounted display of claim 1, wherein the communications circuit is configured to receive one or more packets of the computer-generated interactive media via the Internet and is configured to depacketize the one or more packets to extract the computer-generated interactive media from the packets.

19. The head mounted display of claim 1, further comprising a video decoder, wherein the video decoder is a device separate from the decoder and is coupled to the decoder.

20. The head mounted display of claim 1, further comprising:

a video audio separator configured to separate the decoded computer-generated interactive media into audio data and video data;

a video audio synchronizer configured to synchronize the video data with the audio data;

an audio codec configured to convert the synchronized audio data to generate audio signals;

one or more speakers configured to play the audio signals to generate sound;

a video decoder configured to convert the synchronized video data into analog video signals,

wherein the game processing circuit is configured to apply a rendering operation to the analog video signals to display an augmented reality scene having the real-world media and the portion of decoded computer-generated interactive media.

21. A method for game play via a game cloud system, the method comprising:

capturing real-world media associated with a game program;

sending the real-world media via the Internet, the real-world media being processed by the game cloud system and streamed directly to the game cloud system via a router without using a game console;

decoding, within a head-mounted display, compressed computer-generated interactive media received from a compressor of the game cloud system via the Internet and the router to provide decoded computer-generated interactive media; and

driving a portion of interactivity associated with the game program by superimposing a portion of the decoded computer-generated interactive media on the real-world media, the compressed computer-generated interactive media generated based on the real-world media.

22. The method of claim 21, further comprising:

displaying one or more wireless networks for communication with the game cloud system;

receiving an indication of a selection of one of the wireless networks; and

connecting to the game cloud system via the one of the wireless networks upon receiving the indication of the selection of the one of the wireless networks.

23. The method of claim 21, further comprising:

executing an operating system to further execute a network access application, wherein the network access application is executed to access one or more wireless networks;

recognizing a gesture to determine that a selection of one of the wireless networks is made; and

connecting to the game cloud system via the one of the wireless networks upon determining that the selection of the one of the wireless networks is made.

24. A head mounted display for interfacing with a cloud system, the head mounted display comprising:

a communications circuit of the head mounted display for sending real-world scenes to a program via the Internet, the program being executed by the cloud system and the real-world scenes are captured via a camera of the head mounted display, the communications circuit of the head mounted display is configured to communicate with the cloud system via a router that connects to the network without using a separate computer to communicate with the router and the network;

a decoder in the head mounted display coupled to the communications circuit, wherein the decoder is configured to decode compressed computer-generated interactive media received from a compressor of the cloud system via the network and the router to provide decoded computer-generated interactive media; and

a processing circuit configured to superimpose a portion of the decoded computer-generated interactive media onto the real-world scenes, the compressed computer-generated interactive media generated based on the real-world scenes.

25. The head mounted display of claim 24, further comprising:

a display device configured to display one or more wireless networks for communication with the cloud system;

a user input circuit configured to receive an indication of a selection of one of the wireless networks,

wherein the communications circuit is configured to couple to the cloud system via the one of the wireless networks in response to the reception of the indication of the selection of the one of the wireless networks by the user input circuit.

26. The head mounted display of claim 24, further comprising:

a microcontroller configured to execute an operating system to further execute a network access application, wherein the network access application is executed to access one or more wireless networks;

a user input device configured to recognize a gesture to facilitate a determination that a selection of one of the wireless networks is made,

wherein the communications circuit is configured to couple to the cloud system via the one of the wireless networks in response to the determination that the selection of the one of the wireless networks is made.