Methods and systems for social sharing head mounted display (HMD) content with a second screen

• US 9,691,181 B2
• Filed: 05/05/2014
• Issued: 06/27/2017
• Est. Priority Date: 02/24/2014
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Black, Glenn, Stafford, Jeffrey, Mikhailov, Anton, Watson, Brian
• Primary Examiner(s): Richer, Joni
• Application Number:
  US14/270,257
• Publication Number:
  US 20150243078A1
• Time in Patent Office: 1,149 Days
• Field of Search: None
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/27 characterised by a large display in a public venue, e.g. in a movie theatre, stadium or game arena
• CPC Class Codes: A63F 13/53 involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 30/34 Stereoscopes providing a stereoscopic pair of separated images corresponding to parallactically displaced views of the same object, e.g. 3D slide viewers
• CPC Class Codes: G06F 3/002 Specific input/output arrangements not covered by G06F3/01 - G06F3/16 other optical apparatus G02B27/00
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 2219/2016 : Rotation, translation, scaling

Abstract

Systems and method for processing video frames generated for display on a head mounted display (HMD) to a second screen are provided. One example method includes receiving the video frames formatted for display on the HMD, and while passing the video frames to the HMD, selecting a portion of content from the video frames and processing the portion of the content for output to a second screen. The video frames viewed in the HMD are a result of interactive play executed for viewing on the HMD. The second screen configured to render an undistorted view of the interactive play on the HMD. In one example, the method and system enable additional content to be rendered on the second screen (e.g., second screen content, such as social interactive play with others, other non-game content, player-player communication, etc.).

Claims

1. A system for processing video frames generated for a head mounted display (HMD) for sharing to a second screen, comprising,

a client system having a processor for generating video frames in response to interactive game play of a video game using the HMD, the generated video frames including in-band regions that include the video game content (VGC) to be displayed on the HMD and out-of-band (OOB) regions that include second screen content (SSC);

and

a system-on-a-chip (SOC) processor having an input interface for receiving the generated video frames, the SOC processor including extract logic for extracting the SSC from the out-of-band regions and a graphics processor unit (GPU) for selecting a portion of the generated video frames and formatting for display on the second screen, the extracted SSC and the portion of the generated video frames being provided to a first output interface of the SOC processor, the generated video frames received via the input interface being provided as a pass-through to a second output interface of the SOC processor;

wherein the first output interface of the SOC processor is connectable to the second screen;

wherein the second output interface of the SOC processor is connectable to the HMD;

wherein the portion of the generated video frames, selected by the GPU includes video game content (VGC) that is predefined for display to one eye in a display of the HMD, the portion further excluding the OOB regions, and wherein the formatting by the GPU includes performing an un-distortion function to the VGC in the portion, the un-distortion function removes graphic effects that are native for display in the HMD.

2. The system of claim 1, wherein the SOC processor includes a codec module for decoding the SSC extracted by the extract logic.

3. The system of claim 1, wherein the formatting by the GPU includes a cropping function and a scaling function, the cropping function and scaling function processed to fit a native display format of the second screen, that is different than a native format of the HMD.

4. The system of claim 1, further comprising,

a camera input received by the client system, the camera input configured to capture image data of a scene that includes the HMD, the captured image data used by the client system to modify the interactive game play of the video game;

a motion pass-through module in the SOC processor for receiving motion data from the HMD, the motion data used by the client system to further modify the interactive game play of the video game;

wherein modifications in the interactive game play are rendered in the video frames produced by the client system, the video frames being pass-through the SOC processor to the HMD and processed by the SOC processor for sharing to the second screen via the SOC processor.

5. The system of claim 1, wherein the extract logic operates on the video frames received from the input interface.

6. A system for processing video frames generated for a head mounted display (HMD) for sharing to a second screen, comprising,

a client system having a processor for generating video frames in response to interactive game play of a video game using the HMD, the generated video frames including in-band regions that include the video game content (VGC) to be displayed on the HMD and out-of-band (OOB) regions that include second screen content (SSC);

and

a system-on-a-chip (SOC) processor having an input interface for receiving the generated video frames, the SOC processor including extract logic for extracting the SSC from the out-of-band regions and a graphics processor unit (GPU) for selecting a portion of the generated video frames and formatting for display on the second screen, the extracted SSC and the portion of the generated video frames being provided to a first output interface of the SOC processor, the generated video frames received via the input interface being provided as a pass-through to a second output interface of the SOC processor;

wherein the first output interface of the SOC processor is connectable to the second screen;

wherein the second output interface of the SOC processor is connectable to the HMD, and wherein the SOC processor further comprises,

an audio pass-through communicated between the input interface and the first output interface.

7. A system for processing video frames generated for a head mounted display (HMD) for sharing to a second screen, comprising,

a client system having a processor for generating video frames in response to interactive game play of a video game using the HMD, the generated video frames including in-band regions that include the video game content (VGC) to be displayed on the HMD and out-of-band (OOB) regions that include second screen content (SSC);

and

a system-on-a-chip (SOC) processor having an input interface for receiving the generated video frames, the SOC processor including extract logic for extracting the SSC from the out-of-band regions and a graphics processor unit (GPU) for selecting a portion of the generated video frames and formatting for display on the second screen, the extracted SSC and the portion of the generated video frames being provided to a first output interface of the SOC processor, the generated video frames received via the input interface being provided as a pass-through to a second output interface of the SOC processor;

wherein the first output interface of the SOC processor is connectable to the second screen;

wherein the second output interface of the SOC processor is connectable to the HMD, and wherein the SOC processor further comprises,

a USB module input for connection with the client system, the USB module configured to receive audio and output motion data obtained from the HMD back to the client system.

8. The system of claim 7, wherein the USB module communicates the received audio from the client system to a 3D audio rendering module for delivery to the HMD, the 3D audio rendering module further configured with an input for receiving audio from the input interface.

9. The system of claim 8, wherein the SOC processor further comprises,

a sensor pass-through module for receiving motion data from the HMD, the motion data being communicated to the USB module for communication to the client system, the motion data identifying motion of the HMD that is used by the client system to modify the interactive game play of the video game.

10. A system for processing video frames generated for a head mounted display (HMD) for sharing to a second screen, comprising,

a client system having a processor for generating video frames in response to interactive game play of a video game using the HMD, the generated video frames including in-band regions that include the video game content (VGC) to be displayed on the HMD;

and

a system-on-a-chip (SOC) processor having an input interface for receiving the generated video frames, the SOC processor including,

a USB input for receiving video frames including second screen content (SSC);

extract logic for extracting the SSC from video frames;

a codec unit configured to receive the video frames and extracted SSC;

a graphics processor unit (GPU) for selecting a portion of the generated video frames and formatting for display on the second screen, the extracted SSC and the portion of the generated video frames being provided to a first output interface of the SOC processor, the generated video frames received via the input interface being provided as a pass-through to a second output interface of the SOC processor;

wherein the first output interface of the SOC processor is connectable to the second screen;

wherein the second output interface of the SOC processor is connectable to the HMD.

11. The system of claim 10, wherein the codec decodes the SSC encoded in H.264 by the client system.

12. The system of claim 10, wherein the portion of the generated video frames, selected by the GPU includes video game content (VGC) that is predefined for display to one eye in a display of the HMD, the portion further excluding out-of-band (OOB) regions.

13. The system of claim 10, wherein the formatting by the GPU includes performing an un-distortion function to the VGC in the portion, the un-distortion function removes graphic effects that are native for display in the HMD.

14. The system of claim 13, wherein the formatting by the GPU includes a cropping function and a scaling function, the cropping function and scaling function processed to fit a native display format of the second screen, that is different than a native format of the HMD.

15. The system of claim 10, wherein the SOC processor further comprises,

an audio pass-through communicated between the input interface and the first output interface.

16. The system of claim 10, wherein the USB input receives audio from the client system and returns output motion data obtained from the HMD back to the client system.

17. The system of claim 16, wherein the USB input communicates the received audio from the client system to a 3D audio rendering module for delivery to the HMD, the 3D audio rendering module further configured with an input for receiving audio from the input interface.

18. The system of claim 10, wherein the SSC is HTML content, the HTML content is configured for output by the first output interface of the SOC processor, for display on the second screen.

19. A system for processing video frames generated for a head mounted display (HMD) for sharing to a second screen, comprising,

a client system having a processor for generating video frames in response to interactive game play of a video game using the HMD, the generated video frames including in-band regions that include the video game content (VGC) to be displayed on the HMD and out-of-band (OOB) regions;

and

a system-on-a-chip (SOC) processor (106b) having an input interface for receiving the generated video frames, the SOC processor including,

a format module for formatting the received frames (408);

a graphics processor unit (GPU) for selecting a portion of the generated video frames and formatting for display on the second screen, the formatted portion of the generated video frames being provided to a first output interface of the SOC processor, the portion of the generated video frames provided to the second screen providing a substantial mirroring of the video frames rendered on the HMD, in substantial real time; and

a pass-through interface for communicating the generated video frames between the input interface to a second output interface of the SOC processor;

wherein the first output interface of the SOC processor is connectable to the second screen;

wherein the second output interface of the SOC processor is connectable to the HMD.

20. The system of claim 19, wherein the portion of the generated video frames, selected by the GPU includes video game content (VGC) that is predefined for display to one eye in a display of the HMD, the portion further excluding the OOB regions.

21. The system of claim 19, wherein the formatting by the GPU includes performing an un-distortion function to the VGC in the portion, the un-distortion function removes graphic effects that are native for display in the HMD.

22. The system of claim 19, wherein the formatting by the GPU includes a cropping function and a scaling function, the cropping function and scaling function processed to fit a native display format of the second screen, that is different than a native format of the HMD.

23. The system of claim 19, wherein the SOC processor further comprises,

an audio pass-through communicated between the input interface and the first output interface.

24. The system of claim 19, wherein the SOC processor further comprises,

a USB module input for connection with the client system, the USB module configured to receive audio and output motion data obtained from the HMD back to the client system.

25. The system of claim 24, wherein the USB module communicates the received audio from the client system to a 3D audio rendering module for delivery to the HMD, the 3D audio rendering module further configured with an input for receiving audio from the input interface.