SYSTEM FOR ENABLING A HANDHELD DEVICE TO CAPTURE VIDEO OF AN INTERACTIVE APPLICATION

• US 20120086631A1
• Filed: 10/12/2010
• Published: 04/12/2012
• Est. Priority Date: 10/12/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Osman, Steven, Zhu, Yunpeng, Stafford, Jeffrey Roger
• Granted Patent: US 8,854,298 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/323 between game devices with different hardware characteristics, e.g. hand-held game devices connectable to game consoles or arcade machines
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/525 Changing parameters of virtual cameras
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/403 Connection between platform and handheld device
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player

Abstract

Methods and systems for enabling a handheld device to capture video of an interactive session of an interactive application presented on a main display are provided. An interactive session of the interactive application defines interactivity between a user and the interactive application. An initial position and orientation of a handheld device operated by a spectator are determined. A current state of the interactive application based on the interactivity between the user and the interactive application is determined The position and orientation of the handheld device are tracked during the interactive session. A spectator video stream of the interactive session based on the current state of the interactive application and the tracked position and orientation of the handheld device is generated. The spectator video stream is rendered on a handheld display of the handheld device.

Claims

1. A method for enabling a handheld device to capture video of an interactive session of an interactive application presented on a main display, the method comprising:

initiating the interactive session of the interactive application, the interactive session defining interactivity between a user and the interactive application;

determining an initial position and orientation of a handheld device operated by a spectator;

determining a current state of the interactive application based on the interactivity between the user and the interactive application;

tracking the position and orientation of the handheld device during the interactive session;

generating a spectator video stream of the interactive session based on the current state of the interactive application and the tracked position and orientation of the handheld device;

rendering the spectator video stream on a handheld display of the handheld device.

2. The method of claim 1, wherein the interactive session includes receiving interactive input from a controller device operated by the user, the interactive input being utilized to determine the current state of the interactive application.

3. The method of claim 1, further comprising,

generating an application video stream based on the current state of the interactive application;

rendering the application video stream on the main display;

wherein the interactive session defines interactivity between the user and the rendered application video stream.

4. The method of claim 1, wherein the position and orientation of the handheld device define a position and orientation of a virtual viewpoint within a virtual space defined by the interactive application;

and

wherein the spectator video stream includes images of the virtual space captured from the perspective of the virtual viewpoint.

5. The method of claim 1, further comprising,

transmitting the spectator video stream from the interactive application to the handheld device so as to enable the rendering of the spectator video stream on the handheld display of the handheld device.

6. The method of claim 1, further comprising,

determining application state data based on the current state of the interactive application;

transmitting the application state data to the handheld device;

wherein the spectator video stream is generated at the handheld device based on the application state data and the tracked position and orientation of the handheld device.

7. The method of claim 1, further comprising,

capturing an environmental video stream of an interactive environment in which the interactivity between the user and the interactive application occurs;

wherein the spectator video stream is generated based on the environmental video stream.

8. The method of claim 7, wherein the spectator video stream is generated by augmenting the environmental video stream with a virtual element.

9. The method of claim 8, wherein the generating of the spectator video stream includes,

detecting the user within the environmental video stream, and

replacing at least a portion of the detected user in the environmental video stream with a virtual element.

10. The method of claim 9, wherein the replacing includes replacing the detected user in the environmental video stream with a character of the interactive application controlled by the user.

11. The method of claim 1, further comprising,

determining an initial position of the user;

tracking the position of the user during the interactive session;

wherein the spectator video stream is generated based on the position and orientation of the handheld device relative to the position of the user.

12. The method of claim 11, further comprising,

mapping an object within a virtual environment of the interactive application to the position of the user;

wherein the position and orientation of the handheld device relative to the position of the user define a position and orientation of a virtual viewpoint within a virtual space defined by the interactive application; and

wherein the spectator video stream includes images of the virtual space captured from the perspective of the virtual viewpoint, the object being included in the images of the spectator video stream when the handheld device is oriented towards the position of the user.

13. The method of claim 12, wherein the object is controlled by the user.

14. The method of claim 13, wherein the object is a character of the interactive application.

15. The method of claim 11, wherein the position of the user is determined based on the position of a controller device operated by the user.

16. The method of claim 12, further comprising,

capturing an environmental video stream of an interactive environment in which the interactivity between the user and the interactive application occurs;

wherein the position of the user and the position and orientation of the handheld device relative to the position of the user are determined at least in part based on the environmental video stream.

17. The method of claim 1, further comprising,

storing the spectator video stream in a memory.

18. A system for capturing video of an interactive session of an interactive application presented on a main display, the system comprising:

(a) a console device for presenting the interactive application, the console device including,

(i) a session module for initiating the interactive session of the interactive application, the interactive session defining interactivity between a user and the interactive application,

(ii) a handheld device module for determining and tracking a position and orientation of a handheld device operated by a spectator during the interactive session,

(iii) a spectator video stream module for generating a spectator video stream of the interactive session based on a current state of the interactive application and the tracked position and orientation of the handheld device; and

(b) a handheld device, the handheld device including a handheld display for rendering the spectator video stream.

19. The system of claim 18, further comprising,

(c) a controller device operated by the user, the controller device providing interactive input, the interactive input being utilized to determine the current state of the interactive application.

20. The system of claim 18, wherein the console device further includes an application video stream module for generating an application video stream based on the current state of the interactive application and rendering the application video stream on the main display;

wherein the interactive session defines interactivity between the user and the rendered application video stream.

21. The system of claim 18, wherein the position and orientation of the handheld device define a position and orientation of a virtual viewpoint within a virtual space defined by the interactive application;

and

wherein the spectator video stream includes images of the virtual space captured from the perspective of the virtual viewpoint.

22. The system of claim 18, wherein the spectator video stream module transmits the spectator video stream from the interactive application to the handheld device so as to enable the rendering of the spectator video stream on the handheld display of the handheld device.

23. The system of claim 18, wherein the handheld device further includes an image capture device for capturing an environmental video stream of an interactive environment in which the interactivity between the user and the interactive application occurs;

and

wherein the spectator video stream module generates the spectator video stream based on the environmental video stream.

24. The system of claim 23, wherein the spectator video stream module generates the spectator video stream by augmenting the environmental video stream with a virtual element.

25. The system of claim 24, wherein the spectator video stream module generates the spectator video stream by detecting the user within the environmental video stream, and replacing at least a portion of the detected user in the environmental video stream with a virtual element.

26. The system of claim 25, wherein the replacing includes replacing the detected user in the environmental video stream with a character of the interactive application controlled by the user.

27. The system of claim 18, further comprising a user tracking module, the user tracking module determining an initial position of the user and tracking the position of the user during the interactive session;

and

wherein the spectator video stream module generates the spectator video stream based on the position and orientation of the handheld device relative to the position of the user.

28. The system of claim 27, wherein the interactive application maps an object within a virtual environment of the interactive application to the position of the user;

wherein the position and orientation of the handheld device relative to the position of the user define a position and orientation of a virtual viewpoint within a virtual space defined by the interactive application; and

wherein the spectator video stream includes images of the virtual space captured from the perspective of the virtual viewpoint, the object being included in the images of the spectator video stream when the handheld device is oriented towards the position of the user.

29. The system of claim 28, wherein the object is controlled by the user.

30. The system of claim 29, wherein the object is a character of the interactive application.

31. The system of claim 27, wherein the position of the user is determined based on the position of a controller device operated by the user.

32. The system of claim 28,

wherein the handheld device includes an image capture device for capturing an environmental video stream of an interactive environment in which the interactivity between the user and the interactive application occurs;

wherein the position of the user and the position and orientation of the handheld device relative to the position of the user are determined at least in part based on the environmental video stream.

33. The system of claim 18, wherein the handheld device includes a memory for storing the spectator video stream.