Projectile object rendering for a virtual reality spectator

• US 10,380,798 B2
• Filed: 09/29/2017
• Issued: 08/13/2019
• Est. Priority Date: 09/29/2017
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: He, Shawn, DeVico, Matthew, Kahn, Mohammed
• Primary Examiner(s): Ge, Jin
• Application Number:
  US15/721,660
• Publication Number:
  US 20190102939A1
• Time in Patent Office: 683 Days
• Field of Search: 345419
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/573 using trajectories of game objects, e.g. of a golf ball according to the point of impact
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof

Abstract

A method is provided, including the following operations: receiving at least one video feed from at least one camera disposed in a venue; processing the at least one video feed to generate a video stream that provides a view of the venue; transmitting the video stream over a network to a client device, for rendering to a head-mounted display; wherein processing the at least one video feed includes, analyzing the at least one video feed to identify a projectile object that is launched in the venue, wherein in the video stream, the projectile object is replaced with a virtual object, the virtual object being animated in the video stream so as to exhibit a path of travel that is towards the head-mounted display as the video stream is rendered to the head-mounted display.

Claims

1. A method, comprising:

receiving at least one video feed from at least one camera disposed in a venue;

processing the at least one video feed to generate a video stream that provides a view of the venue;

transmitting the video stream over a network to a client device, for rendering to a head-mounted display;

wherein processing the at least one video feed includes,

analyzing the at least one video feed to identify a projectile object that is launched in the venue,

responsive to identifying the projectile object and identifying a virtual location of a user of the head-mounted display in the venue, then replacing the projectile object in the video stream with a virtual object, using the identified virtual location of the user to adjust a path of travel of the virtual object toward the virtual location of the user of the head-mounted display in the venue, and animating the virtual object in the video stream so as to exhibit the path of travel to be towards the user of the head-mounted display as the video stream is rendered to the head-mounted display.

2. The method of claim 1, wherein the replacement of the projectile object with the virtual object and the animation of the virtual object are in response to a detected gesture by the user of the head-mounted display.

3. The method of claim 2, wherein the gesture is defined by raising of a hand of the user.

4. The method of claim 1, wherein analyzing the at least one video feed further includes tracking movement of the projectile object.

5. The method of claim 4, wherein analyzing the at least one video feed further includes using the tracked movement to determine a predicted landing location of the projectile object in the venue.

6. The method of claim 5, wherein the replacement of the projectile object with the virtual object and the animation of the virtual object are in response to the predicted landing location being within a predefined distance of the virtual location of the user of the head-mounted display.

7. The method of claim 6, wherein the view of the venue that is provided by the video stream is from a perspective that is substantially defined by the virtual location of the user of the head-mounted display.

8. The method of claim 7, wherein the virtual location of the user of the head-mounted display is defined by a seat location in the venue.

9. The method of claim 1, wherein the replacement of the projectile object with the virtual object occurs along a trajectory of the projectile object.

10. The method of claim 1, wherein the at least one video feed includes two or more video feeds, and wherein processing the at least one video feed includes stitching the two or more video feeds together.

21. The method of claim 1, wherein replacing the projectile object with the virtual object includes processing the at least one video feed to remove the projectile object in the at least one video feed.

11. A non-transitory computer readable medium having program instructions embodied thereon that, when executed by at least on computing device, cause said at least one computing device to perform a method including the following operations:

receiving at least one video feed from at least one camera disposed in a venue;

processing the at least one video feed to generate a video stream that provides a view of the venue;

transmitting the video stream over a network to a client device, for rendering to a head-mounted display;

wherein processing the at least one video feed includes,

analyzing the at least one video feed to identify a projectile object that is launched in the venue,

responsive to identifying the projectile object and identifying a virtual location of a user of the head-mounted display in the venue, then replacing the projectile object in the video stream with a virtual object, using the identified virtual location of the user to adjust a path of travel of the virtual object toward the virtual location of the user of the head-mounted display in the venue, and animating the virtual object in the video stream so as to exhibit the path of travel to be towards the user of the head-mounted display as the video stream is rendered to the head-mounted display.

12. The non-transitory computer readable medium of claim 11, wherein the replacement of the projectile object with the virtual object and the animation of the virtual object are in response to a detected gesture by the user of the head-mounted display.

13. The non-transitory computer readable medium of claim 12, wherein the gesture is defined by raising of a hand of the user.

14. The non-transitory computer readable medium of claim 11, wherein analyzing the at least one video feed further includes tracking movement of the projectile object.

15. The non-transitory computer readable medium of claim 14, wherein analyzing the at least one video feed further includes using the tracked movement to determine a predicted landing location of the projectile object in the venue.

16. The non-transitory computer readable medium of claim 15, wherein the replacement of the projectile object with the virtual object and the animation of the virtual object are in response to the predicted landing location being within a predefined distance of the virtual location of the user of the head-mounted display.

17. The non-transitory computer readable medium of claim 16, wherein the view of the venue that is provided by the video stream is from a perspective that is substantially defined by the virtual location of the user of the head-mounted display.

18. The non-transitory computer readable medium of claim 17, wherein the virtual location of the user of the head-mounted display is defined by a seat location in the venue.

19. The non-transitory computer readable medium of claim 11, wherein the replacement of the projectile object with the virtual object occurs along a trajectory of the projectile object.

20. The non-transitory computer readable medium of claim 11, wherein the at least one video feed includes two or more video feeds, and wherein processing the at least one video feed includes stitching the two or more video feeds together.