VIRTUAL SPECTATOR EXPERIENCE WITH A PERSONAL AUDIO/VISUAL APPARATUS

• US 20130083173A1
• Filed: 06/29/2012
• Published: 04/04/2013
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Robert L. Crocco Jr
• Inventors: Geisner, Kevin A., Perez, Kathryn Stone, Latta, Stephen G., Sugden, Ben J., Vaught, Benjamin I., Kipman, Alex Aben-Athar, Scavezze, Michael J., McCulloch, Daniel J., Bennett, Darren, Scott, Jason, Hastings, Ryan L., Crocco, Robert L. Jr., Lamb, Mathew J., Miles, Christopher E., Keane, Brian E.
• Granted Patent: US 9,268,406 B2
• Time in Patent Office: Days
• US Class Current: 348/51
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 21/2143 located in a single building, e.g. hotel, hospital or museum arrangements specially adapted for plural spots in a confined site in broadcast systems H04H20/63; adaptations for transmission by electric cable for domestic distribution in television systems H04N7/106
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/8146 involving graphical data, e.g. 3D object, 2D graphics
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Technology is described for providing a virtual spectator experience for a user of a personal A/V apparatus including a near-eye, augmented reality (AR) display. A position volume of an event object participating in an event in a first 3D coordinate system for a first location is received and mapped to a second position volume in a second 3D coordinate system at a second location remote from where the event is occurring. A display field of view of the near-eye AR display at the second location is determined, and real-time 3D virtual data representing the one or more event objects which are positioned within the display field of view are displayed in the near-eye AR display. A user may select a viewing position from which to view the event. Additionally, virtual data of a second user may be displayed at a position relative to a first user.

Claims

1. A method for providing a virtual spectator experience of an event for viewing with a near-eye, augmented reality display of a personal audiovisual (A/V) apparatus comprising:

receiving in real time one or more positions of one or more event objects participating in the event occurring at a first location remote from a second location;

mapping the one or more positions of the one or more event objects in the first 3D coordinate system for the first location to a second 3D coordinate system for a second location remote from the first location;

determining a display field of view of a near-eye, augmented reality display of a personal A/V apparatus being worn by a user at the second location; and

sending in real time 3D virtual data representing the one or more event objects which are within the display field of view to the personal A/V apparatus at the second location.

2. The method of claim 1 wherein the near-eye, augmented reality display is a near-eye, see-through, augmented reality display.

3. The method of claim 1 further comprising receiving in real time 3D virtual data of the one or more event objects which include real objects over a communication network from one or more computer systems which generated the 3D virtual data based on image and depth data captured of the one or more real event objects.

4. The method of claim 1 wherein the event is an animation of a past event and the method further comprises:

receiving in real time 3D virtual data of the one or more event objects including virtual event objects for the animation over a communication network from one or more computer systems which generated the event objects.

5. The method of claim 1 wherein the second location and the first location are at least one of stadiums or theaters.

6. The method of claim 1 further comprising:

receiving a request for a user selected viewing position of the event;

generating 3D video data of the second location from a perspective of the user selected viewing position of the event including 3D virtual data of the one or more event objects participating in the event occurring at the first location.

7. The method of claim 1 further comprising:

receiving a request for a user selected viewing position of the event;

generating 3D video data of the first location from a perspective of the user selected viewing position of the event occurring in real time at the first location.

8. An apparatus for providing a virtual spectator experience of an event for viewing with a near-eye, augmented reality display of a personal audiovisual (A/V) apparatus comprising:

a first set of one or more processors having access to one or more memories storing a first three-dimensional (3D) coordinate system for a first location where an event is occurring in a first event space and storing a second (3D) coordinate system for a second location having a second event space for hosting a same type of event as the event occurring at the first location;

the first set of one or more processors being communicatively coupled to a second set of one or more processors for receiving in real time 3D virtual data representing one or more event objects participating in the event at the first location and one or more positions of the one or more objects in the first 3D coordinate system for the first location;

the first set of one or more processors mapping the 3D virtual data representing one or more objects participating in the event at the first location to one or more positions in the second 3D coordinate system for the second location; and

the first set of one or more processors being communicatively coupled to the personal A/V apparatus including the near-eye, augmented reality display located at the second location, and the first set of one or more processors sending the 3D virtual data representing one or more event objects participating in the event with their respective one or more positions in the second 3D coordinate system for the second location to the personal A/V apparatus.

13. The apparatus of claim 8 further comprising:

a plurality of capture devices positioned throughout the first location to capture the data of the objects in the event from multiple perspectives at the first location;

the second set of one or more processors having a second set of one or more memories and being communicatively coupled to the plurality of capture devices for receiving image data and depth data of the one or more objects in the event at the first location; and

the second set of one or more processors generating the 3D virtual data representing the one or more objects participating in the event based on the received image and depth data of the one or more objects.

14. The apparatus of claim 8 further comprising:

the second set of one or more processors further being in communication with position sensors of the one or more event objects at the first location for mapping the one or more positions of the one or more event objects in the 3D coordinate system for the first location.

15. The apparatus of claim 8 further comprising:

the first set of one or more processors identifying a set of viewing positions for at least one of the locations for which 3D video of its respective view of the event is available; and

responsive to receiving a request from the personal A/V apparatus for 3D video for a user selected viewing position from the set, the first set of one or more processors sending the requested 3D video for the selected viewing position to the personal A/V apparatus.

16. The apparatus of claim 15 further comprising:

the first set of one or more processors receiving 3D video data for different viewing positions of the event at the first location;

the first set of one or more processors determining a starting position indicator in the 3D video data for the user selected viewing position for display by the near-eye, augmented reality display based on a head position and orientation for the user indicated in inertial data in the sensor data received from the personal A/V apparatus and a gaze direction for the user indicated in gaze data received from the personal A/V apparatus;

the first set of one or more processors selecting 3D video data for the user selected viewing position from the received 3D video data based on the starting position indicator and sending the selected 3D video data to the personal A/V apparatus;

the first set of one or more processors continuously tracking the head position and orientation as indicated in the sensor data continuously received from the personal A/V apparatus, updating the selected 3D video data in accordance with any tracked head position and orientation changes based on the starting position indicator for the user selected viewing position, and sending updates of the selected 3D video data to the personal A/V apparatus.

17. The apparatus of claim 15 wherein the set of viewing positions includes at least one of the following:

a viewing position of a spectator of the event at the first location;

or

a viewing position of a participating object in the event at the first location.

19. The one or more processor readable storage devices of claim 16 wherein the method further comprises:

receiving user input by the first personal A/V apparatus from the first user indicating a position relative to the first user at which to display 3D virtual data representing a second user;

transforming the 3D virtual data representing the second user and a current action being performed by the second user at the position relative to the first user; and

displaying the 3D virtual data representing the second user and the current action being performed by the second user in the near-eye, augmented reality display responsive to the position relative to the first user being within a field of view of the near-eye, augmented reality display.

10. The apparatus of claim 9 further comprising:

the 3D coordinate system for the first location and the 3D coordinate system for the second location are view independent coordinate systems;

the first set of one or more processors determining a display field of view for the near-eye, augmented reality display of the personal A/V apparatus in the second 3D coordinate system based on sensor data received from the personal A/V apparatus; and

the first set of one or more processors transforming the 3D virtual data representing the one or more event objects participating in the event from the view independent 3D coordinate system for second location to a view dependent coordinate system for the near-eye, augmented reality display of the personal A/V apparatus and sending the transformed 3D virtual data to the personal A/V apparatus.

11. The apparatus of claim 10 wherein the sensor data received from the personal A/V apparatus includes image and depth data from at least one capture device positioned on the personal A/V apparatus.

12. The apparatus of claim 10 further comprising:

the first set of one or more processors generating and storing in the one or more memories virtual overlay data representing boundary data indicating boundaries of the 3D coordinate system for the first location with respect to boundaries for the 3D coordinate system for the second location; and

responsive to a request from the personal A/V apparatus requesting display of the virtual overlay data when the boundary data is in the field of view of the near-eye, augmented reality display, including the virtual overlay data in the transformed 3D virtual data responsive to boundary data being in the field of view of the display.

18. One or more processor readable storage devices comprising instructions which cause one or more processors to execute a method for providing a virtual spectator experience of an event using a near-eye, augmented reality display of a first personal A/V apparatus, the method comprising:

receiving user input selecting a viewing position at a first location where the event is occurring; and

requesting and receiving three-dimensional (3D) virtual data of the event including 3D virtual data for the viewing position at the first location over a communication network from one or more computer systems which generate the 3D virtual data of the event for the viewing position based on received image data and depth data from capture devices at the first location;

determining a head position and orientation of a first user wearing the near-eye, augmented reality display;

selecting 3D virtual data as current 3D virtual data for display in the near-eye, augmented reality display based on the head position and orientation; and

displaying the current 3D virtual data in the near-eye, augmented reality display.

20. The one or more processor readable storage devices of claim 18 wherein the first location is at least one of the following:

the location at which the event is occurring;

or

a location remote from the event.