DRIVING A PROJECTOR TO GENERATE A SHARED SPATIAL AUGMENTED REALITY EXPERIENCE

• US 20160098862A1
• Filed: 04/09/2015
• Published: 04/07/2016
• Est. Priority Date: 10/07/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wilson, Andrew D., Benko, Hrvoje
• Granted Patent: US 10,297,082 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: H04N 9/3147 Multi-projection systems displays in general H04N9/12; video walls G06F3/1446, G09G2300/026
• CPC Class Codes: H04N 9/3179 Video signal processing therefor
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14

Abstract

Various technologies pertaining to shared spatial augmented reality (SSAR) are described. Sensor units in a room output sensor signals that are indicative of positions of two or more users in the room and gaze directions of the two or more users. Views of at least one virtual object are computed separately for each of the two or more users, and projectors project such views in the room. The projected views cause the two or more users to simultaneously perceive the virtual object in space.

Claims

1. A computing system comprising:

at least one processor; and

memory that comprises a plurality of components that are executed by the at least one processor, the plurality of components comprising;

a tracker component that is configured to compute a first location of a first user and a second location of a second user in a room, the tracker component computes the first location and the second location based upon at least one signal generated by at least one sensor device; and

a render component that is configured to;

compute a first view based upon the first location of the first user, the first view to be presented to the first user, the first view comprises a virtual object;

compute a second view based upon the second location of the second user, the second view to be presented to the second user, the second view comprises the virtual object; and

transmit at least one frame to at least one projector that causes the at least one projector to project imagery onto projection surfaces in the room, the imagery includes at least one of;

at least a portion of the first view;

or

at least a portion of the second view, such that both the first user and the second user perceive the virtual object in space between the first user and the second user.

2. The computing system of claim 1, the tracker component is further configured to compute a first gaze direction of the first user and a second gaze direction of the second user based upon the at least one signal generated by the sensor device, the render component is further configured to compute the first view based upon the first gaze direction and the second view based upon the second gaze direction.

3. The computing system of claim 1, the memory further comprises a computer-implemented three-dimensional geometric model of the room, the render component is further configured to compute the first view and the second view based upon the computer-implemented three-dimensional geometric model of the room.

4. The computing system of claim 3, the render component is further configured to update the computer-implemented three-dimensional geometric model in real-time based upon the at least one signal generated by the at least one sensor device.

5. The computing system of claim 1, the at least one frame comprises a first frame and a second frame, the at least one projector comprises a first projector and a second projector, the render component is configured to transmit the frame to the first projector and to transmit the second frame to the second projector, the first frame causes the first projector to project the first view, the second frame causes the second projector to project the second view.

6. The computing system of claim 5, the at least one frame further comprises a third frame, the at least one projector further comprises a third projector, the render component is configured to transmit the third frame to the third projector, the third frame causes the third projector to project a portion of the first view and a portion of the second view.

7. The computing system of claim 1, the tracker component is configured to detect a gesture performed by the first user relative to the virtual object, and the render component is further configured to compute the first view and the second view based upon the gesture detected by the tracker component.

8. The computing system of claim 1, the tracker component is configured to receive a word spoken by the first user, and the render component is further configured to compute the first view and the second view based upon the word received by the tracker component.

9. The computing system of claim 1, the virtual object being a three-dimensional object.

10. The computing system of claim 1, the at least one signal comprises at least one of a red-green-blue image or a depth image.

11. A method comprising:

computing a first gaze direction of a first user in a room based upon at least one sensor signal generated by at least one sensor unit;

computing a second gaze direction of a second user in the room based upon the at least one sensor signal generated by the at least one sensor unit; and

transmitting at least one frame to at least one projector in the room based upon the first gaze direction of the first user and the second gaze direction of the second user, the at least one frame causes the at least one projector to project imagery on surfaces in the room such that the first user and the second user perceive a virtual object in space between the first user and the second user.

12. The method of claim 11, further comprising:

computing a first view for the first user based upon the first gaze direction of the first user and a computer-implemented geometric model of the room;

computing a second view for the second user based upon the second gaze direction of the second user and the computer-implemented geometric model of the room; and

generating the at least one frame based upon the first view for the first user and the second view for the second user.

13. The method of claim 12, further comprising:

updating the geometric model in real-time based upon the at least one sensor signal generated by the at least one sensor unit.

14. The method of claim 13, wherein updating the geometric model in real-time comprises updating the geometric model based upon detected positions of the first user and the second user in the room.

15. The method of claim 11, further comprising:

estimating at least one of;

reflectivity of a surface in the room based upon the at least one sensor signal generated by the at least one sensor unit;

or

color properties of the surface in the room based upon the at least one sensor signal generated by the at least one sensor unit; and

generating the at least one frame based upon the at least one of the reflectivity of the surface or the color properties of the surface.

16. The method of claim 11, further comprising:

detecting a gesture performed by one of the first user or the second user with respect to the virtual object; and

generating the frame based upon the gesture.

17. The method of claim 16, the imagery causes the first user and the second user to perceive that the virtual object is moving in the space between the first user and the second user.

18. The method of claim 11, the at least one frame comprises a first frame and a second frame, and wherein transmitting the at least one frame comprises:

transmitting the first frame to a first projector that causes the first projector to project imagery for a first eye of the first user; and

transmitting the second frame to a second projector that causes the second projector to project imagery for a second eye of the first user.

19. The method of claim 11, further comprising continuously performing the acts of computing the first gaze direction, computing the second gaze direction, and transmitting the at least one frame to the at least one projector in the room such that the imagery is updated in real-time as a function of alteration of at least one of the first gaze direction or the second gaze direction.

20. A computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform acts comprising:

detecting motion of at least one of a first user or a second user in a room, the first user and the second user facing one another in the room;

computing a first view for the first user based upon a computer-implemented geometric model of the room and the motion of the at least one of the first user or the second user in the room, the first view comprises a virtual object;

computing a second view for the second user based upon the computer-implemented geometric model of the room and the motion of the at least one of the first user or the second user in the room, the second view comprises the virtual object; and

transmitting at least one frame to at least one projector, the at least one frame causes the at least one projector to project at least one of;

at least a portion of the first view in the room;

or

at least a portion of the second view in the room, such that the first user and the second user perceive that the virtual object is moving in space between the first user and the second user.