Driving a projector to generate a shared spatial augmented reality experience

• US 10,297,082 B2
• Filed: 04/09/2015
• Issued: 05/21/2019
• Est. Priority Date: 10/07/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wilson, Andrew D., Benko, Hrvoje
• Primary Examiner(s): Sun, Hai Tao
• Application Number:
  US14/682,546
• Publication Number:
  US 20160098862A1
• Time in Patent Office: 1,503 Days
• Field of Search: 345419
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: H04N 9/3147 Multi-projection systems displays in general H04N9/12; video walls G06F3/1446, G09G2300/026
• CPC Class Codes: H04N 9/3179 Video signal processing therefor
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14

Abstract

Various technologies pertaining to shared spatial augmented reality (SSAR) are described. Sensor units in a room output sensor signals that are indicative of positions of two or more users in the room and gaze directions of the two or more users. Views of at least one virtual object are computed separately for each of the two or more users, and projectors project such views in the room. The projected views cause the two or more users to simultaneously perceive the virtual object in space.

Claims

1. A computing system comprising:

at least one processor; and

memory that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform acts comprising;

computing a first location of a first user and a second location of a second user in a room, the first location and the second location computed based upon at least one signal generated by at least one sensor device;

computing a first view of a virtual object based upon the first location of the first user, the first view of the virtual object to be presented to the first user, the first view of the virtual object is to be projected onto the second user for viewing by the first user;

computing a second view of the virtual object based upon the second location of the second user, the second view of the virtual object is different from the first view of the virtual object, the second view of the virtual object is to be projected onto the first user for viewing by the second user; and

transmitting at least one frame to at least one projector that causes the at least one projector to simultaneously project the first view of the virtual object onto the second user in the room and the second view of the virtual object onto the first user in the room, wherein the virtual object appears in space, to the first user and the second user, between the first user and the second user due to the first view of the virtual object and the second view of the virtual object being simultaneously projected onto the second user and the first user, respectively.

2. The computing system of claim 1, the acts further comprising computing a first gaze direction of the first user and a second gaze direction of the second user based upon the at least one signal generated by the sensor device, wherein the first view of the virtual object is computed based upon the first gaze direction and the second view of the virtual object is computed based upon the second gaze direction.

3. The computing system of claim 1, the memory further comprises a computer-implemented three-dimensional geometric model of the room, the first view of the virtual object and the second view of the virtual object computed based upon the computer-implemented three-dimensional geometric model of the room.

4. The computing system of claim 3, the acts further comprising updating the computer-implemented three-dimensional geometric model in real-time based upon the at least one signal generated by the at least one sensor device.

5. The computing system of claim 1, the at least one frame comprises a first frame and a second frame, the at least one projector comprises a first projector and a second projector, the acts further comprising:

transmitting the first frame to the first projector; and

transmitting the second frame to the second projector, the first frame causes the first projector to project the first view of the virtual object, the second frame causes the second projector to project the second view of the virtual object.

6. The computing system of claim 5, the at least one frame further comprises a third frame, the at least one projector further comprises a third projector, the acts further comprising transmitting the third frame to the third projector, the third frame causes the third projector to project a portion of the first view of the virtual object and a portion of the second view of the virtual object.

7. The computing system of claim 1, the acts further comprising:

detecting a gesture performed by the first user relative to the virtual object; and

computing the first view of the virtual object and the second view of the virtual object based upon the detected gesture.

8. The computing system of claim 1, the acts further comprising:

receiving a word spoken by the first user; and

computing the first view of the virtual object and the second view of the virtual object based upon the received word.

9. The computing system of claim 1, the virtual object being a three-dimensional object.

10. The computing system of claim 1, the at least one signal comprises at least one of a red-green-blue image or a depth image.

20. The computing system of claim 1, wherein the first view of the virtual object and the second view of the virtual object are computed in real-time based upon a computer-implemented geometry of the room being updated, and further wherein the computer-implemented geometry of the room is updated in real-time as the first user and the second user move about in the room.

11. A method comprising:

computing a first gaze direction of a first user in a room based upon at least one sensor signal generated by at least one sensor unit;

computing a second gaze direction of a second user in the room based upon the at least one sensor signal generated by the at least one sensor unit;

computing a first view of a virtual object for viewing by the first user based upon the first gaze direction of the first user and a computer-implemented geometric model of the room;

computing a second view of the virtual object for the second user based upon the second gaze direction of the second user and the computer-implemented geometric model of the room, wherein the first view of the virtual object and the second view of the virtual object are different; and

transmitting at least one frame to at least one projector in the room based upon the first view of the virtual object for the first user and the second view of the virtual object for the second user, the at least one frame causes the at least one projector to project the first view of the virtual object onto the second user in the room such that the first view of the virtual object is presented to the first user and to project the second view of the virtual object onto the first user in the room such that the second view of the virtual object is presented to the second user, and further wherein simultaneous presentment of first view of the virtual object to the first user and the second view of the virtual object to the second user causes the virtual object to appear in space, to the first user and the second user, between the first user and the second user.

12. The method of claim 11, further comprising:

updating the geometric model in real-time based upon the at least one sensor signal generated by the at least one sensor unit.

13. The method of claim 12, wherein updating the geometric model in real-time comprises updating the geometric model based upon detected positions of the first user and the second user in the room.

14. The method of claim 11, further comprising:

estimating at least one of;

reflectivity of a surface in the room based upon the at least one sensor signal generated by the at least one sensor unit;

or

color properties of the surface in the room based upon the at least one sensor signal generated by the at least one sensor unit; and

generating the at least one frame based upon the at least one of the reflectivity of the surface or the color properties of the surface.

15. The method of claim 11, further comprising:

detecting a gesture performed by one of the first user or the second user with respect to the virtual object; and

generating the at least one frame based upon the gesture.

16. The method of claim 15, wherein simultaneous presentment of the first view of the virtual object to the first user and the second view of the virtual object to the second user causes the first user and the second user to perceive that the virtual object is moving in the space between the first user and the second user.

17. The method of claim 11, the at least one frame comprises a first frame and a second frame, and wherein transmitting the at least one frame comprises:

transmitting the first frame to a first projector that causes the first projector to project the first view of the virtual object for the first user; and

transmitting the second frame to a second projector that causes the second projector to project the second view of the virtual object for the second user.

18. The method of claim 11, further comprising continuously performing the acts of computing the first gaze direction, computing the second gaze direction, computing the first view of the virtual object, computing the second view of the virtual object, and transmitting the at least one frame to the at least one projector in the room such that at least one of the first view of the virtual object or the second view of the virtual object is updated in real-time as a function of alteration of at least one of the first gaze direction or the second gaze direction.

19. A computing system comprising:

a processor; and

memory storing instructions that, when executed by a processor, cause the processor to perform acts comprising;

detecting motion of at least one of a first user or a second user in a room, the first user and the second user facing one another in the room;

updating a computer-implemented geometric model of the room based upon the detected motion of the at least one of the first user or the second user in the room;

responsive to updating the computer-implemented geometric model of the room, computing a first view of a virtual object for presentment to the first user, the first view of the virtual object computed based upon the computer-implemented geometric model of the room;

responsive to updating the computer-implemented geometric model of the room, computing a second view of the virtual object for presentment to the second user, the second view of the virtual object computed based upon the computer-implemented geometric model of the room; and

transmitting at least one frame to at least one projector, the at least one frame causes the at least one projector to simultaneously project a portion of the first view of the virtual object in the room on the second user and at least a portion of the second view of the virtual object in the room on the first user, wherein the virtual object appears, to the first user and the second user, to be moving in space between the first user and the second user.