Constraint based information inference

• US 9,035,970 B2
• Filed: 06/29/2012
• Issued: 05/19/2015
• Est. Priority Date: 06/29/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lamb, Mathew J., Sugden, Ben J., Crocco, Robert L. Jr., Keane, Brian E., Miles, Christopher E., Perez, Kathryn Stone, Massey, Laura K., Kipman, Alex Aben-Athar
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US13/538,848
• Publication Number:
  US 20140002496A1
• Time in Patent Office: 1,054 Days
• Field of Search: 345/633
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 2380/14 Electronic books and readers
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

A system for identifying an AR tag and determining a location for a virtual object within an augmented reality environment corresponding with the AR tag is described. In some environments, the location of a virtual object corresponding with a particular AR tag may be determined by identifying a predefined object, determining an orientation and a scale of the predefined object relative to a head-mounted display device (HMD) based on a model of the predefined object, and inferring the location of the virtual object based on the orientation and the scale of the predefined object. In some cases, an identification of the particular AR tag corresponding with the virtual object may be acquired by aggregating and analyzing individual identity determinations from a plurality of HMDs within an augmented reality environment.

Claims

1. A method for generating and displaying one or more virtual objects, comprising:

identifying a particular object within an environment using a first mobile device, the particular object comprises one of an augmentable book, an augmentable magazine, or an augmentable piece of literature;

acquiring a 3D model of the particular object at the first mobile device, the particular object is associated with one or more tags;

acquiring an identification of a first tag of the one or more tags from a second mobile device different from the first mobile device, the second mobile device comprises a second head-mounted display device, the first tag is viewable within a field of view of the second mobile device, the first mobile device comprises a first head-mounted display device, the first tag is not viewable within a field of view of the first mobile device;

identifying a virtual object corresponding with the first tag based on the identification of the first tag;

acquiring a first location relative to the 3D model from the second mobile device, the acquiring a first location relative to the 3D model from the second mobile device includes acquiring the first location in response to the second mobile device detecting that the particular object has been opened to a particular page that includes the first tag for at least a threshold period of time; and

displaying at the first mobile device the virtual object such that the virtual object is perceived to exist at a point in space corresponding with the first location.

2. The method of claim 1, wherein:

the acquiring an identification of a first tag includes aggregating a plurality of individual identity determinations from a plurality of mobile devices within the environment, the plurality of mobile devices includes the second mobile device, an individual identity determination corresponding with the second mobile device comprises the identification of the first tag.

3. The method of claim 1, wherein:

the acquiring a first location includes acquiring the first location in response to the second mobile device detecting that a phrase associated with the particular object has been spoken.

4. The method of claim 1, further comprising:

determining an orientation and a scale of the particular object relative to the first mobile device based on the 3D model, the determining an orientation and a scale of the particular object includes determining the orientation of the particular object within the environment relative to the first mobile device and determining a size of the particular object within the environment.

5. The method of claim 1, further comprising:

capturing a first set of images of the environment using the first mobile device, the identifying a particular object includes identifying the particular object based on the first set of images.

6. The method of claim 1, wherein:

the virtual object comprises a holographic visual aid.

7. The method of claim 1, wherein:

the first tag is associated with a particular page of the particular object; and

the virtual object comprises a holographic visual aid associated with the particular page.

8. The method of claim 1, wherein:

the 3D model includes a plurality of plates connected to a single axis hinge.

9. The method of claim 1, wherein:

the 3D model includes a plurality of pages connected to a single axis hinge.

10. One or more storage devices containing processor readable code for programming one or more processors to perform a method for generating and displaying one or more virtual objects comprising:

identifying a predefined object within an environment, the predefined object is associated with an AR tag, the identifying is performed by a first mobile device, the predefined object comprises one of an augmentable book, an augmentable magazine, or an augmentable piece of literature;

acquiring a 3D model of the predefined object;

acquiring an identification of the AR tag, the acquiring an identification includes aggregating a plurality of individual identity determinations from a plurality of mobile devices within the environment, the plurality of mobile devices includes a second mobile device different from the first mobile device, an individual identity determination corresponding with the second mobile device comprises the identification of the AR tag, the second mobile device comprises a second head-mounted display device, the AR tag is viewable within a field of view of the second mobile device, the first mobile device comprises a first head-mounted display device, the AR tag is not viewable within a field of view of the first mobile device;

determining a virtual object corresponding with the AR tag based on the identification of the AR tag;

acquiring a first location relative to the 3D model from the second mobile device, the acquiring a first location relative to the 3D model from the second mobile device includes acquiring the first location in response to the second mobile device detecting that the predefined object has been opened to a articular page that includes the AR to for at least a threshold period of time; and

displaying at the first mobile device the virtual object such that the virtual object is perceived to exist at a point in space corresponding with the first location.

11. The one or more storage devices of claim 10, wherein:

the acquiring a first location includes acquiring the first location in response to the second mobile device detecting that a phrase associated with the predefined object has been spoken.

12. The one or more storage devices of claim 10, wherein:

the virtual object comprises a holographic visual aid associated with the reading object.

13. The one or more storage devices of claim 10, wherein:

the AR tag is associated with a particular page of the predefined object; and

the virtual object comprises a holographic visual aid associated with the particular page.

14. The one or more storage devices of claim 10, wherein:

the 3D model includes a plurality of plates connected to a single axis hinge.

15. The one or more storage devices of claim 10, wherein:

the 3D model includes a plurality of pages connected to a single axis hinge.

16. An electronic device for generating and displaying one or more virtual objects, comprising:

a memory, the memory stores a 3D model of a particular object, the particular object is associated with one or more tags, the particular object comprises one of an augmentable book, an augmentable magazine, or an augmentable piece of literature;

one or more processors in communication with the memory, the one or more processors identify the particular object within an environment, the one or more processors acquire an identification of a first tag of the one or more tags from a second mobile device different from the electronic device, the second mobile device comprises a second head-mounted display device, the first tag is viewable within a field of view of the second mobile device, the first mobile device comprises a first head-mounted display device, the first tag is not viewable within a field of view of the first mobile device, the one or more processors determine a virtual object corresponding with the first tag based on the identification of the first tag, the one or more processors acquire a first location relative to the 3D model from the second mobile device, the one or more processors acquire the first location relative to the 3D model from the second mobile device in response to the second mobile device detecting that the particular object has been opened to a particular page that includes the first tag for at least a threshold period of time; and

a see-through display in communication with the one or more processors, the see-through display displays one or more images of the virtual object such that the virtual object is perceived to exist at a point in space corresponding with the first location.

17. The electronic device of claim 16, wherein:

the one or more processors acquire the identification of the first tag by aggregating a plurality of individual identity determinations from a plurality of mobile devices within the environment, the plurality of mobile devices includes the second mobile device, an individual identity determination corresponding with the second mobile device comprises the identification of the first tag.

18. The electronic device of claim 16, wherein:

the virtual object comprises a holographic visual aid.