Mechanism to give holographic objects saliency in multiple spaces

• US 10,643,389 B2
• Filed: 03/28/2016
• Issued: 05/05/2020
• Est. Priority Date: 06/29/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lamb, Mathew J., Sugden, Ben J., Crocco, Jr., Robert L., Keane, Brian E., Miles, Christopher E., Perez, Kathryn Stone, Massey, Laura K., Kipman, Alex Aben-Athar
• Primary Examiner(s): Le, Michael
• Application Number:
  US15/083,229
• Publication Number:
  US 20160210789A1
• Time in Patent Office: 1,499 Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/1431 using a single graphics controller
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 2380/14 Electronic books and readers
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: H04N 2213/008 Aspects relating to glasses for viewing stereoscopic images

Abstract

A system for allowing a virtual object to interact with other virtual objects across different spaces within an augmented reality (AR) environment and to transition between the different spaces is described. An AR environment may include a plurality of spaces, each comprising a bounded area or volume within the AR environment. In one example, an AR environment may be associated with a three-dimensional world space and a two-dimensional object space corresponding with a page of a book within the AR environment. A virtual object within the AR environment may be assigned to the object space and transition from the two-dimensional object space to the three-dimensional world space upon the detection of a space transition event. In some cases, a dual representation of the virtual object may be used to detect interactions between the virtual object and other virtual objects in both the world space and the object space.

Claims

1. A method for generating and displaying one or more virtual objects, comprising:

identifying an environment including a two-dimensional space and a three-dimensional space;

acquiring a virtual object

assigning at least a portion of the virtual object to a two-dimensional space associated with a real object;

displaying a first set of images corresponding with the at least a portion of the virtual object such that the at least a portion of the virtual object is perceived to exist within the two-dimensional space as a two-dimensional object;

detecting a space transition event for the at least a portion of the virtual object from the two-dimensional space to a three-dimensional world space, the detecting a space transition event includes detecting an interaction between the at least a portion of the virtual object existing as a two-dimensional object within the two-dimensional space and a different virtual object existing within the three-dimensional world space;

assigning the at least a portion of the virtual object to the three-dimensional world space in response to detecting the interaction between the at least a portion of the virtual object within the two-dimensional space and the different virtual object existing within the three-dimensional world space; and

displaying a second set of images corresponding with the at least a portion of the virtual object such that the at least a portion of the virtual object is perceived to exist within the three-dimensional world space as a three-dimensional object.

2. The method of claim 1, wherein:

the two-dimensional space corresponds with a surface of a page of the real object; and

the displaying the first set of images includes displaying using a mobile device the first set of images such that the at least a portion of the virtual object is perceived to be attached to the surface of the page.

3. The method of claim 1, further comprising:

determining a location for the at least a portion of the virtual object within the three-dimensional world space subsequent to assigning the at least a portion of the virtual object to the three-dimensional world space; and

the displaying the second set of images includes displaying the second set of images such that the at least a portion of the virtual object is perceived to exist within the three-dimensional world space at the location.

4. The method of claim 3, further comprising:

acquiring a wind speed and a wind direction associated with a physical location of a mobile device; and

the determining a location for the at least a portion of the virtual object includes determining the location for the at least a portion of the virtual object within the three-dimensional world space based on the wind speed and wind direction; and

the displaying the second set of images includes displaying the second set of images using the mobile device.

5. The method of claim 4, wherein:

the detecting a space transition event is performed by the mobile device, the mobile device comprises a see-through head-mounted display device.

6. The method of claim 1, wherein:

the detecting an interaction includes generating a dual representation for the virtual object, the dual representation comprises a first location for the virtual object relative to the two-dimensional space and a second location for the virtual object relative to the three-dimensional world space.

7. The method of claim 1, wherein:

the detecting a space transition event includes detecting that the at least a portion of the virtual object has reached a boundary of the two-dimensional space due to the interaction between the at least a portion of the virtual object and the different virtual object.

8. The method of claim 1, wherein:

the real object comprises an augmentable book and the two-dimensional space associated with the real object comprises a bounded area attached to a page of the augmentable book.

9. An augmented reality system, comprising:

a display; and

one or more processors in communication with the display, the one or more processors configured to identify an environment including a two-dimensional space and a three-dimensional space, the one or more processors configured to acquire a virtual object associated with a real object the one or more processors configured to assign at least a portion of the virtual object to a two-dimensional space associated with the real object and cause a first set of images corresponding with the at least a portion of the virtual object to be displayed using the display such that the at least a portion of the virtual object is perceived to exist within the two-dimensional space as a two-dimensional object, the one or more processors configured to detect an interaction between the at least a portion of the virtual object existing as a two-dimensional object within the two-dimensional space and a different virtual object existing within a three-dimensional world space, the one or more processors configured to assign the at least a portion of the virtual object to the three-dimensional world space in response to detecting the interaction between the at least a portion of the virtual object within the two-dimensional space and the different virtual object existing within the three-dimensional world space, the one or more processors configured to cause a second set of images corresponding with the at least a portion of the virtual object to be displayed using the display such that the at least a portion of the virtual object is perceived to exist within the three-dimensional world space as a three-dimensional object.

10. The augmented reality system of claim 9, wherein:

the one or more processors configured to determine a location for the at least a portion of the virtual object within the three-dimensional world space and cause the second set of images corresponding with the at least a portion of the virtual object to be displayed using the display such that the at least a portion of the virtual object is perceived to exist within the three-dimensional world space at the location.

11. The augmented reality system of claim 9, wherein:

the display is part of a mobile device; and

the one or more processors configured to acquire a wind speed and a wind direction associated with a physical location of the mobile device and determine the location for the at least a portion of the virtual object within the three-dimensional world space based on the wind speed and wind direction.

12. The augmented reality system of claim 11, wherein:

the mobile device comprises a see-through head-mounted display device.

13. The augmented reality system of claim 9, wherein:

the one or more processors configured to generate a dual representation for the virtual object, the dual representation comprises a first location for the virtual object relative to the two-dimensional space and a second location for the virtual object relative to the three-dimensional world space.

14. The augmented reality system of claim 9, wherein:

the one or processors configured to detect that the at least a portion of the virtual object has reached a boundary of the two-dimensional space due to the interaction between the at least a portion of the virtual object and the different virtual object.

15. The augmented reality system of claim 9, wherein:

the real object comprises an augmentable book and the two-dimensional space associated with the real object comprises a bounded area attached to a page of the augmentable book.

16. The augmented reality system of claim 9, wherein:

the virtual object comprises a holographic visual aid associated with the real object.

17. One or more storage devices containing processor readable code for programming one or more processors to perform a method for generating and displaying one or more virtual objects comprising the steps of:

identifying an environment including a two-dimensional space and a three-dimensional space;

acquiring a virtual object associated with a real object;

assigning the at least a portion of the virtual object to a two-dimensional space associated with the real object;

displaying a first set of images corresponding with the at least a portion of the virtual object such that the at least a portion of the virtual object is perceived to exist within the two-dimensional space as a two-dimensional object;

detecting an interaction between the at least a portion of the virtual object existing as a two-dimensional object within the two-dimensional space and a different virtual object existing within a three-dimensional world space;

assigning the at least a portion of the virtual object to the three-dimensional world space in response to detecting the interaction between the at least a portion of the virtual object within the two-dimensional space and the different virtual object existing within the three-dimensional world space; and

displaying a second set of images corresponding with the at least a portion of the virtual object such that the at least a portion of the virtual object is perceived to exist within the three-dimensional world space as a three-dimensional object.

18. The one or more storage devices of claim 17, further comprising:

determining a location for the at least a portion of the virtual object within the three-dimensional world space; and

the displaying a second set of images includes displaying using a mobile device the second set of images corresponding with the at least a portion of the virtual object such that the at least a portion of the virtual object is perceived to exist within the three-dimensional world space at the location.

19. The one or more storage devices of claim 18, further comprising:

acquiring a wind speed and a wind direction associated with a physical location of the mobile device; and

the determining a location for the at least a portion of the virtual object includes determining the location for the at least a portion of the virtual object within the three-dimensional world space based on the wind speed and wind direction.

20. The one or more storage devices of claim 17, further comprising:

the real object comprises an augmentable book and the two-dimensional space associated with the reading object comprises a bounded area attached to a page of the augmentable book.

21. A method for generating and displaying one or more virtual objects, comprising:

acquiring a virtual object, the virtual object comprises a first portion of the virtual object and a second portion of the virtual object;

assigning the second portion of the virtual object to a two-dimensional space associated with a reading object;

displaying a first set of images corresponding with the second portion of the virtual object such that the second portion of the virtual object is perceived to exist within the two-dimensional space as a two-dimensional object;

detecting a space transition event for the second portion of the virtual object from the two-dimensional space to a three-dimensional world space, the detecting a space transition event includes detecting an interaction between the second portion of the virtual object existing as a two-dimensional object within the two-dimensional space and a different virtual object existing within the three-dimensional world space;

assigning the second portion of the virtual object to the three-dimensional world space in response to detecting the interaction between the second portion of the virtual object within the two-dimensional space and the different virtual object existing within the three-dimensional world space; and

displaying a second set of images corresponding with the second portion of the virtual object such that the second portion of the virtual object is perceived to exist within the three-dimensional world space as a three-dimensional object.

22. The method of claim 21, wherein:

the two-dimensional space corresponds with a surface of a page of the reading object; and

the displaying the first set of images includes displaying using a mobile device the first set of images such that the second portion of the virtual object is perceived to be attached to the surface of the page.

23. The method of claim 21, further comprising:

determining a location for the second portion of the virtual object within the three-dimensional world space subsequent to assigning the second portion of the virtual object to the three-dimensional world space; and

the displaying the second set of images includes displaying the second set of images such that the second portion of the virtual object is perceived to exist within the three-dimensional world space at the location.

24. The method of claim 23, further comprising:

acquiring a wind speed and a wind direction associated with a physical location of a mobile device; and

the determining a location for the second portion of the virtual object includes determining the location for the second portion of the virtual object within the three-dimensional world space based on the wind speed and wind direction; and

the displaying the second set of images includes displaying the second set of images using the mobile device.

25. The method of claim 24, wherein:

the detecting a space transition event is performed by the mobile device, the mobile device comprises a see-through head-mounted display device.

26. The method of claim 21, wherein:

the detecting an interaction includes generating a dual representation for the virtual object, the dual representation comprises a first location for the virtual object relative to the two-dimensional space and a second location for the virtual object relative to the three-dimensional world space.

27. The method of claim 21, wherein:

the detecting a space transition event includes detecting that the second portion of the virtual object has reached a boundary of the two-dimensional space due to the interaction between the second portion of the virtual object and the different virtual object.

28. The method of claim 21, wherein:

the reading object comprises an augmentable book and the two-dimensional space associated with the reading object comprises a bounded area attached to a page of the augmentable book.

29. An augmented reality system, comprising:

a display; and

one or more processors in communication with the display, the one or more processors configured to acquire a virtual object associated with a reading object, the virtual object comprises a first portion of the virtual object and a second portion of the virtual object, the one or more processors configured to assign the second portion of the virtual object to a two-dimensional space associated with the reading object and cause a first set of images corresponding with the second portion of the virtual object to be displayed using the display such that the second portion of the virtual object is perceived to exist within the two-dimensional space as a two-dimensional object, the one or more processors configured to detect an interaction between the second portion of the virtual object existing as a two-dimensional object within the two-dimensional space and a different virtual object existing within a three-dimensional world space, the one or more processors configured to assign the second portion of the virtual object to the three-dimensional world space in response to detecting the interaction between the second portion of the virtual object within the two-dimensional space and the different virtual object existing within the three-dimensional world space, the one or more processors configured to cause a second set of images corresponding with the second portion of the virtual object to be displayed using the display such that the second portion of the virtual object is perceived to exist within the three-dimensional world space as a three-dimensional object.

30. The augmented reality system of claim 29, wherein:

the one or more processors configured to determine a location for the second portion of the virtual object within the three-dimensional world space and cause the second set of images corresponding with the second portion of the virtual object to be displayed using the display such that the second portion of the virtual object is perceived to exist within the three-dimensional world space at the location.

31. The augmented reality system of claim 29, wherein:

the display is part of a mobile device; and

the one or more processors configured to acquire a wind speed and a wind direction associated with a physical location of the mobile device and determine the location for the second portion of the virtual object within the three-dimensional world space based on the wind speed and wind direction.

32. The augmented reality system of claim 31, wherein:

the mobile device comprises a see-through head-mounted display device.

33. The augmented reality system of claim 29, wherein:

the one or more processors configured to generate a dual representation for the virtual object, the dual representation comprises a first location for the virtual object relative to the two-dimensional space and a second location for the virtual object relative to the three-dimensional world space.

34. The augmented reality system of claim 29, wherein:

the one or processors configured to detect that the second portion of the virtual object has reached a boundary of the two-dimensional space due to the interaction between the second portion of the virtual object and the different virtual object.

35. The augmented reality system of claim 29, wherein:

the reading object comprises an augmentable book and the two-dimensional space associated with the reading object comprises a bounded area attached to a page of the augmentable book.

36. The augmented reality system of claim 29, wherein:

the virtual object comprises a holographic visual aid associated with the reading object.

37. One or more storage devices containing processor readable code for programming one or more processors to perform a method for generating and displaying one or more virtual objects comprising the steps of:

acquiring a virtual object associated with a reading object, the virtual object comprises a first portion of the virtual object and a second portion of the virtual object;

assigning the second portion of the virtual object to a two-dimensional space associated with the reading object;

displaying a first set of images corresponding with the second portion of the virtual object such that the second portion of the virtual object is perceived to exist within the two-dimensional space as a two-dimensional object;

detecting an interaction between the second portion of the virtual object existing as a two-dimensional object within the two-dimensional space and a different virtual object existing within a three-dimensional world space;

assigning the second portion of the virtual object to the three-dimensional world space in response to detecting the interaction between the second portion of the virtual object within the two-dimensional space and the different virtual object existing within the three-dimensional world space; and

displaying a second set of images corresponding with the second portion of the virtual object such that the second portion of the virtual object is perceived to exist within the three-dimensional world space as a three-dimensional object.

38. The one or more storage devices of claim 37, further comprising:

determining a location for the second portion of the virtual object within the three-dimensional world space; and

the displaying a second set of images includes displaying using a mobile device the second set of images corresponding with the second portion of the virtual object such that the second portion of the virtual object is perceived to exist within the three-dimensional world space at the location.

39. The one or more storage devices of claim 38, further comprising:

acquiring a wind speed and a wind direction associated with a physical location of the mobile device; and

the determining a location for the second portion of the virtual object includes determining the location for the second portion of the virtual object within the three-dimensional world space based on the wind speed and wind direction.

40. The one or more storage devices of claim 37, further comprising:

the reading object comprises an augmentable book and the two-dimensional space associated with the reading object comprises a bounded area attached to a page of the augmentable book.