Methods and devices for rendering interactions between virtual and physical objects on a substantially transparent display

• US 9,784,971 B2
• Filed: 02/19/2015
• Issued: 10/10/2017
• Est. Priority Date: 10/05/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Wong, Adrian, Miao, Xiaoyu
• Primary Examiner(s): HONG, STEPHEN S
• Assistant Examiner(s): Robinson, Marshon
• Application Number:
  US14/626,294
• Publication Number:
  US 20150177518A1
• Time in Patent Office: 964 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Disclosed are methods and devices for rendering interactions between virtual and physical objects on a substantially transparent display are disclosed. In one embodiment, the method includes displaying a user-interface on a substantially transparent display of a wearable computing device. The method further includes displaying a virtual object in the view region at a focal length along a first line of sight and detecting a physical object at a physical distance along a second line of sight. The method still further includes determining that a relationship between the focal length and the physical distance is such that the virtual object and the physical object appear substantially co-located in a user-view through the view region and, responsive to the determination, initiating a collision action between the virtual object and the physical object.

Claims

1. A computer-implemented method comprising:

displaying a user-interface on a substantially transparent display of a wearable computing device, wherein the user-interface comprises a view region;

displaying a virtual object in the view region at a focal length along a first of sight;

determining a physical distance, along the first line of sight, to a physical object;

determining that the focal length is substantially equal to the physical distance to the physical object, such that the virtual object and the physical object appear substantially co-located, in a user-view through the view region, at the physical distance; and

responsive to the determination, initiating a collision action between the virtual object and the physical object, wherein the collision action comprises a graphical modification of the virtual object.

2. The method of claim 1, wherein determining the physical distance to the physical object comprises:

emitting a series of pulsed signals;

detecting reflections of at least some of the pulsed signals; and

based on the reflections, detecting the physical object.

3. The method of claim 1, wherein determining the physical distance to the physical object comprises:

emitting optical beams forming a three-dimensional lattice;

detecting disturbances in the lattice; and

based on the disturbances, detecting the physical object.

4. The method of claim 1, wherein determining the physical distance to the physical object comprises:

capturing a first image of the physical object; and

based on the first image, determining the physical distance to the physical object.

5. The method of claim 4, further comprising:

capturing a second image of the physical object, wherein the first image is captured at a first time and the second image is captured at a second time; and

based on the first image, the second image, the first time, and the second time, determining a velocity of the physical object.

6. The method of claim 1, further comprising:

using at least one sensor to detect a velocity of the physical object.

7. The method of claim 1, wherein the graphical modification of the virtual object is based on a velocity of the physical object.

8. The method of claim 7, wherein:

displaying the virtual object comprises displaying the virtual object as moving with a velocity;

wherein the graphical modification of the virtual object is further based on the velocity of the virtual object.

9. The method of claim 1, wherein the graphical modification of the virtual object comprises at least one of the virtual object moving away from the physical object and the virtual object changing shape.

10. The method of claim 1, wherein:

displaying the virtual object comprises displaying the virtual object as moving in a first direction;

wherein the graphical modification of the virtual object comprises moving the virtual object in a second direction that differs from the first direction.

11. The method of claim 1, wherein the graphical modification of the virtual object comprises moving the virtual object out of the view region.

12. The method of claim 1, further comprising:

in response to the virtual object moving away from the physical object, displaying the virtual object at a second focal length from the wearable computing device along a second line of sight;

displaying an additional virtual object at a third focal length from the wearable computing device along a third line of sight;

determining that a relationship between the second focal length and the third focal length is such that the virtual object and the additional virtual object appear substantially co-located in the user-view through the view region; and

responsive to the determination, initiating a collision action between the virtual object and the additional virtual object.

13. The method of claim 1, wherein the user-interface further comprises a content region that is located outside of the view region.

14. The method of claim 13, wherein the user-interface is initially configured such that the at least one content region is substantially arranged along an at least partial ring above the view region and wherein the at least partial ring is substantially centered above the wearable computing device.

15. A wearable computing device comprising:

a substantially transparent display;

at least one processor; and

data storage comprising logic executable by the at least one processor to;

display a user-interface on the substantially transparent display, wherein the user-interface comprises a view region;

determining a physical distance, along the first line of sight, to a physical object;

determine that the focal length is substantially equal to the physical distance to the physical object, such that the virtual object and the physical object appear substantially co-located, in a user-view through the view region, at the physical distance; and

responsive to the determination, initiate a collision action between the virtual object and the physical object, wherein the collision action comprises a graphical modification of the virtual object.

16. The wearable computing device of claim 15, further comprising:

an emitter configured to emit a series of pulsed signals; and

a detector configured to detect reflections of at least some of the pulsed signals, wherein the logic is further executable by the at least one processor to detect the physical object based on the reflections.

17. The wearable computing device of claim 15, further comprising:

at least one laser configured to emit a three-dimensional lattice; and

at least one detector configured to detect disturbances in the lattice, wherein the logic is further executable by the at least one processor to detect the physical object based on the disturbances.

18. The wearable computing device of claim 15, wherein the graphical modification of the virtual object comprises at least one of the virtual object moving away from the physical object and the virtual object changing shape.

19. A non-transitory computer-readable medium having stored therein instructions executable by a computing device to cause the computing device to perform functions comprising:

displaying a user-interface on a substantially transparent display of a wearable computing device, wherein the user-interface comprises a view region;

displaying a virtual object in the view region at a focal length along a first line of sight;

determining a physical distance, along the first line of sight, to a physical object;

determining that the focal length is substantially equal to the physical distance to the physical object, such that the virtual object and the physical object appear substantially co-located, in a user-view through the view region, at the physical distance; and

responsive to the determination, initiating a collision action between the virtual object and the physical object, wherein the collision action comprises a graphical modification of the virtual object.

20. The non-transitory computer-readable medium of claim 19, wherein the user-interface further comprises a content region that is located outside of the view region, and wherein the user-interface is initially configured such that the at least one content region is arranged along an at least partial ring above the view region and wherein the at least partial ring is substantially centered above the wearable computing device.