Head-mounted display that displays a visual representation of physical interaction with an input interface located outside of the field of view

• US 8,217,856 B1
• Filed: 07/27/2011
• Issued: 07/10/2012
• Est. Priority Date: 07/27/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Petrou, David
• Primary Examiner(s): Patel, Nitin
• Assistant Examiner(s): Edwards, Carolyn R
• Application Number:
  US13/191,873
• Time in Patent Office: 349 Days
• Field of Search: 345 7- 9
• US Class Current: 345/8
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface

Abstract

A head-mounted display (HMD) displays a visual representation of a physical interaction with an input interface that is located outside of the field of view. In one embodiment, the visual representation includes symbols that indicate when close proximity or physical contact is made with the input interface. In another embodiment, the visual representation is a simulation of the physical interaction with the input interface. The visual representation displayed by the HMD can beneficially enable the wearer to interact with the input interface more efficiently.

Claims

1. A head-mounted display (HMD), comprising:

a see-through display, wherein the see-though display is configured to provide a field of view of a real-world environment and to display generated images;

a touch pad interface mounted on the HMD in relation to the see-through display such that the touch pad interface is located outside of the field of view; and

a controller, wherein the controller is configured to;

(a) receive, at least in part from a camera directed toward the touch pad interface, a record of a physical interaction outside of the field of view between a manipulator and the touch pad interface;

(b) generate, based at least in part on the record, a visual representation of the physical interaction between the manipulator and the touch pad interface; and

(c) cause the see-through display to show, inside of the field of view, the visual representation of the physical interaction between the manipulator and the touch pad interface.

2. The HMD of claim 1, wherein the controller is configured to cause the see-through display to display the visual representation of the physical interaction between the manipulator and the touch pad interface as the physical interaction occurs.

3. The HMD of claim 1, wherein the HMD further comprises the camera.

4. The HMD of claim 3, wherein the record of the physical interaction comprises a visual record captured by the camera, and wherein at least part of the visual representation comprises at least part of the visual record.

5. The HMD of claim 4, wherein the visual record comprises one of:

a still image, a video, or both a still image and a video.

6. The HMD of claim 1, wherein the touch pad interface further comprises a keypad.

7. The HMD of claim 1, wherein the see-through display comprises at least one lens that defines the field of view, and wherein the field of view includes a portion of the real-world environment in front of the at least one lens.

8. The HMD of claim 7, further comprising at least one side portion that extends back from the at least one lens.

9. The HMD of claim 8, wherein the touch pad interface is configured to be mounted on the at least one side portion.

10. The HMD of claim 1, wherein the physical interaction between the manipulator and the touch pad interface comprises the manipulator touching the touch pad interface.

11. The HMD of claim 10, wherein the visual representation of the physical interaction between the manipulator and the touch pad interface comprises a first graphic representing the manipulator and a second graphic representing the touch pad interface.

12. The HMD of claim 11, wherein the manipulator comprises a hand of a wearer of the HMD.

13. The HMD of claim 12, wherein the first graphic comprises a realistic representation with at least one of a wearer'"'"'s skin tone, a wearer'"'"'s hand shape, and a wearer'"'"'s hand size.

14. The HMD of claim 13, wherein the controller is configured to determine at least one of the wearer'"'"'s skin tone, the wearer'"'"'s hand shape, and the wearer'"'"'s hand size based on an image of the wearer'"'"'s hand.

15. The HMD of claim 11, wherein the visual representation of the physical interaction between the manipulator and the touch pad interface shows the first graphic moving relative to the second graphic to indicate movement of the manipulator relative to the touch pad interface during the physical interaction.

16. The HMD of claim 15, wherein the movement of first graphic relative to the second graphic shown in the visual representation provides an indication of when input data is received by the data interface.

17. The HMD of claim 16, wherein the movement of first graphic relative to the second graphic shown in the visual representation provides an indication of when the manipulator makes physical contact with the touch pad interface.

18. The system of claim 11, wherein the first graphic comprises one or more colored dots overlaying the second graphic at positions that indicate where the manipulator has made physical contact with the touch pad interface.

19. A method performed in a head-mounted display (HMD), wherein the HMD includes a see-through display configured to provide a field of view of a real-world environment and to display generated images, the method comprising:

(a) receiving, at least in part from a camera directed toward the touch pad interface, a record of a physical interaction between the manipulator and a touch pad interface, wherein the touch pad interface is mounted on the HMD in relation to the see-through display such that the physical interaction is located outside of the field of view;

(b) generating, based at least in part on the record, a visual representation of the physical interaction between the manipulator and the touch pad interface; and

(c) displaying in real-time on the see-through display, inside of the field of view, the visual representation of the physical interaction between the manipulator and the touch pad interface.

20. A wearable computing device, comprising:

a see-through display, wherein the see-though display is configured to provide a field of view of a real-world environment and to display generated images;

a touch pad interface mounted on the wearable computing device in relation to the see-through display such that the touch pad interface is located outside of the field of view;

at least one processor;

a non-transitory computer-readable medium; and

program instructions stored in the non-transitory computer-readable medium and executable by the at least one processor to;

(a) receive, at least in part from a camera directed toward the touch pad interface, a record of a physical interaction outside the field of view between a manipulator and the touch pad interface;

(b) generate, based at least in part on the record, a visual representation of the physical interaction between the manipulator and the touch pad interface; and

(c) cause the see-through display to show, inside of the field of view, the visual representation of the physical interaction between the manipulator and the touch pad interface.