Wearable computer with superimposed controls and instructions for external device

• US 8,941,560 B2
• Filed: 09/21/2011
• Issued: 01/27/2015
• Est. Priority Date: 09/21/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Wong, Adrian, Miao, Xiaoyu
• Primary Examiner(s): SIMPSON, LIXI CHOW
• Application Number:
  US13/238,557
• Publication Number:
  US 20130069985A1
• Time in Patent Office: 1,224 Days
• Field of Search: 345/7, 345/8
• US Class Current: 345/8
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0189 Sight systems
• CPC Class Codes: G05B 19/124 using tapes, cards or discs with optically sensed marks or codes G05B19/128, G05B19/14 take precedence
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 2200/1613 Supporting arrangements, e.g. for filters or documents associated to a laptop display
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units

Abstract

A wearable computing device includes a head-mounted display (HMD) that provides a field of view in which at least a portion of the environment of the wearable computing device is viewable. The HMD is operable to display images superimposed over the field of view. When the wearable computing device determines that a target device is within its environment, the wearable computing device obtains target device information related to the target device. The target device information may include information that defines a virtual control interface for controlling the target device and an identification of a defined area of the target device on which the virtual control image is to be provided. The wearable computing device controls the HMD to display the virtual control image as an image superimposed over the defined area of the target device in the field of view.

Claims

1. A method comprising:

determining that a controllable target device is within an environment of a wearable computing device comprising a head-mounted display (HMD), wherein the HMD provides a field of view in which at least a portion of the environment is viewable, and wherein the HMD is operable to display images superimposed over the field of view;

obtaining target device information related to the controllable target device, wherein the target device information defines a virtual control interface for controlling the controllable target device and identifies a defined area of the controllable target device on which the virtual control interface is to be provided, wherein the target device information further includes status information related to an operational status of the controllable target device;

controlling the HMD to display the virtual control interface as an image superimposed over the defined area of the controllable target device in the field of view and to display an indication of the operational status of the controllable target device;

recognizing a gesture corresponding to interaction with the virtual control interface superimposed over the defined area of the controllable target device;

recognizing a control instruction for the controllable target device based on the recognized gesture; and

transmitting the recognized control instruction to the controllable target device.

2. The method of claim 1, wherein the wearable computing device further comprises a camera configured to obtain images from a point-of-view.

3. The method of claim 2, wherein the point-of-view of the camera substantially corresponds to the field of view provided by the HMD.

4. The method of claim 2, wherein determining that a controllable target device is within an environment of a wearable computing device comprises:

receiving a point-of-view image obtained by the camera; and

identifying the controllable target device in the point-of-view image.

5. The method of claim 4, wherein the point-of-view image is a video image.

6. The method of claim 1, wherein determining that a controllable target device is within an environment of a wearable computing device comprises:

receiving a beacon from the controllable target device.

7. The method of claim 1, wherein determining that a controllable target device is within an environment of a wearable computing device comprises:

recognizing a radio frequency identification (RFID) tag on the controllable target device.

8. The method of claim 1, wherein determining that a controllable target device is within an environment of a wearable computing device comprises:

recognizing an optical identification on the controllable target device.

9. The method of claim 8, wherein the optical identification is a barcode or a quick response (QR) code.

10. The method of claim 1, wherein determining that a controllable target device is within an environment of a wearable computing device comprises:

the wearable computing device obtaining environment information regarding the environment of the wearable computing device; and

the wearable computing device identifying the controllable target device based on the environment information.

11. The method of claim 1, wherein determining that a controllable target device is within an environment of a wearable computing device comprises:

the wearable computing device obtaining environment information regarding the environment of the wearable computing device;

the wearable computing device transmitting the environment information to a server network; and

the wearable computing device receiving an identification of the controllable target device from the server network.

12. The method of claim 11, wherein the environment information comprises a physical location of the wearable computing device.

13. The method of claim 1, wherein obtaining target device information related to the controllable target device comprises:

the wearable computing device retrieving the target device information from a target device database.

14. The method of claim 1, wherein obtaining target device information related to the controllable target device comprises:

the wearable computing device receiving the target device information from a server network.

15. The method of claim 1, wherein the defined area of the controllable target device is an external surface of the controllable target device.

16. The method of claim 15, further comprising:

determining a perspective appearance of the defined area in the field of view; and

generating the image of the virtual control interface based on the perspective appearance of the defined area in the field of view.

17. The method of claim 15, further comprising:

determining a distance from the HMD to the defined area of the controllable target device; and

generating the image of the virtual control interface based on the distance.

18. The method of claim 1, wherein transmitting the recognized control instruction to the controllable target device comprises the wireless computing device transmitting the recognized control instruction to the controllable target device via a WiFi, Bluetooth, or infrared link.

19. The method of claim 1, wherein transmitting the recognized control instruction to the controllable target device comprises the wireless computing device transmitting the recognized control instruction to a network for transmission to the controllable target device.

20. The method of claim 1, further comprising:

the HMD displaying instructions related to the controllable target device.

21. The method of claim 1, wherein the virtual control interface comprises a plurality of visual elements and each visual element is associated with a respective control instruction for the controllable target device.

22. The method of claim 21, wherein the recognized gesture corresponds to interaction with a particular visual element of the virtual control interface, and wherein the particular visual element is associated with the recognized control instruction.

23. The method of claim 1, further comprising:

receiving the status information from the controllable target device.

24. A non-transitory computer readable medium having stored therein instructions executable by a computing device to cause the computing device to perform functions comprising:

determining that a controllable target device is within a field of view provided by a see-through display controlled by the computing device;

obtaining target device information related to the controllable target device, wherein the target device information defines a virtual control interface for controlling the controllable target device and identifies a defined area of the controllable target device on which the virtual control interface is to be provided, wherein the target device information further includes status information related to an operational status of the controllable target device;

controlling the see-through display to display the virtual control interface as an image superimposed over the defined area of the controllable target device in the field of view and to display an indication of the operational status of the controllable target device;

recognizing a gesture corresponding to interaction with the virtual control interface superimposed over the defined area of the controllable target device;

recognizing a control instruction for the controllable target device based on the recognized gesture; and

transmitting the recognized control instruction to the controllable target device.

25. The non-transitory computer readable medium of claim 24, wherein the virtual control interface comprises a plurality of visual elements and each visual element is associated with a respective control instruction for the controllable target device.

26. The non-transitory computer readable medium of claim 25, wherein the recognized gesture corresponds to interaction with a particular visual element of the virtual control interface, and wherein the particular visual element is associated with the recognized control instruction.

27. The non-transitory computer readable medium of claim 24, wherein the functions further comprise:

receiving the status information from the controllable target device.

28. A wearable computing device, comprising:

a head-mounted display (HMD), wherein the HMD is configured to provide a field of view in which at least a portion of an environment of the wearable computing device is viewable, and wherein the HMD is operable to display images superimposed over the field of view; and

a controller, wherein the controller is configured to;

determine that a controllable target device is within the environment of the wearable computing device;

obtain target device information related to the controllable target device, wherein the target device information defines a virtual control interface for controlling the controllable target device and identifies a defined area of the controllable target device on which the virtual control interface is to be provided, wherein the target device information further includes status information related to an operational status of the controllable target device;

control the HMD to display the virtual control interface as an image superimposed over the defined area of the controllable target device in the field of view and to display an indication of the operational status of the controllable target device;

recognize a gesture corresponding to interaction with the virtual control interface superimposed over the defined area of the controllable target device;

recognize a control instruction for the controllable target device based on the recognized gesture; and

transmit the recognized control instruction to the controllable target device.

29. The wearable computing device of claim 28, further comprising:

a camera configured to image a portion of the environment located in front of the HMD, wherein the controller is configured to identify the controllable target device from one or more images obtained by the camera.

31. The wearable computing device of claim 28, further comprising:

a communication interface, wherein the controller is configured to obtain the target device information by communicating with a server network via the communication interface.

32. The wearable computing device of claim 28, further comprising:

a communication interface, wherein the controller is configured to transmit the recognized control instruction to the controllable target device via the communication interface.

33. The wearable computing device of claim 28, wherein the virtual control interface comprises a plurality of visual elements and each visual element is associated with a respective control instruction for the controllable target device.

34. The wearable computing device of claim 33, wherein the recognized gesture corresponds to interaction with a particular visual element of the virtual control interface, and wherein the particular visual element is associated with the recognized control instruction.

35. The wearable device of claim 28, wherein the controller is further configured to receive the status information from the controllable target device.

30. The wearable computing device of 28, further comprising:

a target device database, wherein the controller is configured to obtain the target device information from the database.