Multimode Input Field for a Head-Mounted Display

• US 20120317484A1
• Filed: 06/21/2012
• Published: 12/13/2012
• Est. Priority Date: 06/09/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Gomez, Luis Ricardo Prada, Wheeler, Aaron
• Granted Patent: US 8,519,909 B2
• Time in Patent Office: Days
• US Class Current: 715/716
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 2203/0339 Touch strips, e.g. orthogonal touch strips to control cursor movement or scrolling; single touch strip to adjust parameter or to implement a row of soft keys
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G09G 2370/20 Details of the management of multiple sources of image data
• CPC Class Codes: G09G 2380/02 Flexible displays

Abstract

Exemplary methods and systems relate to a multimode input field for a head-mountable display (HMD). An exemplary multimode input field is operable to display content based on a number of modalities of input data, which may be received from a number of different input sources. Further, the input sources and/or content for the multimode input field may be selected based on implicit information in input data from the input sources, explicit instructions, and/or context information.

Claims

1. A non-transitory computer-readable medium having program instructions stored thereon that are executable by at least one processor, the program instructions comprising:

instructions for causing a graphical display of a head-mountable display (HMD) to display a multimode input field, wherein the multimode input field is a single graphical interface element that is controllable to display content based on a plurality of modalities of input data received from any of a plurality of input sources, and wherein the plurality of modalities comprises point-of-view video and at least one of audio, image, text, and video;

instructions for receiving input data from the plurality of input sources;

instructions for monitoring the received input data for one or more data patterns indicated by input-selection criteria, wherein the one or more data patterns comprise;

(a) at least one data pattern that corresponds to implicit information that is indicative of desired content for the multimode input field and (b) at least one data pattern corresponding to an explicit input-content instruction that indicates desired content for the multimode input field;

instructions for selecting at least one input source from the plurality of input sources in response to detection of a data pattern in the received input data, wherein the detected data pattern matches a corresponding one of the data patterns indicated by the input-selection criteria; and

instructions for displaying content in the multimode input field, wherein the displayed content is based on input data received from the at least one selected input source.

2. The non-transitory computer-readable medium of claim 1, wherein the program instructions further comprise:

instructions for determining one or more context signals; and

instructions for using the one or more context signals as a further basis to select the at least one input source.

3. The non-transitory computer-readable medium of claim 1, wherein the program instructions further comprise:

instructions for receiving data comprising a drag-and-drop instruction to drag content from an application window to the multimode input field; and

instructions for, in response to receipt of the drag-and-drop instruction, causing the graphic display to display the content indicated by the drag-and-drop instruction in the multimode input field.

4. The non-transitory computer-readable medium of claim 1, wherein the program instructions further comprise:

instructions for receiving data comprising an input-content instruction, wherein the input-content instruction indicates particular content for the multimode input field, and wherein the input-content instruction comprises at least one of;

(a) a drag-and-drop instruction to drag content from an application window to the multimode input field, (b) data corresponding to a gesture on a touchpad interface, and (c) input data comprising a spoken command captured in audio data; and

instructions for, in response to receipt of the input-content instruction;

(i) disabling the monitoring of the received input data for the one or more data patterns indicated by the input-selection; and

(ii) causing the graphic display to display the content indicated by the input-content instruction in the multimode input field.

5. The non-transitory computer-readable medium of claim 1, wherein the program instructions further comprise:

instructions for selecting a video input source as the input source for the multimode input field; and

instructions for causing the graphic display to display video from the video input source in the graphical display of the HMD, wherein the video is displayed such that the multimode input field encloses a portion of the video.

6. The non-transitory computer-readable medium of claim 1, wherein the program instructions further comprise:

instructions for selecting a video input source as the input source for the multimode input field;

instructions for causing the graphic display to display at least a portion of the video from the video input source in the multimode input field;

instructions for receiving a capture instruction and responsively capture a snapshot of the video displayed in the multimode input field;

instructions for initiating an image-based search based on the snapshot of the video; and

instructions for causing the graphic display to display one or more search results of the image-based search.

7. The non-transitory computer-readable medium of claim 1, wherein an image is displayed in the multimode input field, and wherein the program instructions further comprise:

instructions for initiating an image-based search based on the displayed image; and

instructions for causing the graphic display to display one or more search results of the image-based search.

8. The non-transitory computer-readable medium of claim 1, wherein text is displayed in the multimode input field, and wherein the program instructions further comprise:

instructions for initiating a search based on the displayed text; and

instructions for causing the graphic display to display one or more search results of the search.

9. The non-transitory computer-readable medium of claim 1, wherein the program instructions further comprise:

instructions for launching an application in the multimode input field; and

instructions for using the received input data from the at least one selected input source as content to the application.

10. The non-transitory computer-readable medium of claim 1, wherein the program instructions further comprise:

instructions for determining one or more current context signals; and

instructions for comparing the one or more current context signals to historical context data; and

instructions for adjusting functionality of the multimode input field based on the comparison of the one or more current context signals to the historical context data.

11. The non-transitory computer-readable medium of claim 1, wherein the program instructions further comprise:

instructions for receiving a first data instruction to switch to an alternate operation mode in which the size of the multimode input field is adjustable via one or more size-adjustment gestures;

instructions for, while in the alternate operation mode;

receiving one or more size-adjustment gestures, wherein at least one of the size-adjustment gestures comprises a multi-touch gesture on a touchpad; and

resizing the multimode input field according to the received size-adjustment gestures; and

instructions for subsequent to receipt of the first data instruction, receiving a second data instruction indicating to select a portion of the video or image that is enclosed by the multimode input field at or near receipt of the second data instruction.

12. The non-transitory computer-readable medium of claim 11, wherein the program instructions further comprise instructions for initiating an image-based search on the selected portion of the video or image.

13. The non-transitory computer-readable medium of claim 11, wherein the program instructions further comprise instructions for initiating a face-detection function on the selected portion of the video or image.

14. A computer-implemented method comprising:

causing a graphical display of a head-mountable display (HMD) to display a multimode input field, wherein the multimode input field is a single graphical interface element that is controllable to display content based on a plurality of modalities of input data received from any of a plurality of input sources, and wherein the plurality of modalities comprises point-of-view video and at least one of audio, image, text, and video;

receiving input data from the plurality of input sources;

monitoring the received input data for one or more data patterns indicated by input-selection criteria, wherein the one or more data patterns comprise;

(a) at least one data pattern that corresponds to implicit information that is indicative of desired content for the multimode input field and (b) at least one data pattern corresponding to an explicit input-content instruction that indicates desired content for the multimode input field;

selecting at least one input source from the plurality of input sources in response to detection of a data pattern in the received input data, wherein the detected data pattern matches a corresponding one of the data patterns indicated by the input-selection criteria; and

displaying content in the multimode input field, wherein the displayed content is based on input data received from the at least one selected input source.

15. The method of claim 14, further comprising:

determining one or more context signals; and

using the one or more context signals as a further basis to select the at least one input source.

16. The method of claim 14, wherein the input-selection criteria comprises an input selection rule indicating to select a video camera as the input source upon detection of the sequence of frames that is characteristic of fixation on a scene, and wherein the method further comprises:

detecting that a sequence of frames in point-of-view video that is captured by the video camera is characteristic of fixation on a scene by the user; and

in response to detection of the sequence of frames that is characteristic of fixation on a scene, selecting the video camera as the input source for the multimode input field.

17. The method of claim 14, further comprising:

determining that audio data received from a microphone comprises speech data;

responsively using the speech-to-text module to convert the speech data to corresponding text data; and

causing the graphic display to display the corresponding text data in the multimode input field.

18. The method of claim 14, further comprising:

selecting a video input source as the input source for the multimode input field;

causing the graphic display to display at least a portion of the video from the video input source in the multimode input field;

receiving a capture instruction and responsively capture a snapshot of the video displayed in the multimode input field;

initiating an image-based search based on the snapshot of the video; and

causing the graphic display to display one or more search results of the image-based search.

19. The method of claim 14, wherein an image is displayed in the multimode input field, and wherein the method further comprises:

initiating an image-based search based on the displayed image; and

causing the graphic display to display one or more search results of the image-based search.

20. The method of claim 14, wherein text is displayed in the multimode input field, and wherein the method further comprises:

initiating a search based on the displayed text; and

causing the graphic display to display one or more search results of the search.