Voice assisted visual search

• US 20110138286A1
• Filed: 08/07/2010
• Published: 06/09/2011
• Est. Priority Date: 08/07/2009
• Status: Abandoned Application

• Current Assignee: Elena Oleinik, Viktor Kaptelinin
• Original Assignee: Elena Oleinik, Viktor Kaptelinin
• Inventors: Kaptelinin, Viktor, Oleinik, Elena
• Application Number:
  US12/852,469
• Publication Number:
  US 20110138286A1
• Time in Patent Office: Days
• US Class Current: 715/728
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

The invention discloses a method and apparatus for (a) processing a voice input from the user of computer technology, (b) recognizing potential objects of interest, and (c) using electronic displays to present visual artefacts directing user'"'"'s attention to the spatial locations of the objects of interest. The voice input is matched with attributes of the information objects, which are visually presented to the viewer. If one or several objects match the voice input sufficiently, the system visually marks or highlights the object or objects to help the viewers direct his or her attention to the matching object or objects. The sets of visual objects and their attributes, used in the matching, may be different for different user tasks and types of visually displayed information. If the user views only a portion of a document and user'"'"'s voice input matches an information object, which is contained in the entire document but not displayed in the current portion, the system displays a visual artefact, which indicates the direction and distance to the object.

Claims

1. A method for assisting a user of a computer system, comprised of at least one electronic display, a user voice input device, and a computer processor with a memory storage, in viewing a plurality of visual objects, the method comprising the method steps of

creating in computer memory a representation of a plurality of visual objects;

and

displaying said plurality of visual objects to the user; and

detecting and processing a voice input from a user; and

establishing, whether an information in the voice input matches one or several representations of visual objects comprising said plurality of visual objects; and

displaying visual artifacts highlighting spatial locations of visual object or visual objects, which match the information in the voice input,

whereby highlighting of said matching visual object or visual objects assists the user in carrying our visual search of visual objects of interest.

2. A method of claim 1, wherein both the plurality of displayed visual objects and the highlighting visual artefacts are displayed on a same electronic display.

3. A method of claim 1, wherein the plurality of displayed visual objects represents a plurality of physical objects observed by the user, and the highlighting visual artefacts are displayed by overlaying said visual artifacts on a visual image of said plurality of displayed visual objects using a head up display.

4. A method of claim 2, wherein the user can set preferences, including at least:

(a) selecting categories of objects used in matching and subsequent highlighting, (b) selecting a set of languages used in matching, (c) selecting types of specific attributes of highlighting visual artifacts, (d) switching voice assisted highlighting on or off, (e) choosing whether or not the highlighted objects are also selected, for subsequent graphical user interface commends, and (f) choosing strict or relaxed criteria for considering an object as matching the voice input.

5. A method of claim 2, wherein language translation means are provided for matching a same representation of a plurality of visual objects to user'"'"'s voice input expressed in a plurality of languages.

6. A method of claim 2, wherein a highlighted visual object is also selected as a potential object of a graphical user interface command.

7. A method of claim 1, wherein a memory representation of a displayed visual object includes a description of visual objects, which can be accessed through operating upon said displayed visual object.

8. A method of claim 1, wherein users are differentiated by their voice attributes, and attributes of the highlighting visual artefacts are individually adjusted to individual users.

9. A method of claim 8, wherein adjusting to individual users employs machine learning algorithms.

10. A method of claim 8, wherein several users, who are using the system generally simultaneously, are provided with different highlighting visual clues.

11. A method of claim 1, wherein only a portion of the plurality of visual objects is displayed to the user and if the voice input matches an object that is not displayed in the portion, then displaying a visual artefact pointing in the direction, in which the display should needs to be moved in order to make the matching object to be displayed to the user.

12. A method of claim 11, wherein the length of the pointing visual artefact is proportional to the distance for which the display needs to be moved in order to make the matching object to be displayed to the user.

13. A method of claim 11, wherein a pointing visual artifact can also be operated by the user to cause the display move to display the matching object.

14. Apparatus, comprising at least an electronic display;

and

a user voice input device; and

a computer processor, and a memory storage, which can be integrated with said computer processor; and

means for creating in computer memory a representation of a plurality of visual objects; and

means for displaying said plurality of visual objects to the user; and

means for detecting and processing a voice input from a user; and

means for establishing, whether an information in the voice input matches one or several representations of visual objects comprising said plurality of visual objects; and

means for displaying visual artifacts highlighting spatial locations of visual object or visual objects, which match the information in the voice input;

whereby highlighting of said matching visual object or visual objects assists the user in carrying our visual search of visual objects of interest.

15. An apparatus of claim 14, further comprising

means for displaying a portion of said plurality of visual objects to the user;

and

means for establishing, whether the voice input matches at least one visual object selected from said plurality of visual objects, said at least selected object not displayed to the user; and

means for displaying a visual artefact pointing in the direction, in which a display needs to be moved to cause said at least selected object to be displayed to the user.

16. An apparatus of claim 14, wherein both the plurality of displayed visual objects and the highlighting visual artefacts are displayed on a same electronic display.

17. An apparatus of claim 14, wherein the plurality of displayed visual objects represents a plurality of physical objects observed by the user, and the highlighting visual artefacts are displayed by overlaying said visual artifacts on a visual image of said plurality of displayed visual objects using a head up display.