Graphic user interface schemes for supporting speech recognition input systems

• US 7,742,923 B2
• Filed: 09/24/2004
• Issued: 06/22/2010
• Est. Priority Date: 09/24/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Bickel, Ryan Terry, Chambers, Robert L., Mowatt, David, Murillo, Oscar E., Scholz, Oliver
• Primary Examiner(s): Opsasnick; Michael N
• Application Number:
  US10/950,020
• Publication Number:
  US 20060074687A1
• Time in Patent Office: 2,097 Days
• Field of Search: 704/231, 704/251, 704/270, 704/270.1, 704/275, 704/256.2
• US Class Current: 704/275
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance

Abstract

A numbering scheme is disclosed for implementation in the context of an application display. A user is able to select an item on the display by speaking a number corresponding to a desired control item. In some cases, the screen can include so many numbers that the user loses context and is unable to identify which number they want to select. For this reason, in one embodiment, a temporal switching mechanism is implemented wherein periodic switches (e.g., second-long intervals) occur between showing numbered items and showing a non-numbered screen. In one embodiment, an optional secondary confirmation step is implemented wherein the user sees only the item they just selected and has the chance to (a) learn the programmatic name of the item they selected and/or (b) either confirm and proceed with their selection, or cancel. In one embodiment, the optional secondary confirmation step is omitted if the user speaks a number followed by a predetermined command word.

Claims

1. A computer-implemented method of facilitating interaction with a user by managing an output display device associated with a computing device, the method comprising:

receiving a request from the user for interaction assistance;

responding to the request by layering a control indication scheme over an application display, wherein the application display is a graphical user interface visibly rendered on the output display device;

wherein layering the control indication scheme comprises visibly rendering, on the output display device, a bounding box around each of a plurality of control items exposed as part of the application display;

wherein layering also comprises rendering, on the output display device, a visible identifier positioned within each bounding box; and

wherein said bounding boxes and visible identifiers are layered over the application display in that they are simultaneously rendered on the output display device with the application display but are not part of the application display itself;

receiving a spoken indication from the user;

identifying a particular one of the plurality of control items that corresponds to the spoken indication; and

altering the appearance of the control indication scheme as rendered on the output display device so as to visually emphasize the particular control item that corresponds to the spoken indication.

2. The method of claim 1, wherein receiving a request comprises receiving a spoken request.

3. The method of claim 1, wherein receiving a request comprises receiving a request communicated through interaction with a physical input device.

4. The method of claim 1, wherein receiving a request comprises recognizing an event sequence.

5. The method of claim 1, wherein layering further comprises rendering said bounding boxes by drawing, on the output display device, a box around each of the plurality of control items.

6. The method of claim 1, wherein layering further comprises rendering said visible identifiers by visibly numbering, on the output display device, the plurality of control items so as to be in a consecutive numerical sequence from a top left portion of the application display to a bottom right portion.

7. The method of claim 1, wherein layering further comprises rendering said visible identifiers by visibly numbering, on the output display device, the plurality of control items with a consistent font.

8. The method of claim 1, wherein layering further comprises rendering said visible identifiers by visibly numbering, on the output display device, the plurality of control items so as to be in a consecutive numerical order from left to right.

9. The method of claim 1, wherein layering further comprises rendering, on the output display device, said visible identifiers by placing a visible indicator over each of the plurality of control items.

10. The method of claim 1, wherein layering further comprises rendering said visible identifiers by placing multiple instances of a same number proximate a single one of the plurality of control items.

11. The method of claim 1, wherein layering further comprises rendering, on the output display device, attention-drawing lines to attract visual attention to a number that is rendered, on the output display device, proximate one of the plurality of control items.

12. The method of claim 1, further comprising:

receiving a spoken predetermined command word immediately after the spoken indication; and

performing a function that is consistent with the particular control item that corresponds to the spoken indication.

13. The method of claim 1, further comprising switching the output display device between a display that does and does not include the control indication scheme to emphasize correlation to a set of underlying graphical representations.

14. The method of claim 13, further comprising altering the appearance of the control indication scheme on the output display device so as to emphasize the particular control item that corresponds to the spoken indication.

15. The method of claim 1, further comprising rendering, on the output display device, a name associated with the particular control item that corresponds to the spoken indication.

16. The method of claim 1, further comprising:

receiving a spoken cancel command; and

re-rendering, on the output display device, the control indication scheme as originally rendered without the alterations made to visually emphasize the particular control item that corresponds to the spoken indication.

17. The method of claim 1, further comprising receiving and responding to a spoken cancel command.

18. The method of claim 1, further comprising:

receiving a confirmation command; and

performing a function that is consistent with the particular control item that corresponds to the spoken indication.

19. The method of claim 1, further comprising:

waiting a predetermined amount of time;

periodically rendering on the output display device so as to switch between a display that includes the control indication scheme and one that does not.

20. A computer-implemented method of facilitating interaction between a user and a computing device, the method comprising:

receiving, at a processor functionally associated with the computing device, a representation of a spoken indicator received from the user, wherein the spoken indicator corresponds to a visual representation of a control item, the visual representation being rendered on an output display device functionally associated with the computing device, and wherein the visual representation is a particular one of a plurality of consecutively numbered numerical identifiers rendered on the output display device, and wherein the spoken indicator is a spoken instance of the particular numerical identifier;

receiving, at the processor, a representation of a spoken predetermined command word received from the user after the spoken indicator;

utilizing the processor to process the spoken indicator and command word so as to identify a corresponding function that is consistent with the control item that corresponds to the visual representation; and

utilizing the processor to perform the function that is consistent with the control item that corresponds to the visual representation.

21. A computer-implemented method of facilitating interaction between a user and a computing device, the method comprising:

receiving, at a processor functionally associated with the computing device, an indication of a spoken indicator received from the user, wherein the spoken indicator corresponds to a representation of a control item that is rendered on a display functionally associated with the computing device;

altering a control indication scheme rendered on the display so as to visually emphasize the representation of the control item; and

responding to receipt of the indication of the spoken indicator by initiating a display, on the display associated with the computing device, of a name associated with the representation of the control item.

22. The method of claim 21, further comprising:

receiving, at the processor, an indication of a spoken cancel command received from the user; and

responding to the received indication of a spoken cancel command by altering a control indication scheme to visually emphasize a plurality of representations of control items rendered on the display.

23. The method of claim 21, further comprising:

receiving, at the processor, an indication of a spoken confirmation command received from the user; and

responding to the received indication of a spoken confirmation command by performing a function that is consistent with the representation of the control item.