Recognition interfaces for computing devices

• US 9,927,949 B2
• Filed: 12/20/2013
• Issued: 03/27/2018
• Est. Priority Date: 05/09/2013
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Gray, Timothy Thomas, Salazar, Marc Anthony, Sommer, Steven Michael, Cummins, Charles Eugene, Rooney, Sean Anthony, Agnetta, Bryan Todd, Park, Jae Pum, Mains, Richard Leigh, Marashi, Suzan
• Primary Examiner(s): Hailu, Tadeese
• Application Number:
  US14/137,783
• Publication Number:
  US 20140337800A1
• Time in Patent Office: 1,558 Days
• CPC Class Codes: G06F 2203/04801 Cursor retrieval aid, i.e. visual aspect modification, blinking, colour changes, enlargement or other visual cues, for helping user do find the cursor in graphical user interfaces
• CPC Class Codes: G06F 3/002 Specific input/output arrangements not covered by G06F3/01 - G06F3/16 other optical apparatus G02B27/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: H04M 1/2755 by optical scanning
• CPC Class Codes: H04M 2250/22 including a touch pad, a touch sensor or a touch detector digitizers for touch screens G06F3/041
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264

Abstract

A computing device can utilize a recognition mode wherein an interface utilizes graphical elements, such as virtual fireflies or other such elements, to indicate objects that are recognized or identified. As objects are recognized, fireflies perform one or more specified actions to indicate recognition. A ribbon or other user-selectable icon is displayed indicates a specific action that the device can perform with respect to the respective object. As additional objects are recognized, additional ribbons are created and older ribbons can be moved off screen and stored for subsequent retrieval or search. The fireflies disperse when the objects are no longer represented in captured sensor data, and can be animated to move towards representations of new objects as features of those objects are identified as potential object features, in order to communicate a level of recognition for a current scene or environment.

Claims

1. A computing device, comprising:

at least one processor;

a display screen;

at least one camera; and

at least one memory including instructions that, when executed by the at least one processor, cause the computing device to;

receive image data using the at least one camera;

present, on the display screen, an image corresponding to the image data;

determine, based on object recognition processing, a first representation of a first object in the image data;

determining, based on object recognition processing, a second representation of a second object in the image data;

cause, on the display screen, a first plurality of graphical elements to indicate the first representation has been determined;

cause, on the display screen a second plurality of graphical elements to indicate the second representation has been determined;

present, on the display screen while the first plurality of graphical elements indicate the first representation has been determined, a first user-selectable icon indicating a first action that the computing device is configured to perform with respect to the first object; and

present, on the display screen while the second plurality of graphical elements indicate the second representation has been determined, a second user-selectable icon indicating a second action that the computing device is configured to perform with respect to the second object, the second user-selectable icon being presented adjacent to the first user-selectable icon.

2. The computing device of claim 1, wherein the instructions, when executed by the at least one processor, further cause the computing device to:

receive audio data using a microphone of the computing device; and

present, on the display screen and based on the audio data, a third user-selectable icon indicating a third action that the computing device is configured to perform.

3. The computing device of claim 1, wherein the instructions, when executed by the at least one processor, further cause the computing device to:

present the first user-selectable icon to appear to be stacked on top of the second user-selectable icon on the display screen.

4. The computing device of claim 1, wherein the instructions, when executed by the at least one processor, further cause the computing device to:

store information for a previously-displayed user-selectable icon; and

present, on the display screen and based on a user performing a swiping motion on the display screen, the previously-displayed user-selectable icon.

5. The computing device of claim 1, wherein the instructions, when executed by the at least one processor, further cause the computing device to:

provide a search option enabling a search to be executed with respect to information stored for any currently displayed user-selectable icon and any previously-displayed user-selectable icon.

6. A computer-implemented method, comprising:

receiving sensor data using at least one sensor of a computing device;

determining, based on object recognition processing, a first representation of a first object in the sensor data;

determining, based on object recognition processing, a second representation of a second object in the sensor data;

causing, on a display screen of the computing device, a first plurality of graphical elements to indicate that the first representation has been determined;

causing, on the display screen, a second plurality of the graphical elements to indicate that the second representation has been determined;

presenting, on the display screen while the first plurality of graphical elements indicate the first representation has been determined, a first user-selectable icon indicating a first action that the computing device is configured to perform with respect to the first object; and

presenting, on the display screen while the second plurality of graphical elements indicate the second representation has been determined, a second user-selectable icon indicating a second action that the computing device is configured to perform with respect to the second object, the second user-selectable icon being presented adjacent to the first user-selectable icon on the display screen.

7. The computer-implemented method of claim 6, further comprising:

presenting the first user-selectable icon on the display screen for at least a period of time after the first representation is no longer presented on the display screen.

8. The computer-implemented method of claim 6, wherein causing the first plurality of graphical elements to indicate the first representation has been determined comprises:

causing the first plurality of graphical elements to form a bounding box around the first a representation on the display screen.

9. The computer-implemented method of claim 6, further comprising:

removing at least one user-selectable icon from the display screen in response to an additional user-selectable icon causing a number of user-selectable icons presented on the display screen to exceed a threshold number of user-selectable icons.

10. The computer-implemented method of claim 9, further comprising:

storing information for any user-selectable icon no longer presented on the display screen, wherein the information is accessible using at least one of a scroll action, a touch-screen-based pull action, or a search request.

11. The computer-implemented method of claim 6, wherein presenting the first user-selectable icon is performed using information previously stored associated with the first object.

12. The computer-implemented method of claim 6, wherein the first user-selectable icon and the second user-selectable icon are presented as at least one of stacked or positioned in a sequence.

13. The computer-implemented method of claim 6, wherein the first object is associated with the first action and a third action, and wherein the first action is conveyed by the first user-selectable icon based at least in part upon historical data associated with a user.

14. The computer-implemented method of claim 6, wherein the first user-selectable icon is presented at a location on the display screen based upon at least one of a first location corresponding to the first representation and a second location corresponding to the second representation, a first type corresponding to the first object, historical user data, a date of last access, or times at which the first user-selectable icon and the second user-selectable icon were generated.

15. A computing device, comprising:

at least one processor;

a display screen;

at least one sensor; and

at least one memory including instructions that, when executed by the at least one processor, cause the computing device to;

receive sensor data using the at least one sensor;

determine, based on object recognition processing, a first representation of a first object in the sensor data;

determine, based on object recognition processing, a second representation of a second object in the sensor data;

cause, on the display screen, a first plurality of graphical elements to indicate the first representation has been determined;

cause, on the display screen, a second plurality of the graphical elements to indicate the second representation has been determined;

present, on the display screen while the first plurality of graphical elements indicate the first representation has been determined, a first user-selectable icon indicating a first action that the computing device is configured to perform with respect to the first object; and

present, on the display screen while the second plurality of graphical elements indicate the second representation has been determined, a second user-selectable icon indicating a second action that the computing device is configured to perform with respect to the second object, the second user-selectable icon being presented adjacent to the first user-selectable icon on the display screen.

16. The computing device of claim 15, wherein the instructions, when executed by the at least one processor to, further cause the computing device to:

present the first user-selectable icon on the display screen for at least a period of time after the first representation is no longer presented on the display screen.

17. The computing device of claim 15, wherein the instructions, when executed by the at least one processor, causing the computing device to cause the first plurality of graphical elements to indicate the first representation has been determined further include instructions to:

cause the first plurality of graphical elements to form a bounding box around the first representation on the display screen.