Graphical behaviors for recognition interfaces

• US 9,671,941 B1
• Filed: 12/20/2013
• Issued: 06/06/2017
• Est. Priority Date: 05/09/2013
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Gray, Timothy Thomas, Cummins, Charles Eugene, Glaser, Russell Edward, Pagan, Tito, Sommer, Steven Michael, Kralyevich, Brian Peter, Warren, Angela Kathleen, Marashi, Suzan, Salazar, Marc Anthony
• Primary Examiner(s): Kawsar, Abdullah Al
• Assistant Examiner(s): Yu, Jian
• Application Number:
  US14/137,674
• Time in Patent Office: 1,264 Days
• Field of Search: 715702, 715764, 715802, 715821, 715822, 715823, 715861
• CPC Class Codes: G06F 2203/04801 Cursor retrieval aid, i.e. visual aspect modification, blinking, colour changes, enlargement or other visual cues, for helping user do find the cursor in graphical user interfaces
• CPC Class Codes: G06F 3/002 Specific input/output arrangements not covered by G06F3/01 - G06F3/16 other optical apparatus G02B27/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: H04M 1/2755 by optical scanning
• CPC Class Codes: H04M 2250/22 including a touch pad, a touch sensor or a touch detector
• CPC Class Codes: H04M 2250/52 including functional features of a camera

Abstract

A computing device can utilize a recognition mode wherein an interface utilizes graphical elements, such as virtual fireflies, to indicate recognized or identified objects. Fireflies can be displayed near an input element to indicate that a recognition mode is available. When a user selects the input element, the fireflies can appear to emanate from the input element and disperse across the display screen. As objects are recognized, fireflies can create bounding boxes around those objects, or otherwise appear proximate those objects, to indicate recognition. The fireflies can again disperse as the objects fall out of view, and can begin moving towards new objects as features of those objects are identified as potential object features. A subsequent selection of the input element to exit recognition mode can cause the fireflies to appear to retreat to their original location in, or near, the input element.

Claims

1. A computing device, comprising:

at least one processor;

a camera configured to generate video data;

a display screen;

a physical button proximate to a portion of the display screen; and

memory including instructions that, when executed by the at least one processor, cause the computing device to;

generate the video data using the camera;

display video content based on at least a portion of the video data on the display screen;

display, on the portion of the display screen proximate to the physical button, a plurality of graphical elements at a first time;

in response to receiving a selection of the physical button at a second time after the first time, causing the plurality of graphical elements to appear to emanate from being bounded from the portion of the display screen to being dispersed across the display screen;

receive an indication that the video data exhibits characteristics corresponding to a recognizable object;

causing at least a portion of the graphical elements to appear proximate to a representation of the recognizable object on the display screen; and

in response to receiving another selection of the physical button, causing the plurality of graphical elements to appear to return to being bounded in the portion of the display screen proximate to the physical button on the display screen.

2. The computing device of claim 1, wherein the instructions when executed further cause the computing device to:

cause the graphical elements to be animated to appear to be dispersed across at least a region of the display screen upon entering a camera mode of the computing device; and

cause the plurality of graphical elements to be animated to appear to move to the portion of the display screen proximate to the physical button, indicating that a recognition mode of the computing device is available, wherein the first selection is capable of activating the recognition mode of the computing device.

3. A computer-implemented method, comprising:

capturing sensor data using at least one sensor of a computing device;

causing, at a first time, a plurality of graphical elements to be displayed in an animated manner on a display of the computing device, and to appear to be associated with a button of the computing device;

in response to receiving input corresponding to the button at a second time after the first time, causing the graphical elements to disperse across a display area of the computing device;

receiving, at a third time after the second time, an indication that the sensor data exhibits characteristics corresponding to a recognizable object; and

causing the plurality of graphical elements to convey information via the display area, in response to recognizing the recognizable object in the sensor data.

4. The computer-implemented method of claim 3, further comprising:

receiving a subsequent input corresponding to the button; and

causing the plurality of graphical elements to be animated to appear to return to being associated with the button.

5. The computer-implemented method of claim 3, wherein:

capturing sensor data comprises capturing from a camera;

the graphical elements include a plurality of flashing elements; and

the method further comprises;

displaying, on a display area of the computing device, video data generated using the camera, and

displaying, on the display area, the flashing elements being animated to move across the display and to form a bounding box around a representation of the recognizable object in response to the object being recognized.

6. The computer-implemented method of claim 3, wherein:

capturing sensor data comprising capturing audio data using a microphone; and

the method further comprises;

animating the plurality of graphical elements to convey information at least in part by forming an animated shape indicating that at least a portion of the audio data is recognized.

7. The computer-implemented method of claim 3, wherein causing the plurality of graphical elements to be animated to appear to be associated with the button comprises at least one of:

illuminating a physical button;

illuminating a physical element proximate to the physical button;

displaying an animation on the physical button;

displaying the graphical elements on a portion of the display area proximate to the physical button;

displaying the graphical elements proximate to a virtual button on the display area;

or

displaying the graphical elements in the virtual button on the display area.

8. The computer-implemented method of claim 3, further comprising:

receiving an initial input corresponding to the button; and

causing the plurality of graphical elements to be displayed on the display area in a portion of the display screen proximate to the button in response to the initial input, wherein at least a portion of a user interface associated with the at least one sensor is removed from the display area while the plurality of graphical elements are displayed.

9. The computer-implemented method of claim 3, further comprising:

causing at least one of a buzzing sound or a vibration to be emitted from the computing device when the plurality of graphical elements are animated to appear to be displayed in a portion of the display screen proximate to the button.

10. The computer-implemented method of claim 3, further comprising:

causing at least one of a buzzing sound or a vibration to be emitted from the computing device when the plurality of graphical elements are animated to appear to be dispersing across at least a portion of the display area.

11. The computer-implemented method of claim 3, wherein the graphical elements are of a type selected from a plurality of different types of graphical elements stored on the computing device.

12. The computer-implemented method of claim 11, wherein the selected type includes a plurality of animated virtual fireflies.

13. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor of a computing device, cause the computing device to:

acquire sensor data using at least one sensor of the computing device;

cause a plurality of graphical elements to be displayed on a portion of a display area proximate to a button of the computing device at a first time;

in response to receiving, at a second time after the first time, input corresponding to the button, cause the plurality of graphical elements to appear to emanate from being bounded from the portion of the display area to being dispersed across the display area;

receive, at a third time after the second time, an indication that the sensor data exhibits characteristics corresponding to a recognizable object; and

cause the plurality of graphical elements to be animated to convey information, via the display area, in response to detecting a representation of the recognizable object in the sensor data.

14. The non-transitory computer-readable storage medium of claim 13, wherein the instructions when executed further cause the computing device to:

animate at least a portion of the graphical elements using at least one of different colors, brightness values, intensity values, sizes, motions, speeds, or shapes.

15. The non-transitory computer-readable storage medium of claim 13, wherein the instructions when executed further cause the computing device to:

animate the graphical elements to appear at different depths of a three-dimensional view displayed via the display area.

16. The non-transitory computer-readable storage medium of claim 13, wherein the instructions when executed further cause the computing device to:

receive a subsequent input corresponding to the button; and

cause the plurality of graphical elements to be animated to appear to return to the portion of a display area proximate to the button.

17. The non-transitory computer-readable storage medium of claim 13, wherein the instructions when executed further cause the computing device to:

cause the graphical elements to be animated to appear to emanate from being bounded from a portion of the display screen proximate to the representation of the recognized object to being dispersed across the display area, in response to the recognized object no longer being represented in the acquired sensor data.

18. The non-transitory computer-readable storage medium of claim 13, wherein the instructions when executed further cause the computing device to:

receive an initial selection of the button; and

cause the at least one sensor to begin acquiring the sensor data in response to the initial selection.

19. The non-transitory computer-readable storage medium of claim 13, wherein the instructions when executed further cause the computing device to:

cause a first subset of the graphical elements to appear to move using animation from one location of the display area to another location; and

cause a second subset of the graphical elements to flash at a respective locations of the display area, wherein the elements of the second subset appear to move by disappearing at a first location and appearing at a second location.