Image analysis and communication device control technology

• US 8,934,719 B1
• Filed: 09/13/2013
• Issued: 01/13/2015
• Est. Priority Date: 09/29/2009
• Status: Active Grant

• Current Assignee: Uber Technologies, Inc.
• Original Assignee: Jason Adam Denise
• Inventors: Denise, Jason Adam
• Primary Examiner(s): BEKELE, MEKONEN T
• Application Number:
  US14/027,131
• Time in Patent Office: 487 Days
• Field of Search: 382/190, 382/199, 382/195, 382/110, 345/786, 345/784, 345/688, 379/265.01, 379/265.02, 379/265.11
• US Class Current: 382/190
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04M 1/72463 to restrict the functionality of the device
• CPC Class Codes: H04M 1/724631 by limiting the access to the user interface, e.g. locking a touch-screen or a keypad
• CPC Class Codes: H04M 1/72484 wherein functions are triggered by incoming communication events
• CPC Class Codes: H04M 19/04 the ringing-current being generated at the substations
• CPC Class Codes: H04M 2250/52 including functional features of a camera
• CPC Class Codes: H04M 3/02 Calling substations, e.g. by ringing selective calling H04Q
• CPC Class Codes: H04W 4/12 Messaging; Mailboxes; Announcements

Abstract

Techniques are described for controlling communication devices using image analysis. For instance, when a communication is received by a communication device, the communication device outputs an alert to notify users of the received communication and accesses one or more images of an area proximate to the communication device. The one or more images cover an area proximate to the communication device at a time during which the communication device is outputting the alert. The communication device analyzes the one or more images to determine whether a user is present in the one or more images and in a position to perceive the received communication. The communication device handles at least one aspect of the received communication based on the determination of whether a user is in a position to perceive the received communication.

Claims

1. A smart phone comprising:

a speaker configured to provide audible output;

a display screen configured to provide display output;

an image capture device configured to capture images of an area in front of the display screen such that the image capture device is oriented to capture images of an area in which a user is in position to perceive the display screen; and

at least one processor configured to control operation of the speaker, the display screen, and the image capture device, the at least one processor being configured to perform operations comprising;

controlling the speaker to initiate audible output and controlling the display screen to initiate visual output associated with the audible output from the speaker;

while providing the audible output through the speaker and the visual output through the display screen;

controlling the image capture device to capture one or more images of the area in front of the display screen;

analyzing the one or more images for a user in a position in front of the display screen;

based on the analysis of the one or more images for a user in a position in front of the display screen, detecting a person within the one or more images of the area in front of the display screen;

based on detecting the person in the one or more images of the area in front of the display screen, analyzing the one or more images of the area in front of the display screen to determine whether the person is looking at the display screen;

based on the analysis of the one or more images of the area in front of the display screen to determine whether the person is looking at the display screen, determining whether the person is looking at the display screen; and

automatically, without human intervention, controlling further handling of the audible output through the speaker and the visual output through the display screen in accordance with the determination of whether the person is looking at the display screen,

wherein automatically, without human intervention, controlling further handling of the audible output through the speaker and the visual output through the display screen in accordance with the determination of whether the person is looking at the display screen comprises, in response to a determination that the person is not looking at the display screen, automatically, without human intervention, adjusting the audible output through the speaker and the visual output through the display screen in a manner suitable for a person that is not looking at the display screen.

2. The smart phone of claim 1:

wherein analyzing the one or more images of the area in front of the display screen to determine whether the person is looking at the display screen comprises detecting the person'"'"'s eyes within the one or more images of the area in front of the display screen; and

wherein determining whether the person is looking at the display screen comprises determining that the person is looking at the display screen based on the detection of the person'"'"'s eyes within the one or more images of the area in front of the display screen.

3. The smart phone of claim 1, wherein controlling the image capture device to capture one or more images of the area in front of the display screen comprises initiating, in response to user manipulation of the smart phone, the image capture device to begin capturing images of the area in front of the display screen.

4. A mobile device comprising:

a display screen configured to provide display output;

an image capture device configured to capture images of an area in front of the display screen such that the image capture device is oriented to capture images of an area in which a user is in position to perceive the display screen;

at least one processor configured to control operation of the display screen and the image capture device, the at least one processor being configured to perform operations comprising;

controlling the image capture device to capture one or more images of the area in front of the display screen;

analyzing the one or more images of the area in front of the display screen for a user in a position that enables perception of the display screen;

based on the analysis of the one or more images of the area in front of the display screen for a user in a position that enables perception of the display screen, determining whether a user is in a position that enables perception of the display screen;

based on a determination that a user is in a position that enables perception of the display screen, automatically, without human intervention, updating output of the display screen to display additional information that was not displayed by the display screen; and

based on a determination that a user is not in a position that enables perception of the display screen, maintaining output of the display screen without displaying the additional information,

wherein controlling the image capture device to capture one or more images of the area in front of the display screen comprises controlling the image capture device to begin capturing images in response to user manipulation of the mobile device.

5. The mobile device of claim 4:

wherein analyzing the one or more images of the area in front of the display screen for a user in a position that enables perception of the display screen comprises detecting a user'"'"'s face within the one or more images of the area in front of the display screen; and

wherein determining whether a user is in a position that enables perception of the display screen comprises determining that a user is in a position that enables perception of the display screen based on the detection of a user'"'"'s face within the one or more images of the area in front of the display screen.

6. The mobile device of claim 5:

wherein detecting a user'"'"'s face within the one or more images of the area in front of the display screen comprises identifying facial features within the one or more images of the area in front of the display screen, comparing the identified facial features to one or more facial templates, and detecting, based on the comparison, that the identified facial features match at least one of the one or more facial templates; and

wherein determining whether a user is in a position that enables perception of the display screen comprises determining that a user is in a position that enables perception of the display screen based on the detection that the identified facial features match at least one of the one or more facial templates.

7. The mobile device of claim 4:

wherein analyzing the one or more images of the area in front of the display screen for a user in a position that enables perception of the display screen comprises detecting a user'"'"'s eyes within the one or more images of the area in front of the display screen; and

wherein determining whether a user is in a position that enables perception of the display screen comprises determining that a user is in a position that enables perception of the display screen based on the detection of the user'"'"'s eyes within the one or more images of the area in front of the display screen.

8. The mobile device of claim 4:

wherein analyzing the one or more images of the area in front of the display screen for a user in a position that enables perception of the display screen comprises;

detecting a person within the one or more images of the area in front of the display screen, and

based on detecting the person in the one or more images of the area in front of the display screen, analyzing the one or more images of the area in front of the display screen to determine whether the person is in a position to perceive the display screen; and

wherein determining whether a user is in a position that enables perception of the display screen comprises determining whether the person is in a position to perceive the display screen.

9. The mobile device of claim 8:

wherein analyzing the one or more images of the area in front of the display screen to determine whether the person is in a position to perceive the display screen comprises analyzing the one or more images of the area in front of the display screen to determine whether a face of the person is oriented in a direction that enables the person to perceive the display screen; and

wherein determining whether the person is in a position to perceive the display screen comprises determining whether the face of the person is oriented in a direction that enables the person to perceive the display screen.

10. The mobile device of claim 8:

wherein analyzing the one or more images of the area in front of the display screen to determine whether the person is in a position to perceive the display screen comprises analyzing the one or more images of the area in front of the display screen to determine whether the person is looking at the display screen; and

wherein determining whether the person is in a position to perceive the display screen comprises determining whether the person is looking at the display screen.

11. The mobile device of claim 4, wherein the operations further comprise controlling the display screen to initiate visual output, wherein automatically, without human intervention, updating output of the display screen to display additional information that was not displayed by the display screen comprises automatically, without human intervention, controlling further handling of the visual output of the display screen by displaying additional information that is additional to the visual output.

12. A communication device comprising:

a display screen configured to provide display output;

an image capture device configured to capture images of an area in front of the display screen such that the image capture device is oriented to capture images of an area in which a user is in position to perceive the display screen; and

at least one processor configured to control operation of the display screen and the image capture device, the at least one processor being configured to perform operations comprising;

controlling the image capture device to capture one or more images of the area in front of the display screen;

analyzing the one or more images of the area in front of the display screen for a user in a position that enables perception of the display screen;

based on the analysis of the one or more images of the area in front of the display screen for a user in a position that enables perception of the display screen, determining whether a user is in a position that enables perception of the display screen;

based on a determination that a user is in a position that enables perception of the display screen, automatically, without human intervention, maintaining the display screen in an on state in which information is displayed; and

based on a determination that a user is not in a position that enables perception of the display screen, automatically, without human intervention, controlling the display screen to change to a blank or an off state.

13. The communication device of claim 12:

wherein analyzing the one or more images of the area in front of the display screen for a user in a position that enables perception of the display screen comprises detecting a user'"'"'s face within the one or more images of the area in front of the display screen; and

wherein determining whether a user is in a position that enables perception of the display screen comprises determining that a user is in a position that enables perception of the display screen based on the detection of a user'"'"'s face within the one or more images of the area in front of the display screen.

14. The communication device of claim 13:

wherein detecting a user'"'"'s face within the one or more images of the area in front of the display screen comprises identifying facial features within the one or more images of the area in front of the display screen, comparing the identified facial features to one or more facial templates, and detecting, based on the comparison, that the identified facial features match at least one of the one or more facial templates; and

wherein determining whether a user is in a position that enables perception of the display screen comprises determining that a user is in a position that enables perception of the display screen based on the detection that the identified facial features match at least one of the one or more facial templates.

15. The communication device of claim 12:

wherein analyzing the one or more images of the area in front of the display screen for a user in a position that enables perception of the display screen comprises detecting a user'"'"'s eyes within the one or more images of the area in front of the display screen; and

wherein determining whether a user is in a position that enables perception of the display screen comprises determining that a user is in a position that enables perception of the display screen based on the detection of the user'"'"'s eyes within the one or more images of the area in front of the display screen.

16. The communication device of claim 12:

wherein analyzing the one or more images of the area in front of the display screen for a user in a position that enables perception of the display screen comprises;

detecting a person within the one or more images of the area in front of the display screen, and

based on detecting the person in the one or more images of the area in front of the display screen, analyzing the one or more images of the area in front of the display screen to determine whether the person is in a position to perceive the display screen; and

wherein determining whether a user is in a position that enables perception of the display screen comprises determining whether the person is in a position to perceive the display screen.

17. The communication device of claim 16:

wherein analyzing the one or more images of the area in front of the display screen to determine whether the person is in a position to perceive the display screen comprises analyzing the one or more images of the area in front of the display screen to determine whether a face of the person is oriented in a direction that enables the person to perceive the display screen; and

wherein determining whether the person is in a position to perceive the display screen comprises determining whether the face of the person is oriented in a direction that enables the person to perceive the display screen.

18. The communication device of claim 16:

wherein analyzing the one or more images of the area in front of the display screen to determine whether the person is in a position to perceive the display screen comprises analyzing the one or more images of the area in front of the display screen to determine whether the person is looking at the display screen; and

wherein determining whether the person is in a position to perceive the display screen comprises determining whether the person is looking at the display screen.

19. The communication device of claim 12:

wherein automatically, without human intervention, maintaining the display screen in an on state in which information is displayed comprises automatically, without human intervention, maintaining the display screen in an on state in which information related to a received electronic communication is displayed; and

wherein automatically, without human intervention, controlling the display screen to change to a blank or an off state comprises controlling the display screen to change from the on state in which information related to the received electronic communication is displayed to a blank state.

20. The communication device of claim 12, wherein the operations further comprise:

receiving an electronic communication; and

based on receiving the electronic communication, controlling the display screen to provide output related to the electronic communication,

wherein controlling the image capture device to capture one or more images of the area in front of the display screen, analyzing the one or more images of the area in front of the display screen for a user in a position that enables perception of the display screen, and determining whether a user is in a position that enables perception of the display screen all occur while the display screen provides output related to the electronic communication,

wherein automatically, without human intervention, maintaining the display screen in an on state in which information is displayed comprises, based on a determination that a user is in a position that enables perception of the display screen, automatically, without human intervention, maintaining the display screen in an on state in which the display screen provides output related to the electronic communication; and

wherein automatically, without human intervention, controlling the display screen to change to a blank or an off state comprises, based on a determination that a user is not in a position that enables perception of the display screen, automatically, without human intervention, controlling the display screen to change from providing output related to the electronic communication to a blank or an off state.