Systems and methods for determining an emotional environment from facial expressions

• US 10,523,991 B2
• Filed: 08/31/2016
• Issued: 12/31/2019
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: OrCam Technologies Ltd
• Original Assignee: OrCam Technologies Ltd
• Inventors: Wexler, Yonatan, Shashua, Amnon
• Primary Examiner(s): Perlman, David
• Application Number:
  US15/253,616
• Publication Number:
  US 20170061201A1
• Time in Patent Office: 1,217 Days
• Field of Search: None
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/18105 related to colour
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/175 Static expression
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04N 21/2668 Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles information retrieval from the Internet by querying with filtering and personalisation G06F16/9535; arrangements for replacing or switching information during the broadcast H04H20/10; push services over packet-switching network H04L12/1859; adaptation of message content in packet-switching networks H04L51/063
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/4223 : Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44218 : Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 23/61 : based on recognised objects
• CPC Class Codes: H04N 23/611 : where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

A wearable apparatus is provided for capturing and processing images from an environment of a user. In one implementation, the wearable apparatus may determine an emotional environment of the user of the wearable apparatus. The wearable apparatus may include an image sensor and capture one or more images from an environment around the user. The wearable device may also be configured to analyze the one or more images to identify facial expressions of a person in the image. In some embodiments, the wearable apparatus may also identify the person in the one or more images. The wearable apparatus may also transmit information associated with the facial expression and/or identity to an external device.

Claims

1. A wearable apparatus for determining an emotional environment of a user of the wearable apparatus, the wearable apparatus comprising:

a wearable image sensor configured to capture one or more images from an environment of the user; and

at least one processor programmed to;

analyze the one or more images to identify a facial expression of a person in the environment of the user;

aggregate and weight features of the facial expression over a period of time to determine a mood of the person;

receive information indicative of a state of the user;

determine, based on the mood and the state of the user, an interaction factor between the user and the person in the environment of the user; and

transmit, to an external device, information associated with the facial expression and the interaction factor to cause the external device to determine a classification associated with the emotional environment based on the information associated with the facial expression and the interaction factor.

2. The wearable apparatus of claim 1, wherein the external device is programmed to analyze the information associated with the facial expression.

3. The wearable apparatus of claim 1, wherein the external device includes at least one of a smartphone, a tablet, and a smartwatch.

4. The wearable apparatus of claim 1, wherein the external device includes a server.

5. The wearable apparatus of claim 1, further comprising a transmitter configured to enable wireless pairing with a receiver in the external device.

6. The wearable apparatus of claim 1, wherein the information associated with the facial expression comprises a number of smiles.

7. The wearable apparatus of claim 1, wherein the information associated with the facial expression comprises a classification of the facial expression.

8. The wearable apparatus of claim 7, wherein the classification comprises an indication of a friendliness level.

9. The wearable apparatus of claim 8, wherein the indication of the friendliness level is based on analysis of changes in the facial expression over a time period.

10. The wearable apparatus of claim 1, wherein the external device is programmed to display information associated with the facial expression on a display of the external device.

11. The wearable apparatus of claim 1, wherein the at least one processor is further programmed to:

analyze the one or more images to identify the person associated with the facial expression; and

transmit, to the external device, information associated with the identified person.

12. The wearable apparatus of claim 11, wherein the external device is configured to update a record associated with the identified person according to the facial expression.

13. The wearable apparatus of claim 12, wherein the record comprises at least one of a profile stored in a database of a plurality of profiles corresponding to individuals and a social graph.

14. The wearable apparatus of claim 1, wherein the at least one processor is further programmed to:

analyze the one or more images to identify a facial expression of a second person in the environment of the user;

analyze the one or more images to identify the second person; and

transmit, to the external device, information associated with the facial expression of the second person and information associated with the second person.

15. The wearable apparatus of claim 14, wherein the external device is configured to update a record associated with the persons according to the facial expressions.

16. The wearable apparatus of claim 15, wherein the record comprises a social graph.

17. The wearable apparatus of claim 1, further comprising a physiological sensor configured to capture data associated with a state of the user, and wherein the transmitted information is also associated with the state of the user.

18. The wearable apparatus of claim 1, wherein the interaction factor includes a degree of a relationship between the user and the person.

19. The wearable apparatus of claim 1, wherein the information indicative of a state of the user is received from at least one component of the wearable apparatus.

20. A method for determining an emotional environment of a user of a wearable apparatus, the method comprising:

obtaining one or more images of at least a portion of an environment of the user;

analyzing the one or more images to identify a facial expression of a person in the environment of the user;

aggregating and weighting features of the facial expression over a period of time to determine a mood of the person;

receive information indicative of a state of the user;

determining, based on the mood and the state of the user, an interaction factor between the user and the person in the environment of the user; and

transmitting, to an external device, information associated with the facial expression and the interaction factor to cause the external device to determine a classification associated with the emotional environment based on the information associated with the facial expression and the interaction factor.

21. The method of claim 20 further comprising:

capturing the one or more images using a wearable image sensor of the wearable apparatus.

22. The method of claim 20 further comprising:

determining a number of smiles over a predetermined time based on the facial expression.

23. The method of claim 20 further comprising:

obtaining a classification of the facial expression based on the facial expression.

24. The method of claim 23, wherein the classification comprises an indication of a friendliness level.

25. The method of claim 20 further comprising:

providing the information associated with the facial expression to the user.

26. The method of claim 20 further comprising:

analyzing the one or more images to identify the person associated with the facial expression.

27. The method of claim 26 further comprising:

updating a record associated with the person according to the facial expression.

28. The method of claim 27, wherein the record comprises at least one of a profile stored in a database of a plurality of profiles corresponding to individuals and a social graph.

29. The method of claim 26 further comprising:

analyzing the one or more images to identify a facial expression of a second person in the environment of the user;

analyzing the one or more images to identify the second person; and

updating a record associated with the persons according to the facial expressions.

30. The method of claim 29, wherein the record comprises a social graph.

31. The method of claim 20 further comprising:

receiving a time associated with the one or more images including the facial expression of the person in the environment of the user;

receiving information about the user indicative of a state of the user; and

determining a relationship between the person and the user based on the one or more images, the time, and the state of the user.

32. A non-transitory computer readable medium storing computer implementable instructions for carrying out the method of claim 20.

33. The method of claim 20, further comprising receiving data captured from a physiological sensor configured associated with a state of the user, and wherein the transmitted information is also associated with the state of the user.

34. The method of claim 20, wherein the interaction factor includes a degree of a relationship between the user and the person.

35. The method of claim 20, wherein the information indicative of a state of the user is received from at least one component of the wearable apparatus.