Wearable apparatus and methods for processing images to identify contextual situations

• US 10,380,425 B2
• Filed: 04/24/2017
• Issued: 08/13/2019
• Est. Priority Date: 01/06/2016
• Status: Expired due to Fees

• Current Assignee: OrCam Technologies Ltd
• Original Assignee: OrCam Technologies Ltd
• Inventors: Wexler, Yonatan, Shashua, Amnon
• Primary Examiner(s): Cattungal, Rowina J
• Application Number:
  US15/495,650
• Publication Number:
  US 20170228596A1
• Time in Patent Office: 841 Days
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 21/6245 Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 3/002 Specific input/output arrangements not covered by G06F3/01 - G06F3/16 other optical apparatus G02B27/00
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06K 7/10366 the interrogation device being adapted for miscellaneous applications
• CPC Class Codes: G06Q 10/103 Workflow collaboration or project management
• CPC Class Codes: G06Q 30/0631 Item recommendations
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/35 : Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 20/41 : Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 40/113 : Recognition of static hand signs
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G08B 21/18 : Status alarms G08B21/02 takes precedence
• CPC Class Codes: H04L 65/61 : for supporting one-way streaming services, e.g. Internet radio
• CPC Class Codes: H04N 1/00342 : with a radio frequency tag transmitter or receiver storage of additional information in an RFID tag attached to an image bearing paper sheet H04N1/32138
• CPC Class Codes: H04N 23/11 : for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/56 : provided with illuminating means
• CPC Class Codes: H04N 23/64 : Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/66 : Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 7/147 : Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 : Conference systems
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control
• CPC Class Codes: H04W 4/02 : Services making use of location information
• CPC Class Codes: H04W 4/029 : Location-based management or tracking services

Abstract

A wearable apparatus is provided for capturing and processing images from an environment of a user. In one implementation, the wearable apparatus is used for identifying a contextual situation related to a wearer. The wearable apparatus includes a wearable image sensor configured to capture a plurality of images from an environment of the wearer. The wearable apparatus further includes at least one processing device. The at least one processing device is programmed to analyze the plurality of images to identify the contextual situation related to the wearer; determine information associated with the contextual situation; and cause the transmitter to transmit the determined information to a device paired with the wearable apparatus to cause the paired device to provide at least one alert to the wearer based on the determined information associated with the contextual situation.

Claims

1. A wearable apparatus for identifying contextual situations related to a user wearing the wearable apparatus, the wearable apparatus comprising:

a wearable image sensor configured to capture a plurality of images from an environment of the user wearing the wearable apparatus;

a transmitter; and

at least one processing device programmed to;

analyze the plurality of images to identify a first contextual situation related to the user wearing the wearable apparatus;

use the plurality of images to determine information associated with the first contextual situation related to the user wearing the wearable apparatus;

use at least of one image-information from previously obtained images and additional information from one or more sources to improve a certainty level in the identification of the first contextual situation;

cause the transmitter to transmit the determined information to a device paired with the wearable apparatus through a short-range communication link to cause the paired device to provide at least one alert to the user wearing the wearable apparatus based on the determined information associated with the first contextual situation identified in the plurality of images;

analyze at least one image captured by the wearable image sensor to identify a second contextual situation related to the user wearing the wearable apparatus;

determine a time difference between the first contextual situation and the second contextual situation; and

based on the first contextual situation, image-information associated with the second contextual situation, and a comparison of the determined time difference with a predetermined threshold associated with the first contextual situation, withhold transmission to the paired device of alerts associated with the second contextual situation.

2. The wearable apparatus of claim 1, wherein the first contextual situation identified in the plurality of images includes the user wearing the wearable apparatus transitioning from indoors to outdoors.

3. The wearable apparatus of claim 2, wherein the at least one alert triggered in the paired device in response to the transmission of the determined information includes information related to a weather condition.

4. The wearable apparatus of claim 2, wherein the at least one alert triggered in the paired device in response to the transmission of the determined information includes a suggestion to remember rain gear.

5. The wearable apparatus of claim 2, wherein the at least one alert triggered in the paired device in response to the transmission of the determined information includes a suggestion to remember a key.

6. The wearable apparatus of claim 2, wherein the at least one alert triggered in the paired device in response to the transmission of the determined information includes a suggestion to change an HVAC setting.

7. The wearable apparatus of claim 1, wherein the first contextual situation identified in the plurality of images includes the user wearing the wearable apparatus entering a grocery store.

8. The wearable apparatus of claim 7, wherein the at least one alert triggered in the paired device in response to the transmission of the determined information includes a suggestion to purchase an item.

9. The wearable apparatus of claim 8, wherein the suggestion to purchase an item is based on a prior determination by the at least one processing device, based on prior captured images, that a container associated with the item was discarded by the user wearing the wearable apparatus.

10. The wearable apparatus of claim 1, wherein the first contextual situation identified in the plurality of images includes the user wearing the wearable apparatus exiting a vehicle.

11. The wearable apparatus of claim 10, wherein the at least one alert triggered in the paired device in response to the transmission of the determined information includes a reminder indicating that a child is present in the vehicle.

12. The wearable apparatus of claim 1, wherein the first contextual situation identified in the plurality of images includes an identification of at least one worker at a work site and the at least one alert triggered in the paired device in response to the transmission of the determined information indicates that the worker is not using and/or wearing safety equipment.

13. The wearable apparatus of claim 1, wherein the paired device is at least one of:

a smartphone, a smarthome controller, a tablet, or a smart watch that is located in proximity to the wearable apparatus.

14. The wearable apparatus of claim 1, wherein the first contextual situation identified in the plurality of images includes a document present in an area in front of the user wearing the wearable apparatus.

15. The wearable apparatus of claim 14, wherein the determined information transmitted from the wearable apparatus through the short-range communication link to the paired device includes an address appearing on the document.

16. The wearable apparatus of claim 15, wherein the at least one alert triggered in the paired device in response to the transmission of the determined information includes the address appearing in a user interface associated with a navigation assistance application.

17. The wearable apparatus of claim 14, wherein the determined information transmitted from the wearable apparatus through the short-range communication link to the paired device includes text from a business card.

18. The wearable apparatus of claim 17, wherein the at least one alert triggered in the paired device in response to the transmission of the determined information includes the text from the business card appearing in a user interface associated with an application on the paired device.

19. The wearable apparatus of claim 14, wherein the determined information transmitted from the wearable apparatus through the short-range communication link to the paired device includes text from a financial document.

20. The wearable apparatus of claim 19, wherein the at least one alert triggered in the paired device in response to the transmission of the determined information includes text from the financial document appearing in a user interface associated with a financial-related application on the paired device.

21. The wearable apparatus of claim 1, wherein the at least one alert triggered in the paired device in response to the transmission of the determined information is associated with at least one of a determined reading speed of the user, a last page read of a book, or progress toward a predetermined reading goal.

22. The wearable apparatus of claim 1, wherein the at least one processing device is further programmed to:

execute at least one machine learning algorithm to determine a likelihood that providing the at least one alert will cause the user wearing the wearable apparatus to perform an action.

23. The wearable apparatus of claim 1, wherein the at least one processing device is further programmed to:

communicate with multiple paired devices; and

select one of the multiple paired devices based on the distance of each paired device from the wearable apparatus.

24. The wearable apparatus of claim 1, wherein causing the paired device to provide at least one alert to the user wearing the wearable apparatus includes causing a display of the paired device to visibly output information.

25. A method for identifying contextual situations related to a user wearing a wearable apparatus, the method comprising:

receiving a plurality of images captured from an environment of the user wearing the wearable apparatus;

analyzing the plurality of images to identify a first contextual situation related to the user wearing the wearable apparatus;

using the plurality of images to determine information associated with the first contextual situation related to the user wearing the wearable apparatus;

causing a device paired with the wearable apparatus through a short-range communication link to provide at least one alert to the user wearing the wearable apparatus based on the determined information associated with the first contextual situation identified in the plurality of images;

after causing the device paired with the wearable apparatus to provide the at least one alert, receiving at least one image captured from an environment of the user wearing the wearable apparatus;

analyzing the at least one image to identify a second contextual situation related to the user wearing the wearable apparatus;

determining a time difference between the first contextual situation and the second contextual situation; and

based on the first contextual situation, image-information associated with the second contextual situation, and a comparison of the determined time difference with a predetermined threshold associated with the first contextual situation, withhold transmission of information that triggers alerts associated with the second contextual situation.

26. The method of claim 25, wherein the second contextual situation is repetitive of the first contextual situation.

27. The method of claim 25, wherein the second contextual situation involves objects associated with the first contextual situation.

28. A non-transitory computer readable medium and comprising data and computer implementable instructions for carrying out a method for identifying contextual situations related to a user wearing a wearable apparatus, the method comprising:

receiving a plurality of images captured from an environment of the user wearing the wearable apparatus;

analyzing the plurality of images to identify a first contextual situation related to the user wearing the wearable apparatus;

using the plurality of images to determine information associated with the first contextual situation related to the user wearing the wearable apparatus;

causing a device paired with the wearable apparatus through a short-range communication link to provide at least one alert to the user wearing the wearable apparatus based on the determined information associated with the first contextual situation identified in the plurality of images;

after causing the device paired with the wearable apparatus to provide the at least one alert, receiving at least one image captured from an environment of the user wearing the wearable apparatus;

analyzing the at least one image to identify a second contextual situation related to the user wearing the wearable apparatus;

determining a time difference between the first contextual situation and the second contextual situation; and

based on the first contextual situation, image-information associated with the second contextual situation, and a comparison of the determined time difference with a predetermined threshold associated with the first contextual situation, withhold transmission of information that triggers alerts associated with the second contextual situation.