Wearable apparatus and method for selectively categorizing information derived from images

• US 10,432,841 B2
• Filed: 07/23/2015
• Issued: 10/01/2019
• Est. Priority Date: 07/23/2014
• Status: Active Grant

• Current Assignee: OrCam Technologies Ltd
• Original Assignee: OrCam Technologies Ltd
• Inventors: Wexler, Yonatan, Shashua, Amnon
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): Noh, Jae N
• Application Number:
  US14/807,055
• Publication Number:
  US 20160026873A1
• Time in Patent Office: 1,531 Days
• Field of Search: 348158
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/18 Packaging or power distribution
• CPC Class Codes: G06F 1/188 Mounting of power supply units
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/53 Querying
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 : Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/038 : Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06Q 30/0246 : Traffic
• CPC Class Codes: G06Q 30/0257 : User requested
• CPC Class Codes: G06Q 30/0267 : Wireless devices
• CPC Class Codes: G06Q 30/0269 : based on user profile or attribute
• CPC Class Codes: G06T 2207/10004 : Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/30232 : Surveillance
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/74 : involving reference images or patches
• CPC Class Codes: G06T 7/97 : Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 10/776 : Validation; Performance evaluation
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/30 : in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/41 : Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 20/647 : by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 2201/07 : Target detection
• CPC Class Codes: G06V 2201/09 : Recognition of logos
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 1/2112 : using still video cameras
• CPC Class Codes: H04N 23/45 : for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/50 : Constructional details
• CPC Class Codes: H04N 23/51 : Housings
• CPC Class Codes: H04N 23/57 : Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/58 : Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/61 : based on recognised objects
• CPC Class Codes: H04N 23/611 : where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/62 : Control of parameters via user interfaces
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/6812 : based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/73 : by influencing the exposure time
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/44 : Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

A wearable apparatus and method are provided for selectively categorizing information derived from images. In one implementation, a wearable apparatus for collecting information related to activities of a user may collect information related to activities of a user. The wearable apparatus may include an image sensor configured to capture a plurality of images from an environment of a user; a communications interface; and at least one processing device. The at least one processing device may be programmed to process the plurality of images to identify an activity occurring in the environment of the user; associate the activity with an activity category; and cause transmission of at least the activity category to a remotely located computing device via the communications interface.

Claims

1. A wearable apparatus for collecting information related to activities of a user, the wearable apparatus comprising:

an image sensor configured to capture a plurality of images from an environment of the user;

a communications interface; and

at least one processing device programmed to;

process the plurality of images to identify an activity occurring in the environment of the user;

associate the activity with an activity category;

determine, based on the plurality of images, a level of interest of the user in the activity category, wherein the level of interest is based, at least in part, on a duration of the activity;

cause transmission of at least the activity category to a remotely located computing device via the communications interface; and

cause a life log to be stored in memory, the life log including information comprising at least part of at least one of the plurality of images depicting the activity and the activity category associated with the at least one of the plurality of images, wherein the information stored in the life log is selectively included based on at least the level of interest of the user in the activity category exceeding a predetermined threshold.

2. The wearable apparatus of claim 1, wherein the at least one processing device is further programmed to determine a location at which the activity occurred and to transmit the location to the computing device via the communications interface.

3. The wearable apparatus of claim 1, wherein the at least one processing device is further programmed to transmit at least one of the plurality of images depicting the activity to the computing device.

4. The wearable apparatus of claim 3, wherein the at least one processing device is further programmed to cause the computing device to display at least one of the plurality of images depicting the activity in association with the activity category.

5. The wearable apparatus of claim 1, wherein the at least one processing device is further programmed to determine an elapsed time of the activity, and the at least one processing device is further programmed to transmit the elapsed time to the computing device.

6. The wearable apparatus of claim 5, wherein the at least one processing device is further programmed to determine an interest level of the user regarding the activity category.

7. The wearable apparatus of claim 1, wherein the at least one processing device is further programmed to determine a number of activities associated with one or more activity categories that were performed by the user over a time period.

8. The wearable apparatus of claim 1, wherein the at least one processing device is further programmed to select a recommendation for the user based on at least the activity category.

9. The wearable apparatus of claim 1, wherein the at least one processing device is further programmed to select at least one of an advertisement and a coupon for presenting to the user based on at least the activity category.

10. The wearable apparatus of claim 1, wherein the at least one processing device is further programmed to receive at least one of an advertisement and a coupon for presenting to the user based on at least the activity category.

11. The wearable apparatus of claim 1, wherein the at least one processing device is further programmed to determine an interest level of the user related to one or more activity categories.

12. The wearable apparatus of claim 11, wherein the at least one processing device is further programmed to cause the computing device to display information regarding activities associated with one or more activity categories for which the user is determined to have at least a certain level of interest.

13. The wearable apparatus of claim 11, wherein the at least one processing device is further programmed to cause the computing device to omit information from the life log regarding activities associated with one or more activity categories for which the user is determined to have less than a certain level of interest.

14. The wearable apparatus of claim 11, wherein the at least one processing device is further programmed to cause the computing device to include or exclude information from the life log regarding activities associated with one or more activity categories based on the determined interest level.

15. The wearable apparatus of claim 1, wherein the at least one processing device is further programmed to determine a preference of the user related to the activity.

16. The wearable apparatus of claim 15, wherein determination of the preference of the user related to the activity is based on whether the user deleted information related to the activity.

17. The wearable apparatus of claim 15, wherein determination of the preference of the user related to the activity is based on whether the user flagged information related to the activity as being not of interest.

18. The wearable apparatus of claim 15, wherein the at least one processing device is further programmed to cause the computing device to delete information related to the activity category based on the preference of the user.

19. The wearable apparatus of claim 12, wherein the at least one processing device is further programmed to cause the computing device to include information in the life log related to a child depicted in one or more of the plurality of images and under the user'"'"'s supervision.

20. The wearable apparatus of claim 12, wherein the at least one processing device is further programmed to cause the computing device to include information in the life log related to images that include text.

21. A wearable apparatus for collecting information related to activities of a user, the wearable apparatus comprising:

an image sensor configured to capture a plurality of images from an environment of the user;

a communications interface; and

at least one processing device programmed to;

process the plurality of images to identify an activity occurring in the environment of the user, wherein the activity is associated with an activity category;

access profile information related to the user to obtain a level of interest of the user in the activity category, wherein the level of interest is based, at least in part, on a duration of the activity determined thorough analysis of the plurality of images;

determine, based on at least the level of interest of the user in the activity category exceeding a predetermined threshold, that images of the activity are to be included in a life log stored in memory; and

based on the determination, transmit at least part of at least one of the plurality of images of the activity to a remotely located computing device via the communications interface for inclusion in the life log.

22. The wearable apparatus of claim 21, wherein the at least one processing device is programmed to select one or more coupons for the user based on the activity and transmit one or more identifiers of the one or more coupons to the computing device.

23. The wearable apparatus of claim 21, wherein the at least one processing device is programmed to select one or more advertisements for the user based on the activity and transmit one or more identifiers of the one or more advertisements to the computing device.

24. A method for collecting information related to activities of a user, the method comprising:

capturing, via an image sensor included in a wearable apparatus, a plurality of images from an environment a user of the wearable apparatus;

processing the plurality of images to identify an activity occurring in the environment of the user;

associating the activity with an activity category;

determining, based on the plurality of images, a level of interest of the user in the activity category, wherein the level of interest is based, at least in part, on a duration of the activity;

transmitting at least the activity category to a remotely located computing device; and

causing a life log to be stored in memory, the life log including information comprising at least part of at least one of the plurality of images depicting the activity and the activity category associated with the at least one of the plurality of images, wherein the information stored in the life log is selectively included based on at least the level of interest of the user in the activity category exceeding a predetermined threshold.

25. A non-transitory computer readable medium storing computer implementable instructions for carrying out the method of claim 24.