Systems and methods for remembering held items and finding lost items using wearable camera systems

• US 10,298,825 B2
• Filed: 07/23/2015
• Issued: 05/21/2019
• Est. Priority Date: 07/23/2014
• Status: Active Grant

• Current Assignee: OrCam Technologies Ltd
• Original Assignee: OrCam Technologies Ltd
• Inventors: Wexler, Yonaton, Shashua, Amnon
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): Belai, Naod W
• Application Number:
  US14/807,373
• Publication Number:
  US 20160028917A1
• Time in Patent Office: 1,398 Days
• Field of Search: 348169
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/18 Packaging or power distribution
• CPC Class Codes: G06F 1/188 Mounting of power supply units
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/53 Querying
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 : Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/038 : Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06Q 30/0246 : Traffic
• CPC Class Codes: G06Q 30/0257 : User requested
• CPC Class Codes: G06Q 30/0267 : Wireless devices
• CPC Class Codes: G06Q 30/0269 : based on user profile or attribute
• CPC Class Codes: G06T 2207/10004 : Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/30232 : Surveillance
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/74 : involving reference images or patches
• CPC Class Codes: G06T 7/97 : Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 10/776 : Validation; Performance evaluation
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/30 : in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/41 : Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 20/647 : by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 2201/07 : Target detection
• CPC Class Codes: G06V 2201/09 : Recognition of logos
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 1/2112 : using still video cameras
• CPC Class Codes: H04N 23/45 : for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/50 : Constructional details
• CPC Class Codes: H04N 23/51 : Housings
• CPC Class Codes: H04N 23/57 : Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/58 : Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/61 : based on recognised objects
• CPC Class Codes: H04N 23/611 : where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/62 : Control of parameters via user interfaces
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/6812 : based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/73 : by influencing the exposure time
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/44 : Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

Apparatuses and methods are provided for storing information related to objects associated with a hand of a user via a wearable camera system. In one implementation, a wearable apparatus for storing the information is provided comprising a wearable image sensor configured to capture a plurality of images from the environment of the user, and at least one processing device programmed to process the images. The processing device may detect the hand of the user, and an object associated with the user'"'"'s hand. The processing device may proceed to store information related to the object. Consistent with disclosed embodiments, the stored information may be used for various purposes, such as warning the user of dangers, catering advertising to the user, and helping the user find objects when they are lost.

Claims

1. A wearable apparatus for providing information related to objects associated with a hand of a user, the wearable apparatus comprising:

a wearable image sensor configured to capture a plurality of images from an environment of the user; and

at least one processing device programmed to;

process at least one image in the plurality of images to detect the hand of the user in at least one of the plurality of images;

process the at least one image to detect an object that is associated with the hand of the user;

process the at least one image to identify at least one action of the hand associated with the object;

in response to identification of the at least one action of the hand associated with the object;

identify a last known location of the object; and

store information related to the object, the information including the last known location;

subsequently, in response to a query from the user indicating that the object is lost, retrieve the stored information including the last known location related to the object; and

transmit the retrieved information for display.

2. The wearable apparatus of claim 1, wherein the information related to the object includes one or more of:

a time that the object was associated with the hand of the user, a location of the user when the object was associated with the hand of the user, a duration in which the object was associated with the hand of the user, and an identifier of the object.

3. The wearable apparatus of claim 1, wherein the object associated with the hand of the user includes one or more of:

an object that the user holds, an object that the user points to, an object that the user touches, an object that the user drops, an object that the user operates, an object that the user manipulates, and an object that the user grabs.

4. The wearable apparatus of claim 1, further comprising a communications interface, and wherein the at least one processing device is further programmed to cause transmission of the information related to the object, via the communications interface, to a remotely located computing device for inclusion in a catalog of objects associated with the user.

5. The wearable apparatus of claim 4, wherein the at least one processing device is further programmed to identify a type of the object, and the catalog of objects is organized according to types of detected objects.

6. The wearable apparatus of claim 4, wherein the at least one processing device is further programmed to identify the object by comparing an image of the object with stored images.

7. The wearable apparatus of claim 1, wherein the at least one processing device is further programmed to identify that the object is a product.

8. The wearable apparatus of claim 7, wherein the at least one processing device is further programmed to identify a brand associated with the product.

9. The wearable apparatus of claim 1, further comprising a communications interface, and wherein the at least one processing device is further programmed to cause transmission of the information related to the object, via the communications interface, to a remotely located computing device for use in the selection of one or more advertisements.

10. The wearable apparatus of claim 9, wherein the one or more advertisements is for a product related to the at least one object.

11. The wearable apparatus of claim 1, further comprising a communications interface, and wherein the at least one processing device is further programmed to cause transmission of the information related to the object, via the communications interface, to a remotely located computing device for use in the selection of one or more warnings to the user.

12. The wearable apparatus of claim 11, wherein the object is food and the one or more warnings is associated with allergy risks or diet concerns.

13. The wearable apparatus of claim 11, wherein the object is a drug and the one or more warnings is associated with drug side effects.

14. The wearable apparatus of claim 1, wherein the at least one processing device is further programmed to cause storage of one or more recent images of the object and delete one or more older images of the object.

15. The wearable apparatus of claim 1, wherein the at least one action of the hand includes one or more of:

a grabbing action associated with the object, and a dropping action associated with the object.

16. The wearable apparatus of claim 1, wherein the wearable apparatus includes a communications interface, and the at least one processing device is programmed to cause transmission of information related to the last known location of the object to a remotely located computing device.

17. The wearable apparatus of claim 15, wherein the information includes an image showing where the object was last grabbed by the user.

18. The wearable apparatus of claim 15, wherein the information includes an image showing where the object was last dropped by the user.

19. A wearable apparatus for determining a last known location of an object, the wearable apparatus comprising:

a wearable image sensor configured to capture a plurality of images from an environment of the user; and

at least one processing device programmed to;

process at least one image in the plurality of images to detect the hand of the user in at least one of the plurality of images;

process the plurality of images to identify at least one action of a hand associated with the object of interest;

in response to identification of the at least one action of the hand associated with the object of interest;

identify a location associated with the detected image and produce location information related to the location, wherein the location information includes an image showing the last known location of the object of interest; and

store, in a memory, the location information with information associated with the object of interest;

subsequently, in response to a query from the user indicating that the object is lost, retrieve the stored information including the last known location related to the object; and

transmit the retrieved information for display.

20. The wearable apparatus of claim 19, wherein the at least one processing device is further programmed to store at least one recent image of the object of interest, and delete at least one older image of the object of interest.

21. A method for providing information related to objects associated with a hand of a user of a wearable device, the method comprising:

processing a plurality of images captured by a wearable image sensor included in the wearable device to detect the hand of the user in at least one of the plurality of images;

processing at least one of the plurality of images to detect an object associated with the hand of the user;

processing at least one of the plurality of images to identify at least one action of the hand of the user;

in response to identification of the at least one action of the hand of the user;

identifying a last known location of the object associated with the hand of the user; and

storing information related to the object, the information including the last known location;

subsequently, in response to a query from the user indicating that the object is lost, retrieving the stored information including the last known location related to the object; and

transmitting the retrieved information for display.

22. The method of claim 21, further comprising:

transmitting the information related to the object to a remotely located computing device for inclusion in a catalog of objects associated with the user.

23. The method of claim 21, further comprising:

transmitting the information related to the object to a remotely located computing device for use in selecting one or more advertisements for the user.

24. A software product stored on a non-transitory computer readable medium and comprising data and computer implementable instructions for carrying out a method, the method comprising:

processing a plurality of images captured by a wearable image sensor included in the wearable device to detect the hand of the user in at least one of the plurality of images;

processing at least one of the plurality of images to detect an object associated with the hand of the user;

processing at least one of the plurality of images to identify at least one action of the hand of the user;

in response to identification of the at least one action of the hand of the user;

identifying a last known location of the object associated with the hand of the user; and

storing information related to the object, the information including the last known location;

subsequently, in response to a query from the user indicating that the object is lost, retrieving the stored information including the last known location related to the object; and

transmitting the retrieved information for display.