Wearable apparatus and method for processing images including product descriptors

• US 10,367,985 B2
• Filed: 07/23/2015
• Issued: 07/30/2019
• Est. Priority Date: 07/23/2014
• Status: Active Grant

• Current Assignee: OrCam Technologies Ltd
• Original Assignee: OrCam Technologies Ltd
• Inventors: Wexler, Yonatan, Shashua, Amnon
• Primary Examiner(s): Moyer, Andrew M
• Application Number:
  US14/807,580
• Publication Number:
  US 20160026868A1
• Time in Patent Office: 1,468 Days
• Field of Search: 382103
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/18 Packaging or power distribution
• CPC Class Codes: G06F 1/188 Mounting of power supply units
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/53 Querying
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 : Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/038 : Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06Q 30/0246 : Traffic
• CPC Class Codes: G06Q 30/0257 : User requested
• CPC Class Codes: G06Q 30/0267 : Wireless devices
• CPC Class Codes: G06Q 30/0269 : based on user profile or attribute
• CPC Class Codes: G06T 2207/10004 : Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/30232 : Surveillance
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/74 : involving reference images or patches
• CPC Class Codes: G06T 7/97 : Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 10/776 : Validation; Performance evaluation
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/30 : in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/41 : Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 20/647 : by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 2201/07 : Target detection
• CPC Class Codes: G06V 2201/09 : Recognition of logos
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 1/2112 : using still video cameras
• CPC Class Codes: H04N 23/45 : for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/50 : Constructional details
• CPC Class Codes: H04N 23/51 : Housings
• CPC Class Codes: H04N 23/57 : Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/58 : Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/61 : based on recognised objects
• CPC Class Codes: H04N 23/611 : where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/62 : Control of parameters via user interfaces
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/6812 : based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/73 : by influencing the exposure time
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/44 : Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

A wearable apparatus and method are provided for processing images including product descriptors. In one implementation, a wearable apparatus for processing images including a product descriptor is provided. The wearable apparatus includes a wearable image sensor configured to capture a plurality of images from an environment of a user of the wearable apparatus. The wearable apparatus also includes at least one processing device programmed to analyze the plurality of images to identify one or more of the plurality of images that include an occurrence of the product descriptor. Based on analysis of the one or more identified images, the at least one processing device is also programmed to determine information related to the occurrence of the product descriptor. The at least one processing device is further configured to cause the information and an identifier of the product descriptor to be stored in a memory.

Claims

1. A wearable apparatus for processing images including a product descriptor, the wearable apparatus comprising:

a wearable image sensor configured to capture a plurality of images from an environment of a user of the wearable apparatus; and

at least one processor programmed to;

analyze the plurality of images to identify one or more of the plurality of images that include an occurrence of the product descriptor;

determine statistical information associated with the product descriptor including a purchase history or a preference related to the product;

based on analysis of the one or more identified images, determine contextual information associated with the product descriptor, wherein the contextual information includes information about a person in the environment of the product descriptor; and

transmit the contextual information and an identifier of the product descriptor to a server.

2. The wearable apparatus of claim 1, wherein the product descriptor includes one or more of:

a logo, an advertisement, an image of the product, and an image of a store.

3. The wearable apparatus of claim 1, wherein the at least one processor is further programmed to:

identify, in one or more of the identified images, a graphic; and

compare the identified graphic with a plurality of stored representations of known graphics; and

select feedback to be provided to the server based on finding a match between the identified graphic and a representation of a known graphic.

4. The wearable apparatus of claim 3, wherein the feedback includes at least one of audible feedback and visual feedback.

5. The wearable apparatus of claim 1, wherein the at least one processor is further programmed to:

identify, in one or more of the identified images, a graphic;

access a database of stored graphics;

compare the identified graphic to the stored graphics; and

trigger execution of an action based on whether a match is found between the identified graphic and a stored graphic in the database.

6. The wearable apparatus of claim 5, wherein the identified graphic includes at least one of a printed image and text.

7. The wearable apparatus of claim 5, wherein the identified graphic includes a logo associated with a brand of the product.

8. The wearable apparatus of claim 5, wherein the action includes storing information including at least one of a location and time at which the graphic was identified.

9. The wearable apparatus of claim 5, further comprising a communications interface, wherein the communications interface is configured to transmit information indicating that the graphic was identified to the server when the action is executed.

10. The wearable apparatus of claim 5, wherein the action includes identifying objects near the identified graphic.

11. The wearable apparatus of claim 5, wherein the identified graphic includes a directional symbol and the action includes providing feedback to the user that includes directing the user in accordance with the directional symbol.

12. The wearable apparatus of claim 5, wherein the action includes providing feedback to the server, and the feedback includes stored information associated with the identified graphic.

13. The wearable apparatus of claim 1, wherein the statistical information includes one or more of:

a continuous duration in which occurrences of the product descriptor are identified in the plurality of images, a total time in which occurrences of the product descriptor are identified in the plurality of images, a total number of images in which occurrences of the product descriptor are identified, and a frequency of images in which occurrences of the product descriptor are identified.

14. The wearable apparatus of claim 1, wherein the contextual information about the person in the environment of the product descriptor includes an age, a gender, a size, a weight, or a height of the person.

15. The wearable apparatus of claim 14, wherein the contextual information about the person includes at least one of an age or a gender of the person, and the age or the gender are determined based on a posture of the person.

16. The wearable apparatus of claim 1, wherein the contextual information further includes a location associated with the product descriptor.

17. The wearable apparatus of claim 1, wherein the contextual information further includes a time or a date when at least one image including the product descriptor was captured.

18. A wearable apparatus for processing images including a product descriptor, the wearable apparatus comprising:

a wearable image sensor configured to capture a plurality of images from an environment of a user of the wearable apparatus; and

at least one processor programmed to;

analyze the plurality of images to identify one or more of the plurality of images that include an occurrence of the product descriptor;

identify, in one or more of the identified images, one or more graphics included in the product descriptor;

access a database of stored graphics;

compare the identified one or more graphics associated with the product descriptor, wherein the contextual information includes information about a person in the environment of the product descriptor; and

transmit the contextual information and an identifier of the product descriptor to a server.

19. The wearable apparatus of claim 18, wherein the identified one or more graphics include at least one of a printed image and text.

20. The wearable apparatus of claim 18, wherein the identified one or more graphics include a logo associated with a brand of the product.

21. The wearable apparatus of claim 18, wherein the at least one processor is further programmed to store information including at least one of a location and time at which the one or more graphics were identified.

22. The wearable apparatus of claim 18, further comprising a communications interface, wherein the communications interface is configured to transmit information indicating that the one or more graphics were identified to the server.

23. The wearable apparatus of claim 18, wherein the at least one processor is further programmed to identify objects near the one or more identified graphics.

24. The wearable apparatus of claim 18, wherein the one or more identified graphics include a directional symbol and the at least one processor is also programmed to provide feedback to the server that includes directing a user in accordance with the directional symbol.

25. The wearable apparatus of claim 18, wherein the at least one processor is also programmed to provide feedback to the server, and the feedback includes presenting to a user stored information associated with the one or more identified graphics.

26. The wearable apparatus of claim 18, wherein the contextual information about the person in the environment of the product descriptor includes an age, a gender, a size, a weight, or a height of the person.

27. The wearable apparatus of claim 26, wherein the contextual information about the person includes at least one of an age or a gender of the person, and the age or the gender are determined based on a posture of the person.

28. The wearable apparatus of claim 18, wherein the contextual information further includes a location associated with the product descriptor.

29. The wearable apparatus of claim 18, wherein the contextual information further includes a time or a date when at least one image including the product descriptor was captured.

30. A method for processing images including a product descriptor, the method comprising:

capturing, via a wearable image sensor, a plurality of images from an environment of a user of a wearable device including the wearable image sensor;

analyzing the plurality of images to identify one or more of the plurality of images that include an occurrence of the product descriptor;

determining statistical information associated with the product descriptor including a purchase history or a preference related to the product;

based on the one or more identified images, determining contextual information associated with the product descriptor, wherein the contextual information includes information about a person in the environment of the product descriptor; and

transmitting the contextual information and an identifier of the product descriptor to a server.

31. A software product stored on a non-transitory computer readable medium and comprising data and computer implemented instructions for carrying out the method of claim 30.

32. The method of claim 30, wherein the contextual information about the person in the environment of the product descriptor includes an age, a gender, a size, a weight, or a height of the person.

33. The method of claim 32, wherein the contextual information about the person includes at least one of an age or a gender of the person, and the age or the gender are determined based on a posture of the person.

34. The method of claim 30, wherein the contextual information further includes a location associated with the product descriptor.

35. The method of claim 30, wherein the contextual information further includes a time or a date when at least one image including the product descriptor was captured.

36. A method for processing images including a product descriptor, the method comprising:

capturing, via a wearable image sensor, a plurality of images from an environment of a user of a wearable device including the wearable image sensor;

analyzing the plurality of images to identify one or more of the plurality of images that include an occurrence of the product descriptor;

identifying, in one or more of the identified images, one or more graphics included in the product descriptor;

accessing a database of stored graphics;

comparing the identified one or more graphics to the stored graphics to determine contextual information associated with the product descriptor, wherein the contextual information includes information about a person in the environment of the product descriptor; and

transmitting the contextual-information and an identifier of the product descriptor to a server.

37. A non-transitory computer readable medium storing computer implemented instructions for carrying out the method of claim 36.

38. The method of claim 36, wherein the contextual information about the person in the environment of the product descriptor includes an age, a gender, a size, a weight, or a height of the person.

39. The method of claim 38, wherein the contextual information about the person includes at least one of an age or a gender of the person, and the age or the gender are determined based on a posture of the person.

40. The method of claim 36, wherein the contextual information further includes a location associated with the product descriptor.

41. The method of claim 36, wherein the contextual information further includes a time or a date when at least one image including the product descriptor was captured.