Computer vision for ambient data acquisition

• US 10,659,247 B2
• Filed: 07/09/2017
• Issued: 05/19/2020
• Est. Priority Date: 07/09/2016
• Status: Active Grant

• Current Assignee: Grabango Co.
• Original Assignee: Grabango Co.
• Inventors: Glaser, William, Van Osdol, Brian
• Primary Examiner(s): Patel, Nirav G
• Application Number:
  US15/644,801
• Publication Number:
  US 20180012072A1
• Time in Patent Office: 1,045 Days
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 13/196 using television cameras
• CPC Class Codes: H04L 12/2809 indicating that an appliance service is present in a home automation network monitoring functionality H04L43/0817; discovery or management thereof, e.g. service location protocol [SLP] or web services, H04L67/51
• CPC Class Codes: H04L 12/2814 Exchanging control software or macros for controlling appliance services in a home automation network arrangements for maintenance or administration involving configuration of the network and network elements H04L41/08
• CPC Class Codes: H04L 12/282 based on user interaction within the home receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source
• CPC Class Codes: H05B 47/105 in response to determined parameters
• CPC Class Codes: H05B 47/125 by using cameras
• CPC Class Codes: H05B 47/19 via wireless transmission
• CPC Class Codes: H05B 47/1965 using handheld communication devices
• CPC Class Codes: Y02B 20/40 Control techniques providing energy savings, e.g. smart controller or presence detection

Abstract

A system and method for extracting uncoupled information from a user interface output that includes collecting image data; processing the image data associated with the device interface source; and exposing the result to the extracted interface representation.

Claims

1. A method for establishing uncoupled information extraction from a user interface output comprising:

collecting image data wherein the image data is collected by a movable imaging device;

tracking an image collection position;

detecting at least one type of device interface and, upon detection of a type of device interface, automatically setting extraction configuration of a device interface source identified in the image data wherein the extraction configuration sets a device position to map to the image collection position;

processing the image data associated with the device interface source into an extracted interface representation according to the extraction configuration and responsive to when the image collection position corresponds to a configured device position of the device interface source; and

exposing at least one interface to the extracted interface representation.

2. The method of claim 1, wherein setting the extraction configuration of the device interface source comprises selecting at least one of a set of extraction modes for the at least one device interface source.

3. The method claim 2, wherein the set of extraction modes comprises a physical state detection mode, an indicator detection mode, a dial extraction mode, an alphanumeric character extraction mode, and a presence extraction mode.

4. The method of claim 2, wherein the set of extracting modes comprises a physical state detection mode, and wherein processing the image data associated with the device interface source comprises parameterizing the physical state of the device interface source into an extracted interface representation when configured in the physical state detection mode.

5. The method of claim 4, wherein the physical state detection mode is a dial extraction mode, wherein parameterizing visual physical state of the device interface source into an extracted interface representation comprises parameterizing the settings of a one dimensional dial.

6. The method of claim 4, wherein the physical state detection mode is a switch extraction mode, wherein parameterizing visual physical state of the device interface source into an extracted interface representation comprises parameterizing the mechanical state of a switch with at least two mechanically stable states.

7. The method of claim 4, wherein the physical state detection mode is an indicator detection mode, wherein parameterizing visual physical state of the device interface source into an extracted interface representation comprises parameterizing the illumination state of an indicator signal with at least two illumination states.

8. The method of claim 2, wherein the set of extracting modes comprises a character extraction mode, and wherein processing the image data associated with the device interface source comprises recognizing visible characters and converting the visible characters into a data format.

9. The method of claim 2, wherein the set of extracting modes comprises a presence detection mode, and wherein processing the image data associated with the device interface source comprises detecting at least one type of object in a region specified in the extraction configuration when set in the presence detection mode.

10. The method of claim 9, further comprising recording metadata at the time of detecting at least one type of object and associating the metadata with the extracted interface representation.

11. The method of claim 2, wherein the set of extracting modes comprises a gesture detection mode, and wherein processing the image data associated with the device interface source comprises detecting at least one type of gesture event in a region specified in the extraction configuration when set in the gesture detection mode.

12. The method of claim 2, wherein the set of extracting modes comprises a device model extraction mode, and wherein processing the image data associated with the device interface source comprises detecting device state of the device interface source according to a selected device model extraction mode and, in accordance with the device state, extracting a parameterized representation of the device interface source from the image data and characterizing the parameterized representation into a labeled data.

13. The method of claim 1, further comprising detecting a type of device interface source in the image data through object recognition, and automatically setting the extraction configuration the detected device interface source according to the type of device interface source.

14. The method of claim 1, wherein setting the extraction configuration of the device interface source comprises:

initializing a pairing mode,

identifying a device indicator in the image data during the pairing mode, and

associating a device position property of the extraction configuration to a region indicated by the device indicator.

15. The method of claim 1, wherein setting extraction configuration of at least one device interface source comprises receiving manual configuration input through a coupled management application.

16. The method of claim 1, further comprising providing a visual marker and wherein setting the extraction configuration of the device interface source comprises detecting the visual marker in the image data and setting at least part of the extraction configuration based on properties of the visual marker.

17. The method of claim 16, wherein properties of the extraction configuration are associated with the visual marker;

and wherein processing of the device interface source comprises detecting the visual marker in the image data and generating an extracted interface representation using the extraction configuration associated with the visual marker.

18. The method of claim 16, further comprising providing a set of visual markers, wherein combined positioning of the set of visual markers in the image data at least partially define extraction configuration of the device interface source.

19. The method of claim 1, wherein the image data is captured by at least a first imaging device and a second imaging device;

and wherein processing the image data associated with the device interface source into an extracted interface representation further comprises processing image data collected from the first imaging device at a first instance into an extracted interface representation of the device interface source during the first instance and processing image data collected from the second imaging device at a second instance into an extracted interface representation of the device interface source during the second instance.

20. The method of claim 19, wherein a first user account is associated with the first imaging device and a second user account is associated with the second imaging device;

and wherein the second user account is prohibited from accessing the extracted interface representation of the device interface source through an exposed interface.

21. The method of claim 1, setting regional permissions of extraction configuration to a first account for a first region;

and wherein setting extraction configuration of a device interface source is restricted to configuration by the first account.

22. The method of claim 1, further comprising:

setting extraction configuration of at least a second device interface source identified in the image data;

for at least the second device interface source, processing the image data associated with the second device interface source into a second extracted interface representation according to the extraction configuration of the second device interface source, and exposing at least one interface to the extracted interface representation of the second device interface source.

23. The method of claim 1, wherein tracking image collection position comprises collecting positioning coordinates and a device orientation measurement of the movable imaging device.

24. The method of claim 23, wherein tracking image collection position further comprises tracking visual mapping of a scene.

25. The method of claim 1, wherein the movable camera is associated with a first account, and wherein exposing at least one interface to the data permits a second account to access the extracted interface representation and prohibits the first account from reading the extracted interface representation.

26. A method for establishing uncoupled information extraction from a user interface output comprising:

collecting image data wherein the image data is collected by a set of movable imaging devices;

tracking image collection positions of the set of movable imaging devices;

setting extraction configuration of a device interface source identified in the image data, which comprises of setting a device position;

for image data collected by a first movable imaging device associated with a first account, processing the image data associated with the device interface source into an extracted interface representation according to the extraction configuration and wherein processing the image data associated with the device interface source is responsive to when an image collection position of the first movable imaging device corresponds to a configured device position of the device interface source; and

exposing at least one interface to the extracted interface representation wherein exposing at least one interface to the data permits a second account to access the extracted interface representation and prohibits the first account from reading the extracted interface representation.

27. The method of claim 26, wherein setting the extraction configuration of the device interface source comprises selecting at least one of a set of extraction modes for the at least one device interface source.

28. The method claim 27, wherein the set of extraction modes comprises a physical state detection mode, an indicator detection mode, a dial extraction mode, an alphanumeric character extraction mode, and a presence extraction mode.

29. The method of claim 27, wherein the set of extracting modes comprises a physical state detection mode, and wherein processing the image data associated with the device interface source comprises parameterizing the physical state of the device interface source into an extracted interface representation when configured in the physical state detection mode.

30. The method of claim 29, wherein the physical state detection mode is a dial extraction mode, wherein parameterizing visual physical state of the device interface source into an extracted interface representation comprises parameterizing the settings of a one dimensional dial.

31. The method of claim 29, wherein the physical state detection mode is a switch extraction mode, wherein parameterizing visual physical state of the device interface source into an extracted interface representation comprises parameterizing the mechanical state of a switch with at least two mechanically stable states.

32. The method of claim 29, wherein the physical state detection mode is an indicator detection mode, wherein parameterizing visual physical state of the device interface source into an extracted interface representation comprises parameterizing the illumination state of an indicator signal with at least two illumination states.

33. The method of claim 27, wherein the set of extracting modes comprises a character extraction mode, and wherein processing the image data associated with the device interface source comprises recognizing visible characters and converting the visible characters into a data format.

34. The method of claim 27, wherein the set of extracting modes comprises a presence detection mode, and wherein processing the image data associated with the device interface source comprises detecting at least one type of object in a region specified in the extraction configuration when set in the presence detection mode.

35. The method of claim 34, further comprising recording metadata at the time of detecting at least one type of object and associating the metadata with the extracted interface representation.

36. The method of claim 27, wherein the set of extracting modes comprises a gesture detection mode, and wherein processing the image data associated with the device interface source comprises detecting at least one type of gesture event in a region specified in the extraction configuration when set in the gesture detection mode.

37. The method of claim 27, wherein the set of extracting modes comprises a device model extraction mode, and wherein processing the image data associated with the device interface source comprises detecting device state of the device interface source according to a selected device model extraction mode and, in accordance with the device state, extracting a parameterized representation of the device interface source from the image data and characterizing the parameterized representation into a labeled data.

38. The method of claim 26, further comprising detecting a type of device interface source in the image data through object recognition, and automatically setting the extraction configuration the detected device interface source according to the type of device interface source.

39. The method of claim 26, wherein setting the extraction configuration of the device interface source comprises:

initializing a pairing mode,

identifying a device indicator in the image data during the pairing mode, and

associating a device position property of the extraction configuration to a region indicated by the device indicator.

40. The method of claim 26, wherein setting extraction configuration of at least one device interface source comprises receiving manual configuration input through a coupled management application.

41. The method of claim 26, further comprising providing a visual marker and wherein setting the extraction configuration of the device interface source comprises detecting the visual marker in the image data and setting at least part of the extraction configuration based on properties of the visual marker.

42. The method of claim 41, wherein properties of the extraction configuration are associated with the visual marker;

and wherein processing of the device interface source comprises detecting the visual marker in the image data and generating an extracted interface representation using the extraction configuration associated with the visual marker.

43. The method of claim 41, further comprising providing a set of visual markers, wherein combined positioning of the set of visual markers in the image data at least partially define extraction configuration of the device interface source.

44. The method of claim 26, wherein the image data is captured by at least a first imaging device and a second imaging device;

and wherein processing the image data associated with the device interface source into an extracted interface representation further comprises processing image data collected from the first imaging device at a first instance into an extracted interface representation of the device interface source during the first instance and processing image data collected from the second imaging device at a second instance into an extracted interface representation of the device interface source during the second instance.

45. The method of claim 44, wherein a first user account is associated with the first imaging device and a second user account is associated with the second imaging device;

and wherein the second user account is prohibited from accessing the extracted interface representation of the device interface source through an exposed interface.

46. The method of claim 26, setting regional permissions of extraction configuration to a first account for a first region;

and wherein setting extraction configuration of a device interface source is restricted to configuration by the first account.

47. The method of claim 26, further comprising:

setting extraction configuration of at least a second device interface source identified in the image data;

for at least the second device interface source, processing the image data associated with the second device interface source into a second extracted interface representation according to the extraction configuration of the second device interface source, and exposing at least one interface to the extracted interface representation of the second device interface source.

48. The method of claim 26, wherein tracking image collection position comprises collecting positioning coordinates and a device orientation measurement of the movable imaging device.

49. The method of claim 48, wherein tracking image collection position further comprises tracking visual mapping of a scene.

50. The method of claim 26, further comprising detecting at least one type of device interface and automatically setting extraction configuration of a device interface upon detection of a type of device interface, wherein the extracting configuration sets device position to map to the image collection position.