Systems and methods for monitoring objects and their states by using acoustic signals

• US 10,185,766 B2
• Filed: 01/15/2016
• Issued: 01/22/2019
• Est. Priority Date: 01/15/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Tannenbaum, Harry, Irvine, Benjamin, Sayadi, Shayan, Singer, James Vanhook
• Primary Examiner(s): Le, Hung D
• Application Number:
  US14/997,456
• Publication Number:
  US 20170206273A1
• Time in Patent Office: 1,103 Days
• Field of Search: None
• CPC Class Codes: G06F 16/61 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/686 using information manually generated, e.g. tags, keywords, comments, title or artist information, time, location or usage information, user ratings

Abstract

A computer system is communicably coupled to one or more sensor devices. The computer system obtains a database of stored acoustic signatures characterizing predefined acoustic signals generated by passive tags in response to physical motion of respective monitored objects associated with the passive tags. A first acoustic signal characterized by a respective acoustic signature and generated by a first passive tag is detected. In response, and based on the respective acoustic signature and information in the database, a first monitored object associated with the respective acoustic signature is identified, and a first state of the first monitored object is determined. The determined first state is stored in the database, and an indication of the first state is provided to an associated monitoring service.

Claims

1. A method of detecting a state of monitored objects, comprising:

at a computer system having one or more processors and memory storing instructions for execution by the one or more processors, wherein the computer system is communicably coupled to one or more sensor devices;

obtaining a database of stored acoustic signatures characterizing predefined acoustic signals generated by electrically non-powered passive tags in response to physical motion of respective monitored objects associated with the passive tags;

receiving a first acoustic signal characterized by a respective acoustic signature and generated by a first passive tag of the passive tags in response to physical motion of a first monitored object of the respective monitored objects, wherein the first monitored object is associated with the first passive tag; and

in response to the receiving;

based on the respective acoustic signature and information in the database;

identifying the first monitored object associated with the respective acoustic signature; and

determining a first state of a plurality of predefined states of the first monitored object;

storing in the database the determined first state of the first monitored object; and

providing an indication of the first state of the first monitored object to a monitoring service associated with the one or more monitored objects.

2. The method of claim 1, wherein the first passive tag comprises a material configured to generate the first acoustic signal in response to physical contact when physically moved in a predetermined manner.

3. The method of claim 1, wherein at least some of the respective monitored objects associated with the passive tags are entryways of a premises.

4. The method of claim 1, wherein identifying the first monitored object and determining the first state of the first monitored object comprises:

comparing one or more respective characteristics of the respective acoustic signature of the first acoustic signal with one or more respective characteristics of one or more of the stored acoustic signatures from the database; and

identifying, from the one or more stored acoustic signatures from the database, a matching acoustic signature that satisfies a match criterion, wherein the matching acoustic signature is associated with the first monitored object and the first state of the first monitored object.

5. The method of claim 1, wherein storing in the database the determined first state of the first monitored object comprises updating a current state of the first monitored object from a previously stored state to the determined first state,

wherein the determined first state and the previously stored state represent distinct states of the first monitored object.

6. The method of claim 5, further comprising:

detecting a second acoustic signal characterized by the same respective acoustic signature as the first acoustic signal, and generated by the first passive tag;

in response to detecting the second acoustic signal, and based on the respective acoustic signature and information in the database, identifying the first monitored object associated with the respective acoustic signature; and

updating the current state of the first monitored object from the first state to the previously stored state.

7. The method of claim 1, wherein identifying the first monitored object and/or determining the first state of the first monitored object is further based on one or more additional inputs received from the one or more sensor devices communicably coupled to the computer system.

8. The method of claim 7, wherein the one or more sensor devices include a camera device, and the one or more additional inputs include video data associated with the detected first acoustic signal.

9. The method of claim 7, wherein the one or more additional inputs include current presence detection data associated with the detected first acoustic signal.

10. The method of claim 7, wherein the one or more sensor devices include a thermostat device, and the one or more additional inputs include temperature data associated with the detected first acoustic signal.

11. The method of claim 7, wherein the one or more sensor devices include an acoustic sensor, and the one or more additional inputs include additional acoustic data associated with the detected first acoustic signal.

12. The method of claim 7, wherein the one or more additional inputs include a user input confirming the first state of the first monitored object.

13. The method of claim 1, wherein:

the one or more sensor devices are located within a premises, and the computer system is a remote server located outside of the premises; and

detecting the first acoustic signal comprises receiving, at the computer system and from the one or more sensor devices, the first acoustic signal.

14. The method of claim 1, wherein acoustic signatures, which include the stored acoustic signatures in the database and the respective acoustic signature of the first acoustic signal, are respectively defined by one or more characteristics of acoustic signatures, the characteristics including at least one of:

an acoustic signature length;

one or more amplitude values; and

one or more spectral coefficients for one or more respective frequencies.

15. The method of claim 14, wherein at least some of the stored acoustic signatures are defined by one or more of the characteristics excluding amplitude values.

16. The method of claim 14, wherein each or any combination of the one or more characteristics respectively defining the stored acoustic signatures is associated with:

a respective object, a type of object, a state or change of state for a respective object, a state or change of state for a respective type of object, and/or a location.

17. The method of claim 16, wherein a first one of the stored acoustic signatures is defined by respective one or more characteristics, the respective one or more characteristics being uniquely associated with a particular object and a particular state of the particular object.

18. The method of claim 1, further comprising, prior to obtaining the database:

creating one or more entries of the database, including;

detecting the predefined acoustic signals, each of the predefined acoustic signals characterized by a respective stored acoustic signature, and generated by a respective passive tag in response to physical motion of a respective object associated with the respective passive tag;

for each predefined acoustic signal that was detected, associating the respective stored acoustic signature with;

the respective object; and

a state of the respective object resulting from the physical motion of the respective object.

19. A method of detecting a state of monitored objects, comprising:

at a sensor device having one or more first processors and first memory storing instructions for execution by the one or more first processors, wherein sensor device is communicably coupled to a remote server;

detecting a first acoustic signal characterized by a respective acoustic signature and generated by a first electrically non-powered passive tag of a plurality of electrically non-powered passive tags; and

at the remote server having one or more second processors and second memory storing instructions for execution by the one or more second processors;

receiving a database of stored acoustic signatures characterizing predefined acoustic signals generated by the plurality of passive tags in response to physical motion of respective monitored objects associated with the plurality of passive tags, wherein the respective monitored objects include a first monitored object associated with the first passive tag; and

in response to the detection of the first acoustic signal by the sensor device;

based on the respective acoustic signature and information in the database;

identifying the first monitored object associated with the respective acoustic signature; and

determining a first state of a plurality of defined states of the first monitored object;

storing in the database the determined first state of the first monitored object; and

providing an indication of the first state of the first monitored object to a monitoring service associated with the one or more monitored objects.

20. A non-transitory computer-readable storage medium storing one or more programs for execution by one or more processors of a computer system, the one or more programs including instructions for:

obtaining a database of stored acoustic signatures characterizing predefined acoustic signals generated by electrically non-powered passive tags in response to physical motion of respective monitored objects associated with the passive tags;

receiving a first acoustic signal characterized by a respective acoustic signature and generated by a first passive tag of the passive tags in response to physical motion of a first monitored object of the respective monitored objects, wherein the first monitored object is associated with the first passive tag; and

in response to the receiving;

based on the respective acoustic signature and information in the database;

identifying the first monitored object associated with the respective acoustic signature; and

determining a first state of a plurality of predefined states of the first monitored object;

storing in the database the determined first state of the first monitored object; and

providing an indication of the first state of the first monitored object to a monitoring service associated with the one or more monitored objects.