Subject sensing in an environment

• US 9,922,256 B2
• Filed: 07/02/2012
• Issued: 03/20/2018
• Est. Priority Date: 06/30/2011
• Status: Active Grant

• Current Assignee: Yale University
• Original Assignee: Yale University
• Inventors: Savvides, Andreas, Teixeira, Thiago
• Primary Examiner(s): Abdi, Amara
• Application Number:
  US14/130,298
• Publication Number:
  US 20140226855A1
• Time in Patent Office: 2,087 Days
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04W 4/021 Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences
• CPC Class Codes: H04W 4/21 for social networking applications

Abstract

A multimodal sensor network is designed to extract a plurality of fundamental properties associated with subject sensing. In one aspect, such network can combine cameras distributed in an environment with inertial sensors available in subjects'"'"' wearable devices. The network can permit anonymous detection, counting, and localization of one or more subjects utilizing the cameras. In one aspect, by fusing such information with positional data from the inertial sensors contained or coupled to wearable devices associated with the one or more subjects, the network can track and can identify each subject carrying a wearable device functionally coupled to inertial sensor(s). In one aspect, the problem of subject sensing can be divided into two parts: (1) a detection, counting, and localization (DCL) layer and (2) a tracking and identification (TI) layer, wherein such layers can be implemented via simulations and a real sensor network deployment.

Claims

1. A method for subject sensing, comprising:

generating a first layer of information associated with sensing of a subject, the first layer of information comprising location information for a plurality of subjects in an environment, wherein the subject is contained in the plurality of subjects;

generating a second layer of information associated with sensing of the subject, the second layer of information comprising tracking information, wherein the tracking information comprises an inertial measurement of the subject;

generating a 2-tuple indicator of a location of the subject that maximizes a likelihood of matching a portion of the first layer of information with a portion of the second layer of information; and

associating, for the subject, identification information with location information based on the 2-tuple indicator.

2. The method of claim 1, wherein generating the first layer of information comprises producing a Boolean image having a value indicative of the likelihood the subject is detected.

3. The method of claim 2, wherein generating the first layer of information further comprises classifying a plurality of pixels in the Boolean image into a set of categories having a cardinality indicative of a number of subjects in the plurality of subjects.

4. The method of claim 3, wherein generating the first layer of information further comprises computing a centroid for each category of the set of categories, each centroid is indicative of a location of each subject in the plurality of subjects.

20. The method of claim 1, wherein the indicator of location of the subject further comprises one or more of identification information for at least one of the plurality of subjects or magnetometer data for at least one of the plurality of subjects.

5. A method for subject sensing, comprising:

detecting a plurality of subjects moving in an environment by a first sensor;

extracting first location information of a subject of the plurality of subjects based at least on the detecting action;

receiving second location information of the subject from a second sensor, wherein the second location information comprises an inertial measurement of the subject;

generating a 2-tuple indicator of a location of the subject that maximizes a likelihood of matching second location information of the subject with the first location information, thereby tracking the subject; and

assigning, based on the generated 2-tuple indicator, a subject location to the subject based on the first location information, and assigning a subject identity (ID) to the subject based on identifying information associated with the second sensor.

6. The method of claim 5, further comprising supplying information indicative of a location-to-ID match comprising the subject location and the subject ID.

7. The method of claim 5, wherein extracting the first location information comprises extracting anonymous coordinates of the person.

8. The method of claim 5, wherein matching the second location information of the subject with the first location information is performed in real-time.

9. The method of claim 5, wherein the first sensor is a camera and the camera is one of a conventional camera or a feature-detecting camera.

10. The method of claim 9, wherein the second sensor is a wearable sensor.

11. The method of claim 5, wherein the second sensor further comprises a radiofrequency identification (RFID) device.

12. A method for subject sensing, comprising:

receiving first location information of a subject of a plurality of subjects, the first location information is generated by first sensor;

generating second location information of the subject, where in the second location information comprises an inertial measurement of the subject;

generating a 2-tuple indicator of a location of the subject that maximizes a likelihood of matching second location information of the subject with the first location information, thereby tracking the subject;

assigning to the subject, in response to generating the 2-tuple indicator, a location based on the 2-tuple indicator, and an identity based on identifying information associated with a second sensor.

13. The method of claim 12, wherein the first sensor is a camera and the camera is one of a conventional camera or a feature-detecting camera.

14. The method of claim 12, wherein the the second sensor is a wearable sensor.

15. The method of claim 12, wherein the second sensor further comprises a radiofrequency identification (RFID) device.

16. A system for subject sensing, comprising:

a plurality of sensors; and

a computing device for information analysis, the computing device functionally coupled to the plurality of sensors and configured to;

transmit first location information of a plurality of subjects to a wearable device, the first location information is generated by at least one of the plurality of sensors,

receive, from the wearable device, second location information associated with a subject of the plurality of subjects, wherein the second location information comprises an inertial measurement of the subject,

generate a 2-tuple indicator of a location of the at least one of the plurality of subjects that maximizes a likelihood of matching the second location information with the first location information and,

assign to the subject, in response to generating the 2-tuple indicator, a subject location based on the 2-tuple indicator, and a subject identity (ID) based on identifying information associated with the wearable device.

17. The system of claim 16, wherein the plurality of sensors comprises a plurality of cameras.

18. The system of claim 16, wherein the wearable device is configured to generate information indicative of a location-to-ID match comprising the subject location and the subject ID.

19. The system of claim 18, wherein the wearable device is configured to supply the information indicative of a location-to-ID match comprising the subject location and the subject ID.

21. The method of claim 16, wherein the wearable device comprises an accelerometer and a wireless transmitter.