System and method for adaptive, rapidly deployable, human-intelligent sensor feeds

• US 10,657,385 B2
• Filed: 03/25/2016
• Issued: 05/19/2020
• Est. Priority Date: 03/25/2015
• Status: Active Grant

• Current Assignee: Carnegie Mellon University, University of Rochester
• Original Assignee: Carnegie Mellon University
• Inventors: Laput, Gierad, Harrison, Christopher, Bigham, Jeffrey P., Lasecki, Walter S., Xiao, Bo Robert, Wiese, Jason
• Primary Examiner(s): Goradia, Shefali D
• Application Number:
  US15/561,407
• Publication Number:
  US 20180107879A1
• Time in Patent Office: 1,516 Days
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 18/2155 characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 5/041 Abduction
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06V 10/7753 Incorporation of unlabelled data, e.g. multiple instance learning [MIL]
• CPC Class Codes: G06V 10/776 Validation; Performance evaluation
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 10/993 Evaluation of the quality of the acquired pattern
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

The disclosure describes a sensor system that provides end users with intelligent sensing capabilities, and embodies both crowd sourcing and machine learning together. Further, a sporadic crowd assessment is used to ensure continued sensor accuracy when the system is relying on machine learning analysis. This sensor approach requires minimal and non-permanent sensor installation by utilizing any device with a camera as a sensor host, and provides human-centered and actionable sensor output.

Claims

1. An adaptive sensor system comprising:

a sensor, wherein the sensor provides a sensor feed of an environment;

a user interface,

wherein a user can enter a question to be answered by the sensor system using the user interface,

wherein the question relates to a condition of the environment;

a dispatcher, wherein the dispatcher directs the sensor feed and question to at least one of a crowd module and a machine learning module,

wherein at least one of the crowd module and the machine learning module generates a labeled sensor feed derived from the sensor feed,

wherein the labeled sensor feed comprises an answer to the question; and

a periodic comparator that evaluates a relative accuracy of the labeled sensor feed generated by the machine learning module as compared to the labeled sensor feed generated by the crowd module.

2. The adaptive sensor system of claim 1, wherein the dispatcher directs the sensor feed to the crowd module upon initiation of the sensor system and prior to the machine learning module reaching a threshold level of accuracy.

3. The adaptive sensor system of claim 2,

wherein the labeled sensor feeds generated by the crowd module are sent to the machine learning module,

wherein the machine learning module uses the labeled sensor feeds to train a classifier.

4. The adaptive sensor system of claim 3, wherein the dispatcher directs the sensor feed to the machine learning module when the machine learning module is operating above a threshold level of accuracy.

5. The adaptive sensor system of claim 1, wherein the crowd module sends the sensor feed and question to crowd workers to generate the labeled sensor feed.

6. The adaptive sensor system of claim 1, wherein the user interface provides a visualization of the labeled sensor feed.

7. The adaptive sensor system of claim 1, wherein the sensor comprises an electronic device having a camera.

8. The adaptive sensor system of claim 1, wherein the sensor feed comprises an image.

9. The adaptive sensor system of claim 1, wherein the question has a data type selected from the group consisting of:

yes/no, number, scale, and multiple choice.

10. The adaptive sensor system of claim 1, further comprising :

an image similarity detector that receives the sensor feed, wherein the image similarity detector maintains a current sensor value if there is no change in the condition of the environment over a period of time.

11. A method of creating an adaptive sensor system, comprising:

providing a sensor that provides a sensor feed;

receiving from a user interface a question, wherein the question is capable of being

answered by observation of the sensor feed;

sending the sensor feed to a crowd,

wherein the crowd generates a labeled sensor feed based on the sensor feed;

using the crowd labeled sensor feed to train a machine learning module;

sending a subsequent sensor feed to the machine learning module to generate the labeled sensor feed when the machine learning module exceeds a threshold level of accuracy; and

sending the labeled sensor feed to a user via the user interface wherein the labeled sensor feed comprises an answer to the question.

12. The method of claim 11, further comprising:

periodically testing the labeled sensor feed generated by the machine learning module against the labeled sensor feed generated by the crowd to determine the relative accuracy of the machine learning module.

13. The method of claim 12, wherein the sensor feed is sent to the crowd if the machine learning module falls below the threshold level of accuracy.

14. The method of claim 11, further comprising:

choosing a region of interest in the sensor feed prior to sending the sensor feed to the crowd.

15. The method of claim 11, further comprising:

obfuscating the sensor feed.

16. The method of claim 11, further comprising:

rejecting a second sensor feed received subsequent to a first sensor feed if there is no difference between the first sensor feed and the second sensor feed.

17. The method of claim 16, wherein the second sensor feed is rejected if a pixel difference threshold does not exceed 10%.

18. The method of claim 16, wherein the second sensor feed is rejected if an image area percentage threshold does not exceed 1%.

19. The method of claim 11, wherein sending the sensor feed to a crowd to generate a labeled sensor feed further comprises:

sending an example sensor feed to the crowd.