Systems and methods for cookware detection

• US 9,449,220 B2
• Filed: 04/21/2014
• Issued: 09/20/2016
• Est. Priority Date: 04/21/2014
• Status: Active Grant

• Current Assignee: Haier US Appliance Solutions, Inc. DBA GE Appliances (Haier Group Company)
• Original Assignee: Haier US Appliance Solutions, Inc. DBA GE Appliances (Haier Group Company)
• Inventors: Guan, Li
• Primary Examiner(s): Patel, Jayesh A
• Application Number:
  US14/257,267
• Publication Number:
  US 20150302276A1
• Time in Patent Office: 883 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 30/1914 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries, e.g. user dictionaries
• CPC Class Codes: G06V 30/2504 Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Systems and methods for cookware detection are provided. One example system includes a vision sensor positioned so as to collect a plurality of images of a cooktop. The system includes a classifier module implemented by one or more processors. The classifier module is configured to calculate a cookware score for each of the plurality of images and to use the cookware score for each of the plurality of images to classify such image as either depicting cookware or not depicting cookware. The system includes a classifier training module implemented by the one or more processors. The classifier training module is configured to train the classifier module based at least in part on a positive image training dataset and a negative image training dataset.

Claims

1. A method for detecting cookware, the method comprising:

identifying, by one or more computing devices, one or more locations respectively associated with one or more burners included in a cooktop, wherein identifying, by one or more computing devices, the one or more locations respectively associated with the one or more burners comprises;

obtaining, by the one or more computing devices, a reference frame of imagery depicting the cooktop without any objects placed thereon;

obtaining, by the one or more computing devices, one or more calibration frames of imagery depicting the cooktop with one or more items of cookware respectively positioned at the one or more locations respectively associated with the one or more burners, wherein the one or more calibration frames of imagery are captured when motion is not detected at the cooktop;

performing, by the one or more computing devices, background subtraction for at least one of the one or more calibration frames of imagery with respect to the reference frame of imagery to identify new imagery; and

segmenting, by the one or more computing devices, the new imagery to identify the one or more locations respectively associated with the one or more burners included in the cooktop;

obtaining, by the one or more computing devices, a frame of imagery captured by a vision sensor, wherein the frame of imagery either depicts cookware on the cooktop, or does not depict cookware on the cooktop;

segmenting, by the one or more computing devices, the frame of imagery into one or more image segments based at least in part on the one or more locations respectively associated with the one or more burners included in the cooktop;

using, by the one or more computing devices, a classifier to provide an initial classification for each of the one or more image segments of the frame of imagery, the initial classification for each of the one or more image segments classifying the one or more image segments into either a first class of images depicting cookware or a second class of images not depicting cookware, wherein classification of one of the one or more image segments into the first class of images corresponds to detection of cookware on the cooktop;

providing, by the one or more computing devices for at least one of the one or more image segments, an indication to a user of whether the initial classification for such image segment comprises the first class or the second class;

after providing the indication for at least one of the one or more image segments, receiving, by the one or more computing devices, a user input indicating whether the initial classification for such image segment is correct; and

when the user input indicates that the initial classification for one of the one or more image segments is not correct, changing, by the one or more computing devices, the initial classification for such image segment to a subsequent classification, wherein the subsequent classification comprises the second class when the initial classification comprises the first class, and wherein the subsequent classification comprises the first class when the initial classification comprises the second class.

2. The method of claim 1, wherein:

the first class of images comprises a positive image training dataset upon which the classifier was trained, the positive image training dataset comprising Images that depict cookware;

the second class of images comprises a negative image training dataset upon which the classifier was trained, the negative image training dataset comprising images that do not depict cookware; and

the method further comprises;

adding, by the one or more computing devices, each of the one or more image segments for which the subsequent classification comprises the first class to the positive image training dataset; and

adding, by the one or more computing devices, each of the one or more image segments for which the subsequent classification comprises the second class to the negative image training dataset.

3. The method of claim 2, wherein the method further comprises, after adding each of the one or more image segments to either the positive image training dataset or the negative image training dataset, re-training the classifier based at least in part on the positive image training dataset and the negative image training dataset.

4. The method of claim 3, wherein:

the classifier provides the initial classification for each of the one or more image segments based on a plurality of planes respectively associated with a plurality of image features; and

re-training the classifier based at least in part on the positive image training dataset and the negative image training dataset comprises recomputing the plurality of planes.

5. The method of claim 1, wherein receiving, by the one or more computing devices, the user input indicating whether the initial classification is correct comprises:

after providing the indication, pausing, by the one or more computing devices, for a response period of time;

interpreting, by the one or more computing devices, a receipt of user input during the response period of time as indicating that the initial classification is not correct; and

interpreting, by the one or more computing devices, an absence of user input during the response period of time as indicating that the initial classification is correct.

6. The method of claim 1, wherein receiving, by the one or more computing devices, a user input comprises receiving, by the one or more computing devices, data describing a voice command provided by the user, data describing one or more buttons pressed by the user, or data describing a gesture performed by the user.

7. The method of claim 1, further comprising, prior to using, by the one or more computing devices, the classifier to provide the initial classification for each of the one or more image segments:

obtaining, by the one or more computing devices, a first plurality of images captured by the vision sensor, wherein the first plurality of images depict cookware upon the cooktop;

storing, by the one or more computing devices, the first plurality of images in a memory as a positive image training dataset;

obtaining, by the one or more computing devices, a second plurality of images captured by the vision sensor, wherein the second plurality of images do not depict cookware upon the cooktop;

storing, by the one or more computing devices, the second plurality of images in the memory as a negative image training dataset; and

training, by the one or more computing devices, the classifier based at least in part on the positive image training dataset and the negative image training dataset.

8. A system for detecting cookware, the system comprising:

a vision sensor positioned so as to collect imagery depicting a cooktop;

one or more processors; and

one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising;

obtaining a first plurality of images, wherein the first plurality of images depict cookware upon the cooktop;

storing the first plurality of images in a memory as a positive image training

dataset;

obtaining a second plurality of images, wherein the second plurality of images do not depict cookware upon the cooktop;

storing the second plurality of images in the memory as a negative image training

dataset;

training a classifier based on the positive image training dataset and the negative image training dataset;

identifying one or more locations respectively associated with one or more burners included in the cooktop, wherein identifying, by one or more computing devices, the one or more locations respectively associated with the one or more burners comprises;

obtaining a reference frame of imagery depicting the cooktop without any objects placed thereon;

obtaining one or more calibration frames of imagery depicting the cooktop with one or more items of cookware respectively positioned at the one or more locations respectively associated with the one or more burners, wherein the one or more calibration frames of imagery are captured when motion is not detected at the cooktop;

performing background subtraction for the one or more calibration frames of imagery with respect to the reference frame of imagery to identify new imagery; and

segmenting the new imagery to identify the one or more locations respectively associated with the one or more burners;

segmenting a frame of imagery captured by the vision sensor into one or more image segments based at least in part on the one or more locations respectively associated with the one or more burners included in the cooktop;

using the classifier to provide an initial classification for each of the one or more image segments, the initial classification for each of the one or more image segments classifying such image segment into either a first class of images depicting cookware or a second class of images not depicting cookware, wherein classification of one or more image segments into first class of images corresponds to detection of cookware on cooktop.

9. The system of claim 8, wherein the operations further comprise:

providing an indication to a user of whether the initial classification for at least one of the one or more image segments comprises the first class or the second class;

after providing the indication, receiving a user input indicating whether the initial classification is correct; and

when the user input indicates that the initial classification is not correct, changing the initial classification for the frame of imagery to a subsequent classification, wherein the subsequent classification comprises the second class when the initial classification comprises the first class, and wherein the subsequent classification comprises the first class when the initial classification comprises the second class.

10. The system of claim 9, wherein the operations further comprise:

adding the frame of imagery to the positive image training dataset when the subsequent classification comprises the first class;

adding the frame of imagery to the negative image training dataset when the subsequent classification comprises the second class; and

re-training the classifier based at least in part on the positive image training dataset and the negative image training dataset.

11. The system of claim 8, wherein the operations further include, prior to obtaining the first plurality of images, providing one or more prompts to a user to place one or more items of cookware on the cooktop.

12. A system for detecting cookware, the system comprising:

a vision sensor positioned so as to collect a plurality of images of a cooktop;

a calibration module implemented by one or more processors, the calibration module configured to;

obtain reference imagery depicting the cooktop without objects placed thereon;

obtain calibration imagery depicting the cooktop with one or more items of cookware respectively placed on one or more burners included in the cooktop;

wherein the calibration imagery is captured only when motion is not detected at the cooktop;

perform background subtraction for the calibration imagery with respect to the reference imagery to identify new imagery; and

segment the new imagery to identify one or more locations respectively associated with the one or more burners;

a classifier module implemented by the one or more processors, the classifier module configured to;

calculate a cookware score for each of the plurality of images; and

use the cookware score for each of the plurality of images to respectively classify each of the plurality of the images as either depicting cookware or not depicting cookware, wherein classification of one of the plurality of images as depicting cookware corresponds to detection of cookware at the cooktop; and

a classifier training module implemented by the one or more processors, the classifier training module configured to train the classifier module based at least in part on a positive image training dataset and a negative image training dataset.

13. The system of claim 12, wherein the calibration module is further configured to:

obtain a first plurality of images captured by the vision sensor, wherein the first plurality of images depict cookware upon the cooktop;

store the first plurality of images in a memory as the positive image training dataset;

obtain a second plurality of images captured by the vision sensor, wherein the second plurality of images do not depict cookware upon the cooktop; and

store the second plurality of images in the memory as the negative image training dataset.

14. The system of claim 12, wherein the classifier module comprises a support vector machine.

15. The system of claim 12, further comprising:

a staus indicator configured to respectively provide an indication of the classification provided for each of the plurality of images by the classifier module;

a user interface for receiving a user input, wherein the user input indicates whether the classification provided for each of the plurality of images by the classifier module is correct;

wherein the classifier module is further configured to change the classification provided for each of the plurality of images for which the corresponding user input indicates that the classification provided for such image is not correct.

16. The system of claim 15, wherein:

the calibration module is further configured to add each of the plurality of images for which the corresponding user input indicates that the classification provided for such image is not correct into either the positive image training dataset or the negative image training dataset, and

the classifier training module is further configured to re-train the classifier module based at least in part on the positive image training dataset and the negative image training dataset after the calibration module has added each of the plurality of images for which the corresponding user input indicates that the classification provided for such image is not correct.