Systems and methods for cookware detection

• US 9,412,048 B2
• Filed: 04/21/2014
• Issued: 08/09/2016
• Est. Priority Date: 04/21/2014
• Status: Active Grant

• Current Assignee: Haier US Appliance Solutions, Inc. DBA GE Appliances (Haier Group Company)
• Original Assignee: Haier US Appliance Solutions, Inc. DBA GE Appliances (Haier Group Company)
• Inventors: Guan, Li, Smith, Charles Ray
• Primary Examiner(s): GORADIA, SHEFALI DINESH
• Application Number:
  US14/257,282
• Publication Number:
  US 20150300652A1
• Time in Patent Office: 841 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: F24C 7/083 on tops, hot plates
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables

Abstract

Systems and methods for cookware detection are provided. One example system includes a vision sensor positioned so as to collect imagery depicting a cooktop. The system includes one or more processors and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations. The operations include obtaining a frame of imagery depicting the cooktop and using a classifier to classify the frame of imagery as either depicting cookware or not depicting cookware. When the frame is classified as depicting cookware, the operations include performing one or more operations in response to the detected cookware.

Claims

1. A system for detecting cookware, the system comprising:

a vision sensor positioned so as to collect imagery depicting a cooktop;

one or more processors; and

one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising;

obtaining a frame of imagery depicting the cooktop;

performing background subtraction for the frame of imagery with respect to a reference frame of imagery depicting the cooktop without any objects placed thereon;

after performing the background subtraction for the frame of imagery, using a classifier to classify the frame of imagery as either depicting cookware or not depicting cookware; and

when the frame is classified as depicting cookware, performing one or more operations in response to the detected cookware.

2. The system of claim 1, wherein the operations further comprise, prior to using the classifier to classify the frame of imagery as either depicting cookware or not depicting cookware, calibrating the system to identify one or more locations respectively associated with one or more burners included in the cooktop.

3. The system of claim 2, wherein calibrating the system to identify one or more locations respectively associated with one or more burners included in the cooktop comprises:

obtaining a reference frame of imagery depicting the cooktop without any objects placed thereon;

obtaining one or more calibration frames of imagery depicting the cooktop with one or more objects respectively positioned at the one or more locations respectively associated with the one or more burners;

performing background subtraction for the one or more calibration frames of imagery with respect to the reference frame of imagery to identify the one or more locations.

4. The system of claim 3, wherein:

the one or more calibration frames of imagery comprise a single calibration frame of imagery;

the one or more objects respectively positioned at the one or more locations respectively associated with the one or more burners comprise a plurality of objects respectively positioned at a plurality of locations respectively associated with a plurality of burners included in the cooktop; and

performing background subtraction for the single calibration frame of imagery with respect to the reference frame of imagery further comprises performing object segmentation to identify the plurality of locations.

5. The system of claim 2, wherein calibrating the system to identify one or more locations respectively associated with one or more burners included in the cooktop comprises:

obtaining a reference frame of imagery depicting the cooktop without any objects placed thereon;

obtaining one or more calibration frames of imagery depicting the cooktop with the one or more burners energized;

performing background subtraction for the one or more calibration frames of imagery with respect to the reference frame of imagery to identify the one or more locations.

6. The system of claim 1, wherein:

the operations further comprise segmenting the frame of imagery according to a plurality of burner locations respectively associated with a plurality of burners included in the cooktop, such that a plurality of frame segments of formed that respectively correspond to the plurality of burners; and

using the classifier to classify the frame of imagery as either depicting cookware or not depicting cookware comprises using the classifier to individually classify each of the plurality of frame segments as either depicting cookware or not depicting cookware, such that cookware detection is performed on a burner-by-burner basis.

7. The system of claim 1, wherein the classifier comprises a support vector machine.

8. The system of claim 1, wherein one or more operations in response to the detected cookware comprise performing a boil detection algorithm.

9. The system of claim 1, wherein one or more operations in response to the detected cookware comprise transmitting a communication to the cooktop that informs the cooktop of the presence of the detected cookware.

10. The system of claim 9, wherein the communication further identifies one or more of a plurality of burners included in the cooktop upon which the detected cookware is positioned.

11. The system of claim 1, wherein:

the vision sensor comprises a wide angle VGA resolution camera; and

the one or more processors and the one or more non-transitory computer-readable media comprise a single System-On-Chip.

12. A method for detecting cookware, the method comprising:

collecting, by a vision sensor, one or more frames comprising imagery depicting a cooktop;

segmenting each of the one or more frames into a plurality of frame segments based on a plurality of locations respectively associated with a plurality of burners included in the cooktop;

classifying, by a binary classifier, each of the plurality of frame segments as either depicting an item of cookware or not depicting an item of cookware, such that cookware detection is performed on a burner-by-burner basis; and

when at least one of the frame segments is classified as depicting an item of cookware, performing one or more operations responsive to the detected item of cookware.

13. The method of claim 12, further comprising, prior to classifying each of the one or more frames, calibrating the system to identify one or more locations respectively associated with one or more burners included in the cooktop, wherein the calibration is performed by comparing reference imagery depicting the cooktop without objects placed thereon with calibration imagery depicting the cooktop with one or more objects respectively placed on the one or more burners.

14. The method of claim 12, further comprising, prior to said segmenting and said classifying, performing a calibration routine to identify the plurality of locations respectively associated the plurality of burners included in the cooktop.

15. The method of claim 14, wherein performing the calibration routine comprises:

obtaining a reference frame of imagery depicting the cooktop without any objects placed thereon;

obtaining one or more calibration frames of imagery depicting the cooktop with a plurality of objects respectively positioned at the plurality of locations respectively associated with the plurality of burners;

performing background subtraction for the one or more calibration frames of imagery with respect to the reference frame of imagery to identify the plurality of locations.

16. The method of claim 15, wherein:

the one or more calibration frames of imagery comprise a single calibration frame of imagery; and

performing background subtraction for the single calibration frame of imagery with respect to the reference frame of imagery further comprises performing object segmentation to identify the plurality of locations.

17. The method of claim 14, wherein performing the calibration routine comprises:

obtaining a reference frame of imagery depicting the cooktop without any objects placed thereon;

obtaining one or more calibration frames of imagery depicting the cooktop with the plurality of burners energized; and

performing background subtraction for the one or more calibration frames of imagery with respect to the reference frame of imagery to identify the plurality of locations.

18. A system for detecting cookware, the system comprising:

a vision sensor positioned so as to collect imagery depicting a cooktop;

one or more processors; and

one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising;

obtaining a frame of imagery depicting the cooktop;

using a classifier to classify the frame of imagery as either depicting cookware or not depicting cookware, wherein the classifier classifies the frame of imagery as either depicting cookware or not depicting cookware based on a plurality of planes respectively associated with a plurality of image features, wherein the plurality of planes are predetermined based on performance of a classifier training algorithm with respect to a positive training image dataset and a negative training image dataset, wherein the positive image training dataset comprises a plurality of images depicting cookware and the negative image training dataset comprises a plurality of images that do not depict cookware, and wherein the plurality of planes are stored in a system memory; and

when the frame is classified as depicting cookware, performing one or more operations in response to the detected cookware.

19. The system of claim 18, wherein the operations further comprise, when the frame is classified as depicting cookware:

adding the frame of imagery to a positive training image dataset; and

recomputing the plurality of planes.

20. The system of claim 18, wherein:

the operations further comprise segmenting the frame of imagery according to a plurality of burner locations respectively associated with a plurality of burners included in the cooktop, such that a plurality of frame segments of formed that respectively correspond to the plurality of burners; and

using the classifier to classify the frame of imagery as either depicting cookware or not depicting cookware comprises using the classifier to individually classify each of the plurality of frame segments as either depicting cookware or not depicting cookware, such that cookware detection is performed on a burner-by-burner basis.