CONNECTED FOOD PREPARATION SYSTEM AND METHOD OF USE

• US 20160327281A1
• Filed: 05/05/2016
• Published: 11/10/2016
• Est. Priority Date: 05/05/2015
• Status: Active Grant

• Current Assignee: June Life LLC
• Original Assignee: June Life, Inc.
• Inventors: Bhogal, Nikhil, Van Horn, Matthew, Park, Seunghoon, Sivalingam, Ravishankar, Clark, Christopher Russell
• Granted Patent: US 10,845,060 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A23L 5/15 using wave energy, irradiation, electrical means or magnetic fields, e.g. oven cooking or roasting using radiant dry heat
• CPC Class Codes: A23L 5/17 in a gaseous atmosphere with forced air or gas circulation, in vacuum or under pressure
• CPC Class Codes: A23V 2002/00 Food compositions, function of food ingredients or processes for food or foodstuffs
• CPC Class Codes: A47J 36/321 the electronic control being performed over a network, e.g. by means of a handheld device
• CPC Class Codes: A47J 37/0623 Small-size cooking ovens, i.e. defining an at least partially closed cooking cavity A47J37/0605 takes precedence
• CPC Class Codes: A47J 37/0664 Accessories
• CPC Class Codes: F24C 15/008 Illumination for oven cavities
• CPC Class Codes: F24C 3/124 Control panels
• CPC Class Codes: F24C 7/062 on stoves
• CPC Class Codes: F24C 7/085 on baking ovens
• CPC Class Codes: F24C 7/086 touch control
• CPC Class Codes: F24C 7/087 of electric circuits regulating heat
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
• CPC Class Codes: H05B 1/0263 Ovens

Abstract

A connected oven, including a set of in-cavity sensors and a processor configured to automatically identify foodstuff within the cooking cavity, based on the sensor measurements; and automatically operate the heating element based on the foodstuff identity.

Claims

1. A method for operating an oven, the oven including a cooking cavity, a camera arranged along a top of the cooking cavity and directed toward a bottom of the cooking cavity, a set of light emitting elements arranged along the top of the cooking cavity, a user interface, computer memory, a processing system, and a weight sensor, the method comprising:

at the oven;

in response to detection of a foodstuff insertion event, automatically recording a first image of the cooking cavity;

extracting foodstuff features from the first image;

determining a foodstuff weight based on measurements from the weight sensor;

determining a probability for each of a plurality of foodstuff classes, based on the foodstuff features and the foodstuff weight, using a classification module stored by the oven;

automatically displaying an identifier for each of the plurality of foodstuff classes in decreasing probability order at the user interface;

receiving a user selection of an identifier for a foodstuff class at the user interface;

sending the selected identifier and the first image to a remote computing system;

automatically controlling cooking elements of the oven based on a first recipe associated with the foodstuff class, wherein the first recipe is stored by the computer memory;

concurrent with controlling the cooking elements based on the first recipe, automatically streaming cooking images of the cooking cavity to the remote computing system;

at the remote computing system;

receiving the streamed cooking images;

identifying a user device associated with the oven; and

streaming the cooking images to the user device in near-real time.

2. The method of claim 1, further comprising, at the remote computing system:

retrieving a copy of the classification module;

in response to the selected identifier matching the foodstuff class having a highest probability, determining a second classification module by reinforcing the copy of the classification module based on the first image and the selected identifier;

in response to the selected identifier differing from the foodstuff class having the highest probability, determining a second classification module by calibrating the copy of the classification module based on the first image and the selected identifier; and

automatically updating the oven with the second classification module.

3. The method of claim 1, wherein extracting the foodstuff features comprises determining a physical foodstuff area, comprising:

identifying adjacent wires of an oven rack within the first image;

determining a pixel distance between the adjacent wires within a foreground of the first image;

determining pixel-to-physical area mapping based on the pixel distance;

identifying a continuous region indicative of foodstuff within the foreground; and

determining a physical foodstuff area for the continuous region based on the pixel-to-physical area mapping;

wherein controlling the cooking elements further comprises determining a set of adjusted control instructions based on the first recipe and the physical foodstuff area, and controlling the cooking elements according to the adjusted control instructions.

4. The method of claim 1, further comprising, concurrently with cooking the foodstuff, periodically, at the oven:

recording a cooking image;

extracting a second set of foodstuff features from a cooking image;

determining a cooking status of the foodstuff based on the second set of foodstuff features;

determining a difference between the cooking status and an expected cooking status;

determining a cooking element control instruction change based on the difference; and

automatically adjusting cooking element operation based on the cooking element control instruction change.

5. The method of claim 4, further comprising, at the oven:

receiving cooking element control instructions from the user device;

controlling the cooking elements based on the received cooking element control instructions;

sending the received cooking element control instructions to the remote computing system, wherein the remote computing system stores the cooking element control instructions in association with a user account associated with the user device;

determining a second recipe based on the first recipe and the received cooking element control instructions;

storing the second recipe in the computer memory; and

in response to subsequent user selection of the identifier for the foodstuff class, controlling the cooking elements based on the second recipe.

6. A method for controlling a connected oven, the oven including a cooking cavity, a camera arranged along a top of the cooking cavity and directed toward a bottom of the cooking cavity, a set of light emitting elements arranged along the top of the cooking cavity, the method comprising, at the oven:

automatically recording a first image of the cooking cavity in response to detection of a foodstuff insertion event;

automatically identifying foodstuff within the cooking cavity, comprising;

segmenting the first image into a foreground and a background; and

classifying the foreground with a foodstuff class using a classification module stored by the oven;

automatically displaying an identifier for the foodstuff class in response to foodstuff identification;

receiving a user classification of the foodstuff class;

automatically cooking the foodstuff in response to receipt of the user classification, comprising controlling cooking elements of the oven based on a recipe associated with the foodstuff class; and

concurrently with cooking the foodstuff, automatically streaming cooking images of the cooking cavity to a remote computing system.

7. The method of claim 6, further comprising, at the remote computing system, automatically streaming images of the cooking cavity to a remote user device.

8. The method of claim 6, wherein:

classifying the foreground with a foodstuff class using the classification module stored by the oven comprises;

determining a set of foodstuff features, wherein the foodstuff features are extracted from the foreground; and

determining a probability for each of a plurality of potential foodstuff classes based on the set of foodstuff features using the classification module;

displaying the identifier for the foodstuff class comprises displaying identifiers for each of the potential foodstuff classes in decreasing probability order;

receiving a user classification of the foodstuff class comprises receiving a user selection of an identifier for a potential foodstuff class; and

the method further comprises;

sending the first image and selected identifier to the remote computing system.

9. The method of claim 8, further comprising, at the remote computing system:

retrieving a copy of the classification module;

in response to the selected identifier matching the potential foodstuff class having a highest probability, determining a second classification module by reinforcing the copy of the classification module based on the first image and the selected identifier;

in response to the selected identifier differing from the potential foodstuff class having the highest probability, determining a second classification module by calibrating the copy of the classification module based on the first image and the selected identifier; and

automatically updating the oven with the second classification module.

10. The method of claim 8, wherein automatically identifying the foodstuff within the cooking cavity further comprises automatically dewarping the first image, using dedicated dewarping hardware of the oven, before segmenting the first image into the foreground and the background.

11. The method of claim 8, wherein determining the set of foodstuff features comprises determining a physical foodstuff size, comprising:

identifying adjacent wires of an oven rack within the foreground;

determining a pixel distance between the adjacent wires within the foreground;

selecting a pixel-to-physical area map from a set of predetermined maps based on the pixel distance;

identifying a continuous region indicative of foodstuff within the foreground; and

determining a physical foodstuff size for the continuous region based on the pixel-to-physical area map;

wherein controlling the cooking elements further comprises determining the recipe based on the physical foodstuff size.

12. The method of claim 6, wherein detecting the foodstuff insertion event comprises detecting an oven door opening event followed by an oven weight increase.

13. The method of claim 6, further comprising, at the oven:

concurrently with cooking the foodstuff, periodically monitoring a cooking status of the foodstuff based on the cooking images;

determining a difference between the cooking status with an expected cooking status;

determining a cooking element control instruction change based on the difference; and

automatically adjusting cooking element operation based on the cooking element control instruction change.

14. The method of claim 13, further comprising, at the oven:

estimating a time to cooking completion based on the cooking status; and

in response to an instantaneous time falling within a predetermined time duration from the time to cooking completion, automatically sending a notification to the user device.

15. The method of claim 6, further comprising, at the oven, retrieving the recipe before automatically cooking the foodstuff, wherein retrieving the recipe comprises retrieving the recipe from computer memory on-board the oven.

16. The method of claim 15, further comprising, at the oven:

receiving cooking element control instructions from a user device;

controlling the cooking elements based on the cooking element control instructions;

sending the cooking element control instructions to the remote computing system;

determining a second recipe, comprising;

updating the recipe, stored by the computer memory on-board the oven, based on the cooking element control instructions; and

in response to subsequent user selection of the identifier for the foodstuff class, controlling the cooking elements based on the second recipe;

wherein the remote computing system stores the cooking element control instructions in association with a user account associated with the user device.

17. The method of claim 16, wherein receiving cooking element control instructions from the user device comprises receiving the cooking element control instructions from the user device at the oven over a short-range wireless communication channel;

wherein sending the cooking element control instructions to the remote computing system from the oven comprises sending the cooking element control instructions to the remote computing system over a long-range wireless communication channel.

18. The method of claim 16, further comprising, at the remote computing system:

receiving a plurality of cooking element control instructions for a food class from each of a plurality of ovens;

extracting a set of common cooking element control instructions from the plurality of cooking element control instructions;

determining a third recipe, comprising updating the recipe associated with the foodstuff class with the set of common cooking element control instructions; and

sending the third recipe to each of the plurality of ovens at a first time;

wherein each oven of the plurality controls the respective cooking elements based on the third recipe in response to subsequent user selection of the identifier for the foodstuff class.

19. The method of claim 6, further comprising, at the oven:

measuring an ambient light intensity; and

automatically controlling light emitting element operation based on the ambient light intensity in response to detection of the foodstuff insertion event.

20. The method of claim 19, wherein automatically controlling light emitting element operation based on the ambient light intensity comprises:

automatically controlling light emitting element operation to meet a target cooking cavity light intensity cooperatively with ambient light.