Ingredient cooking-operation recognition system and ingredient cooking-operation recognition program

• US 20070114224A1
• Filed: 03/17/2005
• Published: 05/24/2007
• Est. Priority Date: 03/17/2004
• Status: Active Grant

• Current Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Nagamitsu, Sachio, Minoh, Michihiko, Yamakata, Yoko
• Granted Patent: US 7,454,390 B2
• Time in Patent Office: Days
• US Class Current: 219/490
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

An ingredient cooking-operation recognition system is provided which is capable of precisely specifying an ingredient that is cooked by a person and a cooking operation for the ingredient, and accurately estimating a cooking recipe for a dish that is prepared by the person. This system includes: a sensing section 200 which has an optical camera 10 that photographs a person'"'"'s cooking situation, a thermal camera 20, and a microphone 30 that acquires an environment sound; a feature-quantity template creation section 106 which creates a feature-quantity template using an ingredient database 103, a cooked-food database 104 and a cooking-operation database 105; and a recognition processing section 110 which calculates, on the basis of observation data which is acquired by the sensing section 200, an observation certainty factor for an ingredient that is cooked by the person and a cooking operation that is conducted by the person, creates a cooking flow based on this observation certainty factors, calculates a relevance factor of this cooking flow on the basis of a template certainty factor which is written in the feature-quantity template, and recognizes the ingredient, the cooking operation and a cooking recipe.

Claims

1-8. -8. (canceled)

9. An ingredient cooking-operation recognition system comprising:

a sensing means for acquiring observation data which includes at least either of an image around the hands of a person who performs a cooking operation and an environment sound that is produced by the cooking operation;

a feature-quantity template in which various cooking operations that are predetermined for various ingredients are stored in a table form, together with a template certainty factor that is a certainty factor which is predetermined for each cooking operation of the ingredients;

a cooking-flow creating means for, based on the observation data which is acquired by the sensing means, calculating an observation certainty factor which indicates the certainty factor of at least either of an ingredient that is cooked by the person and the cooking operation of the ingredient, and based on this observation certainty factor, creating a cooking flow for a dish which is prepared by the person;

a primary certainty-factor calculating means for, based on the template certainty factor and the observation certainty factor, calculating a primary certainty factor which is a certainty factor for each of objects that make up the cooking flow which is created by the cooking-flow creating means; and

a cooking-operation recognizing means for calculating a final certainty factor based on the primary certainty factor which is calculated by the primary certainty-factor calculating means, updating the feature-quantity template by considering this final certainty factor as a new template certainty factor, and based on this feature-quantity template, recognizing the ingredient and the cooking operation that correspond to the observation data which is acquired by the sensing means.

10. The ingredient cooking-operation recognition system according to claim 9, further comprising:

a cooking-flow database which stores cooking flows which are created in advance for various dishes; and

a relevance-factor calculating means for, using flow matching, calculating a relevance factor that indicates which of the cooking flows that are stored in the cooking-flow database is closest to the cooking flow which is created by the cooking-flow creating means,

the cooking-operation recognizing means recognizing an ingredient and a cooking operation, based on the relevance factor which is calculated by the relevance-factor calculating means and the primary certainty factor.

11. The ingredient cooking-operation recognition system according to claim 9, further comprising:

a table storing means for storing at least any one of;

an ingredient table which is formed by an ingredient record that has a field where data that indicates an ingredient is stored and a field where various kinds of data on the ingredient are stored together with a template certainty factor that is predetermined for the data;

a cooked-food table which is formed by a cooked-food record that has a field where data that indicates a cooking operation is stored and a field where various kinds of data on a change in an ingredient by the cooking operation are stored together with a template certainty factor that is predetermined for the data; and

a cooking-operation table which is formed by a cooking-operation record that has a field where data that indicates a cooking operation is stored and a field where various kinds of data on an environment sound that is produced by the cooking operation are stored together with a template certainty factor that is predetermined for the data;

a cooking-recipe database which stores data that indicates an ingredient which is used for various cooking recipes and data that indicates a cooking operation for the ingredient;

an extracting means for extracting all kinds of ingredients and all kinds of cooking operations which are stored in the cooking-recipe database; and

a feature-quantity template creating means for creating, as a feature-quantity template, a table which includes, as items, each ingredient that is extracted by the extracting means and each cooking operation that is extracted by the extracting means,

the feature-quantity template creating means writing, in an optional field of the fields which make up the feature-quantity template, as the template certainty factor, the greatest certainty factor of the template certainty factors which are stored in each field of at least any record of the ingredient record of the ingredient table that stores the data which indicates the ingredient that corresponds to the optional field, the cooked-food record of the cooked-food table that stores the data which indicates the cooking operation that corresponds to the optional field and the cooking-operation record of the cooking-operation table that stores the data which indicates the cooking operation that corresponds to the optional field.

12. The ingredient cooking-operation recognition system according to claim 9, characterized in that:

the sensing means has a thermal camera and an optical camera; and

an image which is acquired by the sensing means includes a thermal image which is photographed by the thermal camera and an optical image which is photographed by the optical camera.

13. The ingredient cooking-operation recognition system according to claim 12, characterized in that the cooking-flow creating means:

removes a background region from the optical image by executing a background difference processing on the optical image;

removes, from the thermal image, as a background region, a region where the temperature is equal to, or higher than, a predetermined value;

calculates an ingredient region which indicates an ingredient by multiplying the thermal image whose background region is removed and the optical image whose background region is removed; and

calculates an observation certainty factor based on this ingredient region.

14. The ingredient cooking-operation recognition system according to claim 9, characterized in that the cooking-flow creating means obtains a histogram of hue and a histogram of saturation from an image which is acquired by the sensing means, and estimates an ingredient by obtaining a correlation between these histograms and a color-feature template which is predetermined for each ingredient.

15. The ingredient cooking-operation recognition system according to claim 10, characterized in that:

the relevance-factor calculating means estimates a cooking recipe based on a relevance factor; and

a guidance means is further provided for giving, to the person, guidance on the cooking recipe which is estimated by the relevance-factor calculating means.

16. A computer-readable recording medium which is recorded with an ingredient cooking-operation recognition program, characterized by allowing a computer to function as:

a sensing means for acquiring observation data which includes at least either of an image around the hands of a person who performs a cooking operation and an environment sound that is produced by the cooking operation;

a feature-quantity template in which various cooking operations that are predetermined for various ingredients are stored in a table form, together with a template certainty factor that is a certainty factor which is predetermined for each cooking operation of the ingredients;

a cooking-flow creating means for, based on the observation data which is acquired by the sensing means, calculating an observation certainty factor which indicates the certainty factor of at least either of an ingredient that is cooked by the person and the cooking operation of the ingredient, and based on this observation certainty factor, creating a cooking flow for a dish which is prepared by the person;

a primary certainty-factor calculating means for, based on the template certainty factor and the observation certainty factor, calculating a primary certainty factor which is a certainty factor for each of objects that make up the cooking flow which is created by the cooking-flow creating means; and

a cooking-operation recognizing means for calculating a final certainty factor based on the primary certainty factor which is calculated by the primary certainty-factor calculating means, updating the feature-quantity template by considering this final certainty factor as a new template certainty factor, and based on this feature-quantity template, recognizing the ingredient and the cooking operation that correspond to the observation data which is acquired by the sensing means.