Ingredient cooking-operation recognition system and computer-readable recording medium which is recorded with ingredient cooking-operation recognition program

• US 7,454,390 B2
• Filed: 03/17/2005
• Issued: 11/18/2008
• Est. Priority Date: 03/17/2004
• Status: Expired due to Fees

• Current Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Nagamitsu, Sachio, Minoh, Michihiko, Yamakata, Yoko
• Primary Examiner(s): Vincent; David R
• Assistant Examiner(s): Fernandez Rivas; Omar F
• Application Number:
  US10/573,387
• Publication Number:
  US 20070114224A1
• Time in Patent Office: 1,342 Days
• Field of Search: 706 45- 48, 706 50- 52, 706/31, 706/61, 426/65, 426/388, 426/390, 426/417, 426/420, 426/443, 426/496, 426/523, 993/24, 993/34, 993/41
• US Class Current: 706/45
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

An ingredient cooking-operation recognition system is provided which is capable of precisely specifying an ingredient that is cooked by a person and a cooking operation for the ingredient, and accurately estimating a cooking recipe for a dish that is prepared by the person. The system includes a sensing section which has an optical camera, a thermal camera, and a microphone that acquires an environment sound. Moreover, the system includes a feature-quantity template creation section which creates a feature-quantity template and a recognition processing section. The recognizing processing section calculates, on the basis of observation data acquired by the sensing section, an observation certainty factor and a cooking operation, creates a cooking flow based on the observation certainty factors, calculates a relevance factor of this cooking flow and recognizes the ingredient, the cooking operation and a cooking recipe.

Claims

1. An ingredient cooking-operation recognition system comprising:

a sensing means for acquiring observation data which includes an image around the hands of a person who performs a cooking operation and an environment sound that is produced by the cooking operation;

a feature-quantity template in which various cooking operations that are predetermined for various ingredients are stored together with a template certainty factor that is a certainty factor which is predetermined for each cooking operation of the ingredients;

a cooking-flow creating means for, based on the observation data, calculating an observation certainty factor which indicates the certainty factor of at least either of an ingredient that is cooked by the person and the cooking operation of the ingredient, and based on the observation certainty factor, creating a cooking flow for a dish which is prepared by the person;

a primary certainty-factor calculating means for, calculating a primary certainty factor which is a certainty factor for each object that makes up the cooking flow based on the template certainty factor and the observation certainty factor;

a cooking-operation recognizing means for recognizing the ingredient and the cooking operation that correspond to the observation data, based on the primary certainty factor; and

a guidance means for providing guidance to the person regarding the cooking flow based on the ingredient and the cooking operation recognized by the cooking-operation recognizing means, wherein the person performs the cooking operation in accordance with the guidance.

2. The ingredient cooking-operation recognition system according to claim 1, further comprising:

a cooking-flow database which stores cooking flows which are created in advance of cooking operations for various dishes; and

a relevance-factor calculating means for, using mapping, calculating a relevance factor that indicates which of the cooking flows that are stored in the cooking-flow database is closest to the cooking flow which is created by the cooking-flow creating means,

wherein the cooking-operation recognizing means calculates a final certainty factor based on the relevance factor which is calculated by the relevance-factor calculating means and the primary certainty factor, updates the feature-quantity template by considering the final certainty factor as a new template certainty factor, and based on the feature-quantity template, recognizes an ingredient and a cooking operation.

3. The ingredient cooking-operation recognition system according to claim 1, further comprising:

a table storing means for storing;

an ingredient table which is formed by an ingredient record that has a field where data that indicates an ingredient is stored and a field where various kinds of data on the ingredient are stored together with the template certainty factor that is predetermined for the data;

a cooked-food table which is formed by a cooked-food record that has a field where data that indicates a cooking operation is stored and a field where various kinds of data on a change in an ingredient by the cooking operation are stored together with the template certainty factor that is predetermined for the data; and

a cooking-operation table which is formed by a cooking-operation record that has a field where data that indicates a cooking operation is stored and a field where various kinds of data on an environment sound that is produced by the cooking operation are stored together with the template certainty factor that is predetermined for the data,

a cooking-recipe database which stores data that indicates an ingredient which is used for various cooking recipes and data that indicates a cooking operation for the ingredient;

an extracting means for extracting all kinds of ingredients and all kinds of cooking operations which are stored in the cooking-recipe database; and

a feature-quantity template creating means for creating, as a feature-quantity template, a table which includes fields defined by items including each ingredient that is extracted by the extracting means and each cooking operation that is extracted by the extracting means, and for specifying, in an optional field of the fields which make up the feature-quantity template, the ingredient record of the ingredient table that stores the data which indicates the ingredient that corresponds to the optional field, the cooked-food record of the cooked-food table that stores the data which indicates the cooking operation that corresponds to the optional field and the cooking-operation record of the cooking-operation table that stores the data which indicates the cooking operation that corresponds to the optional field, and writing the greatest template certainty factor among the template certainty factors which are stored in each field of the specified records in the optional field.

4. The ingredient cooking-operation recognition system according to claim 1, wherein:

the sensing means has a thermal camera and an optical camera; and

an image which is acquired by the sensing means includes a thermal image which is photographed by the thermal camera and an optical image which is photographed by the optical camera.

5. The ingredient cooking-operation recognition system according to claim 4, wherein the cooking-flow creating means:

removes a background region from the optical image by executing a background difference process on the optical image;

removes, from the thermal image, as a background region, a region where the temperature is equal to, or higher than, a predetermined value;

calculates an ingredient region which indicates an ingredient by multiplying the thermal image whose background region is removed and the optical image whose background region is removed; and

calculates an observation certainty factor based on the ingredient region.

6. The ingredient cooking-operation recognition system according to claim 1, wherein the cooking-flow creating means obtains a histogram of hue and a histogram of saturation from an image which is acquired by the sensing means, and estimates an ingredient by obtaining a correlation between the histograms and a color-feature template which is predetermined for each ingredient.

7. The ingredient cooking-operation recognition system according to claim 2, wherein:

the relevance-factor calculating means estimates a cooking recipe based on a relevance factor; and

the guidance means provides guidance to the person regarding the cooking recipe.

8. A computer-readable recording medium which has recorded thereon an ingredient cooking-operation recognition program, for causing a computer to function at least as:

a sensing means for acquiring observation data which includes an image around the hands of a person who performs a cooking operation and an environment sound that is produced by the cooking operation;

a feature-quantity template in which various cooking operations that are predetermined for various ingredients are stored together with a template certainty factor that is a certainty factor which is predetermined for each cooking operation of the ingredients;

a cooking-flow creating means for, based on the observation data, calculating an observation certainty factor which indicates the certainty factor of at least either of an ingredient that is cooked by the person and the cooking operation of the ingredient, and based on the observation certainty factor, creating a cooking flow for a dish which is prepared by the person;

a primary certainty-factor calculating means for, calculating a primary certainty factor which is a certainty factor for each object that makes up the cooking flow based on the template certainty factor and the observation certainty factor; and

a cooking-operation recognizing means for recognizing the ingredient and the cooking operation that correspond to the observation data, based on the primary certainty factor; and

a guidance means for providing guidance to the person regarding the cooking flow based on the ingredient and the cooking operation recognized by the cooking-operation recognizing means, wherein the person performs the cooking operation in accordance with the guidance.