Information processing apparatus and information processing method

• US 10,013,755 B2
• Filed: 05/19/2016
• Issued: 07/03/2018
• Est. Priority Date: 10/30/2012
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Mochizuki, Daisuke, Sato, Tatsuhito
• Primary Examiner(s): Azarian, Seyed
• Application Number:
  US15/159,281
• Publication Number:
  US 20160284074A1
• Time in Patent Office: 775 Days
• Field of Search: 382100, 382103, 382106, 382110, 382113, 382140, 382143, 382154-155, 382162, 382168, 382173, 382181, 382199, 382209, 382219, 382222, 382232, 382254, 382274, 382276, 382282-291, 382305, 382312, 600316, 434127
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 2207/30128 Food products
• CPC Class Codes: G06T 7/001 using an image reference approach
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata

Abstract

There is provided an information processing apparatus including an image acquisition unit configured to acquire a dish image obtained by shooting a single or multiple dishes, and a first dish recognition unit configured to recognize the single or multiple dishes included in the dish image with reference to dish data selected, from dish data registered in advance, based on a condition regarding at least one of a person relating to the dish image, a shooting environment of the dish image, a shooting place of the dish image, and a shooting time of the dish image.

Claims

1. An information processing apparatus comprising:

a display;

an imaging device; and

circuitry configured to

acquire data related to a recognized dish image that is based on one or more images captured by the imaging device,

control a displaying on the display of an indication according to an area of the recognized dish image,

acquire an input of a user, wherein the input is related to a correction of the area in association with a region within the recognized dish image,

replace the data related to the recognized dish image by data related to the corrected area, for storage within a non-transitory computer-readable storage medium, based on the acquired input of the user,

manage a volume of dish data that is stored in the non-transitory computer-readable storage medium by deleting, based on a predetermined criterion when a predetermined volume of the dish data is accumulated within the non-transitory computer-readable storage medium, a part of the dish data from the non-transitory computer-readable storage medium, wherein an amount of the dish data stored in the non-transitory computer-readable storage medium is limited by the management of the volume so as not to exceed a predetermined volume,

recognize one or more foods in an additionally-captured image, based on the volume-managed dish data that is stored in the non-transitory computer-readable storage medium, wherein the dish data that is referred to for the recognition of the one or more foods is limited by the circuitry so as to prevent increase in processing load, and

output information to a user regarding a result of the recognition of the one or more foods in the additionally-captured image,

wherein the information processing apparatus is a mobile device.

2. The information processing apparatus according to claim 1, wherein the circuitry is further configured to

register data related to the corrected area into the non-transitory computer-readable storage medium.

3. The information processing apparatus according to claim 2, wherein the circuitry is further configured to

recognize the dish image using the data related to the corrected area.

4. The information processing apparatus according to claim 1, wherein the recognized dish image is obtained by shooting a single or multiple dishes using a camera as the imaging device.

5. The information processing apparatus according to claim 1, wherein the circuitry is further configured to

recognize the dish image with reference to dish data selected, from dish data registered in advance, based on a condition regarding at least one of a person relating to the dish image, a shooting environment of the dish image, a shooting place of the dish image, and a shooting time of the dish image.

6. The information processing apparatus according to claim 5, wherein the circuitry is further configured to

integrate a first recognition result, obtained based on the recognition of the dish image, with a second recognition result obtained by another recognition of the dish image without using the condition.

7. The information processing apparatus according to claim 6, wherein the circuitry is further configured to

register the dish data in an additional manner based on the second recognition result.

8. The information processing apparatus according to claim 7,

wherein, when the dish image is not automatically recognized, the second recognition result is decided based on an input of a user, and

wherein the dish data is registered in an additional manner based on the second recognition result decided based on the input of the user.

9. The information processing apparatus according to claim 7,

wherein, the second recognition result is a result obtained by automatically recognizing the dish image or a result obtained by correcting the result based on information that is input by a user, and

wherein the dish data is registered in an additional manner based on the second recognition result decided through correction based on the information that is input by the user.

10. The information processing apparatus according to claim 6,

wherein the information processing apparatus is a client used by a user who shoots the dish image,

wherein the acquired data related to the recognized dish image is transmitted to a server, and

wherein the second recognition result is received from the server.

11. The information processing apparatus according to claim 5, wherein the circuitry is further configured to

register the dish data in an additional manner.

12. The information processing apparatus according to claim 11,

wherein the single or multiple dishes are recognized by matching the dish image with the selected dish data.

13. The information processing apparatus according to claim 12,

wherein a part of the dish data is deleted based on a number indicating how many times the dish data is used for matching with the dish image, or a date and time indicating when the dish data is used for matching with the dish image.

14. The information processing apparatus according claim 12,

wherein a part of the dish data is deleted based on a number indicating how many times the dish data successfully matches with the dish image, or a date and time indicating when the dish data successfully matches the dish image.

15. The information processing apparatus according to claim 5,

wherein the dish image is acquired that entails metadata, and

wherein the circuitry is further configured to

extract the information regarding at least one of the person relating to the dish image, the shooting environment of the dish image, the shooting place of the dish image, and the shooting time of the dish image from the metadata.

16. The information processing apparatus according to claim 15,

wherein information regarding the person relating to the dish image is extracted, and

wherein the dish image is recognized with reference to the dish data registered in association with the person.

17. The information processing apparatus according to claim 15,

wherein information regarding the shooting environment of the dish image is extracted, and

wherein the dish image is recognized by using the dish data registered in association with information indicating an environment common to the shooting environment.

18. The information processing apparatus according to claim 15,

wherein information regarding the shooting place of the dish image is extracted, and

wherein the dish image is recognized by using the dish data registered in association with the shooting place.

19. The information processing apparatus according to claim 15,

wherein information regarding the shooting time of the dish image is extracted, and

wherein the dish image is recognized by using the dish data registered in association with a time zone including the shooting time.

20. The information processing apparatus according to claim 5,

wherein a characteristic of a whole of the dish image is treated as information regarding at least one of the shooting environment of the dish image, the shooting place of the dish image, and the shooting time of the dish image.

21. The information processing apparatus according to claim 20,

wherein the dish data is data of an image that is cut away from a source image, and is registered in association with information regarding the source image, and

wherein, when the characteristic of the whole of the dish image is similar to a characteristic of the source image, the dish image is recognized by using the data of the image that is cut away from the source image.

22. The information processing apparatus according to claim 1, wherein the circuitry is further configured to

recognize a respective dish area corresponding to each of the foods included in the dish image.

23. An information processing method executed via an information processing apparatus of a mobile device, the method comprising:

acquiring data related to a recognized dish image that is based on one or more images captured by an imaging device of the mobile device;

displaying, on a display of the mobile device, an indication according to an area of the recognized dish image;

acquiring an input of a user, wherein the input is related to a correction of the area in association with a region within the recognized dish image;

replacing the data related to the recognized dish image by data related to the corrected area, for storage within a non-transitory computer-readable storage medium, based on the acquired input of the user;

managing a volume of dish data that is stored in the non-transitory computer-readable storage medium by deleting, based on a predetermined criterion when a predetermined volume of the dish data is accumulated within the non-transitory computer-readable storage medium, a part of the dish data from the non-transitory computer-readable storage medium, wherein an amount of the dish data stored in the non-transitory computer-readable storage medium is limited by the management of the volume so as not to exceed a predetermined volume;

recognizing one or more foods in an additionally-captured image, based on the volume-managed dish data that is stored in the non-transitory computer-readable storage medium, wherein the dish data that is referred to for the recognition of the one or more foods is limited so as to prevent increase in processing load; and

outputting information to a user regarding a result of the recognition of the one or more foods in the additionally-captured image.

24. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer of a mobile device causes the computer to execute a method, the method comprising:

acquiring data related to a recognized dish image that is based on one or more images captured by an imaging device of the mobile device;

displaying, on a display of the mobile device, an indication according to an area of the recognized dish image;

acquiring an input of a user, wherein the input is related to a correction of the area in association with a region within the recognized dish image;

replacing the data related to the recognized dish image by data related to the corrected area, for storage within a non-transitory computer-readable storage medium, based on the acquired input of the user;

managing a volume of dish data that is stored in the non-transitory computer-readable storage medium by deleting, based on a predetermined criterion when a predetermined volume of the dish data is accumulated within the non-transitory computer-readable storage medium, a part of the dish data from the non-transitory computer-readable storage medium, wherein an amount of the dish data stored in the non-transitory computer-readable storage medium is limited by the management of the volume so as not to exceed a predetermined volume;

recognizing one or more foods in an additionally-captured image, based on the volume-managed dish data that is stored in the non-transitory computer-readable storage medium, wherein the dish data that is referred to for the recognition of the one or more foods is limited so as to prevent increase in processing load; and

outputting information to a user regarding a result of the recognition of the one or more foods in the additionally-captured image.