User terminal apparatus and control method thereof

• US 10,521,903 B2
• Filed: 11/25/2016
• Issued: 12/31/2019
• Est. Priority Date: 11/25/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kang, Seong-min, Han, Heung-woo
• Primary Examiner(s): Mehta, Bhavesh M
• Assistant Examiner(s): Lemieux, Ian L
• Application Number:
  US15/361,119
• Publication Number:
  US 20170148162A1
• Time in Patent Office: 1,131 Days
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables
• CPC Class Codes: G09B 19/0092 Nutrition
• CPC Class Codes: G16H 20/60 relating to nutrition control, e.g. diets
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS

Abstract

A user terminal apparatus is provided. The user terminal apparatus includes a camera configured to obtain a captured image; a storage configured to store a food intake history; a processor configured to extract a food image from the captured image and determine a food type of a food item included in the captured image based on feature information of the extracted food image and the previously stored food intake history; and a display configured to display relevant information about the determined food type.

Claims

1. An electronic device comprising:

a display;

a camera;

a storage; and

a processor configured to;

control the camera to capture a plurality of images,

store the plurality of captured images in the storage,

classify the plurality of stored images into a first group of images and a second group of images according to whether or not at least one food item is included in the plurality of stored images,

control the display to display the first group of images including at least one food item,

based on receiving a user input for selecting an image among the first group of images, control the display to display the image,

based on receiving a user input for selecting an area of the displayed image, obtain a food image by cropping a food item corresponding to the selected area of the displayed image,

identify a food type of the obtained food image as a recognition result based on first feature information of a plurality of food types, a food intake history stored in the storage and second feature information of the obtained food image,

based on the recognition result corresponding to food to be eaten by a user, control the display to display relevant information corresponding to the identified food type, and

based on the recognition result not corresponding to the food to be eaten by the user, receive a user input for selecting a food type, and update the first feature information of the plurality of food types and the food intake history by matching the selected food type and the second feature information.

2. The electronic device as claimed in claim 1, wherein the processor is further configured to:

apply a weight to each food type based on the food intake history, and,

based on the food type of the obtained food image being identified to be one of a plurality of different food types, identify the food type of the obtained food image to be of a food type having a highest weight among the plurality of different food types.

3. The electronic device as claimed in claim 2, wherein the food intake history comprises food types that have been eaten by the user and respective intake frequencies associated with the food types, and

wherein the processor is further configured to apply the weight to each food type in accordance with an order of intake frequency and store the food intake history in the storage.

4. The electronic device as claimed in claim 3, wherein the processor is further to:

control the display to display a list of food types identified based on the second feature information of the obtained food image, the list being sorted in order of weight, and

identify the food type of the obtained food image to be of a food type selected from the list according to an additional user input.

5. The electronic device as claimed in claim 1, wherein the processor is further configured to:

based on a preset event, control to display, on the display, a user interface (UI) in a timeline in which the first group of images are aligned and displayed in order of a capturing time.

6. The electronic device as claimed in claim 1, wherein the relevant information includes at least one of the identified food type and calorie information.

7. The electronic device as claimed in claim 1, wherein the processor is further configured to control to update the food intake history according to at least one of an additional user input and the identified food type.

8. The electronic device as claimed in claim 1, wherein the processor is further configured to, based on a plurality of food images being obtained from the area, control the display to provide a user interface (UI) for selecting at least one of the obtained plurality of food images and identify the food type of the selected food image through the UI.

9. The electronic device as claimed in claim 6, wherein the processor is further configured to:

control to identify a calorie amount that the user has eaten based on an amount of the identified food type, the amount of the identified food type being input by the user,

control the display to display the identified calorie amount on the display, and

control to store, in the storage, the input amount of the identified food type as information for identifying an amount of food of a same food type as the identified food type.

10. The electronic device as claimed in claim 1, wherein the processor is further configured to:

control to transmit, through a communication interface of the electronic device, the second feature information of the obtained food image to an external server,

control to receive, from the external server, relevant information about a food type identified by the external server based on the second feature information of the obtained food image and the food intake history, and

control to display the received relevant information.

11. A control method of an electronic device, the method comprising:

obtaining a plurality of images through a camera of the electronic device;

classifying the plurality of images into a first group of images and a second group of images according to whether or not at least one food item is included in the plurality of images;

displaying the first group of images including at least one food item, on a display of the electronic device;

based on receiving a user input for selecting an image among the first group of images, displaying the image;

based on receiving a user input for selecting an area of the displayed image, obtaining a food image by cropping a food item corresponding to the selected area of the displayed image;

identifying a food type of the obtained food image as a recognition result based on first feature information of a plurality of food types, a food intake history stored in the electronic device and second feature information of the obtained food image;

based on the recognition result corresponding to food to be eaten by a user, displaying relevant information corresponding to the identified food type; and

based on the recognition result not corresponding to the food to be eaten by the user, receiving a user input for selecting a food type and updating the first feature information of the plurality of food types and the food intake history by matching the selected food type and the second feature information.

12. The method as claimed in claim 11, further comprising applying a weight to each food type based on the stored food intake history,

wherein the identifying the food type comprises based on the food type of the obtained food image being identified to be one of a plurality of different food types, identifying the food type of the obtained food image to be of a food type having a highest weight among the plurality of different food types.

13. The method as claimed in claim 12, wherein the food intake history comprises food types that have been eaten by a user and respective intake frequencies associated with the food types, and

wherein the applying of the weight comprises applying the weight to each food type in accordance with an order of intake frequency, and storing the food intake history.

14. The method as claimed in claim 13, wherein the identifying the food type comprises:

displaying a list of food types identified based on the second feature information of the obtained food image, the list being sorted in order of weight; and

identifying the food type of the obtained food image to be of a food type selected from the list according to an additional user input.

15. The method as claimed in claim 11, further comprising storing the plurality of images in the electronic device,

wherein the displaying the first group of images comprises:

displaying a user interface (UI) in a timeline in which the first group of images are aligned and displayed in order of a capturing time.

16. The method as claimed in claim 11, wherein the relevant information includes at least one of the identified food type and calorie information.

17. The method as claimed in claim 11, further comprising updating the food intake history according to at least one of an additional user input and a result of the identifying the food type.

18. The method as claimed in claim 11, wherein the identifying the food type comprises, based on a plurality of food images being obtained from the area, providing a user interface (UI) for selecting at least one of the obtained plurality of food images and identifying the food type of the selected food image through the UI.

19. The method as claimed in claim 16, wherein the displaying further comprises:

identifying a calorie amount that the user has eaten based on an amount of the identified food type, the amount of the identified food type being input by the user;

displaying the identified calorie amount on the display of the electronic device; and

storing the input amount of the identified food type as information for determining an amount of food of a same food type as the identified food type in the electronic device.

20. A non-transitory computer-readable medium having recorded thereon a computer program for executing the control method as claimed in claim 11.