USER TERMINAL APPARATUS AND CONTROL METHOD THEREOF

• US 20170148162A1
• Filed: 11/25/2016
• Published: 05/25/2017
• Est. Priority Date: 11/25/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: KANG, Seong-min, HAN, Heung-woo
• Granted Patent: US 10,521,903 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables
• CPC Class Codes: G09B 19/0092 Nutrition
• CPC Class Codes: G16H 20/60 relating to nutrition control, e.g. diets
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS

Abstract

A user terminal apparatus is provided. The user terminal apparatus includes a camera configured to obtain a captured image; a storage configured to store a food intake history; a processor configured to extract a food image from the captured image and determine a food type of a food item included in the captured image based on feature information of the extracted food image and the previously stored food intake history; and a display configured to display relevant information about the determined food type.

Claims

1. A user terminal apparatus comprising:

a display;

a camera configured to capture an image;

a storage configured to store a food intake history;

a processor configured to extract a food image from the captured image, determine a food type of a food item included in the captured image based on feature information of the extracted food image and the previously stored food intake history, and control the display to display relevant information about the determined food type.

2. The user terminal apparatus as claimed in claim 1, wherein the processor is further configured to:

apply a weight to each food type based on the food intake history, and,

in response to the food item being determined to be one of a plurality of different food types, determining the food item included in the captured image to be of a food type having a highest weight among the plurality of different food types.

3. The user terminal apparatus as claimed in claim 2, wherein the food intake history comprises food types that have been eaten by a user and respective intake frequencies associated with the food types, and

wherein the processor is further configured to apply the weight to each food type in accordance with an order of intake frequency and store the food intake history in the storage.

4. The user terminal apparatus as claimed in claim 3, wherein the processor is further to:

control the display to display a list of food types determined based on the feature information of the extracted food, the list being sorted in order of weight, and

determine the extracted food image to be of a food type selected from the list according to a user input.

5. The user terminal apparatus as claimed in claim 1, wherein the processor is further configured to:

store the captured image in the storage,

in response to a preset event, select captured images that include the food image from captured images stored in the storage, and control the display to display a user interface (UI) in a timeline in which the selected captured images are aligned and displayed in order of a capturing time.

6. The user terminal apparatus as claimed in claim 1, wherein the relevant information includes at least one of the determined food type and calorie information.

7. The user terminal apparatus as claimed in claim 1, wherein the processor is further configured to update the food intake history according to at least one of a user input and a result of the determination.

8. The user terminal apparatus as claimed in claim 1, wherein the processor is further configured to, in response to a plurality of food images being extracted from the captured image, provide a user interface (UI) for selecting at least one of the extracted plurality of food images and determine the food type of the selected food image through the UI.

9. The user terminal apparatus as claimed in claim 6, wherein the processor is further configured to:

determine a calorie amount that a user has eaten based on an amount of food included in the captured image, the amount of food being input by the user,

control the display to display the determined calorie amount, and

store the input amount of food as information for determining an amount of food of a same food type as the determined food type.

10. A user terminal apparatus comprising:

a display;

a camera configured to capture an image;

a communication interface configured to communicate with an external server storing a food intake history;

a processor configured to;

extract a food image from the captured image,

transmit feature information of the extracted food image to the external server,

receive, from the external server, relevant information about a food type determined by the external server based on the feature information of the extracted food image and the food intake history, and

control the display to display the received relevant information.

11. A control method of a user terminal apparatus, the control method comprising:

obtaining a captured image;

extracting a food image from the captured image;

determining a food type of a food item included in the captured image based on feature information of the extracted food image and a previously stored food intake history; and

displaying relevant information about the determined food type.

12. The control method as claimed in claim 11, further comprising applying a weight to each food type based on the previously stored food intake history,

wherein the determining the food type comprises in response to the food item being determined to be one of a plurality of different food types, determining the food item included in the captured image to be of a food type having a highest weight among the plurality of different food types.

13. The control method as claimed in claim 12, wherein the food intake history comprises food types that have been eaten by a user and respective intake frequencies associated with the food types, and

wherein the applying of the weight comprises applying the weight to each food type in accordance with an order of intake frequency, and storing the food intake history.

14. The control method as claimed in claim 13, wherein the determining the food type comprises:

displaying a list of food types determined based on the feature information of the extracted food, the list being sorted in order of weight; and

determining the extracted food image to be of a food type selected from the list according to a user input.

15. The control method as claimed in claim 11, further comprising storing the captured image,

wherein the displaying comprises:

in response to a preset event, selecting captured images that include the food image from the stored captured images; and

displaying a user interface (UI) in a timeline in which the selected captured images are aligned and displayed in order of a capturing time.

16. The control method as claimed in claim 11, wherein the relevant information includes at least one of the determined food type and calorie information.

17. The control method as claimed in claim 11, further comprising updating the food intake history according to at least one of a user input and a result of the determining the food type.

18. The control method as claimed in claim 11, wherein the determining the food type comprises, in response to a plurality of food images being extracted from the captured image, providing a user interface (UI) for selecting at least one of the extracted plurality of food images and determining the food type of the selected food image through the UI.

19. The control method as claimed in claim 16, wherein the displaying further comprises:

determining a calorie amount that a user has eaten based on an amount of food included in the captured image, the amount of food being input by the user;

displaying the determined calorie amount; and

storing the input amount of food as information for determining an amount of food of a same food type as the determined food type.

20. A non-transitory computer-readable medium having recorded thereon a computer program for executing the control method as claimed in claim 11.