Estimation of food volume and carbs

• US 9,892,501 B2
• Filed: 12/22/2015
• Issued: 02/13/2018
• Est. Priority Date: 07/02/2013
• Status: Expired due to Fees

• Current Assignee: Roche Diabetes Care Inc. (Roche Holding AG), The University Of Bern, University Clinic For Endocrinology And Arotorg Center
• Original Assignee: Roche Diabetes Care Inc. (Roche Holding AG), The University of Bern
• Inventors: Dehais, Joachim, Greenburg, Alan, Shevchick, Sergey, Soni, Abhishek, Anthimpoulos, Marios, Mougiakakou, Stavroula
• Primary Examiner(s): Bhatnagar, Anand
• Application Number:
  US14/978,433
• Publication Number:
  US 20160163037A1
• Time in Patent Office: 784 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30128 Food products
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G16H 20/60 relating to nutrition control, e.g. diets

Abstract

There is disclosed a system for estimating the volume of food on a plate, for example meal, with a mobile device. The system uses a camera and a light pattern projector. Images of the food with and without a projected light pattern on it enable to compute the tridimensional shape and volume, while image segmentation and recognition steps estimate one or more food types in said images. By applying accessible knowledge databases, the carbs content is estimated and the associated insulin bolus doses are provided. Developments comprise coding of the light pattern, different light sources and associated wavelengths, motion compensations, additional optics, estimation of fat content and associated multi-wave boluses. The invention can be implemented in a glucometer or in an insulin pump controller provided with a test strip port or with a mobile phone.

Claims

1. A system for estimating the carbs content of food on a plate (single or multiple), said system comprising instructions which when performed by a processor causes the system to operate to:

receive a first image of the food from an image acquisition component;

project a light pattern on the surface of the food with a light pattern projection component;

receive a second image of the food being highlighted with said light pattern from the image acquisition component;

identify a movement of the image acquisition component between the first and second image;

correct the first and/or the second image by an operation which compensates the movements of the image acquisition component throughout the image acquisition operation based on the artifacts to get a corrected pair of images;

subtract the first and second images of the corrected pair of images;

identify the projected light pattern on the food based upon a subtraction of the first and second corrected image; and

compute the tridimensional shape and the volume of the food given the deformations of the projected light pattern;

make segmentation and recognition of the food from the first image; and

compute the carbs content of the food based upon the volume, segmentation and recognition of food via use of nutritional databases.

2. The system of claim 1, wherein the irradiation power of the light pattern projection component is inferior to 5 mW and operates continuously, or in impulse mode with pulse frequencies between around 0.5 Hz up to 10 Hz.

3. The system of claim 1, wherein the light pattern projection component comprises one or more light sources chosen from the list comprising a low intensity semiconductor LASER diode, a LED, an organic LED (OLED), a pre-existing mobile phone flash such as a white LED or a miniaturized halogen lamp or a combination thereof with an electrical power consumption less than 0.7 W.

4. The system of claim 1, wherein the light pattern projection component comprises a light source and an optical objective adapted to form and project and/or to focus the light pattern onto the food, wherein the power density is inferior to the predefined value of 55 mWt/cm².

5. The system of claim 1, wherein the relative pose and orientation of the light pattern projection component and the image acquisition component is static during image acquisition or light pattern projection.

6. The system of claim 1, wherein the light pattern has a bright area in which the power density is inferior to 55 mWt/cm².

7. The system of claim 1, wherein the light pattern is composed of geometrical motifs such as sequences of stripes, or dots, or repetitive graphical elements or of a combination thereof.

8. The system of claim 1, wherein the light pattern is coded by color and/or phase and/or amplitude modulation.

9. The system of claim 8, wherein the coding of the light pattern is predefined and is synchronized with the image acquisition component and data processing.

10. The system of claim 1, wherein the compensation to the movements of the image acquisition component throughout the image acquisition operation is performed by processing data received from a motion sensor.

11. The system of claim 1, wherein the compensation of the movements of the image acquisition component throughout the image acquisition operation is performed by multi-view geometry methods, by projective warping, by piecewise linear, projective, or higher order warping, by deconvolution, by oriented image sharpening, or by optical flow detection before or after or in an iterative refinement process with the subtraction of the images.

12. The system of claim 1, wherein the food segmentation and recognition comprise one or more of the operations of segmenting the image, identifying color and/or texture features of segmented parts of the image and performing machine learning based classification for one or more segmented parts of the image or a combination thereof.

13. The system of claim 12, further comprising instructions which when performed by a processor causes the system to operate to estimate one or more meal characteristics of the meal captured in the first or second image by multiplying the estimated volumes of the determined food types by unitary volumetric values retrieved from a database, said database being accessed from the Internet, and/or stored locally on the device, and/or determined from food labels by using OCR and/or associated with geolocation data and/or provided by the user.

14. The system of claim 13, wherein the one or more characteristics, of the meal or of parts thereof, are one or more of carbs content, fat content, protein content, Glycemic Index (GI), Glycemic Load (GL) and/or Insulin Index (II) or a combination thereof.

15. The system of claim 14, further comprising instructions which when performed by a processor causes the system to operate to provide an insulin dose recommendation and/or a bolus profile advice based on said one or more meal characteristics.

16. The system of claim 1, wherein the system is configured to run automatically and project the light pattern via a voice command or a gesture command and/or by a touchscreen command and/or by a geo position and/or by following a predefined time schedule.

17. The system of claim 1, wherein the image acquisition component and the light pattern projection component are embedded in a mobile device such as a glucometer or an insulin pump controller provided with a test strip port or a mobile phone or a smartphone.

18. The system of claim 17 wherein the light pattern projection component is attachable to the mobile device via a clip or via insertion into an electrical contact slot in particular such as a power charge slot or a USB slot.

19. The system of claim 1, wherein the first or the second image are based upon a video frame.

20. The system of claim 1, wherein one or more operations of the system are continuously automatically repeated until the acquisition of the first and/or the second image is considered as sufficient based on predefined thresholds associated with criteria that comprises one or more of image quality, associated measurements of handshakes, time delays between still images or video frames, resulting light pattern or a combination thereof.

21. The system of claim 1, wherein the image acquisition component is embedded in a handheld device and has a sensitivity between around 0.3 up to around 3 lux.

22. The system of claim 1, wherein the light pattern projection component has an electrical power consumption between 0.05 W and 0.7 W.

23. A non-transitory computer readable medium encoded with an information processing program for use in an information processing device, said information processing program comprising instructions which when executed by a processor in the information processing device causes the information processing device to perform the operations of claim 1.

24. A non-transitory computer product comprising the non-transitory computer readable medium according to claim 23.