Method, apparatus and system for food intake and physical activity assessment

• US 10,006,896 B2
• Filed: 11/14/2012
• Issued: 06/26/2018
• Est. Priority Date: 11/14/2011
• Status: Active Grant

• Current Assignee: University of Pittsburgh of The Commonwealth System of Higher Education
• Original Assignee: University of Pittsburgh of The Commonwealth System of Higher Education
• Inventors: Fernstrom, John, Fernstrom, Madelyn, Jia, Wenyan, Sclabassi, Robert, Sun, Mingui
• Primary Examiner(s): Layno, Carl H
• Assistant Examiner(s): Pahakis, Manolis
• Application Number:
  US13/676,468
• Publication Number:
  US 20130267794A1
• Time in Patent Office: 2,050 Days
• Field of Search: None
• CPC Class Codes: A61B 5/0022 Monitoring a patient using a global network, e.g. telephone networks, internet
• CPC Class Codes: A61B 5/0024 for multiple sensor units attached to the patient, e.g. using a body or personal area network
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/01 Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue clinical contact thermometers G01K13/20
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/1112 Global tracking of patients, e.g. by using GPS
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/14551 for measuring blood gases
• CPC Class Codes: A61B 5/24 Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
• CPC Class Codes: A61B 5/6802 Sensor mounted on worn items
• CPC Class Codes: A61B 5/6805 Vests
• CPC Class Codes: A61B 5/6816 Ear lobe
• CPC Class Codes: A61B 7/00 Instruments for auscultation
• CPC Class Codes: G01N 21/00 Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light G01N3/00 - G01N19/00 take precedence
• CPC Class Codes: G01N 33/02 Food
• CPC Class Codes: G16H 20/30 relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
• CPC Class Codes: G16H 20/60 relating to nutrition control, e.g. diets
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: Y02A 90/10 Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Abstract

Electronic systems, devices and methods are provided to accurately record and analyze food intake and physical activity in a subject. A device is provided to be placed on a subject which records video using at least two video cameras positioned to record stereoscopic image pair, as well as other physiological and/or environmental data including, for example, oxygen saturation, heart rate, and environmental factors such as physical location, temperature, and humidity. Video data is analyzed along with other data obtained by the device to determine food consumption and/or physical activity of the subject, much of which is accomplished by automated computer-implemented processes.

Claims

1. A system for remote monitoring of food intake and physical activity of a subject, comprising:

a) a device configured to be worn on the subject for remote monitoring of food intake and physical activity by the subject, comprising;

a housing;

a bus inside the housing;

one or more physiological sensors connected to the bus;

two or more imaging sensors connected to the bus and positioned in the housing to record a stereo image;

an eating detector in the housing and connected to the bus, the eating detector comprising a light beam source configured in the device to pass a light beam in a path in front of the subject'"'"'s mouth and a light sensor able to measure light intensity data representative of movement of an object in the path of the light beam;

a microprocessor connected to the bus;

a data storage device connected to the bus and comprising a data storage medium;

programming instructions, stored on the data storage device, that when executed by the microprocessor cause the microprocessor to;

receive the light intensity data from the light sensor;

identify based on changes in the received light intensity data when the object is in the path of the light beam to determine when the subject is eating;

generate a signal to control the imaging sensors to obtain images of a food item being eaten by the subject when it is determined that the subject is eating; and

generate, from the images, a stereo image of the food item;

and a power supply connected to the bus;

b) a computer; and

c) a data communication interface for transferring data between the device and the computer,

wherein one or both of the device and the computer comprises one or more programs for determining a dimension of the food item based on;

the stereo image of the food item, the distance between the two or more imaging sensors, and a fitting algorithm which fits a virtual template shape to the stereo image of the food item;

and determining a physical activity of the subject wearing the device based on data from at least one of the physiological sensors, or the imaging sensors.

2. The system of claim 1, in which the fitting algorithm for determining the dimension of the food item causes the microprocessor to generate a representation of the stereo image of the food item in a three dimensional perspective, scales and orients the virtual template shape to fit the food item depicted in the stereo image representation and estimates a volume of the food item by determining the volume of the template shape.

3. The system of claim 1, in which the computer comprises a program for determining the physical activity of the subject, which comprises comparing measured data from the one or more physiological sensors to known values for various physical activities;

estimating a type of physical activity performed by the subject based on comparison between measured data and known values; and

determining physical and nutritional data including calories burned based on the type of physical activity performed and duration of the physical activity performed.

4. The system of claim 3, in which the known values for various physical activities are defined according to a series of Eigen activity values for comparison by the one or more programs for determining the physical activity of the subject wearing the device.

5. The system of claim 1, in which the one or more computer programs further comprise a program for distinguishing data associated with food and drink consumption from data obtained from one or both of the one or more physiological sensors, associated with other behaviors, or a program for distinguishing data associated with physical activity of the subject from other data obtained from the device.

6. The system of claim 3, further comprising a database comprising dietary information for different food items, and a program for determining nutritional information from food dimension data obtained by the computer program for determining a dimension of the different food items in image data obtained from the imaging sensors.

7. The system of claim 1, wherein the device further comprises a global positioning sensor and the system comprises a database comprising environmental quality information for different locations and a computer program for estimating environmental exposure of the subject based on location data obtained from the global positioning sensor, weather data obtained from environmental monitoring sensors on the device, and the database comprising environmental quality information.

8. The system according to claim 1, in which the physiological sensors comprise one or more of an accelerometer, a gyroscope, an oximeter, and a UV sensor.

9. The system of claim 8, in which one or more of the accelerometer, gyroscope, oximeter, and UV sensor are configured to record activity of the device.

10. The system of claim 1, in which the one or more physiological sensors are configured to record activity.

11. The system of claim 1, wherein the housing comprises a front surface, a rear surface, and a side surface extending between the front surface and the rear surface, and wherein a field of view of the two or more imaging sensors passes through at least one opening on the front surface of the housing to capture images of items in front of the subject, and wherein the light beam source directs the light beam through an opening on the side of the housing, such that the light beam passes in front of the subject'"'"'s mouth.