METHOD AND APPARATUS FOR TRACKING OF FOOD INTAKE AND OTHER BEHAVIORS AND PROVIDING RELEVANT FEEDBACK

• US 20180300458A1
• Filed: 02/27/2017
• Published: 10/18/2018
• Est. Priority Date: 01/28/2016
• Status: Active Grant

• Current Assignee: Medtronic Minimed Incorporated (Medtronic Plc)
• Original Assignee: Savor Labs, Inc.
• Inventors: Vleugels, Katelijn, Marianetti II, Ronald
• Granted Patent: US 10,102,342 B1
• Time in Patent Office: Days
• CPC Class Codes: A61B 2505/07 Home care
• CPC Class Codes: A61B 2560/0209 adapted for power saving
• CPC Class Codes: A61B 2562/0219 Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
• CPC Class Codes: A61B 5/0022 Monitoring a patient using a global network, e.g. telephone networks, internet
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/1121 Determining geometric values, e.g. centre of rotation or angular range of movement
• CPC Class Codes: A61B 5/1123 Discriminating type of movement, e.g. walking or running A61B5/1116, A61B5/112 take precedence
• CPC Class Codes: A61B 5/16 Devices for psychotechnics using teaching or educational appliances G09B1/00 - G09B7/00; Testing reaction times ; Devices for evaluating the psychological state
• CPC Class Codes: A61B 5/4866 Evaluating metabolism A61B5/083 takes precedence
• CPC Class Codes: A61B 5/4875 Hydration status, fluid retention of the body
• CPC Class Codes: A61B 5/7282 Event detection, e.g. detecting unique waveforms indicative of a medical condition A61B5/4094, A61B5/4818, A61B5/349, A61B5/0823 take precedence
• CPC Class Codes: G16H 20/60 relating to nutrition control, e.g. diets
• CPC Class Codes: G16H 40/63 for local operation
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: G16H 50/70 for mining of medical data, e.g. analysing previous cases of other patients
• CPC Class Codes: G16H 70/60 relating to pathologies

Abstract

A sensing device monitors and tracks food intake events and details. A processor, appropriately programmed, controls aspects of the sensing device to capture data, store data, analyze data and provide suitable feedback related to food intake. More generally, the methods might include detecting, identifying, analyzing, quantifying, tracking, processing and/or influencing, related to the intake of food, eating habits, eating patterns, and/or triggers for food intake events, eating habits, or eating patterns. Feedback might be targeted for influencing the intake of food, eating habits, or eating patterns, and/or triggers for those. The sensing device can also be used to track and provide feedback beyond food-related behaviors and more generally track behavior events, detect behavior event triggers and behavior event patterns and provide suitable feedback.

Claims

1. A method of sensing wearer activity using an electronic device worn by a wearer, the method comprising:

providing storage for event-specific parameters, initialized for a food intake event;

providing storage for an event state value, wherein the event state value is one of an out-of-event state or an in-event state, with the event state value initialized to the out-of-event state;

determining, using a processor in the electronic device or communicatively coupled to the electronic device, a set of sensor readings from a set of sensors in the electronic device, wherein at least one sensor reading of the set of sensor readings measures a movement of a body part of the wearer and is usable to modify the event state value for use in initiating recording gestures and thus alter an event state of the electronic device;

determining, from the set of sensor readings and using the processor, a first potential gesture of the wearer including a gesture type of the first potential gesture, wherein some of the gesture types are members of a first set of gesture types, the first set of gesture types including a sip gesture type;

determining, using the processor, if the first potential gesture has a sip gesture type by distinguishing a sip gesture from other gestures, thereby determining a determined gesture type of the first potential gesture;

determining, from the set of sensor readings and using the processor, a confidence level related to the first potential gesture, wherein the confidence level relates to a level of confidence that the gesture type of the first potential gesture was correctly determined and would be indicative of a change of the event state;

if the confidence level is above or at a threshold and the gesture type is a member of the first set of gesture types;

(a) modifying, using the processor, the event state value in the storage for the event state value from the out-of-event state to the in-event state;

(b) recording the first potential gesture as a first gesture of the food intake event; and

(c) recording the determined gesture type;

changing, using the processor, the electronic device to a higher performance state in response to the event state value being modified from the out-of-event state to the in-event state, wherein the higher performance state comprises one or more of additional power supplied to sensors of the set of sensors, additional power being drawn by the sensors of the set of sensors, a reduced latency of a communications channel, an increased sensor sampling rate of the sensors of the set of sensors, additional computations performed by the processor, and/or activation of a data recorder;

determining, from a set of additional sensor readings and using the processor, additional gestures of the wearer, each having a respective gesture type; and

if the event state value stored in the storage for the event state value is the in-event state;

(a) recording the first gesture and the additional gestures as a gesture sequence of the food intake event; and

(b) deriving the event-specific parameters about the food intake event from at least some gestures of the gesture sequence.

3. The method of claim 1, wherein the first set of gesture types consists of liquid intake gesture types and a second set of gesture types consists of gesture types unrelated to liquid intake.

4. The method of claim 3, wherein the second set of gesture types include a bite gesture type, wherein distinguishing sip gestures from other gestures includes distinguishing sip gestures from bite gestures.

7. The method of claim 1, wherein, if the determined gesture type is the sip gesture type, the event-specific parameters are measures of the wearer'"'"'s intake of liquids.

8. The method of claim 1, further comprising:

if the confidence level is below the threshold and if the gesture type is a member of the first set of gesture types;

(a) determining, from the set of sensor readings and/or the set of additional sensor readings, a subsequent gesture that follows the first potential gesture, the subsequent gesture having a respective gesture type that is a member of the first set of gesture types; and

(b) determining an updated confidence level related to the first potential gesture and the subsequent gesture, wherein the updated confidence level reflects an update of the level of confidence that the gesture type of the first potential gesture was correctly determined.

11. The method of claim 1, wherein the gesture sequence includes, for gestures in the gesture sequence, a label of the gesture type and a time of occurrence of the gesture, the method further comprising:

determining, from the gesture sequence of the food intake event and/or from times of occurrence of gestures in the gesture sequence of the food intake event, an end of the food intake event; and

if the end of the food intake event is determined, updating the event-specific parameters based on portions of the gesture sequence that occurred between a start of the food intake event and the end of the food intake event.

12. The method of claim 11, further comprising modifying the event state value to the out-of-event state if the end of the food intake event is determined and further changing, using the processor, the electronic device to a lower performance state in response to the event state value being modified from the in-event state to the out-of-event state, wherein the lower performance state comprises one or more of power reduction to sensors of the set of sensors, a reduction in power being drawn by the sensors of the set of sensors, increased latency of a communications channel, and/or a decreased sensor sampling rate of the sensors of the set of sensors.

13. The method of claim 1, wherein the set of sensor readings includes sensor readings fa) from one or more accelerometer that measures movement of an arm of the wearer, (b) from a gyroscope that measures rotation of the arm of the wearer, or (c) both.

16. The method of claim 1, further comprising:

determining that the first potential gesture has the sip gesture type;

determining, from a duration of a hand of the wearer near a mouth of the wearer and/or an amount of rotation of a wrist of the wearer, a sip size; and

using the sip size in a hydration tracking process.

30. The method of claim 1, wherein the sensors of the set of sensors that are supplied additional power, that draw additional power, and/or use the increased sensor sampling rate when in the higher performance state is less than all of the set of sensors, such that some of the set of sensors might draw power, be supplied power, or use a sensor sampling rate that is the same or similar when in the higher performance state as when not in the higher performance state.

2. (canceled)

18. A system for sensing wearer activity, comprising:

a set of sensors of an electronic device worn by a wearer for sensing the wearer activity or portions thereof including a movement of a body part of the wearer;

storage for event-specific parameters, initialized for a food intake event;

storage, within the electronic device, for an event state value, wherein the event state value is one of an out-of-event state or an in-event state, with the event state value initialized to the out-of-event state;

a processor in the electronic device that determines a set of sensor readings from the set of sensors, the set of sensor readings usable to modify the event state value to initiate recording gestures and thus alter an event state of the electronic device; and

program code, stored in the electronic device or in a component of the system in communication with the electronic device, executable by the processor in the electronic device or or communicatively coupled to the electronic device, comprising;

a) program code for determining, from the set of sensor readings and using the processor, a first potential gesture of the wearer including a gesture type of the first potential gesture, wherein some of the gesture types are members of a first set of gesture types, the first set of gesture types including a bite gesture type and a sip gesture type;

b) program code for determining, using the processor, if the first potential gesture has a gesture type that is a member of the first set of gesture types and, when the gesture type of the first potential gesture has a gesture type that is a member of the first set of gesture types, determining if the first potential gesture has a bite gesture type or a sip gesture type by distinguishing a bite gesture from a sip gesture;

c) program code for determining, from the set of sensor readings and using the processor, a confidence level related to the first potential gesture, wherein the confidence level relates to a level of confidence that the gesture type of the first potential gesture was correctly determined and would be indicative of a change of the event state;

d) program code for determining if the confidence level is above or at a threshold and the gesture type is a member of the first set of gesture types and when the confidence level is above or at the threshold and the gesture type is a member of the first set of gesture types, modifying the event state value in the storage for the event state value from the out-of-event state to the in-event state and recording the first potential gesture as a first gesture of the food intake event;

e) program code for changing, using the processor, the electronic device to a higher performance state in response to the event state value being modified from the out-of-event state to the in-event state, wherein the higher performance state comprises one or more of additional power supplied to sensors of the set of sensors, additional power being drawn by the sensors of the set of sensors, a reduced latency of a communications channel, an increased sensor sampling rate of the sensors of the set of sensors, additional computations performed by the processor, and/or activation of a data recorder;

f) program code for determining, from a set of additional sensor readings and using the processor, additional gestures of the wearer, each having a respective gesture type; and

g) program code for determining if the event state value stored in the storage for the event state value is the in-event state and when the event state value stored in the storage for the event state value is the in-event state, recording the first gesture with its gesture type and the additional gestures as a gesture sequence of the food intake event and deriving the event-specific parameters about the food intake event from at least some gestures of the gesture sequence.

20. The system of claim 18, wherein the first set of gesture types consists of food intake gesture types and a second set of gesture types consists of gesture types unrelated to food intake.

21. The system of claim 18, further comprising determining, for gestures that are of the bite gesture type, which type of a utensil the wearer is using for the gesture type.

22. The system of claim 18, wherein the gesture type is the sip gesture type and the event-specific parameters are measures of the wearer'"'"'s intake of liquids.

23. The system of claim 18, further comprising program code for determining a condition where the confidence level is below the threshold the gesture type is a member of the first set of gesture types and when the condition is present, determining, from the set of sensor readings and/or the set of additional sensor readings, a subsequent gesture that follows the first potential gesture, the subsequent gesture having a respective gesture type that is a member of the first set of gesture types and determining an updated confidence level related to the first potential gesture and the subsequent gesture, wherein the updated confidence level reflects an update of the level of confidence that the gesture type of the first potential gesture was correctly determined.

24. The system of claim 18, wherein the gesture sequence includes, for gestures in the gesture sequence, a label of a gesture type and a time of occurrence of the gesture, and where the storage for event-specific parameters comprises storage for an indication of an end of the food intake event, determined from the gesture sequence of the food intake event and/or from times of occurrence of gestures in the gesture sequence of the food intake event, with the event-specific parameters being updated based on portions of the gesture sequence that occurred between a start of the food intake event and the end of the food intake event.

25. The system of claim 18, further comprising controls for changing the electronic device to a lower performance state in response to the event state value changing from the in-event state to the out-of-event state, wherein the lower performance state comprises one or more of power reduction to sensors of the set of sensors, a reduction in power being drawn by the sensors of the set of sensors, increased latency of a communications channel, and/or a decreased sensor sampling rate of the sensors of the set of sensors.

26. The system of claim 18, wherein at least one sensor comprises one or more accelerometer that measures movement of an arm of the wearer and a gyroscope that measures rotation of the arm of the wearer.

27. The system of claim 26, wherein the program code for distinguishing the bite gesture type from the sip gesture type uses data from the one or more accelerometer and the gyroscope and takes into account changes in an angle of roll of the body part of the wearer and/or variances of accelerometer or gyroscope readings across one or more axis for a duration of time and, based on a duration of a hand of the wearer near a mouth of the wearer and/or an amount of rotation of a wrist of the wearer, detects a sip size usable in a hydration tracking process.

31. The system of claim 18, wherein the sensors of the set of sensors that are supplied additional power, that draw additional power, and/or use the increased sensor sampling rate when in the higher performance state is less than all of the set of sensors, such that some of the set of sensors might draw power, be supplied power, or use a sensor sampling rate that is the same or similar when in the higher performance state as when not in the higher performance state.

19. (canceled)

28. A method of sensing wearer activity using an electronic device worn by a wearer, the method comprising:

providing storage for event-specific parameters, initialized for a food intake event;

providing storage for an event state value, wherein the event state value is one of an out-of-event state or an in-event state, with the event state value initialized to the out-of-event state;

determining, using a processor in the electronic device or communicatively coupled to the electronic device, a set of sensor readings from a set of sensors in the electronic device, wherein at least one sensor reading of the set of sensor readings measures a movement of a body part of the wearer and is usable to modify the event state value for use in initiating recording gestures and thus alter an event state of the electronic device;

determining, from the set of sensor readings and using the processor, a first potential gesture of the wearer including a gesture type of the first potential gesture, wherein some of the gesture types are members of a first set of gesture types, the first set of gesture types including a bite gesture type and a sip gesture type;

determining, using the processor, if the first potential gesture has a bite gesture type or a sip gesture type by distinguishing a bite gesture from a sip gesture, thereby determining a determined gesture type of the first potential gesture;

determining, from the set of sensor readings and using the processor, a confidence level related to the first potential gesture, wherein the confidence level relates to a level of confidence that the gesture type of the first potential gesture was correctly determined and would be indicative of a change of the event state;

if the confidence level is above or at a threshold and the gesture type is a member of the first set of gesture types;

(a) modifying, using the processor, the event state value in the storage for the event state value from the out-of-event state to the in-event state;

(b) recording the first potential gesture as a first gesture of the food intake event; and

(c) recording the determined gesture type;

changing, using the processor, the electronic device to a higher performance state in response to the event state value being modified from the out-of-event state to the in-event state, wherein the higher performance state comprises one or more of additional power supplied to sensors of the set of sensors, additional power being drawn by the sensors of the set of sensors, a reduced latency of a communications channel, an increased sensor sampling rate of the sensors of the set of sensors, additional computations performed by the processor, and/or activation of a data recorder;

determining, from a set of additional sensor readings and using the processor, additional gestures of the wearer, each having a respective gesture type; and

if the event state value stored in the storage for the event state value is the in-event state;

(a) recording the first gesture and the additional gestures as a gesture sequence of the food intake event; and

(b) deriving the event-specific parameters about the food intake event from at least some gestures of the gesture sequence.

5. The method of claim 28, further comprising determining, for gestures that are of the bite gesture type, which type of a utensil the wearer is using for the gesture.

6. The method of claim 28, wherein the first set of gesture types consists of food intake gesture types and a second set of gesture types consists of gestures unrelated to food intake, the method further comprising distinguishing between a first food intake gesture type and a second food intake gesture type, wherein at least one parameter of the event-specific parameters is derived in dependence upon the distinguishing between the first food intake gesture type and the second food intake gesture type.

9. The method of claim 28, wherein deriving the event-specific parameters about the food intake event from at least some gestures of the gesture sequence comprises:

determining which subset of gestures of the gesture sequence are relevant to a parameter; and

deriving the event-specific parameters considering the subset of gestures.

10. The method of claim 9, wherein the subset of gestures consists of gestures of the bite gesture type and the event-specific parameters comprise a bite count and a measure of the wearer'"'"'s pace of eating.

14. The method of claim 28, further comprising:

distinguishing a gesture having the bite gesture type from a gesture having the sip gesture type based on accelerometer sensor data and gyroscope sensor data, wherein sensing takes into account changes in an angle of roll of the body part of the wearer and/or variances of accelerometer or gyroscope readings across one or more axis for a duration of time.

15. The method of claim 28, further comprising:

distinguishing, based on detected rotation of a wrist of the wearer, between the bite gesture type and the sip gesture type.

29. A method of sensing wearer activity using an electronic device worn by a wearer, the method comprising:

providing storage for event-specific parameters, initialized for a food intake event;

providing storage for an event state value, wherein the event state value is one of an out-of-event state or an in-event state, with the event state value initialized to the out-of-event state;

determining, using a processor in the electronic device or communicatively coupled to the electronic device, a set of sensor readings from a set of sensors in the electronic device, wherein at least one sensor reading of the set of sensor readings measures a movement of a body part of the wearer and is usable to modify the event state value for use in initiating recording gestures and thus alter an event state of the electronic device;

determining, from the set of sensor readings and using the processor, a first potential gesture of the wearer including a gesture type of the first potential gesture, wherein some of the gesture types are members of a first set of gesture types, the first set of gesture types including a bite gesture type;

determining, using the processor, if the first potential gesture has a bite gesture type by distinguishing a bite gesture from a sip gesture, thereby determining a determined gesture type of the first potential gesture;

determining, from the set of sensor readings and using the processor, a confidence level related to the first potential gesture, wherein the confidence level relates to a level of confidence that the gesture type of the first potential gesture was correctly determined and would be indicative of a change of the event state;

if the confidence level is above or at a threshold and the gesture type is a member of the first set of gesture types;

(a) modifying, using the processor, the event state value in the storage for the event state value from the out-of-event state to the in-event state;

(b) recording the first potential gesture as a first gesture of the food intake event; and

(c) recording the determined gesture type;

changing, using the processor, the electronic device to a higher performance state in response to the event state value being modified from the out-of-event state to the in-event state, wherein the higher performance state comprises one or more of additional power supplied to sensors of the set of sensors, additional power being drawn by the sensors of the set of sensors, a reduced latency of a communications channel, an increased sensor sampling rate of the sensors of the set of sensors, additional computations performed by the processor, and/or activation of a data recorder;

determining, from a set of additional sensor readings and using the processor, additional gestures of the wearer, each having a respective gesture type; and

if the event state value stored in the storage for the event state value is the in-event state;

(a) recording the first gesture and the additional gestures as a gesture sequence of the food intake event; and

(b) deriving the event-specific parameters about the food intake event from at least some gestures of the gesture sequence.

17. The method of claim 29, further comprising:

determining that the first potential gesture is has the bite gesture type;

determining, from a duration of a hand of the wearer near a mouth of the wearer and/or an amount of rotation of a wrist of the wearer, a bite size; and

using the bite size in an eating tracking process.