Optimal deep brain stimulation therapy with Q learning

• US 8,774,923 B2
• Filed: 03/18/2010
• Issued: 07/08/2014
• Est. Priority Date: 03/22/2009
• Status: Expired due to Fees

• Current Assignee: Sorin CRM SAS (MicroPort Scientific Corp.)
• Original Assignee: Sorin CRM SAS (MicroPort Scientific Corp.)
• Inventors: Rom, Rami
• Primary Examiner(s): Bertram, Eric D.
• Assistant Examiner(s): Bays, Pamela M
• Application Number:
  US13/257,303
• Publication Number:
  US 20120016435A1
• Time in Patent Office: 1,573 Days
• Field of Search: 607/2, 607/3, 607/45, 607/62, 607115-117, 600/544, 600/545
• US Class Current: 607/45
• CPC Class Codes: A61B 5/375 using biofeedback
• CPC Class Codes: A61B 5/6846 specially adapted to be brought in contact with an internal body part, i.e. invasive
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61N 1/36082 Cognitive or psychiatric applications, e.g. dementia or Alzheimer's disease

Abstract

A closed loop Deep Brain Stimulation (DBS) system constituted of: a physiological sensor; a multi-electrode DBS lead; an adaptive control system in communication with the physiological sensor; and an implantable pulse generator (IPG) responsive to the adaptive control system, the adaptive control system comprising a learning module operable to learn to find the optimal stimulation parameters, classify and associate patient conditions responsive to the physiological sensor with optimal stimulation parameters in a plurality of patient conditions. The adaptive DBS device control system learns to deliver the optimal stimulation parameters based on Watkins and Dayan Q learning recursive formula, the closed loop adaptive DBS control system thus finds the optimal stimulation parameters online.

Claims

1. A closed loop Deep Brain Stimulation (DBS) system comprising:

a physiological sensor, said physiological sensor comprising at least one of a brain activity sensor and a mechanical body motion sensor;

an adaptive control system in communication with said physiological sensor; and

an implantable pulse generator (IPG), said IPG arranged to provide pulses selectively to one or more electrodes of a multi-electrode DBS lead responsive to stimulation parameters provided by said adaptive control system,

said adaptive control system comprising;

a learning module arranged to determine stimulation parameters to be output to said IPG responsive to feedback received from said physiological sensor wherein said stimulation parameters comprise a selection of a particular set of said one or more electrodes of said multi-electrode DBS lead; and

a deterministic algorithmic module arranged to maintain safety limits for determined stimulation parameters provided to said IPG,

wherein said adaptive control system is arranged to operate in a plurality of states, the states comprising;

a non-adaptive state wherein said adaptive control system controls said IPG to deliver stimulation in accordance with received input stimulation parameters;

an adaptive gradient ascent (GA) learning state wherein said input stimulation parameters are modified by said adaptive control system and optimal parameters for varying patient conditions are learned responsive to feedback received from said physiological sensor, said learning comprising storing reward function values responsive to said modified stimulation parameters and resultant inputs from said physiological sensor in a Q-learning (QL) look up table; and

an adaptive QL state in which said adaptive control system operates said IPG alternately responsive to inputs from said physiologic sensor and inputs stored in said QL look up table, said alternate selection responsive to a probabilistic scheme.

2. The closed loop DBS system of claim 1, wherein said stimulation parameters further comprise at least one of a stimulation frequency, amplitude, a pulse width, and time sequences.

3. The closed loop DBS system according to claim 1, further comprising an external programmer in communication with said adaptive control system, said received input stimulation parameters provided via said external programmer.

4. The closed loop DBS system according to claim 1, wherein said physiological sensor comprises at least one of an accelerometer, a tremor sensor and a rigidity sensor.

5. The closed loop DBS system according to claim 1, wherein said physiological sensor is invasive.

6. The closed loop DBS system according to claim 1, wherein said physiological sensor is non-invasive.

7. The closed loop DBS system according to claim 1, wherein said adaptive control system comprises a neural network type processor.

8. The closed loop DBS system according to claim 1, wherein said adaptive control system further comprises:

an adaptive FAIL QL state in which a pattern recognition sigmoid neural network is operative to converge back to the received input stimulation parameters when a change of state occurs and the pattern recognition networks fails to associate the new pattern with an optimal solution.

9. The closed loop DBS system according claim 8, wherein said adaptive control system exhibits a built in priority to operate in said adaptive QL state.

10. The closed loop DBS system according to claim 9, wherein said adaptive control system transits from said adaptive QL state to one of said adaptive GA learning state and non-adaptive state responsive to one of:

a pre-defined failure; and

an override input.

11. A method for deep brain stimulation (DBS) delivered to a patient in a closed loop system, said method comprising:

providing a physiological sensor;

providing a multi-electrode DBS lead;

determining stimulation parameters to be provided via said provided multi-electrode DBS lead responsive to feedback received from said provided physiological sensor wherein said stimulation parameters comprise a selection of a particular set of said one or more electrodes of said provided multi-electrode DBS lead; and

maintaining safety limits for said stimulation parameters provided to said multi-electrode DBS lead,

wherein said determining stimulation parameters comprises;

delivering brain stimulation via said provided multi-electrode DBS lead according to programmed stimulation parameters;

machine learning the programmed stimulation parameters;

switching to an adaptive gradient ascent (GA) learning state when the machine learning converges;

modifying the programmed stimulation parameters thereby learning optimal stimulation parameters responsive to feedback from said provided physiological sensor;

storing reward function values responsive to said modified stimulation parameters and resultant inputs from said physiological sensor in a Q-learning (QL) look up table; and

in the event that said learning optimal stimulation parameters converges within predetermined parameters, switching to an adaptive QL state in which brain stimulation is provided via said provided multi-electrode DBS lead alternately responsive to inputs from said physiologic sensor and inputs stored in said QL look up table, said alternate selection responsive to a probabilistic scheme.

12. The method of claim 11, further comprising receiving said programmed stimulation parameters, said programmed stimulation parameters comprising at least one of a stimulation frequency, amplitude, a pulse width, a particular electrode of said multi-electrode DBS lead and a time sequence.

13. The method of claim 12, further comprising:

converging back to the received programmed stimulation parameters when a change of state occurs which is not associated with the optimal stimulation parameters.

14. The method of claim 13, wherein the method exhibits a built in priority to operate in said adaptive QL state.

15. The method of claim 13, further comprising transiting from said adaptive QL state to one of said adaptive GA state and non-adaptive state responsive to one of:

a pre-defined failure; and

an override input.

16. The method of claim 11, wherein said provided physiological sensor comprises at least one of a brain activity sensor and a mechanical body motion sensor.

17. The method of claim 11, wherein said provided physiological sensor comprises at least one of an accelerometer, a tremor sensor and a rigidity sensor.