SALIENCY-BASED APPARATUS AND METHODS FOR VISUAL PROSTHESES

• US 20090112287A1
• Filed: 03/06/2008
• Published: 04/30/2009
• Est. Priority Date: 03/08/2007
• Status: Active Grant

• Current Assignee: Cortigent Incorporated, Doheny Eye Institute
• Original Assignee: Second Sight Medical Products, Inc. (Vivani Medical, Inc.)
• Inventors: Meilstrup, Peter, Parikh, Neha Jagdish, Weiland, James D., Greenberg, Robert J., McClure, Kelly H., Roy, Arup, Horsager, Alan Matthew, McMahon, Matthew J., Humayun, Mark S., Zhou, Chunhong
• Granted Patent: US 9,061,150 B2
• Time in Patent Office: Days
• US Class Current: 607/54
• CPC Class Codes: A61F 9/08 Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
• CPC Class Codes: A61N 1/36046 of the eye
• CPC Class Codes: G06V 10/20 Image preprocessing
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G09B 21/008 using visual presentation of the information for the partially sighted

Abstract

Saliency-based apparatus and methods for visual prostheses are disclosed. A saliency-based component processes video data output by a digital signal processor before the video data are input to the retinal stimulator. In a saliency-based method, an intensity stream is extracted from an input image, feature maps based on the intensity stream are developed, plural most salient regions of the input image are detected and one of the regions is selected as a highest saliency region.

Claims

1. A visual prosthesis comprising:

an image processor to process video data from an imager;

a visual stimulator to stimulate visual neurons on a patient; and

a saliency-based component to process the video data before the video data are input to the visual stimulator, the saliency-based component being configured to apply a saliency algorithm on the video data.

2. The visual prosthesis of claim 1, wherein the image processor and the saliency-based component are part of a video data processing unit.

3. The visual prosthesis of claim 1, wherein the image processor and the saliency-based component operate in parallel.

4. The visual prosthesis of claim 1, wherein the image processor is a digital signal processor (DSP), a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

5. The visual prosthesis of claim 1, wherein the saliency-based component operates on the video data after or before the image processor.

6. The visual prosthesis of claim 1, wherein the saliency-based component is a software component.

7. The visual prosthesis of claim 1, wherein the image processor runs on a limited amount of power compliant with power limits of devices operating on visual prostheses.

8. The visual prosthesis of claim 1, wherein the image processor has a processing rate of 3 to 30 frames per second.

9. The visual prosthesis of claim 8, wherein the image processor has a processing rate of 3 to 5 frames per second.

10. The visual prosthesis of claim 1, wherein the saliency-based component, although being able to process multiple information streams of the video data, only processes an intensity stream of the video data.

11. The visual prosthesis of claim 1, wherein the saliency-based component is adapted to be turned on and turned off on demand.

12. The visual prosthesis of claim 11, wherein turning on and turning off the saliency-based component on demand is effected by a patient using the visual prosthesis.

13. The visual prosthesis of claim 1, wherein the video data pertain to multiple images.

14. The visual prosthesis of claim 13, wherein the saliency-based component cues on motion.

15. The visual prosthesis of claim 1, wherein the imager is sensible also to infrared, ultraviolet, or X-ray spectra, or ultrasound.

16. The visual prosthesis of claim 1, further comprising a pattern recognition component.

17. The visual prosthesis of claim 16, wherein the pattern recognition component comprises a table of salient stored patterns for comparison with the results of the saliency component.

18. A method of providing video data to a subject with an implanted visual stimulator, the implanted visual stimulator associated to an image processor processing video data from an input image, comprising:

extracting only an intensity stream from the input image;

developing feature maps based on the intensity stream;

detecting a plurality of most salient regions of the input image based on the feature maps;

selecting one salient region of the most salient regions as a highest saliency region; and

providing highest saliency region information to the subject.

19. The method of claim 18, wherein the plurality of most salient regions are three most salient regions.

20. The method of claim 18, further comprising constructing an image pyramid for the intensity stream after extracting only an intensity stream from the input image and before developing feature maps based on the intensity stream.

21. The method of claim 18, further comprising constructing conspicuity maps after developing feature maps based on the intensity stream and before detecting a plurality of most salient regions of the input image based on the feature maps.

22. The method of claim 18, further comprising averaging each region of the plurality of most salient regions after detecting the plurality of most salient regions and before selecting one salient region of the most salient regions as a highest saliency region.

23. The method of claim 18, the method being performed in absence of video data normalization.

24. The method of claim 18, wherein also information in the infrared, ultraviolet, or X-ray spectra, or ultrasound information is provided.

25. An apparatus for providing stimulation signals to a visual stimulator, comprising:

a saliency map generator to process a video input coming from a camera image;

a downsampling component to downsample saliency map resolution to visual stimulation resolution; and

a stimulation component to provide stimulation signals based on the downsampled saliency map resolution.

26. The apparatus of claim 25, wherein the visual stimulator is implanted on a subject.

27. The apparatus of claim 25, further comprising:

an image fusion component processing i) the video input coming from the camera image and ii) the saliency map, and outputting a signal to the downsampling component, the downsampling component downsampling the signal output from the image fusion component.

28. The apparatus of claim 27, wherein the image fusion component is configured to apply wavelet fusion to i) and ii).

29. The apparatus of claim 25, wherein also information in the infrared, ultraviolet, or X-ray spectra, or ultrasound information is provided.

30. An apparatus for providing stimulation signals to a visual stimulator, comprising:

a saliency map generator to process a video input coming from a camera image and generate a saliency map; and

a stimulation component to provide stimulation signals to the visual stimulator based on the saliency map generated by the saliency map generator.

31. The apparatus of claim 30, wherein the stimulation signals are provided to the visual stimulator according to a rastering pattern.

32. The apparatus of claim 31, wherein the visual stimulator comprises groups of electrodes, each group being assigned a unique stimulation pattern.

33. The apparatus of claim 31, wherein the visual stimulator comprises a plurality of electrodes, and wherein electrodes corresponding to higher saliency regions of the saliency map are stimulated earlier than and separately from electrodes corresponding to lower saliency regions of the saliency map.

34. A method for providing stimulation signals to a visual stimulator, comprising:

generating a saliency map of an input image;

separating the input image into multiple regions according to saliency values of the saliency map; and

separating stimulation of the multiple regions through the visual stimulator in time in accordance with a rastering process.

35. The method of claim 34, wherein separating the input mage into multiple regions generates high saliency regions, medium saliency regions and low saliency regions.

36. The method of claim 35, wherein the low saliency regions are grouped together in time during stimulation, while the high saliency regions and the medium saliency regions are administered independently of each other.

37. The method of claim 35, wherein the high saliency regions are administered before the medium saliency regions and the medium saliency regions are administered before the low saliency regions.

38. The method of claim 34, wherein regions of the input image falling below a saliency threshold are not input to the visual stimulator.

39. A method of providing video data to a subject with an implanted visual stimulator, the implanted visual stimulator associated to a processor processing video data from an input image, the method comprising:

detecting salient regions of the input image through a discrete wavelet transform process.

40. An apparatus for providing video data to a subject with an implanted visual stimulator, the implanted visual stimulator associated to a processor processing video data from an input image, the apparatus comprising:

a module configured to detect salient regions of the input image through a discrete wavelet transform process.

41. The apparatus of claim 40, wherein the module comprises a plurality of levels of filters.

42. The apparatus of claim 41, wherein each level of filters comprises a lowpass-and downsample component and a highpass-and-downsample component, each component of a lower level receiving a same input from the lowpass-and-downsample component of a higher level.

43. The apparatus of claim 40, wherein the discrete wavelet process operates on monodimensional data.

44. An apparatus for providing video data to a subject with an implanted visual stimulator, the implanted visual stimulator associated to a processor processing video data from an input image, the apparatus comprising:

a module configured to detect salient regions of the input image through a discrete wavelet transform process operating on two-dimensional data, the module comprising a plurality of levels of filters, each level comprising a first sub-level operating on data in a first dimension and a second sub-level operating on data in the second dimension.

45. The apparatus of claim 44, wherein each first sub-level of filters comprises a lowpass-and downsample component and a highpass-and-downsample component, and each second sub-level of filters comprises two lowpass-and-downsample components and two highpass-and-downsample components, each component of a lower level receiving a same input from the lowpass-and-downsample component of a higher level.

46. A method of providing video data to a subject, comprising:

receiving video data having a first resolution;

processing the video data in accordance to a saliency method to obtain processed video data having a second resolution, lower than the first resolution, the processing being adapted to prefer peripherally located video data to centrally located video data once the peripherally located video data is more salient than the centrally located video data; and

sending the processed video data to a visual stimulator implanted on the subject.

47. The method of claim 46, further comprising:

providing the subject with cues to alert the subject of salient region falling outside a central field of view.

48. The method of claim 47, wherein the cues are selected from the group consisting of:

video cues, audio cues, and tactile cues.

49. The method of claim 47, wherein the cues are in directions selected from the group consisting of right direction, left direction, top direction, bottom direction, top-left direction, bottom-left direction, top-right direction and bottom-right direction with respect to the central field of view.

50. The method of claim 46, wherein the video data pertain to multiple images.

51. The method of claim 50, wherein the saliency method cues on motion.

52. A visual prosthesis comprising:

a camera sensitive to ultraviolet light;

a visual stimulator to stimulate visual neurons on a patient; and

means to process video data output from the camera, the video data including data corresponding to the ultraviolet light, before the video data are input to the visual stimulator.