Haptic system for providing sensory augmentation to a subject and method thereof

• US 10,359,855 B1
• Filed: 03/15/2018
• Issued: 07/23/2019
• Est. Priority Date: 03/15/2018
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Inventors: Vonikakis, Vasileios
• Primary Examiner(s): Nguyen, Hung T
• Application Number:
  US15/922,476
• Time in Patent Office: 495 Days
• Field of Search: 3404071, 340435, 340436, 340438, 340903, 345157, 345173, 701 1, 701301
• CPC Class Codes: B60N 2/002 Seats provided with an occupancy detection means mounted therein or thereon detection means responsive to presence or absence of children B60N2/266
• CPC Class Codes: B60N 2002/981 Warning systems, e.g. the seat or seat parts vibrates to warn the passenger when facing a danger
• CPC Class Codes: B60W 50/16 Tactile feedback to the driver, e.g. vibration or force feedback to the driver on the steering wheel or the accelerator pedal
• CPC Class Codes: G06F 2203/014 Force feedback applied to GUI
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G08B 5/36 using visible light sources
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: G09B 21/003 using tactile presentation of the information, e.g. Braille displays

Abstract

There is provided a haptic system for providing sensory augmentation to a subject, the haptic system including a haptic surface for interacting with the subject, the haptic surface including an array of haptic portions, each haptic portion configured to be controllable to perform a haptic action; an image generator configured to generate a depth map based on information obtained on a scene captured of a surrounding environment; and a controller configured to control each of the array of haptic portions based on a property of a corresponding portion of the depth map. There is also provided a corresponding method for providing sensory augmentation to a subject and a vehicle having installed therein the haptic system.

Claims

1. A haptic system for providing sensory augmentation to a driver of a potential collision, the haptic system comprising:

a haptic surface for interacting with the driver, the haptic surface comprising an array of haptic portions, each haptic portion configured to be controllable to perform a haptic action;

an image generator configured to generate a depth map based on information obtained on a scene captured of a surrounding environment by a sensor; and

a controller configured to control each of the array of haptic portions based on a property of a corresponding portion of the depth map.

2. The haptic system according to claim 1, wherein the depth map comprises a plurality of portions, the property of each portion being determined based on a degree of proximity associated with a corresponding portion of the scene captured in the depth map.

3. The haptic system according to claim 2, wherein the degree of proximity associated with a corresponding portion of the scene relates to a distance of a surface of one or more scene objects captured in the corresponding portion of the scene from a viewpoint.

4. The haptic system according to claim 2, wherein each portion of the depth map corresponds to a pixel or a group of adjacent pixels, and the property of each portion has a value corresponding to a pixel value.

5. The haptic system according to claim 1, wherein the image generator is configured to generate the depth map continuously for a plurality of consecutive time instances based on the respective information obtained on the scene continuously for the plurality of consecutive time instances, and the controller is configured to control each of the array of haptic portions continuously such that for each time instance, each of the array of haptic portions is controlled based on the property of the corresponding portion of the depth map generated for the time instance.

6. The haptic system according to claim 2, wherein each of the array of haptic portions is configured to be capable of performing the haptic action within a range of an attribute based on a control signal from the controller.

7. The haptic system according to claim 6, wherein the haptic action comprises vibration, pulsation or applied pressure.

8. The haptic system according to claim 6, wherein the attribute comprises frequency and/or amplitude.

9. The haptic system according to claim 6, wherein each of the array of haptic portions corresponds to a respective one of the plurality of portions of the depth map, and the controller is configured to control each of the array of haptic portions to perform the haptic action at a particular level of the attribute corresponding to the value of the property of the corresponding one of the plurality of portions of the depth map.

10. The haptic system according to claim 9, wherein the range of the attribute for the haptic action corresponds to a range of values of the property related to the depth map, and each attribute level within the range of the attribute for the haptic action is predetermined to correspond to a value or a sub-range of values of the range of values of the property related to the depth map such that the haptic action performed at the particular level of the attribute by each of the array of haptic portions is indicative of the degree of proximity associated with the corresponding portion of the scene captured in the depth map.

11. The haptic system according to claim 9, wherein the range of the attribute for the haptic action is adjustable based on a characteristic associated with a portion of the driver.

12. The haptic system according to claim 1, wherein the image generator is configured to generate an initial depth map based on the information obtained on the scene, and adjust a resolution of the initial depth map based on a configuration of the haptic surface to generate the depth map.

13. The haptic system according to claim 12, wherein the resolution of the initial depth map is adjusted to generate the depth map to have a resolution that corresponds to a resolution of the haptic surface formed such that each of the array of haptic portions corresponds to a respective one of the plurality of portions of the depth map.

14. The haptic system according to claim 13, wherein the resolution of the haptic surface is configured based on a spatial acuity of a portion of the driver which the haptic surface is configured to interact with.

15. The haptic system according to claim 1, wherein the haptic surface is configured to topographically map a predetermined space in the surrounding environment so that each haptic portion spatially corresponds to a respective portion of the predetermined space, the predetermined space corresponding to the scene captured.

16. The haptic system according to claim 1, wherein the information obtained on the scene is a point cloud data of the scene captured by the sensor.

17. A vehicle comprising:

a seat;

a sensor configured to obtain information on a scene of a surrounding environment; and

a haptic system for providing sensory augmentation to a driver of a potential collision, the haptic system comprising;

a haptic surface for interacting with the driver, the haptic surface comprising an array of haptic portions, each haptic portion configured to be controllable to perform a haptic action;

an image generator configured to generate a depth map based on the information obtained on the scene captured of the surrounding environment by the sensor; and

a controller configured to control each of the array of haptic portions based on a property of a corresponding portion of the depth map,

wherein the haptic surface is arranged on the seat of the vehicle to provide sensory augmentation to the driver when seated on the seat.

18. The vehicle according to claim 17, wherein the scene includes the surrounding environment located rearward of the vehicle.

19. A method for providing sensory augmentation to a driver of a potential collision, the method comprising:

providing a haptic surface for interacting with the driver, the haptic surface comprising an array of haptic portions, each haptic portion configured to be controllable to perform a haptic action;

generating a depth map based on information obtained on a scene captured of a surrounding environment by a sensor; and

controlling each of the array of haptic portions based on a property of a corresponding portion of the depth map.

20. The method according to claim 19, wherein the depth map comprises a plurality of portions, the property of each portion being determined based on a degree of proximity associated with a corresponding portion of the scene captured in the depth map.

21. The method according to claim 20, wherein the degree of proximity associated with a corresponding portion of the scene relates to a distance of a surface of one or more scene objects captured in the corresponding portion of the scene from a viewpoint.

22. The method according to claim 20, wherein each portion of the depth map corresponds to a pixel or a group of adjacent pixels, and the property of each portion has a value corresponding to a pixel value.

23. The method according to claim 19, wherein the generating a depth map comprises generating the depth map continuously for a plurality of consecutive time instances based on the respective information obtained on the scene continuously for the plurality of consecutive time instances, and the controlling each of the array of haptic portions comprises controlling each of the array of haptic portions continuously such that for each time instance, each of the array of haptic portions is controlled based on the property of the corresponding portion of the depth map generated for the time instance.

24. The method according to claim 20, wherein each of the array of haptic portions is configured to be capable of performing the haptic action within a range of an attribute based on a control signal.

25. The method according to claim 24, wherein the haptic action comprises vibration, pulsation or applied pressure.

26. The method according to claim 24, wherein the attribute comprises frequency and/or amplitude.

27. The method according to claim 24, wherein each of the array of haptic portions corresponds to a respective one of the plurality of portions of the depth map, and the controller is configured to control each of the array of haptic portions to perform the haptic action at a particular level of the attribute corresponding to the value of the property of the corresponding one of the plurality of portions of the depth map.

28. The method according to claim 27, wherein the range of the attribute for the haptic action corresponds to a range of values of the property related to the depth map, and each attribute level within the range of the attribute for the haptic action is predetermined to correspond to a value or a sub-range of values of the range of values of the property related to the depth map such that the haptic action performed at the particular level of the attribute by each of the array of haptic portions is indicative of the degree of proximity associated with the corresponding portion of the scene captured in the depth map.

29. The method according to claim 27, further comprising adjusting the range of the attribute for the haptic action based on a characteristic associated with a portion of the driver.

30. The method according to claim 20, wherein the generating a depth map comprises generating an initial depth map based on the information obtained on the scene, and adjusting a resolution of the initial depth map based on a configuration of the haptic surface to generate the depth map.

31. The method according to claim 30, wherein the resolution of the initial depth map is adjusted to generate the depth map to have a resolution that corresponds to a resolution of the haptic surface formed such that each of the array of haptic portions corresponds to a respective one of the plurality of portions of the depth map.

32. The method according to claim 31, wherein the resolution of the haptic surface is configured based on a spatial acuity of a portion of the subject which the haptic surface is configured to interact with.

33. The method according to claim 19, wherein the haptic surface is configured to topographically map a predetermined space in the surrounding environment so that each haptic portion spatially corresponds to a respective portion of the predetermined space, the predetermined space corresponding to the scene captured.

34. The method according to claim 19, wherein the information obtained on the scene is a point cloud data of the scene captured by the sensor.