Mobility assist device

• US 6,977,630 B1
• Filed: 07/18/2000
• Issued: 12/20/2005
• Est. Priority Date: 07/18/2000
• Status: Expired due to Term

• Current Assignee: University of Minnesota
• Original Assignee: University of Minnesota
• Inventors: Lim, Heon Min, Newstrom, Bryan, Pardy, Sameer, Cheng, Pi-Ming, Alexander, Lee, Donath, Max, Gorjestani, Alec, Shankwitz, Craig R.
• Primary Examiner(s): Shankar, Vijay
• Assistant Examiner(s): Shapiro, Leonid
• Application Number:
  US09/618,613
• Time in Patent Office: 1,981 Days
• Field of Search: 345 7- 9, 701/1, 701 23- 24, 701/200, 701207-225, 701300-302, 359/13, 359/630, 359/63, 250/280.1, 250/280.2, 342/1, 342/42, 342/35, 342/357, 342/450, 356/3, 356/27
• US Class Current: 345/7
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/205 using a head-up display
• CPC Class Codes: B60R 2300/301 combining image information with other obstacle sensor information, e.g. using RADAR/LIDAR/SONAR sensors for estimating risk of collision
• CPC Class Codes: B60R 2300/302 combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
• CPC Class Codes: B60R 2300/305 merging camera image with lines or icons
• CPC Class Codes: B60R 2300/307 virtually distinguishing relevant parts of a scene from the background of the scene
• CPC Class Codes: B60R 2300/60 characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective
• CPC Class Codes: B60R 2300/804 for lane monitoring
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: B60T 2201/08 Lane monitoring; Lane Keeping Systems
• CPC Class Codes: B60T 2201/086 using driver related features
• CPC Class Codes: G01C 21/365 Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself

Abstract

The present invention is directed to a visual mobility assist device which provides a conformal, augmented display to assist a moving body. When the moving body is a motor vehicle, for instance (although it can be substantially any other body), the present invention assists the driver in either lane keeping or collision avoidance, or both. The system can display objects such as lane boundaries, targets, other navigational and guidance elements or objects, or a variety of other indicators, in proper perspective, to assist the driver.

Claims

1. A mobile assist device for use with a motorized vehicle having an operating position at which an operator controls the vehicle, the device comprising:

a data storage system containing stored objects each defining a location of corresponding real world objects; and

a display configured to display virtual images representing the stored objects in a position that generates a visual effect of the virtual images being projected on the corresponding real world objects when the virtual images are viewed by the operator in the operating position.

2. The device of claim 1, wherein:

the stored objects define a location of lane boundaries of a road; and

the virtual images presented on the display include a line that substantially overlays the lane boundary of the road corresponding to the stored objects when viewed by the operator in the operating position.

3. The device of claim 2, wherein the stored objects correspond to at least one of a center lane boundary, a left lane boundary, and a right lane boundary of the road.

4. The device of claim 1, wherein the display includes a screen on which the virtual images are displayed.

5. The device of claim 4, wherein the screen comprises a liquid crystal display film.

6. A mobility assist device for use with a motorized vehicle having an operating position at which an operator controls the motorized vehicle, the device comprising:

a data storage system;

object data contained in the data storage system, the object data including location information of a real world object in a three dimensional coordinate system; and

a display attached to the motorized vehicle and including a graphical image that is positioned to produce a visual effect of the graphical image being projected on the corresponding real world object when viewed from a perspective of the operator in the operating position.

7. The device of claim 6, wherein the graphical image is positioned in line with a line of sight between the operator in the operating position and the corresponding real world object.

8. The device of claim 6, wherein a size of the graphical image changes in response to a change in the position of the operating position relative to the corresponding real world object.

9. The device of claim 8, wherein the size of the graphical image becomes smaller as the operating position moves away from the corresponding real world object.

10. The device of claim 8, wherein the size of the graphical image becomes larger as the operating position moves toward the corresponding real world object.

11. The device of claim 6, wherein the graphical image is representative of the corresponding real world object.

12. The device of claim 11, wherein the real world object includes a lane marking on a road and the graphical image includes a line having a shape that substantially matches the shape of the lane marking such that it appears to be projected on the lane marking when viewed from the perspective of the operator in the operating position.

13. The device of claim 11, wherein the graphical image corresponds to a lane marking on a road.

14. The device of claim 11, wherein the corresponding real world object is located adjacent to a road on which the motorized vehicle is located.

15. The device of claim 14, wherein the corresponding real world object includes a guardrail alongside the road, and the graphical image includes a representation of the guardrail, which appears to be projected on the guardrail when viewed from the perspective of the operator in the operating position.

16. The device of claim 6, wherein the display includes a screen on which the graphical image is presented.

17. The device of claim 16, wherein the screen is positioned between a window of the motorized vehicle and the operating position.

18. The device of claim 17, wherein the screen is transparent.

19. The device of claim 18 including a projector configured to project the graphical image onto the screen.

20. The device of claim 6, wherein the position of the graphical image is based on the object data, a location of the operating position, and a location of the display in the three dimensional coordinate system.

21. The device of claim 6 including an object detection system configured to detect objects outside of the motorized vehicle that are within a field of view of the operator, wherein the display includes tracking graphical images that are positioned in the display to produce a visual effect of being projected on the real world objects that are detected by the object detection system.

24. The device of claim 12, wherein a size of the graphical images changes in response to a change in the position of the operating position relative to the corresponding real world objects.

25. The device of claim 24, wherein the size of the graphical images becomes smaller as the operating position moves away from the corresponding real world objects.

26. The device of claim 24, wherein the size of the graphical images becomes larger as the operating position moves toward the corresponding real world objects.

22. A mobility assist device for use with a motorized vehicle having an operating position at which an operator controls the motorized vehicle, the device comprising:

a data storage system;

objects data contained in the data storage system, the objects data including location information of real world objects in a three dimensional coordinate system; and

a display attached to the motorized vehicle and including graphical images that are positioned in the display such that they appear to be projected on the corresponding real world objects when viewed from a perspective of the operator in the operating position.

23. The device of claim 22, wherein the graphical images are positioned in line with a line of sight between the operator in the operating position and the corresponding real world objects.

27. The device of claim 22, wherein the graphical images are representative of the corresponding real world objects.

28. The device of claim 27, wherein the real world objects include first and second lane markings on a road and the graphical images include first and second lines having a shape that substantially matches a shape of the first and second lane markings, respectively, such that the first and second lines appear to be projected on the first and second lane markings when viewed from the perspective of the operator in the operating position.

29. The device of claim 27, wherein the graphical images correspond to real world lane markings of a road.

30. The device of claim 27, wherein the corresponding real world objects include an object that is located adjacent to a road on which the motorized vehicle is located.

31. The device of claim 30, wherein the corresponding real world object located adjacent the road includes a guardrail alongside the road, and the graphical images include a representation of the guardrail, which appears to be projected on the guardrail when viewed from the perspective of the operator in the operating position.

32. The device of claim 22, wherein the display includes a screen on which the graphical image is presented.

33. The device of claim 32, wherein the screen is positioned between a window of the motorized vehicle and the operating position.

34. The device of claim 32, wherein the screen is transparent.

35. The device of claim 32 including a projector configured to project the graphical image onto the screen.

36. The device of claim 22, wherein the positions of the graphical images are based on the objects data, a location of the operating position, and a location of the display in the three dimensional coordinate system.

37. A mobility assist device comprising:

a motorized vehicle having an operating position at which an operator controls the motorized vehicle;

a data storage system accessible from the motorized vehicle;

objects data contained in a data storage system, the objects data including location information of real world objects in a three dimensional coordinate system; and

a display attached to the motorized vehicle and including graphical images that are displayed in a position such that they appear to be projected on the corresponding real world objects when viewed by the operator in the operating position, wherein the positions of the graphical images are based on the objects data, a location of the operating position, and a location of the display in the three dimensional coordinate system.

38. The device of claim 37, wherein the motorized vehicle is an automobile and the real world objects correspond to features of a road on which the motorized vehicle is located.

39. The device of claim 37, wherein the graphical images are positioned in line with a line of sight between the operator in the operating position and the corresponding real world objects.

40. The device of claim 37, wherein a size of the graphical images changes in response to a change in the position of the operating position relative to the corresponding real world objects.

41. The device of claim 40, wherein the size of the graphical images becomes smaller as the operating position moves away from the corresponding real world objects.

42. The device of claim 40, wherein the size of the graphical images becomes larger as the operating position moves toward the corresponding real world objects.

43. The device of claim 37, wherein the graphical images are representative of the corresponding real world objects.

44. The device of claim 43, wherein the real world objects include first and second lane markings on a road and the graphical images include first and second lines having a shape that substantially matches a shape of the first and second lane markings, respectively, such that the first and second lines appear to be projected on the first and second lane markings when viewed from the perspective of the operator in the operating position.

45. The device of claim 43, wherein the graphical images correspond to real world lane markings of a road.

46. The device of claim 43, wherein the corresponding real world objects include an object that is located adjacent to a road on which the motorized vehicle is located.

47. The device of claim 46, wherein the corresponding real world object located adjacent the road includes a guardrail alongside the road, and the graphical images include a representation of the guardrail, which appears to be projected on the guardrail when viewed from the perspective of the operator in the operating position.

48. The device of claim 37 wherein the display includes a screen on which the graphical image is presented.

49. The device of claim 48, wherein the screen is positioned between a window of the motorized vehicle and the operating position.

50. The device of claim 48, wherein the screen is transparent.

51. The device of claim 48 including a projector configured to project the graphical image onto the screen.