MULTIMEDIA NEAR TO EYE DISPLAY SYSTEM

• US 20140152530A1
• Filed: 12/03/2012
• Published: 06/05/2014
• Est. Priority Date: 12/03/2012
• Status: Abandoned Application

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Venkatesha, Sharath, Au, Kwong Wing
• Application Number:
  US13/692,509
• Publication Number:
  US 20140152530A1
• Time in Patent Office: Days
• US Class Current: 345/8
• CPC Class Codes: G01B 11/00 Measuring arrangements characterised by the use of optical techniques
• CPC Class Codes: G01P 15/00 Measuring acceleration; Measuring deceleration; Measuring shock, i.e. sudden change of acceleration
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G09G 3/001 using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background slide projectors per se G03B23/00 = 42 HP

Abstract

A system and method include receiving video images based on field of view of a wearer of a near to eye display system, analyzing the video images to identify an object in the wearer field of view, generating information as a function of the identified objects, and displaying the information on a display device of the near to eye display system proximate the identified object.

Claims

1. A method comprising:

receiving video images based on fields of view of a near to eye display system;

applying video analytics to enhance the video images and to identify regions of interest (ROI) on the video images;

generating user assistance information as a function of at least one characteristic of the regions of interest; and

augmenting the enhanced video with the derived information proximate to corresponding regions of interest via visual displays and audio of the near to eye display system.

2. The method of claim 1, wherein the user assistance information displayed on the near to eye display system is derived from:

interactive video analysis and user inputs from voice and signals from hand held devices;

information stored in memory; and

information retrieved from cloud storage and the World Wide Web.

3. The method of claim 2, wherein the user assistance information comprises images, video clips, text, graphics, symbols including use of color, transparency, shading, and animation.

4. The method of claim 2, wherein the user assistance information is communicated to the user as audio, including:

descriptions of the video images, identified regions of interest and their characteristics; and

pre-recorded audio instructions, based on outputs of the video analysis.

5. The method of claim 1 wherein the at least one characteristic of regions of interest are selected from the group consisting of textural, spatial, structural, temporal and biometric features including appearance, shape, object identity, identity of person, motion, tracks, and events.

6. The method of claim 5 wherein the events further comprise application specific activities, industrial operations including identifying tools, determining a stage of an activity, operation, and the status of a stage.

7. The method of claim 1 wherein the video analytics to enhance the video images includes modifying the appearance, brightness and contrast by color and local intensity corrections on pixels in the images.

8. The method of claim 1 wherein characteristics of regions of interest further comprise estimated distance to the region of interest, a surface descriptor, and 3D measurements including at least one of volume, surface areas, length, width and height.

9. The method of claim 8 wherein the user assistance information is displayed adjacent the corresponding region of interest in the video.

10. The method of claim 9 wherein augmenting user assistance information further includes:

a distance scale indicating the projected distances of the pixels from the near to eye display system; and

a geometric object of same size as the corresponding region of interest, proximate the ROI.

11. A multi-media visual system comprising:

near-to-eye displays supported by a frame adapted to be worn by a user such that each display is positioned proximate an eye of the user;

speakers coupled to deliver audio of user assistance information;

a set of cameras supported by the frame, capturing video images of a scene in a field of view;

a microphone receiving inputs from the wearer; and

a processor coupled to receive images from the cameras and adapted to apply video analytics to enhance the video images, to identify regions of interest (ROI) on the video images and to generate user assistance information as a function of the characteristics of the regions of interest.

12. The multi-media visual system of claim 11 wherein the near to eye display consists of a transparent LCD for text display overlaid on LCD/LCoS/Light-Guide-Optics (LOE) for video display.

13. The multi-media visual system of claim 11 wherein the cameras are receptive to different spectra including visible, near infrared (NIR), ultraviolet (UV), short wave infrared bands, mid wave infrared or long wave infrared.

14. The multi-media visual system of claim 11 and further comprising:

a MEMS accelerometer to provide orientation of the frame;

cameras capturing images of the eyes of the user including pupil position; and

remote input devices to receive requests from the wearer.

15. The multi-media visual system of claim 14 wherein the processor is further adapted to generate user assistance information based on inputs representing the frame orientation, pupil locations and user requests.

16. The multi-media visual system of claim 15 wherein user assistance information comprises:

at least one of textural, spatial, structural, temporal and biometric features including appearance, shape, object identity, identity of person, motion, tracks, and events; and

at least one of application specific activities, industrial operations including identifying tools, determining the stage of the activity, operation, and the status of the stage

17. The multi-media visual system of claim 16 wherein user assistance information further includes at least one of estimated distance to the region of interest, its surface descriptor, and 3D measurements including volume, surface areas, length, width, and height.

18. The multi-media visual system of claim 17 wherein the user assistance information is displayed proximate the corresponding region of interest in the video.

19. The multi-media visual system of claim 18 wherein the user assistance information further comprises:

a distance scale indicating the projected distances of the pixels from the near to eye display system; and

a geometric object of same size as the corresponding region of interest, proximate the ROI.

20. The multi-media visual system of claim 19 wherein the video analytics to enhance the video images includes at least one of modifying the appearance, brightness and contrast by color, and local intensity corrections on the pixels in the images.