SYSTEM AND METHOD TO ASSIST USERS HAVING REDUCED VISUAL CAPABILITY UTILIZING LIGHTING DEVICE PROVIDED INFORMATION

• US 20160335917A1
• Filed: 05/13/2015
• Published: 11/17/2016
• Est. Priority Date: 05/13/2015
• Status: Active Grant

• Current Assignee: ABL IP Holding LLC (Acuity Brands, Inc.)
• Original Assignee: ABL IP Holding LLC (Acuity Brands, Inc.)
• Inventors: LYDECKER, Stephen H., RAMER, David P., RAINS, Jack C. Jr., AGGARWAL, Januk
• Granted Patent: US 9,536,452 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A61H 2003/063 with tactile perception
• CPC Class Codes: A61H 2201/0188 Illumination related features
• CPC Class Codes: A61H 2201/1604 Head
• CPC Class Codes: A61H 2201/165 Wearable interfaces
• CPC Class Codes: A61H 2201/5007 computer controlled
• CPC Class Codes: A61H 2201/5043 Displays
• CPC Class Codes: A61H 2201/5048 Audio interfaces, e.g. voice or music controlled
• CPC Class Codes: A61H 2201/5082 Temperature sensors
• CPC Class Codes: A61H 2201/5092 Optical sensor
• CPC Class Codes: A61H 2201/5097 wireless
• CPC Class Codes: A61H 3/061 with electronic detecting or guiding means
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09B 21/007 : using both tactile and audible presentation of the information
• CPC Class Codes: G09B 21/008 : using visual presentation of the information for the partially sighted
• CPC Class Codes: G09G 2354/00 : Aspects of interface with display user
• CPC Class Codes: G09G 5/12 : Synchronisation between the display unit and other units, e.g. other display units, video-disc players
• CPC Class Codes: G10L 15/22 : Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 : Execution procedure of a spoken command

Abstract

A lighting device obtains data related to objects and boundaries in an area in the vicinity of the lighting device, and a user wearable device provides a display (e.g. an augmented reality display based on the data related to the objects and the area boundaries) for a user/wearer. The lighting device includes a mapping sensor that collects data related to the objects and boundaries in the area. The user wearable device includes a camera or other optical sensor and wireless communication capability. The user wearable device is provided with mapping data that is presented on a display of the user wearable device. The communications and display capabilities allow the user wearable device to obtain room mapping information related to area in the vicinity of the lighting device in order to provide navigational assistance to a visually impaired person in the area.

Claims

1. A system, comprising:

a lighting device in an area of a premises including a processor, a wireless communication transceiver, a light source, a memory and a mapping sensor, wherein the lighting device processor is coupled to the wireless communication transceiver, the memory, the light source, and the mapping sensor, the processor upon execution of programming stored in the memory is configured to control operations of the lighting device including the mapping sensor, the mapping sensor being configured to collect mapping data related to the area of the premises in which the lighting device is located; and

a data storage for maintaining area mapping data collected by the mapping sensor of an area in which the lighting device is located in the premises, processed mapping data which is area mapping data that has been processed using image processing techniques, and mapping information including navigational information or indications, wherein the database is accessible by the lighting device processor;

wherein the lighting device memory stores programming, and execution of the programming by the lighting device processor configures the lighting device processor to perform functions, including functions to;

receive a request from a user wearable device for processed mapping data for the area of the premises in which the lighting device is located; and

in response to the request, deliver processed mapping data for the area of the premises in which the lighting device is located to the user wearable device.

2. The system of claim 1, wherein the delivered processed mapping data includes one or more of object identifications, object boundaries indications, or navigation instructions or navigation indicators.

3. The system of claim 1, further comprising:

a server, coupled to the lighting device processor and the database, the server configured to provide image processing services, the image processing services including identifying objects, object boundaries, and inserting as area mapping information navigation instructions and indicators in the processed mapping data; and

wherein the lighting device memory stores programming, and execution of the programming by the lighting device processor configures the lighting device processor to perform further functions, including functions to;

instruct the mapping sensor to collect mapping data of the area of the premises in which the lighting device is located; and

send the collected mapping data to the server for image processing.

4. The system of claim 1, wherein the lighting device memory stores programming, and execution of the programming by the lighting device processor configures the lighting device processor to perform further functions, including functions to:

instruct the mapping sensor to collect mapping data of the area of the premises in which the lighting device is located;

generate processed mapping data by image processing the collected mapping data; and

store the processed mapping data in the lighting device memory.

5. The system of claim 4, wherein the lighting device memory stores programming, and execution of the programming by the lighting device processor configures the lighting device processor to perform further functions, including functions to:

after generating the processed mapping data, forward a copy of the processed mapping data to the data storage.

6. The system of claim 1, wherein the wireless communication transceiver is configured to send or receive data via radio frequency wireless communication.

7. The system of claim 1, wherein the wireless communication transceiver is configured to perform as least one of sending or receiving data as a communication via light.

8. The system of claim 3, wherein the lighting device memory stores programming, and execution of the programming by the lighting device processor configures the lighting device to perform functions, including functions to:

instruct the mapping sensor to collect mapping data of the area;

receive the collected mapping data from the mapping sensor; and

forwarding the collected mapping data including a lighting device identifier to the server for image processing, wherein the lighting device identifier uniquely identifies the location of the lighting device in the premises.

9. The system of claim 1, further comprising:

the user wearable device, the user wearable device including an output device, a user wearable device processor, a user wearable device wireless communication transceiver, and a user wearable device memory,

wherein the user wearable device processor is coupled to the user wearable device wireless communication transceiver and the user wearable device memory, the user wearable device processor being configured to control operations of the user wearable device, wherein the user wearable device memory stores programming, and execution of the programming by the user wearable device processor configures the user wearable device to perform functions, including functions to;

send a request for mapping data, via the user wearable device wireless communication transceiver, related to a premises location of the user wearable device from the lighting device processor, wherein the premises location of the user wearable device is the area in which the lighting device is located;

receive, from the lighting device, the processed mapping data specific to the determined location of the user wearable device; and

based on the processed mapping data, output, via the output device, user detectable stimuli suitable for navigating a user through the area of the premises in which the lighting device is located.

10. The system of claim 9, wherein the output device of the user wearable device includes:

an audio output device, and the outputted user detectable stimuli is an audio prompt indicating a direction of travel for a user wearing the user wearable device.

11. The system of claim 9, wherein the output device of the user wearable device comprises:

an audio output device, and the outputted user detectable stimuli is an audio prompt indicating objects in an area surrounding the determined location of the user wearable device.

12. The system of claim 9, wherein the output device of the user wearable device comprises:

a display configured to provide an augmented reality display of an area surrounding the determined location of the user wearable device, and the outputted user detectable stimuli are visual prompts indicating locations of objects with the area surrounding the determined location of the user wearable device.

13. The system of claim 9, wherein the user wearable device includes:

a tactile device that outputs tactile stimuli indicative of a navigation path around objects in an area surrounding the determined location of the user wearable device.

14. The system of claim 9, wherein the user wearable device comprises an eyewear frame having a structure to configure the user interface device to be worn as goggles or eyeglasses.

15. A method, comprising:

collecting, by a lighting device sensor, mapping data of an area of a premises to be illuminated by the lighting device;

identifying from the collected mapping data objects in the area of the premises;

storing the collected mapping data and an indication of the identified objects in memory associated with a lighting device;

in response to a request from a user device, providing, as first area mapping data, the collected mapping data and the indications of the identified objects from the lighting device to the user device; and

enable the user device to present navigational instructions that indicate an unobstructed pathway through the area based on the first area mapping data.

16. The method of claim 15, further comprising:

updating the mapping data of the area of the premises to identify locations of moveable objects within the area as the moveable objects move within the area;

using the updated mapping data, identifying objects in the area;

storing the updated mapping data and updated indications of the identified objects in memory;

comparing the first area mapping data including identified object indications to the updated mapping data and the updated indications of the identified objects stored in memory; and

based on the results of the comparison, providing updated mapping data and updated indications of the identified objects to the user device;

enabling the user device to update navigational instructions to indicate an updated unobstructed pathway through the area based on the provided updated mapping data and updated indications of the identified objects.

17. The method of claim 15, further comprising:

determining a location of the user device in the premises using visible light communication between the lighting device and the user device; and

based on the determined position of the user device, accessing mapping data corresponding to the determined location of the user device in the premises to be provided to the user device.

18. The method of claim 17, wherein determining a location of the user device, comprises:

transmitting, by the lighting device, an identifier of the at least one lighting device in visual light communication signals transmitted into the area of the premises;

receiving as part of the request from the user device the identifier of the at least one lighting device;

comparing the identifier to a premises plan that identifies locations of each of the number of lighting devices, and

based on the results of the comparison, determining a location of the at least one lighting device and the user device.

19. The method of claim 15, further comprising:

receiving the first area mapping data;

determining a viewing perspective of the area based on an input from the user device;

transforming the first area mapping data based on the determined viewing perspective of the area; and

rendering a two dimensional view of the area based on the first area mapping data.

20. A user wearable device, comprising:

a framework configured to enable a visually impaired user to wear the user wearable device;

a display supported by the framework so as to be viewable by at least one eye of the visually impaired user when wearing the user wearable device;

a camera supported by the framework;

a wireless communication transceiver supported by the framework;

a processor, supported by the framework, coupled to the display, the camera, the transceiver and the input, the processor being configured to control operations of the user wearable device;

a memory accessible to the processor and supported by the framework; and

programming in the memory, wherein execution of the programming by the processor configures the user wearable device to perform functions, the programming including;

an operating system to control general operating functions of the user wearable device; and

a navigation application for execution via the operating system, to configure the user wearable device to provide one or more navigational instructions or indications for presentation on the display, wherein the navigational instructions or indications are viewable by the visually impaired user.

21. The user wearable device of claim 20, further comprising:

an audio output device; and

the processor, upon execution of programming in the memory, is further configured to;

output, via the audio output device, user detectable audio stimuli as an audio prompt indicating a direction of travel for a user wearing the user wearable device.

22. The user wearable device of claim 20, further comprising:

an audio output device, and

the processor, upon execution of programming in the memory, is further configured to;

output, via the audio output device, user detectable audio stimuli as an audio prompt indicating objects in an area surrounding the determined location of the user wearable device.

23. The user wearable device of claim 20, further comprising:

a tactile device; and

the processor, upon execution of programming in the memory, is further configured to;

output, via the tactile device, user detectible tactile stimuli indicative of a navigation path around objects in an area surrounding the determined location of the user wearable device.

24. The user wearable device of claim 20, wherein the framework comprises:

an eyewear frame having a structure to configure the user wearable device to be worn as goggles or eyeglasses.

25. The user wearable device of claim 20, wherein the processor upon execution of programming in the memory is further configured to:

cause the generation of an augmented reality representation of an area surrounding a determined location of the user wearable device, and

output to the display as the navigational instructions and indications, visual prompts indicating locations of objects within the area surrounding the determined location of the user wearable device.

26. The user wearable device of claim 25, wherein the displayed visual prompts are outlines of objects, and the processor upon execution of programming in the memory is further configured to:

overlaying markings on the edges of the object outlines, and

display boundaries of unobstructed pathways into and through the area surrounding the determined location of the user wearable device.

27. The user wearable device of claim 25, wherein the displayed visual prompts are markings viewable by the visually impaired user overlaying the edges of the objects and indicating unobstructed pathways into and through the area surrounding the determined location of the user wearable device.

28. A lighting device located in an area of a premises, comprising:

a processor;

a wireless communication transceiver coupled to the processor and to a network;

a light source responsive to commands from the processor;

a memory accessible to the processor; and

programming is stored in the memory, wherein execution of the programming by the processor configures the lighting device to perform functions, including functions to;

in response to a request from a user device in the area of the premises, retrieve mapping data of the area;

deliver the mapping data to the user device via the wireless communication transceiver;

update mapping data of the area with locations of objects in the area based on user inputs indicating an object location in the area; and

in response to a user query, generate an indication of an object location based on the updated mapping data.

29. The lighting device of claim 28, wherein the generated indication is at least one of a visible light communication signal receivable by a user device in communication with the lighting device within the area, a non-visible light communication signal receivable by a user device in communication with the lighting device within the area, a radio frequency signal receivable by a user device in communication with the lighting device within the area, or an audio signal.

30. The lighting device of claim 28 further comprising:

a mapping sensor, coupled to the processor;

wherein upon execution of the programming stored in the memory by the processor configures the lighting device to perform further functions, including functions to;

instruct the mapping sensor to collect mapping data related to the area in which the lighting device is located;

process the collected mapping data to identify objects within the area; and

store the processed mapping data including indicators of objects within the area in the memory.

31. The lighting device of claim 30, wherein upon execution of the programming stored in the memory by the processor configures the lighting device to perform further functions, including functions to:

process the collected mapping data to identify unobstructed pathways within the area; and

update mapping data of the area with locations of unobstructed pathways within the area.

32. The lighting device of claim 30, further comprising:

a microphone for receiving voice inputs;

programming stored in the memory for performing a voice recognition algorithm, wherein the processor upon executing the programming stored in the memory by the processor configures the lighting device to perform further functions, including functions to;

recognize spoken mapping commands that identify the location of objects within the area; and

based on the recognized mapping commands, the processor updates the mapping data based on the recognized spoken mapping command.