SYSTEM, APPARATUS, AND METHOD FOR AUGMENTED REALITY GLASSES FOR END-USER PROGRAMMING

• US 20100164990A1
• Filed: 08/15/2006
• Published: 07/01/2010
• Est. Priority Date: 08/15/2005
• Status: Abandoned Application

• Current Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: Van Doorn, Markus Gerardus Leonardus Maria
• Application Number:
  US12/063,145
• Publication Number:
  US 20100164990A1
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 8/00 Arrangements for software engineering testing or debugging G06F11/36; administrative, planning or organisation aspects of software project management G06Q10/06
• CPC Class Codes: G09G 3/003 to produce spatial visual effects

Abstract

A system, apparatus, and method is provided for augmented reality (AR) glasses (131) that enable an end-user programmer to visualize an Ambient Intelligence environment having a physical dimension such that virtual interaction mechanisms/patterns of the Ambient Intelligence environment are superimposed over real locations, surfaces, objects and devices. Further, the end-user can program virtual interaction mechanisms/patterns and superimpose them over corresponding real objects and devices in the Ambient Intelligence environment.

Claims

1. An apparatus (131140150) for an end-user to program an Ambient Intelligence environment to include at least one programmable component, comprising:

a pair of augmented reality (AR) glasses (131) having a see-through field-of-view (132) to visualize therein, for the end-user when wearing the AR glasses, the at least one programmable component proximate to a corresponding real world entity seen by the user in the see-through field-of-view;

a user programming interface (600-900) that appears in the field-of-view (132) of the AR glasses (131) of the end-user wearing the AR glasses for the end-user to view, create and modify at least one program for the at least one programmable component; and

at least one user input device (133-135 140 150) for a user to direct and react to the user programming interface (600-900) when it appears in the field-of-view (132).

2. The apparatus (131140150) of claim 1, wherein the AR glasses further comprise a capability to “

read”

the at least one programmable component as the end-user interacts with the Ambient Intelligence environment and display the end-user interaction with the Ambient Intelligence environment in the field-of-view as would be seen in the end-user without wearing the AR glasses (131).

3. The apparatus (131140150) of claim 1, wherein the user programming interface (600-900) is combined with the at least one user input device (133-135140150) and thereby comprises a capability to “

write”

that includes each of the following;

create, retrieve and modify/delete, and name and store, individually and in combination any of icons (103), beats (200), areas (101) and links (104).

4. The apparatus (131140150) of claim 3, wherein the AR glasses (131) further comprise a capability to “

read”

the at least one programmed component as the end-user interacts with the Ambient Intelligence environment and display a view of the at least one component of the Ambient Intelligence environment as seen by the end-user.

5. The apparatus (131140150) of claim 1, wherein:

the user programming interface comprises a graphical user interface (600-900) presented in the field-of-view of the AR glasses (131); and

the user input device comprises a combination of devices selected from the group consisting of a headset for voice input/output (140);

a button-device/mobile-mouse (150) including a left button (151) and a right button (153) and a menu button (152);

a handheld audio input-output wand including a microphone and a speaker for voice input and audio feedback;

wheel mouse incorporated in the AR glasses (133-135); and

a left (135) and right (134) button incorporated into the AR glasses.

6. The apparatus (131140150) of claim 2, wherein, the user programming interface is combined with the at least one user input device and thereby comprises a capability to “

write”

that includes each of the following;

create, retrieve and modify/delete, and name and store, individually and in combination any of icons (103), beats (200), areas (101) and links (104).

7. The apparatus (131140150) of claim 2, further comprising:

a means for providing information concerning the position and orientation of the end-user wearing the AR glasses to determine a scene being viewed by the user wearing the AR glasses (131); and

a means for acquiring component position information to visualize at least the corresponding real world entity proximate to the at least one component in the field-of-view (132).

8. The apparatus (131140150) of claim 7, wherein:

the means for providing information concerning the position and orientation of the end-user is a camera mounted in the AR glasses (131); and

the means for acquiring component position information is selected from the group consisting of a retrieving position information from a database of component positions and obtaining position information from a sensor network deployed to sense the components.

9. The apparatus (131140150) of claim 8, wherein, the user programming interface is combined with the at least one user input device and thereby comprises a capability to “

write”

that includes each of the following;

create, retrieve and modify/delete, and name and store, individually and in combination any of icons (103), beats (200), areas (101) and links (104).

10. The apparatus (131140150) of claim 9, wherein:

the user programming interface comprises a graphical user interface (600-900) presented in the field-of-view of the AR glasses (131); and

the user input device is a combination of devices selected from the group consisting of a headset for voice input/output (140);

a button-device/mobile-mouse (150) including a left button (151) and a right button (153) and a menu button (152);

a handheld audio input-output wand including a microphone and a speaker for voice input and audio feedback;

wheel mouse (133-135) incorporated in the AR glasses (131), and a left (135) and right (134) button incorporated into the AR glasses.

11. A system for end-user programming of an Ambient Intelligence environment comprising:

an augmented reality system (402-408) including;

i. a pair of augment reality (AR) glasses (131 402) according to claim 11 that are worn by an end-use; and

ii. a beat sequencing engine (300) to “

read”

programmable components of an Ambient Intelligence environment triggered by the end-user while wearing the AR glasses (131 402), wherein the triggered components are visualized in a field-of-view of the AR glasses (131 402) worn by the end-user,

an authoring tool (502) to collect end-user input an interfaced to the AR system (402-408) for an end-user to “

write”

the programmable components and associated programs of the Ambient Intelligence environment using a user-interface displayed in the field-of-view of the AR glasses (131 402).

12. A method for an end-user in an Ambient Intelligence environment to program the Ambient Intelligence environment to include at least one programmable component, comprising:

providing a pair of augmented reality (AR) glasses (131) having a see-through field-of-view (132);

when an end-user wears the AR glasses in the Ambient Intelligence environment, visualizing in the field-of-view, the at least one programmable component proximate to a corresponding real world entity seen in the see-through field of view;

displaying an end-user programming interface (600-900) in the field-of-view (132) that enables the end-user to “

read” and

“

write”

at least one program for the at least one programmable component having an “

undo”

/“

debugging”

mode; and

providing at least one user input device (133-135 140 150) for the end-user to direct and react to the displayed end-user programming interface (600-900) when it appears in the field-of-view (132) to program the at least one programmable component.

13. The method of claim 12, further comprising the steps of:

providing information concerning the position and orientation of the end-user wearing the AR glasses;

determining a scene being viewed by the end-user wearing the AR glasses (131) from the provided position and orientation information of the end-user; and

acquiring programmable component position information; and

visualizing the at least one programmable component in the field-of-view (132) proximate to the corresponding real world entity seen in the see-through field-of-view (132).

14. The method of claim 13, wherein:

the step of providing information concerning the position and orientation of the end-user further comprises the step of providing s a camera mounted in the AR glasses (131); and

the step of acquiring component position information further comprises the step of acquiring information from a source selected from the group consisting of a database of positions and a sensor network deployed to sense component positions.

15. The apparatus method of claim 14, further comprising the step of combining the steps of displaying the end-user interface with providing the at least one user input device in a step of “

writing”

a program for a programmable component wherein the step of “

writing”

comprises the substeps of creating, retrieving and modifying/deleting, and naming and storing, individually and in combination, any of icons (103), beats (200), areas (101) and links (104).

16. The method of claim 15, wherein:

the step of displaying an end-user programming interface further comprises the step of displaying a graphical user interface (600-900) presented in the field-of-view of the AR glasses (131); and

the step of providing a user input device further comprises the step of providing a combination of devices selected from the group consisting of a headset for voice input/output (140);

a button-device/mobile-mouse (150) including a left button (151) and a right button (153) and a menu button (152);

a handheld audio input-output wand including a microphone and a speaker for voice input and audio feedback, wheel mouse (133-135) incorporated in the AR glasses (131); and

a left (135) and right (134) button incorporated into the AR glasses.