Generating a virtual-room of a virtual room-based user interface

• US 10,146,398 B2
• Filed: 06/08/2015
• Issued: 12/04/2018
• Est. Priority Date: 06/03/2009
• Status: Active Grant

• Current Assignee: Savant Systems, LLC
• Original Assignee: Savant Systems, LLC
• Inventors: Madonna, Robert P., Cipollo, Nicholas J.
• Primary Examiner(s): Casillas, Roland J
• Application Number:
  US14/733,428
• Publication Number:
  US 20150301716A1
• Time in Patent Office: 1,275 Days
• CPC Class Codes: G05B 19/042 using digital processors G05B19/05 takes precedence
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: H04L 12/2816 Controlling appliance services of a home automation network by calling their functionalities arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00
• CPC Class Codes: H05B 47/155 Coordinated control of two or more light sources
• CPC Class Codes: H05B 47/175 by remote control

Abstract

In one embodiment, a virtual room-based user interface includes one or more virtual rooms. Each virtual room is rendered from one or more images captured of a corresponding physical room of a structure, and includes depictions of one or more light fixtures within the physical room, one or more furnishings within the physical room and one or more boundaries of the physical room. A user selects a particular depiction of a particular light fixture within a particular virtual room. In response, a state of the particular light fixture within the corresponding physical room is changed. Also, appearance of the particular virtual room is updated such that the depiction of the particular light fixture shows the particular light fixture with the changed state and the depictions of the one or more boundaries or the one or more furnishings show lighting effects resulting from the changed state.

Claims

1. A method for generating a virtual-room of a virtual room-based user interface that controls a plurality of devices within a physical space, comprising:

capturing, by a camera, a plurality of images of the physical space from a same location, the plurality of images including a first image showing none of the plurality of devices within the physical space activated, and a plurality of additional images that each show a corresponding different one of the plurality of devices within the physical space activated;

rendering the virtual room in a state in which two or more of the plurality of devices are activated based on a combination of multiple ones of the plurality of images of the physical space captured from the same location, the state of the virtual room not depicted by the first image or by any of the additional images of the plurality of images, the rendering to

pass the first image,

pass two or more additional images that correspond to each of the two or more devices that are activated, and

combine the passed first image and the passed two or more additional images to produce an output that depicts the virtual room in the state; and

using the output to represent the virtual room in the virtual room-based user interface that controls the plurality of devices within the physical space.

2. The method of claim 1, wherein the rendering renders the virtual room in a plurality of states to produce a plurality of outputs that depict the virtual room in different states, and the method further comprises:

selecting an output that depicts the virtual room in the state from the plurality of different outputs.

3. The method of claim 2, wherein a number of the plurality of different outputs produced by the rendering is greater than a number of the plurality of images captured of the physical space.

4. The method of claim 1, wherein the plurality of devices are n devices and the plurality of images are no more than n+1 images.

5. The method of claim 1, wherein the rendering further comprises:

adjusting one or more properties of the passed additional images.

6. The method of claim 5, wherein the plurality of devices include light fixtures, and the adjusting properties includes adjusting brightness of the passed additional images in response to dimmer selections.

7. The method of claim 1, wherein the plurality of devices include light fixtures, and the output that depicts the virtual room in the state shows reflections or shadows caused by the two or more devices on at least furnishings or boundaries of the physical space.

8. The method of claim 1, wherein the physical space is a physical room of a structure.

9. The method of claim 1, wherein the physical space is an outdoor space.

10. The method of claim 1, wherein the plurality of images are still photographs of the physical space.

11. The method of claim 1, further comprising:

displaying the output on a display screen;

receiving a selection of a particular device of the plurality of devices within the physical space;

in response to selection of the particular device, activating or deactivating the particular device to change the particular device'"'"'s activation state; and

displaying an updated output on the display screen, the updated output to depict the virtual room in an updated state that includes the particular device'"'"'s changed activation state.

12. The method of claim 11, wherein the particular device is a light fixture, and the updated output depicts reflections or shadows on at least furnishings or boundaries of the physical space caused by the particular device'"'"'s changed activation state.

13. A system for generating a virtual-room of a virtual room-based user interface that controls a plurality of devices within a physical space, comprising:

a camera disposed at a location within the physical space;

a memory configured to store a plurality of images of the physical space captured by the camera from the location, the plurality of images including a first image showing none of the plurality of devices within the physical space activated, and a plurality of additional images that each show a corresponding different one of the plurality of devices within the physical space activated;

a processor configured to implement controls that pass or block the additional images;

a graphical processing unit (GPU) configured to render the virtual room in a state based on a combination of the first image and one or more passed additional images of the plurality of images of the physical space captured by the camera from the same location, the state of the virtual room not depicted by the first image or the one or more passed additional images of the plurality of images, the GPU implementing

an adjustment routine that adjusts one or more properties of the one or more passed additional images, and

a filter that combines the passed first image and the one or more passed additional images to produce an output that depicts the virtual room in the state.

14. The system of claim 13, wherein the GPU is configured to render the virtual room in a plurality of states by implementing adjustment routines and filters to produce a plurality of outputs that depict the virtual room in different states, and the GPU further implements a selector configured to select an output that depicts the virtual room in the state from the plurality of different outputs produced.

15. The system of claim 14, wherein a number of the plurality of different outputs produced is greater than a number of the plurality of images captured of the physical space.

16. The system of claim 14, wherein the plurality of devices are n devices and the plurality images are no more than n+1 images.

17. The system of claim 14, wherein the plurality of devices include light fixtures.

18. The system of claim 17, wherein the one or more properties adjusted by the adjustment routine includes adjusting brightness of the passed additional images in response to dimmer selections.

19. The system of claim 17, wherein the output that depicts the virtual room in the state shows reflections or shadows caused by one or more of the plurality of devices on at least furnishings or boundaries of the physical space.

20. A non-transitory computer-readable medium storing program instructions, the program instructions when executed on one or more processing units operable to:

access a plurality of images of a physical space captured from a same location, the plurality of images including a first image showing none of the plurality of devices within the physical space activated, and a plurality of additional images that each show a corresponding different one of the plurality of devices within the physical space activated;

render a virtual-room of a virtual room-based user interface that controls a plurality of devices within the physical space based on a combination of multiple ones of the plurality of images of the physical space captured from the same location, the rendered virtual room showing a state of the virtual room that is not depicted by the first image or by any of the additional images of the plurality of images, by

passing the first image,

passing one or more additional images that each correspond to a device that is activated in the state,

adjusting one or more properties of the one or more passed additional images, and

combining the passed first image and the one or more adjusted additional images to produce an output that depicts the virtual room in the state; and

use the output to represent the virtual room in the virtual room-based user interface that controls the plurality of devices within the physical space.

21. The non-transitory computer readable medium of claim 20, wherein the state depicts two or more of the plurality of devices activated.

22. The non-transitory computer readable medium of claim 20, wherein the plurality of devices include light fixtures, and the one or more properties include brightness of the one or more passed additional images in response to dimmer settings.

23. The non-transitory computer readable medium of claim 20, wherein the plurality of devices include light fixtures and the output that depicts the virtual room in the state shows reflections or shadows caused by the one or more of the plurality of devices on at least furnishings or boundaries of the physical space.

24. The non-transitory computer readable medium of claim 20, wherein the program instructions when executed on one or more processing units are further operable to:

display the output;

receive a selection of a particular device of the plurality of devices within the physical space;

in response to selection of the particular device, activate or deactivate the particular device to change the particular device'"'"'s activation state; and

display an updated output that depicts the virtual room in an updated state that includes the particular device'"'"'s changed activation state.