System and Method for Virtual Object Placement

• US 20120290987A1
• Filed: 05/10/2012
• Published: 11/15/2012
• Est. Priority Date: 05/13/2011
• Status: Active Grant

• Current Assignee: Knexus Research Corporation
• Original Assignee: Knexus Research Corporation
• Inventors: Gupta, Kalyan M.
• Granted Patent: US 8,893,048 B2
• Time in Patent Office: Days
• US Class Current: 715/848
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 30/13 Architectural design, e.g. computer-aided architectural design [CAAD] related to design of buildings, bridges, landscapes, production plants or roads

Abstract

A computer system and method according to the present invention can receive multi-modal inputs such as natural language, gesture, text, sketch and other inputs in order to manipulate graphical objects in a virtual world. The components of an agent as provided in accordance with the present invention can include one or more sensors, actuators, and cognition elements, such as interpreters, executive function elements, working memory, long term memory and reasoners for object placement approach. In one embodiment, the present invention can transform a user input into an object placement output. Further, the present invention provides, in part, an object placement algorithm, along with the command structure, vocabulary, and the dialog that an agent is designed to support in accordance with various embodiments of the present invention.

Claims

1. A system for manipulating virtually displayed objects in a virtual environment, comprising:

a display;

at least one input device adapted to receive at least one of speech, gesture, text and touchscreen inputs; and

a computer processor adapted to execute a program stored in a computer memory, the program being operable to provide instructions to the computer processor including;

receiving user input via the at least one input device, wherein the user input underspecifies a command to place at least one target object in the virtual environment;

interfacing with the virtual environment;

sensing the at least one object in the virtual environment; and

finding at least one valid location for the at least one object in the virtual environment.

2. The system of claim 1 wherein interfacing with the virtual environment and sensing the at least one object in the virtual environment are conducted by a virtual agent.

3. The system of claim 2 wherein finding at least one valid location includes retrieving a linguistic placement constraint, identifying the virtual agent'"'"'s intent for a target object and identifying a placement preference for the virtual agent.

4. The system of claim 2 wherein finding at least one valid location further includes determining one or more object properties of the target object and determining candidate placement surfaces for placement of the target object.

5. The system of claim 2 wherein finding at least one valid location includes determining a candidate activity surface on which the virtual agent is located while interacting with at least one target object.

6. The system of claim 2 wherein finding at least one valid location includes determining one or more object properties of the target object, determining candidate placement surfaces for placement of the target object, determining at least one candidate activity surface on which the virtual agent is located while interacting with the at least one target object, and, for all candidate placement surfaces and the at least one candidate activity surface, identifying all objects in the virtual environment that are in contact with any of the candidate placement surfaces or the at least one candidate activity surface.

7. The system of claim 6 wherein finding at least one valid location further includes determining, for every object identified as being in contact with any of the candidate placement surfaces or the at least one candidate activity surface, an object per station cardinality constraint representing how many objects of the type identified are allowed in a particular activity station, and further determining whether the object is a self-category attractor.

8. The system of claim 6 wherein finding at least one valid location further includes determining, for every pair of objects identified as being in contact with any of the candidate placement surfaces or the at least one candidate activity surface, whether the pair of objects can be co-located within the reach of the same activity stations.

9. The system of claim 6 wherein finding at least one valid location further includes determining any surface-to-surface constraints as between the target object and the candidate placement surfaces.

10. The system of claim 6 wherein finding at least one valid location further includes determining one or more most probable activity regions for the target object among the candidate placement surfaces by placing a grid of cells over the candidate placement surfaces and assigning a score or vote for each cell on the grid.

11. The system of claim 10 wherein finding at least one valid location further includes determining one or more most probable placement regions for the target object among the one or more most probable activity regions.

12. The system of claim 1 wherein the program is further operable to find at least one valid orientation for the at least one target object.

13. The system of claim 12 wherein finding at least one valid orientation for the at least one target object includes selecting a preferred geo-orientation based on one or more surfaces of the target object and an intent of the target object.

14. The system of claim 13 wherein finding at least one valid orientation for the at least one target object includes identifying one or more most probable interactive orientations of the target object.

15. The system of claim 11 wherein the program is further operable to find at least one valid orientation for the at least one target object by identifying one or more most probable interactive orientations of the target object.

16. The system of claim 15 wherein the program is further operable to determine a final placement and orientation for the target object by determining scores for the one or more most probable placement regions and the one or more most probable orientations of the target object.

17. The system of claim 1 wherein the at least one input device is adapted to receive at least two of speech, gesture, text and touchscreen inputs.

18. A computer-implemented method, comprising:

receiving, by a computer engine, user input that underspecifies a command to place at least one target object in a virtual environment;

interfacing, via the computer engine, with the virtual environment;

sensing, via a virtual agent associated with the computer engine, the at least one target object in the virtual environment; and

determining at least one valid location for the at least one object in the virtual environment.

19. The method of claim 18, wherein determining at least one valid location includes retrieving a linguistic placement constraint, identifying the virtual agent'"'"'s intent for a target object, identifying a placement preference for the virtual agent, determining one or more object properties of the target object and determining candidate placement surfaces for placement of the target object, determining at least one candidate activity surface on which the virtual agent is located while interacting with the at least one target object, and, for all candidate placement surfaces and the at least one candidate activity surface, identifying all objects in the virtual environment that are in contact with any of the candidate placement surfaces or the at least one candidate activity surface, determining one or more most probable activity regions for the target object among the candidate placement surfaces by placing a grid of cells over the candidate placement surfaces and assigning a score or vote for each cell on the grid, determining one or more most probable placement regions for the target object among the one or more most probable activity regions, finding at least one valid orientation for the at least one target object includes identifying one or more most probable interactive orientations of the target object, and determining a final placement and orientation for the target object by determining scores for the one or more most probable placement regions and the one or more most probable orientations of the target object.

20. A system for manipulating virtually displayed objects in a virtual environment, comprising:

a display;

at least one input device adapted to receive at least one of speech, gesture, text and touchscreen inputs; and

a computer processor adapted to execute a program stored in a computer memory, the program being operable to provide instructions to the computer processor including;

receiving user input via the at least one input device, wherein the user input underspecifies a command to place at least one target object in the virtual environment;

interfacing with the virtual environment;

sensing the at least one object in the virtual environment; and

finding at least one valid orientation for the at least one object in the virtual environment.