Interacting with a model via a three-dimensional (3D) spatial environment

• US 9,582,933 B1
• Filed: 12/28/2012
• Issued: 02/28/2017
• Est. Priority Date: 06/26/2012
• Status: Active Grant

• Current Assignee: The MathWorks, Inc.
• Original Assignee: The MathWorks, Inc.
• Inventors: Mosterman, Pieter J., Mestchian, Ebrahim, Torgerson, Jay R., Higgins, Daniel F., Kinnucan, Paul F.
• Primary Examiner(s): Zalalee, Sultana M
• Application Number:
  US13/730,279
• Time in Patent Office: 1,523 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 30/20 Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

A device may provide, for display by a technical computing environment (TCE), a group of model elements of a model. The model, when executed, may simulate behavior of a system. The group of model elements may correspond to a group of physical elements of the system. The device may further detect interaction with a three-dimensional (3D) structure located within a spatial environment. The 3D structure may correspond to one or more physical elements of the group of physical elements of the system. The device may further cause the TCE to modify the model based on the detected interaction.

Claims

1. A method comprising:

receiving spatial information associated with a three-dimensional (3D) structure that is located within a spatial environment,

the spatial information identifying characteristics of the 3D structure,

the characteristics including information identifying at least one of;

dimensions of the 3D structure,

surface contours of the 3D structure,

a location of the 3D structure within the spatial environment, or

an orientation of the 3D structure within the spatial environment,

the receiving the spatial information being performed by a computing device, and

the computing device being associated with a technical computing environment (TCE);

determining at least one of a movement, a gesture, or a signal within the spatial environment,

the at least one of the movement, the gesture, or the signal being associated with at least one of;

all or a portion of a body of a user that is within the spatial environment,

an inanimate physical object within the spatial environment, or

the 3D structure,

the determining the at least one of the movement, the gesture, or the signal being performed by the computing device;

performing an operation on a model based on the at least one of the movement, the gesture, or the signal,

the model being a topological model that, when executed, simulates behavior of a dynamic system,

the 3D structure being associated with one or more model elements of the model,

the performing the operation comprising;

navigating the one or more model elements, associated with the 3D structure, based on at least one of the characteristics of the 3D structure identified by the spatial information, the movement, the gesture, or the signal,

the one or more model elements, when executed, simulating behavior of a part of the dynamic system,

the navigating the one or more model elements being performed by the computing device; and

causing, based on navigating the one or more model elements and based on the spatial information, at least one model element of the model to be displayed in the TCE,

the causing the at least one model element of the model to be displayed being performed by the computing device.

2. The method of claim 1, where the 3D structure includes a 3D physical structure.

3. The method of claim 1, where the 3D structure includes a 3D virtual representation of at least a portion of the dynamic system.

4. The method of claim 1, further comprising:

identifying, based on the spatial information, the part of the dynamic system,

the part of the dynamic system being linked to the one or more model elements of the model, and

where causing the at least one of the model elements to be displayed is based on identifying the part of the dynamic system.

5. The method of claim 1, where the characteristics include the location and the orientation of the 3D structure within the spatial environment,

where the 3D structure corresponds to a portion of the dynamic system, and

where the method further comprises:

obtaining information associated with a 3D spatial representation of the dynamic system,

the 3D spatial representation being based on a viewing angle, the location, and the orientation, and

the information, associated with the 3D spatial representation, representing a stereoscopic projection of the dynamic system; and

providing, for display, the 3D spatial representation based on the information associated with the 3D spatial representation,

the 3D spatial representation being different than the one or more model elements and the 3D spatial representation being provided for display simultaneously with the one or more model elements.

6. The method of claim 5, further comprising:

identifying a spatial element, associated with the 3D spatial representation, based on the viewing angle, the location, and the orientation,

the spatial element corresponding to the part of the dynamic system,

the part of the dynamic system being visible when the dynamic system is viewed from the viewing angle, while being positioned at the location and orientation; and

causing the spatial element to be displayed in a 3D display medium or in a 3D volume of space.

7. The method of claim 1, further comprising:

receiving other spatial information associated with the 3D structure;

determining, based on the other spatial information, that the 3D structure has changed from a first location or orientation, within the spatial environment, to a second location or orientation within the spatial environment,

the first location or orientation being identified by the spatial information, and

the second location or orientation being identified by the other spatial information; and

causing one or more different model elements to be displayed in the TCE based on the other spatial information.

8. The method of claim 1, further comprising:

obtaining a 3D spatial representation of the dynamic system,

the 3D spatial representation including a plurality of spatial elements;

identifying, based on determining the movement, the gesture, or the signal, an instruction to perform an operation on the 3D spatial representation; and

performing the operation on the 3D spatial representation based on identifying the instruction.

9. The method of claim 8, where the instruction corresponds to at least one of:

a first instruction to delete a spatial element of the plurality of spatial elements,

a second instruction to zoom in to the spatial element when the 3D spatial representation is being displayed,

a third instruction to zoom out from a first portion of the 3D spatial representation, that includes the spatial element, to a second portion of the 3D spatial representation when the 3D spatial representation is being displayed,

a fourth instruction to change a viewing angle associated with the spatial element,

a fifth instruction to change a viewing angle associated with the 3D spatial representation, or

a sixth instruction to change a location associated with the 3D spatial representation.

10. The method of claim 1, where performing the operation includes at least one of:

deleting a particular model element of the model;

causing a user interface, associated with the TCE, to navigate to a particular model element of the model;

causing the user interface to navigate from a first portion of the model to a second portion of the model;

causing the user interface to zoom into or out of a particular model element of the model;

causing a parameter, associated with a particular model element of the model to change;

causing one or more equations to be added to or removed from a particular model element of the model;

or

causing one or more equations, associated with a particular model element of the model, to be changed.

11. The method of claim 1, further comprising:

receiving the signal from the inanimate physical object; and

identifying, based on receiving the signal, the operation to be performed.

12. The method of claim 1,

where causing the at least one model element of the model to be displayed includes at least one of:

causing the at least one model element to be superimposed on a 3D representation of a portion of the dynamic system, or

causing the at least one model element to be displayed on the 3D structure.

13. The method of claim 1, where the model is at least one of a time-based block diagram, a discrete-event based diagram, a dataflow diagram, a state transition diagram, a software diagram, or a noncausal block diagram.

14. The method of claim 1, where the model is a block diagram and the one or more model elements include blocks logically connected by connectors.

15. The method of claim 1, where the 3D structure includes at least one of:

a 3D physical structure,

a user device,

the all or the portion of the body of the user that is within the spatial environment, or

the inanimate physical object within the spatial environment.

16. A computing device comprising:

one or more processors to;

provide, for display by a technical computing environment (TCE), a first portion of a model,

the model being a topological model that, when executed, simulates behavior of a dynamic system, and

the model including a plurality of model elements that, when executed, simulate behavior of parts of the dynamic system,

receive first spatial information associated with a three-dimensional (3D) structure that is located within a spatial environment,

the first spatial information including information identifying a first location and a first orientation of the 3D structure within the spatial environment,

determine at least one of a movement, a gesture, or a signal within the spatial environment,

the at least one of the movement, the gesture, or the signal being associated with at least one of;

all or a portion of a body of a user that is within the spatial environment,

an inanimate physical object within the spatial environment, or

the 3D structure;

perform an operation on the first portion of the model based on the at least one of the movement, the gesture, or the signal,

the 3D structure being associated with one or more model elements of the first portion of the model,

where the one or more processors, when performing the operation, are to;

navigate the one or more model elements associated with the 3D structure, based on at least one of the spatial information, the movement, the gesture, or the signal,

and

cause, based on navigating the one or more model elements associated with the 3D structure, a second, different portion of the model to be displayed in the TCE.

17. The computing device of claim 16, where the 3D structure includes at least one of:

an inanimate physical object,

a 3D virtual representation of the system, or

the all or the portion of the body of the user that is within the spatial environment.

18. The computing device of claim 16, where the 3D structure includes a user device, and

where, when receiving the first spatial information, the one or more processors are to:

receive a signal from the user device, and

obtain, from the received signal, the first location and the first orientation.

19. The computing device of claim 16, where the one or more processors are further to:

receive other spatial information associated with the 3D structure,

determine, based on the other spatial information, that the 3D structure has changed from the first location or the first orientation, within the spatial environment, to a second location or a second orientation within the spatial environment,

identify a different part of the system based on the second location or the second orientation, and

cause the TCE to navigate to one or more different model elements based on identifying the different part of the system.

20. The computing device of claim 16, where the one or more processors, when determining at least one of the movement, the gesture, or the signal, are to:

detect a particular gesture within the spatial environment.

21. The computing device of claim 16, where the one or more processors, when determining at least one of the movement, the gesture, or the signal, are to:

detect a particular gesture or movement within the spatial environment.

22. A non-transitory computer-readable medium for storing instructions, the instructions comprising:

a plurality of instructions that, when executed by a processor of a computing device, cause the processor to;

provide, for display by a technical computing environment (TCE), a plurality of model elements of a model,

the model being a topological model that, when executed, simulates behavior of a dynamic system,

the model including a plurality of model elements that, when executed, simulate behavior of parts of the dynamic system,

receive spatial information associated with a three-dimensional (3D) structure located within a spatial environment,

the 3D structure being associated with one or more model elements of the plurality of model elements of the model,

determine at least one of a movement, a gesture, or a signal within the spatial environment,

the at least one of the movement, the gesture, or the signal being associated with at least one of;

all or a portion of a body of a user that is within the spatial environment,

an inanimate physical object within the spatial environment, or

the 3D structure;

perform an operation on the model based on the at least one of the movement, the gesture, or the signal,

where the plurality of instructions to perform the operation include;

a plurality of instructions to navigate the one or more model elements associated with the 3D structure, based on at least one of the spatial information, the movement, the gesture, or the signal; and

cause, based on navigating the one or more model elements and based on the spatial information, at least one model element of the plurality of model elements of the model to be displayed in the TCE.

23. The non-transitory computer-readable medium of claim 22, where the instructions further include:

a plurality of instructions that, when executed by the processor, cause the processor to;

execute the model,

obtain first results based on executing the model,

modify the model to generate a modified model,

execute the modified model, and

obtain second results based on executing the modified model,

the second results being different than the first results.

24. The non-transitory computer-readable medium of claim 23,

where the plurality of instructions to cause the processor to modify the model include:

a plurality of instructions that, when executed by the processor, cause the processor to;

change a parameter associated with at least one model element, of the plurality of model elements, based on the movement, the gesture, or the signal.

25. The non-transitory computer-readable medium of claim 23,

where the plurality of instructions to cause the processor to modify the model include:

a plurality of instructions that, when executed by the processor, cause the processor to;

delete at least one model element, of the plurality of model elements, based on the movement, the gesture, or the signal.