SPATIAL COOPERATIVE PROGRAMMING LANGUAGE

• US 20170255450A1
• Filed: 03/04/2016
• Published: 09/07/2017
• Est. Priority Date: 03/04/2016
• Status: Abandoned Application

• Current Assignee: RPX Corporation
• Original Assignee: DAQRI, LLC
• Inventors: Mullins, Brian, Kammerait, Matthew
• Application Number:
  US15/061,865
• Publication Number:
  US 20170255450A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 8/315 Object-oriented languages
• CPC Class Codes: G06F 8/34 Graphical or visual programming
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06T 2219/028 Multiple view windows (top-side-front-sagittal-orthogonal)

Abstract

A system and method for a mixed reality, spatial, cooperative programming language is described. A sensor of a device detects a first physical object and a second physical object. An augmented reality application identifies the first and second physical objects and a physical state of the first and second physical objects, generates a programming logic associated with the identification and physical state of the first and second physical objects, generates augmented or virtual reality information related to the programming logic, and displays the augmented or virtual reality information in the display.

Claims

1. A device comprising:

a sensor configured to detect a first physical object and a second physical object;

a display; and

one or more processors comprising an augmented reality (AR) application,

the AR application configured to identify the first and second physical objects and a physical state of the first and second physical objects, to generate a programming logic associated with the identification and physical state of the first and second physical objects, to generate augmented reality information related to the programming logic, and to display the augmented reality information in the display.

2. The device of claim 1, wherein the physical state of the first physical object includes an identification of a first location of the first physical object, a first relative position of the first physical object with respect to the second physical object, and a first interaction of a user of the device with the first physical object, the first physical object not electronically connected to the second physical object and the device, and

wherein the physical state of the second physical object includes an identification of a second location of the second physical object, a second relative position of the second physical object with respect to the first physical object, and a second interaction of the user of the device with the second physical object, the second physical object not electronically connected to the first physical object and the device.

3. The device of claim 1, wherein the first or second interaction includes a motion of the first or second physical object in a direction and a rotation of the first or second physical object along an axis, the programming logic based on the motion and the rotation of the first and second physical objects.

4. The device of claim 1, wherein the physical state identifies that the first physical object is aligned with the second physical object along an axis, the programming logic based on the alignment of the first and second physical object along the axis.

5. The device of claim 1, wherein the physical state identifies that the first physical object is grouped with the second physical object based on the distance between the first and the second physical object being within a threshold distance, and

wherein the AR application is configured to generate a second programming logic based on the grouping of the first and second physical object.

6. The device of claim 5, wherein the sensor comprises an optical sensor configured to capture an image of the first and second physical object and a third physical object within a same field of view, and

wherein the AR application is configured to generate a third programming logic associated with the identification and the physical state of the third physical object, and the distance between the third physical object exceeding the threshold distance, the third programming logic configured to perform an operation on a result of the second programming logic.

7. The device of claim 1, wherein the augmented reality information includes a visual link between the first and second physical object, textual information related to the programming logic of the first and second physical objects and to the visual link.

8. The device of claim 7, wherein the AR application dynamically updates the augmented reality information based on the physical states of the first and second physical objects and a contextual information of the first and second physical objects, the contextual information comprising a user profile, a location of the device, and a task selected by a user of the device.

9. The device of claim 1, wherein the AR application is configured to:

receive a selection of a first programming logic for the first physical object from a user of the device, a selection of a second programming logic for the second physical object from the user of the device,

associate the first programming logic with the identification of the first physical object and a physical state of the first physical object,

associate the second programming logic with the identification of the second physical object and a physical state of the second physical object,

generate a first augmented reality visual indicator associated with the first programming logic and display the first augmented reality visual indicator in the display, the first augmented reality visual indicator perceived by the user of the device as displayed next to the first physical object, and

generate a second augmented reality visual indicator associated with the second programming logic and display the second augmented reality visual indicator in the display, the second augmented reality visual indicator perceived by the user of the device as displayed in relation to the second physical object.

10. The device of claim 1, wherein the first and second physical objects include non-electric physical objects, wherein the programming logic includes an operator of a programming language, wherein the programming logic affects at least one of the first and second physical objects.

11. A method comprising:

detecting a first physical object and a second physical object with a device;

identifying, using an augmented reality application implemented in a hardware processor of the device, the first and second physical objects and a physical state of the first and second physical objects;

generating a programming logic associated with the identification and physical state of the first and second physical objects;

generating augmented reality information related to the programming logic; and

displaying the augmented reality information in a display of the device.

12. The method of claim 11, wherein the physical state of the first physical object includes an identification of a first location of the first physical object, a first relative position of the first physical object with respect to the second physical object, and a first interaction of a user of the device with the first physical object, the first physical object not electronically connected to the second physical object and the device, and

wherein the physical state of the second physical object includes an identification of a second location of the second physical object, a second relative position of the second physical object with respect to the first physical object, and a second interaction of the user of the device with the second physical object, the second physical object not electronically connected to the first physical object and the device.

13. The method of claim 11, wherein the first or second interaction includes a motion of the first or second physical object in a direction and a rotation of the first or second physical object along an axis, the programming logic based on the motion and the rotation of the first and second physical objects.

14. The method of claim 11, wherein the physical state identifies that the first physical object is aligned with the second physical object along an axis, the programming logic based on the alignment of the first and second physical object along the axis.

15. The method of claim 11, wherein the physical state identifies that the first physical object is grouped with the second physical object based on the distance between the first and the second physical object being within a threshold distance, and

wherein the method further comprises generating a second programming logic based on the grouping of the first and second physical object.

16. The method of claim 15, further comprising:

capturing an image of the first and second physical object and a third physical object within a same field of view of anoptical sensor of the device; and

generating a third programming logic associated with the identification and the physical state of the third physical object, and the distance between the third physical object exceeding the threshold distance, the third programming logic configured to perform an operation on a result of the second programming logic.

17. The method of claim 11, wherein the augmented reality information includes a visual link between the first and second physical object, textual information related to the programming logic of the first and second physical objects and to the visual link.

18. The method of claim 17, further comprising:

dynamically updating the augmented reality information based on the physical states of the first and second physical objects and a contextual information of the first and second physical objects, the contextual information comprising a user profile, a location of the device, and a task selected by a user of the device; and

updating a state of at least one of the first and second physical objects based on the programming logic.

19. The method of claim 11, further comprising:

receiving a selection of a first programming logic for the first physical object from a user of the device, a selection of a second programming logic for the second physical object from the user of the device;

associating the first programming logic with the identification of the first physical object and a physical state of the first physical object;

associating the second programming logic with the identification of the second physical object and a physical state of the second physical object;

generating a first augmented reality visual indicator associated with the first programming logic and display the first augmented reality visual indicator in the display, the first augmented reality visual indicator perceived by the user of the device as displayed next to the first physical object; and

generating a second augmented reality visual indicator associated with the second programming logic and display the second augmented reality visual indicator in the display, the second augmented reality visual indicator perceived by the user of the device as displayed in relation to the second physical object.

20. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

detecting a first physical object and a second physical object with a sensor of a device;

identifying, using an augmented reality application implemented in a hardware processor of the device, the first and second physical objects and a physical state of the first and second physical objects;

generating a programming logic associated with the identification and physical state of the first and second physical objects;

generating augmented reality information related to the programming logic; and

displaying the augmented reality information in a display of the device.