Augmented reality skin manager

• US 9,953,462 B2
• Filed: 01/31/2014
• Issued: 04/24/2018
• Est. Priority Date: 01/31/2014
• Status: Active Grant

• Current Assignee: Koninklijke Philips N.V.
• Original Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Inventors: Ur, Shmuel, Medan, Yoav
• Primary Examiner(s): Buttram, Todd
• Application Number:
  US14/384,825
• Publication Number:
  US 20160240004A1
• Time in Patent Office: 1,544 Days
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2024 Style variation

Abstract

Technologies are generally described for systems, devices and methods effective to set up image data to be processed by an augmented reality device. In some examples, a processor may generate image data from light reflected from a real object. The image data may represent, or may be a digital representation of, the real object. The processor may analyze a skin stack of augmented reality skins and identify one of the augmented reality skins stored in the memory. The augmented reality skin may include instructions effective to modify some of the image data when the image data includes a feature. The processor may create metadata related to the image data. The metadata may relate to the presence of the feature in the image data. The processor may store the metadata in the memory.

Claims

1. A method to set up image data to be processed by an augmented reality device, the method comprising, by a processor:

generating image data from light reflected from a real object, wherein the image data represents the real object;

storing the image data in a memory;

identifying an augmented reality skin stored in the memory, wherein the augmented reality skin comprises instructions to identify a reference point of a feature of the real object within the image data;

analyzing the augmented reality skin to identify the reference point of the feature of the real object within the image data on which the augmented reality skin would register upon execution of the augmented reality skin;

creating metadata related to the image data, wherein the metadata relates to the presence of the feature, on which the augmented reality skin would register upon execution of the augmented reality skin, in the image data;

storing the metadata related to the image data that includes the identification of the presence of the feature in the image data, in the memory; and

updating the metadata after execution of the augmented reality skin.

2. The method of claim 1, wherein the augmented reality skin includes a first augmented reality skin, wherein the feature includes a first feature of the real object within the image data, and wherein the method further comprises, by the processor:

receiving a request to execute a second augmented reality skin;

analyzing the second augmented reality skin to identify a reference point of a second feature of the real object within the image data on which the second augmented reality skin would register upon execution of the second augmented reality skin;

determining whether the second feature is present in the metadata;

executing the second augmented reality skin on the image data to produce augmented data, in response to a determination that the second feature is present in the metadata; and

rendering the augmented data on a display to produce an augmented image.

3. The method of claim 2, further comprising, by the processor, updating the metadata based on differences between the augmented data and the image data.

4. The method of claim 1, wherein the augmented reality skin includes a first augmented reality skin, and wherein the method further comprises, by the processor:

receiving a first request to execute the first augmented reality skin;

receiving a second request to execute a second augmented reality skin;

determining a first amount of time between when the image data was generated and when the first augmented reality skin is to be executed;

determining a second amount of time between when the image data was generated and when the second augmented reality skin is to be executed; and

executing the first augmented reality skin prior to the second augmented reality skin when the first amount of time is less than the second amount of time.

5. The method of claim 1, further comprising, by the processor:

receiving a request to execute the augmented reality skin;

determining whether the feature, upon which the augmented reality skin registers, is present in the metadata; and

denying execution of the augmented reality skin based on a determination that the feature, upon which the augmented reality skin registers, is not present in the metadata.

6. The method of claim 1, wherein the metadata relates to a location, color, position, size, transparency, depth and/or identification of features in the image data.

7. The method of claim 1, further comprising, by the processor, defining one or more restrictions on execution of the augmented reality skin.

8. The method of claim 7, wherein the one or more restrictions comprise one or more of memory, processing power, latency guarantees and/or refresh rates allocated for the augmented reality skin.

9. The method of claim 7, wherein the one or more restrictions comprise one or more global constraints that prevent the augmented reality skin from altering at least a portion of the image data.

10. The method of claim 9, wherein the one or more global constraints relate to one or more of security, privacy or visual appearance of one or more objects represented by the image data.

11. The method of claim 1, further comprising capturing, by a camera, the light reflected from the real object.

12. A method to determine whether to execute an augmented reality skin by an augmented reality device, the method comprising, by a processor:

generating image data from light reflected from a real object, wherein the image data represents the real object;

storing the image data in a memory;

identifying a first augmented reality skin to be applied to the image data;

analyzing the first augmented reality skin to identify a feature on which the first augmented reality skin would register upon execution of the first augmented reality skin;

receiving a first request to execute the first augmented reality skin;

receiving a second request to execute a second augmented reality skin;

defining one or more restrictions on the execution of the first augmented reality skin or the second augmented reality skin;

determining whether the execution of the first augmented reality skin or the second augmented reality skin would violate at least one of the one or mroe restrictions, wherein the determination includes;

determining a first amount of time between when the image data was generated and when the first augmented reality skin is to be executed, and

determining a second amount of time between when the image data was generated and when the second augmented reality skin is to be executed;

executing the first augmented reality skin prior to the second augmented reality skin, when the determination indicates that the execution of the first augmented reality skin or the second augmented reality skin would not violate the at least one of the one or more restrictions, wherein the one or more restrictions include that the first amount of time is less than the second amount of time; and

updating the metadata after executing the augmented reality skin.

13. The method of claim 12, wherein the one or more restrictions relate to one or more of memory, processing power, latency guarantees and/or refresh rates.

14. The method of claim 12, wherein the one or more restrictions further include one or more global constraints that prevent one or more augmented reality skins of the first augmented reality skin and the second augmented reality skin from altering at least a portion of the image data.

15. The method of claim 14, wherein the one or more global constraints relate to one or more of security, privacy or visual appearance of one or more objects represented by the image data.

16. The method of claim 12, wherein:

the one or more restrictions further include allocated resources; and

the method further comprises denying execution of the first augmented reality skin or the second augmented reality skin, when the determination indicates that the execution of the first augmented reality skin or the second augmented reality skin would violate the at least one of the one or more restrictions.

17. The method of claim 12, further comprising, by the processor, monitoring resource use of the first augmented reality skin or the second augmented reality skin during the execution.

18. The method of claim 17, further comprising, by the processor, terminating the execution of the first augmented reality skin or the second augmented reality skin when the resource use exceeds allocated resources identified in the one or more restrictions.

19. The method of claim 12, further comprising capturing, by a camera, the light reflected from the real object.

20. An augmented reality device, comprising:

a memory that includes an augmented reality skin;

a processor configured to be in communication with the memory; and

a camera configured to be in communication with the processor,

wherein the camera is effective to capture light reflected from a real object,

and wherein the processor is effective to;

generate image data from the light, wherein the image data represents the real object;

store the image data in the memory;

identify the augmented reality skin stored in the memory;

analyze the augmented reality skin to identify a feature, on which the augmented reality skin would register upon execution of the augmented reality skin, wherein the augmented reality skin comprises instructions to identify a reference point of the feature of the real object within the image data;

create metadata related to the image data, wherein the metadata relates to the presence of the feature, on which the augmented reality skin would register upon execution of the augmented reality skin, in the image data;

update the metadata upon the execution of the augmented reality skin; and

store the metadata related to the image data that includes the identification of the presence of the feature in the image data, in the memory.

21. The augmented reality device of claim 20, wherein the processor is further effective to:

receive a request to execute the augmented reality skin;

determine whether the feature is present in the metadata;

execute the augmented reality skin to produce augmented data, based on a determination that the feature is present in the metadata; and

render the augmented data on a display to produce an augmented image.

22. The augmented reality device of claim 20, wherein the processor is further effective to update the metadata based on differences between the augmented data and the image data.