Trimming content for projection onto a target

• US 9,484,005 B2
• Filed: 12/20/2013
• Issued: 11/01/2016
• Est. Priority Date: 12/20/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Rezaiifar, Ramin, Bernarte, Joel Simbulan, Padovani, Niccolo Andrew, Keating, Virginia Walker
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US14/136,310
• Publication Number:
  US 20150179147A1
• Time in Patent Office: 1,047 Days
• Field of Search: 345/625
• US Class Current: 1/1
• CPC Class Codes: A61B 5/1114 Tracking parts of the body
• CPC Class Codes: G03B 21/13 Projectors for producing special effects at the edges of picture, e.g. blurring
• CPC Class Codes: G03B 21/142 Adjusting of projection optics
• CPC Class Codes: G03B 2206/00 Systems for exchange of information between different pieces of apparatus, e.g. for exchanging trimming information, for photo finishing
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 2210/32 Image data format
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G09G 2340/04 Changes in size, position or resolution of an image
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/373 for modifying the size of the graphic pattern
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

Systems and methods are provided for trimming content for projection within the bounds of a projection target. The systems and methods trim the content for projection based on one or more characteristics of the projection target, including a shape, outline, and distance to the projection target. Moreover, the systems and methods designate void areas where no content will be projected based on the one or more characteristics, and the void areas will be generated or otherwise projected along with the content so that the content is projected onto the projection target and the void areas are projected outside of the projection target such that the projected content does not significantly spill onto surfaces or objects outside of the projection target.

Claims

1. A method, comprising:

determining a projection target by using object recognition to recognize an object in one more image frames, the object recognition based on information associated with the target;

determining one or more bounds of the projection target;

determining, by one or more processors, a content area where content is to be projected and a void area where no content is to be projected, the content area and the void area determined based on the one or more bounds, wherein the content area is further determined based on a designated content area associated with the object;

determining, by the one or more processors, a scaling factor;

generating content for projection within the content area based on the scaling factor; and

generating the void area.

2. The method of claim 1, wherein determining one or more bounds of the projection target comprises:

receiving one or more images of the projection target; and

determining a shape of the projection target from the one or more images.

3. The method of claim 2, further comprising determining an outline of the projection target based on the determined shape of the projection target.

4. The method of claim 2, further comprising determining a depth map of an area including the projection target based on the one or more images.

5. The method of claim 4, wherein determining a scaling factor comprises determining a distance to the projection target based on the determined depth map.

6. The method of claim 4, wherein determining a shape of the projection target comprises segmenting the projection target out from a background in the determined depth map.

7. The method of claim 2, wherein generating the void area comprises creating a virtual mask based on the determined shape of the projection target.

8. The method of claim 1, further comprising projecting, by projector, the generated content on the projection target within the determined content area.

9. The method of claim 1, further comprising projecting, by a projector, the generated void area.

10. The method of claim 9, wherein projecting the generated void area comprises projecting dark pixels.

11. The method of claim 1, further comprising:

tracking the projection target; and

adjusting the determined content area and the determined void area based on the tracking.

12. The method of claim 1, further comprising:

detecting a gesture; and

determining, based on the detected gesture, the void area to have no area.

13. The method of claim 1, further comprising:

detecting a gesture;

determining a second projection target based on the detected gesture;

determining one or more bounds of the second projection target;

determining a second content area and a second void area based on the determined one or more bounds of the second projection target;

determining a second scaling factor based on a distance to the second projection target; and

projecting, by a projection device, content onto the second projection target within the determined content area.

29. The method of claim 1, wherein the information associated with the target comprises at least one of information provided by a user, information provided by a content provider, information provided by a retailer, or information provided by a manufacturer.

14. An apparatus, comprising:

a projector configured to project a content area and a void area based on one or more projection parameters of a projection target, wherein the projection target is determined by using objection recognition to recognize an object in one more image frames, the object recognition based on information associated with the target, wherein the content area is further determined based on a designated content area associated with the object;

a camera configured to capture one or more images of a field of view including the projection target;

one or more processors coupled to the projector and the camera, the one or more processors configured to determine the one or more projection parameters; and

a memory coupled to the one or more processors.

15. The apparatus of claim 14, wherein the one or more processors are configured to determine the one or more projection parameters by processing the one or more images and determining a distance to the projection target, a shape of the projection target, and an outline of the projection target.

16. The apparatus of claim 14, wherein the void area comprises dark pixels projected outside of the determined outline of the content area.

17. The apparatus of claim 14, wherein the camera comprises at least one of a Digital Light Processing (DLP) projector, a laser beam-steering (LBS) projector, a liquid crystal on silicon (LCoS) projector.

18. The apparatus of claim 14, wherein the camera is further configured to capture gestures made by a user.

19. The apparatus of claim 18, wherein the gestures include gestures for switching between a private mode and a public mode.

30. The apparatus of claim 14, wherein the information associated with the target comprises at least one of information provided by a user, information provided by a content provider, information provided by a retailer, or information provided by a manufacturer.

20. A system, comprising:

means for determining a projection target, by using object recognition to recognize an object in one more image frames, the object recognition based on information associated with the target;

means for determining one or more bounds of the projection target;

means for determining a content area where content is to be projected and a void area where no content is to be projected based on the determined one or more bounds, wherein the content area is further determined based on a designated content area associated with the object;

means for determining a scaling factor;

means for generating content for projection within the content area based on the scaling factor; and

means for generating the void area.

21. The system of claim 20, wherein the means for determining one or more bounds of the projection target comprises:

means for receiving one or more images of the projection target; and

means for determining a shape of the projection target from the one or more images.

22. The system of claim 21, further comprising means for determining a depth map of an area including the projection target based on the one or more images.

23. The system of claim 20, further comprising a means for projecting the generated content on the projection target within the determined content area.

24. The system of claim 20, further comprising means for projecting the generated void area.

25. The system of claim 20, further comprising:

means for tracking the projection target; and

means for adjusting the determined content area and the determined void area based on the tracking.

26. The system of claim 20, further comprising:

means for detecting a gesture; and

means for determining, based on the detected gesture, the void area to have no area.

27. The system of claim 20, further comprising:

means for detecting a gesture;

means for determining a second projection target based on the detected gesture;

means for determining one or more bounds of the second projection target;

means for determining a second content area and a second void area based on the determined one or more bounds of the second projection target;

means for determining a second scaling factor based on a distance to the second projection target; and

means for projecting, by a projection device, content onto the second projection target within the determined content area.

31. The system of claim 20, wherein the information associated with the target comprises at least one of information provided by a user, information provided by a content provider, information provided by a retailer, or information provided by a manufacturer.

28. A non-transitory computer-readable medium including instructions that when executed by one or more processors cause a system including the one or more processors to perform a method, comprising:

determining a projection target, by using object recognition to recognize an object in one more image frames, the object recognition based on information associated with the target, wherein the content area is further determined based on a designated content area associated with the object;

determining one or more bounds of the projection target;

determining, by one or more processors, a content area where content is to be projected and a void area where no content is to be projected based on the determined one or more bounds;

determining, by the one or more processors, a scaling factor; and

generating content for projection within the content area based on the scaling factor; and

generating the void area.

32. The non-transitory computer-readable medium of claim 28, wherein the information associated with the target comprises at least one of information provided by a user, information provided by a content provider, information provided by a retailer, or information provided by a manufacturer.