Dynamic selection of surfaces in real world for projection of information thereon

• US 9,245,193 B2
• Filed: 07/13/2012
• Issued: 01/26/2016
• Est. Priority Date: 08/19/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Kulkarni, Tejas Dattatraya
• Primary Examiner(s): Harold, Jefferey
• Assistant Examiner(s): Haiem, Sean
• Application Number:
  US13/549,388
• Publication Number:
  US 20130044193A1
• Time in Patent Office: 1,292 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

One or more devices capture a scene of real world, and process one or more image(s) which include distances to points on surfaces in the real world. The distances are used to automatically identify a set of surfaces in the real world. Then, the one or more devices check whether a surface in the set is suitable for display of an element of information to be projected into the scene. On finding that a surface is suitable, a transform function is automatically identified, followed by automatic application of the transform function to the element of the information. A transformed element, which results from automatically applying the transform function, is stored in a frame buffer coupled to a projector, at a specific position in the frame buffer identified during the check for suitability. When no surface is suitable, user input is obtained, followed by projection of information as per user input.

Claims

1. A method comprising:

receiving an image of a scene in real world, the image comprising a plurality of distances to points in the real world;

using at least the plurality of distances, to identify a set of surfaces in the scene;

selecting an element of information to project on to the scene, the selected element of information having a size, the information comprising multiple elements that are independent of one another and can be projected on different surfaces;

comparing the size of the selected element to a size of a surface in the set of surfaces;

selecting the surface from the set of surfaces, on which the selected element of information can be projected, selection of the surface being based at least in part on the size of the surface being found by the comparing to be sufficiently large to accommodate the element;

one or more processors identifying a transform function to use the surface;

applying to at least the selected element of information, the transform function; and

storing in a frame buffer in a memory, at a specific position, a transformed element resulting from the applying.

2. The method of claim 1 wherein:

an area of a rectangle to display the selected element of information is compared to a total number of pixels in a region of the image corresponding to the surface.

3. The method of claim 1 wherein:

the selection of the surface comprises testing whether all pixels in a rectangle to be used to project the selected element of information are present in a region of the image corresponding to the surface.

4. The method of claim 1 wherein:

during selection of the surface, another specific position is identified on the surface, and stored in the memory.

5. The method of claim 3 wherein:

the testing is performed repeatedly, to associate with the selected element of information, multiple surfaces that form a subset of the set of surfaces, such that the selected element of information can be projected on each of the multiple surfaces;

the method further comprises assigning projection of the selected element of information to the surface which is selected from the subset.

6. The method of claim 1 further comprising:

operating a projector to project the transformed element in the frame buffer;

receiving a user input, subsequent to the operating; and

in response to receipt of the user input, repeating for a different surface, the identifying of the transform function, the applying and the storing.

7. The method of claim 6 wherein:

the user input identifies the different surface.

8. The method of claim 6 wherein:

the user input is identified as a hand gesture in another image.

9. The method of claim 1 further comprising:

receiving user input identifying the different surface when the finding indicates the selected element of information cannot be projected on any surface in the set of surfaces; and

repeating for the different surface, the identifying of the transform function, the applying and the storing.

10. The method of claim 1 further comprising:

receiving user input identifying a shape; and

using the shape in the transform function, to mask out a portion of the selected element of information.

11. One or more non-transitory computer readable storage media comprising:

instructions to receive an image of a scene in real world, the image comprising a plurality of distances to points in the scene;

instructions to use at least the plurality of distances in the image, to identify a set of surfaces in the scene;

instructions to select an element of information to project on to the scene, the selected element of information having a size, the information comprising multiple elements that are independent of one another and can be projected on different surfaces;

instructions to compare the size of the selected element to a size of a surface in the set of surfaces;

instructions to select the surface from the set of surfaces, on which the selected element of information can be projected, selection of the surface being based at least in part on the size of the surface being found by execution of the instructions to compare to be sufficiently large to accommodate the element;

instructions to one or more processors to identify a transform function to use the surface

instructions to apply to at least the selected element of information, the transform function; and

instructions to store in a frame buffer in a memory, at a specific position, a transformed element resulting from execution of the instructions to apply.

12. The one or more non-transitory computer readable storage media of claim 11 wherein:

the instructions to select comprise instructions to compare a total number of pixels in a region of the image corresponding to the surface with an area of a rectangle to display the selected element of information.

13. The one or more non-transitory computer readable storage media of claim 11 wherein:

the instructions to select comprise instructions to test whether all pixels in a rectangle to be used to display the selected element of information are present in a region of the image corresponding to the surface.

14. The one or more non-transitory computer readable storage media of claim 13 wherein:

another specific position is identified on the surface by execution of the instructions to select.

15. The one or more non-transitory computer readable storage media of claim 13 wherein:

the instructions to test are configured to be executed repeatedly, to associate with the selected element of information, multiple surfaces that form a subset of the set of surfaces, such that the selected element of information can be projected on each of the multiple surfaces;

the one or more non-transitory computer readable storage media further comprise instructions to assign projection of the selected element of information to the surface which is selected from the subset.

16. The one or more non-transitory computer readable storage media of claim 11 wherein:

instructions to operate a projector to project the transformed element in the frame buffer; and

instructions to receive a user input, subsequent to the operating; and

instructions to execute for a different surface, the instructions to identify the transform function, the instructions to apply and the instructions to store.

17. The one or more non-transitory computer readable storage media of claim 16 wherein:

the user input identifies the different surface.

18. The one or more non-transitory computer readable storage media of claim 16 wherein:

the user input is identified in at least another image.

19. The one or more non-transitory computer readable storage media of claim 11 further comprising:

instructions to receive a user input identifying the different surface, configured to be executed when execution of the instructions to check indicates the selected element of information cannot be projected on any surface in the set of surfaces; and

instructions to repeat for the different surface, the instructions to identify the transform function, the instructions to apply and the instructions to store.

20. The one or more non-transitory computer readable storage media of claim 11 further comprising:

instructions to receive a user input identifying a shape; and

instructions to use the shape in the transform function, to mask out a portion of the selected element of information.

21. One or more devices comprising:

a camera;

one or more processors operatively coupled to the camera;

memory operatively coupled to the one or more processors;

a projector operatively coupled to a frame buffer in the memory, to project information therefrom; and

software held in the memory that when executed by the one or more processors, causes the one or more processors to;

receive from the camera an image of a scene in real world, the image comprising a plurality of distances to points in the real world;

use at least the plurality of distances, to identify a set of surfaces in the scene;

select an element of information to project on to the scene, the selected element of information having a size, the information comprising multiple elements that are independent of one another and can be projected on different surfaces;

compare the size of the selected element to a size of a surface in the set of surfaces;

select the surface from the set of surfaces, on which the selected element of information can be projected, selection of the surface being based at least in part on the size of the surface being found by comparison to be sufficiently large to accommodate the element;

identify a transform function to use the surface;

apply to at least the selected element of information, the transform function; and

store in the frame buffer, at a specific position, a transformed element resulting from application of the transform function.

22. The one or more devices of claim 21 wherein:

the software to check causes the one or more processors to test whether all pixels in a rectangle to be used to display the selected element of information are present in a region of the image corresponding to the surface.

23. The one or more devices of claim 22 wherein:

the software to test is executed repeatedly, to associate with the selected element of information, multiple surfaces that form a subset of the set of surfaces, such that the selected element of information can be projected on each of the multiple surfaces;

the software further causes assignment of projection of the selected element of information to whichever surface in the subset has fewest number of elements associated therewith.

24. A system comprising a processor operatively coupled to a memory and a camera, the system comprising:

means for receiving an image of a scene in real world, the image comprising a plurality of distances to points in the real world;

means for using at least the plurality of distances, to identify a set of surfaces in the scene;

means for selecting an element of information to project on to the scene, the selected element of information having a size, the information comprising multiple elements that are independent of one another and can be projected on different surfaces;

means for comparing the size of the selected element to a size of a surface in the set of surfaces;

means for selecting the surface from the set of surfaces, on which the selected element of information can be projected, selection of the surface being based at least in part on the size of the surface being found by the means for comparing to be sufficiently large to accommodate the element;

means for identifying a transform function to use the surface;

means for applying to at least the element of information, the transform function; and

means for storing in a frame buffer in the memory, at a specific position, a transformed element output by the means for applying.

25. The system of claim 24 wherein the means for finding comprises:

means for testing whether all pixels in a rectangle to be used to display the selected element of information are present in a region of the image corresponding to the surface.

26. The system of claim 25 wherein:

the means for testing is configured to be operated repeatedly, to associate with the selected element of information, multiple surfaces that form a subset of the set of surfaces, such that the selected element of information can be projected on each of the multiple surfaces; and

projection of the selected element of information is assigned to whichever surface in the subset has fewest number of elements associated therewith.