Annotation method and apparatus

• US 9,237,263 B2
• Filed: 10/05/2012
• Issued: 01/12/2016
• Est. Priority Date: 10/05/2012
• Status: Active Grant

• Current Assignee: BigTinCan Holdings Limited
• Original Assignee: Vidinoti SA (BigTinCan Holdings Limited)
• Inventors: Rime, Laurent, Monney, Mathieu, Ayer, Serge, Vetterli, Martin
• Primary Examiner(s): ELAHI, SHAN E
• Application Number:
  US13/645,762
• Publication Number:
  US 20140098191A1
• Time in Patent Office: 1,194 Days
• Field of Search: 348/46
• US Class Current: 1/1
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image

Abstract

The present invention relates to an annotating method including the steps of capturing (100) data representing a light field with a plenoptic image capture device (4); matching (101) the captured data with a corresponding reference data; retrieving an annotation associated with an element of the reference data (102); rendering (103) a view generated from the captured data and including at least one annotation.

Claims

1. An annotating method for adding one or more annotations to data which represent a light field, comprising the steps of:

capturing data representing a light field with a plenoptic camera in a device;

matching, using a processor, the captured data with reference data;

retrieving from a memory, using the processor, an annotation which is associated in the memory with an element of said reference data, wherein said annotation is information related to the element;

adding, using the processor, said retrieved annotation to said captured data, so as to render a view in which said captured data is shown with the retrieved annotation associated with an element shown in the captured data which corresponds to said element of said reference data;

wherein since said captured data was captured with a plenoptic camera more robust matching of said captured data with reference data is achieved, thereby providing for more reliable annotation of element(s) shown in said captured data.

2. The method of claim 1, wherein said reference data represents a reference light field.

3. The method of claim 2, including the step of generating said reference data from a 3D model of said scene.

4. The method of claim 2, wherein said step of matching includes matching said captured data with one piece of data among a plurality of pieces of reference data, each of the plurality of pieces of reference data representing a different light view.

5. The method of claim 1, further comprising a step of detecting local features in said captured data.

6. The method of claim 5, wherein said step of detecting local features includes detecting areas where pixels corresponding to a first depth of the light field have a predefined relationship with pixels corresponding to a different depth of the light field.

7. The method of claim 5, wherein said step of detecting local features includes detecting disparities in the captured data.

8. The method of claim 5, wherein said step of detecting local features includes computing epipolar volumes or lines.

9. The method of claim 5, comprising a step of describing said local features.

10. The method of claim 9, said local features being described with descriptors in a binary form.

11. The method of claim 10, comprising a step of computing a Hamming distances between said descriptors.

12. The method of claim 5, comprising a step of matching said local features in said captured data with local features in said reference data.

13. The method of claim 12, comprising a step of selecting one or more limited number of pieces of reference data depending on the position of said device, on a selection made by the user, or on a received signal, prior to said matching.

14. The method of claim 5, comprising a step of computing a descriptor of said local features in said captured data, and comparing said descriptor with a descriptor of local features in said reference data.

15. The method of claim 5, comprising a step of registering said plenoptic data with said reference data, using said local features.

16. The method of claim 5, comprising a step of detecting a type of scene and determining a type of local features to detect depending on said type of scene.

17. The method of claim 1, said reference data including a global model of a scene.

18. The method of claim 17, comprising a step of minimizing a cost function representing the error of projection of said captured data onto said reference data.

19. The method of claim 1, said step of rendering a view including rendering a 2D view from the captured data, and superimposing annotations on the 2D view.

20. The method of claim 19, said step of rendering a 2D view from the captured data including displaying annotated objects or features of an annotated image so that they are in focus, while the remaining elements of the scene are made blurry.

21. A computer program product comprising a non-transitory tangible apparatus readable medium for causing said processor to carry out the method of claim 1.

22. An annotating method comprising the steps of:

capturing data representing a light field with a plenoptic camera;

sending data by a processor, to a remote server;

matching the captured data with reference data;

retrieving from a memory, by the processor, annotations or annotated images which are associated in the memory with an element of said reference data, wherein said annotation is information related to the element;

adding, using the processor, said retrieved annotations or annotated images to said captured data, so as to render a view in which said captured data is shown with the retrieved annotations or annotated images associated with an element shown in the captured data which corresponds to said element of said reference data;

wherein since said captured data was captured with a plenoptic camera more robust matching of said captured data with reference data is achieved, thereby providing for more reliable annotation of element(s) shown in said captured data.

23. The method of claim 22, wherein said step of sending data to a remote server includes sending captured data in plenoptic format to said remote server.

24. The method of claim 22, comprising a step of computing a descriptor of local features in said captured data, and wherein said step of sending data to a remote server includes sending said descriptor to said remote server.

25. The method of claim 24, wherein said step of computing a descriptor includes detecting areas where pixels corresponding to a first depth of the light field have a predefined relationship with pixels corresponding to a different depth of the light field.

26. The method of claim 24, wherein said step of computing a descriptor includes detecting disparities in the captured data.

27. The method of claim 24, wherein said step of computing a descriptor includes computing epipolar volumes or lines.

28. An annotating method comprising the steps of:

retrieving data representing a light field, wherein said data comprises plenoptic information;

executing at a processor, for registering the retrieved data with one reference model or with one among a plurality of pieces of reference light field data;

retrieving annotations at the processor, associated in a memory with said reference model or with said one among a plurality of pieces of reference light field data, wherein said annotation is information related to an element of the reference model or reference light field data,

adding, using the processor, said retrieved annotations to said data representing a light field, so as to render a view in which said data representing a light field is shown with the retrieved annotations associated with an element shown in the data representing a light field which corresponds to said element of said of the reference model or reference light field data;

wherein since said data comprises plenoptic information more robust registering of said retrieved data with said one reference model or with one among a plurality of pieces of reference light field data is achieved, thereby providing for more reliable annotation of element(s) shown in said data representing a light field.

29. The method of claim 28, comprising a step of retrieving local features in the retrieved data.

30. An apparatus for capturing and annotating data corresponding to a scene, comprising:

a plenoptic camera for capturing data representing a light field;

a processor;

a display;

wherein said processor is configured such that it can, match data captured by the plenoptic camera with reference data;

retrieve, from a memory, an annotation which is associated in the memory with an element of said reference data, wherein said annotation is information related to the element;

add said retrieved annotation to said captured data, so as to render a view in which said captured data is shown with the retrieved annotation associated with an element shown in the captured data which corresponds to said element of said reference data;

wherein since said captured data is captured with a plenoptic camera the processor can perform more robust matching of said captured data with reference data, thereby providing for more reliable annotation of element(s) shown in said captured data.

31. The apparatus of claim 30, said processor further detecting local features present in the captured data.

32. The apparatus of claim 30, said processor further arranged for describing each detected local feature with a binary vector.

33. An apparatus for determining annotations, comprising:

a processor;

a store;

wherein said processor receives data representing a light field the data comprising plenoptic information, to match said data with one reference data in said store, determines an annotation associated in said store with said reference data the annotation being information related to an element of the reference data, and sends said annotation to a remote device,

wherein since said received data comprises plenoptic information the processor can perform more robust matching of said data with reference data, thereby providing for more reliable annotation of element(s) shown in said data representing a light field.

34. An annotating method comprising the steps of:

a) capturing data representing a light field to be annotated with a plenoptic camera in a device;

b) executing by a processor, for matching the captured data with reference data;

c) executing by the processor, for registering the captured data with respect to reference data;

d) executing by the processor, for retrieving an annotation associated in a memory with an element of said reference data, the annotation being information related to an element of the reference data;

e) executing by the processor, for adding said retrieved annotation to said captured data, so as to render a view in which said captured data is shown with the retrieved annotation associated with an element shown in the captured data which corresponds to said element of said reference data;

wherein since said captured data was captured with a plenoptic camera more robust matching of said captured data with reference data is achieved, thereby providing for more reliable annotation of element(s) shown in said captured data.

35. A method of claim 34, comprising a step of computing a geometrical transformation mapping local features in the captured data with the reference data.