Editing digital notes representing physical notes

• US 9,274,693 B2
• Filed: 10/15/2014
• Issued: 03/01/2016
• Est. Priority Date: 10/16/2013
• Status: Active Grant

• Current Assignee: 3M Innovative Properties Company (3M Company)
• Original Assignee: 3M Innovative Properties Company (3M Company)
• Inventors: Moore, Richard J., Leighton, Michael C., Somasundaram, Guruprasad, kerlund, Linus, Persson, Olof Roland, Aybar Lpez, Hctor M., Wolk, Diane R., Axelsson, Pontus, Kjellberg, Staffan H.
• Primary Examiner(s): BHARGAVA, ANIL K
• Application Number:
  US14/514,766
• Publication Number:
  US 20150106755A1
• Time in Patent Office: 503 Days
• Field of Search: 715/765
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/106 Display of layout of documents; Previewing
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text

Abstract

In one example, a method includes receiving a digital note of a plurality of digital notes generated based on image data comprising a visual representation of a scene that includes a plurality of physical notes such that each of the plurality of digital notes respectively corresponds to a particular physical note of the plurality of physical notes, wherein each of the physical notes includes respective recognizable content. In this example, the method also includes receiving user input indicating a modification to one or more visual characteristics of the digital note. In this example, the method also includes editing, in response to the user input, the one or more visual characteristics of the digital note. In this example, the method also includes outputting, for display, a modified version of the digital note that includes the one or more visual characteristics.

Claims

1. A method comprising:

receiving, by one or more processors of a device, image data comprising a visual representation of a scene that includes a plurality of physical notes, each of the physical notes comprising a separate physical object having recognizable content thereon;

extracting, by the one or more processors and from the image data, the recognizable content of a particular one of the plurality of physical notes;

creating, by the one or more processors, a digital note representative of the particular one of the plurality of physical notes, wherein the particular one of the plurality of physical notes includes a first colored region and a second colored region;

associating, by the one or more processors, the extracted recognizable content with the digital note representative of the particular one of the plurality of physical notes;

identifying, for the digital note, the first colored region and a second colored region;

determining an angle formed by an axis of the digital note and a line connecting a feature of the first colored region and a feature of the second colored region;

rotating the digital note based on the determined angle;

receiving, by the one or more processors, user input indicating a modification to one or more visual characteristics of the digital note;

editing, by the one or more processors and in response to the user input, the one or more visual characteristics of the digital note; and

outputting, by the one or more processors and for display, a modified version of the digital note that includes the one or more visual characteristics.

2. The method of claim 1, wherein:

receiving the user input comprises receiving user input that indicates additional content for the digital note,

editing the one or more visual characteristics of the digital note comprises creating a layer object that includes the additional content, and

outputting the modified updated version of the digital note comprises outputting the modified version of the digital note such that the layer object that includes the additional content is displayed on top of content corresponding to the respective recognizable content of the physical note to which the digital note corresponds.

3. The method of claim 2, wherein the additional content is first additional content, the layer object is a first layer object of a plurality of layer objects, and the modified version of the digital note is a first modified version of the digital note, the method further comprising:

responsive to receiving user input indicating second additional content;

creating a second layer of the plurality of layer objects that includes the second additional content; and

outputting, for display, a second modified version of the digital note such that the second layer object that includes the second additional content is displayed on top of the first layer object that includes the first additional content.

4. The method of claim 3, further comprising:

responsive to receiving user input to remove the second layer object from the updated version of the digital note, outputting, for display, a third modified version of the digital note such that the second additional content included in the second layer object is not displayed.

5. The method of claim 2, wherein receiving the user input that indicates the additional content comprises receiving one or more of:

a sequence of typed characters, and

a stroke gesture.

6. The method of claim 2, wherein:

receiving the user input further comprises receiving user input that indicates an updated background color,

editing the one or more visual characteristics of the digital note further comprises setting a background color of the digital note to the updated background color.

7. The method of claim 1, wherein:

receiving the user input comprises receiving user input that indicates an updated background color,

editing the one or more visual characteristics of the digital note comprises setting a background color of the digital note to the updated background color.

8. The method of claim 7, further comprising:

identifying, for the digital note, a foreground region that includes content corresponding to recognizable content of the physical note to which the digital note corresponds, wherein at least a portion of the content of the foreground region is associated a foreground color; and

modifying the foreground color.

9. The method of claim 8, wherein:

receiving the user input comprises receiving user input that indicates an updated foreground color, and

modifying the foreground color comprises setting the foreground color to the updated foreground color in response to receiving user input that indicates the updated foreground color.

10. The method of claim 8, wherein modifying the foreground color comprises automatically modifying the foreground color as a function of the updated background color.

11. The method of claim 10, wherein prior to modifying the foreground color, a plurality of pixels included in the foreground region have current foreground pixel colors, and wherein modifying the foreground color comprises modifying colors of the plurality of pixels included in the foreground region by at least:

selecting, by the one or more processors and for each pixel of the plurality of pixels included in the foreground region, an updated foreground pixel color such that a contrast ratio between the updated foreground pixel color and the updated background color is greater than a contrast ratio between the current foreground pixel color and the updated background color, and setting each respective pixel of the plurality of pixels to the selected respective updated foreground pixel color.

12. The method of claim 8, further comprising:

identifying, subsequent to identifying the foreground region of the digital note, a background region of the digital note by at least identifying pixels of the digital note not included in the foreground region as the background region.

13. The method of claim 8, further comprising:

identifying a background region of the digital note by at least identifying pixels of the digital note not included in the foreground region as the background region, wherein identifying the foreground region is performed subsequent to identifying the background region, and wherein identifying the foreground region for the particular digital note comprises;

identifying pixels of the particular digital note not included in the background region as the foreground region.

14. The method of claim 1, wherein receiving the image data comprises:

receiving, by the one or more processors and from a camera of the device, the image data.

15. The method of claim 1, further comprising:

enhancing, by the one or more processors, the digital note of the plurality of digital notes by at least;

identifying, for the digital note, a foreground region that includes content corresponding to the respective recognizable content of the physical note to which the digital note corresponds;

identifying a background region of the digital note; and

setting pixel values from the background region of the particular digital note to a uniform background pixel value.

16. A computing device comprising:

a memory configured to store digital notes; and

one or more processors configured to;

receive image data comprising a visual representation of a scene that includes a plurality of physical notes, each of the physical notes comprising a separate physical object having recognizable content thereon;

extract, from the image data, the recognizable content of a particular one of the plurality of physical notes;

create a digital note representative of the particular one of the plurality of physical notes, wherein the particular one of the plurality of physical notes includes a first colored region and a second colored region;

associate the extracted recognizable content with the digital note representative of the particular one of the plurality of physical notes;

identify, for the digital note, the first colored region and a second colored region;

determine an angle formed by an axis of the digital note and a line connecting a feature of the first colored region and a feature of the second colored region;

rotate the digital note based on the determined angle;

receive user input indicating a modification to one or more visual characteristics of the digital note;

edit, in response to the user input, the one or more visual characteristics of the digital note; and

output, for display, a modified version of the digital note that includes the one or more visual characteristics.

17. The computing device of claim 16, wherein the one or more processors are configured to:

receive the user input by at least receiving user input that indicates additional content for the digital note,

edit the one or more visual characteristics of the digital note by at least creating a layer object that includes the additional content, and

output the modified updated version of the digital note by at least outputting the modified version of the digital note such that the layer object that includes the additional content is displayed on top of content corresponding to the respective recognizable content of the physical note to which the digital note corresponds.

18. The computing device of claim 17, wherein:

the additional content is first additional content,

the layer object is a first layer object of a plurality of layer objects,

the modified version of the digital note is a first modified version of the digital note, and responsive to receiving user input indicating second additional content, the one or more processors are configured to;

create a second layer of the plurality of layer objects that includes the second additional content; and

output, for display, a second modified version of the digital note such that the second layer object that includes the second additional content is displayed on top of the first layer object that includes the first additional content.

19. The computing device of claim 17, wherein the one or more processors are configured to receive the user input that indicates the additional content by at least receiving one or more of:

a sequence of typed characters, and

a stroke gesture.

20. The computing device of claim 17, wherein the one or more processors are configured to:

receive the user input by at least receiving user input that indicates an updated background color,

edit the one or more visual characteristics of the digital note by at least setting a background color of the digital note to the updated background color.

21. The computing device of claim 16, wherein the one or more processors are configured to:

receive the user input by at least receiving user input that indicates an updated background color,

edit the one or more visual characteristics of the digital note by at least setting a background color of the digital note to the updated background color.

22. The computing device of claim 21, wherein the one or more processors are configured to:

identify, for the digital note, a foreground region that includes content corresponding to recognizable content of the physical note to which the digital note corresponds, wherein at least a portion of the content of the foreground region is associated a foreground color; and

modify the foreground color.

23. The computing device of claim 22, wherein the one or more processors are configured to:

receive the user input by at least receiving user input that indicates an updated foreground color, and

modify the foreground color by at least setting the foreground color to the updated foreground color in response to receiving user input that indicates the updated foreground color.

24. The computing device of claim 22, wherein the one or more processors are configured to modify the foreground color by at least automatically modifying the foreground color as a function of the updated background color.

25. The computing device of claim 24, wherein prior to modifying the foreground color, a plurality of pixels included in the foreground region have current foreground pixel colors, and wherein the one or more processors are configured to modify the foreground color by at least:

selecting, for each pixel of the plurality of pixels included in the foreground region, an updated foreground pixel color such that a contrast ratio between the updated foreground pixel color and the updated background color is greater than a contrast ratio between the current foreground pixel color and the updated background color, and

setting each respective pixel of the plurality of pixels to the selected respective updated foreground pixel color.

26. The computing device of claim 16, wherein the one or more processors are configured to receive the image data by at least:

receiving, from a camera of the device, the image data.

27. The computing device of claim 16, wherein the one or more processors are configured to:

enhance the digital note of the plurality of digital notes by at least;

identifying, for the digital note, a foreground region that includes content corresponding to the respective recognizable content of the physical note to which the digital note corresponds;

identifying a background region of the digital note; and

setting pixel values from the background region of the particular digital note to a uniform background pixel value.

28. A computing system comprising:

means for receiving image data comprising a visual representation of a scene that includes a plurality of physical notes, each of the physical notes comprising a separate physical object having recognizable content thereon;

means for extracting, from the image data, the recognizable content of a particular one of the plurality of physical notes;

means for creating a digital note representative of the particular one of the plurality of physical notes, wherein the particular one of the plurality of physical notes includes a first colored region and a second colored region;

means for associating the extracted recognizable content with the digital note representative of the particular one of the plurality of physical notes;

means for identifying, for the digital note, the first colored region and a second colored region;

means for determining an angle formed by an axis of the digital note and a line connecting a feature of the first colored region and a feature of the second colored region;

means for rotating the digital note based on the determined angle;

means for receiving user input indicating a modification to one or more visual characteristics of the digital note;

means for editing, in response to the user input, the one or more visual characteristics of the digital note; and

means for outputting, for display, a modified version of the digital note that includes the one or more visual characteristics.

29. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors of a device to:

receive image data comprising a visual representation of a scene that includes a plurality of physical notes, each of the physical notes comprising a separate physical object having recognizable content thereon;

extract, from the image data, the recognizable content of a particular one of the plurality of physical notes;

create a digital note representative of the particular one of the plurality of physical notes, wherein the particular one of the plurality of physical notes includes a first colored region and a second colored region;

associate the extracted recognizable content with the digital note representative of the particular one of the plurality of physical notes;

identify, for the digital note, the first colored region and a second colored region;

determine an angle formed by an axis of the digital note and a line connecting a feature of the first colored region and a feature of the second colored region;

rotate the digital note based on the determined angle;

receive user input indicating a modification to one or more visual characteristics of the digital note;

edit, in response to the user input, the one or more visual characteristics of the digital note; and

output, for display, a modified version of the digital note that includes the one or more visual characteristics.