Mobile device image capture and image modification including filters, superimposing and geofenced comments in augmented reality

• US 10,142,496 B1
• Filed: 12/27/2017
• Issued: 11/27/2018
• Est. Priority Date: 01/26/2013
• Status: Active Grant

• Current Assignee: Airspace Reality, Co.
• Original Assignee: IP Holdings Incorporated (IP Holdings LLC)
• Inventors: Rao, Sunil K., Rao, Rekha K., Rao, Raman K.
• Primary Examiner(s): Ayotunde, Ayodeji
• Application Number:
  US15/854,754
• Time in Patent Office: 335 Days
• Field of Search: 455566
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/0219 Special purpose keyboards
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04L 45/306 Route determination based on the nature of the carried application
• CPC Class Codes: H04N 1/00095 Systems or arrangements for the transmission of the picture signal
• CPC Class Codes: H04N 1/00214 details of transmission
• CPC Class Codes: H04N 1/00244 : with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 1/00307 : with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 2007/145 : Handheld terminals
• CPC Class Codes: H04N 21/2353 : specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
• CPC Class Codes: H04N 21/41407 : embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 7/14 : Systems for two-way working H04N7/12, H04N7/173 take precedence
• CPC Class Codes: H04N 7/147 : Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 : Conference systems
• CPC Class Codes: H04W 40/02 : Communication route or path selection, e.g. power-based or shortest path routing

Abstract

A camera may capture real world content which may be displayed in a mobile device or glasses. This content may be modified in real time. The image capture of objects and transformation of those objects are displayed. The images obtained may be processed and changes may be enacted such that they are displayed virtually different than they are physically. Various micro-expressions and emotions may be detected by the facial expressions of a person. Digital notes may be saved and associated with objects. These notes and environments may be saved by a user and viewable by anyone in the geofenced area or all users. An individual may capture an image of an object using a mobile device/intelligent electronic glasses/headset, tap the object on the display screen, and write or speak notes regarding the object. A keyboard, keypad, image, icons, or other items may be virtually placed on to a surface.

Claims

1. A non-transitory computer readable storage medium that stores a set of instructions that are executable by at least one processor of a mobile device to cause the mobile device to perform a method comprising:

obtaining one or more images captured by an application utilizing an image acquisition device;

identifying one or more sections of the one or more images showing an object and a background;

creating at least one modified view based on changes to the one or more sections of the one or more images; and

displaying the at least one modified view on the application.

2. The non-transitory computer readable storage medium of claim 1, wherein the at least one modified view comprises of at least one or more modified images.

3. The non-transitory computer readable storage medium of claim 2, wherein identifying one or more sections of the one or more images showing the object and the background includes:

providing for transmission to a remote server the one or more images including the object and the background; and

acquiring the identification of the one or more sections of the one or more images from the remote server.

4. The non-transitory computer-readable storage medium of claim 1, wherein creating at least one modified view based on changes to one or more sections of the one or more images includes:

providing a toolbar in the application to enable annotations and/or substitutions to the one or more sections of the one or more images.

5. The non-transitory computer-readable storage medium of claim 1, wherein creating at least one modified view based on changes to one or more sections of the one or more images includes:

substituting the one or more sections of the one or more images with one or more related images accessible by the application.

6. The non-transitory computer-readable storage medium of claim 5, wherein substituting one or more sections of the one or more images with the one or more related images includes:

modifying a background of the one or more images.

7. The non-transitory computer-readable storage medium of claim 1, wherein creating at least one modified view based on changes to one or more sections of the one or more images includes:

overlaying an image on top of the one or more images.

8. The non-transitory computer-readable storage medium of claim 1, wherein creating at least one modified view based on changes to one or more sections of the one or more images includes:

associating the at least one modified view to a geofenced area; and

providing the at least one modified view to be displayed on a second mobile device when a determination is made that the second mobile device is located in the geofenced area.

9. The non-transitory computer readable storage medium of claim 1, wherein creating at least one modified view based on changes to one or more sections of the one or more images includes:

adding a virtual object to the at least one modified view.

10. The non-transitory computer-readable storage medium of claim 9, wherein adding the virtual object to the at least one modified view includes:

enabling the virtual object to be placed onto a surface shown in the one or more images.

11. The non-transitory computer-readable storage medium of claim 1, wherein the set of instructions that are executable by the at least one processor of mobile device to cause the mobile device to further perform:

adding a plurality of virtual notes to be associated with the object shown in the one or more images.

12. The non-transitory computer-readable storage medium of claim 11, wherein adding the plurality of virtual notes to be associated with the object includes:

associating expiration conditions to the plurality of virtual notes.

13. A method performed by a mobile device having one or more processors, the method comprising:

obtaining one or more images of real world content captured by an application utilizing an image acquisition device;

identifying one or more sections of the one or more images showing an object and a background;

creating at least one modified view based on changes to the one or more sections of the one or more images; and

displaying the at least one modified view on the application.

14. The method of claim 13, wherein identifying one or more sections of the one or more images showing an object and a background includes:

recognizing a face using face recognition program.

15. The method of claim 13, wherein creating at least one modified view includes:

providing for transmission to a remote server the one or more images;

posting at least one image to a social network; and

obtaining from a remote database a related image.

16. The method of claim 15, wherein creating at least one modified view based on changes to one or more sections of the one or more images includes:

enabling digital comments to save as part of the at least one modified view.

17. The method of claim 16, wherein creating at least one modified view based on changes to one or more sections of the one or more images includes:

super imposing one or more new objects by substituting the one or more sections of the one or more images with one or more related images accessible by the application.

18. The method of claim 17, wherein substituting the one or more sections of the one or more images with the one or more related images includes:

modifying the background of the one or more images.

19. The method of claim 13, wherein creating at least one modified view based on changes to one or more sections of the one or more images showing an object includes:

providing for display an advertisement based at least in part on the object and/or location of the mobile device.

20. The method of claim 19, wherein creating at least one modified view based on changes to one or more sections of the one or more images includes:

associating the at least one modified view to a geofenced area; and

providing through a server the at least one modified view to be displayed on a second mobile device when a determination is made that the second mobile device is located in the geofenced area.

21. The method of claim 13, wherein creating at least one modified view based on changes to one or more sections of the one or more images includes:

adding a virtual object to the modified view.

22. The method of claim 21, wherein adding the virtual object to the at least one modified view includes:

enabling the virtual object to be placed onto a surface shown in the one or more images.

23. The method of claim 13, wherein creating at least one modified view based on changes to one or more sections of the one or more images includes:

adding a plurality of virtual notes to be associated with the object shown in the one or more images.

24. The method of claim 23, wherein adding a plurality of virtual notes to be associated with the object includes:

associating expiration conditions to the plurality of virtual notes.

25. A mobile device having an installed application, the mobile device comprising:

a memory storing a set of instructions; and

one or more processors configured to cause the mobile device to perform;

acquire one or more images captured by an image acquisition device;

identify one or more sections of the one or more images which show an object and a background;

create at least one modified view based on changes to the one or more sections of the one or more images; and

display the at least one modified view.

26. The mobile device of claim 25, wherein creation of at least one modified view based on changes to one or more sections of the one or more images by at least one processor of mobile device causes the mobile device to further perform:

detect one or more facial expressions of a person;

add an avatar of a person; and

generate at least one virtual image that is displayed differently than physical appearance of the person.

27. The mobile device of claim 25, wherein creation of at least one modified view based on changes to one or more sections of the one or more images by at least one processor of mobile device causes the mobile device to further perform:

scan an environment to determine a map of a plurality of physical objects; and

display the one or more virtual objects relative to the environment.

28. The mobile device of claim 25, wherein creation of at least one modified view based on changes to one or more sections of the one or more images by at least one processor of mobile device causes the mobile device to further perform:

add a virtual object to the at least one modified view; and

substitute the object by superimposing the virtual object on the object.

29. The mobile device of claim 25, wherein the mobile device is a wearable glasses device, the wearable glasses device comprising:

one or more microphones;

one or more cameras;

a gyroscope;

a see thru lens;

a wireless transmit and receive unit;

a processor; and

a gesture control.

30. The mobile device of claim 25, wherein the mobile device is enabled to project at least one image on a retina.