Environmental protection message conveyed through AR filter

Algorithm recognizing you

How does Instagram recognize our faces and put filters on them?

How does Instagram recognize our faces, and put a filter over them?

Mechanism of the AR filter

In Instagram, human faces are recognized mainly through algorithms called DeepMask and SharpMask. These two are convolutional neural networks, which are a type of deep neural network, and are algorithms trained to specialize in image recognition.

If you have ever used an Instagram AR filter, it is likely the work of DeepMask and SharpMask algorithms. These algorithms are a convolutional neural network (CNN), which is a type of deep neural network. In the case of DeepMask and SharpMask, they are specialized in the area of image recognition. The process goes as follows.

First of all, the photo is pixelated in the following format and recognized by the algorithm.

First, an image is inputted to the algorithm after it has been pixelated as the image below.

However, in the case of Instagram, in reality, when the user turns on the camera, it receives more information, such as various shades and shades, as well as a screen that moves in real time. And this information is first input into the DeepMask algorithm.

But in the case of Instagram, the amount of information and its complexity is far greater, due to the constant camera movement and the various colors and shadows. All the information of the image is inputted initially in the DeepMask algorithm. 

The DeepMask algorithm classifies pixels with similar shades and colors according to previously learned information and groups them into a patch. In addition, the DeepMask algorithm divides each object by classifying whether there is another patch within each patch and, if so, how the correlation between the newly classified patch and the existing patch is, and finds out the correlation between each object, return this

The DeepMask algorithm is able to patch together pixels with similar colors and shadows, through the information it trained with. Furthermore, the algorithm is able to determine segments of the patches, meaning that it is able to determine whether if there is a distinguishable patch within a patch, and further identify the mutual relationship between the nested patches, and return the results.

The information returned by DeepMask is now fed into the SharpMask algorithm for further refinement. If the DeepMask algorithm divides and extracts information from a large picture called a picture and then makes it into the most basic form, SharpMask now recombines the information to reconstruct the picture. SharpMask also distinguishes between the provided information and the learned information about what object is in the picture and whether the object the program wants is correct.

Information returned from DeepMask is further polished through the SharpMask algorithm. If DeepMask divided and extracted specific informations from the big picture (quite literally) and organized them into small segments of information, SharpMask then reorganizes the information by connecting them together to recreate the image, in an understandable form by a machine. The SharpMask algorithm is able to identify what the objects from the image actually are through learned variables and weights, and further determine whether if the object is the target object of the program as a whole. 

Social Effect of AR Filters

If the face is recognized through the above process, a filter is applied appropriately on the person's face, and the picture desired by the user is displayed on the screen. The result of this AR filter is as follows.

If the algorithmic process was able to successfully recognize a human face, then the program will put a filter on to the identified face, and print it on the screen. The results of this process can be seen below.

The reason why AR filters have recently been in the limelight is that they have excellent virality that can spread through these individuals. Take Instagram as an example. If this AR filter gets attention through the first person, others will follow suit and use the filter, which will go through the eyes of regular Instagram users and become famous.

The reason why AR filters are catching people's eyes are due to their virality though networking effect. This can be seen frequently on Instagram. If a person uses a certain AR filter, and manages to get it on to other people's feeds, and further manages to get other people to join in on using the filter, the filter will spread out to other people's Instagram feeds, becoming viral. 

As a result of actually using and sharing the filter, various questions about the filter came in. There were also questions about the design aspects of the filter itself, such as the meaning contained in the filter and the reason for the specific design, but mainly asked about the meaning contained in the filter. And when he knew that the filter had good intentions, his acquaintances also joined in and tried the filter. As such, if the spotlighted AR filter has good intentions, it will become a positive virality that has a good social impact.

After using the filter, I was able to receive many questions about the filter. There were a few questions regarding the design of the filter itself, but the questions were mostly about the meaning within the filters. When the questioner knew about the positive meaning behind the filters, they too were more than welcomed to join in and spread the filter. Just like this, if the filters have a positive intentions and meanings, then it may become a positive virality, spreading positive influence over our society. 

The filter used as an example above has the potential to become the aforementioned positive virality. This filter was designed based on the public campaign “LOVE EARTH, SWITCH OFF (Darkness that Saves the Earth)” conducted by CUZ.

LOVE EARTH, SWITCH OFF (The Darkness That Saves the Earth)

“LOVE EARTH, SWITCH OFF (Darkness that Saves the Earth)” is a multi-participatory interactive media content project in progress at Hanam Starfield. The core planning intention of this content is to deliver the message 'my small actions of turning off the switch gather to save the earth'. Visitors turn off the light switch in the Media Tower space through their smartphones, and when the lights in 10 spaces in the Media Tower are turned off, nature is reproduced and beautiful and fantastic images that revive the earth are played. At the end, the name of the person who contributed to saving the earth and the face of the person with the AR filter applied appear in the media tower video ending credits. Anamorphic technology, multi-participation interaction technology, and AR technologies make the above project more diverse.

The project “LOVE EARTH, SWITCH OFF. The darkness that saves the earth” is a multi-participation interaction media content project, planned to happen in Hanam Starfield. The core intention of the project is to deliver the message, where “saving the earth can start from a small action of turning off a switch”. The audience can turn off a light switch on the media tower, and when all 10 switches are turned off, a wondrous video of nature regenerating and earth becoming alive is played, and at the end, the participant's name and picture, with the AR filter applied, is shown on the credits. The anamorphic technology, multi-participation interaction technology, and AR technology makes the project overall more vivid. 

When we look at the exponential development of technology, it seems that technology is ahead of us, but in the end, we decide the direction. Through projects such as “LOVE EARTH, SWITCH OFF (Darkness that Saves the Earth)”, we hope that various social problems including the environment can be solved through technological development. 

With the exponential advancements in technology, we often think that technology is overwhelming us. However, it is us who sets the direction of the advancements. With projects such as “LOVE EARTH, SWITCH OFF (Darkness that Saves the Earth)”, we hope to see how advancements of technology can contribute in resolving various social problems that humanity faces.