Physics and emote design: Quantifying clarity in digital images

When analyzing artworks, understanding the visual clarity of compositions is crucial. Inspired by digital artists, Okinawa Institute of Science and Technology (OIST) researchers from the Mechanics and Materials Unit have created a metric to quantify clarity in digital images. As a result, scientists can accurately capture changes in structure during artistic processes and physical transformations.  

This new metric can improve analysis and decision-making across the scientific and creative domains, potentially transforming how we understand and evaluate the structure of images. It has been tested on digital artworks and physical systems. The research is published in the journal PNAS.  

Defining clarity in art with math 

At the heart of the methodology lies a straightforward approach, inspired by the practice of digital artists zooming out to evaluate their work. The researchers developed a high-school-level mathematical method to quantify "clarity" in digital images by measuring how clear visual elements stay when blurred. This metric bridges the gap between physics and art, providing a flexible analytical tool for the scientific analysis and artistic creation of digital images.  

The method involves blurring images by randomly swapping neighboring pixels and then comparing original images with their blurred versions. The researchers measure how much of the original structure remains intact, assigning higher scores to images with structural elements that remain recognizable even after blurring and zero to images with chaotically distributed pixels or solid colors.  

“In our study, we defined ‘clarity’ as the resistance to blurring or structural degradation. Artistically, this definition aligns with a common artistic practice of zooming out of a canvas to assess how clear an artwork appears. Mathematically, clarity can be expressed in terms of the contrast of colors and their spatial distribution,” Prof. Eliot Fried, head of the Mechanics and Material Unit, explained. 

The metric preserves the color diversity of an image and is effective even after image compression. It is versatile, useful for analyzing different images, detecting structural changes in physical systems, and aligns with color theory, which studies how colors interact and affect our perceptions. Essentially, it shows how much of the pattern within an image survives random changes. 

“Artists are always experimenting with their techniques and compositions. The idea came to me while drawing an emote. As an experimental physicist, I was inspired to create metrics that can measure artistic qualities like clarity, balance, and harmony. We hope these metrics will allow art researchers to experiment with different compositions and report their findings in an objective, quantitative way,” Dr. San To Chan, researcher and first author, said.  

In digital art, an "emote" typically describes small, expressive images or animations that convey emotions, especially within online communities and on streaming platforms.  

Video games, Vtubers, and emotes 

Guided by their metric, the researchers designed an emote for the video game Holo X Break. This practical use proved that their research can be useful even in commercial art projects. 

“I chose to portray the virtual YouTuber (VTuber) Sakura Miko because I am a ‘35P’; a member of her fanbase. Together with several other 35Ps, we have previously animated her for a billboard display in New York Times Square to celebrate her 5th anniversary. Following this experience, I felt that featuring her in this research would be an exciting way to connect my enthusiasm as a fan with my professional experience as a scientist," Dr. Chan added. 

Illustration of clarity reduction: Effects of blurring and color contrast. The first row demonstrates the impact of blurring on clarity, with constant color contrast. The second row illustrates the effect of reduced color contrast on clarity, with fixed spatial color arrangement. 

Chan et al., 2024

Measuring image quality and stability. (A) Original artwork: Pixel art of Himemori Luna and her knight mascots (Hololive Production) on a raft; (B) Blurred version after 650,000 pixel flips – the artwork's structure degrades, but the raft shape remains visible; (C) Color Distance Map: A grayscale image showing structural stability – black areas are stable, white areas are not; (D) Degradation score: Shows how the artwork's structure changes with pixel flips, compared to a completely shuffled version; (Inset in D) Color distribution: A 3D scatter plot of the 50 most frequent colors in the artwork, showing their distribution in red-green-blue space; (E) Structural stability: Defines a metric (∆S) to measure structural stability, which is maximized when the number of pixel flips is about equal to the image size.  

Chan et al., 2024; Artwork by Ronin (@zeth_total)

As a member of the WorldWide 35P Network of fans, Dr. San To Chan helped animate the VTuber Sakura Miko for a billboard display in Times Square, New York, for the NYahello2023 project – a fan-led project commemorating Sakura Miko’s 5th anniversary. His research was conducted during the development phase of the video game Holo X Break, when the developer of the game invited him to contribute several emotes.

@MarkV_R

Outside of his research, Dr. Chan is a freelance artist who has contributed to renowned indie game development projects such as Holocure. This study draws strength from significant professional expertise by incorporating artworks from peer artists, providing a strong foundation for the analysis. 

Dr. Chan and Prof. Fried are exploring potential applications of their clarity metric beyond its initial scope. They believe their metric could revolutionize art analysis. Just as engines convert thermal energy to mechanical work, artists convert clarity in exchange for artistic qualities such as harmony, balance, and rhythm.  

Such a thermodynamic perspective on art could help researchers understand the rationales behind certain creative decisions made by artists, providing knowledge that will be useful to human artists to perfect their crafts further.