Listen to the story:
In the early 2000s, coming out as transgender wasn’t an option in Kolkata, India.
Roni Sengupta didn’t have the language for why she secretly dressed up in her mother’s clothes when she was 10 years old. There wasn’t a word for why she felt feminine or why it made her happy. While dating as a teenager, Sengupta told her partners she knew she was a woman, but never her parents. When she met her wife in college, she still wasn’t ready to medically commit to the transition.
“I hoped initially it would go away, like if I pretended to be masculine enough it would eventually go away,” Sengupta says. “But I always knew I was a woman.”
Sengupta felt ready to start the transition in Spring 2023. To her surprise, she began developing feminine features within just a few months of treatment. By fall, the UNC-Chapel Hill computer scientist came out in full. Her colleagues, friends, and wife have been incredibly supportive.
At UNC Chapel Hill, Sengupta works in computer vision, a field of artificial intelligence (AI) that trains computers to see like a human eye. More specifically, it develops AI algorithms that learn from visualizations like graphics, videos, and images. And it’s being used by a variety of industries, from health care to TV and film.
One focus of Sengupta’s lab is to create a 3D model that tracks and predicts changes in someone’s face as they age. In the next five years, she hopes it will help people who are transitioning from one gender to another.
“There is a sense of concern and anxiety for a lot of transgender people,” Sengupta says. “We just don’t know what is going to happen.”
Sengupta hopes to ease this anxiety for others — and that drives her research at Carolina.
The heart of her work
Like many people born, raised, and educated in India, Sengupta found her preferred career after getting an engineering degree.
“In India, we have this fun saying that everyone becomes an engineer first, and then they do what they want to do,” she says.
During the early 2000s, she studied electronics and telecommunication engineering at Jadavpur University in Kolkata. After applying to graduate programs globally, she landed a spot at the University of Maryland to complete her PhD in electrical engineering.
While hardware development was interesting, it wasn’t where she wanted to be.
“I grew up watching a lot of movies, and I used to love animation,” Sengupta shares. “I always had a knack for these things, but I didn’t have the creative skills.”
During 2015, as a research intern at the Weizmann Institute for Science in Israel, she learned 3D reconstruction techniques. Then, she interned at Snapchat to learn how to digitally reconstruct faces, and at Nvidia, where mentors taught her cutting-edge computer graphics techniques and, most importantly, how to utilize AI.
Sengupta graduated from her PhD program in 2019. While most of her peers moved into private sector jobs, she stayed in academia and pursued a postdoctoral research position at the University of Washington, where she was given creative freedom to explore projects and work with mentors on other graphic technology.
“I want to tell a story,” Sengupta says. “I have always felt like images, videos, and other visual mediums were the best way to do that.”
AI for everyone
One of Sengupta’s current projects tackles facial modeling. She and her lab aim to create a high-quality model that the AI on someone’s phone or computer could efficiently read to design a realistic face or make realistic changes to someone’s face.
This could be revolutionary for independent filmmakers and other creatives who lack the funds to produce realistic faces when they can’t use actors. Sengupta also sees a place for facial modeling in virtual reality.
“Say I’m wearing a headset, and my parents in India are using a mobile phone to make the call,” she describes. “If they’re using a video camera, I need this kind of technology to experience them realistically in 3D.”
The AI component in Sengupta’s current work relies on artificial neural networks. These are made up of individual algorithms, written by researchers like Sengupta and her students, that work together to produce 3D models or visuals.
Much like a human network of neurons, they are separate but collaborating.
Sengupta’s lab is trying to create two models derived from a neural network that produces realistic human faces. One model uses the cloud of images provided to AI by the internet. The other is trickier.
This model would run without using the cloud. A small, separate neural model just for one person that no one else can access. Someone could use it to change their facial expression, create a realistic avatar of themself, or privately track physical changes and create personalized predictions.
“Most trans women and men capture photos of themselves every day just to notice the smallest changes,” Sengupta shares. “This data exists, and to build accurate models from this would help physicians see how many changes have happened because of the hormone replacement therapy dosage. And they would have a better understanding of the entire timeline.”
While health agencies have developed guidelines for safe medical transition practices, the physical outcomes of treatment remain unknown until a few years after starting the prescription. This may include changes in face shape, hair, and skin.
Using selfies, Sengupta’s algorithm will track the smallest daily changes in someone’s physical features and create a visual prediction of their future appearance. While she’s comfortable sharing her own story with others, many aren’t ready to hand over their private data to researchers.
Once the technology and trans community become more secure, Sengupta wants to give people undergoing hormone therapy — a process that can take anywhere from two to five years — a personalized image of what they might look like at the end of their treatments.
While Sengupta has specific goals for how her facial remodeling technology is used, its applications go beyond that. It has the potential to help people recovering from injuries see what they’ll look like after healing or facial reconstruction surgery. Additionally, it can create realistic aging effects to provide emotional support for families with missing or deceased relatives.
Sengupta also uses AI to produce background matting and relighting, creating green screens without a green screen to make realistic video backgrounds. The AI turns the user’s background into a panorama, reading the colors behind the subject to light their face. For a forest background, the AI will cast a greenish light on the face to make it look like there’s foliage in the foreground too.
“It can virtually relight you, so if the lighting on your face is bad then you can use this to make the lighting look good in real time for your Zoom calls,” she says.
Ethical unknowns
This kind of advanced computer vision, specifically facial modeling, is expensive, high-powered, and already available to corporations like Disney or Netflix. Sengupta wants to make the technology publicly accessible, providing it to anyone with a personal device to encourage creativity.
“We are in a state where creativity has been hugely democratized, but the tools that allow average people to create haven’t been,” Sengupta says.
Her intentions are good, but this is a tricky field to work in right now. And Sengupta doesn’t have all the answers.
How can anyone know for certain when AI creation has only just begun to unfold?
Sengupta recognizes that accessible AI can be dangerous but points out that it’s not the first technological invention or innovation the public has been skeptical of. Telephones, airplanes, nuclear power, and the internet all came with their own doubts, debates, and eventually policies to counter issues as they arose.
“Tools that we are developing for people to improve their communication or their ability to create stories more independently are also tools that are going to be used for bad purposes,” she says.
For Sengupta, it’s about the good this technology can offer. It may help a creative gain some career traction, and eventually, it could soothe the anxiety of a young trans person during hormone therapy.