A group of CU Boulder artists and technologists, many of whom share connections with the ATLAS Institute, contributed to the Museum of Boulder’s newest exhibit, “Convivial Machines,” which opened Saturday, Oct. 30.

The gallery showcase is the first museum installation for Boulder Experiments in Art and Technology (B.E.A.T), says the exhibit's curator, Jiffer Harriman (ATLS PhD '16), who started B.E.A.T. in 2019 to “foster creativity and collaboration at the intersection of art and technology.”  Museum visitors can expect to see a variety of interactive systems, including musical installations and video art, as well kinetic sculptures, digital games, bioluminescent algae and more, Harriman says.

“The exhibit highlights a local community of artists and creative technologists looking for new ways to see the world,” says Harriman. “It explores the differences and balance between what people do well—expressiveness and creativity—and what machines are good at–repetition and connection.”

One of the emerging themes of the show is the tension between the benefits that technology offers and the negative impacts, Harriman says. Some installations imagine new possibilities, while others point toward technology’s negative effects, including an installation of "useless boxes" that contain a single switch, which when turned on, is immediately switched off by a robotic finger, he adds.

The ATLAS Connection
Inspired by bioluminescent algae and the idea of engaging with something “live,” Harriman has blended research led by Mirela Alistar, assistant professor of computer science with the ATLAS Institute, into the installation. Alistar, also the director of the Living Matter Lab, and Netta Ofer, an ATLAS master’s student, have created an exhibit, “Bioluminescent Touch,” where visitors can interact in the darkness with dinoflagellates, algae that produce light when exposed to oxygen through physical stimulation.

“The bioluminescent response to human touch creates unexpected connection and empathy with the microorganism, drawing us into the wonder of communicating and interacting with nonhuman beings,” Ofer says.

Slaton Spangler, now a software engineer with LASP who, as an undergraduate majoring in computer science was a very active member of the BTU Lab, made the interactive installation, “Amorphous Breeze.” The piece displays a network of undulating “blobs” that fade in and out following a set of underlying rules. When a viewer presses a glowing green button, a fan in the real world appears to blow away the blobs, replacing them with new blobs that follow new rules. The patterns that emerge from the waves of blobs tend to be mesmerizing in the same fashion as a lava lamp, Spangler says.

Spangler also constructed the installation, “Clear the Void,” which uses an XBox Kinect to create an interactive space of digital images provided by Matt Corones, an ATLAS master's student (Creative Industries) in the ACME Lab, Dajira Medić, Jeanne Kipke, Andy DiLallo and Shannon Derthick. 

In a third installation, Spangler created a hardware platform to loop an installation of video art from the B.E.A.T. community that includes a range of experimental techniques and themes.

Priyanka Makin, an ATLAS master's student, fabricated the “Nature Mobile,” a kinetic sculpture of nature shapes that explores the future relationship between technology, engineering, art and nature. Makin used a laser cutter to cut the pieces and built and programmed the motor. With the help of natural sunlight in the lobby, the sculpture scatters colors throughout the entry to the museum.

Perry Owens, an undergraduate CTD student and member of the ACME Lab, created the sculpture, "The Future of Problem Solving," which expresses the duality between man and machine as they are tasked with solving a puzzle. To succeed, man and machine must achieve harmony through trust and coordination. Owens made the "human" from resin casts of his own hands. Using a 3D modeling software, he designed the mechanical arms and 3D printed them with fingers that can be posed. He also 3D printed the Rubik's cube, which can be rotated, and added a wooden exterior, while the LED-lined portals and mirrors were laser-cut.

"I imagine the future of our world is one in which there is collaboration between technology and humanity, instead of competition and war," Owens says. "This piece was fabricated using human skills combined with technology, akin to its message of unity between man and machine."

A collaborative effort of seven created the musical arcade, “Galaga's Ghost,” a multi-channel musical instrument that up to six people can play with buttons, joysticks and air guitar moves. The interactive musical composition was created by Sean Winters, a CMCI lecturer in the Department of Critical Media Practices who performs regularly in the ATLAS Black Box. Winters is also on the B2 advisory committee. Torin Hopkins, a PhD student in the ACME Lab and the instructor for the SOUND class, helped design the interface layout and assisted with the final fabrication of the piece. 

Off

Long before the pandemic sent people scrambling into isolation, musicians have longed to jam virtually with others across the globe. Now researchers from CU Boulder’s ATLAS Institute’sACME Lab and Ericsson Research are developing ways for musicians to play together remotely through the AR Drum Circle project.

The difficulty with online jamming has always been latency, the tiny delay that occurs when data is transmitted from one point to the next, says Torin Hopkins, an ATLAS PhD student who leads the ATLAS team. Video conferencing participants don’t detect the delay because they generally take turns when speaking, but any lag greater than 20 milliseconds makes synchronous singing or performing unworkable, he says.

“There’s no room for delay in musical collaborations,” says Hopkins, adding that the virtual choir videos popular during the pandemic were mixed in post production.  “Yet real-time music-making with zero lag and a consistent video stream currently doesn’t exist.”

In the AR Drum Circle project, ATLAS researchers and Ericsson project collaborators are exploring ways in which remote drumming experiences can be made more enjoyable despite the latency, says Colin Soguero, the project’s app developer and an undergraduate student studying Creative Technology & Design.

“Latency is one of the biggest issues with remote collaboration, and it can be very frustrating for musicians who rely so heavily on precise coordination,” he says. 

Jamming with Avatars
Some of AR Drum Circle’s research focuses on avatars, computer-generated figures that in this case replicate the actions of real drummers participating remotely in drum circles. The avatars appear in another musician’s surroundings using augmented reality (AR), a technology that superimposes a computer-generated image on a user's view of the real world.

Using the AR Drum Circle application, Musician A prints a QR code and places it to position Musician B’s avatar in the augmented reality view. Musician A’s Android cell phone runs the application, and displays B’s avatar where the coded picture was placed. Musician B does the same. When either musician strikes a drum pad connected to their computers, the computers send that information through the internet, the corresponding avatar drummer then strikes its drum, and a drum beat is heard in both locations. The technology employs  a Musical Instrument Digital Interface (MIDI) controller, which, when one drummer strikes their drum pad, sends information to their computer, which then sends the data over the internet to the other drummers’ locations.

While live video of the drumming partners might be best, using avatars mediates the perception of the latency and—potentially—provides visual and audio information for a more satisfying musical exchange, Hopkins says. It takes just small bits of data to trigger an avatar's hand to move, whereas rendering videos requires large amounts of data to transmit every pixel of the moving images.

Adding to this, the core idea of this project is not merely collaboration, but how to minimize or leverage the effects of the inevitable latency and jitter (the deviation from true periodicity of a presumably periodic signal) in order to make collaborations that are highly sensitive to timing more successful or fun, says Mark Gross, professor of computer science, ATLAS director and a member of the project’s advisory team.

“Latency cannot be avoided, but its effects can be mitigated by being clever in portraying avatars and by anticipating future actions,” Gross says.

Sending the drum pad information over the internet to a receiving computer is “incredibly complex,” Hopkins adds. The data travels a long journey and encounters many checkpoints along the way, and small packets of information travel much faster across the network than video with sound.  Because the avatar's motion needs to be realistic, complex information is kept on the receiving device and only “start animation” messages are sent over the internet. 

Enjoying the experience
Just watching and hearing an avatar strike a drum doesn’t provide adequate information for remote drummers to synchronize, says Ellen Do, professor of computer science with ATLAS, who also participates in several drum circles. Drummers often use gestures, such as head motions and eye contact, to indicate part changes, turn taking and solos, she says. They also use striking force to control volume and hand position to control the timbre; they need to recognize the patterns of the rhythms (e.g., focusing on the down beats, space in-between the beats, the speed, embellishments, harmony, etc.) to play with others, she says.

A large part of the team’s research focuses on determining which of those gestures and expressive features might help remote drummers feel immersed in the collaborative musical experience and experience the enjoyment of feeling connected with each other, she says.

Hopkins, who plays guitar, piano, ukulele, bass, and drums, as well as sings, has missed jamming with other musicians during the pandemic.

“Meeting new people, sharing new ideas and the audiences– those are the things that I really, really miss,” Hopkins says. “Part of the project is figuring out how to incorporate that. Every time I hit the drum, is that enough to make you feel like I’m listening to you? That you feel connected, and that we feel in-sync with each other?”

Connecting in an isolated world

Soguero adds that the researchers are also exploring looping, which allows a player to record a drum beat and play it back later, as well as pseudo-haptics, visual effects created in a virtual environment that trick the brain to believe that it’s receiving information about touch and feel.

Regardless of the pandemic, connecting with people who are geographically distant allows for rich, connected, experiences with others who have a variety of talents, come from different cultures and have different perspectives, Hopkins says. 

Lessons learned from the AR Drum Circle study about human-human communication, or human-agent communication (with an avatar, agent or robot) could also possibly inform other computer-supported collaborative work scenarios, such as remotely collaborating in medical procedures or auto-repairs, Do says.  

“Our research raises the question, ‘Why collaborative musical experiences?’ ” says Hopkins. “Are we doing it to enjoy the company of others or because we enjoy music? How much can you strip away from either experience before you realize they are so intimately connected that designing for collaboration or musical expression alone feels disingenuous?

“Therefore, when designing the AR Drum Circle application, we focus on player-centered design strategies. Maximizing play, given the constraints of the mediating technology (augmented reality) and activity (drum circles), enables the players to feel a sense of contribution in a musical collective, giving us a much needed sense of connection in an isolated world.” 

AR Drum Circle's ATLAS Team: Torin Hopkins, ATLAS PhD student, is the project manager; Darren Sholes, ATLAS PhD student, is the technical lead/network engineer; Peter Gyory, ATLAS PhD student, was the former technical lead; Hooman Hedayati, PhD student in computer science, is the project's network engineer and advisor for human-robot (avatar) interaction; and Colin Soguero, an undergraduate student studying Creative Technology & Design, is the app developer. The advisory team consists of Mark D. Gross, ATLAS director and professor of computer science; Ellen Do, ATLAS and computer science professor; Amy Banic, associate professor of computer science at the University of Wyoming and visiting ATLAS professor; and Dan Szafir, ATLAS and computer science assistant professor.

Ericsson Research Project Collaborators: Amir Gomroki, head for 5G, North America; Héctor Caltenco, senior researcher; Per-Erik Brodin, research engineer;  Ali El Essaili, senior research engineer; Chris Phillips, master researcher; Alvin Jude Hari Haran, senior researcher; Per Karlsson, director, media technology research at Ericsson and head of Ericsson Research in Silicon Valley; Gunilla Berndtsson, senior researcher at Ericsson Research, Media Technologies.

[video:https://www.youtube.com/watch?v=e7PhJRmLt1w&feature=youtu.be]

Off

Jam Station promotes musical improvisation, also known as "jamming," among those who have difficulties learning to play instruments, said Pete Pascente, CTD master's student, who along with  researchers Torin Hopkins, ATLAS PhD student, Wayne Seltzer, ATLAS technologist-in-residence and Kellie Masterson, developed the project. Jamming is considered difficult even for experienced musicians, he said. 

"We wanted to allow people who know nothing about music to experience the joy of a jam - to collaborate and feel like they're part of a something a little bigger than themselves,"  Pascente said.  

The station is a 7-by-4 feet rectangular PVC cuboid, whereby four instruments crafted from everyday materials are situated so that each jam member faces a vertically oriented neopixel display.  Embedded sensors in Jam Station's instruments trigger complex and colorful light displays as collaboration between musicians improves, and these visual cues help novice musicians improvise music together.

"The instruments are designed to not resemble what we normally think of as instruments, so they're more approachable," Pascente said.

Initial survey results showed overall positive reactions to the experience, Seltzer said, with all six self-described "nonmusicians" rating their experience as "good."

Off