001: Song switcher + Visualizer

Like many recent popular experiments in TouchDesigner, this one began with Torin Blankensmith’s MediaPipe plug-in. What I sought to achieve with this program changed many times throughout its development, but ultimately my use of the plug-in stayed more or less the same (only adding extra functionalities as time went on).

As the header image shows, this is a visualizer on the surface, showing the live waveform of TouchDesigner’s sound output at a given time. The waveform is tethered to my hands—its functionality is such that when I pull them further apart, the speed increases (and the pitch gets higher), and when I bring them closer together it the speed decreases (pitch gets lower) within a range from 0 to 2. That pitch data is displayed on the bottom left of the display.

The second functionality here is switching between tracks. I loaded three into this example. My right (your left) hand breaking the 180º mark is the switch here, but in MediaPipe terms it is ‘h1’, or the hand that enters the frame first.

That’s the entire program, so let’s break down how it works.

MediaPipe Setup

If its your first time using MediaPipe, look here for Torin’s video on how to properly set it up. Generally, it relies on sending hand data into a Hold CHOP, and then into a Base that limits and then lags the input, so that all of your input data stays within the bounds of 0 and 1. The lag is there to prevent jittering, as happens in most fast-changing CHOP situations. This is all addressed in the linked video.

From there, I output two separate CHOPs. The relative position data I ran through this Base, but the general position data I did not. I don’t know if there was a reason for that choice.

Here are how they appear:

Audio Processing

The fingertip position data (top image, above) powers the speed for the output sound, whose network is demonstrated here:

There are three AudioFileIn CHOPs (you can ignore that error), each loaded with a .wav file. Their speed is referenced with ‘op('two_finger')['distance']’, but if a larger project is at work here you could also just send that data to one of these CHOPs and then reference that for each with ‘op('audiofilein1').par.speed’. Even better, you could bind all of the speed parameters so that they’re all guaranteed to change at the same rate.

From there, they are all sent into a Cross CHOP, an AudioFilter, an Audio and an AudioDevOut. Both the Cross and the Filter are controlled by data stemming from the other finger position CHOP, which I’ll address below. Generally, however, this is the construction for a simple song selector that only moves in one direction.

The CHOP labelled ‘h2_pinch_distance’ leads here. I use a Select CHOP to isolate the channel ‘h1:pinch_midpoint:rotation’ (I apologize for the confusing naming conventions). This then enters an Expression CHOP, which uses the expression ‘me.inputVal < 180’ to determine if the value of the channel is less than 180º. I split that signal into a Null CHOP and a Count CHOP. That Count counts from 0 to 2, and then resets. Fed into a null, that signal is what powers the Cross CHOP of the song selector.

That’s the entire network for that functionality, which would work as is. The overlaid circles on my fingertips comes from the MediaPipe plug-in itself (addressed in Torin’s video).

Visual Processing

As I’m rather green with the program, I couldn’t fully wrap my head around how to use instancing to generate this audio waveform. There are many good videos, but I may never find the rather obscure one that helped me understand it as it relates to this specific concept. I’ll try and be as specific as possible here, for both of us.

First we process the audio output, which comes from the AudioDevOut CHOP:

Audio Processing for the instancing set-up.

What this does is take the stereo output of the CHOP, and then convert it into mono via a Math CHOP that sets 'Combine Channels’ to ‘Add. Then, its changed from a frequency to a spectrum with an AudioSpectrum CHOP. Then it goes into a Resample CHOP, which makes sure the sample rate is equal to the lines which will be instanced (I will address this next). Then it goes into a Math CHOP, which just sets the range from 0.3 to 1 in order to remove dead space in the very low frequencies. Finally, it goes into an Audio Dynamics CHOP which adjusts the volume based on pinch distance. Finally, I rename this frequency to ‘ty’, for clarity.

Tethering the line endpoints to pinch midpoints.

The Line SOP is tethered to the pinch midpoint channels so that it moves with my hands in a basic way. The number of points here is important. It will determine how dense the resulting waveform is. Simply: more points means more available frequencies when you instance it.

This line is sent into a SOP to CHOP, whose output is the second input for the resampling channel in the above audio processing. It is also sent to a Select CHOP, which just selects the ‘tx’ and ‘tz’ channels. These are then merged together with a Merge CHOP to have a resulting three-channel CHOP.

Essentially, this process replaces the default ‘ty’ value of the line with the audio spectrum.

Importantly, I set two Null CHOPs. One is connected to the Merge CHOP (‘null7’), but the other is connected to the SOP to CHOP (‘null8’). This is so that we have one CHOP that is modified, and one which is default.

Next, lets set up the Geometry. First, add a Rectangle SOP, and set its ‘sizex’ to 0.02. Wire that to a Transform SOP, and set its uniform scale to 0.2. Add a Geometry, and enable instancing. Enter your values as such:

And that’s it! You should have the audio waveform working nicely. The way this works:

The key is having two different CHOPs powering the instancing. If you just use ‘null8’, the resulting waveform will be just a series of equal sized rectangles. The two CHOPs work in tandem, ‘null7’ instancing the xyz values of the Line SOP’s points, while ‘null8’ keeps it as a line by default. I don’t fully understand how that makes this, but alas, it works.

Enjoy.

MediaPipe Setup

Audio Processing

Visual Processing

Blog Post Title Two