Latest Version

V. 1.1 - DOWNLOAD

GENERATOR
  • Generator now surfaces songBuilder functions
  • Generator no longer scrolls automatically when adding methods to noteBuilder and methodBuilder.
  • Generator has fixes for several import/export bugs
GENERAL
  • Video file framerate was lowered to 30 from 60 fps for increased performance
  • C2M now has rudimentary tempo change recognition when songs are swapped with a songChange trigger. The tempo must be changed in Ableton Live before the song change.
  • Mid-song or on song change video input source changes have been temporarily removed.
  • You can now turn off face/body detection with the enableFaces setting in the C2M tab
  • Midi output settings have been moved to instrumentBuilder
  • Cosine has been moved to instrumentBuilder. A document-level cosine value in settings is the default for each instrumentBuilder, and thus a similar effect to a single document cosine
  • Normalizer utility has been replaced with the eviousocessing map function
  • numberBetweenIntegers function was removed
  • randomNumberArray function was removed
  • roundToHalf function was removed
  • Video warps have been moved to their own tab
  • Waves function has been renamed to setCosineValues
  • javax.sound.midi.MidiMessage is no longer imported
  • javax.sound.midi.SysexMessage is no longer imported
  • javax.sound.midi.ShortMessage is no longer imported
SONGBUILDER
  • songBuilder can now have attached messageBuilder and triggerBuilder. They are activated with no colorMin or colorMax on every 1/128th note
  • Default time signature is now 4/4
  • Default scale is now all notes instead of cMajor
INSTRUMENTBUILDER
  • Midi output settings have been moved to instrumentBuilder to allow multiple buses to route midi — and thus infinite midi channels
  • beatRestrict has a new default: 1, on
  • beatTime has new default: 32 (1/4 notes)
  • beatTime is now an array, each integer in the array will activate the instrument. Even though beatTime is now a different primitive, it is a non-breaking change
  • Cosine has been moved to instrumentBuilder.
  • instrumentBuilder with no attached noteBuilder objects now only send 1 error message to the console
  • luma, cosine and faces are now added to the colorSum rather than the colorMin. The corresponding code was removed.
  • legatoOverlap allows notes overlap for Live instruments that calculate legato in increments of 1/128th note
NOTEBUILDER
  • mapToScale now maps to setNoteRange when lockToScale is unset
  • The default minimum length for notes is now 16 (1/8th note)
  • The default maximum length for notes is now 32 (1/4th note
  • learnedNoteCount for noteSets. After x plays, the average note that has been played up to that point for that particular note will be set.
  • learnedNoteResetCount the count for a learned note that will cause it to be relearned with the learnedNoteCount
  • cosineAffectsVelocity and lumaAffectsVelocity functions alter default noteBuilder functionality.
  • noteRelative can now be set to 5, which makes the note value relative to the last note that was played on the parent InstrumentBuilder.
SPAWNBUILDER
  • SpawnBuilder code has been rewritten and refactored
  • No longer have beatTime or beatRestrict properties
  • Since cosine, luma, and faces are added to colorSum, the code for cosine and luma affectations in Spawn has been removed.
  • legatoOverlap allows notes overlap for Live instruments that calculate legato in increments of 1/128th note
TRIGGERBUILDER
  • New trigger keys:
  1. onActiveCount
  2. onCamStart*
  3. onColorJitter
  4. onCosine
  5. onCount
  6. onDisable
  7. onEnable
  8. onFaceDetect*
  9. onFaces
  10. onInactiveCount
  11. onInstrumentCount*
  12. onLearnedNote
  13. onLuma
  14. onMessageLength
  15. onMessageSecondByteValue
  16. onMessageThirdByteValue
  17. onNoteCount
  18. onNoteLength
  19. onNoteStartTime
  20. onProgrammaticNote
  21. onProgrammaticNoteBase
  22. onRawMidi*
  23. onSongStart*
  24. onTempoSelect*
  25. onUncount
  26. onVelocity
  27. onVideoFileStart*
  28. onVideoStart*
  • Starred Trigger Keys apply to triggers attached directly to the songBuilder, not to instrumentBuilder
MESSAGEBUILDER
  • MessageBuilder code has been rewritten and refactored
  • No longer have beatTime or beatRestrict properties
  • messageLengthProportional function was removed
  • messages now have an enable parameter
  • messages now have an enableOnCount parameter
  • messages now have an disableOnCount parameter
  • noteLockLength for messageBuilder has been renamed messageLockLength
  • tickTockNoteLength was removed in favor of starting at the previous value when resetOnLoop is set to 0
  • resetOnLoop is now 1 (on) by default
  • setMessageLength is now 2 arrays, a min array and a max array
  • Stasis message functions have been renamed with “start” instead of “stasis”
KNOWN BUGS
  • If Live is playing when you attempt to launch C2M from processing, it will not launch.

V. 1.0 - DOWNLOAD

Overview

Introduction

C2M is a Processing sketch that evaluates the color of video input to create MIDI output. Unlike other systems that have a similar goal, C2M is designed to create MIDI output that is musical, rather than “noise,” and for the created sounds to be extremely customizable.

C2M works by users writing C2M code. As C2M analyzes video input for its color data, this code interprets that color data to create MIDI output. For example, the code could allow a certain saturation of a color to cause a MIDI signal.

MIDI signals are then routed to Ableton Live where they can be processed as MIDI data from a real-world instrument.

There is also the beginning stages of face and body tracking present in the C2M.

Basics

Timer

C2M evaluates the color of video by looping a timer at a very fast constant rate. Every time the timer loops the video colors are evaluated.

This timer can be thought of like a tempo. Each loop of the timer is equivalent to 1/128th note.

The speed of the timer is determined by the BPM value in Ableton Live’s open composition.

Beat

The looping timer also keeps track of elapsed time, called beatTime. The timer keeps track of beatTime in 2 ways :

When writing C2M code, you can choose which beatTime system to use, allowing for a great variety of musical arrangements.

Color Data

C2M evaluates the color data in a video stream on every 1/64th beat.

As the color is evaluated color data is produced that determines the playing video frame’s red, green and blue color values. These values range from 0-255.

A 0 value for any color would be near black, a 255 value would be full saturation.

 
 
 

After calculating the color data for each pixel, the colors are averaged to determine the quantity of each part of the color spectrum present in the entire frame.

Objects

Once C2M has created averaged color data for the playing video frame, user-created C2M code utilizes the data to create MIDI output. Code is written by creating ‘objects’.

C2M has 6 objects, each with a distinct purpose and are activated and deactivated based on the beatTime, color and other objects.

Objects

Object Basics

Objects can be thought of like things in the real-world: a dog has characteristics (name, breed, color) and behaviors (barking wagging and running). C2M objects also have characteristics and behaviors.

There is no limit to the number of objects that can be present in code. By creating objects in different ways, an unlimited number of complex MIDI arrangements can be created.

Each object has several characteristics that must be defined for each and numerous optional characteristics that can be tweaked through methods.

Some objects must be contained within other objects.

songBuilder

Think of songBuilders like a song. They provide the overall structure and contain all the ingredients that make up a piece of music.

SongBuilder have two primary tasks:

Firstly, they control the tempo, video source and any properties that control the entire document and all objects within it..

Secondly, they bundle together all the other objects into a single package.

You can have more than one songBuilder with as many objects within it as you like, however only one can be active at a time — just like you wouldn’t want to play two real-world songs at the same time.

Advanced Usage

instrumentBuilder

An instrumentBuilder can be thought of like an instrument. With the help of other objects, they play sounds via MIDI signals.

An instrumentBuilder has two primary functions:

Firstly, instrumentBuilder house all the other objects (with the exception of songBuilder) and determine when those objects are activated.

Secondly, instrumentBuilder objects produce MIDI signals when activated.

instrumentBuilder objects are activated when two things occur:

  • Firstly, each instrumentBuilder has a beatTime: the beat that will allow the instrumentBuilder to activate. For example, every 1/4 note in a measure.
  • Secondly, each instrumentBuilder is activated on the basis of the video color data. Each instrumentBuilder is assigned a color — red, green or blue — and a subset of that color, creating a color range. The playing video frame must have enough color within that range, based on a set minimum to maximum range, to activate.

InstrumentBuilder must have an attached noteBuilder that defines the notes tha that are sent via MIDI message.

Advanced Usage

noteBuilder

noteBuilder can be thought of as sheet music. They have a scale, a series of notes and the length of those notes.

An instrumentBuilder plays a single sequential note, defined in a noteBuilder,  each time it is activated.

The pitch of each note can be set outright, relative to the instrument that is playing it or automated based on the instrument’s defined color range.

Advanced Usage

spawnBuilder

spawnBuilder are similar to instrumentBuilder.

instrumentBuilder only play one note at a time. spawnBuilder, when attached to an instrumentBuilder, allow the instrumentBuilder to play additional notes, like chords.

spawnBuilder are activated when several things occur:

  • When the instrumentBuilder the spawnBuilder is attached to activates
  • When spawnBuilder’s defined color intensity criteria is met
  • Optionally: when spawnBuilder’s beatTime criteria is met

spawnBuilder must have an attached noteBuilder that defines the notes tha that are sent via MIDI message.

Advanced Usage

triggerBuilder

triggerBuilder objects are like events.

When an event occurs, a triggerBuilder can change the settings of any instrumentBuilder object in the composition. For example the beatTime of an instrument could be changed to activate more frequently.

Trigger Builder are activated based on three criteria:

  • When a connected instrumentBuilder is activated.
  • Based on color intensity.
  • If the previous two criteria are met, the triggerBuilder has an additional user-defined threshold that must be met — for example it could be that the connected instrument must have been activated 10 times or hit a minimum velocity.

Advanced Usage

messageBuilder

messageBuilder are advanced objects. They send customized MIDI messages that can be translated by a digital audio workstation to perform automation tasks.

Messages can be 7 bit or 14 bit (like pitch bend messages).

Advanced Usage

Setup

General

This setup guide is written for Mac OS X, though C2M should be compatible with Windows without alteration, though it has not been tested.

It involves installing Processing, installing C2M and the necessary libraries, setting up Mac OS X’s IAC Bus, and interfacing with Ableton Live.

1. Install Processing

  1. Download Processing — It’s free.
  2. Unzip the downloaded file.
  3. Drag the unzipped Processing app to your applications folder.

2. Install Processing Libraries

C2M uses several free libraries for Processing that can be downloaded from within the app itself.

  1. Launch the Processing app.
  2. Go to Tools -> Add Tool from the top menu bar.
  3. Click the “Libraries” tab.
  4. Find “Signal Filter” by Raphael de Courville. Select it and click the install button.
  5. Find “The MidiBus” by Severin Smith. Select it and click the install button.
  6. Find “Video” by The Processing Foundation. Select it and click the install button.
  7. Find “OpenCV for Processing” by Greg Borenstein. Select it and click the install button.

3. Download C2M

The C2M package below below is always up to date.

  1. Download C2M (v1.1)
  2. Unzip the downloaded file
  3. Launching any .PDE file within the unzipped file will launch C2M

4. Setup IAC Driver and Bus

Mac OS X’s IAC bus allows C2M to communicate with a digital audio workstation.

  1. Launch Audio MIDI Setup from Applications -> Utilities
  2. Open Window -> Show MIDI Studio from the top menu bar
  3. Double-click the “IAC Driver” icon and check the option “Device is online”
  4. Under “Ports”, add or rename the top item to “IAC Bus 1” (without quotes). If you must use an alternate name, you can set it by changing the default name under the SongBuilder tab of C2M in Processing.
  5. If you are using Ableton Live, you must create a second port under “Ports” due to buggy MIDI routing within Ableton Live. Name it “IAC Bus 2” (or whatever else you would like).

5. Setup Ableton Live

Ableton Live has buggy MIDI routing that causes MIDI routed to the app to be unable to produce sound. We can get around this by routing MIDI through a second MIDI bus.

  1. Download MidiPipe
  2. Unzip the download and drag the folder to your applications folder
  3. Launch MidiPipe
  4. Download C2M MidiPipe save file
  5. Unzip and launch the downloaded file.
  6. On the right side of the MidiPipe window, you’ll see Midi In and Midi Out.
  7. Click “Midi In.” On the bottom of the window under “Midi Input” make sure “IAC Driver – IAC Bus 1” is selected. If you have changed the name of “IAC Bus 1”, select that option.
  8. Click “Midi Out.” On the bottom of the window under “Midi Output” make sure “IAC Driver – IAC Bus 2” is selected. If you have changed the name of “IAC Bus 2”, select that option.
  9. Under “Midi Out” make sure “Use Note Off Velocity” and “Pass Through” are both checked.
  10. Download and install MIDI Monitor. By selecting only “IAC Bus 2” under “Sources” within MIDI monitor, you can make certain MIDI is being routed properly.
  11. This should allow the setup MIDI to make it to Ableton Live.

Due to Ableton Live’s buggy MIDI routing, sometimes MidiMonitor routing fails. A hard restart of the computer fixes this, usually.

6. Launching Processing Sketches

When you launch a processing sketch with user code, nothing will happen. Once the sketch has fully launched, you must press the play button in Live, which then will communicate the tempo to Processing and evaluate the colors from that point forward.

In order for Live to communicate with Processing, IAC Bus 2 output sync must be set to “on” in live preferences. Any IAC bus that is being used for input must have a “track” parameter set to “on” in Live preferences as well.

External instruments may have active track parameters also and will not interfere.

Screen Shot 2016-06-26 at 4.49.47 PM

Code

Basics

C2M code is copied into the A_USERCODE tab of Processing. It can be written manually or generated.

Code should always be wrapped with with void setupInstruments(){ and }, respectively.

Between those lines any number of songs, and within those songs any number of additional objects, can be added.

Example Two new songs called “song1” and “song2” are added to the document. You can name a song anything you would like as long as it is a valid java variable name.

							

void setupInstruments(){

 

/* song1 */
SongBuilder song1 = new SongBuilder(); /* New song called song1 */

/* Write objects for song1 here */

addSong(song1); /* Add song to document  */

 

/* song2 — Make as many or few songs as you like */
SongBuilder song2 = new SongBuilder(); /* New song called song2 */

/* Write objects for song2 here */

addSong(song2); /* Add song to document  */

 

}

Creating an instrumentBuilder

Adding an instrumentBuilder to a song takes 6 lines of code. All lines are required. Additional methods can be added to change the object behavior.

The first line of the new instrument code deals with creating the instrumentBuilder object. The last line adds it to a songBuilder object.

The lines in-between, prefixed with the name of the instrument “in1” are methods. They change the behavior of the instrumentBuilder.

See the instrumentBuilder reference for additional explanation.

This code in itself will do nothing. It requires an additional noteBuilder object to create sound.

 

							

void setupInstruments(){

/* song1 */
SongBuilder song1 = new SongBuilder(); /* New Song called song1 */

 

/* Create a new instrument called in1 */
InstrumentBuilder in1 = new InstrumentBuilder();
in1.setColorRange(0,150,50);
in1.setNoteBase(60);
in1.setBeatTime(4);
in1.setColorMinMax(new int[]{0},new int[]{255});
addInstrument(in1, song1); /*Add instrument to song1 */

 

addSong(song1); /* Add song to document  */

}

Adding Methods

Methods change the way that an object behaves. See the documentation for each object for an explanation of what each method does.

Methods are added by starting with the name of the object the method will be applied to followed calling the method by name. For example: objectname.setColorRange(); objectname.setBeatTime();

Between the parenthesis one or more required values for the method are added. The value required by each method is described in the documentation. For example, a method requiring 3 integers:
objectname.setColorRange(0,150,50); A method requiring 2 integer arrays: objectname.setColorMinMax(new int[]{0},new int[]{255});

Methods can be added to an object any time after the new object line of code, but organizationally are best kept within the same block of code as the other object methods.

							

void setupInstruments(){
/* song1 */
SongBuilder song1 = new SongBuilder(); /* New Song called song1 */

 

/* Create a new instrument called in1 */
InstrumentBuilder in1 = new InstrumentBuilder();
in1.setColorRange(0,150,50);
in1.setNoteBase(60);
in1.setBeatTime(4);
in1.setColorMinMax(new int[]{0},new int[]{255});
/* ADDITIONAL METHODS BELOW */
in1.setNoteBaseRange(0,127);
in1.setMidiEnable(1);
in1.setLuma(0,0);
in1.setVelocityIncrement(0);
in1.setBeatRestrict(1);
/* ADDITIONAL METHODS ABOVE */
addInstrument(in1, song1); /*Add instrument to song1 */

 

addSong(song1); /* Add song to document */
}

Adding Notes

							

void setupInstruments(){

/* song1 */
SongBuilder song1 = new SongBuilder(); /* New Song called song1 */

 

/* New noteBuilder called note1 */
NoteBuilder note1 = new NoteBuilder();
note1.setNoteArray(new int[]{0});
note1.setNoteLength(new int[]{0},new int[]{16});

 

/* Create a new instrument called in1 */
InstrumentBuilder in1 = new InstrumentBuilder();
in1.setColorRange(0,150,50);
in1.setNoteBase(60);
in1.setBeatTime(4);
in1.setColorMinMax(new int[]{0},new int[]{255});

in1.addNote(n0_yioi); /* Add note to instrument */

addInstrument(in1, song1); /*Add instrument to song1 */

 

addSong(song1); /* Add song to document  */

}

Adding Other Objects

							

void setupInstruments(){

/* song1 */
SongBuilder song1 = new SongBuilder(); /* New Song called song1 */

 

/* Add objects before the instrument they are added to */

 

/* spawnBuilder objects */
SpawnBuilder spawn1 = new SpawnBuilder();
spawn1.setColorMinMax(new int[]{0},new int[]{255});

 

/* triggerBuilder objects */
TriggerBuilder trigger1 = new TriggerBuilder();
trigger1.setColorMinMax(new int[]{0},new int[]{255});
trigger1.setConnection(0,”onActiveCount”);
trigger1.setOperation(“divisibleBy”);
trigger1.setThresholds(0,0,0);
trigger1.setParameter(“midiEnable”,new int[]{0});

 

/* messageBuilder objects */
MessageBuilder message1 = new MessageBuilder();
message1.setFirst(new int[]{0xB0});
message1.setSecond(new int[]{0},new int[]{0},new int[]{0});
message1.setThird(new int[]{0},new int[]{0},new int[]{0});
message1.setColorMinMax(new int[]{0},new int[]{255});

 

/* Create a new instrument called in1 */
InstrumentBuilder in1 = new InstrumentBuilder();
in1.setColorRange(0,150,50);
in1.setNoteBase(60);
in1.setBeatTime(4);
in1.setColorMinMax(new int[]{0},new int[]{255});
addInstrument(in1, song1); /*Add instrument to song1 */

/* ADD OBJECTS TO INSTRUMENT */

in1.addSpawn(spawn1);
in1.addMessage(message1);
in1.addTrigger(trigger1);

 

addSong(song1); /* Add song to document  */

}

Other

Cosine

Luma

Video Filters