Music-reactive worlds with Pulse

Pulse listens to the music of your world and turns it into numbers your shaders can read every frame: the level of four bands (bass to treble) and their last 1.4 seconds, the spectrum, smoothed levels, a musical time that runs faster on loud beats, and four colors taken from the notes that dominate. The package also has a tablet that players hold to tune all of it during a party.

It is the Unity package NodeWorld Pulse (com.nodeworld.pulse), under the MIT licence. It needs NodeWorld SDK Worlds, which it brings.

Add the package#

1. Add the NodeWorld registry to your project, as for the SDK (install the SDK).
2. Open Window, Package Manager, choose My Registries, select NodeWorld Pulse, then Install.
3. In your world's NW World Config, add the seven scripts of Packages/com.nodeworld.pulse/Runtime/Flux/ to Flux Sharp Sources (in the Project window, they are under Packages, NodeWorld Pulse, Runtime, Flux): PulseAudio, PulseTablet, PulseHandle, PulseSharedSlider, PulseSharedToggle, PulseThemeColors, PulseLabels.

Connect the music#

1. Drag the Pulse prefab (Packages, NodeWorld Pulse, Runtime, Prefabs) into your scene. It holds the engine: an FL# Script component with the class Pulse.PulseAudio.
2. In its field source, put the Audio Source that plays your music: a speaker of the scene, or the sound output of a video screen.

Pulse reads what the source plays, whatever its volume: a player who turns the music down still sees the lights react. When the source is silent or stopped, every value falls back to 0.

One engine per world is enough: every shader of the scene reads the same data.

Put the tablet in the world#

1. Drag the PulseTablet prefab into your scene and place it: its stand goes on a table, at about 1 m from the floor.
2. On the root of the prefab, the FL# Script with the class Pulse.PulseTablet has a field pulse: put the Pulse object of your scene in it.

The tablet stays on its stand until someone takes it. A player takes it by one of the two grips at the bottom, or by both, with one hand each; let go, it stays where it is. Its place is the same for every player.

What the screen does:

The lower card shows the last 1.4 seconds of each band (four rows of dots, now on the right) and, under them, a wave that grows with the music.

The texts of the screen follow the language of each player's game (English or French).

Who changes what#

The settings are shared: when a player moves a slider, everyone in the instance sees and hears the same. The field controllerSyncMode of Pulse.PulseTablet chooses what is shared:

Read the music in a shader#

Include Pulse.cginc from the package, then read the values. A complete shader, to save as a .shader file in your project: the first color of the music glows with the bass.

Shader "MyWorld/Pulse Glow"
{
    Properties
    {
        _MainTex ("Texture", 2D) = "white" {}
        _Glow ("Glow", Range(0, 4)) = 1
    }
    SubShader
    {
        Tags { "RenderType" = "Opaque" }
        Pass
        {
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #include "UnityCG.cginc"
            #include "Packages/com.nodeworld.pulse/Runtime/Shaders/Pulse.cginc"

            sampler2D _MainTex;
            float4 _MainTex_ST;
            float _Glow;

            struct appdata
            {
                float4 vertex : POSITION;
                float2 uv : TEXCOORD0;
                UNITY_VERTEX_INPUT_INSTANCE_ID
            };

            struct v2f
            {
                float4 pos : SV_POSITION;
                float2 uv : TEXCOORD0;
                UNITY_VERTEX_OUTPUT_STEREO
            };

            v2f vert (appdata v)
            {
                v2f o;
                UNITY_SETUP_INSTANCE_ID(v);
                UNITY_INITIALIZE_OUTPUT(v2f, o);
                UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(o);
                o.pos = UnityObjectToClipPos(v.vertex);
                o.uv = TRANSFORM_TEX(v.uv, _MainTex);
                return o;
            }

            float4 frag (v2f i) : SV_Target
            {
                float4 col = tex2D(_MainTex, i.uv);
                if (PulseIsOn())
                    col.rgb += PulseThemeColor(0) * PulseSmooth(0, 8) * _Glow;   // the first color of the music, with the smoothed bass
                return col;
            }
            ENDCG
        }
    }
}

The data texture#

The engine computes the global texture _PulseTex (128 × 32 floats) on the graphics card every frame, as AudioLink does: its script only sends the music's samples and the settings. The bands, the theme colors and the autocorrelation follow the notes one frame later. Its places, as #define in Pulse.cginc:

The engine's fields#

The FL# Script of the Pulse prefab (Pulse.PulseAudio) starts with these values; the tablet changes them during the game.

Licence#

The package is under the MIT licence. Its LICENSE.md also holds the notices of the code it is derived from (MIT) and of its font, Geist (SIL Open Font License).