audio-reactive AudioWorklet analysis for visualizers

From an AudioNode to numbers a renderer can use.

An AudioWorkletProcessor that computes an FFT, harmonic/percussive separation, spectral features and per-band beat envelopes off the main thread, plus the pure functions behind it for band levels, transients and beat tracking that run anywhere, and a Hermite interpolator that turns 43 Hz packets into a continuous curve for 240 Hz displays. Zero dependencies. The spectrum bytes match AnalyserNode.getByteFrequencyData, so it drops in without retuning anything downstream. Born in Stims, the visualizer at toil.fyi.

npm install audio-reactive Try the live demo worklet.js

Signal path

Everything inside the dashed box runs on the audio thread with no per-frame allocation; one packet per FFT block crosses to the main thread with its buffers transferred, not copied.

Signal path: AudioNode into the worklet (Hann window, FFT, dB bytes, HPSS, beat envelopes), a packet to the main thread, then interpolator, band levels and spectral features into the renderer. AudioNode mic, media element, stream, oscillator AudioWorkletGlobalScope: FrequencyAnalyserProcessor 128-frame quanta into fftSize ring Hann window, radix-2 FFT, magnitude to dB to byte HPSS time and frequency medians, Wiener masks band averages bass / mid / treble subBass / kick spectral features flux, crest, centroid, rolloff beat envelopes per band, transients packet port.postMessage transferred buffers main thread interpolator Hermite between packets band levels getFrequencyBandLevels features, beats extractSpectralFeatures renderer recycle-buffers
Each box is a named export or a packet field; the live demo below shows every one of them in motion.

Live demo

The real FrequencyAnalyserProcessor, loaded from lib/worklet.js (the package's worklet entry bundled into one file), analysing audio in this tab. Nothing leaves the page.

idle packets 0 · – · – · fft –

Band energy packet.energy

bass–running mean –
mid–running mean –
treble–running mean –

Weighted averages of the dB-scaled bytes over 24–320, 320–2800 and 2800–12000 Hz. energyAverages is the mean over the last 64 packets.

Beats packet.beatDetection

beats
0
beatIntensity
–
bass mid treble

transientMetrics –

The dot fills with beatIntensity; the badges light while a band's own envelope is above 0.3. Each band compares itself to its running average with a 150 ms refractory interval.

Harmonic / percussive packet.harmonicPercussive

harmonic–
percussive–
harmonicpercussiveRatio –percussive

percussive by region –

Median-filter HPSS over the last 7 spectra: sustained tonal content versus broadband transients. 0.5 in silence.

Spectral features

spectralCentroid
–
spectralFlatness
–
spectralRolloff
–
spectralFlux
–
spectralCrest
–
rms
–
zeroCrossingRate
–

Centroid and rolloff in Hz; flatness 0 (a tone) to 1 (noise); flux is the positive magnitude change since the previous packet.

Spectrum and waveform packet.frequencyData, packet.waveformData

512 bytes on the AnalyserNode scale ([-100, -30] dB), drawn on a log-frequency axis; below it the 1024-byte waveform, 128 = silence.

Interpolator createAudioReactivityInterpolator

Latest packet

Reading energy.bass straight off the newest packet, once per frame. Steps at the ~43 Hz block rate.

raw–

Hermite spline

interpolator.sample(performance.now()) on every requestAnimationFrame. C1-continuous, extrapolates up to 25 % past the newest packet.

smooth–

Wiring, as the page does it

const context = new AudioContext();
await context.audioWorklet.addModule('./lib/worklet.js'); // audio-reactive/worklet, bundled
const node = new AudioWorkletNode(context, 'frequency-analyser', {
  numberOfInputs: 1, numberOfOutputs: 1, outputChannelCount: [1],
  processorOptions: { fftSize: 1024, sampleRate: context.sampleRate, messageEvery: 1 },
});
source.connect(node);
node.connect(context.destination); // output is silence; keeps the node alive

node.port.onmessage = ({ data: packet }) => {
  const frequencyData = new Uint8Array(packet.frequencyData); // 512 bytes
  const waveformData = new Uint8Array(packet.waveformData);   // 1024 bytes
  const timeDomainData = new Float32Array(packet.timeDomainData);
  draw(packet.energy, packet.beatDetection, packet.harmonicPercussive, frequencyData);
  node.port.postMessage(
    { type: 'recycle-buffers', freq: [frequencyData.buffer], wave: [waveformData.buffer], timeDomain: [timeDomainData.buffer] },
    [frequencyData.buffer, waveformData.buffer, timeDomainData.buffer],
  );
};

Offline, no browser audio

The README's offline example, run by this page on load from lib/audio-reactive.js (the main entry). Two seconds of a 440 Hz sine at -20 dBFS with a 10 ms click every 500 ms, analysed block by block with the same functions the worklet uses. The same code runs under Bun or Node.

weighted energy per 1024-sample blockbeat detected by createBeatTracker

running…

Frames 20–22, around the second click

…

The frame holding the click reads as broadband (flatness near 1, centroid near 11 kHz) and percussive (ratio near 1); the frames around it read as a tone near 440 Hz with a percussive ratio near 0.4.

import {
  analyseBlockBytes, buildHannWindow, buildTwiddleTable, createBeatTracker,
  createHarmonicPercussiveAnalyser, extractSpectralFeatures, getFrequencyBandLevels, getWeightedEnergy,
} from 'audio-reactive';

const window = buildHannWindow(fftSize);
const twiddles = buildTwiddleTable(fftSize);
const scratch = { real: new Float32Array(fftSize), imag: new Float32Array(fftSize) };
const spectrum = new Uint8Array(fftSize / 2);
const hpss = createHarmonicPercussiveAnalyser();
const beats = createBeatTracker();

for (let start = 0; start + fftSize <= samples.length; start += fftSize) {
  const block = samples.subarray(start, start + fftSize);
  analyseBlockBytes(block, window, twiddles, spectrum, scratch);
  const bands = getFrequencyBandLevels(spectrum, sampleRate);
  const energy = getWeightedEnergy(bands);
  const hp = hpss.analyse(spectrum, sampleRate);
  const beat = beats.update({ bands, weightedEnergy: energy, deltaMs: blockMs }, (start / sampleRate) * 1000);
}

API at a glance

Everything comes from audio-reactive except the processor, which is the audio-reactive/worklet entry. The README has the full table and every type.

Spectrum maths

ExportPurpose
buildHannWindow(length)Hann window as a Float32Array.
buildTwiddleTable(length)Precomputed { cos, sin } twiddles for analyseBlockBytes.
validateFftSize(value)A power-of-two FFT size or a RangeError; defaults to 1024.
analyseBlockBytes(block, window, twiddles, out, scratch)Hann, radix-2 FFT, magnitude / binCount, dB-to-byte; writes block.length / 2 bytes into out.
computeBandAverage(bytes, sampleRate, fftSize, minHz, maxHz, bandType)Weighted mean of a byte spectrum over a Hz range, 0..1.
createHarmonicPercussiveAnalyser(options?)Median-filter HPSS over successive byte spectra; analyse(bytes, sampleRate) returns HarmonicPercussiveLevels.

Features and reactivity

ExportPurpose
extractSpectralFeatures(timeDomain, amplitudes, sampleRate, fftSize?)RMS, centroid, flatness and rolloff in one pass; each also exported alone (computeRms, computeSpectralCentroid, …).
getFrequencyBandLevels(bytes, sampleRate?, bandRanges?, fftSize?){ bass, mid, treble } in 0..1 from Hz ranges (DEFAULT_FREQUENCY_BAND_RANGES).
getExtendedFrequencyBandLevels(...)Adds subBass (24–60 Hz) and kick (60–250 Hz).
getFourBandTransientMetrics(bytes, previousBytes?, sampleRate?){ subBassEnv, kickTransient, vocalMidEnv, snareSnap } from frame-to-frame deltas.
getWeightedEnergy(bands, { weights?, boost? })A single 0..1 energy from weighted bands.
updateEnergyPeak(currentPeak, weightedEnergy, { decay?, floor? })Decaying peak tracker.
createBeatTracker(options?)Multi-band onset tracker; update({ bands, weightedEnergy, deltaMs }, timeMs) returns a BeatTrackerUpdate. mode: 'projectM' selects a single running-average threshold.
createWaveformAutoGain(options?)Boost-only auto-gain for byte waveforms: measure(wave), apply(in, out, gain).
createAudioReactivityInterpolator()Hermite spline over { bass, mid, treble, rms }: pushSample(sample, timeMs), sample(timeMs), reset().

Lifecycle and the worklet

ExportPurpose
AudioLifecycleTracker, AudioGenerationTokenState machine for the AudioContext lifecycle with leak and ordering assertions, and a generation counter for superseded async mounts. Intended for tests.
isAudioAwaitingGesture(), subscribeAudioGestureGate(fn), …Observable flag for "audio is set up but suspended until a user gesture".
FrequencyAnalyserProcessor /workletThe AudioWorkletProcessor; the module registers it as 'frequency-analyser' on load. Options: fftSize, sampleRate, messageEvery. Stereo input adds frequencyDataL/R and waveformDataL/R.

Design notes

Condensed from the source docblocks; the README carries the long form.

Zero allocation in the worklet

process() runs on the real-time audio thread every 128 frames. Ring buffers, window, twiddles, scratch arrays and HPSS history are allocated once in the constructor. The only per-packet allocation is the set of ArrayBuffers posted to the main thread, and those are transferred, not copied; send them back with recycle-buffers and the processor pools up to 16 of each, so a steady state allocates nothing.

The AnalyserNode dB window

AnalyserNode.getByteFrequencyData maps magnitude through 20·log10 into [-100, -30] dB. Band levels, beat thresholds and the downstream signal processors in Stims were tuned against those bytes. An earlier linear mapping left realistic music (-20 to -40 dBFS) at byte values of 0–12 and flatlined every band, so analyseBlockBytes applies the same window and the worklet can replace an AnalyserNode on the same scale.

Median-filter HPSS

Fitzgerald (2010): a median across time (per bin, over 7 frames) suppresses transients and estimates sustained content; a median across frequency (9 neighbouring bins) suppresses narrow peaks and estimates broadband content. Wiener-style soft masks split each frame's energy into harmonic and percussive. It is a spectrogram measurement, not stem separation. Medians over at most 9 values use insertion sort. percussiveRatio is 0.5 in silence so it never emits NaN.

Hermite interpolation between packets

Packets arrive at the FFT block rate, about 43 Hz at 44.1 kHz / 1024, while displays refresh at 60–240 Hz; sampling the last packet produces staircase stepping. The interpolator keeps the last two packets with finite-difference velocities and evaluates a cubic Hermite spline between them: C1-continuous, clamped at zero, allowed to extrapolate 25 % past the newest packet so rendering never waits on audio.