Situation
Models such as MIDI-GPT can already add or rewrite a track around a musician’s sketch. What is unclear is how a model should develop a musician’s own draft, and what musicians take from it.
FLUX, my final-year project, is a browser MIDI workbench and a study. My October proposal asks:
- Modes. How do varying a draft and rewriting it from the other tracks differ, and does the model follow those tracks?
- Musicians. What do musicians take, change or reject from each mode, and how do they experience control and ownership?
- Fit. Can a learned measure of fit predict musicians’ own judgements?
Role
FLUX is an individual project; scope, product and research decisions, and hands-on testing are mine.
Constraints
- A draft can develop in many good ways, so the study compares strategies, not one right answer.
- Test drafts are my own, because public pop MIDI is in the model’s training data.
- The scope is short multitrack loops and one melodic line at a time.
- Anything involving participants waits for ethics approval.
Process and architecture
A workbench before a model
The earlier prototype had no sound, and its suggestion buttons ran on a random stub. I built editing, playback and chord analysis first, so a suggestion can be heard in context.
Chords and keys stay suggestions until the musician confirms them; candidates are planned as muted tracks to audition or copy from.
Varying or rewriting a draft
Variation keeps the draft’s rhythm and resamples some of its pitches in context. Rewriting withholds the draft; the model writes the part from the other tracks alone. The model, sampling and interface stay fixed.
Kept notes can flatter a model, so only the notes it writes will be scored against random in-key notes and a chord-tone rule. Removing or transposing the other tracks will test whether it follows them.
A learned measure of fit
I will train a small model to score how well a part fits its accompaniment, from real Lakh parts and shifted or mismatched copies. Musicians’ ratings of the same candidates will test it.
Related systems
- MIDI-GPT (opens in a new tab) infills tracks and bars under attribute controls; FLUX uses it at a pinned checkpoint.
- Composer’s Assistant 2 (opens in a new tab) already fills a user’s rhythm with pitches; FLUX starts from a musician’s own draft.
- Krol, Llano and McCormack (opens in a new tab) found that a variation tool relying on musicians’ input supported ownership.
- Huang and colleagues (opens in a new tab) scored stem compatibility with stems from one song as positives, as FLUX’s fit model will.
Results and evidence
On unusual harmonic requests, writing the chords into the context as a guide track met 41% of requests; the built-in controls tested met under 1%. (development set of 25 eight-bar Lakh MIDI excerpts with the human bass removed; 332 requests, three seeds each; pinned MIDI-GPT weights; compliance scored by a rule-based verifier, not by listeners; the verifier's agreement with human judgement so far rests on my own blind ratings) That is why both modes give the model explicit material rather than soft controls.
The Phase A workbench edits and plays multi-track MIDI in the browser, detects chords and suggests a key; its core logic passes 102 unit tests. (automated tests; they check that audio renders, not how it sounds; six need a separately downloaded sound bank)
The instrument constraint masks logits to an allowed instrument set and is not a target-track guarantee. (2x2 pilot of the Anticipatory Music Transformer on one melody, one run per condition; neither constrained run produced a bass note, though bass was allowed; top_p also differed between free and constrained runs)
Reflection and next
I once described the constraint as guaranteeing the target track; the pilot showed it does not. I also assumed the old Regenerate button gave a new suggestion each time; nothing ensured it. The study now records variety instead of assuming it.
Planned:
- Probe both modes, checking that kept notes stay kept.
- Freeze the modes; run the analyses on development drafts, then once on new test drafts.
- Train the fit model and freeze it before participants rate anything.
- After ethics approval, run sessions with musicians on their own drafts, then write up.
Boundaries
No participant data has been collected, and feedback on the workbench will not be reported as user research. MIDI-GPT’s weights are licensed for non-commercial use; the study does not fine-tune them. The guide-track records are not yet public; the code is public for review with all rights reserved.