Music+code (LLM) assist workshop

[pikurasa]

Let's workshop the design for Music Ideation Through Generative Idea together.

[pikurasa]

I asked https://matrix.to/#/@edubot:matrix.org what parameters we might want to have a user specify, and this was its response:

Here's a list of some of the most important musical parameters that could be included in such an interface:

1. Style/Genre: The musical style or genre can greatly influence the composition. This could range from classical, jazz, rock, pop, country, electronic, and so on.

2. Key: The key of the piece will determine the tonal center and the scale used for the melody and harmony.

3. Tempo: This is the speed of the piece, usually measured in beats per minute (BPM).

4. Time Signature: This determines the rhythmic feel of the piece, such as whether it has a "three feel" (like a waltz) or a "four feel" (like most pop and rock music).

5. Structure/Form: The overall structure of the piece, such as verse-chorus-verse, AABA, etc.

6. Instrumentation: The instruments that will be used in the piece.

7. Melodic Themes: Any specific melodic ideas or themes to be used in the piece.

8. Harmonic Progression: The sequence of chords or harmonic structure of the piece.

9. Dynamics: The variations in loudness throughout the piece.

10. Lyrics/Theme: If the piece is a song, the lyrics or overall theme might be specified.

11. Mood/Emotion: The overall mood or emotion that the piece should convey.

12. Complexity: The level of complexity or simplicity desired in the piece.

These parameters can provide a good starting point for generating a piece of music, but there's always room for creativity and deviation from the initial parameters.

[pikurasa]

I think this is a good list. The most critical of these are probably: style/genre, mood/emotion, instrumentation, tempo, time signature, key, and structure (and probably in about this order).

[kshitijdshah99]

Hey, @pikurasa I am not well versed with how the musical terms you mentioned above can be generated using LLMs, because I lack experience in the field of music. However, I can tell you how can LLMs work by explaining you with an analogy.
Whether its text or music the way in which LLMs generate is the same. It's true they consider a lot of parameters before generating data. In case of text we can set the fluency of grammar, vocabulary or the tasks we want to perform (eg. email writing, blog generation, code generation etc) in a similar way for generating music we can set predefined hyper-parameters, agents and system prompts to manipulate the style, tempo, melody, lyrics, dynamics and structure as per our need.
You know that for generating music we need proper alignment of all above mentioned musical parameters. This alignment is actually the optimization and fine-tuning of the hyper-parameters of LLM. This is a quite experimental procedure to optimize our model. In case of music it is even more difficult than generating text because you have to adjust lot's of constraints to generate proper music else it may generate noise instead of music. This noise is referred to as hallucinations of LLMs.
Note:- Parameters and Hyper Parameters are 2 different terms used while working with LLMs. To fine-tune & optimize our model we adjust the hyper parameters and not the parameters. So for generating optimized music we would be fine-tuning hyper-parameters. I feel generating music is all about fine-tuning our LLM and removing hallucinations.

[pikurasa]

The list I gave above are some parameters that would be important, and they can be prompted wither explicitly or implicitly.

For example, if someone says, "Make me a blues", there's a lot that that can be inferred from just that short sentence, such as form (12 or 16 bars), chord progression (primarily I, IV, and V, in a particular pattern), instrumentation (probably guitar, voice, and drums), melody (use of a blues scale in a way that compliments the chord progression), and emotion (sad, but "must go on").

Other prompts a user may give may need more information, such as, "Make me a song about potatoes". The LLM could kind of riff on that, or it could ask follow up questions, like "That sounds great. Any particular style of music, such as pop, rock, or jazz?"

And then, once the LLM serves up some music, I think it would be nice if all the choices for each parameter were summarized, such as: "Ok, I'm done with a song about potatoes. I made it in the style of pop, as you requested. To make it sound especially poppy, I chose a lot of electronic synths in the background. The form is: Intro, Verse, Pre-chorus, Chorus, Verse, Pre-chorus, Chorus, Outro. I chose the key of Bb major, because that sounded a little jazzier. [Blah, blah, blah, blah, blah]." And then it could ask some follow up questions, like "Do you like it? Feel free to change anything you feel could be improved in Music Blocks. I coded it with Moveable Do, so changing the key, for example, is easy." (or something like that.)

Doing it this way, I think, would help make the creation process more educational, as the user would start to pick up an the language of some of these parameters, and see that they could be changed via their code.

[pikurasa]

And then I asked it about adding computation to the mix...

Me:
And what if you were planning to add computation to the mix? What secondary "coding-specifc" parameters may be good for a user to specify?

EduBot:
Adding computation to the music creation process opens up a new set of parameters that users might specify. These could include:

1. Algorithm Type: The user might choose between different algorithmic composition techniques, such as Markov chains, genetic algorithms, neural networks, or rule-based systems.

2. Randomness: The user could specify the amount of randomness or unpredictability in the generated music. This could be a global setting, or it could vary for different aspects of the music.

3. Training Data: If the system uses machine learning, the user might specify the training data. For example, they could train the system on a specific genre, composer, or personal music library.

4. Constraints: The user could specify constraints that the generated music should adhere to. For example, they might specify a maximum and minimum pitch, a specific rhythmic pattern, or a set of allowed chords.

5. Interaction: If the system is interactive, the user might specify how they will interact with it. For example, they might choose to input a melody and have the system generate a harmony, or they might choose to generate music in real-time based on their movements or other data.

6. Post-Processing: The user might specify how the generated music should be post-processed. For example, they might choose to quantize the rhythms, apply a specific sound effect, or adjust the dynamics.

7. Output Format: The user might choose the format of the generated music. This could be sheet music, a MIDI file, an audio file, or a live performance.

Remember, these are just possibilities. The specific parameters would depend on the capabilities of the system and the goals of the user.

[kshitijdshah99]

Yeah, this explanation seems quite up to the mark from what we are expecting from the model. Just for your reference try referring to ChatMusician its a LLM for understanding and generating music. It's accompanied with examples of some generated music notes.

These are the configurations and system prompts we can pre-define in our model to generate music pertaining to our needs.
generation_config = GenerationConfig(
temperature=0.2,
top_k=40,
top_p=0.9,
do_sample=True,
num_beams=1,
repetition_penalty=1.1,
min_new_tokens=10,
max_new_tokens=1536
)

instruction = """Develop a musical piece using the given chord progression.
'Dm', 'C', 'Dm', 'Dm', 'C', 'Dm', 'C', 'Dm'
"""
I hope this helps!!

[nilaygit-10721]

I’ve been following this discussion, and it’s fascinating how we’re exploring ways to combine generative AI with music ideation. I’d like to share some thoughts and ideas that might add value to this conversation:
->Instead of static prompts, the system could have a conversational interface. It could progressively refine the parameters by asking follow-up questions based on the user’s initial input, much like how “Make me a song about potatoes” was discussed.
->Allow users to tweak parameters in real-time (e.g., adjusting tempo or key live) and hear the immediate impact on the music. This would be highly educational for beginners learning musical concepts.
->Letting users experiment with algorithm types (e.g., Markov chains vs. neural networks) through a visual or simplified coding interface could make this approachable for users with varying coding expertise.

[pikurasa]

Of the three you mention, the last seems the most promising: "Letting users experiment with algorithm types (e.g., Markov chains vs. neural networks) through a visual or simplified coding interface could make this approachable for users with varying coding expertise."

This would help this be of educational value, IMO.

As for the first comment, yes, I suppose that could add some value. We do kind of have some of this to varying degrees in our gen-AI projects

As for tweaking parameters, we kind of have that already to a certain extent.

At any rate, I'm interested to explore how we might expose Markov chains vs. neural networks vs. LLMs to a learner.

[ac-mmi]

@pikurasa I have made some progress with getting GPT-4 to help with musical ideas in Music Blocks. I set up a Node.js backend that works with the model. It takes user input from turtle-singer.js, sends it to GPT-4, and then organizes the output into JSON with notes, percussion, and movements.

By tweaking the prompts and organizing the outputs better, I got more consistent musical sequences. Now, the model generates melodies, drum patterns, and turtle movements that sync up nicely. Plus, the organized JSON makes it easier to work with the outputs in the code.

One issue I ran into was keeping the music consistent throughout each sequence. Adjusting the prompts helped, but I’m still trying to figure out how to keep a theme going or add variations without making the music sound random.

Based on this setup, what do you think would be the best next step to improve the musical flow? And do you have any suggestion on how we could experiment with different generation methods (like Markov chains or neural networks) to make the music more interesting?

[pikurasa]

Hi @ac-mmi

I recommend you join us on Matrix at https://matrix.to/#/#musicblocksdev:matrix.org

Also, are you aware of https://github.com/sugarlabs/GSoC/blob/master/Ideas-2025.md ?

[ac-mmi]

Yes already been working on internationalisation js idea ,sent a pr

[ac-mmi]

@pikurasa In my Demo the user chooses a few random notes on the PhraseMaker grid.The AI then completes the phrase in a way that forms a meaningful melody rather than just random sounds.Right now, percussion and movements are mandatory, ensuring rhythm and structure.You could make them optional to allow more flexibility in experimentation.

To improve AI-generated music, we can use Markov Chains to predict the next note based on past sequences, ensuring patterns emerge naturally. Also usin neural networks we could make it learn famous composition to make it more engaging and fun for kids.

demo.mp4

Disclaimer: Sorry if the generated composition isn’t always melodic—I'm still refining the model. With some tweaks, I can improve it to generate more meaningful music. I'm also researching open-source models to enhance the AI’s understanding of musical patterns and theory.

[kshitijdshah99]

@ac-mmi This looks pretty cool but I didn't get this point "Also usin neural networks we could make it learn famous composition to make it more engaging and fun for kids.". I really doubt if neural networks can perform so well.

[ac-mmi]

@kshitijdshah99 combining NN them with rule-based systems or probability models will make AI Music more structured and recognizable.

[pikurasa]

In my humble experience: I think NN would be sufficient for this sort of task -- and may even be preferable (as it is simpler and uses fewer resources.

Suggestion: Please remember that the rhythm and pitch space for the Phrase Maker can be changed. Perhaps experiment with longer lengths and larger ranges in pitch.