{"title":"8.6 Music Generation","description":"","content":"# 8.6 Music Generation\n\n## The Problem\n\nThink about the last time you watched a short-form video, played a mobile game, or sat through an ad. Now imagine the music was missing. The content falls flat. Music sets the emotional tone, controls pacing, and makes content memorable. It is one of the most important production elements, and also one of the hardest to source.\n\nIf you are building a content creation platform, you have three options for background music. You can license existing tracks, which is expensive and legally complex. You can use royalty-free libraries, but those tracks are generic and overused. Or you can hire a composer, which is slow and does not scale. None of these options work well when your users need unique, on-demand audio that matches a specific mood, scene, or brand.\n\nThis is where AI music generation changes the equation. In the last two years, models like MusicGen, Stable Audio, Suno, and Udio have gone from producing toy-like MIDI sequences to generating full, multi-instrument tracks that sound genuinely professional. A text prompt like \"upbeat lo-fi hip hop with soft piano and vinyl crackle, 90 BPM\" produces a usable 30-second clip in under a minute.\n\nFor AI engineers, this opens up a new class of product features. A video editor that auto-generates soundtracks from scene descriptions. A game engine that creates adaptive music based on gameplay state. A meditation app that produces personalized ambient tracks based on user mood. The underlying technology is accessible through APIs, which means you do not need to be an audio engineer to integrate it.\n\nBut generating good music from text is harder than generating good images. Music has temporal structure, harmonic rules, rhythm, and cultural expectations that make the problem uniquely challenging. Understanding how these models work, where they fail, and how to prompt them effectively will help you build products that actually sound good.\n\n**What you will learn:**\n\n- How music generation models represent and produce audio (spectrograms, audio diffusion, codec-based generation)\n- The major platforms and models: MusicGen, Stable Audio, Suno, Udio\n- Prompt engineering techniques for controlling genre, mood, tempo, and instrumentation\n- Stem separation and remixing for post-processing generated audio\n- The legal and copyright landscape for AI-generated music\n- Integration patterns for adding music generation to your applications\n\n---\n\n## How Music Generation Models Work\n\nBefore we look at specific tools, you need to understand how a model turns text into audio. This is not as straightforward as text-to-image generation, because audio has a unique challenge: it is a continuous signal sampled at high frequency. A 10-second audio clip at CD quality (44,100 Hz, stereo) contains nearly 900,000 individual sample values. Generating that sample-by-sample would be painfully slow.\n\nMusic generation models solve this by working in compressed representations rather than raw audio. There are three main approaches, and modern systems often combine them.\n\n### Spectrogram-Based Generation\n\nThe oldest approach treats audio as an image. A spectrogram is a visual representation of sound where the x-axis is time, the y-axis is frequency, and the color intensity represents amplitude. If you convert audio into a spectrogram, you can use image generation techniques (like diffusion models) to create new spectrograms, then convert them back to audio.\n\n\n```mermaid\nflowchart LR\n A[Text Prompt]:::primary --> B[Text Encoder]:::teal\n B --> C[Diffusion Model]:::orange\n C --> D[Mel Spectrogram]:::purple\n D --> E[Vocoder]:::green\n E --> F[Audio Waveform]:::primary\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe text prompt gets encoded into a numerical representation. A diffusion model (similar to those used in image generation) progressively denoises a random spectrogram, guided by the text embedding, until a clean spectrogram emerges. Then a vocoder (a neural network trained to convert spectrograms back to waveforms) produces the final audio.\n\nStable Audio uses this approach. It works well for ambient textures and sound effects, but spectrograms lose some fine-grained detail, which can make the output sound slightly \"washy\" on complex arrangements.\n\n### Neural Audio Codec Generation\n\nThe more recent and powerful approach, used by Meta's MusicGen, avoids spectrograms entirely. Instead, it compresses raw audio into discrete tokens using a neural audio codec (like EnCodec). Think of it as \"tokenizing\" audio the same way a text model tokenizes words.\n\n\n```mermaid\nflowchart TD\n A[Raw Audio]:::primary --> B[EnCodec Encoder]:::teal\n B --> C[Discrete Audio Tokens]:::orange\n C --> D[Transformer Model]:::purple\n D --> E[New Audio Tokens]:::orange\n E --> F[EnCodec Decoder]:::teal\n F --> G[Generated Audio]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nEnCodec compresses audio into a sequence of codebook tokens at multiple levels. The lower levels capture coarse structure (rhythm, melody, harmony) and the higher levels add fine detail (timbre, texture). MusicGen uses a transformer that is conditioned on text to generate these tokens autoregressively, one step at a time, just like a language model generates text tokens.\n\nThis approach produces higher-quality results than spectrogram methods because the codec preserves more audio detail. It is also faster, since the token sequence is much shorter than the raw sample count.\n\n### End-to-End Diffusion on Latent Audio\n\nStable Audio 2 and some newer models combine both ideas. They use a variational autoencoder (VAE) to compress audio into a latent space, then run a diffusion process in that latent space (similar to how Stable Diffusion works for images). The result gets decoded back to audio through the VAE decoder.\n\nThe key difference from spectrogram-based methods is that the latent space is learned end-to-end, so it captures whatever audio features matter most, rather than being constrained by the fixed time-frequency grid of a spectrogram.\n\n**Exercise:** Listen to a few samples from MusicGen and Stable Audio (both have free demos online). Can you hear differences in quality? Pay attention to how cleanly instruments are separated, whether the rhythm stays consistent, and how the track handles transitions. Write down which model sounds better for different genres.\n\n---\n\n## The Model Landscape\n\nThe music generation space is moving fast. Here is where the major players stand and what each is best suited for.\n\n\n| Model / Platform | Type | Access | Strengths | Limitations |\n|---|---|---|---|---|\n| MusicGen (Meta) | Open-source, codec-based | Hugging Face, local | Free, customizable, good for instrumentals | 30s max, no vocals, requires GPU |\n| Stable Audio | Latent diffusion | API + open weights | Long-form (up to 3 min), variable length | Weaker on complex arrangements |\n| Suno | Proprietary | Web + API | Full songs with vocals and lyrics, very high quality | Closed-source, subscription pricing |\n| Udio | Proprietary | Web + API | Realistic vocals, genre diversity | Closed-source, subscription pricing |\n\n\n### MusicGen (Meta)\n\nMusicGen is the most accessible option for developers. It is open-source, runs locally, and is available through Hugging Face's transformers library. Meta released four model sizes: small (300M parameters), medium (1.5B), large (3.3B), and melody (1.5B, conditioned on a melody input).\n\nThe trade-off is clear: MusicGen produces instrumentals only (no vocals), clips max out at 30 seconds, and the small model sounds noticeably thin. The large model produces solid results for background music, ambient tracks, and instrumental pieces.\n\n\n```python\n# Generate music with MusicGen using Hugging Face transformers\nfrom audiocraft.models import MusicGen\nfrom audiocraft.data.audio import audio_write\n\n# Load the model (options: small, medium, large, melody)\nmodel = MusicGen.get_pretrained(\"facebook/musicgen-large\")\n\n# Set generation parameters\nmodel.set_generation_params(\n duration=15, # Duration in seconds (max 30)\n top_k=250, # Top-k sampling for diversity\n top_p=0.0, # Nucleus sampling (0 = disabled)\n temperature=1.0, # Higher = more creative, lower = more conservative\n cfg_coeff=3.0 # Classifier-free guidance strength\n)\n\n# Generate from a text description\ndescriptions = [\n \"upbeat lo-fi hip hop with soft piano chords and vinyl crackle, 85 BPM\",\n \"dark ambient electronic with deep bass and ethereal pads, slow tempo\",\n \"acoustic folk guitar fingerpicking, warm and nostalgic, medium tempo\"\n]\n\n# Generate one clip per description\nwav = model.generate(descriptions)\n\n# Save each generated clip\nfor i, one_wav in enumerate(wav):\n audio_write(\n f\"output_track_{i}\",\n one_wav.cpu(),\n model.sample_rate,\n strategy=\"loudness\",\n loudness_compressor=True\n )\n print(f\"Saved track {i}: {descriptions[i][:50]}...\")\n```\n\n\nThe `cfg_coeff` parameter (classifier-free guidance) is worth understanding. Higher values make the model follow your text description more closely, but can reduce audio quality and diversity. Lower values produce more natural-sounding music but may drift from your prompt. A value between 3.0 and 5.0 is a good starting range.\n\n### Suno and Udio\n\nSuno and Udio are the heavyweights for full song generation. Both can produce complete tracks with vocals, lyrics, multiple instruments, and song structure (verse, chorus, bridge). The quality is remarkable, often indistinguishable from human-produced music for casual listeners.\n\nThe catch is that both are closed-source and API access requires a subscription. Suno's API provides programmatic access to song generation, which is what you would use in a production application.\n\n\n```python\n# Example: Generating music with Suno's API\n# Note: Requires a Suno API key (paid subscription)\nimport requests\nimport time\n\nSUNO_API_BASE = \"https://api.suno.ai/v1\"\nSUNO_API_KEY = \"your-api-key-here\"\n\nheaders = {\n \"Authorization\": f\"Bearer {SUNO_API_KEY}\",\n \"Content-Type\": \"application/json\"\n}\n\ndef generate_song(prompt, style, duration=30):\n \"\"\"Generate a song using Suno's API.\"\"\"\n payload = {\n \"prompt\": prompt,\n \"style\": style,\n \"duration\": duration,\n \"instrumental\": False # Set True for no vocals\n }\n\n # Start generation\n response = requests.post(\n f\"{SUNO_API_BASE}/generate\",\n headers=headers,\n json=payload\n )\n task_id = response.json()[\"task_id\"]\n print(f\"Generation started: {task_id}\")\n\n # Poll for completion\n while True:\n status = requests.get(\n f\"{SUNO_API_BASE}/tasks/{task_id}\",\n headers=headers\n ).json()\n\n if status[\"status\"] == \"completed\":\n return status[\"audio_url\"]\n elif status[\"status\"] == \"failed\":\n raise Exception(f\"Generation failed: {status.get('error')}\")\n\n time.sleep(5)\n\n# Generate a track\naudio_url = generate_song(\n prompt=\"A chill sunset drive through empty highways\",\n style=\"synthwave, retrowave, dreamy, 100 BPM\"\n)\nprint(f\"Audio available at: {audio_url}\")\n```\n\n\n**Exercise:** If you have access to Suno or Udio, generate the same concept (\"relaxing jazz for a coffee shop\") on both platforms. Compare the results: Which handles the instruments better? Which produces more natural-sounding audio? How do the vocals differ?\n\n---\n\n## Prompt Engineering for Music\n\nPrompting a music model is different from prompting a language model. You are describing an auditory experience, not asking a question. The model needs concrete musical parameters, not abstract descriptions. \"Make something cool\" produces mediocre results. Specific, structured prompts produce dramatically better output.\n\n### The Anatomy of a Good Music Prompt\n\nA strong music prompt has five components:\n\n\n```mermaid\nflowchart LR\n A[Genre]:::primary --> F[Complete Prompt]:::green\n B[Mood/Energy]:::orange --> F\n C[Instruments]:::teal --> F\n D[Tempo/BPM]:::purple --> F\n E[Production Style]:::red --> F\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n**Genre** gives the model a musical framework. \"Lo-fi hip hop,\" \"orchestral cinematic,\" or \"80s synthwave\" each activate very different patterns in the model.\n\n**Mood and energy** control the emotional feel. \"Melancholic,\" \"triumphant,\" \"tense,\" or \"playful\" shape the harmonic choices and dynamics.\n\n**Instruments** specify what you want to hear. \"Soft piano, muted trumpet, upright bass\" is far more useful than \"jazz instruments.\"\n\n**Tempo** anchors the rhythm. Specifying BPM (beats per minute) gives the model a concrete target. 60-80 BPM is slow and relaxed, 100-120 is moderate, 140+ is energetic.\n\n**Production style** describes the sonic texture. \"Lo-fi with vinyl crackle,\" \"clean studio recording,\" \"reverb-heavy atmospheric,\" or \"raw and gritty\" all shape the final sound.\n\n### Prompt Comparison\n\nHere is how prompt specificity affects output quality:\n\n\n| Prompt Quality | Example | Expected Result |\n|---|---|---|\n| Vague | \"happy music\" | Generic, unfocused, could be anything |\n| Better | \"upbeat pop music, happy, fast\" | Recognizable genre but still generic |\n| Good | \"upbeat indie pop with acoustic guitar, handclaps, and bright synths, 120 BPM, feel-good summer vibe\" | Focused, clear musical direction |\n| Excellent | \"indie pop, acoustic guitar strumming 8th notes, layered handclaps on beats 2 and 4, bright analog synth pad, female vocal oohs, 122 BPM, warm lo-fi production, inspired by MGMT\" | Specific enough to produce consistent, high-quality results |\n\n\n### Building Prompts Programmatically\n\nIn a real application, you would not ask users to write detailed music prompts. Instead, you map user-friendly inputs (mood, energy, use case) to structured prompts behind the scenes.\n\n\n```python\nfrom dataclasses import dataclass\nfrom typing import Optional\n\n@dataclass\nclass MusicRequest:\n mood: str # \"happy\", \"sad\", \"tense\", \"peaceful\"\n energy: str # \"low\", \"medium\", \"high\"\n use_case: str # \"background\", \"intro\", \"workout\", \"meditation\"\n duration: int = 15 # seconds\n genre: Optional[str] = None # Override auto-detected genre\n\n# Map moods and energy levels to musical parameters\nMOOD_MAP = {\n \"happy\": {\n \"descriptors\": \"bright, uplifting, warm, cheerful\",\n \"harmony\": \"major key\",\n \"instruments_default\": \"acoustic guitar, light percussion, soft synth pads\"\n },\n \"sad\": {\n \"descriptors\": \"melancholic, gentle, bittersweet, reflective\",\n \"harmony\": \"minor key\",\n \"instruments_default\": \"solo piano, soft strings, ambient pads\"\n },\n \"tense\": {\n \"descriptors\": \"dark, suspenseful, building, uneasy\",\n \"harmony\": \"dissonant, chromatic\",\n \"instruments_default\": \"low cello drone, sparse percussion, distorted textures\"\n },\n \"peaceful\": {\n \"descriptors\": \"calm, serene, floating, meditative\",\n \"harmony\": \"open chords, pentatonic\",\n \"instruments_default\": \"soft piano, nature sounds, gentle flute, ambient pads\"\n }\n}\n\nENERGY_TO_BPM = {\n \"low\": \"60-75 BPM\",\n \"medium\": \"90-110 BPM\",\n \"high\": \"120-140 BPM\"\n}\n\nUSE_CASE_STYLE = {\n \"background\": \"subtle, non-intrusive, suitable for background listening\",\n \"intro\": \"attention-grabbing opening, builds momentum in the first 5 seconds\",\n \"workout\": \"driving rhythm, heavy beat, motivating energy\",\n \"meditation\": \"ambient, spacious, minimal, no sudden changes\"\n}\n\ndef build_music_prompt(request: MusicRequest) -> str:\n \"\"\"Convert a user-friendly music request into a detailed generation prompt.\"\"\"\n mood_info = MOOD_MAP[request.mood]\n\n genre = request.genre or \"cinematic\"\n bpm = ENERGY_TO_BPM[request.energy]\n style = USE_CASE_STYLE[request.use_case]\n\n prompt = (\n f\"{genre}, {mood_info['descriptors']}, {mood_info['harmony']}, \"\n f\"{mood_info['instruments_default']}, {bpm}, \"\n f\"{style}\"\n )\n return prompt\n\n# Example usage\nrequest = MusicRequest(mood=\"tense\", energy=\"medium\", use_case=\"background\")\nprompt = build_music_prompt(request)\nprint(prompt)\n# cinematic, dark, suspenseful, building, uneasy, dissonant, chromatic,\n# low cello drone, sparse percussion, distorted textures, 90-110 BPM,\n# subtle, non-intrusive, suitable for background listening\n```\n\n\nThis pattern decouples your user interface from the underlying model's prompt format. When models improve or you switch providers, you only need to update the prompt builder, not your entire application.\n\n**Exercise:** Extend the `MusicRequest` class to include a `scene_description` field (e.g., \"a rainy city at night\" or \"a spaceship launching\"). Write a function that uses an LLM to convert the scene description into musical parameters (mood, energy, genre, instruments) before passing them to `build_music_prompt`. Test with five different scenes and evaluate whether the generated prompts capture the right feeling.\n\n---\n\n## Stem Separation and Remixing\n\nGenerated music often needs post-processing. Maybe you want to remove the drums to create a softer version, isolate a melody line to layer over different backing, or extract vocals from a generated song to remix with a different instrumental. This is where stem separation comes in.\n\nStem separation (also called source separation) splits a mixed audio track into individual components: vocals, drums, bass, and other instruments. The most widely used open-source tool for this is Demucs, developed by Meta.\n\n\n```mermaid\nflowchart LR\n A[Mixed Audio Track]:::primary --> B[Demucs Model]:::orange\n B --> C[Vocals]:::teal\n B --> D[Drums]:::purple\n B --> E[Bass]:::green\n B --> F[Other
Instruments]:::red\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n### Using Demucs for Stem Separation\n\n\n```python\n# Separate a music track into stems using Demucs\nimport torch\nimport torchaudio\nfrom demucs.pretrained import get_model\nfrom demucs.apply import apply_model\n\ndef separate_stems(audio_path: str, output_dir: str = \"stems\"):\n \"\"\"Separate an audio file into vocals, drums, bass, and other.\"\"\"\n import os\n os.makedirs(output_dir, exist_ok=True)\n\n # Load the pre-trained model (htdemucs is the best quality option)\n model = get_model(\"htdemucs\")\n model.eval()\n\n # Load audio\n wav, sr = torchaudio.load(audio_path)\n\n # Resample to model's expected sample rate if needed\n if sr != model.samplerate:\n wav = torchaudio.transforms.Resample(sr, model.samplerate)(wav)\n\n # Add batch dimension and run separation\n wav = wav.unsqueeze(0) # Shape: (1, channels, samples)\n\n with torch.no_grad():\n sources = apply_model(model, wav)\n\n # sources shape: (1, num_sources, channels, samples)\n # Default sources: drums, bass, other, vocals\n source_names = [\"drums\", \"bass\", \"other\", \"vocals\"]\n\n saved_paths = {}\n for i, name in enumerate(source_names):\n stem = sources[0, i] # Shape: (channels, samples)\n output_path = os.path.join(output_dir, f\"{name}.wav\")\n torchaudio.save(output_path, stem.cpu(), model.samplerate)\n saved_paths[name] = output_path\n print(f\"Saved {name} stem to {output_path}\")\n\n return saved_paths\n\n# Separate a generated track\nstems = separate_stems(\"output_track_0.wav\")\n```\n\n\n### Remixing Stems\n\nOnce you have separated stems, you can recombine them with different volume levels, apply effects, or swap stems between tracks. This is useful for creating variations of generated music without re-generating from scratch.\n\n\n```python\nimport torchaudio\nimport torch\n\ndef remix_stems(stem_paths: dict, mix_levels: dict, output_path: str):\n \"\"\"\n Remix separated stems with custom volume levels.\n\n Args:\n stem_paths: Dict mapping stem name to file path\n mix_levels: Dict mapping stem name to volume multiplier (0.0 to 2.0)\n output_path: Where to save the remixed audio\n \"\"\"\n mixed = None\n sample_rate = None\n\n for name, path in stem_paths.items():\n wav, sr = torchaudio.load(path)\n sample_rate = sr\n\n # Apply volume level\n level = mix_levels.get(name, 1.0)\n wav = wav * level\n\n if mixed is None:\n mixed = wav\n else:\n # Ensure same length (trim to shorter)\n min_len = min(mixed.shape[1], wav.shape[1])\n mixed = mixed[:, :min_len] + wav[:, :min_len]\n\n # Normalize to prevent clipping\n peak = mixed.abs().max()\n if peak > 1.0:\n mixed = mixed / peak\n\n torchaudio.save(output_path, mixed, sample_rate)\n print(f\"Saved remix to {output_path}\")\n\n# Create a \"chill\" version: keep bass and other, reduce drums, remove vocals\nremix_stems(\n stem_paths=stems,\n mix_levels={\n \"drums\": 0.3, # Quiet drums\n \"bass\": 1.0, # Full bass\n \"other\": 1.2, # Boost melody/harmony\n \"vocals\": 0.0 # Remove vocals entirely\n },\n output_path=\"remix_chill.wav\"\n)\n```\n\n\nStem separation is not perfect. You will hear artifacts, especially on complex mixes where instruments overlap in frequency range. But for post-processing AI-generated music, where the original mix is relatively clean, the results are usually good enough for production use.\n\n**Exercise:** Generate a track with MusicGen, separate it into stems using Demucs, then create three remixes: one with only drums and bass (rhythm-only), one with only the melodic elements (no drums), and one where you boost the bass by 50% and reduce everything else. Listen to all three. How clean is the separation?\n\n---\n\n## Licensing and Copyright\n\nThis is the section that will save you from legal trouble. The copyright landscape for AI-generated music is unsettled, evolving, and varies by jurisdiction. Here is what you need to know as an engineer building products with music generation.\n\n**Who owns AI-generated music?** In the United States, the Copyright Office has ruled that works created entirely by AI, with no meaningful human creative input, cannot be copyrighted. This means a raw output from Suno with no human editing may not be protectable. However, if a human selects, arranges, curates, and modifies the output, the resulting work may qualify for copyright protection. The exact threshold is still being defined through case law.\n\n**Training data lawsuits.** Several major music labels have filed lawsuits against Suno and Udio, alleging that these models were trained on copyrighted music without permission. The outcome of these cases will significantly shape the legal landscape. As of early 2026, these cases are ongoing.\n\n**Platform-specific terms.** Each platform has different licensing terms for generated content:\n\n\n| Platform | Commercial Use | Ownership | Key Restrictions |\n|---|---|---|---|\n| MusicGen | Yes (MIT license) | You own outputs | No restrictions on model outputs |\n| Stable Audio | Depends on plan | Varies by tier | Free tier: non-commercial only |\n| Suno | Yes (paid plans) | You own outputs on paid plans | Free tier: Suno retains rights |\n| Udio | Yes (paid plans) | You own outputs on paid plans | Free tier: limited commercial use |\n\n\n**Practical recommendations for production applications:**\n\n1. **Use open-source models when possible.** MusicGen's MIT license gives you the clearest legal standing. You own the outputs with no strings attached.\n2. **Keep generation logs.** Record the prompts, parameters, and timestamps for every piece of generated music. If ownership is ever questioned, you want documentation of the creative process.\n3. **Add human curation.** Select, edit, arrange, and modify generated outputs. This strengthens your copyright claim and generally improves the quality anyway.\n4. **Check platform terms before launch.** Terms change frequently. Verify that your intended commercial use is covered by your current subscription tier.\n5. **Avoid reproducing specific artists.** Prompting for \"a song that sounds exactly like [specific artist]\" is a legal gray area. Use genre and style descriptors instead of artist names.\n\n---\n\n## Integration Patterns\n\nWhen you are adding music generation to an application, the architecture matters as much as the model choice. Music generation is computationally expensive and slow compared to text generation. A 30-second clip can take 30-120 seconds to generate depending on the model and hardware. You need to design around this latency.\n\n### Pattern 1: Async Generation with Job Queue\n\nThe most common pattern for production applications. The user submits a request, gets back a job ID, and polls (or receives a webhook) when the audio is ready.\n\n\n```mermaid\nflowchart LR\n A[User Request]:::primary --> B[API Server]:::teal\n B --> C[Job Queue]:::orange\n C --> D[GPU Worker]:::purple\n D --> E[Object Storage]:::green\n E --> F[Audio URL]:::primary\n F --> A\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n\n```python\nimport uuid\nimport json\nfrom dataclasses import dataclass, asdict\nfrom datetime import datetime\nfrom typing import Optional\n\n# In a real application, use Redis, RabbitMQ, or a cloud queue service\n# This is a simplified in-memory example to show the pattern\n\n@dataclass\nclass MusicJob:\n job_id: str\n prompt: str\n duration: int\n status: str = \"pending\" # pending, processing, completed, failed\n audio_url: Optional[str] = None\n created_at: str = \"\"\n error: Optional[str] = None\n\n# Simulated job store (use Redis or a database in production)\njobs: dict[str, MusicJob] = {}\n\ndef submit_generation_job(prompt: str, duration: int = 15) -> str:\n \"\"\"Submit a music generation job and return the job ID.\"\"\"\n job = MusicJob(\n job_id=str(uuid.uuid4()),\n prompt=prompt,\n duration=duration,\n created_at=datetime.utcnow().isoformat()\n )\n jobs[job.job_id] = job\n\n # In production: push to a message queue (Redis, SQS, RabbitMQ)\n # queue.publish(\"music_generation\", asdict(job))\n print(f\"Job submitted: {job.job_id}\")\n return job.job_id\n\ndef get_job_status(job_id: str) -> dict:\n \"\"\"Check the status of a generation job.\"\"\"\n job = jobs.get(job_id)\n if not job:\n return {\"error\": \"Job not found\"}\n return asdict(job)\n\ndef process_job(job_id: str):\n \"\"\"Worker function that processes a music generation job.\n In production, this runs on a GPU-equipped worker.\"\"\"\n job = jobs[job_id]\n job.status = \"processing\"\n\n try:\n # Generate music (using MusicGen or an API call)\n prompt = job.prompt\n # audio = generate_with_musicgen(prompt, job.duration)\n # audio_url = upload_to_storage(audio)\n\n # Simulated result\n audio_url = f\"https://storage.example.com/music/{job_id}.wav\"\n\n job.status = \"completed\"\n job.audio_url = audio_url\n except Exception as e:\n job.status = \"failed\"\n job.error = str(e)\n\n# Usage\njob_id = submit_generation_job(\n prompt=\"ambient electronic, calm, soft pads, 70 BPM\",\n duration=30\n)\nprocess_job(job_id)\nprint(get_job_status(job_id))\n```\n\n\n### Pattern 2: Pre-Generated Library with Semantic Search\n\nFor applications where latency matters more than uniqueness, pre-generate a library of tracks and use semantic search to find the best match for a user's request.\n\n\n```python\nfrom openai import OpenAI\nimport numpy as np\n\nclient = OpenAI()\n\n# Pre-generated library: each track has a text description and embedding\ntrack_library = [\n {\"id\": \"track_001\", \"description\": \"upbeat pop with acoustic guitar and handclaps, 120 BPM, summer vibe\"},\n {\"id\": \"track_002\", \"description\": \"dark ambient electronic with deep bass, 70 BPM, mysterious\"},\n {\"id\": \"track_003\", \"description\": \"classical piano solo, melancholic, slow, rubato timing\"},\n {\"id\": \"track_004\", \"description\": \"lo-fi hip hop, mellow, vinyl crackle, soft piano, 85 BPM\"},\n {\"id\": \"track_005\", \"description\": \"epic orchestral, triumphant, full brass and strings, 140 BPM\"},\n]\n\ndef embed_text(text: str) -> list[float]:\n \"\"\"Get embedding for a text description.\"\"\"\n response = client.embeddings.create(\n model=\"text-embedding-3-small\",\n input=text\n )\n return response.data[0].embedding\n\ndef cosine_similarity(a: list[float], b: list[float]) -> float:\n a_np, b_np = np.array(a), np.array(b)\n return float(np.dot(a_np, b_np) / (np.linalg.norm(a_np) * np.linalg.norm(b_np)))\n\n# Pre-compute embeddings for all tracks\nfor track in track_library:\n track[\"embedding\"] = embed_text(track[\"description\"])\n\ndef find_matching_tracks(query: str, top_k: int = 3) -> list[dict]:\n \"\"\"Find the best matching tracks for a user's mood or scene description.\"\"\"\n query_embedding = embed_text(query)\n\n scored = []\n for track in track_library:\n score = cosine_similarity(query_embedding, track[\"embedding\"])\n scored.append({\"track\": track, \"score\": score})\n\n scored.sort(key=lambda x: x[\"score\"], reverse=True)\n return scored[:top_k]\n\n# User describes a scene, we find matching pre-generated tracks\nresults = find_matching_tracks(\"a quiet rainy evening reading a book\")\nfor r in results:\n print(f\"Score: {r['score']:.3f} | {r['track']['description']}\")\n```\n\n\nThis hybrid approach works well in practice. You pre-generate hundreds or thousands of tracks offline (where latency does not matter), then serve them instantly via semantic search. For edge cases where no pre-generated track is close enough, fall back to on-demand generation.\n\n**Exercise:** Design the architecture for a podcast editing tool that needs background music. The user uploads a podcast episode, and the system automatically suggests background music that matches the mood of different segments. What combination of the patterns above would you use? Sketch the data flow and identify where each API call happens.\n\n---\n\n## Putting It All Together\n\nLet's build the mood-based soundtrack generator from the module exercise. The user describes a scene or mood, the system generates three music clips using different prompt strategies, and a rating system lets you compare which approach works best.\n\n\n```python\nimport uuid\nimport json\nfrom dataclasses import dataclass, field\nfrom datetime import datetime\nfrom typing import Optional\nfrom audiocraft.models import MusicGen\nfrom audiocraft.data.audio import audio_write\n\n# --- Prompt Strategy Definitions ---\n\ndef strategy_direct(scene: str) -> str:\n \"\"\"Strategy 1: Pass the scene description directly as the prompt.\"\"\"\n return scene\n\ndef strategy_structured(scene: str) -> str:\n \"\"\"Strategy 2: Map the scene to structured musical parameters.\"\"\"\n # In production, use an LLM to extract mood/energy/genre from the scene\n # Here we use a simplified keyword-based approach\n scene_lower = scene.lower()\n\n if any(w in scene_lower for w in [\"rain\", \"night\", \"dark\", \"quiet\"]):\n return (\"ambient electronic, melancholic, soft pads, gentle rain texture, \"\n \"minor key, 70 BPM, atmospheric, reverb-heavy\")\n elif any(w in scene_lower for w in [\"battle\", \"chase\", \"action\", \"fight\"]):\n return (\"orchestral action, intense, driving percussion, brass stabs, \"\n \"fast tempo, 150 BPM, cinematic, powerful\")\n elif any(w in scene_lower for w in [\"happy\", \"sun\", \"beach\", \"summer\"]):\n return (\"indie pop, upbeat, acoustic guitar, bright synths, handclaps, \"\n \"major key, 120 BPM, warm production\")\n else:\n return (f\"cinematic soundtrack for: {scene}, \"\n \"orchestral, emotive, medium tempo, 100 BPM\")\n\ndef strategy_llm_enhanced(scene: str) -> str:\n \"\"\"Strategy 3: Use an LLM to generate a detailed music prompt from the scene.\"\"\"\n from openai import OpenAI\n client = OpenAI()\n\n response = client.chat.completions.create(\n model=\"gpt-4o-mini\",\n messages=[\n {\n \"role\": \"system\",\n \"content\": (\n \"You are a music director. Given a scene description, output a detailed \"\n \"music generation prompt. Include: genre, mood descriptors, specific \"\n \"instruments, tempo in BPM, production style, and harmonic character. \"\n \"Output ONLY the prompt, no explanation.\"\n )\n },\n {\"role\": \"user\", \"content\": f\"Scene: {scene}\"}\n ],\n max_tokens=150,\n temperature=0.7\n )\n return response.choices[0].message.content.strip()\n\n# --- Generation and Rating System ---\n\n@dataclass\nclass GeneratedTrack:\n track_id: str\n scene: str\n strategy: str\n prompt: str\n file_path: str\n rating: Optional[int] = None # 1-5 scale\n notes: Optional[str] = None\n\n@dataclass\nclass SoundtrackSession:\n session_id: str\n scene: str\n tracks: list[GeneratedTrack] = field(default_factory=list)\n created_at: str = \"\"\n\ndef generate_soundtrack_variations(\n scene: str,\n duration: int = 15,\n model_size: str = \"facebook/musicgen-medium\"\n) -> SoundtrackSession:\n \"\"\"Generate 3 music clips for a scene using different prompt strategies.\"\"\"\n\n session = SoundtrackSession(\n session_id=str(uuid.uuid4())[:8],\n scene=scene,\n created_at=datetime.utcnow().isoformat()\n )\n\n # Load model once for all generations\n model = MusicGen.get_pretrained(model_size)\n model.set_generation_params(duration=duration, top_k=250, cfg_coeff=3.5)\n\n strategies = {\n \"direct\": strategy_direct,\n \"structured\": strategy_structured,\n \"llm_enhanced\": strategy_llm_enhanced\n }\n\n for name, strategy_fn in strategies.items():\n prompt = strategy_fn(scene)\n print(f\"\\n--- Strategy: {name} ---\")\n print(f\"Prompt: {prompt[:100]}...\")\n\n # Generate audio\n wav = model.generate([prompt])\n file_path = f\"soundtrack_{session.session_id}_{name}\"\n audio_write(file_path, wav[0].cpu(), model.sample_rate, strategy=\"loudness\")\n\n track = GeneratedTrack(\n track_id=str(uuid.uuid4())[:8],\n scene=scene,\n strategy=name,\n prompt=prompt,\n file_path=f\"{file_path}.wav\"\n )\n session.tracks.append(track)\n print(f\"Saved: {track.file_path}\")\n\n return session\n\ndef rate_track(session: SoundtrackSession, strategy: str, rating: int, notes: str = \"\"):\n \"\"\"Rate a generated track on a 1-5 scale.\"\"\"\n for track in session.tracks:\n if track.strategy == strategy:\n track.rating = rating\n track.notes = notes\n print(f\"Rated '{strategy}': {rating}/5 - {notes}\")\n return\n print(f\"Strategy '{strategy}' not found in session\")\n\ndef compare_results(session: SoundtrackSession):\n \"\"\"Display comparison of all generated tracks and their ratings.\"\"\"\n print(f\"\\n{'='*60}\")\n print(f\"Scene: {session.scene}\")\n print(f\"{'='*60}\")\n\n for track in session.tracks:\n rating_str = f\"{track.rating}/5\" if track.rating else \"Not rated\"\n print(f\"\\nStrategy: {track.strategy}\")\n print(f\" Prompt: {track.prompt[:80]}...\")\n print(f\" File: {track.file_path}\")\n print(f\" Rating: {rating_str}\")\n if track.notes:\n print(f\" Notes: {track.notes}\")\n\n rated = [t for t in session.tracks if t.rating is not None]\n if rated:\n best = max(rated, key=lambda t: t.rating)\n print(f\"\\nBest strategy: {best.strategy} ({best.rating}/5)\")\n\n# --- Run the full pipeline ---\n\n# Generate three variations for a scene\nsession = generate_soundtrack_variations(\n scene=\"a lonely astronaut floating through the rings of Saturn\",\n duration=15\n)\n\n# After listening to each track, rate them\nrate_track(session, \"direct\", 2, \"Too literal, model did not understand the visual\")\nrate_track(session, \"structured\", 3, \"Good mood but generic, could be any space scene\")\nrate_track(session, \"llm_enhanced\", 4, \"Best match, captured the loneliness and wonder\")\n\n# Compare all results\ncompare_results(session)\n```\n\n\nThis system gives you a framework for systematically testing prompt strategies. Over time, you can build a dataset of scenes, prompts, and ratings that tells you which strategy works best for different types of content. That feedback loop is what turns a music generation feature from a novelty into a reliable product capability.\n\n---\n\n## Summary\n\nMusic generation has reached a point where it is practical for production applications, but the technology has clear boundaries you need to respect. Here are the key takeaways:\n\n- Music generation models work by operating on compressed audio representations, either spectrograms, neural codec tokens, or learned latent spaces, rather than raw waveforms. This makes generation feasible but introduces quality trade-offs.\n- **MusicGen** is the best choice for developers who need open-source, no-strings-attached instrumental generation. **Suno** and **Udio** produce higher-quality full songs with vocals but require paid subscriptions and come with platform-specific licensing terms.\n- Prompt engineering for music requires specificity across five dimensions: genre, mood, instruments, tempo, and production style. Programmatic prompt builders that map user-friendly inputs to detailed prompts are essential for production applications.\n- **Stem separation** with tools like Demucs lets you post-process generated music by isolating and remixing individual components. This extends the usefulness of each generated track.\n- The **legal landscape** is unsettled. Use open-source models for the clearest ownership story, keep generation logs, add human curation, and avoid prompting for specific artist imitations.\n- For production architectures, use **async job queues** for on-demand generation and **pre-generated libraries with semantic search** for latency-sensitive applications. Most real systems combine both patterns.\n\n---\n\n## References\n\n- [Meta MusicGen: Simple and Controllable Music Generation](undefined) - Original MusicGen paper by Copet et al.\n- [Audiocraft by Meta Research](undefined) - Open-source library for MusicGen and EnCodec\n- [Stable Audio: Fast Timing-Conditioned Latent Audio Diffusion](undefined) - Stability AI's approach to audio diffusion\n- [Demucs: Music Source Separation](undefined) - Meta's open-source stem separation model\n- [U.S. Copyright Office: Copyright and AI-Generated Works](undefined) - Official guidance on AI content ownership\n- [Suno API Documentation](undefined) - API reference for Suno music generation","pageType":"ai-engineering"}

8.6 Music Generation

Ashish Pratap Singh

Get Premium