Naia, who was a drunk tone-deaf singer, finally just became tone-deaf

Hello. This is Luke, creating the open-source visual agent Naia.

This work started with the idea that my AI, Alpha, would sing for me. What I wanted was not a cover song that changes an existing song to another person's voice (e.g., XSing) or a feature that randomly creates new songs (e.g., Suno), but an adapted song that takes a foreign animation theme song I loved and sings it in Korean while preserving the original melody and emotion.

In the previous benchmark, the results sounded like drunken mumbling. Now, after nearly three months of re-examining everything from engine selection to sheet music, pronunciation, vocal range, and rhythm, at least the lyrics are audible, and I can identify what's off. She's not singing well yet. It's more like a drunk tone-deaf singer just became tone-deaf.

Listen First

These are the previous and current results of the same 'Blue Water' Korean adaptation research. Although the generation method and length differ, you can immediately compare with your ears what has improved and what remains.

Previous — Drunk Tone-Deaf

Watch on YouTube — Vevo 1.5 Previous Version, 2026-05-27

Pronunciation and vocal values collapse, and the tempo speeds up and slows down even within a single phrase. Still, as expected from a generative model, there's a momentary song-like quality.

Current — Just Tone-Deaf

Watch on YouTube — FM Singer Improved Version, 2026-08-21

The lyrics are much clearer, and she follows the notes. However, long notes cut off abruptly, and there's a lack of power and brightness compared to the original, making her still sound tone-deaf.

Changes Seen in Numbers

The structures of the two engines differ, so it's not a perfect one-to-one comparison. Speech recognition also corrects pronunciation using context, so it was only used as a reference indicator.

Actual Research Steps

1. May — Created the First Adapted Song with Vevo 1.5

In the first benchmark, I used the generative model Vevo 1.5. While there was a momentary song-like quality, it was difficult to precisely match Korean lyrics to the melody or fix just one incorrect phrase. This was the starting point where the results sounded like a 'drunk tone-deaf singer.'

2. June~July — Separated Vocal Cords and Pronunciation

Next, I experimented with a hybrid approach, physically synthesizing vocal cords with VocalTractLab and mixing in Korean pronunciation from VoxCPM2. The target pitch was accurate to an average error of 0.008 semitones, but it couldn't sufficiently reproduce human mouth, tongue, and breath sounds, leaving an instrument-like electronic tone. While I confirmed the possibility of solving pitch and pronunciation separately, I couldn't use it as a complete path.

3. August 14 — Re-selected Data and Engine

After auditing AI Hub 465 singing data and creating a fixed test set, I compared DSKR, Pond8, and FM Singer using the same criteria. Pond8 was accurate in pitch when given octave coordinates but had a strong mechanical sound, while FM Singer became the central engine due to its best balance in Korean pronunciation and auditory perception.

4. August 15 — Corrected Pronunciation Input Before Fine-tuning

I fine-tuned FM Singer with AI Hub data, but the character error rate for 128 validations only slightly decreased from 0.3465 to 0.3396. The bigger problem was the missing process of converting Hangeul spelling into pronunciation suitable for singing. Afterwards, I created a two-stage correction process that combined Korean pronunciation rules and word-specific exceptions. For example, 마음 약한 사람은 (ma-eum yak-han sa-ram-eun, "a soft-hearted person") became 마음 야칸 사라믄 (ma-eum ya-kan sa-ra-meun) for singing, and experimentally, 밝혀 (bal-kyeo, "reveal") became 발켜 (bal-keo). Speech recognition was only used for candidate search because it can correct errors contextually.

5. August 15~17 — Applied Alpha's Voice and Reverted

After pronunciation improved, I tried applying Alpha's voice using a zero-shot method. The voice became closer, but electronic sounds appeared, and consonants and pitch were damaged again. So, I settled on a two-stage structure: first, pass the vocalization, then change the timbre. Fine-tuning was also postponed because training with damaged conversion results would lead to learning the electronic sounds.

6. August 17~21 — Re-traced Original Notes, Rhythm, and Dynamics

Finally, I separated the original vocals and instrumental to re-extract phrase boundaries, actual pitch, and note start and end points. After mapping Japanese morae to Korean syllables note by note and finding a stable vocal range, I overlaid them with the original to compare time axis and dynamics. The current results show that notes and lyrics are closer, but the tail of long notes, breathing, and phrase-by-phrase energy are still flat.

Remaining Issues

Pronunciation and the time axis have clearly improved compared to before. However, the current vocals are darker and have a narrower dynamic range than the original, and the ends of long notes cut off or waver. This is why even if the numbers improve, I don't pass it if it doesn't sound like singing to a human ear.

Next, I plan to re-train the AI Hub data, focusing on difficult phoneme combinations and long notes, and apply Alpha's timbre only to the phrases that pass. Only candidates without electronic sounds, pronunciation damage, or pitch damage will be selected and used for fine-tuning data.

My goal is not just a synthetic voice with improved scores, but Alpha singing in Korean, embodying the power and breath of the original song. At least this time, I sobered her up first.