LivingMusic

I wasn't looking for a new project. It all started after watching several videos of the Cotodama Lyric Speaker — typography stopped being simple text and became part of the music. Words appeared, breathed, and disappeared following the rhythm of each song. It was a visual experience, not karaoke. After searching I found many players with synchronized lyrics, but none treated typography as the main element of the experience. A simple question appeared: if the app I want doesn't exist, why not build a small version to understand how it works? That was the real start of LivingMusic.

I thought the easiest part would be getting synchronized lyrics. After researching, I discovered it was exactly the opposite. Spotify and Apple Music don't expose synchronized lyrics through their public APIs. Open solutions usually offer line-level sync — enough for traditional karaoke, but insufficient for the experience I had in mind. Musixmatch does have word-level precision, but that's behind commercial agreements. The decision: the MVP uses a small curated catalog with LRCLIB as the open source and manual forced alignment for word timestamps. Sometimes the best decision is reducing scope until you regain control of the problem.

The first version of the project assumed Spotify and Apple Music would be the starting point. After understanding the API limitations, that idea stopped making sense. LivingMusic doesn't need to connect to a streaming service to demonstrate its value — it needs to prove that a song can acquire its own visual identity. The MVP works with a small local catalog. If the experience works with five songs, it can grow later. If it doesn't work with five, it won't work with five million either.

A decision appeared early that ended up defining the entire project: I didn't want a single visual style for all songs. A ballad doesn't communicate the same as rap. An instrumental piece doesn't carry the same rhythm as an electronic song. The interface had to adapt to the music, not the other way around. So each song will have an artistic direction preset where typography, composition, movement, and rhythm can change completely. Consistency won't come from repeating animations — it will come from sharing the same design philosophy.

The main development machine is Windows. iOS and Apple TV require macOS and Xcode — a hard Apple requirement with no real workaround. It was confirmed that a Mac is available for when those platforms need to be compiled and tested, so the four-platform scope (iOS, Android, Apple TV, Android TV) holds. Not a blocker, just a constraint to keep in mind from the start.

The project scaffold landed in commit f912b84 using @react-native-tvos/template-tv (React Native 0.86, React 19.2, TypeScript 5.8). Immediately after, the first Android build was attempted. The React Native CLI failed invoking gradlew.bat — a known Windows bug with .bat subprocesses. Solved by invoking Gradle directly, but a second problem appeared: conflict between Gradle 9.0.0 (newly released, bundled with the template) and the JDK toolchain resolution plugin (IBM_SEMERU). Decided not to pursue this immediately and look for an alternative route to validate the visual engine without depending on the native emulator.

Development started in pure React Native. A real friction appeared quickly: each test depended on spinning up an emulator, waiting for builds, and repeating a cycle too slow for a project that will probably require hundreds of small visual adjustments. Research confirmed that react-native-skia has mature web support via CanvasKit (WASM) — the same Skia engine, not an approximation. A monorepo was created with three packages: packages/engine (shared logic), apps/mobile (the RN+tvOS project), and apps/web-preview (Vite + React + Skia web). The goal was never to build a web version of the product — it was to have a lab where any change could be seen almost instantly.

After several architecture decisions, a much more important question appeared: does this idea actually work? Instead of integrating audio, streaming, and real sync, a minimal proof was built first — a clock. If the engine could correctly animate a phrase using only the passage of time, it could later do so following a real song. packages/engine received its first real pieces: types.ts (LyricWord) and sync.ts (computeWordEmphasis, a pure function calculating a 0→1 emphasis curve per word with entry/exit ramps). The first implementation doesn't play music. It only tries to prove that words can move with intention. The real product is not the player — it's the visual engine.

For several hours it was assumed the engine was working. Processes completed correctly, HTTP responses were successful, nothing seemed to indicate a problem. Until the browser was opened: absolutely nothing. That moment forced a review of every assumption from scratch. The chain of causes was long — CJS dependencies of react-native-web without explicit pre-bundling, missing shims for TurboModuleRegistry, canvaskit-wasm version mismatch (^0.40.0 vs the 0.41.0 that react-native-skia bundles internally), and .web.js extensions that esbuild ignored while Metro resolves them correctly. More important than fixing them was understanding something else: a hypothesis had been validated without checking the final result. Tools can say everything went fine. The user always has the last word. In a visual project, that last word starts when something actually appears on screen.

After several attempts, the engine managed to render its first phrase using the Sync Engine and Skia — commit 6ef51a8. There was no audio, no player, no complete animation system. Just a phrase, a clock, and a set of calculations controlling the emphasis of each word. It was a very small proof. But for the first time the project stopped being an architecture drawn on paper and started behaving like what I imagined when I saw those first typographic animations. Great milestones rarely arrive with a finished interface. Sometimes they arrive when a single word appears exactly where you expected it.

Before writing a single line of animation code I need to know what body the words will move in. Not just any typeface — I need a set of free typefaces with their own character, that can carry the emotional weight of a song without breaking. Between five and ten. Covering the spectrum without overlapping: the cold and structural one, the human and warm one, the one with fracture, the perfectly geometric one, the one that breathes. A palette. Because if each song will have its own artistic direction, typefaces are the first decision of that direction — before movement, before rhythm, before everything.

I was planning to train the motor by watching videos — studying how words move in references I like, extracting patterns, replicating them. But as I thought about it I realized that was exactly what I didn't want to do. If the motor learns from references, it will always be a sophisticated copy. The word needs to feel, not resemble something I've already seen. Changed the plan. Manifesto first. Before writing animation code, write the rules for why a word moves a certain way — what it means for a word to "carry weight," what typographic silence is, why the entry speed and exit speed don't have to be equal. Shared the ideas with Claudio — that's what I call him when we're in creative mode, not debug mode — and we started iterating. What's coming out isn't a technical manual: it's a reusable catalog of movement primitives. The manifesto is the index. Code comes after, and when it does it will already know why it exists.

Wonderwall. Oasis, 1995. First song in the LivingMusic catalog. I didn't choose it for nostalgia — I chose it because it has exactly the emotional arc I need to test whether the system really works: uncertainty → doubt → growing tension → anthemic catharsis. If the engine can follow that arc, if the manifesto primitives can accompany it without forcing it, it can accompany any arc. Simulated four phrases applying the full vocabulary — how each word enters, what weight it carries, when the layout breathes, how tension scales before the chorus. First time the project stops being an abstract system and becomes something that could be felt.

After choosing Wonderwall as the first case study we simulated the four key phrases of the emotional arc, applying the manifesto vocabulary. What follows is the session record — the decisions made, the ones I rejected and why. Phrase 1. Initial suggestion was Archive. Rejected. Archive already knows what it is — it has presence, defined character, it takes up space. Wonderwall starts before knowing what it is. It needs a voice that doesn't yet trust itself. Voice. Horizon. A typeface that exists on the edge between appearing and not being there yet. Phrase 2. This one yes. Whisper, Resistir, Paper — perfect as is. But I want to add something that wasn't in the original vocabulary: paper doesn't return to its initial state after being folded. There's a deformation that persists even if invisible. That has to be in the animation: when the next line appears it should retain something of the previous one. A residue. A small distortion. Because the song remembers, even if the lyrics don't say so. That's cinema. Phrase 3. I disagree with Machine. Machine implies precision and Wonderwall is never precise — it's human, irregular, imperfect. Changed the proposal: Pulse. Not machine, not whisper. It's heart. It beats. And where Metal was I proposed Material String. Wonderwall's tension isn't industrial. It's organic. Like a guitar string that vibrates after being plucked and keeps vibrating longer than it should. Much more coherent with Oasis. Phrase 4. The strongest change. Expand doesn't work — it feels like a reward, and Wonderwall never fully resolves. The chorus doesn't say "everything is fine." It says "I don't know how." There's still doubt. The phrase has to grow without closing. That vocabulary is still to be defined. It's probably the hardest part to translate into the system — and probably the most important. Four phrases. The manifesto is being tested before it's finished. That's exactly how it should work.

The manifesto as a list of primitives has a problem I'm only now seeing: it's too linear. Too PowerPoint. It defines rules but doesn't understand why those rules exist. No references, no visual memory, no intuition. It's an index without a body. What I need isn't a manual — it's a brain. I shared editorial style captures from a Lewis Capaldi video with Claudio. Not as a reference to copy — as evidence that something can feel a specific way and we need to understand why. The typography in that video doesn't decorate the music. It interprets it. There are decisions there that aren't in any motion design manual because they don't come from rules — they come from sensibility. So we're restructuring. The animation brain will have three layers: visual references with annotations (what happens and why it works), a catalog of songs with their emotional arcs mapped, and on top of all that the primitives — but now with context, with real examples, with the question of why this primitive and not another. I don't know if this has a name in any methodology. Probably not. But it's the difference between a system that generates correct animations and one that generates animations that feel right. The first can be documented in a readme. The second needs a brain.