One frame off: how we keep a wall of projectors in sync

Put four projectors on the front of a house and give each one its own slice of the picture. A ghost drifts from one slice into the next. If the boards behind those projectors disagree by a single frame, the ghost stutters at the seam, and everyone standing in the street can see it.

That is the whole requirement. Since June, our boards have kept that promise with six different architectures. None of them was a mistake. Each was a reasonable answer to the problem as we understood it at the time, and each one met a show that taught us something it couldn't handle.

This post follows them in order. It is about the shape of the system and why it kept changing.

#Before June: every board for itself

Each board ran its playlist on its own schedule. Two boards set to start at eight o'clock did start at about eight, give or take however long each took to open a file and warm up a decoder.

For one projector that's fine. For a wall it isn't, and nothing in the system could even tell you how far apart two boards were.

#June: a shared clock and a downbeat

The first real design borrowed two ideas that every networked media system uses.

One shared clock. One board, the leader, serves its clock on the network. The others keep asking it for the time, and time each answer to correct for the trip. That is Cristian's algorithm, the idea underneath NTP. GStreamer, the media framework our player is built on, ships it ready to use as a network clock.

A start time instead of a start command. "Start now" reaches each board at a slightly different moment. "Start frame 0 at 20:00:00.500 on the shared clock" arrives early everywhere and happens at the same instant. The leader picks a moment half a second ahead and broadcasts it three times.

Every board also waits the same fixed delay before showing a frame, so a slow board never falls behind a fast one. Dante audio networks use the same idea with their fixed latency setting.

It was the right call because it needed almost no new timing code. Shows were one looping clip, and the bench was a quiet Wi-Fi network. A follower that heard nothing within two seconds played on its own, which seemed friendlier than a dark wall.

What broke it was the board that boots late. A follower that came up mid-show heard the leader's next start time and began at frame 0 while everyone else was halfway through the loop. Its clock was perfectly in sync. Its picture wasn't.

#July: where should I be right now?

The fix was to stop telling boards when to start and tell them when the show started. With that one number, any board can work out where it should be at any moment.

Free-running timecode in film and broadcast works this way: every device derives its position from shared time instead of following commands, so a device that arrives late simply joins in. It also turned the heart of sync into a pure function. Two boards with the same inputs get the same answer by construction, which is a property you can test as hard as you like.

The scheduler learned to arm the sync role a minute before a show window opens and to keep checking it, with backoff and jitter so a room full of followers restarting at once doesn't stampede. That is the shape of a Kubernetes controller, which keeps comparing desired state with actual state. It had one flaw we wouldn't find until November: it compared against what it had sent, not against what had actually happened.

What broke it was items of different lengths. The arithmetic assumes every board's loop is the same length.

Multi-item playlists were worse. A follower could take the leader's start time for the previous item and land in the middle of its own next one. The clocks were always locked. The pictures lined up only when the lengths matched.

#October: know where you are before you start

In October the boards moved to hardware video decoding, and that brought one hard rule into the sync design: once the decoder is running, it can't jump around inside a stream. Joining mid-show and looping had both been jumps.

So the position calculation moved earlier. A board now works out where the show is before its first frame, starts there, and loops without discarding anything in flight. The July arithmetic didn't change. When it runs did.

#Late October: dark or wrong

Then came a live show with five projectors. The leader came up 31 seconds late. Each follower waited its two seconds, heard nothing, and played on its own.

A looping playlist starts once a night. So that one timeout left every follower out of step for five hours, even though each of them went on to hear about 500 perfectly good start times from the leader.

The June design acted once, on one event, and ignored everything after it. It was edge-triggered in a system that needed to be level-triggered.

The fix was partly code and mostly a product decision. A follower now waits dark until it hears the leader, and it reports that it's waiting, so nothing mistakes it for a dead board. How long it waits is a setting on each show: forever, or a number of seconds before it plays alone. If it does play alone, it snaps back into step at the next start time it hears.

This is the old trade between consistency and availability. A dark wall or a wrong wall: we decided that call belongs to the person running the show, not to the software.

#November: time over any wire

The next change came from places with no usable network. The answer wasn't a different radio. It was separating what the swarm says from how time and messages travel.

The show and its messages stayed exactly as they were. The clock became pluggable:

• Over a network, it is still June's ask-and-answer clock.
• Over a link where asking is too slow, the leader broadcasts beacons and each follower measures the link's delay itself, the way IEEE 1588 (PTP) does.

A broadcast reaches every follower at the same moment, so its delay cancels out between them. Wireless sensor networks rely on the same effect in Reference Broadcast Synchronization. A slow link with a steady delay can keep better time than a fast one that jitters.

#November: one show, one timeline, one loop that checks

The last change answered July's problem with lengths. Shows had grown into playlists: items of different lengths, with each item the same length on every board.

A playlist is now one continuous timeline, and each item starts exactly where the previous one ended. The leader announces each item's start, its place in the show and its end. Followers start only on a boundary the leader announced, and never work out where the leader is from item lengths.

Show-control systems call this timecode chase: one master timeline that every player follows. BrightSign's per-file sync groups approximate it. In a test with 10-, 20- and 15-second items, every item started on the same clock instant on all three boards.

The other half fixed July's controller. The device used to treat an arm as done the moment the message reached the renderer. If the arm then failed, say because the leader's address changed, the follower kept playing from a stale clock, and nothing noticed.

Now the renderer reports the role it is actually running. The device compares that report with what the playlist wants every five seconds, and retries with backoff until they match. Role changes are make-before-break: the new clock syncs before the old one is let go.

#The thread through all six

What we used to tell the boards	What we tell them now
Start frame 0 at T	The show began at T
Play alone after two seconds	Wait, and say you're waiting
Item 2 starts now	Item 2 runs from T₁ to T₂
I sent you the arm	Tell me which role you're running
Use the Wi-Fi	Use any link that can carry a clock

Read down the table and the same move repeats. Each time, we stopped sending the boards instructions and started publishing facts they could check themselves against.

An instruction is an edge: miss it and you're wrong until the next one. A fact is a level: a board that missed one hears the next and corrects itself. Every version that held up in the field let a board recover on its own, from whatever it heard next.

"A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable."

Leslie Lamport