8 pm: the backend is gone. Noon: the demo goes live.
A partner lost the entire backend of a bilingual video job interview system the night before launch, and every backup with it. Around EUR 90,000 of EU funding depended on a working demo at noon the next day. We rebuilt the system from scratch overnight.
Something just broke? Let's talk
What was left: a brandbook and a brief
The partner was about to launch a bilingual job portal built on a video job interview system. While the system was being moved out of the development environment, a step meant as maintenance deleted the entire backend.
Then the second blow. Every backup sat on the same server, so they went too. What remained was the brandbook and the brief. Nothing else.
What was riding on it
A cross-border project funded from EU sources, worth around EUR 90,000. The first-round demo was set for noon the next day. No working system at the demo meant the full amount would be withdrawn and had to be paid back.
Sixteen hours, and an empty server.
The night, hour by hour
8 pm. The call. One of our partners. From the first sentence we could hear that something was very wrong.
Within half an hour. Six at the keyboard. Six of us were working on it before the half hour was up.
Through the night. Rebuilt from almost nothing. Working from the brandbook and the brief, in two languages, we put together something that had looked impossible a few hours earlier.
Somewhere in the middle of it, our partner sent us just this:
“You guys are not normal! Thank you, you saved us!”
Noon. The demo. The partner presented the cross-border project, built on the video job interview system, in two languages. Sixteen hours after the call.
One week later: the real fix
The wrong command was not the real cause. The real cause was that one wrong command could take everything.
So a week later we introduced the partner’s company to version control and set up backups kept separately, on different continents. The same mistake can no longer take the code and the backups at once.
What this night teaches
- A backup on the same server is not a backup. It is a copy that goes down with the machine.
- Version control is not a luxury. If the code lives in one place, one wrong move is enough.
- A night like this takes a team. Six people within half an hour, through to morning. Nobody does that alone.
Where are your backups right now?
If the honest answer is "on the same server", let's fix that before you need it. We reply within one business day.