Blog

Outage of October 6, 2021

Geoffrey Bilder

Geoffrey Bilder – 2021 October 06

In Data CenterPost Mortem

On October 6 at ~14:00 UTC, our data centre outside of Boston, MA went down. This affected most of our network services- even ones not hosted in the data centre. The problem was that both of our primary and backup network connections went down at the same time. We’re not sure why yet. We are consulting with our network provider. It took us 2 hours to get our systems back online.

Lesson learned, the hard way: Let’s not do that again!

TL;DR We missed an error that led to resource resolution URLs of some 500,000+ records to be incorrectly updated. We have reverted the incorrect resolution URLs affected by this problem. And, we’re putting in place checks and changes in our processes to ensure this does not happen again. How we got here Our technical support team was contacted in late June by Wiley about updating resolution URLs for their content. It’s a common request of our technical support team, one meant to make the URL update process more efficient, but this was a particularly large request.
RSS Feed

Recent Posts

More new faces at Crossref

2021 October 21

Outage of October 6, 2021

2021 October 06

2021 Board Election

2021 September 28

Archives