HotSchedules in Austin, Texas, serves close to 2 million employees of the hospitality, restaurant, and retail industries worldwide. Users, whose employers subscribe to the service, rely on the company’s cloud-based software to tell them when and with whom they’re scheduled to work.
With more than 2 million daily log-ins, HotSchedules has become “a mission critical part of an hourly worker’s day-to-day activities,” says CEO Anthony Lye. “We can be the difference between having and not having a job, and we take that responsibility very seriously.” The platform is so popular, in fact, that he claims it is among the top revenue generators in the app store.
That’s why a 45-hour distributed denial of service attack that commenced on a Sunday night in May could have crippled not just HotSchedules, but also thousands of restaurants, hotels, and stores. The malicious action by an unknown perpetrator attemtped to shut down the business by flooding the HotSchedules servers with traffic. “When you get hit by a DDoS attack, it’s not coming from a single place, but from a large number of places, and it’s very difficult to stop it,” Lye says. Experts who assessed the situation predicted Lye’s business would be offline at least 5 days.
But instead of devastating it, the experience became a source of pride for the company. The HotSchedules workforce jumped to action to email, fax, and call the 2 million users who were suddenly unable to check their schedules on the site. “The team was able to come together to address both the infrastructure and the customer service issues within hours,” Lye says. “Our employees put the customer front and center and took full responsibility and accountability for the service we offer them.”
Lye also took the steps to prevent future attacks. “We were already paying good insurance and had good bandwidth,” Lye says. His site was equipped to handle 20 times the number of user logins that it typically experienced, but the buffer wasn’t big enough to absorb the denial of service attack.
“We could accept about 40 million logins a day, but we were hit by traffic that was as high as 12 gigabytes per second; 300 times greater than our capacity.” Physically unable to handle that many connections and attempts to connect, the app was disabled. Even moving the software to another IP address didn’t work: “They were watching us move. It was a very sophisticated attack and it costs someone a fair amount of money to attack a service like ours.”
Now HotSchedules relies on a service used by online gaming businesses, considered the platinum standard. “It would take an enormous amount to even get close to us. Now feel we can take on large attacks,” he says.
It’s hard to say why HotSchedules was the target of an attack in the first place. “We didn’t receive any ransom notes or direct commentary,” Lye says. “Part of the population likes to cause high-impact damage. We’re responsible for a lot of people and getting them to work on time. That makes us an obvious target that would be newsworthy.”
In hindsight, Lye calls the DDoS a great learning experience. “You’re not expecting somebody to do that kind of thing to you. It focuses the whole company on how they should respond, communicate, and interact. A lot of people didn’t sleep for two days. They chose to come in on the weekend and work overnight. People were sleeping on the data center floor while others were reconfiguring systems.”
And while some companies might hide such an event from customers, HotSchedules put the word out on social media, answered every user comment, and took 20 times more customer service calls than usual. “It was stressful, but the company performed far above and beyond anything I’ve experienced,” Lye says.
Lye calls such attacks “a byproduct of success for any business. If you are successful and have a loyal customer following and make a difference to an industry, then there are people who will have fun attacking the way you run your business. It’s part of life on the Internet. If companies aren’t prepared for this they need to start thinking about it.” His advice? “Manage your Web servers and hope your people will do the right things.”Follow Adrienne Burke @adajane