Server CPU Load At 97%

When your server CPU load is at 97% for 30–60 minutes then it’s not good. Especially if it happens more times a week. And this is what happened last week. I started to monitor the situation more closely, and I saw very weird user actions. I had no idea what was going on. My first thought was that somebody did it with malicious intent and it was a kind of a DOS attack. I contacted the user. She explained what she wanted to achieve. I don’t want to get into the details, but it turned out that it was a valid but very rare and weird user scenario and she was doing something in the wrong order, so the unusual load was cased by a user error. Finally I advised her how to do the steps in the proper order to achieve her goal but not causing an extreme load on the server. Even after this things didn’t go smoothly, it turned out that there was a bug in the “Find Duplicates” feature. There were bookmarks with empty urls among the user’s imported bookmarks, which caused an exception. Today I fixed the bug.

DOS attack

So I was lucky it wasn’t a DOS attack. But what I learned from this that I need to prepare to handle a possible DOS attack. I haven’t thought of it until now. This week my number one priority task is to develop and implement some defense mechanism against a DOS attack.

Auto scaling

Another thing that came to my mind was auto scaling. I think the nature of this web app doesn’t require auto scaling but utilizing it is something I might consider in the future. AWS has a pretty good dynamic auto scaling service.

Advertisements

2 comments

  1. AWS has nice services that can be used in order to achieve scale and balancing. I guess you have more than one ec2 instance to prevent single point of failure. Why shouldn’t you try those services in order to reduce costs and complexity with scaling the system?

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s