I have been running this blog on Linode for a few months now. I got to say that I absolutely love Linode. While I would consider myself fairly technical, there are some aspects of the Linux environment that I am either lacking, or absolutely clueless on. I, however, have the motivation to learn new things and have vowed that completely understanding the Linux environment is one of those things. So when I have issues, I am more than happy to investigate, troubleshoot, and fix them on my own before opening a support ticket. The great thing about Linode is that they have taken the time to put together an awesome set of documentation and FAQ. And if those don’t help you the community support surely will.
I do know enough about Linux to be dangerous and understand most of the technologies such as MySQL and Apache that I utilize on a daily basis. I also understand the my.cnf and httpd.conf files enough to configure database and web servers to do what I want them to do efficiently. At the beginning of December I posted some video of a fire that was linked to from several media organizations. Needless to say the traffic to the blog spiked dramatically for the days surrounding the post. What also dramatically spiked was server utilization and an increase in response time for the server. The server also began to crash at random times, and not necessarily during those spikes in traffic.
The nice thing about the Linode Dashboard is that they provide you with several graphs (CPU Utilization, Network Traffic, and Disk I/O rates) right up front. When I had configured my Linode I also set up alerts (another nice feature of their dashboard) to let me know when these parameters exceeded certain thresholds. As the traffic piled in to the server I started receiving alerts at random times for both CPU Utilization and Disk I/O. I knew that this most likely meant the drive was thrashing with as the memory utilization resulted in heavy swapping. But what was causing this issue specifically I wondered. The traffic was not that dramatic that a VPS with 512MB RAM, 16GB storage, and 200GB monthly throughput shouldn’t be able to handle. I began digging into the problem. My immediate fix was just to reboot the VPS and watch the charts to try to determine a pattern, while I troubleshooted on the backend.
Returning to the Linode support documentation I quickly found two articles that might solve my issue. Besides running this blog on the server, I have also been doing some development and had a small requirement for the 64bit version of CentOS. When I set up the VPS I had installed this version knowing that it would be a little bit of a memory hog, but I figured that the 512MB would be able to adequately handle both the blog traffic and my development work. Of course the first article I came across was a warning NOT to install the 64-bit version for this reason. I was not convinced that this was the root cause, however, and continued searching. I did put it in the back of my head and would make a reinstall with the 32-bit version a last resort.
The next article I came across appeared to directly address the issue I was experiencing. High CPU and disk utilization from the swap file. Again, this article addressed the 64-bit version, but also provided additional steps for troubleshooting and resolving the issue.
Going in to the article I had already determined that both Apache and MySQL were consuming the majority of the memory, and I figured that both the my.cnf file and httpd.conf files needed to be adjusted. I was a little puzzled that, with Linode’s complete documentation and ease of install, they had not already set these options to at least a base install. The developers of MySQL have been kind enough to include four sample my.cnf files that users can utilize based on available resources.
Now before I continue, I must advise you that these settings work optimally for my server. They are presented as a guideline for you. If you are not familiar with MySQL optimization, to accurately optimize the settings in my.cnf, you should either consult with someone who knows, talk to your host, or if you are a one that is willing to play with the settings yourself you can always set some baselines off the sample files provided by the developers of MySQL and work from there. These statements also apply for the httpd.conf file. I can say that if you are experiencing issues on Linode similar to mine, that these settings resolved every issue I had.
Continuing on, the first file we will edit is the my.cnf file. If you are running CentOS the path to the file is going to be /etc/my.cnf. I use the vi editor to edit text files in Linux. If you are not familiar with the vi editor, you can do a search on this blog for some help. The four files that MySQL comes with are my-huge.cnf, my-large.cnf, my-medium.cnf, and my-small.cnf. Each file was developed to be a baseline configuration for various server configurations with different levels of resource (mainly memory) availability.
The comments at the top of each file describe what server baseline it is designed for, and a quick review of each will reveal that that the one closest to a Linode 512MB running MySQL and a web server is most likely the my-medium.cnf file. Reviewing the file and comparing the recommended options to those in the Linode support document we can ascertain that the following settings will need to be adjusted. Quick note here, when I reviewed the base my.cnf file included on my Linode install these sections were missing entirely or limited so you will most likely have to add them.
Your my.cnf should look similar to this for a Linode 512 server based on the my-medium.cnf file from MySQL:
[mysqld]
key_buffer = 16M
max_allowed_packet = 1M
thread_stack = 64K
table_cache = 64
sort_buffer = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
After you have made the appropriate edits and saved the my.cnf file, you will need to restart MySQL. To restart issue the following command at the prompt:
/etc/init.d/mysqld restart
Reminder that these instructions are based on CentOS and may be slightly different for your OS.
The next file we will edit is the httpd.conf file. Again on CentOS the path to this file will be /etc/httpd/conf/httpd.conf and again you can use the vi editor to make the necessary changes. Now there is a good chance that if you are running CentOS 5 you are using the “worker MPM” so you can find the section in the httpd.conf that address the worker MPM (similar to below). Again, I found that my install had a very limited configuration here and after a little research was able to determine some baseline settings that worked well for my server configuration: (For more information on each of these settings you can check out the Apache documentation on the worker MPM at http://httpd.apache.org/docs/2.0/mod/worker.html)
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
The Linode documentation gives a slightly different configuration from above, but I went with the above based on my own experience and a little digging around the documentation.
After you have made the changes to the httpd.conf file you will need to restart the Apache process. To do so, issue the following command from the prompt:
/etc/init.d/httpd restart
You should now be good to go. Continue to monitor your performance and adjust the settings above accordingly.
