Check bots success POSTs in apache access log
Thu, 05/19/2011 - 16:20 — sandipHere is a one liner to check for IPs of bots that are misusing your site.
$ awk '$6 ~ /POST/ && $9 ~ /200/ {print $1 " " $7}' /var/log/httpd/access_log | sort | uniq -c | sort -n | tail
This will give you the top 10 IPs and URIs with a hit count.
Consider blocking those rogue IPs with a high hit count via iptables.
Unicode keyword translation
Mon, 05/09/2011 - 11:41 — sandipDo you see garbled text, unicode keywords in your referral web stats reports like below?
à ¹€à ¸£à ¸µà ¸¢à ¸⠄¢ à ¸—à ¸µà ¹ˆà ¹„à ¸ «à ¸™
Use the below url for the conversion:
Here is the simple php source code that uses google translate for translation to English:
<?php header("Content-Type: text/html; charset=iso-8859-1" n style="color: #007700">); ?>r />
<html>
<head>
<title>Unicode keyword translation</title>
</head>
<body>
<br />
Copy/paste unicode keyword:
<br />
<br />
<form action="convert_keywords. php">
<input size="100" name="input" value="<?= $_GET['input']; ?>" type="text">
<input value="Go" type="submit">
/> </form>
<br />
<br />
<?php
$pieces=explode("=", $_SERVER['QUERY_STRING' ]);
$query=$pieces[1];
$translate_url ="http://translate.goo gle.com/?hl=en#auto|en|" . $query;
if(!empty($query)) {
?>r />
<iframe src="<?= $translate_url; ?>" width="100%" height="100%">
/> Iframe unsupported...
</iframe>
<?php } ?>r />
</body>
</html>
Check and tune network speed
Fri, 05/06/2011 - 16:56 — sandipI've had to do this on a couple of servers to check on the network speeds and sysctl tuning.
On the receiving end (192.168.10.1) bring up netcat listening to a port:
while true; do nc -l 8001 >/dev/null ; done
Use the below one liner to send over some data via netcat from 192.168.10.2 which prints out the transfer speed.
( dd if=/dev/zero bs=64K count=1000 | nc 192.168.10.2 8001 ) 2>&1 | awk '/MB/{print $8*8 " " tolower($9)}'
Increase the default maximum TCP buffer size and rerun the above test for proper tuning. Make sure to restart netstat listening socket upon sysctl changes.
The following are recommended:
##
# max TCP buffer size: 16MB (16 * 1024 * 1024).
# Could increase to 32MB for GigE.
#
# Increasing the TCP send buffers will increase the performance
# if you have large files to send.
#
net.core.wmem_max = 16777216
# If you have a lot of large file uploads,
# increasing the receive buffers will help.
#
net.core.rmem_max = 16777216
# increase Linux autotuning TCP buffer limits:
# min, default, and max number of bytes to use
# (only change the 3rd value, and make it 16 MB or more)
#
net.ipv4.tcp_rmem = 4096 &n bsp; 87380 16777216
net.ipv4.tcp_wmem = 4096 &n bsp; 65535 16777216
# Support for the above large TCP send and receive windows.
# Needs to be set to 1 if the Max TCP Window is over 65535 (64K).
#
net.ipv4.tcp_window_scaling = 1
# Increase backlog to avoid dropped packets and increase throughput.
# Check with `netstat -st | grep packets` and check for
# dropped and packet errors.
#
net.core.netdev_max_backlog = 5000
ssh keygen RSA versus DSA
Fri, 05/06/2011 - 10:55 — sandipWhile generating ssh keys, I usually use RSA type since it can be used to generate 2048 bits key, while DSA is restricted to exactly 1024 bits.
ssh-keygen -t rsa -b 2048
smartctl with 3ware RAID controller
Fri, 05/06/2011 - 10:44 — sandipThe below information is extracted from the smartctl man page.
To look at ATA disks behind 3ware SCSI RAID controllers:
smartctl -a -d 3ware,N /dev/sda
smartctl -a -d 3ware,N /dev/twe0
smartctl -a -d 3ware,N /dev/twa0
Where N is the disk number.
Devices /dev/sda and /dev/twe0 is used with 3ware series 6000, 7000, and 8000 controllers that use the 3x-xxxx driver.
/dev/sda form is deprecated starting with the Linux 2.6 kernel.
/dev/twa0 is used with 3ware 9000 series controllers, which use the 3w-9xxx driver.
You can also check on the series with the below command:
lspci |grep 3ware
Find files used for htauth
Wed, 04/06/2011 - 15:39 — sandipBelow will list all of the files that are used for apache authentication in /var/www/html file path:
find /var/www/html -name .htaccess | xargs awk '{sub(/^[ \t]+/,"")};/File/{pr int $2}' | sort | uniq
Here is the breakdown:
find /var/www/html -name .htaccess
Find all files named ".htaccess" at path "/var/www/html"
xargs awk '{sub(/^[ \t]+/,"")};/File/{pr int $2}'
The search output gets piped via xargs to awk, deleting leading whitespace (spaces and tabs) from front of each line and output is of only the second field of lines containing the text "File".
sort | uniq
Awk output is further piped through sort and uniq which results in the files being used for apache authentication.
Converting Shared to Exclusive IP for Plesk Domain
Wed, 04/06/2011 - 08:35 — sandipFirst go into "Clients", select the corresponding client IP Addresses and assign the additional IP address.
Then go into "Domains", select the domain that should be converted to exclusive IP and modify the "Web Host Settings" and assign it the new IP.
VIM with Ruby support on CentOS-5.5
Mon, 03/28/2011 - 15:33 — sandipWhile trying to get command-t VIM plugin installed I noticed that the default vim-7.0.x installed from CentOS Base repository does not come compiled with ruby support and needed to be upgraded to version 7.2.x:
$ vim --version|grep ruby
+python +quickfix +reltime +rightleft -ruby +scrollbind +signs +smartindent
Notice "-ruby" which states that it is without ruby support.
Once the rpmbuild environment is setup, install the source rpm:
# su - rpmbuild
$ cd ~/rpm/SRPMS/
$ wget http://ftp.redhat.com/pub/redh at/linux/enterprise/6Server/en /os/SRPMS/vim-7.2.411-1.6.el6. src.rpm
$ rpm -Uvh --nomd5 ~/rpm/SRPMS/vim-7.2.411-1.6.el 6.src.rpm
Edit the vim.spec file, line 496 and remove perl-devel from the requirement list.
BuildRequires: python-devel ncurses-devel gettext perl-devel
Then build rpm binary:
$ rpmbuild -bb ~/rpm/SPEC/vim.spec
Install the rpms via:
# rpm -Uvh ~/rpm/RPMS/x86_64/vim-{m,c,e}*
jps and jstat for tomcat on jdk-1.6.0_24
Fri, 03/25/2011 - 16:53 — sandipRecently upgrading to java version "1.6.0_24", jps and jstat seemed to be broken to get monitoring information from running tomcat process.
By default java.io.tmpdir is /tmp, however tomcat usually uses it's own temp directory. That is where jps/jstat looks for hsperfdata_* dirs. If java.io.tmpdir is not set, it would look in /tmp. If jps can't find the hsperfdata directory, it won't report anything.
jps/jstat however allows to specify java.io.tmpdir in case you're using a JVM that places those directories in a different location.
So to get it to work:
$JAVA_HOME/bin/jps -J-Djava.io.tmpdir=/path/to/to mcat/temp -l
$JAVA_HOME/bin/jstat -J-Djava.io.tmpdir=/path/to/to mcat/temp -gc $PID
Django HTTPS Redirects
Fri, 03/25/2011 - 15:04 — sandipThis works for both HTTP and HTTPS where any front end web server such as nginx which handles the actual request sets a header when request comes via HTTPS. In Apache configuration you then use mod_setenvif to set the HTTPS variable, which Django then picks up to use for redirection.
With front end nginx server which handles SSL, set header "X-Forwarded-Proto=https" via:
proxy_set_header X-Forwarded-Proto https;
On Apache, add directive:
SetEnvIf X-Forwarded-Proto https HTTPS=1
The HTTPS variable is picked up as being special by mod_wsgi and it will fix the wsgi.url_scheme in WSGI environment which Django then uses for redirection.
This way you don't need to customize Django stack.