Viktors Rotanovs » High Performance, Security, Operations

HBase Easy Import/Export

viktors — Thu, 13 May 2010 21:35:46 +0000

Export a table from HBase into local filesystem:
bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export table_name /local/path

Export a table from HBase into HDFS:
bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export table_name hdfs://namenode/path

Import a table from a local dump into existing HBase table:
bin/hbase org.apache.hadoop.hbase.mapreduce.Driver import table_name /local/path

It’s a good idea to count and compare number of rows before exporting and after importing:
bin/hbase org.apache.hadoop.hbase.mapreduce.Driver rowcounter table_name
Number of rows is visible in Hadoop counter called ROWS, like in output below:
10/05/13 21:26:41 INFO mapred.JobClient: org.apache.hadoop.hbase.mapreduce.RowCounter$RowCounterMapper$Counters 10/05/13 21:26:41 INFO mapred.JobClient: ROWS=103821

Alternatively, you can use Hadoop Tool interface, but it may complain about missing classes if hadoop-env.sh is not configured properly. For example, when launched without arguments, it displays available options:
hadoop jar hbase-0.20.3.jar An example program must be given as the first argument. Valid program names are: export: Write table data to HDFS. hsf2sf: Bulk convert 0.19 HStoreFiles to 0.20 StoreFiles import: Import data written by Export. rowcounter: Count rows in HBase table

HBase dump is one ore more Hadoop SequenceFiles, you can inspect its contents with something like:
hadoop fs -fs local -text table_name/part-m-00000

Lighttpd vs Tux: nice() patch

viktors — Fri, 25 Mar 2005 21:34:35 +0000

Lighttpd by default can be slower than TUX. Why? Because TUX runs inside Linux kernel and has the highest possible priority, while lighttpd is just ordinary userspace process. Solution? Make lighttpd not so nice:
nice -20 /opt/lighttpd/sbin/lighttpd ...
Lighttpd’s Adaptive Spawning users will immediately notice a problem: all processes spawned by lighttpd will also run with the highest priority. That’s not good, since any PHP or other FastCGI process will be able to slow down lighttpd. Solution is quite simple: to restore back niceness level before launching a child process. Here’s a patch for that:

lighttpd_nice.patch

Any other ideas how this could be done?

Lighttpd initgroups()

viktors — Mon, 21 Mar 2005 23:37:45 +0000

While converting my website to lighttpd, I noticed that php processes launched by adaptive spawning can’t access certain files anymore. Small investigation has shown that the problem was caused by user httpd not joining groups specified in /etc/groups. Adding initgroups() call before calling setuid() helped, here’s the patch:

Please let me know if you’ve experienced the same problem with spawn-fcgi program.

Lighttpd Config Includes

viktors — Wed, 16 Feb 2005 15:06:51 +0000

Some of us techies have to maintain 50 or more webservers, which usually leads to in-house Linux distros. With Apache, you could throw in .htaccess for each website without having to change base system. It’s not that simple with Lighttpd, that’s why I needed some script to join global configuration file with a local one. Some perl coding and it’s ready:

http://rotanovs.com/lighttpd/joincfg

Instructions:

Install Tie-IxHash perl module:
- Gentoo users: emerge Tie-IxHash
- Other distros:
  perl -MCPAN -eshell cpan> install Tie::IxHash
Backup your configuration
Call joincfg like follows, then start lighttpd:
joincfg global.conf local.conf > new.conf lighttpd -f new.conf

As usually, NO WARRANTIES!

Lighttpd DoS Tool

viktors — Tue, 01 Feb 2005 13:00:04 +0000

After seeing lighttpd grow to hundreds of megabytes I couldn’t sleep. It was like tiny chicken has grown into huge monster. After some experimenting I managed to grow monsters myself:

http://rotanovs.com/lighttpd/lighttpd_dos_tool

Basically the source of the problem is that lighttpd ignores max request size setting and reads data until it becomes huge and eats all available memory.

Return to Apache? No way. Lighttpd problems are usually solved lightning fast – so a little bit of patience and the cure will be ready.