ELC Code Feed

Auto-solved simple_captcha while developing

Dejan Simic — Wed, 07 Jan 2009 03:57:16 +0000

If you’re developing an app with the excellent simple_captcha plugin do yourself a favor and monkey patch it in config/environments/development.rb:

ActionView::Base.module_eval do
  def simple_captcha_field(options={})
    options[:object] ?
      text_field(options[:object], :captcha, :value => simple_captcha_value(simple_captcha_key)) +
        hidden_field(options[:object], :captcha_key, {:value => options[:field_value]}) : 
      text_field_tag(:captcha, simple_captcha_value(simple_captcha_key))
  end
end

Convert Filesize Bytes to Readable String in Javascript

Cary Dunn — Tue, 06 Jan 2009 20:34:44 +0000

function readablizeBytes(bytes) {
    var s = ['bytes', 'kb', 'MB', 'GB', 'TB', 'PB'];
    var e = Math.floor(Math.log(bytes)/Math.log(1024));
    return (bytes/Math.pow(1024, Math.floor(e))).toFixed(2)+" "+s[e];
}

readablizeBytes(5000); // 4.88 kb

Converting Rails 'created_at' DateTime's to Flot Timestamps

Cary Dunn — Tue, 06 Jan 2009 20:10:42 +0000

As far as graphing libraries go, Flot is simple, powerful, and produces quality graphs.

Graphing straight numerical data is very straightforward but there are times where you might want to graph numerical data over arbitrary periods of time. Converting timestamps from ActiveRecord to the timestamps used in Flot is as simple as...

objekt.created_at.to_time.to_i*1000

You can now pair these UTC timestamps with your data-point and input your array of tuples directly into Flot for readable time based labels.

Here is a simple example of a method I threw in a model to get a Flot compatible data string of some stat data for the object.

def get_stats_since(timeago=1.month.ago)
  ps = Stat.find(
    :all,
    :conditions => ["post_id = ? AND created_at > ?", self.id, timeago],
    :order => "created_at asc"
  )
  stat_str = ps.inject([]) do |result, p| result << "[#{p.created_at.to_time.to_i*1000},#{(p.hit_count.blank? ? 0 : p.hit_count)}]" end
  "[#{stat_str.join(",")}]"
end
=> "[[1231027849000,102204],[1231263741000,102374],[1231263746000,102454],[1231263753000,102194]]"

Flot is a pure Javascript plotting library for jQuery. It produces graphical plots of arbitrary datasets on-the-fly client-side.

Running MySQL on your Amazon EBS volume with RightScale

Danny LaBare — Mon, 05 Jan 2009 18:27:08 +0000

EBS (elastic block storage) is one of Amazon's newer innovations. It essentially offers the user a virtual hard drive of 1GB up to 1TB that can be mounted to any EC2 instance. There are several benefits to such a service, including replication of data (handled by Amazon) as well as the ability to snapshot a volume at any point in time and shoot it up to S3 (handled by the user). It is the latter benefit that interested me the most, and the reason it makes a lot of sense to store your db on this mounted volume.

Getting Setup

Here at ELC we are big supporters of RightScale™ so this tutorial will be driven with that in mind. To start off I simply created a new EC2 instance equipped with Nginx and Mongrel, and setup a very simple Rails app with one table (blobs) just to show the db was working properly.

Ok great, now we have our working Rails app and MySQL database running on EC2

Configuring the EBS Volume

RightScale™ makes it quite easy to create and attach an EBS volume to your instance. Simply click on the Volumes tab, then Create and Attach Volume. Choose a name and a size for it and click create, and in just a few minutes, we can see that the volume has been created at /dev/sdj:

[dlabare@domU-12-31-38-00-2D-82:/dev] ls -la
...
brw-r-----  1 root    disk   8,   3 Jan  5 12:50 sda3
brw-r-----  1 root    disk   8, 144 Jan  5 15:01 sdj
drwxr-xr-x  2 root    root       40 Jan  5 12:50 shm
...

Now let's format it.

[dlabare@domU-12-31-38-00-2D-82:/dev] sudo /sbin/mkfs.ext3 /dev/sdj 
mke2fs 1.39 (29-May-2006)
/dev/sdj is entire device, not just one partition!
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
6553600 inodes, 13107200 blocks
655360 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
400 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Great, now we've got our drive, let's mount it.

[dlabare@domU-12-31-38-00-2D-82:/dev] sudo mkdir /mnt/ebs
[dlabare@domU-12-31-38-00-2D-82:/dev] sudo mount /dev/sdj /mnt/ebs
[dlabare@domU-12-31-38-00-2D-82:/dev] df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             10320888   2080172   7716380  22% /
/dev/sda2            153899044    252388 145829032   1% /mnt
/dev/sdj              51606140    184272  48800428   1% /mnt/ebs

Cool, now we've got our EBS volume located at /mnt/ebs (it's only 50 gigs for demo purposes, you'd probably choose a larger size for anything serious)

MySQL Config

First thing to do is stop mysql for now

[dlabare@domU-12-31-38-00-2D-82:/mnt/ebs] sudo /etc/init.d/mysqld stop
Stopping MySQL:                                            [  OK  ]

Now let's move over the database files onto the EBS volume

sudo mkdir /mnt/ebs/lib /mnt/ebs/log
sudo mv /var/lib/mysql /mnt/ebs/lib/
sudo mv /var/log/mysql* /mnt/ebs/log/

Then we need to tell MySQL about where to find these files. Edit your /etc/my.cnf file to point to the correct locations.

[client]
#password       = your_password
#port           = 3306
socket          = /mnt/ebs/lib/mysql/mysql.sock

[mysqld]
datadir=/mnt/ebs/lib/mysql
socket=/mnt/ebs/lib/mysql/mysql.sock
...

##Inno DB
innodb_data_home_dir = /mnt/ebs/lib/mysql/
innodb_log_group_home_dir = /mnt/ebs/lib/mysql
innodb_log_arch_dir = /mnt/ebs/lib/mysql
...

[mysql.server]
user=mysql
basedir=/mnt/ebs/lib

[mysqld_safe]
log-error=/mnt/ebs/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

...

Now go ahead and restart mysql:

[dlabare@domU-12-31-38-00-2D-82:/mnt/ebs] sudo /etc/init.d/mysqld start
Starting mysqld daemon with databases from /mnt/ebs/lib/mysql
Starting MySQL:                                            [  OK  ]

Great, it worked*. Now save this config file to your EBS volume so it doesn't get lost if your instance crashes.

*crap, it didn't work? Check your mysql logs for more info (/mnt/ebs/logs/mysqld.log)

sudo cp /etc/my.cnf /mnt/ebs/my.cnf-bak

Volume Snapshots

You can manually take snapshots of your volume through the RightScale™ interface when needed, as well as have a cron job run back ups periodically. Eric Hammond has written a nice little backup script that that uses EC2 API tools, the script can be found here.

You can also read his detailed post on how to do this whole setup using the command line tools, which was a great help for me when getting setup.

Why this is a good thing

There are many good reasons to try this method of deployment:

Easily bump up (or down) your instance size without having to copy your database each time. Simply mount the volume to your new instance, copy over your mysql config, and you're good to go.
Rest assured knowing your snapshot data is backed up in multiple geographic locations
Know that your database storage is unaffected by your instance failing
Grow your volume size easily when space gets low

Xapian Vs Sphinx in Rails

Yuanyi Zhang — Tue, 06 Jan 2009 07:11:26 +0000

Xapian and Sphinx are both great open source full text searching solutions, but I'm wonder whose performance is better, Xapian has an official comparison to solr, but no sphinx, so I do it myself.

Dataset

My dataset is about 12,000 blog records like below:

class Blog < ActiveRecord::Base
  # Thinking sphinx indexes
  define_index do
    indexes :title
    indexes :description
  end
  
  # Acts_as_xapian indexes
  acts_as_xapian :texts => [ :title, :description ]

  ...
end

Environment

And here's the environment I used to do the testing:

Sphinx: 0.9.8
Xapian: 1.0.1
Sphinx plugin: thinking_sphinx
Xapian plugin: acts_as_xapian
Ruby: 1.8.6 p114
Rails: 2.1.0
Mysql: 5.0.51
OS: OS X 10.5.6
CPU: Intel Core 2 Duo 2.0G
Ram: 4G

Indexing

Let's start from indexing at first:

$ time rake xapian:rebuild_index models="Blog"
real    2m12.942s
user    0m24.569s
sys    0m12.798s
$ time rake thinking_sphinx:index
real    1m53.543s
user    0m4.725s
sys    0m0.887s

Sphinx is win in this section, and because sphinx has a progress report, I feel it's shorted than it occupied indeed.

Querying

Then is querying, below is my testing code:

keywords = ['key1', 'key2', ..., 'key30'] 

Benchmark.bm do |x|
  x.report("Xapian") { 30.times {|i| ActsAsXapian::Search.new "Blog", keywords[i], :limit => 10 } }
  x.report("Sphinx") { 30.times {|i| Blog.search keywords[i], :per_page => 10, :page => 1 }}
end

And here's the results, the winner goes .......

      user     system      total        real
Xapian  0.020000   0.020000   0.040000 (  0.042381)
Sphinx  0.150000   0.030000   0.180000 (  0.285173)

Yes, Xapian wins, but I'm afraid most of people will still choose sphinx, because acts_as_xapian is really hard to use than thinking sphinx.

At last, my testing is only based on a very small dataset, so it may be not correct in other situations.

Benchmarking with AutoBench

Max Murphy — Mon, 05 Jan 2009 23:50:19 +0000

You've got a rails app, an environment to deploy to, but how do you figure out if that environment can sustain the demand that your users create?

Enter httperf, autobench and gnuplot

To performance/load test rails applications, I've become a big fan of httperf. It has a multitude of configuration options and other various bells and whistles that really make it a jack-of-all-trades when it comes to application performance testing and load testing(both hardware and software)

The httperf package can be one of those intimidating command line tools, with more switches and arguments than one person can easily remember without using the software daily. AutoBench is a great wrapper for this that does some amazing extras. Let's dive in and see what we can do, starting with the config file:

# Autobench Configuration File

# host1, host2 
# The hostnames of the servers under test
# Eg. host1 = iis.test.com 
#     host2 = apache.test.com

host1 = test1.example.com 
host2 = test2.example.com

# uri1, uri2
# The URI to test (relative to the document root).  For a fair comparison
# the files should be identical (although the paths to them may differ on the 
# different hosts)

uri1  = /foo/bar/1
uri2  = /foo/bar/1.xml

# port1, port2
# The port number on which the servers are listening

port1 = 80
port2 = 80
# low_rate, high_rate, rate_step
# The 'rate' is the number of number of connections to open per second.
# A series of tests will be conducted, starting at low rate,
# increasing by rate step, and finishing at high_rate.
# The default settings test at rates of 20,30,40,50...180,190,200

low_rate  = 10
high_rate = 200
rate_step = 10

# num_conn, num_call
# num_conn is the total number of connections to make during a test
# num_call is the number of requests per connection
# The product of num_call and rate is the the approximate number of
# requests per second that will be attempted.

num_conn  = 500
num_call  = 10

# timeout sets the maximimum time (in seconds) that httperf will wait
# for replies from the web server.  If the timeout is exceeded, the
# reply concerned is counted as an error.

timeout   = 5

# output_fmt
# SET TO tsv to use with included bench2graph 
# sets the output type - may be either "csv", or "tsv";

output_fmt = tsv

## Config for distributed autobench (autobench_admin)
# clients  
# comma separated list of the hostnames and portnumbers for the
# autobench clients.  No whitespace can appear before or after the commas.
# clients = bench1.foo.com:4600,bench2.foo.com:4600,bench3.foo.com:4600

clients = localhost:4600

The above mentioned config file is created upon the first invocation of the autobench utility, which by default places it in ~/.autobench.conf. Once you have modified that file it's as easy as running:

 autobench --file output.tsv

My preference is to run a test suite stepping up from 500 connections making 1 request per second in 1 request intervals. This provides me what could be considered overkill on data samples but gives a very fine resolution of exactly where app/server starts to fall apart(requests fail, 500s thrown, etc.). This makes it very easy to find that performance sweet spot within the application, and gives a perfect baseline with which to test different server stacks against (mongrel vs. thin vs. passenger seems to be the debate of the day. Also, if you are able to access your data objects in multiple formats, be sure to test each of them, you'd surprised at what kind of overhead templating engines actually have.

Q:Now what do I do with all these ghasp Tab-Delimited-Files?

A: Generate Graphs!

The sparkling gem within autotest is the included bonus utility...... bench2graph!

  Usage          : bench2graph PARAM_1 PARAM_2 [ PARAM_3 ... PARAM_N ]
  Parameter 1    : autobench results file
  Parameter 2    : name of the postscript output file
  Parameter 3..N : [optional] columns to display in the graphic

This utility takes your data from the autotest suite(only if you save tsv and not csv) runs it through gnuplot and outputs a pretty little graph as a postscript file that looks like the one below:
Have fun with finding just how much can be thrown at your app :)

Weekly Bookmark Recap (01/04)

ELC Team — Sun, 04 Jan 2009 05:00:38 +0000

Thank you to everyone out there who maintains a personal or team blog to share their knowledge and findings! Here are a few of our teams favorite bookmarks from last week.

You can search our favorite links, bookmarks, as well as our own ELC Core content by using our new Explore feature. Check it out!

Tips and Tricks From a Self Proclaimed Cybernomad

Nicholaus — Sat, 03 Jan 2009 03:25:29 +0000

Last time I wrote about why I think distributed development is a viable way to work. Now I'm going to write about why it's an amazing way to work. Leading static lives can lead to static thinking. Dynamic lives lead to dynamic thinking. Remote development can free you from the dregs of traffic and 'the office', but can enslave you to your home (office). If you have worked from home before, you will know exactly what I am talking about. Some days you won't even leave your house. Office work will blend with house work and the next thing you know you're whole day feels like work. Ok, so you have the good fortune of being on a high functioning distributed team (like team jetpack) and you want to take it to the next level, here is some advice for how to travel, live and work without skipping a beat or missing a meeting.

The Setup

Obviously not everybody can just take off for long periods of time. It helps to be in a low/no rent situation at home base and not have any dependents (kids, pets or bf(s)/gf(s)). Subletting your room or apartment will help cover the bills while you are away. Direct deposit and internet bill pay are a must.

The Gear

First rule: Pack light (major bonus points if you can fly carry on)

Backpacking Pack

I think the backpack is a step up from the roller bags.
(supressed_rant) They are expandable, because you WILL collect more stuff as you travel around. The thing I really like about them, other than not having to roll them around, is that they are long and (can be) skinny. Which is prefect for fitting into the over-head bins and the compression straps make them easy to change their shape.

Pelican Case

The pelican case is a must for me. It's bigger, heavier and more expensive than a normal computer case, but still worth it. You must protect your computer. The Pelican case will keep your life line safe and sound. It locks, has this rad break-away foam so you can customize the interior shape. The bottom is a great place to store important documents like passport or extra cash. Toting around my Pelican adds mystique to my travel persona. Put stickers on it to gain some travel-cred.

Sound/Noise Isolating headphones

They are good for two reasons: you can tune out the world and the person next to you doesn't have to listen to the newest Diplo mix tape (which is absolutely awesome btw).

Microfiber towel

BOYTowel. Its small, super absorbent, quick drying and anti-microbial. I have this one made by Waboba (the waboba ball is also a big hit).

Chargers

My cell phone charger also works for my camera, so that cuts down on the clutter. But don't neglect the power adapters if you are leaving the country.

Geek Phone

Get a good geek phone. AT&T offers a Blackberry unlimitedinternational data plan for $80/month. I know, I know, what about the Iphone. Try and get unlimited international data for the Iphone. Then get a good jabber client and Google Maps. I can't stress how valuable Google Maps and its positioning feature are. Even the transit directions are super useful in the bigger cities, where it's supported.

The Travel

Second Rule: Don't let traveling interfere with your team involvement.

Okay, so now you are all geared up and ready to see the world. But don't forget you still have to work. In my first post on distributed teams, one of the team agreements was temporal. So you have to plan your travel times around your team. This goes beyond when you fly or ride the train. It also influences where you go, because working with a 12 hour time difference will start to affect your ability to work and communicate. I would not recommend more than +/- 8 time zones away from your teams agreed upon time zone.

Take the train; it's a great way to see the country side and get some quality hacking done. Save some money, byob.

Buy one-way tickets, because you never know when you might want to stay longer or leave earlier. Also one-way plane tickets can be really cheap, if you troll the right internet sites.

Keep offline documentation and e-books. You'll find yourself without internet and want to look something up.

Where to Work

It's obviously ideal to have internet where you are staying. But that's not always the case and what fun is it if you just stay holed up somewhere? Besides the obvious coffee shops and internet cafes, I also enjoy working from libraries, universities, and outside of random apartment buildings. Keep in mind noise levels, your own and others, when choosing a place for the day.

Where to Stay

I can't afford to stay in hotels every night; that's just not feasible. So be creative and opportunistic with where you travel and stay. Go and visit all the friends you haven't seen in a while. Make sure to be an awesome guest: tell rad stories, bring wine, do dishes, clean up. One of the best ways to fit in is to not require constant entertainment, get out and do things on your own.

Hostels are great, cheap ways to stay places where you don't know anybody. They usually have free internet and you are guaranteed to meet other cool travelers.

There are also international traveling communities online like Couch Surfing which is full of generous travelers offering and looking for places to stay for free. The high level of member verification makes it a safe and reliable way to travel, if you are willing to participate by sharing your home also.

Subletting is a great way to make a long term stay comfortable and cost effective. Craigslist has a lot of good opportunities to rent furnished apartments or rooms in houses, though not heavily used in Europe. Usually private and internet ready, they provide a stable and independent environment.

It doesn't matter if you are a globe trekker or just a local explorer, the point is to approach life like your software development. Become cybernomadic, be agile, have fun, live and learn. You'll be a better person and better developer for it.

[Solved] rake gems:refresh_specs to fix this

Jonathan Siegel — Sun, 28 Dec 2008 22:31:48 +0000

$ rake gems:refresh_specs
(in ./project)
config.gem: Unpacked gem encrypted_strings-0.3.1 in vendor/gems has no specification file. Run 'rake gems:refresh_specs' to fix this.
rake aborted!
uninitialized constant Rails::Initializer::EncryptedStrings

(See full trace by running task with --trace)

My vote for most circuitously unhelpful error recovery hint. Here's how to fix:

Hand-unpacked gem

The gem at issue here was copied by me like:

cp -r /Library/Ruby/Gems/1.8/gems/encrypted_strings-0.3.1 Projects/project/vendor/gems/

Usually this "just works" but for encrypted strings I needed the .specification file that is in the gem source. This .specification file was not installed in my /Library... copy. The solution was to add the gem to my config/environment.rb gem list (as I should have for clear dependency recording) and then let rake do the work.

  # In config/environment.rb:
  config.gem "encrypted_strings", :source => "git://github.com/pluginaweek/encrypted_strings.git"

And the unpack itself:

$ rake gems:unpack
(in ./project)
WARNING:  Installing to ~/.gem since /Library/Ruby/Gems/1.8 and
	  /usr/bin aren't both writable.
Unpacked gem: 'Projects/project/vendor/gems/encrypted_strings-0.3.1'

Now rake runs fine and the only file-level difference is the appearance of the .specifications file in vendor/gems:

$ git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	new file:   vendor/gems/encrypted_strings-0.3.1/.specification
</file>

Preview RDoc with Vim

Husein Choroomi — Wed, 31 Dec 2008 07:11:24 +0000

Many times I want to view RDoc HTMLized quickly without having to switch to terminal, to see if it looks good. I wrote a small method to do it from Vim.

Add the following snippet to ~/.vimrc or add it to ~/.vim/plugin/rdoc.vim

command! -nargs=0 RDocPreview call RDocRenderBufferToPreview()

function! RDocRenderBufferToPreview()
  let rdocoutput = "/tmp/vimrdoc/"
  call system("rdoc " . bufname("%") . " --op " . rdocoutput)
  call system("open -a Safari ". rdocoutput . "index.html")
endfunction

Restart Vim, open a ruby file and type:

:RDocPreview

And voilà!