Array.prototype.binSearch = function(needle, case_insensitive) {
    if (!this.length) return -1;

	var high = this.length - 1;
	var low = 0;
	case_insensitive = (typeof(case_insensitive) !== 'undefined' && case_insensitive) ? true:false;
	needle = (case_insensitive) ? needle.toLowerCase():needle;

	while (low <= high) {
		mid = parseInt((low + high) / 2)
		element = (case_insensitive) ? this[mid].toLowerCase():this[mid];
		if (element > needle) {
			high = mid - 1;
		} else if (element < needle) {
			low = mid + 1;
		} else {
			return mid;
		}
	}

	return -1;
};

Related posts:

1. Javascript For-in Loops
2. LongURL Integration For Your Website

When in HTTP mode HAProxy’s default health check is a simple OPTIONS request. This has the advantage of being a very lightweight request, and is easy to identify and filter from server logs. Consider this scenario though: HAProxy balances the load between several web servers running nginx and PHP-FastCGI. If nginx is up but PHP-FastCGI goes down, nginx will still properly handle the OPTIONS request from HAProxy, giving the impression that all is well. HAProxy continues sending requests to the ill server which in turn get a 504 Gateway Timeout (or similar) response. Not a very good situation.

A solution would be to use a deeper health check, one that goes beyond nginx to the PHP-FastCGI process. That way if PHP-FastCGI goes down, the whole server is presumed ‘down’.

backend appservers

 mode http

 option httpchk HEAD /health_check.php HTTP/1.1\r\nHost:\ example.com

 server web1 x.x.x.x:80 weight 5 check inter 2000

 server web2 x.x.x.x:80 weight 5 check inter 2000

 server web3 x.x.x.x:80 weight 5 check inter 2000

In the above example I’m using a custom health check request which will be processed by PHP-FastCGI. health_check.php is a lightweight script that contains simply <?php echo "I'm healthy"; ?>. I also added a host header so that the health check will be handled by a specific nginx virtual host. The nginx vhost config has this in it:

location = /health_check.php {

 access_log		off;

 fastcgi_pass	127.0.0.1:9000;

 fastcgi_index	index.php;

 include	/etc/nginx/fastcgi_params;

}

And there you have it–a better HAProxy health check for dynamic websites.

Related posts:

1. High Performance Comet on a Shoestring

This will search all files recursively for SEARCH_STRING and replace all occurrences of SEARCH_STRING with REPLACE_STRING throughout each unique file found. It also creates a backup of each modified file so that FILE is backed-up as FILE~ (with a tilde).

grep -R --files-with-matches 'SEARCH_STRING' . | sort | uniq | xargs perl -pi~ -e 's/SEARCH_STRING/REPLACE_STRING/'

No related posts.

Coming up with a suitable name for a business, product, or website is something I do on a fairly regular basis. In brainstorming a name I often make lists of words I’d like to use, like adjectives and nouns than relate the product. Then I start combining the words to create a unique name and check to see if the related domain is taken or not. The problem is that even though I may have come up with a name I really like, if the domain name is taken, it isn’t worth keeping.

These days a LOT of domains are taken, either by people using them or companies squatting them. So to make this whole process a little easier I built a website that takes most of the work out of hunting for a good domain: Trionym.The idea is fairly simple: enter up to three lists of words, choose which Top-Level Domains you’re willing to use, and search. Trionym will then create all the possible word combinations and check whois databases to see if the domains are registered. It’s relatively simple for now, but I’m considering adding more options, so if there’s a feature you’d like to see just let me know.

Related posts:

1. A Better HAProxy Health Check For Dynamic Websites

Introducing Comet to an existing architecture assumes there is already a web server in the neighborhood, and that it is, in one way or another, receiving traffic from port 80. Due to the fact that many site visitors will likely be positioned behind a firewall unwilling to accept connections on ports other than 80 or 443, we also need to get our Comet server running on port 80 as well. This normally wouldn’t be much of a problem at all, unless you don’t want to fork over the money for an extra IP address. I don’t & I didn’t. So let me show you how I did so.

As I eluded to above, to solve this problem of running two services on the same port in the same server environment you would normally have two different IP addresses assigned to the same front-end server. This is typically a load balancer or firewall, but these could also be running on the same machine as a web server and Comet server. The load balancer would then accept requests for x.x.x.1:80 and send them to the web server, and requests for x.x.x.2:80 would go to the Comet server. However if we only have one IP address that means we have to route requests based on a higher network layer, the Application Layer (7). Now we route by domain name.

In fact, that is something most web servers can handle using name-based virtual hosts. “So why not set-up Apache to reverse-proxy requests to the Comet server?”, you ask. Well, that would work. The reason Comet servers even exists though, is because web server connection threads are too heavy to support the level of concurrency Comet requires (for a decent number of users). This is where the “high performance” part comes in. HAProxy is a fantastic high performance layer 7 load balancer. Using HAProxy’s ACL feature we can basically mimic Apache virtual hosts. Consider this example snippet from haproxy.cfg:

frontend www *:80
    mode http
    acl comet hdr_beg(host) comet.
    use_backend meteor if comet

default_backend apachebackend meteor
    mode http
    server server1 127.0.0.1:4670

backend apache
    mode http
    server server2 127.0.0.1:8080

As you can see, I set up a front-end to accept all connections on port 80. Then I use an ACL to examine the HOST header and see if it begins with comet. (e.g. http://comet.example.com). If it does, the request is sent to the comet server on port 4670, and if not, requests go to Apache on port 8080. And there you have it, a high performance Comet installation with no money out-of-pocket.

Related posts:

1. A Better HAProxy Health Check For Dynamic Websites

This is just a quick note for anyone who has to deal with Network Solutions hosting (for clients or otherwise). Apperantly Network Solutions doesn’t run PHP with Apache’s mod_php, but rather as a CGI application. That means if you try to put PHP configuration directives in a .htaccess file it will cause a 500 Internal Server Error. My helpful reference: Because Network Solutions Sucks.

You have been warned.

Related posts:

1. High Performance Comet on a Shoestring

I still need to add support of internationalized TLDs and HTML. So, although it only works with plain text for now, I think it’s off to a good start. Check out the demo or download the source.

No related posts.

I noticed that Twitter search lets you expand URLs from some of the most popular shortening services (though, if they used LongURL they could support a whole lot more ). I liked their UI approach; it’s clear, simple, and effective. So I pretty much mimicked their UI for my plugin.

So here it is, you can add support for LongURL to your site in just a few lines of code:

<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.2.6/jquery.min.js"></script>
<script type="text/javascript" src="http://longurl.org/static/jquery.longurl.js"></script>
<script type="text/javascript">
$(function() {
    $('a').longurl();
});
</script>

Yup, that’s it. Also, if you don’t like the Ajax working indicator that I chose, you can use your own like so:

$(function() {
    $('a').longurl({working_image: 'http://example.com/images/ajaxy_animation.gif'});
});

So, that’s how you use the plugin. You can see a demo of it in action, and download it from the jQuery site. Hope you find it useful.

Related posts:

1. LongURL—Restoring Order to the Universe

I decided CouchDB would be a good fit for my next project (which I should be releasing sometime this week BTW) and rolled up my sleeves. Because of the amount of data I’m working with, I hit a few snags along the way with regard to CouchDB view performance. Some of the things I learned, although they make sense, were not what I was expecting initially (even after reading all the docs). So for the benefit of others, I thought it’d be a good idea to share my current understanding of the way views work in CouchDB, and share some of the tips & tricks Jan, Chris, and others have given me along the way.

Importing Data for Speed and Glory

Most of the work I’ve done with CouchDB this past week has been related to importing a fair amount of data (600k+ documents). Initially I tried creating one document at a time. This worked, but each request has associated with it a certain amount of overhead and latency. For example, creating 33,847 documents, one at a time, took 726 seconds (~12 min). Thankfully CouchDB has a bulk create mode. Creating the same documents 1,000 at a time took 58 seconds. That’s a 1,250% improvement! An added benefit of using bulk create is that it consumes less hard disk space (28.2MB vs 213.2MB).

CouchDB Views vs RDBMS Tables

Now that I could get my data in the database in a decent amount of time, I wanted to aggregate some of it together in a view. Before I get into too many details, let me explain how I think about CouchDB views. I’m a very spatial thinker, and so visualizing the similarities and differences between CouchDB views and traditional RDBMS tables helps me to understand how they work. It may be stupid, it may be naive, it may even be wrong, but here goes: Imagine a RDBMS database. Imagine a handful of tables in that database, each with different columns. Now imagine that every row in every one of those tables is just a document in CouchDB, all lumped into the same bucket (database) and with no hierarchy. Views are what filter and aggregate documents together to create (in a very limited sense) the equivalent of a table.

You don’t join views with each other because you’re already essentially “joining” documents (rows) to create the view. This might get you thinking that CouchDB views relate better to RDBMS views. In some ways that is true, but RDBMS views are a one-time snapshot of the underlying tables, and so for the sake of this discussion I’m leaving RDBMS views out.

Indexing, the  Slowdown

CouchDB view indexes are generated when the view is first called. More than one view can be stored inside a design document, but as long as they’re in the same doc they get generated (and updated) at the same time. After that initial creation, updating the view indexes is incremental based on what documents in the database are added, edited, or deleted. Notice I said indexes are updated based on what documents are modified in the database. This is an important point!, something that wasn’t obvious to me initially. There are no tables. This is no hierarchy. No isolation. Modifying any documents in the database means that all view indexes in all design documents have to be updated.

The only time view generation is isolated is when a new design document is created or updated. In this case, though, the process is not incremental. For this reason, if you plan to store a large number of documents I strongly suggest that you work out and create your design documents before populating your database. Although CouchDB is a schema-less database, creating views for a large data set is currently much like designing a schema: you mostly do it before filling your database with data, and you generally don’t change it often. Why? Because generating views is currently a slow process, and gets slower the more documents you have. As a point of reference, for my example of 33,847 documents it took me 6,705 seconds (~1.86 hours) to generate a view for the first time. Retrieving the view after that took 0.006 seconds.

Improving Speed Now, and in the Future

In some cases it is possible speed up view generation by priming the view as you create documents. This method has great results. For example, if I create 33,847 documents in batches of 1,000, calling my view after every bulk create, the whole process takes 219 seconds (~3.65 min). If we compare the time it takes to insert the documents and then generate a view separately vs doing them at the same time, the latter is 3,088% faster (58 + 6,705 / 219 = 30.88).

CouchDB uses an implementation of MapReduce for generating views. Currently though, view generation cannot be distributed across several nodes. I’ve been told this feature is on the development roadmap, and so chances are view generation will get much, much faster in the (hopefully) not too distant future. Also worth noting, is that CouchDB has not yet been optimized, and Damien is quite optimistic about its potential, as am I.

Now, I have only been working with CouchDB for a week, so it’s quite possible my understanding of something might be off. If so, please correct me. Working with CouchDB has been a load of fun (and education). I’m really looking forward to where it goes in the future, and I hope to do what I can get help get it there.

UPDATE: For anyone that might be interested, you can get the import script I’m using from my Launchpad.net repository. Depending on what you’re doing it might be a decent start. The script has a few nice features like resuming interrupted imports, bulk inserts with priming, and graceful handling of failed bulk inserts.

No related posts.

After getting feedback from some helpful beta testers (thanks Marjolein and Børge!) and making a few tweaks I was happy yesterday to release the LongURL Mobile Expander Greasemokey script and Firefox extension which use the LongURL API to expand shortened URLs on any web page. I haven’t had much feedback from others yet, but for me personally, the extension tremendously improves my user experience. Sorry guys, no more rickrolling me!

Also, with the release of the incredibly awesome Ubiquity extension from Mozilla Labs, I started work on a LongURL command which I hope to finish/release in the next day or so I just released. You can subscribe to the command from the LongURL Tools page. Does that sound like something that would be of interest to you? Have any ideas on how you’d like it to work? If so, leave a comment.

I would like to mention too that this has been the first time I’ve really used the launchpad.net platform for managing my project. It’s been a super experience; I don’t know why I didn’t start using it sooner, especially since I’ve been so close to it. And bazaar? Well, let me just say that I’ll no longer be using Subversion as my VCS of choice. Bazaar rocks, and if you haven’t given it a try, I strongly suggest you do so. You won’t regret it.

A final note: as you can tell, I don’t always get around to blogging about cool stuff I’m working on right away (if at all), so if you’d like to get updates as soon as possible I suggest following me on Twitter (iamseanmurphy) and/or identi.ca (seanmurphy). 140 characters just seems so much more digestible to me.

Related posts:

1. LongURL Integration For Your Website