Documentation for setting up a MongoDB Replicaset with OpsWorks was a bit on the sparse side and took a bit of trial and error. I ended up with this solution.

The Chef Recipe

The chef recipe used for this post is https://github.com/netinlet/chef-mongodb_replicaset

OpsWorks allows you to use custom recipes. All dependent recipes should be included in the custom cookbook.

The mongodb recipe depends on apt, python, runit and yum recipes. Several recipes require the build-essentials recipes and the runit recipe depends upon yum-epl.

The dependencies for this cookbook were pulled from the OpsCode public cookbooks. https://github.com/opscode-cookbooks

The mongodb recipe is custom. I ended up using this recipe referenced in the presenation AWS OpsWorks under the hood.

The custom recipe is based off of https://github.com/edelight/chef-mongodb with OpsWorks specific code to properly setup replicasets in AWS OpsWorks.

OpsWorks Setup

I’m going to assume some degree of familiarity with OpsWorks. If you are new to AWS OpsWorks, you can get an overview here: http://aws.amazon.com/opsworks/

Add a MongoDB layer to your stack

Create a “Custom” layer named MongoDB and add it to the stack.

Configure the MongoDB layer

A custom layer is pre-defined with 12 built-in recipes. Nothing to do or change here.

Custom Chef Recipe Section

Repository URL Set the repository URL to the location of your chef recipe. This repository can be a link to a Git or Subversion repository or an http or S3 url. The custom recipe is configured in the Stack settings in which your layer lives. In this case, I have it pointed to my Github chef recipe.

OpsWorks has 5 lifecycle events which can be used to execute Chef recipe steps. These events are setup, configure, deploy, undeploy and shutdown. For this recipe, we will use the setup and configure events.

Setup event Add mongodb::10gen_repo and mongodb::default. The former configures the service to use the 10Gen MongoDB builds. The latter installs MongoDB and does a basic single node configuration.

Configure event Add mongodb::replicaset. This event configures the MongoDB instance as a member of a replicaset.

Custom Chef JSON In order for the replicaset to be configured properly, we have to specify the name of the replicaset, cluster and tell the instance to auto-configure.

The replicaset_name and cluster_name must match. You must also set the auto_configure:replicaset hash value to true so the replicaset is auto-configured on boot.

{
  "mongodb" : {
    "replicaset_name": "mongo-rs",
      "cluster_name": "mongo-rs",
      "auto_configure": {
        "replicaset": true
      }
  }
}

Screenshot of editing the Stack where the custom recipe and Chef JSON are configured

Boot your MongoDB Instances

Add some instances to the MongoDB layer and boot them. An odd number of instances (3 or more) is the proper way to configure a replicaset without an arbiter.

Unfortunately, on first boot the primary replicaset member is unable to communicate with the secondary replicaset member. I was unable to determine exactly why this happens but suspect a timing issue with the availability of the secondary replicaset member.

This is evident in the logs of the second instance. This error indicates the node was unreachable at the time it was being configured. See line 4.

[2014-01-12T23:15:37+00:00] INFO: ruby_block[run_config_replicaset] sending create action to ruby_block[config_replicaset] (delayed)
[2014-01-12T23:15:37+00:00] INFO: Processing ruby_block[config_replicaset] action create (mongodb::replicaset line 178)
[2014-01-12T23:15:42+00:00] INFO: Configuring replicaset with members adrastea, venus
[2014-01-12T23:15:42+00:00] ERROR: configuring replicaset returned: #<BSON::OrderedHash:0x3fa7491cb9f4 {"errmsg"=>"exception: need most members up to reconfigure, not ok : ip-172-31-0-241.ec2.internal:27017", "code"=>13144, "ok"=>0.0}>
[2014-01-12T23:15:42+00:00] INFO: ruby_block[config_replicaset] called

The work-around is to run the mongodb::replicaset recipe again from the Stack section of OpsWorks with the Run Command button.

*Note: MongoDB instance names changed because I removed the nodes before realizing the original screenshot did not get saved properly.

Verifying the configuration

If everything was successful, you should be able to ssh to your MongoDB instances and look at the MongoDB configuration with the commands mongo to connect to the instance and rs.conf() to show the current replicaset configuration.

This screenshot shows the before and after of the above mentioned mongodb::replicaset command work-around.

A less common method for creating and using Ruby hashes is to initialize the hash with a block. This allows you to return and/or assign a value for a missing key. You can think of this as a parallel to using method_missing in Ruby classes and modules.

The documentation for Hash.new with a block suggests

If a block is specified, it will be called with the hash object and the key, and should return the default value. It is the block’s responsibility to store the value in the hash if required.

Most often for the case where I wanted a default value returned, I would use fetch and set the default value for individual keys being looked for.

my_hash = {}
my_hash.fetch(:foo, "Not Set")
=> "Not Set"

my_hash[:foo] = 'bar'
my_hash.fetch(:foo, "Not Set")
=> "bar"

The same result can be achieved by initializing the hash with a block - any unknown keys requested from the hash will be initialized with a default value.

my_hash = Hash.new{|hash,key| hash[key] = "Not Set"}
my_hash[:foo]
=> "Not Set"

my_hash[:foo] = "bar"
=> "bar"

An alternative to initializing Hash with a block is to set the default_proc= on an existing hash.

my_hash = {}
my_hash[:foo]
=> nil

my_proc = proc{|hash,key| hash[key] = "Not Set"}
my_hash.default_proc = my_proc
my_hash[:foo]
=> "Not Set"

So now we have a way of always returning a default value for a hash. Not very interesting in and of itself. What else could we do with our block backed hash?

##A simple loadbalancer…

This is one of my favorite hacks. At MileMeter, we were using our Google Apps account to send email (reminders, confirmation emails, etc). At the time (not sure what the limit is now) we were limited to 500 emails per day from a single account. At some point, we started bumping up against the 500 email per day limit.

It was easy enough to create another email account to send emails through, but ActionMailer only provided a single configuration and we didn’t want to upgrade Google Apps for all MileMeter users because of a single account limitation.

John Riney and I looked at our options went in search of a solution which would allows us to send more emails without spending any additional money.

John found the Hash documentation and realized we had a solution in what he deemed “Untrustworty Ruby Hashes”

It was easy enough to create additional emails accounts. Armed with a Hash initialized with a block, we can now tell ActionMailer to pick a random user_name from a list of accounts each time it sent an email, thus effectively load-balancing our email.

A typical ActionMailer configuration is

config.action_mailer.delivery_method = :smtp
config.action_mailer.smtp_settings = {
  address:              'smtp.gmail.com',
  port:                 587,
  domain:               'example.com',
  user_name:            '<username>',
  password:             '<password>',
  authentication:       'plain',
  enable_starttls_auto: true
}

Our new load-balanced ActionMailer config is initialize with a block and configured to return a different user_name each time the user_name key is accessed.

# list of email accounts to send from
ACCOUNTS = %w{system1 system2 system3 system4}.map{|name| "#{name}@example.com"}
# => ["system1@example.com", "system2@example.com", "system3@example.com", "system4@example.com"]

# NOTE: user_name key not set
email_config = {
  address:              'smtp.gmail.com',
  port:                 587,
  domain:               'example.com',
  password:             '<password>',
  authentication:       'plain',
  enable_starttls_auto: true
}

config.action_mailer.delivery_method = :smtp
config.action_mailer.smtp_settings = Hash.new do |hash,key|
  ACCOUNTS[rand(ACCOUNTS.size)] if :user_name == key
end.merge!(email_config)

Assuming the password is set the same for all accounts you want to use, when an email is sent, the user_name for the account is looked up and set to a different value every time, effectively a simple load-balancing technique.

irb(main):037:0> smtp_settings[:user_name]
=> "system1@example.com"
irb(main):038:0> smtp_settings[:user_name]
=> "system4@example.com"
irb(main):039:0> smtp_settings[:user_name]
=> "system2@example.com"
irb(main):040:0> smtp_settings[:user_name]
=> "system2@example.com"
irb(main):041:0> smtp_settings[:user_name]
=> "system3@example.com"
irb(main):042:0> smtp_settings[:user_name]
=> "system1@example.com"
irb(main):043:0> smtp_settings[:user_name]
=> "system3@example.com"
irb(main):044:0> smtp_settings[:user_name]
=> "system1@example.com"
irb(main):045:0> smtp_settings[:user_name]
=> "system3@example.com"

I just finished moving all our repositories from subversion to git.  There is lots of documentation on the web, but none of it ties everything together.  Specifically, once you get your subversion repository including all branches and tags imported into a local copy of git, how do you push your git repository including all branches and tags to a remote git server?   This was important to us because we have ongoing work in some branches which are not ready to be pushed to trunk yet.

I started out by following the excellent instructions at http://scie.nti.st/2007/11/14/hosting-git-repositories-the-easy-and-secure-way to setup our own git repository.  Follow these instructions and create a simple git repository with only a couple of files to make sure it works and become familiar with it.  At this point, you have a working central git repository but you are not yet ready to import your subversion repository.  If using github, you can skip this step.

Next, go grab git2svn from github.  This will download an entire subversion project, including branches and tags, into a local git repository.  Depending on the size of your codebase and number of branches/tags, etc, this may take a while.  Our smaller projects took about 10 minutes.  Our larger project took almost an hour.  It differs from git-svn in that it converts anything in subversion tags to actual git tags.

Now for the magic of pushing your local git repository imported from subversion to the central git repository your team will use.   At this point, you should have a working central/remote git repository you are going to use.

Inside the root of your local git project, run these commands, replacing everything which is in caps. git remote add origin git@YOUR_SERVER_HOSTNAME:YOUR_PROJECT_NAME.git git push origin master:refs/heads/master

After this runs, you have only pushed the trunk to the remote repository (now called master).  You will also want to push all your tags and branches.  With git, you have to explicitly push your tags.

And now for the branches.  This was the only piece I could not find on the interweb.  This is achieved by the command

The --all command per the documentation specifies that all refs under $GIT_DIR/refs/heads/ be pushed rather than naming each ref to push.

That should be all there is to it. Our other developers were able to clone the newly minted git repository and start developing where they left off on their branch with the command:

The one thing that had been getting me was the fact that the branch I created locally did not always stick to following the remote branch. It would start out that way, but once I checked out the master branch again, it would not always and definitely not consistently point toward the remote branch once I re-checked out my branch.

I finally figured out that the local and remote branch must be named differently. The character case matters! So if I had a remote branch of ITR-FOO and a local branch of itr-foo, it would loose track of the remote branch. The easy solution is to name the branches differently by using a “-” remotely and a “_” locally for names or just name the branches differently. Now, I use ITR-FOO remotely and itr_foo locally, or shorten the local name to itr_promos if I am using ITR-PROMOTIONS remotely.

For merging branching, using git is so much simpler, even if it is very retarded sometimes.

A big hats off to Chris Gay for working so hard for so many years so that we can could get to this point. Insurance rules had to be changed and it was difficult for many people to see the benefit selling insurance this way.

We are also one of very few, if not the only insurance company powered entirely by Ruby It certainly gave us an advantage as we were able to get a robust, well tested application out the door so quickly.

It quickly turned into a somewhat fun conversation with lots of people and jokes. Rather quickly a google group named Pradipta’s Rolodex was created. Also followed was a Wikipedia entry (quickly deleted by Wikipedia admins), a FaceBook group, at least one domain registration as well as posts to Digg, Reddit and others.

Oh, the friday fun…..

@response.should have_text(" ")

We are proud to be 1 of 7 Finalist for the Amazon.com AWS Startup Challenge and will give our presentation next week at Amazon.com HQ. Milemeter is an innovative insurance start-up that will offer “auto insurance buy the mile”.

Please Vote for Milemeter!

Update: Congratulations to Ooyala for winning the AWS Startup Challenge.

MyClass.send(:public, *MyClass.protected_instance_methods)
MyClass.send(:public, *MyClass.private_instance_methods)


Just place this code in your testing class substituting your class name. Include the namespace if applicable.