Mapping (Multiple) Legacy Databases With Datamapper

We're starting in on a new project at work and it comes with some unique requirements (don't they all?)

To get things going so the rest of the team can take over and run with it I spent a couple days figuring out Datamapper the ruby ORM and mapping a number of legacy data tables.

This application is unique in that it will run against 3 separate databases. Two of them come from an older version of the application that the client built several years ago, we'll call these databases db1 and db2. The third database is where any custom data will be held for the web interface itself. We'll call it web1.

The first trick to getting datamapper to work in a rails applications is knowing what to actually install. Datamapper is a general purpose ORM and can be used quite effectivley outside of rails, as such the documentation on datamapper.org doesn't really focus on rails. No problem, take a look at the dm-rails gem.

dm-rails overrides a number of ruby on rails default generators, patches rake db:migrate to work with datamapper upgrades and automigrates and even integrates well with rspec and cucumber. Also worth noting that it worked fine on the latest and greatest (as of this blog post) - Rails 3.1.3 with Ruby 1.9.3.

Part of setting up dm-rails is getting your database.yml configured correctly. The documentation on the github page is great but as an added example here is a version of my config (with secure info removed.)

db1_defaults: &db1_defaults
  adapter: mysql
  username: [USERNAME]
  password: [PASSWORD]
  host: localhost
  database: db1

db2_defaults: &db2_defaults
  adapter: mysql
  username: [USERNAME]
  password: [PASSWORD]
  host: localhost
  database: db2

web1_defaults: &web1_defaults
  adapter: mysql
  username: [USERNAME]
  password: [PASSWORD]
  host: localhost
  database: web1

development:
  database: web1
  <<: *web1_defaults
  repositories: 
    db1: 
      <<: *db1_defaults
    db2: 
      <<: *db2_defaults

test: &test
  database: web1_test
  <<: *web1_defaults
  repositories: 
    db1: 
      database: db1_test
      <<: *db1_defaults
    db2: 
      database: db2_test
      <<: *db2_defaults

Once your database is configured you should be able to run rake db:migrate and see success messages. At this point you have no models configured but you can see that it can connect correctly.

The next step is mapping your existing tables.

db1 and db2 have quite a bit of existing data, their own data formats and a somewhat inconsistant naming structure. This resulted in a lot of custom mapping and eventually I just decided to explicitly spell out every mapping for consistency (even if the datamapper conventions would have applied.)

Here's an example model file. Again I've removed business context by renaming some things. I'll call out some pitfalls after.

class Procedure
  include DataMapper::Resource

  def self.default_repository_name
    :db1
  end
  storage_names[:db1] = "procedures"

  property :id, Serial, :field => "id"
  property :name, String, :field => "procnam"
  property :length, Integer, :field => "avgminlen"

  validates_length_of :name, :max => 25
end

The first important thing to note is this method declaration.

  def self.default_repository_name
    :db1
  end

This is how you tell Datamapper which database this model's table is in. In this case Procedures come from the db1 database.

Next thing worth noting is:

  storage_names[:db1] = "procedures"

Which is how you set the table name. In this case the table was pluralized. If you find that all of your tables follow a naming standard (something like tblProcedure) there are ways to override the naming globally. In this application naming standards were fairly inconsistant so I ended up adding this line in every model even when it was technically unnecessary.

Also for each varchar type column I added a length validator. The length of the varchar varied wildly in my application from 1 char fields up to 50 varchar fields. The best way I could come up with to deal with that was to simply validate that it never overflowed. Would love suggestions on any other config flags I could add if need be so datamapper knows what it is structurally.

One last thing to note. The override to default_repository_name MUST BE BEFORE THE PROPERTY DECLARATIONS. Sorry for the all caps. I spent about 2 hours fighting datamapper before figuring that one out. I even made a SO post on it before i figured it out.

Codemash 2.0.1.2

2012 marks my fourth year at Codemash. I've written about this conference before but in short, if you don't go to Codemash you should. For your (or your companies) dollar it is the best value you will get. If you run a company you should send your people to Codemash, or sponsor it, maybe even host a bacon bar (thanks DI).

So the promotion is out of the way, what I really want to do with this year's recap is talk about each of the sessions I attended because they were all phenomenal.

The Scala Koans - Dianne Marsh & Daniel Hinojosa


The first precompiler was a half day session that covered the Scala Koans. The Scala Koans are a Scala implementation of the Ruby Koans.

Koans, in the context of computer programming, are "short, enigmatic stories that usually involve some kind of epiphany. (Jim Weirich)." The idea of a language koan exercise is that by completing a series of short questions you will learn the core of a language. Jim Weirich and Joe O'Brien really started this approach to leaning programming languages with the aforementioned Ruby Koans but now implementations exist in Scala, Objective-C, C# and many other languages.

Scala is a pseudo-functional, pseudo-object oriented language that has static typing with a dynamic style syntax. So.. it's a Mutt, but a very interesting Mutt. As a ruby / C# dev I saw a lot of promise in having the power of a runtime like the JVM and static typing with the comfort of ruby-style syntax.

If nothing else it's worth checking out the Scala Koans to get a really quick overview of another language.

Advanced Patterns with Ruby on Rails - Jeff Casimir

This precompiler was targeted at an existing ruby developer with a few projects under their belt that wanted to learn more about applying various design patterns or concepts to their ruby (and rails) applications.

Among the topics covered was isolating business logic and limiting law of demeter violations with Draper by using the decorator pattern.  Also slimming down controllers by refactoring calls to activerecord into one method and limiting the number of instance variables used in the view.

Keynote: Rethinking Enterprise - Ted Neward


Thursday's morning keynote focused on how enterprise software needs to reevaluate what it values. How reuse is no longer as significant as it was in the past, how we should be not only looking at the tools we are comfortable but looking to find the best tool even if it isn't part of out "enterprise stack." It warned agains being technist "This platform is always better than that platform."

It's the Little Things - Brad Colbow


This talk focused on the little issues in UX design and how when put together they can add up to larger problems with your applications. He used the example of the Cleveland Public Library (and really Overdrive.) He previously did a comic strip to express his frustrations with their interface for downloading audiobooks which became quite a popular strip. How taken individually the issues he ran into were minor but as a collection completely infuriating. He advocated small changes like remembering to turn off autocorrect or autocapitalize on a login form for iphone users.


Brad also talked a bit about his experience building an iPad magazine and how so many of the other iDevice magazines tend to make bad assumptions about their audience and use custom gestures that are confusing and easily forgot.

Building Windows 8 Applications with HTML5 and jQuery - Rich Dudley


Rich covered the new Windows 8 developer preview and how to use HTML5 to craft a metro style application that can be deployed via the Windows Store.

Regardless of what you think about Windows 8 I'm really excited about the prospect of using HTML5 and CSS to layout the UI of a client application. I've always thought that Microsoft's previous attempts to handle UI layout were really poorly implemented. HTML in my mind is a more intuitive way to make a UI and I prefer working at a markup level than using a designer (yes i know you could layout xaml in an text editor too.)

It looks like Microsoft is serious about not breaking the standards based nature of the markup too, which is encouraging. Looking at the examples shown they make liberal use of the data- attributes in order to bind data and actions but all of that is standards compliant. I saw no custom metro only tags only attributes and data attributes. WinRT is implemented through a javascript layer that can be granted system level access.

A number of precautions are taken with the applications including making the developer select one of two modes, essentially net or client. Net mode sandboxes the application and it's data and prevents access to most system resources. Client mode provides system access but isolated most network calls. The apps run like mobile apps, they do not remain in memory when switched away from but instead must have their state stored and recovered when focus changes.

In short metro looks promising, especially for non-.net devs who want to build something on windows but don't want to get too far out of the comfort zone of web development.

Mastering Change with Mercurial - Kevin Berridge

So to be perfectly honest I wasn't going to go to this talk except Kevin's an old college friend of mine and I wanted to be there to support his first talk at a larger conference like Codemash. Before I left though I was completely blown away and realized I was very much being a technist like Ted Neward had described.

See there's a bit of a git / mercurial rivalry. Kind of like vim / emacs or other opposing tools. Developers love rivalries and I very clearly have fallen on the git side. Truth be told though that has more to do with Github being awesome than git itself. So i spent most of Thursday harassing kevin (jokingly) about mercurial and asking if he was going to admit at the end of his talk that git was simply better. Instead he came out swinging and knocked everything I knew about distributed version control on it's proverbial ass.

Mercurial is awesome... dammit... it pains me to admit it and then publicly post it in this blog but it really is.

So first off kevin did the best "intro to the thing I'm about to talk about" that I've ever seen. Most "advanced" talks end up spending 30 minutes doing an intro and barely ever get into advanced content. Kevin played a minute long video and narrated it and if you had any experience with distributed version control (including git) you were now up to speed. Seriously i basically never used hg but the video was all i needed to make the connection to my normal workflow in git.

He then talked about stuff like viewing and searching history, simple aliases or tricks that can make your life better, and ended with some of the coolest history management stuff I've ever seen using hgs queue's extension. I may also be convinced that hg's way of labeling branches (by default) is better than how git does it.

The plan now is to get Kevin's slides, convert them to git and then jointly present the topic at burning river devs as how to do these things in both worlds.

CoffeeScript is for Closers - Brandon Satrom

Brandon's talk was an honest evaluation of why Coffeescript is worth looking at. He made every attempt to avoid the CoffeeScript > Javascript false debate people tend to have online. Instead he made a number of points as to why, even if you stick with normal JS for your daily work, you should give coffeescript a shot. The argument that best resonated with me was that using CS and then looking through the JS code is compiles into you can get an idea of how to be a better JS programmer. CS doesn't compiled into impossible to read minified JS code, instead it compiles into very readable code that makes every attempt to follow best practices. If you're a bit of a JS noob it might be worth rewriting some of your stuff in CS, compile it and then see how it turns out.

Dealing with Information Overload - Scott Hanselman

So i went to this talk because Scott Hanselman was speaking. I really didn't even read the title or abstract. It was added as a talk kind of last minute because of some other cancellations.

Scott is a freaking great presenter, which makes sense as that's what he does. This talk could have been a keynote. It focues on real world hacks to make your life easier to handle. Things like not answering email until a set time so that you can establish the precident that you work in the morning when your most "fresh." One hack I actually did right away was setting up an inbox rule to redirect email where I am cced into another folder meaning I focus on the email that I am directly referenced on.

Be The Input - Kinect With Your Computer - Ben Barefield

Ben's talk was on the Microsoft Kinect API for windows. Ben was lead dev on a small racing game that uses the Kinect sensor to control the carts on the screen. His company, SRT Solutions, had the game on display at their booth.

The thing that most surprised me about this talk was how easy it is to get up and running with both Kinect and the XNA platform for game development.  Ben was able to build a small test app that tracked a user's hand on the screen using Kinect in about 20 minutes during the talk. A really impressive demo and now I need to learn a bit more about this platform.

Everything Else


Some of the best parts of a conference like this don't happen in the session but rather in the side conversations you have with friends, peers and strangers in the hallways. I learned about dynamic vs static typing on a more fundamental level, closures, JVM backed languages, the logistics of spinning up dynamic servers with chef and vagrant and a million more things meeting with people in the halls and chatting over beers at the evening parties.

Codemash is worth it for just the inbetweens. The 20 mins between sessions and the events that occur between Thursday and Friday. You can't possibly make it to everything but the joy of codemash is you can always go next year.




Edit: Left off the speakers for the precompilers. Sorry.