Monday, September 13, 2010

Messaging Is Not For Queries

As I’ve stated, I’m willing to eat crow on my path to success. This post reflects what I have unfortunately learned the hard way. In some of our applications we have used messaging in a Send / Reply pattern using messages like FindAddressesForCustomerMessage and FindAddressesForCustomerResponseMessage. In the beginning this worked just fine. However, after we began to grow and scale became more and more of a concern this began to cause us a lot of pain. I’ll state below the pain points of using a service bus / messaging in this manner.

To understand the needs of our system we have  ~10 application / web servers and ~1000 users of our CRM system. Collectively they all participate in service bus communication.

Messaging with any of the .NET service bus projects has many things to prevent failure of any kind, can withstand restarts, failover, load balancing, transactional message processing, etc. This is great if you take a step back for a moment to consider where this is important. Capturing the commands and events of users into a persisted store somewhere. When a user does something we’re pretty dang sure it will be successful. No longer needing to display the message to our user, sorry this failed please try again later.

Using messaging for queries forces an asynchronous pattern when it doesn’t necessarily need to be, making the system much more complex to work with. This doesn’t mean you shouldn’t use messaging at all. I could see using a message when the application starts up that has a response used to populate a list of users / password hashes in a local cache for example. This doesn’t mean that the query side is asynchronous though it just means that it looks to a cache for the information and has a dependency that a response is returned before someone can successfully logon.

The continual retry on failure for messaging implementations causes problems when user endpoints aren’t online. For example, we had a shift that would leave and the system would slow down. Why? The outgoing messages would retry over time up to as many as 2 days causing bottlenecks for legitimate messages that needed to get through.

Messaging doesn’t handle large messages very well. Let’s face it sometimes messages for queries aren’t simple things that are composed of 3 or 4 fields. This forces us to work around the size limitations put in place for all messaging solutions.

Can these things be overcome? Sure, I could invest my time heavily in addressing my concerns as they relates to the service bus code-base. However, my needs might not be compatible with the needs over others and therefore having to maintain my own version. It’s okay to admit when we were wrong in our assumptions. You can disagree with me and tell me about all of the various changes or tweaks that could be made. However, all I will be thinking is that you haven’t actually tried this for yourself. Next time it won’t be me eating crow.

So what is the right solution? My recommendation is to spend some time reading and learning about CQRS (Command Query Responsibility Segregation) and pay attention to the distinct separation of queries and commands.

Tuesday, September 7, 2010

MongoDB for .NET with NoRM DbReference clarification

In my last post. I mentioned DbReference and how I wasn’t fond that there didn’t appear to be any way to specify that a property was in fact a reference to another document inside an implementation of MongoConfigurationMap. To be fair, this is likely not a common need when dealing with document databases and likely not needed at all for what I’m considering using a document database for, the query piece of CQRS where I will persist a view model directly as it is displayed on the screen so that no translation or multiple reads are needed. Below is a sample of the current API in NoRM for DbReference. Then to retrieve an order and it’s products.

So if you use nHibernate you’re likely seeing the same things that bother me with this API. The class Order is not POCO, and you must explicitly pass a Func<IMongo> in order to load the related products. Ideally I think the API below makes more sense and keeps things cleaner.

Then the query to eager load the related documents.

To be honest I’m not sure how difficult my proposed change would be, but I may take some time to see what it would take regardless of my current needs. I want to understand more of the internals and what a better way to understand than to contribute.