Messaging Is Not For Queries

As I’ve stated, I’m willing to eat crow on my path to success. This post reflects what I have unfortunately learned the hard way. In some of our applications we have used messaging in a Send / Reply pattern using messages like FindAddressesForCustomerMessage and FindAddressesForCustomerResponseMessage. In the beginning this worked just fine. However, after we began to grow and scale became more and more of a concern this began to cause us a lot of pain. I’ll state below the pain points of using a service bus / messaging in this manner.

To understand the needs of our system we have  ~10 application / web servers and ~1000 users of our CRM system. Collectively they all participate in service bus communication.

Messaging with any of the .NET service bus projects has many things to prevent failure of any kind, can withstand restarts, failover, load balancing, transactional message processing, etc. This is great if you take a step back for a moment to consider where this is important. Capturing the commands and events of users into a persisted store somewhere. When a user does something we’re pretty dang sure it will be successful. No longer needing to display the message to our user, sorry this failed please try again later.

Using messaging for queries forces an asynchronous pattern when it doesn’t necessarily need to be, making the system much more complex to work with. This doesn’t mean you shouldn’t use messaging at all. I could see using a message when the application starts up that has a response used to populate a list of users / password hashes in a local cache for example. This doesn’t mean that the query side is asynchronous though it just means that it looks to a cache for the information and has a dependency that a response is returned before someone can successfully logon.

The continual retry on failure for messaging implementations causes problems when user endpoints aren’t online. For example, we had a shift that would leave and the system would slow down. Why? The outgoing messages would retry over time up to as many as 2 days causing bottlenecks for legitimate messages that needed to get through.

Messaging doesn’t handle large messages very well. Let’s face it sometimes messages for queries aren’t simple things that are composed of 3 or 4 fields. This forces us to work around the size limitations put in place for all messaging solutions.

Can these things be overcome? Sure, I could invest my time heavily in addressing my concerns as they relates to the service bus code-base. However, my needs might not be compatible with the needs over others and therefore having to maintain my own version. It’s okay to admit when we were wrong in our assumptions. You can disagree with me and tell me about all of the various changes or tweaks that could be made. However, all I will be thinking is that you haven’t actually tried this for yourself. Next time it won’t be me eating crow.

So what is the right solution? My recommendation is to spend some time reading and learning about CQRS (Command Query Responsibility Segregation) and pay attention to the distinct separation of queries and commands.

MongoDB for .NET with NoRM DbReference clarification

In my last post. I mentioned DbReference and how I wasn’t fond that there didn’t appear to be any way to specify that a property was in fact a reference to another document inside an implementation of MongoConfigurationMap. To be fair, this is likely not a common need when dealing with document databases and likely not needed at all for what I’m considering using a document database for, the query piece of CQRS where I will persist a view model directly as it is displayed on the screen so that no translation or multiple reads are needed. Below is a sample of the current API in NoRM for DbReference. Then to retrieve an order and it’s products.

So if you use nHibernate you’re likely seeing the same things that bother me with this API. The class Order is not POCO, and you must explicitly pass a Func<IMongo> in order to load the related products. Ideally I think the API below makes more sense and keeps things cleaner.

Then the query to eager load the related documents.

To be honest I’m not sure how difficult my proposed change would be, but I may take some time to see what it would take regardless of my current needs. I want to understand more of the internals and what a better way to understand than to contribute.

MongoDB for .NET in 10 minutes with NoRM

Technorati Tags: MongoDB,NoRM

There is a lot of talk lately in the .NET community of using a document database instead of the traditional relational database. Although I’ve kept up on the reading I haven’t invested any time trying it out for myself. Today I decided to give it a shot and considering the amount of time it took me to get things up and running I’m impressed.

First off go install MongoDB. Download Page

The install couldn’t be much simpler for me. Extract the files and run the server.

Then go download the source for NoRM and compile.

Start up a new project and begin define your class(es).

 

Then define your mapping. This step is optional it appears, however, I’m a fan of the NHibernate style of keeping persistence depedencies like identifier types, etc out of your model so I’m starting off this way.

There currently doesn’t appear to be a lot of options for mappings, mostly because there isn’t a need for lots of options. However, I would like to specify DbReference<T> in this class as well. DbReference<T> is a way to specify that the relationship of <T> is another document. Unfortunately it doesn’t appear to be supported (unless I missed something). If I get further along using NoRM this is likely something I would want to add.

Next is a sample usage of the simple model defined.

Obviously this sample isn’t ground breaking. However, I was able to download and install, compile NoRM, and put together a sample in under 10 minutes. I would say try doing that with SQL Server. The install alone takes longer than 10 minutes.

Open questions that I have that I intend to cover in the coming weeks.

• What is NoRM doing under the covers?
  
• How easy will it be to extend the behavior I described earlier when relating to other documents?
  
• How does searching against a document database change things. Can I do things like p.FirstName.StartsWith(“f”)?
  
• How well will a document database fit in with event sourcing. If it does, is it enough to have “persisted index” approach to get the current document(s) or is better to still keep that code close to the command persisting the event?
  
• Do you readers have others?

Why eating crow?

During my experience in software thus far, I have discovered that software development is full of opinionated people not excluding myself. Many times I find myself “eating crow” because of something I’ve said or misunderstood along the way. However, any success I’ve had is because I’m perfectly willing to put myself out there, wrong or right. I see others always trying to show perfection, not willing to admit any mistakes and missing out on the experience in the process. There is nothing wrong with stumbling down the path to success.