One key requirement of the Gibraltar Agent is to be able to manage the data files it creates on disk to ensure that they can’t grow out of control.  After looking at a lot of options, we determined that we need a central index to track the locally generated files.  The problem was that it had to be absolutely safe to use from multiple processes without risk.  We had some simple xml-based ideas to solve the problem, but early prototypes were not encouraging.

Fortunately, before we launched into getting more and more aggressive with solving the problem someone on the team stumbled over VistaDB, a fully managed database that we could merge into our agent.  We wrote a quick technical prototype and were impressed:  Not only could we safely throw all the data we needed to track into it even in an extended run of our torture test, but we were able to do the evaluation of whether we needed to prune files or not (and what files to prune) within it which made for a fast, clean, maintainable index.

Even with the success of this prototype, we were very reluctant to go down this road.  Our previous generation solution had use MS Jet for a data store, and it had been a source of problems.  We had ported that to SQL Server, and SQL had become a source of problems for some of our clients.  We’d internalized the lesson that databases and logging do not mix if you want an easy to deploy, foolproof system.  We decided to cast a wider net and look at a range of options:

• SQL Express / SQL Server Embedded: Microsoft’s free offerings.  SQL Express was right out because it was a windows service and would make our deployment complicated and huge.  SQL Server Embedded couldn’t solve our problem because only one process can access the database files at a time.  And oh yeah, it would make our deployment somewhat complicated and large.
• Other third parties: Without getting into an exhaustive list, we decided it really had to be a completely .NET native, managed implementation that we could merge with our assembly because we were only going to ship one assembly for the agent.  Furthermore, it needed to support syntax at least largely similar to SQL Server so we wouldn’ t have to master multiple environments.

A Dashboard of sessions run locally

In the back of our minds was another consideration:  We’ve always intended to grow Gibraltar into offering a larger version for enterprises with centralized log storage and management for many computers.  It was a lead pipe cinch that this solution was going to use SQL Server, so the closer we could be to that on the client the better off we’d be.  The more we worked with VistaDB, the more we started to wonder:  Did we really need SQL Server even for our future larger version?  Could we perhaps just use VistaDB?  We did more prototyping and came to the conclusion that it could work very well technically.

At this point, we knew we had a winner in VistaDB.  A quick prototype showed that we could easily target both it and SQL Server with exactly the same code at each level.  Nothing else could do that:  We could use the same schema, stored procedure code and database access code and switch between SQL Server and VistaDB with none the wiser.

If I see so far, it’s because I stand on the shoulders of giants

The best was yet to come.  With VistaDB it’s easy to create a new instance of a database anytime we needed our index data structure.  Opening and closing databases is relatively fast, so we were able to have a common relational database structure available everywhere.  This meant that we could have just one data model everywhere instead of separate ones for data collection and session data management.  Better yet, we could have one set of database access code for both cases, even if we ultimately supported SQL Server.

This opened a lot of opportunities:

• The same data bound charts and views could be generated against any repository, allowing immediate analysis without having to copy data first into a central store.
• Very large repositories could be supported because the index data didn’t need to be loaded into memory for processing
• We were able to redirect the time saved not developing our own XML persistence format and structure into customer value-add features.
• We could add a reporting system that would require rich, hierarchal data structures.

In short, if it wasn’t for VistaDB most of the features in the repository view and the reporting system of Gibraltar would only be available in a future release that used a central server.  In a small way VistaDB did cause us to spend more development time than we would have – we got so excited about the potential for some of the features that we’d written off as being too expensive to implement that we held the release until we could get them done.

Now, we could have written our own thing for a lot of these pieces, but it’d taken a lot more time (particularly since we target .NET 2.0, not 3.5 so no LINQ for us) and there are some parts that we’d have always been worried about – namely the fundamentally hard problem of having may processes accessing the same file doing reads and writes.  That’s just a hard problem to get right period, and it was great to hand it off to someone who worried as deeply about that as we do about logging and metrics.  In the end, it’s a great example of not reinventing the wheel unless you want to learn a lot about wheels.  We already knew a lot about databases and shared files, enough to know that we didn’t want to learn any more.  We were much more interested in digging into the areas that added unique value to Gibraltar.

Beware the gift Trojan horse

One last concern we had was that adopting anything into our agent would create a long term obligation for us to support it.  Customers would expect that multiple agent versions would safely interoperate on the same computer.  Furthermore, because we would use the same format in our Package files used to send session data between computers we’d be required to support it for years.  We’d have to get comfortable that this was feasible.  To this end, we needed the solution to either be available completely through source code we had or through a company that we were absolutely sure would support it long term.

A great incentive for long term support is revenue.  People in general and for-profit companies in particular are motivated most directly by the idea that people will pay them to do something.   This was another place where we were a bit concerned with using one of the options from Microsoft because they were all free.   Microsoft’s motivation for creating these products is primarily defensive – prevent people from using other free options in the hope that they’d eventually upgrade to one of the nicely expensive server options.  That’s fine and good if you’re targeting one of those options and just want a free scale down option, but this was central to our product.

We could have gone open source, but there are a few issues there for us:

1. No open source option even came close to offering what VistaDB did – namely the ability to support stored procedures compatible with SQL Server.
2. We’d then largely be on the hook for our own source code support if the community wasn’t doing what we needed, and the whole point here was to not have to write something.
3. We’d have to very carefully scrutinize the open source license to make sure we didn’t get GPL’d into oblivion.

Part of our concerns were mitigated by VistaDB offering a source code license, so we could get source code to make our own changes if we had to.  But really, I wasn’t looking to ever need to write this code, that’s why I wanted to get someone else’s solution.  With VistaDB, we found not only a strong community of folks that have built products around it, just like we were, but also a small company owned by a guy that believed fundamentally that folks like us had to succeed with his product for him to stay in business.  His goals tightly aligned with our needs, and that’s a great precondition for success.  That’s good, but there’s another big condition for success:  Was this company focused on listening to its customers?  Would it respond to our concerns?

As is our tradition, we sent an unsolicited block of product feedback to VistaDB.  There were some good things, but we had some concerns as well.  We got a point by point response from Jason Short, the CEO of VistaDB.  Better yet, we got an invitation to a conference call.  Now, at this point VistaDB had gotten a total of $300 from us.  We spent over 7 times that on the fancy graphing component we use, and compared to the criticality of it to our system I’d have spent a lot more to solve the problem we had.   I was impressed by a series of conversations we’ve had with Jason and his team, and how dedicated they are to doing the right thing in solving the challenges they’re up to.

When you think about how much money it costs to create software of any sophistication, it’s great to be able to pull off a piece of your complexity and hand it to someone who will care more about it than you do.  We’re so happy with how much VistaDB added to what we could offer our customers that you’ll notice in our About Gibraltar page that we’re proudly Powered by VistaDB.