Getting to Know Gibraltar – Making a Start
By · CommentsHello there. My name is Gary Short and I’ve recently joined Gibraltar Software. Amongst the other things I’ll be doing (more details on those later) I’ll be evangelising the Gibraltar product.
Now I’ve never used the product and I don’t know anything about it, so I guess I’m on the same sort of learning curve as a new customer when they first download a trial version.
We all learn in different ways and at different speeds, but the way I learn best is through action centred learning, in other words, I like to “do stuff” to learn how “stuff works”. So, the best way for me to learn how to do something with Gibraltar, is to do something with Gibraltar. 
So what’s the simplest thing I can do to get something working? Well after having a quick look at the documentation, it seems to me that the simplest thing to do is:
- Configure a minimalist application for tracing
- Hook up Gibraltar
- Examine the data in Gibraltar.
Okay, so let’s do that!
Step 1. Configure a Minimalist Application For Tracing
Let’s work with the simplest thing, namely a console application with one method and a couple of calls to the Tracing API. Something like this should fit the bill:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Diagnostics;
using Gibraltar.Agent;
namespace GibraltarGettingStartedApp
{
class Program
{
static void Main(string[] args)
{
Trace.WriteLine("Entered method!");
Console.WriteLine("In the body of the method!");
Trace.WriteLine("Leaving method!");
}
}
}
And when we run this, it does pretty much as we’d expect it to do:
Step 2. Hook up Gibraltar
Firstly, start Gibraltar and from the and click the “Add Gibraltar Now” button:
Then, simply work your way through the wizard. Starting with selecting the project file for your application:
Next, since we’re doing the simplest thing possible, accept the default configuration:
Then deselect all the checkboxes.With the exception of “Allow Sessions to be Emailed to you”, which I can guess at, I have no idea what these do, but we’ll find out later. For now, we just want to do the simplest thing that’ll work.
And we’re done:
Okay so let’s run our application again.
Hmm, notice that the console window stays open this time, and we have to physically close it to make our application end – what’s happening here I wonder? We’ll come back and investigate that in a moment, but for now, let’s see what information is available in Gibraltar.
Step 3. Examine the Data in Gibraltar
Open Gibraltar and select “New Sessions”
and in the right hand pane, we see that we do indeed have information from our application!
Now it’s time to investigate that “application is still running” oddity. Looking at the information we can see that the “status” is “crashed”, even though we closed it down properly.
My guess as to what is going on here is that the Gibraltar Agent is holding open the application after the execution path has reached the end of the Main() method. We then closed the window manually, and because our application didn’t end in the way Gibraltar expected, it’s marked this event as a “crash”. It’s time to look up the documentation and find out what’s going on.
A quick read of the documentation at:
Shows us that indeed, when we are using a console application, we are responsible for ending the session. We can also see that there are a number of ways to do this, but since we are trying the simplest possible thing that will work, we’ll just add the Trace.Close() call to the end off our application. So now our code looks like this:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Diagnostics;
using Gibraltar.Agent;
namespace GibraltarGettingStartedApp
{
class Program
{
static void Main(string[] args)
{
Trace.WriteLine("Entered method!");
Console.WriteLine("In the body of the method!");
Trace.WriteLine("Leaving method!");
Trace.Close();
}
}
}
If we run our application now, we can see that our application closes when we get to the end of the Main() method. If we look in Gibraltar we can also see that our application has closed normally:
Okay, I think that is enough for this post. To help us learn how Gibraltar works, we’ve build a minimalist application and enabled Gibraltar to record our trace information. In the next post, we’ll build on these beginnings by looking at what happens when our application throws and error. We’ll also find out if Gibraltar deals with handled and unhandled exceptions differently.
Until next time, happy coding!
After a long time baking we’re happy to announce that Gibraltar 3.0 Beta 2 is available for immediate download. While this is still a beta release, we’ve run it through more testing than normal so it’d be ready for you to try out. This is just a quick overview of what’s new in Beta 2; stay tuned for drill in articles on the high points.
What’s it Mean?
Our focus with Gibraltar 3.0 is:
- Scalability: Big sessions, long running sessions, large data sets, lots of sessions… We want to up the limits in every direction.
- Easy Adoption: We want every team member from the project managers down to the junior developers to be able to get into Gibraltar and be comfortable.
- Servers and Data Centers: Where Gibraltar 2.0 was focused on applications shipped out to hundreds or thousands of computers, Gibraltar 3.0 adds features specifically for monitoring just a few servers in your own data center.
Scalability
When we originally designed Gibraltar we had to pick some target limits for the number of log messages in a session and the number of sessions in a repository. Since our initial target was smart clients deployed to hundreds of systems (this was the customer base that we’d been working with for years that motivated creating the product in the first place) we went back and looked at the size and scale of what had been done in the past. We applied some scale up factors and came up with a target of 500,000 log messages per session and 2000 sessions in a repository.
This turned out to be laughably low.
We now know that people will cheerfully log millions of messages (biggest we’ve seen: 55 million messages in one session that was just 12 hours of data) and will gather tens of thousands of sessions per day. We’ve nibbled around the edges of this problem before – we added paging to the log viewer which let us get from around 1 million to about 2 million messages that still could be displayed.
For 3.0 we’ve made fundamental changes to how data is moved and tracked to make sure we’re ready for large, long-running sessions and lots of them. Still, this is more of a journey than a destination – if there’s one thing we’ve learned it’s that people will push scale until the infrastructure pushes back. We’re working hard to move those limits well out so you can get the most from Gibraltar.
Now with a splash screen - doesn't every app need one?
Making it Easy
We’ve done some usability testing with Gibraltar and the results weren’t what we wanted to hear. The feedback we got was that we’d designed a tool for experts. Not expert developers, but experts in logging and performance monitoring. In one usability session we saw a junior team member be visibly startled by the information overload presented by Analyst. The developer closed it and didn’t want to go in again – too much information to find the detail they needed to work on. Now, we haven’t had a chance to incorporate this research into 3.0 Beta 2 but we have work already underway for the 3.0 release that is squarely aimed at making sure everyone on the team can get value out of Gibraltar. Stay tuned and wait for Beta 3 for a preview of these changes.
Servers and Data Centers
To help out with monitoring your own server applications we’ve added new session categorization features for tracking the environment and promotion level of each session so you can view information by where it was gathered (environment) or the stage in your development process (promotion level). You don’t have to use these at all, but when you want them they’re ready. We’ve enabled long running sessions like windows services to work much better with Gibraltar and added a real time log viewing capability specifically for internal server scenarios so you can see the contents of your web server logs as things are happening, all without affecting your application’s performance.
What’s Changed?
Internally we’ve made a few big changes:
- The Agent works just off files and directories without an index: We’ve dropped the index database approach Gibraltar 2.0 used to instead rely on scanning directories for their contents. This is much more resilient in the face of manipulating files and directories and eliminates the problem of upgrading the index database schema.
- The file format is now much smaller: We’ve extensively optimized our serialization format based on sampling hundreds of thousands of real-world data files. In many cases, log files will be around 30% the size they would be under Gibraltar 2.0. The benefits are particularly visible on large log files.
- Sessions are no longer merged in the Agent: Previously if a session was recorded into multiple files, these files were merged together in the agent before the data could be sent anywhere. This could take 5x as much memory as the file fragments in question, a real problem for long running applications. Now these individual fragments are handled individually all the way down to Analyst.
- There is a new live streaming protocol: In addition to writing data to a file it can now be sent across the wire via TCP/IP to be viewed in real time.
- Gibraltar Hub can now use SQL Server or VistaDB for its database: For larger customers, or situations where you want to do your own reporting and analysis, you can elect to have the Hub use SQL Server for tracking session data instead of VistaDB, which is included by default.
- Analyst is now 64-bit ready: We reworked the graphing in Analyst so we could make the whole application run in 64-bit, helping with large data sets.
Try it Out Today
You can download 3.0 Beta 2 right from our site. To install it you’ll need to have a valid Gibraltar license that’s up to date with upgrade assurance. If you’ve let your upgrade assurance lapse and want to give 3.0 a try, contact us to discuss your options.
If you aren’t a Gibraltar customer, you can get a trial key and then use that to try out 3.0 as well.
We’re planning on one more round of beta for 3.0 after this so there is still time for your feedback to make a difference. Don’t be shy, tell us where we’ve missed the mark by emailing support or post your comments on the forums.
GiveCamp UK – Start a CRM in a Weekend
By · Comments
As part of my recent trip to the UK I got a chance to participate in GiveCamp UK. Gibraltar Software was a sponsor, but more than that we like to really participate in the community. I got selected as a team lead which made it doubly interesting – leading a GiveCamp team is always challenging but made that much more so when you account for leading a team of UK developers.
When talking with the charities that were interested in having something done at GiveCamp a common thread showed up – many needed a basic customer relationship management (CRM) capability to replace their existing paper or Excel process. Nearly every charity has the need to track donors, volunteers, and people they help. Additionally, these charities often have little or no in-house IT talent so self-hosting an option of any complexity wasn’t a good option. Instead of helping one or two of them by implementing an existing CRM option (like SugarCRM or a commercial option) we saw an opportunity to use GiveCamp to kick off a greater initiative: A free hosted CRM tuned just to small charities.
The core idea is to provide just what small charities need in the easiest to use fashion through a software-as-a-service platform. This way the charities wouldn’t have to worry about security or backups and could shave off a lot of time that otherwise went into basic recordkeeping. In many ways, this CRM would be distinguished by what it didn’t do more than what it did. The primary goal is simplicity of setup and operation and setup, not flexibility to handle any scenario.
Before GiveCamp UK started, I’d had a range of conversations with the organizers to make sure we had a clear reason for why we weren’t taking something off the shelf. We looked at a few open source offerings to see if we could just make a customization engine on top of them but in the end it was clear that the risk and complexity of those options overweighed the head start they would give us. We didn’t want to risk suitability for small charities in exchange for capabilities they wouldn’t need.
What’s a Minimum Viable CRM?
The initial requirements we got from charities were, not surprisingly, very general. Fortunately, we had a few members of our team that had experience developing CRM solutions and were able to give us a starting point that made a lot of sense. Our goal for the weekend was to get just far enough to handle the most essential requirement: track donors and campaigns.
Besides making the application useful we had to make it easy enough to set up that individual charities could create their own CRM instances. Immediately this pushed us to design it as a Software-as-a-Service (SaaS) to eliminate the need for installation and deployment by the charity. Instead, they would just go to a web site and enter the bare minimum information to get going and receive instructions.
Once we had the functionality outlined we wrote out exactly what we wanted to demo on Sunday. You don’t want to end up with nothing to show at the end of the weekend, so it’s useful for everyone to have clarity on exactly what you’re going to show. Things that the application will ultimately need but aren’t part of that demo are lower priority.
Have a demo plan before you start building the application so everyone knows what you have to get done.
Building the App
Day One (Friday night)
At first it appears that you have 43 hours to get it done – from 5 PM on Friday through to Sunday noon. But, knock out the time to form the teams (about two hours, including the introduction) and then six hours a night for sleep (sleep is a weapon!) plus another 5 hours total for eating and breaks and you’re down to 24 hours. Given that, we broke it down into requirements, planning, and framework (Friday night), construction (Saturday) and integration/defect fixing (Sunday morning).
With one day to build out the user interface and process flows it was essential to make sure we had the foundation set up Friday night. We worked largely from back-to-front with the database and data access layer as well as a generalized Excel import/export capability being done by the end of the day Friday while the rest of the developers worked out the exact flow and wireframes for the UI. I made tea and asked for status updates.
Break your project into independent modules that can run in parallel. The Team Leader should keep a close eye on whichever module is currently the critical path.
Day Two (Saturday)
Saturday morning began bright and early with most of the team in before breakfast. We had made our goal for Friday with a complete data access layer for the application set up and tested. It’s very tempting to drop parts of your normal development process when working in such a hurry, but basic unit tests put in as you go dramatically reduce the odds you end up with nothing to show on Sunday because things don’t work when you put them together.
The main applications were being built using ASP.NET MVC3 which most of the team was at least somewhat familiar with. Since we expected the system to have significant data storage requirements that would grow over time we opted for a SQL Server back end. For data access, Simple Data was selected over other options in part because Mark Rendle was on the team which made it emphatically the fastest way to get our goals accomplished. Several of the team were impressed with it and I suspect will use it on some of their future projects.
That mug isn't going to fill itself now, is it.
We were fortunate enough to be able to draft in another UI developer that could work strictly at the HTML/JavaScript layer. To accelerate development he recommended and we adopted the Twitter Bootstrap library. This gave us a pre-packaged set of styles and scripts that define the visual language of the application. We now were working down several fronts at the same time – the provisioning application that would be used to set up new users, the main web application, Excel import & export, and the critical search capability for creating campaigns. Meanwhile, I made tea and asked for status updates.
Saturday is the long day where you want to end with each of the components of your system feature complete and working at least in isolation. Once you leave on Saturday you’ll only have a few more hours available on Sunday to get everything wrapped up. As the day progressed things were looking up that we’d be able to deliver on our goal for the weekend.
To help with testing and prove that our application could scale we wanted to load it up with test data that would feel like the real thing. There were two approaches that occurred to us – write a programmatic code generator to run on top of the data layer, or use a data generation tool (like Red Gate’s SQL data generator). We weren’t sure if the off-the-shelf tool would get us good enough results in time, particularly since no one on the team had used it. Alternately, it wasn’t clear how much time it’d take to generate plausible data on our own. So we did both for a while – running down both paths for a few hours to see which would bear out. In the end we found the Red Gate tool did great with people and addresses, but our custom coded tool did better with donations and more interesting edges of the data.
When you aren’t sure which way to go – do both. You have more people than time. Pick a winner when it’s indisputable or you can’t spare the effort any more.
Around dinner time we did a walkthrough of the complete demo we’d laid out and noted the missing elements and problems. We’d come a long way, but there were several minor changes that appeared useful for the best experience. With the user interface coming together, I went around to other teams to find a graphic artist that could make us a logo for our project. We’d settled on the name GiveCRM and a logo would be a great touch for the demo. Fortunately another team was already done and was willing to lend us a hand with a family of logo graphics we could incorporate. Meanwhile, I made tea and asked for status updates.
Day Three (Sunday morning)
Sunday morning was a later start for most people since they’d stayed late to accomplish our Saturday goals. Once there was no more tea to make I’d left and gotten a good night’s rest so I was able to whip up a presentation on Sunday morning for our demo later that day and review where we were. We started off the day with another demo walkthrough to see what gaps remained. It was clear one thing we wanted to have done – a proper authentication system – was not going to make it in time. Instead of risking the demo we decided to have that team continue working on a branch while we finished up the rest.
About every hour we continued this pattern – do a demo walk through, note the issues, make progress. Naturally, I made tea and asked for status updates. By the time it was pencils down at high noon we had the whole thing put together, along with some extra bells and whistles that weren’t in the original plan.
Keep tracking your progress against your demo. This keeps you looking at the whole picture instead of just the individual modules and function points
Well, that didn’t work
The monkey wrench that got thrown into our plans was challenges with configuration management. We’d opted to use GitHub as the central repository for the code despite very few on the team being familiar with it. This was my mistake – it was mentioned by one person and I assumed I was the only one that was in the dark on how to use it on a real project.
DVCS tools are very powerful for distributed development but can get you in real trouble. Several times we had to stop work and enlist a Git expert (Ben Hall) to rescue us from merge oblivion. Ultimately we addressed this with external locking: I had a squeeze toy which you had to come get if you were going to change project or solution files (or add a new file to the solution). Combined with a fair bit of manual coordination we mostly avoided merging problems going forward, but it did cost us a lot of time.
Think hard before relying on tools your team doesn’t know. Time is your greatest enemy.
Where’d we end up after one weekend?
When I originally discussed the project with the organizers of Give Camp UK I didn’t give us much odds of having an interesting demo at the end of the weekend. Our first goal was just to make progress that could be built on in the future. After our Friday night planning it seemed quite possible we’d have something ready for charities which was quite exciting indeed.
The main application was in pretty good shape: You could import data, export it, create and manage campaigns. Donors could be individually added, donations recorded, and an easy-to-use campaign generator let you build up a filtered list of donors. Even the authentication system that had been such a concern earlier had picked up its pace and was ready to be merged in. We decided not to do it for the demo in case it introduced a problem, but it was checked in and ready for merge.
The provisioning system was similarly looking good: It gathers the necessary information, creates the right database entries, generated a nice introduction email (in HTML and text formats no less) and could import initial donor data from Excel. Like the main application it too needed to be merged with the authentication system which we left out for the demo.
What’s Next?
There were a few things missing before we could release it to charities for use. Most notably, there are some back end features missing in the provisioning system for creating new charities and you can only have one account for each charity. For the most part these issues will be straightforward to address and the team is set up to get these handled.
Before we go live, we need to find the right place to host the system. We have a test server up and running at GiveCRM.org.uk but we need a real home – one that can scale up to handle hundreds if not thousands of charities. Our goal is that the service is free for these small charities. The incremental cost to host each will be quite small, but any charge will shift how charities will approach a decision to adopt it.
Interested in Helping Out?
This is just the beginning of the development of GiveCRM. Our goal is to continue development with a volunteer team to refine the existing features (and address the gaps we need to go live) but then at each of the coming GiveCamp UK events add another module to the system. While we believe we’ve hit the most critical feature charities need, it’s far from a complete solution. If this is interesting to you, be sure to sign up to hear more about the schedule for the next Give Camp UK event.
Conversation with Scott Hanselman at Hanselminutes
By · Comments
I had the good fortune to be invited to chat with Scott Hanselman about what it’s like to create a software company as part of his ongoing series on startups. In a quick 30 minutes we talked through anumber of the angles of what it’s like to go all in on a startup. It was great to hear that even Microsoft, with all of their resources, has many of the same problems we do just at a much larger scale.
I’m going to be doing an extended version of a talk on the same subject at the New York City Code Camp next weekend called Creating Your Own Software Company: A Survival Guide. Come on out and you can hear that session, grill me at length afterwards, and hear four even better speakers all in one day. Code Camps are the best kept secret in software development today; if you aren’t hunting them out in your area you’re missing a great opportunity…
Later in October I’m on a pretty intense two week tour – starting in Philadelphia and ending in Coventry, UK. You can see the full list for an event near you on our events page.
