For Gibraltar 3.0 we’ve introduced a new way of seeing your log data in real time: Live Sessions.  When you enable live sessions you can view log data remotely, efficiently, in real time.  In Analyst you’ll see a new node in the tree view called Live sessions.  When you select that, you’ll see a list of all of the available sessions which you can pick from.

When you first click the Live View tab a data connection is established and you’ll get the last thousand messages that were logged.  Once that data’s come across, new messages are shown as they are recorded.  You can filter the display to show just messages of a particular severity or to search their text for a particular value.

This is your server, live!

If you switch away from one session to another the live view stream will continue to run in the background – keeping it current with the most recent thousand messages.  Gibraltar takes steps to keep this memory efficient so you can have multiple live streams running at the same time without performance issues.

If you want to keep watching an application while you work you can double click it and get an independent window which will update in real time as long as the remote application is running.

Efficient over the Network

The data stream that’s used by Live Sessions is designed to be very efficient over networks.  It is highly compressed (using the same techniques employed by the Agent to write log files), connection based, and only running when there is an active viewer monitoring the session.  In real world scenarios the live stream for a busy server can be easily viewed over a low bandwidth, intermittent connection like a tethered phone or Mifi.

• If the connection is lost, the data stream restarts with the last message you received so it can show you all of the messages
• If no one is watching a live session then the data is buffered in the Agent but not sent across the network.

Safety First

There are a number of features designed to make sure live sessions don’t cause a problem to your application:

• Asynchronous, buffered transmission:  The Agent publishes data to any set of listeners asynchronously from recording the log file to ensure slow network connections or network connectivity problems won’t slow down recording the main log file.
• Drop on overflow:  If there are more log messages waiting to be sent than can be sent over the network new messages will be dropped instead of holding up the application.  This is in contrast to the log file which will switch to synchronous while it catches up.
• Encryption optional:  You can choose to enable SSL for live session communication so it will be encrypted end-to-end.

Live Sessions and Firewalls

The Live Session protocol is designed to work with most firewall configurations:

• All connections are outbound: The Agent and Analyst establish connections to the central server to communicate.  No connections are directly created between the Agent and Analyst.  This works well with NAT and often requires no explicit firewall rules or exceptions.
• Separate ports are used for Agent and Analyst:  If you need to create a firewall rule you can create separate rules for Agent and Analyst traffic (ports 29970 and 29971 respectively)
• Connections are asynchronous:  Many firewalls will simply fail to respond to attempts to connect to ports that aren’t open, causing very long timeouts.  By performing all connections asynchronously this prevents the application from being unresponsive.

Try it out Today

If you’d like to check out Live Sessions for yourself, you’ll need to download the latest build of Gibraltar 3.0 and upgrade to the included Agent, Analyst and Hub.  The Hub Server does the central relaying between agents and analysts so it has to be set up in the center.  Live Sessions are not available with the subscription Hub Service operated by us at this time, you have to have your own private Gibraltar Hub.

using System.Collections.Generic;
using System.Linq;
using System.Diagnostics;

namespace GibraltarGettingStartedApp
{

	class Program
	{
		static int Main(string[] args)
			{
				Trace.WriteLine("About to Blow up...");
				int zero = 0;
				Trace.Close();
				return 1 / zero;
			}
		}
	}
}

That should get the runtime’s panties in a bunch, let’s try it.

Yep, that did it! Now, let’s see what Gibraltar logged…

Gibraltar knows we crashed and if we open up the record…

We know what the problem was, we can click the link and go right to the line in the source code and Gibraltar tells us what the user experience is going to be, “This fatal error will not be reported to the user, then the application will exit”. All and all, pretty cool.

As an aside, let me close down VS, I want to test if I click the link, will it open the file and go to the line? It does! That’s cool, though I notice that it only opens the actual file and not the solution or project that the file is in. Hmm, I’ll need to investigate that more as it’d be better if the entire solution was opened, don’t you think?

Let’s try running the same unhandled exception in a winforms application and see what happens. We’ll need to run the exe from the commandline, as the VS debugger will “get in the way” and try to do “clever stuff”, but when we do, we see the following dialog:

Here Gibraltar is giving us even more help. It’s popped up a dialog which will actually help us recover from the crash and carry on using our application. If we select “Restart Application” Gibraltar will restart the application for us and enable us to carry on using our application. Of course, this facility should only be used in a testing environment. In the real world, I don’t think it would be wise to restart and to continue using an application which had just suffered from a crash, as there is no way to know what state the underlying model is in. However, from a testing perspective, it’s an excellent tool.

Of course, the same information is recorded in Gibraltar:

Notice that the “Status” has been recorded as “Normal”. If we open the record we can see why:

It’s because we chose to continue the execution and shut down the application normally after.

Okay, let’s change our console application code to handle the exception…

using System;
using System.Collections.Generic;
using System.Linq;
using System.Diagnostics;

namespace GibraltarGettingStartedApp
{
	class Program
	{
		static int Main(string[] args)
		{
			Trace.WriteLine("About to Blow up...");
			int zero = 0;
			Trace.Close();
			try { return 1 / zero; }
			catch (DivideByZeroException e) { }
			return 0;
		}
	}
}

And let’s see what difference that made to Gibraltar…

Hmm, as far as Gibraltar is concerned everything is normal, there is no sign of the handled exception. For a clue as to why that is, let’s examine logging best practices, you can read them at…

http://www.gibraltarsoftware.com/Support/Documentation/Logging_BestPractices.html

And they quite clearly state:

“We strongly recommend that in all cases where an exception is consumed within a catch block that you log the exception with an Informational severity rather than silently swallowing the error.”

Okay, so this states that if we handle an exception, and we want that fact recorded by Gibraltar, then we have to log it. That makes sense really. So, let’s change our code to the following:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Diagnostics;

namespace GibraltarGettingStartedApp
{
	class Program
	{
		static int Main(string[] args)
		{
			Trace.WriteLine("About to Blow up...");
			int zero = 0;
			try { return 1 / zero; }
			catch (DivideByZeroException e)
			{
				Trace.WriteLine("Error: " + e.Message);
			}
			Trace.Close();
			return 0;
		}
	}
}

Now let’s see how that’s affected Gibraltar:

This time you can plainly see that the error has been recorded, which is good, but notice too that there’s no exception tab:

Of course, we should be able to do better and we can. If we use the TraceInformation method, and pass a formatted string, along with the exception, then Gibraltar will give us a much richer experience.

So, let’s change our code to use that method:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Diagnostics;

namespace GibraltarGettingStartedApp
{
	class Program
	{
		static int Main(string[] args)
		{
			Trace.WriteLine("About to Blow up...");
			int zero = 0;
			try { return 1 / zero; }
			catch (DivideByZeroException e)
			{
				Trace.TraceInformation("Error: {0}",e.Message,e);
			}
			Trace.Close();
			return 0;
		}
	}
}

And let’s look at the information provided in Gibraltar:

This time, as you can see, we get the “Exceptions” tab. This is because Gibraltar scans the insertion variables and, if it finds one that is an exception, then it uses that as the exception for the log message (regardless of its severity). Pretty cool eh?

In this post we added an unhandled exception to your console application to see what effect that had on Gibraltar. We then added an unhandled exception to a winforms application so we could see the enhanced features available to us there and then we finished up by handling that exception and seeing why best practice is to log handled exceptions.

I hope you’ll join me next time as we continue our exploration of Gibraltar, until then, happy coding!

Download VistaDB 4.3

We are happy to announce the biggest update to the VistaDB engine since Gibraltar Software took over the product last year. It’s actually our fifth update, though our earlier releases were more limited in scope consisting of a new streamlined licensing system and a number of bug fixes—often providing closer compatibility with SQL Server scripts.

The main focus of VistaDB 4.3 is query performance, particularly with multiple JOINs. We’re pleased to report 2.5x improvements in many cases and discuss below what’s happening inside the VistaDB engine to achieve these results. Let’s start with an overview of how VistaDB produces query results.

How Does My Query Work?

VistaDB queries are executed in three general phases:

• Parse: The SQL query text is parsed into a tree of objects representing each language element.
• Prepare: The statement/expression tree is recursively processed to identify table and column references and to determine the data types of results.
• Execute: The statement/expression tree is recursively processed to execute statements and provide results in order (returning to the calling application as each row is ready).

If there were no optimizations at all, the engine would walk every row of the parent table. And for each of those rows, then walk every row of the first joined table looking for matches to the ON clause. For each of those rows it would then repeat the process for every row of every additional table referenced in the query. Obviously, this would be ridiculously slow when joining multiple tables of any real size, or even when querying a single table with a large number of rows when you don’t actually want most of them.

To be more efficient, VistaDB performs an additional optimization step at the start of the Execute phase. This step recursively walks the WHERE clause and ON clauses and converts the comparison expressions (and special functions such as BETWEEN) into a more efficient representation as constrains in which each constraint specifies a range of values for a particular column based on constants, parameter values, or the value of a column from an earlier table in the parse tree.

In optimizing the parse tree, VistaDB simplifies the execution plan into a series of constrained tables you could imagine as being evaluated left-to-right. Constraints that can’t be resolved are declared non-optimizable and must be handled by testing the WHERE clause. Similarly, target columns for which there isn’t an available index are also non-optimizable and must be tested row-by-row. These optimized conditions are then processed for logical ANDs and ORs to calculate an overall optimized filter for each table.

Building On What Already Works Well

VistaDB has always done well with queries in which a range of rows can be retrieved on a single-column index with an identifiable starting and ending value based on the current rows of tables “to the left” of it. For example, if the parent (FROM) table is restricted by a single comparison in the WHERE clause such as: WHERE ParentTable.ColA = @ChosenValue; (with an index on ColA), then the engine doesn’t need to walk every row of ParentTable, it can start with the first row in the index with a value of @ChosenValue for ColA, and walk each row in the index until it passes the last row with a value of @ChosenValue for ColA. If another table is then joined in it doesn’t need to consider any combinations outside of that range on ParentTable; they’re already certain to be excluded by that condition in the WHERE clause.

Improvements in VistaDB 4.3

We use VistaDB extensively in our Gibraltar application monitoring system and noticed that VistaDB performance left something to be desired for some of our more complex queries. For example, it is common to use a placeholder ID (perhaps a UNIQUEIDENTIFIER) as a foreign key into a small lookup table which can contain additional information fields universal to that value. In Gibraltar we have tables such as Application_Type and Boot_Mode which provide caption and description labels for display purposes. They can be joined directly into a query about one or more sessions, like so:

SELECT * FROM Session_Details SD
	JOIN Processor_Architecture OSA
		ON OSA.PK_Architecture_Id = SD.FK_OS_Architecture_Id
	JOIN Boot_Mode BM ON BM.PK_Boot_Mode_Id = SD.FK_OS_Boot_Mode_Id
	JOIN Processor_Architecture RA
		ON RA.PK_Architecture_Id = SD.FK_Runtime_Architecture_Id

The joined tables are tiny, only 5 or so rows each, so (in theory) this should be very efficient. Each joined table can have its unique matching row directly looked up based on the corresponding column value in the parent table. Perfect, right? But this query was taking several seconds. The base query (SELECT * FROM Session_Details) took less than half a second, and that’s querying the entire table! What we found in VistaDB 4.2 was that as each JOIN was added to this query, the overall query time nearly doubled! Something was clearly less efficient than it should be.

As we analyzed the engine internals, we found a lot of opportunities to improve performance which we’ll be implementing over the coming year. As a first step, we decided to focus in VistaDB 4.3 on reducing the overhead for multiple joins and optimize for the most common cases.

We expect that the majority of joins will be on a single equality between a single column from each table with a foreign key relationship between them. Since this should by definition identify a single value (and often a single row), it should be the most efficient filter to narrow down the joined table based on those “to the left”. So the optimization logic will now catch these top-priority conditions early on and bypass the rest of the expensive reduction pass.

And in queries like our example, we integrated column value caching to eliminate the need to search for the same rows over and over again. When a table is joined on a UNIQUE single column index, the table can cache the row in a Dictionary keyed by the column value from the other table, and each time it comes back to that value, it grabs the row from cache instead of searching the index and reading it from disk again.

We also coded our caching to ensure that it doesn’t consume excessive memory when processing large tables. The cache only holds hard references to the most recently accessed rows. By using weak references to less recently used rows they stick around when memory is plentiful but can be garbage-collected if necessary. For more info on on weak references, check out Kendall’s Code Project article and sample code on creating a single instance string store.

As shown in the graph above, VistaDB 4.3 is several times faster for many common queries. More importantly, in queries such as above, performance degrades linearly as more tables are joined, rather than exponentially as before.

Stay Tuned for More to Come

The query optimizations we’ve introduced in VistaDB 4.3 are just a start. Subsequent releases will have additional query optimizations as well as other performance improvements such as support for bulk insert and enhanced multi-user scalability. We also will be adding new features such as enhanced support for Entity Framework, enhanced compatibility with SQL Server and improvements to our development tools (Data Builder, Data Migration Wizard, etc).

We’ll be writing additional blog posts about our adventures taking VistaDB to the next level, so check back often or leave a question/comment below or in our support forums –we’d love to hear from you!

Once you start using Gibraltar with server applications – like web sites or even windows services then a few things happen:

• Long Life:  Gibraltar equates a session lifetime to the application domain it runs in.  For windows services this could be months, possibly even years.  For web sites it’s typically a day or less.
• Session Size: It’s hard to imagine in a single user smart client recording more than 50MB of information.  Our analysis of log files we receive and our customers use indicates this is a pretty fair assumption.  For a busy corporate line of business web application you can easily generate 500MB of log data each day.
• Submitting Sessions: By default, Gibraltar waits for a session to end before sending in the data.  This works well with end-user applications but doesn’t match up with operating your own web sites or services.  What use is it to find out a month later your Windows Service was having a problem?  You could trigger sending data manually and even when an error happened, but if your session got to large this could be dangerous because we merged together all of the data first to send it.

The biggest problems people have with using Gibraltar 2.x in these situations is:

• If you log a lot of data, you need a lot of memory and processor to submit the data to be able to view it – about five times as much as the log data.  To add insult to injury, every time it submitted the data it pushed it all to the server, and that got completely re-analyzed for issues then copied down to each Analyst.  Very inefficient.
• You had to add code to your application to explicitly push the session to the server to see any of the data.  And when you did you could run into the first problem.

No question about it, this sucks and it had to stop.

Editor’s Note: Not all of the optimizations discussed below have shipped in Gibraltar 3.0 Beta 2.

Smaller Data Files

Our original compression approach was based on a sampling of files between 5 and 50MB.  It turns out that the files we really should worry about were in the 50 to 500MB range.  Saving 25% on a 5MB file isn’t likely to impress anyone.  Cutting 100MB off a log file makes a real difference.  We’ve substantially revised how we compress files and will get an average of 65% reduction in size for typical files.  That’s right – they’re now 1/3rd as big.  This is all done with no impact on the throughput of the log system because of the asynchronous, queued approach Gibraltar uses to record data.

We also reduced the memory it takes in the Agent to perform this compression.  Previously it was possible to use up to your maximum log file size in RAM (in extreme cases) while logging.  To avoid the problem of merging fragments (which uses even more memory) some customers ran into this when they dramatically upped their log file sizes. Now the compression uses a fixed buffer size measured in KB to ensure it’s both efficient and predictable.

These smaller files have another advantage:  We previously zipped files before they were sent to the hub to save network time.  This would get us about a 40% reduction in file size but took time, processor, and disk space.  With the new scheme there’s no advantage to doing this – the files would actually get slightly larger.  So we can skip this step which saves time on both the client and server.

We’ve also added a feature to let you understand why your data files are as big as they are.  In Analyst, you can open a session and then click File Stats on the Session Details tab.

Here’s the analysis we get for one of our Gibraltar Hubs we’re using for testing.  It logged about 700,000 messages in a 24 hour period.

You can see that the average size of each log message is 57 bytes.  Now that’s good stuff from a compression standpoint.  The process also recorded a fair number of metrics and performance counters.  There are metrics for each data operation, web service call, etc. and these add up to a total of about 350,000 individual samples at an average size of 31 bytes.  All of the performance counter samples – 35,000 of them – came in at an average of 10 bytes.  All in, the file was just over 50MB.

From this information you can get a good feel for what’s driving your log file size.  Now, you may never care – but when you get into logging millions of messages and millions of metrics then it’s good to have around.

Incremental Data Files

For Gibraltar 3.0 we’ve changed it so that the individual file fragments Gibraltar stores data into (each representing a period of time) are never merged together – all the way down to Analyst.  When data needs to be made available Gibraltar has always rolled over the current log file (called a session fragment) and started a new one.  Previously these fragments all had to be merged together before the data could be sent anywhere off the computer.  Now they’re each managed individually.  Thanks to this change, each time your application wants to send data to the Hub only the fragments that haven’t already been sent are processed.  Likewise when this data needs to be analyzed for issues or sent down to clients that want to view the details it’s just the new information.

We’ve also gone through the processes that send data – whether it’s to the Hub server, email, or files – and worked to eliminate unnecessary file copies that were being done previously in an overabundance of copies (these get expensive if you have a 300MB file) and minimize the memory required.  The bottom line is that you can safely send session data at any time – regardless of how much data you’ve logged or how long your application’s been running.

Let Your Application Run Forever

Making the session files smaller and keeping them in fragments helps with large sessions, but when you have a really long application there are a few other things that kick into play.  First, you want to make sure you can discard older log data off the computer (to keep down local disk space) even while the application was still running.  In many cases, Gibraltar 2.x would end up throwing away the start of the session and all you’d have is the last 150MB of data (if you used the default limits).   For 3.0, session fragments can be sent automatically as soon as the log file rolls over for any reason.  The default limits of 50MB and one day mean that for each Windows Service you’d at least get a fragment every day – which would be sent up to the server and then pruned locally.  You can still trigger sending data more often from within the application like you used to and it’ll work great.

The Hub server works better with this as well – it will analyze each fragment without having to load up information it’s previously worked through.  This means you get steady updates and the memory required on the server is dramatically reduced.  We’ve had customers with Gibraltar 2.x that had servers perpetually behind analyzing sessions because by the time they analyzed a large file they received another copy of it with just a modest amount of new information.  Now this is a non-issue because the server can load just the new data and check it.

Of Course, You’d Like to see All of That

The final problem area for Gibraltar with very large sessions was viewing them in Analyst.  Historically you could load up a session with between 1.2 and 2 million messages.  More than that and it’d fail with an out of memory exception.  This was because Gibraltar Analyst was a 32-bit only application.  We’ve rewritten our graphing system to use the DevExpress XtraCharts like we use for Charting and that means we’re good to go for 64-bit in Analyst.  If you have 8GB of RAM, you can use all of it for session data if you want.  In practice this means you’re still limited to a session with perhaps 10-15 million messages.  We’re going to work on that in a future release to really achieve our goal of infinite logging.

This is all fine and good, but it’s still missing something… The ability to handle the server-on-fire-right-now scenario we’ve all experienced at some time in our career.  For the story on that, stay tuned for our next deep dive!