Author Archive
First, Do No Harm – Designing Robust Infrastructure
Posted by: | Comments
Hippocrates’ Primum non nocere, “First do no harm”
Several customers have requested a notification mechanism to be alerted when errors are detected in their programs. Simply raising an event is straightforward, but our promise to our customers is that we’ll do the hard thinking that ensures Gibraltar is safe and robust in production systems. Our mantra is: first, do no harm.
In this case, we asked ourselves questions like:
- What if a customer’s error notification logic is slow? How do we ensure that it doesn’t slow down the application as a whole?
- What if the program starts screaming thousands of errors? How do we ensure that we don’t swamp the error notification handler?
- What if there are errors in the customer’s error notification handler? What if it throws an exception? What if it hangs?
This resulted in a design that ensures that the logging infrastructure (including Gibraltar itself AND customer logic that interfaces with it) will be robust and safe.
Our central Log object in Gibraltar Agent now has a MessageAlert event that is raised when warning, error, or critical messages are recorded. This event has a number of safety features such as:
- Asynchronous: The event is raised on a background thread that is not part of the logging path, ensuring that time spent handling the event will not slow down logging or affect other threads.
- Batching: When a burst of messages are recorded that qualify they will typically be raised together to allow more efficient processing
- Throttling: A minimum delay between events can be easily specified to ensure the event isn’t raised too frequently, particularly in error cascade scenarios. Messages are batched up until the next time the event can be raised.
- Hang Protection: If the event handler never returns the Agent will continue to process messages and not queue them, allowing them to be released from memory.
- Loop Protection: Messages that are recorded by your event handler will not cause additional events to be raised. This prevents notification loops where an event handler records an error during notification which subsequently causes the message alert notification to be raised again.
- Low Overhead: We don’t spin up anything (the threading, queue, etc.) until someone subscribes to the event so if you don’t use this feature it doesn’t take up resources either.
The MessageAlert event is particularly useful for automatically triggering immediate data transmission in the case of an error and implementing your own error notification mechanism. The full detail of each log message is available in the event.
Check out our recent post on charting enhancements for more examples of how we are incorporating customer feedback to ensure that Gibraltar provides a robust logging infrastructure allowing you to build rock solid .NET software.
Announcing Gibraltar Hub for Easy CEIP
Posted by: | CommentsGibraltar Hub is our new server-based product that works with Gibraltar Agent and Gibraltar Analyst to deliver an end-to-end solution for creating a customer experience improvement programs (CEIP) as well as remote debugging for customer support. We have been quietly developing and testing Hub for months and are thrilled with how well it’s working. We’ll be releasing it commercially later this Fall and are now inviting existing Gibraltar customers to participate in our beta testing program.
We’ve created a short video tour (3 min, below) to give you a sense of Gibraltar Hub as well as a podcast (8 min) of a conversation between Kendall and me talking about Gibraltar Hub and the problems it solves. You can read an abridged version of the interview below and we’ll be posting more technical details later this week.
If you like what you see and want to participate in our beta program, please shoot me an email.
More about Gibraltar Hub
What problems does Gibraltar Hub solve?
Gibraltar Hub is designed to address a couple key scenarios:
- Collecting data from many application instances even past firewalls such as commercial software products.
- Customer Experience Improvement Programs (CEIP) for proactively gathering feedback on application performance in the field through continuous data collection and analysis.
How does Gibraltar Hub complement Gibraltar Agent and Gibraltar Analyst?
Gibraltar Hub sits between Gibraltar Agent and Gibraltar Analyst making it easier to get data from users to the development team. It’s a web service providing two interfaces: one for Agents to submit logs, the other allowing logs to stream down into Analysts. With Gibraltar Hub you can collect, manage and analyze thousands of logs and provide every member of your development team with a consistent, near real-time view of all that data as well as simple, powerful tools to analyze the data and gain new insights into the areas of your applications most needing improvement.
Is Hub required to use Gibraltar?
No, Hub is totally optional. The existing email and file transfer mechanisms in Agent will continue to be supported. However, we believe Hub will provides the best user experience because both Analyst and Agent have been enhanced to support secure, reliable, background data transfers with Hub. This means the applications can be configured to silently stream logs in the background and the development team sees new data automagically appear like new mail popping into your inbox.
How does Hub help development teams?
Gibraltar Analyst has always made it easy to import and export packages containing logs. But some of our customers with large user communities or development teams with multiple members found it challenging to ensure that everyone had a consistent view of all the relevant data. With Gibraltar Hub each team member can subscribe to a shared feed and have all the data available and continuously updated.
What is a Customer Experience Improvement Program (CEIP) and how does Gibraltar Hub help?
Microsoft coined the term Customer Experience Improvement Program (CEIP) and describe it like this:
CEIP collects information about how our customers use Microsoft programs and about some of the problems they encounter. Microsoft uses this information to improve the products and features customers use most often and to help solve problems. Participation in the program is voluntary, and the end results are software improvements to better meet the needs of our customers.
The three components of Gibraltar correspond directly with the three key challenges for development teams wishing to create their own CEIP:
- Agent efficiently collects data about how customers use programs and records details on problems they encounter.
- Hub provides reliable, secure transmission of log data from end-users to each member of the development team. Data is highly compressed and the transfer protocol is firewall-friendly and reliable even when network connectivity is limited and intermittent.
- Analyst indexes all that data and provides powerful visualization tools that help team members identify broad patterns spanning many logs as well as the ability to drill into each log to point the root cause of a single issue.
Does automatic transmission of logging data raise any privacy concerns?
Yes, dealing responsibly with information privacy is extremely important. At the same time, the considerations vary widely between different applications so we think that it’s important for Gibraltar to provide the flexibility to fit within a broad range of usage scenarios. For example, having a dialog for CEIP opt-in is a recommended best practice for a commercial software product, but for an in-house corporate application that is only used by employees, their employment agreement or computer login screen may already require informed consent to certain information being monitored. With this in mind, the default configuration settings for Gibraltar only transmit data on demand with explicit user consent. And we also make it easy to enable automatic background log transmission when appropriate.
UPDATE: Kendall has written a nice follow-up post on what Gibraltar Hub is, how it works and why we created it.
Want to get started?
Upon release you can either purchase your own private Hub to run on your own server or subscribe to the Gibraltar Hub Service we host. If you’re a registered user of Gibraltar you can try out the Gibraltar Hub Service right now. Just contact us to get your preview account set up. Depending on the final user feedback from this preview we expect the general release to be in the next 4 to 8 weeks.
Learning Camtasia Screencasting
Posted by: | CommentsI love being part of a startup, except when I don’t. One thing I love – and occasionally hate – is wearing 47 hats. Unlike big corporate life where there are specialists for everything, in a startup, it’s just you and your team. There’s an amazing range of tasks that you simply must do for yourself. You live by your wits and versatility.
In a heady rush of uninformed confidence, I set off last week to take my nascent screencasting skills to a new level.
I just created my second screencast and I learned a ton in the process though it felt, at times, like having my head pounded with bricks.
My adventure started with me searching around to learn that Camtasia is the tool of choice for creating screencasts. Plus, I found some great tips for using it to create YouTube HD screencasts. I was very relieved, for instance, not to have to figure out for myself that H.264 is a really good codec but that it’s only compatible with MOV and not MP4. I think, hey this won’t be too hard. And with my wife and two boys away at the beach, I decided to make the most of my bachelor weekend by ratcheting up my ambitions and taking this opportunity to also play with the awesome Adobe CS4 Production Premium suite.
Before delving into the details of my many trials and tribulations, let me tell you how everything turned out.
The Oracle Speaks
After three days of nearly round-the-clock work – and a number of kitten-startling screams at my computer using language I was happy my two boys weren’t around to hear – I posted it to YouTube and requested feedback on the Business of Software (BoS) discussion group. A few hours later, I was shocked to discover that Ian Ozsvald has posted a comment linking to a critique of my screencast. I recognized his name immediately as the guru behind thescreencastinghandbook.com which I subscribed to last week. Uh, oh!
Fortunately, Ian generally liked my newbie effort – even applauding some controversial choices I wavered over like including my never-been-mistaken-for-Brad-Pitt face on the opening and close.
However, Ian had a few suggestions for me in a section of his critique entitled “The Bad”. Well, he is absolutely right on every point. But there is a funny behind-the-scenes story as to why…
- when webcam is displayed, the video and audio aren’t quite in sync
- there are some unwanted volume fluctuations in the audio track
- there are a couple quick cuts that make the conclusion look and sound chopped up.
Growing Pains
I learned a lot on this project, but man, what a PITA! For example, I learned that Adobe SoundBooth is great for applying EQ and equalizing volume, but the 48K mono wav files I produced aren’t supported by Camtasia. Although they play fine in Windows, when I imported them into Camtasia the waveform showed a flatline and, sure enough, nothing but silence when I tried to play them.
So, back to SoundBooth to convert them to 44.1K – only to learn that it doesn’t have the batch processing capabilities of Adobe’s earlier Audition product – hence conversion requires a tedious series of 33 one-at-a time Save As operations.
Ian accurately noted that my video and audio weren’t quite aligned. I only wanted PIP video for the beginning and end of the screencast and just audio in the middle. I also wanted to use Soundbooth to EQ and equalize all the audio (I’m also a musician and my business partner use to be an AV engineer so we’re picky about sound).
I tried recording audio in Soundbooth and it sounded great so I decided to use it for ALL my audio recording, thus restricting the use of my cheesy Logitech WebCam to only recording video. I used Premiere Pro to align the two together. This turned out to be an extremely painful, time-consuming process. And, despite hours of painstaking tweaking, I never could get everything quite in sync. Never again.
Talking Heads Stop Making Sense
Recording the video was also painful. I couldn’t memorize my entire intro/wrap-up scripts. So instead, I shot them a line at a time and used Premiere Pro to splice the clips together because I wanted to use Camtasia’s sweet fade in/out on the PIP. Unfortunately, when I concatenated multiple clips in Camtasia, to form one larger sequence, the clips kept fading in/out at each of the seams.
So, I finally get my audio and video clips together and pull everything in Camtasia. I put the PIP clips on the PIP tracks and the audio-only clips on the Audio 1 track. But, now, for reasons I don’t grasp, the audio levels for the PIP video clips are much softer than the audio-only clips. WTF! I theorize I lost some gain in the conversion from WAV audio to WMV video. This might be because Premiere Pro can import 44.1K mono WAV files, but it won’t export audio in that format when creating WMV video files. The closest I could come was 48K stereo audio (funny that Camtasia doesn’t mind 48K stereo WMV but won’t play 48K mono WAV).
Next, I tried to use Camtasia to equalize the volume to no avail – it won’t equalize across the PIP Audio and Audio 2 tracks. I end up pulling the audio-only clips onto the PIP Audio track. Camtasia only allows this after I agree to permanently unlink PIP Audio from PIP Video — thus creating new opportunities for video and audio to get out of sync.
Eventually, everything seems to be more-or-less fixed. Not perfect, but pretty good. After granting my computer an hour of quiet-time rendering and encoding my four minute video, I’m finally ready to upload to YouTube. As a final check, I watch the video one more time. It’s 3 AM and, OMG, everything starts perfectly, but about a minute in the audio totally drops out. WHAT!? I go back to Camtasia and, sure enough, see the same thing – but only when I play the video from the beginning . If I click in the timeline and start at other points in the video, the audio is there. An hour later, after poking various rocks with an assortment of sticks, I notice some weird little soundwave artifacts on a couple audio clips that must have gotten introduced somewhere in the cascade of tools. I trim them off and finally get the audio to play from start to finish.
One Last Edit
I watch it again and feel like it’s finally good-to-go except – ugh! — when I see the whole thing in context, I feel like the conclusion looses steam right at the end because of stutter in my delivery. At this point, reshooting that clip is absolutely out of the question. I have exactly zero confidence I’ll be able to match the audio levels, the light in the video will be wrong (it’s the wee hours of the night now vs mid afternoon when I originally recorded. And, most significantly, at this point there is no f@%*ing way I’m going to risk hosing everything up messing with those audio video tracks again. Still, this is the punchline of the whole video and it’s falling flat right at the finish line.…
So, yeah, I go back to Premiere Pro and splice away telling myself that it’s the lesser evil and maybe even is kinda hip — so many youtube videos use fast-cut video splicing… yeah. This isn’t a defect, it’s cool. And besides, with a little luck I can get to bed before my neighbor’s rooster starts mocking me.
So, I make the changes, set Camtasia to rendering/encoding again and catch some shut eye. In the light of day, after confirming that my audio still plays end-to-end, I decide not to tempt fate any further with additional tweaks and upload to YouTube, warts and all.
It’s a Wrap
Well, that’s my tale of woe. This is only my second screencast and I (obviously) still have a lot to learn from Ian and his Screencast handbook — especially about using tools optimally in a workflow that is efficient, yet resilient enough to accomodate change during the creative process.
I also think there are some interesting parallels between my experiences in miniature creating this screencast and the challenges I’ve faced on teams building large software projects (platform selection, change management, agile vs waterfall, etc). But, that’s a topic for another day.
