Archive for Demoware
Reworking the Gibraltar AddIn API
Posted by: | Comments
There’s a classic saying in designing software that you should plan on throwing the first one away. I’ve never been very comfortable with that because it can be easily abused by a team to justify half-baked thinking, inadequate requirements and design, and just general hope instead of solid engineering.
That said, it’s generally been my experience that no matter how good and diligent you are when designing an API you’ll inevitably find that you’ve missed the mark as soon as the first set of users start solving real problems with it. So, you’ll want to leave room to redo and rethink your original approach before you’re married to it. This is the major reason that we shipped Gibraltar 2.2.0 as a Beta back in early April with our first vision of a Gibraltar Add In API.
Feedback from inside & outside
To make sure we got solid evidence that the design was sound we worked hard to get some existing customers as well as outsiders to try solving their challenges with the API. We also did our own internal Red Team approach: we asked Jay (who isn’t doing any day-to-day development on Gibraltar, but has extensive commercial development experience) to scratch his favorite itch with the API as well. We’re going to publish more about what he built soon, but without going into details something significant emerged from his work and discussions with our customers.
Instead of people using the API to solve a few tactical problems (like data export or an alternate view of the data) people wanted to use it to extensively integrate Gibraltar with external systems. This went way beyond our original intent; we have a big block of features on our roadmap for creating an external integration engine and hadn’t ever intended this to be the way that came about. Despite that, it’s the problem people kept wanting to solve with the API.
Unfortunately, there are several things that we just didn’t allow for, but are common problems in these scenarios:
- Configuration Management: Whenever you’re working between two systems, you’ll want some metadata to describe endpoints and map data between the systems.
- Credential Management: Nobody likes hardcoding credentials, or doing something simple like popping them in a file without encryption. We all know the right things to do, but it’s annoying and overhead. It should be built in.
- Coordination: All the various user interface and process extensions don’t exist in a vacuum; they need easy ways to coordinate their work to enable new processes, not just isolated features.
- More Extensibility: People want to be able to extend other parts of the UI again with the goal of connecting processes together.
- Development Debugging: In a twist of irony, it’s hard to log and view data from within an Add in because you’re running within the Gibraltar environment; you can’t use the Agent to log because you’re running inside of the Agent. This made development of extensions a pain.
Powerful capabilities, not demoware
As we’ve said before, we hate demoware. Although we’ve heard from a few parties that we just need to “check the feature box” with the API (have it but no one will use it), we just aren’t willing to accept that measure for success. Instead, we see another angle: We’re going to leverage the add in API ourself to enable us to create targeted features that only apply to some part of our audience so we can create them faster and keep control over the product’s complexity. In short, we’ll make sure that it’ll solve problems for you by having it solve problems for us.
Fortunately, we didn’t have to throw the first one away in a literal sense. We did end up discarding virtually our entire implementation of it from the Gibraltar side, but the change to the API you see is fairly modest.
We’re still finalizing the details, and deciding whether we need to do another beta round of 2.2 or should go ahead and formally release 2.2 so that everyone can get the benefit of all of the improvements that have happened over the past several months beyond the API. But, we can speak to a few great features:
Configuration Management
Configuration data can be shared between the Hub and all of its connected Analysts. This makes it more practical for add ins that connect systems together but need everyone on the team to share metadata on how items in both systems relate. Since add ins can run on both the Hub and in the Analyst we made it so you can change the Hub configuration for an add in remotely. This means you don’t have to run to IT and get on the server to say add new mappings for new projects or do whatever other configuration your favorite add in requires. The central data is automatically synchronized between the Hub and Analyst and is available even if the Hub isn’t (say because you’re running disconnected from the network)
For example, say you want to integrate with your defect tracking system. It’ll have some form of heirarchy for managing things – probably with different areas for the current development release vs. the production releases, etc. You need to be able to relate the data Gibraltar has (like product and application names as well as versions) to those areas so when it opens or updates an issue it shows up in the right place and in the right workflow.
Credential Management
In a perfect world, your Windows credentials would give you access to any external system your Add In needs to access, so no credentials need to be stored. Unfortunately, most of the third party systems you’ll want to integrate with (particularly ones that aren’t designed for enterprises) just flat won’t integrate that way. You need to be able to store a user id and password but do so in a way that doesn’t create an unreasonable security risk.
We’ve baked in credential management leveraging the data protection API in Windows (DPAPI) so that you can encrypt account names and passwords so they can be accessed only by the current user, computer, or the hub. Here again you can manage it all from Analyst, so if you need to change the account the Hub is using to login into your defect tracking system or something you don’t have to go to the server to do it.
Development Debugging
We’re taking advantage of improvements in the Live Log Viewer control to offer a real-time view of what’s happening with Add Ins, both the Gibraltar side and the messages logged by each Add In. We log a lot of information as Analyst runs, and we don’t want to bury you in all of that detail so it automatically filters down to just information needed to monitor and support Add Ins.
This is done using only capabilities that are exposed through our normal API, so you can do the same thing in your WinForms/WPF application as well. You’ll notice we’ve updated the live viewer to include a number of capabilities formerly only available in Analyst, like source code lookup and exception / detail viewing.
More Extensibility
We’ve added a number of places you can extend Gibraltar to the original list:
- Analyze Sessions on Hub: Any analysis engine you create can run in Analyst or Hub. We always intended it, but it wasn’t ready for the beta. This means you can have data flowing from Gibraltar to external systems without anyone being logged in or having to leave something running.
- Global Commands: You can extend the main Analyst menu bar with your own menu items to create global commands end users can easily access.
- Controllers: We’ve added some key controller interfaces so you can hold state in the right places (think MVC) and substantially improve the efficiency of integrating with remote systems that may be expensive to query.
- And More: Frankly, we’re going to continue adding items as we head towards release based on our internal experimentation of what we can safely get done and commit to. Our ultimate goal is that you can extend each of the different views in Analyst so you can have it seamlessly participate in end-user workflow.
Real World Experience
We’ve been internally using the new API to help us automate our CEIP process for Gibraltar itself and it’s given us a lot of confidence not just in the new implementation but also in the vision it represents. We’re going to be able to ship extensions outside of our release cycle, add optional modules in the box for a great initial experience, and help create a space for you to show your peers the cool things you can do where Gibraltar + Your System = Awesome.
Is there a specific system you’d like to see us integrate with? Drop us a line; we love to engage with our peers to understand new & creative ways to make a difference in how developers create rock solid .NET software.
Charting and analysis updates from real-world experience
Posted by: | CommentsI spend most of my time on the development side of Gibraltar – I lead the team writing the code and supporting our customers. I sometimes get involved in presales activities – if you click our web site chat link, it’s a good bet I’m on the other side – but the bulk of our marketing and presales work is done by my partner Jay. A few times a week we discuss what he’s hearing from folks trying out Gibraltar and we decide if we need to make any adjustments to our development plans. We give first priority to development requests from our customers, then prospects, then the fun cool ideas we have. Every once in a while those ideas collide at the same time and we know we’re onto something.
Several weeks ago I was handling a support request from a new customer and Jay had a conversation with a prospect that hit on the same issues: Our charting features for metrics looked really good on the surface, but fell down in real life:
- Really Long Strings: Our customer wanted to track the execution characteristics of every database query, and used dynamic SQL – so the query could be really long. If a label was longer than you could show diagonally on the screen, the whole chart wouldn’t show.
- Outlier Filtering: In any set of data some weird points will fall. Perhaps your code was just spinning up or whatever, it should be ignored.
- Show Me the Money: It’s all great you can summarize the information, but what details went into that? To take actions on the information most people ended up replicating the chart in the Metric Grid tool to see the individual elements of data to work with.
- Bad Topping: When you added secondary data to a chart you were topping (restricting the display to the Top N values) it re-topped the secondary data, causing downright bizarre results.
Now, we knew about the last two of them but the first two came out from real users solving problems with Gibraltar. Frankly, our test data always generated relatively short strings partly because we knew how our tool worked and how to get good results from it. The killer comment came from our prospective customer, and it hurt:
I mean, it all looks nice but basically it’s demoware until you address these issues
Ouch.
We hate demoware. We will not make demoware. This hurt even more because we really believe in the need for great charting – it’s important, it’s one of the things really unique about Gibraltar, and yet we also had a list of things we wanted to do with it. The trouble is we have a list of things we’d like to do with just about every part of the product; we’re insanely passionate about application diagnostics.
So we wrote out a set of things we had to address and moved this to the front of the development queue. We got sample data from a few customers we knew were using the feature and having trouble and went to work.
Previously we prepared the data and fed it to the chart control for analysis. Unfortunately, to fix the topping problem would be really problematic with the chart doing the work – the control vendor disagreed that what we wanted was sensible and working around it was going to be complicated. Additionally, we were jittery about having the same calculation done two different ways – one way to show the details and another to display the data. In the end, we chose to rewrite the analysis into a central set of code we could check and control. That way we could guarantee the results were consistent, and tailor them to our needs.
That’s just the way the breaks go – some things are easy, some are hard, but in the end it’s about what it’s worth to our customers. Here’s what’s fun and new:
See the Details
As the data is grouped up you can see all of the raw data that went into it. Don’t like one outlier? You can suppress it and immediately see the chart change to show how that affects the analysis. Curious as to why there’s a big spike in the chart? Click it and look at the individual rows to see the little details to know what to do.
This is one place I really love what we do vs. traditional performance profiling: You can see not just the overall time used by a method but the exact parameters used for each call and their individual time to know if it’s a problem of just one particular set of options taking a lot of time or the method is generally slow. This is useful to eliminate false leads so you know what methods are really worth profiling with your full up performance tool. The great part is that it’s safe for use in production – which gives you much more accurate information on what really matters.
Have hundreds of group items? That’s OK too – we’ll automatically scroll the chart to keep things at a sensible size. Otherwise, you can use the Top feature to just show the most significant information, whether that’s based on a top count, percentage, or threshold.
Without the Noise
In the real world there’s always some noise in the data – points that will lead you astray. In particular, if you’re looking at the duration of something you really shouldn’t judge its performance on the average or maximum; the maximum is likely a worst case scenario that reflects first time startup or a transient and the average will hide operations that are often slow, and often fast. The problem is that you’re still upsetting users with the slow ones.
To resolve this, we’ve implemented a 95th percentile performance summarization that gives you a good real-world feel for performance data. Here’s an example from our web site:
Notice that if you sorted by the average page duration instead of the 95th percentile you wouldn’t get the same pages floating to the top. It’s clear from this that we have a few pages that are consistently slow for some users – we’ll definitely be taking a look at that!
Work with Any Data
We’re often… impressed… with what people do with Gibraltar. We’ve seen proof positive that if you design a product to go to X, users will immediately take it to 3X. We’ve previously addressed some edge cases with the log viewer, now we’ve applied the same lessons to Metric Charting. Want to group by raw SQL Statements that are a page long? No problem. It’ll be fast, it’ll display, and you won’t have to worry about tooltips trying to go off the edges of the screen.
Need to still be able to see the full page of SQL when you find that slow request? Yep, you can do that too.
With more Freedom
Previously, you could only chart a single metric at a time. This was a big coding shortcut, and it worked for us internally because we designed our metrics to work with it. But, in the real world things aren’t quite that simple. Our own Agent for PostSharp, which is probably the way most people start trying out metrics, records data in multiple metrics and really can’t be charted well if you can’t throw a bunch of them together.
We’ve updated the charting to let you throw just about any combination of metrics together you think could be sensibly put together – and it’ll figure out what columns are common enough and can be graphed. In the end there are a range of reasons why you might want to record metrics in different ways in your application – and we’d rather you could do what was convenient and minimize how much you have to worry about how it shows up in Analyst.
We’ve also made the charting more configurable – you can change most of the different labels if you don’t like the autogenerated values and show or hide various elements to get the chart the way you want it. When you’re done, you can export it right to the format of your choice
Bringing it All Together
I hope you get a sense from the pictures above just how much you can do with the new metric charting features, but there’s no substitute for downloading a trial and seeing it for yourself. All of the data behind the charts in this example was collected using the Gibraltar Agent for ASP.NET so no coding was necessary to get these results. Try it and you’ll see that Gibraltar can help you solve problems in the real world you live in, not just in some theoretical abstract place where demos happen.
Check out our recent post on error notification for another example of how we are incorporating customer feedback to help you build rock solid .NET software.
Beware Demoware
Posted by: | CommentsI’m becoming convinced that there is a scourge in the Microsoft development ecosystem, and it’s called the 10 minute demo. Here’s the problem: No matter what the tool is, you have to make a demo that, in 5-10 minutes, purely using point & click design tools, shows off the power of the tool and technology.
Now, the problem isn’t that you have to get your message across in 5 or 10 minutes; I’ve found that the harder I work at getting the message down the better the message gets. Instead, I wonder how many product managers have had to sit back and say “fine fine, but enough about what customers are telling us it really has to do: how am I going to make a point and click demo that will go from user interface to database through any number of tiers and makes it all look easy?” The problems that we really want vendors to tackle are hard: If they were easy, we wouldn’t be looking for solutions.
Take Entity Framework. This is a very complicated set of technology that’s oriented at a very complicated problem. If you have the problem they’re solving, it can be a huge time and cost saver. Developing a technology like this is extraordinarily expensive, and you’ll be married to it for a long time so it’s good to have a major vendor stand behind it. I’ve been in environments where this would have cut major projects in half and eliminated piles of risk. I can imagine the challenge the Product Manager for Entity Framework had when we had to make a few short demos that show off how great this technology is.
This is a high-concept technology for integrating disparate data sources into a common model. Any demo you can do purely visually in 5 minutes of the product is going to be contrived at best. Most of the demos I’ve seen have been how you can take Entity Framework and point it at your schema and then get your client to bind to it. You know, the same thing you could do with ADO.NET, LINQ to SQL, and any number of other technologies… And shouldn’t do. It does a disservice to the tool because:
- You end up walking away with an underwhelming impression of what the technology is uniquely enabling; something that you couldn’t easily do before.
- The demo shows an approach that really doesn’t scale to real-world problems.
The last one is the real problem. Folks who are approaching using your technology to solve real world problems take the methods you use in your demo very seriously. They’ll pick it apart and use it as a trump card in design conversations: ”but here’s how they did it in the demo, just strait from that little part of the UI back to the database. So we make our huge application by just doing that over and over. It’s how they did it!”. I think of all those early demos of .NET that showed people dragging a few tables onto a designer, then dragging that onto a UI, click click click and *poof*, there’s the list of customers! But seriously, would you really build anything of scale that way? Unlikely – maintenance challenges, performance, production considerations (like common error management and security) all argue against it.
As an experienced team of developers, even we get caught by this some times – we look at the buzz and a demo around a visual design tool like LINQ to SQL and then charge out to try it on a skunkworks project. Within a short period of time we’re in trouble with it – everything started well but now we can’t update the model, or we keep losing our changes, or we can’t get it to handle the full application lifecycle. We’ll talk back and forth trying to decide if:
- We just don’t get it. We’re just too old and set in our ways, and can’t see the great brilliance of the new vision.
- The demo approach is demoware. A feature you’d only use in a demo.
In the end, we nearly always prove it’s the second. We pick it apart, talk with other (non Microsoft evangelists) folks and come to the consensus that yes indeed the natural laws of computing haven’t been set aside by this new strategy or tool.
So please, I’m begging you – if you work for Microsoft or in the Microsoft ecosystem, put an end to demoware. Your product may not make a 5 minute end to end sample – so show us how it fits conceptually. I’ll get more out of your 10 minute video on Scrum that’s funny and conceptual than I will on a contrived sample that just shows that your product can solve a problem too simple to justify its use.
A final note: I’m not really singling out Entity Framework as having bad demo videos. There’s a lot that I disagree with about how EF was rolled out (A service pack? Really? How about .NET 3.9…). They actually did create a set of conceptual videos, but the high traffic stuff in MSDN has a definite feeling of demoware.
