BI Estimation - "A more appropriate metahpor"?

http://peterjamesthomas.com/2009/03/18/a-more-approprate-metaphor-for-business-intelligence-projects/

So this leads me to suggest a different metaphor for BI projects. Major elements of them are much more like archaeological digs than traditional building. The extent and importance of a dig is very difficult to ascertain before work starts and both may change during the course of a project. It is not atypical that an older site is discovered underneath an initial dig, doubling the amount of work required.

That's so obvious it kills me that I hadn't thought of it before.

Let the DB Handle It

Perhaps I'm pointing out the obvious, but software developers don't always seem to experience data the same way that, say, a DBA or a BI developer does.

I'm not saying that they don't process in rows and columns, or that they all have Dyscalculia or something.  I'm saying that I frequently detect a reluctance in developers to let the database handle it.  I theorize that this issue can come from a number of possible sources, but is usually a symptom of childhood trauma related to set-based math...

The rationale provided is often one that I consider an artificial barrier in this day and age, which is the old saw that "business logic can't go in the database".  In my decade-plus of experience maintaining, designing and building business software, most of the databases I've encountered have been fairly inextricable from their front-end line-of-business apps from the beginning.  And most of these apps are not going, say, make a switch between platforms or OS's without a wholesale rewrite of the code, anyway.  So using PL/SQL packages or a set of T-SQL stored procs to help encapsulate logic is not a very serious crime.  As long as it's documented, of course.  Also, the best iron in the whole solution always goes to the DB machine. Why not leverage it to accomplish the work of the application? Save some clock and RAM utilization in your web farm and let the database do it.

Another scenario is when the RDBMS is considered a "Mystery Hole," where strange, nigh-impossible things happen in the darkness and hum of the server closet.  I find this understandable enough from developers. I don't see much evidence that good data theory is being imparted on software dev students.  And SQL itself, as a language, operates in a different paradigm from any procedural or OO language, and it requires those nasty set math skills to become accomplished with it.  Advanced SQL techniques do take years to learn, and even to recognize when they can be brought to bear. 
That said, a Pivot operation takes a lot more Java or C# code to implement than it does SQL.  And frankly, being good with SQL means bever having to write a loop. 
It makes sense (to the point of being mandatory, IMHO) for software developers to learn how data works, how SQL and databases work, and when you can leverage them for processing power by letting the database handle it.

A variation on this "Mystery Hole" thing is where it's not about being unfamiliar with the DB environment, but instead it's all about control.  The developer simply can't have any of the logic NOT in code that she has her hands in.  Maybe I'm wrong but, on a big enough project, that way lies madness, methinks.  And again, it fails to heed the wisdom of "rendering unto Caesar that which is Caesar's."  For certain tasks/functions, it's often more expedient and design-time and run time to just let the DB handle it.

Now, I have encountered on many occasions scenes where DBAs and application developers are at odds.  Usually over who gets to do what in whose environment.  In many of these, I wouldn't be surprised to find one of the syndromes above involved.  But there are many times when the obstinacy and regimentation of a DBA will make developers want to punch them dead in the face rather than get involved in DB coding.   So it's certainly a two-way street, and I'm not going to applaud overly-controlling DBAs any more than overly-controlling Developers.  Clearly both parties are on the same team, and should be cooperating to solve the business goals they support, yes?  Rather than playing parochial games over fictitious territories?

Maybe the fact that I'm partial to such compromise (or even averse to conflict) is why I ended up in the BI world.  But as an application developer I always felt it was critical to have insight into the DB layer in order to be a well-rounded architect-level software professional. And if you could get it, working knowledge of the the network/infrastructure layer, too. 

But let's not get ahead of ourselves.  I suppose it's enough to expect that software developers know when to -- well, you know.

Why I switched from Android to Windows Phone | PhoneDog

Why I switched from Android to Windows Phone PhoneDog

"The Tip of the Spear"

When talking about information hierarchy with clients, I often use the analogy of the "tip of the spear".  What I want to express to them is the idea that reporting is most effective when it answers the most important questions first.  Once that's done, it's best to make available increasing levels of detail, on-demand.  In other words, the focus of reporting should start narrow, and widen to include more specific data.  The POV to focus on is that of the user, and their critical questions.

Of course, this isn't the way things develop organically.  If left to itself, any reporting or data operation in a company will follow an "accretion" model, where functionality and capability gradually builds up in an organization as a function of time -- not a function of efficiently doing the job. Databases with unrelated clusters of tables that enter the system as functionality arises.  Reports made to meet an immediate demand, but then left behind to be rebuilt later by someone who didn't realize the original existed.  The scenario accumulates junk, just like a hippie's... well, you know.   And so it is that the first step to avoiding or fixing this situation is in organizing a coherent information architecture. 

Aim at "first things first." Identify the top users, the top data interaction workflows, the top metrics/KPIs, and the top reports.  Then, you can proceed in order of priority down into the details of who needs to see what data, and how.  Essentially, you want to put together something like chart of business metrics which might overlay an Org chart.

This "information hierarchy org chart" should help you identify who needs to see what data, at what detail and business level.  It should help identify drilldown/drillthrough paths that might be necessary in resulting reports and/or dashboards.  It should help shape the reporting offering being made at each level of the business.  And it can also help prevent redundant efforts by showing where common views of data are required and how they are connected to other views. 

You can usually expect that a summary level of KPIs can be decomposed into lower level metrics, with each lower level being of importance to somebody (if not everybody).  So follow this trail from the data at the bottom up as well.  Because it's somewhat likely that natural "fault lines" within the data will also exert an "upward" force counter to the business rules, that might actually suggest reporting elements that are not as easily revealed from a purely business perspective.

The goal is to arrive at a catalog of reporting elements aimed at answering the user audience's most important questions first, and then facilitating answers to the "why?" questions that come next.  As an added bonus, the implementation of those elements can easily be organized into several different development workpaths that allow flexibility.  Would the client like to implement the entire summary level first, to satisfy executive needs?  Or is it more critical to build an entire "stack" of reports around one area, in order to fully explore that particular metric from summary level down to ultimate detail?  Either direction is possible -- provided the appropriate foundational knowledge of the wider system goals is in place.

Hippie's Backyard Redux: Data vs Information

What's the difference between data and information?  
It's not a trick question.  Data are (or "is", depending on your grammarian tendencies) unadorned facts,  while information consists of facts evaluated and/or interpreted to provide a particular meaning. It's the difference between plotted points on a graph, and a trend-line drawn to connect them.  The points are interesting, but the line tells a story.  Data points have value, but to really get answers you want information. Therefore in order to answer business questions, a typical reporting user needs and wants plenty of information at their disposal, all based on dependable data. 

So then why is it that these users typically end up with a sea of data points, but are left on their own to derive any usable information?

Tough one...

I know there are some organizations that simply don't have the processes or mandate to find and fix inefficient processes -- including reporting processes.  The scope and depth of a problem process simply sits in an organizational "blind spot".  Northridge has been called in to "fix" some serious manual-extract, chicken-wire-and-duct-tape Excel situations, fed through a shadowy underworld of data connections hidden to the non-adept, where if users can get any kind of realistic data at all, they're thankful.  Of course, if that kind of solution is already causing pain, you're too late for duct tape.  But somehow the company can't see the larger-scale syndrome at work.

Also I've seen more than one "bottom-up" reporting system, where the shape and structure of the user-facing solution is based largely on the data and the structures it occupies, with little or no recognition of business-based reporting needs.  In these situations, the user frequently must sift through arbitrarily-named (from their perspective) metrics, segmented in senseless ways (again, from their perspective) in order to get answers.  Or they get data access along lines skew to the business roles they fill, restricting them along axes of data volume or visibility.  "Data, data everywhere, nor any conclusions to be derived."  Of course, by then time this has been built, there's neither budget nor guts to actually build a bridge to the users.

And of course you can never underestimate the sheer inertial power of "We've always shown it that way"  ...

But in all cases, let's face it -- the goal is to get OUT of that situation where you're giving the right stuff in the wrong format, and the right people can't find it anyway.  So, the key manuever is to evaluate one's reporting offerings through the lens of business context. Context, which sums up in one word the difference between data and information, gives users the background they need to derive conclusions, make decisions, and ultimately take action when they look at a report.   In practical terms, you're showing users graphs with trend lines instead of grids of numbers.  Using indicator graphics to show whether costs are up for current period over previous period.  It also means organizing your reporting data in a way ammenable reporting needs.

Detailed data will still have its place since certain types of analysis may require a deeper look into the data behind an informational trend.  But that's why the good lord invented drill-down logic.  I refer to it as a "tip of the spear" strategy.  You want to start with a summary, and allow further exploration along established data relationships.  Click the "up vs down over prior period" indicator and it shows a grid-style report with daily details.

Accounting for users workflow-oriented interactions with data can help establish a more coherent and useful representation of data.  This structuring of information based on users' needs is called Information Architecture, which is defined by the Information Architecture Institute as "The art and science of organizing and labeling web sites, intranets, online communities, and software to support findability and usability."  And it can mean the difference between efficient and effective information delivery, and a hippie's backyard.

Article link: Ten Mistakes to Avoid When Estimating ROI for Business Intelligence

I like this article, smart stuff:
http://www.bi-bestpractices.com/view-articles/4781

Keep Politics From Overwhelming Your Project

Have you found yourself on a project trapped in a client political struggle?  With no dog in the fight except your burning, itching desire to reach code complete and deployment?  Well, you aren't alone.  Consultants often occupy some unique political positions with respect to their clients' organizations. 

Since we are not part of the company, we frequently have outsiders' perspectives.  Over the course of our engagements, we survey how relationships work in these organizations, summarily critique the entire culture, and usually end up just shaking our heads.  Other times we feel just as mired in tar pits as the client stakeholders we work with.  Waiting on legal signatures, approval documents, an internal IT resource to complete a precedent task, etc.  And of course, sometimes we defect because our clients' have better gigs...  ;-)   

Now, if you are one (like me) who specifically tries to avoid the whole subject of internal corporate politics then you must eventually realize, as I did, that in real life not everyone is completely earnest or imbued with a puritan work ethic.  Learning how the political game works for your clients, and knowing who the players are, is imminently valuable even if you don't want to join in.

Because the fact is that our client companies' politics are effectively a force of nature when it comes to running projects.  It behooves us to look objectively at this factor EARLY on in a project -- preferably before getting contracted into an intractable situation.  I have learned through a long chain of missed tells and heartaches to pay close attention to the interpersonal dynamics going on with clients.  Those sorts of things can reveal dangers to be avoided or otherwise defended against throughout the project.  Yell-y bosses, ineffective or power-tripping gatekeepers, impossible-to-please approvers; you can often see it in the very earliest project meetings--if you're paying attention. 

That said, it's frankly quite difficult to track all the various wavelengths of information transmitted by clients in early project meetings: technical context and detail, political context, business context, interpersonal relationship context.  You need at least two people to effectively document/remember/witness it all.  I think it even helps to record audio where possible, too.  As throw-away lines your clients utter can be the missing piece that later keeps the puzzle from being completed. 

No matter how it's done, the point is that you should harvest not only the technical details, but as much of the human context as possible.  And keep those tidbits of info in mind as you proceed.  They can aid you in project planning and in managing the overall relationship on a number of levels:

• allowing for slow-moving companies in your timeline
• selecting which client stakeholders you want to deal with and how often
• identifying your allies and enemies within the organization,
• identifying potential problem decision makers
• finding friends where it's advantageous
• saving you from gaffes that jeopardize your project 

Any of these elements can and will, regardless of your project's technical merit or value, blow it right out of the sky.  Seriously, some person up the line is holding the purse strings for your project, and they can be influenced in many ways that are entirely disconnected from whether you are technically succeeding.  I see very experienced and good consultants spend more time and resources managing these elements than they do on the technical details.  Because in some cases, the political value of the overall consulting relationship is equally if not more valuable than the technical help ostensibly being provided...

Now, if this sounds a little Machiavellian, or perhaps a little "Art of War"-ish, well... it is.  Congratulations: this is consulting. You don't necessarily have to jump in and get your Iago on.  But at least knowing about the game can help you play smart when you have to, and to avoid some potential pitfalls and obstacles to doing your "real" job.