How multi-million dollar decisions are made

I've been traveling recently and spending an unusual amount of time in hotel rooms as a result. The TV in the hotel room I find to be pretty much useless. Why? Well, first of all, I can get everything I need from the hotel's internet connection. News, sports, movies (Netflix), etc it's all there and I don't have to wait (and wade) through countless commercials to get what I'm looking for. Not to mention that there's usually nothing on worth watching. I typically travel Mon-Wed and except for Monday night football (of course) I don't watch anything on the useless screen provided by the hotel. Until a few weeks ago...

I cannot remember the exact circumstance, but after a very, very trying day I did turn to the otherwise useless box in my room for an evening of "brainless" entertainment. And what did I find? The HBO's series "Hard Knocks".

For those of you who have never seen the show, the premise is a (invisible) camera crew that follows and films the New York Jets through training camp and their pre-season schedule. From a "tabloid" TV perspective it's great entertainment. We watch as men who are dedicated to the game of professional football do everything possible to make an NFL squad. We watch the trials and tribulations of grown men driving Masarati's with state of the art GPS systems that still lack the ability to be on time to practice and team events (sorry Jason, had to take the shot). The impact their dedication and success (or failure) has on their psyches and on their friends/family makes for a compelling story. On the other hand, you also get a unique perspective on the "proven" players, what they do during training camp (including hazing rookies) and the drama involved in re-signing a top cornerback to a new contract (Darrelle Revis).

(Side note, I find the "Revis island" commercial very entertaining.)

The connection this show provides to Real Sports Analytics is actually something that I've been talking about here for quite some time. Identifying and judging (grading) talent on "Hard Knocks" is done by guys who have been coaching and been involved with the game of football for more years than I've been alive. Rex Ryan probably has more football knowledge in his left pinky finger than I have in my entire body. But you know what? Sometimes I think all of that knowledge gets in the way of good decisions...

One of the most important, critical benefits that I bring to my clients in my consulting practice is my view from "outside the box". Call it a third-person perspective or "30,000 foot" view, whatever - basically when anyone, including businessmen, coaches, scientist, etc look at a problem for too long they get a form of "blinders" on their subject matter. I'm sure this has happened to you.

At the end of most training sessions during the Jets training camp the coaches would sit together (in a somewhat spartan environment) to discuss the days events and provide roman-emperor-like "thumbs up" or "thumbs down" on certain players. There are a few things that I find interesting about this...

First of all, no one brings up grading. Not once do I hear a coach say "Player X graded out at 90% in the practice session, maybe we should try him with the first team tomorrow". The feedback from the coaches almost entirely consists of "I like this guy because..." or "I don't like this guy because...". The performance measurement that takes place during player evaluation consists of the guy either doing a "good job" or a "bad job". At one point a player is considered a spot on the roster because he's a "hitter" (i.e. he made a great hit in a kickoff coverage situation in the previous pre-season game). I mean, it's incredible to think that those "multi-million dollar" decisions are being made by a bunch of guys sitting around in a locker room and (virtually) putting a wet finger in the air to see which way the wind is blowing.

Secondly, we really are talking about multi-million dollar decisions. I'm not joking, the total salary for the Jets for the 2010 season is reported to be $116,910,097. These guys are sitting around in a concrete block room on what look to be folding chairs around what looks to be a folding table discussing $100 million freaking dollars!!!

Third, most of the decisions really seem to be about whether a certain position coach "likes" the player being discussed. I mean, I always suspected that a portion of the decision on whether or not to sign a guy to the squad was based on "gut instinct", "feel", "intangibles", etc - what I didn't realize was that it make up close to 100 % of the decision making process. I will give them credit, sometimes they look at film so at least 20th century technology is involved. But analyzing a player by the different skill sets they are looking for and putting something MEASURABLE around that? Not once.

That last one really floors me. This is a BIG business and it's obvious to me you can tell the teams that really care about player performance measurement and those that don't. Coach Ryan, I gave you props a moment ago regarding your knowledge on the game of football, but it's silly to think about doing this same type of player evaluation/selection over and over again each year and expecting different results (a common definition for insanity).

What I also find interesting (count this as #4 above) is that the Jets ownership thinks they can really "buy" themselves into contention this year. If you watch the show, coach Ryan talks a lot about putting together a "superbowl" team and how all of the "pieces are in place". Well, I'll only make one observation on that... Ask Dan Snyder down in Washington how that worked out for him the last few years. Last I checked the Miami Dolphins spend much more on their team salary - what's their record the last few years?

My 3rd person perspective on this? Individual players are not enough - you've got to find, evaluate and field a good TEAM. You've got to put all of the necessary components in place to make your team successful and you can't do that without measuring what you NEED. Blindly buying the "most expensive" guy out there isn't necessarily the right thing to do if he doesn't grade out well with what your TEAM is trying to do. Spending $46 million on a guy for four years of service doesn't help if he's out with a hamstring injury (get well soon Mr. Revis). Might it have been better to have found THREE guys at say $14 million a piece that work more cohesively as a UNIT?

My advice to the audience? Watch closely, there are certain teams that are really starting to get this concept and you can tell who they are. Ironically, in all of my discussions with professional sports teams I've only received ONE distinct "NO" (as in don't call back, we're not interested, don't contact us again). Yep, folks in Cleveland, you're in for another long, long season.

Hindsight is 20/20. Having a unique, third person perspective on your day-to-day evaluation processes can be priceless (hint: there's a team up in Boston that does just this).

It's called Real Sports Analytics.

Enterprise Performance “M” (NOT statistics)

I’m on my way back from the MIT Sloan Sports Analytics conference in Boston.  I’ll post a review and thoughts on the conference later on when I have a chance to compile my notes.

Something that came up in conversation during the conference and during the reception afterwards was the definition of EPM.  Basically, EPM is something that I’ve heard tossed around in the last few years and is a term that has been defined and redefined by just about every services firm, software company, industry analyst and talking head on the planet.  I’ve personally heard it defined as “Enterprise Performance Management” and equally as much as “Enterprise Performance Measurement”.

Now, here’s a disclaimer.  There are numerous wonderful books out there that try to help people define and establish EPM practices for their business.  These books are fantastic in that if you’re ever having a problem sleeping at night pick up one of these.  Problem solved.  (I would especially recommend the Norton/Kaplan books.)

What I also find comical about these books is that you would really trust your knowledge (and your organizations health) to the concepts presented in a book!  I would compare that to having your teenager read the owner’s manual of your car to learn how to drive.

Here’s what I can add to the already muddied waters though.  This is based purely on experience as I myself have fallen asleep on many of the above-mentioned book.

Performance measurement is the process by which you define measures to help you analyze the effectiveness of your business.  This “effectiveness” ratio can be financial, operational, managerial or any other “ial” that you feel is of value to your business (I’ll provide a simplistic example below).

Performance management on the other hand, is taking the information provided by performance metrics (as defined in the previous paragraph) and doing something that then (hopefully) has a positive impact.

One of the most common examples is a simple inventory management example.  Let’s say a major retailer has 50 white t-shirts on the shelves at a local store.  Based on what someone can see in the inventory management system the quantitative observation can be made that the inventory level is 50.  But, WHAT DOES 50 MEAN?  Is 50 good, bad or about right?

This is one of the main elements of a performance measurement system – the difference between quantitative data and qualitative information.

Back to our example.  If the “ideal” inventory level for those white shirts is 50 then we’re right where we should be.  If the ideal level is 10 then we are 5x in an overstock condition (not tragic, but not good).  If the ideal level is 200, then we are woefully under stocked which could potentially end up in a dreaded “out of stock” condition which results in a direct loss of revenue – which is obviously a bad thing.

A scoring possibility in this scenario and one that is commonly employed in the retail space is a scale of –10 to 10 with zero being a neutral middle ground.  -10 would report a dangerous understock condition and 10 would report an overstock, very high level.  Now we have a qualitative analysis that qualifies as a performance measurement and can be subsequently used in performance management.

Advanced performance measurement systems (like what Real Sports Analytics is based on) can take the inventory levels for all of the these t-shirts and for that matter everything in the store, compare them against their ideal levels and produce usable information on the inventory health of the product, the department, the store, a region of stores, a distribution center, etc, etc, etc.

Now, let’s explore how this concept fits into sports.  Let’s say a very talented wide receiver makes a 10 yard catch.  Great, we have quantitative data similar to the 50 t-shirt example above.  But, what does that 10 yards mean?  Was the play supposed to go for 15 yards and the receiver ran an improper route?  Was the play only supposed to go for 5 yards and he picked up extra yards on effort?  Was this near the end of the game and it was 3 and 8 and they needed the ten yards for a first down and the possibility of running out the clock?  Or did he need to get out of bounds to stop the clock and didn’t make it?

These are all factors that allow us to take this simple 10 yard catch and, usually with the help of coaches or scouts, put a qualitative value on this action.

When you observe this activity over hundreds or (hopefully) over thousands of observations you can readily pick out the noise and put a quality metric on the performance of any player, set of players, offensive, defensive or complete team.  This is valid whether it’s a wide receiver, quarterback, offensive linemen or defensive player.   You just need to define the right metrics for YOUR organization.

When these metrics are defined correctly and adjusted over time you can create a very, very effective performance management systems that can have direct, measurable results on the playing field.

So, what metrics are of the most value to your organization?

Metrics for the sporting world

Recently I was approached by someone looking for a system to track instance-based metrics for a soccer organization.  Soccer is a fantastic sport and (obviously) has a huge global audience.  While soccer hasn’t quite “caught on” here in the states, I’ve traveled enough overseas to see first-hand how passionate soccer fans can be.  I also realized that my past postings have mostly been focused on American football – my apologies to all soccer, hockey, basketball, etc fans that may be reading this.  Performance, instance-based metrics are alive and well and extremely valid in the world on what I call “continuous” sports.

Let me explain that last part very quickly.  American football, baseball, even rugby can be considered what I call “play” based sports.  A play is initiated, certain actions are taken, results occur, play stops and then the process is repeated.  For something like soccer or basketball the game is much more of a “possession” based event with continuous action (with the exception of events like out-of-bounds or injuries). 

There are exceptions to these classifications of course.  Take for example a soccer match with a corner kick.  A player kicks the ball from the corner position, usually to the middle of the field where several, dynamic things may occur.  Players from different teams can be interacting with the ball through a series of actions – headers, kicks, catching (goalie only of course), etc.  So how do you differentiate a “possession” or a “play” in a so-called continuous sport?

Well, it’s quite easy actually.  Every single player on the field, regardless of whether or not they are interacting with the ball, is performing some action for almost every second of the game.  Players are jockeying for position, the goalie is setting up his defense and positioning himself (or herself), the kicker is setting up the offensive players, etc.  There are some cases (again exceptions) where this is not valid.  For example, the opposing goalie may not be doing a whole heck of a lot during a corner kick on the other end of the field.  But, for the most part everyone is doing something.  Each of these actions is measurable.  Just like in the business world where top-tier organizations examine each and every transaction for actionable metrics, every action during a game can be observed and graded.

To put it mildly, what might seem “normal” or even “boring” to the untrained eye is anything but (baseball is another great example, but we’ll save that discussion for another day).

As you can also imagine, watching every single player involved in the action, over tens or possibly hundreds of skills needed to perform at the top level and doing that for an entire match is going to generate a very, very large amount of raw data.  Now, imagine that same raw data being gathered for both teams for every game they play.

Well, I’ve already talked a lot about the capabilities of Real Sports Analytics and its expansive data handling capabilities.

Is YOUR organization ready to take performance to the next level?

2010 NFL Draft

No, it’s not too early to start talking about it.

By this time in the season we already have a wealth of information on who is succeeding in the NFL and who is not.  With a fair degree of certainty we know who is going to make the playoffs and who is going to advance to the divisional and championship games.  Sure, there may be an outlier here and there, a team with a .500 record who’ll barely make the playoffs and survive for a week or two there (or like the Giants two years ago win the whole thing), but what strikes me about this year in particular is the “distance” between the top tier teams and the bottom.  Since the introduction of the salary cap the NFL has become more and more competitive every year – this year is the anomaly.  There were games that were played in the last two weeks where we could all but certainly predict the winner.  All that was left for chance was the point spread.

My point here is that except for the top tier teams that KNOW they are going to the playoffs (Saints, Patriots, Broncos, Colts, etc) everyone else should be starting to look at building their team for the next year.  How is that done?

Much like any multi-step self-help process, the first step is the admission you have a problem.  Is your problem on offense, defense, special teams or all of the above?  Then, break that problem down into positions – quarterback, offense line, defensive line, backfields, gunners, kickers, etc.  Ask yourself this:

 What is it specifically about these units that are not functioning as expected? 

More importantly, define and standardize the performance metrics that are most important to your team being successful. 

What elements (grading) are showing where individuals or groups of individuals are not performing for your team?

Once these “holes” are identified how do you plug them?  Off-season trades are a good way.  As teams drop out of the post season picture, in-season trades may even work well (it’s also a great way to improve your draft position).  But, the one instance that occurs EVERY year for EVERY team is the combine and draft.  Since this is the most important event, shouldn’t we start talking about it now?  Where do most of these players come from?  You bet, they’re in action every week just like the NFL squads with one exception – they play on Saturday.

Over the next couple of months I will be building my special “draft edition” of Real Sports Analytics.  I’m going to take the top 100 draft possibilities and score all of their 2009 collegiate games.  I’ll use the base metrics (already qualified vs. on-field results) and make small modifications along the way to adjust the weights to get a finely tuned model.

This information can be critical to teams already looking ahead for next year and I’m looking forward to seeing the results myself!

Numbers don’t lie

The saying “There are three kinds of lies: lies, damned lies, and statistics” is normally credited to either Mark Twain or 19th century British Prime Minister Benjamin Disraeli.  It is a saying that I have often used especially when watching politics since it seems there is a statistic for every argument.  (I will NOT get into that discussion here though!)

Nevertheless, I have learned from experience that this saying is NOT valid in the world of business.  No matter how many excuses you make or how many people you point your finger at – NUMBERS DO NOT LIE!  I once worked with a VP of sales who told me “SELL and all of your problems go away” and he was right.

In the world of sports you can “feel good” about a player, you can like his values and beliefs, you can even like him as a person or as a family man, but in the world of professional sports performance matters more than anything else.

Remember the scenario I outlined in my previous post?  An NFL team in the southeast was having difficulties in the preseason picking a starting quarterback.  The initial 1st string guy was a long-time NFL veteran with a very, very mediocre career given his physical talents.  The 2nd string guy will probably be a perpetual backup in the NFL and has never started AND WON an NFL game in his career.  Then, there’s  the rookie…

Well who knows how the season will play out and I certainly do not have any “insider” knowledge with this particular organization, but I can tell you this.  The 3rd string rookie consistently graded out better in preseason play than the other two.  You’ve also got to remember the challenges faced by a rookie trying to make a squad: limited snaps with the 1st string offensive live, little access to the 1st string skill players and all sorts of practice limitations and lack of game day experience, etc, etc….

So guess what happened this weekend?  The kid finally got a chance to play!

Now, this particular team is still facing some very, very large challenges in turning their season around, but given the hostile environment faced this weekend, the kid didn’t do half bad.  The question I always have for NFL coaches is “Would you rather have a veteran player in there making rookie mistakes over and over again or would you rather have the rookie in there making mistakes, learning and getting better.”  As I mentioned before, I just don’t think (if history is any lesson) that the first two quarterbacks are going to get any better.  But, this rookie gets better every time he touches the ball.  His Real Sports Analytics scoring tells me that he gets one or two more starts and he’s going to have his first 300 yard day and the team will get one in the “W” column.

I guess only time will tell.  Ironically, I do know that this organization has some sort of “computer” system to help in evaluating their player performance.  Apparently it is NOT working too well for them this season.

Time for some home cookin’…

I was watching some pre-season games this weekend and became interested by an all too typical story line.  It’s the preseason in the NFL for those of you who don’t watch football and many teams this week have to make drastic cuts to trim their roster to 75 players and then to the league maximum of 53 by next Saturday.  Some guys will make the roster, others won’t and will either go back to their “day” jobs or end up on a scout team making far below the league minimum. 

By the way (side note here), the league minimum for a rookie is currently (2009) set at $310,000.  The salary range, again minimum wage goes up to $535,000 for a guy with 3 years experience in the league and a measly $845,000 for a veteran with over 10 years experience…

Unlike some fans, I actually like this time of year, most people think “oh, it’s only pre-season, it doesn’t matter”, but for these young men playing their guts out trying to make a squad it makes an incredible amount of difference.  Unlike college teams where there may be some favoritism involved in many “cut” or “travel” decisions the NFL is all about business.  What 53 players will give me the best chance of getting to the playoffs and making a run for the Super Bowl?  Sometimes questionable decisions are made – players are released and go on to have extremely productive careers in other organizations.  How does this happen?  Does one player “fit” poorly with one team and exponentially better with another?  How are these decisions made?  Do the coaches and front office people have the correct information all the time to make these business decisions?

Those are the same types of questions that I’ve encountered the last 15 years in banking, finance, retail, supply chain, transportation, etc.  What skill sets need to be fostered in your organization so that these decisions can be made?  What differentiates organizations that make “good” decisions versus “bad” ones?  What tools do I need to ensure my personnel have all of the information necessary to make these decisions?

This is what REAL SPORTS ANALYTICS can do for your organization!  Take the guess work out of the statistical analysis of your grading system and allow your coaches to focus on the intangibles that are so important.  Give your personnel the tools they need to make better decisions.  And most importantly use 21st century technology to give your organization the competitive advantage to get on that road to the playoffs.

Now, for the “home cooking” part….

There is a QB battle going on right now with an NFL team in the southeast (I won’t name the team).  There is a veteran who has had a mediocre career at best.  There are two younger guys who have some NFL experience, but are still very unproven.  Then there’s the rookie.  Who will win?  That’s where this gets interesting.

I am going to use the Real Sports Analytics application that I have created, powered by Essbase and Dodeca to grade out these four quarterbacks.  I can already tell you that the current 1st string veteran did NOT grade out well the other night.  He consistently missed open receivers (mostly high), his decision making ability was not sound and in a  couple of instances did not make a clean handoff or pitch to the running back.  Basically, this “veteran” is a train wreck waiting to happen…  But, the coaching staff is new, they are still trying to use an antiquated system for player grading and these miscues may be overlooked in the long run, we’ll see.  I also have some concerns about the rookie’s arm strength, just not convinced yet he can hit those 15 yard “out routes” from a cross-field position.

I’m going to watch the game again to score the 2nd, 3rd and 4th string guys – I’ll let you know how that turns out.

Why Real Sports Analytics is BETTER than your current system.

I was watching a pre-season game a few nights ago and saw what was probably the best example of why Real Sports Analytics is better.

Here’s what happened – Team B kicked off to Team A.  The kickoff was a touchback so Team A took over at the 20-yard line.  After a commercial break, Team A’s quarterback called the play in the huddle and the team lined up.  Team B’s defense was obviously coming up for a blitz.  The safety was near the line of scrimmage, the outside linebackers were on the line and watching the football closely, everyone was obviously trying to time the snap.  The quarterback stopped his snap count and called an audible.  He turned to the running back to make sure he could hear, you saw several offensive lineman look toward him to catch the new play as well, the receivers acknowledged the change also.   The quarterback took the snap, took a quick three stop drop and delivered the ball perfectly to the receiver on the right side who had cut off his route to run a quick slant.  At this point the quarterback has done everything perfectly, he’s read the defense, adjusted his blocking, adjusted the WR routes, took the correct drop and delivered a perfect throw.

But, you know what they say about “best laid plans”…

Here’s what happened next.  The receiver bobbled the ball over his left shoulder, the cornerback (Team B) grabbed the football and took off for an easy score.

“Traditional” scoring systems (i.e. passer rating) would ding the QB for the interception as well as the pick six even though he did nothing wrong.  The receiver on the other hand might be charged with a “dropped ball” which is pretty misleading as well since it usually doesn’t take into account the game situation, accuracy of the pass or the result of the play.

Now, let’s look at this scenario from the Real Sports Analytics perspective.

The QB gets a good score for reading the defense, calling the correct audible, adjusting his blocking, taking the correct drop (3 instead of 5 steps) and delivering a perfectly thrown pass.

The RECEIVER gets a good score for communication with the QB and running a correct route.  He get a BAD score for not catching the ball and allowing the interception.  Depending on if he was in a position to make a play on the return, he might get a good or bad score on his tackling ability.

The CORNERBACK for Team B gets a good score for coverage, identifying and jumping the route and for catching the interception and turning it into 6 points.

What?  What was that last statement?  Why am I grading the OPPOSITION?

Well, there are really two points to be made in this post:

1)  Score how players actually perform doing the specific tasks that are necessary on the field to win.  This includes decision making ability, mechanics, vision and of course athletic ability.  Offensive coordinators and most coaches know what play is being called in the huddle.  Because they are on the field they can most likely hear an audible that’s called by the QB at the line of scrimmage.  These are known items, grades, metrics… whatever you want to call them, they are collectable and measurable and over time can pinpoint strengths and weaknesses of every player on your team.

2) Speaking of strengths and weaknesses – wouldn’t it be nice to look at past performance of your opponents to see how they are grading out?  You may discover that a certain defensive left tackle is having an incredible year, but only in the first 3 quarters – could it be a conditioning issue that the opposing coaches haven’t quiet figured out yet?  Well, if your team knows this they’ll know to run to the opposite side of the field for the first 3 quarters.  But the 4th quarter, when it comes down to crunch time, that’s when games are won or lost.  And that’s when you most need the competitive advantage and Real Sports Analytics brings to the table.

Is your team using an outdated grading system?

If you’re already using technology to score items like this are you limited to how many metrics you can track, how many players you can view, how quickly and easily you can get this data when you need it?

Have you ever even though about tracking your opponents to discover strengths and weaknesses beyond the “traditional” performance scoring systems?