Principles of Scoring Models

December 11, 2012 By 1 Comment

When I was running the IT-PMO at PeopleSoft we faced an interesting dilemma. As we finished work on the integration of JD Edwards there was a ton of unmet demand for IT work from all corners of the enterprise. This ranged from tweaks to the purchasing system to an all-new global training environment. We quickly realized even our ability to analyze the demand would be swamped by the incoming flood of work.

So, we devised a scoring system. Why? There were three main reasons, all of which really comprise some fundamental principles when creating a scoring model.

First was the need to analyze and separate the wheat from the chaff quickly. Our primary driver was to be able to make an initial cut from 120+ requests to something more manageable for more in-depth analysis. So we needed a way to make quick judgment calls to find the top 20-30 project requests with the most merit.

We further realized that any analysis that came up with a specific number (like $300K for changing the purchasing program), even with a caveat of +/- 100%, would become sticky. That is to say, if the $300K estimate was later revised to $400K – well within the +/- 100-% – the executives would still want to hold us to the $300K! “I thought you said $300K 2 months ago – what changed!” was a familiar refrain. Scoring models, on the other hand, place estimates in ranges. So as long as you don’t exceed the top range it’s all good.

Many project-driven organizations today face this same dilemma on an ongoing basis. Scoring models meet this challenge well. So, to create a scoring model that will quickly find the projects with the most merit without being nailed down to estimates too early, keep these key principles in mind:

  1. Group your scoring criteria into around three buckets – these will be used as axis on a bubble chart later. My favorites are benefits, cost/size, and risk. Others include impact, and for product development groups may include market share, technical feasibility, and margin.
  2.  Scoring criteria should comprise ranges.  An example would be a 1-5 rating of potential revenue increase, with 0 = none, 1 = less than $1 million, 2= 1-5 million, 3 = 5-10 million, etc. Same goes for project cost or other financial metric. For criteria like risk, an example would be a rating on project familiarity with 1 = very familiar with this type of project and 5 = never done this kind of work before. Make sure all the criteria produce the same range of scores (e.g. 0 – 5) so you can create weighted averages for each group and a weighted average total project score.
  3.  Scoring criteria should fit the company’s strategic direction and business needs. A retailer will be concerned about increasing market share, while a SaaS company like Daptiv is concerned with customer satisfaction.
  4.  Bubble charts are a great tool for graphically envisioning which projects will produce the most bang for the buck. While the simplicity of a single chart is more efficient, I have seen new product development organizations with up to 6 criteria groupings used on 2-3 bubble charts.
  5.  Back test the model. Take the scoring model produced and score the current slate of active projects. When I did this with a major retailer a couple of years ago, we knew we had it right when the only current projects that wouldn’t have made the cut turned out to be problem children that should never have been launched.
  6.  Always analyze requests in cycles. Applying a scoring model to each request as it comes in negates the comparative process. It also leads to new priorities interrupting live projects, which results in project and resource churn. We typically recommend quarterly cycles. Monthly can work in an environment with larger quantities of shorter lifespan projects. Generally annual cycles are too long as too much work comes up in the interim. However, an annual planning process for the larger, more strategic work can be coupled with a quarterly cycle for the smaller work.
  7.  Scoring models work best when there is a cross-functional team empowered with the ability to make decisions. This means they will be high enough level in the company to not be second-guessed by colleagues or superiors.

Once requests are reviewed and sorted using a scoring model, decisions can be made about which should proceed for further analysis. Those that pass muster then pass into the more traditional initiation process for projects, ensuring that valuable analysis time is not wasted while allowing the focus necessary to properly present the best projects for funding.

 

Filed Under: The Business-Driven PMO Tagged With: , , , , ,

Effective Approaches to Project Prioritization

September 24, 2012 By Leave a Comment

‘The key is not to prioritize what’s on your schedule, but to schedule your priorities’- Stephen R. Covey

Like most of us, I have a tendency to attend to tasks that interest me the most, especially when I am expected to deliver on multiple projects within the same time frame. The problem here is that, what interests me the most may not necessarily be the most important thing that needs to get done. Personal priorities help us direct our attention to activities that give us the greatest value as individuals while organizational priorities are intended to focus more on teams and on matters that provide optimal value to the organization.  Without organizational priorities, individuals will set their own priorities (like doing what they like), hence the important stuff doesn’t get done on time. Conversely, clear organizational priorities guide us better to make decisions that are ‘directionally’ correct.

Well-defined governance outlines who sets priorities for what.  While all the work in an enterprise gets prioritized by someone, there are often multiple managers or management groups that are independently prioritizing that work. While this is totally appropriate for tactical operational work, using the same approach for projects – especially cross-functional efforts – can derail project progress while project work waits for resources that are working on locally defined priorities.

Project prioritization is not tactical – it’s strategic and, as such, it is a function of portfolio governance.  Portfolio Governance Committees or Teams are typically made up of senior-level managers and as a team they are responsible for representing the broader interests of the enterprise – rather than the individual groups that they lead.  The portfolio governance team meets periodically to review the progress of high-value or strategically significant projects, consider new project
proposals, and establish or revise the priorities of projects in the portfolio.

It is important to note that the Portfolio Governance Committee is not concerned with day-to-day prioritization of work efforts but instead focuses on longer term, strategic prioritization of projects.  Short-term issues or resource conflicts are handled by tactical management teams guided by the project priorities that have been set.  For example, if a lower level project is in trouble the tactical team may temporarily redeploy resources from a higher priority project — but only to the extent that it does not impact the performance of that higher priority effort. The priority of the projects does not change – only the tactical priority of some of the work does.

The second attribute of an effective project prioritization process is having a consistent set of criteria for evaluating projects by themselves and against others.  Using consistent criteria make the prioritization process more efficient and effective since we know what information is needed before we take a project to our governing body for consideration.

While there are a number of techniques for achieving consistency in evaluation, the most popular is accomplished with a scoring model.  The scoring model consists of a set standard criteria that reflect the key attributes of a project to be considered in the prioritization process and a set of measures or ‘scores’ that will assign a value to each criteria.  For example, let’s say that one of our evaluation criteria is cost reduction. In building our scoring model we’ve decided that any project expected to reduce costs by more than $1M will receive the highest score value and any cost savings less that $10K should receive the lowest.  We can also establish intermediate score values to represent the levels between the highest and lowest levels between our highest and lowest values.   By providing the criteria with pre-defined values we have the ability to compare projects on an ‘apples to apples’ basis.

Good scoring models take a while to create – and the best ones I’ve seen generally have evolved over time as the governance team refines their view of the portfolio and the individual projects within.  If you’re building a scoring model, there are a few things to keep in mind:

  1.  Identify criteria that reflect both the potential upside and downside of the project.  While, the example cited earlier used a scoring criteria based upon a benefit (cost reduction) you also want criteria that evaluate project cost, duration or risk dimensions of the project.
  2. Consider strategic objectives in defining benefit criteria.  While many projects may seem like a good idea, if they don’t support your organizational strategy they shouldn’t score as well as those that do.
  3. Clearly define both criteria and scoring values.  Take some extra time to both define and validate the definition of your criteria – ambiguity will only create confusion and reduce the credibility of the scoring process.

While using a scoring model to evaluate projects is a high-value component of project prioritization, the score of a project and the priority of a project may not be exactly the same thing.  While the score will inform the prioritization process, external factors such as regulatory deadlines, seasonal demand, or unforeseen business opportunities may dictate that a lower-scoring project is prioritized above one with a higher score.   For example, if a project is almost done it generally makes more sense to complete it rather than putting it on hold to start a new one. While you could build all these factors into your scoring model it would be highly unlikely that you could anticipate everything and every time something came up all the projects would need to be re-scored, adding chaos and rework to the process. By establishing a priority rank (1 to n) or priority value (high,medium, low) in addition to the score you have the flexibility to accommodate minor prioritization changes without having to re-score the entire portfolio.   Even here we need to be cautious: While minor adjustments may be needed, continual, significant re-prioritization can be disruptive.

Last but not least is communication – project priorities are only useful if the people impacted by those priorities know what they are.  The individuals responsible for priority-setting are in the best position to communicate those priorities across the organization and to communicate the business rationale for them and typically will craft the message regarding initial priorities. But one-time communication is only part of the story, regular reminders are necessary to maintain focus and commitment to priorities.  Here is where an the PMO can play a significant role by reinforcing the message in obvious ways like written announcements and meetings or more subtly, like always showing the project priority in listings and reports.  Constant visibility of project priorities can go a long way to making these priorities an integral part of how projects are viewed and executed in the organization.

Establishing a project prioritization process is neither simple nor fast.  It requires a great deal of organizational buy-in and support, especially from individuals who may see prioritization as a threat to their authority and control. However, organizations that invest the time in establishing repeatable, fact-based processes for prioritizing projects find it quickly yields positive
results and is well-worth the effort.

Filed Under: The Business-Driven PMO Tagged With: , , , , , ,

Communicating Effectively With Stakeholders: What Does IT Portfolio Management Success Look Like?

July 6, 2012 By Leave a Comment

Success is relative.  Just yesterday I was working with an organization where the CEO said, “If I could just get a manageable list of the current active projects and their current status that would be the first step to success.” As you can see, this CEO’s idea of success is incremental.  That’s the key.

I’ve witnessed successful and, unfortunately, unsuccessful attempts at implementing IT Portfolio governance models.  Typically most of these failures are:

  • a result of a narrow vision of the end state;
  • implementing a PMO with the exclusive mission of projects and methods. This is a situations where the focus is less on delivery and more on creating the Project Lifecycle Management (PLM) deliverables regardless of the size/duration of the proposed project, i.e., “The Process Police.”
  • And finally, the “Big Bang” approach to implementing technology in an attempt to buy a process out of the box.

Success is more of an organizational change management effort than simply installing a tool. Or, as I like to say “Build it, they will come.”  Unfortunately what really happens is just the opposite: Build it and they will RUN.

This notion of a change management approach is important because success is more about identifying the “change plateaus” and building a strategy to execute those plateaus instead of just implementing a tool that everyone resists 6 weeks after you go-live.  You have to walk before you run.

Key to that is having an understanding of the pain the organization is experiencing. Like the example of the CEO above, his pain was around portfolio transparency.  That was his first change plateau.

So what’s the next plateau?  That’s not always clear. That’s why it is imperative to have access to senior management, because macro-environmental influences can most definitely change any leader’s course of action.

IT Portfolio Management or Governance can be viewed in three categories.

  • Asset Management – this isn’t necessarily the way we currently view assets like servers, PC’s, etc. although they can be part of the asset pool.  For IT, I look at assets more broadly, focused on business outcomes.  It really about applications and how applications influence processes to achieve positive business outcomes.
  • Project Management – we execute projects to create, renew, and divest assets.
  • Resource Management – self-explanatory – we need people (in the right place at the right time) to operate the process/ tools and execute the projects.

The big challenge is to not let the governance of these three categories be developed and operated in silos.

Oh, and by the way, this doesn’t come in a box.  You just can’t buy software and expect the software to immediately solve everything.

So what are the change plateaus for each of these categories?  Every organization is different and each stakeholder will bring a different lens upon that pain.  But here are some ideas:

Like the CEO above, it could simply be an inventory for all three categories and the status of that inventory.

Identifying and implementing standard operating practices and service agreements for your assets, for IT ITIL (Information Technology Library) is a great place to start.

Project Management and Resource Management has really evolved in understanding to Project Portfolio Management (PPM). With PPM we are dealing with:

  • Timeliness – having the right number and the right balance of projects, completed on time without gridlock.
  • High Value – projects that reflects business strategy and are aligned with business objectives
  • Quality – projects are executed with predictable outcomes
  • Right People – having the right person at the right time to execute projects

Success is in understanding your pain points and which ones to take on first for immediate success, recognizing they all can’t be tackled overnight. Success is getting early and quick victories like the CEO example sought. Then build on to those early successes by identify the change plateaus and executing a roadmap with incremental capability and success.

Remember, IT Portfolio Management is transitional NOT transformational!

 

Filed Under: The Business-Driven PMO Tagged With: , , ,

What Does The CEO Want From The PMO?

May 24, 2012 By 1 Comment

Project Management Offices, and those with an enterprise flavor (or e-PMOs), tend to think they’re all about project portfolio management and project execution and performance. But these topics can sometimes be lost on a CEO as they don’t innately sound like they relate to the company’s vision, strategy, or performance. Thus, when cost-cutting time comes, the PMO looks like one more piece of G&A bureaucracy ripe for the chopping block.

What does a CEO really look for? To a CEO, the company is driving toward a vision – hopefully a compelling vision to the marketplace; a good example is Google’s “Organizing the World’s Information.” That vision translates into strategies to implement and further the vision. Those strategies require changes to be made in the corporation to either implement the strategy or to course-correct. The CEO is more than willing to invest in these changes to help drive the company forward.

Projects, of course, are the main vehicles used to implement change. They aren’t just isolated business cases proposed as pet projects by department heads – they should actually be creating the changes required to drive the company into the future. So, what PMO directors really want to show the CEO is how the PMO helps orchestrate and execute these changes.

First, the PMO needs to ensure that all changes connect to strategic goals and investment priorities. Each change should carry an investment classification – Run, Grow, Transform is a popular model. I like Strategic, Improvement, and Run. It provides a quick justification for the change – we need this change to keep the business running, make it more efficient, or to further corporate strategy. For strategic projects it is important they align with the stated corporate strategies. If not, is it really furthering the vision of the company?

Key PMO Value: Making sure the right changes (projects) are selected

On the orchestration side, the PMO’s involvement with portfolio planning is critical. There are two basic modes: the Annual Operating Plan (AOP) and the funneled intake process. Often, both are employed. The AOP usually emanates from the executive team as a set of strategies and initiatives with proposed funding amounts attached. The PMO’s job is to make sure these initiatives get fleshed out, planned, funded, and finally executed. They are an invaluable resource to the CEO in making sure business plans become reality.

The intake model is more typical and involved, sorting through numerous requests for changes and deciding which to pursue. Here again the PMO is valuable as the facilitator of the process, providing objective information on the alignment and relative value of each request to enable steering committees or executives to make sound business-focused investment decisions.

Either way, if the CEO sees the PMO as effectively aiding the planning of organizational changes they will be in good stead.

Best Practices for Success: Effective Reporting, Performance Metrics

When reporting on execution progress, don’t just group a project list under investment classes or strategies. Do you really think the CEO has time to wade through all that data? There are better ways of reporting progress more suited to the executive suite.

First – they are interested in how much has been invested and the returns from those investments. To highlight the investment side, pie charts dividing the budgets by investment class, strategy, and business unit work really well. Returns are trickier as not all changes result in direct dollars to the bottom line. Some are measured in market share, customer satisfaction, or other non-financial metrics. The key is creating a benefits scoring model that rationalizes the various stats into benefit units. Then report the expected benefit points out per portfolio and follow up – if you can! – with the actual results post-change.

Next up on the CEO’s list – how are the investments faring? Now we’re talking standard project performance metrics, but grouped up to the portfolio level. Budget pie charts of project health (Red, Yellow, Green) work well for a high level view. If they want more info on critical projects the single-page or 4-square status report serves the purpose.

One question frequently asked by the C-Suite is when will these changes hit the organization? The best tool I’ve seen  is the project landing map. This usually has a Gantt bar for the life of the project and milestone markers for only the go-live events. It’s a graphical timeline of change impacts across the enterprise.

If you want to put this all together in a nice neat package for corporate execs the portfolio deck does the trick. I’ve seen this used at Fortune 500 and even Fortune 100 companies at the CEO level to great effect. This is a published (often pdf) document that  starts high level but has more detail in the body for reference if needed. A common format would be:

-          Investment Pie Charts

-          Project Landing Map

-          Portfolio list (not project list) showing budget, actual, variance totals along with average health and average schedule variance.

-          Program list (largest and/or most important programs/projects only) grouped by strategic portfolio with standard health, financial, and schedule status indicators

-          4-square project status reports for all projects listed above

So to give the CEO what they want, think like a CEO instead of a project or portfolio manager. When you’re running a company, you want to know you’ve made the right investments in strategic change and that those investments are paying off.

Filed Under: The Business-Driven PMO Tagged With: , , , , ,

The Top 5 Reasons PMOs Fail

May 21, 2012 By Leave a Comment

There are many reasons a PMO either thrives or fails, but the most common reasons for failure are often not process or technology issues, but rather related to “people issues” in an organization. Ultimately, a PMO’s success rests in the hands of its team members, spanning across all levels of the org chart – it can’t fall solely on a PMO director or within a single isolated department.

 Here are the top five reasons we see PMOs fail:

  • Executive stakeholders that are not fully committed to the PMO. It’s a familiar story: The executive team realizes there’s a problem with projects not executing properly, contending for too few resources, or just not delivering results. So they authorize a PMO, and hope it solves the problem. But come time to attend steering committee meetings and make hard decisions, they send lower-level functionaries and don’t give them decision-making authority. Then, when projects are once again performing poorly due to resource contention and prioritization conflicts, they blame the PMO and disband it.
  • PMO leaders that don’t know how to adapt. What worked in one company most likely won’t work in another, despite similarities on paper. If a PMO leader isn’t adaptable and tries a “cookie cutter” approach based on experience from a prior job, then a PMO will likely fail. A better approach is to understand the unique drivers and pain points for the organization, the various personalities and motivations of key stakeholders, and whether the culture of the organization supports the ability to develop a plan with achievable goals.
  • The PMO becomes a project manager’s worst enemy. In many ways a PMO director has to be part cheerleader, part salesperson and part coach, making sure that the PMO’s mission/charter is well articulated and all stakeholders buy into it. However, if a PMO becomes part auditor and part methodology police, forcing adoption of ill-fitting methodologies or gathering unnecessary information, it will become a hindrance to project managers and rejected by an organization. 
  • No strategic vision. Focusing on tactical, day-to-day execution is fine, but PMO directors and teams need to not lose sight of the bigger, strategic picture. If a PMO leader can’t grasp and articulate the business challenges faced by senior executives, and help facilitate organizational change and alignment to meet those challenges, they will become less relevant in an organization.
  • Lack of a metric-based approach. Unless a PMO leader has an analytical mindset and is comfortable with metrics, a PMO won’t be successful.  For example, when it comes to deciding how many of the top projects can actually be done, discussion too often turns to pure guess work. Without a metrics-based understanding of resource capacity, it is impossible to match demand with the actual supply of human resources.

Let us know what you think and if you have others that you’d include in your top 5!

Filed Under: The Business-Driven PMO Tagged With: , ,