Innovation, Product Development and Commercialization by Dariush Rafinejad, Ph.D.[edit | edit source]
3 comments/questions from each individual per chapter. Redaktors should be completed for Monday's class. Please have comments completed by Sunday 1700. I just threw these times/dates in, if we want to come to a consensus on an actual time to submit the redaktor notes that is fine. (Ho)
Chapter 1: High Technology - Redaktor Richey[edit | edit source]
1. (Skiple) The author stresses the importance of doing the "whole job" in product development. This is a big investment for any size of an organization. Can it be sustained in the long run, or will having such a large structure in place lead to more short-term decision-making?
- Good question...I suppose it really depends upon how the organization expects to succeed. If product development is their "purpose" in life, then yes, it would probably be sustained. If it isn't, it will likely lead to what you have surmised. Is the AF in the business of product development or in another business...you can draw your own conclusions. JRW 17:39, 5 January 2011 (UTC)
2. (Ho) Mention of the development team for new products. Who exactly is on the development team and what are their roles?
- That is a function of what the business has determined it to be. Obviously, in the context of this book, we'll get an idea of what roles we would expect to be on the team. But I wouldn't expect it to be much different than traditional roles such as program manager, engineer, marketing, etc. JRW 17:39, 5 January 2011 (UTC)
3. (Pauli) Systems engineering pops up in a few different ways (pg 3, 17) highlighting the importance it plays in R&D (and why we have it in our GRD program). Reasons why: 1) Early decisions in the design phase impact a large portion of the costs in execution (affecting profitability and ability to fund other projects in the portfolio in the long run 2) Building in modularity to accommodate newer technologies assists in market penetration agility and momentum (2 of the success indicators).
- Correct - glad you're seeing why certain things were included in the GRD program. JRW 17:39, 5 January 2011 (UTC)
4. (Richey) The author makes a pretty big deal about IT as an enabler. IT is valuable to high-tech R&D as an enabling tool for many reasons mentioned in chapter 1. However, I believe IT's ulimate value is that it enables globalization. Virtual product development and global marketing teams accentuate this point, as they are key strategic initiatives.
- Fair comment. I have a couple of PhD references to global/virtual product development. Let me know if you're interested. JRW 17:39, 5 January 2011 (UTC)
5. (Skiple) Using the metaphor "digital nervous system"—moving information (nearly) instantly throughout the organization—the author claims that in large organizations (like, for example, the Air Force) this can improve communications that would otherwise be slowed down due to their large size. Isn't there some danger, when the information moves that quickly, of losing the context of that information? Is there always going to be some need to filter information to some degree?
- Absolutely! This is where the idea of cyberops/cyber warfare has already eclipsed the ideas of the author. We all have seen examples of where an email goes "viral" with unintended consequences. In this case, the author is a bit altruisitc. JRW 17:39, 5 January 2011 (UTC)
6. (Ho) I find it odd that figure 1.6 would not have the customer as the center of the network. One would think that a marketing system or value chain network would centralize the customer. There really isn't a business if there's no customer.
- Agreed! Now you're catching on to some of the dangers of focusing on product development merely for the sake of product development. That said, there is a known phenomenon of being too close to your customer and missing the "market" in the long run. Hopefully you'll remember back in our initial GRD class our discussions about 'disruptive technologies' and 'lead users.' JRW 17:39, 5 January 2011 (UTC)
7. (Pauli) I think one of the "success indicators" should specify "Customer Acceptance" with perhaps the strategic enabler being "prototyping". I think spelling out "customer acceptance" as a success indicator draws attention to the importance of identifying who the customer is, what they want, and does the product satisfy that want.
- Good! "Build it and they will come" isn't always the best product development strategy. JRW 17:39, 5 January 2011 (UTC)
8. (Richey) I believe the Air Force struggles with implications of customer value. Looking past the basics of whether a customer (the warfighter) received a capability on-time and on-cost, how do we quantify success? What about interoperability?? Are we managing high-tech acquisitions in a manner that creates synergy across the services or among allies? Yes, there is guidance - but is there control? Oftentimes systems are developed within stovepipes, which impacts interoperability.
- Excellent observation and one that all of the armed forces have struggled with. We have ongoing research on the topic of interoperability. JRW 17:39, 5 January 2011 (UTC)
9. (Skiple) HP reported in the early 1990s that "77% of its revenue" came from products that had been introduced in the previous 2 years (page 9); the author also points out the shortening business cycle (fig. 1.2) and the small window of opportunity to market new technology. Do you think HP and other "high tech" companies actually have truly long-range plans (ten years? Fifteen?) or has the whole concept of a long range plan become less useful?
- Yes. The business cycle is definitely contracting (while the defense acquisition cycle is extending). I honestly don't know about long range plans of companies. I suspect they are becoming much more along the lines of "vision" statements and broadly worded objectives. JRW 17:39, 5 January 2011 (UTC)
10. (Ho) Figure 1.2 and the discussion of "high technology" reminds me of disruptive innovations - new innovations in an unstable/niche market. The chapter uses the same examples Christensen's books use (memory/semicoductor industry). Makes you think if authors are talking about the same thing, but name it different just to publish the work.
- Perhaps...but I suspect Christensen has been able to pin it down better. JRW 17:39, 5 January 2011 (UTC)
11. (Pauli) I like the impact on U.S. economy section that provides examples of how technology creates more jobs than it eliminates. Often, arguments against technology stem from worry that technology replaces people but in reality, while it may replace functions previously performed by people, it tends to create more jobs because of the new wave of people required to continuously develop the technology, market it, and operate it.
12. (Richey) Perhaps high-tech portfolio management can be considered both a strategic enabler and a success indicator? After all, a portflio is an organization tool. Companies can analyze portfolio projects from resource allocation as well as financial and schedule perspectives. Portfolio health feeds into probabilities of success, which leaders use to make enterprise-wide decisions.
- It all depends on the focus of the organization. If "high-tech" is the bread and butter of the business, then yes, it could be used as both. That way, an organization can always be seeking to keep the portfolio focused on the "high-tech" aspects. JRW 17:39, 5 January 2011 (UTC)
Chapter 2: Product Development Strategies - Redaktor Ho[edit | edit source]
Overview: Chapter 2 discusses the culmination of various types of strategy (business, market and technology) and their "holistic framework" in product development. The business, market and technology strategy form an organization's product strategy which guides business units in various activities. A business strategy identifies an organization's mission and vision through competitive advantages in its operational plan and resources. Market strategy is a plan and method for the organization to enter the marketplace in an advantageous position over its competitors, while creating the most benefit for the customer. Technology strategy guides an organization's efforts to acquire, develop, and apply technology for a competitive advantage. The key to product strategy is to produce a roadmap for the development, commercialization, and life cycle evolution of the product while meeting the needs of the market.
Is it beneficial adding more positions in the chain of command/management?
- (Pauli) The problem with nice sounding ideas suggested by additional positions like "strategic officers" appointed at each level in the organization is that it just adds another stakeholder to the process and grows the bureacracy even more causing the org to be even less agile. Of course, the problem with not having such a role could lead to accomplishment of the wrong things.
- (Skiple) The author states that organizational subunits of large companies often have their own strategies, which may be needed to deal “locally” with problems and issues but are difficult to keep aligned with the overall corporate strategy. But as the number of layers each “doing strategy” increases, it logically becomes more difficult to maintain a consistent strategy across the organization, and there may in fact be a theoretical (or at least practical) limit, and if so, the government will probably reach it before any company does.
- (Ho) Strategic Officers - is this not a role that leadership and positions among the C-suite are supposed to accomplish? It seems that there would be a need for these positions only if leadership has failed to convey their strategy downstream. We're sometimes quick to create a new position, but maybe organizations should improve their communication instead.
- Counter: (Richey) The concept is new to me, but I like the idea of strategy officers. Dynamic business environments demand agility in propagating a current strategy/objectives. Most organizations employ a "cascade" method to disseminate information, including strategy. As such, communication is challenging, and business units can be preoccupied with details of execution. The strategy officer can help maintain proper emphasis.
Thoughts about the customer:
- (Pauli) How much power does the customer(i.e. COCOMM) actually/should have in influencing the AF acquisition/market leadership?
- (Skiple) Customers’ buying behavior can be influenced by a mismatch of the new technology offered and the operating environment, but for a given system, the operating environment will almost always have evolved over time to be optimized for the old technology. With the development of a new weapon system, since a new technology can not exactly replace the existing technology, a change to the operating environment is unavoidable.
- (Richey) On p. 23, in developing a firm's aggregate strategy, the organization should refrain from attempting to be "all things to all customers" in order to achieve sustained leadership. This calls for tough decisions. Trade-offs are important in program management - however, Air Force program managers often find it difficult to say "no" to lead MAJCOM pressures. Perhaps resulting inefficiencies may lead to the enterprise's strategic failure?
Comments about strategy:
- (Pauli) "Strategy must be concise in vision and long in detail", "...should include objectives", and "is particularly important for large orgs." Maybe we are getting better at this. While it may be easy to bash the AF vision statement (i.e. Aim High; Fly, Fight, Win), at least its concise. Moreover, Gen Schwartz outlined the 5 priorites or objectives for us (pg 37 at http://www.defense.gov/pubs/FY_2011_SMP_12302010.pdf) which might need some quantifiables in there, but at least its relatively concise. And as far as our actual Strategy, I think it was over 100 pages last I saw. However, I'm not sure any concise words can be expressed to capture the enterprise strategy we hope for. Maybe the strategy template the author suggests or some other "strategy consultant's" template is a way forward.
- (Ho) I don't agree with the author's framework for a technology strategy (Fig. 2.6). Its completely linear with technology strategy at the tail end. Companies may create new technologies and core competencies which should feedback into either the business objectives or business/market strategy.
- (Skiple) Are there any areas where the Air Force is clearly “driving the market” or where is it being driven by the market? Perhaps cyberspace is one (or will become one) of the areas in which the market is waiting for a force to emerge.
- (Ho) I though the chapter as a whole was very much representative of what a general undergraduate course should cover, but after a year at AFIT I found the chapter to be very cursory. The author cited various authors we've read about (Porter, Drucker, Christensen), but I felt that reading & discussing those individual documents/articles were more helpful...hands down more informative.
Chapter Case study:
- (Richey) The HyTec supplier management case study (p. 29) demonstrates what seem to be possible ethical concerns. I don't believe that government contract negotiators could legally operate by using such tactics. Truth in negotiations are typically backed by historical performance documentation. Additionally, suppliers are usually aware of their competition - so the "alternative source" tactic seems weak.
Chapter 3: Marketing Management - Redaktor Pauli[edit | edit source]
Overview: This chapter discusses marketing as a means of creating value in the eyes of the customer and provides some roles, responsibilities, and ideas to achieve that end.
Comments about Marketing:
- (Skiple) The author makes several analogies to "war" when it comes to markeing. While the case could be made that for DOD the players (Air Force, Army, and Navy product developers) should not waste resources competing against each other, I would maintain that such healthy competition is far more beneficial than the internal costs to conduct this "war" by bringing out better products.
- (Ho) I understand that everyone being involved in marketing is beneficial. I also agree with it, but when you've got a scientist doing pure R&D deep in the weeds do you really want them distracted with marketing? Would that not take away from their research?
- (Richey) The author asserts that marketing must define the "whole product." p. 72 provides examples of whole products, which are analogous to products and services provided by DoD weapon system acquisition programs. However, I believe one significant difference is that DoD customers don't have as much power when they rate "part of the whole product" as inadequate.
- (Richey) Pages 65–67 list effective marketing tasks and questions that teams must answer when developing product marketing plans. Even though USAF acquisition policy and regulation bypasses marketing, as you read through these pages it's clear that we ask some of the questions and perform certain tasks.
Comments regarding the customer and stakeholders:
- (Skiple) Although it is obvious that understanding the customer is paramount, what may be less obvious is the potential danger in the extreme case. A sort of "Stockholm Syndrome" could very well arise where the marketers are so close to the customer that they only see things from their point of view. They could end up sharing the customer's fears, uncertainties and doubts (FUDs), to the point where they may stop offering up solutions that involve anything but the smallest of risk.
- (Ho) I think DoD acquisitions could benefit most from the "Understanding Stakeholder Needs and Priorities" section. Though this is one of many areas the DoD can improve on, if we just got products that satisfied the needs and priorities of the warfighter we'd succeed at acquisitions. To me, this is one of the most important things we do.
- (Skiple) The "value proposition" is widely varied in the DOD, so much more so than in the private sector. There are some stakeholders to whom the F-22 is a bargain at twice the price, but of course others feel differently.
Comments about being a salesman:
- (Pauli) It's a good point to consider the "subjective utility" of products/prototypes and not just focus on the "objective utility" of new products/technology. To the extent possible, designing a product or prototype to have the "shiny penny" appeal may seem fluffy at first, but I think might make a difference in customer acceptance of a new idea or product. Ex: iPods vs. other mp3 players.
- (Pauli) The importance of networking seems to be important to marketing and developing new products. Although the word networking isn't used, there's a lot of discussion on building and maintaining relationships with customers, suppliers, and other people. "No news is bad news!" is an interesting heuristic. Maybe that's one reason why networking was one of the things that separated "star performers" from novices at Bell labs.
- (Pauli) If anyone exhibited all or even most of the characteristics of a salesman on pg 108–109, not only would you be a good salesman, but you'd probably be a hero in any role of any job. This highlights the fact that everyone can benefit from studying sales and seeing themselves as a salesman.
- (Ho) I find the "The Sales Person" section funny. To me, sales people have a negative connotation (think used car salesman). The marketplace nowadays is filled with educated buyers that know what they want, and the salesperson is sometimes viewed as the middleman that gets in the way of the purchase.
Comment about S-Curves:
- (Richey) Looking at the S-Curve Model (p. 83), one can make connections to technology and product roadmaps. Strategically speaking, you can also make the case where DoD might adopt this structure to enhance the FYDP by applying a "capabilty roadmap" overlay. The USAF's history of sustainment, rather than recapitalization, resembles a series of stairsteps between each S-Curve.
Chapter 4: Product Platform and Knowledge Integration - Redaktor Skiple[edit | edit source]
Overview: This chapter discusses the nature of products and processes, and how these concepts play into decisions made during product development.
Comments regarding the use of knowledge
- (Skiple) The topic of a "time value of knowledge" (similar to the "time value of money" from ECON 101) was short-changed. There are plenty of reasons why a short time-to-market is not plausible, but I do agree that it is important to be aggressive when you do have "knowledge" that is of value. It is usually only a matter of time before the next guy figures it out.
- (Richey) Knowledge generation, retention, reuse, and integration is a BIG DEAL in acquisition. Perhaps the greatest challenge facing AF acquisition is the loss of corporate knowledge as our civilian force reaches retirement eligibility. Our community falls into the "large company" category, and the knowledge management infrastructure is questionable at best. Tacit knowledge remains untapped, existing CoPs are underutilized, and policy is weak. If we could fix this, it would be a significant strategic advantage.
Comments regarding modularity
- (Ho) Sections on platform and modularity were particularly important in today's global marketplace. No one wants a one-off platform, we want the ability to "extend" platforms and upgrade, which creates more value to the customer. I think modularity contributes to today's use of "System of Systems" for product development.
- (Ho) I couldn't help to think "systems engineering" the entire time I was reading this chapter. Much of what is discussed, if not all, is touched by SE.
- (Pauli) When it comes to acquisition, do we go for “home runs” such that we try to include “everything” in the baseline? Would a series of “singles” or “doubles” work better where we aim for faster to market, high modularity products that can be improved upon as technology develops and is built into future replacement modules? Or is our acquisition process primarily designed to produce “home runs”?
Comments on Defense acquisition
- (Richey) A quick look at metrics on p. 137 reveals some similarity with DoD acquisition. COO, product quality, and project management in particular.
- (Richey) Perhaps section 4.12, Baseline and Derivative Products, describes spiral development?
- (Pauli) The book mentions the importance of flexibility especially in light of unproven technology. How do we do this in the AF? Do we engage in “cheap” prototyping or do we tend to prefer the full mock up, expensive “ford style” approach to developing products?
- (Pauli) I liked the emphasis on devising metrics as a means of satisfying not just internal but external stakeholders as well. Since things that get measured get attention, some of the potential metrics listed on p. 137 as they relate to various categories (internal + external) may be helpful to reference.
Comments on Communication
- (Skiple) A "virtual integrated worldwide cross-functional team" is a great idea, but rare in the DoD. With all the resources available, this should be where the government shines. Is it possible that the government is simply too large to be able to communicate, even in this day and age?
- (Richey) The chapter emphasizes product development team flexibility in handling high degrees of uncertainty in technologies and market requirements. Does the Air Force do this well, or does "the system" prevent success? Overruns and failed programs may provide key lessons learned. However, new policy may not be the answer to every problem.
- (Skiple) I'm not sure I agree with the statement that "usually the scope of a type C development project is large" (type C being disruptive/discontinuous technology). Yes, high potential for growth but also a high risk of failure (I've seen figures in the 5~10% range discussed in other writings as the average "hit" rate for research). Maybe for a start-up who has all their eggs in one basket this would relatively large, but overall I would imagine that a typical firm would be more inclined to invest where the odds are better.
- (Ho) Overall, I wasn't too impressed with this chapter. Nothing really stood out and made me think "oh, that’s interesting." We've covered, I believe, everything in this chapter at one point in time during the program.
Chapter 5: The Product Development Process - Redaktor Richey[edit | edit source]
Overview: This chapter covers the product development process, beginning with a look at several organizations' preferred frameworks. The author subsequently presents a process for "new" product development and commercialization, with a description of its phases - including various applications. Cross-functional tasks and responsibilities are also addressed, before the presentation of software development and configuration/change management processes.
Comments on PD framework/process:
- (Ho) Scaling based on the dynamic nature of PD is an interesting section. Should all DoD acquisitions go through the same cookie cutter process outlined in the DAG? Do we currently have a process in place to scale back the "PDCP?"
- (Pauli) It seems like the final stage (Figure 5.2 pg 161) for several product development(PD) frameworks is a "lessons learned", "knowledge management", or "Capture" if you are Tidd and Bessant. It's a common theme in PD yet it seems like it takes a back seat similar to continuity binders. Perhaps a natural fit to do this would be something like a "wiki-acquisitions" interface such that it's as simple and user friendly as this wikiversity page for our class (maybe CAC card enabled)and perhaps tailorable to at least each "wing" or directorate if it's not feasible on a larger scale. Do we have a database that attempts this currently and does it work like it's supposed to?
- (Skiple) The book notes that in general the software design process is lagging in maturity, but it is difficult to imagine how this can be. Time-to-market is inherently short (extremely so, in fact), there is little lead time in manufacturing, and distribution channels are far more efficient than for any "physical" product. And yet, most software development seems almost totally isolated from the user.
- (Richey) Figure 5.4 (p. 162) depicts the accelerated PD process at Applied Materials Co. What potential issues come with implementing a “customer release” at the beginning of the beta phase? Does similar risk-taking exist in Air Force acquisition??
- (Skiple) The Carnegie Mellon University Software Engineering Institute model defining "maturity levels" in capability could easily be applied to any product development process. Likewise, Hewlett Packard's "FURBS" (functionality, usability, reliability, performance and supportability) also make just as much sense from a hardware development perspective as software.
Comments on $$:
- (Richey) The case study on pp. 176–177 is an example of a large company wasting R&D resources on developments that outdistance market readiness for the product. Rafinejad implies that small companies are more judicial with their R&D portfolios, so they can avoid bankruptcy. Perhaps the Air Force is not very prudent in its pursuit of R&D projects? Is our technology at risk of being surpassed by a “start-up”?
- (Skiple) The need for “concurrent” development, which includes manufacturing technology and processes, implies that the developer is incentivized to lower costs. The emphasis with military procurement tends to be the technology itself, with the concurrent things outsourced to the prime. As long as cost targets are met, there is no reason to question whether or not the prime is being diligent enough. The DOD is both the “venture capitalist” and the customer with a real stake in manufacturability, but we give away most of the leverage we have.
Comments on flexibility:
- (Ho) Our acquisitions process is a stage gate process - linear and sequential. However, in order to be flexible the author illustrates in fig. 5.9 an overlapping and iterative process. Is there anyway to overcome the stage gate process in DoD acquisitions to be more flexible?
- (Pauli) Section 5.6 is entirely dedicated to discussing the importance of "Flexibility" given a high tech environment and high difficulty in forecasting future conditions. I'd guess that most would agree it sounds great and is nice to have but it seems like a bureaucracy like ours is designed to be inflexible because a bureaucracy is inherently risk adverse (i.e. a rule for everything that can go wrong, and if something slips between the cracks, it becomes a rule in the next edition of the instruction, thus increasing inflexibility).
- (Richey) Rafinejad considers program managers as the linchpin of product development. This chapter asserts that PMs are key to product commercialization success (p. 167), PD knowledge integration (p. 170), and PD flexibility (p. 172). Looking at Table 5.1 (pp. 188–191), it is easy to imagine the immense challenge of leading PD. Considering these tasks and responsibilities, do you feel that the application of IPTs enhances or hinders the chances of PD success?
Comments on configuration management:
- (Ho) I think change/configuration management is a major problem in DoD acquisitions..more specifically prior to product deployment. We see requirements creep in many programs which leads drastic impacts on cost, schedule, and performance of the final product. It would be interesting to see what percentage requirements creep has on the DoD acquisition projects busting the cost, schedule, and performance objectives.
- (Pauli) How does Configuration and Change Management work in AF? Is it someone's entire job? Why is it seen as a "necessary evil" as mentioned on pg. 207?
Chapter 6: Excellence in Design and Product Reliability - Redaktor Ho[edit | edit source]
Overview: Discussed various aspects of product design as well as the deliverables and things we can do to excel in PDCP. Some focus on reliability modeling and various failure modes in PDCP.
The customer and stakeholders in PDCP:
- (Pauli) P. 214 lists many heuristics including the importance of understanding an engineering specification's “purpose” and “the why.” I would add that any justification should talk about what it does for the customer and the associating assumptions behind each specification. If you can't do that, then question why it's there.
- (Richey) As mentioned in Monday's class, it is interesting that the authors take such little time to discuss PDCP quality - even while admitting the 100-fold increase in costs to fix problems found by the end user. On the F-22 program, the lead command embedded maintainers to perform in-process inspections on the production assembly line. Beyond that, I cannot say what the end user's level of involvement was during initial development. However, I will attest that DT and OT&E professionals at Edwards & Nellis were engaged on the development of increments 3.1 and 3.2.
- (Skiple) Regarding customer participation in NPD (6.20), Rafinejad only briefly mentions the need to safeguard intellectual property (IP) as a “concern.” I would suspect that this can be a very difficult balancing act for a firm, and one that would naturally need to involve some fairly high levels of management.
Maintainability, reliability, failures, and safety:
- (Pauli) Many of the “design for _____” can be incentivized from within (manufacturability, safety, environmental, reliability, predictability, robustness) but it seems that when it comes to “maintainability”, it is harder to do well. When the CV-22 came online in 2006, the technical orders were notorious for being extremely vague, incomplete, and relied on the user to debug rather than the manufacturer. Additionally, certain components, like the engines, were “power by the hour” and required sending them back to Rolls Royce for overhaul (no intermediate level maintenance capability which makes you wonder how cost effective that can be for sustainment). On the other hand, how do you incentivize maintainability? Poka-yoke?
- (Richey) F-22 program sustaining engineering employs FMECA to analyze DRs and T&E findings. In fact, maintainers can access much of the necessary data through the integrated maintenance information system, which performs a fault tree analysis to identify all sub-systems that were involved in a failure. Engineering finds this "smart product" information very useful toward completing FMECAs, which are key inputs to the reliability and maintainability maturation program.
- (Richey) According to Rafinejad, the "best" design for safety practice is to apply the most stringent standards. Can there be a more significant driver of moral dilemmas regarding business ethics? Look at even the "simpler" products on the market...when can we expect to see the next car seat or high chair recall?? Consider potential constraints on even the most aggressive customer safety risk mitigation strategies. F-22 example.
- (Skiple) Rafinejad discusses how taking a corrective action can be undertaken too hastily (section 6.3.2) and possibly leading to “fixing the wrong problem.” This is arguably one area where the federal government does an admirable job. The NTSB is a great example; the focus in every accident investigation is the root cause, which is more often than not a succession of errors.
- (Ho) A corrective action plan can be applied to more than just engineering and manufacturing issues. Author mentions sometimes we "fix" the wrong problems and don't address the root cause. We kind of do the same thing when it comes to management and firefighting. Sometimes we need to take a step back to look at the whole system as compared to just a part of the system.
- (Pauli) Prototyping is noted to be important to “integration of components, verify designs, identify shortcomings, and characterizing performance.” However, I would add that it seems very important for marketing as well, and would be a great way to market basic research findings to the boss through the use of “bench prototypes.”
- (Ho) The three types of prototyping seem to be very similar. It would seem rather difficult at times to differentiate between the three and when specifically should we use one or the other.
- (Skiple) Section 6.17 (Product Documentation) acknowledges that “engineers usually do not like to complete the product documents,” and offers as a best practice to do it “as you go.” I see a paradox in that the more complex the system (and the more cutting edge the technology) the more important the documentation becomes but the less likely engineers will have the time to document it, or possess enough knowledge to even document.
- (Ho) For product development textbook, I find section 6.16 on design stages lacking and should warrant a separate chapter. In my engineering programs (both undergrad and grad), the design process was an entirely separate course, normally done through a capstone design course.
Chapter 7: Flawless Execution and Global Resources Management - Redaktor Pauli[edit | edit source]
Summary: No matter how much thought has been given to a firm's strategy, processes, marketing plan, technology, success will not materialize without excellence in execution.
Excellence in Execution:
- (Ho) Figure 7.1 says concept limitations were the leading cause of failure. The explanation the author has says this is rooted in project execution. Not sure how the two relate, concept limitation to me sounds like a design issue.
- (Pauli) "Common problems in project management often stem from the local optimization of each task." I wonder if part of the reason it may be so hard to get things done as a program manager is that everybody is an expert in their domain...maybe too much of an expert. Expert bias can then take over such that they feel like they know every rule so well and eventually develop a strong sense that their interpretation is the right one and that outsider's (non-experts) interpretations get written off as inaccurate or wrong. Thus, maybe the problem is when we tell everyone to do their best...they do...except we don't realize we are encouraging them to suboptimize at the expense of project objectives.
- (Pauli) For those with acquisition experience, is there "one thing" that seems to separate those that have "excellent execution" vs. those that can't seem to do that? Assuming everyone is trying their best to do a good job, is "excellent execution" a random event? Is the "one more thing" factor according to Lt Col Dan Ward the random occurrence that determines if "excellent execution" happens or not?
- (Skiple) The author says that “project management is about managing interfaces between people, tasks, and product subsystems.” Sounds like systems engineering could be used as a metaphor for project management, in that a project's objectives and stakeholders are in effect a “system of systems.” Just like in complex hardware or software, it is important to ensure other people's expectations match what you are providing them.
Product development processes:
- (Richey) This chapter touches on concurrent development. In DoD acquisition, we do not focus on the submerged iceberg that is concurrent development between prime and subcontractors. However, we are somewhat familiar with concurrent development of subsystems as end units. This chapter also covers the importance of interface control, which is vital to concurrent development of subsystems. Concurrent development is a tremendous enabler if done correctly, but a significant risk when not properly handled.
- (Ho) The author says the management of a product development process must have "teeth" to ensure the process is followed. Sometimes it seems in DoD acquisitions we don't exercise those "teeth" to ensure the KTR is on CSP.
- (Ho) Section 7.8 talks about the critical chain scheduling by Goldratt, i think this is Theory of Constraints. TOC is just one management technique available to PMs today, which was pushed down at Edwards in the test community. I find it humorous the author says multitasking should be minimized, normally part of being efficient is being able to multitask.
- (Pauli) Regarding the importance of having metrics as part of the success criteria for alpha (functionality, reliability), beta(manufacturing cycle time and labor cost), and gamma phases (costs/gross margin n months after product launch), do we use similar metrics in our acquisition process at the various milestones?
Budget as a sense of urgency
- (Skiple) I found the discussion on large- versus small-companies in the “Managing Constraints” section interesting. Smaller companies are constrained by smaller budgets (and often shorter time to show a positive cash flow) and operate with a sense of urgency. In the DoD we can artificially insert a funding “constraint” on a project in hopes of spurring creativity, but how do you recreate that heightened urgency component?
- (Skiple) It's nice to know that unrealistic schedules are proposed just as frequently outside of the DoD as within it. I especially liked the executive's quote that “it is OK to present and unrealistic schedule to top management because an aggressive schedule is what it takes to win their support.” My experience in the labs is that even with relatively short, straightforward projects, for every project that finishes on time there are three to five that slip, and rarely are any done early.
- (Richey) On p. 240, the author quotes a high-level executive in the high-tech industry as saying "It is OK to present an unrealistic schedule to top management, because an aggressive schedule is what it takes to win support for your plan!" This philosophy may very well be a significant driver of DoD acquisition failure. For example, I've had several experiences with contractors that cannot meet their worst-case schedules. Oftentimes, there were no groundshaking problems leading to the schedule delays. I believe that contractors either use faulty modeling in their proposals, or they outright lie in order to gain contract awards. In the end, it's up to the DoD to validate proposed schedules before award. Perhaps we need to fix the proposal evaluation process in order to "fix acquisitions."
Dual Source Strategy
- (Richey) 7.19.2 covers dual-source strategy. This is an important consideration for legacy Air Force platforms as OEMs move on to other work. One of the problems we face is that prime contractors control too much, and the government has little influence over dual-source strategies. One way to combat this is to purchase large amounts of data in order to qualify sources for system sustainment, but this is very expensive (if it can be done at all). Also, contractors are smart enough to claim proprietary rights on key subsystems and components. In the end, the government can pay (again) for technical data procurement in order to re-compete parts such as T-38 wings.
Chapter 8: Project Management in Product Development - Redaktor Skiple[edit | edit source]
Overview: Project management is where the rubber meets the road in product development. Dr. Rafinejad presents some of the more pertinent methodologies that will lead to a project team and its supporting organization to a successful program.
Comments: Staff Deployment and the Christensen Chart
- (Pauli) (Regarding the Christensen task model, p. 306). The next time the boss asks me to take on another project when I'm on 2 already, I'll be sure to say I'm sorry Sir/Ma'am, but Christensen says if I accept that task, my efficiency is going to drop 30%. Or maybe I'll scan a copy of his chart, post it in my cubicle, and draw a dot to indicate how many projects I'm working so everyone will know what my current efficiency is at any given time and maybe won't ask me to take anything else.
- (Richey) Staff deployment is something that junior and even mid-level Air Force PMs have virtually no experience with. However, CGOs are often assigned to multiple projects. Fig 8.9 (p. 306) is accurate by my experience, where I've been somewhat bored with one project, comfortable with additional membership on another IPT, but downright ineffective with responsibilities on three or more projects.
- (Skiple) The author offers a rule of thumb regarding time not spent in projects of 10%, which is on the order of 200 hours year. That's probably understated in the DoD by a factor of two (or more). I recall some admittedly anecdotal analysis that there is close to 200 hours of just mandatory training in the Air Force. In any case, what would Christensen's chart (Fig 8.9) look like with even that extra 10% of your life back?
- (Pauli) (p. 298-299) Kernzer's formula for estimating task completion and standard deviation is handy and seems like it could serve a variety of applications beyond task estimation (e.g. cost, risk). In a task saturated world, it seems like it would be pretty easy to use on the job. Maybe not ideal, but good enough when the boss wants an estimate of something on Friday at 4:59.
- (Ho) I would caution using Kerzner's equations and building our project Gantt charts from it. Many times it's a civilian who has pulled the numbers off the top of his/her head. We may get too fixated on meeting the chart's schedule when no real science went into the inputs... GIGO
Getting Things Done
- (Pauli) (p. 297) God's lesson to Noah was an interesting parallel to project management and the point is both generalizable and in accordance with goal setting literature which suggests that specific, challenging, and attainable goals spark performance. I would add that it's a bonus if you can involve people in establishing the goal since individuals tend to work more toward the goals that they have helped developed. However, sometimes that luxury isn't an option.
- (Ho) At the lowly CGO level as program managers, we tend to have more influence than we do authority to carry out an action. The best we can do is influence the true decision maker (our commander/rating supervisor). This influence may be our only way to "control" Project team members who are less inclined to take direction from us (the program manager) since they're trying to make their boss happy, not the program manager.
- (Ho) This chapter has a lot of corollaries to systems engineering and the SE perspective. Makes me think if SEs make the best program managers.
- (Skiple) “Most of the actions of project managers take place in project meetings.” But how well-run are these meetings typically? There is a real art to running an effective meeting, but it also requires planning, which takes time that program managers typically don't have.
- (Skiple) The author recommends that management review meetings be held monthly. The roughly equivalent level in AFRL is the Laboratory Management Review (LMR) which is held once a year. Granted, there is no reason a division chief can't request reviews at a greater frequency, but realistically that rarely happens unless a project is in trouble already. Obviously there should be room for tailoring, but there is a lot to be said for a regular “battle rhythm” in meetings.
- (Richey) This chapter begins with a very interesting perspective that "effective program mgmt is not only dependent on the PM, team members, & methods, but the organization as a whole (structure, processes & culture). Do you think any high-tech company follows guidance like DoD 5000? They wouldn't compete if they did. However, the USAF continues to stay (slightly) ahead of the competition.
- (Richey) On p. 300, the author states that it is reasonable for a total cost uncertainty of up to 30% in the early stages of PDCP. This would not fly by Nunn-McCurdy Provision standards (25% is a kill). This chapter also states that significant investments in unsuccessful products is not uncommon in large firms. I think that's probably also true of DoD acquisition, in the context of underperforming systems. Will it come down to making sure that our underperforming systems are less underperforming than our adversary?
Chapter 9: Best Practices for Product Development Managers - Redactor TBD[edit | edit source]
Summary: The author presents some straightforward tools and guidelines to help managers (and others) understand the implications of their business decisions and direction.
- (Richey) Fig 9.1 recommends a flow for developing and adopting a new business process. Selecting tools is the fourth step. In the F-22 program, we had a database for managing everything from technical documentation and supply status to site activation reports. Clearly, some genius "made" their appraisal by requiring business processes to be built around the database. However, other tools (such as AFKN) were oftentimes more appropriate - but we were "required" to use the LM-managed database.
- (Skiple) This chapter is one of the few times I’ve heard “bureaucracy” referred to as anything but universally evil – and possibly the first time I’ve seen it in print. We’re quick to label any ineffective process as bureaucratic, and yet as the author puts it, a bureaucracy keeps us from repeating mistakes.
- (Ho) Figure 9.2 lists three decision types. I'm sure there's more than three out there. What about decisions that are indifferent? Only a right decision can result in a good outcome. Aren’t there times when we make wrong decisions that may result in a good outcome?
- (Ho) Glad to see more than one paragraph go towards risk management. In the five steps listed, I notice the exclusion of "implementing the mitigation plan." I see we develop one and then we're tracking it. I assume it's implied to implement the plan.
- (Skiple) I liked the statement, “calculated risk-taking must be rules based.” And yet, how often do those rules get trumped by gut feel (or, more correctly, the boss's gut feel)? In the labs I have this might be more the case than a program office, since the stakes are smaller.
- (Richey) Sec 9.6.2 touches on the "5 whys" method of root cause identification. This is a particularly useful method for solving tangible problems, such as those encountered in aircraft maintenance. My IPT was very good at asking the 5 whys to help solve depot flow problems (even if the contractor would sometimes lose patience!). I'm not sure the 5 whys would be a preferred method for developing or testing immature technology, where relationships are ill-defined.
- (Pauli) The "after-sales" support is a major determinant of customer loyalty and trust. I'd agree with that. It's expected that sales people will put on their best face before a transaction...but once you get sucked in...it's the companies that have a support structure that earn loyalty. Although I didn't work with them, having the CV-22 engineers on site was essential to work through issues.
Meetings Bloody Meetings
- (Ho) I'm a little surprised 5 pages was dedicated towards running a meeting and presentations. Granted a lot of time is probably wasted in DoD by ineffective meetings. I remember at my first assignment we had 1 hour meetings everyday in which the division was briefed on things that did not apply to them.
- (Pauli) I do not miss the 0600 morning meeting, to prepare for the 0700 morning meeting, to prepare for the 0800 morning meeting...which sometimes had meetings after the meetings...and oh yeah, the 1600 meeting—to recap. Daily! However, why did it seem like these meetings often took on a 5th category of meeting: the "who will we throw a spear at today" meeting.
- (Skiple) My personal belief is that if we could get meetings right, we could solve world hunger. In my (yes, long) career I have been in maybe a half-dozen truly well-run meetings. What makes the difference between a good meeting and “that’s two hours of my life I’ll never get back” is definitely an art, no doubt about it. The first step, though, is asking the question, “do we really need a meeting?”
- (Richey) Problem-solving meetings (in sec 9.9) are akin to working groups (which are very common in AF acq). My experience with working groups is that they often include elements of management-decision meetings (and vice-versa). Therefore, I believe the author provides a decent description of meeting categories - except for the "people relationship meeting." Mention of such meetings is a tad out of place in this book.
- (Skiple) We’ve all had the “presentation skills” classes, and the unsolicited pointers from the bosses. Sometimes briefings need to be entertaining, but other times they just need to be “brief.” Colin Powell used to say that in most briefings he would “tell you what I know, tell you what I don’t know, and tell you what I think.”
- (Pauli) Effective presentations are the key to many of the best practices and jobs in general. Impromptu, formal, casual-->the human interface determines the effectiveness of the most advanced PD tools/techniques and people. Maybe we should all get a minor in acting, Toastmasters, comm, etc. What is more important to PD...soft or hard sciences? It's like asking if Mom or Dad is more important to create a family.
[edit | edit source]
Summary: Tools are no substitute for judgment. There is no “right” answer when it comes to managing a portfolio. You will never get it perfect, but there are guidelines on how to deal with the tension within an organization as the various parts compete for scarce resources.
- (Pauli) "Establishing metrics for strategy and tactical objectives is essential." Key starting point is strategy. Do people even know what the strategy is (transparent)? Is the strategy clearly linked to the customer? This was clearly missing in the case study.
- (Ho) The three objectives to PPP really show how important a clear, concise strategy statement can be for an organization....no matter how big. I look back to LtCol Fass's class, and really find value in a lot of what we had to read on strategy.
- (Richey) The case study (p. 348) demonstrates several key issues detrimental to portfolio success. The BU GMs reveal some infighting, the CEO and corporate strategy VPs seem somewhat hesitant to act. The strategy and RD objectives do not appear integrated, which leads to further indecision and wasted resources. As the case demonstrates, this drives R&D portfolio inefficiencies on a global scale.
Portfolio Assessment and Decision Making
- (Skiple) Section 10.6 recommends against attempting a portfolio analysis (the fourth step in the framework in fig. 10.3) unless you are willing to take action based on that analysis. It also says to keep it simple and concise, then goes on to show 9 or 10 bubble charts that slice and dice the data. Seems to me that such a wide range of analysis could support just about any position; in other words, it's possible to just do the analysis until the answer comes up that you wanted to hear.
- (Skiple) One of the three key questions that PPP is supposed to answer is: "are there adequate resources to execute the planned product projects? (Sections 10.3 and 10.5.4) My experience in the labs is that very often this question is rarely addressed properly, if at all. There is ample room to "reverse engineer" the portfolio to match the resources (not just funding and overall headcount but specialized facilities and skill sets).
- (Pauli) Regarding equity, it seems problematic to assess your strategy based on stockholder returns. Who are the customers—stockholders? [Return on Equity - how many dollars of income are produced for each dollar invested by common stockholders.]
- (Pauli) It's one thing to implement a rigorous methodology for portfolio management, but it seems that public sector lacks the control that private sector has over the portfolio. Even if we wanted to shut down a dog, the Chief of Staff (or MAJCOM CC) can't necessarily just say "make it happen." As we've stated in class before, programs that enter the acquisition funnel tend to continue (unless you have a Robert Gates who stays on board for consecutive administrations). However, on a more optimistic note, a rigorous methodology can make the "best" decisions more obvious.
- (Richey) The strategic alignment analysis template (Fig 10.5) looks like a handy tool. I can see this as a means to analyze aggregate capabilities across acquisition portfolios (e.g. fighter-bomber). It would be nice to gain some insight on how the strategic value is calculated. Perhaps strategic objectives and alignment codes have assigned values.
- (Ho) We've spent a lot of time in GRD talking about the importance of portfolio management. Besides the 2x2 matrix of cash cows, pearls, white elephants, etc....I'm a little worried if/when I'm thrown into the "real (Acquisitions) air force" and have to manage my own portfolio. I feel like we've grazed over a lot of techniques, but Im not sure if I can successfully implement one.
- (Richey) Fig 10.15, Product Line Differentiation, could be used to categorize weapon system's war-fighting efficacy. Fig 10.24 is a nice example of our risk management decision tree.
Intellectual Property (IP)
- (Ho) You would think safeguarding information would be one of the things the government does best, but with the rise in wikileaks incidents it really makes you second guess. It seems like the biggest enemy we need to worry about is ourselves.
- (Skiple) The government has a strange role when it comes to intellectual property (IP). The way I understand is that the government does not buy IP, just the rights to use it as it. That makes virtually every contract a Joint Development Project (JDP). The author recommends marking everything "confidential" in order to raise people's sensitivities but what that will also do is slow down the information flow (similar to what happens when a project becomes "SECRET," albeit not to the same degree).
Case Studies[edit | edit source]
Ch 7 Case Studies[edit | edit source]
Case Study 1[edit | edit source]
Product: Copper thickness mapping (CTM) instrument for process control in IC manufacturing
My Initial Concerns:
- After CTM concept originated in the central research department, the company outsourced phase 3 development to small local supplier
- CTM being developed for incorporation with products of two internal BUs. (short timeline ~ 6 mos).
- Aggressive Beta test goal of ~ one year.
By July 2002, CTM had not passed BU qualification testing. Prototypes were out of spec and modifications had not addressed the problem. Root cause was supplier incompetence, yet it went unnoticed for two years. Meanwhile, CTM ownership had been transferred several times throughout this period, leading to loss of oversight and expertise. By 2002, market conditions had changed, the effort failed, and another division had already developed a real-time copper thickness measuring instrument. Company president called a meeting to determine whether the CTM project should be cancelled.
The decision to terminate should be a no-brainer. Looking back, this is an excellent opportunity for the company to use lessons learned and adjust their internal business strategy. Project management is already difficult enough when teams remain stable. However, project success is threatened even moreso when ownership transfers among business units. Recognizing that the development would be outsourced, the company should have left the original internal project team in-place. Another reason for this is the fact that the CTM was already pressured by the requirements of two BUs.
Another consideration is the aggressive schedule, which reflects the importance of time-to-market for this product. How did leadership lose sight of the fact that the product was undergoing a two-year slip? I can see where changing the project ownership delayed the realization that the supplier was incompetent. However, the source selection process should have provided some indication of that possibility. Worse yet, the company had a comparable solution coming out of another division and they didn't terminate this project sooner. The company needs tighter controls.
Case Study 2[edit | edit source]
Product: A dielectric etch product to serve all market segments
My Initial Concerns:
- Did the GM set an unreasonable market goal for FinEtch? (he was aware of tough technology requirements)
- Product development team given broad direction.
- Spent few years in concept design, finding the “etch all” to be impractical
Company president hired consultant to do root cause analysis of what happened. Generally, the conclusion was that marketing lacked clarity and focus, while the development process was flawed. Specifically, project priority and direction had changed several times, project lacked an empowered, accountable PM, infighting among engineering and marketing managers, insertion of high-level “helpers” created a political environment that demoralized project members. In the end, the company has lost market share so they decided to pursue high and low end (large market) technology.
It seems as though political pressure was overwhelming from the start. Perhaps marketing and development engineering managers should have approached leadership sooner with the fact that this concept was not feasible. Maybe they did not do this because of the changes in priority and direction, in hope that the idea would “just go away.” Regardless, high-profile projects require high-profile leadership, and the company also appeared to fail by not assigning the best PM to the effort. A strong PM may have been able to keep marketing and engineering focused on problem solving. Lastly, if not handled properly, any project is at risk of failure once it is politicized. In the end, the GM should have requested a feasibility study for this concept before sending out the troops.
Case Study 3[edit | edit source]
Factory Interface Product Development
Setting: SPEq Corp is under pressure in 1998 to accelerate developed of 300mm wafer fab product line. A key subsystem was the factory interface module (FIn) composed of two robots and two door openers. Systems and Automation Organization (SAO) was the internal supplier within SPEq with the engineering resources to develop FIn in order to ship to Intel. After a year of design and deliberation among engineering gurus and executives, an "optimal" design was selected and named FIn.0. Two firms were chosen as suppliers, E-Corp (robots) and Auto-Fab (door openers). Alpha exit was successfully completed on schedule in Dec 1999.
- Disastrous field performance: MTBF was 10 hrs vs the 300 hrs reported in alpha exit - validity of alpha exit test methods in question.
- Manufacturing cost was $200K vs the MRS target of $100K.
- FIn PM responsibilities transferred to a new division in the company (Fab-Pro) in charge of expanding company's SAM - old-timer, influential, self-proclaimed "best-product manager" VP was in charge
- VP directed complete re-engineer of the FIn.0 - named FIn.1 (radically different due to leagacy sys and the VP wanted a new design)
- Formed a corporate-wide committee of top engineers to ensure new system was sound and company was committed to design - proved to be ineffective
Recommended COA: From the case study, it sounds like the VP failed to communicate with Intel and focus on the reliability of FIn.0. She assumed the problem was inherent in the design rather than possible supplier issues. Alpha test models were probably hand-built, one-off designs which probably did produce the reported MTBFs. However, there could be issues with production and the interface of the robot and openers. Instead of pouring all my resources into a complete re-design, I would have an engineering team examine E-Corp and Auto-Fab's production facilities and re-evaluate the currently design. A lot is left out to truly pinpoint the exact deficiency in the process. SPEq seems to have failed in managing the critical interfaces. Things were constantly rushed through out the PDCP and SPEq failed "doing things right the first time." SAO should have been responsible for the issues, so shouldn't they be the ones to fix the issues instead of a new division?
Case Study 4[edit | edit source]
Tool Buffer Storage Product Marketing - A Strategy in Disarray
Setting: Continuation of Case Study 3. The Fab-Pro division has developed a new Tool Buffer Storage (TBS) product that allows customers to store wafers locally instead of at a central fab location. This would radically change the traditional approach and reduce manufacturing time by as much as 30%. The benefit of TBS lied not in the revenue, but the competitive advantage through the integration of current process equipment at SPEq.
- President gave OK to market directly to customer independent of the BUs- ran risk of TBS integration with competitor equipment.
- BUs did not have complete buy-in of TBS integration with their equipment.
- BU gross profit margins were impacted
- BUs were responsible for delivery and reliability issues of the TBS since customers believed the TBS was an integral part of the process equipment.
Recommended COA: It seems obvious to me that the issue here was giving the VP of Fab-Pro the OK to independently market the TBS. Competitive advantages belong to the entire corporation and should be utilized by the entire corporation. These competitive advantages should be integrated into a company's strategy throughout the business units. In this case, TBS was not integrated properly with the BUs' process equipment prior to marketing resulting in issues down the road. Had the company taken time to gain BU buy-in and tested the TBS integration with BU process equipment prior to launch, many of the early headaches could have been avoided.
Case Study 5[edit | edit source]
Setting: A tale of Schedule vs. Quality. Discusses product development issues of the TBS caused by accelerating the time to market of a new product by bypassing the beta customer qualification process and introducing the product straight into the pilot production of customers prematurely.
- Before the project, Fab-Pro VP committed to unrealistic "critical to success" goals with the customer at project initiation
- Despite the development process unsuccessfully exiting the alpha phase and containing numerous engineering issues, the product shipped to the first customer.
- Beta qualification was skipped.
- Engineering was essentially told to put on their management hat (i.e. Engineering program manager set up "soft date" to finish alpha phase to appease marketing. Soft date treated as written in stone.)
- Bypassed company policy by invoking a technicality in the shipping/install time in the fab which was supposed to allow enough time for alpha testing to finish.
- Bypassed company's multiple bid policy and went with supplier that Fab VP knew of in order to speed up the process.
- Reliability, safety, software problems emerged causing lots of expensive rework.
- Nearly met an overly aggressive schedule demanded by the market but due to poor quality, the product did not meet customer expectations and thus, failed to gain traction past the few few customers. The TBS product line was discontinued.
- What would I have done? The ideal answer is I would have negotiated a more realistic deadline with the customer to ensure I could build quality into my design (i.e. finish alpha/beta testing) and hope that delayed market timing won't be a mortal wound. However, in this case, the key point is that quality HAS to be there, otherwise I lose the customer. Thus, if my competitor simply could not accept a delay and had to have quality, then I would chose option c, the "Don't do it" option. This product development team gambled and lost. If quality would have "happened to work out" given this schedule, then this could have been a success story. If they focused on quality but delayed the product to market, they may have forfeited the first mover advantage to a competitor and perhaps lost the ability to even try and compete at all. If failure were not an option such that an unsuccessful outcome meant I could not not absorb this lost opportunity, then I would be inclined to roll the dice as well (i.e. get rich or die trying). However, if this endeavor only meant additional profits to my company, it would not be worth the risk to accept the customer's aggressive timeline if I could not negotiate a more mutually acceptable alternative.
Case Study 6[edit | edit source]
Setting: Zylex developed the highly successful FutureEtch in 1987. However, during the technology boom of the 1990s, competitors launched better rivalry prodcuts which spurred Zylex to come up with solutions that eventually got them tied up with NeXT as an attempt to revitalize their product line.
- Competitor products had superior appearance
- So impressed with NexT's GUI capability, Zylex failed to address the suitability of NeXT OS for their real time application needs.
- Overpromised and underdelivered the Enable to customers...Enable supposed to be "a lot better." It was not at all (i.e. reliability and GUI). Sales did not want to sell Enable.
- NeXT meant a lot more to Zylex then Zylex meant to NeXT. Zylex was just a small customer.
- NeXT retained rights to software code during contract negotiations and thus prevented Zylex from being able to troubleshoot and fix issues.
- NeXT was showing signs of not being committed to fix its OS shortcomings and threatened discontinuing its OS product altogether.
- The future of Zylex—Enable and Symphony—were tied up with the ambivalent NeXT.
Impact: Zylex's mismanagement of technology led to a slow process of patching old problems and cleaning up a mess that costed them a lot of money they could have used on newer innovations.
- As exciting an opportunity to work with the "cool" Steve Jobs technology that was supposed to save the Enable and Symphony products, I hope I would have had the foresight to scrutinize this partnership more wisely. Hopefully, I would have realized the danger of relying too much on a supplier for my success such that if I am just a small customer, then I lost my leverage. Additionally, I hoped I would have ensured that I have negotiated the right to fix problems that were associated with my brand name. This was perhaps the biggest problem for Zylex.
Case Study 7[edit | edit source]
Setting: A late entrant to the chemical mechanical planarization (CMP) industry, NATL Corporation was viewed as a rising star in the due to their product's versatility and reliability. To forge a massive deal with a major European customer, negotiations were conducted by NATL executives and the regional sales force.
- The deal was sealed “with minimal consultation with the CMP business unit.”
- The NATL sales force recognized that the CMP tool was “relatively immature.”
- Still a newcomer to the industry, NATL pushed for a vertically integrated solution to the customer's need.
- Customer had yet to sign off on final payment for tools purchased and received at the time the new order was placed.
Not a lot of good came from this deal for either NATL or the customer. While not explicitly stated, the company's reputation took a serious hit when it couldn't meet the customer's expectations. Also, by packaging the CMP tools with other NATL products into a single, huge contract, the company's cash flow was affected when the customer refused to make final payment on tools purchased under an earlier contract.
Recommended COA: .
Obviously, the salesmen were overzealous, and managed to get their top leadership excited at the prospect of a huge sale. As a relative newcomer to the CMP game, NATL should have been more aware of the risks to their reputation, which was good but not based on a lot of experience. This would go beyond cross-checking specifications between required and demonstrated performance.
Case Study 8[edit | edit source]
Our friends at the NATL Corporation shipped ten thin film deposition systems to a large 300-mm wafer laboratory, and shortly after production ramp-up all ten of these highly complex systems failed at 50 times the rate the customer specified. Moreover, the increased failures occurred in the same component, the heating chamber.
NATL executives pulled together the A-team to solve the puzzle. Through a methodical approach, the team collected data and in a matter of three weeks identified a number of probable causes each contributing to the failures.
Systems Engineering Issues:
- From a design perspective, they discovered that two machine specifications (the transfer plane of the robot arm moving the wafers through the system, and the wafer lifter travel inside the heating chamber) were out of spec.;
- From an operator perspective, they found that the customer had dialed up the speed beyond what was recommended by NATL (but, presumably, still within operating limits);
- From an application perspective, they found that the customers was basically reusing wafers, which likely were on the edge of, if not completely out of, tolerance.
In short, the machines were whizzing out-of-spec wafers from one misaligned component to the next.
Other parts of the process were also found to be contributing to the increased breakage rates; it was implied that these were within tolerance but that the variability would have had much less impact had the other alignment and operating issues not been present.
I am going to assume – based on case study seven, and given the timeframe (2001) – that NATL was also an emerging player in this part of the industry as well. NATL demonstrated in that case study a willingness to package together pieces and parts without consulting the affected business units. Even validating specifications, however, would probably not have completely warded off these problems.
As stated in the previous chapter on excellence in design, “a robust system is tolerant of inaccuracy and imprecision in the input variables.” The inputs were all (I assume) within acceptable limits, and each component system was also produced within specification, but had a bias in one direction or another that effectively changed the initial conditions for the next component.
The author makes an excellent case for systems engineering rigor throughout product development, but it may be a little unfair to blame NATL for not foreseeing the customer using recycled wafers and higher speeds. A full-up beta prototype tested under “battlefield” conditions, while a good idea (assuming that this was NATL's first attempt at this particular market segment) would likely not have resulted in the same failure rates unless the customer was up-front about the assumed operating conditions.
Still, I would hope that NATL has since adopted systems analysis as a design philosophy, not just for post-mortems.
Homework Assignments[edit | edit source]
Homework 1 - Tools to Invent a Whole Product[edit | edit source]
Ho[edit | edit source]
The Kano Model
- Great way to see where we are with certain parameters of a project. Illustrates if we meeting our objective and within threshold. I'm not aware if the USAF actually uses this model.
The Conjoint Analysis Method
- This method seems to be driven by the numbers. It sounds great and would make a great powerpoint slide to brief leadership, however, it seems extremely resource-intensive requiring surveys of small sample groups. What happens when those small sample groups don't represent the population correctly?
The Product Value Matrix
- This seems like it would be a great systems engineering tool. Gives a cursory view of stakeholders and needs/priorities. There's a lot of information displayed in a neat, organized illustration to help guide decision making.
The Lead User Method
- A very customer-centric model. Assuming communication between manufacturers and lead users is perfect, the lead user method seems it could potentially address all customer needs. We talked about being too close to the customer, potentially an issue especially with this method. I can see this method being used somewhat in DoD acquisitions.
Quality Function Deployment (QFD)
- QFD is a very logical flow of actions that take place in order to fill a requirement. It seems like our acquisition system can follow these processes fairly well:
MSA-> Tech Dev -> EMD -> Production/Deployment
Pauli[edit | edit source]
The Kano Model
- While I've never used the Kano Model or seen it used, I think it's a great way to illustrate the extent to which a product is going to satisfy a customer. I think the exercise we did in class was quite useful to visibly see how Objectives, Thresholds, and KPPs fit into the Kano Model quadrants. Moreover, the fifth category "reverse quality" attributes at first seems unnecessary but could be useful to explicitly illustrate what's been considered.
The Conjoint Analysis Method
- A quantitative tool always seems to be preferred by decision makers in which case, this would be a useful tool. Geared toward consumer products, this tool attempts to comprehensively determine customer needs. Unfortunately, it may require a lot of time to use this method correctly (ID all combos of a product features + devise a survey + wait for survey data + analyze it)
The Product Value Matrix
- I think the product value matrix is a great tool to identify all stakeholders and describe the effect each stakeholder has on the process. This should probably be the first step in a program manager/product developer role whereby the other tools can be used in light of how they contribute to the stakeholders interests. The reality is that without stakeholder buy-in, the process slows down or fails to launch.
The Lead User Method
- Since asking a potential customer what they want doesn't exactly work for developing new products, this technique addresses that by attempting to invent products based on user trends. However, in Defense, do we have lead users to collaborate with? Perhaps areas like Cyber may be more fertile ground to tap lead users (i.e. collaborate with the "hackers" to develop new protection/attack methods).
Quality Function Deployment (QFD)
- Out of all other methods, this is one that I've heard of and wonder if this is the one used in DoD more than the others. It's a good concept to ensure all steps in product development are driven by customer needs but seems like this could get overly complicated and difficult to use efficiently. However, the process of going through something like this might be valuable simply to ensure all activities are oriented correctly—and to challenge those that do not tie to a customer need activity.
Richey[edit | edit source]
QFD appears to be the most beneficial of these five methods to understand, categorize, and prioritize stakeholder requirements for whole products. The method provides a product team with sound means by which to correlate product design with customer satisfaction, enabling requirements traceability. Traceability is an important advantage as the team progresses from the product plan to the production plan, because it enables informed decision-making. Trade-offs effect projects during all phases, representing the greatest risk management opportunity early on...and correlation factors allow decision-makers to quantify proposed trade-offs by their customer importance weighting. Perhaps you could also apply weights to the customers themselves (e.g., ACC vs. PACAF). So, traceability enables trade-offs, which (should) reduce risk and improve decision-making timelines in agile acquisition...to capture and hold markets.
Skiple[edit | edit source]
- The Kano Model: I was not familiar with this model, but it is straightforward and quite easy to see how different attributes of a product map to the customer needs, regardless of what we call them (KPPs, “delighters,” etc.). I think the trouble we get into is making assumptions without always understanding what's important to a customer (“we can make a 90 minute battery, the customer will love it.”)
- Cojoint Analysis: To me, this arguably along the lines of “cost as an independent variable,” and in that form I’ve seen it used successfully in proposals coming in from contractors. That way it more or less goes beyond the assumptions about a customer's requirements versus “desirements.”
- Product Value Matrix: This seems to me to be the “government way.” I’ve seen a number of proposals for system upgrades, for example, where all the “ilities” have to get addressed. There is a need many times to “market” to a wide variety of stakeholders, some of whom may not even realize they are stakeholders. Some squeaky wheels will get more attention than they merit, and overall this can be a lot of work.
- Lead User: Makes me think of joint capability technology demonstrations (JCTD). A great way to let the user “play” with the technology and come up with potentially novel (innovative) ways of employing it, something we should probably do more of.
- QFD: I confess I did not think of this as “marketing,” but it actually does just that. I’ve seen it bandied about all over, and in certain circles it is sworn by, but I’ve never personally gotten my hands dirty in it.
Homework 2 - Sidharth Rupani MIT Dissertation - Standardization of Product Development Processes in Multi-Project Organizations[edit | edit source]
Links[edit | edit source]
- ESD Symposium Presentation Slides - http://esd.mit.edu/symp09/presentations/day3.session1b.rupani.pdf
- LAI Summit Presentation Slides - http://lean.mit.edu/component/docman/doc_details/2284-product-development-process-standardization-in-multi-project-organizations
Homework 3 - Lt Col Dan Ward article - My Big, Slow Fail: A comedy of errors[edit | edit source]
- (Pauli) Wow. Is this what I should expect in my near acquisitions future? Yet somehow, this is easily conceivable. A million forms and rules, task saturation, unexpected and expected delays. Murphy's Law could be defined as this story. I liked how he pointed out how he avoided the rookie mistake of failing to get face time right away. Maybe that's my take away in a world of task saturation—doing things in person so that you stand out to the contracting officer, specialist, etc. who is probably drowning in other tasks and emails. Sometimes what seems like the long way is the quickest way.
- (Skiple) Colonel Ward was at least partially sarcastic when he talks about his half day field trip to the CO's office, but I’ve see that same hesitance to travel in my office, where the CO is a block away. But hey, it's uphill, there are no sidewalks, and the geese are getting more aggressive every season. There is never enough communication going on, and there never will be enough, in project management. Government contracting is an incredibly complex process, but expecting everything (make that anything) to flow smoothly in a series of seamless handoffs is just not realistic. As much as we'd all like to think that the “one more thing” phenomenon could be replaced by a foolproof recipe, it's not likely to happen any time soon. Best approach is to maintain a sense of humor.
- (Ho) Sounds like the acquisition system we all know and love. This is the reason the system is broke. A task that was supposed to be simple, was turned into an 11-month ordeal. Luckily in this situation the warfighter or customer did not suffer. I can't help to think that this is partly due to the fact that we have a very complicated, standardized process that is unknown to a large majority. Does this come back on training acquisition officers or does this fall on a broken system?
- (Richey) Although it seems to be embellished, the story is an accurate reflection of my acquisition experience. By that I mean acquisition guidance is so convoluted and subject to varying interpretations that it makes an acq professional's life very difficult. You can really get bogged down by some personalities who supposedly "know the law." (even legal interpretations can differ) In the end, people won't play if they don't want to, and will still begrudge you if they are directed to cooperate. Oh yeah, and stakeholders can get really nasty!
Homework 4 - LAI Whitepaper on Program Management and Waste in Lean Product Development[edit | edit source]
- (Pauli) The Product development process is less wasteful when it enjoys the same courtesies as thesis development. To avoid rework, unscheduled work, defective information, and an unstable process 1) You don't want to overproduce/process too much information without consulting the customer/boss/thesis advisor (CBT) 2) Stick to a schedule—don't deliver information too early or too late, however...3) Be respectful—you're not everyone's highest priority nor should you be. 3) You don't want to stockpile too much research too long without processing it with the customer/boss/thesis advisor. 4) Don't overuse email...email should be for simple, routine messages...best to discuss complex matters in person/on phone. The simpler the communication structure, the better (you don't want too many cooks in the kitchen). 5) Save other people's time by digesting raw information into what's relevant to the recipient before delegating). 6) Try not to be a bottleneck with signatures/easy yes/no approvals.
- (Pauli) Can we discuss the following single biggest actions that reduced program duration?: 3. Reduction of technical uncertainty; 4. Improve success at Critical Design Reviews and 5: Eliminate wait time for Acquisition Panel activities. Combined actions showed the biggest impact. What combinations?
- (Pauli) Can we discuss the specific recommendations for Air Force acquisition policy makers and practitioners?: 1. Make system representations and adaptability part of acquisition planning; 2. Involve the operational user in the design phase; 3. Create effective system representations; 4. Make effective use of system representations; and 5. Create a “zone of novelty”, i.e. a mix of flexibility and structure for the program.
- (Skiple) Lots of papers discussing “learning organizations” and team performance; one mentioned the impact of the DoD's aging technical workforce and how much of the organizational knowledge is in danger of being lost. Are we in danger of “leaning” the system too much by relying on the superstar PMs to do too much who may not be around a lot longer?
- (Skiple) Is it really 2/3rds of project time spent waiting for something to happen? Is this really “waste” or is it more analogous to, say, thermal efficiencies of internal combustion engines (which are far less than 100%)? In other words, is there a theoretical limit on the work that can be produced?
- (Ho) Understandably, there's a lot of emphasis on the value stream in Lean Engineering. Since the AF has gone towards lean thinking, it seems like we sometimes fail at fully committing to it, specifically eliminating waste in the value stream. In my short career and experience, it seems as though we do alright identifying the waste, but cannot fully commit to eliminating it due to "tenured" civilians in certain positions or just because it's the way we've done it in the past. This point seems counterintuitive from the Program Management paper that cited low MIL tenure detrimental to PD.
- (Richey) Very good addition to our "toolkit." Touched on several of our lessons throughout the GRD program, from Dual Ladder intellectual capital investments to team vs. functional leadership. Makes me wonder (once again) about the state of Air Force acquisition technology transition! It would be helpful if we could gain access to the linked references.
- (Richey) Correcting information wastes a tremendous amount of time, especially later in a project's schedule. It seems to me that information "config control" plays a huge part in this. For example, in the F-22 program, you could find two different technical specs on support equipment in the data library. It was always like "hey, who's going to fix this?" - but nothing came of it. I'm sure similar problems have plagues other programs, resulting in more serious issues.
Homework 5 - LAI Risk Management Paper [edit | edit source]
- (Richey) Good overview of ISO 31000 (also touched on in QMGT). Section 4.3, The Organizationa Context, highlights some of the prevailing issues regarding the proper implementation & execution of risk management across AF acq programs. Specifically, point four calls out the need for sufficient resources - which most program offices do not have. However, (like everything else) if leadership wants it to be done, it can/will be done...but, perhaps our culture perpetuates inadequate risk management.
- (Pauli) Gratuitous risk analogy for the day: Good risk management is a filter that prevents the most costly risks from passing through the system while retaining the ones that hopefully won't be too harmful. A filter that is too restrictive costs too much but so is one that lets too much through. It's important to not only get a "BOGSAAT" to walk through the risk framework but to decide how much each risk will cost to mitigate and thus, which risks deserve money to be mitigated (if any).
- (Ho) The first sentence is of particular importance to DoD: the culture is just as important as the process. We touched on this in the last class concerning culture and the organizational context, we live in a culture that is sometimes (or most times) a hindrance to risk management and sometimes lack the vision (or know how) to change this. Maybe we need to start at the micro-level and work our way up...start training an acquisition corp in risk mitigation, I would as far as to have a unified training plan/career development plan for acquisitions. Pilots are trained early at the onset of UPT with risk identification and mitigation concerning flying ops.
- (Skiple) The "real options" section was interesting. I had not thought of that strictly in terms of risk management probably because of the negative connotation that "risk" has. Real options involve building flexibility and robustness into your plan, but doing so smartly (i.e. not just making something bulletproof because you can, regardless of whether it will ever see a bullet).
Topics for a future session[edit | edit source]
Slingshot method[edit | edit source]
Are there other methods used for idea generation?[edit | edit source]
Organizing for communicating[edit | edit source]
Reinertsen book[edit | edit source]
Prototyping/Test Cycles[edit | edit source]
Wheelwright/Clark book[edit | edit source]
Gate decision as key to managing risk[edit | edit source]
Chapter 21 of PDMA, 2nd ed handbook[edit | edit source]
Acquisition's Greatest Hits[edit | edit source]
Evaluating an Effective Risk Mitigation Plan[edit | edit source]
(Ho) Should identify top risks and its impact to the program
(Ho) Resources and stakeholders/key players should be identified and given responsibilities through the risk mitigation process
(Ho) Description of the implementation process must be clear to everyone reading the plan and must include steps taken should a risk be realized.
(Richey) Risk communication – are the right risk statements conveyed (by owners) to the right people (stakeholders) at the right time (in time to act), by the right means (e-mail, memo, contracting correspondence, etc.)
(Richey) The RMP should be synchronized with current program baseline (e.g. risk workshops convened in advance of program management reviews)
(Richey) Legitimacy - RMP is endorsed (signed) by leaders at the appropriate level within an enterprise
(Skiple) Project personnel responsible for risk management are qualified and/or properly trained; any necessary additional training is scheduled to take place soon enough to influence the risk management development.
(Skiple) The RM program is adequately resourced; a budget for establishing the program, required training, risk meetings, etc., is set aside out of the project budget.
(Skiple) Risk reviews are event-driven, integral to overall project management schedule.
(Skiple) A plan to maintain documentation (including software tools) is in place to guard against personnel turnover (i.e., the risks are not all on a massive Excel spreadsheet that "Joe" created and no one else understands).
(Pauli) - Risk responses are actionable within a timeframe that allows risk to be tackled effectively.
(Pauli) - Risk responses are achieveable or feasible (they are within the capability and responsibility to make happen)
(Pauli) - Risk responses are assessed (comparing pre and post response positions)
(Pauli) - Back up risk strategies are identified to the extent possible
(Pauli) - there is a single point of responsibility and accountability for implementing response
The "Short" written summary[edit | edit source]
- Top Risk Identification - Are the top risks identified? How are they identified? How do they impact the program?
- Stakeholder Analysis - Are stakeholders/key players specified? Are stakeholders prioritized? Are these the right people? Is there a communication structure established to facilitate stakeholder communication (e-mail, memo, contracting correspondence)?
- Rigorous Risk Process - Is the implementation process clear to everyone reading the plan? Are risk response steps specified? Are risk responses actionable within an appropriate timeframe (enough time to implement)? Are the steps within the risk process achieveable and within the capability and authority to carry out? How are risk responses assessed? Is there a way to compare pre and post responses? Are back up strategies to risk responses addressed?
- Resources - Is the risk management program adequately resourced? Is there a budget for establishing the program? Does the schedule incorporate risk meetings, training, etc.? Are risk management personnel qualified and/or properly trained? Is any additional training required? Will additional training occur in time to influence the risk management process?
- Reviews - Is there a plan to document risks (including software tools)? Is there a plan to guard against personnel turnover (e.g. the risks are not all on a massive Excel spreadsheet that "Joe" created and no one else understands). Is the RMP synchronized with the current program baseline (e.g. Are risk workshops convened in advance of program management reviews)? Are risk reviews conducted? Are risk reviews event-driven? Are risk reviews integrated in the overall project management schedule? Are risks assigned with enough time to address them?
- Risk Ownership - Is the risk management plan endorsed/signed by leaders at the appropriate level of authority (establishing legitimacy/support to the RMP)? Are risk activities assigned a single owner? What are their responsibilities involved in the risk management process?
Method used to develop criteria: We used a process that resembled nominal grouping in order to develop our criteria:
- First, we individually devised criteria using lecture/course material/internet research that could be used to assess the quality of a RMP (Reviewed risk response effectiveness criteria (e.g. appropriate, affordable, actionable, achieveable, etc.)
- Second, we got together and combined our individual lists. As a group, we reviewed each contribution and mapped each input that perhaps tied with other inputs. This process also led to clarification by uncovering the reasoning behind each individual's input.
- Next, we devised appropriate criteria by assigning categories (or criteria headings) to encapsulate the grouped inputs.
- Finally, we developing prompting questions using the description of each input that could be used to assess a risk management plan.
Analysis of Draft Risk Management Plan: Using our RMP criteria, we assessed the Hyperion RMP. As a whole, it appears the Hyperion RMP is high quality and adequately addresses the concerns associated with risks and how to mitiage them. Below are our specific findings:
Top Risk Identification
- Good: 2.0 Continuous Risk Management Overview - addresses that only top 20% of risks will get resources; others will be watched/accepted which is appropriate and affordable.
- Good: 4.0 Risk Classification - establishes the criteria of impact, likelihood, and timeframe in order to prioritize and assess risk.
- Good: 7.6 Risks and Mitigation Strategies for implementation - identifies not only responses but contingencies which is a great idea.
Stakeholder/Key Role Analysis
- Good: 2.1.1 Identifying Risks - discusses capturing the context (who, what, when, where, why) which is important for risk identification and to understand how the risks are connected to stakeholders. This risk would also received a "good" rating for risk ownership since it ties a name to a risk.
- Good: Table 1 highlights key roles and responsibilities in order to communicate and document risk information more efficiently.
- Good: Table 2 structures how risks should be communicated and when/who they should be communcated to.
- Bad: 7.2 Roles and Responsibilities - doesn't address conflicts of interest that may occur since one person "may fulfill multiple roles"
- Good: 7.2.2 Project Personnel Roles - "When training in tailored risk management practice is made available, all project personnel are expected to take risk management training."
Rigorous Risk Process
- Good: 1.0 Introduction - this identified the intent to design a structured approach to risk management.
- Good: 2.1 Risk Management Process and Data Flow - this outlined the activities of the CRM process which at least alludes to a structured approach to risk management. Moreover, meetings and individual activities are tied to the processes which suggests accountability.
- Good: 2.1.2 Analyzing Risks - uses a multivoting to prioritize risk and Tri-level attribute evaluation method to determine impact, probability of occurrence, and timeframe.
- Good: 7.5 Evaluation Measures and Completion Criteria - success criteria is important to identify and this section outlines it well.
Resources (Budget, Schedule, People)
- Good: 2.1.3 Planning Risks - allocating resources to only the most substantive risks is prudent (20%)
- Bad: 6.0 Hyperion Resources - could be more specific and provide more details on resource allocations such as training. As it is, it only generally addresses that resources (cost, staff, equipment, software) shall be identified but doesn't discuss any further other than establishing two categories to place these resources for risk management.
- Good: 7.1.2 Reporting Requirements - identifies who requests for resources should be made to (OSSMA).
- Bad: 7.3.1 Detailed Transition Schedule Milestones - should at least specify a date for the draft RMP to be documented)
- Good: 126.96.36.199 Basic RM Practice Phase - great job of outlining RM training and dates/POCs with those activities.
- Good: 1.0 Introduction - this established a regular 6 month review meeting which is important to show how well the RMP is working.
- Good: 5.0 Hyperion Risk Reporting - maps risk management activities against project milestones including baselines,major reviews of risk status, and routing activities. Also outlines weekly project and IPT meetings and monthly project meetings that will view stoplight charts.
- Good: 7.1.2 Reporting Requirements - monthly progress reports on the successes/difficulties of implementing risk management.
- Good: 1.0 Introduction - this identified a single accountable POC which is important to establish clear lines of authority and accountability
- Good: 7.1. Sponsorship - identifies Davey Jones as the project sponsor [clear]. Also, 7.1.1 - "Jack Sparrow"
- Bad: 7.1.1 Sponsorship Roles and Responsibilities - "All sponsors' written endorsement and encouragement of this effort to all Hyperion project personnel." The last part--"all personnel" is unclear (The General? Guy that takes out trash?)
- Bad: 7.2.1 Infrastructure Roles to filled - does a good job of assigning names to roles for most part but doesn't specificy where the two facilitators from outside the project should come from.
Recommendations for improving the quality of the Hyperion RMP:
- Bad: 7.2 Roles and Responsibilities - address conflicts of interest that may occur since one person "may fulfill multiple roles"
- Bad: 6.0 Hyperion Resources - provide more specific details on resource allocations such as training.
- Bad: 7.1.1 Sponsorship Roles and Responsibilities - "All sponsors' written endorsement and encouragement of this effort to all Hyperion project personnel." The last part--"all personnel" is unclear (The General? Guy that takes out trash?). Need to specify this.
- Bad: 7.2.1 Infrastructure Roles to filled - does a good job of assigning names to roles for most part but should specify where the two facilitators from outside the project should come from.