CONTRIBUTIONS OF THE RESTRUCTURING OF THE

ELECTRIC POWER INDUSTRY TO THE

AUGUST 14, 2003 BLACKOUT

By Jack Casazza

Frank Delea

George Loehr

Members, Power Engineers Supporting Truth

                                                            August 2005

I.      Purpose  3

II.     Basic Approach  4

III.            Executive Summary  5

IV.     Change in Industry Structure  6

A.     Prior Procedures  6

B.     Changes with Restructuring  7

C.        Increases in Complexity  8

V.     Lessons Learned from the Analyses of the August 2003 Blackout  9

A.     Past Blackout Reviews  9

B.     The DOE Report 10

C.     The NERC Reports  11

VI.        Change in Focus from Coordination to Competition. 14

A.     Prior Coordination Procedures  14

B.     Changes Caused by Restructuring  14

VII.    Expenditure Reductions to Improve Profits  15

A.        Reductions in Transmission Additions and Maintenance  15

B.        Reductions in Personnel 15

VIII.            Changes in Technical Qualification of Those Managing Electric Power Organizations in Government and Industry  16

IX.        Failure to Pass on Past Knowledge  16

A.        Investigations of Past Blackouts  16

B.        Investigation of August 14, Blackout 17

C.         Departure of Key Personnel 19

X.     Summary  19

XI.            Recommendations  20

I.          Purpose

This report addresses whether the restructuring of the electric utility industry in the United States, and most specifically in the Midwest Independent System Operator (MISO) area, contributed to the August 2003 blackout in northeastern North America.

There have been numerous after-the-fact reviews of the August 2003 blackout. The various institutional reports by the Department of Energy (DOE), North American Electric Reliability Council (NERC), East Central Area Reliability Coordination Agreement (ECAR), etc., addressed the cause of the blackout by looking one step beyond the most immediate causes.  As noted in Chapter 3, page 17 of the April 2004 Final Report of the U.S. Canada Power System Outage Task Force:

“A dictionary definition of “cause” is “something that produces an effect, result, or consequence.”¹ In searching for the causes of the blackout, the investigation team looked back through the progression of sequential events, actions and inactions to identify the cause(s) of each event. The idea of “cause” is here linked not just to what happened or why it happened, but more specifically to the entities whose duties and responsibilities were to anticipate and prepare to deal with the things that could go wrong. Four major causes, or groups of causes, are identified (see box on page 18).

Although the causes discussed below produced the failures and events of August 14, they did not leap into being that day. Instead, as the following chapters explain, they reflect long-standing institutional failures and weaknesses that need to be understood and corrected in order to maintain reliability.”

Although this quote suggests that the investigation focused on underlying causes of the blackout, the post-blackout reviews did not specifically address the true root causes, as illustrated by the following extracts from two ECAR reports.

“ECAR INVESTIGATION OF AUGUST 14, 2003 BLACKOUT by MAJOR SYSTEM DISTURBANCE ANALYSIS TASK FORCE

Technical Report”

“It should be noted that the pursuit of the root causes to the above factors was not within the scope of this MSDATF effort, and hence, root causes are not addressed in the [Major System Disturbance Analysis Task Force] MSDATF Technical Report.”

“ECAR INVESTIGATION OF AUGUST 14, 2003 BLACKOUT by MAJOR SYSTEM DISTURBANCE ANALYSIS TASK FORCE

Recommendations Report”

“The technical analysis of what happened in ECAR was done by the ECAR Major System Disturbance Analysis Task Force (MSDATF), which focused specifically on the system behavior during the events leading up to and through the blackout, on the system response to the various events, and on the behavior of the protective relaying systems as the events progressed throughout the afternoon of August 14.”

The post-blackout reviews attempted to determine what happened or didn’t happen from a technical perspective.  They did not take the next step and ask why – what managerial decisions were made, or not made, that brought about the more immediate causes of the blackout. 

For example, tree contact was identified as an immediate cause of power line tripouts.  This was explained by unacceptable right-of-way maintenance practices.  What was not pursued was why the companies involved had decided on their right-of-way maintenance practices.  Could it have been to maximize immediate profits?

Similarly, inadequate situation awareness has been identified as another immediate cause.  This was explained, in part, by deficiencies in the analytical capabilities of control centers, communication protocols, training, etc.  However, the reasons for these deficiencies were not pursued.  Could the reasons have included decisions to keep costs down so as to show better financial results?  Could there have been a decision to “speed things up” when establishing MISO, in reaction to external pressures to establish a new market structure, before all physical and management systems were in place?

Unfortunately, the information required to answer these broader questions is not publicly available and these questions have not even been asked, nor have secondary sources of information (e.g., maintenance expenditures as found in required company submissions to the Federal Energy Regulatory Commission (FERC)) been reviewed.

This report attempts to lay out some of the pertinent questions that are still unanswered and, where possible, supply relevant information and conclusions related to these questions.

II.          Basic Approach

The review presented in this report was conducted by a group of engineers with extensive high-level experience in the electric power industry and access to information from several hundred individuals involved in the industry.  While it was triggered by the August 14, 2003 blackout, it has necessarily involved a more general concern with how national policies have affected the reliability of electricity service to American consumers.   It considers the roles of industry and government, particularly FERC.  It focuses on reliability, examines the responsibilities and failures of NERC, includes brief discussions of the government’s investigation of the August 14, 2003 blackout, and calls attention to some as-yet unanswered questions.  It is based on two essential ingredients:

Ø     When comparing alternatives, characterize them correctly.  In reviewing the effect of deregulation on the August 14th blackout, it is essential that comparisons be made based on accurate information about prior procedures.  Unfortunately, many who make such comparisons often have very little knowledge of past procedures. 

Ø     Avoid being influenced by political, commercial, or personal power concerns.  The most precious thing that anyone can bring to a review of policy, such as this report, is to be an honest witness to what he or she knows.

III.          Executive Summary

Deregulation and the concomitant restructuring of the electric power industry in the U.S. have had a devastating effect on the reliability of North American power systems, and constitute the ultimate root cause of the August 14, 2003 blackout.  Specifically, deregulation and restructuring have led to:

Ø     Changes in focus from long-term optimization and inter-system coordination and reliability to total dependence on immediate profits and the efficacy of “the market.”

Ø     Change in technical qualifications of those holding management positions in electric power organizations and government policy makers and regulators; this change affects entire organizations.

Ø     Reductions in personnel at electric power organizations and companies.  

Ø     Failure to make adequate technical analyses including risks when setting government policies.

Ø     Increased complexity of operations because of separation of generation and transmission functions, the large increase in the number of organizations involved, and the establishment of additional levels of responsibility in the operation/control process.

Ø     Dilution of management responsibility, including too many entities in the management structure with veto power.

Ø     An almost fundamentalist reliance on markets to solve even the most scientifically complex problems.

Ø     Decreased emphasis on the importance of strong reliability standards, and a trend toward lower standards;  this is most pronounced in the very organization charged with maintaining reliability –NERC – aided and abetted by FERC.

Ø     Dispersed, fragmented control of the bulk power system in the Midwest.

Ø     A patchwork quilt of overlapping jurisdictions among marketing areas, Independent System Operators/Regional Transmission Organizations (ISOs/RTOs), and regional reliability councils in the midwest.

Ø     Reductions in, or outright elimination of, training including training of operators.

Ø     Continuation of the historical problem of geo-electrically small control areas in the Midwest, despite the creation of the MISO, which, in the context of operations on August 14, 2003, appeared to be little more than a toothless shell.

Unless the root causes of the August 14, 2003 blackout are addressed and the trend toward lower standards reversed, the likelihood of future blackouts will increase.

The DOE/Canadian report demonstrates the dominance of market participants and lack of government concern about the root causes of the blackout. [1]^{, [2]}  Both are also clearly illustrated by the almost two-year delay in the investigations, and discussions that are taking place through the competition and reliability study of which this paper is part. 

Despite its “spin”, the Energy Policy Act of 2005 does nothing to address the root causes of the 2003 blackout, and hence will do nothing to enhance reliability.

IV.     Change in Industry Structure

To understand the changes that have taken place in the structure of the electricity industry, it is necessary to compare current and prior procedures and examine the increased complexity of today’s system relative to the system of the past.

A.       Prior Procedures

From the installation of inter-regional transmission ties in the 1960s until the start of restructuring/deregulation in the 1990s, generation and transmission were planned, designed, and operated on a regionally coordinated basis.[3] ^[4]  Most individual power companies were vertically integrated, so development of generation and transmission was coordinated.  Installation of generation took into account transmission limits, and installation of transmission took into account generation needs.  Hierarchal organizations (power pools) evolved covering multi-company areas and each having limited numbers of participants; e.g., the Pennsylvania-New Jersey-Maryland Interconnection (PJM), New York Power Pool (NYPP), and New England Pool (NEPOOL).  Power pools in turn combined to form regional reliability councils; e.g., Mid-Atlantic Area Council ((MAAC), ECAR, and Northeast Power Coordinating Council (NPCC).  These councils coordinated through organizations such as MEN (consisting of MAAC, ECAR, and NPCC), VEM (consisting of VACAR, ECAR, and MAAC), and VAST (consisting of VACAR, American Electric Power [AEP], Southern, and Tennessee Valley Authority [TVA]).  Reliability criteria, monitoring and enforcement were accomplished by the regional councils, which had boundaries congruent with planning and operating organizations.  Inter-regional coordination was accomplished by organizations such as MEN, VEM, and VAST.

The various organizations communicated often and effectively and cooperated both on real-time operating basis and on a longer-term planning basis.  Sales and purchases of power were conducted through these hierarchal organizations, and each organization was well aware of conditions in other systems that could affect it.  Each system’s plans were coordinated with those of its neighbors.

B.          Changes Resulting From Restructuring

However, during the past 15 years, the structure – indeed the very underpinnings – of the electric power industry have changed.  The following phenomena are key indicators of s the changes in the industry’s structure:

Ø     The functional separation of generation and transmission within companies as mandated by Order 888.  Reliable planning and operation of a bulk supply system requires full coordination between generation and transmission; this functional separation made coordination much more difficult.^[5]  In most companies system planning departments were split up or disbanded.  Typical organizational impacts were:

·       The diffusion of best technical knowledge which in the past was centered in planning departments.

·       Severe reductions in personnel in generation and transmission, including encouragement of senior personnel to take early retirement.  These reductions effectively ended the transfer of essential expertise from one generation to the next.[6]^{, [7], [8], [9], [10]}

·       Reductions in training as a means of reducing costs. 

·       The divestiture by many private utilities of their generation resources in response to regulatory pressures. Many companies either “spun-off” their generating assets into unregulated affiliates or sold them to third parties. This increased number of “players” greatly complicated the system planning process and diffused responsibility for maintaining a reliable system.

Ø     The transfer of control of transmission assets in response to federal regulatory requirements to ISOs/RTOs, the majority of whose boards were made up of individuals with no knowledge of power system operational or reliability issues.[11]

Ø     Entrance of merchant power plants into the power system.  This also complicated the system planning process and diffused responsibility for maintaining a reliable system.

Ø     New market areas were established that were inconsistent with the boundaries of responsible operating entities and/or the regional councils responsible for reliability standards and enforcement.  For example, the PJM and PJM West marketing area stretches across three reliability councils, at least three ISO/RTO-type organizations, and numerous control areas.

Ø     An increase in system and decision making complexity, with more opportunities for delay and the likelihood of “watering down” decisions to the lowest common denominator.  On the day of the August 14, 2003 blackout, MISO had neither the authority nor technical means to operate a generation and transmission grid in the region.  Since formal spot-markets had not been established, a large number of bi-lateral contract trades originated with IPPs, complicating system operations[12]. These IPPs had little incentive to provide needed reactive power on the day of the blackout. 

C.          Increases in Complexity

The changes described above have created a more complicated and compartmentalized industry structure than was the case in the past.  One example of the increased complication is the extraordinary increase in the membership of three “old line” power pools.  The New York Independent System Operator (NYISO) 2004 Annual Report[13] cites 245 “market participants” significantly more than the eight members that made up NYPP prior to restructuring.  Each of these 245 “market participants” can make decisions about buying and selling electric power that affect the transmission system in New York and in other regions.  In 1993, PJM had 10 members and served 23 million people in five states and the District of Columbia [14].  Today, PJM has more than 350 members and operates in 13 states and the District of Columbia.  ISO New England now has 237 market organizations, 150 of which are members of the Participants’ Committee[15].  The complexity of the decision-making process has increased on the same scale in other regions of the country.  There have not been significant changes since the 2003 Bblackout.

In addition to an absolute increase in the number of participants, the inter-relationships between and among the participants have changed.  No longer are decisions made by a relatively small number of non-competing organizations; today, decisions are made by a large number of entities, most of which are competitors and each of which has more interest in profit than in bulk-power-system reliability.  Procedural rules established between and among the various parties are no longer matters of overall corporate policy, but rather of contractual arrangements based on the parties’ financial self-interest.

In sum, the complexity of planning and operating the electric power system has significantly increased with the growth in the number of participants whose decisions affect the overall system.  It has further increased because the objective of many of these organizations is short-term profits rather than long-term reliability.[16]

V.          Lessons Learned from Analyses of the

August 2003 Blackout

For an overview of lessons learned from analysis of the 2003 blackout, we examine reviews of major blackouts in the past as well as the reports by DOE and NERC on the 2003 blackout.

A.       Past Blackout Reviews

Following the August 14, 2003 blackout, a number of reviews were conducted, the most visible of which was by DOE and the Canadian government.  Separate reports were prepared by NERC; by ECAR, MAAC, and NPCC; by the NY, New England and PJM ISOs; and by state regulatory commissions.  Many of these documents focused narrowly on technical issues.  However, the DOE/Canadian and NERC reports raise issues that could and should have been explored more deeply to determine how the complicated organizational structure of the current “deregulated” industry, with its heightened focus on commercial concerns, might have contributed to the problems that led to the blackout.  Our approach is to highlight some of the conclusions/recommendations of these reports and to raise follow-up questions that have not been addressed.

B.      The DOE Report

The following italicized material[17] is excerpted from the Final Report on the August 14, 2003 Blackout in the United States and Canada - Causes and Recommendations - April 2004.  The inserted questions are raised by the authors of this report.  

Chapter 3, “Causes of the Blackout and Violations of NERC Standards.” 

Page 19. “Group 2: Inadequate situational awareness at FirstEnergy. FE did not recognize or understand the deteriorating condition of its system.

Violations (Identified by NERC):

• Violation 7: FE’s operational monitoring equipment was not adequate to alert FE’s operators regarding important deviations in operating conditions and the need for corrective action as required by NERC Policy 4, Section A, Requirement 5.

Other Problems:

• FE’s operational monitoring equipment was not adequate to provide a means for its operators to evaluate the effects of the loss of significant transmission or generation facilities as required by NERC Policy 4, Section A, Requirement 4.

• FE’s operations personnel were not provided sufficient operations information and analysis tools as required by NERC Policy 5, Section C, Requirement 3.

• FE’s operations personnel were not adequately trained to maintain reliable operation under emergency conditions as required by NERC Policy 8, Section 1.”

In the aggregate, the above problems raise the question of why adequate equipment, information and training were not provided.

Page 20. “Group 4: Failure of the interconnected grid’s reliability organizations to provide effective diagnostic support.

Violations (Identified by NERC):

• Violation 5: MISO was using non-real-time data to support real-time operations, in violation of NERC Policy 9, Appendix D, Section A, Criteria 5.2.”

Why weren’t real time data used?  Was there consideration given to its use and, if so, why was the decision made not to use it?

·       “Violation 6: PJM and MISO as reliability coordinators lacked procedures or guidelines between their respective organizations regarding the coordination of actions to address an operating security limit violation observed by one of them in the other’s area due to a contingency near their common boundary, as required by Policy 9, Appendix C. Note: Policy 9 lacks specifics on what constitutes coordinated procedures and training. “

When MISO was established and approved, why wasn’t this most basic function of an ISO in place?

“Other Problems:

·       MISO did not have adequate monitoring capability to fulfill its reliability coordinator responsibilities as required by NERC Policy 9, Appendix D, Section A. “

Again, when MISO was established and approved, why wasn’t this basic function of an ISO in place?

·       “American Electric Power (AEP) and PJM attempted to use the transmission loading relief (TLR) process to address transmission power flows without recognizing that a TLR would not solve the problem. “

What instructions/directions were given to the operators when trade-offs between reliability and commerce occurred?

Page 21. “Institutional Issues

2.   NERC and the industry’s reliability community were aware of the lack of specificity and detailing some standards, including definitions of Operating Security Limits, definition of planned outages, and delegation of Reliability Coordinator functions to control areas, but they moved slowly to address these problems effectively. “

What impediments does the new stakeholder process place in the path of the “industry’s reliability community when it tries to move effectively and expeditiously?

C.      The NERC Reports

NERC in its report “August 14, 2003 Blackout: NERC Actions to Prevent and Mitigate the Impacts of Future Cascading Blackouts – February 10, 2004” approved 14 recommendations for corrective action. Included are the following directives to MISO:

“B. Corrective Actions to Be Completed by MISO

MISO shall complete the following corrective actions no later than June 30, 2004.

1. Reliability Tools. MISO shall fully implement and test its topology processor to provide its operating personnel real-time view of the system status for all transmission lines operating and all generating units within its system, and all critical transmission lines and generating units in neighboring systems. Alarms should be provided for operators for all critical transmission line outages. MISO shall establish a means of exchanging outage information with its members and neighboring systems such that the MISO state estimation has accurate and timely information to perform as designed. MISO shall fully implement and test its state estimation and real-time contingency analysis tools to ensure they can operate reliably no less than every ten minutes. MISO shall provide backup capability for all functions critical to reliability.

2. Visualization Tools. MISO shall provide its operating personnel tools to quickly visualize system status and failures of key lines, generators or equipment. The visualization shall include a high level voltage profile of the systems at least within the MISO footprint.

3. Training. Prior to June 30, 2004 MISO shall meet the operator training criteria stated in NERC Recommendation 6.

4. Communications. MISO shall reevaluate and improve its communications protocols and procedures with operational support personnel within MISO, its operating members, and its neighboring control areas and reliability coordinators.

5. Operating Agreements. MISO shall reevaluate its operating agreements with member entities to verify its authority to address operating issues, including voltage and reactive management, voltage scheduling, the deployment and redispatch of real and reactive reserves for emergency response, and the authority to direct actions during system emergencies, including shedding load.”

Collectively, these directives raise the question of why approval was given by FERC for MISO to become “operational” in the first place if so many basic operational issues have not been resolved.

The following italicized text was excerpted from NERC’s “Technical Analysis of the August 14, 2003, Blackout: What Happened, Why, and What Did we Learn? – Report to the NERC Board of Trustees by the NERC Steering Group July 13, 2004”. 

The first page of the July NERC report states that “… the NERC investigation did not address regulatory, economic, market structure or policy issues” related to the blackout. However, on pages 94 and 95 under the section titled “Causal Analysis Results”, a tantalizing statement is made that reinforces our view that additional investigation is warranted (emphasis added).

“The causes of the blackout described here did not result from inanimate events, such as ‘the alarm processor failed’ or ‘a tree contacted a power line.’ Rather, the causes of the blackout were rooted in deficiencies resulting from decisions, actions, and the failure to act of the individuals, groups, and organizations involved. These causes were preventable prior to August 14 and are correctable. Simply put — blaming a tree for contacting a line serves no useful purpose. The responsibility lies with the organizations and persons charged with establishing and implementing an effective vegetation management program to maintain safe clearances between vegetation and energized conductors.

“Each cause identified here was verified to have existed on August 14 prior to the blackout. Each cause was also determined to be both a necessary condition to the blackout occurring and, in conjunction with the other causes, sufficient to cause the blackout. In other words, each cause was a direct link in the causal chain leading to the blackout and the absence of any one of these causes could have broken that chain and prevented the blackout. This definition distinguishes causes as a subset of a broader category of identified deficiencies. Other deficiencies are noted in the next section; they may have been contributing factors leading to the blackout or may present serious reliability concerns completely unrelated to the blackout, but they were not deemed by the investigators to be direct causes of the blackout. They are still important; however, because they might have caused a blackout under a different set of circumstances.”

The following italicized sections are some of the General Conclusions also from page 94 of the NERC report.  The questions inserted between the quoted texts are those that we believe need to be addressed.

• Reliability and control areas have adopted differing interpretations of the functions, responsibilities, authorities, and capabilities needed to operate a reliable power system.”

The reliability coordinator function is relatively new (post restructuring). Why was it approved with such apparent weaknesses in its mission?

“• Deficiencies identified in studies of prior large-scale blackouts were repeated, including deficiencies in vegetation management, operator training, and tools to help operators better visualize system conditions. “

What are the reasons for the inattention to these prior problems?

The following quotations come from page 101 of the NERC report, in the section entitled “Summary of Other Deficiencies in the Blackout Investigation”:

“22. Operating entities and reliability coordinators demonstrated an over-reliance on the administrative levels of the [transmission loading relief] TLR procedure to remove contingency and actual overloads, when emergency redispatch of other emergency actions were necessary.  TLR is a market based congestion relief procedure and is not intended for removing an actual violation in real time.”

This observation raises the question of what managerial directions/guidance/instructions operators had been given vis-à-vis the relative importance of reliability and the market.

“NERC’s technical analysis of the August 14 blackout leads it to fully concur with the Task Force Interim Report regarding the direct causes of the blackout. The report stated that the principal causes of the blackout were that FE did not maintain situational awareness of conditions on its power system and did not adequately manage tree growth in its transmission rights-of-way. Contributing factors included ineffective diagnostic support provided by MISO as the reliability coordinator for FE and ineffective communications between MISO and PJM.”

Why was MISO authorized by FERC if MISO was unable to provide effective diagnostic support, or if there were ineffective communications capabilities between MISO and PJM?

The remaining sections of this report attempt, to the extent feasible, to answer the questions raised above and to explain how restructuring caused the blackout, as well as an overall national decline in the reliability of electric power systems.

VI.          Change in Focus from Coordination to Competition.

To understand how industry restructuring has led to a change in focus from coordination to competition, we summarize the coordination procedures of the past (regulated) industry and the changes that have resulted from deregulation.

A.       Prior Coordination Procedures

Electric power systems require investments in major facilities typically costing from tens of millions to billions of dollars.  These facilities have long lead times, requiring many years from start to completion, and often remain in service for up to 40 years.  Regulation provided for the return of the investment (depreciation) and the return on the investment (earnings) over the facilities’ lifetimes.

Electricity systems were interconnected to take advantage of diversity in times of peak use, equipment outages and emergencies.  The industry’s focus was on reliability and long-term cost minimization.  In that environment, a high degree of cooperation developed among those involved in owning, managing, planning, and operating electric power systems.[18]  This level of coordination and cooperation was accelerated in the years following the November 9, 1965 blackout.

B.          Changes Caused by Restructuring

With deregulation and restructuring, the emphasis shifted from technical knowledge and competence to financial and marketing knowledge.  Economic theory replaced engineering fact.  The new managers are driven by the desire for “immediate profit.”  This has sometimes led to conflicts between marketers focused on profits and system operators responsible for reliability, and disputes have been arbitrated by top management.[19]

In brief, restructuring fostered policies that involved increased reliability risk taking in order to improve profits.

VII.    Expenditure Reductions to Improve Profits

Deregulation has resulted in reductions in expenditures on transmission facilities, maintenance, and personnel in the industry.

A.          Reductions in Transmission Additions and Maintenance

There has been a 25 percent reduction in expenses for maintenance of power-system facilities[20] (including but not limited to tree trimming), and in personnel, (including operating personnel).  In many companies, the time between routine maintenance schedules has more than doubled since deregulation.[21]  Between 1990 and 2000, transmission investment fell at a rate of about $50 million a year.[22]

B.          Reductions in Personnel

The labor force at investor-owned utilities decreased from 480,000 to 350,000 between 1990 and 1999.  U.S. Department of Labor data show that from 1999 to 2000 the numbers of utility employees working in power generation dropped from 350,000 to 280,000, and in transmission and distribution from 196,000 to 156,000, while electricity consumption continued to increase.  Among the consequences were drastic reductions in training.  At a FERC Technical Conference in Philadelphia, one system operator observed, “We have downsized quite a bit in our operating staff.... There is not a whole lot of time left for training.”[23]  Many systems had no operator training programs, relying solely upon “on-the-job” experience.  An independent European analysis has concluded that personnel reductions played an important part in recent blackouts there.

VIII.          Changes in Technical Qualification of Those Managing Electric Power Organizations in Government and Industry

During the past 15 years there have been major shifts in the qualifications, experience, and knowledge required of those who control electric power policy and manageing electric power activities.[24]^{, [25]}

Past experience has shown that technical standards and procedures are much less important than the qualifications of the individuals who apply and enforce those standards and procedures.  Nonetheless, in response to the new preeminence of market concerns, appointments to key industry regulatory and reliability organizations have increasingly downplayed technical knowledge and experience.[26]  Many appointees to key electric energy policy positions, such as FERC Commissioners, show a complete lack of the experience relevant for their positions, or are beholden to certain segments of the industry.[27]

IX.          Failure to Pass on Past Knowledge

Deregulation has led to a failure to transfer knowledge gained from past blackouts even though such knowledge could help prevent, or accelerate recovery from, future outages.  This lack of knowledge transfer contributed to the 2003 blackout.  We delineate the failure in knowledge transfer by reviewing investigations of past blackouts as well as the 2003 blackout.

A.          Investigations of Past Blackouts

Between 1965 and 1977 there were three major blackouts affecting the eastern U.S.  In 1978 a major blackout shut down all of France, an outage in scope close to the size of the August 14, 2003 blackout.  Reviews of the 1965 northeastern U.S. and 1967 PJM blackouts led to the realization that extensive regional coordination of planning and operations was required to improve reliability. 

Other lessons were learned from reviews of prior blackouts, e.g., the need to make certain that relay settings and transmission ratings were consistent and communicated to operating personnel (Northeast 1965), the need for “black start” capability (Northeast 1965), the vital need for an Energy Management System (EMS) to analyze potential problems (PJM 1967), the need for improved system restoration procedures (Northeast 1965, PJM 1967, Con Edison 1977), the need for adequate communication within and between control areas (Con Edison 1977), the need for adequate reactive supply (France 1978), and the need to make certain that line clearances on rights of way are maintained (West Coast 1996).

Following the blackouts mentioned above, many technical reports and papers were prepared, presentations were made at various public and technical committee meetings, and magazine and newspaper stories were published.  Some of the lessons were specifically addressed in reliability council documents.  However, these lessons have been ignored by the new, post-deregulation policy makers in today’s electric power industry.  The DOE report on the blackout identifies these failures to transfer lessons learned from past blackouts as an important contributor to the August 2003 blackout.

The head of the DOE’s Office of Transmission and Distribution[28] commented that the restoration of power after the August 14th blackout in about 2½ days was a remarkable achievement.  This was a prime example of two of our observations: an especially uninformed regulator and the failure to pass on past knowledge. Almost 40 years earlier (November 1965), a system almost as large was completely restored in 13 hours.  Following the blackout of all of France in 1978, the entire system was restored in four hours!  Understanding how and why these restorations were accomplished is important knowledge, but was totally ignored by policy makers after August 14, 2003 – even though a three-volume report on the 1965 blackout was published by the U.S. government (one of this paper’s authors having served on the study group) and a report on the 1978 French Blackout (written by another of this paper’s authors on commission from the DOE).

B.          Investigation of August 14, Blackout

The effect of the lack of technical competence on reliability and the August 14th blackout can be illustrated by a few examples:

1.  FERC had approved the operation of the Midwest Independent System Operator (MISO), stretching across all or parts of three reliability councils.  While FERC’s purpose was to facilitate market procedures, no analysis of the technical adequacy of MISO was attempted.  There was no appraisal of whether MISO was prepared to assume operating responsibilities, whether the MISO control center was complete, and whether its operators were properly trained or qualified.  In April 2003, MISO prepared its “Regional Transmission Organization (RTO) Reliability Plan.”  It was vital that the procedures involved to coordinate MISO’s operations with existing reliability councils and security coordinators be carefully reviewed.  Quoting from the DOE Interim Blackout Report:

• “Before approving MISO, FERC had asked NERC for a formal assessment of whether reliability could be maintained under the arrangements proposed by MSIO and PJM.  NERC replied affirmatively but provisionally.  NERC conducted audits in November and December of 2002 of the MISO and PJM reliability plans, and some of the recommendations are still being addressed.  The adequacy of the plans and whether the plans were being implemented ads written are factors in NERC’s on-going investigation.”

Even though the plans had not been deemed adequate, FERC approved the operation of MISO.

2.  The chairman of FERC recently created a new department to address reliability matters, and assume national control of reliability standards and their enforcement.  Additional engineers were hired for this purpose.  The new staffers will work in the Office of Markets, Tariffs, and Rates, under the management of those responsible for enhancing market procedures.  The creation of this department is a recognition of past FERC failures, but the oversight of the department’s work by personnel whose focus is to enhance markets demonstrates FERC’s inability to understand the problem.

3.  The government’s blackout investigation is another example of the failure to allow technically competent advisors to contribute.  The government carefully selected personnel and orchestrated the investigation’s limited content.[29]  The government controlled the writing of the report, the public hearings, and workshops conducted after the blackout.  Technically competent participants were given bare minimum opportunities to comment.  The government even required those involved in the investigation to sign confidentiality agreements, an action unprecedented in the history of electric power in the U.S.  By contrast, following the 1965 blackout, Joseph C. Swidler, then chairman of the Federal Power Commission (FERC’s predecessor), was instructed by President Lyndon Johnson to have the nation’s best engineering talent available to supervise the investigation – and that is exactly what happened.[30]  The U.S.- Canada Power Outage Task Force and the Electric System Working Group for the August 2003 blackout were composed almost entirely of individuals either in federal or state regulatory positions.  Few appear to have had any technical experience in planning or operating a power company. The NERC steering and working groups were staffed by highly qualified technical people; however, these participants did not oversee preparation of the study’s report which was a DOE staff effort.

C.           Departure of Key Personnel

Often essential knowledge is held by one individual or a very few teams and cannot be passed on except though direct contact, i.e., a “doctor-intern” type relationship.  Programs that encouraged early retirement in electric power companies facilitated the departure of personnel with extensive experience causing a breakdown in the essential transfer of knowledge.[31]  NERC’s approach of writing voluminous procedures is not sufficient to correct this problem.  As we have noted, the best procedures are only as good as the experience and expertise of the parties applying them.

X.     Summary  

The stage was set for the events of the 2003 blackout by changes in the structure of the electric power industry.[32]^{, [33]}  The federal government, mostly through FERC, had mandated untried and inappropriate structural changes.  It established new rules and procedures that facilitated bad behavior (e.g., Enron), with no analyses of the potential effect on reliability.[34]^{, [35]}  In a survey, more than half of the utility executives polled (along with many others) expressed the belief that industry restructuring has caused a decline in reliability.[36]  Only participants with no technical background argued that market forces could somehow produce good engineering designs and operations.

In the wake of these failures, “spin” has, perhaps predictably, replaced substance, for example:

Ø     NERC’s publication of the so-called “Version 0” of its reliability standards was promoted to the press and public as a direct response to the August 14, 2003 blackout – when, in fact, it was nothing more than the restatement (in a somewhat different format) of the same reliability standards NERC had been using for more than a decade.

Ø     NERC continues to deny that changes proposed to its transfer capability definitions are in effect a lowering, and watering down, of its standards.  Yet this is demonstrably so in the view of anyone familiar with the subject.[37] Unless this trend is reversed, more blackouts will happen.

The recently proposed Reliability First Corp., which would merge the regional councils -- MAAC, ECAR, Mid-Atlantic Interconnected Network (MAIN), and possibly Midwest Reliability Organization (MRO) -- into a single new reliability council, is being presented as a “fix” to some of the problems that led to the August 14, 2003 blackout.  But this merger in no way addresses the important question of the host of geo-electrically small control areas in the midwest, which contributed significantly to the 2003 blackout.  Had Reliability First Corp., as now proposed, been in existence on August. 14, 2003, nothing would have been different.  This is a solution searching for a problem, spin rather than substance.

XI.          Recommendations[38]

The authors have been asked to provide recommendations, a difficult assignment.  In the massive effort to “deregulate” and “restructure” the electric power industry, the Laws of Physics were ignored, replaced by a blind conviction that the Laws of Economics could provide all things – including a reliable system.  Unfortunately, this has been proven to be a tragic mistake.  The problem with correction, however, is that a fundamentalist market philosophy has so permeated the entire industry, from the Federal Government and its regulatory officials to the industry’s own organizations, that to undo the damage will likely take an effort well beyond a few simple recommendations.  The problem cannot easily be fixed since the problem is an innate attitude or belief system, not an error or two in procedures or protocols.  An indication of this is the fact that, despite such evidence as the California Meltdown, unprecedented price spikes, the criminal actions of Enron and others, and the most devastating blackout in our history, policy makers still steadfastly deny that deregulation and restructuring had anything at all to do with any of it.  Sociologists call this “cognitive dissonance.” 

Recognizing this difficulty, there are a number of steps that could be taken to start the nation on its difficult corrective path:

Ø     Before approving any new ISO/RTOs, ensure and demonstrate that the entity is fully functional.

Ø     Investigate and recommend guidelines for the geo-electrical characteristics of control areas.

Ø     Require NERC to roll back the reductions in reliability standards implemented since 1998.

Ø     Prohibit NERC from implementing any further reductions in reliability standards.

Ø     Permit any state or reliability entity to mandate more stringent reliability standards than NERC’s.  In other words, make sure that NERC standards are a floor, but not a ceiling.

Ø     Before implementing a new market design, ensure and demonstrate that the design’s impacts on the reliable operation of the power system have been fully evaluated.

Ø     To make markets work more efficiently and effectively, emphasize in policy standards the need to foster cooperation between organizations.

Ø     Develop standards for technical qualifications required for key government and industry positions, including those responsible for establishing electric power policies, and for management, design, and operation of the transmission grid.

Ø     Require that appointments to FERC and the new DOE Office of Electricity of Delivery and Energy Reliability, and to the NERC Board and senior management positions, have demonstrated expertise and experience in electric power and are vetted by the National Academy of Engineers, with input from the Institute of Electrical and Electronic Engineers (IEEE), Edison Electric Institute (EEI), the American Public Power Association (APPA), and National Rural Electric Cooperative Association (NRECA).

Ø     Mandate that DOE, in consultation with FERC, NARUC, and NERC, undertake a biannual “National Power Survey” modeled after the 1964 survey.  This survey should give emphasis to reliability risks, including such incidents as the loss of major gas pipelines, as well as economic constraints.[39]

Ø     Investigate and develop new programs for encouraging and improving the transfer of technical experience and expertise in the electric power industry and universities; such efforts could be enhanced by utilizing experienced retired engineers from the electric power industry.

Ø     Investigate the effects that extensive labor reductions have had on overall national reliability, and on the ability to cope with national disasters and acts of terrorism.

Ø     Require that marketing areas and reliability council areas be consistent.

Ø     Support the reporting and exchanging of information related to system reliability.  (Concerns exist about the consistency of some information, --and the availability of data to the entire electric power industry.)  The Federal Government could play an important role in enhancing the definition, collection, and sharing of information.

Ø     When adjusting generation because of transmission economic constraints, insure that such adjustments minimize reliability risks.

Ø     Investigate and monitor reductions of maintenance expenditures as indicated in reports to FERC as a part of FERC’s reliability monitoring function.

Additional references can be obtained at the following web sites:

www.PEST-03.org

www.ameredinst.org