Getting it Right the Second or Third Time Rather than the Sixth or Seventh

Craig Zimring, College of Architecture, Georgia Institute of Technology, Atlanta, GA

Thierry Rosenheck, Office of Foreign Buildings Operations, US Department of State

Federal building delivery organizations are facing intense pressures. They not only must provide buildings on time and within budget, they are called on to deliver buildings that are improved in many ways: more sustainable, accessible, maintainable, responsive to customer needs, capable of improving customer productivity and safer. And in most cases they must achieve these goals with fewer staff. With these kinds of demands, organizational learning has become a core competence.

By organizational learning, we mean that organizations are able to constantly improve the ways that they operate under routine conditions, such as providing standard office space more efficiently that is also higher quality, and are able to respond to change quickly and effectively when needed . Learning is "organizational" if it about the core mission of the organization and is infused through the organization rather than residing in a few individuals.

For example, Disney evaluates everything they do, and has been doing so since the 1970s. Disney has at least three evaluation programs and three corresponding databases: 1) They keep track of the performance of materials and equipment and record their findings in a technical database; 2) Guest services staff interview guests about facilities and services and record predictors of Disney's "key business drivers:" the intention to return; 3) A 40-person industrial engineering team conducts continuous research that is aimed at refining guidelines and rules-of-thumb. They explore optimal conditions: When does Main Street feel pleasantly crowded but not oppressive? When are restrooms full without undue wait? When are gift shops most productive? This research allows Disney to make direct links between "inputs" such as the proposed number of people entering the gates to "outputs" such as the width of Main Street.

The Disney databases are not formally linked but are used extensively during design and renovation projects. They have been so effective that the senior industrial engineer works as a co-equal with the Imagineering project manager during the programming of major new projects.

Disney is quite rare. It uses an evaluation program to do all four process that are key to organizational learning :

• Monitoring changes in the internal and external business environment and assessing performance;
• Interpreting and discussing the implications of results;
• Consolidating results into an organizational memory;
• Widely distributing findings and conclusions.

POE mostly focuses on individual project support and analysis rather than on lessons-learned

While post-occupancy evaluation (POE) potentially provides a methodology for all four of these processes, POE practice has historically focused on case studies and supporting decision-making for individual projects rather than for more general lessons-learned. Even when evaluators have been able to create databases of findings they have often been used to benchmark single cases rather than for developing more general conclusions.

And organizational learning is hard to do. Learning requires the will to collect data about performance, and the time to interpret and draw conclusions from it. And of course we're often too busy to do these things, despite the best of intentions. But more fundamentally, learning involves risk and change. To learn we have to expose when we make mistakes so we can improve. Most organizations don't reward exposing shortcomings. Imagine for a minute learning to play golf but not knowing whether the ball went toward the flag or not! Learning brings change, and organizations are usually better at trying to ensure stability than at supporting change.

In this paper we explore how a variety of public agencies and some private ones have used POE successfully for organizational learning. We discuss the "lessons-learned" role of evaluation rather that the project support and analysis role. We have examined written materials from 18 POE programs and interviewed participants wherever possible. We explored whether POE-based organizational learning appeared to be going on, whether the organizations had established support for learning and what the nature of the learning was.

Did POE-enabled organizational learning occur?

For example, in looking for evidence of feed-forward we asked:

• Are participants in building projects, including internal project managers, consultants and clients aware of POEs or POE results, either from personal participation or from written results?
• If so, were POE results consciously used in decision-making about buildings? For example, are they used for programming, planning, design, construction and facilities management?
• Can we see evidence that POE results are part of reflection and discussions about how to do a good job, among peers and with supervisors?
• Are POE results consciously used to refine processes for delivering buildings either in terms of formal process reflected in manuals or informal rules of thumb and customs?
• Are people who make policy about buildings, such as policy directives, design guidelines, specifications, etc., aware of POEs?
• If so is POE explicitly used in formulating policy?

Were the conditions for organizational learning present?

As we attempted to document feed-forward we were trying to understand the conditions that foster or thwart it:

• Does the organization have an infrastructure for feed-forward? For example, are results from POEs consolidated in some way, such as in reports or databases? Is this consolidated information distributed, either internally or to consultants or the public?
• Is there a mechanism for ensuring that this information is kept current?
• If lessons-learned are made available, do they support the kinds of decisions that are made by the organization? Are they likely to seem authentic and important to decision-makers? Are the implications of results made clear or do busy decision-makers need to make translations between results and their needs?
• Are there incentives for accessing the data, using the results and contributing to the lessons-learned knowledge base? For example, are internal staff or consultants evaluated on use of lessons-learned? Are consultants rewarded for participation? Are they rewarded for good performance as judged by the POE?
• Are there disincentives for participating in lessons-learned programs? If an innovative initiative receives a negative evaluation is it treated as an opportunity for organizational learning or as a personal failure?
• Is there a perception of high-level support for the lessons-learned program? Many organizations create frequent new initiatives and seasoned staff often perceive these as the management infatuation du jour: wait a day and it will change.

How can organizations develop useful learning content?

We also assessed the content of the organizational knowledge. We see the development of organizational knowledge about buildings as similar to other kinds of learning. When someone goes from a naïve knowledge of physics to a more sophisticated one their knowledge moves from fragmentary, idiosyncratic knowledge to knowledge that is better tested and more integrated. It is similar with organizational knowledge about buildings and building delivery. We asked:

• Has the organization produced a shared view of what makes a good building, either in terms of process or product? For example, has the organization been clear about key design and programming decisions and about how these link to the client's needs? Are these contributed to by POE?
• Has the organization created an organizational memory of significant precedents? Are these precedents described, analyzed or evaluated in meaningful ways?
• Is this view tested and refined through POE or other similar processes?

In this paper we briefly report on our findings and analysis. We discuss four topics:

• What is post-occupancy evaluation? What is its history and how has this contributed both to its potential for and difficulties in achieving organizational learning?
• Do organizations do POE-enabled organizational learning?
• How have organizations to created the appropriate conditions for learning through POE
• How have they create a knowledge base for building delivery and management?

Brief Introduction to Post-Occupancy Evaluation

Post-occupancy evaluation (POE) grew out of the extraordinary confluence of interests among social scientists, designers and planners in the 1960s and 1970s (see for example,. Early POE researchers were strongly interested in understanding the experience of building users and in representing the "non-paying" client . Many early POEs were conducted by academics focusing on settings that were accessible to them, such as housing, college dorms and residential institutions .

During the 1980s, many large public agencies developed more formal processes to manage information and decisions in their building delivery processes. As planning, facilities programming, design review and value engineering became more structured, agencies such as Public Works Canada and the US Postal Service added building evaluation as a further step in gathering and managing information about buildings .

This growth of POE occurred while politicians and policy analysts were advocating the evaluation of public programs more generally. Campbell and many others had been arguing at least since the 1960s that public programs could be treated as social experiments and that rational, technical means could contribute to, or even replace, messier political decision-making . A similar argument was applied to POE: statements of expected performance could be viewed as hypotheses that POE could test .

The term "post-occupancy" evaluation was intended to reflect that assessment takes place after the client had taken occupancy of a building; this was in direct contrast to some design competitions where completed buildings were disqualified from consideration, or other kinds of assessment such as "value engineering" that reviewed plans before construction. Some early descriptions focused on POE as a stand-alone practice aimed at understanding building performance from the users' perspectives. Some methodologists have advocated the development of different kinds of POEs, with different levels of activity and resource requirements . For example, Preiser advocated three levels of POEs: brief indicative studies; more detailed investigative POEs; and, diagnostic studies aimed at correlating environmental measures with subjective user responses . Whereas there was little agreement about specific methods and goals, most early POEs focused on systematically assessing human response to buildings and other designed spaces, using methods such as questionnaires, interviews and observation, sometimes linking these to physical assessment (Zimring, 1987).

Over the years, many theorists and practitioners have grown uncomfortable with the term "POE;" it seems to emphasize evaluation done a single point in the process. Friedmann, Zimring and Zube (1978) proposed the term "environmental design evaluation." Other researchers and practitioners have suggested terms such as "environmental audits" or "building-in-use assessment" . More recently, "building evaluation" and "building performance evaluation" have been proposed . Nonetheless, for historical reasons the term "post-occupancy evaluation" remains common and I use it in this paper for clarity.

Other discussions of evaluation emphasized the importance of embedding POE in a broader program of user-based programming, discussion and design guide development, proposing terms such as "pre-occupancy evaluation" , "process architecture" and "placemaking" . As early as the 1970s the Army Corps of Engineers conducted an ambitious program of programming and evaluation that resulted in some 19 design guides for facilities ranging from drama and music centers to barracks and military police stations . More recently, POE has been seen as part of a spectrum of practices aimed at understanding design criteria, predicting the effectiveness of emerging designs, reviewing completed designs, and supporting building activation and facilities management . With growing concerns about health and sustainability, several programs have also linked user response to the physical performance of buildings, such as energy performance or indoor air quality .

POE methodologists and practitioners have identified several potential benefits of POE . POE:

• Aids communications among stakeholders such as designers, clients, end-users and others;
• Creates mechanisms for quality monitoring, similar to using student testing to identify under-performing schools, where decision-makers are notified when a building does not reach a given standard;
• Supports fine-tuning, settling-in and renovation of existing settings;
• Provides data that informs specific future decisions;
• Supports the improvement of building delivery and facility management processes;
• Supports development of policy as reflected in design and planning guides;
• Accelerates organizational learning by allowing decision-makers to build on successes and not repeat failures.
  In this paper we primarily focus on the use of POE for improving organizational learning.

Do organizations do POE-enabled organizational learning?

As we discussed above, we reviewed materials from some 18 organizations that are currently doing POEs or who have done so in the past. In looking at organizations that have active POE programs, we found that members of project teams, including project managers, consultants and clients, tend not to be aware of POEs, unless a special evaluation has been conducted to address a problem that the team is facing. Where they are aware of the POEs, team members often do not have the reports from past POEs at hand and do not apparently use POE results in daily decision-making.

Mid-level staff tend to be more aware of the POEs. In particular, staff responsible for developing guidelines and standards are often aware of the POE results. For example, in the US Postal Service the staff who maintain guidelines also administer POEs; the POEs conducted by the Administrative Office of the US Courts are directly used by the Judicial Conference to test and update the US Courts Design Guide.

We were not able to find situations where senior management used POEs for strategic planning. POEs have the potential for supporting "double-loop learning" : for not only evaluating how to achieve existing goals better but to reflect on whether goals themselves need to be reconsidered. But we were not able to find cases where this actually occurred.

We were not able to find many compilations of POE findings, though several organizations such as the Army Corps of Engineers, US Postal Service, Administrative Office of the US Courts, GSA and others have incorporated POEs into design guides. Disney and the US Department of State have incorporated POE into databases of information. These are discussed in more detail below.

It does appear that POEs are not used for their full potential for organizational learning. There are two major reasons for this:

• Learning is fragile and difficult and many organizations have not created the appropriate conditions for learning. If learning is to be genuinely "organizational" a wide number of staff must have the opportunity to participate and to reflect on the results in a way that enables them to incorporate the results into their own practice. Potential participants must see the value for themselves: there must be incentives for participating. Also, evaluation will sometimes reveal that building performance does not reach a desired standard. This is, of course, the value of POE, but many organizations punish people when innovations do not work. In addition, many organizations simply do not make information available in a format that is clear and useful to decision-makers.
• Many organizations have not created a body of knowledge that is valuable in the sense that it provides a coherent, integrated body of knowledge that is helpful in everyday decision-making. Knowledge tends to be informal and individual.

Ways to create the appropriate conditions for learning through POE

Create broad opportunities for participation and reflection

Our research suggests that POE-based knowledge is not widely shared within most organizations. One way to achieve this sharing is through direct participation in evaluations. Seeing how a facility works and hearing directly from users is a memorable experience. And, the process of analyzing and writing up the results from an evaluation can help decision-makers reflect on the implications of the results and make links to their own practice.

A group of evaluators in New Zealand developed a "touring interview" methodology to allow decision-makers to actively participate in evaluations with little training and only modest commitment of time . For example, in an active program of evaluation with over 80 completed evaluations, Bill Watson, a consultant to public and private clients, takes building users on a tour of the building and asks open-ended questions such as "what works here?" as well as more specific probes about the functions of spaces and systems. POE reports are mostly verbatim comments by users and are sorted into categories such as "action for this building" or "change in guidelines for future buildings." This approach is quite inexpensive and can be completed with several person-days of effort. The experience is vivid for the participants and produces results that are imageable and articulate. It also allows participants to discuss relative priorities and values. However, because each touring interview group varies it is somewhat more difficult to compare evaluations of different settings

This kind of participatory evaluation can be an extension of existing processes for getting feedback from customers. Project managers in Santa Clara County (California) were tired of getting a storm of requests from users after they moved into a building. These were difficult to direct to contractors, suppliers and others. They contracted with consultants Cheryl Fuller and Craig Zimring to create a "Quick Response Survey" (QRS) aimed at organizing and prioritizing user needs about three months after buildings were occupied. All building users fill out a one-page questionnaire and project managers follow up with a half-day walkthrough interview of the building with the facility manager and staff representatives. The project managers would then prioritize requests and meet with the client organizations. The State of California Department of General Services is further developing the QRS and will have evaluators enter results into a lessons-learned database.

A lessons-learned program initiated in 1997 for New York City to examine the success of school projects in the state was aimed at participation by consultants. The School Construction Authority (SCA), whose membership is appointed by the governor, the mayor and the NY City Board of Education, was charged with the program. To get the program approved SCA, under the leadership of consultant Ralph Steinglass, adopted a simple methodology—require the architect/engineer of record to conduct the POE. The rationale was that this would guarantee that designers would confront how users responded to their designs, and force a lessons-learned loop in the design process. About 20 POEs have been completed. To ensure reliability, SCA reviewed the results before approving the POEs. In some cases, the architects or engineers had to re-schedule their interviews when they were suspected of introducing a bias, or continue their investigation if they failed to include critical areas required in the study.

The three programs described above involve evaluation by the people who designed and managed the project. As such, participatory evaluation is well suited to supporting learning by in-house project managers and consultants. Whereas the New Zealand projects are led by consultants, the quick response projects and the SCA projects are entirely conducted by the consultants or project managers.

Create incentives for participation

Most building professionals are interested in doing a good job and see value in POE. However, as personal time management consultant Stephen Covey has argued, things that are merely important often lose out to things that are urgent: general benefits such as long-term learning often lose out when building professionals are faced with the pressing matters of everyday life. When more specific incentives are offered Iit often increases participation in a POE program program when more specific incentives are offered.

The drug company Ciba-Geigy has used direct monetary incentives. The A/E firm HLW and the contractor Sordoni Skansa Construction put their design and construction profits ($300,000 and $1.2M) at risk based on performance on schedule, cost and user satisfaction for the new $39 million Ciba-Geigy Corp's Martin Dexter Laboratory in Tarrytown, NY. One-third of the profits was based on user satisfaction responses to 14 survey questions: HVAC, acoustics, odor control, vibration, lighting, fume-hood performance, quality of construction (finishes), building appearance and user-friendliness. The questions were binary-choice (acceptable-not acceptable) and the building had to reach 70% satisfaction to pass the test. Some aspects such as sound transmission were also assessed using physical measures; if the user satisfaction measures didn't reach criterion, physical measures could be substituted . The designers and contractors consulted the scientists throughout the process, showing them alternatives of the façade design and full-scale mockups of the range hoods. The building passed on all criteria other than satisfaction with the range hoods, which were modified after the evaluation as a response to user input. Whereas some aspects of this testing process might be questionable—should you evaluate and entire building on 14 yes-no scales? Should maintenance and operating experience be included?—this process gained from participation throughout. The design firm, contractors, management and scientists all participated in establishing the criteria at the outset and the financial incentive encouraged the contractor and designers to consult the users at every step in the process. It is difficult to document the learning benefit of this process but the contractor, Sordoni Skansa, has since used POE-based incentives in several other projects and has refined the way in which they deliver buildings.

The US General Services Administration and the California Department of General Services are planning to include the results of POEs as part of the review of qualifications when selecting consultants and contractors. This has strongly increased the interest in POEs by participating firms. We are unaware of any POE programs that provide incentives for internal staff members to participate in evaluations, though several programs have discussed such incentives, such as providing a free vacation day as a reward for adding data to the knowledge base or , providing a mini-sabbatical for participating in evaluations or lessons-learned program. Disney provides a powerful, if indirect incentive: knowledge. Only the industrial engineers have access to key data and they then become valuable members of the design team.

Reduce disincentives: Create protected opportunities for innovation and evaluation

Organizational learning consultants have long pointed to an inherent contradiction in many organizations. Whereas most organizations espouse innovation and learning they behave in ways that limit it. We recently participated in a meeting where an organization had used an innovative building delivery strategy. They were not familiar with it and had left out a key review step. When this became clear, a senior manager turned to the project manager and said: "We would have expected someone a your level to do better." The message to everyone in the room was clear: avoid innovation and avoid evaluation! This syndrome—focusing on the individual rather than the performance, blaming the innovator rather than learning from the innovation—is pervasive among organizations more generally . However, some building-delivery organizations have used POE to at least partially overcome it.

Some organizations have done this by explicitly sanctioning "research" with the attendant acknowledgment that innovations might not succeed. For example, the US General Services Administration (GSA) Public Buildings Service has recently appointed a "Director of Research." The first director, Kevin Kampschroer, has a budget to conduct, synthesize and distribute research, including POE. The use of the term "research" carries with it the understanding that not all efforts are successful and the budget provides some time for reflection about findings. To date, much of the research is conducted by academic consultants who bring in outside learning into GSA. However, GSA is also looking at ways to broaden internal ownership of the research program.

GSA has also created an active "officing" laboratory in its own headquarters building. The lab, supervised by Kampschroer, is one floor of actual workspace that includes an innovative raised floor heating, ventilating and air-conditioning system and several brands of modular office furniture systems. It also explores design to support teamwork, with many small conference rooms and meeting areas. The workers are frequently surveyed and observed, and the lab also becomes a place where clients can see alternative office layouts.

The US Courts and the US General Services Administration Courthouse Management Group are considering developing a different kind of laboratory: a full-scale courtroom mockup facility where new courtroom layouts and technologies can be tested and refined at relatively low cost and risk. This facility, to be constructed at Georgia Tech, would allow mock trials to be conducted and would provide training for judges, staff and lawyers.

Another way to reduce the personal and organizational cost of experimentation is by starting small with projects that have an experimental component. The innovation can be evaluated and considered for broader adoption. For example the US Department of State Office of Foreign Buildings Operations (FBO) tries out innovations on a limited number of projects before rolling out the innovation to the larger organization. For example, FBO has recently used Building Serviceability Tools and Methods for programming and design review for the new embassies in Dar es salaam and Nairobi.

The State of Minnesota Department of Natural Resources has used POE to evaluate two innovative regional centers. In each of these cases the organizational learning effort provides some additional resources for data collection and reflection as well as the clear designation that it was an innovative effort that might not be fully successful.

Although the prototype and laboratory efforts are quite different, all reduce the disincentives for innovation and evaluation by allowing innovation and evaluation at relatively low personal and organizational cost.

In many organizations it is risky to be the first one to try an innovation. MIT organizational consultant Edgar Scheiien has proposed that while organizations may benefit greatly from consultants they often find the experience of peers more helpful when they actually move to implementing an innovation.often learn best from peers rather than from consultants, at least when they move to implementing it. He Schein has called for "learning consortia" where people can get advice from others peers in other organizations and learn from their experience . He argues that while such learning consortia may be effective at all levels of an organization they are particularly effective among CEOs or upper-mid-level managers. Hopefully the Federal Facilities Council Symposium will be a first step in creating a consortium where organizations can reduce their disincentives by creating a network where organizations can share innovations.

Although the prototype, laboratory and learning consortium efforts are quite different, all reduce the disincentives for innovation and evaluation by allowing innovation and evaluation at relatively low personal and organizational cost.

Provide access to knowledge for different audiences

The simplest disincentive to using POE for organizational learning is when POE results are simply not available to decision makers. Many organizations produce POEs as case study reports that are not widely distributed. Part of this may be due to the history of POE, which has focused on single case studies, and part may be because of the perceived disincentives to distributing information that might be seen as critical of internal efforts or individuals. Some of the problem is the simple technical difficulty of distributing printed information and this has become a lot easier with the internet and intranet and virtual private networks. NASA makes it lessons-learned database available to all authorized staff and contractors. In the UK Adrian Leaman and Bill Bordass have created a interactive website for the 18 buildings they have evaluated as part of the PROBE project. Funded by the Building Research Establishment and Building Services Journal, Probe stands for Post-occupancy review of buildings and their engineering.

Some organizations have overcome some of these issues by creating design guides and databases of POE information. Agencies such as the Administrative Office of the US Courts, the US Postal Service, the US General Services Administration have created design guides that are broadly distributed.

As we have suggested, the problem with organizational learning is only partially technical. The tools for creating web sites and databases are now widely available and inexpensive. A useful web site requires the initiative to collect the information, the time to make sense of it and the will to share it.

Part of the problem with building delivery organizations and design projects is that they represent so many different professional cultures. Engineers tend to take a technical problem-solving approach. A while architects are often interested in form. Clients might be interested in the usability and experience of the building. Senior managers might be searching for help in setting strategic directions whereas project managers might be interested in lessons-learned about specific materials or equipment. Part of the challenge in creating any database or reports is translating between these different professional cultures, and evaluators have not always been successful at doing this.

Reduce uncertainty by upper management commitment

Participants in POE programs report that uncertainty about senior management's commitment to the program is a key disincentive to participation. Sometime the lack of commitment is seen in lack of resources but is often manifest in lack of visible endorsement for the program by senior management and lack of commitment to the 2-5 year time span necessary to see results in terms of organizational learning. Savvy staff have learned not to genuinely commit to the management's infatuation du jour, knowing that it will change quickly.

Create a knowledge base for building delivery and management

Most fundamentally, organizational learning for a building delivery organization is producing better buildings more effectively. Given the large number of POEs that have been completed—longtime researcher and Environment and Behavior journal editor Robert Bechtel estimates that over 50,000 have been completed—one would expect that there would have been many books and guides that synthesize the results of POEs and tie them into a coherent guide for key programming and design decisions. But such guides are relatively rare. In part, this is because of the focus of POE researchers and consultants on case studies. But much knowledge about buildings has been built up incrementally through negotiation on individual projects and programs. Organizations seldom take the time to identify the key strategic decisions that most affect their clients. However, efforts such as the ASTM Building Serviceability Tools and Methods have begun to do this for programming and portfolio management. But we have seldom done this for POE.

In this section we examine several strategies that have proven successful for beginning to create this kind of knowledge base. In several cases organizations have built on POEs that have been initiated for different purposes (or seem to us to be able to do so reasonably easily).

POE can be particularly successful in feed-forward if it links strategic facilities decisions to the "key business drivers" of the client organization. In the 1970s the US Army was shifting to an all-volunteer army. Potential recruits said that the aging facilities were a significant impediment to recruiting and retention and the Army sought to renovate or rebuild many of its buildings. To help guide the multi-billion dollar investment, the Army Corps of Engineers created a large program of participatory programming and evaluation, resulting in some 19 design guides .

In the 1980s, the newly-privatized US Postal Service (USPS) was losing customers to private competitors such as FedEx and UPS . Focusing initially on the customer experience with lobbies, the USPS contracted with Min Kantrowitz and Jay Farbstein and Associates to conduct focus group evaluations. This has led to a large and continuing program of evaluations and design guide development. New concepts of post office design are developed, such as the retail-focused "postal store," innovative projects are designed, the projects are evaluated and the ideas are refined then incorporated into design guides. This program has sustained an ongoing process of testing and refining the design guides through evaluation and experience. More recently, the USPS has de-emphasized on-site evaluations. According to POE manager Mark Nedzbala, most POEs now involve having facility managers fill out relatively brief mail-out surveys. Nedzbala has found that the open-ended responses to the questionnaire are often most valuable in refining the USPS' design guidelines because they are more specific than the scaled satisfaction responses.

Building on existing evaluations

An organization can begin to rationalize its knowledge base by building on evaluations that occur for other reasons. As the experienced evaluator Bob Shibley has said, evaluations are easiest to justify if they bring both project support and analysis benefits and lessons-learned benefits .

Building on diagnoses of troubled settings

Sometimes a building is the subject of complaints or controversy; a POE can help to diagnose the source of problems and prioritize solutions. For example, the new San Francisco central library was a landmark when it opened in 1996, but faced immediate controversy. Some of the initial programmatic assumptions continued to be debated—such as the wisdom of moving books to closed stacks to create room for computers. As a result, the mayor appointed an audit commission that recommended a POE, led by architect Cynthia Ripley and including the director of the Los Angeles library system. After interviewing staff and users, observing use and analyzing records, the POE team highlighted problems with wayfinding, flexibility and public access to books. The POE recommended detailed renovations to reorganize the stacks and collection . The POE is quite thorough in suggesting detailed specific changes and the basis of these recommendations could potentially be turned into planning principles. Most significantly, this raises issues of programming process where the (former) library director went against the recommendations of his planning committee to reduce access to books in favor of closed stacks. It can support broader reflection about the role of libraries and physical structures in providing to information in the age of computers.

Whereas the focus on understanding problems and failure provides a clear direction to POEs, and has a history in case studies of blast, earthquakes and other building failures, these kinds of POEs carry special risks of becoming ways to focus (or deflect) blame.

Capitalizing on evaluations of innovations

Evaluation can help decide whether innovative buildings or building components should be considered for additional capitol investment. For example, as was mentioned above, the State of Minnesota Department of Natural Resources (DNR) has recently changed the way in which it manages the environment. Rather than organizing their staff by discipline, they now use a matrix management system where decisions are made by a multidisciplinary group organized by eco-system. The DNR is creating new regional centers that include wildlife biologists, air and water specialists and others concerned with a given area. The centers are intended to encourage multi-disciplinary collaboration and to be very "green." The DNR contracted with a university team led by Julia Robinson to evaluate two of the initial projects. The team made numerous recommendations. The POE was included as an appendix for the funding request for the third center. This was the first time in DHR's history that a capital request was fully funded on the first attempt, and the DHR was told that the POE was a major reason: it showed a high level of understanding of the project. This result provided an important incentive for DHR as an organization. However, the project also raised some issues about sustainability and the internal staff did not feel that they had been fully consulted in the POE process. An additional team was hired to create design guidelines in close consultation with staff . Issues like such as sustainability, which are undergoing rapid change, are particular candidates for "double-loop" learning where both goals and methods for achieving them are being developed, if appropriate conditions are established for discussion, reflection and action.

Focus on "learning moments"

The Administrative Office of the US Courts (AO) conducts a POE program that informs design guidelines (the US Courts Design Guide). However, the AO has achieved feed-forward by linking the design guide to a "strategic learning moment" in the development of courthouses: the negotiation between judges and the building agent (the US General Services Administration) about the scope and quality level for new courthouses. In the early 1990s the US Government initiated the largest civilian construction program since the Second World War, projecting to spend over $10 billion on 160 new courthouses. (The creation of new judgeships in the 1980s, concerns for increased security and new technologies all necessitated new courthouses or major renovations.) However, both the judiciary and GSA were being criticized by Congress for creating marble-clad "Taj Mahals." The AO initiated the POE Program to identify necessary changes to the standards in the first edition of the Design Guide, to defend the judiciary against attack by documenting the efficacy of the design standards, and to inform the negotiation about issues such as the dimensions and materials of courtrooms and chambers. Information from POEs was also used in training workshops for judges and staff who were becoming involved in new courthouse design and construction. This program is run by the AO but the design guide is actually created and vetted by a committee of the Judicial Conference, the group that sets broad policy within the federal judiciary. This program is quite unusual: it is the only case that we are aware of where a POE and design guide are developed by a client organization that does not build its own buildings.

The Minnesota Department of Natural Resources project also focuses on strategic moments: especially the approval of the funding package by the key legislative committees.

The focus on strategic learning moments is similar to Shibley's reminder that information is most likely to be used when it's asked for . A strategic learning moment is a critical time when information or a POE can help resolve a problem or issue that is of considerable importance to the participants. The focus on learning moments can also be used in developing policy documents or targeting POEs toward decisions.

Creating organizational memory for precedents

A key part of organizational memory is simply knowing what the organization has done. But few POE programs have been linked to recording and analysis of precedent. There is a real opportunity to link evaluation to a record of past projects. This record can include simple plans and photos and some analyses of cost, size and materials. These descriptions can be linked to evaluations.

Lessons from POE programs: Enhancing feed-forward

We have suggested that POE has a large potential for lessons-learned as well as for project support and analysis. Because of the historic focus of much POE research and the difficulty of finding resources for organizational learning and sensitivities in exposing problems, relatively few organizations have created effective POE-enabled organizational learning systems that include all four components of:

• Monitoring and assessing performance;
• Interpreting and discussing the implications of results;
• Consolidating results into an organizational memory;
• Widely distributing findings and conclusions.

Despite the difficulties, we have found that some building delivery organizations have created the conditions for learning:

• Create opportunities for participation
• Add incentives
• Remove disincentives
• Provide access to information
• Provide upper-level management support

And for creation of knowledge:

• Clarify key strategic choices
• Build on existing evaluations
• Focus on strategic moments
• Record precedents

The lessons from these 18 POE programs, which influence billions of dollars of construction, suggest that the solution to creating a lessons-learned program is partly technical, such as using information technology to reduce costs in gathering and distributing information. But, at its core the problem is organizational, creating a setting where decisions can be evaluated, discussed and learned from.