Keynote address for Inspire’96, European Software
Institute, Bilbao Spain, September 1996
Closing the feedback loop
Essence and accident of software engineering education
Daniel M. Roy
Software Technology,
Process & People
20 Forest Rd. Bradford
Woods, PA 15015 USA
Visiting scientist,
SEI
Abstract: In a landmark paper
written over 10 years ago, Frederick Brooks discussed the inherent difficulties
of engineering software and examined
various possible solutions to the software crisis [Brooks-87]. Since
then, a volley of bullets and other
projectiles of various silver contents have been fired amidst beaucoup noise
and smoke.
This talk explores
some of the advances in software process, technology, and applied psychology
that have produced measurable positive results in software engineering
practice. It is proposed that further
advances hinge on the disciplined application of a self improvement paradigm
that scales up from the personal, to the team, to the organizational level.
Already, a shift
can be noticed in software engineering education toward a more disciplined
approach based on the experimental paradigm of the scientific method. It is
proposed that the same principles can be applied to the very education and
training of software practitioners.
The evolution of
organizational paradigms from Taylor to McGregor, to Lewin and Weisbord is used
to show that our greatest challenge goes even beyond inspiring. In these times
of fast paced and incessant change, it is proposed that the essence of our
mission is to coach our students into inspiring themselves for their entire
professional life.
Essence vs. accident
‘The problem of
quality management is not what people don’t know about it. The problem is what
they think they do know... In this regard, quality has much in common with sex.
Everybody is for it. (Under certain conditions of course.) Everyone feels they
understand it. (Even though they wouldn’t want to explain it.) Everyone thinks
execution is only a matter of following natural inclinations. (After all, we do
get along somehow.) And, of course, most people feel that problems in these
areas are caused by other people. (if only they
would take the time to do things right.) [Crosby-79]
This rather... unorthodox analogy
by one of the founding fathers of the quality movement seems applicable to
software education and training as well. In particular, both industry and
academia may very well feel that software expertise problems are caused by the
other. Interestingly, it is a well known characteristic of immature
organizations to blame the people when they should be fixing their process. Do
industry and academia have a formal mechanism, a cooperative process, to fully
prepare students for their professional life and keep their skills current and
sharp thereafter? If product quality is directly dependent on the quality of
the process, what about the continuous process of education and training
itself? As Weinberg puts it in his landmark book about the psychology of
programming: “Good programmers are made not born; therefore, we should turn our
attention to the manufacturing, or training process.” [Weinberg-71] Like with
other processes, the first step in doing this is to establish a baseline of
known strengths and difficulties.
In his famous paper, Brooks
borrowed from Aristotle’s philosophy to divide the difficulties facing software
practitioners “...into essence, the
difficulties inherent in the nature of software, and accidents, those difficulties that today attend its production but
are not inherent... Past progress has so reduced the accidental tasks that
future progress now depend upon addressing the essence.” [Brooks-87]
I propose that the same kind of
distinction (between essence and accident) can be made with the problems facing
software engineering educators in university and industry alike. If this holds,
then the same kind of discussion and potential solutions may apply to fostering
continuous progress in our field. By essential difficulties, I mean the
difficulties inherent in the nature of communicating software engineering
knowledge. The accidental difficulties are those that attend the production and
delivery of software engineering courses but are not inherent. I will
concentrate on the essence.
Why it is hard - Essential Difficulties
Following Brooks, I can see four
categories of reasons why teaching software engineering is and will remain
inherently difficult (I borrowed the first one from Mary Shaw):
1.
Immaturity of
computer science
2.
Complexity of interlocking concepts
3.
Conformity to a broad range of requirements
4.
Changeability of the technology
Immaturity of computer science
Mary Shaw makes the point that
computer science is still too immature to meet current engineering needs
[Shaw-90]. To show this, she distinguishes five phases in the evolution of any engineering (i.e. scientific)
discipline:
1.
Ad hoc solutions:
practices are entered in the folklore if they work
2.
Systematic
folklore: some folklore analysis is made, some procedures and heuristics
are written down
3.
Codification:
crisper procedures and rules are systematically written down and passed along
4.
Models and
theories: experiments are made to validate models, practice is improved by
using models
5.
Experience from
practice: new theories and principles are continuously discovered,
validated and put into practice.
Mary Shaw argues that even though
some part of computer science such as basic “algorithm, data structures, and
compiler construction theory can be considered mature, current demands is far
ahead of the scientific base” [Shaw-90]. Clearly, the electronics engineer and
the chemist for instance may draw on centuries of scientific developments in
physics, mathematics, and chemistry in their daily practice. More importantly,
it is by building on this solid scientific base that new models and theories
continuously enrich their field. The software engineer often has to be content
with commonly accepted practices (at
best up to codification in Mary’s model). This, in turn, poses a formidable
problem to the educator. On what science can we base teachings that will endure
the test of time?
Complexity of interlocking concepts
According to Brooks,
“the essence of a
software entity is a construct of interlocking concepts: data sets, relationships
among data items, algorithms, and invocations of functions. I believe the hard part of building software to be the
specification, design, and testing of this conceptual construct, not the labor
of representing it and testing the fidelity of the representation.” [Brooks-87]
In the field of education, the
complexity of the interlocking concepts that we must introduce, is compounded
by the three dimensions of technology, process and people. In their outstanding
book on software technology innovations, the experimental scientists of IBM
Santa Theresa laboratories remark that,
“in general, a
quality strategy works best if the leadership, process, and technology
innovations are in balance. A program that is overly skewed to one at the
expense of the others can produce dysfunctional results. For example, too much
leadership can result in a dictatorial group. Too much process emphasis can
lead to bureaucracy. Too much technology can lead to technical virtuosity
without a consistent direction.” [Kaplan-95]
This concept of balance is
fundamental. I like to compare it to the concept of impedance matching in
electronics. Already, experiments teaching process in the classroom have shown
the importance of this concept of balance
in practice. I have felt the difficulties to implement this concept with
students of the studio project for the masters of software engineering at CMU,
I have seen teams of my PSP students struggle with it at SEI as well as in
industry, and others have reported the same kind of findings. For instance, in
discussing the results of student team projects at the last CSEE conference,
Robillard et al. confessed:
“We were lured by
the ability of the students to deliver process oriented documentation. It seems
that team cohesion and communications is better than process rules to obtain a
good product.” [Robillard-94]
Teaching the optimum balance of
technology, process and human aspects of
the complex interlocking concepts of software engineering is and will
remain inherently hard.
Conformity to a broad range of requirements
Another essential complexity of
software education lies in the broad range of conflicting requirements that are
constantly being hurled at us.
We claim to be teaching analysis,
design, coding, testing, management, teamwork, process, programming languages,
etc. for any domain of application, all in one swoop. Do they teach masonry in
architecture schools? Our broadside approach is at least in part a consequence
of the immaturity of our field. Other professions, say electronics, make a
distinction between specialties (from various grades of technicians to
certified engineers) and in different domains such as microelectronics, power
systems, microwaves, digital electronics, and so on. At the very least, while
teaching design for instance, we should follow Budgen’s advice to take into
account the difference between “the minority of students who will have to produce designs and the majority that
will simply have to comprehend them”
[Budgen-95]. This remark applies to other phases of the life cycle, and to
teaching process concepts as well.
Furthermore, in teaching software
engineering, the usual tension between theory and practice, is compounded by
the requirement to balance whatever science we can muster, with learning the
programming language, OS, or process model, currently popular in industry. Even
though it makes sense to give our students the best chance at finding a job,
there is a danger here for academia to abdicate our mission of education for
the reinforcement of what has been called ‘lemming-gineering’. In my view, the
lemming-like behavior of blindly espousing whatever is in demand at the time is
responsible for the slow adoption of
better and proven solutions in industry. It is Lemming-gineering that
makes industry mandate C for all applications. It is lemming-gineering that
makes us cave in to the ‘demand’ and use the C language instead of Ada to teach
good programming practices. In so doing we show the wrong example; we turn our
students into leming-gineers. No wonder they carry this learned behavior on the
job, and when they are promoted to management, they end up slowing down the
progress of their organization.
Striking the right educational
balance to meet these numerous evolving requirements without producing leming-gineers
is and will remain, for the foreseeable future, an essential (albeit mainly
political) difficulty of our job.
Changeability of the technology
We often hear how slow our
progress is compared to the neck breaking speed of hardware engineering. Brooks
observes that “...the anomaly is not that software progress is so slow, but that computer hardware
progress is so fast.” [Brooks-87] Indeed, progress in software technology is
fast enough that teaching evolving methods, languages, operating systems,
networks, tools, and so on require vigilantly tracking a multitude of moving
targets. Blending process concepts, and psychological factors, with this
technology, all in careful balance, make the evolution of curriculum a
formidable challenge. The pace of evolution in all three dimensions of
technology, process and people issues is such that the half life of a software
engineer education is probably less than three years and it is clearly
decreasing.
Since knowledge and technology
are both fueling the knowledge inflation, this essential difficulty of our
mission, can be expected to get worse. This is already having a profound effect
on the value of our degrees, the meaning of education, and the role of
educators in the work place.
Past progress - Accidental difficulties
Like in software engineering,
past progress has been achieved by addressing the accidental complexity of the
field.
Software engineering vs. computer science curriculum
For instance, the ACM/IEEE-CS
joint curriculum task force [JCTF-91] has produced documents that help in the
production and delivery of undergraduate software education. It has also
generated a healthy debate. Lin Zucconi provides an excellent summary of the issues in [Zucconi-95]. She states that
the proposed curriculum
“...is overwhelmed
by the CS component and should include:
SQA and related activities such as CM
software testing-theory and practice
software maintenance
software measurement
software safety, reliability, security
and integrity
systems engineering
a practicum of a minimum of one
semester in industry or an industry-like setting
a minimum of a full year team
project-oriented course during which software life cycle issues of process
definition and product development and production are experienced first hand.”
Addressing most of these issues,
the CMU MSE core curriculum [MSE-95] is organized along discipline lines
instead of along life cycle activities. It achieves a good balance between
theory and practice and is taught in 5 semester courses, elective/directive
courses and the studio team project to cover:
Models of software systems
Methods of software development
Management of software development
Analysis of software artifacts
Architectures of software systems
Advanced curriculum supported by
teamwork on real projects (more along
the line of French ‘grandes écoles’ than universities) are indeed becoming
increasingly popular. One can cite the SE integrated curriculum at Embry Riddle
Aeronautical University, the Real World Lab at the Georgia Institute of
technology, the SE curriculum at Rochester Institute of Technology, the
capstone software engineering course at the University of Alabama, and the
master of software engineering at Carnegie Mellon University to name a few. It
is only when the concepts are practiced on real projects that students can
judge their own progress. This in turn helps to close the feedback loop between
instructors and students, curriculum and evolving industrial needs. I believe
this principle to be central to finally addressing the essence of software engineering education.
Academia-industry cooperation
In the paper quoted earlier
[Zucconi-95], Lin Zucconi also lists
“...five areas of
knowledge and capability necessary for the software engineer to achieve broad
competency:
1.
Systems and software engineering and computer science
2.
Hardware platform-specific knowledge (hw, OS, network,
etc.)
3.
Application domain
4.
Personal and interpersonal skills, ethics and
5.
Business culture (better learned by experience with the
help of a mentor)”
Clearly, such an education cannot
be offered without tight University-Industry cooperation. Lin views
universities as having primary responsibility for the first 2 skills, industry
for the last 3 with some shared and complementary responsibilities for skills 3
and 4.
In fact, active collaboration
between industry and universities from traditional education to full-fledged
partnerships is happening [Carpenter-95]. Examples are the Software Engineering
Laboratory (SEL) at NASA Goddard Space Flight Center (GSFC), the Research
Institute for Computing and Information Systems (RICIS) in Texas, the Software
Center at Embry Riddle Aeronautical University and the University of Southern
California Center for Software Engineering under Barry Boehm.
Such cooperation is vital to
closing the feedback loop between the producers of software engineers and their
customer/partners in industry.
Addressing the essence: Closing the feedback loop
In his book on instructional
design Gagné lists the external factors that foster effective learning
[Gagné-79]:
Contiguity- Stimulus and desired
response must be contiguous in time
Repetition- Necessary but not
sufficient for true learning
Reinforcement- Not only a reward for
following desired behavior but in the sense of Premak’s definition “A new act
(N) is most readily learned when it is immediately followed by an old act (O)
which the individual performs readily, in such a way that doing O is made
contingent on doing N.” [Premak-65]
He also notes the internal
factors that may be influenced by the instruction (fig. 1):
Factual information (and knowing where
to find it)
Intellectual skills (such as those
needed for Algebra)
Cognitive strategies (knowledge
storage and retrieval, self learning)
Inner drive (attitude, self confidence
and motivation)
Figure 1: External
and internal learning factors
The need for feedback
I believe that educators will
increasingly bring to bear all external factors to particularly emphasize self learning
and inner drive in their students, helping them close the feedback loop on
themselves through real-world engineering assignments. In a paper given at CSEE
94, Melody Moore calls on cognitive psychology to make the point:
“In an influential
book, Donald Schon [Schon-83] presents evidence that ...expertise is the
interplay of two competencies: core competencies that permit the practitioner
to act and respond effectively in familiar problem situations, and reflective
skills that let the practitioner reason about his/her skills and knowledge when
the most immediate course of action seems likely to be unfruitful.... Our
position is that software engineering educators have a responsibility, to
accelerate that learning process and that this can best be achieved by means of
integrating project classes into the curriculum.” [Moore-94]
Needless to say, such assignments
won’t be easy to grade. How do we reconcile Schon’s principles of ‘reflection
in action’ with the student’s inner drive to get ‘good grades’? How do we
reconcile structured learning and free discovery? The danger is great that we
will end up encouraging students to prefer book work or playing the teacher’s
‘hot buttons’ instead of concentrating
in creative work such as analysis and design. And how do we reward leadership
and risk taking in the face of project failure? Weinberg remarks that
“The fear of new
things, the expectation of failure, and the reluctance to admit weakness all
have a direct retarding effect on learning, whether in a formal classroom
situation or on the job...” [Weinberg-71]
Closing the feedback loop between
teacher and student, and between the industrial customer and the educational
institution will prove central to the learning experience of all parties
involved. We must nurture an environment where students are encouraged to admit
and openly discuss their errors. Only then can we expect to beat cognitive
dissonance at its own game[1]
because only guilt free coaching and sensitive feedback can lead to true
egoless programming in the classroom and on the job.
Weinberg remarks “It is a well
known psychological principle that in order to maximize the rate of learning,
the subject must be fed back information of how well or poorly he is doing.”
[Weinberg-71] Indeed, experiments have repeatedly shown that people who know
they are being judged but are not sure exactly how, will try variations on
their performance to get some feedback even at the risk of a poor evaluation! I
remember doing this myself as a student. I even tried this later at SEI with
various degree of success!
Experience factory: the outer feedback loop
A most powerful closed loop
improvement paradigm is Vic Basili’s experience factory (Fig. 2). Here, two
complementary organizations work in synergy to the fulfillment of the long term
and short term goals of the company. The project organization’s goals are
pragmatic and short term: develop the product on cost and on schedule with the
minimum of overhead. The experience factory organization’s goals are every bit
as pragmatic but also long term: help the project organization select the right
process, tools and components so that they do not re-invent the wheel, provide
developers with real-time feedback on the state of the project, and be the
repository of all project data and lessons learned to do even better next time.
This repository is the experience base. In practice it consists of reports,
models, metrics data bases, memos, handbooks, libraries, process elements, and
other artifacts that collectively constitutes a true corporate memory.
[Basili-94]
Figure 2: The
experience factory
This closed loop improvement
paradigm was most successful in practice as demonstrated by the software
engineering laboratory (SEL) at the NASA Goddard Space Flight Center (GSFC).
The SEL is a joint venture of the University of Maryland, Computer Sciences
Corporation, and the flight dynamics branch at GSFC. Over a period of 15 years,
this 200+ organization doubled its productivity and divided by 4 its number of
post delivery errors on real and complex projects, at a cost of 11% of its
total development effort. This achievement was recognized in 1994 with the
award to SEL of the first IEEE process achievement award. [McGarry-94]
By establishing specific goals, documenting
hypotheses and quantifying expectations, by treating each project as an
experiment, by measuring the results objectively to compare them to
expectations, and by documenting what was learned, the experience factory
brings the scientific method to software engineering. Indeed the SEL status
report offered each year at the Software Engineering Workshop at GSFC shows
that new models, “new theories and principles are continuously discovered,
validated and put into practice” within the SEL, as described by Mary Shaw at
the highest level of her model. [Shaw-90]
Furthermore, one of the most
interesting ideas behind the experience factory is the movement of people and
ideas between the factory proper and the project organization. The scientists
join their engineering colleagues in the project to help implement previous
knowledge and new concepts. During the post mortem phase, engineers join
scientists to write down lessons learned, updating process and enactment data.
This circulation of people and ideas drastically reduces the endemic resistance
that so often hampers software improvement in practice.
I believe the concepts of
experience factory to be eminently applicable to establishing a truly
cooperative process between industry and academia. I believe that only such a
tight cooperation will allow us to tackle the essential complexity of both
software engineering and software education.
PSP: the inner feedback loop
At the other end of the scale,
that of the individual engineer, Watts Humphrey offers a novel paradigm to
learn disciplined software engineering by developing programs and other
software work products in a closed loop framework.
This paradigm detailed in Watts’
latest work on the Personal Software Process (PSP) [Humphrey-95] is strikingly
similar to that of the experience factory. While the PSP is described as
downscaling the Capability Maturity Model (CMM), I believe that it can also be seen as a downscaled experience
factory [Roy-94], as an ‘experience workshop’ (fig. 3).
Like with each project in the
experience factory, each PSP exercise embodies a specific set of improvement
goals. The student applies a well defined process to derive a plan to achieve
these goals and then executes the plan. All along the student enacts the
process and collects data (such as time spent by process phase, defects
injected and corrected, etc.). Finally, a post mortem is performed at the end
of the exercise. Lessons learned are contributed to a private experience base
(PSP data, reports, process elements) to feed the next cycle of improvement.
This effectively closes the feedback loop on the student himself [Roy-96].
Note that the PSP measurement
framework, upward compatible processes and contiguous feedback, cover all
external and internal factors listed by Cagné (fig. 1) with an emphasis on
inner drive and self improvement. Furthermore, by emphasizing process
improvement, the PSP also directly implements Schon’s principles of reflection
in action.
Figure 3: The PSP:
Downscaling the experience factory?
As an example of the power of the
PSP closed loop paradigm, figure 4 shows the effectiveness (measured in number
of defects removed per hour) of design reviews, code reviews, compilation, and
unit test for all ten exercises of the PSP and for all students of a class I
taught at the Lawrence Livermore National Laboratory in California. The data
shows that reviews are typically 2 to 4 times more effective than test in
removing defects.
The effect on students is
striking. Seeing such results on their own data convince them forever not only of the value of personal
reviews, but (in the light of experienced cognitive dissonance) of the
importance of walkthroughs and formal inspections as well.
Figure 4: Reviews vs.
test LLNL data
A possible future
The path seems clearly marked
from the PSP to the CMM, and from the PSP to the experience factory. I believe
that further advances in both software engineering and its attending continuous
education hinge on the disciplined application of a self improvement paradigm
that scales up from the personal, to the team, to the organizational level.
Here again, we can borrow from applied psychology to sketch a probable
evolution of our field.
We have known since Frederick
Taylor that a rigorous process based on the scientific method can improve
productivity while reducing stress in the work place. We have known since Kurt
Lewin about the importance of learning by doing and doing by learning, the
cornerstone in fact of Lewin’s learning organization. We have known since
Douglas McGregor that people will exercise self control toward objectives they
feel committed to. What is next?
In his marvelous book on the
evolution of industrial organizations [Weisbord-88], Marvin Weisbord sheds a
brilliant light on the evolution of management paradigms (figure 5).
Figure 5: The
evolution of industrial organizations
Starting at the turn of the
century with Frederick Taylor we have the myth of the omniscient expert solving
engineering problems piecemeal. I see the analogy with our field as a teacher
running the classroom according to theory X. In the 50’s, participative
management and McGregor’s Theory Y make everyone attack problems piecemeal
albeit in teams. I remember such a situation in 1970 when, as a EE student, I
participated in the creation of our bio medical electronics section. The next
development occurs when experts start looking at the entire system, similar in
a way to what we are doing ourselves today at this conference. This is followed
by the collective study and resolution of system wide problems along true
democratic principles (third wave management).
Pursuing the analogy, I believe
that the total involvement of the educator, students, and the industrial
customer in the creation and continuous evolution of the software engineering
curriculum is the future of our field and the key to a mature software
engineering profession [Ford-96].
The stakes
The creation of software
engineering institutes in nearly all major countries, the funding of software
centers of excellence and process
improvement efforts in corporations of all sizes are signs that the continuous
improvement of software engineering has become a strategic concern for
governments and corporations alike. Marvin Weisbord describes the stakes in
these terms:
“Informed self
control, not close supervision is the only way to operate new technologies....
Knowledge and skills can’t be pumped into people the way traditional schools
have done it. They can be mastered only by applying theory directly on the job,
to real problems, here and now. That requires the learner’s direct
involvement... We better fight for the integration of social, technical and
economic change. Otherwise, we surely will put ourselves on the losing side of
the knowledge revolution.” [Weisbord-88]
Educators, like managers, are
seeing their role evolve away from directing, toward mentoring when in school
and coaching thereafter. But in these times of fast paced and incessant change,
our role as educators goes even beyond coaching. Since we are on the first line
of this knowledge revolution, the essence of our mission is to inspire our
students into inspiring themselves for their entire professional life.
The closing sentences in Watts Humphrey’s
last book (A Discipline for Software Engineering) offer a powerful example
of such an inspiring message. It is
with this quote that I conclude each of my PSP courses. Frankly, I don’t see
what I could add to this:
“Deciding what you
want from your chosen field is like asking what you want from life.
Surprisingly often, people achieve their objectives, but in ways they did not
expect. Life rarely turns out the way we plan. While our carefully developed
strategies may go down in flames, a new and more rewarding opportunity shows up
in the ashes. The key is to keep an open mind and to keep looking. In life, we
all reach the same end, so we need to concentrate on the trip. Just as with a
process, once you decide how you want to live, the rest will follow. Devote
yourself to excellence, and you just might achieve it. That would be worth the
trip”. [Humphrey-95]
Bibliography
[Basili-89] Victor Basili,
`Software Development: A Paradigm for the Future’, Proceedings of the
thirteenth Annual International Computer Software & Applications
Conference, Orlando, FL, September 20-22, 1989.
[Brooks-87] Frederick P. Brooks,
‘No Silver Bullet: Essence and Accidents of Software Engineering’, Computer,
April 1987 (first published in Information
Processing’86)
[Budgen-95] David Budgen, ‘Is
Teaching Software Design a “Wicked” Problem Too?’, proceedings of the 8th SEI
CSEE Conference on Software Engineering Education, New Orleans, LA, March 1995,
Springer.
[Carpenter-95] Maribeth B.
Carpenter, ‘Directory of Industry and University Collaborations with a Focus on
Software Engineering Education’, special report CMU/SEI-95-SR-011, SEI, March
1995.
[Crosby-79] Phil Crosby, ‘Quality
is Free’, McGraw-Hill, 1979.
[Ford-96] Gary Ford, Norman E.
Gibbs, “A Mature Profession of Software Engineering’, technical report
CMU/SEI-96-TR-004, January, 1996.
[Gagné-79] Robert M. Gagné and
Leslie J. Briggs, ‘Principles of Instructional Design’, Holt, Rinehart and
Winston, 1979.
[Humphrey-95] Watts S. Humphrey,
`A Discipline for Software Engineering',
Addison Wesley, 1995.
[JCTF-91] Allen B. Tucker, Ed, ‘A
Summary of the ACM/IEEE-CS Joint Curriculum Task Force Report- Computing
Curricula 1991’, CACM, Vol. 34, No 6, June 1991.
[Kaplan-95] Craig Kaplan, Ralph
Clark, Victor Tang, ‘Secrets of Software Quality’, McGraw-Hill, 1995.
[McGarry-94] Frank McGarry et
al., ‘Software Process Improvement in the NASA Software Engineering
Laboratory’, technical report CMU/SEI-94-TR-22, December 1994.
[Moore-94] Melody Moore and Colin
Potts, ‘Learning by Doing: Goals and Experience of Two Software Engineering
Project Courses’, Proceedings of the 7th SEI CSEE Conference on Software
Engineering Education, San Antonio, TX, January 1994, Springer-Verlag.
[MSE-95] David Garlan et al.,
‘The CMU Master of Software Engineering Core Curriculum’, proceedings of the
8th SEI CSEE Conference on Software Engineering Education, New Orleans, LA,
March 1995, Springer.
[Premak-65] D. Premak,
‘Reinforcement Theory’, Proceedings of
Nebraska Symposium on Motivation, Lincoln: University of Nebraska Press, 1965.
[Robillard-94] Pierre N.
Robillard, Jean Maynrand, Jean-Normand Drouin, ‘Process Self Assessment in an
Educational Context’, Proceedings of the 7th SEI CSEE Conference on Software
Engineering Education, San Antonio, TX, January 1994, Springer-Verlag.
[Roy-94] Daniel M. Roy, ‘The PSP:
Downscaling the Factory?’, proceedings of the nineteenth annual Software
Engineering Workshop, GSFC, December 1994.
[Roy-96] Daniel M. Roy, ‘The PSP:
An Ego-Centered SPI Paradigm’, proceedings of the 1996 SEPG conference,
Atlantic city, May 1996.
[Schon-83] Donald A. Schon, ‘The
Reflective Practitioner: How Professionals Think in Action’, Basic books, 1983.
[Shaw-90] Mary Shaw, ‘Prospects
for an Engineering Discipline of Software’, SEI, September 1990. See also, IEEE
Software, November 1990.
[Weinberg -71] Gerald M.
Weinberg, ‘The Psychology of Computer Programming’, Van Nostrand Reinhold,
1971.
[Weisbord-88] Marvin R. Weisbord,
'Productive Workplaces - Organizing and Managing for Dignity, Meaning, and
Community', Jossey-Bass, 1988.
[Zucconi-95] Lin Zucconi,
‘Essential Knowledge for the Practicing Software Engineer and the
Responsibilities of University and Industry for Her Education’, proceedings of
the 8th SEI CSEE Conference on Software Engineering Education, New Orleans, LA,
March 1995, Springer.