Software Engineering came in when the Software Crisis became an issue and the idea was to put more Engineering into software rather than ad hoc programming, coding or hacking. It turns out that the part of the work of writing the line of code is just a small part of the job, and the quality of the product has a lot to do with everything else that goes on around the writing of the code or programming. Also eventually degrees in computer science became the way to get into the field more and more so that people were trained like engineers are trained to produce software more and more all the time. It was also the influence of the Software Engineering Institute whose purpose was to bring more discipline to software development. They came up with the CMM and then the CMMI and started appraising organizations as to their software maturity, and you could not get a contract with the government unless you were appraised to be of maturity level Three at least. This has not spread to Systems Engineering and Hardware Engineering despite much resistance. The resistance was so great that it spawned Agile and then Lean approaches which were directly opposed to the Software Process approach espoused by Watts Humphrey based on the work of Deming. Agile and now Lean have attempted to throw off the process yoke, but many of those issues that caused Software Engineering to be founded, to make Software production more than a craft are coming back with Agile at Scale issues. In other words if you are creating web sites you just do not need to do things that a traditional engineering approach would say are necessary, like Requirements Engineering, Architectural Design, High Level Planning, Documentation, etc. But if you are creating larger systems where multiple teams have to cooperate and synchronize and collaborate then these things become necessary again in some form, so people are working out the Agile way to do these things that were first introduced under the guise of engineering. But actually Hardware Engineers never did the things that were developed in Software Engineering until they were forced to by the CMMI when it was applied to everyone. The point is no matter what you call it, creating software is a special kind of craft which is hard to raise to commercial and production standards.
The reason why, which I go into in my paper on Software Engineering Ontology that is part of my book Wild Software Meta-systems (see http://works.bepress.com/kent_palmer) is that software has a special ontological status. There are different meta-levels of Being, and these have been described in Continental Philosophy much to the chagrin of Analytical Philosophers. These are as follows:
Pure Being — Parmenides Static
Process Being — Heraclitus Dynamic
Hyper Being — Plato’s Third kind of Being in the Timaeus, DifferAnce of Derrida
Wild Being — discovered by Merleau-Ponty in The Visible and InVisible
Ultra Being — The existential core of Being
What I discovered when I finally understood how CPUs work which I learned by programming designing and programming real time systems in assembly language was that they are based on the interaction of pointers and accumulators, and thus exemplified the pointing of Pure Being and the Grasping of Process Being that Merleau-Ponty talks about in the Phenomenology of Perception where he tries to make more concrete the present-at-hand and ready-to-hand of dasein in Heidegger’s Being and Time. It turns out that hardware embodies the first two meta-levels of Being, and that means that Software is the embodiment of the third meta-level of Being, i.e. Hyper Being, e.g. Differance of Derrida which is an interval of differing and deferring, or what Paul Simon calls “slip-sliding away”. We get this analogy when code is compared to nailing jelly to a tree
Object Oriented Design
Code is something very malleable because it has very few physical limits and most of its limits are discrete mathematical or to do with relativistic space and time constraints, or underlying computing machinery constraints. Basically you can do almost anything, represent almost anything in code if you are clever enough. And if you cannot do it in a given language you can invent another language that allows you to do whatever it is you want to represent. If you cannot directly do something in code then most likely you can simulate it. In code because the same formalism represents both the Turing machine and the Universal Turing machine (operating system) it means that there is a degree of freedom in the production of source code representations that is normally just not there in most material things. It is interesting that the difference engine of Babbage which Lady Ada programmed at least conceptually was based on a Jacquard Loom by which patterns in fabric were mass produced. Basically any kind of pattern you want to produce can be produced by our modern difference engines that we call computers with software in them acting as little conceptual machines that control the hardware. We can think of software as basically a state machine and a stack (read tape) that manipulate pointers and execute hardware operations on operands in accumulators. We impose on this micro-dance that weaves together patterns into larger patterns many layers of conceptualization until we get up to a software design that we can understand which is doing something for us and providing some accordance in our world we would not have otherwise, like the cell phone in our hand or the world wide web, or a working ATM machine.
But when we think about software itself what comes to mind eventually is that software can rewrite software transforming it, and it is in this case where it starts to become like jelly. In some sense the nails of our design concepts have a hard time pinning it down and so we develop an object oriented approach and package the jello in boxes and use it that way.
Now the interesting thing is that software is the only object in our culture that embodies Hyper Being. And what is amazing is that no one appears to notice the emergence of a new kind of cultural object that never before existed, until Lady Ada put her pin to paper and tried to figure out how to program the difference engine. Notice that Differance (differing and deferring) runs on a difference engine. Our calling them computers after the ladies who used to do calculations by hand and whose jobs were taken over by these mechanical beasts because if there is no error in the program then the answers are far more accurate than any computationalist could manage. Difference engines are actually a better name for them than computers. They weave threads of pointer and accumulator operations together one use of which is to calculate, but this is just one very narrow use, predominately we see them as communication devices rather than calculating devices. But the uses of them are actually endless, as we find more and more uses of them to offer accordance and thus transform our lifeworld.
Now, when we realize that the nature of the computer and its code together is to push us into Hyper Being and out of the machine dominant culture that preceded the computer dominated culture, where computers are machines that can be reprogrammed and that can actually reprogram themselves, which is why I believe my Grand Mother called them “re-puters”. Once something takes on a life of its own in this way that pushes us further into Wild Being in which things take on a seeming life of their own, and we chuck those things out of Software Engineering into Artificial Intelligence, Artificial Life or Artificial Sociality. It is this ability of code to reference itself and rewrite itself that makes it like jello not only the fact that it can represent almost anything either directly or as simulations with very few restrictions. Coders are those who dealt directly with the encoding and decoding of the assembly level instructions originally via punch cards. Programmers appeared when people could first set at terminals and enter the code directly into the machine via a coded interface like ASCII characters. Programmers dealt with languages discovered by Chomsky rather than the coding and encoding of assembly instructions. Then we started building bigger and bigger systems and SEI tried to get us to put some discipline into our work and used Engineering as a paradigm for transforming programming all the other work necessary to build large systems which software engineers did, of which programming was only a small part. Eventually it was realized that development had a lifecycle and that there were certain phases that you just had to go through in order to get a product that worked in the end. And that is why the term software developer came in, as someone who does everything that is necessary to get working code out the other end and there was a dichotomy introduced between developers and testers because it was thought best to have someone else different from the developer do the testing of the code. But of course testers were also writing code to do the testing. So software engineer became the generic term. Basically you can decide if you want something that is coded, programmed, developed and tested or engineered. These are both levels of abstraction and they are also levels of rigor and discipline as currently understood. There are hackers who try all sorts of things to achieve their goals, and are not necessarily systematic about what they are doing. Coders suggest some level of systematization because you are taking a model of what you want to software to do and are encoding it into the language of the machine. Programmers have a little broader scope in as much as they are dealing with higher level languages and there is some sense that they are not just coding in a model but actually programming the system to do something that is desired by someone else. Developers on the other hand along with Testers are creating larger systems of many parts that work together, and they develop a larger scale system that works because many different things are done beyond just programming the system, like testing, like reviews, like documentation. They develop a whole system that end users will use and customers will buy, but the scale of it may not be very large and the performance aspects not the emphasis. But if you decide you need something realtime and large scale then you pretty much have to fall back into engineering practices with their extra rigor and discipline. Ironically it was software engineers who introduced this rigor and discipline which is now being absorbed by other disciplines such as Hardware engineering, or at a higher level of abstraction Systems Engineering. It is not as one might expect that these terms are random, but they developed as the discipline of software engineering developed step by step and based on the technology available at different stages and the rigor that is needed for the task at hand. However, we still use all the terms and the semantics has continued to shift as new terms are added to the mix. For instance, Software Engineering includes within it requirements engineering and architectural design as up front activities that developers might gloss over and programmers would not necessarily see as part of the work of programming software. Also the heart of Software Engineering is in my view Realtime systems that become relativistic if there is no global clock in the system. Very sophisticated architectures are necessary to make those kinds of system work. But with multicores this is becoming commonplace on desktop systems. All the new languages have tasking built in. So what were simple batch type systems, or then applications, are now very sophisticated in how they farm out the work to tasks and tasks to cores, and so something that started off as only necessary in Software Engineering has now become ubiquitous in our attempt to sustain Moores law when it becomes more and more challenging to continue increasing performance exponentially.
Software Engineering puts limits on the jello aspect of software, and excludes anything that is strange like self-rewriting code, neural networks, expert rule systems and other artificial intelligence techniques. Software engineering as it exists with computer science as its basis is a real research program that has gone through several paradigm shifts from just code it, to structural and functional models of programs, to object oriented, and now in things like Scala (read Haskell) back to a mathematically functional model of how programs should work. And also with Agile in a reaction to the CMM and CMMI from SEI there is a social ideological revolution as well that is trying to take us back to the roots of coding and programming when they were black arts that no one understood, and thus really little control could be exerted on how the work was done as we have attempted to do by introducing engineering models into software. But as we move out of that social paradigm change into Lean and Agile at Scale models then the same problems that Software Engineering was meant to deal with come back to haunt us and so some of the deprecated practices that Agile wished to throw out are slowly being realized that they are necessary in a transformed way within Agile at Scale and Second Generation Lean (Reinertsen Flow) environments. So now we see hybrids like Agile Software Engineers rather than Agile Developers being sought by larger companies. Can you get the befits of Agile and stay true to the paradigm shift of the Agile Manifesto and Agile principles as well as apply the rigor and disciple necessary for large scale software product development. All this is a trial and error process unfortunately as the pendulum swings back and forth between effectiveness (Agile) and efficiency (Lean) and between freedom from managerial control (Ashby’s law) and necessary discipline taught as a harsh lesson due to repeated failure. City codes only exist because of disasters that happened without them. Outliers can produce extraordinary results but Probabilities kick in on large scale, sometimes truly global, systems that the CMMI was designed to handle in the face of factors like learning curve, employee turnover, the fact that the system needs to last for decades, etc.
What I believe is going to happen is that Agile and Lean are going to be reabsorbed by Software Engineering and then eventually even Systems Engineering and the paradigm shift will fundamentally alter how we work. But whatever benefits we can gain in the short term will be overwhelmed by the ever increasing scale of what we are building so that rigor and discipline will have to be part of it as well, and practices like Requirements Engineering, Architectural Design, Verification and Validation will come back with a vengeance to become part of our transformed way of working. The escape from rigor and discipline as well as necessary at scale practices like Requirements Engineering, Architectural Design, High level Planning are all coming back as we can see in Leffingwell’s Agile at Scale Framework as seen in Agile Software Requirements. An excellent book where you can see this happening right before our eyes is Lean Architecture: for Agile Software Development James O. Coplien, Gertrud Bjørnvig. Here the Use Case is reintroduced and justified piece at a time and then a new design discipline called Data Context Interaction (DCI) is introduced based on code injection that better models the actual use of the software by the users in the code. So the culture wars in Software are far from over as the new paradigm of Agile/Lean/Flow slowly takes hold. It is a social change that seems to me inevitable because it reintroduces the concepts of human scale back into software development, the model of humans as computers is slowly losing its grip on us, and we are starting to realize that we have talents not yet dreamed of by by our cyborg friends who are our creations. They have not yet started to dream of Electric Sheep. But we are dreaming of Beautiful code, teams, design etc. and actually smelling the problems in the code. I think we are entering a truly fascinating chapter of the development of software expertise whatever the title is that you associate with it. But that is a social paradigm shift were we are rediscovering our human limitations but also the creativity and the miracles we perform when we overcome the impossible in the technological infrastructure we ourselves have built, and as we continually bootstrap ourselves to keep up with and to keep reinventing the technological infrastructure based on software that is re-factoring our worldview.