=========================================================================== Organic Computing: Towards structured design of processes ========================================================= Interdisciplinary Symposium, Paderborn, November 23-24, 2001 Session III: Towards a science of organization ============================================== by Laurenz Wiskott, Berlin, December 2001. This is an attempt to summarize the third session of the symposium on organic computing. I have tried to bring the different contributions into a logical order rather than listing them in chronological order. I have also added the names of those who have contributed. However, this has to be taken with some caution, since I certainly misunderstood some of the arguments, so that I got it right probably only in about 90% of the cases. Thus, don't cite anybody based on this summary. First contact the person and double check what he/she really meant to say. "Q" and "A" indicate questions and answers, respectively. There are also some arguments which I left out because I didn't understand them acoustically or intellectually. That doesn't mean that I considered them unimportant, so please forgive me if I haven't summarized one of your contributions. I hope that overall this summary can give an impression of the discussion and the people who contributed. I) Self-Organization ==================== Definition of terms ------------------- "Organization" is a process in which some form of order, pattern, or regularity develops [Schöner]. "Self-organization" is organization without a central organizer. Example: stimulated emission in a laser generates coherent light [Schöner]. Self-organization is typically characterized by local mechanisms that create global patterns. This makes it difficult to establish a connection between the (local) interactions and the (global) functional properties of a system [Schuster, ...]. "Emergence": Closely related to the notion of self-organization is the term emergence. This term has often been misused for results that we don't understand. In a proper sense the term emergence refers to patterns that are inherent to processes [Schöner]. Levels of self-organization --------------------------- There are different levels of self-organization. a) The simplest form of self-organization is pattern formation with given boundary conditions. This usually leads to stationary spatial patterns that can be predicted if the dynamics and the boundary conditions are known. Only patterns that self-stabilize once they are formed emerge. [Schöner] Well known examples are the formation of stripes and hexagon patterns on a two-dimensional sheet with localized excitatory and less localized inhibitory interactions. b) A more complex level of self-organization is found in systems that modify their own boundary conditions. Flames in a forest fire are an example. This type of self-organization can be found in developmental or evolutionary systems and pose a major challenge to the theoretician. These systems are much more flexible, but there is no general mathematical theory for available. [Schöner] c) On an even higher level are systems that modify not only their boundary conditions but also their interaction types. This can lead to more open and more powerful systems. [McCaskill] d) Finally different processes and levels of self-organization can be / need to be coupled to form truly powerful systems. Example: In vision, many different cues need to be combined to come to the right interpretation of a visual scene. [von der Malsburg] Self-organization is usually multi-causal and there are multiple paths that may lead to a pattern [Schöner]. Theories of self-organization ----------------------------- Theory is worked out only for the simple forms of self-organization. The more complex forms are still awaiting a good formalization [Schöner]. Theory is particularly needed in understanding how different systems could be combined and integrated [von der Malsburg]. Self-organization and computation --------------------------------- In general it may be misleading to think of self-organization in terms of computation. Thinking in terms of processes might be more appropriate. [Schöner] In some cases self-organization can be used for computation and understood in these terms. However, the relation is highly asymmetric. It is much easier to go from self-organization to computation (understand what a self-organizing system 'computes') than the other way around (design a self-organizing system for a specific computational task). The latter is the true challenge. [McCaskill] Today's computer programs already have some degree of self-organization and are not simply input-output functions. For instance, it could offer insights to look at GUIs, which are decentralized and distributed. [Meyer] Self-organization in evolutionary systems ----------------------------------------- Pattern formation in evolution, in contrast to physical systems, has a memory. [Schuster] In developing systems it is not always desirable that self-organizing patterns are stable. Sometimes old patterns must be destroyed to give way to new patterns. Limited lifetime of patterns permits flexibility. [Meinhardt] II) Efficient Evolution ======================= One important mechanism for efficient evolution is to separate mutation and selection. Mutation is coupled to the genotype, while selection is coupled to the phenotype. There are many mutations which don't change the phenotype. This permits exploration of the genotype space without any selection pressure, i.e. without risk. This leads to plateaus during the exploration phase without changing the phenotype significantly and innovations in evolution when a breakthrough occurs and the phenotype changes significantly. [Schuster] For this principle of exploration by neutral mutations to work, the regions in genotype space with same phenotype must be sufficiently connected. Usually there exist several alternative bottlenecks to go from one phenotype to a higher phenotype. Evolution has to find these bottlenecks and will follow a trajectory through a sequence of bottlenecks. Each bottleneck corresponds to an innovation in phenotype. [Q: von Seelen, Sendhoff, A: Schuster] It is particularly interesting that the number of innovations to go from one phenotype to another one apparently depends only on the initial and final phenotype and not on the evolutionary trajectory, which can be quite different from case to case. [Q: von der Malsburg, A: Schuster] Recombination is useful if the problem or the genetic coding is modular. However, the mathematics gets very complicated if mutation and recombination are combined. [Q: Schwefel, A: Schuster] DNA vs. computers ----------------- >From computers to DNA? In general it doesn't make sense to use DNA directly to solve problems of genetic algorithms, except for cases for which DNA computing is well suited, such as finding molecules with particular properties. [Q: Palm, A: Schuster] Computation with DNA could be done for more problems, but this requires a different computational paradigm. Usually people try to transfer the algorithmic paradigm to DNA computing and that is inappropriate. [McCaskill] >From DNA to computers? One lesson to learn from evolution is that we first need to classify the fitness landscape of a problem, e.g. analyse the distribution of fitness values, to decide which genetic algorithm variants to use. [Q: Schaal, A: Schuster] In order to make computers themselves evolvable we need completely different computer architectures. It is not a good idea to copy nature too closely, but the following principles may be useful: Do computation highly parallel and redundant, e.g. let the same operation be computed thousands of times in different ways and perform selection on these processes. However, don't be too fuzzy. Use neutral mutations first coarsely but then more strictly. [Q: Rechenberg, A: Schuster] III) Modules ============ Modules seem to be essential for complex systems to evolve and work well. Prominent examples are the brain as well as modern computer languages, which are both highly modular. Brains are subdivided into areas, layers, columns, synapses, molecules etc. In software development complex problems must be subdivided into sub-problems over and over again until one arrives at simple sub-problems that can be solved easily. Object oriented languages have a high degree of modularity. However, it is not really clear what a module is in general and it is much less clear how modules could self-organize. [von der Malsburg] What are modules? ----------------- There are different types of modules: a) Structural modules, such as domains in molecules. [Schuster] b) Functional modules, such as groups of proteins in metabolic networks that are switched on together. [Schuster] c) Modules in task space. For example, a cell is divided into sub-parts, which are dedicated to specific tasks. This type of modularity is evolutionary most relevant. [Schuster] d) Temporal modules [Pasemann]. For example modules could be separated by the time scales of their dynamics. Why do modules emerge? ---------------------- Modularity might emerge because they allow the system to use something over and over again. Thus exploitation might be the central concept and modularity only a consequence of it. [Sendhoff] Such form of exploitation requires generalization. Since a module is usually applied to many different variants of a similar task it must have a general purpose mechanism rather than a special solutions. Thus generalization permits modularization to pay off. [McCaskill] Structured modules in artificial neural networks can have very different functions and physiological neurons are much more flexible than the modules in physics, such as elementary particles. [Pasemann] In some cases the function of modules can actually emerge a posteriori. For example, insect and vertebrate limbs are made from the same genes, although the common ancestors did not have limbs. [Meinhardt] Synthesis vs. analysis ---------------------- Modules are probably very useful in synthesis as well as analysis, but their role can be very different in these two cases [Palm]. One example of this may be that in brains anatomical and functional modules can be very different and very difficult to map onto each other [Triesch]. For synthesizing a system it may be useful to start with modules. These modules however may disappear by completely merging or integrating with each other, so that the modular origin becomes invisible in an analysis of the system. [Pasemann] A good example is a computer program, which is typically highly modular. However, if you analyse the final product, namely the binary code, these modules are not visible anymore. [Douglas] In analysis modules could be identified based on their strong internal and weak external coupling [Giese, von der Malsburg, ...] or based on closeness [Giese]. For instance, in a car parts of the engine must fit together tightly while the luggage can fit loosely in the trunk. Tightly coupled elements must be protected while one can play around with loosely coupled elements. This defines the modules. [von der Malsburg] Dynamics of modules ------------------- Modules must be hierarchical, must interact, must cooperate to make up for each others failures, and must find partners to interact with [von der Malsburg]. Modules can be time dependent, i.e. emerge and disappear again, which may make it difficult to find them [Kummer]. This might be how the brain works, creating dynamic cliques of modules that emerge and disappear [Triesch]. Modularity must be flexible, since different tasks require different hierarchies. Top and bottom may change from situation to situation [von Seelen]. In biology, results of self-organization must have finite life-time to permit flexibility, see Section II [Meinhardt]. It is obviously necessary to move from the static picture to a dynamic one [Gerisch, von der Malsburg], chaotic interactions between elements leading to the emergence of order and dynamic patterns [von der Malsburg]. Could it be even possible to understand the formation of modules through these dynamic mechanisms, which can be as simple as aggregation [Ritter]. The analogy between biological modules and objects in computers may not be helpful. There may be a better analogy with pattern matching systems (string rewriting systems), which permit dynamic and self-organizing interactions between parts in a program. Pattern matching in feedback loops may be more important than modularity and the latter only a consequence of the former. [Rojas] Arguments against modules ------------------------- It is not clear why modules should be prevailing in nature since there is no direct mechanism that would favor modules per se. [Herzel] The definition of modules can be very dependent on the perspective or the space that we consider [Schuster, Schöner]. Cars, for example, are very similar modules from a technical or traffic perspective. From a functional perspective, however, they can be very different. Some are for transportation of people, some are for extinguishing a fire, some are for rescuing people, etc. [Schuster]. Gate patterns of walking, for example, are usually separated by instability. However, going another route in parameter space the same two gate patterns could be connected smoothly [Schöner]. It may help to find the space in which modularity is most prominent [Schuster, Schöner]. Modules in regulatory networks are partly artificial. [Herzel] Modules in synthesis and analysis can be very different, see above. >From a pragmatic point of view, modules are just a method that may be useful in some tasks depending on the tools used. [von Seelen] Too strong encapsulation can cause errors. For instance, in autonomous robotics the subsumption architecture is a modular architecture. If two modules are in conflict with each other, one module may dominate the other and lead to fatal behavior. Modules must communicate well to avoid such failures. [Schöner] Another problem in biology can be that sub-modules might actually threaten the host system. The system must keep the modules under control but alive. [Schuster] IV) Epilogue ============ As it becomes clear from the summary above and probably also from the summary by Rolf Würtz and Bernhard Sendhoff, the discussions of the symposium were rather conceptual and little was said about concrete steps towards organic computing. The evening after the first of the two days of the symposium I though about what the next more concrete steps in this venture could be and, after my summary, I presented the following six points. 1) Which hardware/software/whateverware architectures for organic computing can we imagine for 2050? (Well, I am afraid that my proposal for concreteness wasn't too convincing by choosing the year 2050 as a deadline. However, I guess I wanted to leave enough time for truly innovative and ambitious ideas. Consider how long it took from the first freely programmable computer of the world built by Konrad Zuse in 1938 until the fast and sophisticated computers we have today. If you don't like 2050 put any other number there.) Despite the far time horizon, I intended this question to be rather concrete. I don't know enough about possible hardware realizations of organic computing but some possibilities might be DNA computing, quantum computing, computing with molecules and nanotubes, VLSI chips, parallel computers with conventional chips, real neurons grown on a chip, etc. In the end, one has to think about how to put a whole system together based on one or more of these techniques. The task is to conceptualize a rough outline of an organic computer or, even better, a number of alternative architectures. 2) What could be the contribution of your field to these envisioned architectures? This is probably fairly clear with respect to hardware issues. Somebody working with DNA could contribute to DNA computing, somebody working with VLSI chips could contribute to that type of architecture, etc. It is much less clear for people like me, who are working on problems of self-organization from an abstract algorithmic point of view. We usually implement our learning rules on conventional computers and can pretty much realize any kind of mechanism we want. We have little idea about what kind of constraints would be imposed on our algorithms if we had to implement them more directly on an organic hardware. 3) How can we combine our techniques? Closely connected to the previous point is that of combining techniques, because you can only contribute something that can be integrated with other techniques. Here I see great need for communication between hardware people and those working on the theory of self-organization in order to figure out which computational paradigms fit to which organic hardware. It needs to be defined what type of computations can be realized with the different hardwares, including the costs that are associated with these computations. This would be a useful interface between hardware and software people. This doesn't say that in an organic computer hard- and software are separated like in a conventional computer. It just defines what a theoretician is allowed to use in his/her simulations if he/she wants to be consistent with what is realizable in a certain type of hardware. 4) Which techniques are required but not developed yet? I imagine that if we think about a complete organic computer, we will find problems which we don't have any solution for yet. One such problem could be that of interfacing different hardware components with each other. 5) How can we put everything together to create a working system? This seems to be simple once we have worked through Steps 1 to 4. However there is an organizational aspect to it, that needs some consideration. What kind of infrastructure do we need? Does it need a big Center for Organic Computing to bring all required know-how to one physical location? Can it be done in smaller institutes that work together over larger distances? What is the role of industry? How can we get scientists and the public interested in organic computing and bring them together to cooperate? To what extent are we, as the participants of the symposium, willing to get involved in this venture ourselves? What are the steps we need to take now to build the infrastructure needed in the future? 6) What tasks do we wish to define to compare systems? It has turned out very fruitful and stimulating in various fields to define some bench mark problems to evaluate and compare the performance of different systems. It might be useful to do something similar for organic computing as well. Such bench mark problems would naturally be problems that cannot be formalized easily. In some sense face and speech recognition are already of that kind. And, indeed, some organic software concepts have proven useful in these tasks. However, the programs still run on conventional machines (rather successfully). It might be a useful exercise to think of problems that are maximally difficult to solve with traditional techniques/hardware and minimally difficult to solve within the paradigm of organic computing. Something that really requires an integrated organic hardware/software system. As an alternative one might have to add some hardware/software constraints to define what organic computing means and exclude the traditional systems, in the hope that, in the long run, organic computing will surpass traditional computers and programming techniques in any case. This Step 6 could logically come also before Step 1.