Using the KeLP infrastructure, we have developed a scalable cell microphysiology simulator based on the highly successful serial MCell simulator. The resultant software, called MCell-K, significantly increases the size of approachable problems, which had previously been limited by the capacity of a single processor, opening up new venues for scientific discovery. For example, MCell-K now makes it possible to simulate extra-cellular dynamics in a specialized neural structure in the cerebellum called the cerebellar glomerulus. We hypothesize that local calcium depletion in this structure could convey information about recent synaptic activity between neighboring neurons in a small volume of tissue. Such a signal would constitute the biophysical basis of the classic back-propagation algorithm, an important form of supervized learning in artificial neural network theory, which has long been criticized as being unrealizable in a biological system.
To our knowledge no one has gotten biological simulations to scale up as well as we have, and we have therefore found a level of description and way to simulate nature that can approach the complexity of nature on its own terms.
Results have been reported at IPDPS 2004 (held in Santa Fe, NM in April 2004) and at SIAM Conference on Parallel Processing for Scinetific Computation (held in San Francisco in Februrary 2004).
For our experiments we used as input a chick ciliary ganglion model (courtesy of J. Coggan, T. Bartol, E. Esquenazi, D. Berg, M. Ellisman, T. Sejnowski), constructed with serial EM tomography. The image below shows the complex geometry with its many valleys and folds.
The following image shows a simulation in progress. Diffusing molecules (shown in green) originate from synaptic vesicle release sites (shown in white) and move through Brownian dynamics random walks. They may react with receptor molecules on the surface (shown in in blue and red) or they may be destroyed by enzymes on the surface (not shown). Approximately 10% of the diffusing molecules cross a processor boundary during each time step. Such molecules are highlighted in yellow.
MCell-K incorporates some important optimizations that enable it to scale well. First, it employs an adaptive communication protocol, which enables inter-processor data transfers to adapt in size according to simulation dynamics. This capability, supported by KeLP, significantly decreases the cost of moving data, which is only about 1% or 2% on 64 processors. As shown in the figure below, running times drop by a factor of 2 for each doubling of total number of processors.
We are in the process of installing a load balancer, which will extend the range of simulations that can be handled by MCell-K, and will also significantly increase its scalability. This load balancer uses an empirically derived performance model which was recently developed.
Although the load balancer is complete, modifying the existing MCell code (that is, the serial portions of the simulation) to accomodate the non-uniform partitions employed by the load balancer has taken longer than expected. This coding was complicated by the need to locate and repair a subtle bug that was causing inconsistent state to arise in distributed data that effectively shared information. The solution has been installed, however the resultant disruption has forced us to reprioritize the visualization interface in order that the load balancer can be installed. We plan to transition the visualization interface to other extramural support.
For the past 3 years, the development of MCell-K has been supported by NPACI. We will soon transition MCell-K to production status, using other extramural support.
Owing to the widespread use of the serial version of the serial simulator, we believe that deployment of MCell-K will stimulate a high demand for large scale simulations which in turn will increase the use of scalable computing facilities by the applications community. In addition, the prodigious data sets generated by MCell-K present new technical and research challenges for conducting data discovery.
This project has also motivated an asynchronous implementation and accompanying research which is the recipient of an NSF ITR award "Asynchronous execution for scalable simulation of cell physiology." Please see the related Tarragon site.
Further information about MCell-K can be found at the Salk Institute's site.
MCell-K and KeLP development were supported by the National Partnership for Advanced Computational Infrastructure (NPACI) under NSF contract ACI9619020. Greg Balls and Scott Baden were supported by the National Partnership for Advanced Computational Infrastructure (NPACI) under NSF contract ACI9619020. MCell is supported by NSF NPACI ACI9619020, NSF IBN-9985964, and the Howard Hughes Medical Institute.