(From Vinoski's paper, "Integrating Diverse Applications Within Distributed Heterogeneous Environments".)
CORBA is a software architecture for distributed object-based computing. It is architecture, location, implementation language, and vendor independent; individual object methods may be implemented in different languages and at different locations if desired. CORBA's design is set up to be very flexible, enabling effective distributed object computation without hindering the programmer's ability to layer additional services on top of CORBA, or to implement CORBA objects with other services at his disposal.
Mechanically speaking, the heart of a distributed-object system, from the programmer's point of view, is the way in which the remote objects are represented locally, and how operations are invoked. CORBA provides abstraction from programming languages through the use of a declarative "interface definition language" (IDL) to declare interfaces only; the supporting software that comes with a CORBA environment provides a compiler that reads the IDL and creates IDL stubs in a supported programming language. The methods and interfaces specified in the IDL declaration are mapped to the implementation in a uniform and well-defined manner, as specified in the CORBA "language mapping" for that language, along with mappings of the IDL types to local implementation types.
How objects are typically used is, assuming the CORBA run-time
environment has been initialized (which is just a few lines of code to
contact the local server for the environment and register the process),
essentially the same way as objects are used natively in the programming
language. In C++, for example, objects are created with calls to
"new", and invoked with the usual syntax; in C, objects are
represented as structure pointers, are created with an explicit
allocation call (provided by the environment), and methods are called by
name, and take the object pointer as an explicit distinguished
parameter.
Since objects are invoked in a way that is natural for the given programming environment, actually using CORBA is fairly easy. The distributed and dynamic nature of the environment is largely hidden behind the method calls; the methods themselves, as seen by the user's code, actually refer to a "phony" implementation of the interface in the local programming language which is generated from the IDL by the IDL compiler. The local methods do not contain an actual implementation of the user's code (of course, since the IDL is purely declarative, and supports no programming constructs), rather they contain "stub" functions which contact the local CORBA environment, handle all parameter marshalling, and use the facilities of the local environment to locate and contact the correct remote object. The remote object has a similar body of code, the IDL "skeleton", also generated automatically by the IDL compiler, which reverses the process by de-marshalling the parameters and eventually invoking the actual object implementation by whatever method calling convention is natural for the language the target object is implemented in. If there is a return value, it is returned through the same communication pathway, back through the IDL stub, where it is eventually returned to the caller as if the object was a regular local object. Exception behavior is a little more complicated, but still specified.
Current/future trends in systems: hardware getting smaller, faster, cheaper; software getting larger, more expensive, arguably slower. Increases in problem size & complexity lead to the use of cooperative aggregations of computing resources, and resource constraints (read: $$) lead to physical and organizational distribution of these resources. This gives rise to the need for distributed applications, to effectively manage computation on distributed resources. This, in turn, gives rise to the need for a systematic approach to developing this distributed software. "Ninja programming" suffices for Supercomputing demos, but doesn't serve the needs of HPC application developers in general.
Developing distributed applications is more complicated than centralized applications, of course. Why? Have to handle communication explicitly (still an issue for local parallel applications), but also have to deal with an unreliable, high-latency WAN. Other issues include bridging architecture and software gaps, and load balancing.
Several programming models have evolved to support distributed applications development; common models include message-passing (MPI & PVM) as well as toolkit-based approaches (Globus being one). While these models can perform well, in that the facilities they provide do their jobs, developing applications on top of them introduces significant complexity to the developer: the developer is forced to decompose data algorithmically, and to write code that takes a direct role in the distribution and communication. Additionally, many of the existing models and toolkits focus on developing applications from a single set of source files, for a handful of programming languages, in a homogenous environment.
The need for distributed applications, and tools to help develop them, is clear. While current tools address some of the need, they leave a lot to be desired from a software engineer's point of view: they tend to expose the complexity of both the distributed and heterogenous aspects of the execution environment, as opposed to hiding these details behind abstractions. Additionally, they do not promote component re-use, except at the implementation source level, which can complicate matters when a development effort spans several groups of programmers.
What is needed is an abstract component framework, which allows distributed software to be developed in a modular manner, around well-defined interfaces which aren't tied to a particular implementation language or platform. These well-defined functional units could, in theory, be re-used in other projects with a minimum of effort.
In general, object-oriented programming has the potential to promote code abstraction around interfaces, and functional unit re-use. Traditional OOP languages such as C++ and Java make strides in this area, but ultimately fall short for several reasons. The interface specification itself is coupled very tightly with the implementation; details of the implementation often creep into the interface specification at several levels, since the two are conjoined in header files. It is entirely up to the programmer to ensure that the declared interfaces aren't interwoven with the underlying implementation, for each and every object; across different classes written by different developers, each with their own style, it ends up a chore to keep the implementation from being exposed.
Artifacts of this interface/implementation coupling are numerous; given the mostly flat namespace of traditional OO languages at both the source file level and at the object file level, care must be taken to avoid namespace conflicts. Changes in an object implementation are generally not invisible to consumer modules -- often they require a re-link at the very least, and may require re-compilation to function properly. One of the most prominent artifacts is that the platform an implementation is compiled for is visible; re-using an object on a separate platform requires at the very least a re-compile, and may programmer effort to deal with platform-specific issues (system headers, libraries, different APIs, etc.). All of these, taken in turn for a single class, aren't too daunting; but for the particle physicist who wishes to re-use multiple families of objects implemented by other parties, the total work to adapt the sources to his environment may be prohibitive -- particularly if (s)he isn't a career programmer. This also presents synchronization problems if multiple versions of the code are "in-flight" between developers.
Beyond the obstacles of re-using traditional objects in a local environment wait the obstacles of using objects in a distributed context. One solution is to have each class define input and output methods which encode and decode the object opaquely, for transmission across networks and remote use. This approach is tedious and error-prone, however; now, not only must the developers keep track of their own source files, they must now worry about what version of the protocols the remote implementations are using, as well as correctly multiplexing object data streams. Since the protocols used are often implicitly specified through function call sequences (ala XDR), there is lots of room for error if the producing and consuming codes are not exactly the same -- a scenario which is quite possible in a distributed, heterogenous environment. A fundamental flaw with this type of approach is that it does not distinguish between an object's state and it's methods. Depending on the internal structure of the object and on the external encoding used to capture object state, using object across language boundaries can become inconvenient.
All in all, while the object-oriented approach has a lot of promise in distributed computing environments, traditional OO programming environments have fallen short in practice because of ties between interface definition and implementation details.
Enter the OMG -- Object Management Group -- a consortium of about 800 member groups, from HP to 3COM to American Airlines, formed in 1989. The OMG is a group independent from any one company's interests, with the common goal of enabling large-scale software re-use. The OMG's vision is to have applications themselves act as objects, so that to an end user, an application operates as reliably as a large appliance, without the user needing to know details about it's innards. To this end, the components making up the objects will be objects themselves, so that application developers needn't be concerned about lower levels of implementation, and so on. The OMG wants software modules to share a common set of "plugs", much as real-world systems have standard interconnections defined to allow vendor-independent inter-operability. (AC power, Ethernet, PCI). The main piece missing from the proverbial puzzle is a set of standard interfaces for re-usable, inter-operable software components -- this is what the OMG set out to address.
The OMG defines an Object Management Architecture, a well-defined, implementation-independent specification for object oriented software systems. The OMA is a broad model describing how classes of objects and services, common facilities, and application/domain-specific facilities fit together and communicate through a common entity, the Object Request Broker. The OMG specifies the Common Object Request Broker Architecture -- CORBA, which includes details on object semantics, an abstract declarative interface definition language (IDL), and formal IDL bindings to "real" programming languages such as C/C++ to ensure portability of user code to work with different ORBs. Other important specifications are protocols for inter-ORB inter-operability "on the wire", particularly the GIOP and IIOP. The OMG produces only standards, not any actual software products, and the specifications are available for anybody to download and implement. Feedback and requests for changes are also solicited through several open channels.
This is the CORBA figure I sketched during my talk:
(From Vinoski's paper, "Integrating Diverse Applications Within Distributed Heterogeneous Environments".)
The figure shows how the various key components of CORBA relate to each other, at the logical module-interface level. The arrows indicate the "primary" flow of information and control. I used this diagram, with some chalk, to illustrate how information passes through the system. One interesting feature of this figure that wasn't in my sketch is the indication of which interfaces are private to the ORB you've chosen, and which are independent (ie, standard). Also of interest is the shading that indicates which pieces of the puzzle are the same for all ORBS and those which vary with the particular object interface in use. The figure doesn't convey any very detailed information on it's own; it's more useful for glancing at while reading a paper (such as Vinoski's), to keep a perspective on how things are connected, or when giving a talk (such as mine) to make sure you don't skip over any key components.
The ORB is the heart of the distributed CORBA system. It accepts object requests from method users, or "clients", with object references; the ORB takes on the responsibility of mapping the object reference and request to a concrete location, delivering the request there, and returning any response. The most important feature of the ORB is that we don't need to know (or care) how it works; it is the ORB that hides most of the messy details of conventional distributed system implementations from it's consumers. The ORB completely hides most of the non-essential information about the target object in a request, including:
Central to the operation of the ORB is the notion of object references. With very few system-level exceptions, every addressable entity one interacts with in the CORBA environment is itself a CORBA object, all of which inherit eventually from a common ancestor, the CORBA::Object class. The ORB itself is an object. Given an object reference, a "client" cannot "open it up" and see what's inside; only the local ORB knows the details of object references, and it is responsible for providing them to clients, interpreting them in requests, and communicating them to/from remote ORBs transparently. Object references may be passed around freely by objects; in this way, objects can accept and return other objects as parameters to method calls. Objects are passed by reference in this manner. ("Objects by value" is the subject of a recent addition to the spec).
Every request operation must have an associated object reference. References are generated when objects are created (by "factory" objects), they are returned from directory-service requests (much like DNS in the Internet context), and they can be generated from strings. CORBA supports the "stringification" and "destringification" of object requests by clients, allowing them to store object references in persistent local storage for later use -- if the target object is still exists, of course.
To get the ball rolling, the local "ORB" provides a built-in name service that provides references to the ORB itself, allowing all of it's services to be accessed programmatically by clients.
It is in the IDL that the key separation of object interface from object implementation is "enabled", even enforced. The OMG IDL is a purely declarative language, similar to a subset of C++, that contains no statements of any kind -- only type and interface declarations. The IDL specifies objects, specifically what operations and types objects support, what requests can be made on objects, and where they inherit from. Multiple explicit inheritance is supported, but method overriding in sub-classes is not.
Among the important features of the OMG IDL is that it is in no way
tied to any underlying programming language; it's basic types are
defined in absolute bit sizes, and data formats for transmission are
also well-defined. Another interesting feature of the OMG IDL is that
of block nested name spaces; one can open a "module" block,
and add interfaces to a particular module's namespace, without fear of
namespace conflicts. Also, since the IDL is vendor-independent, one
doesn't need to worry about massaging the declarations to fit any
particular vendor's compiler or OS.
OMG IDL supports defining "interface"s inside of "module"s; interfaces can contain method declarations as well as explicit attributes (read/write or read-only). In OMG IDL, specifying an attribute is equivalent to defining a set of accessor methods to manipulate internal data; they're just syntactic sugar for those simple manipulations.
An example IDL interface definition for, say, some devices in a practical home security system might look like this:
// OMG IDL declaration for my high-tech home security system
module HomeSecurity {
exception NotAuthorized {
string auth_fail_reason;
};
interface LightControls {
attribute boolean on;
boolean set_power ( in boolean new_state );
...
};
interface Alarm {
attribute boolean armed;
boolean arm ( in string requester, in long auth_token )
raises ( NotAuthorized );
...
};
typedef sequence<float,3> Vect3D;
interface Sensor {
attribute boolean triggered;
attribute Vect3D location, forward, up;
};
interface IRDevice {
attribute float sensitivity;
attribute boolean saturated;
};
interface MotionSensor : Sensor, IRDevice {
void reset_gain ( void );
};
typedef sequence<Sensor> SensorArray;
...
interface IntruderKiller {
boolean locate_baddy( in SensorArray sa, out Vect3D loc );
boolean attack_baddy( in Vect3D loc, in float severity );
};
...
};
Author's note: These next few sections start to get pretty technical,
and because this is getting pretty long, I won't go into too much detail
-- I'll touch on them in class if I have time.
In the distributed environment, typically each local set of object implementations has an ORB associated with them. The specifics are flexible, but for the most part each "server machine" has an ORB running. Given that the ORBs handle all of the communication, there is the need for them to inter-operate smoothly. Since ORBs from different vendors may be used at different locations, there is a set of standard protocol definitions that ensure inter-operability. The CORBA architecture handles this at two layers: the General Inter-ORB Protocol (GIOP), defines the format of the messages to be transmitted between ORBS, but not a mechanism for transporting the messages in any particular manner. The GIOP is meant to be layered on another protocol that has knowledge of a particular communications domain; in this way, ORBs themselves can use different communication mechanisms transparently. To support CORBA use on IP networks, the Internet Inter-ORB Protocol (IIOP) is specified. It provides a transport layer for sending GIOP messages over TCP/IP.
Object method calls are handled in a manner that is as natural as possible to the implementation language. As outlined in the introduction, the IDL compiler generates stubs and skeletons in order to hide the distributed nature of the objects from both the caller and the object implementation. There are three sets of semantics specified for method calls: synchronous invocation, deferred synchronous invocation, and one-way invocation. Synchronous invocation is straightforward; the caller is blocked at the method call, and does not continue until the call completes and any values are returned. Synchronous invocation is supported by the IDL-generated static stubs. Deferred synchronous invocation provides for communication/computation overlap. The request is dispatched with an explicit dynamic method call, and the caller does not block; instead the caller collects responses explicitly as they arrive. This allows for multiple outstanding requests to proceed in parallel. One-way methods are non-blocking method calls that do not return values. The request is dispatched and essentially forgotten by the client; in fact, the caller has no specified way to tell if or when the request finishes processing.
The Dynamic Invocation Interface allows for method calls to be constructed explicitly at run-time. This can be useful for invoking objects with more flexible semantics, such as deferred-synchronous or one-way semantics; it can also be used to implement a module that maps CORBA objects to some other object storage environment without the need to re-compile static IDL stubs at the CORBA consumer end for every change of the target object's interface.
The Object Adapter serves to link the local ORB to the actual object implementation code. The OA just adapts one object's interface to another, namely the object interface for ORB requests to the interface for the local implementation of the target object, be it C++ object code or a shell script. The OA links to the ORB through both the IDL skeleton, creating the necessary invocation environment, and also directly with the object for operations such as creation.
CORBA supports directory services with object references; in particular, there is a standardized naming service, which maps plain-text (and meaningful) names to object references, much as DNS servers map host names to IP addresses in the Internet. The directory services can also support "trading" of objects, allowing a consumer for a particular object service to locate a suitable target object without a priori knowledge of what objects are available. This is accomplished by having objects (at the server end) register themselves with a trading service when created, and having the clients make explicit pattern-based queries to the trading service to locate suitable target object. The name service and trading service both act as directories of various sorts, and return CORBA object references which the consumer can then invoke methods upon.
CORBA is all quite nifty, providing what is truly a vendor-independent, language-independent, platform-independent, network-independent system for object oriented programming. As it is, it's a very powerful environment for developing robust, distributed applications. However, some of CORBA's strengths -- primarily, the simplification of the programming model through the hiding of information by the ORB -- seems like it could get in the way of the HPC application developer, who has developed his distributed application, and now wants it to run fast. The ability to effectively utilize homogenous HPC resources depends on competent scheduling algorithms which are able to match resources to computation based on the distribution of data, and a-priori knowledge about the structure of the application and properties of the systems. (Vector machine vs. Blue Horizon, etc.).
CORBA tries to act as an "enabling technology", that enables programmers to do useful things without restricting them arbitrarily. The class of scheduling mechanism described could actually be implemented on top of a distributed CORBA system, by making use of the trading services CORBA provides -- objects could contain explicit information about their location and resources, specified in the interface, perhaps as attributes. The scheduler could examine available objects -- perhaps, each local scheduler queue could be an object -- to gather info about them, and can also select specific objects by properties from a directory.
All the niftiness and abstractness of CORBA does come at a cost, however. There is a performance penalty associated with going to the ORB for every object request, and having it forward communication, possibly to other ORBs. Particularly in the case of dynamic invocation, clients must programmatically construct argument lists, and at the server end an object adapter may have to repeatedly query an Interface Repository to locate the correct object method to call, and construct a skeleton on the fly before the request actually gets to the target. All is not completely lost for local applications, though: CORBA does not strictly dictate all of the communication steps that have to take place over given links; they've left the door open for ORBs to be implemented using shared memory or other local, fast communication methods that may be available. The primary concern is that ORBs be able to communicate amongst themselves, and this communication is tightly specified by the ORB interoperability protocols.
Unless CORBA was being used in a manner that totally disregarded performance considerations -- overuse of blocking, dynamic methods in an inner loop, or some such -- the impact of the overhead will probably be worth it, on the whole, because of the programmer time and pain saved in developing the application initially, maintaining it for the time being, and adapting it for the future. This is especially true in business settings, since programmer time costs so much more then conventional CPU time; it's not quite as true in HPC environments where CPU time can be very expensive, and the applications are exotic enough that programmers would have to spend a lot of time on the problems anyway. But in the presence of well-written ORBs and applications that are written conscious of the potential overhead in ORB requests, even HPC environments will benefit from adoption of the model -- perhaps most significantly from the code re-use that well-written CORBA software allows.
Unfortunately for the HPC folks, however, CORBA implementations to date have been centered around business needs, being transaction and communication-oriented versus computation-oriented. They all seem to just communicate over GIOP over IIOP over TCP/IP, which works great for web servers but can be starkly sub-optimal for a supercomputer's toroidal mesh, for example. Since CORBA standards tend to not get in anybody's way, they don't provide a mechanism for explicitly handling high-speed parallel communication channels -- the ORB just delegates all of the communication in an abstract manner, expecting the lower levels to handle things in an effective way. In an HPC environment, this abstraction layer can hurt performance, since the application and the network could conceivably work together for better performance, but the ORB prevents them from doing so.
Yet still, as an HPC application developer, we'd like to get more out of the system. CORBA gives us distributed, interoperable object-oriented programming. While this is a giant leap from, say, local C++ over MPI, it still leaves us with the task of constructing an object system to perform our computation -- so are we really better off then when we started? (Yes: but we're greedy!) It would be nice to have some higher-level building blocks for HPC application developers to work from. As in other cases, CORBA and the OMG do not provide this directly, but they "enable" it by paving over the nagging lower-level details to allow these toolkits to be built.
CCA -- the Common Component Architecture -- is a software architecture proposed by the CCA Forum to define a standard for interoperability amount HPC software components. HPC applications often demand simulation, computation, and analysis techniques that spawn from many different fields of science. Traditionally, HPC applications are "just written" by the groups that need them, perhaps making use of standard static API's such as the LAPACK linear algebra package. However, given the specialized nature of the applications, and that many of the interdisciplinary folks are not software engineers who daydream about clean interfaces and abstraction, the code bodies are for the most part separate between projects, largely due to the pain level of sharing source-level implementations of components without narrow interfaces.
Rather than rope big groups of the HPC folks into rooms and preach an approach like CORBA to them, the CCA Forum is proposing standards for ready-made scientific computing components to be built, for the HPC folks to use and benefit from. The HPC folks themselves stand to benefit because, by using these standard components, they have reliable, portable building blocks to work from and their code gets simpler. This is especially the case with exotic multidisciplinary work: the particle physicist who needs to do some molecular interaction simulation as part of their application can rely on the efficient, smart molecular interaction simulator component written by some chemistry guru and his cadre of interns, which in turn uses the I/O and load-balancing cluster allocation scheme devised specifically for that simulation by an Application-LEvel Scheduling guru. . The hope is that the particle physicist will buy into this model and write his own application-specific bits as CCA components, which someone else can pick up and use down the line.
CCA addresses two main problems, which the CCA Forum perceives as shortcomings of current distributed object models (including CORBA). The first is the lack of efficient parallel communication, the second is the lack of an explicit component model. Other problems include a lack of convenient data types for "science-like" things, such as dynamic multi-dimensional sparse arrays and complex numbers. These can of course be implemented on top of CORBA (or just about anything else), but the HPC folks want them to be built-in to the system they get, and fast -- this may imply some low-level ties to the underlying programming language and/or architecture.
The CCA Forum has defined their own object-and-component-oriented programming model. Given that scientific codes often have very regular data flow characteristics -- raw data goes in, some kind of computation is done, cooked data comes out -- it makes sense to model these computations in some sort of pipeline. What CCA does is consider each computation entity (at the software level -- say, a linear equation solver -- not at the hardware level) as a component. Each component has ports where data flows in and out of; the ports are then interconnected by registration calls. This does not seem wholly unlike CORBA's object reference system; indeed, "compatible" ports are those which are type-compatible at the OO level. Where CCA differs, however, is that it provided explicit support for "direct port connections" -- direct connections are cheap to set up, and have very low communication overhead.
Interestingly enough, CCA's distributed object model, aside from the "ports" business, is very similar to CORBA's. They define their own interface language, Scientific IDL (SIDL), which is based upon CORBA's IDL, as well as the Java language. Their ports use a unidirectional "provides/uses" connection model, which seems quite similar to what the Nexus library provides, although at a higher and more abstract level. The CCA folks also define IDL-to-language bindings and use static IDL stubs and skeletons generated by an IDL compiler, much like CORBA. They don't seem to be as worried about dynamic invocation, probably because that is such a costly proposition in a distributed object scenario, and also because scientific data is often highly parameterized and uniformly formatted at the input and output stages, obviating some of the need for dynamic method invocation.
CORBA itself is not completely out of the running; there are several CORBA folks researching high-performance CORBA implementations, and there is an "official" high-performance CORBA working group. However, it will probably be a while (a few years?) before high-performance CORBA implementations viable for HPC use come out, and upcoming CORBA specifications under review by the OMG do support a component model similar to CCA's. Interestingly enough, the upcoming CORBA 3.0 specification includes such a "provides/use" communication model very much like CCA's, and a "CCA over CORBA" implementation is being planned by some CCA folks.