MPI I/O - by Otto Sievert

I've been using logging of the form 

  fprintf(stderr, "%d: some data = %d\n", rank, data);

in my MPI programs, but have been running into the reordering
problem Shava mentioned on the web page such that my output gets 
garbled and ends up looking like

  0: some 1: data = 32
  some data = 34 
  0: 
  ...

But I noticed that the garbling always happened at parameter boundaries 
(the '%d' stuff - I'm not exactly sure what they are called formally),
so I wrote a simple logging function that copies the fprintf()
arguments into a single string, then prints the string on stderr/stdout:

  #include <stdarg.h>

  void tlog(char *s, ...)
  {
    char     tmpbuf[1024]; /* hard-coded maximum message length! */
    va_list  ap;

    va_start(ap, s);
    vsprintf(tmpbuf, s, ap);
    va_end(ap);

    fprintf(stderr, tmpbuf);
  }


  tlog("%d: some data = %d\n", rank, data):

("tlog" for "terminal log", just in case you were wondering. :)

This has completely cleared up the garbled nature of the messages!

I think something about how fprintf() is implemented allows delays/
separations around parameters that allow other fprintf() calls to 
sneak in. I don't know if this trick will always work, but it sure is
a lot better than before.