More chpldoc updates to modules/standard/Random.chpl

With Lydia's latest chpldoc improvements, my top-2 CHPLDOC FIXME
issues are closed (so removing the comments related to those).  In
reviewing the output, two others came up, though, making my new top 3:

* for arguments of unknown/generic type with default values, we
  should render 'x = y' rather than 'x -> = y'

* in general, I think we should render type qualifiers as ':' rather
  than '->' to be consistent with Chapel concepts.

* (holdover): have chpldocs follow module declaration order.

/*
 * Copyright 2004-2015 Cray Inc.
 * Other additional copyright holders may be indicated within.
 *
 * The entirety of this work is licensed under the Apache License,
 * Version 2.0 (the "License"); you may not use this file except
 * in compliance with the License.
 *
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

/*
   Support for pseudorandom number generation

   This module defines an abstraction for a stream of pseudorandom
   numbers, :chpl:class:`RandomStream`.  It also provides a
   convenience function, :chpl:proc:`fillRandom` that can be used to
   fill an array with random numbers in parallel.

   The current pseudorandom number generator (PRNG) implemented by
   this module uses the algorithm from the NAS Parallel Benchmarks
   (NPB, available at: http://www.nas.nasa.gov/publications/npb.html).
   The longer-term intention is to add knobs to select between a menu
   of PRNG algorithms, such as the Mersenne twister.

   Paraphrasing the comments from the NPB reference implementation:

     This generator returns uniform pseudorandom real values in the
     range (0, 1) by using the linear congruential generator

       x_{k+1} = a x_k  (mod 2**46)

     where 0 < x_k < 2**46 and 0 < a < 2**46.  This scheme generates
     2**44 numbers before repeating.  The generated values are
     normalized to be between 0 and 1, i.e., 2**(-46) * x_k.

     This generator should produce the same results on any computer
     with at least 48 mantissa bits for `real(64)` data.

   Here is a list of currently open issues (TODOs) for this module:

   1. This module is currently restricted to generating `real(64)`,
   `imag(64)`, and `complex(128)` complex values.  We would like to
   extend this support to include other primitive types as well.

   2. We plan to support general serial and parallel iterators on the
   RandomStream class; however, providing the full suite of iterators
   is not possible with our current parallel iterator framework.
   Specifically, if :chpl:class:`RandomStream` is a follower in a
   zippered iteration context, there is no way for it to update the
   total number of random numbers generated in a safe/sane/coordinated
   way.  We are exploring a revised leader-follower iterator framework
   that would support this idiom (and other cursor-based ones).

   3. If no seed is provided by the user, one is chosen based on the
   current time in microseconds, allowing for some degree of
   pseudorandomness in seed selection.  The intent of
   :chpl:record:`SeedGenerators` is to provide a menu of other options
   for initializing the random stream seed, but only one option is
   implemented at present.

   4. As noted above, we plan to add support for additional PRNG
   algorithms over time.

*/
module Random {

  /*
     Note for developers on "private" symbols:

     It is the intent that once Chapel supports a notion of 'private'
     symbols, everything prefixed with RandomPrivate will be made
     private to this module and everything prefixed with
     'RandomStreamPrivate_' will be made private to the RandomStream
     class.
  */

  /* Internal TODOs for developers:

     - is the parSafe on each of the three main calls overkill?

     - should RandomStream be parameterized by the type to return?
       it seems odd that currently getNext() and getNth() return
       reals always...
  */

/*
   An instance of :chpl:record:`SeedGenerators` that provides a
   convenient means of generating seeds when the user does not wish to
   specify one manually.
*/
const SeedGenerator: SeedGenerators;

// CHPLDOC FIXME: Respect ordering of things in the file

/*
  Provides methods to help generate seeds when the user doesn't want
  to create one.  It currently only supports one such method, but the
  intention is to add more over time.

  (Note: once Chapel supports static class methods,
  :chpl:const:`SeedGenerator` and :chpl:record:`SeedGenerators` should
  be combined into a single record type with static methods).
*/

record SeedGenerators {
  /*
    Generate a seed based on the current time in microseconds as
    reported by :chpl:proc:`Time.getCurrentTime`, ensuring that it
    meets the PRNG's requirements.
  */
  proc currentTime {
    use Time;
    const seed: int(64) = getCurrentTime(unit=TimeUnits.microseconds):int(64);
    return (if seed % 2 == 0 then seed + 1 else seed) % (1:int(64) << 46);
  }
};

// CHPLDOC FEEDBACK: If easy, I'd suggest either deprecating the
// :arg <type> <name>: form or else switching the order to
// :arg <name> <type>: as it's an easy trap to simply type the same
// argument as in the function's signature.

// CHPLDOC FIXME: if the first line below is shifted one character to the
// left, it ends up being rendered like a method

// CHPLDOC FIXME: if the first line below is shifted one character to the
// right, it ends up causing a warning (promoted to error) in the .rst
// file.  It'd be preferable to have it declare the issue in terms of
// the .chpl line numbers.
//
// CHPLDOC FIXME: When a formal argument is of inferred/unknown type,
// and has a default value, it is currently getting both '->' and '='
// I think it should only get the '='.
//
// CHPLDOC FIXME: For that matter, it seems to me that we should be
// rendering types as ':' rather than '->'.  I'm not seeing the
// rationale for using non-syntax in this context and while it
// seemed acceptable in some cases I was seeing it before (return
// values, variables without default values), the more I think about
// it, the more it seems like ':' would be more clear for Chapel
// (even in the context where I was just looking past it earlier).
//

/*
  Fill an array of `real(64)`, `imag(64)`, or `complex(128)` elements
  with pseudorandom values in parallel using a new
  :chpl:class:`RandomStream` created specifically for this call.  The
  first `arr.size` values from the stream will be assigned to the
  array's elements in row-major order for `real` and `imag` elements.
  For `complex` elements, consecutive pairs of random numbers are
  assigned to the real and imaginary components, respectively.  The
  parallelization strategy is determined by the array.

  :arg arr: The array to be filled, where T is real(64), imag(64), or complex(128).
  :type arr: [] T

  :arg seed: The seed to use for the PRNG.  Defaults to :chpl:proc:`SeedGenerator.currentTime <SeedGenerators.currentTime>`.
  :type seed: int
*/

proc fillRandom(arr: [], seed: int(64) = SeedGenerator.currentTime)
  where (arr.eltType == real || arr.eltType == imag || arr.eltType == complex) {
  var randNums = new RandomStream(seed, parSafe=false);
  randNums.fillRandom(arr);
  delete randNums;
}

pragma "no doc"
proc fillRandom(arr: [], seed: int(64) = SeedGenerator.currentTime) {
  compilerError("Random.fillRandom is only defined for real(64), imag(64), and complex(128) arrays");
}

/*
  Models a stream of pseudorandom numbers.  See the module-level
  notes for :chpl:mod:`Random` for details on the PRNG used.
*/
class RandomStream {
  /*
    Indicates whether or not the RandomStream needs to be
    parallel-safe by default.  If multiple tasks interact with it in
    an uncoordinated fashion, this must be set to `true`.  If it will
    only be called from a single task, or if only one task will call
    into it at a time, setting to `false` will reduce overhead related
    to ensuring mutual exclusion.
  */
  param parSafe: bool = true;
  /*
    The seed value for the PRNG.  It must be an odd integer in the
    range (1, 2**46).
  */
  const seed: int(64);

  /*
    Constructs a new stream of random numbers using the specified seed
    and parallel safety.  Ensures that the seed value meets the PRNG's
    constraints.

    :arg seed: The seed to use for the PRNG.  Defaults to :chpl:proc:`SeedGenerator.currentTime <SeedGenerators.currentTime>`..
    :type seed: int

    :arg parSafe: The parallel safety setting.  Defaults to `true`.
    :type parSafe: bool
  */
  proc RandomStream(seed: int(64) = SeedGenerator.currentTime,
                   param parSafe: bool = true) {
    if seed % 2 == 0 || seed < 1 || seed > 1:int(64)<<46 then
      halt("RandomStream seed must be an odd integer between 0 and 2**46");
    RandomStreamPrivate_init(seed);
  }

  /*
    Returns the next value in the random stream as a `real(64)`

    :arg parSafe: Permits :chpl:param:`RandomStream.parSafe` to be overridden for this call.  Defaults to :chpl:param:`this.parSafe <RandomStream.parSafe>`.
    :type parSafe: bool

    :returns: The next value in the random stream as a `real(64)`.
   */
  proc getNext(param parSafe = this.parSafe): real(64) {
    if parSafe then
      RandomStreamPrivate_lock$ = true;
    RandomStreamPrivate_count += 1;
    const result = RandomPrivate_randlc(RandomStreamPrivate_cursor);
    if parSafe then
      RandomStreamPrivate_lock$;
    return result;
  }

  /*
    Advances/rewinds the stream to the `n`-th value in the sequence.

    :arg n: The position in the stream to skip to.  Must be non-negative.
    :type n: integral

    :arg parSafe: Permits :chpl:param:`RandomStream.parSafe` to be overridden for this call.  Defaults to this.parSafe.
    :type parSafe: bool
   */

  proc skipToNth(n: integral, param parSafe = this.parSafe) {
    if n <= 0 then
      halt("RandomStream.skipToNth(n) called with non-positive 'n' value", n);
    if parSafe then
      RandomStreamPrivate_lock$ = true;
    RandomStreamPrivate_count = n;
    RandomStreamPrivate_cursor = RandomPrivate_randlc_skipto(seed, n);
    if parSafe then
      RandomStreamPrivate_lock$;
  }

  /*
    Advance/rewind the stream to the `n`-th value and return it.  This
    is equivalent to :chpl:proc:`skipToNth()` followed by
    :chpl:proc:`getNext()`.

    :arg n: The position in the stream to skip to.  Must be non-negative.
    :type n: integral

    :arg parSafe: Permits :chpl:param:`RandomStream.parSafe` to be overridden for this call.  Defaults to this.parSafe.
    :type parSafe: bool

    :returns: The `n`-th value in the random stream as a `real(64)`.
  */

  proc getNth(n: integral, param parSafe = this.parSafe): real(64) {
    if (n <= 0) then
      halt("RandomStream.getNth(n) called with non-positive 'n' value", n);
    if parSafe then
      RandomStreamPrivate_lock$ = true;
    skipToNth(n, parSafe=false);
    const result = getNext(parSafe=false);
    if parSafe then
      RandomStreamPrivate_lock$;
    return result;
  }

  /*
    Fill the argument array with pseudorandom values.  This method is
    identical to the standalone :chpl:proc:`fillRandom` procedure,
    except that it consumes random values from the
    :chpl:class:`RandomStream` object on which it's invoked rather
    than creating a new stream for the purpose of the call.

    :arg arr: The array to be filled, where T is real(64), imag(64), or complex(128).
    :type arr: [] T

    :arg parSafe: Permits :chpl:param:`RandomStream.parSafe` to be overridden for this call.  Defaults to this.parSafe.
    :type parSafe: bool
  */

  proc fillRandom(arr: [], param parSafe = this.parSafe) {
    if arr.eltType != complex && arr.eltType != real && arr.eltType != imag then
      compilerError("RandomStream.fillRandom is only defined for real(64), imag(64), and complex(128) arrays");
    forall (x, r) in zip(arr, iterate(arr.domain, arr.eltType, parSafe)) do
      x = r;
  }

  pragma "no doc"
  proc iterate(D: domain, type resultType=real, param parSafe = this.parSafe) {
    if resultType != complex && resultType != real && resultType != imag then
      compilerError("RandomStream.iterate is only defined for real(64), imag(64), and complex(128) result types");
    param cplxMultiplier = if resultType == complex then 2 else 1;
    if parSafe then
      RandomStreamPrivate_lock$ = true;
    const start = RandomStreamPrivate_count;
    // NOTE: Not bothering to check to see if D.numIndices can fit into int(64)
    RandomStreamPrivate_count += cplxMultiplier * D.numIndices:int(64);
    skipToNth(RandomStreamPrivate_count, parSafe=false);
    if parSafe then
      RandomStreamPrivate_lock$;
    return RandomPrivate_iterate(resultType, D, seed, start);
  }

  pragma "no doc"
  proc writeThis(f: Writer) {
    f <~> "RandomStream(parSafe=";
    f <~> parSafe;
    f <~> ", seed = ";
    f <~> seed;
    f <~> ")";
  }

  ///////////////////////////////////////////////////////////// CLASS PRIVATE //
  //
  // It is the intent that once Chapel supports the notion of
  // 'private', everything in this class declared below this line will
  // be made private to this class.
  //

  pragma "no doc"
  var RandomStreamPrivate_lock$: sync bool;
  pragma "no doc"
  var RandomStreamPrivate_cursor: real;
  pragma "no doc"
  var RandomStreamPrivate_count: int(64);

  pragma "no doc"
  proc RandomStreamPrivate_init(seed: int(64)) {
    this.seed = seed;
    RandomStreamPrivate_cursor = seed;
    RandomStreamPrivate_count = 1;
  }
}

////////////////////////////////////////////////////////////// MODULE PRIVATE //
//
// It is the intent that once Chapel supports the notion of 'private',
// everything declared below this line will be made private to this
// module.
//

//
// NPB-defined constants for linear congruential generator
//
pragma "no doc"
const RandomPrivate_r23   = 0.5**23,
      RandomPrivate_t23   = 2.0**23,
      RandomPrivate_r46   = 0.5**46,
      RandomPrivate_t46   = 2.0**46,
      RandomPrivate_arand = 1220703125.0; // TODO: Is arand something that a
                                          // user might want to set on a
                                          // case-by-case basis?

//
// NPB-defined randlc routine
//
pragma "no doc"
proc RandomPrivate_randlc(inout x: real, a: real = RandomPrivate_arand) {
  var t1 = RandomPrivate_r23 * a;
  const a1 = floor(t1),
    a2 = a - RandomPrivate_t23 * a1;
  t1 = RandomPrivate_r23 * x;
  const x1 = floor(t1),
    x2 = x - RandomPrivate_t23 * x1;
  t1 = a1 * x2 + a2 * x1;
  const t2 = floor(RandomPrivate_r23 * t1),
    z  = t1 - RandomPrivate_t23 * t2,
    t3 = RandomPrivate_t23 * z + a2 * x2,
    t4 = floor(RandomPrivate_r46 * t3),
    x3 = t3 - RandomPrivate_t46 * t4;
  x = x3;
  return RandomPrivate_r46 * x3;
}

// Wrapper that takes a result type (two calls for complex types)
pragma "no doc"
proc RandomPrivate_randlc(type resultType, inout x: real) {
  if resultType == complex then
    return (RandomPrivate_randlc(x), RandomPrivate_randlc(x)):complex;
  else
    return RandomPrivate_randlc(x):resultType;
}

//
// Return a value for the cursor so that the next call to randlc will
// return the same value as the nth call to randlc
//
pragma "no doc"
proc RandomPrivate_randlc_skipto(seed: int(64), in n: integral): real {
  var cursor = seed:real;
  n -= 1;
  var t = RandomPrivate_arand;
  RandomPrivate_arand;
  while (n != 0) {
    const i = n / 2;
    if (2 * i != n) then
      RandomPrivate_randlc(cursor, t);
    if i == 0 then
      break;
    else
      n = i;
    RandomPrivate_randlc(t, t);
    n = i;
  }
  return cursor;
}

//
// iterate over outer ranges in tuple of ranges
//
pragma "no doc"
iter RandomPrivate_outer(ranges, param dim: int = 1) {
  if dim + 1 == ranges.size {
    for i in ranges(dim) do
      yield (i,);
  } else if dim + 1 < ranges.size {
    for i in ranges(dim) do
      for j in RandomPrivate_outer(ranges, dim+1) do
        yield (i, (...j));
  } else {
    yield 0; // 1D case is a noop
  }
}

//
// RandomStream iterator implementation
//
pragma "no doc"
iter RandomPrivate_iterate(type resultType, D: domain, seed: int(64),
                          start: int(64)) {
  var cursor = RandomPrivate_randlc_skipto(seed, start);
  for i in D do
    yield RandomPrivate_randlc(resultType, cursor);
}

pragma "no doc"
iter RandomPrivate_iterate(type resultType, D: domain, seed: int(64),
                          start: int(64), param tag: iterKind)
      where tag == iterKind.leader {
  for block in D._value.these(tag=iterKind.leader) do
    yield block;
}

pragma "no doc"
iter RandomPrivate_iterate(type resultType, D: domain, seed: int(64),
                          start: int(64), param tag: iterKind, followThis)
      where tag == iterKind.follower {
  param multiplier = if resultType == complex then 2 else 1;
  const ZD = computeZeroBasedDomain(D);
  const innerRange = followThis(ZD.rank);
  var cursor: real;
  for outer in RandomPrivate_outer(followThis) {
    var myStart = start;
    // NOTE: Not bothering to check to see if this can fit into int(64)
    if ZD.rank > 1 then
      myStart += multiplier * ZD.indexOrder(((...outer), innerRange.low)):int(64);
    else
      myStart += multiplier * ZD.indexOrder(innerRange.low):int(64);
    if !innerRange.stridable {
      cursor = RandomPrivate_randlc_skipto(seed, myStart);
      for i in innerRange do
        yield RandomPrivate_randlc(resultType, cursor);
    } else {
      // NOTE: Not bothering to check to see if this can fit into int(64)
      myStart -= innerRange.low:int(64);
      for i in innerRange {
        cursor = RandomPrivate_randlc_skipto(seed, myStart + i:int(64) * multiplier);
        yield RandomPrivate_randlc(resultType, cursor);
      }
    }
  }
}

} // close module Random

/*
 * Copyright 2004-2015 Cray Inc.
 * Other additional copyright holders may be indicated within.
 *
 * The entirety of this work is licensed under the Apache License,
 * Version 2.0 (the "License"); you may not use this file except
 * in compliance with the License.
 *
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

/*
   Support for pseudorandom number generation

   This module defines an abstraction for a stream of pseudorandom
   numbers, :chpl:class:`RandomStream`.  It also provides a
   convenience function, :chpl:proc:`fillRandom` that can be used to
   fill an array with random numbers in parallel.

   The current pseudorandom number generator (PRNG) implemented by
   this module uses the algorithm from the NAS Parallel Benchmarks
   (NPB, available at: http://www.nas.nasa.gov/publications/npb.html).
   The longer-term intention is to add knobs to select between a menu
   of PRNG algorithms, such as the Mersenne twister.

   Paraphrasing the comments from the NPB reference implementation:

     This generator returns uniform pseudorandom real values in the
     range (0, 1) by using the linear congruential generator

       x_{k+1} = a x_k  (mod 2**46)

     where 0 < x_k < 2**46 and 0 < a < 2**46.  This scheme generates
     2**44 numbers before repeating.  The generated values are
     normalized to be between 0 and 1, i.e., 2**(-46) * x_k.

     This generator should produce the same results on any computer
     with at least 48 mantissa bits for `real(64)` data.

   Here is a list of currently open issues (TODOs) for this module:

   1. This module is currently restricted to generating `real(64)`,
   `imag(64)`, and `complex(128)` complex values.  We would like to
   extend this support to include other primitive types as well.

   2. We plan to support general serial and parallel iterators on the
   RandomStream class; however, providing the full suite of iterators
   is not possible with our current parallel iterator framework.
   Specifically, if :chpl:class:`RandomStream` is a follower in a
   zippered iteration context, there is no way for it to update the
   total number of random numbers generated in a safe/sane/coordinated
   way.  We are exploring a revised leader-follower iterator framework
   that would support this idiom (and other cursor-based ones).

   3. If no seed is provided by the user, one is chosen based on the
   current time in microseconds, allowing for some degree of
   pseudorandomness in seed selection.  The intent of
   :chpl:record:`SeedGenerators` is to provide a menu of other options
   for initializing the random stream seed, but only one option is
   implemented at present.

   4. As noted above, we plan to add support for additional PRNG
   algorithms over time.

*/
module Random {

  /*
     Note for developers on "private" symbols:

     It is the intent that once Chapel supports a notion of 'private'
     symbols, everything prefixed with RandomPrivate will be made
     private to this module and everything prefixed with
     'RandomStreamPrivate_' will be made private to the RandomStream
     class.
  */

  /* Internal TODOs for developers:

     - is the parSafe on each of the three main calls overkill?

     - should RandomStream be parameterized by the type to return?
       it seems odd that currently getNext() and getNth() return
       reals always...
  */

/*
   An instance of :chpl:record:`SeedGenerators` that provides a
   convenient means of generating seeds when the user does not wish to
   specify one manually.
*/
const SeedGenerator: SeedGenerators;

// CHPLDOC FIXME: Respect ordering of things in the file

/*
  Provides methods to help generate seeds when the user doesn't want
  to create one.  It currently only supports one such method, but the
  intention is to add more over time.

  (Note: once Chapel supports static class methods,
  :chpl:const:`SeedGenerator` and :chpl:record:`SeedGenerators` should
  be combined into a single record type with static methods).
*/

record SeedGenerators {
  /*
    Generate a seed based on the current time in microseconds as
    reported by :chpl:proc:`Time.getCurrentTime`, ensuring that it
    meets the PRNG's requirements.
  */
  proc currentTime {
    use Time;
    const seed: int(64) = getCurrentTime(unit=TimeUnits.microseconds):int(64);
    return (if seed % 2 == 0 then seed + 1 else seed) % (1:int(64) << 46);
  }
};

// CHPLDOC FEEDBACK: If easy, I'd suggest either deprecating the
// :arg <type> <name>: form or else switching the order to
// :arg <name> <type>: as it's an easy trap to simply type the same
// argument as in the function's signature.

// CHPLDOC FIXME: if the first line below is shifted one character to the
// left, it ends up being rendered like a method

// CHPLDOC FIXME: if the first line below is shifted one character to the
// right, it ends up causing a warning (promoted to error) in the .rst
// file.  It'd be preferable to have it declare the issue in terms of
// the .chpl line numbers.

//
// CHPLDOC FIXME: Do we want default values to print in the function
// prototype?  (I think we do, given that it reflects the code in the
// file)
//

/*
  Fill an array of `real(64)`, `imag(64)`, or `complex(128)` elements
  with pseudorandom values in parallel using a new
  :chpl:class:`RandomStream` created specifically for this call.  The
  first `arr.size` values from the stream will be assigned to the
  array's elements in row-major order for `real` and `imag` elements.
  For `complex` elements, consecutive pairs of random numbers are
  assigned to the real and imaginary components, respectively.  The
  parallelization strategy is determined by the array.

  :arg arr: The array to be filled, where T is real(64), imag(64), or complex(128).
  :type arr: [] T

  :arg seed: The seed to use for the PRNG.  Defaults to :chpl:proc:`SeedGenerator.currentTime <SeedGenerators.currentTime>`.
  :type seed: int
*/

proc fillRandom(arr: [], seed: int(64) = SeedGenerator.currentTime)
  where (arr.eltType == real || arr.eltType == imag || arr.eltType == complex) {
  var randNums = new RandomStream(seed, parSafe=false);
  randNums.fillRandom(arr);
  delete randNums;
}

pragma "no doc"
proc fillRandom(arr: [], seed: int(64) = SeedGenerator.currentTime) {
  compilerError("Random.fillRandom is only defined for real(64), imag(64), and complex(128) arrays");
}

/*
  Models a stream of pseudorandom numbers.  See the module-level
  notes for :chpl:mod:`Random` for details on the PRNG used.
*/
class RandomStream {
  /*
    Indicates whether or not the RandomStream needs to be
    parallel-safe by default.  If multiple tasks interact with it in
    an uncoordinated fashion, this must be set to `true`.  If it will
    only be called from a single task, or if only one task will call
    into it at a time, setting to `false` will reduce overhead related
    to ensuring mutual exclusion.
  */
  param parSafe: bool = true;
  /*
    The seed value for the PRNG.  It must be an odd integer in the
    range (1, 2**46).
  */
  const seed: int(64);

  //
  // CHPLDOC FIXME: More missing default values below
  //

  /*
    Constructs a new stream of random numbers using the specified seed
    and parallel safety.  Ensures that the seed value meets the PRNG's
    constraints.

    :arg seed: The seed to use for the PRNG.  Defaults to :chpl:proc:`SeedGenerator.currentTime <SeedGenerators.currentTime>`..
    :type seed: int

    :arg parSafe: The parallel safety setting.  Defaults to `true`.
    :type parSafe: bool
  */
  proc RandomStream(seed: int(64) = SeedGenerator.currentTime,
                   param parSafe: bool = true) {
    if seed % 2 == 0 || seed < 1 || seed > 1:int(64)<<46 then
      halt("RandomStream seed must be an odd integer between 0 and 2**46");
    RandomStreamPrivate_init(seed);
  }

  //
  // CHPLDOC FIXME: the type of parSafe below is printed out as _unknown
  //

  /*
    Returns the next value in the random stream as a `real(64)`

    :arg parSafe: Permits :chpl:param:`RandomStream.parSafe` to be overridden for this call.  Defaults to :chpl:param:`this.parSafe <RandomStream.parSafe>`.
    :type parSafe: bool

    :returns: The next value in the random stream as a `real(64)`.
   */
  proc getNext(param parSafe = this.parSafe): real(64) {
    if parSafe then
      RandomStreamPrivate_lock$ = true;
    RandomStreamPrivate_count += 1;
    const result = RandomPrivate_randlc(RandomStreamPrivate_cursor);
    if parSafe then
      RandomStreamPrivate_lock$;
    return result;
  }

  /*
    Advances/rewinds the stream to the `n`-th value in the sequence.

    :arg n: The position in the stream to skip to.  Must be non-negative.
    :type n: integral

    :arg parSafe: Permits :chpl:param:`RandomStream.parSafe` to be overridden for this call.  Defaults to this.parSafe.
    :type parSafe: bool
   */

  proc skipToNth(n: integral, param parSafe = this.parSafe) {
    if n <= 0 then
      halt("RandomStream.skipToNth(n) called with non-positive 'n' value", n);
    if parSafe then
      RandomStreamPrivate_lock$ = true;
    RandomStreamPrivate_count = n;
    RandomStreamPrivate_cursor = RandomPrivate_randlc_skipto(seed, n);
    if parSafe then
      RandomStreamPrivate_lock$;
  }

  /*
    Advance/rewind the stream to the `n`-th value and return it.  This
    is equivalent to :chpl:proc:`skipToNth()` followed by
    :chpl:proc:`getNext()`.

    :arg n: The position in the stream to skip to.  Must be non-negative.
    :type n: integral

    :arg parSafe: Permits :chpl:param:`RandomStream.parSafe` to be overridden for this call.  Defaults to this.parSafe.
    :type parSafe: bool

    :returns: The `n`-th value in the random stream as a `real(64)`.
  */

  proc getNth(n: integral, param parSafe = this.parSafe): real(64) {
    if (n <= 0) then
      halt("RandomStream.getNth(n) called with non-positive 'n' value", n);
    if parSafe then
      RandomStreamPrivate_lock$ = true;
    skipToNth(n, parSafe=false);
    const result = getNext(parSafe=false);
    if parSafe then
      RandomStreamPrivate_lock$;
    return result;
  }

  /*
    Fill the argument array with pseudorandom values.  This method is
    identical to the standalone :chpl:proc:`fillRandom` procedure,
    except that it consumes random values from the
    :chpl:class:`RandomStream` object on which it's invoked rather
    than creating a new stream for the purpose of the call.

    :arg arr: The array to be filled, where T is real(64), imag(64), or complex(128).
    :type arr: [] T

    :arg parSafe: Permits :chpl:param:`RandomStream.parSafe` to be overridden for this call.  Defaults to this.parSafe.
    :type parSafe: bool
  */

  proc fillRandom(arr: [], param parSafe = this.parSafe) {
    if arr.eltType != complex && arr.eltType != real && arr.eltType != imag then
      compilerError("RandomStream.fillRandom is only defined for real(64), imag(64), and complex(128) arrays");
    forall (x, r) in zip(arr, iterate(arr.domain, arr.eltType, parSafe)) do
      x = r;
  }

  pragma "no doc"
  proc iterate(D: domain, type resultType=real, param parSafe = this.parSafe) {
    if resultType != complex && resultType != real && resultType != imag then
      compilerError("RandomStream.iterate is only defined for real(64), imag(64), and complex(128) result types");
    param cplxMultiplier = if resultType == complex then 2 else 1;
    if parSafe then
      RandomStreamPrivate_lock$ = true;
    const start = RandomStreamPrivate_count;
    // NOTE: Not bothering to check to see if D.numIndices can fit into int(64)
    RandomStreamPrivate_count += cplxMultiplier * D.numIndices:int(64);
    skipToNth(RandomStreamPrivate_count, parSafe=false);
    if parSafe then
      RandomStreamPrivate_lock$;
    return RandomPrivate_iterate(resultType, D, seed, start);
  }

  pragma "no doc"
  proc writeThis(f: Writer) {
    f <~> "RandomStream(parSafe=";
    f <~> parSafe;
    f <~> ", seed = ";
    f <~> seed;
    f <~> ")";
  }

  ///////////////////////////////////////////////////////////// CLASS PRIVATE //
  //
  // It is the intent that once Chapel supports the notion of
  // 'private', everything in this class declared below this line will
  // be made private to this class.
  //

  pragma "no doc"
  var RandomStreamPrivate_lock$: sync bool;
  pragma "no doc"
  var RandomStreamPrivate_cursor: real;
  pragma "no doc"
  var RandomStreamPrivate_count: int(64);

  pragma "no doc"
  proc RandomStreamPrivate_init(seed: int(64)) {
    this.seed = seed;
    RandomStreamPrivate_cursor = seed;
    RandomStreamPrivate_count = 1;
  }
}

////////////////////////////////////////////////////////////// MODULE PRIVATE //
//
// It is the intent that once Chapel supports the notion of 'private',
// everything declared below this line will be made private to this
// module.
//

//
// NPB-defined constants for linear congruential generator
//
pragma "no doc"
const RandomPrivate_r23   = 0.5**23,
      RandomPrivate_t23   = 2.0**23,
      RandomPrivate_r46   = 0.5**46,
      RandomPrivate_t46   = 2.0**46,
      RandomPrivate_arand = 1220703125.0; // TODO: Is arand something that a
                                          // user might want to set on a
                                          // case-by-case basis?

//
// NPB-defined randlc routine
//
pragma "no doc"
proc RandomPrivate_randlc(inout x: real, a: real = RandomPrivate_arand) {
  var t1 = RandomPrivate_r23 * a;
  const a1 = floor(t1),
    a2 = a - RandomPrivate_t23 * a1;
  t1 = RandomPrivate_r23 * x;
  const x1 = floor(t1),
    x2 = x - RandomPrivate_t23 * x1;
  t1 = a1 * x2 + a2 * x1;
  const t2 = floor(RandomPrivate_r23 * t1),
    z  = t1 - RandomPrivate_t23 * t2,
    t3 = RandomPrivate_t23 * z + a2 * x2,
    t4 = floor(RandomPrivate_r46 * t3),
    x3 = t3 - RandomPrivate_t46 * t4;
  x = x3;
  return RandomPrivate_r46 * x3;
}

// Wrapper that takes a result type (two calls for complex types)
pragma "no doc"
proc RandomPrivate_randlc(type resultType, inout x: real) {
  if resultType == complex then
    return (RandomPrivate_randlc(x), RandomPrivate_randlc(x)):complex;
  else
    return RandomPrivate_randlc(x):resultType;
}

//
// Return a value for the cursor so that the next call to randlc will
// return the same value as the nth call to randlc
//
pragma "no doc"
proc RandomPrivate_randlc_skipto(seed: int(64), in n: integral): real {
  var cursor = seed:real;
  n -= 1;
  var t = RandomPrivate_arand;
  RandomPrivate_arand;
  while (n != 0) {
    const i = n / 2;
    if (2 * i != n) then
      RandomPrivate_randlc(cursor, t);
    if i == 0 then
      break;
    else
      n = i;
    RandomPrivate_randlc(t, t);
    n = i;
  }
  return cursor;
}

//
// iterate over outer ranges in tuple of ranges
//
pragma "no doc"
iter RandomPrivate_outer(ranges, param dim: int = 1) {
  if dim + 1 == ranges.size {
    for i in ranges(dim) do
      yield (i,);
  } else if dim + 1 < ranges.size {
    for i in ranges(dim) do
      for j in RandomPrivate_outer(ranges, dim+1) do
        yield (i, (...j));
  } else {
    yield 0; // 1D case is a noop
  }
}

//
// RandomStream iterator implementation
//
pragma "no doc"
iter RandomPrivate_iterate(type resultType, D: domain, seed: int(64),
                          start: int(64)) {
  var cursor = RandomPrivate_randlc_skipto(seed, start);
  for i in D do
    yield RandomPrivate_randlc(resultType, cursor);
}

pragma "no doc"
iter RandomPrivate_iterate(type resultType, D: domain, seed: int(64),
                          start: int(64), param tag: iterKind)
      where tag == iterKind.leader {
  for block in D._value.these(tag=iterKind.leader) do
    yield block;
}

pragma "no doc"
iter RandomPrivate_iterate(type resultType, D: domain, seed: int(64),
                          start: int(64), param tag: iterKind, followThis)
      where tag == iterKind.follower {
  param multiplier = if resultType == complex then 2 else 1;
  const ZD = computeZeroBasedDomain(D);
  const innerRange = followThis(ZD.rank);
  var cursor: real;
  for outer in RandomPrivate_outer(followThis) {
    var myStart = start;
    // NOTE: Not bothering to check to see if this can fit into int(64)
    if ZD.rank > 1 then
      myStart += multiplier * ZD.indexOrder(((...outer), innerRange.low)):int(64);
    else
      myStart += multiplier * ZD.indexOrder(innerRange.low):int(64);
    if !innerRange.stridable {
      cursor = RandomPrivate_randlc_skipto(seed, myStart);
      for i in innerRange do
        yield RandomPrivate_randlc(resultType, cursor);
    } else {
      // NOTE: Not bothering to check to see if this can fit into int(64)
      myStart -= innerRange.low:int(64);
      for i in innerRange {
        cursor = RandomPrivate_randlc_skipto(seed, myStart + i:int(64) * multiplier);
        yield RandomPrivate_randlc(resultType, cursor);
      }
    }
  }
}

} // close module Random


Fixing a small think-o in studies/graph500/kristyn/Graph500_1D_onV.

[not reviewed]

git-svn-id: 88467cb1fb04b8a755be7e1ee1026be4190196ef@22226 3a8e244f-b0f2-452b-bcba-4c88e055c3ca

module Graph500_main
{
  proc main () {

  use BlockDist;
  use Time;
  use Random;

  use Graph500_defs;
  use Scalable_Graph_Generator;
  use Construct_Graph;
  use Create_Parent_Tree;
  use Verify;

  const A: real = 0.57;
  const B: real = 0.19;
  const C: real = 0.19;
  const D: real = 0.05;

  extern proc output_results (scale: int, NV: int, edgefactor:int,
                const A: real, const B: real, const C: real, const D: real,
                generation_time: real, construction_time: real,
                nbfs: int,
                inout BFS_time_array: real, inout BFS_nedges_traversed: int );

  writeln ( "Problem Dimensions");
  writeln ( "              Scale: ", SCALE );
  writeln ( " Number of Vertices: ", N_VERTICES );
  writeln ( "        Edge factor: ", EDGEFACTOR );

  if (PRINT_LOCALE_INFORMATION) {
    coforall loc in Locales do on loc {
      writeln("Locale ID: ", loc.id, " of ", numLocales);
      writeln("Locale ID: ", loc.id, " Number of cores " , loc.numCores);
    }
  }

  const edge_range =  1..N_RAWEDGES;

  var Edges:[edgelist_domain] directed_vertex_pair;
  var Histogram:[vertex_domain] vertex_id=0;

  var generation_time: Timer;
  var construction_time: Timer;
  var BFS_time: [1..NUMROOTS] Timer;
  var BFS_time_array: [1..NUMROOTS] real;
  var BFS_nedges_traversed: [1..NUMROOTS] int;

  generation_time.start();
  Scalable_Data_Generator( SCALE, N_VERTICES, N_RAWEDGES, Edges);
  generation_time.stop();

  if(PRINT_TIMING_STATISTICS){
    writeln("Time for Scalable Data Generator = ", generation_time.elapsed());
  }

// Check min/max Node ID's

  var min_vertex_ID_start = min reduce Edges.start;
  var min_vertex_ID_end = min reduce Edges.end;
  var max_vertex_ID_start = max reduce Edges.start;
  var max_vertex_ID_end = max reduce Edges.end;

  var min_vertex_ID = min (min_vertex_ID_start, min_vertex_ID_end);
  var max_vertex_ID = max (max_vertex_ID_start, max_vertex_ID_end);

  writeln ();
  writeln ("Check for valid minimum and maximum vertex IDs ");
  writeln ("Minimum vertex ID ", min_vertex_ID);
  writeln ("Maximum vertex ID ", max_vertex_ID);
  writeln ();

  construction_time.start();

  var G = new Graph (vertex_domain);

  constructGraph (Edges, G);

  construction_time.stop();
  if(PRINT_TIMING_STATISTICS){
    writeln("Kernel 1, Graph Construction Time = ", construction_time.elapsed());
  }

  // Generate a list of valid starting roots
  // Valid starting roots have at least one edge to another node
  var Rand_Gen = new RandomStream (seed = 9);
  var Unif_Random: [1..NUM_CANDIDATES] real;
  Rand_Gen.fillRandom ( Unif_Random );
  var runID: int = 1;
  var candidate: int = 1;
  var root: vertex_id;

  var ParentTree:[vertex_domain] vertex_id=-1;
  var nedges_traversed: int;
  var nTEPs: real;

  writeln("Select candidate root and run BFS if criteria satisfied");
  writeln();

  while ( (runID <= NUMROOTS) && (candidate <= NUM_CANDIDATES )) do
  {
     root = floor (1+ Unif_Random[candidate]* N_VERTICES): vertex_id;
     writeln("Candidate root = ", root);

     // Verify that root is a valid candidate, ie has at least one edge
     if ( G.Vertices[root].neighbor_count > 0 ){

       BFS_time[runID].start();
       BFS ( root, ParentTree, G);
       BFS_time[runID].stop();

       BFS_time_array[runID] = BFS_time[runID].elapsed();
       BFS_nedges_traversed[runID] = verify_bfs_tree ( root, ParentTree, Edges);

       nTEPs = BFS_nedges_traversed[runID] / BFS_time[runID].elapsed();

       writeln("Root =  ", root );
       writeln("  Number of traversed edges = ", BFS_nedges_traversed[runID]);

       if (PRINT_TIMING_STATISTICS) {
          writeln("Time for BFS = ", BFS_time[runID].elapsed());
          writeln("TEPS = ", nTEPs);
       }

       runID += 1; candidate += 1;
       ParentTree = -1;

     }
     else
     {
        writeln("Root not a valid candidate");
        candidate += 1;
     }

  }

      if (PRINT_TIMING_STATISTICS) {
        output_results (SCALE, N_VERTICES, EDGEFACTOR, A, B, C, D,
             generation_time.elapsed(), construction_time.elapsed(), NUMROOTS,
             BFS_time_array(1), BFS_nedges_traversed(1) );
      }

}

}

module Graph500_main
{
  proc main () {

  use BlockDist;
  use Time;
  use Random;

  use Graph500_defs;
  use Scalable_Graph_Generator;
  use Construct_Graph;
  use Create_Parent_Tree;
  use Verify;

  const A: real = 0.57;
  const B: real = 0.19;
  const C: real = 0.19;
  const D: real = 0.05;

  extern proc output_results (scale: int, NV: int, edgefactor:int,
                const A: real, const B: real, const C: real, const D: real,
                generation_time: real, construction_time: real,
                nbfs: int,
                inout BFS_time_array: real, inout BFS_nedges_traversed: int );

  writeln ( "Problem Dimensions");
  writeln ( "              Scale: ", SCALE );
  writeln ( " Number of Vertices: ", N_VERTICES );
  writeln ( "        Edge factor: ", EDGEFACTOR );

  if (PRINT_LOCALE_INFORMATION) {
  coforall loc in Locales do
    {
    on loc do
      writeln("Locale ID: ", loc.id, " of ", numLocales);
      writeln("Locale ID: ", loc.id, " Number of cores " , loc.numCores);
    }
  }

  const edge_range =  1..N_RAWEDGES;

  var Edges:[edgelist_domain] directed_vertex_pair;
  var Histogram:[vertex_domain] vertex_id=0;

  var generation_time: Timer;
  var construction_time: Timer;
  var BFS_time: [1..NUMROOTS] Timer;
  var BFS_time_array: [1..NUMROOTS] real;
  var BFS_nedges_traversed: [1..NUMROOTS] int;

  generation_time.start();
  Scalable_Data_Generator( SCALE, N_VERTICES, N_RAWEDGES, Edges);
  generation_time.stop();

  if(PRINT_TIMING_STATISTICS){
    writeln("Time for Scalable Data Generator = ", generation_time.elapsed());
  }

// Check min/max Node ID's

  var min_vertex_ID_start = min reduce Edges.start;
  var min_vertex_ID_end = min reduce Edges.end;
  var max_vertex_ID_start = max reduce Edges.start;
  var max_vertex_ID_end = max reduce Edges.end;

  var min_vertex_ID = min (min_vertex_ID_start, min_vertex_ID_end);
  var max_vertex_ID = max (max_vertex_ID_start, max_vertex_ID_end);

  writeln ();
  writeln ("Check for valid minimum and maximum vertex IDs ");
  writeln ("Minimum vertex ID ", min_vertex_ID);
  writeln ("Maximum vertex ID ", max_vertex_ID);
  writeln ();

  construction_time.start();

  var G = new Graph (vertex_domain);

  constructGraph (Edges, G);

  construction_time.stop();
  if(PRINT_TIMING_STATISTICS){
    writeln("Kernel 1, Graph Construction Time = ", construction_time.elapsed());
  }

  // Generate a list of valid starting roots
  // Valid starting roots have at least one edge to another node
  var Rand_Gen = new RandomStream (seed = 9);
  var Unif_Random: [1..NUM_CANDIDATES] real;
  Rand_Gen.fillRandom ( Unif_Random );
  var runID: int = 1;
  var candidate: int = 1;
  var root: vertex_id;

  var ParentTree:[vertex_domain] vertex_id=-1;
  var nedges_traversed: int;
  var nTEPs: real;

  writeln("Select candidate root and run BFS if criteria satisfied");
  writeln();

  while ( (runID <= NUMROOTS) && (candidate <= NUM_CANDIDATES )) do
  {
     root = floor (1+ Unif_Random[candidate]* N_VERTICES): vertex_id;
     writeln("Candidate root = ", root);

     // Verify that root is a valid candidate, ie has at least one edge
     if ( G.Vertices[root].neighbor_count > 0 ){

       BFS_time[runID].start();
       BFS ( root, ParentTree, G);
       BFS_time[runID].stop();

       BFS_time_array[runID] = BFS_time[runID].elapsed();
       BFS_nedges_traversed[runID] = verify_bfs_tree ( root, ParentTree, Edges);

       nTEPs = BFS_nedges_traversed[runID] / BFS_time[runID].elapsed();

       writeln("Root =  ", root );
       writeln("  Number of traversed edges = ", BFS_nedges_traversed[runID]);

       if (PRINT_TIMING_STATISTICS) {
          writeln("Time for BFS = ", BFS_time[runID].elapsed());
          writeln("TEPS = ", nTEPs);
       }

       runID += 1; candidate += 1;
       ParentTree = -1;

     }
     else
     {
        writeln("Root not a valid candidate");
        candidate += 1;
     }

  }

      if (PRINT_TIMING_STATISTICS) {
        output_results (SCALE, N_VERTICES, EDGEFACTOR, A, B, C, D,
             generation_time.elapsed(), construction_time.elapsed(), NUMROOTS,
             BFS_time_array(1), BFS_nedges_traversed(1) );
      }

}

}


Promote my latest changes to SSCA2 to the release version

Promote r20441 and r20536 to the release version:

- Always use on clause in the forward pass
- More remote value forwarding
- Use Rachel's new barrier implementation
- Handle numTPVs > numLocales

git-svn-id: 88467cb1fb04b8a755be7e1ee1026be4190196ef@20557 3a8e244f-b0f2-452b-bcba-4c88e055c3ca

module SSCA2_kernels

//  +==========================================================================+
//  |  Polymorphic Implementation of SSCA #2, Kernels 2-4                      |
//  |                                                                          |
//  |  Each kernel takes a graph argument which provides for each vertex       |
//  |  1.  an iterator for its set of neighbors                                |
//  |  2.  a parallel integer array of edge weights, which can be zipper       |
//  |      iterated with the set of neighbors                                  |
//  |  3.  the number of neighbors                                             |
//  |                                                                          |
//  |  These are the only requirements on the representation of the graph.     |
//  |                                                                          |
//  |  Filtering in Kernel 4 is turned on or off by a compilation time param.  |
//  +==========================================================================+

{
  use SSCA2_compilation_config_params, Time;

  var stopwatch : Timer;

  // ========================================================
  //                           KERNEL 2:
  // ========================================================
  // Find the edges with the largest edges.  Return a list of
  // edges, all of which have the largest weight.
  // ========================================================

  proc largest_edges ( G, heavy_edge_list :domain )

    // edge_weights can be either an array over an associative
    // domain or over a sparse domain.  the output  heavy_edge_list
    // can either kind of domain or something else purpose-built
    // for this task.
    {
      if PRINT_TIMING_STATISTICS then stopwatch.start ();

      // ---------------------------------------------------------
      // find heaviest edge weight in a single pass over all edges
      // ---------------------------------------------------------

      proc maxweight(s) {
	var mymax = 0;
	for w in G.edge_weight[s] do
	  if w > mymax then mymax = w;
	return mymax;
      }

      const heaviest_edge_weight = max reduce [s in G.vertices] maxweight(s);

      // ---------------------------------------------
      // in a second pass over all edges, extract list
      // of all edges  matching the heaviest weight
      // ---------------------------------------------

      forall s in G.vertices do
        forall (t, w) in G.NeighborPairs (s) do

	  if w == heaviest_edge_weight then {
	    heavy_edge_list.add ( (s,t) );
	  };

      if PRINT_TIMING_STATISTICS then {
	stopwatch.stop ();
	writeln ( "Elapsed time for Kernel 2: ", stopwatch.elapsed (),
		  " seconds");
	stopwatch.clear ();
      }

      // ------------------------------------------------
      // should be able to write a user-defined "maxlocs"
      // reduction more efficiently than this scheme
      // ------------------------------------------------

      if DEBUG_KERNEL2 then {
	writeln ();
	writeln ( "Heaviest weight      : ", heaviest_edge_weight);
	writeln ( "Number of heavy edges:", heavy_edge_list.numIndices );
	writeln ();
	writeln ( "Edges with largest weight and other neighbors:" );
	for (s,t) in heavy_edge_list do {
	  writeln ("edge   ", (s,t));
	  for (v,w) in G.NeighborPairs (s) do
	    writeln ("      ", v, " ", w);}
      }
    };

  // ===================================================================
  //                              KERNEL 3:
  // ===================================================================
  // For each root (heavy) edge, find the subgraph (vertices and edges)
  // defined by directed paths of length no greater than max_path_length
  // in which the first edge traversed is the root edge
  // ===================================================================

  proc rooted_heavy_subgraphs ( G,
                                Heavy_Edge_List     : domain,
                                Heavy_Edge_Subgraph : [],
                                in max_path_length  : int )

    // -------------------------------------------------------------------------
    // there is a classic space versus time tradeoff.  if the subgraphs expanded
    // by breadth first search are small, it would make sense to use a hash
    // table or an associative domain to represent the search.  If the subgraphs
    // are large, using a full length vector to represent the search is more
    // appropriate.  We expect small diameters for power law graphs, so we
    // expect large subgraphs.
    // -------------------------------------------------------------------------

    {
      if PRINT_TIMING_STATISTICS then stopwatch.start ();

      const vertex_domain = G.vertices;

      forall ( x, y ) in Heavy_Edge_List do {
	var Active_Level, Next_Level : domain ( index (vertex_domain) );
	var min_distance             : [vertex_domain] atomic int;
        forall m in min_distance do m.write(-1);

	if DEBUG_KERNEL3 then
	  writeln ( " Building heavy edge subgraph from pair:", (x,y) );
	Active_Level.add ( y );
	Next_Level.clear ();
	Heavy_Edge_Subgraph ( (x, y) ).nodes.clear ();
	Heavy_Edge_Subgraph ( (x, y) ).edges.clear ();
	min_distance ( y ).write(0);

	Heavy_Edge_Subgraph ( (x, y) ).edges.add ( (x, y) );
	Heavy_Edge_Subgraph ( (x, y) ).nodes.add ( x );
	Heavy_Edge_Subgraph ( (x, y) ).nodes.add ( y );

	for path_length in 1 .. max_path_length do {

	  forall v in Active_Level do {

	    forall w in G.Neighbors (v) do {

              if min_distance(w).compareExchangeStrong(-1, path_length) then {
                Next_Level.add (w);
                Heavy_Edge_Subgraph ( (x, y) ).nodes.add (w);
	      }

	      if min_distance(w).read() == path_length then {
		Heavy_Edge_Subgraph ( (x, y) ).edges.add ( (v, w) );
	      }
	    }
	  }

	  if path_length < max_path_length then {
	    Active_Level = Next_Level;
	    Next_Level.clear ();
	  }
	}
      }

      if PRINT_TIMING_STATISTICS then {
	stopwatch.stop ();
	writeln ( "Elapsed time for Kernel 3: ", stopwatch.elapsed (),
		  " seconds");
	stopwatch.clear ();
      }
    } // end of rooted_heavy_subgraphs

  use BlockDist;
  // For task-private temporary variables
  config const defaultNumTPVs = 16;
  config var numTPVs = min(defaultNumTPVs, numLocales);
  // Would be nice to use PriavteDist, but aliasing is not supported (yet)
  const PrivateSpace = {LocaleSpace} dmapped Block(boundingBox={LocaleSpace});

  // ==================================================================
  //                              KERNEL 4
  // ==================================================================
  // Calculate Betweenness Centrality for simple unweighted directed or
  // undirected graphs, using Madduri, et.al.'s modification of
  // Brandes's 2001 algorithm
  // ==================================================================

  proc approximate_betweenness_centrality ( G, starting_vertices,
                                            Between_Cent : [] real,
                                            out Sum_Min_Dist : real )

    // -----------------------------------------------------------------------
    // The betweenness centrality metric for a given node  v  is defined
    // as the double sum over s not equal to v and  t not equal to
    // either s or v of the ratio of the number of shortest paths from s to t
    // passing through v to the number of shortest paths from s to t.
    //
    // Brandes's algorithm decomposes the computation of this metric into,
    // first, separate sums for each vertex s, which can be computed
    // independently in parallel, and
    // two, a recursive, tree-based, calculation of the path counts for
    // any particular s.
    // The complexity of this algorithm is O ( |V||E| ) time for an unweighted
    // graph.  The algorithm requires O ( |V| ) temporary space for each
    // process that executes instances of the outermost loop.
    // -----------------------------------------------------------------------
    {
      const vertex_domain = G.vertices;

      // Considering using a dense 1-d array instead.  This would
      // complicate the Level_Set implementation a little, but would
      // probably be more efficient.
      type Sparse_Vertex_List = domain(index(vertex_domain));

      var Between_Cent$ : [vertex_domain] atomic real;
      var Sum_Min_Dist$ : atomic real;

      //
      // Throughout kernel 4, we use distributed arrays that are
      // private to each iteration of the outer loop.  Because set-up
      // and clean-up of such variables is very expensive, it is
      // desirable to do it at most only once per task versus once
      // per iteration, as a task executes multiple iterations of a
      // forall loop.
      //
      // To avoid this per-iteration overhead, we have implemented
      // an optimization that make certain variables private to each
      // task.  These variables are reinitialized each iteration of
      // the loop, but not reallocated, reprivatized, etc.
      //
      // Note that the Chapel group and collaborators are currently in
      // the process of bringing the idea of task private variable to
      // the language.
      //

      numTPVs = min(numTPVs, starting_vertices.size);
      const TPVSpace = {0..#numTPVs};
      var TPVLocales: [TPVSpace] locale;

      if numTPVs <= numLocales {
        // Here we set up the locales array for our task private temporary
        // variables.  We use _computChunkStartEnd() to distribute the
        // variables approximately evenly across the 1d locales array.
        for t in TPVSpace {
          TPVLocales[t] = Locales[_computeChunkStartEnd(numLocales,
                                                        numTPVs, t+1)[1]-1];
        }
      } else {
        for t in TPVSpace {
          TPVLocales[t] = Locales[((t-1)/numTPVs)/numLocales];
        }
      }
      const TPVLocaleSpace = {TPVSpace} dmapped Block(boundingBox={TPVSpace},
                                                      targetLocales=TPVLocales);

      // There will be numTPVs copies of the temps, thus throttling the
      // number of starting vertices being considered simultaneously.
      var TPV: [TPVLocaleSpace] taskPrivateData(domain(index(vertex_domain)),
                                                vertex_domain.type);

      // Initialize
      coforall (t, i) in zip(TPVLocaleSpace, TPVSpace) do on TPVLocaleSpace.dist.idxToLocale(t) {
          TPV[i] = new taskPrivateData(domain(index(vertex_domain)), vertex_domain);
          var tpv = TPV[i];
          forall v in vertex_domain do
            tpv.BCaux[v].children_list.nd = {1..G.n_Neighbors[v]};
          for loc in Locales do on loc {
            tpv.Active_Level[here.id] = new Level_Set (Sparse_Vertex_List);
            tpv.Active_Level[here.id].previous = nil;
            tpv.Active_Level[here.id].next = new Level_Set (Sparse_Vertex_List);;
            tpv.Active_Level[here.id].next.previous = tpv.Active_Level[here.id];
            tpv.Active_Level[here.id].next.next = nil;
          }
      }
      var TPVM: TPVManager(TPV.type) = new TPVManager(TPV);

      // ------------------------------------------------------
      // Each iteration of the outer loop of Brandes's algorithm
      // computes the contribution (the "dependency" metric) for
      // one particular vertex  (s)  independently.
      // ------------------------------------------------------

      if PRINT_TIMING_STATISTICS then stopwatch.start ();

      forall s in starting_vertices do on vertex_domain.dist.idxToLocale(s) {

        const shere = here.id;

	if DEBUG_KERNEL4 then writeln ( "expanding from starting node ", s );

        // Initialize task private variables
        const tid = TPVM.gettid();
        const tpv = TPVM.getTPV(tid);
        var BCaux => tpv.BCaux;
        pragma "dont disable remote value forwarding"
        inline proc f1(BCaux, v) {
          BCaux[v].path_count$.write(0.0);
        }
        forall v in vertex_domain do {
          BCaux[v].depend = 0.0;
          BCaux[v].min_distance.write(-1);
          f1(BCaux, v);
          BCaux[v].children_list.child_count.write(0);
        }
	var Lcl_Sum_Min_Dist: atomic real;

	// The structure of the algorithm depends on a breadth-first
	// traversal. Each vertex will be marked by the length of
	// the shortest path (min_distance) from s to it. The array
	// path_count$ will hold a count of the number of shortest
	// paths from s to this node.  The number of paths in moderate
	// sized tori exceeds 2**64.

        //
        // Used to check termination of the forward pass
        //
        // We could use a task-private reduction variable here.  This
        // is yet another concept that the Chapel group is planning to
        // implement.
        //

        var remaining = true;

        //
        // Each locale will have its own level sets.  A locale's level set
        // will only contain nodes that are physically allocated on that
        // particular locale.
        //
        var Active_Level => tpv.Active_Level;
        pragma "dont disable remote value forwarding"
        inline proc f2(BCaux, s) {
          BCaux[s].path_count$.write(1.0);
        }

        const barrier = tpv.barrier;

        coforall loc in Locales do on loc {
          Active_Level[here.id].Members.clear();
          Active_Level[here.id].next.Members.clear();
          if vertex_domain.dist.idxToLocale(s) == here {
            // Establish the initial level sets for the breadth-first
            // traversal from s
            Active_Level[here.id].Members.add(s);
            BCaux[s].min_distance.write(0);
            f2(BCaux, s);
          }
          barrier.barrier();

          var current_distance : int = 0;

          while remaining do {
	    // ------------------------------------------------
	    // expand the neighbor sets for all vertices at the
	    // current distance from the starting vertex  s
	    // ------------------------------------------------

	    current_distance += 1;

            // The Chapel compiler is still a bit conservative when it
            // comes to forwarding read-only variables to remote
            // locales, so we make a const copy here to insure it is
            // forwarded to the remote locale in the following
            // coforall loop.
            const current_distance_c = current_distance;
            pragma "dont disable remote value forwarding"
            inline proc f3(BCaux, v, u, current_distance_c, Active_Level, Lcl_Sum_Min_Dist) {

                  // --------------------------------------------
                  // add any unmarked neighbors to the next level
                  // --------------------------------------------

                  if  BCaux[v].min_distance.compareExchangeStrong(-1, current_distance_c) {
                    Active_Level[here.id].next.Members.add (v);
                    if VALIDATE_BC then
                      Lcl_Sum_Min_Dist.add(current_distance_c);
                  }

                  // ------------------------------------------------
                  // only neighbors of  u  that are in the next level
                  // are on shortest paths from s through v.  Some
                  // task will have set  min_distance (v) by the
                  // time this code is reached, whether  v  lies in
                  // the previous, the current or the next level.
                  // ------------------------------------------------

                  if BCaux[v].min_distance.read() == current_distance_c {
                    BCaux[v].path_count$.add(BCaux[u].path_count$.read());
                    //f3(BCaux, v, u);
                    BCaux[u].children_list.add_child (v);
                  }

            }

            forall u in Active_Level[here.id].Members do {
              forall v in G.FilteredNeighbors(u) do on vertex_domain.dist.idxToLocale(v) {
                      f3(BCaux, v, u, current_distance_c, Active_Level, Lcl_Sum_Min_Dist);
                }
            };

            // This barrier is needed to insure all updates to the next
            // level are completed before updating to use the next level

            // do some work while we wait
            // barrier.notify(); // This is expensive without network atomics
            //  for now, just do a normal barrier

            if Active_Level[here.id].next.next == nil {
              Active_Level[here.id].next.next = new Level_Set (Sparse_Vertex_List);
              Active_Level[here.id].next.next.previous = Active_Level[here.id].next;
              Active_Level[here.id].next.next.next = nil;
            } else {
              Active_Level[here.id].next.next.Members.clear();
            }
            // barrier.wait(); // ditto
            barrier.barrier();
            if here.id==shere {
              remaining = false;
            }

            barrier.barrier();
            Active_Level[here.id] = Active_Level[here.id].next;
            if Active_Level[here.id].Members.numIndices:bool then
              remaining = true;

            barrier.barrier();

          };  // end forward pass

          if VALIDATE_BC then
            if here.id==0 then
              Sum_Min_Dist$.add(Lcl_Sum_Min_Dist);

          // -------------------------------------------------------------
          // compute the dependencies recursively, traversing the vertices
          // of the graph in non-increasing order of distance (reverse
          // ordering from the initial traversal)
          // -------------------------------------------------------------

          const graph_diameter = current_distance - 1;

          if DEBUG_KERNEL4 then
            if here.id==0 then
              writeln ( " graph diameter from starting node ", s,
                        "  is ", graph_diameter );

          pragma "dont disable remote value forwarding"
          inline proc f4(BCaux, Between_Cent$, u) {
            BCaux[u].depend = + reduce [v in BCaux[u].children_list.Row_Children[1..BCaux[u].children_list.child_count.read()]]
              ( BCaux[u].path_count$.read() /
                BCaux[v].path_count$.read() )      *
              ( 1.0 + BCaux[v].depend );
            Between_Cent$(u).add(BCaux[u].depend);
          }

          // back up to last level
          var curr_Level =  Active_Level[here.id].previous;

          for current_distance in 2 .. graph_diameter by -1 {
            curr_Level = curr_Level.previous;

            for u in curr_Level.Members do on vertex_domain.dist.idxToLocale(u) {
                f4(BCaux, Between_Cent$, u);
            }

            barrier.barrier();
          }
        }

        TPVM.releaseTPV(tid);

      }; // closure of outer embarassingly parallel forall

      if PRINT_TIMING_STATISTICS then {
	stopwatch.stop ();
	var K4_time = stopwatch.elapsed ();
	stopwatch.clear ();
	writeln ( "Elapsed time for Kernel 4: ", K4_time, " seconds");

	const n_edges          = G.num_edges;
	const N_VERTICES       = vertex_domain.numIndices;
	const N_START_VERTICES = if starting_vertices == G.vertices
                                 then N_VERTICES
                                      - + reduce [v in vertex_domain]
                                        (G.n_Neighbors (v) == 0)
                                 else starting_vertices.numIndices;
	const TEPS             = 7.0 * N_VERTICES * N_START_VERTICES / K4_time;
	const Adjusted_TEPS    = n_edges * N_START_VERTICES / K4_time;

	writeln ( "                     TEPS: ", TEPS );
	writeln ( " edge count adjusted TEPS: ", Adjusted_TEPS );
      }

      if VALIDATE_BC then
        Sum_Min_Dist = Sum_Min_Dist$.read();

      Between_Cent = Between_Cent$.read();

      if DELETE_KERNEL4_DS {
        coforall t in TPVSpace do on t {
          var tpv = TPV[t];
          delete tpv.barrier;
          var al = tpv.Active_Level;
          coforall loc in Locales do on loc {
            var level = al[here.id];
            while level != nil {
                var l2 = level.next;
                delete level;
                level = l2;
            }
          }
          delete TPV[t];
        }
        delete TPVM;
      }

    } // end of Brandes' betweenness centrality calculation

  //
  // Addition support data structures for kernel 4
  //

  // ------------------------------------------------------------
  // The set of vertices at a particular distance from s form a
  // level set.  The class allows the full set of vertices to be
  // partitioned into a linked list of level sets.  Each instance
  // of the outer loop in kernel 4 creates such a partitioning.
  // ============================================================

  class Level_Set {
    type Sparse_Vertex_List;
    var Members  : Sparse_Vertex_List;
    var previous : Level_Set (Sparse_Vertex_List);
    var next : Level_Set (Sparse_Vertex_List);
  }

  //
  // Data structure to save Children lists between the forward
  //  and backwards passes
  //
  record child_struct {
    type vertex;
    var nd: domain(1);
    var Row_Children: [nd] vertex;
    var child_count: atomic int;

    // This function should only be called using unique vertices
    proc add_child ( new_child: vertex ) {
      var c = child_count.fetchAdd(1)+1;
      Row_Children[c] = new_child;
    }
  }

  //
  // Implementation of task-private variables for kernel 4
  //
  record taskPrivateArrayData {
    type vertex;
    var min_distance  : atomic_int64; // used only on home locale
    var path_count$   : atomic real;
    var depend        : real;
    var children_list : child_struct(vertex);
  }

  class taskPrivateData {
    type Sparse_Vertex_List;
    const vertex_domain;
    var used  : atomic bool;
    var BCaux : [vertex_domain] taskPrivateArrayData(index(vertex_domain));
    var barrier = new Barrier(numLocales);
    var Active_Level : [PrivateSpace] Level_Set (Sparse_Vertex_List);
  }

  // This is a simple class that hands out task private variables from the
  // array of task private variables, TPV.
  class TPVManager {
    const TPV;
    var currTPV: atomic int;
    proc gettid() {
      const tid = this.currTPV.fetchAdd(1)%numTPVs;
      on this.TPV[tid] do
        while this.TPV[tid].used.testAndSet() do chpl_task_yield();
      return tid;
    }
    proc getTPV(tid) {
      return this.TPV[tid];
    }
    proc releaseTPV(tid) {
      this.TPV[tid].used.clear();
    }
  }

  //
  // reusable barrier implementation
  //
  class Barrier {
    param reusable = true;
    var n: int;
    var count: atomic int;
    var tasksFinished: [PrivateSpace] atomic bool;

    proc Barrier(_n: int) {
      reset(_n);
    }

    inline proc reset(_n: int) {
      on this {
        n = _n;
        count.write(n);
        for loc in tasksFinished {
          loc.write(false);
        }
      }
    }

    inline proc barrier() {
      const myc = count.fetchSub(1);
      if myc <= 1 {
        if tasksFinished[here.id].read() {
          halt("too many callers to barrier()");
        }
        for loc in tasksFinished {
          loc.write(true);
        }
        if reusable {
          count.waitFor(n-1);
          count.add(1);
          for loc in tasksFinished {
            loc.clear();
          }
        }
      } else {
        tasksFinished[here.id].waitFor(true);
        if reusable {
          count.add(1);
          tasksFinished[here.id].waitFor(false);
        }
      }
    }

    inline proc notify() {
      const myc = count.fetchSub(1);
      if myc <= 1 {
        if tasksFinished[here.id].read() {
          halt("too many callers to notify()");
        }
        for loc in tasksFinished {
          loc.write(true);
        }
      }
   }

    inline proc wait() {
      tasksFinished[here.id].waitFor(true);
      if reusable {
        var sum = count.fetchAdd(1);
        if sum == n-1 then tasksFinished.clear();
        else tasksFinished[here.id].waitFor(false);
      }
    }

    inline proc try() {
      return tasksFinished[here.id].read();
    }
  }
}

module SSCA2_kernels

//  +==========================================================================+
//  |  Polymorphic Implementation of SSCA #2, Kernels 2-4                      |
//  |                                                                          |
//  |  Each kernel takes a graph argument which provides for each vertex       |
//  |  1.  an iterator for its set of neighbors                                |
//  |  2.  a parallel integer array of edge weights, which can be zipper       |
//  |      iterated with the set of neighbors                                  |
//  |  3.  the number of neighbors                                             |
//  |                                                                          |
//  |  These are the only requirements on the representation of the graph.     |
//  |                                                                          |
//  |  Filtering in Kernel 4 is turned on or off by a compilation time param.  |
//  +==========================================================================+

{
  use SSCA2_compilation_config_params, Time;

  var stopwatch : Timer;

  // ========================================================
  //                           KERNEL 2:
  // ========================================================
  // Find the edges with the largest edges.  Return a list of
  // edges, all of which have the largest weight.
  // ========================================================

  proc largest_edges ( G, heavy_edge_list :domain )

    // edge_weights can be either an array over an associative
    // domain or over a sparse domain.  the output  heavy_edge_list
    // can either kind of domain or something else purpose-built
    // for this task.
    {
      if PRINT_TIMING_STATISTICS then stopwatch.start ();

      // ---------------------------------------------------------
      // find heaviest edge weight in a single pass over all edges
      // ---------------------------------------------------------

      proc maxweight(s) {
	var mymax = 0;
	for w in G.edge_weight[s] do
	  if w > mymax then mymax = w;
	return mymax;
      }

      const heaviest_edge_weight = max reduce [s in G.vertices] maxweight(s);

      // ---------------------------------------------
      // in a second pass over all edges, extract list
      // of all edges  matching the heaviest weight
      // ---------------------------------------------

      forall s in G.vertices do
        forall (t, w) in G.NeighborPairs (s) do

	  if w == heaviest_edge_weight then {
	    heavy_edge_list.add ( (s,t) );
	  };

      if PRINT_TIMING_STATISTICS then {
	stopwatch.stop ();
	writeln ( "Elapsed time for Kernel 2: ", stopwatch.elapsed (),
		  " seconds");
	stopwatch.clear ();
      }

      // ------------------------------------------------
      // should be able to write a user-defined "maxlocs"
      // reduction more efficiently than this scheme
      // ------------------------------------------------

      if DEBUG_KERNEL2 then {
	writeln ();
	writeln ( "Heaviest weight      : ", heaviest_edge_weight);
	writeln ( "Number of heavy edges:", heavy_edge_list.numIndices );
	writeln ();
	writeln ( "Edges with largest weight and other neighbors:" );
	for (s,t) in heavy_edge_list do {
	  writeln ("edge   ", (s,t));
	  for (v,w) in G.NeighborPairs (s) do
	    writeln ("      ", v, " ", w);}
      }
    };

  // ===================================================================
  //                              KERNEL 3:
  // ===================================================================
  // For each root (heavy) edge, find the subgraph (vertices and edges)
  // defined by directed paths of length no greater than max_path_length
  // in which the first edge traversed is the root edge
  // ===================================================================

  proc rooted_heavy_subgraphs ( G,
                                Heavy_Edge_List     : domain,
                                Heavy_Edge_Subgraph : [],
                                in max_path_length  : int )

    // -------------------------------------------------------------------------
    // there is a classic space versus time tradeoff.  if the subgraphs expanded
    // by breadth first search are small, it would make sense to use a hash
    // table or an associative domain to represent the search.  If the subgraphs
    // are large, using a full length vector to represent the search is more
    // appropriate.  We expect small diameters for power law graphs, so we
    // expect large subgraphs.
    // -------------------------------------------------------------------------

    {
      if PRINT_TIMING_STATISTICS then stopwatch.start ();

      const vertex_domain = G.vertices;

      forall ( x, y ) in Heavy_Edge_List do {
	var Active_Level, Next_Level : domain ( index (vertex_domain) );
	var min_distance             : [vertex_domain] atomic int;
        forall m in min_distance do m.write(-1);

	if DEBUG_KERNEL3 then
	  writeln ( " Building heavy edge subgraph from pair:", (x,y) );
	Active_Level.add ( y );
	Next_Level.clear ();
	Heavy_Edge_Subgraph ( (x, y) ).nodes.clear ();
	Heavy_Edge_Subgraph ( (x, y) ).edges.clear ();
	min_distance ( y ).write(0);

	Heavy_Edge_Subgraph ( (x, y) ).edges.add ( (x, y) );
	Heavy_Edge_Subgraph ( (x, y) ).nodes.add ( x );
	Heavy_Edge_Subgraph ( (x, y) ).nodes.add ( y );

	for path_length in 1 .. max_path_length do {

	  forall v in Active_Level do {

	    forall w in G.Neighbors (v) do {

              if min_distance(w).compareExchangeStrong(-1, path_length) then {
                Next_Level.add (w);
                Heavy_Edge_Subgraph ( (x, y) ).nodes.add (w);
	      }

	      if min_distance(w).read() == path_length then {
		Heavy_Edge_Subgraph ( (x, y) ).edges.add ( (v, w) );
	      }
	    }
	  }

	  if path_length < max_path_length then {
	    Active_Level = Next_Level;
	    Next_Level.clear ();
	  }
	}
      }

      if PRINT_TIMING_STATISTICS then {
	stopwatch.stop ();
	writeln ( "Elapsed time for Kernel 3: ", stopwatch.elapsed (),
		  " seconds");
	stopwatch.clear ();
      }
    } // end of rooted_heavy_subgraphs

  use BlockDist;
  config param useOnClause = CHPL_NETWORK_ATOMICS!="ugni";
  // For task-private temporary variables
  config const defaultNumTPVs = 16;
  config var numTPVs = min(defaultNumTPVs, numLocales);
  // Would be nice to use PriavteDist, but aliasing is not supported (yet)
  const PrivateSpace = {LocaleSpace} dmapped Block(boundingBox={LocaleSpace});

  // ==================================================================
  //                              KERNEL 4
  // ==================================================================
  // Calculate Betweenness Centrality for simple unweighted directed or
  // undirected graphs, using Madduri, et.al.'s modification of
  // Brandes's 2001 algorithm
  // ==================================================================

  proc approximate_betweenness_centrality ( G, starting_vertices,
                                            Between_Cent : [] real,
                                            out Sum_Min_Dist : real )

    // -----------------------------------------------------------------------
    // The betweenness centrality metric for a given node  v  is defined
    // as the double sum over s not equal to v and  t not equal to
    // either s or v of the ratio of the number of shortest paths from s to t
    // passing through v to the number of shortest paths from s to t.
    //
    // Brandes's algorithm decomposes the computation of this metric into,
    // first, separate sums for each vertex s, which can be computed
    // independently in parallel, and
    // two, a recursive, tree-based, calculation of the path counts for
    // any particular s.
    // The complexity of this algorithm is O ( |V||E| ) time for an unweighted
    // graph.  The algorithm requires O ( |V| ) temporary space for each
    // process that executes instances of the outermost loop.
    // -----------------------------------------------------------------------
    {
      const vertex_domain = G.vertices;

      // Considering using a dense 1-d array instead.  This would
      // complicate the Level_Set implementation a little, but would
      // probably be more efficient.
      type Sparse_Vertex_List = domain(index(vertex_domain));

      var Between_Cent$ : [vertex_domain] atomic real;
      var Sum_Min_Dist$ : atomic real;

      //
      // Throughout kernel 4, we use distributed arrays that are
      // private to each iteration of the outer loop.  Because set-up
      // and clean-up of such variables is very expensive, it is
      // desirable to do it at most only once per task versus once
      // per iteration, as a task executes multiple iterations of a
      // forall loop.
      //
      // To avoid this per-iteration overhead, we have implemented
      // an optimization that make certain variables private to each
      // task.  These variables are reinitialized each iteration of
      // the loop, but not reallocated, reprivatized, etc.
      //
      // Note that the Chapel group and collaborators are currently in
      // the process of bringing the idea of task private variable to
      // the language.
      //

      numTPVs = min(numTPVs, starting_vertices.size);
      const TPVSpace = {0..#numTPVs};
      var TPVLocales: [TPVSpace] locale;

      // Here we set up the locales array for our task private temporary
      // variables.  We use _computChunkStartEnd() to distribute the
      // variables approximately evenly across the 1d locales array.
      for t in TPVSpace {
        TPVLocales[t] = Locales[_computeChunkStartEnd(numLocales,
                                                      numTPVs, t+1)[1]-1];
      }
      const TPVLocaleSpace = {TPVSpace} dmapped Block(boundingBox={TPVSpace},
                                                      targetLocales=TPVLocales);

      // There will be numTPVs copies of the temps, thus throttling the
      // number of starting vertices being considered simultaneously.
      var TPV: [TPVLocaleSpace] taskPrivateData(domain(index(vertex_domain)),
                                                vertex_domain.type);

      // Initialize
      coforall (t, i) in zip(TPVLocaleSpace, TPVSpace) do on TPVLocaleSpace.dist.idxToLocale(t) {
          TPV[i] = new taskPrivateData(domain(index(vertex_domain)), vertex_domain);
          var tpv = TPV[i];
          forall v in vertex_domain do
            tpv.BCaux[v].children_list.nd = {1..G.n_Neighbors[v]};
          for loc in Locales do on loc {
            tpv.Active_Level[here.id] = new Level_Set (Sparse_Vertex_List);
            tpv.Active_Level[here.id].previous = nil;
            tpv.Active_Level[here.id].next = new Level_Set (Sparse_Vertex_List);;
            tpv.Active_Level[here.id].next.previous = tpv.Active_Level[here.id];
            tpv.Active_Level[here.id].next.next = nil;
          }
      }
      var TPVM: TPVManager(TPV.type) = new TPVManager(TPV);

      // ------------------------------------------------------
      // Each iteration of the outer loop of Brandes's algorithm
      // computes the contribution (the "dependency" metric) for
      // one particular vertex  (s)  independently.
      // ------------------------------------------------------

      if PRINT_TIMING_STATISTICS then stopwatch.start ();

      forall s in starting_vertices do on vertex_domain.dist.idxToLocale(s) {

        const shere = here.id;

	if DEBUG_KERNEL4 then writeln ( "expanding from starting node ", s );

        // Initialize task private variables
        const tid = TPVM.gettid();
        const tpv = TPVM.getTPV(tid);
        var BCaux => tpv.BCaux;
        pragma "dont disable remote value forwarding"
        inline proc f1(BCaux, v) {
          BCaux[v].path_count$.write(0.0);
        }
        forall v in vertex_domain do {
          BCaux[v].depend = 0.0;
          BCaux[v].min_distance.write(-1);
          f1(BCaux, v);
          BCaux[v].children_list.child_count.write(0);
        }
	var Lcl_Sum_Min_Dist: atomic real;

	// The structure of the algorithm depends on a breadth-first
	// traversal. Each vertex will be marked by the length of
	// the shortest path (min_distance) from s to it. The array
	// path_count$ will hold a count of the number of shortest
	// paths from s to this node.  The number of paths in moderate
	// sized tori exceeds 2**64.

        //
        // Used to check termination of the forward pass
        //
        // We could use a task-private reduction variable here.  This
        // is yet another concept that the Chapel group is planning to
        // implement.
        //

        var remaining = true;

        //
        // Each locale will have its own level sets.  A locale's level set
        // will only contain nodes that are physically allocated on that
        // particular locale.
        //
        var Active_Level => tpv.Active_Level;
        pragma "dont disable remote value forwarding"
        inline proc f2(BCaux, s) {
          BCaux[s].path_count$.write(1.0);
        }

        var barrier = new Barrier(numLocales);

        coforall loc in Locales do on loc {
          Active_Level[here.id].Members.clear();
          Active_Level[here.id].next.Members.clear();
          if vertex_domain.dist.idxToLocale(s) == here {
            // Establish the initial level sets for the breadth-first
            // traversal from s
            Active_Level[here.id].Members.add(s);
            BCaux[s].min_distance.write(0);
            f2(BCaux, s);
          }
          barrier.barrier();

          var current_distance : int = 0;

          while remaining do {
	    // ------------------------------------------------
	    // expand the neighbor sets for all vertices at the
	    // current distance from the starting vertex  s
	    // ------------------------------------------------

	    current_distance += 1;

            // The Chapel compiler is still a bit conservative when it
            // comes to forwarding read-only variables to remote
            // locales, so we make a const copy here to insure it is
            // forwarded to the remote locale in the following
            // coforall loop.
            const current_distance_c = current_distance;
            pragma "dont disable remote value forwarding"
            inline proc f3(BCaux, v, u) {
              BCaux[v].path_count$.add(BCaux[u].path_count$.read());
            }

            forall u in Active_Level[here.id].Members do {
              forall v in G.FilteredNeighbors(u) do
                on if useOnClause then vertex_domain.dist.idxToLocale(v) else here {
                  // --------------------------------------------
                  // add any unmarked neighbors to the next level
                  // --------------------------------------------

                  if  BCaux[v].min_distance.compareExchangeStrong(-1, current_distance_c) {
                    var aloc = if useOnClause then here.id else vertex_domain.dist.idxToLocale(v).id;
                    Active_Level[aloc].next.Members.add (v);
                    if VALIDATE_BC then
                      Lcl_Sum_Min_Dist.add(current_distance_c);
                  }

                  // ------------------------------------------------
                  // only neighbors of  u  that are in the next level
                  // are on shortest paths from s through v.  Some
                  // task will have set  min_distance (v) by the
                  // time this code is reached, whether  v  lies in
                  // the previous, the current or the next level.
                  // ------------------------------------------------

                  if BCaux[v].min_distance.read() == current_distance_c {
                    f3(BCaux, v, u);
                    BCaux[u].children_list.add_child (v);
                  }

                }
            };

            // This barrier is needed to insure all updates to the next
            // level are completed before updating to use the next level

            // do some work while we wait
            // barrier.notify(); // This is expensive without network atomics
            //  for now, just do a normal barrier

            if Active_Level[here.id].next.next == nil {
              Active_Level[here.id].next.next = new Level_Set (Sparse_Vertex_List);
              Active_Level[here.id].next.next.previous = Active_Level[here.id].next;
              Active_Level[here.id].next.next.next = nil;
            } else {
              Active_Level[here.id].next.next.Members.clear();
            }
            // barrier.wait(); // ditto
            barrier.barrier();
            if here.id==shere {
              remaining = false;
            }

            barrier.barrier();
            Active_Level[here.id] = Active_Level[here.id].next;
            if Active_Level[here.id].Members.numIndices:bool then
              remaining = true;

            barrier.barrier();

          };  // end forward pass

          if VALIDATE_BC then
            if here.id==0 then
              Sum_Min_Dist$.add(Lcl_Sum_Min_Dist);

          // -------------------------------------------------------------
          // compute the dependencies recursively, traversing the vertices
          // of the graph in non-increasing order of distance (reverse
          // ordering from the initial traversal)
          // -------------------------------------------------------------

          const graph_diameter = current_distance - 1;

          if DEBUG_KERNEL4 then
            if here.id==0 then
              writeln ( " graph diameter from starting node ", s,
                        "  is ", graph_diameter );

          pragma "dont disable remote value forwarding"
          inline proc f4(BCaux, Between_Cent$, u) {
            BCaux[u].depend = + reduce [v in BCaux[u].children_list.Row_Children[1..BCaux[u].children_list.child_count.read()]]
              ( BCaux[u].path_count$.read() /
                BCaux[v].path_count$.read() )      *
              ( 1.0 + BCaux[v].depend );
            Between_Cent$(u).add(BCaux[u].depend);
          }

          // back up to last level
          var curr_Level =  Active_Level[here.id].previous;

          for current_distance in 2 .. graph_diameter by -1 {
            curr_Level = curr_Level.previous;

            for u in curr_Level.Members do on vertex_domain.dist.idxToLocale(u) {
                f4(BCaux, Between_Cent$, u);
            }

            barrier.barrier();
          }
        }

        TPVM.releaseTPV(tid);

      }; // closure of outer embarassingly parallel forall

      if PRINT_TIMING_STATISTICS then {
	stopwatch.stop ();
	var K4_time = stopwatch.elapsed ();
	stopwatch.clear ();
	writeln ( "Elapsed time for Kernel 4: ", K4_time, " seconds");

	const n_edges          = G.num_edges;
	const N_VERTICES       = vertex_domain.numIndices;
	const N_START_VERTICES = if starting_vertices == G.vertices
                                 then N_VERTICES
                                      - + reduce [v in vertex_domain]
                                        (G.n_Neighbors (v) == 0)
                                 else starting_vertices.numIndices;
	const TEPS             = 7.0 * N_VERTICES * N_START_VERTICES / K4_time;
	const Adjusted_TEPS    = n_edges * N_START_VERTICES / K4_time;

	writeln ( "                     TEPS: ", TEPS );
	writeln ( " edge count adjusted TEPS: ", Adjusted_TEPS );
      }

      if VALIDATE_BC then
        Sum_Min_Dist = Sum_Min_Dist$.read();

      Between_Cent = Between_Cent$.read();

      if DELETE_KERNEL4_DS {
        coforall t in TPVSpace do on t {
          var tpv = TPV[t];
          var al = tpv.Active_Level;
          coforall loc in Locales do on loc {
            var level = al[here.id];
            while level != nil {
                var l2 = level.next;
                delete level;
                level = l2;
            }
          }
          delete TPV[t];
        }
        delete TPVM;
      }

    } // end of Brandes' betweenness centrality calculation

  //
  // Addition support data structures for kernel 4
  //

  // ------------------------------------------------------------
  // The set of vertices at a particular distance from s form a
  // level set.  The class allows the full set of vertices to be
  // partitioned into a linked list of level sets.  Each instance
  // of the outer loop in kernel 4 creates such a partitioning.
  // ============================================================

  class Level_Set {
    type Sparse_Vertex_List;
    var Members  : Sparse_Vertex_List;
    var previous : Level_Set (Sparse_Vertex_List);
    var next : Level_Set (Sparse_Vertex_List);
  }

  //
  // Data structure to save Children lists between the forward
  //  and backwards passes
  //
  record child_struct {
    type vertex;
    var nd: domain(1);
    var Row_Children: [nd] vertex;
    var child_count: atomic int;

    // This function should only be called using unique vertices
    proc add_child ( new_child: vertex ) {
      var c = child_count.fetchAdd(1)+1;
      Row_Children[c] = new_child;
    }
  }

  //
  // Implementation of task-private variables for kernel 4
  //
  record taskPrivateArrayData {
    type vertex;
    var min_distance  : atomic int;
    var path_count$   : atomic real;
    var depend        : real;
    var children_list : child_struct(vertex);
  }

  class taskPrivateData {
    type Sparse_Vertex_List;
    const vertex_domain;
    var used  : atomic bool;
    var BCaux : [vertex_domain] taskPrivateArrayData(index(vertex_domain));
    var Active_Level : [PrivateSpace] Level_Set (Sparse_Vertex_List);
  }

  // This is a simple class that hands out task private variables from the
  // array of task private variables, TPV.
  class TPVManager {
    const TPV;
    var currTPV: atomic int;
    proc gettid() {
      const tid = this.currTPV.fetchAdd(1)%numTPVs;
      on this.TPV[tid] do
        while this.TPV[tid].used.testAndSet() do chpl_task_yield();
      return tid;
    }
    proc getTPV(tid) {
      return this.TPV[tid];
    }
    proc releaseTPV(tid) {
      this.TPV[tid].used.clear();
    }
  }

  //
  // reusable barrier implementation
  //
  record Barrier {
    param reusable = true;
    var n: int;
    var count: atomic_int64;
    var done: atomicflag;

    proc Barrier(_n: int) {
      reset(_n);
    }

    inline proc reset(_n: int) {
      on this {
        n = _n;
        count.write(n);
        done.write(false);
      }
    }

    inline proc barrier() {
      on this {
        const myc = count.fetchSub(1);
        if myc<=1 {
          if done.testAndSet() then
            halt("Too many callers to barrier()");
          if reusable {
            count.waitFor(n-1);
            count.add(1);
            done.clear();
          }
        } else {
          done.waitFor(true);
          if reusable {
            count.add(1);
            done.waitFor(false);
          }
        }
      }
    }

    inline proc notify() {
      on this {
        const myc = count.fetchSub(1);
        if myc<=1 {
          if done.testAndSet() then
            halt("Too many callers to notify()");
        }
      }
    }

    inline proc wait() {
      on this {
        done.waitFor(true);
        if reusable {
          const myc = count.fetchAdd(1);
          if myc == n-1 then
            done.clear();
          done.waitFor(false);
        }
      }
    }

    inline proc try() {
      return done.read();
    }
  }

}


Second pass of norm code.

git-svn-id: 88467cb1fb04b8a755be7e1ee1026be4190196ef@8586 3a8e244f-b0f2-452b-bcba-4c88e055c3ca

module Norm{
  enum normType {norm1, norm2, normInf, normFrob};

  def norm(x: [], p: normType) where x.rank == 1 {
    select (p) {
      when norm1 do return + reduce abs(x);
      when norm2 do return sqrt(+ reduce (x*x));
      when normInf do return max reduce abs(x);
      when normFrob do halt("From-norm not defined for 1D arrays");
      otherwise halt("Unexpected norm type");
    }
  }

  def norm(x:[?D], p: normType) where x.rank == 2 {
    select (p) {
      when norm1 do
        return max reduce [j in D(2)] (+ reduce abs(x[D(1), j]));

      when norm2 do
        halt("Haven't implemented 2-norm for 2D arrays yet");

      when normInf do
        return max reduce [i in D(1)] (+ reduce abs(x[i, D(2)]));

      when normFrob do return sqrt(+ reduce abs(x));

      otherwise halt("Unexpected norm type");
    }
  }

  def norm(x:[?D], p: normType) where x.rank > 2 {
    compilerError("Norms not implemented for array ranks > 2D");
  }

  def norm(x: []) {
    select (x.rank) {
      when 1 do return norm(x, norm2);
      when 2 do return norm(x, normFrob);
      otherwise compilerError("Norms not implemented for array ranks > 2D");
    }
  }
}

module TestNorm {
  use Norm;

  def main() {
    const D1 = [1..5];
    var a:[D1] real;
    a = 2.0;
    writeln("Test of vector norms.  x = ");
    writeln(a);
    writeln("1-norm = ", norm(a, norm1));
    writeln("2-norm = " , norm(a, norm2));
    writeln("infinity norm = ", norm(a, normInf));
    writeln("default norm = ", norm(a));
    writeln();

    const D2 = [1..2,1..2];
    var b:[D2] real;
    b = 2.0;
    writeln("Test of matrix norms.  A = ");
    writeln(b);
    writeln("1-norm = ", norm(b, norm1));
    writeln("infinity norm = ", norm(b, normInf));
    writeln("frobenius norm = ", norm(b, normFrob));
    writeln("default norm = ", norm(b));
    writeln();

    const D3 = [1..3,1..2];
    var c:[D3] real;
    c = 2.0;
    writeln("Test of matrix norms.  A = ");
    writeln(c);
    writeln("1-norm = " , norm(c, norm1));
    writeln("infinity norm = ", norm(c, normInf));
    writeln("frobenius norm = ", norm(c, normFrob));
    writeln("default norm = ", norm(c));
    writeln();

    var d:[D2] int;
    d = 1;
    writeln("Test of matrix norms.  A = ");
    writeln(d);
    writeln("1-norm = ", norm(d, norm1));
    writeln("infinity norm = ", norm(d, normInf));
    writeln("frobenius norm = ", norm(d, normFrob));
    writeln("default norm = ", norm(d));
    writeln();

    var e:[D2] complex;
    e = 1.0 + 1.0i;
    writeln("Test of matrix norms.  A = ");
    writeln(e);
    writeln("1-norm = ", norm(e, norm1));
    writeln("infinity norm = ", norm(e, normInf));
    writeln("frobenius norm = ", norm(e, normFrob));
    writeln("default norm = ", norm(e));
    writeln();
  }
}

module Norm{
  enum normType {norm1, norm2, normInf, normFrob};

  def norm(x: [], p: normType) where x.rank == 1 {
    select (p) {
      when norm1 do return + reduce abs(x);
      when norm2 do return sqrt(+ reduce (x*x));
      when normInf do return max reduce abs(x);
      when normFrob do halt("From-norm not defined for 1D arrays");
      otherwise halt("Unexpected norm type");
    }
  }

  def norm(x:[?D], p: normType) where x.rank == 2 {
    select (p) {
      when norm1 {
        var maxColSum = abs(x(1,1));
        for j in D(2) {
          maxColSum = max(maxColSum,+ reduce abs(x[D(1),j]));
        }
        return maxColSum;
      }

      when norm2 {
        halt("Haven't implemented 2-norm for 2D arrays yet");
      }

      when normInf {
        var maxRowSum = abs(x(1,1));
        for i in D(1) {
          maxRowSum = max(maxRowSum,+ reduce abs(x[i,D(2)]));
        }
        return maxRowSum;
      }

      when normFrob do return sqrt(+ reduce abs(x));

      otherwise halt("Unexpected norm type");
    }
  }

  def norm(x:[?D], p: normType) where x.rank > 2 {
    compilerError("Norms not implemented for arrays greater than 2D");
  }

  def norm(x: []) {
    select (x.rank) {
      when 1 do return norm(x, norm2);
      when 2 do return norm(x, normFrob);
      otherwise compilerError("Norms not implemented for arrays greater than 2D");
    }
  }

}

module TestNorm {
  use Norm;

  def main() {
    const D1 = [1..5];
    var a:[D1] real;
    a = 2.0;
    writeln("Test of vector norms.  x = ");
    writeln(a);
    writeln("1-norm = ", norm(a, norm1));
    writeln("2-norm = " , norm(a, norm2));
    writeln("infinity norm = ", norm(a, normInf));
    writeln("default norm = ", norm(a));
    writeln();

    const D2 = [1..2,1..2];
    var b:[D2] real;
    b = 2.0;
    writeln("Test of matrix norms.  A = ");
    writeln(b);
    writeln("1-norm = ", norm(b, norm1));
    writeln("infinity norm = ", norm(b, normInf));
    writeln("frobenius norm = ", norm(b, normFrob));
    writeln("default norm = ", norm(b));
    writeln();

    const D3 = [1..3,1..2];
    var c:[D3] real;
    c = 2.0;
    writeln("Test of matrix norms.  A = ");
    writeln(c);
    writeln("1-norm = " , norm(c, norm1));
    writeln("infinity norm = ", norm(c, normInf));
    writeln("frobenius norm = ", norm(c, normFrob));
    writeln("default norm = ", norm(c));
    writeln();

    var d:[D2] int;
    d = 1;
    writeln("Test of matrix norms.  A = ");
    writeln(d);
    writeln("1-norm = ", norm(d, norm1));
    writeln("infinity norm = ", norm(d, normInf));
    writeln("frobenius norm = ", norm(d, normFrob));
    writeln("default norm = ", norm(d));
    writeln();

    var e:[D2] complex;
    e = 1.0 + 1.0i;
    writeln("Test of matrix norms.  A = ");
    writeln(e);
    writeln("1-norm = ", norm(e, norm1));
    writeln("infinity norm = ", norm(e, normInf));
    writeln("frobenius norm = ", norm(e, normFrob));
    writeln("default norm = ", norm(e));
    writeln();
  }
}