Instead of using Chapel's writeln() to print the value returned by getenv(), call C's printf() directly.  Chapel writeln() doesn't work for printing c_string arguments on non-0 locales, because writeln() executes on locale 0 and the pointer for a string on some other locale isn't meaningful on locale 0 (and we don't deep-copy the string).

extern proc getenv(const name: c_string): c_string;
extern proc printf(const fmt: c_string, const arg1: c_string);
for loc in Locales do on loc do printf('%s\n', getenv('GETENV_ENV_VAR'));

extern proc getenv(const name: c_string): c_string;
for loc in Locales do on loc do writeln(getenv('GETENV_ENV_VAR'));


Fixed my jacobi test that was incorrect with parallel default iterators.

There was an assignment to a non-sync var in a parallel loop. Changed the
problem loop to a + reduction and an array assignment.

git-svn-id: 88467cb1fb04b8a755be7e1ee1026be4190196ef@16201 3a8e244f-b0f2-452b-bcba-4c88e055c3ca

config var N = 4;
config var epsilon = 1.0e-5;
var D = [1..N, 1..N];
var BD= [0..N+1, 0..N+1];

def main(){
  var A : [BD] real(64);
  var tmp : [D] real(64);
  var delta : real(64);

  // set bottom border to 1.0
  [i in 1..N] A(N+1, i) = 1.0;
  do {
    [ij in D] tmp(ij) = (A(ij + (0, 1))  +
                         A(ij + (1, 0))  +
                         A(ij + (0, -1)) +
                         A(ij + (-1, 0))) / 4.0;

    delta = + reduce abs(tmp - A[D]);
    A[D] = tmp;
  } while (delta > epsilon);
  writeln(A(D));
}

def +(a: (int, int), b: (int, int)){
  return (a(1) + b(1), a(2) + b(2));
}

config var N = 4;
config var epsilon = 1.0e-5;
var D = [1..N, 1..N];
var BD= [0..N+1, 0..N+1];

def main(){
  var A : [BD] real(64);
  var tmp : [BD] real(64);
  var delta : real(64);

  // set bottom border to 1.0
  [i in 1..N] A(N+1, i) = 1.0;
  do {
    [ij in D] tmp(ij) = (A(ij + (0, 1))  +
                         A(ij + (1, 0))  +
                         A(ij + (0, -1)) +
                         A(ij + (-1, 0))) / 4.0;
    delta = 0;

    [ij in D] {
      delta = delta + abs(tmp(ij) - A(ij));
      A(ij) = tmp(ij);
    }
  } while (delta > epsilon);
  writeln(A(D));
}

def +(a: (int, int), b: (int, int)){
  return (a(1) + b(1), a(2) + b(2));
}


Minor comment/variable tweak.

Minor comment/variable tweak.

git-svn-id: 88467cb1fb04b8a755be7e1ee1026be4190196ef@19202 3a8e244f-b0f2-452b-bcba-4c88e055c3ca

// In this file:
//  * the "replicated" distribution
//  * support for user-level "replicated" variables

/*****************************************************************************
*** THE REPLICATED DISTRIBUTION ***

This ReplicatedDist distribution causes a domain and its arrays
to be replicated across the desired locales (all the locales by default).
An array receives a distinct set of elements - a "replicand" -
allocated on each locale.

In other words, mapping a domain with ReplicatedDist gives it
an implicit additional dimension - over the locales,
making it behave as if there is one copy of its indices per locale.

Replication over locales is observable:
- when iterating over a domain or array
- when printing with write() et al.
- when zippering and the replicated domain/array is
  the first among the zippered items
- when assigning into the replicated array
- when inquiring about the domain's numIndices
  or the array's numElements
- when accessing array element(s) from a locale that was not included
  in the array passed explicitly to the ReplicatedDist constructor,
  an out-of-bounds error will result

Only the replicand *on the current locale* is accessed
(i.e. replication is not observable):
- when examining certain domain properties:
  dim(d), dims(), low, high, stride;
  but not numIndices
- when indexing into an array
- when slicing an array  TODO: right?
- when zippering and the first zippered item is not replicated
- when assigning to a non-replicated array,
  i.e. the replicated array is on the right-hand side of the assignment
- when there is only a single locale (trivially: only one replicand)

E.g. when iterating, the number of iterations will be (the number of
locales involved) times (the number of iterations over this domain if
it were distributed with the default distribution).

Note that the above behavior may change in the future.

Features/limitations:
* Consistency/coherence among replicands' array elements is NOT maintained.
* Only rectangular domains are presently supported.
* Serial iteration over a replicated domain (or array) visits the indices
  (or array elements) of all replicands *from the current locale*.
* When replicating over user-provided array of locales, that array
  must be "consistent" (see below).

"Consistent" array requirement:
* The array of locales passed to the ReplicatedDist constructor, if any,
  must be "consistent".
* A is "consistent" if for each ix in A.domain, A[ix].id == ix.
* Tip: if the domain of the desired array of locales cannot be described
  as a rectangular domain (which could be strided, multi-dimensional,
  and/or sparse), make that array's domain associative over int.

Examples:

const Dbase = [1..5];
const Drepl: domain(1) dmapped ReplicatedDist() = Dbase;
var Abase: [Dbase] int;
var Arepl: [Drepl] int;

// only the current locale's replicand is accessed
Arepl[3] = 4;

// these iterate over Dbase, so
// only the current locale's replicand is accessed
forall (b,r) in (Abase,Arepl) b = r;
Abase = Arepl;

// these iterate over Drepl;
// each replicand will be zippered against (and copied from) the entire Abase
forall (r,b) in (Arepl,Abase) r = b;
Arepl = Abase;

// sequential zippering will detect difference in sizes
// (if multiple locales)
for (b,r) in (Abase,Arepl) ... // error
for (r,b) in (Arepl,Abase) ... // error

Potential extensions:
- support other kinds of domains
- allow run-time change in locales
*/

/*****************************************************************************
*** SUPPORT FOR USER-LEVEL REPLICATED VARIABLES ***

A "replicated" variable is a variable for which there is a copy on each locale.
Referencing a replicated variable (in a stylized way, see below)
accesses its copy on the current locale.

Features:
* The variable's copies are not synchronized automatically among the locales.

Limitations:
* It is "user-level", i.e. the user is required to handle the variable
  in specific ways to achieve the desired result.
* Tree-shape communication (like for reductions) is not provided.
* Using a replicated variable of an array type is not straightforward.
  Workaround: declare that array itself as replicated, then access it normally,
  e.g.:
   var replArray: [YOUR DOMAIN dmapped ReplicatedDist()] YOUR ELEMENT TYPE;

How to use replicated variables:
(Note: names that start with 'rc' are provided by this module.)

    use ReplicatedDist;

    // declare a replicated variable of the type 'MyType'
    var myRepVar: [rcDomain] MyType;

    // access its copy on the current locale (read or write) (either option)
    ... myRepVar(1) ...
    ... rcLocal(myRepVar) ...

    // "replicate": assign 'valToRep' to copies on all locales
    rcReplicate(myRepVar, valToRep);

    // "collect": assign from each copy of 'myRepVar' to
    // corresponding element of an array 'collected'
    var collected: [LocaleSpace] MyType;
    rcCollect(myRepVar, collected);

    // access directly a remote copy on the locale 'remoteLoc' (read or write)
    ... rcRemote(myRepVar, remoteLoc) ...

Advanced: replicate over a subset of locales 'myLocales'.
// 'myLocales' must be "consistent", as defined for ReplicatedDist above.
Use it as above, except modify the variable declarations as follows:

    var myRepVar: [rcDomainBase dmapped ReplicatedDist(myLocales,
                     "over which to replicate 'myRepVar'")] MyType;
    var collected: [myLocales.domain] MyType;

See also rcExample() and rcExampleOverLocales() in this file.
*/

/*****************************************************************************/
// THE REPLICATED DISTRIBUTION IMPLEMENTATION

// include locale information when printing out domains and arrays
config param printReplicatedLocales = true;

// trace certain DSI methods as they are being invoked
config param traceReplicatedDist = false;

/////////////////////////////////////////////////////////////////////////////
// distribution

//
// (global) distribution class
//
class ReplicatedDist : BaseDist {
  // the desired locales (an array of locales)
  const targetLocales;
  // "IDs" are indices into targetLocales
  proc targetIds return targetLocales.domain;

  // privatized object id
  var pid: int = -1;
}

// constructor: replicate over the given locales
// (by default, over all locales)
proc ReplicatedDist.ReplicatedDist(targetLocales: [] locale = Locales,
                 purposeMessage: string = "used to create a ReplicatedDist")
{
  if traceReplicatedDist then
    writeln("ReplicatedDist constructor over ", targetLocales);
  _localesCheckHelper(purposeMessage);
}

// helper to check consistency of the locales array
// TODO: going over all the locales - is there a scalability issue?
proc ReplicatedDist._localesCheckHelper(purposeMessage: string): void {
  // ideally would like to make this a "eureka"
  forall (ix, loc) in (targetIds, targetLocales) do
    if loc.id != ix {
      halt("The array of locales ", purposeMessage, " must be \"consistent\".",
           " See ReplicatedDist documentation for details.");
    }
}

// privatization

proc ReplicatedDist.dsiSupportsPrivatization() param return true;

proc ReplicatedDist.dsiGetPrivatizeData() {
  if traceReplicatedDist then writeln("ReplicatedDist.dsiGetPrivatizeData");

  // TODO: return the targetLocales array by value,
  // to reduce communication needed in dsiPrivatize()
  // perhaps by wrapping it in a class (or tuple?).
  return targetLocales;
}

proc ReplicatedDist.dsiPrivatize(privatizeData: this.targetLocales.type)
  : this.type
{
  if traceReplicatedDist then writeln("ReplicatedDist.dsiPrivatize on ", here);

  const pdTargetLocales = privatizeData;
  // make private copy of targetLocales and its domain
  const privTargetIds: domain(pdTargetLocales.domain.rank,
                              pdTargetLocales.domain.idxType,
                              pdTargetLocales.domain.stridable
                              ) = pdTargetLocales.domain;
  const privTargetLocales: [privTargetIds] locale = pdTargetLocales;
  return new ReplicatedDist(privTargetLocales, "used during privatization");
}

/////////////////////////////////////////////////////////////////////////////
// domains

//
// global domain class
//
class ReplicatedDom : BaseRectangularDom {
  // to support rectangular domains
  param rank: int;
  type idxType;
  param stridable: bool;
  // we need to be able to provide the domain map for our domain - to build its
  // runtime type (because the domain map is part of the type - for any domain)
  // (looks like it must be called exactly 'dist')
  const dist; // must be a ReplicatedDist

  // this is our index set; we store it here so we can get to it easily
  var domRep: domain(rank, idxType, stridable);

  // local domain objects
  // NOTE: 'dist' must be initialized prior to 'localDoms'
  // => currently have to use the default constructor
  // NOTE: if they ever change after the constructor - Reprivatize them
  var localDoms: [dist.targetIds] LocReplicatedDom(rank, idxType, stridable);

  proc numReplicands return localDoms.numElements;

  // privatized object id
  var pid: int = -1;
}

//
// local domain class
//
class LocReplicatedDom {
  // copy from the global domain
  param rank: int;
  type idxType;
  param stridable: bool;

  // our index set, copied from the global domain
  var domLocalRep: domain(rank, idxType, stridable);
}

// No explicit ReplicatedDom constructor - use the default one.
// proc ReplicatedDom.ReplicatedDom(...){...}

// Since we piggy-back on (default-mapped) Chapel domains, we can redirect
// a few operations to those. This function returns a Chapel domain
// that's fastest to access from the current locale.
// With privatization this is in the privatized copy of the ReplicatedDom.
//
// Not a parentheses-less method because of a bug as of r18460
// (see generic-parenthesesless-3.chpl).
proc ReplicatedDom.redirectee(): domain(rank, idxType, stridable)
  return domRep;

// The same across all domain maps
proc ReplicatedDom.dsiMyDist() return dist;

// privatization

proc ReplicatedDom.dsiSupportsPrivatization() param return true;

proc ReplicatedDom.dsiGetPrivatizeData() {
  if traceReplicatedDist then writeln("ReplicatedDom.dsiGetPrivatizeData");

  // TODO: perhaps return 'domRep' and 'localDoms' by value,
  // to reduce communication needed in dsiPrivatize().
  return (dist.pid, domRep, localDoms);
}

proc ReplicatedDom.dsiPrivatize(privatizeData): this.type {
  if traceReplicatedDist then writeln("ReplicatedDom.dsiPrivatize on ", here);

  var privdist = chpl_getPrivatizedCopy(this.dist.type, privatizeData(1));
  return new ReplicatedDom(rank=rank, idxType=idxType, stridable=stridable,
                           dist = privdist,
                           domRep = privatizeData(2),
                           localDoms = privatizeData(3));
}

proc ReplicatedDom.dsiGetReprivatizeData() {
  return tuple(domRep);
}

proc ReplicatedDom.dsiReprivatize(other, reprivatizeData): void {
  assert(this.rank == other.rank &&
         this.idxType == other.idxType &&
         this.stridable == other.stridable);

  this.domRep = reprivatizeData(1);
}

proc ReplicatedDist.dsiClone(): this.type {
  if traceReplicatedDist then writeln("ReplicatedDist.dsiClone");
  return new ReplicatedDist(targetLocales);
}

// create a new domain mapped with this distribution
proc ReplicatedDist.dsiNewRectangularDom(param rank: int,
                                         type idxType,
                                         param stridable: bool)
  : ReplicatedDom(rank, idxType, stridable, this.type)
{
  if traceReplicatedDist then writeln("ReplicatedDist.dsiNewRectangularDom ",
                                      (rank, typeToString(idxType), stridable));

  // Have to call the default constructor because we need to initialize 'dist'
  // prior to initializing 'localDoms' (which needs a non-nil value for 'dist'.
  var result = new ReplicatedDom(rank=rank, idxType=idxType,
                                 stridable=stridable, dist=this);

  // create local domain objects
  coforall (loc, locDom) in (targetLocales, result.localDoms) do
    on loc do
      locDom = new LocReplicatedDom(rank, idxType, stridable);

  return result;
}

// create a new domain mapped with this distribution representing 'ranges'
proc ReplicatedDom.dsiBuildRectangularDom(param rank: int,
                                          type idxType,
                                          param stridable: bool,
                                          ranges: rank * range(idxType,
                                                BoundedRangeType.bounded,
                                                               stridable))
  : ReplicatedDom(rank, idxType, stridable, this.dist.type)
{
  // could be made more efficient to avoid visiting each locale twice
  // but perhaps not a big deal, for now anyways
  var result = dist.dsiNewRectangularDom(rank, idxType, stridable);
  result.dsiSetIndices(ranges);
  return result;
}

// Given an index, this should return the locale that owns that index.
// (This is the implementation of dmap.idxToLocale().)
// For ReplicatedDist, we point it to the current locale.
proc ReplicatedDist.dsiIndexToLocale(indexx): locale {
  return here;
}

/*
dsiSetIndices accepts ranges because it is invoked so from ChapelArray or so.
Most dsiSetIndices() on a tuple of ranges can be the same as this one.
Or that call dsiSetIndices(ranges) could be converted following this example.
*/
proc ReplicatedDom.dsiSetIndices(rangesArg: rank * range(idxType,
                                          BoundedRangeType.bounded,
                                                         stridable)): void {
  if traceReplicatedDist then
    writeln("ReplicatedDom.dsiSetIndices on ranges");
  dsiSetIndices([(...rangesArg)]);
}

proc ReplicatedDom.dsiSetIndices(domArg: domain(rank, idxType, stridable)): void {
  if traceReplicatedDist then
    writeln("ReplicatedDom.dsiSetIndices on domain ", domArg);
  domRep = domArg;
  coforall locDom in localDoms do
    on locDom do
      locDom.domLocalRep = domArg;
}

proc ReplicatedDom.dsiGetIndices(): rank * range(idxType,
                                                 BoundedRangeType.bounded,
                                                 stridable) {
  if traceReplicatedDist then writeln("ReplicatedDom.dsiGetIndices");
  return redirectee().getIndices();
}

// Iterators over the domain's indices (serial, leader, follower).
// Our semantics: yield each of the domain's indices once per each locale.

// Serial iterator: the compiler forces it to be completely serial
iter ReplicatedDom.these() {
  // compiler does not allow 'on' here (see r16137 and nestedForall*)
  // so instead of ...
  //---
  //for locDom in localDoms do
  //  on locDom do
  //    for i in locDom.domLocalRep do
  //      yield i;
  //---
  // ... so we simply do the same a few times
  var dom = redirectee();
  for count in 1..#numReplicands do
    for i in dom do
      yield i;
}

iter ReplicatedDom.these(param tag: iterator) where tag == iterator.leader {
  coforall locDom in localDoms do
    on locDom do
      // there, for simplicity, redirect to DefaultRectangular's leader
      for follow in locDom.domLocalRep._value.these(tag) do
        yield follow;
}

iter ReplicatedDom.these(param tag: iterator, follower) where tag == iterator.follower {
  // redirect to DefaultRectangular
  for i in redirectee()._value.these(tag, follower) do
    yield i;
}

/* Write the domain out to the given Writer serially. */
proc ReplicatedDom.dsiSerialWrite(f: Writer): void {
  // redirect to DefaultRectangular
  redirectee()._value.dsiSerialWrite(f);
  if printReplicatedLocales {
    f.write(" replicated over ");
    dist.targetLocales._value.dsiSerialWrite(f);
  }
}

proc ReplicatedDom.dsiDims(): rank * range(idxType,
                                           BoundedRangeType.bounded,
                                           stridable)
  return redirectee().dims();

proc ReplicatedDom.dsiDim(dim: int): range(idxType,
                                           BoundedRangeType.bounded,
                                           stridable)
  return redirectee().dim(dim);

proc ReplicatedDom.dsiLow
  return redirectee().low;

proc ReplicatedDom.dsiHigh
  return redirectee().high;

proc ReplicatedDom.dsiStride
  return redirectee().stride;

// here replication is visible
proc ReplicatedDom.dsiNumIndices
  return redirectee().numIndices * numReplicands;

proc ReplicatedDom.dsiMember(indexx)
  return redirectee().member(indexx);

proc ReplicatedDom.dsiIndexOrder(indexx)
  return redirectee().dsiIndexOrder(indexx);

/////////////////////////////////////////////////////////////////////////////
// arrays

//
// global array class
//
class ReplicatedArr : BaseArr {
  // These two are hard-coded in the compiler - it computes the array's
  // type string as '[dom.type] eltType.type'
  type eltType;
  const dom; // must be a ReplicatedDom

  // the replicated arrays
  // NOTE: 'dom' must be initialized prior to initializing 'localArrs'
  var localArrs: [dom.dist.targetIds]
              LocReplicatedArr(eltType, dom.rank, dom.idxType, dom.stridable);

  // privatized object id
  var pid: int = -1;
}

//
// local array class
//
class LocReplicatedArr {
  // these generic fields let us give types to the other fields easily
  type eltType;
  param rank: int;
  type idxType;
  param stridable: bool;

  var myDom: LocReplicatedDom(rank, idxType, stridable);
  var arrLocalRep: [myDom.domLocalRep] eltType;
}

// ReplicatedArr constructor.
// We create our own to make field initializations convenient:
// 'eltType' and 'dom' as passed explicitly;
// the fields in the parent class, BaseArr, are initialized to their defaults.
//
proc ReplicatedArr.ReplicatedArr(type eltType, dom: ReplicatedDom) {
  // initializes the fields 'eltType', 'dom' by name
}

// The same across all domain maps
proc ReplicatedArr.dsiGetBaseDom() return dom;

// privatization

proc ReplicatedArr.dsiSupportsPrivatization() param return true;

proc ReplicatedArr.dsiGetPrivatizeData() {
  if traceReplicatedDist then writeln("ReplicatedArr.dsiGetPrivatizeData");

  // TODO: perhaps return 'localArrs' by value,
  // to reduce communication needed in dsiPrivatize().
  return (dom.pid, localArrs);
}

proc ReplicatedArr.dsiPrivatize(privatizeData) {
  if traceReplicatedDist then writeln("ReplicatedArr.dsiPrivatize on ", here);

  var privdom = chpl_getPrivatizedCopy(this.dom.type, privatizeData(1));
  var result = new ReplicatedArr(eltType, privdom);
  result.localArrs = privatizeData(2);
  return result;
}

// create a new array over this domain
proc ReplicatedDom.dsiBuildArray(type eltType)
  : ReplicatedArr(eltType, this.type)
{
  if traceReplicatedDist then writeln("ReplicatedDom.dsiBuildArray");
  var result = new ReplicatedArr(eltType, this);
  coforall (loc, locDom, locArr)
   in (dist.targetLocales, localDoms, result.localArrs) do
    on loc do
      locArr = new LocReplicatedArr(eltType, rank, idxType, stridable,
                                    locDom);
  return result;
}

// Return the array element corresponding to the index - on the current locale
proc ReplicatedArr.dsiAccess(indexx) var: eltType {
  return localArrs[here.id].arrLocalRep[indexx];
}

// Write the array out to the given Writer serially.
proc ReplicatedArr.dsiSerialWrite(f: Writer): void {
  var neednl = false;
  for locArr in localArrs {
//  on locArr {  // may cause deadlock
      if neednl then f.write("\n"); neednl = true;
      if printReplicatedLocales then
        f.write(locArr.locale, ":\n");
      locArr.arrLocalRep._value.dsiSerialWrite(f);
//  }
  }
}

// iterators

// completely serial
iter ReplicatedArr.these() var: eltType {
  for locArr in localArrs do
//  on locArr do // compiler does not allow; see r16137 and nestedForall*
      for a in locArr.arrLocalRep do
        yield a;
}

iter ReplicatedArr.these(param tag: iterator) where tag == iterator.leader {
  // redirect to ReplicatedDom's leader
  for follow in dom.these(tag) do
    yield follow;
}

iter ReplicatedArr.these(param tag: iterator, follower) var where tag == iterator.follower {
  // redirect to DefaultRectangular
  for a in localArrs[here.id].arrLocalRep._value.these(tag, follower) do
    yield a;
}

/////////////////////////////////////////////////////////////////////////////
// slicing, reindexing, etc.

// This supports reassignment of the array's domain.
/*
This gets invoked upon reassignment of the array's domain,
prior to calling this.dom.dsiSetIndices().
So this needs to adjust anything in the array that won't be taken care of
in this.dom.dsiSetIndices(). In our case, that's nothing.
*/
proc ReplicatedArr.dsiReallocate(d: domain): void {
  if traceReplicatedDist then
    writeln("ReplicatedArr.dsiReallocate ", dom.domRep, " -> ", d, " (no-op)");
}

// array slicing
proc ReplicatedArr.dsiSlice(sliceDef: ReplicatedDom) {
  if traceReplicatedDist then writeln("ReplicatedArr.dsiSlice on ", sliceDef);
  const slicee = this;
  const result = new ReplicatedArr(slicee.eltType, sliceDef);

  // ensure sliceDef and slicee are over the same set of locales/targetIds
  assert(sliceDef.localDoms.domain == slicee.localArrs.domain);

  coforall (loc, sliceDefLocDom, sliceeLocArr, resultLocArr)
   in (sliceDef.dist.targetLocales, sliceDef.localDoms,
       slicee.localArrs, result.localArrs) do
    on loc do
      resultLocArr = new LocReplicatedArr(eltType,
        sliceDef.rank, sliceDef.idxType, sliceDef.stridable,
        myDom = sliceDefLocDom,
        arrLocalRep => sliceeLocArr.arrLocalRep[sliceDefLocDom.domLocalRep]);

  return result;
}

// array reindexing
// very similar to array slicing
proc ReplicatedArr.dsiReindex(sliceDef: ReplicatedDom) {
  if traceReplicatedDist then writeln("ReplicatedArr.dsiReindex on ", sliceDef);
  var result = new ReplicatedArr(eltType, sliceDef);
  var slicee = this;

  // ensure 'dom' and 'slicee' are over the same set of locales/targetIds
  assert(sliceDef.localDoms.domain == slicee.localArrs.domain);

  coforall (loc, sliceDefLocDom, sliceeLocArr, resultLocArr)
   in (sliceDef.dist.targetLocales, sliceDef.localDoms,
       slicee.localArrs, result.localArrs) do
    on loc do
     {
      var locAlias: [sliceDefLocDom.domLocalRep] => sliceeLocArr.arrLocalRep;
      resultLocArr = new LocReplicatedArr(eltType,
        sliceDef.rank, sliceDef.idxType, sliceDef.stridable,
        myDom = sliceDefLocDom,
        arrLocalRep => locAlias);
     }

  return result;
}

// rank-change slicing
// very similar to slicing
proc ReplicatedArr.dsiRankChange(sliceDef: ReplicatedDom,
                                 param newRank: int,
                                 param newStridable: bool,
                                 args) {
  if traceReplicatedDist then writeln("ReplicatedArr.dsiRankChange");
  var result = new ReplicatedArr(eltType, sliceDef);
  var slicee = this;

  // ensure 'dom' and 'slicee' are over the same set of locales/targetIds
  assert(sliceDef.localDoms.domain == slicee.localArrs.domain);

  coforall (loc, sliceDefLocDom, sliceeLocArr, resultLocArr)
   in (sliceDef.dist.targetLocales, sliceDef.localDoms,
       slicee.localArrs, result.localArrs) do
    on loc do
      resultLocArr = new LocReplicatedArr(eltType,
        sliceDef.rank, sliceDef.idxType, sliceDef.stridable,
        myDom = sliceDefLocDom,
        arrLocalRep => sliceeLocArr.arrLocalRep[(...args)]);

  return result;
}

// todo? these two seem to work (written by analogy with DefaultRectangular)
proc ReplicatedDist.dsiCreateReindexDist(newSpace, oldSpace) return this;
proc ReplicatedDist.dsiCreateRankChangeDist(param newRank, args) return this;

/*****************************************************************************/
// SUPPORT FOR USER-LEVEL REPLICATED VARIABLES

const rcDomainIx   = 1;
const rcDomainBase = [rcDomainIx..rcDomainIx];
const rcLocales    = Locales;
const rcDomainMap  = new ReplicatedDist(rcLocales);
const rcDomain     = rcDomainBase dmapped new dmap(rcDomainMap);
const rcCollectDomaim = rcLocales.domain;
param _rcErr1 = " must be 'rcDomain' or 'rcDomainBase dmapped ReplicatedDist(an array of locales)'";

proc _rcTargetLocalesHelper(replicatedVar: [?D])
  where replicatedVar._value.type: ReplicatedArr
{
  return replicatedVar._value.dom.dist.targetLocales;
}

proc rcReplicate(replicatedVar: [?D] ?MYTYPE, valToReplicate: MYTYPE): void
{ compilerError("the domain of first argument to rcReplicate()", _rcErr1); }

proc rcReplicate(replicatedVar: [?D] ?MYTYPE, valToReplicate: MYTYPE): void
  where replicatedVar._value.type: ReplicatedArr
{
  assert(replicatedVar.domain == rcDomainBase);
  coforall loc in _rcTargetLocalesHelper(replicatedVar) do
    on loc do
      replicatedVar[rcDomainIx] = valToReplicate;
}

proc rcCollect(replicatedVar: [?D] ?MYTYPE, collected: [?CD] MYTYPE): void
  where ! replicatedVar._value.type: ReplicatedArr
{ compilerError("the domain of first argument to rcCollect()", _rcErr1); }

proc rcCollect(replicatedVar: [?D] ?MYTYPE, collected: [?CD] MYTYPE): void
  where replicatedVar._value.type: ReplicatedArr
{
  var targetLocales = _rcTargetLocalesHelper(replicatedVar);
  assert(replicatedVar.domain == rcDomainBase);
  assert(collected.domain == targetLocales.domain);
  coforall (loc, col) in (targetLocales, collected) do
    on loc do
      col = replicatedVar[rcDomainIx];
}

proc rcLocal(replicatedVar: [?D] ?MYTYPE) var: MYTYPE {
  return replicatedVar[rcDomainIx];
}

proc rcRemote(replicatedVar: [?D] ?MYTYPE, remoteLoc: locale) var: MYTYPE {
  var result: MYTYPE;
  on remoteLoc do
    result = replicatedVar[rcDomainIx];
  return result;
}

/////////////////////////////////////////////////////////////////////////////
// code examples

// This example illustrates the use of a replicated variable as follows:
// - create a replicated variable of the type MyType,
// - initializes all its copies to 'initVal',
// - stores 'newVal' into its copy on the locale 'newLocale',
// - collects all its copies into an array 'collected'.
//
proc rcExample(initVal: ?MyType, newVal: MyType, newLocale: locale): void {
  writeln("starting rcExample");

  // declare a replicated variable
  var myRepVar: [rcDomain] MyType;

  // initialize all copies to 'initVal'
  rcReplicate(myRepVar, initVal);

  writeln("\nafter initialization, myRepVar copies are:\n", myRepVar);

  // go to 'newLocale' and update its copy to 'newVal'
  on newLocale {
    // myRepVar[1] references myRepVar's copy on the current locale
    writeln("\non ", newLocale, ", myRepVar is:\n",
            "  before assignment: ", myRepVar[1]);

    // update the current locale's copy
    myRepVar[1] = newVal;

    writeln("  after assignment:  ", myRepVar[1]);
  }

  writeln("\nafter update, myRepVar copies are:\n", myRepVar);

  // collect all copies of 'myRepVar' into an array
  var collected: [LocaleSpace] MyType;
  rcCollect(myRepVar, collected);

  writeln("\ncollected copies of myRepVar are:\n", collected);
}

// This is the same as 'rcExample', except the user can provide
// specific locales to replicate over. The two differences are marked.
//
proc rcExampleOverLocales(initVal: ?MyType, newVal: MyType, newLocale: locale,
                          localesToReplicateOver: [] locale = Locales): void {
  writeln("starting rcExampleOverLocales over ", localesToReplicateOver);

  // declare a replicated variable
  // DIFFERENT: the domain in myRepVar's type
  var myRepVar: [rcDomainBase dmapped ReplicatedDist(localesToReplicateOver,
   "over which to replicate 'myRepVar' in rcExampleOverLocales()")] MyType;

  // initialize all copies to 'initVal'
  rcReplicate(myRepVar, initVal);

  writeln("\nafter initialization, myRepVar copies are:\n", myRepVar);

  // go to 'newLocale' and update its copy to 'newVal'
  on newLocale {
    // myRepVar[1] references myRepVar's copy on the current locale
    writeln("\non ", newLocale, ", myRepVar is:\n",
            "  before assignment: ", myRepVar[1]);

    // update the current locale's copy
    myRepVar[1] = newVal;

    writeln("  after assignment:  ", myRepVar[1]);
  }

  writeln("\nafter update, myRepVar copies are:\n", myRepVar);

  // collect all copies of 'myRepVar' into an array
  // DIFFERENT: the domain in collected's type
  var collected: [localesToReplicateOver.domain] MyType;
  rcCollect(myRepVar, collected);

  writeln("\ncollected copies of myRepVar are:\n", collected);
}

// In this file:
//  * the "replicated" distribution
//  * support for user-level "replicated" variables

/*****************************************************************************
*** THE REPLICATED DISTRIBUTION ***

This ReplicatedDist distribution causes a domain and its arrays
to be replicated across the desired locales (all the locales by default).
An array receives a distinct set of elements - a "replicand" -
allocated on each locale.

In other words, mapping a domain with ReplicatedDist gives it
an implicit additional dimension - over the locales,
making it behave as if there is one copy of its indices per locale.

Replication over locales is observable:
- when iterating over a domain or array
- when printing with write() et al.
- when zippering and the replicated domain/array is
  the first among the zippered items
- when assigning into the replicated array
- when inquiring about the domain's numIndices
  or the array's numElements
- when accessing array element(s) from a locale that was not included
  in the array passed explicitly to the ReplicatedDist constructor,
  an out-of-bounds error will result

Only the replicand *on the current locale* is accessed
(i.e. replication is not observable):
- when examining certain domain properties:
  dim(d), dims(), low, high, stride;
  but not numIndices
- when indexing into an array
- when slicing an array  TODO: right?
- when zippering and the first zippered item is not replicated
- when assigning to a non-replicated array,
  i.e. the replicated array is on the right-hand side of the assignment
- when there is only a single locale (trivially: only one replicand)

E.g. when iterating, the number of iterations will be (the number of
locales involved) times (the number of iterations over this domain if
it were distributed with the default distribution).

Note that the above behavior may change in the future.

Features/limitations:
* Consistency/coherence among replicands' array elements is NOT maintained.
* Only rectangular domains are presently supported.
* Serial iteration over a replicated domain (or array) visits the indices
  (or array elements) of all replicands *from the current locale*.
* When replicating over user-provided array of locales, that array
  must be "consistent" (see below).

"Consistent" array requirement:
* The array of locales passed to the ReplicatedDist constructor, if any,
  must be "consistent".
* A is "consistent" if for each ix in A.domain, A[ix].id == ix.
* Tip: if the domain of the desired array of locales cannot be described
  as a rectangular domain (which could be strided, multi-dimensional,
  and/or sparse), make that array's domain associative over int.

Examples:

const Dbase = [1..5];
const Drepl: domain(1) dmapped ReplicatedDist() = Dbase;
var Abase: [Dbase] int;
var Arepl: [Drepl] int;

// only the current locale's replicand is accessed
Arepl[3] = 4;

// these iterate over Dbase, so
// only the current locale's replicand is accessed
forall (b,r) in (Abase,Arepl) b = r;
Abase = Arepl;

// these iterate over Drepl;
// each replicand will be zippered against (and copied from) the entire Abase
forall (r,b) in (Arepl,Abase) r = b;
Arepl = Abase;

// sequential zippering will detect difference in sizes
// (if multiple locales)
for (b,r) in (Abase,Arepl) ... // error
for (r,b) in (Arepl,Abase) ... // error

Potential extensions:
- support other kinds of domains
- allow run-time change in locales
*/

/*****************************************************************************
*** SUPPORT FOR USER-LEVEL REPLICATED VARIABLES ***

A "replicated" variable is a variable for which there is a copy on each locale.
Referencing a replicated variable (in a stylized way, see below)
accesses its copy on the current locale.

Features:
* The variable's copies are not synchronized automatically among the locales.

Limitations:
* It is "user-level", i.e. the user is required to handle the variable
  in specific ways to achieve the desired result.
* Tree-shape communication (like for reductions) is not provided.
* Using a replicated variable of an array type is not straightforward.
  Workaround: declare that array itself as replicated, then access it normally,
  e.g.:
   var replArray: [YOUR DOMAIN dmapped ReplicatedDist()] YOUR ELEMENT TYPE;

How to use replicated variables:
(Note: names that start with 'rc' are provided by this module.)

    use ReplicatedDist;

    // declare a replicated variable of the type 'MyType'
    var myRepVar: [rcDomain] MyType;

    // access its copy on the current locale (read or write) (either option)
    ... myRepVar(1) ...
    ... rcLocal(myRepVar) ...

    // "replicate": assign 'valToRep' to copies on all locales
    rcReplicate(myRepVar, valToRep);

    // "collect": assign from each copy of 'myRepVar' to
    // corresponding element of an array 'collected'
    var collected: [LocaleSpace] MyType;
    rcCollect(myRepVar, collected);

    // access directly a remote copy on the locale 'remoteLoc' (read or write)
    ... rcRemote(myRepVar, remoteLoc) ...

Advanced: replicate over a subset of locales 'myLocales'.
// 'myLocales' must be "consistent", as defined for ReplicatedDist above.
Use it as above, except modify the variable declarations as follows:

    var myRepVar: [rcDomainBase dmapped ReplicatedDist(myLocales,
                     "over which to replicate 'myRepVar'")] MyType;
    var collected: [myLocales.domain] MyType;

See also rcExample() and rcExampleOverLocales() in this file.
*/

/*****************************************************************************/
// THE REPLICATED DISTRIBUTION IMPLEMENTATION

// include locale information when printing out domains and arrays
config param printReplicatedLocales = true;

// trace certain DSI methods as they are being invoked
config param traceReplicatedDist = false;

/////////////////////////////////////////////////////////////////////////////
// distribution

//
// (global) distribution class
//
class ReplicatedDist : BaseDist {
  // the desired locales (an array of locales)
  const targetLocales;
  // "IDs" are indices into targetLocales
  proc targetIds return targetLocales.domain;

  // privatized object id
  var pid: int = -1;
}

// constructor: replicate over the given locales
// (by default, over all locales)
proc ReplicatedDist.ReplicatedDist(targetLocales: [] locale = Locales,
                 purposeMessage: string = "used to create a ReplicatedDist")
{
  if traceReplicatedDist then
    writeln("ReplicatedDist constructor over ", targetLocales);
  _localesCheckHelper(purposeMessage);
}

// helper to check consistency of the locales array
// TODO: going over all the locales - is there a scalability issue?
proc ReplicatedDist._localesCheckHelper(purposeMessage: string): void {
  // ideally would like to make this a "eureka"
  forall (ix, loc) in (targetIds, targetLocales) do
    if loc.id != ix {
      halt("The array of locales ", purposeMessage, " must be \"consistent\".",
           " See ReplicatedDist documentation for details.");
    }
}

// privatization

proc ReplicatedDist.dsiSupportsPrivatization() param return true;

proc ReplicatedDist.dsiGetPrivatizeData() {
  if traceReplicatedDist then writeln("ReplicatedDist.dsiGetPrivatizeData");

  // TODO: return the targetLocales array by value,
  // to reduce communication needed in dsiPrivatize()
  // perhaps by wrapping it in a class (or tuple?).
  return targetLocales;
}

proc ReplicatedDist.dsiPrivatize(privatizeData: this.targetLocales.type)
  : this.type
{
  if traceReplicatedDist then writeln("ReplicatedDist.dsiPrivatize on ", here);

  const pdTargetLocales = privatizeData;
  // make private copy of targetLocales and its domain
  const privTargetIds: domain(pdTargetLocales.domain.rank,
                              pdTargetLocales.domain.idxType,
                              pdTargetLocales.domain.stridable
                              ) = pdTargetLocales.domain;
  const privTargetLocales: [privTargetIds] locale = pdTargetLocales;
  return new ReplicatedDist(privTargetLocales, "used during privatization");
}

/////////////////////////////////////////////////////////////////////////////
// domains

//
// global domain class
//
class ReplicatedDom : BaseRectangularDom {
  // to support rectangular domains
  param rank: int;
  type idxType;
  param stridable: bool;
  // we need to be able to provide the domain map for our domain - to build its
  // runtime type (because the domain map is part of the type - for any domain)
  // (looks like it must be called exactly 'dist')
  const dist; // must be a ReplicatedDist

  // this is our index set; we store it here so we can get to it easily
  var domRep: domain(rank, idxType, stridable);

  // local domain objects
  // NOTE: 'dist' must be initialized prior to 'localDoms'
  // => currently have to use the default constructor
  // NOTE: if they ever change after the constructor - Reprivatize them
  var localDoms: [dist.targetIds] LocReplicatedDom(rank, idxType, stridable);

  proc numReplicands return localDoms.numElements;

  // privatized object id
  var pid: int = -1;
}

//
// local domain class
//
class LocReplicatedDom {
  // copy from the global domain
  param rank: int;
  type idxType;
  param stridable: bool;

  // our index set, copied from the global domain
  var domLocalRep: domain(rank, idxType, stridable);
}

// No explicit ReplicatedDom constructor - use the default one.
// proc ReplicatedDom.ReplicatedDom(...){...}

// Since we piggy-back on (default-mapped) Chapel domains, we can redirect
// a few operations to those. This function returns a Chapel domain
// that's fastest to access from the current locale.
// With privatization this is in the privatized copy of the ReplicatedDom.
//
// Not a parentheses-less method because of a bug as of r18460
// (see generic-parenthesesless-3.chpl).
proc ReplicatedDom.redirectee(): domain(rank, idxType, stridable)
  return domRep;

// The same across all domain maps
proc ReplicatedDom.dsiMyDist() return dist;

// privatization

proc ReplicatedDom.dsiSupportsPrivatization() param return true;

proc ReplicatedDom.dsiGetPrivatizeData() {
  if traceReplicatedDist then writeln("ReplicatedDom.dsiGetPrivatizeData");

  // TODO: perhaps return 'domRep' and 'localDoms' by value,
  // to reduce communication needed in dsiPrivatize().
  return (dist.pid, domRep, localDoms);
}

proc ReplicatedDom.dsiPrivatize(privatizeData): this.type {
  if traceReplicatedDist then writeln("ReplicatedDom.dsiPrivatize on ", here);

  var privdist = chpl_getPrivatizedCopy(this.dist.type, privatizeData(1));
  return new ReplicatedDom(rank=rank, idxType=idxType, stridable=stridable,
                           dist = privdist,
                           domRep = privatizeData(2),
                           localDoms = privatizeData(3));
}

proc ReplicatedDom.dsiGetReprivatizeData() {
  return tuple(domRep);
}

proc ReplicatedDom.dsiReprivatize(other, reprivatizeData): void {
  assert(this.rank == other.rank &&
         this.idxType == other.idxType &&
         this.stridable == other.stridable);

  this.domRep = reprivatizeData(1);
}

proc ReplicatedDist.dsiClone(): this.type {
  if traceReplicatedDist then writeln("ReplicatedDist.dsiClone");
  return new ReplicatedDist(targetLocales);
}

// create a new domain mapped with this distribution
proc ReplicatedDist.dsiNewRectangularDom(param rank: int,
                                         type idxType,
                                         param stridable: bool)
  : ReplicatedDom(rank, idxType, stridable, this.type)
{
  if traceReplicatedDist then writeln("ReplicatedDist.dsiNewRectangularDom ",
                                      (rank, typeToString(idxType), stridable));

  // Have to call the default constructor because we need to initialize 'dist'
  // prior to initializing 'localDoms' (which needs a non-nil value for 'dist'.
  var result = new ReplicatedDom(rank=rank, idxType=idxType,
                                 stridable=stridable, dist=this);

  // create local domain objects
  coforall (loc, locDom) in (targetLocales, result.localDoms) do
    on loc do
      locDom = new LocReplicatedDom(rank, idxType, stridable);

  return result;
}

// create a new domain mapped with this distribution representing 'ranges'
proc ReplicatedDom.dsiBuildRectangularDom(param rank: int,
                                          type idxType,
                                          param stridable: bool,
                                          ranges: rank * range(idxType,
                                                BoundedRangeType.bounded,
                                                               stridable))
  : ReplicatedDom(rank, idxType, stridable, this.dist.type)
{
  // could be made more efficient to avoid visiting each locale twice
  // but perhaps not a big deal, for now anyways
  var result = dist.dsiNewRectangularDom(rank, idxType, stridable);
  result.dsiSetIndices(ranges);
  return result;
}

// Given an index, this should return the locale that owns that index.
// (This is the implementation of dmap.idxToLocale().)
// For ReplicatedDist, we point it to the current locale.
proc ReplicatedDist.dsiIndexToLocale(indexx): locale {
  return here;
}

/*
dsiSetIndices accepts ranges because it is invoked so from ChapelArray or so.
Most dsiSetIndices() on a tuple of ranges can be the same as this one.
Or that call dsiSetIndices(ranges) could be converted following this example.
*/
proc ReplicatedDom.dsiSetIndices(rangesArg: rank * range(idxType,
                                          BoundedRangeType.bounded,
                                                         stridable)): void {
  if traceReplicatedDist then
    writeln("ReplicatedDom.dsiSetIndices on ranges");
  dsiSetIndices([(...rangesArg)]);
}

proc ReplicatedDom.dsiSetIndices(domArg: domain(rank, idxType, stridable)): void {
  if traceReplicatedDist then
    writeln("ReplicatedDom.dsiSetIndices on domain ", domArg);
  domRep = domArg;
  coforall locDom in localDoms do
    on locDom do
      locDom.domLocalRep = domArg;
}

proc ReplicatedDom.dsiGetIndices(): rank * range(idxType,
                                                 BoundedRangeType.bounded,
                                                 stridable) {
  if traceReplicatedDist then writeln("ReplicatedDom.dsiGetIndices");
  return redirectee().getIndices();
}

// Iterators over the domain's indices (serial, leader, follower).
// Our semantics: yield each of the domain's indices once per each locale.

// Serial iterator: the compiler forces it to be completely serial
iter ReplicatedDom.these() {
  // compiler does not allow 'on' here (see r16137 and nestedForall*)
  // so instead of ...
  //---
  //for locDom in localDoms do
  //  on locDom do
  //    for i in locDom.domLocalRep do
  //      yield i;
  //---
  // ... so we simply do the same a few times
  var dom = redirectee();
  for count in 1..#numReplicands do
    for i in dom do
      yield i;
}

iter ReplicatedDom.these(param tag: iterator) where tag == iterator.leader {
  coforall locDom in localDoms do
    on locDom do
      // there, for simplicity, redirect to DefaultRectangular's leader
      for follow in locDom.domLocalRep._value.these(tag) do
        yield follow;
}

iter ReplicatedDom.these(param tag: iterator, follower) where tag == iterator.follower {
  // redirect to DefaultRectangular
  for i in redirectee()._value.these(tag, follower) do
    yield i;
}

/* Write the domain out to the given Writer serially. */
proc ReplicatedDom.dsiSerialWrite(f: Writer): void {
  // redirect to DefaultRectangular
  redirectee()._value.dsiSerialWrite(f);
  if printReplicatedLocales {
    f.write(" replicated over ");
    dist.targetLocales._value.dsiSerialWrite(f);
  }
}

proc ReplicatedDom.dsiDims(): rank * range(idxType,
                                           BoundedRangeType.bounded,
                                           stridable)
  return redirectee().dims();

proc ReplicatedDom.dsiDim(dim: int): range(idxType,
                                           BoundedRangeType.bounded,
                                           stridable)
  return redirectee().dim(dim);

proc ReplicatedDom.dsiLow
  return redirectee().low;

proc ReplicatedDom.dsiHigh
  return redirectee().high;

proc ReplicatedDom.dsiStride
  return redirectee().stride;

// here replication is visible
proc ReplicatedDom.dsiNumIndices
  return redirectee().numIndices * numReplicands;

proc ReplicatedDom.dsiMember(indexx)
  return redirectee().member(indexx);

proc ReplicatedDom.dsiIndexOrder(indexx)
  return redirectee().dsiIndexOrder(indexx);

/////////////////////////////////////////////////////////////////////////////
// arrays

//
// global array class
//
class ReplicatedArr : BaseArr {
  // These two are hard-coded in the compiler - it computes the array's
  // type string as '[dom.type] eltType.type'
  type eltType;
  const dom; // must be a ReplicatedDom

  // the replicated arrays
  // NOTE: 'dom' must be initialized prior to initializing 'localArrs'
  var localArrs: [dom.dist.targetIds]
              LocReplicatedArr(eltType, dom.rank, dom.idxType, dom.stridable);

  // privatized object id
  var pid: int = -1;
}

//
// local array class
//
class LocReplicatedArr {
  // these generic fields let us give types to the other fields easily
  type eltType;
  param rank: int;
  type idxType;
  param stridable: bool;

  var myDom: LocReplicatedDom(rank, idxType, stridable);
  var arrLocalRep: [myDom.domLocalRep] eltType;
}

// ReplicatedArr constructor.
// We create our own to make field initializations convenient:
// 'eltType' and 'dom' as passed explicitly;
// the fields in the parent class, BaseArr, are initialized to their defaults.
//
proc ReplicatedArr.ReplicatedArr(type eltType, dom: ReplicatedDom) {
  // initializes the fields 'eltType', 'dom' by name
}

// The same across all domain maps
proc ReplicatedArr.dsiGetBaseDom() return dom;

// privatization

proc ReplicatedArr.dsiSupportsPrivatization() param return true;

proc ReplicatedArr.dsiGetPrivatizeData() {
  if traceReplicatedDist then writeln("ReplicatedArr.dsiGetPrivatizeData");

  // TODO: perhaps return 'localArrs' by value,
  // to reduce communication needed in dsiPrivatize().
  return (dom.pid, localArrs);
}

proc ReplicatedArr.dsiPrivatize(privatizeData) {
  if traceReplicatedDist then writeln("ReplicatedArr.dsiPrivatize on ", here);

  var privdom = chpl_getPrivatizedCopy(this.dom.type, privatizeData(1));
  var result = new ReplicatedArr(eltType, privdom);
  result.localArrs = privatizeData(2);
  return result;
}

// create a new array over this domain
proc ReplicatedDom.dsiBuildArray(type eltType)
  : ReplicatedArr(eltType, this.type)
{
  if traceReplicatedDist then writeln("ReplicatedDom.dsiBuildArray");
  var result = new ReplicatedArr(eltType, this);
  coforall (loc, locDom, locArr)
   in (dist.targetLocales, localDoms, result.localArrs) do
    on loc do
      locArr = new LocReplicatedArr(eltType, rank, idxType, stridable,
                                    locDom);
  return result;
}

// Return the array element corresponding to the index - on the current locale
proc ReplicatedArr.dsiAccess(indexx) var: eltType {
  return localArrs[here.id].arrLocalRep[indexx];
}

// Write the array out to the given Writer serially.
proc ReplicatedArr.dsiSerialWrite(f: Writer): void {
  var neednl = false;
  for locArr in localArrs {
//  on locArr {  // may cause deadlock
      if neednl then f.write("\n"); neednl = true;
      if printReplicatedLocales then
        f.write(locArr.locale, ":\n");
      locArr.arrLocalRep._value.dsiSerialWrite(f);
//  }
  }
}

// iterators

// completely serial
iter ReplicatedArr.these() var: eltType {
  for locArr in localArrs do
//  on locArr do // compiler does not allow; see r16137 and nestedForall*
      for a in locArr.arrLocalRep do
        yield a;
}

iter ReplicatedArr.these(param tag: iterator) where tag == iterator.leader {
  // redirect to ReplicatedDom's leader
  for follow in dom.these(tag) do
    yield follow;
}

iter ReplicatedArr.these(param tag: iterator, follower) var where tag == iterator.follower {
  // redirect to DefaultRectangular
  for a in localArrs[here.id].arrLocalRep._value.these(tag, follower) do
    yield a;
}

/////////////////////////////////////////////////////////////////////////////
// slicing, reindexing, etc.

// This supports reassignment of the array's domain.
/*
This gets invoked upon reassignment of the array's domain,
prior to calling this.dom.dsiSetIndices().
So this needs to adjust anything in the array that won't be taken care of
in this.dom.dsiSetIndices(). In our case, that's nothing.
*/
proc ReplicatedArr.dsiReallocate(d: domain): void {
  if traceReplicatedDist then
    writeln("ReplicatedArr.dsiReallocate ", dom.domRep, " -> ", d, " (no-op)");
}

// array slicing
proc ReplicatedArr.dsiSlice(sliceDef: ReplicatedDom) {
  if traceReplicatedDist then writeln("ReplicatedArr.dsiSlice on ", sliceDef);
  var result = new ReplicatedArr(eltType, sliceDef);
  var slicee = this;

  // ensure 'dom' and 'slicee' are over the same set of locales/targetIds
  assert(sliceDef.localDoms.domain == slicee.localArrs.domain);

  coforall (loc, sliceDefLocDom, sliceeLocArr, resultLocArr)
   in (sliceDef.dist.targetLocales, sliceDef.localDoms,
       slicee.localArrs, result.localArrs) do
    on loc do
      resultLocArr = new LocReplicatedArr(eltType,
        sliceDef.rank, sliceDef.idxType, sliceDef.stridable,
        myDom = sliceDefLocDom,
        arrLocalRep => sliceeLocArr.arrLocalRep[sliceDefLocDom.domLocalRep]);

  return result;
}

// array reindexing
// very similar to array slicing
proc ReplicatedArr.dsiReindex(sliceDef: ReplicatedDom) {
  if traceReplicatedDist then writeln("ReplicatedArr.dsiReindex on ", sliceDef);
  var result = new ReplicatedArr(eltType, sliceDef);
  var slicee = this;

  // ensure 'dom' and 'slicee' are over the same set of locales/targetIds
  assert(sliceDef.localDoms.domain == slicee.localArrs.domain);

  coforall (loc, sliceDefLocDom, sliceeLocArr, resultLocArr)
   in (sliceDef.dist.targetLocales, sliceDef.localDoms,
       slicee.localArrs, result.localArrs) do
    on loc do
     {
      var locAlias: [sliceDefLocDom.domLocalRep] => sliceeLocArr.arrLocalRep;
      resultLocArr = new LocReplicatedArr(eltType,
        sliceDef.rank, sliceDef.idxType, sliceDef.stridable,
        myDom = sliceDefLocDom,
        arrLocalRep => locAlias);
     }

  return result;
}

// rank-change slicing
// very similar to slicing
proc ReplicatedArr.dsiRankChange(sliceDef: ReplicatedDom,
                                 param newRank: int,
                                 param newStridable: bool,
                                 args) {
  if traceReplicatedDist then writeln("ReplicatedArr.dsiRankChange");
  var result = new ReplicatedArr(eltType, sliceDef);
  var slicee = this;

  // ensure 'dom' and 'slicee' are over the same set of locales/targetIds
  assert(sliceDef.localDoms.domain == slicee.localArrs.domain);

  coforall (loc, sliceDefLocDom, sliceeLocArr, resultLocArr)
   in (sliceDef.dist.targetLocales, sliceDef.localDoms,
       slicee.localArrs, result.localArrs) do
    on loc do
      resultLocArr = new LocReplicatedArr(eltType,
        sliceDef.rank, sliceDef.idxType, sliceDef.stridable,
        myDom = sliceDefLocDom,
        arrLocalRep => sliceeLocArr.arrLocalRep[(...args)]);

  return result;
}

// todo? these two seem to work (written by analogy with DefaultRectangular)
proc ReplicatedDist.dsiCreateReindexDist(newSpace, oldSpace) return this;
proc ReplicatedDist.dsiCreateRankChangeDist(param newRank, args) return this;

/*****************************************************************************/
// SUPPORT FOR USER-LEVEL REPLICATED VARIABLES

const rcDomainIx   = 1;
const rcDomainBase = [rcDomainIx..rcDomainIx];
const rcLocales    = Locales;
const rcDomainMap  = new ReplicatedDist(rcLocales);
const rcDomain     = rcDomainBase dmapped new dmap(rcDomainMap);
const rcCollectDomaim = rcLocales.domain;
param _rcErr1 = " must be 'rcDomain' or 'rcDomainBase dmapped ReplicatedDist(an array of locales)'";

proc _rcTargetLocalesHelper(replicatedVar: [?D])
  where replicatedVar._value.type: ReplicatedArr
{
  return replicatedVar._value.dom.dist.targetLocales;
}

proc rcReplicate(replicatedVar: [?D] ?MYTYPE, valToReplicate: MYTYPE): void
{ compilerError("the domain of first argument to rcReplicate()", _rcErr1); }

proc rcReplicate(replicatedVar: [?D] ?MYTYPE, valToReplicate: MYTYPE): void
  where replicatedVar._value.type: ReplicatedArr
{
  assert(replicatedVar.domain == rcDomainBase);
  coforall loc in _rcTargetLocalesHelper(replicatedVar) do
    on loc do
      replicatedVar[rcDomainIx] = valToReplicate;
}

proc rcCollect(replicatedVar: [?D] ?MYTYPE, collected: [?CD] MYTYPE): void
  where ! replicatedVar._value.type: ReplicatedArr
{ compilerError("the domain of first argument to rcCollect()", _rcErr1); }

proc rcCollect(replicatedVar: [?D] ?MYTYPE, collected: [?CD] MYTYPE): void
  where replicatedVar._value.type: ReplicatedArr
{
  var targetLocales = _rcTargetLocalesHelper(replicatedVar);
  assert(replicatedVar.domain == rcDomainBase);
  assert(collected.domain == targetLocales.domain);
  coforall (loc, col) in (targetLocales, collected) do
    on loc do
      col = replicatedVar[rcDomainIx];
}

proc rcLocal(replicatedVar: [?D] ?MYTYPE) var: MYTYPE {
  return replicatedVar[rcDomainIx];
}

proc rcRemote(replicatedVar: [?D] ?MYTYPE, remoteLoc: locale) var: MYTYPE {
  var result: MYTYPE;
  on remoteLoc do
    result = replicatedVar[rcDomainIx];
  return result;
}

/////////////////////////////////////////////////////////////////////////////
// code examples

// This example illustrates the use of a replicated variable as follows:
// - create a replicated variable of the type MyType,
// - initializes all its copies to 'initVal',
// - stores 'newVal' into its copy on the locale 'newLocale',
// - collects all its copies into an array 'collected'.
//
proc rcExample(initVal: ?MyType, newVal: MyType, newLocale: locale): void {
  writeln("starting rcExample");

  // declare a replicated variable
  var myRepVar: [rcDomain] MyType;

  // initialize all copies to 'initVal'
  rcReplicate(myRepVar, initVal);

  writeln("\nafter initialization, myRepVar copies are:\n", myRepVar);

  // go to 'newLocale' and update its copy to 'newVal'
  on newLocale {
    // myRepVar[1] references myRepVar's copy on the current locale
    writeln("\non ", newLocale, ", myRepVar is:\n",
            "  before assignment: ", myRepVar[1]);

    // update the current locale's copy
    myRepVar[1] = newVal;

    writeln("  after assignment:  ", myRepVar[1]);
  }

  writeln("\nafter update, myRepVar copies are:\n", myRepVar);

  // collect all copies of 'myRepVar' into an array
  var collected: [LocaleSpace] MyType;
  rcCollect(myRepVar, collected);

  writeln("\ncollected copies of myRepVar are:\n", collected);
}

// This is the same as 'rcExample', except the user can provide
// specific locales to replicate over. The two differences are marked.
//
proc rcExampleOverLocales(initVal: ?MyType, newVal: MyType, newLocale: locale,
                          localesToReplicateOver: [] locale = Locales): void {
  writeln("starting rcExampleOverLocales over ", localesToReplicateOver);

  // declare a replicated variable
  // DIFFERENT: the domain in myRepVar's type
  var myRepVar: [rcDomainBase dmapped ReplicatedDist(localesToReplicateOver,
   "over which to replicate 'myRepVar' in rcExampleOverLocales()")] MyType;

  // initialize all copies to 'initVal'
  rcReplicate(myRepVar, initVal);

  writeln("\nafter initialization, myRepVar copies are:\n", myRepVar);

  // go to 'newLocale' and update its copy to 'newVal'
  on newLocale {
    // myRepVar[1] references myRepVar's copy on the current locale
    writeln("\non ", newLocale, ", myRepVar is:\n",
            "  before assignment: ", myRepVar[1]);

    // update the current locale's copy
    myRepVar[1] = newVal;

    writeln("  after assignment:  ", myRepVar[1]);
  }

  writeln("\nafter update, myRepVar copies are:\n", myRepVar);

  // collect all copies of 'myRepVar' into an array
  // DIFFERENT: the domain in collected's type
  var collected: [localesToReplicateOver.domain] MyType;
  rcCollect(myRepVar, collected);

  writeln("\ncollected copies of myRepVar are:\n", collected);
}


Remove tabs

Remove tabs

/*
 * Copyright 2015 Cray Inc.
 * Other additional copyright holders may be indicated within.
 *
 * The entirety of this work is licensed under the Apache License,
 * Version 2.0 (the "License"); you may not use this file except
 * in compliance with the License.
 *
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

/*

   Support for the visualization tool "chplvis"

   This module provides access to and enables hooks to dump out
   task and communication information for post-run visualization
   of the tasks and communication.

*/
module VisualDebug
{

  use String;

  pragma "no doc"
  extern proc chpl_now_time():real;

  //
  // Data Generation for the Visual Debug tool  (offline)
  //

  pragma "no doc"
  extern proc chpl_vdebug_start ( rootname: c_string, time:real);

  pragma "no doc"
  extern proc chpl_vdebug_stop ();

  pragma "no doc"
  extern proc chpl_vdebug_tag ( tagname: c_string);

  pragma "no doc"
  extern proc chpl_vdebug_pause ();

  pragma "no doc"
  extern proc chpl_vdebug_nolog ();

/* Tree "coforall procedure .... calls one of the above rotunes */

pragma "no doc"
  enum vis_op {v_start, v_stop, v_tag, v_pause};

pragma "no doc"
  proc VDebugTree (what: vis_op, name: string, time: real, id: int = 0) {
      var child = id * 2 + 1;
      chpl_vdebug_nolog();
      cobegin {
         /* left */
         if child < numLocales then
             on Locales[child] do VDebugTree (what, name, time, child);
         /* right */
         if child+1 < numLocales then
             on Locales[child+1] do VDebugTree (what, name, time, child+1);
     }
         /* Do the op at the root  */
         select what {
            when vis_op.v_start    do chpl_vdebug_start (name.c_str(), time);
            when vis_op.v_stop     do chpl_vdebug_stop ();
            when vis_op.v_tag      do chpl_vdebug_tag (name.c_str());
            when vis_op.v_pause    do chpl_vdebug_pause ();
         }
  }

/*
  Start logging events for VisualDebug.  Open a new set of data
  files, one for each locale, for chplvis.  This routine should be
  called only once for each program.  It creates a directory with the
  rootname and creates the files in that directory.  The files are
  named with the rootname and "-n" is added where n is the locale
  number.

  :arg rootname:  Directory name and rootname for files.
*/
  proc startVdebug ( rootname : string ) {
    var now = chpl_now_time();
    //coforall l in Locales do
    //  on l do chpl_vdebug_start (rootname.c_str(), now);
    VDebugTree (vis_op.v_start, rootname, now);
  }

/*
  Add a tag to the data for chplvis to allow "view points" in the data.

  :arg tagname: name of the tag
*/
  proc tagVdebug ( tagname : string ) {
    //coforall l in Locales[1..] do
    //  on l do chpl_vdebug_tag (tagname.c_str(), 0);
    //chpl_vdebug_tag (tagname.c_str(), 0);
    VDebugTree (vis_op.v_tag, tagname, 0);
  }

/*
  Stop collecting VisualDebug data and close the data files.
*/
  proc stopVdebug () {
    // chpl_vdebug_stop();
    // coforall l in Locales[1..] do
    //  on l do chpl_vdebug_stop();
    VDebugTree (vis_op.v_stop, "", 0);
  }

/*
  Suspend collection of VisualDebug data.
*/
  proc pauseVdebug () {
    //coforall l in Locales[1..] do
    //  on l do chpl_vdebug_tag (tagname.c_str(), 1);
    //chpl_vdebug_tag(tagname.c_str(), 1);
    VDebugTree (vis_op.v_pause, "", 0);
  }

/*
  Resume collection of VisualDebug data for chplvis after a pauseVdebug().
  This also generates a tag record visible by chplvis.

  :arg tagname: name of the tag
*/
  proc resumeVdebug ( tagname : string ) {
    //coforall l in Locales[1..] do
    //  on l do chpl_vdebug_resume ();
    //chpl_vdebug_resume(tagname);
    VDebugTree (vis_op.v_tag, tagname, 0);
  }

}


Added more default methods to Collections

/*
  A 'Collection' is an interface for a data structure that is a 'contract' between
  the user and implementer that it fulfills the following:

  1. Is safe parallel-safe, hence is safe to use across multiple tasks across multiple locales.
  2. Supports the basic operations that any data structure needs to be truly useful, that is:
    a. Insertion of an arbitrary element
    b. Removal of an arbitrary element
    c. Lookup of an specific element
*/

/*
  A 'Collection' is a data structure, a container for elements that provide support
  for insert, lookup, remove, and iteration operations.
*/
module Collection-Interface {
  class Collection {
    /*
      The type of element that this Collection holds.
    */
    type eltType;

    /*
      Adds an element to this data structure.
    */
    proc add(elt : eltType) : bool {
      halt("'proc add(elt : eltType) : bool' is not supported...");
    }

    /*
      Add all elements in bulk to this data structure. If the data structure
      rejects an element, we cease to offer more. We return the number of elements
      successfully added to this data structure.
    */
    proc addBulk(elts) : int {
      var successful : int;
      for elt in elts {
        if !add(elt) {
          break;
        }

        successful += 1;
      }
    }

    /*
      Removes an arbitrary element from this data structure.

      **BUG:** Compiler will segfault if the returned value is not captured at callsite.
      Issue: #6542

      **FIX:** Ensure that you always capture the return value...

      ::

        var capturedRetval = c.remove()

      **BUG:** Loop Invariant Code Motion causes undefined behavior if assigned to a
      variable declared outside of loop. Issue: #7003

      **FIX:** Use the `--no-loop-invariant-code-motion` to disable LICM.
      Otherwise, just make sure you always capture the return value inside of a loop
      in a variable not declared outside of loop...

      ::

        for i in 1 .. N {
          var retval = c.remove();
        }

    */
    proc remove() : (bool, eltType) {
      halt("'proc remove() : (bool, eltType)' is not supported...");
    }

    /*
      Removes `nElts` elements from this data structure, returning them as an array.
      If the data structure fails to produce a new element, we cease and shrink the array
      to its appropriate size and return that.
    */
    proc removeBulk(nElts : int) {
      var dom = {0..#nElts};
      var arr : [dom] eltType;
      var idx = 0;

      for 1 .. nElts {
        var (hasElt, elt) = remove();
        if !hasElt {
          dom = {0..#idx};
          break;
        }

        dom[idx] = elt;
        idx += 1;
      }

      return arr;
    }

    /*
      Determine whether an element exists in this collection.
    */
    proc contains(elt : eltType) : bool {
      var found = false;

      // Iterate to find it... note that we cannot break early due to
      // issues with leaking during serial iteration.
      for e in these() {
        if e == elt {
          found = true;
        }
      }

      return found;
    }

    /*
      Clears all elements in this collection.
    */
    proc clear() {
      while true {
        var (hasElt, elt) = remove();
        if !hasElt {
          break;
        }
      }
    }

    /*
      Check if this data structure is empty.
    */
    proc isEmpty() : bool {
      return getSize() == 0;
    }

    /*
      Syntactic sugar for `getSize`.
    */
    proc length : int {
      return getSize();
    }

    /*
      Syntactic sugar for `getSize`.
    */
    proc size : int {
      return getSize();
    }

    /*
      Obtain the number of elements contained in this collection.
    */
    proc getSize() : int {
      halt("'proc size() : int' is not supported...");
    }

    /*
      Iterate over all elements in the data structure.

      **BUG:** Compiler does not currently allow overriding standalone or leader/follower
      iterators, and as such only serial iterators may be used with the base type. See
      issue #6998

      **BUG:** Resources are not properly cleaned up when the user breaks or returns
      from a serial iterator, and so this *must* be avoided at all cost. See issue #6912
    */
    iter these() : eltType {
      halt("'iter these() : eltType' is not supported...");
      yield _defaultOf(eltType);
    }
  }

  /*
    Syntactic sugar for 'add'.

    **BUG:** Compiler produces a warning that `c` should be a `ref`, when it is not needed
    because it is a class and can be used by value. Need a pragma to disable this?
    As such, this cannot be used where `ref` intents cannot be used, such as ``forall``
    and ``coforall`` loops.
  */
  inline proc +=(const ref c : Collection(?eltType), elt : eltType) {
    c.add(elt);
  }
}

/*
  A 'Collection' is a data structure, a container for elements that provide support
  for insert, lookup, remove, and iteration operations.
*/
class Collection {
  /*
    The type of element that this Collection holds.
  */
  type eltType;

  /*
    Adds an element to this data structure.
  */
  proc add(elt : eltType) : bool {
    halt("'proc add(elt : eltType) : bool' is not supported...");
  }

  /*
    Removes an arbitrary element from this data structure.

    **BUG:** Compiler will segfault if the returned value is not captured at callsite.
    Issue: #6542

    **FIX:** Ensure that you always capture the return value...

    ::

      var capturedRetval = c.remove()

    **BUG:** Loop Invariant Code Motion causes undefined behavior if assigned to a
    variable declared outside of loop. Issue: #7003

    **FIX:** Use the `--no-loop-invariant-code-motion` to disable LICM.
    Otherwise, just make sure you always capture the return value inside of a loop
    in a variable not declared outside of loop...

    ::

      for i in 1 .. N {
        var retval = c.remove();
      }

  */
  proc remove() : (bool, eltType) {
    halt("'proc remove() : (bool, eltType)' is not supported...");
  }

  /*
    Determine whether an element exists in this collection.
  */
  proc contains(elt : eltType) : bool {
    halt("'proc contains(elt : eltType) : bool' is not supported...");
  }

  /*
    Clears all elements in this collection.
  */
  proc clear() {
    while !remove()[1] do ;
  }

  /*
    Check if this data structure is empty.
  */
  proc isEmpty() : bool {
    return getSize() == 0;
  }

  /*
    Syntactic sugar for `getSize`.
  */
  proc length : int {
    return getSize();
  }

  /*
    Syntactic sugar for `getSize`.
  */
  proc size : int {
    return getSize();
  }

  /*
    Obtain the number of elements contained in this collection.
  */
  proc getSize() : int {
    halt("'proc size() : int' is not supported...");
  }

  /*
    Iterate over all elements in the data structure.

    **BUG:** Compiler does not currently allow overriding standalone or leader/follower
    iterators, and as such only serial iterators may be used with the base type. See
    issue #6998

    **BUG:** Resources are not properly cleaned up when the user breaks or returns
    from a serial iterator, and so this *must* be avoided at all cost. See issue #6912
  */
  iter these() : eltType {
    halt("'iter these() : eltType' is not supported...");
    yield _defaultOf(eltType);
  }
}

/*
  Syntactic sugar for 'add'.

  **BUG:** Compiler produces a warning that `c` should be a `ref`, when it is not needed
  because it is a class and can be used by value. Need a pragma to disable this?
  As such, this cannot be used where `ref` intents cannot be used, such as ``forall``
  and ``coforall`` loops.
*/
inline proc +=(ref c : Collection(?eltType), elt : eltType) {
  c.add(elt);
}