© 2013, NVIDIA CORPORATION. All rights reserved.
Code and text by Sean Baxter, NVIDIA Research.
(Click here for license. Click here for contact information.)
Parallel computing is important because it enables much higher throughput than conventional systems. The GPU is revolutionary because it does this affordably.
Massive parallelism is the future of computing, but it comes with some challenges. Without good tools, targeting these systems complicates development and slows productivity—we'd need to see serious performance gains to justify the added effort. Fortunately, the CUDA software ecosystem has a number of quality libraries which ease development, improve productivity, and help users see super-charged performance. These packages solve many common problems, are reliable, and abstract away the complexities of the device. With them, realizing the high-performance of GPU computing is made easy for software professionals and made possible for the non-ninja domain expert. If you aren't already familiar with them, check out these pre-compiled libraries that ship with the CUDA Toolkit:
CUBLAS - The NVIDIA CUDA Basic Linear Algebra Subroutines library is a GPU-accelerated version of the complete standard BLAS library that delivers 6x to 17x faster performance than the latest MKL BLAS.
CUSPARSE - The NVIDIA CUDA Sparse Matrix library provides a collection of basic linear algebra subroutines used for sparse matrices that delivers up to 8x faster performance than the latest MKL.
CURAND - The NVIDIA CUDA Random Number Generation library delivers high performance GPU-accelerated random number generation (RNG). The cuRAND library delivers high quality random numbers 8x faster using hundreds of processor cores available in NVIDIA GPUs.
CUFFT - The NVIDIA CUDA Fast Fourier Transform library provides a simple interface for computing FFTs up to 10x faster. By using hundreds of processor cores inside NVIDIA GPUs, cuFFT delivers the floating‐point performance of a GPU without having to develop your own custom GPU FFT implementation.
NPP - The NVIDIA Performance Primitives library is a collection of GPU-accelerated image, video, and signal processing functions that deliver 5x to 10x faster performance than comparable CPU-only implementations. Using NPP, developers can take advantage of over 1900 image processing and approx 600 signal processing primitives to achieve significant improvements in application performance in a matter of hours.
Pull the latest revisions of these versatile template libraries developed by my colleagues at NVIDIA Research:
Thrust - Thrust is a parallel algorithms library which resembles the C++ Standard Template Library (STL). Thrust's high-level interface greatly enhances programmer productivity while enabling performance portability between GPUs and multicore CPUs.
CUB - CUB is a library of high-performance parallel primitives and other utilities for constructing CUDA kernel software. CUB enhances productivity, performance, and portability by providing an abstraction layer over complex block-level, warp-level, and thread-level operations.
Modern GPU was developed to help ambitious CUDA programmers push out the frontier. Every user's problem is unique; real applications require functionality that existing libraries don't provide. Programmers will have to solve original problems by getting their hands dirty and writing device code. This project covers algorithms, idioms, and strategies to help users craft the best kernels they can.
Modern GPU is code and commentary intended to promote new and productive ways of thinking about GPU computing. The library, moderngpu, was designed with four goals:
The great selling point of GPUs is their extraordinary floating-point throughput, and this has been well exploited with compute libraries like CUBLAS and visualization APIs like OpenGL. Users have been successful in writing kernels that chew through dense, regular, numerical problems. These are often domain-specific operations like particle-particle interactions for physics and chemistry or image filters for computer vision.
Difficulty occurs between the calls to these float-heavy routines. How do you gather and utilize your results? How do you prepare inputs for the next launch? I want to push the GPU further into this territory. The GPU has high memory bandwidth and an amazing latency-hiding architecture that is well suited for fine-grained manipulation of data. MGPU focuses on the most generic of problems: manipulation of arrays and fundamental CS algorithms. We look at sorting and searching, the bricks and mortar of programming.
The biggest challenge impeding wide-spread adoption of this technology is the difficulty in finding and exposing parallelism in irregular problems. I introduce strategies for dealing with this, and demonstrate how to put these ideas into execution by building a high-performance, comprehensive array-processing library.
Most of the ideas in Modern GPU are new and may seem unusual even to CUDA veterans. Although the code is very high performance, it isn't overtly concerned with the traditional nuts and bolts of GPU programming. MGPU de-emphasizes low-level programming and focuses on making CUDA more expressive. It emphasizes geometric reasoning and develops an idiom for solving problems by breaking solutions into two distinct phases:
Find a coarse-grained partitioning of the problem that exactly load-balances work over each thread. Scheduling—the exercise of mapping work items to CTAs (cooperative thread arrays) and threads on the GPU—is handled in this phase.
Execute simple, work-efficient, sequential logic that solves the problem. Because scheduling is part of the partitioning phase, this sequential phase runs embarrassingly parallel.
Partitioning involves a search over one or more input sequences. All the characteristic trickiness of parallel computing is isolated to this phase. Specific problem-solving code is run independently by each thread, and resembles code you'd have written for a CPU. By decoupling partitioning from the work logic, we improve modularity and make both phases easier to reason about.
This two-phase idiom comprises a new style for programming GPUs. Programmed with this idiom in mind, a number of functions are elegantly expressed:
Segmented sort is a work-efficient mergesort on multiple variable-length arrays.
Interval move schedules multiple variable-length coarse-grained memcpys.
Load-balancing search coordinates work-items with the objects that scheduled them, allowing perfect load-balancing for functions that expand and contract data.
Relational joins compute the outer products of variable-length duplicate ranges from two sorted inputs. Although this functionality is very difficult to express using conventional CUDA strategies, it comes out effortlessly with these new primitives.
Performance tuning is simple and well-understood in this framework. Each thread processes VT (Values per Thread) items. Increasing VT assigns more work to each thread, amortizing the cost of partitioning, which is constant per thread. Although this increases work-efficiency, it decreases parallelism and the GPU's ability to hide latency. As described below, programmers who use this framework may benchmark to find an optimal VT parameter for a particular device architecture, input type, size, and distribution, without having to modify any device code.
The central content of MGPU is the source code. Accordingly I've made clarity and conciseness a point of pride. The code is organized in a very flat directory structure, in a single namespace, with no dependencies other than the CUDA Toolkit. Routine operations that offer little of interest to the reader have been factored out, resulting in code that is dense with algorithms. The user shouldn't have to chase a function's flow of execution through more than a couple of files to see its entire implementation.
For flexibility the functions are heavily templated, each parameterized over input and output types, to support Boost-style iterators, comparators, and predicate objects. Still, this library is not an exercise in meta-programming, and the use of generics is limited.
A call to each host-level function is demonstrated in a simple form at the top of its corresponding web page (drawn from tests/demo.cu), and in a more complex form in its benchmark code, which generates the performance charts. The articles are organized so that functions are rolled out with increasing complexity and build on one another. Readers with CUDA experience who take the pages in order should be should not be unprepared for anything presented.
A quality of software that's been undervalued in recent years is hackability. Under schedule pressure, programmers grab and modify whatever works. The world runs on code spliced from Numerical Recipes and Sedgewick, O'Reilly books, Wikipedia samples, conference talks, powerpoints, and message boards.
Much emphasis has been put on the idea of composability in CUDA software. For example, Thrust combines simple functions with powerful iterators to increase functionality. While MGPU supports these same iterators and is largely interoperable with Thrust, my philosophy is that composability is just one aspect of software reuse. This library is designed to be forked and modified. If a function does nearly what you desire, but not exactly, creating a derivative function will not be a burden.
A major challenge in parallel programming—and especially in the massively fine-grained parallelism that GPUs provide—is deciding which work to run on which processor, and when to run it. This is a problem of scheduling or decomposition. Attempts to implement a function and manage decomposition with a single strategy can be unwieldy or inefficient, because the user is trying to solve two problems at once.
CPU Merge implementation
template<typename T, typename Comp> void CPUMerge(const T* a, int aCount, const T* b, int bCount, T* dest, Comp comp) { int count = aCount + bCount; int ai = 0, bi = 0; for(int i = 0; i < count; ++i) { bool p; if(bi >= bCount) p = true; else if(ai >= aCount) p = false; else p = !comp(b[bi], a[ai]); dest[i] = p ? a[ai++] : b[bi++]; } }
Consider this sequential merge implementation. It takes two sorted inputs and loops over each output. During each iteration, two inputs are compared and the smaller one is emitted. The implementation is simple and work-efficient. Because it's totally sequential, there's no real consideration given to scheduling or decomposition.
template<int NT, typename InputIt1, typename InputIt2, typename OutputIt, typename Comp> __global__ void ParallelMergeA(InputIt1 a_global, int aCount, InputIt2 b_global, int bCount, OutputIt dest_global, Comp comp) { typedef typename std::iterator_traits<InputIt1>::value_type T; int gid = threadIdx.x + NT * blockIdx.x; if(gid < aCount) { T aKey = a_global[gid]; int lb = BinarySearch<MgpuBoundsLower>(b_global, bCount, aKey, comp); dest_global[gid + lb] = aKey; } } template<int NT, typename InputIt1, typename InputIt2, typename OutputIt, typename Comp> __global__ void ParallelMergeB(InputIt1 a_global, int aCount, InputIt2 b_global, int bCount, OutputIt dest_global, Comp comp) { typedef typename std::iterator_traits<InputIt2>::value_type T; int gid = threadIdx.x + NT * blockIdx.x; if(gid < bCount) { T bKey = b_global[gid]; int ub = BinarySearch<MgpuBoundsUpper>(a_global, aCount, bKey, comp); dest_global[gid + ub] = bKey; } }
Now consider this first attempt at parallel merge. There are two kernels: ParallelMergeA
, which assigns one thread to each element in A, binary searches for the lower-bound in B, and outputs A keys to the destination; and ParallelMergeB
, which assigns one thread to each element in B, binary searches for the upper-bound in A, and outputs B keys to the destination.
Although this implementation is highly concurrent, it's also highly inefficient. The O(n)-efficiency sequential code is now O(n log n), as each output requires a binary search over the input. The code only runs quickly if the number of processors is large compared to the input size. Additionally, the new code looks nothing like the sequential version. Because we've baked scheduling logic into the problem-solving logic, we've written code that is difficult to optimize. Perhaps even worse, it's difficult to extend—for example, how would we support multiset operations with this PRAM-style scheduling?
template<MgpuBounds Bounds, typename It1, typename It2, typename Comp> MGPU_HOST_DEVICE int MergePath(It1 a, int aCount, It2 b, int bCount, int diag, Comp comp) { typedef typename std::iterator_traits<It1>::value_type T; int begin = max(0, diag - bCount); int end = min(diag, aCount); while(begin < end) { int mid = (begin + end)>> 1; T aKey = a[mid]; T bKey = b[diag - 1 - mid]; bool pred = (MgpuBoundsUpper == Bounds) ? comp(aKey, bKey) : !comp(bKey, aKey); if(pred) begin = mid + 1; else end = mid; } return begin; }
MGPU focuses on a two-phase strategy for execution. In the first phase we address partitioning: this phase handles scheduling and decomposition. The goal is to map work onto each thread in a load-balanced and work-efficient manner. All the functions in this library are parameterized over a grain size (the parameter VT), which controls the amount of work scheduled per thread. Increasing the grain size amortizes partitioning costs, improving work-efficiency, while simultaneously reducing occupancy and potentially sacrificing execution efficiency. Because partitioning is not tied up with the problem-solving logic, we can reuse this code in many functions, and optimize it by searching the grain-size parameter space.
Examples of partitioning functions used in MGPU are Merge Path, Balanced Path, and load-balancing search. The MergePath
implementation above decomposes merge-like problems, which includes mergesort, vectorized sorted search, and the load-balancing search scheduling function. This function binary searches both input arrays simultaneously, producing a decomposition that maps a uniform amount of work to each thread.
template<int VT, bool RangeCheck, typename T, typename Comp> MGPU_DEVICE void SerialMerge(const T* keys_shared, int aBegin, int aEnd, int bBegin, int bEnd, T* results, int* indices, Comp comp) { T aKey = keys_shared[aBegin]; T bKey = keys_shared[bBegin]; #pragma unroll for(int i = 0; i < VT; ++i) { bool p; if(RangeCheck) p = (bBegin >= bEnd) || ((aBegin < aEnd) && !comp(bKey, aKey)); else p = !comp(bKey, aKey); results[i] = p ? aKey : bKey; indices[i] = p ? aBegin : bBegin; if(p) aKey = keys_shared[++aBegin]; else bKey = keys_shared[++bBegin]; } __syncthreads(); }
The second phase in the strategy is the actual problem-solving logic. For efficiency and clarity we prefer that this code resembles sequential functions. The CUDA device function SerialMerge
resembles the sequential CPU functon CPUMerge
. Because concurrency was addressed in the partitioning phase, this phase can implement GPU-specific optimizations without complicating the decomposition. In addition to its improved work efficiency (we process VT inputs for each binary search), SerialMerge
realizes extra throughput by unrolling the serial merge loop and storing intermediates to register to conserve shared memory and improve occupancy.
Kernels written using the two-phase decomposition are more involved than direct solutions like the naive parallel merge, but are more efficient, easier to optimize, and more flexible; they easily accommodate algorithmic changes to solve related problems. MGPU's functions continually stress the same few decomposition strategies, turning them into boilerplate code that sets up the problem-specific second phase.
This figure benchmarks the two-phase implementation of Merge (square markers) against the naive parallel version (round markers). Due to the device's advantage in memory bandwidth, even the unoptimized GPU code beats STL by 10x (run on a Sandy Bridge i7 at 2.8ghz). The two-phase implementation beats the naive code by 5x for large inputs.
The two-phase implementation's throughput grows as the workload increases, better filling the device. The naive code hits its highest throughput for small problem sizes. While wider workloads run with better execution efficiency on the device (more concurrency means better latency hiding), this benefit is counteracted by the O(n log n) work-efficiency—the cost of the binary search grows with the log of the input size.
Two-phase design delivers consistently high throughput of merge-like functions, while promoting code reuse and readability.
Most early work on GPU algorithms reduce to scan or scan-like patterns. Scan is a miracle of efficient parallel communication. Radix sort, perhaps the most successful general-purpose CS algorithm to build on GPU, is essentially a very intricate scan: because of the mechanical and regular nature of radix sort, scan manages to both evenly distribute work over threads and solve the key-ranking problem.
Conventional wisdom is to lower every problem to scan, because we've proven that scan helps solve problems with cooperative parallelism. The flaw in this reasoning is that we don't need to actually solve most problems in parallel—it is simpler and more efficient to partition problems in parallel, then solve them sequentially. Modern GPU pushes out the frontier by experimenting with new idioms and only using scan when it is the right tool for the job.
We introduce a new pattern, load-balancing search, which can be thought of as a particular type of inverse of scan. Access to this operator makes certain problems trivial and generally helps reduce the circumlocutions of scan-centric parallel programming. The load-balancing search uses two-phase decomposition to make certain dependencies explicit and further ease scheduling burdens.
Consider a vectorized fill function. It replicates each input, in order, a variable number of times. We'll call it expand.
CPU Expand example.
template<typename T> void Expand(int numOutput, const int* scan, int numInput, const T* values, T* output) { for(int i = 0; i < numInput; ++i) { int offset = scan[i]; int end = (i + 1 < numInput) ? scan[i + 1] : numOutput; std::fill(output + offset, output + end, values[i]); } } int Scan(const int* counts, int numTerms, int* scan) { int x = 0; for(int i = 0; i < numTerms; ++i) { scan[i] = x; x += counts[i]; } return x; } int main(int argc, char** argv) { const char* Alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"; const int Counts[26] = { 3, 1, 0, 0, 7, 3, 2, 14, 4, 6, 0, 2, 1, 5, 3, 0, 5, 1, 6, 2, 0, 0, 9, 3, 2, 1 }; // Scan the counts int Offsets[26]; int total = Scan(Counts, 26, Offsets); std::vector<char> results(total + 1); Expand(total, Offsets, 26, Alphabet, &results[0]); printf("%s\n", &results[0]); return 0; }
AAABEEEEEEEFFFGGHHHHHHHHHHHHHHIIIIJJJJJJLLMNNNNNOOOQQQQQRSSSSSSTTWWWWWWWWWXXXYYZ
We start with a set of input values (the alphabet) and a corresponding set of counts. Scan the counts to compute the offsets for each fill operation. Loop through the list of offsets and call std::fill
to copy each input, values[i]
, Counts[i]
times.
Thrust provides a set of primitives (transform, scan, gather, scatter, compact) that are composed with iterators, operators, and comparators to solve problems. The user typically calls transform, gather, and scatter to prepare intermediate values, scans or compacts them, and uses transform, gather, and scatter to complete the function. The difficulty is that there is no separation between two basically distinct challenges—partitioning and work logic.
Consider this implementation of expand written with Thrust:
// This example demonstrates how to expand an input sequence by // replicating each element a variable number of times. For example, // // expand([2,2,2],[A,B,C]) -> [A,A,B,B,C,C] // expand([3,0,1],[A,B,C]) -> [A,A,A,C] // expand([1,3,2],[A,B,C]) -> [A,B,B,B,C,C] // // The element counts are assumed to be non-negative integers template <typename InputIterator1, typename InputIterator2, typename OutputIterator> OutputIterator expand(InputIterator1 first1, InputIterator1 last1, InputIterator2 first2, OutputIterator output) { typedef typename thrust::iterator_difference<InputIterator1>::type difference_type; difference_type input_size = thrust::distance(first1, last1); difference_type output_size = thrust::reduce(first1, last1); // scan the counts to obtain output offsets for each input element thrust::device_vector<difference_type> output_offsets(input_size, 0); thrust::exclusive_scan(first1, last1, output_offsets.begin()); // scatter the nonzero counts into their corresponding output positions thrust::device_vector<difference_type> output_indices(output_size, 0); thrust::scatter_if (thrust::counting_iterator<difference_type>(0), thrust::counting_iterator<difference_type>(input_size), output_offsets.begin(), first1, output_indices.begin()); // compute max-scan over the output indices, filling in the holes thrust::inclusive_scan (output_indices.begin(), output_indices.end(), output_indices.begin(), thrust::maximum<difference_type>()); // gather input values according to index array // (output = first2[output_indices]) OutputIterator output_end = output; thrust::advance(output_end, output_size); thrust::gather(output_indices.begin(), output_indices.end(), first2, output); // return output + output_size thrust::advance(output, output_size); return output; }
Counts: 0: 3 1 0 0 7 3 2 14 4 6 10: 0 2 1 5 3 0 5 1 6 2 20: 0 0 9 3 2 1 Result of exclusive_scan: 0: 0 3 4 4 4 11 14 16 30 34 10: 40 40 42 43 48 51 51 56 57 63 20: 65 65 65 74 77 79 Result of scatter_if: 0: 0 0 0 1 4 0 0 0 0 0 10: 0 5 0 0 6 0 7 0 0 0 20: 0 0 0 0 0 0 0 0 0 0 30: 8 0 0 0 9 0 0 0 0 0 40: 11 0 12 13 0 0 0 0 14 0 50: 0 16 0 0 0 0 17 18 0 0 60: 0 0 0 19 0 22 0 0 0 0 70: 0 0 0 0 23 0 0 24 0 25 Result of inclusive_scan with thrust::maximum(): 0: 0 0 0 1 4 4 4 4 4 4 10: 4 5 5 5 6 6 7 7 7 7 20: 7 7 7 7 7 7 7 7 7 7 30: 8 8 8 8 9 9 9 9 9 9 40: 11 11 12 13 13 13 13 13 14 14 50: 14 16 16 16 16 16 17 18 18 18 60: 18 18 18 19 19 22 22 22 22 22 70: 22 22 22 22 23 23 23 24 24 25 Result of gather: AAABEEEEEEEFFFGGHHHHHHHHHHHHHHIIIIJJJJJJLLMNNNNNOOOQQQQQRSSSSSSTTWWWWWWWWWXXXYYZ
As in the CPU code, an exclusive scan converts item counts to output indices. The meaning of the rest of the implementation is somewhat obscure; I had to print the intermediate arrays to understand it. Scan is not the most natural primitive to use here, but when all you have is a hammer...
Temporary space to hold one integer per output is allocated and zeroed. scatter_if
outputs the index of each input value to the start of the corresponding output run (the exclusive scan of count) if and only if the count is non-zero. Because zero counts cause consecutive output indices to match, multiple threads would attempt to store to the same address. scatter_if
avoids this race condition by giving priority to the input with the non-zero count. The
call to inclusive_scan
specialized over the maximum
functor fills the zeros with the largest indices encountered to the left. A gather
loads input values at these indices and and stores them to the output, completing the expand.
Expand is a trivial function, but the use of scatter_if
and inclusive_scan
on maximum
is far from an obvious solution. Adopting scan as the primary cooperatively-parallel function is more puzzle-solving than problem-solving. Scan is highly composable and Thrust lets you solve problems without writing new kernels. However, because you're trying to satisfy the logic of the scan function instead of targeting your specific needs, it may require
non-intuitive design.
MGPU introduces the Load-Balancing Search, a pattern that helps developers write elegant implementations of functions like expand. Although this search is available as a host-callable function, it is best invoked from inside a
kernel. The MGPU idiom is less composable than Thrust's: users will need to write their own kernels. The solutions are much more intuitive, however, because the parallel demands of the architecture (i.e. scheduling) are satisfied in the partitioning phase, and the problem-specific logic is executed in a simple, sequential
fasion. The implementation of IntervalExpand
is a more direct solution to the expand problem: it loads input elements just once from global into shared memory and cooperatively fills the output arrays. The function makes only a single pass over the data and requires no auxiliary storage.
As problems become less scan-like, a gather/scatter/scan solution becomes more difficult to understand and express, and the value of composability decreases.
include/kernels/intervalmove.cuh
template<typename Tuning, typename IndicesIt, typename ValuesIt, typename OutputIt> MGPU_LAUNCH_BOUNDS void KernelIntervalExpand(int destCount, IndicesIt indices_global, ValuesIt values_global, int sourceCount, const int* mp_global, OutputIt output_global) { typedef MGPU_LAUNCH_PARAMS Tuning; const int NT = Tuning::NT; const int VT = Tuning::VT; typedef typename std::iterator_traits<ValuesIt>::value_type T; union Shared { int indices[NT * (VT + 1)]; T values[NT * VT]; }; __shared__ Shared shared; int tid = threadIdx.x; int block = blockIdx.x; // Compute the input and output intervals this CTA processes. int4 range = CTALoadBalance<NT, VT>(destCount, indices_global, sourceCount, block, tid, mp_global, shared.indices, true); // The interval indices are in the left part of shared memory (moveCount). // The scan of interval counts are in the right part (intervalCount). destCount = range.y - range.x; sourceCount = range.w - range.z; // Copy the source indices into register. int sources[VT]; DeviceSharedToReg<NT, VT>(NT * VT, shared.indices, tid, sources); // Load the source fill values into shared memory. Each value is fetched // only once to reduce latency and L2 traffic. DeviceMemToMemLoop<NT>(sourceCount, values_global + range.z, tid, shared.values); // Gather the values from shared memory into register. This uses a shared // memory broadcast - one instance of a value serves all the threads that // comprise its fill operation. T values[VT]; DeviceGather<NT, VT>(destCount, shared.values - range.z, sources, tid, values, false); // Store the values to global memory. DeviceRegToGlobal<NT, VT>(destCount, values, tid, output_global + range.x); }
AAABEEEEEEEFFFGGHHHHHHHHHHHHHHIIIIJJJJJJLLMNNNNNOOOQQQQQRSSSSSSTTWWWWWWWWWXXXYYZ
The first half of KernelIntervalMove
is boilerplate. mp_global
points to coarse-grained partitioning information computed in
prior to the launch. CTALoadBalance
uses this to subdivide the input and output ranges into intervals that fit exactly in CTA shared memory.
0: 0 0 0 1 4 4 4 4 4 4 10: 4 5 5 5 6 6 7 7 7 7 20: 7 7 7 7 7 7 7 7 7 7 30: 8 8 8 8 9 9 9 9 9 9 40: 11 11 12 13 13 13 13 13 14 14 50: 14 16 16 16 16 16 17 18 18 18 60: 18 18 18 19 19 22 22 22 22 22 70: 22 22 22 22 23 23 23 24 24 25 80: 0 3 4 4 4 11 14 16 30 34 90: 40 40 42 43 48 51 51 56 57 63 100: 65 65 65 74 77 79
CTALoadBalance
fills shared memory with two non-descending sequences: references to the generating source object for each destination object (in green), and the scan of source object item counts (in black). It's not coincidence that the array of source references is exactly the same as the array of gather indices computed by Thrust's expand function. With one boilerplate call we've already solved the problem! CTALoadBalance
does as much work as the scatter_if
and inclusive scan on maximum
, yet never has to materialize
intermediates into global memory, and so requires no storage.
The kernel moves on to load the source indices into register, freeing up shared memory. It cooperatively loads the referenced source values into shared memory (the 26 letters of the alphabet). Each thread uses the source indices to gather up to VT values, then stores them to the output array. This implementation is much faster than the scan-based version using Thrust. As developers gain more exposure to the patterns involved, code written in this idiom will become easy to read and write.
Because it delivers gather indices so readily, it may seem that CTALoadBalance
is just part of an expand implementation. In fact, the load-balancing search is a highly general tool. Rather than solving the expand problem, it simply partitions for this class of problems. In the case of expand, the code that is executed per work-item is trivial (we just copy from the source to the destination). Interval move (a vectorized memcpy) and relational join (including full outer join) are implemented with the same partitioning boilerplate, but with additional problem-specific logic. The load-balancing search introduces a new pattern for GPU computing, one that I hope will push out the frontier and allow users to run more ambitious calculations.
The algorithms in this project primarily operate on multiple sorted inputs and produce one sorted output. The collection comprises an attempt at addressing the lack of data structures on GPU. Although we don't have self-balancing trees to serve as a data store, we can use multiset union and intersection to add and remove records by key. We can use bulk insert and bulk remove for fine-grained modification of arrays given sorted indices. Vectorized sorted search is a high-throughput search with desirable work-complexity characteristics to help locate records quickly.
Although many of these functions take sorted inputs, this is not an impractical requirement. The load-balancing search pattern, introduced in the expand example, takes a sorted array, but this is typically generated by scanning a sequence of non-negative work-item counts.
Modern GPU covers eleven functions:
Reduce and Scan - Reduce and scan are core primitives of parallel computing. These simple implementations are compact for maximum legibility, but also support user-defined types and operators.
Bulk Remove and Bulk Insert - The first routines that use coarse-grained partitioning. Remove and insert items given a sorted sequence of indices. Merge Path partitioning is introduced to serve bulk insert.
Merge - Uses Merge Path for fine-grained partitioning. The first routine that does not use scan. Develops many patterns for the routines that follow.
Mergesort - Recursively merge sorted sequences. Develops a useful and reusable CTA blocksort. Mergesort's throughput is usually beaten by radix sort for uniform random inputs, but the highly-organized structure of mergesort allows for optimizations on conditioned inputs.
Segmented Sort and Locality Sort - Segmented sort is probably the most versatile GPU sort. This allows us to sort many variable-length arrays in parallel. A list of segment head indices or an array of head flag bits is provided to define segment intervals. Segmented sort is fast: not only is segmentation supported for negligible cost, the function takes advantage of early-exit opportunities to improve throughput over vanilla mergesort. Locality sort is a useful variant that detects regions of approximate sortedness without requiring annotations.
Vectorized Sorted Search - Run many concurrent searches where both the needles and haystack arrays are sorted. This input condition lets us recast the function as a sequential process resembling merge, rather than as a traditional binary search. Complexity improves from A log B to A + B, and because we touch every input, a search can retrieve not just the lower-bound of A into B but simultaneously the upper-bound of B into A, plus flags for all elements indicating if matches in the other array exist.
Load-Balancing Search - Load-balancing search is a specialization of vectorized sorted search. It coordinates output items with the input objects that generated them. The CTA load-balancing search is a fundamental tool for partitioning irregular problems.
IntervalExpand and IntervalMove - Schedule multiple variable-length fill, gather, scatter, or move operations. Partitioning is handled by load-balancing search. Small changes in problem logic enable different behaviors. These functions are coarse-grained counterparts to Bulk Remove and Bulk Insert.
Relational joins - Sort-merge joins supporting inner, left, right, and outer variants. Uses vectorized sorted search to match keys between input arrays and load-balancing search to manage Cartesian products.
Multisets - Replace Merge Path partitioning with the sophisticated Balanced Path to search for key-rank matches. The new partitioning strategy is combined with four different serial set operations to support CUDA analogs of std::set_intersection
, set_union
, set_difference
, and set_symmetric_difference
.
Segmented reduce - .


{% extends "info_base.html" %} {% load hunt_tags %} {% block title %}Not Available{% endblock title %} {% block content %}
The resource or page you are attempting to access is associated with a puzzlehunt that is not yet started. Please refresh this page after the hunt has started.
{% endif %} {% if reason == "team" %}You must be participating in the current hunt to view this resource or page, and it does not appear you are on a team for this hunt. Please ensure you are signed in correctly and that you have properly added yourself to a team on the registration page.
{% endif %} {% if reason == "puzzle" %}This resource or page is part of an ongoing puzzlehunt and is not yet unlocked for your team. Please either continue playing to unlock it or wait until the hunt is public.
{% endif %}For assistance, email puzzlehunt staff..


Community
Modify
delete
name
date
title
content
Return to list
close
Password Check
password
Passwords do not match.
close
Password Check
password
Passwords do not match.


Hooray!
Your vote has been successfully submited. {{ candidate }}
You can now check your vote, but you cannot modify it or vote again.
{% extends "voting/main.html" %} {% load static %} {% block title %} BlockVote {% endblock %} {% block body %}
Your vote has been successfully submited. {{ candidate }}
You can now check your vote, but you cannot modify it or vote again.


Genesis 30:36
And he set three days' journey betwixt himself and Jacob: and Jacob fed the rest of Laban's flocks.


Question: Consider the following data below. How many variables does this dataset contain?
| males | females |
|---|---|
| 4 | 1 |
| 2 | 5 |
The way the table is presented, it seems like there are only two variables. However, the correct answer is 3. The variables are injured?
, count
and gender
.
A dataset is said to be tidy if it satisfies the following conditions
Tidy data makes it easy to carry out data analysis.
Let us explore some common causes of messiness by inspecting a few datasets.
Income Distribution by Religion
Our first dataset is based on a survey done by Pew Research that examines the relationship between income and religious affiliation.
Read the dataset into your R session and inspect the first few rows to assess if it is tidy.
pew <- read.delim(
file = "http://stat405.had.co.nz/data/pew.txt",
header = TRUE,
stringsAsFactors = FALSE,
check.names = F
)
| religion | <$10k | $10-20k | $20-30k | $30-40k | $40-50k | $50-75k | $75-100k | $100-150k | >150k |
|---|---|---|---|---|---|---|---|---|---|
| Agnostic | 27 | 34 | 60 | 81 | 76 | 137 | 122 | 109 | 84 |
| Atheist | 12 | 27 | 37 | 52 | 35 | 70 | 73 | 59 | 74 |
| Buddhist | 27 | 21 | 30 | 34 | 33 | 58 | 62 | 39 | 53 |
| Catholic | 418 | 617 | 732 | 670 | 638 | 1116 | 949 | 792 | 633 |
Except for religion
, the rest of the columns headers are actually values of a lurking variable income
. This dataset violates the second rule of tidy data.
Tuberculosis Incidence
tb <- read.csv(
file = "http://stat405.had.co.nz/data/tb.csv",
header = TRUE,
stringsAsFactors = FALSE
)
| iso2 | year | new_sp | new_sp_m04 | new_sp_m514 | new_sp_m014 | new_sp_m1524 |
|---|---|---|---|---|---|---|
| ZW | 2004 | 14581 | 187 | 833 | ||
| ZW | 2005 | 13155 | 210 | 837 | ||
| ZW | 2006 | 12718 | 215 | 736 | ||
| ZW | 2007 | 10583 | 6 | 132 | 138 | 500 |
| ZW | 2008 | 9830 | 127 | 614 |
Except for iso2
and year
, the rest of the columns headers are actually values of a lurking variable, in fact combination of two lurking variables, gender
and age
.
Weather Data
weather <- read.delim(
file = "http://stat405.had.co.nz/data/weather.txt",
stringsAsFactors = FALSE
)
| id | year | month | element | d1 | d2 | d3 | d4 | d5 | d6 | d7 |
|---|---|---|---|---|---|---|---|---|---|---|
| MX000017004 | 2010 | 10 | TMIN | 140 | 129 | |||||
| MX000017004 | 2010 | 11 | TMAX | 313 | 272 | 263 | ||||
| MX000017004 | 2010 | 11 | TMIN | 163 | 120 | 79 | ||||
| MX000017004 | 2010 | 12 | TMAX | 299 | 278 | |||||
| MX000017004 | 2010 | 12 | TMIN | 138 | 105 |
This dataset seems to have two problems. First, it has variables in the rows in the column element
. Second, it has a variable d
in the column header spread across multiple columns.


Sign Up!
First :
Last :
Gender :
Select A Gender
Male
Female
Other
Email :
Password :
Re-enter Password :


Thierry Henry predicts where Arsenal will finish this season
Arsenal legend Thierry Henry is still hoping Unai Emery's side can finish in the Premier League's top four
by The Sport Review staffThierry Henry says he is still hopeful Arsenal can finish in the Premier League’s top four this season.
The Gunners are currently in sixth spot in the Premier League table and eight points adrift of defending champions Manchester City in fourth place.
Arsenal lost 2-0 to Leicester City at the King Power Stadium last weekend following goals from Jamie Vardy and James Maddison.
The north London side have only managed to win one of their last six Premier League games to heap pressure on Gunners boss Unai Emery.
Arsenal are nine points adrift of Leicester City and Chelsea FC, who appear to be the most likely candidates to claim the final two top-four spots.
Former Gunners striker Henry is still attempting to remain optimistic that Arsenal can secure a long-awaited return to the Champions League.
“For me, as a fan, I would like us to get back into the top four and get Champions League football back at the Emirates.” Henry told Yahoo Sport.
“Right now, it looks like not just us who are struggling but also United and Tottenham are struggling, so you have to hope we can go on a run.”
Henry added: “What do they need to change? I’m not in the dressing room every day to know what is happening in training. That’s the job of Emery or whoever is in charge.
“As a fan, I would like to see them back in the Champions League, that will be a start because catching Liverpool at the minute seems very complicated. Even for Manchester City. Let’s see where we can finish at the end of the season.”
Arsenal will have the opportunity to restore some feel-good factor at The Emirates when the Gunners take on Premier League strugglers Southampton in their next fixture on Saturday 23 November.
The Gunners signed six new players in the summer transfer window as Emery looked to bolster his squad ahead of a potential top-four challenge.
Arsenal snapped up Gabriel Martinelli, William Saliba, Dani Ceballos, Kieran Tierney, David Luiz and Nicolas Pepe in the summer.


About Sports Weather
Sports Weather uses the free popular weather API from Open Weather API to get the current up to date weather from thousands of places. Using simple fetch requests in Javascript, Sports Weather is able to display the current weather in cities where sports are being played for the user. Thanks for using my application and enjoy!


Watch King Princess perform ‘1950’ and ‘Hit the Back’ in her ‘SNL’ debut
She joined Will Ferrell who hosted episode 878 of the comedy series
King Princess served as the musical guest on Saturday Night Live this weekend – watch her performances below.
Joining Will Ferrell, who returned to host the show for a fifth time, King Princess performed her debut single ‘1950’ along with ‘Hit the Back’ from her recently released album, ‘Cheap Queen’.
In a four-star NME review, Thomas Hobb said of Mikaela Straus and ‘Cheap Queen’: “The Mark Ronson-endorsed New Yorker recalibrates the traditional love song for an LGBTQ audience and her debut studio album soars at its most experimental.”
Watch clips of King Princess’ SNL performances below:
The pair previously collaborated on ‘Pieces Of Us’, a cut from Ronson’s latest record ‘Late Night Feelings’. Their remake of the 1967 Turtles hit was recorded for Tiffany & Co.’s latest campaign for its new line of fragrance, Tiffany & Love.
Taking her Big Little Lies fandom to the next level, she sampled a moment from season two for her latest record, ‘Mary-Luiz (Plz Plz)’.


ADOBE GURO | Retouch images
ADOBE PS TUTORIALS
Learn how to remove unwanted objects, add objects by cloning, and fix other imperfections in your images with retouching tools in Adobe Photoshop.
-
Remove small objects
Use the Spot Healing Brush tool to remove objects
-
Add objects by cloning
Use the Clone Stamp tool to add objects
-
Remove a large object
Use the Content-Aware fill command to remove an object


| CS 341 - Survey of Programming Languages
Homework #2 |
Note: This work is to be done in pairs. Each group will submit one assignment. Although you may divide the work, both team members should be able to present/describe their partner's work upon request.
To submit work on our systems, change into the directory where your hw2.scm resides and enter the shell command "submit341 hw2.scm". This assignment will be manually graded and there will be no immediate feedback.
Read the tutorial up through "Pairs and Lists", and complete the following companion exercises in the commented sections of file hw2.scm:
REPL Exercises
3-4
Operators and Operands Exercises 2-4
Identifiers, Whitespace, and Comments Exercise 3
Fundamental Types Exercises 1-3, 6-7
Pairs and Lists Exercises 3-6


You can use this for attaching directives to the resolve container dynamically (when using ngIf / ngSwitch /
ngFor). This is very useful if you have multiple places where you use the same resolver and they can be
conditionally visible.
This means that we at a certain point we can have only one visible, or have them both or have none. If we have
none of the visible basically nothing happens.
If we have only one visible no matter which one a request is dispatched to load the data and after it gets loaded
it gets presented to the user. In the case when we have
both visible only the first one sends a request to get the data resolved and the other one uses the first one as a
delegate (it doesn't do anything else than get the data)
Current value: {{showUserPostsCount}}
Current
value: {{showUserPosts}}
Note: UserService loadUsers and PostService loadPosts have a delay operator
applied!
Also if you go to the user
service you can uncomment to throwError line in order to view how the error will be displayed


REPORT_VIEW privileges are required to perform this operation. You don't have the required permissions.
TracGuide — The Trac User and Administration Guide


Felipe Silveira
Profile
Felipe Silveira has more than 15 years of experience in the IT industry, working for Brazilian and Global companies as a Senior IT Consultant, Infrastructure Specialist, and System Analyst, with large experience delivering infrastructure solutions for monitoring systems, performance & capacity planning and pre-sales, and since 2017 working with Kubernetes and building microservices with Golang.
Experience
IBM
Hortolandia, Brazil
Application Developer
Jun 2020 - Present
- Assigned as Application Developer on M&EaaS (Monitoring & Event Automation as a Service) team, being responsible for creation and maintenance of an API synchronization micro-service written in Go Language -- which extracts information from different monitoring tools and consolidates data from multiple sources into a single API.
- Responsible for CI/CD tools such as Jenkins, UrbanCode Deploy and for the build and deployment pipeline currently in use by the team, being also responsible for development and support of all the Helm Charts and Dockerfiles currently used by the team for packaging/deploying applications.
- Responsible for creating tools and scripts used by development team to increase productivity, such as Command Line utilities for facilitating interaction and load tests on different API's and Ansible Playbooks for automating deployment of services in general.
- IT Infrastructure
- Automation
- Monitoring
- APM
- Linux
- Docker
- Kubernetes
- OpenShift Container Platform
- Jenkins
- UrbanCode Deploy
- Bash
- Python
- GoLang
- Ansible
- Prometheus
- Grafana
- ELK
- Tivoli Monitoring
- Agile'
- Git
Site Relieability Engineer
Nov 2017 - Jun 2020
- Part of the M&EaaS (Monitoring & Event Automation as a Service) DevOps team, which responsible for the M&EaaS Portal, which provides integrated monitoring services to multiple IBM customers from Europe.
- I'm currently responsible for the deployment and availability of the solution, supporting both Development and Ops teams by automating and developing better ways of doing things and also on-boarding new customers..
- Linux
- Kubernetes
- Docker
- Jenkins
- UrbanCode Deploy (UCD)
- Bash
- Python
- Ansible
- Ansible Tower
- Prometheus
- Tivoli Monitoring
- IPM8
Lead System Engineer
Jan 2015 - Nov 2017
- Responsible for developing new monitors and automation to support Michelin North America monitoring infrastructure in order to guarantee quality and availability using ITM6 and BlueCARE.
- IT Infrastructure
- Automation
- Monitoring
- Red-Hat
- SUSE
- AIX
- Bash
- Python
- Tivoli Monitoring
- ITM6
- BMC Patrol
- IBM BlueCARE Portal
Subject Matter Expert
Jul 2011 - Jan 2015
- Assigned as a Subject Matter Expert for Enterprise Automation team being responsible for automating and supporting the monitoring infrastructure using ITM5, ITM6 and BlueCARE. supporting Global Accounts such as Whirlpool, Honda Motor Company, GAP, IBM Canada, Michelin North America and Morgan Stanley.
- Automation
- Monitoring
- Red-Hat
- SUSE
- AIX
- Bash
- Tivoli Monitoring
- ITM5
- ITM6
- ITCAM RRT
- BMC Patrol
- BMC ProactiveNet
IT Specialist assigned at IBM e-Business Hosting
Feb 2009 - Jul 2011
- Specialist on IT infrastructure solutions, part of the Enterprise Automation team at IBM Brazil. Supported a large BMC Patrol environment with over 4000 monitoring agents integrated with different Tivoli monitoring tools on multiple environments, using Tivoli Configuration Management, Tivoli Distributed Monitoring and ITM 6.2 solutions for a pool with over 100 off-shore accounts in different sectors such as Banking, Health-care, Oil & Gas, Automotive, Energy, Retail, News Media and Public Sector.
- Also responsible for the development and delivery of new solutions focusing on automation of manual tasks for cost reduction and revenue assurance
- Automation
- Configuration Management
- Monitoring
- Red-Hat
- SUSE
- AIX
- Bash
- Tivoli
- ITM5
- ITM6
- ITCAM
- TCM
- BMC Patrol
BearingPoint
Rio de Janeiro, Brazil
Senior Infrastructure Consultant
Oct 2007 - Nov 2008
- Specialist in IT infrastructure solutions, part of the Technology & Solutions team. Responsible for helping on pre-sales and on implementations of IT infrastructure projects on BearingPoint Brazil.
- Acted as part of the Knowledge Management team at VALE. Being responsible for managing new demands and generating the Key Performance Indicators for the mining assets fuel supply.
- IT Infrastructure
- Outsourcing
- Configuration Management
- Pre-Sales
- Project Management
- Knowledge Management
IBM Brazil
Rio de Janeiro, Brazil
IT Specialist - Assigned to Carrefour Brazil
Aug 2006 - Oct 2007
- Specialist in IT infrastructure solutions, part of the Brazilian Automation service -line at IBM Brazil. Helping to support non IBM solutions on Brazil and on off shore accounts.
- Responsible for development and delivery of new solutions focusing automation of manual tasks and operations for cost reduction and revenue assurance.
- IT Infrastructure
- Automation
- Outsourcing
- Configuration Management
- Pre-Sales
- Project Management
- Knowledge Management
EDS
Sao Caetano, Brazil
Infrastructure Specialist
Apr 2006 - Aug 2006
- IT specialist with focus on supporting the EDS clients, using solutions such as CA Unicenter TNG / NSM, BMC PATROL and BMC PATROL Performance.
- Part of the Automation and Tools off-shore group, supporting accounts in the US like NASD and Nextel.
- Automation
- Configuration Management
- Monitoring
- Red-Hat
- SUSE
- AIX
- Bash
- BMC Patrol Performance
AUM Management (BMC Software Partner)
Rio de Janeiro, Brazil
IT Consultant
Aug 2003 - Apr 2006
- IT consultant/Pre-sales, with focus on Infrastructure Management and Service Desk solutions using BMC Software and Computer Associates solutions in large/medium accounts like Petrobras, Aracruz Celulose, Light, Oi(Telemar), Blah!(TIM), Andima, Unimed, White Martins and SITA.
- Automation
- Configuration Management
- Monitoring
- Red-Hat
- SUSE
- Solaris
- Bash
- BMC Patrol
- BMC Patrol Performance
- Unicenter NSM
- Unicenter Service Desk
Education
Computer Science
Estacio de Sa University (Rio de Janeiro/Brazil)
Incomplete/Dropout
- +55 19 99979-6185
-<EMAIL_ADDRESS>- https://silveiralexf.github.io
- Brazil
Skills
-
-
OS/Platforms
- RedHat
- Ubuntu
- SUSE
- AIX
- Solaris
- Kubernetes
- IBM Cloud
- IBM Cloud Private
- OpenShift Container Platform
-
Programming
- Golang
- Python
- Bash
- Git
-
Tools
- Ansible
- ELK
- Grafana
- Docker
- Grafana
- AWS
-
Monitoring
- Prometheus
- NewRelic
- ITM5
- ITM6
- IPM8
- ITCAM
- BMC Patrol
- Unicenter TNG/NSM
Languages
-
PortugueseNative
-
EnglishFluent
-
SpanishBasic
Awards
- Cognitive Innovation Leadership Award (Dec/2019)
- IBMer's Essential Award (Apr/2017)
- IBM Manager's Choice Award (Nov/2017)
- IBM Manager's Choice Award (Nov/2016)


type Domain struct { Address string `json:"domain"` Users []User `json:"users"` }
Domains are identified by their Address (unique) and contain a slice of all the registered Users for that domain
type Proxy struct { Domains []Domain }
Proxy data model of a slice of Domains searched when authentication request is made. Proxy is the highest level data model in proxyauth, the golang type analogous to users.json
func NewProxy(filePath string) (*Proxy, error)
Parse the json users file (filePath) and return the proxy data type.
func (p *Proxy) Authenticate() http.Handler
Proxy Authentication handler expects: domain (mux url variable) username and password (query parameters). HTTP response returns the appropriate json response and Status Code as specified in SPEC.md
type Response struct { Success bool `json:"access_granted"` Reason string `json:"reason,omitempty"` }
Response is used for communicating the result from an attempted authentication
type User struct { Username string `json:"username"` Password string `json:"password"` }
Users are identified by their Username (unique) and each has a Base 64 encoded, SHA 256 digest of the users password. See SPEC.md or b64sha256 for specific implementation details


Introduction to Arrays and Hashes
A beginner's guide to the differences between arrays and hashes
in Ruby and when to use them
May 17, 2015
It won't be long after you begin learning Ruby that you realize how useful arrays and hashes are. You can think of an array as a list, a collection of data in some kind of order. Every slot in the list acts like a variable, where you can see what object each slot points to and identify an element, or piece of data, by using its slot position on the list. Since programming counts starting from zero, the first slot in the array is indexed as slot zero. The following is an example of an array:
puppies = ['LaSalle', 'Stella', 'Sophie', 'BeBop']
puts puppies[0] # => LaSalle
puts puppies[1] # => Stella
puts puppies[2] # => Sophie
puts puppies[3] # => BeBop
puts puppies # => LaSalle
Stella
Sophie
BeBop
Both arrays and hashes are collections used to store and retrieve data. A hash (sometimes called an associative array) is similar to an array, except that a hash works as a dictionary-like collection of unique keys and their values. Where an array uses integers as its index, a hash allows you to use any object type. You can think of a hash as a collection of data pairs, where each element (name) in the hash is linked to a specific result (value). According to ruby-doc.org, "Hashes enumerate their values in the order that the corresponding keys were inserted." An example of a hash is as follows, where we want to know what kind of breed mix each puppy is by linking it to the dog's name in a hash:
breed_mix = { LaSalle: "Labrador", Stella: "Bulldog", Sophie: "Boxer", BeBop: "Hound" }
puts breed_mix[:LaSalle] # => Labrador
puts breed_mix[:Stella] # => Bulldog
puts breed_mix[:Sophie] # => Boxer
puts breed_mix[:BeBop] # => Hound
How do you know when it's better to use an array or a hash? It depends on the kind of list you want to create and access. A simple array will work for a collection of elements that you just need to call by the index (order) number to list out, such as words in a sentence, days of the week, and so on. A hash will be more useful where you want to retrieve data using a key value, such as with grades assigned to particular students or a table of contents with corresponding page numbers. You can store any type of value in both arrays and hashes. Moreover, you can have arrays inside of arrays, arrays inside of hashes, hashes inside of arrays and hashes inside of hashes, depending on what you want to program or index. It is worth experimenting with the basic forms of both of these in order to better understand their differences and how to apply each one in a given situation.


David Greymaax
1 Minutes ago
15 Minutes ago
Just a reminder that you have an event today
Update dashboard


AMAZING PEOPLE ARE MAKING AMAZING DESIGNS IN A FUN ENVIRONMENT
When you work with us, you are part of ateam of committed collaborators. All aspects of our software development process are intergrated, interactive and agile.
Our approach unifies design, development and product management to create exceptional products.
We start each project by understanding your business goals, the user's needs and the product requirements
Then we help translate them into effective and elegant solutions.
Before we sign a contract or write a line of code our team will spend the necessary time needed to understand yoyur product vision
The outcome of this scoping session is a proposal or high-level statement of work.
Based on your requirements, we can provide a complete offering or just the pieces you need.
Our design practice offers a full range of services including brand strategy, interaction and visual design and user experience testing. Throughout your project, our designers create and implement visual design and workflows, solicit user feedback and work with you to make sure what gets built is what is needed.
All engineers are fluent in the latest enterprise, mobile and web development technologies. They collaborate with your team to write, and improve code on a daily basis, using proven practices such as test-driven development and pair programming.
Planning and development is iterative because we are constantly coding and testing, the products we build are always ready to go live. This iterative process allows for changes as business requirements evolve.


Welcome to my Phase 0
page.
This is where I will be posting all of my online content during Phase 0
, and also where I will be linking to my blogs and other related projects. This is definitely still a work in process, I am still working through the kinks, I am definitely on my way to redesigning my website in the way that I want to, and I will be adding some cool stuff when I find some cool resources later on. Thank you for your patience!
Cultural Blogs:
This one's my
Time and Habits blog, where I talk about making good habits and timeboxing.
My
Reflection blog., where I talk about my first reflections of the Fireside Chat with Shereef video.
Here another one: my
Emotional Intelligence blog, where I reflect on the first chapter of the book
Search Inside Yourself.
This is my
Meditation Process blog, where I reflect on Chapter 2 of
Search Inside Yourself and the process over product concept.
Here's my
Daily Mindfulness blog, where I reflect on Chapter 3 of
Search Inside Yourself.
Here's my
Self-Confidence blog, where I reflect on Chapter 4 of
Search Inside Yourself.
This is my
Control blog, where I reflect on Chapter 5 of
Search Inside Yourself.
Here is my
Motivation blog, where I reflect on Chapter 6 of
Search Inside Yourself.
This one is my
Connecting blog, where I reflect on Chapter 7 of
Search Inside Yourself.
Here is my
Leadership blog, where I reflect on
Search Inside Yourself, Chapter 8.
Technical Blogs
Here's my
Technical 1 blog, where I reflect on my first Udacity lessons on frameworks and boxifying.
My
Technical 2 blog, where I reflect on semantic structuring and responsive design.
Here's my
Design to Web blog post, where I discuss my page design, frameworks, wireframes and page responsiveness.
Here's my
Javascript Basics blog, where I discuss my first interactions with JS.
This is my
Problem Solving blog, where I talk about my methods for problem solving and what has worked for me so far in this course.
Here's my
Understanding Javascript Language blog, where I reflect on what various symbols mean in JavaScript.
This one's my
Scope blog, where I talk about what scope in JS is.
For my tutors who are marking this, and also for my own use, here is my
main blog template.


Roots
List #10
| Root | Meaning | Origin | Example(s) |
|---|---|---|---|
| -phys- | nature | Greek | physics, physiology, physiocratic |
| -plur- | more | Latin | pluralist, plural, plurality |
| -port- | to carry | Latin | portal, export, portfolio |
| -pos- | to place / put / lay | Latin | jjuxtapose, compose, expose, posture |
| -poten- | powerful | Latin | potent, despot, potential, bellipotent |
| -pyr- | heat / fire | Greek | pyroclastic, pyromania, pyrite |
| -quer- | -ques- | -quir- | -quis- | to ask or seek | Latin | query, conquest, exquisite |
| -rog- | to ask or demand | Latin | interrogate, aarrogant, prerogative |
| -rot- | wheel | Latin | rotisserie, rotation, rototiller |
| -rupt- | to break or burst | Latin | interrupt, corruption, abrupt, erupt |
Previous List Next List


| lesson 59 Collecting |
| People tend to amass possessions, sometimes without being aware of doing so. |
| Indeed they can have a delightful surprise when they find something useful which they did not know they owned. |
| Those who never have to move house become indiscriminate collectors of what can only be described as clutter. |
| They leave unwanted objects in drawers, cupboards and attics for years, in the belief that they may one day need just those very things. |
| As they grow old, people also accumulate belongings for two other reasons, |
| lack of physical and mental energy, both of which are essential in turning out and throwing away, and sentiment. |
| Things owned for a long time are full associations with the past, perhaps with relatives who are dead, |
| and so they gradually acquire a value beyond their true worth. |
| Some things are collected deliberately in the home in an attempt to avoid waste. |
| Among these I would list string and brown paper, |
| kept by thrifty people when a parcel has been opened, to save buying these two requisites. |
| Collecting small items can easily become a mania. |
| I know someone who always cuts sketches out from newspapers of model clothes that she would like to buy if she had the money. |
| As she is not rich, the chances that she will ever be able to afford such purchases are remote; |
| but she is never sufficiently strong-minded to be able to stop the practice. |
| It is a harmless habit, but it litters up her desk to such an extent that every time she opens it, loose bits of paper fall out in every direction. |
| Collecting as a serious hobby is quite different and has many advantages. |
| It provides relaxation for leisure hours, as just looking at one's treasures is always a joy. |
| One does not have to go outside for amusement, since the collection is housed at home. |
| Whatever it consists of, stamps, records, first editions of books china, glass, |
| antique furniture, pictures, model cars, stuffed birds, toy animals, |
| there is always something to do in connection with it, from finding the right place for the latest addition, to verifying facts in reference books. |
| This hobby educates one not only in the chosen subject, but also in general matters which have some bearing on it. |
| There are also other benefits. |
| One wants to meet like-minded collectors, to get advice, to compare notes, to exchange articles, to show off the latest find. |
| So one's circle of friends grows. |
| Soon the hobby leads to travel, perhaps to a meeting in another town, possibly a trip abroad in search of a rare specimen, |
| for collectors are not confined to any one country. |
| Over the years, one may well become an authority on one's hobby and will very probably be asked to give informal talks to little gatherings and then, |
| if successful, to larger audiences. |
| In this way self-confidence grows, first from mastering a subject, then from being able to talk about it. |
| Collecting, by occupying spare time so constructively, makes a person contented, with no time for boredom. |


| TEP: | 104 |
|---|---|
| Group: | Core Working Group |
| Type: | Experimental |
| Status: | Draft |
| Author: | Kevin Klues |
| Draft-Created: | 12-Oct-2004 |
| Draft-Version: | 1.3 |
| Draft-Modified: | 2005-02-14 |
| Draft-Discuss: | TinyOS Developer List <tinyos-devel at mail.millennium.berkeley.edu> |
Note
This memo documents a part of TinyOS for the TinyOS Community, and requests discussion and suggestions for improvements. Distribution of this memo is unlimited. This memo is in full compliance with TEP 1.
This TEP proposes a hardware abstraction architecture for Radio Components used in TinyOS. The ideas introduced in this document borrow from those introduced in TEP 2 regarding the three-layer Hardware Abstraction Architecture (HAA).
The hardware abstraction of a radio component can be divided into three separate layers. Following the tradition of the three-layer Hardware Abstraction Architecture (HAA) these layers are labeled as the Hardware Physical Layer (HPL), Hardware Adaptation Layer (HAL), and the Hardware Interface Layer (HIL).
The block diagram of the proposed architecture, including example interfaces provided by each layer, can be seen in the diagram below:
| | RadioData | | RadioControl | | ---------------------------------------- | HILXXXRadio | ---------------------------------------- | | HALXXXRadioData | | HALXXXRadioControl | | ---------------------------------------- | HALXXXRadio |------- BusInterface ---------------------------------------- | | | HPLXXXRadioData | HPLXXXBusComm | HPLXXXRadioControl | | | ---------------------------------------- | HPLXXXRadio | ----------------------------------------
In the following sections, each layer will be discussed in detail, and the details of the interfaces required/provided by each will be examined.
Note
The naming convention for the different interfaces provided by the various layers in the architecture, as well as the names of the components themselves will be used throughout this document as shown in the diagram above.
The three Layers are divided in such a way that the HPL and HIL layers are kept solely radio dependent, while the HAL is not just radio dependent, but also microcontroller dependent.
With this in mind, the HPL and HIL layers can be thought of as platform independent, in that they do not rely on a specific microcontroller for their implementation. They only need to be implemented once, and stored somewhere for use. These files, however, cannot be used until they are wired to another file which implements the interfaces they require. This file IS microcontroller dependent, and takes the form of the HAL layer in the radio abstraction heirarchy.
The next few subsections describe how each of these layers fit together, and how one might go about implementing them.
At the bottom most layer, the implementation of the radio HPL is highly radio dependent. At this layer, all functionality offered by the radio in terms of configuring it or controlling its various modes of operation SHOULD be somehow provided. A means in which to send data over the radio MUST also be provided. In order to accomplish this, the radio HPL MUST connect directly to the TinyOS bus component that is used to physically communicate with the radio hardware. To keep the radio HPL separated from how this communication is actually performed, a radio specific BusComm interface is introduced that is used to encapsulate the commands necessary to perform reads and writes to the radio hardware over the physical bus. The radio HPL MUST use this interface to perform all reads and writes to the actual radio hardware.
An example of such an interface for a non-register-based, byte interface radio is given below:
interface HPLXXXBusComm { async command result_t tx(uint8_t byte); async command result_t rxDone(uint8_t byte); }
An example of such a BusComm interface for a register-based, byte interface radio is given below:
interface HPLXXXBusComm { async command result_t writeRegByte(uint8_t address, uint8_t byte); async command result_t writeRegWord(uint8_t address, uint16_t word); async command result_t readRegByte(uint8_t address, uint8_t byte); async command result_t readRegWord(uint8_t address, uint16_t word); async command result_t tx(uint8_t byte); async command result_t rxDone(uint8_t byte); }
No other interfaces other than the BusComm interface SHOULD need to be used by the HPL implementation, but the HPL implementation is not necessarily restricted to just this single interface. Which interfaces it needs to use will be dictated by the hardware specifications of the chip and need to be accomodated for as necessary.
A control interface and a data interface MUST also be provided by the radio HPL layer. These interfaces MUST provide a path for configuration/control information and data information to flow to and from the radio. The actual definitions of these interfaces are completely hardware dependent, and need only be defined so as to reliably communicate with the HAL layer. Because of this fact, these interfaces may not necessarily be just single interfaces, but rather multiple interfaces that are divided into whatever may be logical for a given radio chip.
An example of the control interface for a register-based, byte interface radio is given below:
interface HPLXXXRadioControl { command void init(); command void HighLNAGain(); command void LowLNAGain(); async command result_t setTxMode(); async command result_t setRxMode(); async command result_t setSleepMode(); ... ... ... command void PowerDown(); command void PowerUp(); }
The data interface for the same type of radio would look like:
interface HPLXXXRadioData { async command result_t tx(uint8_t byte); async command result_t rxDone(uint8_t byte); }
For a packet based radio, multiple data and control interfaces may exist.
[These example interfaces were taken directly from the TEP covering Link Layer Primitives in TinyOS (TEP 105)]:
interface HPLXXXCmd { async command uint8_t cmd(uint8_t addr); async command uint8_t write(uint8_t addr, uint16_t data); async command uint16_t read(uint8_t addr); } interface HPLXXXCapture { async command result_t enableCapture(bool low_to_high); async event result_t captured(uint16_t val); async command result_t disable(); } interface HPLXXXFIFO { async command cc2420_result_t readRXFIFO(uint8_t length, uint8_t *data); async command cc2420_result_t writeTXFIFO(uint8_t length, uint8_t *data); async event result_t RXFIFODone(uint8_t length, uint8_t *data, cc2420_result_t success); async event result_t TXFIFODone(uint8_t length, uint8_t *data, cc2420_result_t success); }
The implementation of these interfaces in the HPL MUST use the commands provided by the BusComm interface, and will therefore be kept independent of which platform they are running on. As will be seen in the next section, the implementation of the BusComm interface in the HAL layer of the radio will provide the required access for getting the processor to run on a specific platform. Because of this fact, applications SHOULD never directly wire themselves to the HPL layer of the radio, unless they are also prepared to define the protocol used for sending data through the BusComm interface.
A portion of the HPL implementation file for the example register-based, byte interface radio is shown below:
module HPLXXXRadioM { provides { interface HPLXXXRadioControl; //Control/Config Communication interface HPLXXXRadioData; //Data Communication } uses { interface HPLXXXBusComm; } } implementation { uint16_t currentConfigReg; command result_t HPLXXXRadioControl.init() { TOSH_SEL_RADIO_EN_IOFUNC(); TOSH_SEL_RADIO_TXRX_IOFUNC(); TOSH_SET_RADIO_TXRX_PIN(); TOSH_SET_RADIO_EN_PIN(); ... currentConfigReg = DEFAULT_CONFIG_VALUE; } ... command void HPLXXXRadioControl.HighLNAGain() { currentConfigReg |= HIGH_LNA_MASK; TOSH_CLR_RADIO_EN_PIN(); TOSH_uwait(1); HPLXXXBusComm.writeRegWord(CONFIG_ADDRESS, currentConfigReg); TOSH_SET_RADIO_EN_PIN(); TOSH_uwait(1); } ... command result_t HPLXXXRadioData.tx(uint8_t byte) { call HPLXXXBusComm.tx(byte); } async event result_t HPLXXXRadioData.rxDone(uint8_t byte) { signal HPLXXXBusComm.rxDone(byte); } }
Unlike the HAA specification (which requires that all HPL implementations be stateless), the implementation of the radio HPL layer MAY be stateless, but is not required to be. This is due to the fact that the radio is an off-chip component and it may be necessary to store some information about the state of the radio that would otherwise be accessible by the internal registers of on-chip components. In the example file above, the "currentConfigReg" variable is used to keep state about the current configuration of the radio.
The HPL implementation is, however, REQUIRED to have exclusive access to the pins that are used for configuring the radio. It is responsible for setting pin directions as well as setting or clearing these pins via MACROs that will be later defined by a given platform.
By looking at the definitions of the various commands in the example file above, it can be seen how the HPLXXXBusComm commands are used to actually write data to the radio. Calling the commands in such a way provides the flexibility to allow the radio HPL layer to prepare itself as necessary both before and after a read or a write, while keeping the implementation independent of the actual protocol being used for performing the read or write.
With the organizational structure described above, the HPL implementation of any given radio is able to be completely self contained. Because of this, all interfaces defined for the HPL layer of a radio as well as the implementation itself should be kept in the radio's local "chips" directory.
The HAL layer of the radio hardware abstraction is responsible for actually connecting the radio to the microcontroller used on a given platform. It is the only layer in the radio heirarchy that is platform DEPENDENT, and will therefore be stored in the given "platform" directory rather than the "chips" directory.
This layer is used to implement the HPLXXXBusComm interface, set up platform specific settings for the radio, and provide any control and data interfaces specific to the given radio. Any required bus arbitration is also performed at this layer, along with any timing issues being resolved for delayed responses to commands issued for the radio.
For most platforms, the bus component used will be designed around a standard communication protocol (such as SPI or I2C) that many processors and radios will be able to natively support.
An example of the HAL implementation file for a register-based, byte interface radio is shown below:
module HPLXXXRadioM { provides { interface SplitControl; interface HALXXXRadioControl; //HAL Control/Config Communication interface HALXXXRadioData; //HAL Data Communication interface HPLXXXBusComm; //BusComm for the HPL layer } uses { interface HPLXXXRadioControl; //HPL Control/Config Communication interface HPLXXXRadioData; //HPL Data Communication interface BusInterface; //Could be SPI or I2C interface BusArbiration; interface TimerJiffy; } } implementation { ... }
Since this layer has intimate knowledge of exactly what process is being used to communicate with the radio, it has alot of flexibility in how it chooses to implement things. The actual implementation and definition of the interfaces provided by this layer is completely arbitrary and must only be made to match those expected by the HIL and HPL layers that it connects to.
In order to hide these implementation details, and only expose those interfaces that the HPL, HIL, or an application would be interested in, a configuration file for the HAL layer MUST be created that does this.
An example configuration file for the HAL implementation file given given above is shown below:
configuration HALXXXRadioC { provides { interface SplitControl; interface HALXXXRadioControl; interface HALXXXRadioData; interface HPLXXXBusComm; } } implementation { components BusArbitrationC, BusInterfaceC, TimerC, HALXXXRadioM, HPLXXXRadioM; SplitControl = HALXXXRadioC; HALXXXRadioControl = HALXXXRadioM; HALXXXRadioControl = HALXXXRadioM; HPLXXXBusComm = HALXXXRadioM; HALXXXRadioM.HALRadioData -> HPLXXXRadioC; HALXXXRadioM.HALRadioControl -> HPLXXXRadioC; HALXXXRadioM.BusInterface -> BusInterfaceC; HALXXXRadioM.BusArbitration -> BusArbitration; HALXXXRadioM.TimerJiffy -> TimerC.TimerJiffy[unique("TimerJiffy")]; }
As stated before, the interfaces provided by the HAL layer for use by the HIL layer or other applications is highly radio dependent and the files shown above are only examples of how such a layer could be built.
It is important to remember though, that only this layer is allowed to connect to any interfaces that are microcontroller (or platform) specific. This layer should be able to simply connect to the HIL and HPL layers after it has been implemented, and all three layers should spring to action for a given platform.
This layer lives in the platform directory of the platform wishing to interface with the given radio.
Note
In principle all microcontroller specific implementations should also be kept in their own "chips" directory just like the radio implementations. If this is done correctly, then files similar to this HAL file for the radio will be the only ones that exist in a platform directory (Other than files such as hardware.h, AM.h, etc..) Their purpose will be to simply provide the code that allows communication between the various hardware chips on a given platform's sensor node to take place.
This layer is used to provide a platform independent interface to the radio hardware. Such an interface does not yet exist, but it is plausible that one could be conceived of in the future. This layer will use the platform DEPENDENT interfaces implemented by the HAL layer to implement the functionality required for providing the platform INDEPENDENT ones. Certain functionality is provided by ALL radios and these functions should be defined in an interface somewhere.
It is not worth speculating which commands and events would be present in these interfaces, but it is worth mentioning showing how such an implementation would be constructed.
Below is an example of what the HIL implmentation file might look like:
module HILXXXRadioM { provides { interface SplitControl; interface RadioControl; //Control/Config Communication interface RadioData; //Data Communication } uses { interface HALXXXRadioControl; //HPL Control/Config Communication interface HALXXXRadioData; //HPL Data Communication } } implementation { ... }
The RadioControl and RadioData interfaces are just examples of what the platform independent interfaces COULD be called. They do not imply that any sort of final decision on HIL interfaces will look anything like this.
Just as with the HPL layer, however, this layer should be only radio dependent and not microcontroller dependent. This means that this file will exist in the radios "chips" directory and will only be implemented once for any given radio.
The radio can be used with either its platform independent interfaces or its platform dependent ones. Applications must decide which interfaces are suitable for their purposes, and simply wire to one set of interfaces or the other through the configuration files described below.
For applications that wish to use the platform INDEPENDENT interfaces of the radio, the configuration file will look something like this:
configuration XXXRadioC { provides { interface SplitControl; interface RadioControl; interface RadioData; } implementation { components HILXXXRadioM, HALXXXRadioC; SplitControl = HILXXXRadioM; SplitControl = HALXXXRadioC; RadioData = HILXXXRadioM.RadioData; RadioControl = HILXXXRadioM.RadioControl; HILXXXRadioM.HALRadioData -> HALXXXRadioC; HILXXXRadioM.HALRadioControl -> HALXXXRadioC; }
The names and number of the intermediary interfaces connecting the HAL to the HPL and HIL files will vary from radio to radio, but the basic structure for any such configuration file will be the same.
Since the intermediary interfaces are completely radio dependent and not microcontroller dependent, this file can be created once, and placed in the radio's "chips" directory along with all of the interface files and the HPL/HIL implementation files.
For applications that wish to use the platform DEPENDENT interfaces of the radio, the configuration file created for the HAL layer of the radio in the platform directory can be used directly.
Note
As of now, all applications will need to link to the HAL (platform dependent) configuration, because the HIL interfaces do not yet exist for radio abstraction.
Radios in TinyOS can be divided into three layers of abstraction. The lowest layer and highest layers can be made to be solely radio depenedent and not microcontroller dependent, while the middle layer is made to be both radio dependent and microcontroller dependent.
Having such a configuration allows radio implementations to be done for the HPL and HIL layers once, and simply recreating the HAL layer each time a new micrcontroller (platform) is introduced.
With such a configuration, applications have the freedom to choose whether they would like to connect to radio specific interfaces or platform independent interfaces of the radio. While these platform independent ones do not exist yet, work is being done to define them.
The reference use of this document is TEP 1, TEP 2, and TEP 105
NONE


Once all of the required annotations are provided, the GUI displays a quick review screen before proceeding to copy the metadata across all report files. Any warnings generated by the automatic form validation that occurs at each step will be displayed here. If multiple files are being converted at once, the total number of input files will also be shown. Users can add additional experiment-level CV terms that would reflect the experiment as a whole but are not appropriate in any of the previous sections. The DAOs and PRIDE Converter 2 itself will automatically add params for ‘XML generation software’ and ‘Original MS data file format’.
Updating the report file (if performing a single conversion) or copying the data across all report files (if converting multiple files) can take anywhere between seconds and several minutes and is directly proportional to the number of files that need to be updated.


About Xin Song
biography about Xin Song
Xin Song is a graduate student at Stevens Institute of Technology. She majors in Computer Science . Her student ID is 10431021. She came to the United State last fall.
Before that, she lived in China. She was born in the city of JingBian in Shaanxi. She got a bachelor's degree in China. The four-year-long study laid a solid foundation in science for her. She decides to pursue a career in Computer science in the US.
favorite TV Shows
- Prison Break
- Friends
- The Vampire Diaries
- The Ellen DeGeneres Show
- Daily Show with Jon Stewart
hobbies
- Travel
- Watching movies
- Listening to music
- Writing
- Swimming


Introduction
This topic explains how to deploy Unravel Server 4.4.X on AWS EC2 instance that will be used to monitor existing or newly created EMR clusters. For instructions that correspond to older versions of Unravel Server, contact Unravel Support.
- Setup the EC2 instance for unravel installation.
- Install the Unravel EMR RPM on the created EC2 instance.
- Log into Unravel Web UI.
Requirements Checklist
| Platform Compatibi lity | Base OS for EC2 | EC2 instance type & size | Security Group / IAM role |
|---|---|---|---|
:sup:`AWS EMR 5.x (5.14 preferred ) ` EMR Support Matrix Hiv e 0.9.x 2.3.x Spa rk 1.3.x, 1.6.x - 2.3.x |
Base OS Redhat/CentOS 6.4 - 7.4 Recomme nded Centos 7.4 AMI e.g. a mi-02e98f78 (in us-east-1) |
Minimum: R4.2xlarge (60 GB Ram) :sup:`Recommen ded: R4.4xlarge (122 GB Ram) ` Virtuali zation type: HVM Root device type: EBS – 100 - 500GB |
Allowed ports
for inbound access
to unravel EC2
node:
Port 3000
Port 4043
EC2 instance profile that has permission to Read the S3 bucket used in the EMR clusters. |
1. Setup the EC2 instance for unravel installation.
Provision an EC2 Instance
Instance Type: r4.2xlarge; recommended r4.4xlarge; the instance type: HVM with EBS.[
OS: centos7.4 (ami-02e98f78)
Minimum root disk space is 100GB. Recommendation: 200GB or above with “Provisioned IOPS SSD (IO1)” storage.
The EC2 instance must be in same region with the target EMR clusters which Unravel EC2 node will be monitoring.
Create a S3 ReadAccess only IAM role and assign it to Unravel EC2 node to READ the archive logs on the S3 bucket configured for the EMR cluster.
For instance, create an IAM role that contains the policy that only READ the specific S3 bucket used on EMR cluster. Then create a EC2 instance profile and add the IAM role to it. SeeCreating IAM RoleandCreate Instance Profile CLI
- Security Group or policy required for Unravel EC2 node:Unravel EC2 node works with multiple EMR clusters, including existing and newly created clusters. A TCP & UDP connection is needed from the EMR master node to Unravel EC2 node. You must
- create a security group that allows port 3000 and port 4043 from EMR cluster nodes IP address, or
- put the member of security group used on EMR cluster in this rule.
| Type | P r o t o c o l | Po rt Ra ng e | Source |
|---|---|---|---|
| All traffic | A l l | Al l | SG ID of this group or Subnet IP block (e.g. <IP_ADDRESS>/16) |
| SSH | T C P | 22 | <IP_ADDRESS>/0 |
| Custom TCP Rule | T C P | 30 00 | SG ID used on EMR cluster or Subnet IP block (if IP block belong to different VPC) is required for VPC peering connection |
| Custom TCP Rule | T C P | 40 43 | SG ID used on EMR cluster or Subnet IP block (if IP block belong to different VPC) is required for VPC peering connection |
Outbound rule
| Type | P r o t o c o l | Po rt Ra ng e | Source |
|---|---|---|---|
| All traffic | A l l | Al l | <IP_ADDRESS>/0 |
Configure the EC2 node at first login
Disable
selinux
.Edit the file to make sure the setting persists after reboot and make sure “SELINUX=permissive”.
Install
libaio.x86_64 and lzop.x86_64.
Start
ntp
and check timeCreate a new user
hadoop
.confluence-information-macro confluence-information-macro-information
In a PoC or evaluation, the minimal root disk size 100GB should be sufficient. When monitoring more EMR clusters or lots of jobs, we recommend to set minimal root disk from 300 - 500GB “Provisioned IOPS” EBS volume with 3000 IOPS.
For production unravel use case, 200GB root disk Provisioned IOPS EBS and RDS are recommended
2. Install the Unravel EMR rpm on the created EC2 instance.
confluence-information-macro has-no-icon confluence-information-macro-warning
Reminder
AWS EMR 5.x support is available via Unravel <IP_ADDRESS> only
Get the Unravel Server RPM; see Download Unravel Software. Look for <IP_ADDRESS> EMR rpm download instruction.
Install the Unravel Server RPM. The precise filename can vary, depending on how it was fetched or copied.
Run the specified
await_fixups.sh
script to make sure background processing is finished before going on to further steps. In a routine upgrade, it is okay to start all Unravel daemons, but do not stop or restart them until theawait_fixups.sh
prints “Done"
. This may take a few minutes.Edit the
/usr/local/unravel/etc/unravel.properties
file to append the following line in it.For monitoring EMR spark service, add the following properties in the
unravel.properties
file.confluence-information-macro confluence-information-macro-information
The installation creates/usr/local/unravel/
which contains the executables, scripts, and settings. Userunravel
is created. The initial internal database and other durable states are put in/srv/unravel/
for larger storage.The master configuration file is in/usr/local/unravel/etc/unravel.properties
and the logs are in/usr/local/unravel/logs/
. The RPM installation creates userunravel (
if not already existing),/etc/init.d/unravel_*
scripts for controlling services, and/etc/init.d/unravel_all.sh
which can be used to manually stop, start, and get status of all daemons in proper order.During initial install, a bundled database is used; you can switch to use AWS RDS for production.confluence-information-macro confluence-information-macro-warning
Security Remind
We do not recommend making Unravel Server UI accessible on the public Internet.
3. Login to Unravel Web UI
Start Unravel daemons.
Create a
ssh
tunnel from your workstation to the Unravel EC2 node.From your workstation browser, navigate to URL http://<IP_ADDRESS>:3000. Login to the unravel UI with user id:
admin
and password:unraveldata.
* *Congratulations! Unravel EC2 node is up and running. You can proceed to connect to your existing or new EMR clusters.* *
Attachments:


BRIT HOTELS
Hotels
CONTACT US
BOOK NOW
menu
BRIT HOTELS
BRIT HOTELS
menu
Hotels
About Us
Book Now
News & Offers
Gallery
Contact Us
back to top
Welcome to London and enjoy
affordable luxury.
about us
Page under construction


Danity - DK3
Visit Danity Kane's Website
Release Date - 2014
Danity Kane Members
Aundrea Fimbres - Vocals
Dawn Richard- Vocals
Aubrey O'Day - Vocals
Tracks
Track #
Track Title
Track Length
1
Lemonade
4:24
2
Rhythm Of Love
3:36
3
All In A Day's Work
3:14
4
Rage
3:26


javanalysis
/
io.disassemble.javanalysis.flow
/
ControlFlowGraph
/
dominatorTree
dominatorTree
val
dominatorTree
:
Array
<
Node
>
Retrieves the dominator tree of this
ControlFlowGraph
.


"I Wasn't Ready" Season 1 Ep. 1
"Bora Bora Bora" Season 1 Ep. 10
"We Have Manners. We're Polite." Season 2 Ep. 13
"Bed Bugs and Beyond" Season 3 Ep. 2
"We Can Be Heroes" Season 3 Ep. 11
"Doctor Psycho" Season 4 Ep. 4


Description : A Short survey to gather journalist feedback
Questions : {{ surveyData.questions_details.length}} questions
Total Responses : {{ submittedResponses.submissions.length }} submissions
You currently do not have a survey on the system. and start surveying.


The workshop was a success! Contact Kimberly Collins at kcollins at techlahoma dot org to get information about the next event.
An optional InstallFest was Friday, August 3, 2018, from 5:30 pm to 7:30 pm.
The workshop was Saturday, August 4, 2018, from 9:30 am to 5:30 pm.
We'll be in the Will Rogers Meeting Room and the lobby at the Residence Inn Oklahoma City Airport at 804 South Meridian Avenue, Oklahoma City, OK
We'll be using the #djangogirls channel in Techlahoma's Slack to share information about the workshop. Here's how to join us:
More information about Slack:
It's extremely important to us that participants feel safe and respected at our events. Techlahoma's Code of Conduct applies in person and in Slack!
Each coach will work with 2-3 students.
I am a physics/history student looking to teach high school physics when I graduate. I'm 3 credits away from an associates in Cyber Forensics, and I enjoy tinkering with technology. I can program at a rudimentary level in C and I am experienced with BASH.
I am a software developer interested in things that make people's life better. I've been in the industry for 4 years and worked on a variety of software stacks, mostly in the web application space. I enjoy teaching, mentoring, learning, and discussing all things tech.
I'm a developer for the University of Oklahoma Libraries, where I work on web infrastructure and content management projects.
I write web code at Mozilla. I help organize some local tech communities. I owe all my best experiences to the open-source community and try to give back.
I am a Data Scientist with Exaptive, an OKC headquartered Data Applications company. I have worked in Data Analytics of one flavor or another for the past 9 years, mostly in Python and R. My professional specialty is Graph Structure Machine Learning and Natural Language Processing.
I'm a recent computer science graduate from Oklahoma State. I work as a Junior Web Developer at the OSU Foundation.
I am a python enthusiast, a hobbyist programmer, and a lover of all things technology. I work at the University of Oklahoma in the Parking Services office and use python almost every day to aggregate data or script away monotonous tasks. With only about 6 months of Python under my belt I'm definitely still green, but I absolutely love every day that I am able to learn something new by using it.
Aerospace Engineering student at OU. Love getting involved into young student outreach for STEM. Experience with C, C++, Matlab, and Python.
Developer, Architect, and Leader in Software Development for 25+ years. Currently focusing on Secure Software development and protecting code in the cloud.
My name is Fernando. I currently work as an IT Specialist, and I'm trying to break into the world of development. I have gone to the community college for computer science, but I haven't had the opportunity to finish my degree. I love to cook and love dogs.


You don't have to do much to get an attractive plot. Create a placeholder, make sure it has dimensions (so Flot knows at what size to draw the plot), then call the plot function with your data.
The axes are automatically scaled.
You don't have to do much to get an attractive plot. Create a placeholder, make sure it has dimensions (so Flot knows at what size to draw the plot), then call the plot function with your data.
The axes are automatically scaled.


- getAllConfigInfo() - Method in class utils.SupervisordClient
-
Undocumented in current supervisord API
- getAllProcessInfo() - Method in class utils.SupervisordClient
-
Get info about all processes
- getAPIVersion() - Method in class utils.SupervisordClient
-
Return the version of the RPC API used by supervisord
This API is versioned separately from Supervisor itself.
- getBytes() - Method in class utils.TailLog
-
Get tail log output
- getDescription() - Method in class utils.StartStopStatus
-
Get status description
- getExitstatus() - Method in class utils.ProcessInfo
-
Get process exit status
- getGroup() - Method in class utils.ProcessInfo
-
Get process group name
- getGroup() - Method in class utils.StartStopStatus
-
Get status process group name
- getIdentification() - Method in class utils.SupervisordClient
-
Return identifiying string of supervisord
- getName() - Method in class utils.ProcessInfo
-
Get process name
- getName() - Method in class utils.StartStopStatus
-
Get status process name
- getName() - Method in class utils.SupervisorProcess
-
Get process name
- getNow() - Method in class utils.ProcessInfo
-
Get process current host unix timestamp
- getOffset() - Method in class utils.TailLog
-
get offset to start reading from for next call
- getPid() - Method in class utils.ProcessInfo
-
Get process pid
- getPID() - Method in class utils.SupervisordClient
-
Return the PID of supervisord
- getProcessInfo(String) - Method in class utils.SupervisordClient
-
Get info about a process named name as a ProcessInfo object
- getProcessInfo() - Method in class utils.SupervisorProcess
-
Get ProcessInfo object of the process
- getSerializer(XmlRpcStreamConfig, Object) - Method in class utils.MyTypeFactory
-
- getSpawnerr() - Method in class utils.ProcessInfo
-
Get process spawnerr
- getStart() - Method in class utils.ProcessInfo
-
Get process start unix timestamp
- getStartDate() - Method in class utils.SupervisorProcess
-
Get last start date of the process
- getState() - Method in class utils.ProcessInfo
-
Get process state
- getState() - Method in class utils.SupervisordClient
-
Return current state of supervisord as a SupervisorState object
- getState() - Method in class utils.SupervisorProcess
-
Get state name of the process (RUNNING,STOPPED,...)
- getStatename() - Method in class utils.ProcessInfo
-
Get process state name
- getStatus() - Method in class utils.StartStopStatus
-
Get status value
- getStderr_logfile() - Method in class utils.ProcessInfo
-
Get process error log file location
- getStdout_logfile() - Method in class utils.ProcessInfo
-
Get process log file location
- getStop() - Method in class utils.ProcessInfo
-
Get process stop unix timestamp
- getStopDate() - Method in class utils.SupervisorProcess
-
Get stop date of the process
- getSupervisor() - Method in class utils.SupervisorProcess
-
Get Supervisor model object linked to this process
- getSupervisorVersion() - Method in class utils.SupervisordClient
-
Return the version of the supervisor package in use by supervisord
- getUsername(Http.Context) - Method in class controllers.Secured
-
- Global - Class in <Unnamed>
-
- Global() - Constructor for class Global
-
- Group - Class in models
-
Group entity managed by Ebean
- Group() - Constructor for class models.Group
-
- group - Variable in class models.Supervisor
-
- groups(int, String, String) - Static method in class controllers.Admin
-
Show group list and action links to create, modify...


|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object ussr.model.Entity ussr.model.Actuator
public class Actuator
An abstract actuator for a modular robot: can actuate two module components relative to each other. Has sensors to sense its state, and embeds its own controller.
| Constructor Summary | |
|---|---|
Actuator(PhysicsActuator actuator)
Construct a new actuator |
| Method Summary | |
|---|---|
boolean |
activate(float goalValue)
Activate this actuator - to go for a particular goal value |
void |
disactivate()
Stop the actuator |
float |
getEncoderValue()
Get the encoder value of this actuator |
java.util.List<? extends PhysicsEntity> |
getPhysics()
Get the physics entities representing this entity |
boolean |
isActive()
Returns true if this actuator is active |
void |
reset()
|
java.lang.String |
toString()
For debugging |
| Methods inherited from class ussr.model.Entity |
|---|
getProperty, readLabels, setProperty, waitForPropertyToExist |
| Methods inherited from class java.lang.Object |
|---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait |
| Constructor Detail |
|---|
public Actuator(PhysicsActuator actuator)
actuator
- the physics model for the actuator| Method Detail |
|---|
public boolean isActive()
public boolean activate(float goalValue)
public void disactivate()
public java.lang.String toString()
toString
in class java.lang.Object
public java.util.List<? extends PhysicsEntity> getPhysics()
Entity
getPhysics
in class Entity
public float getEncoderValue()
public void reset()
|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |