Our goal is to work through examples that demonstrate how to explore, process and manipulate genomic interval files (e.g., BED, VCF, BAM) with the bedtools
software package.
Some of our analysis will be based upon the Maurano et al exploration of DnaseI hypersensitivity sites in hundreds of primary tissue types.
Maurano et al. Systematic Localization of Common Disease-Associated Variation in Regulatory DNA. Science. 2012. Vol. 337 no. 6099 pp. 1190-1195.
www.sciencemag.org/content/337/6099/1190.short
This tutorial is merely meant as an introduction to whet your appetite. There are many, many more tools and options than presented here. We therefore encourage you to read the bedtools documentation.
From the Terminal, create a new directory on your Desktop called bedtools-demo
(it doesn’t really matter where you create this directory).
mkdir -p ~/workspace/monday/bedtools
Navigate into that directory.
cd ~/workspace/monday/bedtools
Download the sample BED files I have provided.
curl -O https://s3.amazonaws.com/bedtools-tutorials/web/maurano.dnaseI.tgz
curl -O https://s3.amazonaws.com/bedtools-tutorials/web/cpg.bed
curl -O https://s3.amazonaws.com/bedtools-tutorials/web/exons.bed
curl -O https://s3.amazonaws.com/bedtools-tutorials/web/gwas.bed
curl -O https://s3.amazonaws.com/bedtools-tutorials/web/genome.txt
curl -O https://s3.amazonaws.com/bedtools-tutorials/web/hesc.chromHmm.bed
Now, we need to extract all of the 20 Dnase I hypersensitivity BED files from the “tarball” named maurano.dnaseI.tgz
.
tar -zxvf maurano.dnaseI.tgz
rm maurano.dnaseI.tgz
Let’s take a look at what files we now have.
ls -1
Your directory should now contain 23 BED files and 1 genome file. Twenty of these files (those starting with “f” for “fetal tissue”) reflect Dnase I hypersensitivity sites measured in twenty different fetal tissue samples from the brain, heart, intestine, kidney, lung, muscle, skin, and stomach.
In addition: cpg.bed
represents CpG islands in the human genome; exons.bed
represents RefSeq exons from human genes; gwas.bed
represents human disease-associated SNPs that were identified in genome-wide association studies (GWAS); hesc.chromHmm.bed
represents the predicted function (by chromHMM) of each interval in the genome of a human embryonic stem cell based upon ChIP-seq experiments from ENCODE.
The latter 4 files were extracted from the UCSC Genome Browser’s Table Browser.
In order to have a rough sense of the data, let’s load the cpg.bed
, exons.bed
, gwas.bed
, and hesc.chromHmm.bed
files into IGV. To do this, launch IGV, then click File->Load from File. Then select the four files. IGV will warn you that you need to create an index for a couple of the files. Just click OK, as these indices are created automatically and speed up the processing for IGV.
Visualization in IGV or other browsers such as UCSC is a tremendously useful way to make sure that your results make sense to your eye. Conveniently, a subset of bedtools is built-into IGV!
Bedtools is a command-line tool. To bring up the help, just type
bedtools
As you can see, there are multiple “subcommands” and for bedtools to work you must tell it which subcommand you want to use. Examples:
bedtools intersect
bedtools merge
bedtools subtract
What version am I using?
bedtools --version
How can I get more help?
bedtools --contact
The intersect
command is the workhorse of the bedtools
suite. It compares two or more BED/BAM/VCF/GFF files and identifies all the regions in the gemome where the features in the two files overlap (that is, share at least one base pair in common).
By default, intersect
reports the intervals that represent overlaps between your two files. To demonstrate, let’s identify all of the CpG islands that overlap exons.
bedtools intersect -a cpg.bed -b exons.bed | head -5
chr1 29320 29370 CpG:_116
chr1<PHONE_NUMBER>63 CpG:_30
chr1<PHONE_NUMBER>29 CpG:_29
chr1<PHONE_NUMBER>29 CpG:_29
chr1<PHONE_NUMBER>29 CpG:_29
The -wa
(write A) and -wb
(write B) options allow one to see the original records from the A and B files that overlapped. As such, instead of not only showing you where the intersections occurred, it shows you what intersected.
bedtools intersect -a cpg.bed -b exons.bed -wa -wb \
| head -5
chr1 28735 29810 CpG:_116 chr1 29320 29370 NR_024540_exon_10_0_chr1_29321_r -
chr1<PHONE_NUMBER>63 CpG:_30 chr1<PHONE_NUMBER>96 NR_039983_exon_0_0_chr1_134773_r 0 -
chr1<PHONE_NUMBER>29 CpG:_29 chr1<PHONE_NUMBER>81 NR_028322_exon_2_0_chr1_324439_f 0 +
chr1<PHONE_NUMBER>29 CpG:_29 chr1<PHONE_NUMBER>81 NR_028325_exon_2_0_chr1_324439_f 0 +
chr1<PHONE_NUMBER>29 CpG:_29 chr1<PHONE_NUMBER>81 NR_028327_exon_3_0_chr1_327036_f 0 +
The -wo
(write overlap) option allows one to also report the number of base pairs of overlap between the features that overlap between each of the files.
bedtools intersect -a cpg.bed -b exons.bed -wo \
| head -10
chr1 28735 29810 CpG:_116 chr1 29320 29370 NR_024540_exon_10_0_chr1_29321_r - 50
chr1<PHONE_NUMBER>63 CpG:_30 chr1<PHONE_NUMBER>96 NR_039983_exon_0_0_chr1_134773_r 0 439
chr1<PHONE_NUMBER>29 CpG:_29 chr1<PHONE_NUMBER>81 NR_028322_exon_2_0_chr1_324439_f 0 439
chr1<PHONE_NUMBER>29 CpG:_29 chr1<PHONE_NUMBER>81 NR_028325_exon_2_0_chr1_324439_f 0 439
chr1<PHONE_NUMBER>29 CpG:_29 chr1<PHONE_NUMBER>81 NR_028327_exon_3_0_chr1_327036_f 0 439
chr1<PHONE_NUMBER>47 CpG:_60 chr1<PHONE_NUMBER>68 NR_033908_exon_6_0_chr1_713664_r 0 84
chr1<PHONE_NUMBER>45 CpG:_115 chr1<PHONE_NUMBER>02 NR_024321_exon_0_0_chr1_761586_r - 486
chr1<PHONE_NUMBER>45 CpG:_115 chr1<PHONE_NUMBER>55 NR_015368_exon_0_0_chr1_762971_f + 185
chr1<PHONE_NUMBER>45 CpG:_115 chr1<PHONE_NUMBER>55 NR_047519_exon_0_0_chr1_762971_f + 185
chr1<PHONE_NUMBER>45 CpG:_115 chr1<PHONE_NUMBER>55 NR_047520_exon_0_0_chr1_762971_f + 185
We can also count, for each feature in the “A” file, the number of overlapping features in the “B” file. This is handled with the -c
option.
bedtools intersect -a cpg.bed -b exons.bed -c \
| head
chr1 28735 29810 CpG:_116 1
chr1<PHONE_NUMBER>63 CpG:_30 1
chr1<PHONE_NUMBER>29 CpG:_29 3
chr1<PHONE_NUMBER>64 CpG:_84 0
chr1<PHONE_NUMBER>44 CpG:_99 0
chr1<PHONE_NUMBER>14 CpG:_94 0
chr1<PHONE_NUMBER>49 CpG:_171 0
chr1<PHONE_NUMBER>47 CpG:_60 1
chr1<PHONE_NUMBER>45 CpG:_115 10
chr1<PHONE_NUMBER>11 CpG:_28 9
Often we want to identify those features in our A file that do not overlap features in the B file. The -v
option is your friend in this case.
bedtools intersect -a cpg.bed -b exons.bed -v \
| head
chr1<PHONE_NUMBER>64 CpG:_84
chr1<PHONE_NUMBER>44 CpG:_99
chr1<PHONE_NUMBER>14 CpG:_94
chr1<PHONE_NUMBER>49 CpG:_171
chr1<PHONE_NUMBER>38 CpG:_24
chr1<PHONE_NUMBER>28 CpG:_50
chr1<PHONE_NUMBER>19 CpG:_83
chr1<PHONE_NUMBER>83 CpG:_153
chr1<PHONE_NUMBER>53 CpG:_28
chr1<PHONE_NUMBER>27 CpG:_15
Recall that the default is to report overlaps between features in A and B so long as at least one basepair of overlap exists. However, the -f
option allows you to specify what fraction of each feature in A should be overlapped by a feature in B before it is reported.
Let’s be more strict and require 50% of overlap.
bedtools intersect -a cpg.bed -b exons.bed \
-wo -f 0.50 \
| head
chr1<PHONE_NUMBER>63 CpG:_30 chr1<PHONE_NUMBER>96 NR_039983_exon_0_0_chr1_134773_r 0 439
chr1<PHONE_NUMBER>29 CpG:_29 chr1<PHONE_NUMBER>81 NR_028322_exon_2_0_chr1_324439_f 0 439
chr1<PHONE_NUMBER>29 CpG:_29 chr1<PHONE_NUMBER>81 NR_028325_exon_2_0_chr1_324439_f 0 439
chr1<PHONE_NUMBER>29 CpG:_29 chr1<PHONE_NUMBER>81 NR_028327_exon_3_0_chr1_327036_f 0 439
chr1<PHONE_NUMBER>11 CpG:_28 chr1<PHONE_NUMBER>26 NR_047519_exon_5_0_chr1_788771_f 0 348
chr1<PHONE_NUMBER>11 CpG:_28 chr1<PHONE_NUMBER>26 NR_047521_exon_4_0_chr1_788771_f 0 348
chr1<PHONE_NUMBER>11 CpG:_28 chr1<PHONE_NUMBER>26 NR_047523_exon_3_0_chr1_788771_f 0 348
chr1<PHONE_NUMBER>11 CpG:_28 chr1<PHONE_NUMBER>26 NR_047524_exon_3_0_chr1_788771_f 0 348
chr1<PHONE_NUMBER>11 CpG:_28 chr1<PHONE_NUMBER>26 NR_047525_exon_4_0_chr1_788771_f 0 348
chr1<PHONE_NUMBER>11 CpG:_28 chr1<PHONE_NUMBER>26 NR_047520_exon_6_0_chr1_788859_f 0 348
So far the examples presented have used the traditional algorithm in bedtools for finding intersections. It turns out, however, that bedtools is much faster when using presorted data.
For example, compare the difference in speed between the two approaches when finding intersections between exons.bed
and hesc.chromHmm.bed
:
time bedtools intersect -a gwas.bed -b hesc.chromHmm.bed > /dev/null
1.10s user 0.11s system 99% cpu 1.206 total
time bedtools intersect -a gwas.bed -b hesc.chromHmm.bed -sorted > /dev/null
0.36s user 0.01s system 99% cpu 0.368 total
-sorted
option groqw as datasets grow larger. For example, compare the runtimes of the sorted and unsorted approaches as a function of dataset size in the figure below. The important thing to remember is that each dataset must be sorted by chromosome and then by start position: sort -k1,1 -k2,2n
.-
As of version 2.21.0, bedtools is able to intersect an “A” file against one or more “B” files. This greatly simplifies analyses involving multiple datasets relevant to a given experiment. For example, let’s intersect exons with CpG islands, GWAS SNPs, an the ChromHMM annotations.
bedtools intersect -a exons.bed -b cpg.bed gwas.bed hesc.chromHmm.bed -sorted | head
chr1 11873 11937 NR_046018_exon_0_0_chr1_11874_f 0 +
chr1 11937 12137 NR_046018_exon_0_0_chr1_11874_f 0 +
chr1 12137 12227 NR_046018_exon_0_0_chr1_11874_f 0 +
chr1 12612 12721 NR_046018_exon_1_0_chr1_12613_f 0 +
chr1 13220 14137 NR_046018_exon_2_0_chr1_13221_f 0 +
chr1 14137 14409 NR_046018_exon_2_0_chr1_13221_f 0 +
chr1 14361 14829 NR_024540_exon_0_0_chr1_14362_r 0 -
chr1 14969 15038 NR_024540_exon_1_0_chr1_14970_r 0 -
chr1 15795 15947 NR_024540_exon_2_0_chr1_15796_r 0 -
chr1 16606 16765 NR_024540_exon_3_0_chr1_16607_r 0 -
Now by default, this isn’t incredibly informative as we can’t tell which of the three “B” files yielded the intersection with each exon. However, if we use the -wa
and wb
options, we can see from which file number (following the order of the files given on the command line) the intersection came. In this case, the 7th column reflects this file number.
bedtools intersect -a exons.bed -b cpg.bed gwas.bed hesc.chromHmm.bed -sorted -wa -wb \
| head -10000 \
| tail -10
chr1 27632676 27635124 NM_001276252_exon_15_0_chr1_27632677_f 0 + 3 chr1 27633213 27635013 5_Strong_Enhancer
chr1 27632676 27635124 NM_001276252_exon_15_0_chr1_27632677_f 0 + 3 chr1 27635013 27635413 7_Weak_Enhancer
chr1 27632676 27635124 NM_015023_exon_15_0_chr1_27632677_f 0 + 3 chr1 27632613 27632813 6_Weak_Enhancer
chr1 27632676 27635124 NM_015023_exon_15_0_chr1_27632677_f 0 + 3 chr1 27632813 27633213 7_Weak_Enhancer
chr1 27632676 27635124 NM_015023_exon_15_0_chr1_27632677_f 0 + 3 chr1 27633213 27635013 5_Strong_Enhancer
chr1 27632676 27635124 NM_015023_exon_15_0_chr1_27632677_f 0 + 3 chr1 27635013 27635413 7_Weak_Enhancer
chr1 27648635 27648882 NM_032125_exon_0_0_chr1_27648636_f 0 + 1 chr1 27648453 27649006 CpG:_63
chr1 27648635 27648882 NM_032125_exon_0_0_chr1_27648636_f 0 + 3 chr1 27648613 27649413 1_Active_Promoter
chr1 27648635 27648882 NR_037576_exon_0_0_chr1_27648636_f 0 + 1 chr1 27648453 27649006 CpG:_63
chr1 27648635 27648882 NR_037576_exon_0_0_chr1_27648636_f 0 + 3 chr1 27648613 27649413 1_Active_Promoter
Additionally, one can use file “labels” instead of file numbers to facilitate interpretation, especially when there are many files involved.
bedtools intersect -a exons.bed -b cpg.bed gwas.bed hesc.chromHmm.bed -sorted -wa -wb -names cpg gwas chromhmm \
| head -10000 \
| tail -10
chr1 27632676 27635124 NM_001276252_exon_15_0_chr1_27632677_f 0 + chromhmm chr1 27633213 27635013 5_Strong_Enhancer
chr1 27632676 27635124 NM_001276252_exon_15_0_chr1_27632677_f 0 + chromhmm chr1 27635013 27635413 7_Weak_Enhancer
chr1 27632676 27635124 NM_015023_exon_15_0_chr1_27632677_f 0 + chromhmm chr1 27632613 27632813 6_Weak_Enhancer
chr1 27632676 27635124 NM_015023_exon_15_0_chr1_27632677_f 0 + chromhmm chr1 27632813 27633213 7_Weak_Enhancer
chr1 27632676 27635124 NM_015023_exon_15_0_chr1_27632677_f 0 + chromhmm chr1 27633213 27635013 5_Strong_Enhancer
chr1 27632676 27635124 NM_015023_exon_15_0_chr1_27632677_f 0 + chromhmm chr1 27635013 27635413 7_Weak_Enhancer
chr1 27648635 27648882 NM_032125_exon_0_0_chr1_27648636_f 0 + cpg chr1 27648453 27649006 CpG:_63
chr1 27648635 27648882 NM_032125_exon_0_0_chr1_27648636_f 0 + chromhmm chr1 27648613 27649413 1_Active_Promoter
chr1 27648635 27648882 NR_037576_exon_0_0_chr1_27648636_f 0 + cpg chr1 27648453 27649006 CpG:_63
chr1 27648635 27648882 NR_037576_exon_0_0_chr1_27648636_f 0 + chromhmm chr1 27648613 27649413 1_Active_Promoter
Many datasets of genomic features have many individual features that overlap one another (e.g. aligments from a ChiP seq experiment). It is often useful to just cobine the overlapping into a single, contiguous interval. The bedtools merge
command will do this for you.
The merge tool requires that the input file is sorted by chromosome, then by start position. This allows the merging algorithm to work very quickly without requiring any RAM.
If your files are unsorted, the merge
tool will raise an error. To correct this, you need to sort your BED using the UNIX sort
utility. For example:
sort -k1,1 -k2,2n foo.bed > foo.sort.bed
Merging results in a new set of intervals representing the merged set of intervals in the input. That is, if a base pair in the genome is covered by 10 features, it will now only be represented once in the output file.
bedtools merge -i exons.bed | head -n 20
chr1 11873 12227
chr1 12612 12721
chr1 13220 14829
chr1 14969 15038
chr1 15795 15947
chr1 16606 16765
chr1 16857 17055
chr1 17232 17368
chr1 17605 17742
chr1 17914 18061
chr1 18267 18366
chr1 24737 24891
chr1 29320 29370
chr1 34610 35174
chr1 35276 35481
chr1 35720 36081
chr1 69090 70008
chr1<PHONE_NUMBER>96
chr1<PHONE_NUMBER>47
chr1<PHONE_NUMBER>66
A more sophisticated approach would be to not only merge overlapping intervals, but also report the number of intervals that were integrated into the new, merged interval. One does this with the -c
and -o
options. The -c
option allows one to specify a column or columns in the input that you wish to summarize. The -o
option defines the operation(s) that you wish to apply to each column listed for the -c
option. For example, to count the number of overlapping intervals that led to each of the new “merged” intervals, one will “count” the first column (though the second, third, fourth, etc. would work just fine as well).
bedtools merge -i exons.bed -c 1 -o count | head -n 20
chr1 11873 12227 1
chr1 12612 12721 1
chr1 13220 14829 2
chr1 14969 15038 1
chr1 15795 15947 1
chr1 16606 16765 1
chr1 16857 17055 1
chr1 17232 17368 1
chr1 17605 17742 1
chr1 17914 18061 1
chr1 18267 18366 1
chr1 24737 24891 1
chr1 29320 29370 1
chr1 34610 35174 2
chr1 35276 35481 2
chr1 35720 36081 2
chr1 69090 70008 1
chr1<PHONE_NUMBER>96 1
chr1<PHONE_NUMBER>47 1
chr1<PHONE_NUMBER>66 1
With the -d
(distance) option, one can also merge intervals that do not overlap, yet are close to one another. For example, to merge features that are no more than 1000bp apart, one would run:
bedtools merge -i exons.bed -d 1000 -c 1 -o count | head -20
chr1 11873 18366 12
chr1 24737 24891 1
chr1 29320 29370 1
chr1 34610 36081 6
chr1 69090 70008 1
chr1<PHONE_NUMBER>66 3
chr1<PHONE_NUMBER>81 10
chr1<PHONE_NUMBER>97 3
chr1<PHONE_NUMBER>34 3
chr1<PHONE_NUMBER>31 3
chr1<PHONE_NUMBER>27 1
chr1<PHONE_NUMBER>67 1
chr1<PHONE_NUMBER>92 2
chr1<PHONE_NUMBER>87 1
chr1<PHONE_NUMBER>60 1
chr1<PHONE_NUMBER>68 1
chr1<PHONE_NUMBER>14 2
chr1<PHONE_NUMBER>29 10
chr1<PHONE_NUMBER>84 9
chr1<PHONE_NUMBER>84 1
Many times you want to keep track of the details of exactly which intervals were merged. One way to do this is to create a list of the names of each feature. We can do with with the collapse
operation available via the -o
argument. The name of the exon is in the fourth column, so we ask merge
to create a list of the exon names with -c 4 -o collapse
:
bedtools merge -i exons.bed -d 90 -c 1,4 -o count,collapse | head -20
chr1 11873 12227 1 NR_046018_exon_0_0_chr1_11874_f
chr1 12612 12721 1 NR_046018_exon_1_0_chr1_12613_f
chr1 13220 14829 2 NR_046018_exon_2_0_chr1_13221_f,NR_024540_exon_0_0_chr1_14362_r
chr1 14969 15038 1 NR_024540_exon_1_0_chr1_14970_r
chr1 15795 15947 1 NR_024540_exon_2_0_chr1_15796_r
chr1 16606 16765 1 NR_024540_exon_3_0_chr1_16607_r
chr1 16857 17055 1 NR_024540_exon_4_0_chr1_16858_r
chr1 17232 17368 1 NR_024540_exon_5_0_chr1_17233_r
chr1 17605 17742 1 NR_024540_exon_6_0_chr1_17606_r
chr1 17914 18061 1 NR_024540_exon_7_0_chr1_17915_r
chr1 18267 18366 1 NR_024540_exon_8_0_chr1_18268_r
chr1 24737 24891 1 NR_024540_exon_9_0_chr1_24738_r
chr1 29320 29370 1 NR_024540_exon_10_0_chr1_29321_r
chr1 34610 35174 2 NR_026818_exon_0_0_chr1_34611_r,NR_026820_exon_0_0_chr1_34611_r
chr1 35276 35481 2 NR_026818_exon_1_0_chr1_35277_r,NR_026820_exon_1_0_chr1_35277_r
chr1 35720 36081 2 NR_026818_exon_2_0_chr1_35721_r,NR_026820_exon_2_0_chr1_35721_r
chr1 69090 70008 1 NM_001005484_exon_0_0_chr1_69091_f
chr1<PHONE_NUMBER>96 1 NR_039983_exon_0_0_chr1_134773_r
chr1<PHONE_NUMBER>47 1 NR_039983_exon_1_0_chr1_139790_r
chr1<PHONE_NUMBER>66 1 NR_039983_exon_2_0_chr1_140075_r
We often want to know which intervals of the genome are NOT “covered” by intervals in a given feature file. For example, if you have a set of ChIP-seq peaks, you may also want to know which regions of the genome are not bound by the factor you assayed. The complement
addresses this task.
As an example, let’s find all of the non-exonic (i.e., intronic or intergenic) regions of the genome. Note, to do this you need a “genome” file, which tells bedtools
the length of each chromosome in your file. Consider why the tool would need this information…
bedtools complement -i exons.bed -g genome.txt \
> non-exonic.bed
head non-exonic.bed
chr1 0 11873
chr1 12227 12612
chr1 12721 13220
chr1 14829 14969
chr1 15038 15795
chr1 15947 16606
chr1 16765 16857
chr1 17055 17232
chr1 17368 17605
chr1 17742 17914
For many analyses, one wants to measure the genome wide coverage of a feature file. For example, we often want to know what fraction of the genome is covered by 1 feature, 2 features, 3 features, etc. This is frequently crucial when assessing the “uniformity” of coverage from whole-genome sequencing. This is done with the versatile genomecov
tool.
As an example, let’s produce a histogram of coverage of the exons throughout the genome. Like the merge
tool, genomecov
requires pre-sorted data. It also needs a genome file as above.
bedtools genomecov -i exons.bed -g genome.txt
This should run for 3 minutes or so. At the end of your output, you should see something like:
genome 0<PHONE_NUMBER><PHONE_NUMBER> 0.976171
genome 1 44120515<PHONE_NUMBER> 0.0140638
genome 2 15076446<PHONE_NUMBER> 0.00480576
genome 3 7294047<PHONE_NUMBER> 0.00232505
genome 4 3650324<PHONE_NUMBER> 0.00116358
genome 5 1926397<PHONE_NUMBER> 0.000614057
genome 6 1182623<PHONE_NUMBER> 0.000376972
genome 7<PHONE_NUMBER>161264 0.000183
genome 8<PHONE_NUMBER>161264 0.000112634
genome 9<PHONE_NUMBER>161264 4.86596e-05
genome 10<PHONE_NUMBER>161264 3.61352e-05
genome 11 57361<PHONE_NUMBER> 1.82844e-05
genome 12 52000<PHONE_NUMBER> 1.65755e-05
genome 13 55368<PHONE_NUMBER> 1.76491e-05
genome 14 19218<PHONE_NUMBER> 6.12592e-06
genome 15 19369<PHONE_NUMBER> 6.17405e-06
genome 16 26651<PHONE_NUMBER> 8.49526e-06
genome 17 9942<PHONE_NUMBER> 3.16911e-06
genome 18 13442<PHONE_NUMBER> 4.28477e-06
genome 19 1030<PHONE_NUMBER> 3.28322e-07
genome 20 6329<PHONE_NUMBER> 2.01743e-06
...
Using the -bg
option, one can also produce BEDGRAPH output which represents the “depth” fo feature coverage for each base pair in the genome:
bedtools genomecov -i exons.bed -g genome.txt -bg | head -20
chr1 11873 12227 1
chr1 12612 12721 1
chr1 13220 14361 1
chr1 14361 14409 2
chr1 14409 14829 1
chr1 14969 15038 1
chr1 15795 15947 1
chr1 16606 16765 1
chr1 16857 17055 1
chr1 17232 17368 1
chr1 17605 17742 1
chr1 17914 18061 1
chr1 18267 18366 1
chr1 24737 24891 1
chr1 29320 29370 1
chr1 34610 35174 2
chr1 35276 35481 2
chr1 35720 36081 2
chr1 69090 70008 1
chr1<PHONE_NUMBER>96 1
Analytical power in bedtools
comes from the ability to “chain” together multiple tools in order to construct rather sophisicated analyses with very little programming - you just need genome arithmetic! Have a look at the examples here.
Here are a few more examples.
@brent_p bedtools genomecov -ibam aln.bam -bga | awk ‘$4==0’ |
| bedtools intersect -a regions -b - > foo
— Aaron Quinlan (@aaronquinlan) January 10, 2014
We will use the bedtools implementation of a Jaccard statistic to meaure the similarity of two datasets. Briefly, the Jaccard statistic measures the ratio of the number of intersecting base pairs to the total number of base pairs in the two sets. As such, the score ranges from 0.0 to 1. 0; lower values reflect lower similarity, whereas higher values reflect higher similarity.
Let’s walk through an example: we would expect the Dnase hypersensivity sites to be rather similar between two samples of the same fetal tissue type. Let’s test:
bedtools jaccard \
-a fHeart-DS16621.hotspot.twopass.fdr0.05.merge.bed \
-b fHeart-DS15839.hotspot.twopass.fdr0.05.merge.bed
intersection union-intersection jaccard n_intersections
81269248 160493950 0.50637 130852
But what about the similarity of two different tissue types?
bedtools jaccard \
-a fHeart-DS16621.hotspot.twopass.fdr0.05.merge.bed \
-b fSkin_fibro_bicep_R-DS19745.hg19.hotspot.twopass.fdr0.05.merge.bed
intersection union-intersection jaccard n_intersections
28076951 164197278 0.170995 73261
Hopefully this demonstrates how the Jaccard statistic can be used as a simple statistic to reduce the dimensionality of the comparison between two large (e.g., often containing thousands or millions of intervals) feature sets.
We are going to take this a bit further and use the Jaccard statistic to measure the similarity of all 20 tissue samples against all other 20 samples. Once we have a 20x20 matrix of similarities, we can use dimensionality reduction techniques such as hierarchical clustering or principal component analysis to detect higher order similarities among all of the datasets.
We will use GNU parallel to compute a Jaccard statistic for the 400 (20*20) pairwise comparisons among the fetal tissue samples.
But first, we need to install GNU parallel.
brew install parallel
Next, we need to install a tiny script I wrote for this analysis.
curl -O https://s3.amazonaws.com/bedtools-tutorials/web/make-matrix.py
Now, we can use parallel
to, you guessed it, compute the 400 pairwise Jaccard statistics in parallel using as many processors as you have available.
parallel "bedtools jaccard -a {1} -b {2} \
| awk 'NR>1' \
| cut -f 3 \
> {1}.{2}.jaccard" \
::: `ls *.merge.bed` ::: `ls *.merge.bed`
This command will create a single file containing the pairwise Jaccard measurements from all 400 tests.
find . \
| grep jaccard \
| xargs grep "" \
| sed -e s"/\.\///" \
| perl -pi -e "s/.bed./.bed\t/" \
| perl -pi -e "s/.jaccard:/\t/" \
> pairwise.dnase.txt
A bit of cleanup to use more intelligible names for each of the samples.
cat pairwise.dnase.txt \
| sed -e 's/.hotspot.twopass.fdr0.05.merge.bed//g' \
| sed -e 's/.hg19//g' \
> pairwise.dnase.shortnames.txt
Now let’s make a 20x20 matrix of the Jaccard statistic. This will allow the data to play nicely with R.
awk 'NF==3' pairwise.dnase.shortnames.txt \
| awk '$1 ~ /^f/ && $2 ~ /^f/' \
| python make-matrix.py \
> dnase.shortnames.distance.matrix
Let’s also make a file of labels for each dataset so that we can label each dataset in our R plot.
cut -f 1 dnase.shortnames.distance.matrix | cut -f 1 -d "-" | cut -f 1 -d "_" > labels.txt
Now start up R.
R
ggplot2
and RColorBrewer
packages installed on your computer. If they are not installed, run both install.packages("ggplot2")
and install.packages("RColorBrewer")
from the R prompt and respond to the prompts that will follow.-
You should see something very similar to this:
R version 2.15.1 (2012-06-22) -- "Roasted Marshmallows"
Copyright (C) 2012 The R Foundation for Statistical Computing
ISBN 3-900051-07-0
Platform: x86_64-apple-darwin12.0.0 (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
>
No paste these commands into the R console:
library(ggplot2)
library(RColorBrewer)
blues <- colorRampPalette(c('dark blue', 'light blue'))
greens <- colorRampPalette(c('dark green', 'light green'))
reds <- colorRampPalette(c('pink', 'dark red'))
x <- read.table('dnase.shortnames.distance.matrix')
labels <- read.table('labels.txt')
ngroups <- length(unique(labels))
pca <- princomp(x)
pdf('pca.pdf')
qplot(pca$scores[,1], pca$scores[,2], color=labels[,1], geom="point", size=1) +
scale_color_manual(values = c(blues(4), greens(5), reds(5)))
dev.off()
You should see this:
Et voila.
So let’s make a heatmap for giggles.
gplots
package installed on your computer. If it are not installed, run install.packages("gplots")
from the R prompt and respond to the prompts that will follow.-
library(gplots)
library(RColorBrewer)
jaccard_table <- x[, -1]
jaccard_matrix <- as.matrix(jaccard_table)
pdf('heat.pdf')
heatmap.2(jaccard_matrix, col = brewer.pal(9,"Blues"), margins = c(14, 14), density.info = "none", lhei=c(2, 8), trace= "none")
dev.off()
You should see this:
Create a BED file representing all of the intervals in the genome that are NOT exonic.
What is the average distance from GWAS SNPs to the closest exon? (Hint - have a look at the closest tool.)
Count how many exons occur in each 500kb interval (“window”) in the human genome. (Hint - have a look at the makewindows
tool.)
Are there any exons that are completely overlapped by an enhancer? If so, how many?
What fraction of the GWAS SNPs are exonic?
What fraction of the GWAS SNPs are lie in either enhancers or promoters in the hESC data we have?
Create intervals representing the canonical 2bp splice sites on either side of each exon (don’t worry about excluding splice sites at the first or last exon). (Hint - have a look at the flank tool.)
What is the Jaccard statistic between CpG and hESC enhancers? Compare that to the Jaccard statistic between CpG and hESC promoters. Does the result make sense? (Hint - you will need grep
).
What would you expect the Jaccard statistic to look like if promoters were randomly distributed throughout the genome? (Hint - you will need the shuffle tool.)
Which hESC ChromHMM state (e.g., 11_Weak_Txn, 10_Txn_Elongation) represents the most number of base pairs in the genome? (Hint: you will need to use awk
or perl
here, as well as the groupby tool.)


{% block content %}
You can delete or update the database by pressing the right button
The application can freeze for a little while
{% if deleted %}
Deleted all the records from the database!
{% endif %}
{% if correct > 0 %}
Inserted {{ correct }} documents into the database!
{% endif %}
{% if correct == 0 and wrong == 0 %}
No documents added into the database! Database is up-to-date!
{% endif %}
{% if correct == 0 and wrong > 0 %}
No documents added into the database!
{% endif %}
{% if wrong > 0 %}
Received {{ wrong }} wrong responses from NBP!
{% endif %}
{% endblock %}


Users will need to sign up for a free account with Joberdesk so there can have full access to the free content that this site has to offer.
Search jobs that interest you or fit your skills set by using the job name, location, and or the job's type or just browse all jobs.
Candidates should consider uploading their resume unto to this site, only candidates with resumes in their profiles can apply for jobs.
Users who have created their accounts have the ability to post jobs to this site just as many as they may wish to. But it should be noted that all jobs posted by user will be vierified before publishing which takes a minimum of 24 hours .


An electronic dictionary is a dictionary whose data exists in digital form and can be accessed through a number of different media. Electronic dictionaries can be found in several forms, including software installed on tablet or desktop computers, mobile apps, web applications, and as a built-in function of E-readers. As well as Latin script, electronic dictionaries are also available in logographic and right-to-left scripts, including Arabic, Persian, Gurmukhi, Chinese, Devanagari, Greek, Hebrew, Japanese, Korean, Cyrillic, and Thai.
Handheld electronic dictionaries, also known as "pocket electronic dictionaries" or PEDs, resemble miniature clamshell laptop computers, complete with full keyboards and LCD screens. Because they are intended to be fully portable, the dictionaries are battery-powered and made with durable casing material.Many manufacturers produce hand held dictionaries that use licensed dictionary content that use a database such as the Merriam Webster Dictionary and Thesaurus while others may use a proprietary database from their own lexicographers. Many devices can be expanded for several languages with the purchase of additional memory cards.


This is your fallback content in case JavaScript fails to load.
Submit
Messages from all topics
get channels


CHOCOLAT (director: Lasse Hallstrom; screenwriter: Robert Nelson Jacobs/based on the novel by Joanne Harris; cinematographer: Roger Pratt; editor: Andrew Mondshein; cast: Juliette Binoche (Vianne), Johnny Depp (Roux), Judi Dench (Armande Voizin), Alfred Molina (Comte De Reynaud), Lena Olin (Josephine Muscat), Carrie-Anne Moss (Caroline Clairmont), Hugh O'Conor (Père Henri), Peter Stormare (Serge Muscat), Leslie Caron (Mme. Audel), John Wood (Guillaume Blerot), Victoire Thivisol (Anouk); Aurelien Parent Koenig (Luc Clairmont), Runtime: 121; Miramax Films; 2000)
Reviewed by Dennis Schwartz
CHOCOLAT by Lasse Hallstrom is a thoroughly manipulative and indigestible art-house, comedy-drama, fluff film. It sets up a war between sensualists and those repressed by their religious beliefs. Every character is a cliché, everything about the story is predictably smug. It is so easy to make fun of the one-dimensional villains and feel morally superior to them. I grew nauseous from how bourgeois this film was in attitude, where even the rebels are merely lovers of sweet things. I found not much about the film that I enjoyed (though the cinematography was beguilling and the cast was adept). It was a labor just to sit through it and not gag, while waiting for the inevitable preachy sermon that comes at the end, so even the church is redeemed for its unchristian attitude toward strangers. Lasse Hallstrom has come down a few pegs after his more engaging recent films "My Life as a Dog" and "The Cider House Rules," as he couldn't pull this fairy tale off without being tainted by the mediocrity of it all. This film got nominated for an Oscar as best picture, which should show you what kind of films Hollywood falls in love with.
Set in 1959, in a small traditional medieval French village, where the centerpiece of the town is the church and the town's nobleman mayor, Comte De Reynaud (Alfred Molina), who is their guiding force to righteousness. He is the one who makes sure everyone knows their moral place. Tranquility and resistance to change are the only mood swings of this unchanging town. Into this serene but rigid climate, when the north wind was blowing, come two drifters in red-hooded coats, Vianne (Binoche) and her school-aged, illegitimate daughter Anouk (Thivisol). They rent a shabby apartment across from the church from a cranky old woman, Armande Voizin (Judi Dench).
Hoping to win over the locals, Vianne, who is an atheist, opens up a specialty chocolate shop, but unfortunately she opens up during Lent, and from the pulpit, the newly arrived, Père Henri (Hugh O' Conor), reluctantly but slavishly follows the dictates of the repressed mayor and warns the parishioners against yielding to temptation. So a war ensues between the pleasure seekers and the followers of the church, with the mayor's aim being to get her to close her shop before Easter.
The shop is painted over in bright pastel colors and when Armande sees what the decor has become, she cleverly states that it's like a "Mexican brothel." But Vianne can bake some extraordinary sweets and soon exerts a sensual influence over some of the townspeople, whom she lures into the shop with free samples and her friendly gestures.
Her magical chocolates bring sexual pleasure to one couple previously having an empty sexual life and an elderly man (John Wood) uses her chocolates as a present to an elderly widow he always admired (Leslie Caron), but was too timid to let her know that.  Vianne will make friends and teach a very troubled woman how to make chocolates, who initially steals from her, Josephine (Lena Olin), the long-suffering wife of the town's abusive cafe owner Serge (Peter Stormare).
The film drags on for an hour setting up how these uninteresting figures battle their natural temptations and all the repression they have been burdened with all their lives, before the arrival of the 'River Rats,' a guitar-strumming boat crew of gypsies, led by the Irish accented Roux (Pitt), invigorates the films with some more clichés of rebels. They are also strangers the church will turn its anger towards as moral sinners. The moral war led by the mayor intensifies, and a near fatal tragedy is narrowly averted by luck, after the mayor's strong words to the piggish Serge, that something must be done about these strangers, brings about the burning down of their boats. These events cause Vianne to give up on the town and want to leave, but something happens that pulls the town magically back to its senses and they all undergo a change of spirit and become more tolerant and better people. This contrived turn-around was just too much for me to digest without getting sick from all the sugar I was taking in.
REVIEWED ON 2/24/2001 GRADE: D
Dennis Schwartz: "Ozus' World Movie Reviews"
http://www.sover.net/~ozus
<EMAIL_ADDRESS>© ALL RIGHTS RESERVED DENNIS SCHWARTZ
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


Here, Napoleon's political ideals; role within the Directory, the Consulate, and the Empire; domestic reforms; military strategy in Europe; downfall of military executions, and personal life events will be discussed
01
Birth (1769)
Born in Corsica 1769, right after the French colonization of a once proudly independent nation that had resisted larger countries influence for centuries.
02
Corsican Roots.
Napoleon was raised with a scathing hatred of anything french, as a result of his Corsican upbringing. Napoleon’s father, to his disgust, rather than fighting to rebel instead decided to submit and work for France.Napoleon viewed this as weak and spineless, led to him favoring his mother, whom he idolized.
03
Napoleon the Scholar.
Napoleon goes to study in France, at it's peak. Additionally, Napoleon, as a result of his father’s role in French diplomacy got a scholarship to the Royal Military School in 1778. He was nine years old at the time. By the time that he turned 15, he was promoted the Royal Military Academy. Napoleon was not very popular at school because of this ambition and egotistical personality. He was also looked down upon, and was antisocial because of his Corsican identity.
04
Rise to power.
Napoleon starts out as a Second Lieutenant in the French Cavalry. Here, he comes to power, and welcomes it. He is not necessairly a revolutionary. He agrees with Robspierre's desire for order, but despises his use of Terror to obtain it. Napoleon jumps into the revolution by returning to Corsica to acquire power in government. Napoleon and his family are banished from moving back after Napoleon trying to retake power from the established government that was against France. Next, Napoleon, turns to France in turmoil, returns to his position in France and achieves success in the military and wins many battles with simple, virtuosic plans. Napoleon, being self-educated man, is liked by the French populace.
05
The Napoleonic Wars. (Timeline)
** Overall, Napoleon served the Directory’s goals as a military general on the battlefield. He was mostly successful but his losses were contributory to dissent towards Directory that he served. Later on, Napoleon dissolved the Directory through a coup alongside Abbe Sieyes and took power over France. France returned to a monarchy through the consulate. (1799) **


Home
About
contact
Activity
Log in
Register
Contact Us
Your Full Name
Email
Phone Number
massage
send massage
RAYYAN SMART SOLUTION
GET TO KNOW MORE ABOUT OUR WEB DESIGN PROGRAM
To know more about our pragram
Join us to know more


Reliable Book Recommendations
For an example of how CloudI can provide fault-tolerance for CloudI services implemented in Java we will explore how books can receive reliable recommendations. We will use books data from Project Gutenberg and Java source code from the Apache Mahout to provide reliable book recommendations.
Why Does Java Source Code Need CloudI To Be Reliable?
Any actively developed source code may have software errors which cause a system to fail but fault-tolerance makes the failure isolated (both in time and area). With a software system gradually changing as requirements are added or removed it is important that reliability remains constant, despite developer mistakes. While Java development relies upon exception handling for any runtime problems the scope of failures that exception handling can resolve is very limited. If you consider the potential for memory leaks or corruption with Java Native Interface (JNI) usage, threading deadlocks or race conditions and memory exhaustion, it should be easy to understand that failures may occur in many unexpected ways.
Limiting the scope of a failure in Java source code requires isolated memory to ensure the failure is isolated. The Java Virtual Machine (JVM) relies on the use of global state with stop-the-world Garbage Collection to manage its use of memory. However, the lack of isolation in the JVM requires external management to limit the scope of failures within Java source code.
Typically Java source code uses an application server to provide external management as an Operating System (OS) process and CloudI provides application server functionality. Many Java application servers include Java Message Service (JMS) and various Enterprise Service Bus (ESB) solutions exist for use in Java source code. CloudI provides its own Enterprise Service Bus specifically for CloudI services so that service messaging and service lifetime can both be managed efficiently while keeping service memory isolated. The approach in CloudI simplifies development so the focus can remain on business logic while CloudI handles any fault-tolerance or scalability concerns of the source code.
What Do Book Recommendations Look Like In CloudI?
The source code used in this tutorial is available at https://github.com/CloudI/cloudi_tutorial_java but we should first consider how this source code works with an overview. A single Java CloudI service is used to provide a REST API that handles book recommendations with the following Uniform Resource Locator (URL) path suffixes:
| URL Path Suffix | HTTP Method | CloudI Service Name Pattern Suffix |
|---|---|---|
item/refresh |
POST |
item/refresh/post |
item/list |
POST |
item/list/post |
language/list |
POST |
language/list/post |
subject/list |
POST |
subject/list/post |
recommendation/refresh |
POST |
recommendation/refresh/post |
recommendation/update |
POST |
recommendation/update/post |
recommendation/list |
POST |
recommendation/list/post |
client |
GET |
client/get |
The table above shows how the HTTP method becomes a suffix on the URL path to create the CloudI service name pattern suffix (cloudi_service_http_cowboy adds the HTTP method as a suffix in lowercase when sending the HTTP request data as a CloudI service request).
When the CloudI service is configured, a service name pattern prefix is provided to describe the scope of the service.
For the book recommendation Java service, the prefix "/tutorial/java/service/" is used so the full service name pattern for the table’s first entry becomes "/tutorial/java/service/item/refresh/post".
A CloudI service name pattern suffix is used with a subscribe
CloudI API function call to declare a service process will accept service requests with a matching service name.
When multiple processes and multiple threads are specified in the configuration of a CloudI service (with a count greater than 1), the same service will naturally call the subscribe
CloudI API function more than once to declare multiple subscriptions.
A CloudI service that sends a service request with a matching service name will have the service request delivered to a service thread that is randomly selected from the available subscriptions.
Using random selection of a destination that matches the service name keeps the fault-tolerance of the receiving service constant (i.e., with the same probability of a failure for each receive).
All CloudI service request receives depend on a previous call to the subscribe
CloudI API function.
However, both the send_sync
and send_async
CloudI API functions send a service request to a single destination.
To send a service request to all destinations that match a service name requires using the function mcast_async
in the CloudI API, which is similar to publish functionality in other messaging systems.
What Is The Structure Of The Book Recommendation Service?
All services are structured for a 3 part sequence: initialization, handling-requests, and termination.
During CloudI service initialization, data structures are initialized to confirm that the service is ready for a runtime that may last indefinitely (making the initialization stage the most critical stage for a service’s reliability).
Inside the service initialization the subscribe
CloudI API function is typically called so that incoming service requests are handled after initialization is done.
The book recommendation service initialization is shown below (from Service.java:88-138):
public void run()
{
try
{
Main.info(this, "initialization begin");
// initialization timeout is enforced
// based on the service configuration value
if (Service.recommend() == null)
{
throw new RuntimeException("Recommender initialization failed");
}
// subscribe to different CloudI service name patterns
if (this.api.process_index() == 0 &&
this.thread_index == 0)
{
// only a single thread of a single OS process
// should handle items refresh due to filesystem usage
this.api.subscribe("item/refresh/post",
this, "itemRefresh");
}
this.api.subscribe("item/list/post",
this, "itemList");
this.api.subscribe("language/list/post",
this, "languageList");
this.api.subscribe("subject/list/post",
this, "subjectList");
if (this.thread_index == 0)
{
// only a single thread in any OS process
// should handle recommendation refresh due to global data
this.api.subscribe("recommendation/refresh/post",
this, "recommendationRefresh");
}
this.api.subscribe("recommendation/update/post",
this, "recommendationUpdate");
this.api.subscribe("recommendation/list/post",
this, "recommendationList");
if (this.thread_index != 0)
{
// persistent connections that lack a standard routing
// identifier in their protocol (like websockets)
// use the same service name for all requests
// which must be routed based on the content of the request
// (do not utilize thread 0, so that it can be used
// as a forward destination, for requests that require it)
this.api.subscribe("client/get",
this, "client");
}
Main.info(this, "initialization end");
The initialization sequence above shows that the recommendation data is initialized first.
Afterwards, all the subscribe
CloudI API function calls occur with a few choosing specific threads for execution.
You will notice that the Service object instant pointer this
is provided with a string function name in the subscribe function call so that the service request will be handled in a specific function (Java 8 allows a single method reference to be used instead of these two separate parameters).
The "itemRefresh" service request can take a few minutes and creates filesystem data, so this request is only handled by a single thread within a single process (to ensure the execution is never parallel with other service threads).
The "recommendationRefresh" service request may take a minute or two with many recommendations but is updating global data in a single process, so it gets its own thread.
The "client" service request is used by the WebSocket interface to handle any of the service’s functionality with a single WebSocket connection and its functionality will never block the occurrence of the "recommendationRefresh" service request.
All of the subscriptions that have been discussed are simply for CloudI service requests that define the interface of a REST API. For the REST API to be used by HTTP requests, a HTTP server that creates CloudI service requests needs to be used. CloudI includes two CloudI services that are HTTP Servers: cloudi_service_http_cowboy and cloudi_service_http_elli. The book recommendation service will use cloudi_service_http_cowboy for handling both basic HTTP requests and WebSocket requests.
The initialization sequence ends when the poll
CloudI API function is called to begin handling requests.
Handling Requests
CloudI service requests are processed in-memory and not persisted to disk because service requests are transient transactional data which may or may not represent a failure (only the sender really knows if it is a failure based on the response).
When a CloudI service request receives a response the transaction is complete.
To identify the transaction during its lifetime a globally unique Transaction Identifier (often named trans_id
or TransId
in the source code) is used within both the request and the response.
All CloudI service requests are able to receive a response.
The CloudI service that handles the request can make the request asynchronous by providing a "null response" (a response that has the response data and response_info data set to a binary of size 0) due to the control of the service request passing to the receiver when the service request is handled.
That means that the send_async
CloudI API function call is an asynchronous send due to not waiting for a response to occur, though a response may be sent in the future if the receiving service decides to send a response.
An example of a service request handling function is below (using the "itemRefresh" function from Service.java:166-212):
public Object itemRefresh(Integer request_type, String name,
String pattern, byte[] request_info,
byte[] request, Integer timeout,
Byte priority, byte[] trans_id,
OtpErlangPid source)
{
// refresh all item data asynchronously
final JSONItemRefreshRequest request_json =
JSONItemRefreshRequest.fromString(new String(request));
if (request_json == null)
{
return JSONItemRefreshResponse
.failure("json")
.toString().getBytes();
}
if (this.item_refresh_pending != null &&
this.item_refresh_pending.isDone() == false)
{
return JSONItemRefreshResponse
.failure("pending")
.toString().getBytes();
}
final String D = System.getProperty("file.separator");
final String executable_path = System.getProperty("user.dir") + D +
"scripts" + D;
final String executable_download = executable_path +
"gutenberg_refresh_download";
final String executable_cleanup = executable_path +
"gutenberg_refresh_cleanup";
final String directory = System.getProperty("java.io.tmpdir") + D +
(new API.TransId(trans_id)).toString();
final RecommendData recommend = Service.recommend();
if (recommend == null)
{
return JSONItemListResponse
.failure("recommend")
.toString().getBytes();
}
// item_refresh takes a long time, so it is done asynchronously
this.item_refresh_pending = this.item_refresh_executor.submit(
new GutenbergRefresh(this.idle,
recommend.dataSource(),
executable_download,
executable_cleanup,
directory));
return JSONItemRefreshResponse.success().toString().getBytes();
}
The contents of the service request is contained within the 9 function parameters to keep service request handling consistent with the avoidance of side-effects present in functional programming.
So, while it may seem a bit scary to have 9 function parameters you normally only need to focus on the request
parameter which contains the request body.
The return value of the function provides the response body for the response to the service request.
HTTP requests do provide HTTP headers which is request meta-data and all request meta-data is put into the request_info
parameter.
An important fault-tolerance constraint is the timeout on a service request (i.e., the timeout
parameter) since it determines how long the service request is valid.
Every CloudI service request contains a specific timeout value in milliseconds which is decremented when the service request encounters any delay (e.g., due to queuing, handling execution, or forwarding).
The priority
parameter controls the priority when a service request is queued and a service request is only queued while the service thread is busy with a separate service request.
The request_type
, name
, and pattern
parameters show how the service request got here by containing whether the send is asynchronous or synchronous, the service name, and the service name pattern, respectively.
The source
contains the source of the service request as it is represented within the CloudI source code (it is the response destination).
While many of the parameters in a service request are just book-keeping for tracking the transaction, the service request can be forwarded through any number of service request handlers before a response is provided.
We can see in the source code example above that a JSON response is always returned as a byte array since the response data is always passed as binary data.
If it was necessary to add HTTP response headers in the response, they would be added to response_info
data with the function returning a 2 element array that contains both the response_info
meta-data and the response
body, respectively.
However, nothing in CloudI enforces the format of the transaction contents (the request_info
meta-data input, the request
body input, the response_info
meta-data output, and the response
body output) to keep services loosely coupled.
By not enforcing the data format used in the transaction, CloudI remains protocol agnostic and allows services to fail naturally when they are unable to handle a format.
This approach keeps service development flexible so services can independently change at their own pace.
How Do Book Recommendations Occur?
To understand how book recommendations occur within the example source code, lets first get the source code running.
Creating Book Recommendations
Make sure you have Git, curl, Maven with Java 7 (or 8) and PostgreSQL running (>= 9.3). Later Java versions are not currently supported by the Apache Mahout dependencies but are supported by the Java CloudI API. Make sure CloudI is installed and running, using the installation and start sequence below, if necessary:
wget https://osdn.net/dl/cloudi/cloudi-2.0.6.tar.gz
tar zxvf cloudi-2.0.6.tar.gz
cd cloudi-2.0.6/src
./configure
make
sudo make install
cd ../..
sudo cloudi start
Get a copy of the Java tutorial repository, build the tutorial with Maven and keep the shell in the repository directory:
git clone https://github.com/CloudI/cloudi_tutorial_java.git
cd cloudi_tutorial_java
mvn clean package
To create the CloudI service configuration for the services used in this tutorial use the following commands in your shell (with the repository directory as the current working directory):
export JAVA=`which java`
export PWD=`pwd`
export USER=`whoami`
cat << EOF > website.conf
[[{prefix, "/"},
{module, cloudi_service_filesystem},
{args,
[{directory, "$PWD/html/"}]},
{dest_refresh, none},
{count_process, 4}],
[{prefix, "/tutorial/java/"},
{module, cloudi_service_http_cowboy},
{args,
[{port, 8080}, {use_websockets, true}]},
{timeout_async, 600000},
{timeout_sync, 600000}]]
EOF
cat << EOF > log4j.properties
log4j.rootLogger=DEBUG,consoleAppender
log4j.appender.consoleAppender=org.apache.log4j.ConsoleAppender
log4j.appender.consoleAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.consoleAppender.layout.ConversionPattern=[%t] %-5p %c %x - %m%n
EOF
cat << EOF > tutorial.conf
[[{prefix, "/tutorial/java/service/"},
{file_path, "$JAVA"},
{args, "-Dfile.encoding=UTF-8 "
"-Dlog4j.configuration=file:$PWD/log4j.properties "
"-server "
"-ea:org.cloudi... "
"-Xms3g -Xmx3g "
"-jar $PWD/target/cloudi_tutorial_java-2.0.6-jar-with-dependencies.jar "
"-pgsql_hostname localhost "
"-pgsql_port 5432 "
"-pgsql_database cloudi_tutorial_java "
"-pgsql_username cloudi_tutorial_java "
"-pgsql_password cloudi_tutorial_java"},
{timeout_init, 600000},
{count_thread, 4},
{options,
[{owner, [{user, "$USER"}]},
{directory, "$PWD"}]}]]
EOF
The configuration for the Java tutorial CloudI services now is split into two separate files: website.conf (to run initial CloudI services using services that are included in CloudI for hosting the interface and handling HTTP connections) and tutorial.conf (to run the Java tutorial source code for handling book recommendation REST API requests). The main reason to split the configuration is to ensure all HTTP requests have a 600000 milliseconds (10 minutes) timeout value. The 10 minute timeout value for a HTTP request allows the Java service for book recommendations to be started dynamically with a 10 minute initialization timeout value for the Apache Mahout model creation (a 1-2 minute startup time is normal with lots of recommendation data).
Make sure you have PostgreSQL setup with a database that matches the Java service configuration arguments:
psql -U postgres << EOF
CREATE DATABASE cloudi_tutorial_java;
CREATE USER cloudi_tutorial_java WITH PASSWORD 'cloudi_tutorial_java';
GRANT ALL PRIVILEGES ON DATABASE cloudi_tutorial_java to cloudi_tutorial_java;
EOF
Create the PostgreSQL schema with some test data:
wget https://osdn.net/projects/cloudi/downloads/72706/schema.sql.bz2
cat SHA256SUM | sha256sum --check
bunzip2 schema.sql.bz2
psql -h localhost cloudi_tutorial_java cloudi_tutorial_java < schema.sql
Now the tutorial services can be started by relying on the default CloudI configuration having cloudi_service_http_cowboy running on port 6464 so that CloudI Service API requests can be made dynamically (using cloudi_service_api_requests) to start new service instances:
curl -X POST -d @website.conf http://localhost:6464/cloudi/api/rpc/services_add.erl
The result will be similar to ["ServiceId1","ServiceId2"]
where ServiceId1
is a UUID (Universally Unique Identifier) for the new service instance of cloudi_service_filesystem and ServiceId2
is a UUID for the new service instance of cloudi_service_http_cowboy.
The HTTP request above starts the new HTTP server service on port 8080 with the 10 minute timeout value and the web interface files.
Now we can use port 8080 for a CloudI Service API request so the 10 minute timeout value is used when initializing the book recommendation service:
curl -X POST -d @tutorial.conf http://localhost:8080/cloudi/api/rpc/services_add.erl
The result will provide a single service ID for the new Java book recommendation service instance. With all the tutorial services running successfully we can now look at the book recommendation interface at http://localhost:8080/tutorial/java/. The interface should look like the screenshot below:
The green box surrounds the "Items Available" which are Gutenberg Project books included as test data within the schema.sql
file.
Add a few ratings to get recommendations:
-
Set a rating for an item by adjusting the slider and click the "Rate" button to store the rating. The "Rate" button will change to an "Update" button to allow the rating to change in the future.
-
Set a second ratings in the same way while keeping the user_id value set to
1
. -
The Apache Mahout recommendation model has not yet been updated based on other user ratings to create suggestions so click the "Refresh Recommendation Model" button (this update action is typically done weekly or daily with online services that have many users, when Apache Mahout is used in production).
The interface will update to show a red box that surrounds the "Recommendations Based On Previous Ratings" with expected ratings values that the recommendation model predicted, as shown in the screenshot below (your output will be based on the ratings you set, though it will look similar):
How Is The Book Recommendation Data Processed?
To understand how book recommendations are occurring in the interface, lets start by looking closer at the REST API in the Java book recommendation service:
| Service Name Pattern Suffix | Service.java function |
message_name |
Processes and Threads |
|---|---|---|---|
item/refresh/post |
itemRefresh |
item_refresh |
thread 0 in process 0 only |
item/list/post |
itemList |
item_list |
Any |
language/list/post |
languageList |
language_list |
Any |
subject/list/post |
subjectList |
subject_list |
Any |
recommendation/refresh/post |
recommendationRefresh |
recommendation_refresh |
thread 0 only |
recommendation/update/post |
recommendationUpdate |
recommendation_update |
Any |
recommendation/list/post |
recommendationList |
recommendation_list |
Any |
client/get |
client |
thread > 0 |
The table above shows the service subscriptions that occur during service initialization (in Service.java:88-138).
The service configuration used for running Service.java
was stored in the tutorial.conf
file and it specified a total of 4 threads with count_thread
being set to 4
.
The service configuration did not provide a count_process
setting so the number of OS processes defaulted to 1
(the thread count is applied to each OS process separately, i.e., count_process
== 2
would have 2 OS processes, each with 4
threads).
The service configuration always specifies a service name pattern prefix and that was set to "/tutorial/java/service/" in the tutorial.conf
file.
When a HTTP request (on port 8080) is made to the REST API through cloudi_service_http_cowboy, a CloudI service request is sent based on the HTTP request URL path using the default timeout that was configured for cloudi_service_http_cowboy service requests (10 minutes). The Javascript interface relies on a WebSocket connection for any usage of the REST API as shown below (index.html:542-550):
var host = "<IP_ADDRESS>";
var port = "8080";
var service_name = "/tutorial/java/service/client";
if (typeof WebSocket != "function") {
alert("WebSocket support is required!");
return;
}
var websocket_url = "ws://" + host + ":" + port + service_name;
websocket = new WebSocket(websocket_url);
This means that the URL path includes the Java book recommendation service prefix of "/tutorial/java/service/" with a suffix of "client/get" (all WebSocket connections are initialized with the GET HTTP method and cloudi_service_http_cowboy is adding the "/get" suffix to the URL path before sending a CloudI service request).
All REST API requests are then sent as JSON data on the WebSocket connection with a message_name
property to distinguish between the different endpoints.
The endpoints can also be used as separate URL paths to receive the same response without a WebSocket connection (which is useful for testing):
curl -X POST -d '{"message_name": "item_list", "user_id": 1, "language": "en", "subject": "Philosophy"}' http://localhost:8080/tutorial/java/service/item/list
Book Recommendation WebSocket Requests
A WebSocket request is sent as a CloudI service request that is randomly load-balanced among the 3 threads that have subscribed with the function named "client" (in Service.java
the subscription occurred when thread_index
> 0 with 4 threads so 3 threads handle the "client" function).
If Service.java
had count_process
set to 2
in its service configuration, the service request would instead be randomly load-balanced among 6 total threads with each group of 3 in a separate OS process.
The service request handler function "client" is shown below (from Service.java:385-508):
public Object client(Integer request_type, String name, String pattern,
byte[] request_info, byte[] request,
Integer timeout, Byte priority, byte[] trans_id,
OtpErlangPid source)
throws API.ForwardAsyncException,
API.ForwardSyncException,
API.InvalidInputException,
API.MessageDecodingException,
API.TerminateException
{
// handle any JSON request based on the "message_name" field
final JSONRequest request_json =
JSONRequest.fromString(new String(request));
if (request_json == null)
{
return JSONResponse
.failure("json")
.toString().getBytes();
}
switch (request_json.getMessageName())
{
case JSONItemRefreshRequest.message_name_valid:
// a single OS process will refresh the single database
final String name_item_refresh =
this.api.prefix() + "item/refresh/post";
this.api.forward_(request_type, name_item_refresh,
request_info, request,
timeout, priority,
trans_id, source);
return null;
case JSONItemListRequest.message_name_valid:
return this.itemList(request_type, name, pattern,
request_info, request,
timeout, priority,
trans_id, source);
case JSONLanguageListRequest.message_name_valid:
return this.languageList(request_type, name, pattern,
request_info, request,
timeout, priority,
trans_id, source);
case JSONSubjectListRequest.message_name_valid:
return this.subjectList(request_type, name, pattern,
request_info, request,
timeout, priority,
trans_id, source);
case JSONRecommendationRefreshRequest.message_name_valid:
// all OS processes need to refresh their recommendation data
final String name_recommendation_refresh =
this.api.prefix() + "recommendation/refresh/post";
final String name_websockets =
this.api.prefix() + "client/websocket";
final int refresh_response_latency_max = 1000; // milliseconds
final int refresh_request_timeout = Math.max(0,
timeout - refresh_response_latency_max);
final ArrayList<API.TransId> refresh_requests =
this.api.mcast_async(name_recommendation_refresh,
request_info, request,
refresh_request_timeout, priority);
if (refresh_requests.isEmpty())
return JSONRecommendationRefreshResponse
.failure("timeout")
.toString().getBytes();
final int timeout_decrement = 500; // milliseconds
final Iterator<API.TransId> refresh_requests_iterator =
refresh_requests.iterator();
API.TransId refresh_request_id =
refresh_requests_iterator.next();
while (timeout > 0)
{
final API.Response refresh_response =
this.api.recv_async(timeout_decrement,
refresh_request_id.id);
if (refresh_response.isTimeout())
{
if (timeout_decrement >= timeout)
timeout = 0;
else
timeout -= timeout_decrement;
}
else
{
final JSONRecommendationRefreshResponse
refresh_response_json =
JSONRecommendationRefreshResponse
.fromString(new String(refresh_response
.response));
if (! refresh_response_json.getSuccess())
{
return refresh_response.response; // failure
}
else if (! refresh_requests_iterator.hasNext())
{
byte[] notification =
JSONRecommendationRefreshOccurredNotification
.success().toString().getBytes();
this.api.mcast_async(name_websockets,
notification);
return refresh_response.response; // last success
}
else
{
refresh_request_id =
refresh_requests_iterator.next();
}
}
}
// timeout already occurred, null response
return ("").getBytes();
case JSONRecommendationListRequest.message_name_valid:
return this.recommendationList(request_type, name, pattern,
request_info, request,
timeout, priority,
trans_id, source);
case JSONRecommendationUpdateRequest.message_name_valid:
return this.recommendationUpdate(request_type, name, pattern,
request_info, request,
timeout, priority,
trans_id, source);
default:
return JSONResponse
.failure("message_name")
.toString().getBytes();
}
}
The source code above shows that most of the REST API functionality is utilized by just a function call with the same function that was provided for the subscription based on the message_name
JSON property.
Any exceptions that are thrown in the service request handler functions get handled by CloudI and result in a null response to the service request.
The "client" function source code that is more complex is focused on the message_name
JSONItemRefreshRequest.message_name_valid
(with the value "item_refresh") and JSONRecommendationRefreshRequest.message_name_valid
(with the value "recommendation_refresh").
To handle when message_name
is "item_refresh" requires the forward
CloudI API function to ensure that only a single thread in a single process is utilized for refreshing all the database items.
The forward
CloudI API function is simply changing the service name for the service request destination so that the service request is sent to a new destination and the "client" function returns without sending a response.
If message_name
is "recommendation_refresh" then the mcast_async
CloudI API function is used to ensure each process has a single thread refresh the Apache Mahout recommendation model that is kept in-memory.
After the mcast_async
, the recv_async
CloudI API function is used to get the response from each asynchronous send (making mcast_async
a little different from typical publish functionality since a response can be received).
If all the responses indicate a success, a second mcast_async
CloudI API function call is used to notify all connected WebSocket connections that the recommendations may have changed.
Then the interface is able to get a new list of recommendations with a separate request on the WebSocket connection.
All the other service request handler functions are returning byte arrays with the response
data which is equivalent (but shorter) than calling the return
function in the CloudI API.
As mentioned earlier (in Handling Requests) a 2 element array could be returned to provide both the response_info
data (response meta-data, e.g., HTTP response headers) and the response
data (response body).
Book Recommendation WebSocket Notifications
The WebSocket notification in the "client" function when message_name
is "recommendation_refresh" (as described above) was a single mcast_async
CloudI API function call because the notification could occur within a service request handler.
Using the CloudI API outside of a service request handler does require some planning to avoid threading problems.
To understand how the CloudI API can be used correctly outside of a service request handler we can look at how the "itemRefresh" function (in Service.java
which we looked at earlier in Handling Requests) can provide data after it has already returned a response from the service request handler.
The "itemRefresh" function response to a service request is only providing whether or not its single processing thread is busy, or if it is idle, it will start a new execution of the item refresh in the single processing thread. The single processing thread is separate from the 4 threads configured in the CloudI service configuration to keep the item refresh processing separate from low-latency service request handling. The refresh of the items takes a few minutes when getting all the current data from Project Gutenberg, uncompressing the data, parsing all the files and updating the PostgreSQL database. However, after the thread is done refreshing the items, the interface would want to know that new items may exist, so it can request a new list of items (similar to how a new list of recommendations is requested by the interface after a notification).
To utilize the result of the "itemRefresh" function processing thread requires that the poll
CloudI API function is interrupted to call a function that is not a service request handler (to allow usage of the mcast_async
CloudI API function for sending a notification to all connected WebSockets, similar to what occurs with a successful "recommendation_refresh" message_name
in the "client" function).
For scheduling an interrupt of the poll
function the timeout parameter is used as shown below (from Service.java:139-144):
// accept service requests
while (this.api.poll(ServiceIdle.INTERVAL))
{
// execute ServiceIdle function objects
this.idle.check();
}
Each interrupt of the poll
function allows execution to check if it can run a chunk of function objects in the ServiceIdle
class as shown below (from ServiceIdle.java:10-74):
public class ServiceIdle
{
public static final int INTERVAL = 1000; // milliseconds
public static final int SIZE_MAX = 1000;
public static final int SIZE_CHUNK = SIZE_MAX / 2;
public static interface Callable
{
public void call(final API api);
}
public static class Queue
{
private final ArrayBlockingQueue<Callable> queue;
public Queue()
{
this.queue = new ArrayBlockingQueue<Callable>(ServiceIdle.SIZE_MAX);
}
public void in(final Callable o)
{
try
{
this.queue.put(o);
}
catch (InterruptedException e)
{
e.printStackTrace(Main.err);
}
}
public LinkedList<Callable> out()
{
final LinkedList<Callable> out = new LinkedList<Callable>();
this.queue.drainTo(out, ServiceIdle.SIZE_CHUNK);
return out;
}
}
private final API api;
private final Queue queue;
public ServiceIdle(final API api)
{
this.api = api;
this.queue = new Queue();
}
public void check()
{
final LinkedList<Callable> idle = this.queue.out();
while (! idle.isEmpty())
{
final Callable o = idle.removeFirst();
o.call(this.api);
}
}
public void execute(final Callable o)
{
this.queue.in(o);
}
}
The result of the "itemRefresh" function processing thread is stored in the ServiceIdle
class instance within the GutenbergRefresh
class which handles the item refresh process (GutenbergRefresh.java:101-104):
if (error == null)
this.idle.execute(GutenbergRefreshDone.success());
else
this.idle.execute(GutenbergRefreshDone.failure(error));
After the poll
function is interrupted and the ServiceIdle
class has function objects to call, the GutenbergRefreshDone
function object is executed to send the "itemRefresh" result notification to all connected WebSockets (GutenbergRefreshDone.java:32-52):
byte[] notification;
if (this.success)
{
notification = JSONItemRefreshOccurredNotification
.success().toString().getBytes();
}
else
{
notification = JSONItemRefreshOccurredNotification
.failure(this.error).toString().getBytes();
}
final String name_websockets = api.prefix() + "client/websocket";
try
{
api.mcast_async(name_websockets,
notification);
}
catch (Exception e)
{
e.printStackTrace(Main.err);
}
The usage of ServiceIdle
after the poll
CloudI API function has returned is required due to the CloudI API instance not providing thread-safe locking to keep its execution efficient (and to help development avoid threading deadlocks).
What If I Want To Create My Own Service?
The Java source code for this tutorial (at https://github.com/CloudI/cloudi_tutorial_java) can help provide a start for your CloudI Java Service development in the future. There is a separate documentation section for service development guidelines which contains information applicable to any programming language ("Guidelines For Creating A CloudI Service"). If you need other examples of Java services there are programming examples below the tutorials list.


Scroll!
1
Things aren't always
#000000
#FFFFFF
2
PUSH
Nice!
3
boop
4
boop
5
6
7
8
9
10
SCR
O
LL
11
12
13
14
15
16
17
18
19
20
20
21
The Ocean.
It's big and blue.
And dangerous.
23
24
Is it getting darker in here?
We're deep into the ocean now.
This is the deep sea.
You are
feet deep.


The briad, or berger de brie, is a large breed of dog traditionally used as a herder and guardian of sheep.
The briard, which is believed to have originated in France, has been bred for centuries to herd and to protect sheep. The breed was used by the French Army sentries, messengers and to search for wounded soldiers because of its fine sense of hearing. Briards were used in the First World War almost to the point of extinction. Currently the population of briards is slowly covering. Charlemagne, Napoleon, Thomas Jefferson and Lafayette all owned briards.
by son to


BetterDoctor Symptom Search Api
Enter your condition below and physicians related to treatment for the will be listed to the right.
Enter your condition here:
Submit
`


I'm Nicolas Campos, a software developer currently living in New York City.
Currently in software industry. In 2018 I recieved a bachelor's degree, joint in Physics and Mathematics with a secondary in Russian Studies, from Harvard. I also received two Language Citations from the university in both the Russian and Polish languages. I love trying new things and simply making stuff work :)


The original data was acquired from Harvard's Dataverse. The data was then processed using a python script to count the number of times each country was mentioned per year and saved the result to a csv file for use with the map.
Using the names of countries within the countries topojson file, each tweet was examined to see if each it contained each country name. If a tweet contained two different country names, each country included would gain a mention. Extra criteria for a mention was created speifically for the USA. If a tweet contained the word "america" and did not contain "north", "south", or "united states of america", it was assumed that Trump was referring to the USA and counted as a mention. In the same vain, if "USA" or "U.S.A." was included in the tweet, it was counted as a mention. It is recognized that this method can not accurately calculate the exact number of country mentions due to language nuances. It is also recognized that country names change and the list generated from the 2018 Natural Earth countries topojson file may not include all country names mentioned.


LastPass has detected that you have used the password for this login on other sites, too. We recommend going to your account settings for this site, and creating a new password. Use LastPass to generate a unique, strong password for this account. You can then save the changes on the site, and to LastPass.
Name:
Username:


|
|
Beginner's guide to soldering an OOK Tx module to emonPi v1.5 |
3 |
50 weeks 6 days ago by dain |
50 weeks 6 days ago by glyn.hudson |
|
|
Buying new emonTXv3.4, what node? |
3 |
51 weeks 4 days ago by joshpond |
51 weeks 3 days ago by joshpond |
|
|
Which CT? |
3 |
51 weeks 4 days ago by wuciwug |
51 weeks 3 days ago by Bbaass_TMH |
|
|
Could not reliably control the 3CH Wifi Relay Control Board |
3 |
1 year 3 weeks ago by jvda |
1 year 3 weeks ago by jvda |
|
|
Idiots guide to voltage calibration |
3 |
1 year 4 weeks ago by webby |
1 year 4 weeks ago by Robert Wall |
|
|
Enclosures/cases for OEM sensors - DIY or bought |
3 |
1 year 5 weeks ago by Vster |
1 year 4 weeks ago by Shoka |
|
|
"EnergyMonitor" does not name a type Error |
3 |
1 year 5 weeks ago by webby |
1 year 4 weeks ago by webby |
|
|
EmonPi Lite? |
3 |
1 year 5 weeks ago by mrscruffy |
1 year 1 week ago by mrscruffy |
|
|
invalid values emonTX /emonBase |
3 |
1 year 7 weeks ago by Roeland |
1 year 7 weeks ago by pb66 |
|
|
Heatpump Monitoring board MBUS, ESP12++ |
3 |
1 year 9 weeks ago by TrystanLea |
1 year 8 weeks ago by fcattaneo |
|
|
avrdude: stk500_recv(): programmer is not responding |
3 |
1 year 10 weeks ago by amgican |
1 year 10 weeks ago by Robert Wall |
|
|
How to Interface NRG #40 Anemometer to Arduino |
3 |
1 year 11 weeks ago by Lee_B |
1 year 11 weeks ago by Robert Wall |
|
|
Not getting [RF12demo.12] on minimum |
3 |
1 year 13 weeks ago by stumpygreg |
1 year 13 weeks ago by pb66 |
|
|
Probleme mit Arduino und 4 CT Sensoren |
3 |
1 year 16 weeks ago by aDm1N |
1 year 16 weeks ago by Robert Wall |
|
|
Adding 2nd Emon TX V3 |
3 |
1 year 19 weeks ago by JBlackham |
1 year 18 weeks ago by Robert Wall |
|
|
CT not working? |
3 |
1 year 20 weeks ago by newms |
1 year 19 weeks ago by Robert Wall |
|
|
pv immersion, data logging, tank temperature and room temp needed. |
3 |
1 year 21 weeks ago by saggers |
1 year 21 weeks ago by calypso_rae |
|
|
Sending big data packet by radio. What consequences ? |
3 |
1 year 24 weeks ago by Eric_AMANN |
1 year 24 weeks ago by Eric_AMANN |
|
|
Voltage and Current reading varies alot[Arduino] |
3 |
1 year 26 weeks ago by Guest |
1 year 26 weeks ago by Robert Wall |
|
|
Help understanding several components in the sch of eMonTx 3.4 |
3 |
1 year 26 weeks ago by Gzero |
1 year 24 weeks ago by whatnick |
|
|
Weak signal? |
3 |
1 year 27 weeks ago by daturach |
1 year 27 weeks ago by Bill Thomson |
|
|
Variation in air conditioner power readings - can anyone explain? |
3 |
1 year 28 weeks ago by baldockery |
1 year 28 weeks ago by Robert Wall |
|
|
EmonTH 1.5 errors in schematic and board files? [Solved - No errors] |
3 |
1 year 29 weeks ago by haden |
1 year 29 weeks ago by haden |
|
|
emonTH - Non-battery powered version? |
3 |
1 year 31 weeks ago by lev400 |
1 year 30 weeks ago by lev400 |
|
|
Arduino and several SCT-013-000 sensors |
3 |
1 year 31 weeks ago by mel |
1 year 31 weeks ago by Robert Wall |
|
|
RFM2PI v2 (ATMEGA328) Automatic upgrade - timeouts |
3 |
1 year 34 weeks ago by stuart |
1 year 34 weeks ago by pb66 |
|
|
Can use one CT simultaneously to measure Volts and Current ? |
3 |
1 year 36 weeks ago by tinamore |
1 year 35 weeks ago by tinamore |
|
|
emonPi 5v form factor not same as Raspberry Pi |
3 |
1 year 37 weeks ago by ianrumford |
1 year 36 weeks ago by hapless |
|
|
Question about the half wave rectifier |
3 |
1 year 40 weeks ago by Gzero |
1 year 40 weeks ago by Robert Wall |
|
|
Converting emontx 2.2 from 3.3v to 5V |
3 |
1 year 40 weeks ago by bruce_miranda |
1 year 40 weeks ago by bruce_miranda |
|
|
DC-DC stepUp |
3 |
1 year 47 weeks ago by Larsjo |
1 year 47 weeks ago by mharizanov |
|
|
why is the DC offset circuit working with the current sensor (YHDC SCT-013-000) but not with the function generator! |
3 |
1 year 47 weeks ago by Sabirmgd |
1 year 47 weeks ago by Robert Wall |
|
|
Burden for ACT-0750 CT and Arduino Due (3.3v) |
3 |
1 year 48 weeks ago by renza |
1 year 48 weeks ago by Robert Wall |
|
|
CT saturation question |
3 |
1 year 50 weeks ago by BraileTrail |
1 year 50 weeks ago by Robert Wall |
|
|
need help about arduino-emontx shield |
3 |
1 year 50 weeks ago by trshark |
1 year 49 weeks ago by trshark |
|
|
CT Sensor sensitivity |
3 |
1 year 51 weeks ago by coldpenguin |
1 year 51 weeks ago by Robert Wall |
|
|
emonHub RX frame node id outside scope |
3 |
2 years 5 weeks ago by bbroge |
2 years 5 weeks ago by pb66 |
|
|
emonTH with relay module |
3 |
2 years 6 weeks ago by aspto |
2 years 6 weeks ago by Robert Wall |
|
|
emontx newbie to measure subsctions |
3 |
2 years 7 weeks ago by Guest |
2 years 7 weeks ago by Robert Wall |
|
|
questions about RFM69Pi |
3 |
2 years 8 weeks ago by haden |
2 years 7 weeks ago by dBC |
|
|
Noise in the Arduino / emonTx front end |
3 |
2 years 11 weeks ago by Robert Wall |
2 years 11 weeks ago by glyn.hudson |
|
|
RFM12Pi V1 vs V2 |
3 |
2 years 12 weeks ago by haden |
2 years 11 weeks ago by haden |
|
|
Flush mount temperature sensor |
3 |
2 years 16 weeks ago by madas |
2 years 15 weeks ago by madas |
|
|
Extending CT wires |
3 |
2 years 16 weeks ago by fitzy89 |
2 years 11 weeks ago by Robert Wall |
|
|
Potential collisions when using many EmonTX V3 RF in the same place |
3 |
2 years 25 weeks ago by bugmax |
2 years 24 weeks ago by calypso_rae |
|
|
Water flow sensor with emonTH |
3 |
2 years 26 weeks ago by bernamorillo |
2 years 26 weeks ago by JBecker |
|
|
Using Efergy CT Sensor with EmonTx V3 |
3 |
2 years 27 weeks ago by amailtoaben |
2 years 27 weeks ago by Robert Wall |
|
|
Wind turbine monitoring |
3 |
2 years 28 weeks ago by gilou |
2 years 2 weeks ago by gilou |
|
|
Arduino and RFM12B connections |
3 |
2 years 29 weeks ago by Guest |
2 years 29 weeks ago by jb79 |
|
|
CTs 2-4 all return a signal of 0 |
3 |
2 years 32 weeks ago by natshiel |
2 years 31 weeks ago by Robert Wall |


Writing (Basic) IRC Scripts in Python
16 Mar 2015This post is intended to be a gentle introduction into the fine and glorious art of writing IRC scripts (for irssi and hexchat, weechat is on the way) in the Python programming language. In this, you will be walked through, line by line, the “shakespeare.py” plugin for various IRC clients that I wrote out of sheer boredom and a desire to insult people on IRC ye olde style.
Hopefully, the following will be comprehensible to all readers, as I try explain every last bit of it so even the total beginner to Python or Computer Science III in general can get the gist of it :)
General Stuff
First off, lets cover some of the general purpose boilerplate crap used in all the scripts.
So, arrays/lists, and how do they work…
Arrays, or lists, in Python, are really simple. Now, I am not a computer scientist, just a bored hacker. So I make no distinction between an array and a list as they look the same to me. Call me a pleb, but I give no fucks.
You basically have an array like the following:
Now, you can probably see clearly that we can address this in a nice fashion.
So we can select elements from our array and print them, without popping them from our array. We can also append items to our array, and remove them from our array, quite easily.
As you can see above, we appended ‘4’ to the array, and popped the first element (which was 2, rememeber, in programming we count from 0). This shows how simple it is to insert and delete items from an array or list. In our program, we make 3 arrays. Each including a bunch of “segments” for an insult.
Random Selection…
In Python, we can easily do random selection with the module named “random”. to randomly select an item from an array, we will simply call the random.choice() function on the array which will return a random element from it.
An example follows:
In our scripts, we use this random selection mechanism to select a random item from each array to construct our insult.
Building our “Insult” from the arrays…
In order to build our insult, we need a random element from each of the arrays, so we can build a string to send.
This is actually really, really, simple. See below.
Interesting Mathematics on how many insults we have…
Now, you may note, that we have 50 insult pieces in each of the arrays.
P(n,r) = number of permutations
P(50,3) = number of permutations
50! / (50 - 3)! = number of permutations
Number of Permutations = 117600
So hypothetically, we can fire off 117,600 unique insults!
irssi specific stuff
Installing irssi-python for development
First off, don’t even bother.  I wasted an entire day trying to get it to build on 2 different Debian boxes, and had absolutely no luck. The command to install irssi and irssi-python is as follows. You are more than welcome to try getting it working on Debian or whatever, but be warned, you might find yourself jumping off the nearest tall structure before long, due to the sheer amount of fuckery involved.
 They basically seem to accept them all as “data”.
See the following:
Note how “data” is the first argument passed. This seems to be “all the arguments given to the command”, as “server” and “witem” are built in (the current server, and current window name). Data is effectively a string object of arguments that I strongly suspect parsing is left to the user.
Sending messages to channel
In the above code snippet, you will have noticed the “server.command” function being called. This function sends a raw IRC command to the server.  You simply do the following at the end of your script.
this binds the command “shakespeare” to the function “cmd_insult”, and calls said function whenever the “shakespeare” command is called.
Installing and using the script
To install, after nearly shooting yourself in the face during the compiling hell that is irssi-python, just move shakespeare.py to ~/.irssi/scripts and add “py load shakespeare” to your ~/.irssi/startup somewhere after “load python”.
To use, just do /shakespeare to insult no one in particular (the open query window), or /shakespeare $nick to insult a specific user in the current channel.
Hexchat Specific Stuff
Installing Hexchat
Hexchat comes with the Python support built in. So… You just install Hexchat, using yaourt/pacman on Arch (I prefer yaourt), or apt on Debian. Those of you on Fedora and other “things I don’t use” are on your own here. As for the Mac users… I guess homebrew or something?
 You have to intiialize your script with a load of boilerplate bollocks giving the name, version, and description of your script. Otherwise it may complain or something.
How Hexchat seems to accept arguments
Now, this is something I have not worked out, exactly. It kind of works. Sometimes. Sorta. Well, it takes arguments as “word”, similar to “data”, except, in what I think is kind of a neat feature, it treats them as an addressable list like sys.argv. So your initial command would be word[0], and first argument word[1], and so on. See the following example.
Sending messages to channel
In the above code snippet, you will have noticed the “hexchat.command” function being called. This function sends a raw IRC command to the server.  You simply do the following at the end of your script.
this binds the command “shakespeare” to the function “cmd_insult”, and calls said function whenever the “shakespeare” command is called.
Installing and using the script
Drop it into your “addons” directory in Hexchat, and load it with “/py load shakespeare.py” or add to autoload. Reasonably painless to make work, compared to the absolute motherfuckery that was configuring irssi-python.
To use, just do /shakespeare to insult no one in particular (the open query window), or /shakespeare $nick to insult a specific user in the current channel.
Source Codes, references.
You can find the source code for the IRSSI Shakespearean Insult Script Here, and source for the Hexchat Shakespearean Insult Script Here. Hexchat and IRSSI are available at their respective websites, and the irssi-python plugin is also available. Documentation for the various clients and such are available on their respective websites.


Description: Remove the whitespace from the beginning and end of a string.
The $.trim()
function removes all newlines, spaces (including non-breaking spaces), and tabs from the beginning and end of the supplied string. If these whitespace characters occur in the middle of the string, they are preserved.
<!DOCTYPE html>

$.trim(" hello, how are you? ");
"hello, how are you?"


#{extends 'main.html' /} #{set title:'Name The Kid - About' /}
Disclaimer: The winning name may or may not be selected as the new baby's name.


Grand Theft Auto
Call Of Duty
Fifa
Hell Blade
Star Wars Battlefront
Crew
BattleField
Darkwood
Previous
Next
HOT THIS WEEK
Trending Games
...View all
TRAILERS
Indie Games


In today's world internet is the most popular way of connecting with the people. So search for films the right way.
Get started today


Jack Grealish to boost Aston Villa ahead of Monday night's Steve Bruce reunion
Villa talisman Grealish missed the defeats to Liverpool and Wolves due to injury but is set to return against his former manager when Newcastle head to Villa Park
by James Nursey, https://www.mirror.co.uk/authors/james-nursey/Jack Grealish is on course to return for Aston Villa in Monday's crunch Premier League home clash with Newcastle when former boss Steve Bruce returns.
Villa skipper Grealish has missed the club's last two top-flight games as Dean Smith's men have lost to Liverpool and Wolves .
He also missed the club's Carabao Cup win over a weakened Wolves side due to a calf injury sustained in the 3-0 loss at Manchester City last month.
It saw Grealish, 24, leave the Etihad Stadium on crutches but he is due back in training later this week after some time off in the international break.
Grealish has been in fine form this term after inspiring his boyhood club to promotion last season.
He has two goals and three assists and is the most fouled player in the top-flight ahead of Wilfried Zaha after also being singled out in the Championship last term.
But the uncapped former England U21 ace continues to be overlooked by Three Lions boss Gareth Southgate who has selected Leicester's James Maddison instead.
Villa have badly missed Grealish after slipping to 17th in the Premier League – four points below Newcastle who are bossed by Bruce.
Bruce coached Villa from October 2016–October 2018 but was sacked after being hounded out by fans who even threw a cabbage at him.
Former Villa manager Tim Sherwood reckons it is a mistake overlooking Grealish ahead of next summer's European Championships after comparing his ability to England legend Gazza.
Sherwood, in charge at Villa Park from February 2015-October 2015, said: “You have two friendlies in March but I think it may be a bit late for Jack.
“I think we've missed a trick because this kid is special - 100% they missed the boat on him.
"Jack takes the ball to the opposition, he goes right at the heart of people, he dribbles like Paul Gascoigne.”


| Interface | Description |
|---|---|
| Edm | |
| EdmAnnotatable | |
| EdmAnnotationAttribute | |
| EdmAnnotationElement | |
| EdmAnnotations | |
| EdmAssociation | |
| EdmAssociationEnd | |
| EdmAssociationSet | |
| EdmAssociationSetEnd | |
| EdmComplexType |
A CSDL ComplexType element.
|
| EdmCustomizableFeedMappings | |
| EdmElement | |
| EdmEntityContainer | |
| EdmEntitySet | |
| EdmEntitySetInfo | |
| EdmEntityType |
A CSDL EntityType element.
|
| EdmFacets | |
| EdmFunctionImport | |
| EdmMappable | |
| EdmMapping | |
| EdmNamed | |
| EdmNavigationProperty | |
| EdmParameter | |
| EdmProperty |
A CSDL Property element
|
| EdmReferentialConstraint | |
| EdmReferentialConstraintRole | |
| EdmServiceMetadata | |
| EdmSimpleType |
EdmSimpleType is a primitive type as defined in the Entity Data Model (EDM).
|
| EdmSimpleTypeFacade | |
| EdmStructuralType | |
| EdmType | |
| EdmTyped |
| Class | Description |
|---|---|
| EdmLiteral | |
| EdmTargetPath | |
| FullQualifiedName |
| Enum | Description |
|---|---|
| EdmAction | |
| EdmConcurrencyMode | |
| EdmContentKind | |
| EdmLiteralKind | |
| EdmMultiplicity | |
| EdmSimpleTypeKind | |
| EdmTypeKind |
| Exception | Description |
|---|---|
| EdmException | |
| EdmLiteralException | |
| EdmSimpleTypeException |
The Entity Data Model is described in the OData protocol specification as well as in the Conceptual Schema Definition for the OData Protocol. Both can be found at http://www.odata.org/.
Many classes and interfaces contain the pragma:" @org.apache.olingo.odata2.DoNotImplement " to indicate these classes should only be used and not implemented.
Copyright © 2013-2017 The Apache Software Foundation. All Rights Reserved.


Dutch salary calculator
Gross yearly salary
Are you entitled to the 30% rulling?
Yes
No
CALCULATE SALARY


This method retrieves the textual content of the range, considering the document content as a string
and then restricting it according to the begin and end locations.
This method retrieves the textual content of the range 1) applying the XPath expression
specified as context to the docuverse (XML) content, 2) taking all the value of all the
nodes (elements, attributes and text nodes only) returned by the expression, considering the
sequence order, and finally restrict this content using the begin and end locations.
This method retrieves the textual content of the range simply applying the XPath expression
specified as context to the docuverse (XML) content and then taking all the value of all the
nodes (elements, attributes and text nodes only) returned by the expression, considering the
sequence order.


{{exception.error.message}}
Please provide a name for this promotion.
Change Product
Any product in the cart can trigger this cross sell
Only if one of the selected products is in the cart can this cross sell be offered
Please make a selection
Please provide a discount percentage greater than 0 and less than or equal to 100.
The percentage discount to apply. For example, for a 20% discount, enter '20'.
'No' discounts the initial payment only, 'Yes' discounts the initial payment and the number of recurring payments you specify.
The number of recurring payments this promotion should apply to. If blank the promotion will apply to all recurring payments.
Please provide a sort priority for this promotion.
If multiple cross sells are available for a cart, the sort priority indicates the order in which they will be presented. A lower number puts an item closer to the top.
Please provide a weight for this promotion.
If you have multiple cross sells configured for the same product, the weight determines how likely this cross sell will be selected compared to others, allowing you to test different cross sell discounts.
Supply a date if you want the promotion to automatically expire, otherwise leave blank.
By setting as inactive, this promotion will immediately be invalidated for all customers. You can enable and disable at any time.
Comments and notes about this promotion for your reference


You can click anywhere on the screen for popunder ads.
Native Banner:
300×160px Banner:
320×50px Banner:
Direct Link
See stats


Nuclear energy is an affordable, highly dense energy source, and the world's supply of fissile material is abundant. The carbon footprint of nuclear power is much smaller than that of coal and natural gas. A major environmental concern with nuclear energy, however, is the associated hazardous radioactive waste that needs to be sequestered in a geological repository for thousands of years.
By reprocessing used nuclear fuel, we can recover fissile material for recycling-- making the most of the fuel we have-- and mitigate the volume of radioactive waste that we need to sequester. For more information, watch this video. An engineering problem associated with used nuclear fuel reprocessing is the release of radioactive gases xenon and krypton. These radioactive noble gases evolve into the air of used nuclear fuel reprocessing facilities in parts per million concentrations and must be captured.
A relatively new class of materials, metal-organic frameworks (MOFs), have nano-sized pores that serve as parking spaces for gas molecules. Depending on the chemistry of the particular MOF, the MOF may highly prefer to adsorb one gas species over another. Because of this property, MOFs hold promise to act as selective sponges and capture radioactive gases from used nuclear fuel reprocessing facilities in a room-temperature process.
The chemistry of MOFs is highly tunable, and over 10,000 different MOFs have been reported. Which of these MOFs exhibit chemistry that is optimal for capturing radioactive noble gases? Due to constrained resources and time, chemists cannot test all of these MOFs.
We used computer simulations to rapidly and cost-effectively sift through thousands of reported MOF structures and predict which MOF is the best for capturing noble gas xenon. Our models predicted that SBMOF-1, shown on the right, would be the most xenon-selective material. Our colleagues at Pacific Northwest National Lab then synthesized SBMOF-1 and measured its capability to selectively adsorb xenon over other gases. SBMOF-1 exhibits the highest xenon selectivity ever reported! Published in Nature Communications, this is a rare case of a computationally-inspired discovery of a material!
The visualization of the structure of SBMOF-1 on the right is interactive. Click and hold to grab, then drag, and/or move your mouse scroll bar to zoom in and out.


当て身 Atemi
A Cybersecurity blog by shinris3n
👊
Writeups
👊
News
👊
Resources
Part of the
Ninpwn
Network
<< home
Resources
Audio Support
Buffer Overflow Prep
John Hammond
Rickie the Pentester
The Cyber Mentor


Community Monetary System
Home
Transactions
Payment
Transactions
Exchange
P2P Credits
Endorse P2P Credit
Give access
Monitor account
Commoner
Community Service
Community Funding
Community
Commoners List
Commons Finances
Commune
Treasury
Administration
Community Service
This page only available to Commoners
Claim provided Hours:
Request PAY
Community Service Goals
Community Goal:
Delivered so far:


Go, Bokke! Check Out Komphela's Message To The Springboks
by SL SocialWith the South African national team set to play in the 2019 Rugby World Cup final this weekend, Golden Arrows coach Steve Komphela took the time to wish the Bokke well and sent them a rallying message on social media. Check it out!
Taking to Twitter, Komphela wished the Springboks well for their crunch tie against England, and in the process, rallied captain Siya Kolisi, who will be recording his 50th cap in the green and gold.
Download The Soccer Laduma Social Media App & Earn Airtime for Selfies!
Translated from Zulu to English, Komphela said: "Hey Siya, this is your 50th cap, bring it on. Hey Faf, and so it is, Mapimpi, my boy and so it is!
"Let's chase these boys! We wish you all the best! The time is now, the moment has arrived! Go Bokke!"
Check it out!
Do you think the Springboks will win the Rugby World Cup?


--- layout: page title: "HTTP Variants" description: "This specification introduces an alternative way to communicate a secondary cache key for a HTTP resource, using the HTTP \"Variants\" and \"Variant-Key\" response header fields. Its aim is to make HTTP proactive content negotiation more cache-friendly." ---
| Document Name: | Internet Draft ietf-httpbis-variants |
|---|---|
| Document URI: |
urn:ietf:id:ietf-httpbis-variants
|
| Online Version: |
https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-variants
|
| Organization: | Internet Engineering Task Force (IETF) |
| Series: | Internet Draft (I-D) |
| Abstract: | This specification introduces an alternative way to communicate a secondary cache key for a HTTP resource, using the HTTP "Variants" and "Variant-Key" response header fields. Its aim is to make HTTP proactive content negotiation more cache-friendly. |
Variant-Key
,
Variants
Return to ( Series | Organization | all Specifications )