The intestine is composed of an epithelial layer containing rapidly proliferating cells that mature into two regions, the small and the large intestine. Although previous studies have identified stem cells as the cell-of-origin for intestinal epithelial cells, no studies have directly compared stem cells derived from these anatomically distinct regions. Here, we examine intrinsic differences between primary epithelial cells isolated from human fetal small and large intestine, after in vitro expansion, using the Wnt agonist R-spondin 2. We utilized flow cytometry, fluorescence-activated cell sorting, gene expression analysis and a three-dimensional in vitro differentiation assay to characterize their stem cell properties. We identified stem cell markers that separate subpopulations of colony-forming cells in the small and large intestine and revealed important differences in differentiation, proliferation and disease pathways using gene expression analysis. Single cells from small and large intestine cultures formed organoids that reflect the distinct cellular hierarchy found in vivo and respond differently to identical exogenous cues. Our characterization identified numerous differences between small and large intestine epithelial stem cells suggesting possible connections to intestinal disease.
The intestine consists of two major subdivisions: the small intestine (SI) and the large intestine (LI), which differ in structure and function. The SI is largely responsible for the digestion and absorption of food while the LI aids in final water absorption and waste removal. Among other signaling pathways, Wnt and Notch control the well-defined epithelial hierarchy in the intestine, helping to maintain stem cell homeostasis. Since these pathways require receptors, ligands and transcriptional regulation, it is unclear whether differences observed between the SI and LI are primarily due to intrinsic or extrinsic mechanisms [ 1, 2]. Understanding these differences is crucial, since failure of intestinal stem cells to properly proliferate and differentiate may lead to cancer, which is 20 times more prevalent in the LI than the SI in humans [ 3]. However, a thorough investigation of the origin of the differences between the SI and LI has yet to be done.
The identification and characterization of stem cells in the intestine has evolved rather rapidly in recent years. In vivo lineage tracing studies have identified leucine-rich repeat-containing G-protein coupled receptor 5 (LGR5) + stem cells in the mouse as cells capable of generating all the epithelial cells of the intestine and forming crypt-like structures in vitro [ 4, 5]. Interestingly, LGR5 is intricately involved in the synergistic activation of the Wnt pathway, via the R-Spondin protein family, which is responsible for homeostatic crypt formation and maintenance in the intestine [ 6– 8]. This pathway is also commonly altered in colon cancer via mutation of adenomatous polyposis coli (APC), causing an accumulation of beta-catenin in the nucleus and enhanced Wnt signaling [ 9, 10]. Rapidly growing adenomas form in the mouse after deletion of APC in LGR5 + intestinal stem cells, suggesting that normal stem cells are the cell-of-origin of intestinal cancer [ 11]. Additionally, murine adenomas revealed continual LGR5 + stem cell activity, providing functional evidence of a cancerous stem cell population in primary intestinal adenomas [ 12].
The Wnt pathway has extensive cross-talk with the Notch pathway in its control over cell fate decisions, proliferation and tumorigenesis [ 1, 13, 14]. More specifically, activation of the Notch pathway represses secretory cell differentiation but inhibition of the Notch pathway leads to activation of atonal homolog 1 (ATOH1) promoting goblet cell differentiation ( S1 Fig.) [ 1, 15, 16]. Thus far, a majority of studies elucidating these pathways in the intestine have not made clear distinctions between the SI and LI, possibly missing differences with important consequences.
The majority of intestinal stem cell characterization has been performed in animal models because cells from normal human intestine has been notoriously difficult to grow in vitro and lineage tracing cannot be performed practically in humans. To overcome these limitations, we used feeder cells as a stromal layer to provide cell-cell interactions with human intestinal cells and promote epithelial cell growth [ 17]. Our lab has previously used this system to isolate and expand tumor cells with stem cell properties (cancer stem cells, CSCs) from human metastatic colon cancer [ 18]. Here, we isolated human fetal intestinal cells from primary tissue and expanded the cells on the feeder layer. Other in vitro models have successfully been used to study and understand stem cell biology such as the three-dimensional system introduced by Sato, et al [ 5]. Importantly, we compared cells expanded from SI and LI isolated from the same donor tissue, minimizing potential discrepancies due to genetic variability. We also expanded the SI and LI cells in identical culture conditions, to allow for the comparison of intrinsically programmed differences. The stem cell enriched cultures from human SI and LI allowed us to compare stem cell and differentiation properties to determine key differences between these two cell populations, with possible implications to disease occurrence in these two regions of the intestine.
To investigate the in vitro potential of cells isolated from the SI and LI, we used human intestinal tissues around gestational week 23. At this stage, crypts in both fetal SI and LI are structurally similar to those observed in adult, consisting of epithelial cells of all lineages, with the exception of a higher number of goblet cells in fetal tissue ( Fig. 1A). We used a feeder layer to expand cells from the SI and LI as it has previously been shown to expand human colon CSCs from metastatic colon cancer specimens [ 18]. In addition, we found that growth of normal fetal intestinal cells on the feeder layer required the addition of R-spondin 2, a Wnt agonist acting via the LGR5 receptor [ 8, 19]. This was not surprising considering that R-spondin 1 is known to be important for the in vitro expansion of intestinal cells and all four R-spondins bind the same receptor family and are expected to be somewhat redundant [ 5, 8]. In total, 29 pairs of SI and LI were isolated and processed for culture on the feeder layer. After plating ∼80,000 primary cells/cm 2, only 18 pairs formed rare colonies after approximately two weeks (62% success rate), indicating that only a small population of cells from primary tissue was capable of surviving selection under these culture conditions. Some samples were much more efficient at forming colonies than others, resulting in an estimated efficiency rate range of 0.00015–0.005%. Serial passaging led to more robust expansion of intestinal cells. Phenotypic differences in colony density, cell size and shape were observed between cells expanded from SI and LI and could be passaged at least 10 times with no apparent morphological changes or oncogenic transformation ( Fig. 1B and S1 Table). After primary cell expansion in vitro, flow cytometric analyses revealed enrichment of epithelial cells, as marked by Epithelial Cell Adhesion Molecule (EPCAM), from 21.6 to 92.3% in cells expanded from SI and from 34.3% to 92.4% in cells expanded from LI ( Fig. 1C&D). Over twenty cell-surface markers were interrogated for all SI and LI epithelial cell cultures. The most reproducible and relevant markers indicated that expanded intestinal cells expressed several markers associated with stemness. CD133.1 (AC133) has been suggested as a marker of CSCs in the intestine [ 20– 22], as well as a marker for normal stem cells such as hematopoietic stem cells [ 23] or neuronal stem cells [ 24]. While primary cells from SI and LI contained less than 1% CD133.1 + cells, approximately 70% of the expanded cells expressed CD133.1 ( Fig. 1E). Other putative stem cell markers showed similar expression profiles, including CD166 [ 25] and CD24 [ 26], both enriched in expression over primary cells, similar to CD133.1 ( Fig. 1F and data not shown) [ 18]. A previous study using human tissue from three months to eighteen years of age reported isolation, characterization and expansion of stem cell populations from single cells [ 27]. Attempts were made to expand fetal primary intestinal cells after cell sorting, but no robust long-term colonies were generated from sorted primary cells. The developmental difference in pre-natal versus post-natal and adult tissues may account for our failure to generate long-term colonies from single cells. The environment is much more dynamic and mucinous in the prenatal developmental state even though adult-like structures are present. We therefore expanded bulk intestinal epithelial cells in vitro directly from dissociated tissue prior to further characterization.
A-F, top panels: small intestine, bottom panels: large intestine ( A) Hematoxylin and eosin paraffin sections of normal fetal intestine at 23 weeks of gestation. ( B) Intestinal cells expanded on the feeder layer (yellow outline). ( C-F) Representative analyses of primary and expanded cells at passages 4, 5 or 6 with the percentage of gated live cells provided. ( C) Flow cytometric analysis (dot plots) of freshly isolated primary intestinal cells. Gating strategy identifies total live cells (left panels) and EPCAM + cells, excluding blood cells (CD45 +CD235a +) (right panels). ( D) Flow cytometric analysis of expanded cells.  ( E) Flow cytometric analysis (5% probability contour plots) of primary and expanded intestinal cells, stained with EPCAM and CD133.1. ( F) Expanded intestinal cells also express other putative stem cell markers. Histograms represent unstained control (blue) and stained with CD166 or CD24 (red) of live EPCAM + cells. Scale bars = 100μm for panel A and B.
We observed greater expression of CD133.1 in cells expanded from LI compared to SI, while CD13, an aminopeptidase responsible for hydrolyzing N-terminal amino acids at the brush border membrane, showed higher expression in the SI ( Fig. 2A). [ 28]. In addition, CD66c, also known as carcinoembryonic antigen-related cell adhesion molecule 6 (CEACAM6), was notably higher in cells expanded from SI (79.8% CD66c hi) compared to cells expanded from LI (89.4% CD66c low) ( Fig. 2A). These trends were consistent in expanded cells from three paired SI and LI samples as seen by staining index profiles ( Fig. 2B). The staining index describes the extent to which a positive population is resolved from the negative control population.
( A) Flow cytometric analysis of CD133.1, CD13 and CD66c expression on expanded cells from one paired SI and LI sample (SI and LI depicted here originated from the same tissue source). The histograms show one representative set of three paired samples examined presenting unstained control (blue) and stained live, EPCAM + cells (red). For CD66c, the percentage of low and high expressing cells is provided. All markers were PE-labeled. The same control was used for SI CD13 and CD66c as well as LI CD133 and CD13 because the staining was done at the same time. ( B) Corresponding staining index of three paired samples showing reproducible results between the paired SI and LI samples (1, 2 and 3) (staining index = D/W, where D is the distance between the positive and negative populations and W is two standard deviations of the negative population).
To demonstrate the functional importance of the differential expression of cell surface markers, we used limiting dilution analysis (LDA) to assess the clonogenic potential of epithelial cell populations derived from SI and LI. LDA is a quantitative measurement that can determine the colony-forming frequency of a specific cell population. Increased colony-forming frequency correlates with stem cell activity [ 29]. Since CD66c showed a heterogeneous and opposite expression pattern in cells expanded from SI and LI, we asked whether it could distinguish colony-forming ability within these two populations of intestinal epithelial cells. Using Fluorescence-Activated Cell Sorting (FACS), we sorted expanded cells from three tissue-matched samples of SI and LI in limiting dilution based on high and low CD66c expression. Four weeks after plating, we calculated the colony-forming unit (CFU) frequency of the cells by linear regression [ 30]. The frequency of the EPCAM + colony-forming cells from SI and LI was similar, and ranged from 1/10 to 1/106. Interestingly, cells expanded from SI that expressed low levels of CD66c had a higher CFU frequency than CD66c hi cells, while LDA results for cells expanded from LI were reversed ( Table 1). Therefore, even though a majority of cells expanded from SI are CD66c hi, as seen in the staining index ( Fig. 2B), the subpopulation of CD66c low cells are the cells with more colony-forming ability ( Table 1). The opposite is true for cells derived from LI. In consequence, the disparity in expression of CD66c can be used to enrich for expanded SI and LI cells with self-renewal potential.
| SI 1 | HLA+/EpCAM+ | 1/60 | 1/73–1/49 |
| HLA+/EpCAM+/CD66c low | 1/11 | 1/15–1/8 | |
| HLA+/EpCAM+/CD66c hi | 1/78 | 1/103–1/58 | |
| SI 2 | HLA+/EpCAM+ | 1/10 | 1/16–1/6 |
| HLA+/EpCAM+/CD66c low | 1/12 | 1/17–1/9 | |
| HLA+/EpCAM+/CD66c hi | 1/25 | 1/30–1/18 | |
| SI 3 | HLA+/EpCAM+ | 1/25 | 1/30–1/20 |
| HLA+/EpCAM+/CD66c low | 1/11 | 1/16–1/8 | |
| HLA+/EpCAM+/CD66c hi | 1/34 | 1/45–1/25 | |
| LI 1 | HLA+/EpCAM+ | 1/22 | 1/27–1/18 |
| HLA+/EpCAM+/CD66c low | 1/24 | 1/32–1/18 | |
| HLA+/EpCAM+/CD66c hi | 1/9 | 1/13–1/7 | |
| LI 2 | HLA+/EpCAM+ | 1/106 | 1/129–1/87 |
| HLA+/EpCAM+/CD66c low | 1/211 | 1/279–1/160 | |
| HLA+/EpCAM+/CD66c hi | 1/15 | 1/20–1/11 | |
| LI 3 | HLA+/EpCAM+ | 1/38 | 1/47–1/31 |
| HLA+/EpCAM+/CD66c low | 1/77 | 1/103–1/58 | |
| HLA+/EpCAM+/CD66c hi | 1/13 | 1/18–1/10 |
We showed that expanded epithelial cells from SI and LI are enriched for intestinal stem cells as evidenced by their dependence on R-Spondin 2, their expression of stem cell markers ( Fig. 1) and their high colony-forming frequency ( Table 1). We then used gene array analysis to determine if cells expanded from SI and LI demonstrated differences in their molecular signatures (NCBI Gene Expression Omnibus: GSE56525). Among the 21,238 total probes on each array, 3274 probes were significantly different. After removal of probes for the same gene, 1118 transcripts had an expression fold change greater than 1.5. Cells expanded from SI and LI clustered separately using unsupervised hierarchical clustering methods where the distances are defined linearly (left axis, Fig. 3A). Therefore, even though we were unable to distinguish subregions in SI tissue (ileum, jejunum, etc.), we did not detect any major differences or heterogeneity among the many SI samples examined. In addition, while our samples ranged in age from 20–23 weeks, this variable was deemed non-essential due to proper clustering of gene expression among the SI and LI samples.
( A) Heat map of differentially expressed transcripts of the expanded epithelial cells from SI versus LI. Unsupervised clustering was performed on log2 gene expression values (K-means algorithm) with q<0.05 (FDR = 5%) cutoff. Expression values are scaled for minimum (blue) and maximum (red) intensity values. Expanded cells formed distinct clusters based on cell origin (SI or LI). ( B-D) Histograms depict significant (q<0.05) differential expression of individual genes between cells expanded from SI and LI classified into canonical pathways using Ingenuity Pathways Analysis (see Methods). All values are reported as mean (box) plus standard deviation (error bar) of raw intensity for each transcript. List of gene names can be found in S2 Table.
We then used Ingenuity Pathway Analysis Software to predict expression changes in the dataset that relate to biological processes and disease. Functional differences in gene expression between SI and LI appeared frequently in known cancer and gastrointestinal disease pathways. The transcripts with the highest fold changes between SI and LI were related to proliferation, adhesion, migration and those that have reported functions in cancer and metastasis. Tumor-related transcripts were expressed significantly higher in cells expanded from the LI compared to SI ( Fig. 3B). We also found differences in Wnt and Notch pathway transcripts ( Fig. 3C&D). Real-time PCR of selected genes confirmed the differential expression found with the gene array analysis ( S2 Fig.). Together, expression analysis of expanded cells indicated distinct intrinsic regulation between SI and LI at the stem cell level.
Intestinal stem cells are partially defined by their multipotentiality, so we investigated the ability of expanded intestinal cells to differentiate into various intestinal cell lineages. We adapted a previously defined three-dimensional culture system to promote cell differentiation in vitro [ 5]. After expansion on the feeder layer to eliminate differentiated cells and create a more homogenous cell population, single cell suspensions were embedded in Matrigel and overlaid with media containing growth factors known to be important for the proliferation and differentiation of intestinal stem cells [ 19] (see Methods and Fig. 4A). Use of a γ-secretase inhibitor (GSI), such as N-[N-(3,5-difluorophenacetyl)-1-alanyl]-S-phenylglycine t-butyl ester (DAPT), blocks the translocation of the Notch Intracellular Domain (NICD) to the nucleus for downstream signaling, causing forced differentiation to goblet cells [ 16, 19, 31]. Expanded intestinal cells retained the ability to differentiate and form a layer of epithelial cells around a central lumen with a cellular hierarchy and architecture similar to that found in vivo ( Fig. 4B). Careful tracking of single cells showed growth into multicellular organoids, which supports the clonal origin for at least some of the SI and LI organoids ( S3 Fig.). Furthermore, we observed morphological differences between SI and LI organoids with the appearance of branching structures in the SI and spherical structures in the LI ( Fig. 4B).
( A) Experimental design of three-dimensional assay. Expanded intestinal cells were embedded as single cells in Matrigel, first expanded in Stem Cell (SC) media, then treated further with SC media, Differentiation media or Differentiation media + DAPT (GSI). ( B) In vitro and H&E pictures of SI (top) and LI (bottom) organoids in each of the three treatments. One paired set of SI and LI organoids is representative of three sets examined. Scale bar = 50μm.
Using immunofluorescent staining for differentiation markers, we analyzed SI and LI organoids for the presence of mature intestinal cells. The expression of Mucin 2 (MUC2), a marker of mature goblet cells, significantly increased upon treatment with differentiation media without and with DAPT (Diff and Diff+DAPT) in both SI and LI organoids ( Fig. 5A-C). This correlated with the appearance of vacuolated cells in both differentiation treatments in SI organoids ( Fig. 4B). SI organoids also showed increased expression of Chromogranin A (CHGA), a marker for enteroendocrine cells, upon differentiation treatment ( Fig. 5A&C). Under the same conditions, LI organoids failed to express any measurable amounts of CHGA ( Fig. 5B). A similar phenomenon was also seen in vivo where a significantly higher number of CHGA + cells was observed in SI compared to LI ( S4 Fig.). Furthermore, enterocyte expression of Villin (VIL1) in SI organoids was expressed mostly in the cytoplasm but was specifically localized on the apical cell surface in differentiated LI organoids ( S5 Fig.).
( A) SI and ( B) LI organoids from one paired sample stained for epithelial marker, EPCAM (green), goblet cell marker, MUC2 (red) and enteroendocrine marker, CHGA (red) treated with SC media, Differentiation media (Diff) or Diff media + DAPT. Counterstain, Hoechst 33342. White arrows indicate positive cells. Scale bar = 50μm. ( C) Quantification of percent goblet or enteroendocrine cells of total cells counted per field of view. Not enough enteroendocrine cells were present in the organoids derived from LI for quantification. This paired set serves as a representative of two sets examined.
Real-time PCR showed increased relative mRNA expression of ATOH1 and MUC2 in SI organoids upon differentiation treatment ( Fig. 6A). In LI organoids, MUC2 expression increased upon exposure to differentiation cues, while ATOH1 expression was negligible ( Fig. 6B). Additionally, while CHGA expression steadily increased with differentiation treatment in SI organoids, no appreciable CHGA expression was observed in LI organoids, confirming our immunofluorescence data ( Fig. 6B and Fig. 5A&B). Furthermore, as expected, the expression of stem cell marker and Wnt pathway participant LGR5 decreased in SI organoids upon differentiation treatment. Surprisingly, however, LGR5 reproducibly increased in LI organoids after the addition of Diff media, and only returned to stem cell (SC) media levels after treatment with DAPT ( Fig. 6C&D). We also found that EPH receptor B2 ( EPHB2), another putative intestinal stem cell marker [ 32], consistently increased upon Diff treatment of LI organoids. In SI organoids, EPHB2 remained unchanged in Diff media but subsequently decreased with the addition of DAPT. In addition, the MYC oncogene, a Wnt target, also decreased after exposing SI organoids to differentiation treatment, but a significant change of expression in LI organoids was not observed.
Real-time PCR analysis of ( A&C) SI and ( B&D) LI organoids derived from expanded cells and treated with SC media, Differentiation (Diff) or Diff media + DAPT. SC media treatment values used as control and set to 1. Diff media and Diff media + DAPT expression values are relative to SC media. See media key in panel A. ( E) Select marker expression comparison between SI and LI organoids in the three treatments. SI values used as control and set to 1. LI organoid expression values are relative to SI organoids. See text for full gene names. Expression was normalized to GAPDH mRNA. Error bars represent upper and lower error limits based on replicate variability (* P<0.05, ** P<0.01, *** P<0.001, # no expression detected). (n = 3 wells per sample/primer pair). One paired SI and LI set serves as a representative of all paired sets examined.
We then compared SI and LI organoids directly at each of the three treatment steps and again found undetectable expression of ATOH1 in LI organoids while SI organoids displayed substantial expression of ATOH1. LGR5 and MYC expression were significantly higher in LI organoids compared to SI organoids in all matrigel conditions ( Fig. 6E). Together, the results from the differentiation assay support the gene array analysis ( Fig. 3C&D) and demonstrate that the cells expanded from SI and LI have the stem cell property of multipotentiality but differ in their response to the same environmental cues.
In this study, we used paired sets of human fetal SI and LI to investigate the differences between the two distinct anatomic regions with potentially different stem cell populations. While others have successfully cultured organoids from mouse and/or human tissue from varying ages [ 5, 19, 27, 32], excluding 20–23 week human fetal tissue as used here, none directly compared the intrinsic differences of the SI and LI. In vivo, a combination of intrinsic and extrinsic factors determine the structural and cellular components of the SI and LI [ 33]. Here, we have removed any differences present in the SI and LI environments by providing identical growth factors and culture conditions for SI and LI cells. Therefore, the differences seen in cell lineage specification in vitro are due to differential intrinsic regulation of stem cells present in SI and LI.
We limited our investigation to human cells and by culturing bulk intestinal cells on a feeder layer with exogenous Wnt factor, R-Spondin 2, we were able to characterize expanded epithelial cells for stem cell properties such as self-renewal and mulitpotentiality without bias of pre-selecting the cells for a particular phenotype. In vivo balance of proliferation and differentiation of the intestinal epithelium, including the stem cell niche, depends on many Wnt factors [ 4, 5, 34]. Only recently was high expression of R-Spondin 2, known to enhance intestinal epithelial precursors and limit differentiation via LGR5, found in intestinal subepithelial myofibroblasts [ 35, 36]. Therefore, the R-Spondin 2 dependence of the intestinal epithelial cells expanded on feeder cells in vitro is consistent with the maintenance of intestinal stem cells in vivo.
The expected shared expression of putative stem cell markers CD133.1, CD166 and CD24 on cells expanded from the two intestinal segments suggested that they were of stem cell nature. However, differences emerged after analysis of the specific expression patterns of CD13, CD133.1 and CD66c. A higher expression of CD13 in cells expanded from SI is consistent with reports showing that it is the most abundant peptidase in mammalian SI, with CD13 expression continually decreasing along the ascending and descending LI [ 37, 38]. Furthermore, considering its relation to colon CSCs, the higher expression of CD133.1 on cells expanded from LI may indicate a predisposition of these cells to neoplastic transformation compared to cells expanded from SI [ 39]. Finally, by functionally exploiting the differential expression pattern of CD66c, we identified opposing populations of cells with colony-forming ability. High CD66c expression in stem cell enriched cultures from the LI confers increased self-renewal and, together with high expression of CD133.1, suggests an intrinsic regulatory difference for why these stem cells may be more susceptible to transformation than their SI counterparts [ 40, 41].
Furthermore, gene array analysis demonstrated global differences between cells expanded from the SI and LI. Differential expression indicated a tendency for cells expanded from the LI to be more closely related to cancer than their SI counterparts. Specifically, serologically defined colon cancer antigen 1 ( SDCCAG1) and tumor-associated calcium signal transducer 2 ( TACSTD2) are defined by their presence and association with cancer. Hypoxia inducible factor 1 ( HIF1A) functions in angiogenesis, ready to feed any new tumor that begins to form [ 42]. Prom1, also known as CD133, is a well-known CSC marker discussed previously [ 20]. The higher expression of insulin-like growth factor binding protein 3 ( IGFBP3), involved in intestinal epithelium homeostasis, may indicate a more delicate balance necessary in the LI compared to the SI [ 43]. Lastly, secreted phosphoprotein 1 ( SPP1), also known as osteopontin ( OPN), was the transcript with the highest fold change between LI and SI cells (142-fold higher in LI cells). Not only is SPP1 expression higher in colorectal cancer cell lines than normal intestinal cell lines, but it is higher in tumor tissue in comparison to normal tissue [ 44, 45]. It also correlates with lymph node metastasis, lymphatic or venous invasion and TNM stage (classification of malignant tumors). Patients with a higher expression of SPP1 had lower 5-year survival rates and it is thought to be an independent prognostic factor. Therefore, it may play a role in CRC progression and could be a prognostic marker for patients [ 45]. The increased expression of these molecules in LI cells and their associated molecular pathways not only confirm the intrinsic differences between the SI and LI, but possibly a functional reason for why there is more cancer in the LI compared to the SI. Therefore, differences at the stem cell level not only contribute to the separate structure and function of the two tissue segments, but also to their differing response to insults leading to disease.
The molecular differences identified between cells expanded from the SI and LI correlated well with the disparities in differentiation using a three-dimensional culture system. Fordham et. al used similar culture conditions for the growth of intestinal stem cells from fetal material at the gestational age of ten weeks [ 46]. However, cells from such an early stage did not require expansion prior to high efficiency formation of organoids from primary tissue, but did require additional factors to sustain cells. At 20–23 weeks, as presented here, the primary tissue structure is much more differentiated, mimicking adult organization and requiring selection and expansion on feeder cells prior to use in organoid formation assays. In addition, in the Fordham et. al study, whole epithelial units were initially embedded in their cultures, whereas in our study, single cells were plated on a feeder layer in order to eliminate the more differentiated cells present at 20–23 weeks gestational age. The feeder layer allowed us to expand and maintain the stem cell population in an undifferentiated state prior to embedding in Matrigel. This expansion allowed for a more homogenous cell population for analysis in the organoid system, without the presence of differentiated cells from primary tissue.
The initial observation of differentiation into goblet cells confirmed the multipotentiality of cells expanded from both SI and LI while demonstrating that their intrinsic differences determine unique differentiation potential. The undetectable expression of ATOH1 in LI organoids was in agreement with the gene expression profiling data, which showed that cells expanded from LI express ATOH1 at a lower level than cells expanded from SI. Interestingly, ATOH1 is required for the efficacy of GSIs [ 47], which may explain the limited MUC2 protein expression seen in DAPT treated LI organoids. Previous studies showed that 70% of colorectal cancer patient samples have significantly decreased expression of ATOH1 compared to tissue-matched normal LI samples, and revealed ATOH1 as a tumor suppressor in vitro and in vivo [ 48]. Therefore, the lack of detectable ATOH1 in cells expanded from the LI could make them more susceptible to transformation than their SI counterparts.
Lastly, the disparity in expression of stem cell related genes EPHB2, LGR5 and MYC might indicate a yet unknown molecular pathway that distinguishes LI stem cells from SI stem cells. The higher expression of LGR5 in LI organoids could mean the presence of cells with higher resistance to damage, DNA repair and growth advantage that, if transformed, could confer resistance to cancer cells [ 49, 50]. Even though LGR5 expression remains high with differentiation treatment in LI cultures, LGR5 likely remains a stem cell marker in both the SI and LI. Lgr5 + cells are present in cycling columnar cells of the mouse at the crypt base and can generate cells of all the epithelial lineages of the intestine, suggesting stem cell nature [ 4]. In addition, as a receptor for R-Spondin 2, LGR5 can mediate the stem cell niche [ 6, 8]. However, the work presented here presents a novel observation in human cells of differential regulation of LGR5 upon differentiation that may indicate a separate function of LGR5 in the LI in addition to its presence as an intestinal stem cell marker.
Functionally, LGR5 is both a receptor for Wnt ligands (R-spondins) and a transcriptional target of the Wnt pathway [ 4, 6, 8]. Here, we show that LGR5 mRNA is expressed differently between SI and LI cells, with LGR5 remaining high in LI cells after the removal of Wnt factors and even under γ-secretase inhibition ( Fig. 6D&E). This seems to be discordant with a parallel increase in goblet cells under the same conditions in LI cells after differentiation treatment ( Fig. 5A-C). Wnt and Notch signaling pathways are known to function cooperatively in intestinal epithelium, intestinal stem cells and tumor development [ 1, 13, 14]. ATOH1, a key regulator of the Notch pathway, can be marked for degradation by GSK3-β in the presence of Wnt signals and is thought to be necessary for goblet cell formation and response to γ-secretase inhibition [ 47]. Here, we show that ATOH1 was not detected in cells derived from LI, even under γ-secretase inhibition ( Fig. 6B&E), suggesting an alternative pathway in the LI. The combined result of high LGR5 coupled with the absence of ATOH1 in the LI, may indicate a possible new pathway potentially involving LGR5/Wnt and ATOH1/Notch for the regulation of differentiation in the LI.
To our knowledge, this is the first comprehensive report characterizing the differences between cells expanded from primary human SI and LI. The expected results observed from experiments in SI samples confirm that combining the expansion of an enriched stem cell population with subsequent three-dimensional differentiation assays reproducibly creates an environment similar to that found in vivo. While there is evidence of extrinsic regulation of intestinal stem cells [ 33] the unexpected results seen with the LI samples suggest key intrinsic differences between human SI and LI stem cells. These differences may account for the increased prevalence of LI cancer compared to SI cancer, as well as highlight crucial protective mechanisms present in the SI that could be harnessed to improve prevention and treatment of human LI cancer.
Mouse experiments were performed under the guidelines of the University of Pittsburgh Institutional Animal Care and Use Committee, which approved the study, and were performed in accordance with the NIH guidelines for the care and use of animals (approval no. 1006567). Human fetal intestines were obtained through Magee Women’s Hospital and the University of Pittsburgh Health Sciences Tissue Bank with approval from the University of Pittsburgh Institutional Review Board (IRB approval no. PRO10090385) after written consent.
LA7 (ATCC: CRL-2283) cells were grown in DMEM/F12 (Mediatech, VA) supplemented with 5% FBS, 1% Pen/Strep (Mediatech, VA), 50nM hydrocortisone (Sigma-Aldrich, MO) and 5μg/mL Insulin (Sigma-Aldrich, MO). Cells were detached from the plate with 0.25% Trypsin/0.1% Ethylenediaminetetraacetic acid (EDTA) (Mediatech, VA) and γ-irradiated at 17,000 rads. Cell culture flasks were seeded at ∼300,000 cells/cm 2 to generate a confluent monolayer of feeder cells prior to seeding human cells.
Small and large intestines were obtained from 19–23 week human fetuses. Intestines were cut longitudinally in HBSS, contents rinsed, cut into 1-inch pieces, transferred to EBSS/1mM EGTA/1% HEPES (Life Technologies, NY/Sigma-Aldrich, MO/Mediatech, VA) and minced. Tissue was then incubated for 5 min at room temperature. After an EBSS wash, the tissue was treated three times with a cocktail containing 1mg/mL collagenase II (Life Technologies, NY), 1mg/mL hyaluronidase (Sigma-Aldrich, MO) and 20μg/mL DNase I (Roche, IN) in HBSS/1% HEPES for 20 min. Tissue/cell suspensions were passed through a 100μm cell strainer (Fisher, PA) to isolate single cells from undigested tissue. Cells were plated on irradiated feeders at ∼80,000 cells/cm 2 in DMEM/F12 supplemented with 0.5% FBS, 25μg/mL gentamicin (Sigma-Aldrich, MO), 1% Insulin-Transferrin Selenium (ITS) (Mediatech, VA) and 0.1ng/mL human R-Spondin2 (R&D, MN). When single cells were plated, 10μM ROCK inhibitor, Y-27632 (Reagents Direct, CA), was added to the media for ∼24 hours. Cultures were passaged at 2–3 weeks post-plating (∼70% confluence) by incubating with EBSS/1mM EGTA/1% HEPES followed by 0.25% Trypsin/0.1% EDTA.
Single cell suspensions were stained with appropriate antibodies ( S3 Table) at 200,000 cells per tube and analyzed on the MACSQuant (Miltenyi Biotec, Germany) or BD FACSAriaII (BD Biosciences, MA). Singlet discrimination was performed as described by Wersto et al. [ 51]. Post-acquisition analysis was carried out in FlowJo (
The differentiation assay used in this study was modified from Sato, et al. [ 19]. Briefly, expanded intestinal cells were suspended in 50μL Growth Factor Reduced (GFR), phenol-red free Matrigel (BD Bioscience, CA) and plated in tissue culture dishes at low density (∼50,000 cells/well of 48-well plate). After solidification at 37°C for 30 min, the Matrigel was overlaid with Stem Cell media (SC) ( S4 Table). For the first 24 hours, the media included Y-27632 to prevent anoikis. After initial plating in SC media and the formation of small organoids, two-thirds of the wells were switched to Differentiation (Diff) media to allow differentiation of the organoids to occur. After further growth of the organoids, DAPT (Reagents Direct, CA), a γ-secretase inhibitor, was added to half the wells containing Diff media at 10μmol/mL for three days. Media was changed on all wells three times a week. Resulting organoids were isolated using 0.2% dispase (Life Technologies, NY), 0.1% collagenase II and 20μg/mL DNase I. Organoids were then rinsed in PBS (Mediatech, VA) for downstream applications.
Total RNA was extracted from frozen cell pellets of expanded human fetal cells from SI or LI (SI; n = 6 and LI; n = 6) ranging from 2–8 passages on the feeder cells. Expression data (q value<0.05, 2 fold change) was imported into Ingenuity software (Ingenuity Pathways Analysis, Ingenuity Systems Inc, CA) for pathway analysis. All figures are plotted using unlogged data to maintain the original biological distribution of expression. See S1 Methods for more detail.
Notch ligand, Delta/Serrate/LAG-2 (DSL), of one cell activates a Notch receptor on a neighboring cell. Notch intracellular domain (NICD) is cleaved by gamma secretase (GS) and activates Hes1. Hes1 blocks Muc2 activation via blockage of ATOH1 and the cell retains a progenitor state. In the cell where Notch is inactive, ATOH1 activates MUC2 and the cell adopts a goblet cell fate.
(TIF)
Click here for additional data file.
Selected genes were assayed using real-time PCR as confirmation of gene array results. RNA from cells expanded from three paired SI and LI samples (1, 2 and 3) were assayed for expression of six genes identified as differentially expressed in the gene array analysis. In 6/6 cases, the real-time PCR analysis confirmed the same expression pattern as the gene array analysis. SI values used as control and set to 1 in each case. LI expression values are relative to SI expression. SI 1 and 3 had no detectable expression of SPP1, therefore, SI2 was used to calculate fold change in LI 1, 2 and 3. Expression was normalized to GAPDH mRNA. Fold changes are presented above bars for clarification. Error bars represent upper and lower error limits based on replicate variability. All expression comparisons between SI and LI cells were significant ( P<0.05). (n = 3 wells per sample/primer pair).
(TIF)
Click here for additional data file.
SI or LI organoids grown in SC media, Differentiation (Diff) media and Diff media supplemented with DAPT for the last three days, followed from a single cell to multicellular organoids. Arrows and arrowheads indicate the growth of organoids from a single cell. Scale bar = 50μm.
(TIF)
Click here for additional data file.
( A) Normal human fetal SI (left) and LI (right) stained for epithelial marker, EPCAM (green), and enteroendocrine marker, CHGA (red). Counterstain, Hoechst 33342. Scale bar = 50μm. ( B) Cells were counted as cells per crypt and depicted in a bar graph (* P<0.05).
(TIF)
Click here for additional data file.
SI (left) and LI (right) organoids grown in SC media, Differentiation (Diff) media and Diff media supplemented with DAPT stained for enterocyte marker, Villin (VIL1) (red). Counterstain, Hoechst 33342. Scale bar = 50μm.
(TIF)
Click here for additional data file.
(PDF)
Click here for additional data file.
(PDF)
Click here for additional data file.
(PDF)
Click here for additional data file.
(PDF)
Click here for additional data file.
(PDF)
Click here for additional data file.
(PDF)
Click here for additional data file.
(DOCX)
Click here for additional data file.
We would like to thank Ms. Christin Sciulli for her technical expertise in processing and running the gene arrays as well as Lynda Guzik of the McGowan Institute Flow Cytometry Facility for her technical expertise in FACS.


filmcritic.com presents a review from staff member Robert Strohmeyer. You can find the review with full credits at http://www.filmcritic.com/misc/emporium.nsf/2a460f93626cd4678625624c007f2b46/688604d4ea5026b28825693800054463?OpenDocument
Long Night's Journey Into Day A film review by Robert Strohmeyer Copyright 2000 filmcritic.com
Aside from a keen enjoyment of the Special AKA song, "Free Nelson Mandela," I've never known much about South African apartheid. And, in my own defense, I don't think I'm entirely to blame for this. Sure, I've seen periodic newscasts over the course of my lifetime. Some kids in my high school used to like wearing pins denouncing apartheid and, as a general sort of thing, I opposed it too. But beyond that, it hasn't figured very strongly into my social consciousness. At least not as much as, say, the meanings of all those little emoticons people use to spice up their otherwise boring e-mails.
Frances Reid and Deborah Hoffman have made up for this, to a degree, with Long Night's Journey into Day. This potent 95-minute documentary visits the ongoing proceedings of the South African Truth and Reconciliation Commission (TRC), by which that nation administers amnesty for the horrors of apartheid crimes on a case-by-case basis. Packed start to finish with heart-wrenching interviews and bleak, sometimes gruesome, unedited footage of murdered bodies bleeding in the streets, Reid and Hoffman's film is a long-overdue eye-opener.
The picture examines four cases brought before the TRC; two in which whites were victimized, and two in which the victims were black. The sense of balance here is largely superficial because it must always be remembered that the blacks were victims before they were murdered, putting the white killers at a distinct disadvantage before the commission. Ironically, however, the film points out that the vast majority of amnesty applicants are black.
Among the subjects visited in this documentary are the parents of slain American Fullbright scholar Amy Biehl, as well as one of her killers, Mongezi Manqina, and his family. A South African security forces officer seeks amnesty for his part in the killing of the "Cradock 4," a group of black South Africans slain for alleged conspiracies against the state. And Robert McBride, another high profile figure in the later days of apartheid, faces the sister of one of his bombing victims before the commission.
What's most striking about these people is the mutual compassion they have for each other. Each of them, perpetrators and victims' families alike, seems genuinely sorrowful for the condition they share. It is refreshing and promising to see the hopeful spirit with which these people confront one another.
While it's obvious that this is only a microscopic view of the larger reconciliation process, the candid eyes of Hoffman and Reid reveal a larger picture of the South African predicament. And, like the reconciliation process itself, it is both painful and rewarding to see.
Rating: 4 stars
Directors: Frances Reid and Deborah Hoffmann Producer: Frances Reid Screenwriter: NA Stars: NA MPAA Rating: NR
http://www.amazon.com/exec/obidos/redirect?tag=filmcriticcom&path=subst/video/sellers/amazon-top-100-dvd.html Movie Fiends: Check out Amazon.com's Top 100 Hot DVDs!
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


WITNESS LIST Human Services Committee March 17, 1997-10:30A HB 606 For: Zbranek, Zeb (Author) Registering, but not testifying: For: Comer, Mary Lee (AARP) Horne, Allen (TX Hospital Assoc.) HB 690 Registering, but not testifying: On: Alexander, Cindy (Comptroller) HB 1914 For: Wise, Miguel "Mike" (Author) On: Fields, Janie (Children's Trust Fund) Winkler, Sarah (Children's Trust Fund) Registering, but not testifying: For: Kibby, Sandy (TX Congress Parents/Tea) Teltow, Wendell (Self) On: Crow, Shannon (Children's Trust Fund) HB 2123 For: Courtney, Chuck (TX Retailer's Assoc.) On: Lovel, Annette (Comptroller) HB 2126 On: Gummerman, Kent (DHS) Registering, but not testifying: For: Comer, Mary Lee (AARP)


<<< Back Submit >>>
Problem 1: Happy or Sad?
On the most recent CCC, Keenan was unsure of whether or not his score of 54 would be enough to get him into CCO. Your task is to take a list of scores and Keenan's score and see if he will get into CCO, which takes the top 20 CCC participants.
Input Specifications
The first line will contain a single integer N, the number of participants in this year's CCC.
The next line will contain N space-separated integers L, representing the scores of all participants, not including Keenan.
The last line will contain a single integer K, Keenan's score.
Output Specifications
Output will consist of a single line containing Keenan is happy!
if K is smaller than at most 19 elements of L and Keenan is sad.
otherwise.
Subtasks
For all of the points, 20 ≤ N ≤ 100 000
Sample Input
20 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 1 |
Sample Output
Sample Input
30 11 67 19 55 63 26 35 46 48 30 21 65 55 7 59 67 2 23 20 42 42 26 50 25 36 25 48 64 49 20 25 |
Sample Output


Authors: erinboyd@, screeley44@, mtanino@
This document presents a proposal for managing raw block storage in Kubernetes using the persistent volume source API as a consistent model of consumption.
By extending the API for volumes to specifically request a raw block device, we provide an explicit method for volume consumption, whereas previously any request for storage was always fulfilled with a formatted filesystem, even when the underlying storage was block. In addition, the ability to use a raw block device without a filesystem will allow Kubernetes better support of high performance applications that can utilize raw block devices directly for their storage. Block volumes are critical to applications like databases (MongoDB, Cassandra) that require consistent I/O performance and low latency. For mission critical applications, like SAP, block storage is a requirement.
For applications that use block storage natively (like MongoDB) no additional configuration is required as the mount path passed to the application provides the device which MongoDB then uses for the storage path in the configuration file (dbpath). Specific tuning for each application to achieve the highest possibly performance is provided as part of its recommended configurations.
Specific use cases around improved usage of storage consumption are included in the use cases listed below as follows:
* An admin wishes to expose a block volume to be consumed as a block volume for the user
* An admin wishes to expose a block volume to be consumed as a block volume for an administrative function such
as bootstrapping
* A user wishes to utilize block storage to fully realize the performance of an application tuned to using block devices
* A user wishes to read from a block storage device and write to a filesystem (big data analytics processing)
Also use cases include dynamically provisioning and intelligent discovery of existing devices, which this proposal sets the
foundation for more fully developing these methods.
The proposed design is based on the idea of leveraging well defined concepts for storage in Kubernetes. The consumption and definitions for the block devices will be driven through the PVC and PV definitions. Along with Storage Resource definitions, this will provide the admin with a consistent way of managing all storage. The API changes proposed in the following section are minimal with the idea of defining a volumeMode to indicate both the definition and consumption of the devices. Since it’s possible to create a volume as a block device and then later consume it by provisioning a filesystem on top, the design requires explicit intent for how the volume will be used. The additional benefit of explicitly defining how the volume is to be consumed will provide a means for indicating the method by which the device should be scrubbed when the claim is deleted, as this method will differ from a raw block device compared to a filesystem. The ownership and responsibility of defining the retention policy shall be up to the plugin method being utilized and is not covered in this proposal.
Limiting use of the volumeMode to block can be executed through the use of storage resource quotas and storageClasses defined by the administrator.
To ensure backwards compatibility and a phased transition of this feature, the consensus from the community is to intentionally disable the volumeMode: Block for both in-tree and external provisioners until a suitable implementation for provisioner versioning has been accepted and implemented in the community. In addition, in-tree provisioners should be able to gracefully ignore volumeMode API objects for plugins that haven’t been updated to accept this value.
It is important to note that when a PV is bound, it is either bound as a raw block device or formatted with a filesystem. Therefore, the PVC drives the request and intended usage of the device by specifying the volumeMode as part of the API. This design lends itself to support of dynamic provisioning by also letting the request initiate from the PVC defining the role for the PV. It also allows flexibility in the implementation and storage plugins to determine their support of this feature. Acceptable values for volumeMode are ‘Block’ and ‘Filesystem’. Where ‘Filesystem’ is the default value today and not required to be set in the PV/PVC.
In the simplest case of static provisioning, a user asks for a volumeMode of block. The binder will only bind to a PV defined with the same volumeMode.
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: myclaim
spec:
volumeMode: Block #proposed API change
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 80Gi
For dynamic provisioning and the use of the storageClass, the admin also specifically defines the intent of the volume by indicating the volumeMode as block. The provisioner for this class will validate whether or not it supports block and return an error if it does not.
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: myclaim
spec:
storageClassName: local-fast
volumeMode: Block #proposed API change
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 80Gi
For static provisioning the admin creates the volume and also is intentional about how the volume should be consumed. For backwards compatibility, the absence of volumeMode will default to filesystem which is how volumes work today, which are formatted with a filesystem depending on the plug-in chosen. Recycling will not be a supported reclaim policy as it has been deprecated. The path value in the local PV definition would be overloaded to define the path of the raw block device rather than the filesystem path.
kind: PersistentVolume
apiVersion: v1
metadata:
name: local-raw-pv
annotations:
"volume.alpha.kubernetes.io/node-affinity": '{
"requiredDuringSchedulingIgnoredDuringExecution": {
"nodeSelectorTerms": [
{ "matchExpressions": [
{ "key": "kubernetes.io/hostname",
"operator": "In",
"values": ["ip-172-18-11-174.ec2.internal"]
}
]}
]}
}'
spec:
volumeMode: Block
capacity:
storage: 10Gi
local:
path: /dev/xvdf
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
This change intentionally calls out the use of a block device (volumeDevices) rather than the mount point on a filesystem.
apiVersion: v1
kind: Pod
metadata:
name: my-db
spec:
containers:
- name: mysql
image: mysql
volumeDevices: #proposed API change
- name: my-db-data
devicePath: /dev/xvda #proposed API change
volumes:
- name: my-db-data
persistentVolumeClaim:
claimName: raw-pvc
For dynamic provisioning, it is assumed that values passed in the parameter section are opaque, thus the introduction of utilizing fstype in the StorageClass can be used by the provisioner to indicate how to create the volume. The proposal for this value is defined here: https://github.com/kubernetes/kubernetes/pull/45345 This section is provided as a general guideline, but each provisioner may implement their parameters independent of what is defined here. It is our recommendation that the volumeMode in the PVC be the guidance for the provisioner and overrides the value given in the fstype. Therefore a provisioner should be able to ignore the fstype and provision a block device if that is what the user requested via the PVC and the provisioner can support this.
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: block-volume
provisioner: kubernetes.io/scaleio
parameters:
gateway: https://<IP_ADDRESS>:443/api
system: scaleio
protectionDomain: default
storagePool: default
storageMode: ThinProvisionned
secretRef: sio-secret
readOnly: false
The provisioner (if applicable) should validate the parameters and return an error if the combination specified is not supported. This also allows the use case for leveraging a Storage Class for utilizing pre-defined static volumes. By labeling the Persistent Volumes with the Storage Class, volumes can be grouped and used according to how they are defined in the class.
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: block-volume
provisioner: no-provisioning
parameters:
DESCRIPTION: An admin wishes to pre-create a series of local raw block devices to expose as PVs for consumption. The admin wishes to specify the purpose of these devices by specifying ‘block’ as the volumeMode for the PVs.
WORKFLOW:
ADMIN:
kind: PersistentVolume
apiVersion: v1
metadata:
name: local-raw-pv
spec:
volumeMode: Block
capacity:
storage: 100Gi
local:
path: /dev/xvdc
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
DESCRIPTION: * A user uses a raw block device for database applications such as MariaDB. * User creates a persistent volume claim with “volumeMode: Block” option to bind pre-created iSCSI PV.
WORKFLOW:
ADMIN: * Admin creates a disk and exposes it to all kubelet worker nodes. (This is done by storage operation). * Admin creates an iSCSI persistent volume using storage information such as portal IP, iqn and lun.
kind: PersistentVolume
apiVersion: v1
metadata:
name: raw-pv
spec:
volumeMode: Block
capacity:
storage: 100Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
iscsi:
targetPortal: <IP_ADDRESS>:3260
iqn: iqn.2017-05.com.example:test
lun: 0
USER:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: raw-pvc
spec:
volumeMode: Block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 80Gi
apiVersion: v1
kind: Pod
metadata:
name: my-db
spec:
containers
- namee: mysql
image: mysql
volumeDevices:
- name: my-db-data
devicePath: /dev/xvda
volumes:
- name: my-db-data
persistentVolumeClaim:
claimName: raw-pvc
DESCRIPTION:
A developer wishes to enable their application to use a local raw block device as the volume for the container. The admin has already created PVs that the user will bind to by specifying ‘block’ as the volume type of their PVC.
BACKGROUND:
For example, an admin has already created the devices locally and wishes to expose them to the user in a consistent manner through the Persistent Volume API.
WORKFLOW:
USER:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: local-raw-pvc
spec:
volumeMode: Block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 80Gi
DESCRIPTION: StorageClass with non-dynamically created volumes
BACKGROUND: The admin wishes to create a storage class that will identify pre-provisioned block PVs based on a user’s PVC request for volumeMode: Block.
WORKFLOW:
ADMIN:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: block-volume
provisioner: no-provisioning
parameters:
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-block-volume
spec:
volumeMode: Block
storageClassName: block-volume
capacity:
storage: 35Gi
accessModes:
- ReadWriteOnce
local:
path: /dev/xvdc
DESCRIPTION: StorageClass with dynamically created volumes
BACKGROUND: The admin wishes to create a storage class that will dynamically create block PVs based on a user’s PVC request for volumeMode: Block. The admin desires the volumes be created dynamically and deleted when the PV definition is deleted.
WORKFLOW:
ADMIN:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: local-fast
provisioner: kubernetes.io/local-block-ssd
parameters:
DESCRIPTION: The developer wishes to request a block device via a Storage Class.
WORKFLOW:
USER:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-local-block
spec:
volumeMode: Block
storageClassName: local-fast
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
Since the PVC object is passed to the provisioner, it will be responsible for validating and handling whether or not it supports the volumeMode being passed
DESCRIPTION: Admin creates network raw block devices
BACKGROUND: Admin wishes to pre-create Persistent Volumes in GCE as raw block devices
WORKFLOW:
ADMIN:
apiVersion: "v1"
kind: "PersistentVolume"
metadata:
name: gce-disk-1
Spec:
volumeMode: Block
capacity:
storage: "10Gi"
accessModes:
- "ReadWriteOnce"
gcePersistentDisk:
pdName: "gce-disk-1"
DESCRIPTION: * A user uses a raw block device for database applications such as mysql to read data from and write the results to a disk that has a formatted filesystem to be displayed via nginx web server.
ADMIN: * Admin creates a 2 block devices and formats one with a filesystem
kind: PersistentVolume
apiVersion: v1
metadata:
name: raw-pv
spec:
volumeMode: Block
capacity:
storage: 100Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
gcePersistentDisk:
pdName: "gce-disk-1"
kind: PersistentVolume
apiVersion: v1
metadata:
name: gluster-pv
spec:
volumeMode: Filesystem
capacity:
storage: 100Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Delete
glusterfs:
endpoints: glusterfs-cluster
path: glusterVol
USER:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: raw-pvc
spec:
volumeMode: Block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 80Gi
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: gluster-pvc
spec:
volumeMode: Filesystem
accessModes:
- ReadWriteMany
resources:
requests:
storage: 50Gi
apiVersion: v1
kind: Pod
metadata:
name: my-db
spec:
volumes:
- name: my-db-data
persistentVolumeClaim:
claimName: raw-pvc
- name: my-nginx-data
persistentVolumeClaim:
claimName: gluster-pvc
containers
- name: mysql
image: mysql
volumeDevices:
- name: my-db-data
devicePath: /var/lib/mysql/data
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- mountPath: /usr/share/nginx/html
name: my-nginx-data
readOnly: false
DESCRIPTION: * A user wishes to read data from a read-only raw block device, an example might be a database for analytics processing.
USER: * User creates pod and specifies ‘readOnly’ as a parameter in the persistent volume claim to indicate they would like to be bound to a PV with this setting enabled.
apiVersion: v1
kind: Pod
metadata:
name: nginx-pod-block-001
spec:
containers:
- name: nginx-container
image: nginx:latest
ports:
- containerPort: 80
volumeDevices:
- name: data
devicePath: /dev/xvda
volumes:
- name: data
persistentVolumeClaim:
claimName: block-pvc001
readOnly: true #flag indicating read-only for container runtime
Note: the readOnly field already exists in the PersistentVolumeClaimVolumeSource above and will dictate the values set by the container runtime options
It is important the values that are passed to the container runtimes are valid and support the current implementation of these various runtimes. Listed below are a table of various runtime and the mapping of their values to what is passed from the kubelet.
| runtime engine | runtime options | accessMode |
| ————– |:—————-:| —————-:|
| docker/runc/rkt | mknod / RWM | RWO |
| docker/runc/rkt | R | ROX |
The accessModes would be passed as part of the options array and would need validate against the specific runtime engine. Since rkt doesn’t use the CRI, the config values would need to be passed in the legacy method. Note: the container runtime doesn’t require a privileged pod to enable the device as RWX (RMW), but still requires privileges to mount as is consistent with the filesystem implementation today.
The runtime option would be placed in the DeviceInfo as such: devices = append(devices, kubecontainer.DeviceInfo{PathOnHost: path, PathInContainer: path, Permissions: “XXX”})
The implementation plan would be to rename the current makeDevices to makeGPUDevices and create a separate function to add the raw block devices to the option array to be passed to the container runtime. This would iterate on the paths passed in for the pod/container.
Since the future of this in Kubernetes for GPUs and other plug-able devices is migrating to a device plugin architecture, there are still differentiating components of storage that are enough to not to enforce alignment to their convention. Two factors when considering the usage of device plugins center around discoverability and topology of devices. Since neither of these are requirements for using raw block devices, the legacy method of populating the devices and appending it to the device array is sufficient.
// BlockVolume interface provides methods to generate global map path
// and pod device map path.
type BlockVolume interface {
// GetGlobalMapPath returns a global map path which contains
// symbolic links associated to a block device.
// ex. plugins/kubernetes.io/{PluginName}/{DefaultKubeletVolumeDevicesDirName}/{volumePluginDependentPath}/{pod uuid}
GetGlobalMapPath(spec *Spec) (string, error)
// GetPodDeviceMapPath returns a pod device map path
// and name of a symbolic link associated to a block device.
// ex. pods/{podUid}}/{DefaultKubeletVolumeDevicesDirName}/{escapeQualifiedPluginName}/{volumeName}
GetPodDeviceMapPath() (string, string)
}
// BlockVolumePlugin is an extend interface of VolumePlugin and is used for block volumes support.
type BlockVolumePlugin interface {
VolumePlugin
// NewBlockVolumeMapper creates a new volume.BlockVolumeMapper from an API specification.
// - spec: The v1.Volume spec
// - pod: The enclosing pod
NewBlockVolumeMapper(spec *Spec, podRef *v1.Pod, opts VolumeOptions) (BlockVolumeMapper, error)
// NewBlockVolumeUnmapper creates a new volume.BlockVolumeUnmapper from recoverable state.
// - name: The volume name, as per the v1.Volume spec.
// - podUID: The UID of the enclosing pod
NewBlockVolumeUnmapper(name string, podUID types.UID) (BlockVolumeUnmapper, error)
// ConstructBlockVolumeSpec constructs a volume spec based on the given
// pod name, volume name and a pod device map path.
// The spec may have incomplete information due to limited information
// from input. This function is used by volume manager to reconstruct
// volume spec by reading the volume directories from disk.
ConstructBlockVolumeSpec(podUID types.UID, volumeName, mountPath string) (*Spec, error)
}
// BlockVolumeMapper interface provides methods to set up/map the volume.
type BlockVolumeMapper interface {
BlockVolume
// SetUpDevice prepares the volume to a self-determined directory path,
// which may or may not exist yet and returns combination of physical
// device path of a block volume and error.
// If the plugin is non-attachable, it should prepare the device
// in /dev/ (or where appropriate) and return unique device path.
// Unique device path across kubelet node reboot is required to avoid
// unexpected block volume destruction.
// If the plugin is attachable, it should not do anything here,
// just return empty string for device path.
// Instead, attachable plugin have to return unique device path
// at attacher.Attach() and attacher.WaitForAttach().
// This may be called more than once, so implementations must be idempotent.
SetUpDevice() (string, error)
}
// BlockVolumeUnmapper interface provides methods to cleanup/unmap the volumes.
type BlockVolumeUnmapper interface {
BlockVolume
// TearDownDevice removes traces of the SetUpDevice procedure under
// a self-determined directory.
// If the plugin is non-attachable, this method detaches the volume
// from devicePath on kubelet node.
TearDownDevice(mapPath string, devicePath string) error
}
Currently, a volume which has filesystem is mounted to the following two paths on a kubelet node when the volumes is in-use. The purpose of those mount points are that Kubernetes manages volume attach/detach status using these mount points and number of references to these mount points.
- Global mount path
/var/lib/kubelet/plugins/kubernetes.io/{pluginName}/{volumePluginDependentPath}/
- Volume mount path
/var/lib/kubelet/pods/{podUID}/volumes/{escapeQualifiedPluginName}/{volumeName}/
Even if the volumeMode is “Block”, similar scheme is needed. However, the volume which
doesn’t have filesystem can’t be mounted.
Therefore, instead of volume mount, we use symbolic link which is associated to raw block device.
Kubelet creates a new symbolic link under the new global map path
and pod device map path
.
Kubelet creates a new symbolic link under the new global map path when volume is attached to a Pod. Number of symbolic links are equal to the number of Pods which use the same volume. Kubelet needs to manage both creation and deletion of symbolic links under the global map path. The name of the symbolic link is same as pod uuid. There are two usages of Global map path.
{volumePluginDependentPath}
during Block volume reconstruction
/var/lib/kubelet/plugins/kubernetes.io/{pluginName}/volumeDevices/{volumePluginDependentPath}/{pod uuid1}
/var/lib/kubelet/plugins/kubernetes.io/{pluginName}/volumeDevices/{volumePluginDependentPath}/{pod uuid2}
...
FC plugin: {wwn}-lun-{lun} or {wwid}
ex. /var/lib/kubelet/plugins/kubernetes.io/fc/volumeDevices/500a0982991b8dc5-lun-0/f527ca5b-6d87-11e5-aa7e-080027ff6387
iSCSI plugin: {portal ip}-{iqn}-lun-{lun}
ex. /var/lib/kubelet/plugins/kubernetes.io/iscsi/volumeDevices/<IP_ADDRESS>:3260-iqn.2001-04.com.example:storage.kube.sys1.xyz-lun-1/f527ca5b-6d87-11e5-aa7e-080027ff6387
Kubelet creates a symbolic link under the new pod device map path. The file of {volumeName} is symbolic link and the link is associated to raw block device. If a Pod has multiple block volumes, multiple symbolic links under the pod device map path will be created with each volume name. The usage of pod device map path is;
Container initialization
and Block volume reconstruction
/var/lib/kubelet/pods/{podUID}/volumeDevices/{escapeQualifiedPluginName}/{volumeName1}
/var/lib/kubelet/pods/{podUID}/volumeDevices/{escapeQualifiedPluginName}/{volumeName2}
...
| PV volumeMode | PVC volumeMode | Result |
|---|---|---|
| unspecified | unspecified | BIND |
| unspecified | Block | NO BIND |
| unspecified | Filesystem | BIND |
| Block | unspecified | NO BIND |
| Block | Block | BIND |
| Block | Filesystem | NO BIND |
| Filesystem | Filesystem | BIND |
| Filesystem | Block | NO BIND |
| Filesystem | unspecified | BIND |
Using dynamic provisioning, user is able to create block volume via provisioners. Currently,
we have two types of provisioners, internal provisioner and external provisioner.
During volume creation via dynamic provisioner, user passes persistent volume claim which
contains volumeMode
parameter, then the persistent volume claim object is passed to
provisioners. Therefore, in order to create block volume, provisioners need to support
volumeMode
and then create persistent volume with volumeMode
.
If a storage and plugin don’t have an ability to create raw block type of volume,
then both internal and external provisioner don't need any update
to support volumeMode
because volumeMode
in PV and PVC are automatically set to Filesystem
as a default when
these volume object are created.
However, there is a case that use specifies volumeMode
as Block
even if both plugin and
provisioner don’t support. As a result, PVC will be created, PV will be provisioned
but both of them will stuck Pending status since volumeMode
between them don’t match.
For this situation, we will add error propagation into persistent volume controller to make
it more clear to the user what’s wrong.
If admin provides external provisioner to provision both filesystem and block volume, admin have to carefully prepare Kubernetes environment for their users because both Kubernetes itself and external provisioner have to support block volume functionality. This means Kubernetes v1.9 or later must be used to provide block volume with external provisioner which supports block volume.
Regardless of the volumeMode, provisioner can set FSType
into the plugin’s volumeSource
but the value will be ignored at the volume plugin side if volumeMode
is Block
.
If internal plugin has own provisioner, the plugin needs to support volumeMode
to provision
block volume. This is the example implementation of volumeMode
support for GCE PD plugin.
// Obtain volumeMode from PVC Spec VolumeMode
var volumeMode v1.PersistentVolumeMode
if options.PVC.Spec.VolumeMode != nil {
volumeMode = *options.PVC.Spec.VolumeMode
}
// Set volumeMode into PersistentVolumeSpec
pv := &v1.PersistentVolume{
Spec: v1.PersistentVolumeSpec{
VolumeMode: &volumeMode,
PersistentVolumeSource: v1.PersistentVolumeSource{
GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{
PDName: options.Parameters["pdName"],
FSType: options.Parameters["fsType"],
...
},
},
},
}
We have a “protocol” to allow dynamic provisioning by external software called external provisioner.
In order to support block volume via external provisioner, external provisioner needs to support
volumeMode
and then create persistent volume with volumeMode
. This is the example implementation
of volumeMode
support for external provisioner of Local volume plugin.
// Obtain volumeMode from PVC Spec VolumeMode
var volumeMode v1.PersistentVolumeMode
if options.PVC.Spec.VolumeMode != nil {
volumeMode = *options.PVC.Spec.VolumeMode
}
// Set volumeMode into PersistentVolumeSpec
pv := &v1.PersistentVolume{
Spec: v1.PersistentVolumeSpec{
VolumeMode: &volumeMode,
PersistentVolumeSource: v1.PersistentVolumeSource{
Local: &v1.LocalVolumeSource{
Path: options.Parameters["Path"],
},
},
},
}
The value used for the plugin to indicate is it provisioning block will be plugin dependent and is an opaque parameter. Binding will also be plugin dependent and must handle the parameter being passed and indicate whether or not it supports block.
Phase 1: v1.9 Feature: Pre-provisioned PVs to precreated devices
Milestone 1: API changes
Milestone 2: Restricted Access
Milestone 3: Changes to the mounter interface as today it is assumed 'file' as the default.
Milestone 4: Expose volumeMode to users via kubectl
Milestone 5: PV controller binding changes for block devices
Milestone 6: Container Runtime changes
Milestone 7: Initial Plugin changes (FC & Local storage)
Phase 2: v1.10 Feature: Discovery of block devices
Milestone 1: Dynamically provisioned PVs to dynamically allocated devices
Milestone 2: Plugin changes with dynamic provisioning support (RBD, iSCSI, GCE, AWS & GlusterFS)


|
|
Unified
Hosts Packages
Below are reformatted hosts packages for Steven Black's
hosts repo.
 The
other files add onto the core Unified Hosts additional blocks.
These hosts packages are updated regularly with Steven Black's hosts
files and are reformatted here for the purposes of improving
compatibility. The
directory structure and naming standards are identical to that of
Steven Black's repo to ensure the widest range of compatibility across
projects. The ZIP links in this repo are perfect for older legacy hosts
file managing
software which require hosted archives to pull from as sources. The
hosts files in this repo have also been normalized to
CRLF to make them more user-friendly for Windows users, while also
being just as compatible for any other system, such as Linux, etc.
If you run into any issues using these, you can also try the
"compressed" format on the Additional
Blacklist Support page.
If
you are running a Windows operating system and would like to keep your
machine up to date automatically with the latest hosts files from
Steven Black, you can also try the Unified Hosts AutoUpdate.
| Hosts
File Type |
Download
|
| Unified
hosts
= (adware + malware) |
Raw
| ZIP
|
| Unified
hosts + fakenews |
Raw
| ZIP
|
| Unified
hosts + gambling |
Raw
| ZIP
|
| Unified
hosts + porn |
Raw
| ZIP
|
| Unified
hosts + social |
Raw
| ZIP
|
| Unified
hosts +
fakenews +
gambling |
Raw
| ZIP |
| Unified
hosts +
fakenews +
porn |
Raw
| ZIP |
| Unified
hosts +
fakenews +
social |
Raw
| ZIP |
| Unified
hosts +
gambling +
porn |
Raw
| ZIP |
| Unified
hosts +
gambling +
social |
Raw
| ZIP |
| Unified
hosts +
porn
+
social |
Raw
| ZIP |
| Unified
hosts +
fakenews +
gambling + porn |
Raw
| ZIP |
| Unified
hosts +
fakenews +
gambling + social |
Raw
| ZIP |
| Unified
hosts +
fakenews +
porn + social |
Raw
| ZIP |
| Unified
hosts +
gambling +
porn + social |
Raw
| ZIP |
| Unified
hosts +
fakenews +
gambling + porn + social |
Raw
| ZIP |
Have questions or comments about the content on this page? ScriptTiger
would love to hear them! Open a new issue in ScriptTiger's
GitHub scripttiger.github.io repository to have your voice
heard! |
|
|


Guidelines to plan tasks into packages on Big Data Processor¶
We suggest the following tips to have your scripts become a protable form.
1. Do expose all output paths as input arguments.¶
All output files/folders are specified through arguments when executing your script/program. Namely, Your scripts/programs should accepts arguments of all output file/folder paths.
Tip
A task may generate many output files. In this case, we can plan to generate a single output folder for this task, and all output files with any complex file structure can then be organized in this folder.
Tip
Intermediate outputs of a workflow are recommended to be generated inside an output folder. When this output folder is set as an argument, users can directly examine these intermediate outputs on Bio-Data Processor.
Warning
Bio-Data Processor does not tightly control the file outputs to maintain light weighted. Developers are suggested to know where the output files are (see the third suggested point below). If the outputs are created in the current working directory
, we provided web interface for you to set the cwd
property for that task. The default cwd
is the $ProjectFolder
that specifies the project folder path.
2. For input/output files, accepts the absolute full path(s) as arguments.¶
It is recommended for a task to accept absolute paths as input/output file arguments.
Tip
If a task only accepts the output file/folder that is relative to the current working directory (e.g. output prefix), you may consider setting the task current working directory as the $ProjectFolder
on the Bio-Data Processor and setting the argument to use only the output file/folder name only instead of the full path.
Tip
If a (batch) task needs many input files, try to plan the task that takes a single input folder as an argument that contains all the input files.
3. Make sure to follow symbolic links¶
If your scripts accepts symbolic links as an input argument, remember to follow the symbolic link (soft link) when this folder is a soft-link to see underlying files.
4. Receive SIGTERM or SIGINT signal for gracefully stop processes¶
This is usually the default behavior for a script/program to stop. If your scripts/programs intercept these two signals, please make sure your task will stop itself gracefully, e.g. removing all intermediate files or stop all child processes.
5. Construct a container image for your scripts to run.¶
We strongly encourage you to plan all your tasks running inside Docker or Signularity containers. In this way, these tasks are cross-platformed and users need not worry about the task running environments. In the BDP, the container images are suggested to contain only the dependent libraries or packages, so that you can edit and then directly test your scripts on the BDP. Also, you do not have to build a new image whenever you modify your scripts. Your scripts are mounted to the container at run time.
Tip
Many Task Adapters of BDP can be developed to deploy containerized tasks on different environments. (e.g. a server with a cluster architecture with SGE/PBS, cloud computing (Google Cloud/Microsoft Azure/Amazon web service, etc.), with CWL or WDL, or simply on a single workstation.)
6. Do NOT handle batch tasks by yourself¶
Some tasks may process a bunch of files with same arguments. This is called a batch task. Most of the time, batch tasks are considered as independent tasks and can be executed parallel. If you hard-coded the for-loop
inside your script, these tasks can only be executed one by one. The Workflow Playbook
help you to configure this kind of batch tasks. You may loop over some kinds of lists, e.g. file globs, list from csv or excel files. Developers should focus on building each task unit and need NOT worry about the parallel task executions, since there may be different kinds of task parallelism or no parallel environments at run time.
Different kinds of adapters can handle the task parallelisms for you. For each task, developers configure ONE single task definition and the task can then be deployed on different run-time environments, e.g. a local workstation, High-Performance computing (HPC) clusters, cloud computings.
See also
Please click here to see how to define a batch task with the introduction of Workflow Playbook.
7. Be careful when outputting files in input folders¶
Usually, we do NOT recommended writing files in input folders, since the file writing conflicts may occur. That is, multiple process may write data into the same file. Usually, the OS will lock the file for the first process, and other process will not have permissions to write data. Hence, the task will fail.
Bio-Data Processor allows writing files into input folders because the output details are not tightly controled. We preserve the most flexible circumstances to run a task. However, be sure to create unique files under the input folders.
Tip
Please write output or temp files in the output files. If the intermediate temp files are not so important, you may consider to write temp files into the /tmp/
folder. Each task is independent in a docker container. Namely, the /tmp
is not shared among different processes. In this way, you can write the temp file using the same name.


{{cooking.user.name}}
{{cooking.created | date:'medium'}}
{{cooking.title}}
{{cooking.description}}
Millions of amazing sites across the web are being built with Bootstrap. Get started on your own with our growing collection of examples or by exploring some of our favorites.
email title
email address
location title
location content


Virgin Gorda, or V.G. as calle by the locals, is the third largest island in the B.V.I. It was named "Fat Virgin,” by Christoper Columbus.
You can learn more about Virgin Gorda by visiting the B.V.I. Tourists Board's website.


Hi,
I would like to connect an EmonTx v3 to RaspberryPI via serial. I found instructions here but they refer to the Rock-solid-gateway forwarder image. In my case, I would like to the last ready to go image (emonSD-13-08-14.img) that include Emonhub.
I guess that the instruction are sill valid for the emonhub config file and the inittab file. But what should I do regarding the comandline.txt file that is really different ?
Can someone have a look on this ?
Thank's
Eric
Re: Serial link RPI<->EMonTX using Emonhub (emonSD-13-08-14.img)
Hi Eric
The cmdline.txt and inittab ashould already be done on the SD image. These 2 changes are done to any Pi running Raspbian OS that has any software that needs to access the serial port as by default Raspbian uses the serial port for a console. The ' # ' at the begining of the last line in /etc/inittab and ensuring the line in /boot/cmdline.txt does not contain " console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 " free's the serial port for other uses reagrdless of if that's a RFM2Pi or a direct serial connection.
Your cmdline.txt should probably read " dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait " if running the SD image.
In the emonhub.conf you will need to disable the RFM2Pi interfacer, which can easily be done by adding a " # " to the beginning of the "Type" line, this will effectively "comment out" the type and emonHub will not create interfacers that do not specify a type. so that line should be edited so it's
and to create a serial interfacer add a new set of settings to the [interfacers] section where the RFM2Pi settings are something like this
Changing the com_baud as required and adding any datacode etc to the runtimesetting. The 2 ....settings headings must be present even if there are no specific settings.
Paul
Re: Serial link RPI<->EMonTX using Emonhub (emonSD-13-08-14.img)
Thank's Paul !
I will give a try in the coming days.
Eric
Re: Serial link RPI<->EMonTX using Emonhub (emonSD-13-08-14.img)
Hi,
I forgot to reply that it works fine with an EmonTX V3 after modifying the emonTxV3_DirectSerial sketch. As explained also here, the nodeID is missing in that sketch. It makes emonhub suddenly fail (because the first value is a float ? Need to be fixed ?)
Just add Serial.print("10 "); on line 68 of that sketch to set the nodeID to 10.
As explained by Paul in the other thread, no need to modify the datacode section of emonhub
Eric
Re: Serial link RPI<->EMonTX using Emonhub (emonSD-13-08-14.img)
Hi Eric, thanks for letting us know.
With regard to the node id I suppose the 2 trains of thought would be
A) Since under normal circumstances there can be only one device communicating over serial there should be no need for a node id to know where the data come from, however since emonCMS expects a node id and there may well be other nodes connected to same emonhub. A node id would need to be added by emonHub before forwarding to emonCMS, which I guess could be a user setting in emonhub.conf, thus removing the need to edit the sketch to change/add a node id.
B) Adding a node id to the sketch would keep it in line with other OEM payloads so that emonHub etc can process the payload in the same way, this way also has flexibility for a serial connected device to use more than 1 node id for example if it is itself a interface for one or more devices (eg rs485)
Maybe a optional "nodeid" setting in emonhub could work. if a "nodeid" is specified in the serialinterfacer runtimesettings emonhub would use that and assume all serial data is values, alternatively if the user does not specify a "nodeid" it uses the first received byte as the node id. How does that sound ?
Paul
Re: Serial link RPI<->EMonTX using Emonhub (emonSD-13-08-14.img)
Hi Paul,
IMHO, adding a node id to the serial sketch is the best solution. No need to change anything in Emonhub, nor in EmonCMS. Keep it simple !
Did you notice that EMonhub fails whithout error message if the first value of the frame is a float ? Can you make emonhub more robust and/or more explicit.
Eric
Re: Serial link RPI<->EMonTX using Emonhub (emonSD-13-08-14.img)
Hi Eric, I wasn't aware of the float causing a failure, thanks for the heads up. I will certainly look into it, however there are some changes on the horizon (for other reasons) that will mean rewriting this part so I'm not overly concerned right now but will aim to make it as robust as possible.
You're probably right about adding the node id to the sketch being the easier, that was my preference too. I won't totally rule out the "optional node id" just yet though as there are other instances where this could be useful too. For example I have a "weather data" interfacer that requires a node id to be set and when I get around to it the "system data" interfacer inspired by your php script will need one too.
Paul
Re: Serial link RPI<->EMonTX using Emonhub (emonSD-13-08-14.img)
I wondered why I hadn't experienced this myself, so I dug out a couple of my old "serial output" sketches to look at and found in the past I have done things a little differently, When I have converted a sketch for serial output, I have added a function to print the "emontx" typedef struct so that the serial output was identical to what the RFM2Pi would normally output, ie a string of byte values to be reconstructed into real values.
I have added a test for an int between 0 and 31 for to first value in the payload, this will cause emonhub to discard the payload if the first value isn't valid as a node id, This should prevent emonhub failing and output a warning.
https://github.com/emonhub/emonhub/commit/366b3dbb3c2870ef1a2beefcff440e9e6685f98c
Paul


{% extends "base_statdev.html" %} {% load crispy_forms_tags %} {% block page_content_inner %} Back to Application
Please provide any required feedback related to this application (in addition to already-proposed conditions):
{% crispy form %}| Name | Sent date | Period (days) | Status | Details | Action |
|---|---|---|---|---|---|
| {{ refer_obj.referee }} | {{ refer_obj.sent_date|date:"d-M-Y" }} | {{ refer_obj.period }} | {{ refer_obj.get_status_display }} | {{ refer_obj.details }} | {% if may_referral_delete == "True" %} Delete {% endif %} |


--- layout: post title: replacing content re-use with source control file syncing date: 2015-01-13 13:48:16.000000000 -08:00 categories: - technical-writing tags: [] status: draft type: post published: false ---
In a recent podcast with Joe Malin, Joe said that source control repositories that are able to keep local files in sync with each other is one of the greatest inventions of this century.
[excerpt quote]
This is a really interesting observation. Today I was struggling with a glossary issue and suddenly realized why this these repositories are so ingenius.
Consider this scenario. You have 4 separate doc sets for similar products (Product A, Product B, Product C, and Product D). You have a glossary that you want to share for all four projects. How do you do it?
Approach one. You could try to reference the files using DITA. You could have a folder structure that looks like the following:
To refer to the glossary topic, you would need to write the reference like this:
And also be sure you include the resource on your ditamap:
However, you'll get the following error when you do this:
The requested URL /resolve/glossary/division.html was not found on this server.
I don't know if the DITA spec ever mentions this, but DITA has problems with file references outside its root directory. (Subfolders are okay.)
There's a parameter in the webhelp transform that gets around this problem:
fix.external.refs.com.oxygenxml
The DITA Open Toolkit usually has problems processing references which point to locations which are outside of the processed DITA Map directory.
This parameter is used to specify whether the application should try to fix up such references in a temporary files folder before the DITA Open Toolkit is invoked on the fixed references.
The fix up has no impact on your edited DITA content.
Only "false" and "true" are valid values. The default is false.
And then the reference to the glossary term works fine.
Approach two. Use file repository syncing to keep the terms in sync.
Create a central repository for the glossary terms, and then clone the repository inside each of the four projects, like this:
project_a
When you want to upate your glossary clone, you browse to the folder location on your command line and enter "hg pull". It syncs all the files.
Now your references in your DITA map just look like this:
and
With the first approach, you have one immutable instance of the content. With the second approach, the glossary clones can be the same, but they can also be forks from the original repository. People can fork the glossary repository and create their own specialized glossaries from that base.
The idea of forking follows common programming practices of inheritance and subclasses. In Java you often have a class that looks like this:
Class Cat extends Animal
Here Cat receives all of the fields and methods of Animal, but it can overwrite them, add to them, and more.
There's a similar concept with "interfaces" in Java. The interface provides blank methods and field names, and then each class that implements the interface basically fills in the blanks for what the fields and methods do (but they can't change the names.)
Of course this is all just the concept of a template. Why start from scratch each time? When you create a resume, you often begin from a template. You have a general resume template that you use to specialize/extend/fork with your own resume.
(Programmers also have classes that can't be overwritten. They're called abstract classes.)
What's interesting to me is how source control syncing essentially duplicates the idea of content re-use. Instead of re-using the same topic across multiple projects, you could merely sync a repository with a central repository and achieve the same effect, with added benefits.
Let's approach doc re-use from another perspective. Let's say you're not using DITA. You're using a static site generator like Jekyll. You have 4 projects, and you're reusing topics 1 and 2 across the 4 projects.
reused topics repo
- topic 1
- topic 2
Here topics 77, 45, 83, and 29 are unique, but topic 1 and 2 are the same.


Name: {{ book_form.name }}
Author: {{ book_form.author }}
Publisher: {{ book_form.publisher }}
Pages: {{ book_form.pages }}
Price: {{ book_form.price }}
Description: {{ book_form.description }}


Toggle navigation
tapsie
Home
About
Menu
Appetizers
Main Course
Desserts
Drinks
Specials
Lunch Buffet
Weekend Brunch
Contact
texto 1
texto 2
texto 3
texto 4
texto 5
texto 6
More on Tapsie
Cooking
Lifestyle
DIY Projects
Follow us on Intragram
Featured in Cooking
see all →
Cooking
Cooking
Cooking
Cooking
Featured in Lifestyle
see all →
Cooking
Cooking
Cooking
Cooking
Featured in DIY Project
see all →
Cooking
Cooking
Cooking
Cooking


More...
|
| bool | HasExpired (DateTime offset) |
| | Check the session has expired against the offset time.- Parameters
-
| offset | The datetime to compare against this session. |
- Returns
- If the session has expired.
|
| |
| bool | HasRefreshExpired (DateTime offset) |
| | Check if the refresh token has expired against the offset time.- Parameters
-
| offset | The datetime to compare against this refresh token. |
- Returns
- If refresh token has expired.
|
| |
| override string | ToString () |
| |
| bool | HasExpired (DateTime offset) |
| | Check the session has expired against the offset time.
|
| |
| bool | HasRefreshExpired (DateTime offset) |
| | Check if the refresh token has expired against the offset time.
|
| |
|
| static ISession | Restore (string authToken, string refreshToken=null) |
| | Restore a session from the auth token.
|
| |
|
| static readonly DateTime | Epoch = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc) |
| |
|
| string | AuthToken [get] |
| | The authorization token used to construct this session.
|
| |
| long | ExpireTime [get] |
| | The UNIX timestamp when this session will expire.
|
| |
| bool | IsExpired [get] |
| | If the session has expired.
|
| |
| bool | IsRefreshExpired [get] |
| | If the refresh token has expired.
|
| |
| long | RefreshExpireTime [get] |
| | The UNIX timestamp when the refresh token will expire.
|
| |
| string | RefreshToken [get] |
| | Refresh token that can be used for session token renewal.
|
| |
| string | IdentityId [get] |
| | The ID of the user who owns this session.
|
| |
| string | AuthToken [get] |
| | The authorization token used to construct this session.
|
| |
| long | ExpireTime [get] |
| | The UNIX timestamp when this session will expire.
|
| |
| bool | IsExpired [get] |
| | If the session has expired.
|
| |
| bool | IsRefreshExpired [get] |
| | If the refresh token has expired.
|
| |
| long | RefreshExpireTime [get] |
| | The UNIX timestamp when the refresh token will expire.
|
| |
| string | RefreshToken [get] |
| | Refresh token that can be used for session token renewal.
|
| |
| string | IdentityId [get] |
| | The ID of the user who owns this session.
|
| |
◆ HasExpired()
| bool Satori.Session.HasExpired |
( |
DateTime |
offset | ) |
|
|
inline |
Check the session has expired against the offset time.
- Parameters
-
| offset | The datetime to compare against this session. |
- Returns
- If the session has expired.
Implements Satori.ISession.
◆ HasRefreshExpired()
| bool Satori.Session.HasRefreshExpired |
( |
DateTime |
offset | ) |
|
|
inline |
Check if the refresh token has expired against the offset time.
- Parameters
-
| offset | The datetime to compare against this refresh token. |
- Returns
- If refresh token has expired.
Implements Satori.ISession.
◆ Restore()
| static ISession Satori.Session.Restore |
( |
string |
authToken, |
|
|
string |
refreshToken = null |
|
) |
| |
|
inlinestatic |
Restore a session from the auth token.
A null
or empty authentication token will return null.
- Parameters
-
| authToken | The authorization token to restore as a session. |
| refreshToken | The refresh token for the session. |
- Returns
- A session.
◆ ToString()
| override string Satori.Session.ToString |
( |
| ) |
|
|
inline |
◆ Epoch
| readonly DateTime Satori.Session.Epoch = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc) |
|
static |
◆ AuthToken
| string Satori.Session.AuthToken |
|
get |
The authorization token used to construct this session.
Implements Satori.ISession.
◆ ExpireTime
| long Satori.Session.ExpireTime |
|
get |
The UNIX timestamp when this session will expire.
Implements Satori.ISession.
◆ IdentityId
| string Satori.Session.IdentityId |
|
get |
◆ IsExpired
| bool Satori.Session.IsExpired |
|
get |
◆ IsRefreshExpired
| bool Satori.Session.IsRefreshExpired |
|
get |
◆ RefreshExpireTime
| long Satori.Session.RefreshExpireTime |
|
get |
The UNIX timestamp when the refresh token will expire.
Implements Satori.ISession.
◆ RefreshToken
| string Satori.Session.RefreshToken |
|
get |
Refresh token that can be used for session token renewal.
Implements Satori.ISession.
The documentation for this class was generated from the following file:


Israeli official: Gaza fighting could end tonight or it could escalate further
Ceasefire negotiations run as both Israel and Gaza militias continue carrying out attacks; more than 400 rockets launched at Israel, Iron Dome intercepts dozens of rockets with 90% success rate
by Itamar Eichner and Yoav ZitunAn Israeli senior official said on Wednesday that negotiation talks over a ceasefire in Gaza, which are conducted despite continued rocket strikes from Gaza militant groups on Israeli communities and IDF retaliation airstrikes in the Strip, could end fighting immediately or lead to further escalation.
"Everything is happening simultaneously, " said the official. "On one hand, there are talks, but on the other hand, they continue firing rockets and we also continue to attack. This could be resolved tonight or tomorrow, but it could also escalate further".
UN Special Coordinator for the Middle East Peace Process Nickolay Mladenov arrived in Cairo on Wednesday as efforts grew to deescalate the worst bout of fighting between Israel and Gaza militants in recent months.
Mladenov landed in the Egyptian capital via Tel Aviv and was scheduled to meet with President Abdel-Fattah el-Sissi, Egyptian officials said.
Islamic Jihad leader Ziad Nahala will also reportedly travel to Cairo to join the negotiation talks brokered by Egypt and the U.N.
Egypt often acts as a mediator between Israel and Gaza militants, and brokered a ceasefire deal to end the previous round of fighting in Gaza in May.
Meanwhile, sirens continue to blare in Gaza border communities as IDF fire jets continue to launch airstrikes on Islamic Jihad targets in the Gaza Strip.
As of Wednesday night, the IDF said 400 rockets were fired at Israel by the Islamic Jihad and that the Iron Dome missile defense system has managed to intercept dozens of rockets with a 90% success rate.