The purpose of this Community Group Report is to
This document is governed by the W3C requirements of the Community and Business Group Process related to deliverables.
Concomitant use of therapeutic drugs and natural products, including vitamins, minerals, herbal medicinal products, and other botanicals, is a frequent occurrence. One study found that approximately 50% of Americans in their midlife were exposed to prescription drugs simultaneously with an dietary supplements, many of which were natural products [[Kiefer-2014]]. Concomitant exposure raises concerns of possible natural product-drug interactions (NPDIs) leading to patient harm. Potential adverse outcomes of NPDIs include transplant rejection, cardiovascular collapse, renal and liver toxicity, and death [[Posadzki-2013]]. A review identified 81 NPDIs between 35 NPs and 31 drug classes [[Wilkinson-2016]].
Research on NPDIs includes studies that characterize natural product chemical constituents, elucidate pharmacologic mechanisms, and identify potential clinical impact of NPDI exposure. The different kinds of datasets that arise from these various kinds of studies range from mass spectrometric to in vitro and in vivo pharmacokinetics. The multidisciplinary nature of NPDI research means that there is a large variety of approaches archiving, sharing, and describing the datasets (i.e., metadata).
In the face of this complexity, there is a growing recognition by both researchers and agencies that fund research that NPDI study datasets should be more findable, accessible, interoperable, and reusable (FAIR). This report proposes a set of recommended approaches to help NPDI researchers make their data more FAIR. After clarifying the intended audience, scope, and context, this report provides several recommendations organized into general and research-specific categories. Busy researchers can jump to to obtain a quick overview of the recommendations and follow links to specific FAIR practices.
This report is written for use by anyone who generates NPDI research data. These include, but are not limited to analytical chemists and clinical pharmacologists. Other natural product stakeholders might also benefit from the information in this report including funding agencies that support NPDI research, biomedical research librarians tasked with searching and/or indexing NPDI datasets, and data scientists working on natural product related analyses.
The following NPDI data are within the scope of this report:
Chemical analytical: Data derived from assays that identify and quantify the chemical constituents present in natural products.
Chemical metabolomics: Data derived from assays that assess of similarity and differences between natural products based on chemical constituents.
In vitro: Data derived from in vitro systems include parameters describing natural product binding (fu plasma, fu mic), metabolism and transport (Km, Vmax), inhibition potency (IC50, Ki), and induction potential (EC50, Emax).
IVIVE: in vitro to in vivo extrapolation approaches encompass numerous mathematical models to predict the magnitude of natural product-drug interactions. Mathematical models include static interaction models, mechanistic static models, and fully mechanistic physiologically-based pharmacokinetic interaction models.
Clinical: Appropriately designed clinical pharmacokinetic interaction studies where the natural product is the precipitant and a drug (or drugs) is the object can provide a definitive assessment of interaction risk. Clinical data include pharmacokinetic outcomes (AUC, Cmax, Tmax, t1/2) describing object drug disposition in the absence and presence of the precipitating natural product.
provides an overview of the context for this report. The contributors to this report have considered both the types of data and the data management workflows specific to chemistry and pharmacology researchers involved in NPDI research. They then reviewed journal articles, white papers, and other types of reports that make recommendations on how to make biomedical datasets more FAIR. They also reviewed specific requirements related to FAIR that are emerging from either organizations that fund NPDI research or journals that publish the NPDI studies. This report makes the recommendations and requirements from these sources more accessible to researchers who study natural product chemistry and NPDI pharmacology. It distinguishes recommendations that apply generally to all types of NPDI research from those that are specific to natural product chemistry and NPDI pharmacology.
In this paper, all recommend approaches are provided using the template below. This template is similar to the template used in the World Wide Web Consortium Data on the Web Best Practices Recommendation [[dwbp]].
Recommendation Template
Short description of the recommendation
This section answers two crucial questions:
A short description of the problem addressed by the recommended approach is also provided.
The benefits for NPDI researchers and the research community are expected if researchers follow the recommended approach. A recommendation can yield one or more benefits.
A description of a possible implementation strategy based on the state-of-the art.
Information on how to test that a researcher has followed the recommended approach. In many cases, testing can be automated.
A short example of the recommended approach applied to data emerging from NPDI research.
| Give a clear description of the experiment |
| Reporting standard experimental conditions |
Assign to a dataset a globally unique and persistent identifier
NPDI researchers should assign to datasets generated by their research a globally unique and persistent identifier. If possible, the identifier should also be resolvable.
Researchers will more easily and reliably locate and cite NPDI datasets over a longer period of time than if no identifiers are assigned. When a dataset’s identifier is globally unique, references to the dataset are non-ambiguous. When the dataset’s identifier is persistent, the dataset is findable for a longer period of time than otherwise.
The NPDI research community stands to benefit from the more rapid and non-ambiguous location of NPDI datasets over a longer period of time than if this recommendation is not followed.
There are many ways to create a globally unique and persistent identifiers including a Globally Unique Identifier (GUID) generator, a program that creates Uniform Resource Identifiers using a Uniform Resource Name (URN), a service that creates Persistent Uniform Resource Locators (PURL), or a service that creates Digital Object Identifiers (DOIs).
If the researcher intends for the dataset identifier to also resolve to a location on the Internet, the researcher could obtain a DOI through an affiliate that works with DataCite.org such as services like Zenodo, Figshare, or Dryad. If the researcher works for an academic library system, the library might have a method to obtain DOIs. For various reasons, a DOI might not be the best solution for a given research dataset. Some NPDI researchers share data on their laboratory website that has a specific domain name. In these cases, it might be possible for the researcher to create globally unique and persistent identifiers by combining the domain name with a GUID, URN, or a globally unique identifier that they create.
In this case, design leads to outcome. If the dataset identifiers are DOIs, GUID, URNs, or PURLs, then they automatically satisfy this recommendation. If not, look to see if the creator(s) of a NPDI dataset explicitly states that the identifiers were designed for persistence and global uniqueness.
TODO: Fang et al. proposed a systems pharmacology infrastructure to predict new drug targets and anticancer indications of natural products.....PMID: 28956927
Refer to the globally unique and persistent identifier in relevant publications
NPDI researchers should include mention of the globally unique and persistent identifier assigned to a given dataset in all publications and reports based on the dataset.
Researchers will more easily and reliably locate and cite NPDI datasets over a longer period of time if it they know unequivocally what specific dataset a publication or report is based on. One of the main benefits from this practice is that some services that index the scientific literature will index mention of the identifier. This leads to the ability for researchers to locate publications related to a given dataset simply by searching with the identifier.
The NPDI research community stands to benefit from the more rapid and non-ambiguous location of NPDI datasets over a longer period of time than if this recommendation is not followed.
Once a globally unique and persistent identifier exists for given dataset (see ), researchers can write the identifier directly into an approprate section in the publication or report that they are authoring. The appropriate location for mention of the identifier depends on the nature of the publication and the relevant formatting requirements. In some cases, researchers can simply mention the identifer within the methods section. Other times, it might be more appropriate to use a data citation formatted according to the norms of the researcher's discipline.[[datacite]] Researchers should check the publisher's guideline for authors for specific data citation instructions.
In this case, design leads to outcome. If all relevant publications mention the dataset identifier, then they automatically satisfy this recommendation.
TODO cite Force 11 and Research Resource Identifiers
Deposit a description of the dataset (metadata) in a public data repository
NPDI researchers should deposit dataset metadata in a public data repository that is indexed by major search engines.
Dataset metadata informs other researchers about who created a given dataset, its purpose, when it was created, and how to access it [[dwbp]]. Depositing dataset metadata in publically availabe resource that is indexed by major search engines means that this information will be more easily found by other researchers.
Depositing dataset descriptions in public repositories is an important artifact to increase the findability, accessibility, and reuse of data generated by the NPDI research community.
There are a number of repositories that a research could use to satisfy this recommendation including various Dataverse sites[[dataverse]], Zenodo[[zenodo]], Figshare[[figshare]], and Dryad[[dryad]]. A common feature of these repostories is that they ensure dataset descriptions are readible to both humans and computer programs such as web search crawlers. It is relatively common for a research lab to maintian a website that provides links to datasets created by lab members. A description of each dataset could be created and deposited in aforementioned repositories. Alternatively, the lab could write a human readable dataset description on in a location that is easy for website visitors to locate. However, this will not enable web search crawlers to index the metadata so that the datasets are more easily found via web search. To accomplish that, the website should provide the dataset description in the HTML head of the website as machine readable metadata using a format such as JSON-LD [[JSON-LD]] or HTML+RDFa [[html-rdfa]]. Examples of both human and machine readable dataset metadata are available in the Metadata section of the Data on the Web Best Practices W3C Recommendation [[dwbp]].
If the researcher deposits a dataset description in a searchable resource, then they automatically satisfy this recommendation.
TODO: A short example of the recommended approach applied to data emerging from NPDI research.
Use an existing machine readable metadata standard if publishing data on a lab website
As mentions, many labs distribute datasets from a website that they maintain. The machine readable metadata that researchers put on their website could be represented a number of different ways in JSON-LD or HTML+RDFa [[html-rdfa]]. However, researchers should use an established metadata standard such as such as Dublin Core, Metadata Initiative [[dcmi]], Schema.org [[schema.org]], or the Data Catalog Vocabulary [[dcat]].
A dataset produced from any kind of NPDI study will be more easily found and reused by other researchers if important facts about the dataset are described using standardized metadata elements that make since for the domain. For example, Google's new Dataset Search indexes sites that prpvide metadata using Schema.org [[google-data-search]].
Researchers should be able to find data on the lab website much more easily. The use of standards that are used by data search engines should increase the specificity of natural product-drug interaction search results. The metadata can provide clarity on data ownership and access rights that will help facilitate re-use.
Established metadata standards include Dublin Core, Metadata Initiative [[dcmi]], Schema.org [[schema.org]], the Data Catalog Vocabulary [[dcat]], and others [[dwbp]]. Researchers with little programming experience can use tools such as Catalog Generator [[catalog-generator]] which can export standardized metadata based in information that a researcher enters into a web form or via uploading a spreadsheet. Other approaches include using the Center for Expanded Data Annotation and Retrieval (CEDAR) workbench [[cedar]] or the tools listed on the Project Open Data metadata resources site [[project-metadata-tools]]. In terms of content, at a minimum the researcher should provide the dataset name, identifier (see ), description, creator, date of last modification, date of publication (if applicable), and distribution formats (if applicable) as structured metadata. While many established metadata standards provide attributes for general use cases, they might not include attributes relevant to a given research domain. In such cases, the researcher may use custom metadata attributes that will be recognizable by research peers and appropriate to use cases relevant to the researchers scientific peers.
Researchers can utilize Google's Structured Data Testing Tool [[google-data-testing]] to ensure that the metadata they create will be processed by Google properly. Researchers can copy and paste their metadata into the tool's form or provide a URL to a website page that has their metadata in the head of the HTML document.
TODO: A short example of the recommended approach applied to data emerging from NPDI research.
Whenever possible, make the full dataset downloadable
Make the full dataset associated with your NPDI experiments downloadable to other interested researchers.
Sharing detailed research data is associated with increased citation rate (Piwowar, Day, and Fridsma 2007). Data sharing is being increasingly required by journals, as a solution to the “replication crisis” where, for exampe, natural product researchers re-isolate previously described molecules (Spicer and Steinbeck 2017).
It helps advance science to have a straightforward access to data discussed in reports and journal articles. It simplifies referee work, makes is easier to compare multiple datasets, and improves data reuse.
For proprietary datasets, which is often the case in NP research, it is recommended to provide at least part of it (Guha and Willighagen 2017) or/and to provide a description. There are numerous repositories, and researchers have to be careful about ones which are not maintained and/or don’t follow standards. Widely used repositories that follow good practices include Dataverse and Zenodo. For MS raw file, results and metadata, NPDI studies could use more specific repository like MetaboLights, which is completely based on open source technology, using open data formats and community based reviews (https://www.ebi.ac.uk/metabolights/). MetaboLights adheres to MSI standards for metadata reporting and uses the ISA-tab format for metadata. For NMR, the COSMOS initiative (Salek et al. 2015) is similar and could be use by NPDI researchers to upload and share their NMR data with the nmrML format. Some users will benefit for the simplicity of a single file while others will want to access only a part of the dataset. Email is very commonly used but does not qualify as making data accessible because it depends on the data owner pushing the data to the recipient.
Researchers should test to download their data and confirm that it is actually open.
TODO: A short example of the recommended approach applied to data emerging from NPDI research.
If your lab website host datasets, serve the data using content negotiation
If your lab website host datasets, serve the data using content negotiation
Content negotiation make it possible to use single URL for each dataset rather than a different URL for each format (e.g., Excel, CSV, XML, JSON, etc.). Clients can request whatever format they need by specifying the format type in the HTTP request.
A single URL to a given dataset makes access much easier. Also, a single URL is easier to reference and track, increasing the potential for re-use.
A possible approach to implementation is to use the Accept header property from the request inside a controller method. The client will set the Accept header to specify the reponse they are expecting, the controller will then redirect to the proper method in order to return the correct reponse.
Check the available representations of the resource and try to get them specifying the accepted content on the HTTP Request header.
TODO: A short example of the recommended approach applied to data emerging from NPDI research.
Make the update frequency of the data explicit
Make the update frequency of the data explicit
See the recommendation Provide data up to date in the W3C report "Data on the Web Best Practices" [[dwbp]].Strive to use standard data elements
Strive to use standard data elements for both dataset metadata and for dataset fields
Standard data elements benefit researchers by making it simpler to find and integrate data from different sources.
The use of standard data elements makes it possible for researchers to quickly integrate datasets that originate from different sources (interoperability). Another benefit is that standardized elements improve data search and retrieval.
Standard data elements should be used for both dataset metadata and the datasets themselves. For metadata, researchers should use broadly accepted metadata standards such as the Data Tag Suite (DATS) [[DATS]], Dublin Core [[DC11]], or PROV [[prov-overview]]. If researchers are contributing their data to a repository, researchers should confirm repository should use a widely accepted metadata standard. Zenodo [[zenodo]], Figshare [[figshare]], and most university/research institute data catalogs satisfy this recommendation.
For datasets (rather than dataset metadata), researchers should aim to maximise the reuse terms from
existing broadly used ontologies and/or terminologies for data fields and values. One simple approach
for data spreadsheets is to maintain a seperate file with a data dictionary for the fields used in the
dataset. The dictionary should provide each data element's name (label), definition, and persistent
identifier. Another approach would be to provide the dataset in a format such as XML, JSON-LD, of RDF
that can explicitly associate each data item with a standard data element. Entire sets of standar data
elements exist for some kinds of experiments. For example, mzML is an open and generic XML (extensible
markup language) representation of MS data [[martens-2011]]. The DIDEO ontology provides standard
definitions for many data elements that arise from
When creating metadata it is crucial to abide by the fields of that standard. For example if you choose to utilize the DATS [[DATS]] format you must use the fields provided here https://datatagsuite.github.io/schema/dataset_schema.json when creating metadata. This can then be validated using the python code located here: https://github.com/datatagsuite/dats-tools.
TODO: A short example of the recommended approach applied to data emerging from NPDI research.
If possible, provide access to data via a well-documented REST-full API
If possible, provide the ability for persons with programming experience to write scripts/programs that can access data via a well-documented REST-full API
Access to datasets and associated metadata via a REST API makes it much simpler for researchers and developers to integrate some portion of the data into web sites and applications.
The primary benefit is simple programmatic access to NPDI data and metadata. APIs result in data that is computable and more likely to be reused in research workflows.
One approach to implement a REST-full API is to utilize an existing framework called Swagger and
integrate it with your application. In the case of a spring maven project the steps would be to add a
dependency such as springfox
and then configure the Docket
class as such:
Your Swagger 2.0 API documentation will now be available at http://myapp/v2/api-docs
.
By visiting http://myapp/v2/api-docs
you should have a UI that you can interact with.
Simply check that every endpoint returns the expected output based on your given input.
TODO: A short example of the recommended approach applied to data emerging from NPDI research.
Provide a license for data usage
Provide a license for data usage
See See the recommendation Provide data license information in the W3C report "Data on the Web Best Practices" [[dwbp]].Give future users of the data a way to provide feedback
Give future users of the data a way to provide feedback
See the two recommendations in the section Feedback in the W3C report "Data on the Web Best Practices" [[dwbp]].Provide raw spectral data using an open file format
NPDI researcher should provide a raw spectral data file derived from assays that identify and quantify the chemical constituents present in natural products, for both mass spectrometry (MS) and nuclear magnetic resonance (NMR) analysis in an open file format.
Chemical analytical spectral data of a NP used for NPDI research should be accessible by the NPDI researcher community. Chemical metabolomic spectral data should be reusable by everyone. Researchers will more easily and reliably trust NPDI datasets if they have the opportunity to work on the raw files. It also provides the later opportunity to launch meta analyses that advance NPDI research by providing a powerful new interpretation based on multiples datasets.
By using an open file format for raw spectral data sharing, NPDI chemical data are more accessible and reusable. Data sharing, repository deposition, and re-analysis of raw spectral data will be easier. Researchers can transfer raw data between institutions and collaborators, without the need for proprietary software (Rocca-Serra et al. 2015).
Previously, mass spectrometry open file formats, mzXML [[Pedrioli-2004]] and mzData [[Orchard-2004]], have been combined to create mzML. mzML is an open and generic XML (extensible markup language) representation of MS data (Martens et al. 2011). The user needs to convert MS raw data files from proprietary formats into an open format such as mzML. To this end, open source software for reading and writing mzML are useful to the user, such as Proteowizard (http://proteowizard.sourceforge.net/) or OpenMS (http://open-ms.sourceforge.net/) , as well as package under R (mzR), Python (pymzML) and Matlab (MSiReader). Specific to MS metabolomics, the mzTab-M is a data standard for sharing quantitative results [[Hoffmann-2019]]. This is a tab-separated text format with a standardized structure through the design of a detailed specification document. For NMR data, nmrML is an XML based, vendor-neutral open exchange and data storage format which the NMR community had agree to use [[Schober-2018]]. Other initiatives are NMReDATA (nmredata.org) and the universal raw data format, the Allotrope Data Format (ADF) developed by the Allotrope Foundation (allotrope.org). In NMR, the Raw Data Initiative is a source of information for the NPDI researcher on how to make FAIR NMR data [[McAlpine-2019]].
Researchers should test if their vendor spectral file conversion to open format spectral file have been done correctly. They could test to upload their data to any software or program using mzXML, mzTab-M, nmrML or nmredata and confirm that the file is correct.
TODO
Publish NMR and MS results following standards
NPDI researcher should share and publish NMR and MS results following standards recognized by the field. For example, spectral images must to have a good resolution, NMR assigned peaks tables or text have to be complete and to follow standards, filtered peak list with intensity in metabolomics have to be shared and identification of compounds have to follow standard procedure.
This practice will make NPDI NMR and MS results FAIR for the community to analyze and interpret. Data will already have been transformed making them accessible for a quick and easy interpretation of the results.
This would benefit the community by facilitating publication and dissemination of the results.
Reporting of NMR spectra is most frequently done using tables or descriptive text including HSQC, HMBC, and NOE correlations, as well as listings of chemical shifts, coupling constants, multiplicities and assignments [[McAlpine-2019]]. Journals provide guidelines, as ACS journal (“NMR Guidelines for ACS Journals,” n.d.). Then, it is recommended to upload the spectral file as an image with a good resolution, often in PDF format. To share high quality data in MS, mass spectra are also shared as an image to visualize fragmentation pattern in the case of tandem MS. Regarding metabolite identification in MS, the Metabolomics Standards Initiative (MSI) had defined four different levels of metabolites identification, which are identified metabolites (level 1), putatively annotated compounds (level 2), putatively characterized compound classes (level 3) and unknown compound (level 4) [[Salek-2013]]. It is recommended to report for the metabolite identification the level of identification, the common name and the structural code like the InChI string. Also, external database identifiers, chemical structure image and chemical formula are frequently reported.
TODO
TODO: A short example of the recommended approach applied to data emerging from NPDI research.
Report experiment metadata along with the spectral raw data file
Experimental metadata have to be reported with the spectral raw data file and results following standards.
Experimental metadata have to follow reporting standards to make NPDI data reproducible and comparable with other NPDI studies. This allow results from different laboratories to be shared and re-interpreted.
TODO
Experiment metadata covers the whole range necessary to understand how the experiment was conducted: the species under investigation, which organism part the metabolite was isolated from, which instruments were used, and which parameters were used. Researcher could use the properly documented metadata format IsaTab [[Salek-2013]] (ISA for Investigation, Study and Assay), as well as the raw data format Allotrope Raw Data format (https://www.allotrope.org) in NPDI studies. Every metadata is backed by a term in a commonly used ontology. Researchers should follow the minimal information standard, as created by the Metabolomics society in 2007, with the metabolomics standards initiative (MSI Board Members et al. 2007). Similar initiatives are undergoing for NMR studies, where the ISA format is also recommended [[McAlpine-2019]].
The researcher could try to re-acquire the spectral data with the same sample only by using experimental metadata reported and compare results.
TODO: A short example of the recommended approach applied to data emerging from NPDI research.
Use reproducible NMR and MS analysis workflows
Use reproducible NMR and MS analysis workflows by clearly documenting the analytical process in an open and sharable manner.
This is specifically relevant to make NPDI data FAIR as the NPDI scientific community can then reproduced exactly the workflow apply on the spectral raw data by using published and open access bioinformatics tools. By using proprietary software or “in house” functions, NPDI raw spectral data are less accessible and reusable.
TODO
The use of free and open tools, software, workflow which are reproducible. It is important to be explicit about the workflow used. The workflow could have its own identifier, for example with a Github directory. In metabolomics, such an open access pipeline exists, https://workflow4metabolomics.org which is a collaborative portal dedicated to metabolomics data processing, analysis and annotation for the Metabolomics community. The GALAXY environment also proposes different open access packages, as well as R with Xcms for MS data and Batman for NMR data.
Researchers may use a test dataset and confirm that the bioinformatics workflow used is applicable to other datasets.
A short example of the recommended approach applied to data emerging from NPDI research.
Give a clear description of the experiment
NPDI pharmacologist should clearly state details about the experiment including the effect tested, experimental system used, natural product/natural product constituant/drugs, and the object and object metabolite measured.
A clear description of the experiment helps ensure that NPDI pharmacology results more reusable by other NPDI researchers.
Researchers interested in the experiment data will have an improved understanding of the experimental context that led to the reported NPDI results. This should help the researchers assess if data from an NPDI experiment informs their research. Also, it should help researchers more easily reproduce the experiments and determine if new experiments are necessary.
A short and efficient title has to be chosen for each experiement, indicating in few words what is the main effect detected, inhibition of an enzyme or not?, and what is the system tested with which natural product. The NPDI pharmacologist is highly encouraged to details which enzymes are involved, what is the test system, the object and it’s metabolite measured as well as the natural product tested.
...
A short example of the recommended approach applied to data emerging from NPDI research.
Reporting standard experimental conditions
...
...
...
...
...
A short example of the recommended approach applied to data emerging from NPDI research.
Development of this Community Group Report was partially supported by a grant from the United States National Center for Complementary and Integrative Health U54 AT008909.


Arcadia Bay
A closer look at the town.
Introduction
Arcadia Bay is a picturesque seaside town in Oregon, United States. The town is experiencing strange weather and other phenomena that is somehow linked to Max Caulfield's time-controlling abilities.
Arcadia Bay has a large and diverse range of wildlife. It varies from different types of birds to the various deer that you see throughout the episodes of the game. However, not much is said regarding the different types of animal species in Arcadia Bay.
Key Locations
Blackwell Academy
Blackwell Academy is a senior high school which specializes in Science and Arts, and is currently led by Ray Wells.
According to Ms. Grant, the school is built on old Native American land, shared between the natives and the early colonial settlers.
Blackwell is partially owned by, and financially supported by, the Prescott family. In 1998, they donated the dormitory building to Blackwell.
Two Whales Diner
The Two Whales Diner is a local diner in Arcadia Bay. Chloe's mother, Joyce, works there as a waitress.
It is clear that the diner is hugely popular, particularly with the students in the area. Police officers, passing truck drivers and other residents of Arcadia Bay also frequent the diner, possibly due to its close proximity to a bus stop, a large parking space next to and opposite the diner and a gas station nearby.
Lighthouse
The Lighthouse is located in the forested outskirts of Arcadia Bay. For locals who want to go to the lighthouse they must follow an uphill pathway in the forest which ends up by the ocean cliff.
Max tells us that she and Chloe used to spend a lot of time there when they were young. Visitors and guests at the lighthouse are encouraged to leave the wildlife in the area alone, as per the sign on the trail. Max confirms it's been there a long time.


Time to hunt for candy! It's Halloween and you are out trick or treating. Your goal is to return home with as much candy as possible. Beware! There are bullies and abandoned houses and dentists in your neighborhood. Be clever and have fun with the rest of the ghoulies out tonight!
  And if anyone disappoints you, you can always {{player.prank}}. There is a house to your left and another to the right. Which do you visit?


--- layout: stats title: Iowa Statistics permalink: "/stats/teams/iowa.html" ---
| Year | Team | Status | Finish | W-L | PPG | PAPG | Margin | TU/G | N/G | BHrd | BPts | PPB |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2019 | Iowa | UG | 44 | 4-10 | 88.9 | 180.7 | -91.8 | 5.57 | 1.93 | 78 | 600 | 7.69 |
| Year | Round | Team | Opponent | Result | Score | 10 | -5 | PPB | Packet |
|---|---|---|---|---|---|---|---|---|---|
| 2019 | 1 (Prelims) | Iowa | North Carolina | Loss | 65 - 275 | 5 | 1 | 4.00 | Yale A + Michigan |
| 2019 | 2 (Prelims) | Iowa | Carnegie Mellon | Win | 130 - 120 | 9 | 2 | 5.56 | UVA + Rutgers + Chicago B |
| 2019 | 3 (Prelims) | Iowa | Michigan State | Loss | -10 - 415 | 0 | 2 | NaN | McGill + UCLA + Oxford |
| 2019 | 4 (Prelims) | Iowa | Florida A | Loss | 90 - 245 | 5 | 2 | 10.00 | Berkeley A + Ohio St |
| 2019 | 6 (Prelims) | Iowa | Gettysburg | Loss | 105 - 110 | 7 | 3 | 7.14 | Minn A + Harvard A + B |
| 2019 | 7 (Prelims) | Iowa | Illinois A | Loss | 35 - 270 | 3 | 1 | 3.33 | Florida A + Michigan St |
| 2019 | 8 (Prelims) | Iowa | Penn State | Loss | 95 - 165 | 6 | 1 | 6.67 | Illinois A + Wright St |
| 2019 | 10 (Playoffs) | Iowa | Wright State | Loss | 80 - 90 | 5 | 4 | 10.00 | Editors 1 |
| 2019 | 11 (Playoffs) | Iowa | New College of Florida | Loss | 60 - 190 | 5 | 0 | 2.00 | Editors 2 |
| 2019 | 12 (Playoffs) | Iowa | Central Florida | Loss | 60 - 220 | 4 | 4 | 10.00 | Editors 3 |
| 2019 | 13 (Playoffs) | Iowa | Stanford B | Win | 165 - 70 | 8 | 1 | 11.25 | Editors 4 |
| 2019 | 15 (Superplayoffs) | Iowa | Oklahoma | Loss | 75 - 210 | 4 | 3 | 12.50 | Editors 6 |
| 2019 | 16 (Superplayoffs) | Iowa | Auburn | Win | 170 - 55 | 9 | 2 | 10.00 | Editors 7 |
| 2019 | 17 (Superplayoffs) | Iowa | Caltech A | Win | 125 - 95 | 8 | 1 | 6.25 | Editors 8 |
| Year | Team | Player | Games | Tossups | Negs | Pts | PPG |
|---|---|---|---|---|---|---|---|
| 2019 | Iowa | Ganon Evans | 14 | 30 | 5 | 275 | 19.64 |
| 2019 | Iowa | Jordan LaCerte | 14 | 16 | 5 | 135 | 9.64 |
| 2019 | Iowa | Eric Brosch | 9 | 10 | 4 | 80 | 8.89 |
| 2019 | Iowa | Tony Cosimini | 14 | 16 | 10 | 110 | 7.86 |
| 2019 | Iowa | Samuel Kitzman | 6 | 6 | 3 | 45 | 7.50 |


This is a tutorial on how to use ocamllex which is distributed with Ocaml language
This document borrowed lots of part from flex manual.
Please mail all comments and suggestions to<EMAIL_ADDRESS>The latest version can be found at https://ohama.github.io/ocaml/ocamllex-ocamlyacc/ocamllex-tutorial/index.html.
The companion tutorial for ocamlyacc is available at https://ohama.github.io/ocaml/ocamllex-ocamlyacc/ocamlyacc-tutorial/index.html.
You can find the source of this document in tutorial directory. For printing, pdf is presented.
For the source of examples used in this document, see ocamllex-examples directory.
This document is just a re-formatting of the old one. There have been many improvements since then in Ocaml programming language but those are not reflected in this document.
First release: 2004-11-16
Last updated: 2019-05-05
| Next | ||
| Introduction |


The Pi and I
Scott Vitale, Founder @
@svvitale
Raspberry Pi
"... is a series of small single-board computers developed in the United Kingdom by the Raspberry Pi Foundation to promote the teaching of basic computer science in schools and in developing countries."
Nerds Minds Blown!
- Fully functional computer
- Full Linux OS
- HDMI + USB + Wifi
- CHEAP! $35 (and now $10 for the Zero W)
Okay, but what's it do?
Home Automation
Beer Brewing
Surveillance
Compute Clusters
Smart Boards
Now a little about
Data Capture for the Modern Event
Applications
At the core of it all...
Tech Stack
- Raspbian OS
- Mostly Python Services
- Managed by Supervisor
- Small amount of C/Python wrapper code
- Connectivity/Power Fault tolerant
Hardware Control
- Polls NFC sensor for a valid tag
- Gives visible/audible feedback when found
- Adds detected tags to work queue (RabbitMQ).
- Synchronous for payments
- Asynchronous for everything else
API Client
- Reads from the work queue
- Posts tag data to Spigot API via Wifi
- On error, requeues tag data.
Resources
- Sparkfun - based in Niwot CO, great tutorials, local source for parts, sensors, help
- Adafruit - more of the same, but with shipping costs
- Oshpark - CHEAP printed circuit boards
Thanks!
Shameless plug: Fritzler Hops n' Husks on 9/16
http://pi.spigotlabs.com


Datasets associated with the 'gap' package. Currently, it includes an example data for regional association plot (CDKN), Manhattan plots with (hr1420, mhtdata) and without (w4) gene annotations.
| Version: | 0.0.2 |
| Depends: | R (≥ 2.10) |
| Published: | 2017-11-27 |
| Author: | Jing Hua Zhao with suggestions from R-Core |
| Maintainer: | Jing Hua Zhao <jinghua.zhao at mrc-epid.cam.ac.uk> |
| License: | GPL-2 | GPL-3 [expanded from: GPL (≥ 2)] |
| URL: | https://jinghuazhao.github.io/ |
| NeedsCompilation: | no |
| Materials: | README |
| CRAN checks: | gap.datasets results |
| Reference manual: | gap.datasets.pdf |
| Package source: | gap.datasets_0.0.2.tar.gz |
| Windows binaries: | r-devel: gap.datasets_0.0.2.zip, r-release: gap.datasets_0.0.2.zip, r-oldrel: gap.datasets_0.0.2.zip |
| OS X binaries: | r-release: gap.datasets_0.0.2.tgz, r-oldrel: gap.datasets_0.0.2.tgz |
Please use the canonical form https://CRAN.R-project.org/package=gap.datasets to link to this page.


{{recipe?.rating}} stars
{{ recipe?.description}}
Poison level: {{ recipe?.poisonLevel}}
There are no ingredients.


NAVIAGATE TO HOME PAGE
Data Insights
Decision Trees
Scored Labels and Probablities
Feature Weights Classification


WELCOME TO WITH
Where culture meets creativity
-
Discover cultural content from around the world
Learn MoreDiscover -
Build using data exposed by the WITH API
Learn MoreBuild -
Create stories, collections and exhibitions
Learn MoreCreate -
Link & Enrich with LoD sources
Learn MoreAnnotate -
Upload, map, transform and publish
Learn MoreAggregate -
Tag, follow, discuss, moderate WITH
Learn MoreParticipate
Partners using with


- hours open: 6:00am - 7:00pm
- location: seattle jawad st bulding #4
- contact: 16187951
- hours open: 6:00am - 7:00pm
- location: tokyo monkey D.luffy st building #21
- contact: 987654191
- hours open: 6:00am - 7:00pm
- location: dubai ammar st building# 44
- contact: 873385897
- hours open: 6:00am - 7:00pm
- location: paris naymar ju st building #1
- contact: 783676549
- hours open: 6:00am - 7:00pm
- location: lima kambodia st building #6
- contact: 948655566


Gamepad Visualizer
Explore gamepads connected to your system
No gamepads detected. If your gamepad is connected and powered on, try pressing a button to activate it.
Using the Gamepad API
Gamepad Reference
Information about popular external gamepads


All paper submissions will be handled electronically, only.
Submit
your paper(s) to E-MediSys 2007
For more information please contact<EMAIL_ADDRESS>Check our web page at http://lifc.univ-fcomte.fr/emedisys07 for the
latest information concerning the conference.
Only .PDF or .DOC
files will be accepted for paper submission.
Instruction for Authors:
- This template is for the standard IEEE submission.
- Latex Template
- For PC and Mac, download IEEEtran.zip and IEEEtranBST.zip
bibliography style file.
- For Unix, download IEEEtran.tar.gz and IEEEtranBST.tar.gz
bibliography style file.
- MS Word Template
- Download the TRANS-JOUR.DOC
author's template for Microsoft Word. This file contains a template for
you to use in creating your paper.
- You may also download the TRANS-JOUR.PDF
file if you just want the instructions and not the template.
- The maximum number of pages is limited to 8
pages (10pt, double column, IEEE format).


Making scientific posters
Mike Harms
November 6, 2019
What is the purpose of a poster presentation?
Have a conversation about your science.
- Get feedback from experts
- Generate excitement
- Find collaborators
- ...
Design your poster for a conversation, not talking at people.
Who is your audience?
Tipsy scientist in a hurry.
Persuade them that your science is worth their time.
Is a poster expository, persuasive, narrative, or descriptive writing?
Scientific presentations are persuasive.
Make a case that:
- Your problem is worth solving
- You have solved it
Outline
- Presenting your poster
- Making your poster
Principles
- Persuade your audience: tell them why they should care about the science
- Be excited about your science
- Make your poster a conversation
In practice
- Ask questions, make sure the person is following.
- Have a 3 sentence version of your poster
- Have a 3 minute version of your poster
- Know your data inside and out
Outline
- Presenting your poster
- Making your poster
Visual principles:
This is a persuasive poster arguing that your problem is important and that you've solved it.
- Every bit of ink should support your case
- Each heading/title should be a claim supported by evidence
- Set up your graphics so the evidence supporting your claim is clear
Some practical rules:
- Minimize text.
- Never go below 30 point font.
- Never have an image below 300 dots per inch, or a graphic smaller than 4x4 inches.
- Use a vector graphics program like Illustrator or Inkscape (free), not Powerpoint.
- If you print the poster out on a normal piece of paper, you should still be able to read it.
Conclusion:
- A poster should be persuasive: "This question is interesting." and "I have answered the question."
- A poster presentation should be a conversation
- A poster should be designed to achieve #1 and #2.
Slides: harmsm.github.io/scientific-posters


File streams allow input and output to files. Unlike the C stdio functions for file I/O, however, file streams follow the idiom which Stroustrup puts forth on page 366 of The C++ Programming Language, 3rd Edition: "Resource acquisition is initialization." In other words, file streams provide an advantage in that you can open a file on construction of a stream, and the file is closed automatically on destruction of the stream. Consider the following code:
void use_file(const char* fileName) { FILE* f = fopen("fileName", "w"); // use file fclose(f); }
If an exception is thrown while the file is in use here, the file is never closed. With a file stream, however, the file is closed whenever the file stream goes out of scope, as in the following example:
void use_file(const char* fileName) { std::ofstream f("fileName"); // use file }
Here the file is closed even if an exception occurs during use of the open file.
There are three class templates that implement file streams: basic_ifstream, basic_ofstream, and basic_fstream. These templates are derived from the stream base class template basic_ios. Therefore, they inherit all the functions for formatted input and output described in Chapter 28, as well as the stream state. They also have functions for opening and closing files, and a constructor that allows opening a file and connecting it to the stream. For convenience, there are the typedefs std::ifstream, std::ofstream, and std::fstream, with std::wifstream, std::wofstream, and std::wfstream for the respective narrow and wide character file streams.
The buffering is implemented by a specialized stream buffer class template, basic_filebuf.
In a large character set environment, a file is assumed to contain multibyte characters. To provide the contents of a such a file as a wide character sequence for internal processing, wifstream and wofstream perform corresponding conversions. The actual conversion is delegated to the file buffer, which relays the task to the imbued locale's code conversion facet.


|
Robot
|
#include <Mech2IndMotors.h>
Public Member Functions | |
| void | Update () override |
| update the output to the mechanism using the current controller and target value(s) More... | |
| void | UpdateTargets (double primary, double secondary) override |
| double | GetPrimaryPosition () const override |
| Return the current position of the primary motor in the mechanism. The value is in inches or degrees. More... | |
| double | GetSecondaryPosition () const override |
| Return the current position of the secondary motor in the mechanism. The value is in inches or degrees. More... | |
| double | GetPrimarySpeed () const override |
| Get the current speed of the primary motor in the mechanism. The value is in inches per second or degrees per second. More... | |
| double | GetSecondarySpeed () const override |
| Get the current speed of the secondary motor in the mechanism. The value is in inches per second or degrees per second. More... | |
| void | SetControlConstants (ControlData *pid) override |
| Set the control constants (e.g. PIDF values). More... | |
| Mech2IndMotors (std::unique_ptr< IDragonMotorController > primaryMotor, std::unique_ptr< IDragonMotorController > secondaryMotor) | |
| Mech2IndMotors ()=delete | |
| ~Mech2IndMotors ()=default | |
| Public Member Functions inherited from IMech2IndMotors | |
| IMech2IndMotors ()=default | |
| virtual | ~IMech2IndMotors ()=default |
Definition at line 34 of file Mech2IndMotors.h.
| Mech2IndMotors::Mech2IndMotors | ( | std::unique_ptr< IDragonMotorController > | primaryMotor, |
| std::unique_ptr< IDragonMotorController > | secondaryMotor | ||
| ) |
|
delete |
|
default |
|
overridevirtual |
Return the current position of the primary motor in the mechanism. The value is in inches or degrees.
Implements IMech2IndMotors.
Definition at line 90 of file Mech2IndMotors.cpp.
|
overridevirtual |
Get the current speed of the primary motor in the mechanism. The value is in inches per second or degrees per second.
Implements IMech2IndMotors.
Definition at line 104 of file Mech2IndMotors.cpp.
|
overridevirtual |
Return the current position of the secondary motor in the mechanism. The value is in inches or degrees.
Implements IMech2IndMotors.
Definition at line 97 of file Mech2IndMotors.cpp.
|
overridevirtual |
Get the current speed of the secondary motor in the mechanism. The value is in inches per second or degrees per second.
Implements IMech2IndMotors.
Definition at line 111 of file Mech2IndMotors.cpp.
|
overridevirtual |
Set the control constants (e.g. PIDF values).
| [in] | ControlData* | pid: the control constants |
Implements IMech2IndMotors.
Definition at line 120 of file Mech2IndMotors.cpp.
|
overridevirtual |
update the output to the mechanism using the current controller and target value(s)
Implements IMech2IndMotors.
Definition at line 58 of file Mech2IndMotors.cpp.
|
overridevirtual |
Implements IMech2IndMotors.
Definition at line 71 of file Mech2IndMotors.cpp.


GridMix
Overview
GridMix is a benchmark for Hadoop clusters. It submits a mix of synthetic jobs, modeling a profile mined from production loads.
There exist three versions of the GridMix tool. This document discusses the third (checked into src/contrib), distinct from the two checked into the src/benchmarks sub-directory. While the first two versions of the tool included stripped-down versions of common jobs, both were principally saturation tools for stressing the framework at scale. In support of a broader range of deployments and finer-tuned job mixes, this version of the tool will attempt to model the resource profiles of production jobs to identify bottlenecks, guide development, and serve as a replacement for the existing GridMix benchmarks.
To run GridMix, you need a MapReduce job trace describing the job mix for a given cluster. Such traces are typically generated by Rumen (see Rumen documentation). GridMix also requires input data from which the synthetic jobs will be reading bytes. The input data need not be in any particular format, as the synthetic jobs are currently binary readers. If you are running on a new cluster, an optional step generating input data may precede the run.
In order to emulate the load of production jobs from a given cluster on the same or another cluster, follow these steps:
- Locate the job history files on the production cluster. This location is specified by the mapred.job.tracker.history.completed.location configuration property of the cluster.
- Run Rumen to build a job trace in JSON format for all or select jobs.
- Use GridMix with the job trace on the benchmark cluster.
Jobs submitted by GridMix have names of the form "GRIDMIXnnnnnn", where "nnnnnn" is a sequence number padded with leading zeroes.
Usage
Basic command-line usage without configuration parameters:
org.apache.hadoop.mapred.gridmix.Gridmix [-generate <size>] [-users <users-list>] <iopath> <trace>
Basic command-line usage with configuration parameters:
org.apache.hadoop.mapred.gridmix.Gridmix \ -Dgridmix.client.submit.threads=10 -Dgridmix.output.directory=foo \ [-generate <size>] [-users <users-list>] <iopath> <trace>
The <iopath> parameter is the working directory for GridMix. Note that this can either be on the local file-system or on HDFS, but it is highly recommended that it be the same as that for the original job mix so that GridMix puts the same load on the local file-system and HDFS respectively.
The -generate option is used to generate input data and Distributed Cache files for the synthetic jobs. It accepts standard units of size suffixes, e.g. 100g will generate 100 * 230 bytes as input data. <iopath>/input is the destination directory for generated input data and/or the directory from which input data will be read. HDFS-based Distributed Cache files are generated under the distributed cache directory <iopath>/distributedCache. If some of the needed Distributed Cache files are already existing in the distributed cache directory, then only the remaining non-existing Distributed Cache files are generated when -generate option is specified.
The -users option is used to point to a users-list file (see Emulating Users and Queues).
The <trace> parameter is a path to a job trace generated by Rumen. This trace can be compressed (it must be readable using one of the compression codecs supported by the cluster) or uncompressed. Use "-" as the value of this parameter if you want to pass an uncompressed trace via the standard input-stream of GridMix.
The class org.apache.hadoop.mapred.gridmix.Gridmix can be found in the JAR contrib/gridmix/hadoop-gridmix-$VERSION.jar inside your Hadoop installation, where $VERSION corresponds to the version of Hadoop installed. A simple way of ensuring that this class and all its dependencies are loaded correctly is to use the hadoop wrapper script in Hadoop:
hadoop jar <gridmix-jar> org.apache.hadoop.mapred.gridmix.Gridmix \ [-generate <size>] [-users <users-list>] <iopath> <trace>
The supported configuration parameters are explained in the following sections.
General Configuration Parameters
| Parameter | Description |
|---|---|
| gridmix.output.directory | The directory into which output will be written. If specified, iopath will be relative to this parameter. The submitting user must have read/write access to this directory. The user should also be mindful of any quota issues that may arise during a run. The default is "gridmix". |
| gridmix.client.submit.threads | The number of threads submitting jobs to the cluster. This also controls how many splits will be loaded into memory at a given time, pending the submit time in the trace. Splits are pre-generated to hit submission deadlines, so particularly dense traces may want more submitting threads. However, storing splits in memory is reasonably expensive, so you should raise this cautiously. The default is 1 for the SERIAL job-submission policy (see Job Submission Policies) and one more than the number of processors on the client machine for the other policies. |
| gridmix.submit.multiplier | The multiplier to accelerate or decelerate the submission of jobs. The time separating two jobs is multiplied by this factor. The default value is 1.0. This is a crude mechanism to size a job trace to a cluster. |
| gridmix.client.pending.queue.depth | The depth of the queue of job descriptions awaiting split generation. The jobs read from the trace occupy a queue of this depth before being processed by the submission threads. It is unusual to configure this. The default is 5. |
| gridmix.gen.blocksize | The block-size of generated data. The default value is 256 MiB. |
| gridmix.gen.bytes.per.file | The maximum bytes written per file. The default value is 1 GiB. |
| gridmix.min.file.size | The minimum size of the input files. The default limit is 128 MiB. Tweak this parameter if you see an error-message like "Found no satisfactory file" while testing GridMix with a relatively-small input data-set. |
| gridmix.max.total.scan | The maximum size of the input files. The default limit is 100 TiB. |
| gridmix.task.jvm-options.enable | Enables Gridmix to configure the simulated task's max heap options using the values obtained from the original task (i.e via trace). |
Job Types
GridMix takes as input a job trace, essentially a stream of JSON-encoded job descriptions. For each job description, the submission client obtains the original job submission time and for each task in that job, the byte and record counts read and written. Given this data, it constructs a synthetic job with the same byte and record patterns as recorded in the trace. It constructs jobs of two types:
| Job Type | Description |
|---|---|
| LOADJOB | A synthetic job that emulates the workload mentioned in Rumen trace. In the current version we are supporting I/O. It reproduces the I/O workload on the benchmark cluster. It does so by embedding the detailed I/O information for every map and reduce task, such as the number of bytes and records read and written, into each job's input splits. The map tasks further relay the I/O patterns of reduce tasks through the intermediate map output data. |
| SLEEPJOB | A synthetic job where each task does nothing but sleep for a certain duration as observed in the production trace. The scalability of the Job Tracker is often limited by how many heartbeats it can handle every second. (Heartbeats are periodic messages sent from Task Trackers to update their status and grab new tasks from the Job Tracker.) Since a benchmark cluster is typically a fraction in size of a production cluster, the heartbeat traffic generated by the slave nodes is well below the level of the production cluster. One possible solution is to run multiple Task Trackers on each slave node. This leads to the obvious problem that the I/O workload generated by the synthetic jobs would thrash the slave nodes. Hence the need for such a job. |
The following configuration parameters affect the job type:
| Parameter | Description |
|---|---|
| gridmix.job.type | The value for this key can be one of LOADJOB or SLEEPJOB. The default value is LOADJOB. |
| gridmix.key.fraction | For a LOADJOB type of job, the fraction of a record used for the data for the key. The default value is 0.1. |
| gridmix.sleep.maptask-only | For a SLEEPJOB type of job, whether to ignore the reduce tasks for the job. The default is false. |
| gridmix.sleep.fake-locations | For a SLEEPJOB type of job, the number of fake locations for map tasks for the job. The default is 0. |
| gridmix.sleep.max-map-time | For a SLEEPJOB type of job, the maximum runtime for map tasks for the job in milliseconds. The default is unlimited. |
| gridmix.sleep.max-reduce-time | For a SLEEPJOB type of job, the maximum runtime for reduce tasks for the job in milliseconds. The default is unlimited. |
Job Submission Policies
GridMix controls the rate of job submission. This control can be based on the trace information or can be based on statistics it gathers from the Job Tracker. Based on the submission policies users define, GridMix uses the respective algorithm to control the job submission. There are currently three types of policies:
| Job Submission Policy | Description |
|---|---|
| STRESS | Keep submitting jobs so that the cluster remains under stress.
In this mode we control the rate of job submission by monitoring
the real-time load of the cluster so that we can maintain a stable
stress level of workload on the cluster. Based on the statistics we
gather we define if a cluster is underloaded or
overloaded. We consider a cluster underloaded if
and only if the following three conditions are true:
|
| REPLAY | In this mode we replay the job traces faithfully. This mode exactly follows the time-intervals given in the actual job trace. |
| SERIAL | In this mode we submit the next job only once the job submitted earlier is completed. |
The following configuration parameters affect the job submission policy:
| Parameter | Description |
|---|---|
| gridmix.job-submission.policy | The value for this key would be one of the three: STRESS, REPLAY or SERIAL. In most of the cases the value of key would be STRESS or REPLAY. The default value is STRESS. |
| gridmix.throttle.jobs-to-tracker-ratio | In STRESS mode, the minimum ratio of running jobs to Task Trackers in a cluster for the cluster to be considered overloaded. This is the threshold TJ referred to earlier. The default is 1.0. |
| gridmix.throttle.maps.task-to-slot-ratio | In STRESS mode, the minimum ratio of pending and running map tasks (i.e. incomplete map tasks) to the number of map slots for a cluster for the cluster to be considered overloaded. This is the threshold TM referred to earlier. Running map tasks are counted partially. For example, a 40% complete map task is counted as 0.6 map tasks. The default is 2.0. |
| gridmix.throttle.reduces.task-to-slot-ratio | In STRESS mode, the minimum ratio of pending and running reduce tasks (i.e. incomplete reduce tasks) to the number of reduce slots for a cluster for the cluster to be considered overloaded. This is the threshold TR referred to earlier. Running reduce tasks are counted partially. For example, a 30% complete reduce task is counted as 0.7 reduce tasks. The default is 2.5. |
| gridmix.throttle.maps.max-slot-share-per-job | In STRESS mode, the maximum share of a cluster's map-slots capacity that can be counted toward a job's incomplete map tasks in overload calculation. The default is 0.1. |
| gridmix.throttle.reducess.max-slot-share-per-job | In STRESS mode, the maximum share of a cluster's reduce-slots capacity that can be counted toward a job's incomplete reduce tasks in overload calculation. The default is 0.1. |
Emulating Users and Queues
Typical production clusters are often shared with different users and the cluster capacity is divided among different departments through job queues. Ensuring fairness among jobs from all users, honoring queue capacity allocation policies and avoiding an ill-behaving job from taking over the cluster adds significant complexity in Hadoop software. To be able to sufficiently test and discover bugs in these areas, GridMix must emulate the contentions of jobs from different users and/or submitted to different queues.
Emulating multiple queues is easy - we simply set up the benchmark cluster with the same queue configuration as the production cluster and we configure synthetic jobs so that they get submitted to the same queue as recorded in the trace. However, not all users shown in the trace have accounts on the benchmark cluster. Instead, we set up a number of testing user accounts and associate each unique user in the trace to testing users in a round-robin fashion.
The following configuration parameters affect the emulation of users and queues:
| Parameter | Description |
|---|---|
| gridmix.job-submission.use-queue-in-trace | When set to true it uses exactly the same set of queues as those mentioned in the trace. The default value is false. |
| gridmix.job-submission.default-queue | Specifies the default queue to which all the jobs would be submitted. If this parameter is not specified, GridMix uses the default queue defined for the submitting user on the cluster. |
| gridmix.user.resolve.class | Specifies which UserResolver implementation to use.
We currently have three implementations:
|
If the parameter gridmix.user.resolve.class is set to org.apache.hadoop.mapred.gridmix.RoundRobinUserResolver, we need to define a users-list file with a list of test users. This is specified using the -users option to GridMix.
A users-list file has one user per line, each line of the format:
<username>
For example:
user1 user2 user3
In the above example we have defined three users user1, user2 and user3. Now we would associate each unique user in the trace to the above users defined in round-robin fashion. For example, if trace's users are tuser1, tuser2, tuser3, tuser4 and tuser5, then the mappings would be:
tuser1 -> user1 tuser2 -> user2 tuser3 -> user3 tuser4 -> user1 tuser5 -> user2
For backward compatibility reasons, each line of users-list file can contain username followed by groupnames in the form username[,group]*. The groupnames will be ignored by Gridmix.
Emulating Distributed Cache Load
Gridmix emulates Distributed Cache load by default for LOADJOB type of jobs. This is done by precreating the needed Distributed Cache files for all the simulated jobs as part of a separate MapReduce job.
Emulation of Distributed Cache load in gridmix simulated jobs can be disabled by configuring the property gridmix.distributed-cache-emulation.enable to false. But generation of Distributed Cache data by gridmix is driven by -generate option and is independent of this configuration property.
Both generation of Distributed Cache files and emulation of Distributed Cache load are disabled if:
- input trace comes from the standard input-stream instead of file, or
- <iopath> specified is on local file-system, or
- any of the ascendant directories of the distributed cache directory i.e. <iopath>/distributedCache (including the distributed cache directory) doesn't have execute permission for others.
Configuration of Simulated Jobs
Gridmix3 sets some configuration properties in the simulated Jobs submitted by it so that they can be mapped back to the corresponding Job in the input Job trace. These configuration parameters include:
| Parameter | Description |
|---|---|
| gridmix.job.original-job-id | The job id of the original cluster's job corresponding to this simulated job. |
| gridmix.job.original-job-name | The job name of the original cluster's job corresponding to this simulated job. |
Emulating Compression/Decompression
MapReduce supports data compression and decompression. Input to a MapReduce job can be compressed. Similarly, output of Map and Reduce tasks can also be compressed. Compression/Decompression emulation in GridMix is important because emulating compression/decompression will effect the CPU and Memory usage of the task. A task emulating compression/decompression will affect other tasks and daemons running on the same node.
Compression emulation is enabled if gridmix.compression-emulation.enable is set to true. By default compression emulation is enabled for jobs of type LOADJOB. With compression emulation enabled, GridMix will now generate compressed text data with a constant compression ratio. Hence a simulated GridMix job will now emulate compression/decompression using compressible text data (having a constant compression ratio), irrespective of the compression ratio observed in the actual job.
A typical MapReduce Job deals with data compression/decompression in the following phases
- Job input data decompression: GridMix generates compressible input data when compression emulation is enabled. Based on the original job's configuration, a simulated GridMix job will use a decompressor to read the compressed input data. Currently, GridMix uses mapreduce.input.fileinputformat.inputdir to determine if the original job used compressed input data or not. If the original job's input files are uncompressed then the simulated job will read the compressed input file without using a decompressor.
- Intermediate data compression and decompression: If the original job has map output compression enabled then GridMix too will enable map output compression for the simulated job. Accordingly, the reducers will use a decompressor to read the map output data.
- Job output data compression: If the original job's output is compressed then GridMix too will enable job output compression for the simulated job.
The following configuration parameters affect compression emulation
| Parameter | Description |
|---|---|
| gridmix.compression-emulation.enable | Enables compression emulation in simulated GridMix jobs. Default is true. |
With compression emulation turned on, GridMix will generate compressed input data. Hence the total size of the input data will be lesser than the expected size. Set gridmix.min.file.size to a smaller value (roughly 10% of gridmix.gen.bytes.per.file) for enabling GridMix to correctly emulate compression.
Emulating High-Ram jobs
MapReduce allows users to define a job as a High-Ram job. Tasks from a High-Ram job can occupy multiple slots on the task-trackers. Task-tracker assigns fixed virtual memory for each slot. Tasks from High-Ram jobs can occupy multiple slots and thus can use up more virtual memory as compared to a default task.
Emulating this behavior is important because of the following reasons
- Impact on scheduler: Scheduling of tasks from High-Ram jobs impacts the scheduling behavior as it might result into slot reservation and slot/resource utilization.
- Impact on the node : Since High-Ram tasks occupy multiple slots, trackers do some bookkeeping for allocating extra resources for these tasks. Thus this becomes a precursor for memory emulation where tasks with high memory requirements needs to be considered as a High-Ram task.
High-Ram feature emulation can be disabled by setting gridmix.highram-emulation.enable to false.
Emulating resource usages
Usages of resources like CPU, physical memory, virtual memory, JVM heap etc are recorded by MapReduce using its task counters. This information is used by GridMix to emulate the resource usages in the simulated tasks. Emulating resource usages will help GridMix exert similar load on the test cluster as seen in the actual cluster.
MapReduce tasks use up resources during its entire lifetime. GridMix also tries to mimic this behavior by spanning resource usage emulation across the entire lifetime of the simulated task. Each resource to be emulated should have an emulator associated with it. Each such emulator should implement the org.apache.hadoop.mapred.gridmix.emulators.resourceusage .ResourceUsageEmulatorPlugin interface. Resource emulators in GridMix are plugins that can be configured (plugged in or out) before every run. GridMix users can configure multiple emulator plugins by passing a comma separated list of emulators as a value for the gridmix.emulators.resource-usage.plugins parameter.
List of emulators shipped with GridMix:
- Cumulative CPU usage emulator: GridMix uses the cumulative CPU usage value published by Rumen and makes sure that the total cumulative CPU usage of the simulated task is close to the value published by Rumen. GridMix can be configured to emulate cumulative CPU usage by adding org.apache.hadoop.mapred.gridmix.emulators.resourceusage .CumulativeCpuUsageEmulatorPlugin to the list of emulator plugins configured for the gridmix.emulators.resource-usage.plugins parameter. CPU usage emulator is designed in such a way that it only emulates at specific progress boundaries of the task. This interval can be configured using gridmix.emulators.resource-usage.cpu.emulation-interval. The default value for this parameter is 0.1 i.e 10%.
- Total heap usage emulator: GridMix uses the total heap usage value published by Rumen and makes sure that the total heap usage of the simulated task is close to the value published by Rumen. GridMix can be configured to emulate total heap usage by adding org.apache.hadoop.mapred.gridmix.emulators.resourceusage .TotalHeapUsageEmulatorPlugin to the list of emulator plugins configured for the gridmix.emulators.resource-usage.plugins parameter. Heap usage emulator is designed in such a way that it only emulates at specific progress boundaries of the task. This interval can be configured using gridmix.emulators.resource-usage.heap.emulation-interval . The default value for this parameter is 0.1 i.e 10% progress interval.
Note that GridMix will emulate resource usages only for jobs of type LOADJOB.
Simplifying Assumptions
GridMix will be developed in stages, incorporating feedback and patches from the community. Currently its intent is to evaluate MapReduce and HDFS performance and not the layers on top of them (i.e. the extensive lib and sub-project space). Given these two limitations, the following characteristics of job load are not currently captured in job traces and cannot be accurately reproduced in GridMix:
- Filesystem Properties - No attempt is made to match block sizes, namespace hierarchies, or any property of input, intermediate or output data other than the bytes/records consumed and emitted from a given task. This implies that some of the most heavily-used parts of the system - text processing, streaming, etc. - cannot be meaningfully tested with the current implementation.
- I/O Rates - The rate at which records are consumed/emitted is assumed to be limited only by the speed of the reader/writer and constant throughout the task.
- Memory Profile - No data on tasks' memory usage over time is available, though the max heap-size is retained.
- Skew - The records consumed and emitted to/from a given task are assumed to follow observed averages, i.e. records will be more regular than may be seen in the wild. Each map also generates a proportional percentage of data for each reduce, so a job with unbalanced input will be flattened.
- Job Failure - User code is assumed to be correct.
- Job Independence - The output or outcome of one job does not affect when or whether a subsequent job will run.
Appendix
Issues tracking the original implementations of GridMix1, GridMix2, and GridMix3 can be found on the Apache Hadoop MapReduce JIRA. Other issues tracking the current development of GridMix can be found by searching the Apache Hadoop MapReduce JIRA


{% extends 'base.html' %} {% import 'bootstrap/wtf.html' as wtf %} {% block content%}
Write a new pitch...


Send
IT
4 New Notifications
your order #22776 has left our warehouse
your order #23455 has been delivered
your order #23455 has been delivered
your order #23455 has been delivered
New Order
Profile
Orders
Logout
New Order
Profile
Orders
Logout
Welcome KELECHI
delivered
10
In Transit
3
Personal Details
Name
Kelechi Ogbonna
Email
<EMAIL_ADDRESS>password
******
Order Details
Order ID
Date
Price
Status
#22338
20-12-2018
N2000
In transit
edit
#22338
20-12-2018
N2000
In transit
edit


--- layout: post title: Starting programs on a remote linux desktop date: 2011-02-09 22:25:44.000000000 +01:00 categories: - linux tags: [] status: publish type: post published: true meta: _edit_last: '1' _wp_old_slug: '' author: ---
This was something that I was looking for a long time and finally found while looking for a completely different thing altogether!
Agreed I always wanted this only for mischief :P Basically what I wanted was to fire up any GUI so that it would show on a remote desktop and not on my local display. clearly I didn't want vnc or ssh -XCYP.
What I needed was this:
ssh -X host
export DISPLAY=:0
and then I can run any gui and it will open on the remote desktop.
would make things look like some hacky mischief ;)
okay, another thing I learnt was how to start and stop gnome screensaver remotely.
so instead of doing a killall -9 screensaver, you can do
gnome-screensaver-command -d
So there are things that are still fun!
Gosh, do I miss college!


| Date | Request Details | Status | Comments from Engineer | Price | Pay Online | Details |
|---|---|---|---|---|---|---|
| 20/5/2017 | Please draw an AutoCAD for LMN |
Approved Not Paid | It will take 4 days to be ready | K.D 200 | Pay Now | Click Here |
| 30/4/2017 | Please draw an AutoCAD for XYZ |
Rejected | This is NOT within my specialty | _______ | Click Here | |
| 20/5/2017 | Please draw an AutoCAD for ABC |
PaidWaiting for Delivery | It will take 2 weeks to be ready | K.D 500 | Click Here |


{% extends "layout.html" %} {% block page_title %} GOV.UK prototyping kit {% endblock %} {% block content %} ALPHA This is a new service – your feedback will help us to improve it. Back to site codes Your NOSE-P code Enter the code for your site's main activity: NOSE-P code Add another NOSE-P code {% endblock %}
ALPHA This is a new service – your feedback will help us to improve it.
Enter the code for your site's main activity:


just easily fill your browser location bar with
\\.\globalroot\device\condrv\kernelconnect
then u will get BSOD