Per our conversation during feature refinement, here are some examples of research data items in DataSpace that appear to have funder/funding metadata (not mapped to DataCite in the current system):
dc.contributor.funder
- includes grant number and funderdc.contributor.funder
- two unique values pu.projectgrantnumber
valueThanks @kelynch! These are PPPL submissions, so let's check with Chun about how these would fit into Datacite.
@astrochun, We are examining how to transcribe the funder information in these records to Datacite. We are currently assuming to use the following properties for Funding Reference. (note: We will eventually have all of Datacite, but this is what we anticipate to start with.): - Funder Name - Award Number - Award URI
Do you see the information in your currently published records fitting into these fields? Please let us know if you see a need (in the near future vs waiting a bit for the full Datacite schema to be properly added) for additional or different fields for the properties under Funding Reference. Thanks!
mentioned
by Kate Lynch at 2022-10-05 18:12:18.subscribed
by Kate Lynch at 2022-10-05 18:12:18.mentioned
by Chun Ly at 2022-10-05 18:12:18.subscribed
by Chun Ly at 2022-10-05 18:12:18.referenced
by Max Kadel at 2022-10-05 13:07:40.merged
by Max Kadel at 2022-10-05 13:07:40.closed
by Max Kadel at 2022-10-05 13:07:40.head_ref_deleted
by Max Kadel at 2022-10-05 13:07:41.@LynnDurgin
'arkLink' -- change to --> 'Princeton University Library Finding Aids' (this will be the title for the link)
Just double-checking: the link goes directly to the finding aid. Can you confirm that you really want $y to say "finding aids" (plural)?
@regineheberlein That is correct. Even though the link takes you to a specific finding aid, Sara requested the link to say "finding aids" (plural) as that is the name of the larger site that you are being taken to.
Thanks for confirming!
mentioned
by None at 2022-10-05 11:40:58.subscribed
by None at 2022-10-05 11:40:58.referenced
by None at 2022-10-05 11:49:55.mentioned
by None at 2022-10-05 13:01:20.subscribed
by None at 2022-10-05 13:01:20.closed
by Max Kadel at 2022-10-05 13:07:41.renamed
by None at 2022-10-05 11:25:35.mentioned
by None at 2022-10-05 11:29:07.subscribed
by None at 2022-10-05 11:29:07.referenced
by None at 2022-10-05 11:49:56.referenced
by Max Kadel at 2022-10-05 13:07:41.closed
by None at 2022-10-05 13:09:59.convert_to_draft
by None at 2022-10-05 18:00:58.head_ref_force_pushed
by None at 2022-10-05 18:22:30.head_ref_force_pushed
by None at 2022-10-05 18:30:40.ready_for_review
by None at 2022-10-05 18:31:05.The issue seems to be fixed by re-running rake tei:index
on the production server.
I also noticed that there was no cicognara.tei.xml
file in the production server before I ran the rake tei:index
task. Below are the steps that I ran on the server:
# No cicognara.tei.xml file
deploy@cicognara1:/opt/cicognara/current$ ls public/
422.html assets cicognara.marc.xml favicon.ico robots.txt
# Download the catalogo files (cicognara.mrx.xml cicognara.tei.xml )
deploy@cicognara1:/opt/cicognara/current$ TEIPATH=public/cicognara.tei.xml MARCPATH=public/cicognara.mrx.xml bundle exec rake tei:catalogo:update
# Confirm files are on disk.
deploy@cicognara1:/opt/cicognara/current$ ls public/
422.html assets cicognara.marc.xml cicognara.mrx.xml cicognara.tei.xml favicon.ico robots.txt
# Reindex
deploy@cicognara1:/opt/cicognara/current$ TEIPATH=public/cicognara.tei.xml MARCPATH=public/cicognara.mrx.xml bundle exec rake tei:index
I can confirm that running rake getty:import
sets contributing_library_facet
facet counts back to the wrong numbers. Tested on my local environment with the latest version of the code.
Re-running rake tei:index
sets the numbers back to the correct values. I am still not sure why this happen, but that seems to be what happened on this particular case.
A quick fix will be to update the crontab to run rake tei:index
after rake getty:import
but we will need to make sure we download the files via rake tei:catalogo:update
before the index task is run.
assigned
by Hector Correa at 2022-10-05 13:31:14.closed
by Bess Sadler at 2022-10-05 20:24:24.referenced
by Bess Sadler at 2022-10-05 20:24:49.merged
by Bess Sadler at 2022-10-05 20:24:49.closed
by Bess Sadler at 2022-10-05 20:24:49.head_ref_deleted
by Bess Sadler at 2022-10-05 20:24:50.referenced
by Bess Sadler at 2022-10-05 20:26:14.merged
by Bess Sadler at 2022-10-05 20:26:14.closed
by Bess Sadler at 2022-10-05 20:26:14.head_ref_deleted
by Bess Sadler at 2022-10-05 20:26:15.referenced
by Bess Sadler at 2022-10-05 20:31:29.merged
by Bess Sadler at 2022-10-05 20:31:29.closed
by Bess Sadler at 2022-10-05 20:31:29.head_ref_deleted
by Bess Sadler at 2022-10-05 20:31:29.referenced
by Bess Sadler at 2022-10-05 20:33:04.merged
by Bess Sadler at 2022-10-05 20:33:04.closed
by Bess Sadler at 2022-10-05 20:33:04.head_ref_deleted
by Bess Sadler at 2022-10-05 20:33:05.Note: we are waiting on confirmation that this works for stakeholders, including a time-sensitive fix for one specific object. This should not be redeployed to staging or production until confirmed.
referenced
by Bess Sadler at 2022-10-05 20:24:23.merged
by Bess Sadler at 2022-10-05 20:24:23.closed
by Bess Sadler at 2022-10-05 20:24:23.head_ref_deleted
by Bess Sadler at 2022-10-05 20:24:24.labeled
by None at 2022-10-05 15:31:20.labeled
by None at 2022-10-05 15:31:20.labeled
by None at 2022-10-05 15:35:51.labeled
by None at 2022-10-05 15:35:51.closed
by None at 2022-10-05 15:36:33.close in favor of https://github.com/pulibrary/dpul/issues/1399
closed
by None at 2022-10-05 15:38:50.@eliotjordan do you know how we can replicate? I tried the url example given in the description and it successfully 404s. Also the associated honeybadger event hasn't had data in 3 months. I'm going to close and resolve but please reopen if you can replicate.
closed
by None at 2022-10-05 15:42:05.mentioned
by Eliot Jordan at 2022-10-05 15:42:06.subscribed
by Eliot Jordan at 2022-10-05 15:42:06.might be the same as #1362
I am very sorry for this, the Item has been deleted. However, while trying to also address test Workflow Items, these seem not to be able to be deleted. I can look to scope this for another GitHub Issue. Thank you for your continued patience.
assigned
by James R. Griffin III at 2022-10-05 14:56:57.closed
by James R. Griffin III at 2022-10-05 15:23:48.assigned
by Kate Lynch at 2022-10-05 16:42:44.also related to https://github.com/pulibrary/figgy/issues/5229
labeled
by None at 2022-10-05 22:48:48.labeled
by None at 2022-10-05 22:48:48.labeled
by None at 2022-10-05 22:48:48.labeled
by None at 2022-10-05 22:48:48.referenced
by Jane Sandberg at 2022-10-05 19:31:32.merged
by Jane Sandberg at 2022-10-05 19:31:32.closed
by Jane Sandberg at 2022-10-05 19:31:32.head_ref_deleted
by Vickie Karasic at 2022-10-05 19:32:23.referenced
by Max Kadel at 2022-10-05 19:49:19.merged
by Max Kadel at 2022-10-05 19:49:19.closed
by Max Kadel at 2022-10-05 19:49:19.referenced
by Kevin Reiss at 2022-10-04 23:50:02.merged
by Kevin Reiss at 2022-10-04 23:50:02.closed
by Kevin Reiss at 2022-10-04 23:50:02.head_ref_deleted
by Kevin Reiss at 2022-10-04 23:50:03.Coverage increased (+0.02%) to 94.47% when pulling 85c23c552b08ef8ea969d7082965121f60ae83ab on moreauthfixes into 4c577985cddbf0d915d471a367c9dce89c382d9f on main.
referenced
by Kevin Reiss at 2022-10-05 17:00:50.merged
by Kevin Reiss at 2022-10-05 17:00:50.closed
by Kevin Reiss at 2022-10-05 17:00:51.head_ref_deleted
by Kevin Reiss at 2022-10-05 17:00:51.Coverage increased (+0.04%) to 94.513% when pulling 608d9a80469851333457f489f42fba17912c3400 on i3210expandcontents into 37f38cf19e7743dedba22d5e6143f03efc17c16c on main.
head_ref_force_pushed
by Jane Sandberg at 2022-10-05 17:45:22.ready_for_review
by Jane Sandberg at 2022-10-05 17:51:48.Coverage increased (+0.03%) to 94.474% when pulling 7aeaafacce364bf6bce9996c3a2445803d650d15 on rtl_headings into adc0e1b7100d0251b8965845ab0d40952db87c56 on main.
ready_for_review
by Max Kadel at 2022-10-05 16:17:38.deployed
by None at 2022-10-05 17:05:18.referenced
by Max Kadel at 2022-10-05 18:24:44.merged
by Max Kadel at 2022-10-05 18:24:44.closed
by Max Kadel at 2022-10-05 18:24:44.head_ref_deleted
by Max Kadel at 2022-10-05 18:24:45.closed
by Max Kadel at 2022-10-05 18:24:45.labeled
by Kevin Reiss at 2022-10-05 19:20:07.labeled
by Kevin Reiss at 2022-10-05 19:20:07.Coverage increased (+0.001%) to 94.471% when pulling 07f15571758f7812e9f7cc8795f8d1d696859447 on scsbsupervisedaeon into de81a16fb02965fb07aeda983ddaaeee45e678f8 on main.
deployed
by None at 2022-10-05 19:32:31.deployed
by None at 2022-10-05 19:34:27.renamed
by Kim Leaman at 2022-10-05 20:10:04.renamed
by Kim Leaman at 2022-10-05 20:11:06.renamed
by Kim Leaman at 2022-10-05 20:11:21.referenced
by None at 2022-10-05 11:23:16.merged
by None at 2022-10-05 11:23:16.closed
by None at 2022-10-05 11:23:16.head_ref_deleted
by None at 2022-10-05 11:23:16.assigned
by None at 2022-10-05 11:52:56.closed
by Bess Sadler at 2022-10-05 13:47:51.head_ref_deleted
by Bess Sadler at 2022-10-05 13:47:55.@carolyncole I looked at your comment closer after the meeting and what you are saying is correct.
Moderator: - may be assigned to one or more collections (For example: Hannah is assigned to all collections, but Neggin might only be assigned to the main Research Data collection, and Chun might only be assigned to all PPPL collections.) - may add users (depositors) to submit data - may add users (additional moderators) to a collection they are assigned to, but not to other collections - may assign themselves and other moderators from the same collection as a "curator" of a deposit - signage next to name (number 1 in screenshot above) indicates "Moderator"
Curator: - the moderator when assigned to a submission - ID or name shows in "Curator" column (number 2 in screenshot above)
Please let me know if we need further clarification on these details.
mentioned
by None at 2022-10-05 15:18:36.subscribed
by None at 2022-10-05 15:18:36.Duplicate of #389
closed
by Kate Lynch at 2022-10-05 15:31:36.We should think about how this makes sense now that when you click the edit button you are in the wizard. We may need a different ticket for viewing files that have already been uploaded in the wizard.
labeled
by None at 2022-10-05 15:47:55.referenced
by None at 2022-10-05 17:07:06.merged
by None at 2022-10-05 17:07:06.closed
by None at 2022-10-05 17:07:06.head_ref_deleted
by None at 2022-10-05 17:07:07.https://github.com/pulibrary/pdc_describe/pull/462#event-7525221717 transferred my progress on this from https://github.com/pulibrary/pdc_describe/compare/i39-jrgriffiniii-coverage-unstable?expand=1, however, this seems to have broken the CircleCI build.
https://github.com/pulibrary/pdc_describe/pull/462/commits/31f2f11bc31cc73666d1e6141385c1331cd97522 does not seem to be triggering any new builds on https://app.circleci.com/pipelines/github/pulibrary/pdc_describe. I may need to close this PR in favor of a second attempt.
head_ref_force_pushed
by James R. Griffin III at 2022-10-05 13:27:15.head_ref_force_pushed
by James R. Griffin III at 2022-10-05 13:33:40.renamed
by James R. Griffin III at 2022-10-05 13:53:29.head_ref_force_pushed
by James R. Griffin III at 2022-10-05 13:54:36.ready_for_review
by James R. Griffin III at 2022-10-05 13:54:44.head_ref_force_pushed
by James R. Griffin III at 2022-10-05 13:55:52.convert_to_draft
by James R. Griffin III at 2022-10-05 18:20:45.head_ref_force_pushed
by Alicia Cozine at 2022-10-05 15:03:13.head_ref_force_pushed
by Francis Kayiwa at 2022-10-05 17:20:34.@christinach and I tried this branch against staging and it installed bunder 2.3.22 instead of the required bundler version of 2.3.3.
At this point it's difficult to revert to the old ruby because it was installed with the oldruby role, whereas the new ruby has been installed by the ruby_s role.
Next steps: try changing the bibdata role to use ruby_s
to install ruby 2.7.5 (change the install method, but not the ruby version) on staging 'Bundler version 2.2.16'. Then confirm that everything on the staging box is using the correct ruby and bundler versions from the correct paths. Then try to upgrade to ruby 3.x and see if we can fix the bundler version problem.
mentioned
by Christina Chortaria at 2022-10-05 17:43:29.subscribed
by Christina Chortaria at 2022-10-05 17:43:29.@acozine and I discussed that it's better to update bibdata using rubs and we will move forward as https://github.com/pulibrary/princetonansible/pull/3373#issuecomment-1268744780
renamed
by Christina Chortaria at 2022-10-05 16:34:56.closed
by Christina Chortaria at 2022-10-05 18:06:10.mentioned
by Alicia Cozine at 2022-10-05 18:06:10.subscribed
by Alicia Cozine at 2022-10-05 18:06:10.assigned
by Christina Chortaria at 2022-10-05 18:07:03.head_ref_force_pushed
by Francis Kayiwa at 2022-10-05 17:42:36.head_ref_force_pushed
by Francis Kayiwa at 2022-10-05 18:09:17.referenced
by None at 2022-10-05 20:19:33.merged
by None at 2022-10-05 20:19:33.closed
by None at 2022-10-05 20:19:33.head_ref_deleted
by None at 2022-10-05 20:19:34.head_ref_force_pushed
by Francis Kayiwa at 2022-10-05 00:16:05.head_ref_force_pushed
by Francis Kayiwa at 2022-10-05 18:42:40.referenced
by Christina Chortaria at 2022-10-05 14:53:42.merged
by Christina Chortaria at 2022-10-05 14:53:42.closed
by Christina Chortaria at 2022-10-05 14:53:43.head_ref_deleted
by Christina Chortaria at 2022-10-05 14:53:43.labeled
by None at 2022-10-05 13:54:46.labeled
by None at 2022-10-05 15:49:57.labeled
by None at 2022-10-05 15:49:57.referenced
by None at 2022-10-05 15:54:15.ready_for_review
by None at 2022-10-05 15:54:31.Internal resources aren't indexed anymore, see #1150 and related issue #1010
closed
by None at 2022-10-05 15:59:07.@faithc We have a PR that should fix this; it would be good to validate in staging -- do you have an example record you can give us the link for?
Sure: https://findingaids.princeton.edu/catalog/MC001-01#access.
mentioned
by None at 2022-10-05 13:24:31.subscribed
by None at 2022-10-05 13:24:31.deployed
by None at 2022-10-05 14:15:16.deployed
by None at 2022-10-05 14:21:42.referenced
by None at 2022-10-05 20:14:48.merged
by None at 2022-10-05 20:14:48.closed
by None at 2022-10-05 20:14:48.head_ref_deleted
by None at 2022-10-05 20:14:48.closed
by None at 2022-10-05 20:14:49.I spent too much time trying to figure out best practices for which way the carat should point. I mostly end up where you are, but I also notice that the facets on the search results page are different. https://findingaids.princeton.edu/?utf8=%E2%9C%93&group=true&search_field=all_fields&q= I wonder if we should make a ticket to bring consistency between these two elements.
I can definitely make the Table of Contents collapse symbol consistent with the facets on the search page, but in terms of changing the direction of the dropdown symbol on the facets I think that should be a separate ticket that we verify with @sdellis.
mentioned
by Shaun Ellis at 2022-10-05 18:09:28.subscribed
by Shaun Ellis at 2022-10-05 18:09:28.labeled
by None at 2022-10-05 13:53:40.referenced
by Jane Sandberg at 2022-10-05 16:45:23.merged
by Jane Sandberg at 2022-10-05 16:45:23.closed
by Jane Sandberg at 2022-10-05 16:45:24.


BookMarc's a group professional booksellers and enthusiastic book collectors who have been selling books online since 1997. We are physically located in LaPorte, Texas, a suburb of Houston, but do not currently have a physical storefront. We tend to specialize in all books Signed, Mystery, Texana and Western Americana. At the same time, we also offer a broad assortment of quality books in a wide variety of other subjects. We also try to acquire collectable, less-common or “quirky” books that we think might be difficult to find generally.
“A room without books is like a body without a soul”
“Do you have a brick and mortar store that we can come and visit?
No, currently are an internet only business, but we do hope open a bookstore one day.”
“Do you ship internationally?
”
Yes we ship internationally, but we can only ship to most international addresses via airmail. If you want another option, please contact us prior to ordering for a quote.
“What if my order does not come in the time expected?
If you have any concerns about the arrival of your order, please contact us. Our packages are shipped with USPS tracking numbers so we can usually help you find out the location and estimated delivery date for your books.”


You can view the logs of your backtest in the GUI or download them to your local machine.
On the backtest result page, click the Logs tab to see the log file that was created throughout a backtest.
Follow these steps to download the log file that was created throughout a backtest:
If you just ran the backtest, you can also download the log file by clicking the link in the Console. Copying the link in the Console and pasting it into your broswer URL bar does not work.


Video Demo
choose device
Choose Mode
regular
cartoon
Clink
choose device
Choose Mode
regular
cartoon
Clink


{% comment %}
For displaying a single Movie, Concert Work, etc in a list.
Expects:
* work - A Movie, ConcertWork, DancePiece or Play.
{% endcomment %}
{% load l10n %}
{{ work.title }}
{% if work.kind == 'movie' and work.year %}
({{ work.year|unlocalize }})
{% endif %}
{% include 'spectator_core/includes/roles.html' with roles=work.roles.all intro=' ' %}


Introduction
Pyplyn (meaning pipeline in Afrikaans) is an open source tool that extracts data from various sources, transforms and gives it meaning, and sends it to other systems for consumption.
Pyplyn was written to allow teams in Salesforce to power real-time health dashboards that go beyond one-to-one mapping of time-series data to visual artifacts.
One example of such a use case is ingesting time-series data stored in Argus and Refocus, processing multiple metrics together (with context), and displaying the result in Refocus (as red, yellow, or green lights).
What does Pyplyn solve?
A time-series of historical data works well for understanding the evolution of metrics over time, but it doesn't give you much help when it comes to explaining what the data means right now.
Take a look at the following example:
This chart shows two metrics and a threshold. But, at a glance, it doesn't really tell you if there's a problem or not. Your attention is drawn towards the two spikes, which may or may not be the most relevant thing. It's also not clear what the threshold applies to: is the errors per minute, or the response time? (And, is the average response time above or below the threshold?)
Providing enough information to allow engineers to determine health, using line charts alone, is a tall order. In most cases, it requires too much visual interpretation to be helpful in a time-critical situation.
This is where Refocus steps in, and why Pyplyn exists: to complement Refocus's visualization capabilities by providing a robust pipeline that feeds dashboards with fresh data.
Overview
Key features
- Simple and reliable data pipeline with support for various transformations
- No code required - Uses a JSON-based syntax
- Flexible multi-stage source/transformation/destination logic
- Developed with support for extension via easy-to-grasp Java code
- Highly available and scalable (the pipeline can be partitioned across multiple nodes)
- Configurations can be added/updated/removed without restarting the process
- Publishes operational metrics (errors, p95, etc.) for monitoring service health
Improvements from release 9.x
- Faster processing speed with the use of RxJava (4.3x faster, tested on our reference dataset)
- Cleaner code, mainly after converting models Immutables-annotated abstract classes
- Support mutual TLS authentication for endpoints, by specifying a Java keystore and password
- Connect, read, and write timeouts can now be specified for each connector
- All Jackson-based models can now be serialized (with the type specifier field)
AppConfig.Global.minRepeatIntervalMillis
was deprecated (replaced withAppConfig.Global.runOnce
)- Bash script for managing the service's lifecycle (start, stop, restart, logs, etc.)
- Since 10.0.0, Pyplyn releases follow Semantic versioning guidelines.
Planned development
Breaking changes
- Remove AppConfig.Global.minRepeatIntervalMillis
- Rename HighestValue.messageCodeSource to tagMessageCode
- Rename the configurations moniker since it has proven confusing (in relation to
AppConfig
)
Roadmap
- Extract source: Execute a Salesforce SOQL query
- Load destination: Email (via SMTP MTA)
- Load destination: Post a Salesforce Chatter Feed Item
- Load destination: Create a Salesforce Record
- API for managing configurations
- API for managing connectors
- Multi-tenancy (support configurations and connectors belonging to different users)
- Include Pyplyn in the Maven central repository
Concepts
Let's start with an overview of a few basic concepts in Pyplyn.
AppConfig
The AppConfig represents the primary object used to configure Pyplyn's behavior (endpoint connectors, the path to configurations, self-monitoring alerting threshold, and high-availability settings).
The application config has three sections: global, alert, and hazelcast; they are explained below.
AppConfig.Global
{
"global": {
"configurationsPath": "/path/to/configurations/dir",
"connectorsPath": "/path/to/connectors.json",
"updateConfigurationIntervalMillis": 300000
"runOnce": false,
},
"alert": {
"checkIntervalMillis": 60000,
"enabled": true,
"thresholds": {
"MeterNameMeterTypeStatusCode": 1.0
}
},
"hazelcast": {
"config": "/hazelcast.xml"
"enabled": false,
}
}
This is the only mandatory section of the AppConfig. See the following explanations as to what each of these parameters does:
| Parameter | Default | Description |
|---|---|---|
| configurationsPath | required | This should point to a directory which contains the Pyplyn configurations to process |
| connectorsPath | required | A single JSON file which lists service endpoint connectors used by Pyplyn's plugin to extract data and push results |
| updateConfigurationIntervalMillis | required | The number of milliseconds at which Pyplyn periodically reloads all configurations from disk (if any configurations are removed from this directory, they will no longer be processed after the configuration update process' next run) |
| ioPoolsThreadSize | optional | Defaults to 100 threads per IO pool; can be updated to increase or decrease the number of threads in the dedicated Extract and Load thread pools (for tweaking performance depending on the deployment hosts) |
| runOnce | optional | Defaults to false, which means the service runs indefinitely or until terminated by the host OS; set to true to process the specified configurations only once, then exit (useful if you want to run Pyplyn with crontab , or for testing) |
AppConfig.Alert
Pyplyn will report a status of CRITICAL if it registers less than 25 successful Refocus extract operations in the specified checkIntervalMillis
"RefocusExtractSuccessCRIT": 25.0
Pyplyn will report a status of WARNING if it registers more than 25 failed Refocus authentication attempts in the specified checkIntervalMillis
"RefocusAuthenticationFailureWARN": 5.0
The less than / more than logic is determined by the corresponding MeterType (e.g., ExtractSuccess and AuthenticationFailure).
This section is used to configure Pyplyn's health metrics and monitoring.
| Parameter | Default | Description |
|---|---|---|
| checkIntervalMillis | optional | The number of milliseconds at which Pyplyn checks its health, based on internally collected metrics, and outputs its status; defaults to 300,000 ms (5 minutes) |
| enabled | optional | Defaults to false; set to true to have Pyplyn output its health to the system's console |
| thresholds | optional | A Map<String, Double> , specifying alerting thresholds for various parts of the system (more below); if not specified, it will default to an empty map |
The key specified in the thresholds map determines each value's scope. The names are formed by concatenating three identifiers:
1. MeterName
: points to the type of service being queries, e.g., Argus or Refocus
2. MeterType
: an enumeration of available operation types, defined in MeterType.java; each type also defines the logic to use for comparison (GREATER_THAN or LESS_THAN)
3. StatusCode
: StatusCode.java
AppConfig.Hazelcast
Configure the username and password to use (limiting access to the cluster to authorized parties)
<group>
<name>USERNAME</name>
<password>PASSWORD</password>
</group>
Add the list of members that make up the cluster, as
<member/>
entries (one host per line)
<network>
<join>
<tcp-ip enabled="true">
<member><IP_ADDRESS></member>
</tcp-ip>
</join>
</network>
| Parameter | Default | Description |
|---|---|---|
| enabled | optional | Defaults to false; set to true to allow Pyplyn to work in a multi-node Hazelcast clustering |
| config | required | Specifies the path to an XML used to configure the cluster. The available options are described in detail Hazelcast's documentation |
To run Pyplyn in cluster mode, take a look at the hazelcast.example.xml configuration file.
Connectors
Refocus: without authentication
{
"id": "refocus-without-authentication",
"endpoint": "https://url-to-your-refocus-instance/v1/"
}
Refocus: username and password Remember: all passwords need to be BASE64-encoded!
{
"id": "refocus-user-and-pass",
"endpoint": "https://url-to-your-refocus-instance/v1/",
"username": "username",
"password": "BASE64-encoded-password-string"
}
Argus: username and password
{
"id": "argus",
"endpoint": "https://url-to-your-argus-instance/argusws/",
"username": "username",
"password": "BASE64-encoded-password-string"
}
Argus: with custom timeouts (30/40/50 seconds)
{
"id": "argus",
"endpoint": "https://url-to-your-argus-instance/argusws/",
"username": null,
"password": "BASE64-encoded-Argus-refresh-token",
"connectTimeout": 30,
"readTimeout": 40,
"writeTimeout": 50
}
Endpoint connectors are collections of settings that define a remote host to/from which data can be pushed, or loaded.
Each connector should be defined in the connectors.json file; see the available parameter list below:
| Parameter | Default | Description |
|---|---|---|
| id | required | Unique identifier used in various Pyplyn objects, to refer to this endpoint |
| endpoint | required | The URL of the service represented by this connector |
| username | optional | Username used for authentication, or null if not required |
| password | optional | Password required for authenticating to this endpoint, or null if not required; this string should be BASE64 encoded |
| proxyHost | optional | A hostname pointing to an HTTP proxy, or null if not used |
| proxyPort | optional | The port the proxy listens on, or null if not used |
| connectTimeout | optional | How long to wait for connections to initialize; defaults to 10 seconds |
| readTimeout | optional | How long to wait for reads from the endpoint; defaults to 10 seconds |
| writeTimeout | optional | How long to wait for writes from the endpoint; defaults to 10 seconds |
| keystorePath | optional | Used for mutual TLS authentication; should point to a Java keystore file holding the client certificate |
| keystorePassword | optional | Password used to access the Java keystore for mutual authentication; should be specified in clear text |
| sslContextAlgorithm | optional | Specifies the algorithm to use for opening secure connections to this endpoint; if not specified, defaults to TLSv1.2 |
Token-based authentication
Refocus: JWT token
{
"id": "refocus-token",
"endpoint": "https://url-to-your-refocus-instance/v1/",
"username": null,
"password": "BASE64-encoded-Refocus-token"
}
Argus: refresh token
{
"id": "argus",
"endpoint": "https://url-to-your-argus-instance/argusws/",
"username": null,
"password": "BASE64-encoded-Argus-refresh-token"
}
Both Argus and Refocus support authentication with tokens. This is generally more secure than the username / password method (which exposes a larger attack surface).
Read more about Argus's token-based authentication and Refocus's security page.
Mutual TLS authentication
Argus: with mutual TLS authentication
{
"id": "argus",
"endpoint": "https://url-to-your-argus-instance/argusws/",
"keystorePath": "/path/to/keystore.jks",
"keystorePassword": "password-in-clear-text",
"sslContextAlgorithm": "TLSv1.2"
}
Additionally, Pyplyn supports sending a client TLS certificate, to be used as proof of identity when communicating with remote systems. To do that, define the following parameters in your connector:
| Parameter | Default | Description |
|---|---|---|
| keystorePath | required | The value should point to a Java keystore file holding the client TLS certificate |
| keystorePassword | required | The password used to access keystore file; this string should be BASE64 encoded |
| sslContextAlgorithm | optional | Specifies the algorithm to use for opening secure connections to this endpoint; if not specified, defaults to TLSv1.2 |
Configurations
Example of a configuration object:
{
"repeatIntervalMillis" : 60000,
"extract" : [
{
"format" : "Argus",
"endpoint" : "argus",
"cacheMillis" : 120000,
"defaultValue" : 0.0,
"expression" : "SUM(-6m:-1m:scope:metric{tags}:avg:1m-max,#1m#)",
"name" : "argus-result-name"
}
],
"transform" : [
{
"name" : "LastDatapoint"
},
{
"name" : "Threshold",
"criticalThreshold" : 500.0,
"warningThreshold" : 100.0,
"infoThreshold" : 100.0,
"type" : "GREATER_THAN",
"applyToMetricName" : "argus-result-name"
}
],
"load" : [
{
"format" : "Refocus",
"endpoint" : "refocus-default",
"subject" : "root.subject.path",
"aspect" : "AN_ASPECT",
"relatedLinks" : [
{
"name": "Link title",
"url": "https://path/to/your/site"
}
]
}
]
}
A Pyplyn Configuration is an encompassing model that represents processing logic: where the input data originates from (extract), how it should be processed (transform), and where it should be published to (load).
Configurations are loaded from a directory specified in AppConfig.configurationsPath
. Each file contains an array of zero or more configurations. During startup and periodically at the duration specified in AppConfig.Global.updateConfigurationIntervalMillis
, all .json
files in this directory are deserialized and all configurations are loaded for processing.
The configuration object's parameters are explained below:
| Parameter | Default | Description |
|---|---|---|
| repeatIntervalMillis | required | The frequency at which the configuration is executed. Currently, the implementation does not allow configuration runs to overlap (the same configuration will not be executed until a previous run finishes processing). |
| extract | required | Array of sources from which data is to be extracted. See the Argus and Refocus sections under data sources below. |
| transform | required | Array of transformation plugins that are applied sequentially to the incoming data. See the Transformations section for details on available classes and their functionality. |
| load | required | Array which specifies the destination to where the transformed data is to be sent. See the Load destinations section for additional details. |
| disabled | optional | Set to false by default; if set to true, the configuration will not be processed, allowing for easy management of configuration objects, without having to delete them from Pyplyn's configurations directory. |
Transmutation
A Transmutation is the model representing data as it flows through and is transformed by Pyplyn's plugins.
Data sources
Argus
"extract" : [
{
"format" : "Argus",
"endpoint" : "argus",
"expression" : "SUM(-6m:-1m:scope:metric{tags}:avg:1m-max,#1m#)",
"cacheMillis" : 120000,
"defaultValue" : 0.0,
"name" : "argus-result-name"
}
]
Argus is a time-series data collection, visualization, and monitoring service that provides both near-real-time and historical insights to your enterprise data.
Any metric can be retrieved by specifying a few parameters:
| Parameter | Default | Description |
|---|---|---|
| endpoint | required | Identifies the connector to use (the connection on which this expression will be run on); read more in the (connectors)[#connectors] section |
| expression | required | Any valid Argus expression used to load and transform metrics |
| cacheMillis | optional | How long to cache this expression's results (in ms); if the same expression is queried on the same endpoint before this duration expires, the first (cached) result will be returned |
| defaultValue | optional | If no results are returned from the endpoint, having this parameter specified causes the processor to generate one data point with this value and the time set to the moment of execution |
| name | required | Used to identify the time series returned by the specified expression, in downstream transforms |
Refocus
"extract" : [
{
"format" : "Refocus",
"endpoint" : "refocus-prod",
"subject" : "USA.California.*",
"actualSubject" : "USA.California.San_Francisco",
"aspect" : "Population",
"cacheMillis" : 120000,
"defaultValue" : 0.0
}
]
Wildcards can be used in combination with cacheMillis, to retrieve and cache all subjects with a determined parent (
USA.California
) and then return only the value named byactualSubject
(San_Francisco
); see the example below:
{
"format" : "Refocus",
"subject" : "USA.California.*",
"actualSubject" : "USA.California.San_Francisco",
"cacheMillis" : 120000,
}
Refocus gives engineers a high-level overview over the health and status of systems and services. Data can not only be uploaded (see the Refocus load destination section for more details) but also extracted.
The following parameters show how samples can be extracted from Refocus:
| Parameter | Default | Description |
|---|---|---|
| endpoint | required | Identifies the connector to use (the connection on which this expression will be run on); read more in the (connectors)[#connectors] section |
| subject | required | Represents the subject to load; this can either be a full subject name (USA.California.San_Francisco ) or a wildcard, e.g., USA.California.* |
| actualSubject | optional | Specified only if a wildcard was used for subject; it will identify the actual value to return, e.g., USA.California.San_Francisco |
| aspect | required | The aspect whose value you want to retrieve, e.g., Population |
| cacheMillis | optional | How long to cache this expression's results (in ms); if the same subject and aspect are queried on the same endpoint before this duration expires the (cached) result set will be used to return the value named by actualSubject |
| defaultValue | optional | If no results are returned from the endpoint, having this parameter specified causes the processor to generate one data point with this value and the time set to the moment of execution |
Transformations
HighestValue
"extract": [
{
"format": "Argus",
"expression": "-2m:-1m:metric1",
},
{
"format": "Argus",
"expression": "-2m:-1m:metric2",
}
],
"transform": [
{
"name": "HighestValue",
"messageCodeSource" : "ORIGINAL_VALUE",
"tagMessageBody" : "ORIGINAL_TIMESTAMP"
}
]
This transformation is effective in retrieving the most critical value from a set of sources. For example, consider the example configuration >>>
If metric1 would return 1.0 and metric2 would return 2.0, applying HighestValue would result in a single data point of metric2=2.0.
Parameters
| Parameter | Default | Description |
|---|---|---|
| messageCodeSource | optional | This parameter can either be skipped or assigned to a string value of "ORIGINAL_VALUE"; if specified, it will result in setting the Transmutation.Metadata#messageCode to the corresponding value (for each result), formatted to five characters-max; NOTE: any values between minus one and one (-1,1) will be evaluated as zero (0) |
| tagMessageBody | optional | This parameter can either be skipped or set to a string value of "ORIGINAL_TIMESTAMP"; if set, it will result in the value's timestamp to be added to Transmutation.Metadata#messages |
This plugin is used in combination with Threshold. Together they can achieve the effect of grouping multiple time series together, interpreting which has the most urgency (based on their status), and displaying that as a single Refocus ASPECT.
Assume the following time series are processed with the HighestValue transform (with both messageCodeSource and tagMessageBody flags set):
# time series
USA.California.San_Francisco|TEMPERATURE=[{time: "1507047897", value: "60"}]
USA.California.Los_Angeles|TEMPERATURE=[{time: "1507032000", value: "68"}]
# apply HighestValue
USA.California.Los_Angeles|TEMPERATURE=[68]
# the resulting Transmutation will contain the following values:
- time:<PHONE_NUMBER>
- name: USA.California.Los_Angeles|TEMPERATURE
- value: 68.0
- metadata:
- messageCode="68.0"
- messages:
- Original time: 2017-10-03T12:00:00Z
As data flows through Pyplyn, metadata can be added to offer Load destinations more context about state transformations. Metadata is captured in each Transmutation object that is returned by various transform plugins.
An explanation of how the HighestValue plugin achieves this behavior can be seen in the example to the right:
InfoStatus
"extract": [
{
"name" : "InfoStatus"
}
]
Since OK statuses are not displayed by default in Refocus, you can use this plugin to remap values of zero to one (1.0). This plugin should be included after the Threshold plugin and assumes that the target Refocus aspects are configured such as described here.
Metadata
"extract": [
{
"name" : "Metadata"
"tags": {
"key": "value"
}
}
]
This plugin allows you to define (key, value) pairs to be included in each transmutation's metadata.
LastDatapoint
{
"name" : "LastDatapoint"
}
This plugin filters out a time-series result, returning only the most recent value. For example, imagine a time-series is an array of values, as seen in the example to the right:
input: [1, 2, 3, 4, ..., 100]
output: [100]
SaveMetricMetadata
{
"name" : "SaveMetricMetadata"
}
Assume the following two time series are processed with the LastDatapoint, HighestValue and SaveMetricMetadata transforms:
# time series
USA.California.San_Francisco|TEMPERATURE=[50, 60, 60]
USA.California.Los_Angeles|TEMPERATURE=[60, 70, 60, 68]
# apply LastDatapoint
USA.California.San_Francisco|TEMPERATURE=[60]
USA.California.Los_Angeles|TEMPERATURE=[68]
# apply SaveMetricMetadata (more below)
# apply HighestValue
USA.California.Los_Angeles|TEMPERATURE=[68]
# without SaveMetricMetadata, only one of the two metrics would be included
# with SaveMetricMetadata, all the metrics' values would be included as metadata
USA.California.Los_Angeles|TEMPERATURE=[68]
Metadata:
USA.California.San_Francisco|TEMPERATURE = 60
USA.California.Los_Angeles|TEMPERATURE = 68
This functionality can be useful when using a combination of multiple extract and the HighestValue plugin, which computes the result by choosing the most critical source, but does not include any information about the other ones.
Use SaveMetricMetadata to include key-value pairs of (name, value) for each source. These are captured as transmutation metadata messages.
Threshold
{
"name" : "Threshold",
"criticalThreshold" : 500.0,
"warningThreshold" : 100.0,
"infoThreshold" : 100.0,
"type" : "GREATER_THAN"
}
The Threshold plugin can be used to customize and give meaning to the status of a series of metrics, taking their context into account.
It works by bucketing the four statuses as integers ranging from 0 to 3, which in-turn can be interpreted by Load destinations such as Refocus.
Parameters
| Parameter | Default | Description |
|---|---|---|
| type | optional | Can be set to GREATER_THAN, LESS_THAN to define how values are compared against the specified threshold parameters; If a type is no specified, it will cause the plugin to return the same inputs it was passed |
| criticalThreshold | optional | Determines where the CRIT interval starts; leave null to skip |
| warningThreshold | optional | Determines where the WARN interval starts; leave null to skip |
| infoThreshold | optional | Determines where the INFO interval starts; leave null to skip |
| okThreshold | optional | Determines what makes an OK status |
| applyToMetricName | optional | If specified, the current transform will only be applied to Transmutation s whose name matches this value |
ThresholdMetForDuration
{
"name" : "ThresholdMetForDuration",
"type" : "GREATER_THAN",
"threshold" : 80.0,
"criticalDurationMillis" : 120000,
"warnDurationMillis" : 60000,
"infoDurationMillis" : 30000
}
Similar to the Threshold plugin, with the difference that this plugin determines the status of a data point by the duration for which a certain value was exceeded (or fell behind).
Parameters
| Parameter | Default | Description |
|---|---|---|
| type | required | Can be set to GREATER_THAN, LESS_THAN to define how values are compared against the specified threshold parameter |
| threshold | required | The value which a value has to exceed or fall behind, depending on the chosen type |
| criticalDurationMillis | required | Determines how long the threshold has to be exceeded to return a status of CRIT |
| warnDurationMillis | required | Determines how long the threshold has to be exceeded to return a status of WARN |
| infoDurationMillis | required | Determines how long the threshold has to be exceeded to return a status of INFO |
Load destinations
At this time Pyplyn only supports publishing results to Refocus. More load destinations are planned in a future release.
You can, however, make use of Pyplyn's ability to post the same results to multiple destinations, and publish to more than one Refocus endpoint!
Refocus as a destination
For a subject of USA.California.San_Francisco and an aspect of Population, the Pyplyn load section of the configuration would look as follows:
"load": [
{
"format" : "Refocus",
"endpoint" : "refocus-default",
"subject" : "USA.California.San_Francisco",
"aspect" : "Population",
"relatedLinks" : []
}
]
Refocus is a platform for visualizing the health and status of systems and services. Data in Refocus is represented by Samples, which point to the value of an aspect for a specified subject at a given point in time (which is usually, very close to real-time).
Subjects introduce the concept of paths, where each can have multiple parents forming a 1:N relationship and allowing developers to structure the resulting dashboard in a way that makes sense for the observed system.
See the Refocus configuration section below, to read about how aspects can be defined in Refocus, to be used in combination with the Threshold plugin.
Parameters
| Parameter | Default | Description |
|---|---|---|
| format | required | Should be set to "Refocus" |
| endpoint | required | Identifies the connector to use when uploading this sample; read more in the (connectors)[#connectors] section |
| subject | required | The subject path that defines the sample |
| aspect | required | The aspect name that defines the sample |
| relatedLinks | optional | A list of links that will be associated with the published Sample; should specify two parameters, name and URL |
Quickstart
This section is meant to provide a step-by-step tutorial, showcasing how Pyplyn can be used to populate a Refocus dashboard using data originating from Argus. For this example we will utilize one of Argus' internal metrics, which specifies how much free memory is available in the system (argus.jvm:mem.physical.free
).
Prerequisites
Argus
Check out Argus' Getting started wiki page on how to install it.
Refocus
Define an aspect
{
"name": "FREE_MEM",
"criticalRange": [3, 3],
"warningRange": [2, 2],
"infoRange": [1, 1],
"okRange": [0, 0]
}
Root subject (Monitor)
{
"name": "Monitor",
"isPublished" : true
}
Child subjects (Monitor.System and Monitor.System.Argus); please refer to the Refocus manual on how to create a subject hierarchy
{
"name" :"System",
"isPublished": true
}
{
"name" :"Argus",
"isPublished": true
}
Create a perspective (after you've uploaded a lens and retrieved its id)
{
"name": "ArgusFreeMemory",
"lensId":"your-lensId-goes-here",
"rootSubject":"Monitor"
}
Refocus can be either installed to Heroku with a one-button deploy, installed locally, or in a Docker container.
Once Refocus was installed, define the PYPLYN_DEMO aspect, either with the Postman Chrome extension (collection here) or with curl
.
Similarly, define the Monitor, Monitor.System, and Monitor.System.Argus subjects.
You will also need to install a lens and create a perspective.
For this demo, you can skip enforcing security in Refocus. For production deployments, check out the Refocus' security options page for additional information!
Maven
Maven is a Java build tool (among other things), and it's used to compile and package the Pyplyn project as an executable.
If you're using a Mac, you can install it with Homebrew; if you're on Linux you can use package managers such as yum
or apt-get
, otherwise check-out the Maven on Windows page. We recommend Apache Maven 3.3.9
.
JAVA 8
This should be available on most modern systems. However, you can install it with the same tools listed above. We recommend at least version 1.8.0_131
.
Configure and run Pyplyn
# Clone the Pyplyn repository
git clone https://github.com/salesforce/pyplyn /tmp/pyplyn
# Build the project with Maven
cd /tmp/pyplyn
mvn clean package
After you finish building Pyplyn, you will have the following files in the target/
directory:
pyplyn-[VERSION].jar
: an executable (shaded) JAR file that contains all its required dependenciesbin/pyplyn.sh
: a script that simplifies starting and stopping Pyplyn as a serviceconfig/
: a directory containing example Pyplyn configurations, so you don't have to write these from scratch; the most important are:config/app-config.example.json
: this is the main config and determines where the connectors and configurations can be foundconfig/connectors.example.json
: all the remote services (and credentials) accessed/required by Pyplyn need to be listed in this file
# 1. Create a directory
mkdir ~/pyplyn
# 2. Copy Pyplyn's files (this assumes you've cloned Pyplyn in `/tmp/pyplyn`)
cp -R /tmp/pyplyn/target/pyplyn-*.jar /tmp/pyplyn/target/config /tmp/pyplyn/target/bin ~/pyplyn/
# 3. Create a new directory for your configurations
mkdir ~/pyplyn/configurations
# 4. Rename app-config.example.json and make the required changes
mv ~/pyplyn/config/app-config.example.json ~/pyplyn/config/pyplyn-config.json
# 5. Rename connectors.example.json and make the required changes (see below)
mv ~/pyplyn/config/connectors.example.json ~/pyplyn/config/connectors.json
5. Define connectors
json [ { "id": "refocus-token-connector", "endpoint": "https://url-to-your-refocus-instance/v1/", "username": null, "password": "BASE64-encoded-Refocus-token" }, { "id": "argus-token-connector", "endpoint": "https://url-to-your-argus-instance/argusws/", "username": null, "password": "BASE64-encoded-Argus-refresh-token" } ]
# 6\. Edit ~/pyplyn/bin/pyplyn.sh and set _LOCATION_ to the absolute path of the _pyplyn_ directory
LOCATION=/path/to/pyplyn
# 7\. Start pyplyn and check logs
bash ~/pyplyn/bin/pyplyn.sh start
# Should not display any exceptions
bash ~/pyplyn/bin/pyplyn.sh logs
1. First, it is important to create a dedicated directory to hold your files (since if left in the default location they would get erased every time you executed mvn clean
)
2. Create a directory (e.g., ~/pyplyn
) and copy the generated artifacts
3. Create an empty directory (e.g., ~/pyplyn/configurations
)
4. Rename app-config.example.json to pyplyn-config.json and set the following:
- configurationsPath
: "../configurations"
- connectorsPath
: "./connectors.json"
- updateConfigurationIntervalMillis
: 10000
5. Rename connectors.example.json to connectors.json, erase existing contents, and define Argus and Refocus connectors; don't forget to update the endpoints, BASE64 encode your access tokens, and update their values in this file!
6. Update the startup script and update LOCATION
7. Run Pyplyn, which should start and not throw any exceptions!
If you do see any Java exceptions, re-check your paths and all steps outlined above; in general, Pyplyn's errors have enough information to point you to the actual problem and give you an idea of what needs to be done to resolve any issues. If however, you are blocked, please open a new issue on GitHub, and we'll help you out as soon as we can!
BASE64-MIME encoding
# One simple way to achieve this is the following command
perl -e 'use MIME::Base64; print encode_base64("PASSWORD");'
Create a configuration for processing
Create a new file named
~/pyplyn/configurations/one.json
and paste the following:
[
{
"repeatIntervalMillis" : 60000,
"extract" : [
{
"format" : "Argus",
"endpoint" : "argus-token-connector",
"cacheMillis" : 0,
"defaultValue" : 0.0,
"expression" : "-60s:-0s:argus.jvm:mem.physical.free:avg",
"name" : "argus-avg-free-mem"
}
],
"transform" : [
{
"name" : "LastDatapoint"
},
{
"name" : "Threshold",
"criticalThreshold" :<PHONE_NUMBER>.0,
"warningThreshold" :<PHONE_NUMBER>.0,
"infoThreshold" :<PHONE_NUMBER>.0,
"type" : "LESS_THAN",
"applyToMetricName" : "argus-avg-free-mem"
}
],
"load" : [
{
"format" : "Refocus",
"endpoint" : "refocus-token-connector",
"subject" : "Monitor.System.Argus",
"aspect" : "FREE_MEM",
"relatedLinks" : []
}
]
}
]
Now that you have Pyplyn up and running, you are ready to write your first configuration.
In Pyplyn terminology, a configuration is a JSON object which specifies where the data is extracted from, how it should be processed (transformed), and where it should be published to.
Let's analyze each section:
- repeatIntervalMillis: is set to repeat every 60000 ms (60 seconds)
- extract: We will retrieve one of Argus' internal metrics that tracks free memory from argus-token-connector (specified in
connectors.json
); if no data is found, we will return adefaultValue
of 0.0 - transform:
- First, we filter all but the last datapoint;
- Then we apply the Threshold plugin
- And consider anything less than 2GB to be CRITICAL, 4GB to be a WARNING, and 8GB to be INFO
- load: Finally, after all the transformations have been applied, we load the resulting data in a Refocus sample named
Monitor.System.Argus|FREE_MEM
, on the endpoint defined by refocus-token-connector
Save this file in the ~pyplyn/configuration/
directory and issue a bin/pyplyn.sh logs
command; within the next 10 seconds, you should see a message showing that a new configuration was loaded and then executed. Open the Refocus perspective you created (ArgusFreeMemory), and you will see an entry for this sample!
Startup script (bin/pyplyn.sh)
Pyplyn includes a startup script which can be used to manage the service's lifecycle.
After building the project with mvn package
, you will find the script in the [PROJECT_HOME]/target/bin/pyplyn.sh
. Also, perform the following steps:
- Edit this file and set the LOCATION parameter to the location of the Pyplyn JAR.
- Make this file executable:
chmod u+rx pyplyn.sh
You can then issue the following commands:
pyplyn.sh start
: starts Pyplyn (configured from[LOCATION]/config/pyplyn-config.json
)pyplyn.sh nohup
: starts Pyplyn in nohup mode (the service keeps running after you log out)pyplyn.sh stop
: stops a running Pyplyn instance, if one existspyplyn.sh status
: checks if the service is running or notpyplyn.sh logs
: examines (tail -f
) current logs, if the service is startedpyplyn.sh restart
: re-starts the service
Extending Pyplyn
This section will give you a quick overview of developing new extract, transform, and load plugins for Pyplyn.
Adding more ways to extract and process data is probably the most common use case you will encounter for extending Pyplyn. However, most of Pyplyn's other functionality can be extended or replaced by changing the list of Guice injector modules defined in AppBootstrap#modules().
While you can easily fork the project and modify its source code, we do not recommend this approach.
Import the
duct
module, which encapsulates the processing and runtime logic
<!-- pom.xml -->
<dependencies>
<dependency>
<groupId>com.salesforce.pyplyn</groupId>
<artifactId>duct</artifactId>
<version>10.0.0</version>
</dependency>
</dependencies>
NOTE: you will need to first clone and install (
mvn clean install
) Pyplyn before being able to use the dependency above, since Pyplyn is not currently deployed in any global Maven repositories.
Instead, we propose you create a new Maven project that imports Pyplyn
as a dependency. This approach has the benefit of not having to maintain a forked project and integrate changes when upgrading to a newer version!
After you've developed and tested a new feature, you can still contribute your changes back into Pyplyn.
Common use-cases
- Add other Extract data sources
- Create new Transform plugins to apply custom logic to your data
- Define other Load destinations for sending data to
- Augment connectors to retrieve credentials from an external system (e.g., a database, remote endpoint, credentials store, etc.)
- Add other configuration objects that supplement the options defined in AppConfig
- Send monitoring data to other systems and APIs
How do I replace/augment Pyplyn's bootstrap?
1. Create a new bootstrap class
import com.google.inject.Module;
import com.salesforce.pyplyn.duct.app.AppBootstrap;
import java.util.ArrayList;
import java.util.List;
public class CustomPyplynBootstrap extends AppBootstrap {
@Override
public List<Module> modules() {
// load default Pyplyn modules
final List<Module> modules = new ArrayList<>(super.modules());
// add new modules
// replace or remove any of the original modules
// return the final list, which will be used during bootstrap
return modules;
}
}
2. Create a new class and reference the newly created bootstrap class
import com.salesforce.pyplyn.duct.app.DuctMain;
public class PyplynExtensionApp {
public static void main(String[] args) {
// this changes the program's name, when _usage_ information is displayed
DuctMain.setProgramName("pyplyn-extension");
// start Pyplyn with a custom bootstrap
DuctMain.execute(CustomPyplynBootstrap.class, args);
}
}
To have Pyplyn start with custom functionality, you will have to (at least) perform the following actions:
1. Create a new bootstrap class that extends AppBootstrap
- override the AppBootstrap#modules()
method and update the list of modules used during bootstrap
- this list will be passed to the Guice injector and control the final behavior of the program
2. Create a new main(String[])
entry-point
- set the program's name (only used in usage help), or skip to use the default value of pyplyn
- call the DuctMain#execute(Class, String[])
method, pointing it do your newly created bootstrap class (step 1 above)
How do I create a new Transform?
3. Create a new class (using Immutables)
import org.immutables.value.Value;
import com.fasterxml.jackson.annotation.JsonTypeName;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
import com.salesforce.pyplyn.annotations.PyplynImmutableStyle;
import com.salesforce.pyplyn.model.Transform;
import com.salesforce.pyplyn.model.Transmutation;
@Value.Immutable
@PyplynImmutableStyle
@JsonDeserialize(as = ImmutableNewTransformPlugin.class)
@JsonSerialize(as = ImmutableNewTransformPlugin.class)
@JsonTypeName("NewTransformPlugin")
public abstract class NewTransformPlugin implements Transform {
private static final long serialVersionUID = -1L; // generate a random value
public abstract String aProperty(); // define properties if required
@Override
public List<List<Transmutation>> apply(List<List<Transmutation>> input) {
// logic goes here
}
}
4. Create a Guice module
import com.google.inject.AbstractModule;
import com.salesforce.pyplyn.util.MultibinderFactory;
public class NewTransformPluginModule extends AbstractModule {
@Override
protected void configure() {
MultibinderFactory.transformFunctions(binder()).addBinding().toInstance(NewTransformPlugin.class);
}
}
5. Add the newly created Guice module in your custom bootstrap
import com.google.inject.Module;
import com.salesforce.pyplyn.duct.app.AppBootstrap;
import java.util.ArrayList;
import java.util.List;
public class CustomPyplynBootstrap extends AppBootstrap {
@Override
public List<Module> modules() {
// load default Pyplyn modules
final List<Module> modules = new ArrayList<>(super.modules());
// add the new module
modules.add(new NewTransformPluginModule());
// return the final list, which will be used during bootstrap
return modules;
}
}
3. Create a new class for the transform plugin
- while not required, using the Immutables library, gives you benefits such as guarded values and auto-generated builders, which is extremely useful if you will need to automate the generation of your configurations with other (JAVA-based) tooling
- define any properties that might be required by your plugin
- implement the Transform#apply(List)
method and write your plugin's logic
- the value specified in @JsonTypeName(String)
should be unique across all registered Pyplyn modules for the current type (Transform); it will be used to correctly serialize and deserialize JSON strings to their corresponding types
4. Create a Guice module and register your plugin with MultibinderFactory#transformFunctions(Binder)
- See the MultibinderFactory.java class for other helper methods you can use to register the various types Pyplyn supports
- You can also consult Pyplyn's extract, transform, load, etc. implementations in the source code, to understand how these features are implemented; a few links are presented in the common use-cases section above
5. Update the custom bootstrap to include the newly created module
Starting/stopping a Pyplyn-extension project
You can manage an extension project's lifecycle similarly as described in the Configure pyplyn and Startup script sections.
Follow these steps:
- copy the script to a new location and rename it to match your project's name
- build the extension project (mvn clean package
)
- copy the resulting JAR file into this new location
- edit the startup script and update the LOCATION
and NAME
variables
Tech stack
Below you can see the amazing open-source technologies that make Pyplyn possible!
Contributing to Pyplyn
Any contributions (code, artwork, documentation, etc.) will be considered as long as it respects our licensing requirements, listed below.
All contributors are required to sign the Salesforce Contributor's License Agreement. Please print this file, sign it, scan it, and send it to<EMAIL_ADDRESS>For more details on the process see CONTRIBUTE.MD in our repository.
We look forward to collaborating with you.
Thank you!
Licensing requirements
For all contributions, we can only accept materials published under the following license types. Please ensure that your submission respects these guidelines, otherwise, we will not be able to accept them.
- Apache 2.0
- BSD
- MIT
- ISC
- MPL


Game Of Thrones
9.2
Nine noble families fight for control over the lands of Westeros, while an ancient enemy returns after
being dormant for millennia.
Friends
8.9Follows the personal and professional lives of six twenty to
thirty-something-year-old friends living in Manhattan.

Breaking Bad
9.4
A high school chemistry teacher diagnosed with inoperable lung cancer turns to manufacturing and selling
methamphetamine in order to secure his family's future.
The Walking Dead
8.2
Sheriff Deputy Rick Grimes wakes up from a coma to learn the world is in ruins and must lead a group of
survivors to stay alive.
The Big Bang Theory
8.1
A woman who moves into an apartment across the hall from two brilliant but socially awkward physicists
shows them how little they know about life outside of the laboratory.
Ozark
8.4
A financial advisor drags his family from Chicago to the Missouri Ozarks, where he must launder money to
appease a drug boss.
Mirzapur
8.4
A shocking incident at a wedding procession ignites a series of events entangling the lives of two
families in the lawless city of Mirzapur.
How I Met Your Mother
8.3
A father recounts to his children - through a series of flashbacks - the journey he and his four best
friends took leading up to him meeting their mother.
New Girl
7.7
After a bad break-up, Jess, an offbeat young woman, moves into an apartment loft with three single men.
Although they find her behavior very unusual, the men support her - most of the time.
Sacred Games
8.6
A link in their pasts leads an honest cop to a fugitive gang boss, whose cryptic warning spurs the
officer on a quest to save Mumbai from cataclysm.
Vikings
8.5
Vikings transports us to the brutal and mysterious world of Ragnar Lothbrok, a Viking warrior and farmer
who yearns to explore - and raid - the distant shores across the ocean.
Kota Factory
9.2
Dedicated to Shrimati SL Loney ji, Shri Irodov ji and Maanniya HC Verma ji, 'Kota Factory' is TVF's
latest original. India's first 'Black and White' show highlights the problems present day IIT-JEE
aspirants face in their day-to-day
The Last Dance
9.1
Charting the rise of the 1990's Chicago Bulls, led by Michael Jordan, one of the most notable dynasties
in sports history.
Suits
8.4
On the run from a drug deal gone bad, brilliant college dropout Mike Ross finds himself working with
Harvey Specter, one of New York City's best lawyers.
Loki
8.4
The mercurial villain Loki resumes his role as the God of Mischief in a new series that takes place
after the events of “Avengers: Endgame.”
Peaky Blinders
8.8
A gangster family epic set in 1900s England, centering on a gang who sew razor blades in the peaks of
their caps, and their fierce boss Tommy Shelby.
Black Mirror
8.8
An anthology series exploring a twisted, high-tech multiverse where humanity's greatest innovations and
darkest instincts collide.


App Store Download Buttons Change From Free To Get As In App Purchases Dominate Revenue Appleinsider
Ios 11 App Store Sketch Freebie Download Free Resource For Sketch Sketch App Sources
Get Line Microsoft Store
Google Play App Store Outline Icons Download From Google Pay Editorial Photography Illustration Of Button Finance 177332032
Line Payアプリをダウンロード Line Pay
Lineアプリをインストールする Lineみんなの使い方ガイド
App Store Icon White Stock Illustrations 15 667 App Store Icon White Stock Illustrations Vectors Clipart Dreamstime
Download App Store Stock Illustrations 2 503 Download App Store Stock Illustrations Vectors Clipart Dreamstime
How To Add App Store Google App Download Links To Your Email Signature Snapp Apps
Apple Relabels Free Download Button As Get On App Stores Technology News
Google Play And Apple App Store Download Buttons Vector Image
How To Download Free Apps On App Store 10 Steps With Pictures
Apple App Store And Google Play Store Icons Sketch Freebie Download Free Resource For Sketch 3 Google Play Store Store Icon App Store
2 Methods How To Download Apps Without App Store No Jailbreak
App Store Icon Free Download On Iconfinder
Share App Store Itunes Store And Apple Books Purchases With Family Sharing Apple Support
How To Download Apps And Games From The App Store Imore
Ios App Store Unable To Download And Update Apps Due To Terms And Conditions Loop Even After Agreeing Ask Different
Google Play Store V8 Brings In Line Changelogs To The Update Screen Apk Download
App Store Preview Templates Sketch Freebie Download Free Resource For Sketch Sketch App Sources
App Store Preview Template Sketch Freebie Download Free Resource For Sketch Sketch App Sources Sketch App App App Store
How To Create An Apple Id Account And Download Apps From Apple App Store
App Store Connect Help
Unable To Download An App From Google Play Store Here S How To Fix It Gadgets Now
Https Encrypted Tbn0 Gstatic Com Images Q Tbn And9gctuxnsknsnidrnxdqxvm7okhawpdpsshcpvx Wtb9hde8 Xdvt1 Usqp Cau
Download Icons App Store Google Play Svg Eps Png Psd Ai App Store Icon App Store Google Play Google Play Apps
App Store Download Page 1 Line 17qq Com
Apple Bumps Up 4g App Store Download Limit For Iphones Ipads To 200mb Appleinsider
How To Download Apps Without App Store On Iphone Ipad
How To Use The Apple App Store On Iphone
App Store Download Free Page 1 Line 17qq Com
How To Download Snapchat Without The App Store Amazeinvent


list_to_bit_array(xs)
Return a bit array initialized from the list of integers xs.
xs
list_to_bit_array([1, 2, 3]) // #b[1, 1, 1] list_to_bit_array([1, 0, 3]) // #b[1, 0, 1]
Core Module Index | Contents


Introduction:
This is a paragraph.
Objective:
This is a paragraph.
Introduction:
This is a paragraph.
Contect: :
This is a paragraph.
Resource:
This is a paragraph.
Contact:


Styles P Announces New Project "Presence" With "Brand New"
Ghost is back.
by Aron A.Brand New
Styles P
Editor Rating: VERY HOTTTTT
User Rating: hottttt - 75% (1)
Styles P went from slanging bricks to juices but the essence of the streets hasn't left his music at all. The rapper is getting ready to release his forthcoming project, Presence which is set to arrive this Friday. Even though there are only a few days left until its release, he gave fans of what to expect with his latest track, "Brand New." Over breezy, soulful production, Styles does what he does best while dishing bars about everything from paranoia in the hood to having an Ital diet and keeping positive energy.
Presence will mark Styles's third project of the year. In May, he dropped off S.P. The GOAT: Ghost Of All Time and shortly after, he unleashed a joint project with his son ahead of Father's Day titled, Generational Excellence.


Policies and Rules -- customize the underlying behavior at compile time
Policy and rule
Policy and rule
Get startback top
Policy is a mechanism to change the default behavior of the library.
Policy is composed by rules.
One rule is just a data type (a struct).
Rules are pre-defined and recognized by the library.
One rule will change one aspect of the default behavior.
For example, by default a meta class requires the class has a default constructor.
If the class doesn't have it, compiling will fail if meta class is being built for it.
However, if feeding the meta class with a policy with the rule of GMetaRuleDefaultConstructorAbsent, the library will know there is no default constructor and will succeed the compiling.
Rules are pre-defined, you can't add any new rules or change any existing rules behavior.
However, with the combination of rules, a new policy can be created.
To create a policy, include the header gpolicy.h, use MakePolicy to make a policy.
typedef MakePolicy<Rule1, Rule2, ..., RuleN> NewPolicy;
For the reflection library, there are several predefined policies in file gmetapolicy.h.
Meta rulesback top
GMetaRuleForbidRead
Apply to: field and property.
Effect: the value of field and property can't be read directly.
Invoking "Get" will cause exception.
GMetaRuleForbidWrite
Apply to: field and property.
Effect: the value of field and property can't be written directly.
Invoking "set" will cause exception.
GMetaRuleDefaultConstructorAbsent
Apply to: class
Effect: the class doesn't have default constructor.
GMetaRuleCopyConstructorAbsent
Apply to: class
Effect: the class doesn't have copy constructor.
GMetaRuleTransferOwnership <N>
Apply to: invokable -- method, constructor, and operator
Effect: the parameter or result will discard the object ownership.
This rule is a template with an integer parameter N.
If N is metaPolicyResultIndex (which value is -1), the rule is for result. The object returned by the invokable should be freed by the caller.
If N is 0 or greater integer, the rule is for parameter N. The object passed to the invokable at parameter N will be freed by the callee (the invokable).
Note: parameter 0 is the first explicit parameter, not "this" pointer.
GMetaRuleCopyConstReference <N>
Apply to: invokable -- method, constructor, and operator
Effect: the parameter or result will copy the object to reference-to-const instead of reference.
If implicit type conversion is required, such like C string to std::string, use this rule.
See Rule 5 in GVariant document for details.
GMetaRuleParamNoncopyable <N>
Apply to: invokable -- method, constructor, and operator
Effect: the parameter or resjiult is not copyable.
GMetaRuleExplicitThis
Apply to: method, operator and property which getter or setter is method
Effect: the first parameter of the method is treated as "this" pointer and it's invisible from the outside.
This policy is used to use a global method to pretend being a member function.
For example, if we have global function,
void myMethod(MyClass * instance, int param);
Applying GMetaRuleExplicitThis will have the same effect as if myMethod is a member function of MyClass and the prototype is void myMethod(int param).
With this policy, global function can be used as property getter or setter.
GMetaRuleGetterExplicitThis and GMetaRuleSetterExplicitThis
Apply to: property getter and setter.
Effect: same as GMetaRuleExplicitThis, but GMetaRuleGetterExplicitThis only has effect on property getter, and GMetaRuleSetterExplicitThis only has effect on property setter.
With these two rules, we can control the explict this on getter and setter separately.
Meta policiesback top
Though you can combine the rules to create the policy you need, there are some pre-defined meta policies in gmetapolicy.h
Below only lists the source code of all of the pre-defined policies.
The meaning can be found by the policy name and the rules.
typedef MakePolicy<
GMetaRuleForbidRead,
GMetaRuleForbidWrite
>
GMetaPolicyNoncopyable;
typedef MakePolicy<
GMetaRuleForbidWrite
>
GMetaPolicyReadOnly;
typedef MakePolicy<
GMetaRuleForbidRead
>
GMetaPolicyWriteOnly;
typedef MakePolicy<
GMetaRuleDefaultConstructorAbsent
>
GMetaPolicyNoDefaultConstructor;
typedef MakePolicy<
GMetaRuleCopyConstructorAbsent
>
GMetaPolicyNoCopyConstructor;
typedef MakePolicy<
GMetaRuleDefaultConstructorAbsent,
GMetaRuleCopyConstructorAbsent
>
GMetaPolicyNoDefaultAndCopyConstructor;
typedef MakePolicy<
GMetaRuleTransferOwnership <metaPolicyResultIndex>
>
GMetaPolicyTransferResultOwnership;
typedef MakePolicy<
GMetaRuleCopyConstReference <metaPolicyResultIndex>,
GPP_REPEAT(REF_MAX_ARITY, POLICY_ALL_INDEX_HELPER, GMetaRuleCopyConstReference)
>
GMetaPolicyCopyAllConstReference;
typedef MakePolicy<
GMetaRuleParamNoncopyable <metaPolicyResultIndex>,
GPP_REPEAT(REF_MAX_ARITY, POLICY_ALL_INDEX_HELPER, GMetaRuleParamNoncopyable)
>
GMetaPolicyAllParamNoncopyable;
typedef MakePolicy<
GMetaRuleExplicitThis
>
GMetaPolicyExplicitThis;


The Tale of Brownie Beaver
by Arthur Scott Bailey
Chapter 8: “Mr. Crow Is Upset”
Additional Information
- Year Published: 1916
- Language: English
- Country of Origin: United States of America
- Source: Bailey, A. S. (1916). The Tale of Brownie Beaver. New York: Grosset and Dunlap.
-
Readability:
- Flesch–Kincaid Level: 5.0
- Word Count: 804
- Genre: Fantasy
- Keywords: 20th century literature, american literature, children's stories, sleepy-time tales
- ✎ Cite This
Downloads
Brownie Beaver couldn’t help feeling that Mr. Crow had not treated him very well, because Mr. Crow hadn’t told him all the news about Frisky Squirrel’s tail. He thought that maybe there were things about a newspaper that even Mr. Crow didn’t know.
Another week had passed.  And there were now seven of them.
Late Saturday afternoon Mr. Crow came back again. He lighted on top of Brownie’s house and called “Paper!” down the chimney, just as he had a week before.
Brownie Beaver came swimming up once more.
“Look here!” he said to Mr. Crow. “I don’t believe yon know much about being a newspaper, do you?”
That surprised Mr. Crow.
“What do you mean?” he asked.
“A newspaper—” said Brownie Beaver—“a newspaper is always left on a person’s doorstep. I’ve talked with a good many people and not one of them ever heard of a paper being left on the roof.”
Mr. Crow’s face seemed to grow blacker than ever, he was so angry.
“How can anybody leave a newspaper on your doorstep, when the step’s under water?” he growled.
Brownie Beaver did not answer that question, for he had something else to say to Mr. Crow.
“I’ve talked with a good many people,” he said once more, “and not one of them ever heard of such rudeness as shouting down a person’s chimney. If there was anybody asleep in the house, it would certainly wake him; and if a person had a fire in his house, shouting down the chimney might put it out.”
Mr. Crow looked rather foolish.
“I’ll try to think of some way of leaving your newspaper that will suit us both,” he said. Then he hemmed and began to tell Brownie the week’s news.
“On Sunday,” said Mr. Crow, “there was a freshet.”
“I knew that before you did,” said Brownie Beaver.
Mr. Crow looked disappointed.
“How?” he asked.
“Why, I live further up the river than you,” said Brownie Beaver. “And since freshets always come down a river, this one didn’t reach you till after it had passed me.”
Something made Mr. Crow peevish.
“I don’t believe you’d care to hear any more of my news,” he said. “You appear to know it already. Perhaps you’ll be kind enough to tell me the sort of news you prefer to hear.”
“Certainly!” Brownie Beaver replied. “Now, there’s the weather! I’ve talked with a good many people and they all say that a good newspaper ought to tell the weather for the next day.”
Mr. Crow cocked an eye up at the sky.
“Tomorrow will be fair,” he said.
“I’m told that a good newspaper ought to tell a few jokes,” Brownie Beaver continued.
But Mr. Crow sneered openly at that. “I’m a newspaper—not a jest-book,” he announced.
“Then you refuse to tell any jokes, do you?” Brownie Beaver asked him.
“I certainly do!” Mr. Crow cried indignantly.
“Very well!” Brownie said. “I see I’ll have to take some other newspaper, though I must say I hate to change—after taking this one so long.”
“I hope you’ll find one to suit you,” Mr. Crow said in a cross voice. And he flew away without another word. He was terribly upset.  He had intended all that week to ask Aunt Polly Woodchuck whether she wore a wig. But he hadn’t been able to find her at home. And now it was too late—for Mr. Crow was a newspaper no longer.
As for Brownie Beaver, he succeeded in getting Jasper Jay to be his newspaper. Though Jasper told him many jokes, Brownie found that he could not depend upon Jasper’s news. And as a matter of fact, Jasper made up most of it himself. He claimed that the newest news was the best.
“That’s why I invent it myself, right on the spot,” he explained.


{% block head %}{% endblock %}
Kilda Openflow Controller
Home
Topology status
Topology Switches
Topology Links
Logout
{% block body %}{% endblock %}


Auth Components¶
Agave is a multi-tenant platform, and each tenant deploys a dedicated auth component while the other components (core and persistence) are shared across tenants. The Ansible installer can support the installation and management of multiple tenants. Each tenant is identified by the value of tenant_id
provided in the inventory file. The default values for the tenant_id is “sandbox”.
The following tables describe variables for use with the Ansible installer that can be used to customize the Auth components:
| Variable | Type | Description |
|---|---|---|
| tenant_id | string | A unique id for this tenant.
Note: This will be replaced by the ``agave_tenant_id``
*field in the near future. Both are currently *
required.
|
| agave_tenant_id | string | A unique id for this tenant.
Note: This will be replace the ``tenant_id``
*field in the near future. Both are currently *
required.
|
| tenant_public_domain_or_ip | string | The public domain name or ip address of the
auth server. This is the address you will use
to connect to your installation.
required.
|
| agave_env | string | (staging/prod). The Agave environment used for
this tenant. Determines the core services that
will be invoked. Note that the IP address for the
core services can be resolved by docker compose
using the extra_hosts stanza for platform
deployments that are not in DNS.
|
| host | string | The public domain that the platform should listen
to API requests on. For example, if this value is
api.example.org , then the core apps servicewill be available at
https://api.example.org/apps/v2 .*Note: This will be replace by the *
``tenant_public_domain_or_ip`` field in the near future.
required.
|
| tenant_admin_role | string | The name of the tenant admine role.
|
| agave_profiles_url | string | URL for the profiles service. When the tenant uses
the Agave hosted identity, this URL should be of
the form
profiles.<tenant_id>.agave.tacc.utexas.edu/profiles and will be resolved automatically; however, for
other tenants such as Cyverse, this URL will be
different.
|
| haproxy_auth_version | string | Version of the HA Proxy auth service to use
(Defaults to latest).
|
| apim_httpd_version | string | Version of the auth httpd service to use
(Ddefaults to latest).
|
| apim19_base_version | string | Version of APIM 1.9 auth service to use
(Defaults to latest).
|
| agave_id_dedicated_version | string | Version of the Agave ID auth service to use
(Defaults to latest).
|
| agave_clients_dedicated_version | string | Version of the Agave clients auth service to use
(Defaults to latest).
|
| admin_services_nginx_version | string | Version of the admin nginx auth service to use
(Defaults to latest).
|
| flask_admin_services_version | string | Version of the admin auth service to use
(Defaults to latest).
|
| update_auth_dns | bool | Whether or not to update the
/etc/hosts filewith the location of the auth mysql server for the
auth containers. Use this when the host name/domain
for the mysql server is not in DNS.
|
| core_api_protocol | string | (http or https). Protocol the auth services will
use when communicating with the core services. Use
http unless the auth services must communicatewith the core services across a firewall.
|
| update_apim_core_dns | bool | Whether or not to update the /etc/hosts file with
the location of the core services within the APIM
container. Use this when the host name/domain for
the core services are not in DNS. NOTE: the APIM
now routes all requests to
<tenant_id>.api.<agave_env>.agaveapi.co so this is the value that must be in DNS.
|
| core_host | string | IP of the core host. Used only when
update_apim_core_dns is true.
|
| deploy_admin_password_grant | bool | Provides a grant type for delegating credentials on
behalf of other users.
|
| access_token_validity_time | string | Time, in seconds, that OAuth access_tokens are valid.
|
| apim_increase_global_timeout | bool | Whether or not to increase the APIM’s global
timeout config for APIs to respond. For very
large file uploads, it is possible that the
APIs will time out before responding to APIM,
resulting in a 502. However, setting this to
True will increase the timeout for all APIs.
It will be possible to just increase the files
API timeout in a future release.
|
| deploy_custom_oauth_app | bool | Whether to deploy a custom-branded version of the
OAuth login application. If true, install the
authenticationendpoint application (in adirectory with the same name) within the tenant
directory.
|
| boutique_apis | string | (Optional). A YAML list of API names from the
catalog of extended APIs available in the platform
(e.g., admin_services). Each API must be defined
as a template in the boutique_apis role, and the
name listed must match the name of the template
file minus the extension (e.g.
admin_services for
admin_services.json.j2 ). |
| ha_deployment | bool | For use in h/a deployments of the auth component
requiring multiple hosts. Value True will deploy
an HA Proxy instance on each auth host to
facilitate an active-active architecture. Requires
additional configuration, see below.
*Note: ha_deployment=True is required, as the
deployment playbooks currently depend on it.
|
| hap_servers | string | YAML mapping of servers to use for HAProxy. At
least two entries are required, though it is
possible to run both servers on the same host
using IP addresses and ports on the Docker0
interface
Each entry must have the following properties:
* name: Name for the server; must be unique.
* ip: IP for the server, addressable by the
* port: Http port for the httpd instance for
this server. Must be reachable by
the HAProxy container(s).
* ssl_port: Https port for the httpd instance
for this server. Must be reachable by the
HAProxy container(s).
|
| mysql_host | string | The host of the Auth MySQL server
|
| mysql_port | string | The port of the Auth MySQL server
|
| use_hosted_id | bool | Whether or not to use Agave hosted identity
services. Cannont be used when the
use_hosted_id setting is True. Use this config when Agave will
be administering its own LDAP.
|
| use_remote_userstore | bool | Whether to configure a remote userstore. Use this
config when the LDAP will be administered by a
third party.
* Cannot be used when the* use_hosted_id
setting is True. Currently, this configuration
supports read only LDAP databases.
|
| agave_id_read_only | bool | Whether the hosted id service should run in read
only mode.
|
| hosted_id_domain_name | string | A unique id for the agaveldap hosted userstore.
Use this config when Agave will be administering
its own LDAP.
|
| remote_id_domain_name | string | A unique id for the remote userstore.
Use this config when the LDAP will be administered
by a third party.
|
| ldap_name | string | URL or service discovery token for the hosted LDAP
instance (including port).
|
| auth_ldap_bind_dn | string | Account to bind to the LDAP db.
|
| ldap_base_search_dn | string | Base search directory for user accounts.
|
| agave_id_check_jwt | string | Whether or not to check the JWT; When this is False,
certain features will not be available such as the
“me” lookup feature since these features rely on
profile information in the JWT.
|
| jwt_header | string | Actual header name that will show up in
request.META; value depends on APIM
configuration, in particular the tenant id
specified in api-manager.xml.
|
| agave_id_apim_pub_key | string | Absolute path to the public key of the APIM
instance; used for verifying the signature of the
JWT.
|
| agave_id_user_admin_role | string | Role required to make updates to the LDAP database.
|
| agave_id_check_user_admin_role | bool | Whether or not the
agave_id_user_admin_role before allowing updates to the LDAP db.
|
| agave_id_app_base | string | Sets the base URL for the hypermedia responses;
Typically this should be the same as host but
should include the protocol.
|
| cert_file | string | Should be a path relative to the httpd directory
contained within the tenant directory for this
tenant: e.g.
deploy/tenants/dev_staging/httpd |
| cert_key_file | string | Should be a path relative to the httpd directory
contained within the tenant directory for this
tenant.
|
| ssl_ca_cert_file | string | Add when mounting in a CA cert (not used for
self-signed certs). Should be a path relative to
the httpd directory contained within the tenant
directory for this tenant.
|
| use_custom_ldap | bool | Use this setting when using a remote userstore
and the ldap has a different schema then the hosted
Agave LDAP (for example, the iPlant and TACC LDAPs).
Note: Only certain schemas are supported. See the
agaveplatform/pyprofiles-api Github repo for more
details.
|
| agave_id_create_notifications | bool | Whether or not the auth services should send
notifications to beanstalk (see settings below).
The auth services must have access to the
beanstalk IP and port.
|
| beanstalk_server | string | IP address of beanstalk instance.
|
| beanstalk_port | string | Port of beanstalk instance.
|
| beanstalk_tube | string | Beanstalk tube name that the auth services will send
messages to.
|
| beanstalk_srv_code | string | Code for the service to use when generating messages.
There is one code per core service.
|
| tenant_uuid | string | The UUID of the tenant; this is only used by the
components when sending messages to beanstalk.
|
| virtualhosts | string | A yaml list of collections for configuring the
virtualhosts that the auth services will listen
on. Defining multiple configuration collections
allows the tenant auth server to respond to multiple
domains (e.g.
agave.example.com andsandbox.tenants.dev.agaveplatform.org ). Eachconfiguration collection should define the
following fields:
* server_name: The domain to respond to.
* base_cert_path: The directory inside the
httpd container that will hold the certs. This
can be anything, and the agave_id container
will create the directory if it does not exist,
but it must be defined.
* cert_file: The cert file used by httpd;
this file must be installed within the
httpd folder within the tenant folder.
* cert_key_file: The cert key file used by
httpd; this file must be installed within the
httpd folder within the tenant folder.* ssl_ca_cert_file: The CA bundle file used
by httpd; this file must be installed within the
httpd folder within the tenant folder. |
| deploy_admin_services | bool | Whether to deploy the tenant admin services
(Defaults to True).
|
| boutique_apis | string | (Optional) A YAML list of strings representing
additional APIs to deploy. Each API in the list
should be an API recognized by the
boutique_apis role. These are the officially supported boutique
APIs. Additional APIs can be added to a the set
of officially supported APIs by simply adding an
<api>.json.j2 template to the
roles/boutique_apis/templates directory. The JSON should be formatted as
required by the Admin services
/apis service.Note: this role requires the admin services to
be deployed on the auth host.
|


Horror
| Title | Author | Type | Date Posted | User Views |
|---|---|---|---|---|
| HOW TO GET BACK TO THE FOREST | by SOFIA SAMATAR | Prose | 10/31/2019 | 109 |
| PATIENT ZERO | by TANANARIVE DUE | Prose | 11/08/2019 | 246 |
| A COLLAPSE OF HORSES | by BRIAN EVENSON | Prose | 09/30/2019 | 674 |
| The Raven | by Edgar Allen Poe | Poetry | 01/29/1845 | 2,109 |
| Omens | by Cecelia Llompart | Poetry | 07/24/2019 | 223 |
Fantasy
| Title | Author | Type | Date Posted | User Views |
|---|---|---|---|---|
| St. Lucy's Home for Girls Raised by Wolves | by Karen Russell | Prose | 05/11/2019 | 55 |
| The City Born Great | by N.K. Jemisin | Prose | 06/16/2019 | 709 |
| The Truth Is a Cave in the Black Mountains | by Neil Gaiman | Prose | 09/23/2019 | 1049 |
| The Ballad Of The Harp Weaver | by Edna St. Vincent Millay | Poetry | 03/29/2019 | 298 |
| Paul Bunyan | by Shel Silverstein | Poetry | 09/26/2019 | 432 |
| Dreams | by Jane A Beresford | Poetry | 10/16/2019 | 889 |
Historical Fiction
| Title | Author | Type | Date Posted | User Views |
|---|---|---|---|---|
| Greenhorns | by Richard Slotkin | Prose | 09/17/2019 | 649 |
| Hard Ride | by Elmer Kelton | Prose | 10/07/2019 | 543 |
| Scenes from the Heartland: Stories Based on Lithographs by Thomas Hart Benton | by Donna Baier Stein | Prose | 11/04/2019 | 42 |
| Pneumonic Poem to remember Henry the Eighth’s wives | by Sathyam-Drew | Poetry | 08/13/2019 | 675 |
| Holocaust | by angelo19 | Poetry | 11/10/2019 | 965 |
| MOTHER AFRICA | by TAMBE MANSFIELD AGBOR | Poetry | 09/18/2019 | 25 |


Fecha_Creacion: 2017-11-10T13:08:46.060Z
Dimension: Dimension C-137
Id_Locacion: 5
Nombre: Anatomy Park
Residentes: ['https://rickandmortyapi.com/api/character/12', 'https://rickandmortyapi.com/api/character/17', 'https://rickandmortyapi.com/api/character/96', 'https://rickandmortyapi.com/api/character/97', 'https://rickandmortyapi.com/api/character/98', 'https://rickandmortyapi.com/api/character/99', 'https://rickandmortyapi.com/api/character/100', 'https://rickandmortyapi.com/api/character/101', 'https://rickandmortyapi.com/api/character/108', 'https://rickandmortyapi.com/api/character/268', 'https://rickandmortyapi.com/api/character/300']
Tipo: Microverse
URL: https://rickandmortyapi.com/api/location/5


# File lib/basset/feature_selector.rb, line 49 def select_features(chi_value = 1.0, classification = nil) classification ||= @docs_in_class.keys.first selected_features = features_with_chi(classification).select do |feature| (docs_with_feature(feature.name) > 1) && (feature.value >= chi_value) end selected_features.sort_by(&:value).reverse.collect(&:name) end


HTTP client
| url | (character) A url. One of |
|---|---|
| opts | (list) curl options, a named list. See
|
| proxies | an object of class |
| headers | (list) a named list of headers |
| handle | A handle, see |
Methods
get(path, query, disk, stream, ...)
Make a GET request
post(path, query, body, disk, stream, ...)
Make a POST request
put(path, query, body, disk, stream, ...)
Make a PUT request
patch(path, query, body, disk, stream, ...)
Make a PATCH request
delete(path, query, body, disk, stream, ...)
Make a DELETE request
head(path, disk, stream, ...)
Make a HEAD request
Possible parameters (not all are allowed in each HTTP verb):
path - URL path, appended to the base URL
query - query terms, as a list
body - body as an R list
encode - one of form, multipart, json, or raw
disk - a path to write to. if NULL (default), memory used.
See curl::curl_fetch_disk()
for help.
stream - an R function to determine how to stream data. if
NULL (default), memory used. See curl::curl_fetch_stream()
for help
... curl options, only those in the acceptable set from
curl::curl_options()
except the following: httpget, httppost,
post, postfields, postfieldsize, and customrequest
post-requests, delete-requests, http-headers, writing-options
(x <- HttpClient$new(url = "https://httpbin.org"))#> <crul connection> #> url: https://httpbin.org #> curl options: #> proxies: #> auth: #> headers:x$url#> [1] "https://httpbin.org"(res_get1 <- x$get('get'))#> <crul response> #> url: https://httpbin.org/get #> request_headers: #> User-Agent: libcurl/7.51.0 r-curl/2.6 crul/<IP_ADDRESS>10 #> Accept-Encoding: gzip, deflate #> response_headers: #> status: HTTP/1.1 200 OK #> connection: keep-alive #> server: meinheld/0.6.1 #> date: Mon, 15 May 2017 18:11:57 GMT #> content-type: application/json #> access-control-allow-origin: * #> access-control-allow-credentials: true #> x-powered-by: Flask #> x-processed-time: 0.00109100341797 #> content-length: 287 #> via: 1.1 vegur #> status: 200res_get1$content#> [1] 7b 0a 20 20 22 61 72 67 73 22 3a 20 7b 7d 2c 20 0a 20 20 22 68 65 61 64 65 #> [26] 72 73 22 3a 20 7b 0a 20 20 20 20 22 41 63 63 65 70 74 22 3a 20 22 2a 2f 2a #> [51] 22 2c 20 0a 20 20 20 20 22 41 63 63 65 70 74 2d 45 6e 63 6f 64 69 6e 67 22 #> [76] 3a 20 22 67 7a 69 70 2c 20 64 65 66 6c 61 74 65 22 2c 20 0a 20 20 20 20 22 #> [101] 43 6f 6e 6e 65 63 74 69 6f 6e 22 3a 20 22 63 6c 6f 73 65 22 2c 20 0a 20 20 #> [126] 20 20 22 48 6f 73 74 22 3a 20 22 68 74 74 70 62 69 6e 2e 6f 72 67 22 2c 20 #> [151] 0a 20 20 20 20 22 55 73 65 72 2d 41 67 65 6e 74 22 3a 20 22 6c 69 62 63 75 #> [176] 72 6c 2f 37 2e 35 31 2e 30 20 72 2d 63 75 72 6c 2f 32 2e 36 20 63 72 75 6c #> [201] 2f 30 2e 33 2e 35 2e 39 31 31 30 22 0a 20 20 7d 2c 20 0a 20 20 22 6f 72 69 #> [226] 67 69 6e 22 3a 20 22 31 35 37 2e 31 33 30 2e 31 37 39 2e 38 36 22 2c 20 0a #> [251] 20 20 22 75 72 6c 22 3a 20 22 68 74 74 70 73 3a 2f 2f 68 74 74 70 62 69 6e #> [276] 2e 6f 72 67 2f 67 65 74 22 0a 7d 0ares_get1$response_headers#> $status #> [1] "HTTP/1.1 200 OK" #> #> $connection #> [1] "keep-alive" #> #> $server #> [1] "meinheld/0.6.1" #> #> $date #> [1] "Mon, 15 May 2017 18:11:57 GMT" #> #> $`content-type` #> [1] "application/json" #> #> $`access-control-allow-origin` #> [1] "*" #> #> $`access-control-allow-credentials` #> [1] "true" #> #> $`x-powered-by` #> [1] "Flask" #> #> $`x-processed-time` #> [1] "0.00109100341797" #> #> $`content-length` #> [1] "287" #> #> $via #> [1] "1.1 vegur" #>res_get1$parse()#>#> [1] "{\n \"args\": {}, \n \"headers\": {\n \"Accept\": \"*/*\", \n \"Accept-Encoding\": \"gzip, deflate\", \n \"Connection\": \"close\", \n \"Host\": \"httpbin.org\", \n \"User-Agent\": \"libcurl/7.51.0 r-curl/2.6 crul/<IP_ADDRESS>10\"\n }, \n \"origin\": \"<IP_ADDRESS>\", \n \"url\": \"https://httpbin.org/get\"\n}\n"(res_get2 <- x$get('get', query = list(hello = "world")))#> <crul response> #> url: https://httpbin.org/get?hello=world #> request_headers: #> User-Agent: libcurl/7.51.0 r-curl/2.6 crul/<IP_ADDRESS>10 #> Accept-Encoding: gzip, deflate #> response_headers: #> status: HTTP/1.1 200 OK #> connection: keep-alive #> server: meinheld/0.6.1 #> date: Mon, 15 May 2017 18:11:56 GMT #> content-type: application/json #> access-control-allow-origin: * #> access-control-allow-credentials: true #> x-powered-by: Flask #> x-processed-time: 0.00133991241455 #> content-length: 323 #> via: 1.1 vegur #> params: #> hello: world #> status: 200res_get2$parse()#>#> [1] "{\n \"args\": {\n \"hello\": \"world\"\n }, \n \"headers\": {\n \"Accept\": \"*/*\", \n \"Accept-Encoding\": \"gzip, deflate\", \n \"Connection\": \"close\", \n \"Host\": \"httpbin.org\", \n \"User-Agent\": \"libcurl/7.51.0 r-curl/2.6 crul/<IP_ADDRESS>10\"\n }, \n \"origin\": \"<IP_ADDRESS>\", \n \"url\": \"https://httpbin.org/get?hello=world\"\n}\n"library("jsonlite") jsonlite::fromJSON(res_get2$parse())#>#> $args #> $args$hello #> [1] "world" #> #> #> $headers #> $headers$Accept #> [1] "*/*" #> #> $headers$`Accept-Encoding` #> [1] "gzip, deflate" #> #> $headers$Connection #> [1] "close" #> #> $headers$Host #> [1] "httpbin.org" #> #> $headers$`User-Agent` #> [1] "libcurl/7.51.0 r-curl/2.6 crul/<IP_ADDRESS>10" #> #> #> $origin #> [1] "<IP_ADDRESS>" #> #> $url #> [1] "https://httpbin.org/get?hello=world" #># post request (res_post <- x$post('post', body = list(hello = "world")))#> <crul response> #> url: https://httpbin.org/post #> request_headers: #> User-Agent: libcurl/7.51.0 r-curl/2.6 crul/<IP_ADDRESS>10 #> Accept-Encoding: gzip, deflate #> response_headers: #> status: HTTP/1.1 200 OK #> connection: keep-alive #> server: meinheld/0.6.1 #> date: Mon, 15 May 2017 18:11:59 GMT #> content-type: application/json #> access-control-allow-origin: * #> access-control-allow-credentials: true #> x-powered-by: Flask #> x-processed-time: 0.00125694274902 #> content-length: 531 #> via: 1.1 vegur #> status: 200## empty body request x$post('post')#> <crul response> #> url: https://httpbin.org/post #> request_headers: #> User-Agent: libcurl/7.51.0 r-curl/2.6 crul/<IP_ADDRESS>10 #> Accept-Encoding: gzip, deflate #> response_headers: #> status: HTTP/1.1 200 OK #> connection: keep-alive #> server: meinheld/0.6.1 #> date: Mon, 15 May 2017 18:12:00 GMT #> content-type: application/json #> access-control-allow-origin: * #> access-control-allow-credentials: true #> x-powered-by: Flask #> x-processed-time: 0.00109601020813 #> content-length: 437 #> via: 1.1 vegur #> status: 200# put request (res_put <- x$put('put'))#> <crul response> #> url: https://httpbin.org/put #> request_headers: #> User-Agent: libcurl/7.51.0 r-curl/2.6 crul/<IP_ADDRESS>10 #> Accept-Encoding: gzip, deflate #> response_headers: #> status: HTTP/1.1 200 OK #> connection: keep-alive #> server: meinheld/0.6.1 #> date: Mon, 15 May 2017 18:12:00 GMT #> content-type: application/json #> access-control-allow-origin: * #> access-control-allow-credentials: true #> x-powered-by: Flask #> x-processed-time: 0.00123000144958 #> content-length: 436 #> via: 1.1 vegur #> status: 200# delete request (res_delete <- x$delete('delete'))#> <crul response> #> url: https://httpbin.org/delete #> request_headers: #> User-Agent: libcurl/7.51.0 r-curl/2.6 crul/<IP_ADDRESS>10 #> Accept-Encoding: gzip, deflate #> response_headers: #> status: HTTP/1.1 200 OK #> connection: keep-alive #> server: meinheld/0.6.1 #> date: Mon, 15 May 2017 18:12:00 GMT #> content-type: application/json #> access-control-allow-origin: * #> access-control-allow-credentials: true #> x-powered-by: Flask #> x-processed-time: 0.00131607055664 #> content-length: 439 #> via: 1.1 vegur #> status: 200# patch request (res_patch <- x$patch('patch'))#> <crul response> #> url: https://httpbin.org/patch #> request_headers: #> User-Agent: libcurl/7.51.0 r-curl/2.6 crul/<IP_ADDRESS>10 #> Accept-Encoding: gzip, deflate #> response_headers: #> status: HTTP/1.1 200 OK #> connection: keep-alive #> server: meinheld/0.6.1 #> date: Mon, 15 May 2017 18:12:00 GMT #> content-type: application/json #> access-control-allow-origin: * #> access-control-allow-credentials: true #> x-powered-by: Flask #> x-processed-time: 0.00135493278503 #> content-length: 438 #> via: 1.1 vegur #> status: 200# head request (res_head <- x$head())#> <crul response> #> url: https://httpbin.org/ #> request_headers: #> useragent: libcurl/7.51.0 r-curl/2.6 crul/<IP_ADDRESS>10 #> response_headers: #> status: HTTP/1.1 200 OK #> connection: keep-alive #> server: meinheld/0.6.1 #> date: Mon, 15 May 2017 18:11:59 GMT #> content-type: text/html; charset=utf-8 #> content-length: 12373 #> access-control-allow-origin: * #> access-control-allow-credentials: true #> x-powered-by: Flask #> x-processed-time: 0.00640797615051 #> via: 1.1 vegur #> status: 200# query params are URL encoded for you, so DO NOT do it yourself ## if you url encode yourself, it gets double encoded, and that's bad (x <- HttpClient$new(url = "https://httpbin.org"))#> <crul connection> #> url: https://httpbin.org #> curl options: #> proxies: #> auth: #> headers:res <- x$get("get", query = list(a = 'hello world'), verbose = TRUE)