This module includes both unit tests, which can run in isolation without connecting to the S3 service, and integration tests, which require a working connection to S3 to interact with a bucket. Unit test suites follow the naming convention Test*.java
. Integration tests follow the naming convention ITest*.java
.
Due to eventual consistency, integration tests may fail without reason. Transient failures, which no longer occur upon rerunning the test, should thus be ignored.
hadoop-aws
module.The Apache Jenkins infrastructure does not run any S3 integration tests, due to the need to keep credentials secure.
This is important: patches which do not include this declaration will be ignored
This policy has proven to be the only mechanism to guarantee full regression testing of code changes. Why the declaration of region? Two reasons
You don’t need to test from a VM within the AWS infrastructure; with the -Dparallel=tests
option the non-scale tests complete in under ten minutes. Because the tests clean up after themselves, they are also designed to be low cost. It’s neither hard nor expensive to run the tests; if you can’t, there’s no guarantee your patch works. The reviewers have enough to do, and don’t have the time to do these tests, especially as every failure will simply make for a slow iterative development.
Please: run the tests. And if you don’t, we are sorry for declining your patch, but we have to.
Some of the tests do fail intermittently, especially in parallel runs. If this happens, try to run the test on its own to see if the test succeeds.
If it still fails, include this fact in your declaration. We know some tests are intermittently unreliable.
The tests and the S3A client are designed to be configurable for different timeouts. If you are seeing problems and this configuration isn’t working, that’s a sign of the configuration mechanism isn’t complete. If it’s happening in the production code, that could be a sign of a problem which may surface over long-haul connections. Please help us identify and fix these problems — especially as you are the one best placed to verify the fixes work.
To integration test the S3* filesystem clients, you need to provide auth-keys.xml
which passes in authentication details to the test runner.
It is a Hadoop XML configuration file, which must be placed into hadoop-tools/hadoop-aws/src/test/resources
.
core-site.xml
This file pre-exists and sources the configurations created under auth-keys.xml
.
For most purposes you will not need to edit this file unless you need to apply a specific, non-default property change during the tests.
auth-keys.xml
The presence of this file triggers the testing of the S3 classes.
Without this file, none of the integration tests in this module will be executed.
The XML file must contain all the ID/key information needed to connect each of the filesystem clients to the object stores, and a URL for each filesystem for its testing.
test.fs.s3a.name
: the URL of the bucket for S3a testsfs.contract.test.fs.s3a
: the URL of the bucket for S3a filesystem contract testsThe contents of the bucket will be destroyed during the test process: do not use the bucket for any purpose other than testing. Furthermore, for s3a, all in-progress multi-part uploads to the bucket will be aborted at the start of a test (by forcing fs.s3a.multipart.purge=true
) to clean up the temporary state of previously failed tests.
Example:
<configuration> <property> <name>test.fs.s3a.name</name> <value>s3a://test-aws-s3a/</value> </property> <property> <name>fs.contract.test.fs.s3a</name> <value>${test.fs.s3a.name}</value> </property> <property> <name>fs.s3a.access.key</name> <description>AWS access key ID. Omit for IAM role-based authentication.</description> <value>DONOTCOMMITTHISKEYTOSCM</value> </property> <property> <name>fs.s3a.secret.key</name> <description>AWS secret key. Omit for IAM role-based authentication.</description> <value>DONOTEVERSHARETHISSECRETKEY!</value> </property> <property> <name>test.sts.endpoint</name> <description>Specific endpoint to use for STS requests.</description> <value>sts.amazonaws.com</value> </property> </configuration>
For S3a encryption tests to run correctly, the fs.s3a.encryption.key
must be configured in the s3a contract xml file or auth-keys.xml
file with a AWS KMS encryption key arn as this value is different for each AWS KMS. Please note this KMS key should be created in the same region as your S3 bucket. Otherwise, you may get KMS.NotFoundException
.
Example:
<property> <name>fs.s3a.encryption.key</name> <value>arn:aws:kms:us-west-2:360379543683:key/071a86ff-8881-4ba0-9230-95af6d01ca01</value> </property>
You can also force all the tests to run with a specific SSE encryption method by configuring the property fs.s3a.encryption.algorithm
in the s3a contract file.
Buckets can be configured with default encryption on the AWS side. Some S3AFileSystem tests are skipped when default encryption is enabled due to unpredictability in how ETags are generated.
After completing the configuration, execute the test run through Maven.
mvn clean verify
It’s also possible to execute multiple test suites in parallel by passing the parallel-tests
property on the command line. The tests spend most of their time blocked on network I/O with the S3 service, so running in parallel tends to complete full test runs faster.
mvn -Dparallel-tests clean verify
Some tests must run with exclusive access to the S3 bucket, so even with the parallel-tests
property, several test suites will run in serial in a separate Maven execution step after the parallel tests.
By default, parallel-tests
runs 4 test suites concurrently. This can be tuned by passing the testsThreadCount
property.
mvn -Dparallel-tests -DtestsThreadCount=8 clean verify
To run just unit tests, which do not require S3 connectivity or AWS credentials, use any of the above invocations, but switch the goal to test
instead of verify
.
mvn clean test mvn -Dparallel-tests clean test mvn -Dparallel-tests -DtestsThreadCount=8 clean test
To run only a specific named subset of tests, pass the test
property for unit tests or the it.test
property for integration tests.
mvn clean test -Dtest=TestS3AInputPolicies mvn clean verify -Dit.test=ITestS3AFileContextStatistics -Dtest=none mvn clean verify -Dtest=TestS3A* -Dit.test=ITestS3A*
Note that when running a specific subset of tests, the patterns passed in test
and it.test
override the configuration of which tests need to run in isolation in a separate serial phase (mentioned above). This can cause unpredictable results, so the recommendation is to avoid passing parallel-tests
in combination with test
or it.test
. If you know that you are specifying only tests that can run safely in parallel, then it will work. For wide patterns, like ITestS3A*
shown above, it may cause unpredictable test failures.
S3A can connect to different regions —the tests support this. Simply define the target region in auth-keys.xml
.
<property> <name>fs.s3a.endpoint</name> <value>s3.eu-central-1.amazonaws.com</value> </property>
Alternatively you can use endpoints defined in core-site.xml.
<property> <name>fs.s3a.endpoint</name> <value>${frankfurt.endpoint}</value> </property>
This is used for all tests expect for scale tests using a Public CSV.gz file (see below)
The TestS3AInputStreamPerformance
tests require read access to a multi-MB text file. The default file for these tests is one published by amazon, s3a://landsat-pds.s3.amazonaws.com/scene_list.gz. This is a gzipped CSV index of other files which amazon serves for open use.
The path to this object is set in the option fs.s3a.scale.test.csvfile
,
<property> <name>fs.s3a.scale.test.csvfile</name> <value>s3a://landsat-pds/scene_list.gz</value> </property>
fs.s3a.scale.test.csvfile
is empty, tests which require it will be skipped.(the reason the space or newline is needed is to add “an empty entry”; an empty <value/>
would be considered undefined and pick up the default)
Of using a test file in an S3 region requiring a different endpoint value set in fs.s3a.endpoint
, a bucket-specific endpoint must be defined. For the default test dataset, hosted in the landsat-pds
bucket, this is:
<property> <name>fs.s3a.bucket.landsat-pds.endpoint</name> <value>s3.amazonaws.com</value> <description>The endpoint for s3a://landsat-pds URLs</description> </property>
S3a supports using Access Point ARNs to access data in S3. If you think your changes affect VPC integration, request signing, ARN manipulation, or any code path that deals with the actual sending and retrieving of data to/from S3, make sure you run the entire integration test suite with this feature enabled.
Check out our documentation for steps on how to enable this feature. To create access points for your S3 bucket you can use the AWS Console or CLI.
Integration test results and logs are stored in target/failsafe-reports/
. An HTML report can be generated during site generation, or with the surefire-report
plugin:
mvn surefire-report:failsafe-report-only
Some tests (specifically some in ITestS3ARemoteFileChanged
) require a versioned bucket for full test coverage.
To enable versioning in a bucket.
Once a bucket is converted to being versioned, it cannot be converted back to being unversioned.
Hadoop supports different policies for directory marker retention -essentially the classic “delete” and the higher-performance “keep” options; “authoritative” is just “keep” restricted to a part of the bucket.
Example: test with markers=delete
mvn verify -Dparallel-tests -DtestsThreadCount=4 -Dmarkers=delete
Example: test with markers=keep
mvn verify -Dparallel-tests -DtestsThreadCount=4 -Dmarkers=keep
Example: test with markers=authoritative
mvn verify -Dparallel-tests -DtestsThreadCount=4 -Dmarkers=authoritative
This final option is of limited use unless paths in the bucket have actually been configured to be of mixed status; unless anything is set up then the outcome should equal that of “delete”
To enable an audit of the output directory of every test suite, enable the option fs.s3a.directory.marker.audit
-Dfs.s3a.directory.marker.audit=true
When set, if the marker policy is to delete markers under the test output directory, then the marker tool audit command will be run. This will fail if a marker was found.
This adds extra overhead to every operation, but helps verify that the connector is not keeping markers where it needs to be deleting them -and hence backwards compatibility is maintained.
There are a set of tests designed to measure the scalability and performance at scale of the S3A tests, Scale Tests. Tests include: creating and traversing directory trees, uploading large files, renaming them, deleting them, seeking through the files, performing random IO, and others. This makes them a foundational part of the benchmarking.
By their very nature they are slow. And, as their execution time is often limited by bandwidth between the computer running the tests and the S3 endpoint, parallel execution does not speed these tests up.
The tests are enabled if the scale
property is set in the maven build this can be done regardless of whether or not the parallel test profile is used
mvn verify -Dscale mvn verify -Dparallel-tests -Dscale -DtestsThreadCount=8
The most bandwidth intensive tests (those which upload data) always run sequentially; those which are slow due to HTTPS setup costs or server-side actions are included in the set of parallelized tests.
Some of the tests can be tuned from the maven build or from the configuration file used to run the tests.
mvn verify -Dparallel-tests -Dscale -DtestsThreadCount=8 -Dfs.s3a.scale.test.huge.filesize=128M
The algorithm is
unset
, then the configuration value is used. The unset
option is used to work round a quirk in maven property propagation.Only a few properties can be set this way; more will be added.
| Property | Meaning |
|---|---|
fs.s3a.scale.test.timeout |
Timeout in seconds for scale tests |
fs.s3a.scale.test.huge.filesize |
Size for huge file uploads |
fs.s3a.scale.test.huge.huge.partitionsize |
Size for partitions in huge file uploads |
The file and partition sizes are numeric values with a k/m/g/t/p suffix depending on the desired size. For example: 128M, 128m, 2G, 2G, 4T or even 1P.
Some scale tests perform multiple operations (such as creating many directories).
The exact number of operations to perform is configurable in the option scale.test.operation.count
<property> <name>scale.test.operation.count</name> <value>10</value> </property>
Larger values generate more load, and are recommended when testing locally, or in batch runs.
Smaller values results in faster test runs, especially when the object store is a long way away.
Operations which work on directories have a separate option: this controls the width and depth of tests creating recursive directories. Larger values create exponentially more directories, with consequent performance impact.
<property> <name>scale.test.directory.count</name> <value>2</value> </property>
DistCp tests targeting S3A support a configurable file size. The default is 10 MB, but the configuration value is expressed in KB so that it can be tuned smaller to achieve faster test runs.
<property> <name>scale.test.distcp.file.size.kb</name> <value>10240</value> </property>
S3A specific scale test properties are
fs.s3a.scale.test.huge.filesize
: size in MB for “Huge file tests”.
The Huge File tests validate S3A’s ability to handle large files —the property fs.s3a.scale.test.huge.filesize
declares the file size to use.
<property> <name>fs.s3a.scale.test.huge.filesize</name> <value>200M</value> </property>
Amazon S3 handles files larger than 5GB differently than smaller ones. Setting the huge filesize to a number greater than that) validates support for huge files.
<property> <name>fs.s3a.scale.test.huge.filesize</name> <value>6G</value> </property>
Tests at this scale are slow: they are best executed from hosts running in the cloud infrastructure where the S3 endpoint is based. Otherwise, set a large timeout in fs.s3a.scale.test.timeout
<property> <name>fs.s3a.scale.test.timeout</name> <value>432000</value> </property>
The tests are executed in an order to only clean up created files after the end of all the tests. If the tests are interrupted, the test data will remain.
Some are designed to overload AWS services with more requests per second than an AWS account is permitted.
The operation of these test maybe observable to other users of the same account -especially if they are working in the AWS region to which the tests are targeted.
There may also run up larger bills.
These tests all have the prefix ILoadTest
They do not run automatically: they must be explicitly run from the command line or an IDE.
Look in the source for these and reads the Javadocs before executing.
The S3A filesystem is designed to work with storage endpoints which implement the S3 protocols to the extent that the amazon S3 SDK is capable of talking to it. We encourage testing against other filesystems and submissions of patches which address issues. In particular, we encourage testing of Hadoop release candidates, as these third-party endpoints get even less testing than the S3 endpoint itself.
Some tests rely on the presence of existing public datasets available on Amazon S3. You may find a number of these in org.apache.hadoop.fs.s3a.test.PublicDatasetTestUtils
.
When testing against an endpoint which is not part of Amazon S3’s standard commercial partition (aws
) such as third-party implementations or AWS’s China regions, you should replace these configurations with an empty space () to disable the tests or an existing path in your object store that supports these tests.
An example of this might be the MarkerTools tests which require a bucket with a large number of objects or the requester pays tests that require requester pays to be enabled for the bucket.
If the endpoint doesn’t support server-side-encryption, these will fail. They can be turned off.
<property> <name>test.fs.s3a.encryption.enabled</name> <value>false</value> </property>
Encryption is only used for those specific test suites with Encryption
in their classname.
When running storage class tests against third party object store that doesn’t support S3 storage class, these tests might fail. They can be disabled.
<property> <name>test.fs.s3a.create.storage.class.enabled</name> <value>false</value> </property>
To test on alternate infrastructures supporting the same APIs, the option fs.s3a.scale.test.csvfile
must either be set to " ", or an object of at least 10MB is uploaded to the object store, and the fs.s3a.scale.test.csvfile
option set to its path.
<property> <name>fs.s3a.scale.test.csvfile</name> <value> </value> </property>
(yes, the space is necessary. The Hadoop Configuration
class treats an empty value as “do not override the default”).
The S3 select tests are skipped when the S3 endpoint doesn’t support S3 Select.
<property> <name>fs.s3a.select.enabled</name> <value>false</value> </property>
If your endpoint doesn’t support that feature, this option should be in your core-site.xml
file, so that trying to use S3 select fails fast with a meaningful error (“S3 Select not supported”) rather than a generic Bad Request exception.
By default, the requester pays tests will look for a bucket that exists on Amazon S3 in us-east-1.
If the endpoint does support requester pays, you can specify an alternative object. The test only requires an object of at least a few bytes in order to check that lists and basic reads work.
<property> <name>test.fs.s3a.requester.pays.file</name> <value>s3a://my-req-pays-enabled-bucket/on-another-endpoint.json</value> </property>
If the endpoint does not support requester pays, you can also disable the tests by configuring the test URI as a single space.
<property> <name>test.fs.s3a.requester.pays.file</name> <value> </value> </property>
Some tests requests a session credentials and assumed role credentials from the AWS Secure Token Service, then use them to authenticate with S3 either directly or via delegation tokens.
If an S3 implementation does not support STS, then these functional test cases must be disabled:
<property> <name>test.fs.s3a.sts.enabled</name> <value>false</value> </property>
These tests request a temporary set of credentials from the STS service endpoint. An alternate endpoint may be defined in fs.s3a.assumed.role.sts.endpoint
. If this is set, a delegation token region must also be defined: in fs.s3a.assumed.role.sts.endpoint.region
. This is useful not just for testing alternative infrastructures, but to reduce latency on tests executed away from the central service.
<property> <name>fs.s3a.delegation.token.endpoint</name> <value>fs.s3a.assumed.role.sts.endpoint</value> </property> <property> <name>fs.s3a.assumed.role.sts.endpoint.region</name> <value>eu-west-2</value> </property>
The default is ""; meaning “use the amazon default endpoint” (sts.amazonaws.com
).
Consult the AWS documentation for the full list of locations.
Logging at debug level is the standard way to provide more diagnostics output; after setting this rerun the tests
log4j.logger.org.apache.hadoop.fs.s3a=DEBUG
There are also some logging options for debug logging of the AWS client
log4j.logger.com.amazonaws=DEBUG log4j.logger.com.amazonaws.http.conn.ssl=INFO log4j.logger.com.amazonaws.internal=INFO
There is also the option of enabling logging on a bucket; this could perhaps be used to diagnose problems from that end. This isn’t something actively used, but remains an option. If you are forced to debug this way, consider setting the fs.s3a.user.agent.prefix
to a unique prefix for a specific test run, which will enable the specific log entries to be more easily located.
New tests are always welcome. Bear in mind that we need to keep costs and test time down, which is done by
No duplication: if an operation is tested elsewhere, don’t repeat it. This applies as much for metadata operations as it does for bulk IO. If a new test case is added which completely obsoletes an existing test, it is OK to cut the previous one —after showing that coverage is not worsened.
Efficient: prefer the getFileStatus()
and examining the results, rather than call to exists()
, isFile()
, etc.
Isolating Scale tests. Any S3A test doing large amounts of IO MUST extend the class S3AScaleTestBase
, so only running if scale
is defined on a build, supporting test timeouts configurable by the user. Scale tests should also support configurability as to the actual size of objects/number of operations, so that behavior at different scale can be verified.
Designed for parallel execution. A key need here is for each test suite to work on isolated parts of the filesystem. Subclasses of AbstractS3ATestBase
SHOULD use the path()
method, with a base path of the test suite name, to build isolated paths. Tests MUST NOT assume that they have exclusive access to a bucket.
Extending existing tests where appropriate. This recommendation goes against normal testing best practise of “test one thing per method”. Because it is so slow to create directory trees or upload large files, we do not have that luxury. All the tests against real S3 endpoints are integration tests where sharing test setup and teardown saves time and money.
A standard way to do this is to extend existing tests with some extra predicates, rather than write new tests. When doing this, make sure that the new predicates fail with meaningful diagnostics, so any new problems can be easily debugged from test logs.
Effective use of FS instances during S3A integration tests. Tests using FileSystem
instances are fastest if they can recycle the existing FS instance from the same JVM.
If you do that, you MUST NOT close or do unique configuration on them. If you want a guarantee of 100% isolation or an instance with unique config, create a new instance which you MUST close in the teardown to avoid leakage of resources.
Do NOT add FileSystem
instances manually (with e.g org.apache.hadoop.fs.FileSystem#addFileSystemForTesting
) to the cache that will be modified or closed during the test runs. This can cause other tests to fail when using the same modified or closed FS instance. For more details see HADOOP-15819.
This is what we expect from new tests; they’re an extension of the normal Hadoop requirements, based on the need to work with remote servers whose use requires the presence of secret credentials, where tests may be slow, and where finding out why something failed from nothing but the test output is critical.
Extend AbstractS3ATestBase
or AbstractSTestS3AHugeFiles
unless justifiable. These set things up for testing against the object stores, provide good threadnames, help generate isolated paths, and for AbstractSTestS3AHugeFiles
subclasses, only run if -Dscale
is set.
Key features of AbstractS3ATestBase
getFileSystem()
returns the S3A Filesystem bonded to the contract test Filesystem defined in fs.s3a.contract.test
AbstractFSContractTestBase
and Assert
for all their methods.Having shared base classes may help reduce future maintenance too. Please use them/
Don’t ever log credentials. The credential tests go out of their way to not provide meaningful logs or assertion messages precisely to avoid this.
This means efficient in test setup/teardown, and, ideally, making use of existing public datasets to save setup time and tester cost.
Strategies of particular note are:
ITestS3ADirectoryPerformance
: a single test case sets up the directory tree then performs different list operations, measuring the time taken.AbstractSTestS3AHugeFiles
: marks the test suite as @FixMethodOrder(MethodSorters.NAME_ASCENDING)
then orders the test cases such that each test case expects the previous test to have completed (here: uploaded a file, renamed a file, …). This provides for independent tests in the reports, yet still permits an ordered sequence of operations. Do note the use of Assume.assume()
to detect when the preconditions for a single test case are not met, hence, the tests become skipped, rather than fail with a trace which is really a false alarm.The ordered test case mechanism of AbstractSTestS3AHugeFiles
is probably the most elegant way of chaining test setup/teardown.
Regarding reusing existing data, we tend to use the landsat archive of AWS US-East for our testing of input stream operations. This doesn’t work against other regions, or with third party S3 implementations. Thus the URL can be overridden for testing elsewhere.
Don’t assume AWS S3 US-East only, do allow for working with external S3 implementations. Those may be behind the latest S3 API features, not support encryption, session APIs, etc.
They won’t have the same CSV test files as some of the input tests rely on. Look at ITestS3AInputStreamPerformance
to see how tests can be written to support the declaration of a specific large test file on alternate filesystems.
As well as making file size and operation counts scalable, this includes making test timeouts adequate. The Scale tests make this configurable; it’s hard coded to ten minutes in AbstractS3ATestBase()
; subclasses can change this by overriding getTestTimeoutMillis()
.
Equally importantly: support proxies, as some testers need them.
S3AFileSystem
and its input and output streams all provide useful statistics in their {{toString()}} calls; logging them is useful on its own.AbstractS3ATestBase.describe(format-stringm, args)
here.; it adds some newlines so as to be easier to spot.ContractTestUtils.NanoTimer
to measure the duration of operations, and log the output.The ContractTestUtils
class contains a whole set of assertions for making statements about the expected state of a filesystem, e.g. assertPathExists(FS, path)
, assertPathDoesNotExists(FS, path)
, and others. These do their best to provide meaningful diagnostics on failures (e.g. directory listings, file status, …), so help make failures easier to understand.
At the very least, do not use assertTrue()
or assertFalse()
without including error messages.
Tests can overrun createConfiguration()
to add new options to the configuration file for the S3A Filesystem instance used in their tests.
However, filesystem caching may mean that a test suite may get a cached instance created with an different configuration. For tests which don’t need specific configurations caching is good: it reduces test setup time.
For those tests which do need unique options (encryption, magic files), things can break, and they will do so in hard-to-replicate ways.
Use S3ATestUtils.disableFilesystemCaching(conf)
to disable caching when modifying the config. As an example from AbstractTestS3AEncryption
:
@Override protected Configuration createConfiguration() { Configuration conf = super.createConfiguration(); S3ATestUtils.disableFilesystemCaching(conf); conf.set(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM, getSSEAlgorithm().getMethod()); return conf; }
Then verify in the setup method or test cases that their filesystem actually has the desired feature (fs.getConf().getProperty(...)
). This not only catches filesystem reuse problems, it catches the situation where the filesystem configuration in auth-keys.xml
has explicit per-bucket settings which override the test suite’s general option settings.
Keeps costs down.
We really appreciate this — you will too.
Tests must be designed to run in parallel with other tests, all working with the same shared S3 bucket. This means
AbstractFSContractTestBase.path(String filepath)
.Tests such as these can only be run as sequential tests. When adding one, exclude it in the POM file. from the parallel failsafe run and add to the sequential one afterwards. The IO heavy ones must also be subclasses of S3AScaleTestBase
and so only run if the system/maven property fs.s3a.scale.test.enabled
is true.
This is invaluable for debugging test failures.
How to set test options in your hadoop configuration rather than on the maven command line:
Most of the base S3 tests are designed to use public AWS data (the landsat-pds bucket) for read IO, so you don’t have to pay for bytes downloaded or long term storage costs. The scale tests do work with more data so will cost more as well as generally take more time to execute.
You are however billed for
The GET/decrypt costs are incurred on each partial read of a file, so random IO can cost more than sequential IO; the speedup of queries with columnar data usually justifies this.
How to keep costs down
fs.s3a.scale.test.huge.filesize
unset, or a few MB (minimum: 5).*hadoop fs -rm -r -f -skipTrash s3a://test-bucket/
hadoop s3guard uploads -abort -force s3a://test-bucket/
Although the auth-keys.xml
file is marked as ignored in git and subversion, it is still in your source tree, and there’s always that risk that it may creep out.
You can avoid this by keeping your keys outside the source tree and using an absolute XInclude reference to it.
<configuration> <include xmlns="http://www.w3.org/2001/XInclude" href="file:///users/ubuntu/.auth-keys.xml" /> </configuration>
Warning do not enable any type of failure injection in production. The following settings are for testing only.
One of the challenges with S3A integration tests was the fact that S3 was an eventually-consistent storage system. To simulate inconsistencies more frequently than they would normally surface, S3A supports a shim layer on top of the AmazonS3Client
class which artificially delays certain paths from appearing in listings. This is implemented in the class InconsistentAmazonS3Client
.
Now that S3 is consistent, injecting inconsistency is no longer needed during testing. However, it is stil useful to use the other feature of the client: throttling simulation.
To enable the fault-injecting client via configuration, switch the S3A client to use the “Inconsistent S3 Client Factory” when connecting to S3:
<property> <name>fs.s3a.s3.client.factory.impl</name> <value>org.apache.hadoop.fs.s3a.InconsistentS3ClientFactory</value> </property>
The inconsistent client will, on every AWS SDK request, generate a random number, and if less than the probability, raise a 503 exception.
<property> <name>fs.s3a.failinject.throttle.probability</name> <value>0.05</value> </property>
These exceptions are returned to S3; they do not test the AWS SDK retry logic.
InconsistentAmazonS3CClient
in downstream integration testsThe inconsistent client is shipped in the hadoop-aws
JAR, so it can be used in integration tests.
As part of the removal of S3Guard from the production code, the tests have been updated so that
There is no need to test S3Guard -and so tests are lot faster. (We developers are all happy)
Tests for the AWS Assumed Role credential provider require an assumed role to request.
If this role is not declared in fs.s3a.assumed.role.arn
, the tests which require it will be skipped.
The specific tests an Assumed Role ARN is required for are
ITestAssumeRole
.ITestRoleDelegationTokens
.ITestDelegatedMRJob
.To run these tests you need:
A role in your AWS account will full read and write access rights to the S3 bucket used in the tests, and KMS for any SSE-KMS tests.
Your IAM User to have the permissions to “assume” that role.
The role ARN must be set in fs.s3a.assumed.role.arn
.
<property> <name>fs.s3a.assumed.role.arn</name> <value>arn:aws:iam::9878543210123:role/role-s3-restricted</value> </property>
The tests assume the role with different subsets of permissions and verify that the S3A client (mostly) works when the caller has only write access to part of the directory tree.
You can also run the entire test suite in an assumed role, a more thorough test, by switching to the credentials provider.
<property> <name>fs.s3a.aws.credentials.provider</name> <value>org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider</value> </property>
The usual credentials needed to log in to the bucket will be used, but now the credentials used to interact with S3 will be temporary role credentials, rather than the full credentials.
Updating the AWS SDK is something which does need to be done regularly, but is rarely without complications, major or minor.
Assume that the version of the SDK will remain constant for an X.Y release, excluding security fixes, so it’s good to have an update before each release — as long as that update works doesn’t trigger any regressions.
hadoop-project/pom.xml
update the aws-java-sdk.version
to the new SDK version.hadoop-aws
tests. This includes the -Pscale
set, with a role defined for the assumed role tests. in fs.s3a.assumed.role.arn
for testing assumed roles, and fs.s3a.encryption.key
for encryption, for full coverage. If you can, scale up the scale tests.ITest*
integration tests from your IDE or via maven.ILoadTest*
load tests from your IDE or via maven through mvn verify -Dtest=skip -Dit.test=ILoadTest\*
; look for regressions in performance as much as failures.mvn site -DskipTests
; look in target/site
for the report.-output.txt
file in hadoop-tools/hadoop-aws/target/failsafe-reports
, paying particular attention to org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance-output.txt
, as that is where changes in stream close/abort logic will surface.mvn install
to install the artifacts, then in hadoop-cloud-storage-project/hadoop-cloud-storage
run mvn dependency:tree -Dverbose > target/dependencies.txt
. Examine the target/dependencies.txt
file to verify that no new artifacts have unintentionally been declared as dependencies of the shaded aws-java-sdk-bundle
artifact.fs.s3a.encryption.algorithm
to ‘CSE-KMS’ and setting up AWS-KMS Key ID in fs.s3a.encryption.key
.We need a run through of the CLI to see if there have been changes there which cause problems, especially whether new log messages have surfaced, or whether some packaging change breaks that CLI.
It is always interesting when doing this to enable IOStatistics reporting
<property> <name>fs.iostatistics.logging.level</name> <value>info</value> </property>
From the root of the project, create a command line release mvn package -Pdist -DskipTests -Dmaven.javadoc.skip=true -DskipShade
;
hadoop-dist/target/hadoop-x.y.z-SNAPSHOT
dir.core-site.xml
file into etc/hadoop
.HADOOP_OPTIONAL_TOOLS
env var on the command line or ~/.hadoop-env
.export HADOOP_OPTIONAL_TOOLS="hadoop-aws"
Run some basic s3guard CLI as well as file operations.
export BUCKETNAME=example-bucket-name export BUCKET=s3a://$BUCKETNAME bin/hadoop s3guard bucket-info $BUCKET bin/hadoop s3guard uploads $BUCKET # repeat twice, once with "no" and once with "yes" as responses bin/hadoop s3guard uploads -abort $BUCKET # --------------------------------------------------- # root filesystem operatios # --------------------------------------------------- bin/hadoop fs -ls $BUCKET/ # assuming file is not yet created, expect error and status code of 1 bin/hadoop fs -ls $BUCKET/file # exit code of 0 even when path doesn't exist bin/hadoop fs -rm -R -f $BUCKET/dir-no-trailing bin/hadoop fs -rm -R -f $BUCKET/dir-trailing/ # error because it is a directory bin/hadoop fs -rm $BUCKET/ bin/hadoop fs -touchz $BUCKET/file # expect I/O error as it is the root directory bin/hadoop fs -rm -r $BUCKET/ # succeeds bin/hadoop fs -rm -r $BUCKET/\* # --------------------------------------------------- # File operations # --------------------------------------------------- bin/hadoop fs -mkdir $BUCKET/dir-no-trailing bin/hadoop fs -mkdir $BUCKET/dir-trailing/ bin/hadoop fs -touchz $BUCKET/file bin/hadoop fs -ls $BUCKET/ bin/hadoop fs -mv $BUCKET/file $BUCKET/file2 # expect "No such file or directory" bin/hadoop fs -stat $BUCKET/file # expect success bin/hadoop fs -stat $BUCKET/file2 # expect "file exists" bin/hadoop fs -mkdir $BUCKET/dir-no-trailing bin/hadoop fs -mv $BUCKET/file2 $BUCKET/dir-no-trailing bin/hadoop fs -stat $BUCKET/dir-no-trailing/file2 # treated the same as the file stat bin/hadoop fs -stat $BUCKET/dir-no-trailing/file2/ bin/hadoop fs -ls $BUCKET/dir-no-trailing/file2/ bin/hadoop fs -ls $BUCKET/dir-no-trailing # expect a "0" here: bin/hadoop fs -test -d $BUCKET/dir-no-trailing ; echo $? # expect a "1" here: bin/hadoop fs -test -d $BUCKET/dir-no-trailing/file2 ; echo $? # will return NONE unless bucket has checksums enabled bin/hadoop fs -checksum $BUCKET/dir-no-trailing/file2 # expect "etag" + a long string bin/hadoop fs -D fs.s3a.etag.checksum.enabled=true -checksum $BUCKET/dir-no-trailing/file2 bin/hadoop fs -expunge -immediate -fs $BUCKET # --------------------------------------------------- # Delegation Token support # --------------------------------------------------- # failure unless delegation tokens are enabled bin/hdfs fetchdt --webservice $BUCKET secrets.bin # success bin/hdfs fetchdt -D fs.s3a.delegation.token.binding=org.apache.hadoop.fs.s3a.auth.delegation.SessionTokenBinding --webservice $BUCKET secrets.bin bin/hdfs fetchdt -print secrets.bin # expect warning "No TokenRenewer defined for token kind S3ADelegationToken/Session" bin/hdfs fetchdt -renew secrets.bin # --------------------------------------------------- # Directory markers # --------------------------------------------------- # require success bin/hadoop s3guard bucket-info -markers aware $BUCKET # expect failure unless bucket policy is keep bin/hadoop s3guard bucket-info -markers keep $BUCKET/path # you may need to set this on a per-bucket basis if you have already been # playing with options bin/hadoop s3guard -D fs.s3a.directory.marker.retention=keep bucket-info -markers keep $BUCKET/path bin/hadoop s3guard -D fs.s3a.bucket.$BUCKETNAME.directory.marker.retention=keep bucket-info -markers keep $BUCKET/path # expect to see "Directory markers will be kept" messages and status code of "46" bin/hadoop fs -D fs.s3a.bucket.$BUCKETNAME.directory.marker.retention=keep -mkdir $BUCKET/p1 bin/hadoop fs -D fs.s3a.bucket.$BUCKETNAME.directory.marker.retention=keep -mkdir $BUCKET/p1/p2 bin/hadoop fs -D fs.s3a.bucket.$BUCKETNAME.directory.marker.retention=keep -touchz $BUCKET/p1/p2/file # expect failure as markers will be found for /p1/ and /p1/p2/ bin/hadoop s3guard markers -audit -verbose $BUCKET # clean will remove markers bin/hadoop s3guard markers -clean -verbose $BUCKET # expect success and exit code of 0 bin/hadoop s3guard markers -audit -verbose $BUCKET # --------------------------------------------------- # Copy to from local # --------------------------------------------------- time bin/hadoop fs -copyFromLocal -t 10 share/hadoop/tools/lib/*aws*jar $BUCKET/ # expect the iostatistics object_list_request value to be O(directories) bin/hadoop fs -ls -R $BUCKET/ # expect the iostatistics object_list_request and op_get_content_summary values to be 1 bin/hadoop fs -du -h -s $BUCKET/ mkdir tmp time bin/hadoop fs -copyToLocal -t 10 $BUCKET/\*aws\* tmp # --------------------------------------------------- # S3 Select on Landsat # --------------------------------------------------- export LANDSATGZ=s3a://landsat-pds/scene_list.gz bin/hadoop s3guard select -header use -compression gzip $LANDSATGZ \ "SELECT s.entityId,s.cloudCover FROM S3OBJECT s WHERE s.cloudCover < '0.0' LIMIT 100" # --------------------------------------------------- # Cloudstore # check out and build https://github.com/steveloughran/cloudstore # then for these tests, set CLOUDSTORE env var to point to the JAR # --------------------------------------------------- bin/hadoop jar $CLOUDSTORE storediag $BUCKET time bin/hadoop jar $CLOUDSTORE bandwidth 64M $BUCKET/testfile
ILoadTestS3ABulkDeleteThrottling
.storediag
etc)A Jenkins run should tell you if there are new deprecations. If so, you should think about how to deal with them.
Moving to methods and APIs which weren’t in the previous SDK release makes it harder to roll back if there is a problem; but there may be good reasons for the deprecation.
At the same time, there may be good reasons for staying with the old code.
When the patch is committed: update the JIRA to the version number actually used; use that title in the commit message.
Be prepared to roll-back, re-iterate or code your way out of a regression.
There may be some problem which surfaces with wider use, which can get fixed in a new AWS release, rolling back to an older one, or just worked around HADOOP-14596.
Don’t be surprised if this happens, don’t worry too much, and, while that rollback option is there to be used, ideally try to work forwards.
If the problem is with the SDK, file issues with the AWS SDK Bug tracker. If the problem can be fixed or worked around in the Hadoop code, do it there too.


nugridpy.nugridse¶
nugridse is a collection of plots of data in se-type h5 files.
Usage¶
start by loading the module,
>>> import nugridse as mp
you can get help,
>>> help mp
next, initiate a se class instance,
>>> pt=mp.se('path/to/dir/with/se-files')
which would read all h5 files from the current directory; again, do help(mp.se) or mp.se? To get info for more options, as for example how to specify a match pattern to select only certain files in the directory; this may be useful if there are very many files and you want only every 10th. For example,
>>> pt =mp.se('.','M1.65Z0.020.00')
Note: Initializing an instance with 120 files with 1000 packets each may take 3-4 minutes, which is too long for many people, therefore the initialization module will generate and write an index file. If such an index file is present initialization will take less then a second. The initialization module will automatically detect various situations in which the index file needs to be rewritten, for example if there is a new file.
look at the data that is available in your instance,
>>> pt.sedir
>>> pt.sefiles
The actual data is in pt.se and the following commands give you access to header and cycle attributes as well as the cycle profile data:
>>> pt.se.cattrs
>>> pt.se.hattrs
>>> pt.se.dcols
Available cycles can be viewed with,
>>> pt.se.cycles
You can get any of the quantities via the ‘get’ method, which is relatively smart to give you things in various ways.
>>> pt.get('rho')
would give you all rho vectors for all cycles, which is maybe more than you want, so try
>>> pt.get(300,'rho')
to get the rho vector for cycle 300. Instead of one cycle you may also supply a cycle list for the first argument.
Use help(pt) (or whatever you instance is) for a full description of what else your instance has to offer, which includes methods to work with data.
Note: A particularly nice feature is that you can work with instances of different types of data in a very flexible way, for example in lists:
>>> cases=[pt1,pt2,pt3] >>> for this_case in cases: ... do something with this_case
There are various plotting methods, including plotting abundance charts (abu_chart), isotopic abundance distributions and the generic ‘plot’ method that lets you plot every quantity against any other quantity that can possibly be plotted. Some of the methods (including the three just mentioned) are available via the super class data_plot, and are equally available in other python modules, as for example mesa.py or ppn.py. Several methods accept lists of cycles which implies to create a series of frames to be written to disk in order to make movies.
>>> pt.plot('mass','rho',fname=3000)
whereas,
>>> pt.plot('mass','rho',fname=[3000,4000])
will produce two png files, one for each cycle, do help(m.plot) for a full list of options.
mppnp output allows to do plots with abundances:
>>> pt.iso_abund(23500)
>>> pt.abu_chart(23500)
Also for iso_abund and abu_chart: if instead of a single cycle the user inputs a list of cycles, the method will then, instead of plotting them, will then save a .png for each cycle. Also if you just want a singular plot saved, the user can input their cycle, in a list like [0]. And that will save their plot.
Functions
set_nice_params () |
|
set_nugrid_path (path) |
This function sets the path to the NuGrid VOSpace directory as a global variable, so that it need only be set once during an inter- active session. |
Classes
se ([sedir, pattern, rewrite, mass, Z, type, …]) |
This class provides easy access to h5 files from the NuGrid project, along with some standard plots. |
-
class
nugridpy.nugridse.
se
(sedir='.', pattern='.h5', rewrite=False, mass=None, Z=None, type='ppd_wind', output='out', code_source='MES', exp_type='delay', data_set='set1ext', verbose=False)¶ This class provides easy access to h5 files from the NuGrid project, along with some standard plots.
Parameters: - sedir (string, optional) – The directory on which the h5 files can be found. The default is ‘.’.
- pattern (string, optional) – A string pattern that the file names have to contain in order to be read. The default is “.h5”.
- rewrite (boolean, optional) – Would the user like to rewrite the preprocessor file. The default is False.
- mass (integer or float, optional) – The user may select a mass and metallicity instead of providing the sedir explicitly, if they are using the VOSpace data. If mass is provided then Z should also be provided. The default is None (i.e. user gives sedir explicitly)
- Z (float, optional) – See ‘mass’ above. The default is None (i.e. user gives sedir explicitly)
- type (string, optional) – If the user gives mass and metallicity instead of giving sedir explicitly, then the type of data should also be given here. For example, type=’ppd_wind’, ‘see_wind’, see_exp’ or ‘ppd_exp’ The default is ‘ppd_wind’.
- output (string, optional) – If the user gives mass and metallicity instead of giving sedir explicitly, then the type of data should also be given here. For example, out=’restart’, ‘out’ or ‘surf’ The default is ‘out’.
- data_set (string, optional) – Coose your data of ‘set1’ or ‘set1ext’. The default is ‘set1ext’.
- exp_type – If type is ‘see_exp’ or ‘ppd_exp’ runs of exp_type are selected. For example ‘delay’ and ‘rapid’ would be choices for Set1.
- verbose (boolean, optional) – If True, print more output
Examples
>>> f=nu.plot_tools('.','260')
reads all h5 files with the string 260 in the name in the present directory.
-
abund_at_masscoordinate
(ini, mass_coo, online=False)¶ Create a file with distribution at a given mass coord, and at a given time step.
This for instance may be used as intial distribution for function trajectory, to reproduce local conditions in ppn.
Parameters: - ini (integer) – Initial model, inital cycle number.
- mass_coo (float or 1x2 list) – Mass coordinate for the traj. If list, return mass-averaged abundances for the region spanned by the list.
- online (boolean, optional) – are you working online in the ipython notebook? If so, you will be given an HTML link to download the file.
-
abup_se_plot
(mod, species)¶ plot species from one ABUPP file and the se file.
You must use this function in the directory where the ABP files are and an ABUP file for model mod must exist.
Parameters: - mod (integer) – Model to plot, you need to have an ABUPP file for that model.
- species (string) – The species to plot.
Notes
The species is set to ‘C-12’.
-
average_iso_abund_marco
(mass_range, cycle, stable, i_decay)¶ Interface to average over mass_range.
Parameters: - mass_range (list) – A 1x2 array required to plot data in a certain mass range. Needed for _read_iso_abund_marco.
- cycle (integer) – which cycle from the h5 file?. Needed for _read_iso_abund_marco
- stable (boolean) – Do you want to plot only stable or not.
- i_decay (integer) – If i_decay is 1, then plot not decayed. If i_decay is 2, then plot decayed. Make sense only if stable is true.
See also
_read_iso_abund_marco()
-
burnstage
(**keyw)¶ This function calculates the presence of burning stages and outputs the ages when key isotopes are depleted and uses them to calculate burning lifetimes.
Parameters: keyw (dict) – A dict of key word arguments. Returns: A list containing the following information: burn_cycles, burn_ages, burn_abun, burn_type and burn_lifetime. Return type: list Notes
The following keywords can also be used:
Keyword Argument Default Value abund “iso_massf” isoa “A” isoz “Z” mass “mass” cycle “cycle” cyclefin 0 All arguments change the name of fields used to read data from HDF5 files, other than cyclefin. Cyclefin is the last timestep to use when reading files.
Cycles contain the cycle numbers for the various points where the abundance is abun. The age of the star at each point and the type of burning is indicated by those arrays. The lifetimes are calculated by.
-
burnstage_upgrade
(**keyw)¶ This function calculates the presence of burning stages and outputs the ages when key isotopes are depleted and uses them to calculate burning lifetimes.
Parameters: keyw (dict) – A dict of key word arguments. Returns: A list containing the following information: burn_cycles, burn_ages, burn_abun, burn_type and burn_lifetime. Return type: list Notes
The following keywords can also be used:
Keyword Argument Default Value abund “iso_massf” isoa “A” isoz “Z” mass “mass” cycle “cycle” cyclefin 0 All arguments change the name of fields used to read data from HDF5 files, other than cyclefin. Cyclefin is the last timestep to use when reading files.
Cycles contain the cycle numbers for the various points where the abundance is abun. The age of the star at each point and the type of burning is indicated by those arrays. The lifetimes are calculated by.
-
cores
(incycle, **keyw)¶ This function uses the abundances as a function of time to return core masses.
Parameters: - incycle (integer) – incycle is the cycle to choose where to take the core masses.
- keyw (dict) – A dict of key word arguments.
Returns: [cores, core_type, core_info] cores contains the core masses for the types found in core_type. Core_info stores a variety of information about the abundances at certain mass coordinates.
Return type: list
Notes
These core masses are:
M_alpha mass of the helium core M_CO mass of the carbon-oxygen core M_si mass of the silicon core M_fe mass of the iron core M_rem mass of the remnant star post-supernova The following keywords can also be used:
Keyword Argument Default Value abund “iso_massf” isoa “A” isom “isomeric_state” isoz “Z” mass “mass” core_opt 1 All arguments change the name of fields used to read data from HDF5 files, other than core_opt. core_opt controls which scheme to use for calculating the core masses.
If core_opt is 0, then cores uses alpha-isotopes to calculate the silicon and iron cores. Use this for stellar evolution output.
Si_core = Si28+S32+Ar36+Ca40+Ti44
Fe_core = Cr48+Fe52+Ni56
If core_opt is 1, then cores uses all elements in a network with proton number Z=23 as a boundary. Use this for MPPNP output.
Si_core = Sum of all isotopes with 14 <= Z <= 22
Fe_core = Sum of all isotopes with 23 <= Z <= 28
-
decay
(mass_frac)¶ this module simply calculate abundances of isotopes after decay.
It requires that before it is used a call is made to _read_iso_abund_marco and _stable_species.
Parameters: mass_frac (list) – alist of mass_frac dicts. See also
_read_iso_abund_marco()
,nuutils.Utils._stable_species()
-
ernst_table_exporter
(cycle, outfname='table_out', sheetname='Sheet 1')¶ This routine takes NuGrid data (model output) for a given cycle and writes it into an Excel sheet.
This is one format as requested by Ernst Zinner in June 2013 (through Marco). If you want all radioactive isotopes, start from the restart file. Empty columns are not written out and you will get a message how many were empty. Please note that only one cycle is written out.
Parameters: - cycle (integer) – Number of the cycle to consider.
- outfname (string, optional) – File name to write it to, .xlsx is appended automatically. The default is ‘table_out’.
- sheetname (string, optional) – Name of the sheet in the excel file. The default is ‘Sheet 1’.
-
get
(cycle_list, dataitem=None, isotope=None, sparse=1)¶ Simple function that simply calls h5T.py get method. There are three ways to call this function.
Parameters: - cycle_list (string, list) –
If cycle_list is a string, then get interpates the argument cycle_list as a dataitem and fetches the dataitem for all cycles.
If cycle_list is a list, then get fetches the dataitem for the cycles in the list.
- dataitem (string, optional) –
fetches the dataitem from the list of cycles. If dataitem is None, then cycle_list must be a string and will be used as dataitem. If dataitem is an isotope in the form ‘H-2’, it then returns the result of,
>>> self.get(cycle_list,'iso_massf',dataitem)
The default is None.
- isotope (string, optional) – The name of the isotope to fetch, it must be in the form ‘H-2’. If isotope is None, then cycle_list or dataitem must be a string. The default is None.
- sparse (integer, optional) – Implements a sparsity factor on the fetched data. The default is 1.
Notes
Calling the get method directly in the form,
>>> self.get(cycle_list,'iso_massf',dataitem)
is depricated, and only included for compatibility.
- cycle_list (string, list) –
-
get_abundance_elem
(cycle)¶ returns the undecayed element profile (all elements that are in elem_names).
Parameters: cycle (integer) – The cycle number
-
get_abundance_iso_decay
(cycle)¶ returns the decayed stable isotopes.
Parameters: cycle (integer) – The cycle.
-
get_decayed
(cycle_list, dataitem=None, isotope=None, sparse=1)¶ This function gives back the fully decayed isotope.
By default, it wants to beta-decays every isotope, however, it then checks if this is okay or not. If not it actually goes into a database (see routine below) to do the proper decay. This database might not be complete, so pay attention to the chart of the nuclides.
Standard input is as in regular get function.
Parameters: - cycle_list (string, list) –
If cycle_list is a string, then get interpates the argument cycle_list as a dataitem and fetches the dataitem for all cycles.
If cycle_list is a list, then get fetches the dataitem for the cycles in the list.
- dataitem (string, optional) –
fetches the dataitem from the list of cycles. If dataitem is None, then cycle_list must be a string and will be used as dataitem. If dataitem is an isotope in the form ‘H-2’, it then returns the result of,
>>> self.get(cycle_list,'iso_massf',dataitem)
The default is None.
- isotope (string, optional) – The name of the isotope to fetch, it must be in the form ‘H-2’. If isotope is None, then cycle_list or dataitem must be a string. The default is None.
- sparse (integer, optional) – Implements a sparsity factor on the fetched data. The default is 1.
Notes
Additional features are:
IN PROGRESS: CONTACT RETO<EMAIL_ADDRESS>the get method directly in the form,
>>> self.get_decayed(cycle_list,'iso_massf',dataitem)
is depricated, and only included for compatibility.
- cycle_list (string, list) –
-
get_elemental_abunds
(cycle, index=None)¶ returns the elemental abundances for one cycle, either for the whole star or a specific zone depending upon the value of ‘index’.
Parameters: - cycle (string or integer) – Model to get the abundances for.
- index (integer or list, optional) – zone number for which to get elemental abundances. If None the entire abundance profile is returned. If a 1x2 list, the abundances are returned between indices of index[0] and index[1]. The default is None.
-
kip_cont
(modstart, modstop, dcoeff_thresh=1000000000000.0, xres=1000, ylims=[0.0, 0.0], xlims=[0.0, 0.0], yres=2000, ixaxis='log_time_left', outfile='', landscape_plot=False, codev='KEP', kepswitch=12555, outlines=False, write_preproc=False, hatching=False)¶ This function creates a Kippenhahn diagram as a contour plot of the se output using a convection flag or diffusion coefficient threshold.
Parameters: - modstop (modstart,) – First and last cycle numbers to plot.
- dcoeff_thresh (float, optional) – Diffusion coefficient threshold, above which assume convection is present (only for MESA, where convection_indicator is not there)). The default is 1.e12.
- yres (xres,) – x and y resolution. The default for xres is 1000, the default for yres is 2000.
- ylims (xlims,) – x and y plot limits. The default is [0., 0.].
- ixaxis (string, optional) – Choose one of ‘age’, ‘model_number’ or ‘log_time_left’. The default is ‘log_time_left’.
- outfile (string, optional) – Name of output file including extension, which will be saved in 300 dpi. The default is ‘’.
- landscape_plot (boolean, optional) – For a landscape plot. The default is False.
- codev (string, optional) – Coose one of ‘KEP’, ‘MES’ or ‘GNV’. The default is ‘KEP’.
- kepswitch (integer, optional) – The default is 12555.
- outlines (boolean, optional) – The default is False.
- write_preproc (boolean, optional) – The default is False.
- hatching (boolean, optional) – The default if False.
Examples
>>> pt.kip_cont(0, -1, xres=1000, yres=1000, ixaxis='log_time_left')
Notes
Unfortunately, although the implicit do loops are faster, I cannot implement a % bar, but this example call should not take more than 1 minute.
-
plot4
(num)¶ Plots the abundances of H-1, He-4, C-12 and O-16.
-
plot4_nolog
(num)¶ Plots the abundances of H-1, He-4, C-12 and O-16.
-
plot_prof_1
(mod, species, xlim1, xlim2, ylim1, ylim2, symbol=None)¶ plot one species for cycle between xlim1 and xlim2
Parameters: - mod (string or integer) – Model to plot, same as cycle number.
- species (list) – Which species to plot.
- xlim2 (xlim1,) – Mass coordinate range.
- ylim2 (ylim1,) – Mass fraction coordinate range.
- symbol (string, optional) – Which symbol you want to use. If None symbol is set to ‘-‘. The default is None.
-
plot_prof_2
(mod, species, xlim1, xlim2)¶ Plot one species for cycle between xlim1 and xlim2
Parameters: - mod (string or integer) – Model to plot, same as cycle number.
- species (list) – Which species to plot.
- xlim2 (xlim1,) – Mass coordinate range.
-
plot_prof_sparse
(mod, species, xlim1, xlim2, ylim1, ylim2, sparse, symbol)¶ plot one species for cycle between xlim1 and xlim2.
Parameters: - species (list) – which species to plot.
- mod (string or integer) – Model (cycle) to plot.
- xlim2 (xlim1,) – Mass coordinate range.
- ylim2 (ylim1,) – Mass fraction coordinate range.
- sparse (integer) – Sparsity factor for points.
- symbol (string) – which symbol you want to use?
-
presnyields
(*cycles, **keyw)¶ This function calculates the presupernova yields of a full structure profile from a remnant mass, mrem, to the surface.
Parameters: - cycles (variadic tuple) – cycle[0] is the cycle to perform the presupernova yields calculations on. If cycle[1] is also specified, the yields are outputted using ‘initial’ abundances from cycle[1], otherwise the ejected masses are outputted.
- keyw (dict) – A dict of key word arguments.
Notes
The following keywords can be used:
Keyword Argument Default Value abund “iso_massf” xm “mass” mrem 0 abund and xm are used when the variables within the input file differ in their names. The default values are set to the output typically found in an MPPNP output file. For example, if the table for the abundances is called “abundances” instead of the default value, use abund = “abundances” as a keyword argument.
mrem is specified using a keyword argument and tells the program where to begin integrating.
-
trajectory
(ini, end, delta, mass_coo, age_in_sec=False, online=False)¶ create a trajectory out of a stellar model
Parameters: - ini (integer) – Initial model, inital cycle number.
- end (integer) – Final model, final cycle number.
- delta (integer) – Sparsity factor of the frames.
- mass_coo (float) – Mass coordinate for the traj.
- age_in_sec (boolean, optional) – Set to True if age in se file is in seconds (like in MESA). The default is False.
Returns: radius_at_mass_coo, density_at_mass_coo, temperature_at_mass_coo, age_all
Return type: float
Notes
plus writes a file with the trajectory information to be used with ppn.
Warning: remove the old trajectory, if you have any for the same mass coordinate. You are appending data, not overwriting.
Update: this method works for output types with indexes going from the outside in (MESA) or the other way around. Also the requested quantities are linearly interpolated in the mass shell. online: boolean, optional are you working online in the ipython notebook? If so, you will be given an HTML link to download the file.
-
windyields
(ini, end, delta, **keyw)¶ This function returns the wind yields and ejected masses.
X_i, E_i = data.windyields(ini, end, delta)
Parameters: - ini (integer) – The starting cycle.
- end (integer) – The finishing cycle.
- delta (integer) – The cycle interval.
- keyw (dict) – A dict of key word arguments.
Returns: The function returns a list of the wind yields(X_i) and a list of the ejected masses(E_i) in the mass units that were used (usually solar masses).
Return type: list
Notes
The following keywords cand also be used:
Keyword Argument Default Value abund “iso_massf” tmass “mass” cycle “cycle” The keyword arguments are used when the variables within the input file differ in name from their default values typically found in an MPPNP output file. If the data table differs in name, use these keywords. For example, if the table for the abundances is called “abundances” instead of “iso_massf”, then use abund = “abundances” as a keyword argument.
-
write_deltatable
(filename='default', decayed=True, dcycle=500, iniabufile='../../frames/mppnp/USEEPP/iniab2.0E-02GN93.ppn')¶ This subroutine is to write out tables with delta values for cosmochemists to use in comparison with their data.
No options are necessarily needed to load this routine, however, it might be useful to specify a filename. This file furthermore searches for thermal pulses, hence, it’s only useful for non-explosive TP-AGB stars!
Parameters: - filename (string, optional) – Choose the filename you want. The default is “default”.
- decayed (boolean, optional) – Value if decayed massfractions of isotope should be taken or not. The default is False.
- dcycle (integer, optional) – Difference between cycles to search for thermal pulses. The default is 500.
- iniabufile (string, optional) – File with initial abundances. As a standard value, the GN93 file is used (in USEEPP folder). Important, input file has to be USEEPP conforming, see NuGrid book! The default is ‘../../frames/mppnp/USEEPP/iniab2.0E-02GN93.ppn’.
Notes
Regardless of what value is used as filename it will be replaced with ‘delta_outfile.txt’.
-
nugridpy.nugridse.
set_nugrid_path
(path)¶ This function sets the path to the NuGrid VOSpace directory as a global variable, so that it need only be set once during an inter- active session.


Documentation for classes includes syntax, usage information, and code samples for methods, properties, and event handlers and listeners for those APIs that belong to a specific class in ActionScript. The classes are listed alphabetically. If you are not sure to which class a certain method or property belongs, you can look it up in the Index.
| Class | Package | Description | |
|---|---|---|---|
| AgglomerativeCluster
| flare.analytics.cluster | Hierarchically clusters a set of items using agglomerative clustering. | |
| AggregateExpression
| flare.query | Base class representing an aggregate query operator. | |
| AlphanumSort
| org.juicekit.util.collection | The AlphanumSort class implements the natural sorting algorithm outlined at http://www.davekoelle.com/alphanum.html. | |
| AnchorControl
| flare.vis.controls | Interactive control for updating a layout's anchor point in response to mouse movement. | |
| And
| flare.query | Expression operator that computes the logical "and" of sub-expression clauses. | |
| AnimateColor
| org.juicekit.effects | ||
| AnimateColorInstance
| org.juicekit.effects.effectClasses | ||
| Arithmetic
| flare.query | Expression operator for arithmetic operations. | |
| ArrayInterpolator
| flare.animate.interpolate |
Interpolator for numeric Array values. |
|
| Arrays
| flare.util | Utility methods for working with arrays. | |
| ArrowType
| flare.vis.data.render | Utility class defining arrow types for directed edges. | |
| AspectRatioBanker
| flare.analytics.optimization | Computes an optimized aspect ratio for drawing a line chart. | |
| AutoDataGrid
| Top Level | ||
| AutoTextField
| org.juicekit.visual.controls | The AutoTextField class supports directly editing a static-like text field. | |
| AutoTextFieldEvent
| org.juicekit.events |
The AutoTextFieldEvent class defines an event to signal a
|
|
| Average
| flare.query | Aggregate operator for computing the average of a set of values. | |
| Axes
| flare.vis.axis | Base class for representing metric data axes. | |
| Axis
| flare.vis.axis | A metric data axis consisting of axis labels and gridlines. | |
| AxisGridLine
| flare.vis.axis | Axis grid line in an axis display. | |
| AxisLabel
| flare.vis.axis | Axis label in an axis display. | |
| AxisLayout
| flare.vis.operator.layout | Layout that places items along the X and Y axes according to data properties. | |
| BetweennessCentrality
| flare.analytics.graph | Calculates betweenness centrality measures for nodes in a graph. | |
| BifocalDistortion
| flare.vis.operator.distortion | Computes a bifocal distortion of space, magnifying a focus region of space and uniformly demagnifying the rest of the space. | |
| BinaryExpression
| flare.query | Base class for binary expression operators. | |
| Browser
| org.juicekit.util.helper | The Browser class provides a set of functions that provide a bridge between the Flash Player hosted application and the containing browser. | |
| BundledEdgeRouter
| flare.vis.operator.layout | Layout that routes edges in a graph so that they form groups, reducing clutter. | |
| CartesianAxes
| flare.vis.axis | Axes class representing 2D Cartesian (X-Y) axes. | |
| CircleLayout
| flare.vis.operator.layout | Layout that places items in a circular layout. | |
| CirclePackingLayout
| flare.vis.operator.layout | Layout that places nodes as circles compacted into a larger circle. | |
| ClickControl
| flare.vis.controls | Interactive control for responding to mouse clicks events. | |
| Clipboard
| org.juicekit.util.helper | The Clipboard class produces string representations of value objects. | |
| ColorEncoder
| flare.vis.operator.encoder | Encodes a data field into color values, using a scale transform and color palette. | |
| ColorInterpolator
| flare.animate.interpolate |
Interpolator for color (uint ) values. |
|
| ColorPalette
| flare.util.palette | Palette for color values, including utility methods for generating both categorical and ordinal color palettes. | |
| ColorPalette
| org.juicekit.flare.util.palette | ||
| Colors
| org.juicekit.flare.util | This class supplies utility methods for working with colors. | |
| Colors
| flare.util | Utility methods for working with colors. | |
| CommunityStructure
| flare.analytics.cluster | Hierarchically clusters a network based on inferred community structure. | |
| Comparison
| flare.query | Expression operator for comparing sub-expression values. | |
| CompositeExpression
| flare.query | Base class for expressions with an arbitrary number of sub-expressions. | |
| Control
| flare.vis.controls | Base class for interactive controls. | |
| ControlList
| flare.vis.controls | A ControlList maintains a sequential chain of controls for interacting with a visualization. | |
| Converters
| flare.data.converters | Factory class for looking up the appropriate IDataConverter for a given data format. | |
| Count
| flare.query | Aggregate operator for counting the number of items in a set of values. | |
| CSSUtil
| org.juicekit.util.helper | The CSSUtil class provides a set of static helper functions for manipulating CSS styles. | |
| Data
| flare.vis.data | Data structure for managing a collection of visual data objects. | |
| DataEvent
| flare.vis.events | Event fired when a data collection is modified. | |
| DataField
| flare.data | A data field stores metadata for an individual data variable. | |
| DataList
| flare.vis.data |
Data structure for a collection of DataSprite instances. |
|
| DataMouseEvent
| org.juicekit.events | The DataMouseEvent class defines an event to signal a mouse event occurred on/over a visual representation of a data object. | |
| DataSchema
| flare.data | A data schema represents a set of data variables and their associated types. | |
| DataSet
| flare.data | A data set is a collection of data tables. | |
| DataSource
| flare.data | A data source provides access to remote data on the Internet. | |
| DataSprite
| flare.vis.data | Base class for display objects that represent visualized data. | |
| DataTable
| flare.data | A data table maintains a collection of data objects, each representing a row of data, and an optional data schema describing the data variables. | |
| DataUtil
| flare.data | Utility class for parsing and representing data field values. | |
| DateInterpolator
| flare.animate.interpolate |
Interpolator for Date values. |
|
| Dates
| flare.util | Utility methods for working with Date instances. | |
| DateUtil
| flare.query |
Utility class providing functions for manipulating Date
objects. |
|
| DelimitedTextConverter
| flare.data.converters | Converts data between delimited text (e.g., tab delimited) and flare DataSet instances. | |
| DendrogramLayout
| flare.vis.operator.layout | Layout that places items in a dendrogram for displaying the results of hierarchical clustering. | |
| DenseMatrix
| flare.util.math | A matrix of numbers implemented using an array of values. | |
| DirtySprite
| flare.display | A Sprite that redraws itself as needed when marked as "dirty". | |
| Displays
| flare.util | Utility methods for working with display objects. | |
| Distinct
| flare.query | Aggregate (group-by) operator for counting the number of distinct values in a set of values. | |
| Distortion
| flare.vis.operator.distortion | Base class for distortion operators that adjust the position and size of visual objects in a visualization. | |
| DMAMapControl
| org.juicekit.visual.controls | The DMAMapControl class visualizes Nielson Designated Market Areas (DMA). | |
| DragControl
| flare.vis.controls | Interactive control for dragging items. | |
| DragForce
| flare.physics | Force simulating frictional drag forces (e.g., air resistance). | |
| Easing
| flare.animate | Collection of easing functions to control animation rates. | |
| EdgeRenderer
| flare.vis.data.render | Renderer that draws edges as lines. | |
| EdgeSprite
| flare.vis.data | Visually represents a connection between two data elements. | |
| Encoder
| flare.vis.operator.encoder | Base class for Operators that perform encoding of visual variables such as color, shape, and size. | |
| ExpandControl
| flare.vis.controls | Interactive control for expaning and collapsing graph or tree nodes by clicking them. | |
| Expression
| flare.query | Base class for query expression operators. | |
| ExpressionIterator
| flare.query | The ExpressionIterator simplifies the process of traversing an expression tree. | |
| FibonacciHeap
| flare.util.heap | A Fibonacci heap data structure for maintaining a sorted priority queue. | |
| FilledRectValueRenderer
| org.juicekit.visual.renderers |
The FilledRectValueRenderer class renderers numeric data
filling up to rectCount number of rectangles. |
|
| Filter
| flare.util | Utility methods for creating filter functions. | |
| FisheyeDistortion
| flare.vis.operator.distortion | Computes a graphical fisheye distortion of a graph view. | |
| FisheyeTreeFilter
| flare.vis.operator.filter | Filter operator that computes a fisheye degree-of-interest function over a tree structure. | |
| FlareControlBase
| org.juicekit.visual.controls |
The class FlareControlBase provides a common implementation
for visual controls based upon the prefure.flare Visualization . |
|
| FlareVis
| flare.flex | Flex component that wraps a Flare visualization instance. | |
| Fn
| flare.query | Expression operator that performs function invocation. | |
| ForceDirectedLayout
| flare.vis.operator.layout | Layout that places nodes based on a physics simulation of interacting forces. | |
| Formatter
| org.juicekit.util.helper | The Formatter class methods produce labeling functions for use as values for a DataGridColumn labelFunction property. | |
| FunctionSequence
| flare.animate | Transition that runs sub-transitions in sequential order while also invoking a function before each sub-transition is run. | |
| Geometry
| flare.util | Utility methods for computational geometry. | |
| GraphDistanceFilter
| flare.vis.operator.filter | Filter operator that sets visible all items within a specified graph distance from a set of focus nodes. | |
| GraphicsUtil
| org.juicekit.util.helper | The GraphicsUtil class provides functions to assist development of visual components that rely on manipulating the Flash Player's scene graph and graphics coontext. | |
| GraphMLConverter
| flare.data.converters | Converts data between GraphML markup and flare DataSet instances. | |
| GraphUtil
| org.juicekit.util.data | Utilities for generating sample data and Graph structures from data. | |
| GravityForce
| flare.physics | Force simulating a global gravitational pull on Particle instances. | |
| HBarRenderer
| org.juicekit.visual.renderers |
The HBarRenderer class draws a text label and a horizontal bar using
the numeric value assigned to this renderer's data property. |
|
| HeapNode
| flare.util.heap | A node in a heap data structure. | |
| HierarchicalCluster
| flare.analytics.cluster | Base class for clustering operators that construct a hierarchical clustering tree. | |
| HLabeledGradient3
| Top Level | ||
| HoverControl
| flare.vis.controls | Interactive control for responding to mouse hover events. | |
| HPercentRenderer
| org.juicekit.visual.renderers | The HPercentRenderer class draws a text label and a horizontal rectangle split into two halves. | |
| IcicleTreeLayout
| flare.vis.operator.layout | Layout that places nodes in an icicle layout, distributing nodes evenly within the display bounds. | |
| IControl
| flare.vis.controls | ||
| IDataConverter
| flare.data.converters | Interface for data converters that map between an external data file format and ActionScript objects (e.g., Arrays and Objects). | |
| IEvaluable
| flare.util | Interface for methods that evaluate an object and return a result. | |
| If
| flare.query | Expression operator for an if statement that performs conditional execution. | |
| IForce
| flare.physics | Interface representing a force within a physics simulation. | |
| ILabelFormatter
| org.juicekit.flare.vis.label | Inteface for LabelFormatters used by a LabelLayout. | |
| IMatrix
| flare.util.math | Interface for a matrix of real-valued numbers. | |
| InactivityEvent
| org.juicekit.events | The InactivityEvent class defines an event to signal a period of inactivity occurred. | |
| InactivityMonitor
| org.juicekit.events |
The InactivityMonitor class monitors other event dispatchers
for keyboard or mouse events in order to signal periods of
inactivity to listeners of jkInactivityPeriod . |
|
| IndentedTreeLayout
| flare.vis.operator.layout | Layout that places tree nodes in an indented outline layout. | |
| IndexCircleRenderer
| org.juicekit.visual.renderers |
The IndexCircleRenderer class draws an indexing circle and label based upon
the numeric value assigned to this renderer's data property. |
|
| Interpolator
| flare.animate.interpolate | Base class for value interpolators. | |
| IOperator
| flare.vis.operator | Interface for operators that perform processing tasks on the contents of a Visualization. | |
| IPredicate
| flare.util | Interface for methods that evaluate an object and return true or false. | |
| IRenderer
| flare.vis.data.render | Interface for DataSprite rendering modules. | |
| IsA
| flare.query | Expression operator that type checks a sub-expression. | |
| IScaleMap
| flare.scale | Interface used by classes which support mapping between spatial (x,y) coordinates and values in a data scale. | |
| ISchedulable
| flare.animate | Interface for "schedulable" objects that can be run by the Scheduler class. | |
| IValueProxy
| flare.util | Interface for classes that get and set named property values of objects. | |
| JSONConverter
| flare.data.converters | Converts data between JSON (JavaScript Object Notation) strings and flare DataSet instances. | |
| JuiceKitEvent
| org.juicekit.events | The JuiceKitEvent class defines the event type string constants for many of the JuiceKit event types, especially custom data binding events. | |
| Labeler
| flare.vis.operator.label | Labeler that adds labels for items in a visualization. | |
| LabelFormat
| org.juicekit.flare.vis.label | The LabelFormat class describes the style attributes to be applied to text labels of a LabelLayout instance. | |
| LabelFormatter
| org.juicekit.flare.vis.label | Implements the ILabelFormatter interface to apply one format to all label instances in a LabelLayout. | |
| Labels
| org.juicekit.flare.vis.label | A LabelLayout places text labels using a DataSprite data property for the text value and DataSprite x and y coordinates to position the text rendering. | |
| Layout
| flare.vis.operator.layout | Base class for all operators that perform spatial layout. | |
| Legend
| flare.vis.legend | A legend describing the visual encoding of a data property. | |
| LegendItem
| flare.vis.legend | An item in a discrete legend consisting of a label and an icon indicating color, shape, and/or size. | |
| LegendRange
| flare.vis.legend | A range in a continuous legend, consisting of a continuous visual scale and value labels. | |
| LinearScale
| flare.scale | Scale that spaces values linearly along the scale range. | |
| LineSprite
| flare.display | A Sprite representing a line. | |
| LinkDistance
| flare.analytics.graph | Calculates the link distance from a source node based on a breadth-first traversal. | |
| Literal
| flare.query | Expression operator for a literal value. | |
| LiveQuery
| org.juicekit.util.data | ||
| LogScale
| flare.scale | Scale that performs a log transformation of the data. | |
| Match
| flare.query | Expression operator for text matching operations. | |
| Maths
| flare.util |
Utility methods for mathematics not found in the Math class. |
|
| MatrixInterpolator
| flare.animate.interpolate |
Interpolator for flash.geom.Matrix values. |
|
| MaxDataGridColumn
| org.juicekit.visual.renderers | The class MaxDataGridColumn provides a maximum value that is used by the HBarRenderer to convert absolute value bars into percentage of maximum value bars. | |
| MaxFlowMinCut
| flare.analytics.graph | Calculates the maximum flow along edges of a graph using the Edmonds-Karp method. | |
| Maximum
| flare.query | Aggregate operator for computing the maximum of a set of values. | |
| Minimum
| flare.query | Aggregate operator for computing the minimum of a set of values. | |
| ModelObject
| org.juicekit.util.model | ||
| MovieClipControlBase
| org.juicekit.visual.controls |
The MovieClipControlBase class provides a wrapper for a Flash
MovieClip that contains SimpleButton children. |
|
| NBodyForce
| flare.physics | Force simulating an N-Body force of charged particles with pairwise interaction, such as gravity or electrical charge. | |
| NodeLinkTreeLayout
| flare.vis.operator.layout | Layout that places nodes using a tidy layout of a node-link tree diagram. | |
| NodeSprite
| flare.vis.data | Visually represents a data element, such as a data tuple or graph node. | |
| Not
| flare.query | Expression operator that returns the logical "not" of a sub-expression. | |
| NullOnAggregate
| flare.query | Aggregate operator for aggregating strings or nonnumberic objects. | |
| NumberInterpolator
| flare.animate.interpolate |
Interpolator for Number and int values. |
|
| NumberRendererBase
| org.juicekit.visual.renderers | The class NumberRendererBase provides a common implementation for custom ActionScript renderers that draw numeric data values. | |
| ObjectInterpolator
| flare.animate.interpolate |
Interpolator for arbitrary Object values. |
|
| Operator
| flare.vis.operator | Operators performs processing tasks on the contents of a Visualization. | |
| OperatorList
| flare.vis.operator | An OperatorList maintains a sequential chain of operators that are invoked one after the other. | |
| OperatorSequence
| flare.vis.operator | An OperatorSequence maintains a list of operators that are run in sequence with an animated transition in between each. | |
| OperatorSwitch
| flare.vis.operator | An OperatorSwitch maintains a list of operators but only runs one of these operators when invoked. | |
| Or
| flare.query | Expression operator that computes the logical "or" of sub-expression clauses. | |
| OrdinalScale
| flare.scale | Scale for ordered sequential data. | |
| Orientation
| flare.util | Constants defining layout orientations. | |
| Palette
| flare.util.palette | Base class for palettes, such as color and size palettes, that map from interpolated scale values into visual properties | |
| PanZoomControl
| flare.vis.controls | Interactive control for panning and zooming a "camera". | |
| Parallel
| flare.animate | Transition that runs multiple transitions simultaneously (in parallel). | |
| Particle
| flare.physics | Represents a Particle in a physics simulation. | |
| Pause
| flare.animate | Transition representing a pause or dwell in which nothing happens. | |
| PieLayout
| flare.vis.operator.layout | Layout that places wedges for pie and donut charts. | |
| PointInterpolator
| flare.animate.interpolate |
Interpolator for flash.geom.Point values. |
|
| Property
| flare.util |
Utility class for accessing arbitrary property chains, allowing
nested property expressions (e.g., x.a.b.c or
x.a[1] ). |
|
| PropertyEncoder
| flare.vis.operator.encoder | Encodes property values for a collection of visual items. | |
| QuantileScale
| flare.scale | Scale that organizes data into discrete bins by quantiles. | |
| QuantitativeScale
| flare.scale | Base class for representing quantitative numerical data scales. | |
| Query
| flare.query | Performs query processing over a collection of ActionScript objects. | |
| RadialLabeler
| flare.vis.operator.label | Labeler that positions labels around a circle in polar coordinates. | |
| RadialTreeLayout
| flare.vis.operator.layout | Layout that places tree nodes in a radial layout, laying out depths of a tree along circles of increasing radius. | |
| RandomLayout
| flare.vis.operator.layout | Layout that places nodes randomly within the layout bounds. | |
| Range
| flare.query | Expression operator that tests if a value is within a given range. | |
| RectangleInterpolator
| flare.animate.interpolate |
Interpolator for flash.geom.Rectangle values. |
|
| RectSprite
| flare.display | A Sprite representing a rectangle shape. | |
| RendererBase
| org.juicekit.visual.renderers | The class RendererBase provides a common implementation of IDataRenderer, IListItemRenderer, and IDropInListItemRenderer for custom ActionScript renderers. | |
| RootScale
| flare.scale | Scale that performs a root transformation of the data. | |
| Scale
| flare.scale | Base class for all data scale types. | |
| ScaleBinding
| flare.vis.data | Utility class that binds a data property to a descriptive scale. | |
| ScaleType
| flare.scale | Constants defining known scale types, such as linear, log, and date/time scales. | |
| Scheduler
| flare.animate | Scheduler that oversees animation and time-based processing. | |
| SelectionControl
| flare.vis.controls | Interactive control for selecting a group of objects by "rubber-banding" them with a rectangular section region. | |
| SelectionEvent
| flare.vis.events | Event fired in response to interactive selection events. | |
| Sequence
| flare.animate | Transition that runs multiple transitions one after the other in sequence. | |
| ShapeEncoder
| flare.vis.operator.encoder | Encodes a data field into shape values, using an ordinal scale. | |
| ShapePalette
| flare.util.palette | Palette for shape values that maps integer indices to shape drawing functions. | |
| ShapeRenderer
| flare.vis.data.render | Renderer that draws shapes. | |
| Shapes
| flare.util | Utility class defining shape types and shape drawing routines. | |
| ShortestPaths
| flare.analytics.graph | Calculates the shortest paths to a source node using Dijkstra's algorithm. | |
| Simulation
| flare.physics | A physical simulation involving particles, springs, and forces. | |
| SizeEncoder
| flare.vis.operator.encoder | Encodes a data field into size values, using a scale transform and a size palette to determines an item's scale. | |
| SizePalette
| flare.util.palette | Palette for size values represeneted as scale factors. | |
| Sort
| flare.util | Utility class for sorting and creating sorting functions. | |
| SortOperator
| flare.vis.operator | A SortOperator sorts a data group. | |
| SpanningTree
| flare.analytics.graph | Calculates a spanning tree for a graph structure. | |
| SparseMatrix
| flare.util.math | A matrix of numbers implemented using a hashtable of values. | |
| Spring
| flare.physics | Represents a Spring in a physics simulation. | |
| SpringForce
| flare.physics | Force simulating a spring force between two particles. | |
| StackedAreaLabeler
| flare.vis.operator.label | Labeler for a stacked area chart. | |
| StackedAreaLayout
| flare.vis.operator.layout | Layout that consecutively places items on top of each other. | |
| Stats
| flare.util | Utility class for computing statistics for a collection of values. | |
| Strings
| flare.util | Utility methods for working with String instances. | |
| StringUtil
| flare.query |
Utility class providing functions for manipulating String
objects. |
|
| StringUtil
| org.juicekit.util.helper | The StringUtil class provides String utility functions. | |
| Sum
| flare.query | Aggregate operator for computing the sum of a set of values. | |
| TextSprite
| flare.display | A Sprite representing a text label. | |
| TimeScale
| flare.scale |
Scale for timelines represented using Date values. |
|
| TooltipControl
| flare.vis.controls | Interactive control for displaying a tooltip in response to mouse hovers exceeding a minimum time interval. | |
| TooltipEvent
| flare.vis.events | Event fired in response to tooltip show, hide, or update events. | |
| Transition
| flare.animate | Base class representing an animated transition. | |
| Transitioner
| flare.animate | Parallel transition with convenience methods for adding new object tweens, helping to incrementally construct a group of transitions. | |
| TransitionEvent
| flare.animate |
Event fired when a Transition
starts, steps, ends, or is canceled. |
|
| Tree
| flare.vis.data | Data structure for managing a collection of visual data objects in a tree (hierarchy) structure. | |
| TreeBuilder
| flare.vis.data | Calculates a spanning tree for a graph structure. | |
| TreeMapControl
| org.juicekit.visual.controls | The TreeMapControl class visualizes large hierarchical data sets. | |
| TreeMapLayout
| flare.vis.operator.layout | Layout that places node in a TreeMap layout that optimizes for low aspect ratios of visualized tree nodes. | |
| TreeMapZoomControl
| org.juicekit.visual.controls |
The TreeMapZoomControl class provides a zoom-in/out
companion control for the TreeMapControl . |
|
| TreeMLReader
| org.juicekit.flare.vis.data | Converts TreeML markup into flare Tree instances. | |
| Tween
| flare.animate | Transition that interpolates (in-betweens) properties of a target object over a time interval. | |
| USMapControl
| org.juicekit.visual.controls | The USMap class visualizes U.S. | |
| Variable
| flare.query | Expression operator that retrieves a value from an object property. | |
| Variance
| flare.query | Aggregate operator for computing variance or standard deviation. | |
| VisibilityFilter
| flare.vis.operator.filter | Filter operator that sets item visibility based on a filtering condition. | |
| Visualization
| flare.vis | The Visualization class represents an interactive data visualization. | |
| VisualizationEvent
| flare.vis.events | Event fired in response to visualization updates. | |
| WeightedAverage
| flare.query | Aggregate operator for computing the weighted average of a set of values. | |
| Xor
| flare.query | Expression operator that computes the exclusive-or ("xor") of sub-expression clauses. |


We created a database of hundreds of 2d layered materials and 1d
molecular
wires.
Click here to download the database!
Layered materials have vast potential in applications, both in the bulk form and in a single layer. Bulk layered materials have been widely used for various technologies, for example in energy storage; most commercial Li-ion batteries today use layered materials such as graphite or layered metal oxides to store lithium. In a single layer form, they provide advantages such as maximum mechanical flexibility and optical transparency. Applications in nanoscale devices are numerous, and atomically thin transistors and resonators have been demonstrated. Even though there is a lot of research interest in layered two-dimensional materials, only a few of them are being studied. Graphene, BN, transition metal dichalcogenides and some others like phosphorene dominate the research on 2D materials.
Examples of 2d layered (left) and 1d molecular wire (right)
structures that our data mining algorithm discovered.
In this work, we use data-mining to screen almost 60,000 materials from the Materias Project database to identify discover 1173 2D layered materials, 487 1D materials, 98 lattice-commensurate heterostructures and 325 materials with piezoelectric monolayers. To do this, we developed a novel algorithm that can determine the dimensionalisy of the strongly bonded subunits in the crystal. We find bonds between atoms by comparing interatomic distances between all pairs of atoms in a unit cell with the sum of their bond radii. If the distances are smaller than the sum of bond radii + tolerance of 0.45 angstroms (to account for bond length variability), we consider the pair bonded. We find all connected clusters of atoms in the unit cell, and do the same for a 2x2x2 supercell of the unit cell. Now, we count the number of atoms in the connected clusters in the original unit cell and the 2x2x2 supercell. If the connected cluster is two-dimensional, the number of atoms in a cluster would have increased to 4 times the number in the original unit cell, because the size of a two-dimensional object scales as length^2.
Using this algorithm, we identified 1173 2D weakly bonded solids and 487 1D weakly bonded solids with unique chemical formulas, consisting of 23 distinct chemical families of 2D weakly bonded solids and eight chemical families of 1D weakly bonded solids. While layered materials have received considerable attention, much less was known about bulk inorganic crystals that consist of weakly bonded one-dimensional chains, or molecular wires. Less than 20 such materials were found in literature, and this work is the first to uncover a variety of 1D weakly bonded solids. The majority of weakly bonded solids identified in this work come from experimentally reported crystals composed of stacks of 2D layers or 1D chains of bonded atoms.
We also identify 98 weakly bonded lattice-commensurate heterostructures, i.e. materials with weakly bonded adjacent units with dissimilar properties. There has been much activity on assembling layers of different materials to make van der Waals heterostructures with new material properties3,4. However, the fabrication of such heterostructures present a significant experimental challenge. The materials identified in our work have the substantial manufacturing advantage of being amenable to growth using more conventional and scalable methods.
We also present data on the band gaps of the 2D and 1D materials, as well as the point groups. Crystal symmetries of bulk and monolayer 2D materials can be different for many layered materials, and the physical properties affected by them are also different. For example, TMDs in the 2H phase with inversion symmetry in the bulk form are not piezoelectric, but single layers of TMDs are piezoelectric. We developed a method to calculate monolayer point group symmetries from AA-stacking of the monolayers. Combining this with the band gap data, we identify 2D piezoelectric material candidates from point group symmetries.
Publication:
Cheon, G., Duerloo, K.-A. N., Sendek, A. D., Porter, C., Chen, Y., Reed, E. J., Data Mining for New Two- and One-dimensional Weakly Bonded Solids and Lattice-commensurate Heterostructures. Nano Letters, doi:10.1021/acs.nanolett.6b05229 (2017).


UNSHARE
Section: Linux Programmer's Manual (2)
Updated: 2013-04-17
Index
Return to Main Contents
NAME
unshare - disassociate parts of the process execution context
SYNOPSIS
#include <sched.h>
int unshare(int flags);
Feature Test Macro Requirements for glibc (see
feature_test_macros(7)):
unshare():
-
- Since glibc 2.14:
-
_GNU_SOURCE
-
-
Before glibc 2.14:
_BSD_SOURCE || _SVID_SOURCE
/* _GNU_SOURCE also suffices */
DESCRIPTION
unshare()
allows a process to disassociate parts of its execution
context that are currently being shared with other processes.
Part of the execution context, such as the mount namespace, is shared
implicitly when a new process is created using
fork(2)
or
vfork(2),
while other parts, such as virtual memory, may be
shared by explicit request when creating a process using
clone(2).
The main use of
unshare()
is to allow a process to control its
shared execution context without creating a new process.
The
flags
argument is a bit mask that specifies which parts of
the execution context should be unshared.
This argument is specified by ORing together zero or more
of the following constants:
- CLONE_FILES
-
Reverse the effect of the
clone(2)
CLONE_FILES
flag.
Unshare the file descriptor table, so that the calling process
no longer shares its file descriptors with any other process.
- CLONE_FS
-
Reverse the effect of the
clone(2)
CLONE_FS
flag.
Unshare filesystem attributes, so that the calling process
no longer shares its root directory
(chroot(2)),
current directory
(chdir(2)),
or umask
(umask(2))
attributes with any other process.
- CLONE_NEWIPC (since Linux 2.6.19)
-
This flag has the same effect as the
clone(2)
CLONE_NEWIPC
flag.
Unshare the System V IPC namespace,
so that the calling process has a private copy of the
System V IPC namespace which is not shared with any other process.
Specifying this flag automatically implies
CLONE_SYSVSEM
as well.
Use of
CLONE_NEWIPC
requires the
CAP_SYS_ADMIN
capability.
- CLONE_NEWNET (since Linux 2.6.24)
-
This flag has the same effect as the
clone(2)
CLONE_NEWNET
flag.
Unshare the network namespace,
so that the calling process is moved into a
new network namespace which is not shared
with any previously existing process.
Use of
CLONE_NEWNET
requires the
CAP_SYS_ADMIN
capability.
- CLONE_NEWNS
-
This flag has the same effect as the
clone(2)
CLONE_NEWNS
flag.
Unshare the mount namespace,
so that the calling process has a private copy of
its namespace which is not shared with any other process.
Specifying this flag automatically implies
CLONE_FS
as well.
Use of
CLONE_NEWNS
requires the
CAP_SYS_ADMIN
capability.
- CLONE_NEWUTS (since Linux 2.6.19)
-
This flag has the same effect as the
clone(2)
CLONE_NEWUTS
flag.
Unshare the UTS IPC namespace,
so that the calling process has a private copy of the
UTS namespace which is not shared with any other process.
Use of
CLONE_NEWUTS
requires the
CAP_SYS_ADMIN
capability.
- CLONE_SYSVSEM (since Linux 2.6.26)
-
This flag reverses the effect of the
clone(2)
CLONE_SYSVSEM
flag.
Unshare System V semaphore undo values,
so that the calling process has a private copy
which is not shared with any other process.
Use of
CLONE_SYSVSEM
requires the
CAP_SYS_ADMIN
capability.
If
flags
is specified as zero, then
unshare()
is a no-op;
no changes are made to the calling process's execution context.
RETURN VALUE
On success, zero returned.
On failure, -1 is returned and
errno
is set to indicate the error.
ERRORS
- EINVAL
-
An invalid bit was specified in
flags.
- ENOMEM
-
Cannot allocate sufficient memory to copy parts of caller's
context that need to be unshared.
- EPERM
-
The calling process did not have the required privileges for this operation.
VERSIONS
The
unshare()
system call was added to Linux in kernel 2.6.16.
CONFORMING TO
The
unshare()
system call is Linux-specific.
NOTES
Not all of the process attributes that can be shared when
a new process is created using
clone(2)
can be unshared using
unshare().
In particular, as at kernel 3.8,
unshare()
does not implement flags that reverse the effects of
CLONE_SIGHAND,
CLONE_THREAD,
or
CLONE_VM.
Such functionality may be added in the future, if required.
SEE ALSO
clone(2),
fork(2),
kcmp(2),
setns(2),
vfork(2)
Documentation/unshare.txt
in the Linux kernel source tree
Index
- NAME
-
- SYNOPSIS
-
- DESCRIPTION
-
- RETURN VALUE
-
- ERRORS
-
- VERSIONS
-
- CONFORMING TO
-
- NOTES
-
- SEE ALSO
-
This document was created by
man2html,
using the manual pages.
Time: 02:54:46 GMT, September 18, 2014


The NetBeans Runtime Inspector facility lets you look at the structure of objects live in the IDE, provided they can be represented as JavaBeans (which many API-related objects can). To use the inspector, Flip to the Runtime tab in the Explorer and look under the NetBeans Runtime node - you will see some important folders in the system filesystem. You will also see the lookup helper.
In the inspector, by default nodes just show up as themselves, but they will show extra children. First of all, their regular children are displayed in a special subnode, to separate them from other things. Other subnodes may be displayed, including:
BeanInfo
is simply ignored, and all getter methods are treated as fair game--even if they
are not public methods. Useful for classes which have BeanInfo
which
does not include all the information you might to debug with.
InstanceCookie
will show that instance as a Bean.
Node
properties will appear as themselves, after an introduction.
Container
properties (i.e. visual beans) will show the AWT
component hierarchy.
FileSystem
properties additionally display the root folder.
This lets you browse to a specific file in the filesystem.
FileObject
properties additionally display the data object
(from which you can also see the node delegate). And you get a list of file
attributes by name.
File objects which are part of a MultiFileSystem
will show the
original filesystem which produced them. In the case of file objects in the
system filesystem, you can see the provenance of the file, indicating
whether it was defined in a module (including the core - generally some XML
layer); the user directory (global setting); the installation directory (rarer);
or the current project.
Clipboard
objects show the objects associated
with each currently supported data flavor in their
transferables. Go to java.awt.Clipboard
under Lookup results for
example. Also ExClipboard
objects will update this
information when it changes, and MultiTransferObject
objects are broken apart into constituent flavors.
Lookup.Provider
method (such as nodes, data objects, and so on)
will be examined for a list of all standard cookie types, and these cookies will be shown
in a separate sublist for examination.
(Details...)
It is possible to recognize your own cookie types, too;
if your module defines a cookie you wish to see listed, simply add an XML file to its layer
underneath Services/Hidden/ matching the DTD at
http://www.netbeans.org/dtds/apisupport-cookie-class-list-1_0.dtd.
FilterNode
s, you can see
the original node.