NOTE: Hadoop’s s3: and s3n: connectors have been removed. Please use s3a: as the connector to data hosted in S3 with Apache Hadoop.
Consult the s3n documentation for migration instructions.
See also:
Apache Hadoop’s hadoop-aws module provides support for AWS integration. applications to easily use this support.
To include the S3A client in Apache Hadoop’s default classpath:
Make sure thatHADOOP_OPTIONAL_TOOLS in hadoop-env.sh includes hadoop-aws in its list of optional modules to add in the classpath.
For client side interaction, you can declare that relevant JARs must be loaded in your ~/.hadooprc file:
hadoop_add_to_classpath_tools hadoop-aws
The settings in this file does not propagate to deployed applications, but it will work for local clients such as the hadoop fs command.
Hadoop’s “S3A” client offers high-performance IO against Amazon S3 object store and compatible implementations.
There other Hadoop connectors to S3. Only S3A is actively maintained by the Hadoop project itself.
S3A depends upon two JARs, alongside hadoop-common and its dependencies.
The versions of hadoop-common and hadoop-aws must be identical.
To import the libraries into a Maven build, add hadoop-aws JAR to the build dependencies; it will pull in a compatible aws-sdk JAR.
The hadoop-aws JAR does not declare any dependencies other than that dependencies unique to it, the AWS SDK JAR. This is simplify excluding/tuning Hadoop dependency JARs in downstream applications. The hadoop-client or hadoop-common dependency must be declared
<properties> <!-- Your exact Hadoop version here--> <hadoop.version>3.0.0</hadoop.version> </properties> <dependencies> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>${hadoop.version}</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-aws</artifactId> <version>${hadoop.version}</version> </dependency> </dependencies>
Amazon S3 is an example of “an object store”. In order to achieve scalability and especially high availability, S3 has —as many other cloud object stores have done— relaxed some of the constraints which classic “POSIX” filesystems promise.
The S3Guard feature attempts to address some of these, but it cannot do so completely. Do read these warnings and consider how they apply.
For further discussion on these topics, please consult The Hadoop FileSystem API Definition.
Amazon S3 is an example of “an object store”. In order to achieve scalability and especially high availability, S3 has —as many other cloud object stores have done— relaxed some of the constraints which classic “POSIX” filesystems promise.
Specifically
The S3A clients mimics directories by:
Here are some of the consequences:
The final three issues surface when using S3 as the immediate destination of work, as opposed to HDFS or other “real” filesystem.
The S3A committers are the sole mechanism available to safely save the output of queries directly into S3 object stores through the S3A filesystem.
The object authorization model of S3 is much different from the file authorization model of HDFS and traditional file systems. The S3A client simply reports stub information from APIs that would query this metadata:
S3A does not really enforce any authorization checks on these stub permissions. Users authenticate to an S3 bucket using AWS credentials. It’s possible that object ACLs have been defined to enforce authorization at the S3 side, but this happens entirely within the S3 service, not within the S3A implementation.
Your AWS credentials not only pay for services, they offer read and write access to the data. Anyone with the credentials can not only read your datasets —they can delete them.
Do not inadvertently share these credentials through means such as
If you do any of these: change your credentials immediately!
Except when interacting with public S3 buckets, the S3A client needs the credentials needed to interact with buckets.
The client supports multiple authentication mechanisms and can be configured as to which mechanisms to use, and their order of use. Custom implementations of com.amazonaws.auth.AWSCredentialsProvider may also be used.
<property> <name>fs.s3a.access.key</name> <description>AWS access key ID. Omit for IAM role-based or provider-based authentication.</description> </property> <property> <name>fs.s3a.secret.key</name> <description>AWS secret key. Omit for IAM role-based or provider-based authentication.</description> </property> <property> <name>fs.s3a.aws.credentials.provider</name> <description> Comma-separated class names of credential provider classes which implement com.amazonaws.auth.AWSCredentialsProvider. These are loaded and queried in sequence for a valid set of credentials. Each listed class must implement one of the following means of construction, which are attempted in order: 1. a public constructor accepting java.net.URI and org.apache.hadoop.conf.Configuration, 2. a public static method named getInstance that accepts no arguments and returns an instance of com.amazonaws.auth.AWSCredentialsProvider, or 3. a public default constructor. Specifying org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider allows anonymous access to a publicly accessible S3 bucket without any credentials. Please note that allowing anonymous access to an S3 bucket compromises security and therefore is unsuitable for most use cases. It can be useful for accessing public data sets without requiring AWS credentials. If unspecified, then the default list of credential provider classes, queried in sequence, is: 1. org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider: supports static configuration of AWS access key ID and secret access key. See also fs.s3a.access.key and fs.s3a.secret.key. 2. com.amazonaws.auth.EnvironmentVariableCredentialsProvider: supports configuration of AWS access key ID and secret access key in environment variables named AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, as documented in the AWS SDK. 3. com.amazonaws.auth.InstanceProfileCredentialsProvider: supports use of instance profile credentials if running in an EC2 VM. </description> </property> <property> <name>fs.s3a.session.token</name> <description> Session token, when using org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider as one of the providers. </description> </property>
S3A supports configuration via the standard AWS environment variables.
The core environment variables are for the access key and associated secret:
export AWS_ACCESS_KEY_ID=my.aws.key export AWS_SECRET_ACCESS_KEY=my.secret.key
If the environment variable AWS_SESSION_TOKEN is set, session authentication using “Temporary Security Credentials” is enabled; the Key ID and secret key must be set to the credentials for that specific sesssion.
export AWS_SESSION_TOKEN=SECRET-SESSION-TOKEN export AWS_ACCESS_KEY_ID=SESSION-ACCESS-KEY export AWS_SECRET_ACCESS_KEY=SESSION-SECRET-KEY
These environment variables can be used to set the authentication credentials instead of properties in the Hadoop configuration.
Important: These environment variables are generally not propagated from client to server when YARN applications are launched. That is: having the AWS environment variables set when an application is launched will not permit the launched application to access S3 resources. The environment variables must (somehow) be set on the hosts/processes where the work is executed.
The standard way to authenticate is with an access key and secret key using the properties in the configuration file.
The S3A client follows the following authentication chain:
S3A can be configured to obtain client authentication providers from classes which integrate with the AWS SDK by implementing the com.amazonaws.auth.AWSCredentialsProvider Interface. This is done by listing the implementation classes, in order of preference, in the configuration option fs.s3a.aws.credentials.provider.
Important: AWS Credential Providers are distinct from Hadoop Credential Providers. As will be covered later, Hadoop Credential Providers allow passwords and other secrets to be stored and transferred more securely than in XML configuration files. AWS Credential Providers are classes which can be used by the Amazon AWS SDK to obtain an AWS login from a different source in the system, including environment variables, JVM properties and configuration files.
There are three AWS Credential Providers inside the hadoop-aws JAR:
| classname | description |
|---|---|
| org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider | Session Credentials |
| org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider | Simple name/secret credentials |
| org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider | Anonymous Login |
There are also many in the Amazon SDKs, in particular two which are automatically set up in the authentication chain:
| classname | description |
|---|---|
| com.amazonaws.auth.InstanceProfileCredentialsProvider | EC2 Metadata Credentials |
| com.amazonaws.auth.EnvironmentVariableCredentialsProvider | AWS Environment Variables |
Applications running in EC2 may associate an IAM role with the VM and query the EC2 Instance Metadata Service for credentials to access S3. Within the AWS SDK, this functionality is provided by InstanceProfileCredentialsProvider, which internally enforces a singleton instance in order to prevent throttling problem.
Temporary Security Credentials can be obtained from the Amazon Security Token Service; these consist of an access key, a secret key, and a session token.
To authenticate with these:
Example:
<property> <name>fs.s3a.aws.credentials.provider</name> <value>org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider</value> </property> <property> <name>fs.s3a.access.key</name> <value>SESSION-ACCESS-KEY</value> </property> <property> <name>fs.s3a.secret.key</name> <value>SESSION-SECRET-KEY</value> </property> <property> <name>fs.s3a.session.token</name> <value>SECRET-SESSION-TOKEN</value> </property>
The lifetime of session credentials are fixed when the credentials are issued; once they expire the application will no longer be able to authenticate to AWS.
Specifying org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider allows anonymous access to a publicly accessible S3 bucket without any credentials. It can be useful for accessing public data sets without requiring AWS credentials.
<property> <name>fs.s3a.aws.credentials.provider</name> <value>org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider</value> </property>
Once this is done, there’s no need to supply any credentials in the Hadoop configuration or via environment variables.
This option can be used to verify that an object store does not permit unauthenticated access: that is, if an attempt to list a bucket is made using the anonymous credentials, it should fail —unless explicitly opened up for broader access.
hadoop fs -ls \ -D fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider \ s3a://landsat-pds/
Allowing anonymous access to an S3 bucket compromises security and therefore is unsuitable for most use cases.
If a list of credential providers is given in fs.s3a.aws.credentials.provider, then the Anonymous Credential provider must come last. If not, credential providers listed after it will be ignored.
Simple name/secret credentials with SimpleAWSCredentialsProvider
This is is the standard credential provider, which supports the secret key in fs.s3a.access.key and token in fs.s3a.secret.key values. It does not support authentication with logins credentials declared in the URLs.
<property> <name>fs.s3a.aws.credentials.provider</name> <value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value> </property>
Apart from its lack of support of user:password details being included in filesystem URLs (a dangerous practise that is strongly discouraged), this provider acts exactly at the basic authenticator used in the default authentication chain.
This means that the default S3A authentication chain can be defined as
<property> <name>fs.s3a.aws.credentials.provider</name> <value> org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider, com.amazonaws.auth.EnvironmentVariableCredentialsProvider, com.amazonaws.auth.InstanceProfileCredentialsProvider </value> </property>
It is critical that you never share or leak your AWS credentials. Loss of credentials can leak/lose all your data, run up large bills, and significantly damage your organisation.
Never share your secrets.
Never commit your secrets into an SCM repository. The git secrets can help here.
Avoid using s3a URLs which have key and secret in the URL. This is dangerous as the secrets leak into the logs.
Never include AWS credentials in bug reports, files attached to them, or similar.
If you use the AWS_ environment variables, your list of environment variables is equally sensitive.
Never use root credentials. Use IAM user accounts, with each user/application having its own set of credentials.
Use IAM permissions to restrict the permissions individual users and applications have. This is best done through roles, rather than configuring individual users.
Avoid passing in secrets to Hadoop applications/commands on the command line. The command line of any launched program is visible to all users on a Unix system (via ps), and preserved in command histories.
Explore using IAM Assumed Roles for role-based permissions management: a specific S3A connection can be made with a different assumed role and permissions from the primary user account.
Consider a workflow in which usera and applications are issued with short-lived session credentials, configuring S3A to use these through the TemporaryAWSCredentialsProvider.
Have a secure process in place for cancelling and re-issuing credentials for users and applications. Test it regularly by using it to refresh credentials.
When running in EC2, the IAM EC2 instance credential provider will automatically obtain the credentials needed to access AWS services in the role the EC2 VM was deployed as. This credential provider is enabled in S3A by default.
The safest way to keep the AWS login keys a secret within Hadoop is to use Hadoop Credentials.
The Hadoop Credential Provider Framework allows secure “Credential Providers” to keep secrets outside Hadoop configuration files, storing them in encrypted files in local or Hadoop filesystems, and including them in requests.
The S3A configuration options with sensitive data (fs.s3a.secret.key, fs.s3a.access.key, fs.s3a.session.token and fs.s3a.server-side-encryption.key) can have their data saved to a binary file stored, with the values being read in when the S3A filesystem URL is used for data access. The reference to this credential provider then declareed in the hadoop configuration.
For additional reading on the Hadoop Credential Provider API see: Credential Provider API.
The following configuration options can be storeed in Hadoop Credential Provider stores.
fs.s3a.access.key fs.s3a.secret.key fs.s3a.session.token fs.s3a.server-side-encryption.key fs.s3a.server-side-encryption-algorithm
The first three are for authentication; the final two for encryption. Of the latter, only the encryption key can be considered “sensitive”. However, being able to include the algorithm in the credentials allows for a JCECKS file to contain all the options needed to encrypt new data written to S3.
A credential file can be created on any Hadoop filesystem; when creating one on HDFS or a Unix filesystem the permissions are automatically set to keep the file private to the reader —though as directory permissions are not touched, users should verify that the directory containing the file is readable only by the current user.
hadoop credential create fs.s3a.access.key -value 123 \ -provider jceks://hdfs@nn1.example.com:9001/user/backup/s3.jceks hadoop credential create fs.s3a.secret.key -value 456 \ -provider jceks://hdfs@nn1.example.com:9001/user/backup/s3.jceks
A credential file can be listed, to see what entries are kept inside it
hadoop credential list -provider jceks://hdfs@nn1.example.com:9001/user/backup/s3.jceks Listing aliases for CredentialProvider: jceks://hdfs@nn1.example.com:9001/user/backup/s3.jceks fs.s3a.secret.key fs.s3a.access.key
At this point, the credentials are ready for use.
The URL to the provider must be set in the configuration property hadoop.security.credential.provider.path, either on the command line or in XML configuration files.
<property> <name>hadoop.security.credential.provider.path</name> <value>jceks://hdfs@nn1.example.com:9001/user/backup/s3.jceks</value> <description>Path to interrogate for protected credentials.</description> </property>
Because this property only supplies the path to the secrets file, the configuration option itself is no longer a sensitive item.
The property hadoop.security.credential.provider.path is global to all filesystems and secrets. There is another property, fs.s3a.security.credential.provider.path which only lists credential providers for S3A filesystems. The two properties are combined into one, with the list of providers in the fs.s3a. property taking precedence over that of the hadoop.security list (i.e. they are prepended to the common list).
<property> <name>fs.s3a.security.credential.provider.path</name> <value /> <description> Optional comma separated list of credential providers, a list which is prepended to that set in hadoop.security.credential.provider.path </description> </property>
This was added to suppport binding different credential providers on a per bucket basis, without adding alternative secrets in the credential list. However, some applications (e.g Hive) prevent the list of credential providers from being dynamically updated by users. As per-bucket secrets are now supported, it is better to include per-bucket keys in JCEKS files and other sources of credentials.
Once the provider is set in the Hadoop configuration, hadoop commands work exactly as if the secrets were in an XML file.
hadoop distcp \ hdfs://nn1.example.com:9001/user/backup/007020615 s3a://glacier1/ hadoop fs -ls s3a://glacier1/
The path to the provider can also be set on the command line:
hadoop distcp \ -D hadoop.security.credential.provider.path=jceks://hdfs@nn1.example.com:9001/user/backup/s3.jceks \ hdfs://nn1.example.com:9001/user/backup/007020615 s3a://glacier1/ hadoop fs \ -D fs.s3a.security.credential.provider.path=jceks://hdfs@nn1.example.com:9001/user/backup/s3.jceks \ -ls s3a://glacier1/
Because the provider path is not itself a sensitive secret, there is no risk from placing its declaration on the command line.
All S3A client options are configured with options with the prefix fs.s3a..
The client supports Per-bucket configuration to allow different buckets to override the shared settings. This is commonly used to change the endpoint, encryption and authentication mechanisms of buckets. S3Guard options, various minor options.
Here are the S3A properties for use in production. The S3Guard options are documented in the S3Guard documenents; some testing-related options are covered in Testing.
<property> <name>fs.s3a.connection.maximum</name> <value>15</value> <description>Controls the maximum number of simultaneous connections to S3.</description> </property> <property> <name>fs.s3a.connection.ssl.enabled</name> <value>true</value> <description>Enables or disables SSL connections to S3.</description> </property> <property> <name>fs.s3a.endpoint</name> <description>AWS S3 endpoint to connect to. An up-to-date list is provided in the AWS Documentation: regions and endpoints. Without this property, the standard region (s3.amazonaws.com) is assumed. </description> </property> <property> <name>fs.s3a.path.style.access</name> <value>false</value> <description>Enable S3 path style access ie disabling the default virtual hosting behaviour. Useful for S3A-compliant storage providers as it removes the need to set up DNS for virtual hosting. </description> </property> <property> <name>fs.s3a.proxy.host</name> <description>Hostname of the (optional) proxy server for S3 connections.</description> </property> <property> <name>fs.s3a.proxy.port</name> <description>Proxy server port. If this property is not set but fs.s3a.proxy.host is, port 80 or 443 is assumed (consistent with the value of fs.s3a.connection.ssl.enabled).</description> </property> <property> <name>fs.s3a.proxy.username</name> <description>Username for authenticating with proxy server.</description> </property> <property> <name>fs.s3a.proxy.password</name> <description>Password for authenticating with proxy server.</description> </property> <property> <name>fs.s3a.proxy.domain</name> <description>Domain for authenticating with proxy server.</description> </property> <property> <name>fs.s3a.proxy.workstation</name> <description>Workstation for authenticating with proxy server.</description> </property> <property> <name>fs.s3a.attempts.maximum</name> <value>20</value> <description>How many times we should retry commands on transient errors.</description> </property> <property> <name>fs.s3a.connection.establish.timeout</name> <value>5000</value> <description>Socket connection setup timeout in milliseconds.</description> </property> <property> <name>fs.s3a.connection.timeout</name> <value>200000</value> <description>Socket connection timeout in milliseconds.</description> </property> <property> <name>fs.s3a.paging.maximum</name> <value>5000</value> <description>How many keys to request from S3 when doing directory listings at a time.</description> </property> <property> <name>fs.s3a.threads.max</name> <value>10</value> <description> Maximum number of concurrent active (part)uploads, which each use a thread from the threadpool.</description> </property> <property> <name>fs.s3a.socket.send.buffer</name> <value>8192</value> <description>Socket send buffer hint to amazon connector. Represented in bytes.</description> </property> <property> <name>fs.s3a.socket.recv.buffer</name> <value>8192</value> <description>Socket receive buffer hint to amazon connector. Represented in bytes.</description> </property> <property> <name>fs.s3a.threads.keepalivetime</name> <value>60</value> <description>Number of seconds a thread can be idle before being terminated.</description> </property> <property> <name>fs.s3a.max.total.tasks</name> <value>5</value> <description>Number of (part)uploads allowed to the queue before blocking additional uploads.</description> </property> <property> <name>fs.s3a.multipart.size</name> <value>100M</value> <description>How big (in bytes) to split upload or copy operations up into. A suffix from the set {K,M,G,T,P} may be used to scale the numeric value. </description> </property> <property> <name>fs.s3a.multipart.threshold</name> <value>2147483647</value> <description>How big (in bytes) to split upload or copy operations up into. This also controls the partition size in renamed files, as rename() involves copying the source file(s). A suffix from the set {K,M,G,T,P} may be used to scale the numeric value. </description> </property> <property> <name>fs.s3a.multiobjectdelete.enable</name> <value>true</value> <description>When enabled, multiple single-object delete requests are replaced by a single 'delete multiple objects'-request, reducing the number of requests. Beware: legacy S3-compatible object stores might not support this request. </description> </property> <property> <name>fs.s3a.acl.default</name> <description>Set a canned ACL for newly created and copied objects. Value may be Private, PublicRead, PublicReadWrite, AuthenticatedRead, LogDeliveryWrite, BucketOwnerRead, or BucketOwnerFullControl.</description> </property> <property> <name>fs.s3a.multipart.purge</name> <value>false</value> <description>True if you want to purge existing multipart uploads that may not have been completed/aborted correctly</description> </property> <property> <name>fs.s3a.multipart.purge.age</name> <value>86400</value> <description>Minimum age in seconds of multipart uploads to purge</description> </property> <property> <name>fs.s3a.signing-algorithm</name> <description>Override the default signing algorithm so legacy implementations can still be used</description> </property> <property> <name>fs.s3a.server-side-encryption-algorithm</name> <description>Specify a server-side encryption algorithm for s3a: file system. Unset by default. It supports the following values: 'AES256' (for SSE-S3), 'SSE-KMS' and 'SSE-C' </description> </property> <property> <name>fs.s3a.server-side-encryption.key</name> <description>Specific encryption key to use if fs.s3a.server-side-encryption-algorithm has been set to 'SSE-KMS' or 'SSE-C'. In the case of SSE-C, the value of this property should be the Base64 encoded key. If you are using SSE-KMS and leave this property empty, you'll be using your default's S3 KMS key, otherwise you should set this property to the specific KMS key id.</description> </property> <property> <name>fs.s3a.buffer.dir</name> <value>${hadoop.tmp.dir}/s3a</value> <description>Comma separated list of directories that will be used to buffer file uploads to.</description> </property> <property> <name>fs.s3a.block.size</name> <value>32M</value> <description>Block size to use when reading files using s3a: file system. </description> </property> <property> <name>fs.s3a.user.agent.prefix</name> <value></value> <description> Sets a custom value that will be prepended to the User-Agent header sent in HTTP requests to the S3 back-end by S3AFileSystem. The User-Agent header always includes the Hadoop version number followed by a string generated by the AWS SDK. An example is "User-Agent: Hadoop 2.8.0, aws-sdk-java/1.10.6". If this optional property is set, then its value is prepended to create a customized User-Agent. For example, if this configuration property was set to "MyApp", then an example of the resulting User-Agent would be "User-Agent: MyApp, Hadoop 2.8.0, aws-sdk-java/1.10.6". </description> </property> <property> <name>fs.s3a.impl</name> <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value> <description>The implementation class of the S3A Filesystem</description> </property> <property> <name>fs.AbstractFileSystem.s3a.impl</name> <value>org.apache.hadoop.fs.s3a.S3A</value> <description>The implementation class of the S3A AbstractFileSystem.</description> </property> <property> <name>fs.s3a.readahead.range</name> <value>64K</value> <description>Bytes to read ahead during a seek() before closing and re-opening the S3 HTTP connection. This option will be overridden if any call to setReadahead() is made to an open stream.</description> </property> <property> <name>fs.s3a.list.version</name> <value>2</value> <description>Select which version of the S3 SDK's List Objects API to use. Currently support 2 (default) and 1 (older API).</description> </property>
The S3A client makes a best-effort attempt at recovering from network failures; this section covers the details of what it does.
The S3A divides exceptions returned by the AWS SDK into different categories, and chooses a differnt retry policy based on their type and whether or not the failing operation is idempotent.
These are all considered unrecoverable: S3A will make no attempt to recover from them.
The status code 400, Bad Request usually means that the request is unrecoverable; it’s the generic “No” response. Very rarely it does recover, which is why it is in this category, rather than that of unrecoverable failures.
These failures will be retried with a fixed sleep interval set in fs.s3a.retry.interval, up to the limit set in fs.s3a.retry.limit.
Some network failures are considered to be retriable if they occur on idempotent operations; there’s no way to know if they happened after the request was processed by S3.
These failures will be retried with a fixed sleep interval set in fs.s3a.retry.interval, up to the limit set in fs.s3a.retry.limit.
Important: DELETE is considered idempotent, hence: FileSystem.delete() and FileSystem.rename() will retry their delete requests on any of these failures.
The issue of whether delete should be idempotent has been a source of historical controversy in Hadoop.
Because S3 is eventially consistent and doesn’t support an atomic create-no-overwrite operation, the choice is more ambigious.
Currently S3A considers delete to be idempotent because it is convenient for many workflows, including the commit protocols. Just be aware that in the presence of transient failures, more things may be deleted than expected. (For anyone who considers this to be the wrong decision: rebuild the hadoop-aws module with the constant S3AFileSystem.DELETE_CONSIDERED_IDEMPOTENT set to false).
When S3A or Dynamo DB returns a response indicating that requests from the caller are being throttled, an exponential back-off with an initial interval and a maximum number of requests.
<property> <name>fs.s3a.retry.throttle.limit</name> <value>${fs.s3a.attempts.maximum}</value> <description> Number of times to retry any throttled request. </description> </property> <property> <name>fs.s3a.retry.throttle.interval</name> <value>1000ms</value> <description> Interval between retry attempts on throttled requests. </description> </property>
Notes
Throttling of S3 requests is all too common; it is caused by too many clients trying to access the same shard of S3 Storage. This generatlly happen if there are too many reads, those being the most common in Hadoop applications. This problem is exacerbated by Hive’s partitioning strategy used when storing data, such as partitioning by year and then month. This results in paths with little or no variation at their start, which ends up in all the data being stored in the same shard(s).
Here are some expensive operations; the more of these taking place against part of an S3 bucket, the more load it experiences. * Many clients trying to list directories or calling getFileStatus on paths (LIST and HEAD requests respectively) * The GET requests issued when reading data. * Random IO used when reading columnar data (ORC, Parquet) means that many more GET requests than a simple one-per-file read. * The number of active writes to that part of the S3 bucket.
A special case is when enough data has been written into part of an S3 bucket that S3 decides to split the data across more than one shard: this is believed to be one by some copy operation which can take some time. While this is under way, S3 clients access data under these paths will be throttled more than usual.
Mitigation strategies
Different S3 buckets can be accessed with different S3A client configurations. This allows for different endpoints, data read and write strategies, as well as login details.
As an example, a configuration could have a base configuration to use the IAM role information available when deployed in Amazon EC2.
<property> <name>fs.s3a.aws.credentials.provider</name> <value>com.amazonaws.auth.InstanceProfileCredentialsProvider</value> </property>
This will become the default authentication mechanism for S3A buckets.
A bucket s3a://nightly/ used for nightly data can then be given a session key:
<property> <name>fs.s3a.bucket.nightly.access.key</name> <value>AKAACCESSKEY-2</value> </property> <property> <name>fs.s3a.bucket.nightly.secret.key</name> <value>SESSIONSECRETKEY</value> </property> <property> <name>fs.s3a.bucket.nightly.session.token</name> <value>Short-lived-session-token</value> </property> <property> <name>fs.s3a.bucket.nightly.aws.credentials.provider</name> <value>org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider</value> </property>
Finally, the public s3a://landsat-pds/ bucket can be accessed anonymously:
<property> <name>fs.s3a.bucket.landsat-pds.aws.credentials.provider</name> <value>org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider</value> </property>
Secrets in JCEKS files or provided by other Hadoop credential providers can also be configured on a per bucket basis. The S3A client will look for the per-bucket secrets be
Consider a JCEKS file with six keys:
fs.s3a.access.key fs.s3a.secret.key fs.s3a.server-side-encryption-algorithm fs.s3a.bucket.nightly.access.key fs.s3a.bucket.nightly.secret.key fs.s3a.bucket.nightly.session.token fs.s3a.bucket.nightly.server-side-encryption.key fs.s3a.bucket.nightly.server-side-encryption-algorithm
When accessing the bucket s3a://nightly/, the per-bucket configuration options for that backet will be used, here the access keys and token, and including the encryption algorithm and key.
S3 Buckets are hosted in different “regions”, the default being “US-East”. The S3A client talks to this region by default, issing HTTP requests to the server s3.amazonaws.com.
S3A can work with buckets from any region. Each region has its own S3 endpoint, documented by Amazon.
While it is generally simpler to use the default endpoint, working with V4-signing-only regions (Frankfurt, Seoul) requires the endpoint to be identified. Expect better performance from direct connections —traceroute will give you some insight.
If the wrong endpoint is used, the request may fail. This may be reported as a 301/redirect error, or as a 400 Bad Request: take these as cues to check the endpoint setting of a bucket.
Here is a list of properties defining all AWS S3 regions, current as of June 2017:
<!-- This is the default endpoint, which can be used to interact with any v2 region. --> <property> <name>central.endpoint</name> <value>s3.amazonaws.com</value> </property> <property> <name>canada.endpoint</name> <value>s3.ca-central-1.amazonaws.com</value> </property> <property> <name>frankfurt.endpoint</name> <value>s3.eu-central-1.amazonaws.com</value> </property> <property> <name>ireland.endpoint</name> <value>s3-eu-west-1.amazonaws.com</value> </property> <property> <name>london.endpoint</name> <value>s3.eu-west-2.amazonaws.com</value> </property> <property> <name>mumbai.endpoint</name> <value>s3.ap-south-1.amazonaws.com</value> </property> <property> <name>ohio.endpoint</name> <value>s3.us-east-2.amazonaws.com</value> </property> <property> <name>oregon.endpoint</name> <value>s3-us-west-2.amazonaws.com</value> </property> <property> <name>sao-paolo.endpoint</name> <value>s3-sa-east-1.amazonaws.com</value> </property> <property> <name>seoul.endpoint</name> <value>s3.ap-northeast-2.amazonaws.com</value> </property> <property> <name>singapore.endpoint</name> <value>s3-ap-southeast-1.amazonaws.com</value> </property> <property> <name>sydney.endpoint</name> <value>s3-ap-southeast-2.amazonaws.com</value> </property> <property> <name>tokyo.endpoint</name> <value>s3-ap-northeast-1.amazonaws.com</value> </property> <property> <name>virginia.endpoint</name> <value>${central.endpoint}</value> </property>
This list can be used to specify the endpoint of individual buckets, for example for buckets in the central and EU/Ireland endpoints.
<property> <name>fs.s3a.bucket.landsat-pds.endpoint</name> <value>${central.endpoint}</value> <description>The endpoint for s3a://landsat-pds URLs</description> </property> <property> <name>fs.s3a.bucket.eu-dataset.endpoint</name> <value>${ireland.endpoint}</value> <description>The endpoint for s3a://eu-dataset URLs</description> </property>
Why explicitly declare a bucket bound to the central endpoint? It ensures that if the default endpoint is changed to a new region, data store in US-east is still reachable.
The original S3A client implemented file writes by buffering all data to disk as it was written to the OutputStream. Only when the stream’s close() method was called would the upload start.
This made output slow, especially on large uploads, and could even fill up the disk space of small (virtual) disks.
Hadoop 2.7 added the S3AFastOutputStream alternative, which Hadoop 2.8 expanded. It is now considered stable and has replaced the original S3AOutputStream, which is no longer shipped in hadoop.
The “fast” output stream
Because it starts uploading while data is still being written, it offers significant benefits when very large amounts of data are generated. The in memory buffering mechanims may also offer speedup when running adjacent to S3 endpoints, as disks are not used for intermediate data storage.
<property> <name>fs.s3a.fast.upload.buffer</name> <value>disk</value> <description> The buffering mechanism to use. Values: disk, array, bytebuffer. "disk" will use the directories listed in fs.s3a.buffer.dir as the location(s) to save data prior to being uploaded. "array" uses arrays in the JVM heap "bytebuffer" uses off-heap memory within the JVM. Both "array" and "bytebuffer" will consume memory in a single stream up to the number of blocks set by: fs.s3a.multipart.size * fs.s3a.fast.upload.active.blocks. If using either of these mechanisms, keep this value low The total number of threads performing work across all threads is set by fs.s3a.threads.max, with fs.s3a.max.total.tasks values setting the number of queued work items. </description> </property> <property> <name>fs.s3a.multipart.size</name> <value>100M</value> <description>How big (in bytes) to split upload or copy operations up into. A suffix from the set {K,M,G,T,P} may be used to scale the numeric value. </description> </property> <property> <name>fs.s3a.fast.upload.active.blocks</name> <value>8</value> <description> Maximum Number of blocks a single output stream can have active (uploading, or queued to the central FileSystem instance's pool of queued operations. This stops a single stream overloading the shared thread pool. </description> </property>
Notes
If the amount of data written to a stream is below that set in fs.s3a.multipart.size, the upload is performed in the OutputStream.close() operation —as with the original output stream.
The published Hadoop metrics monitor include live queue length and upload operation counts, so identifying when there is a backlog of work/ a mismatch between data generation rates and network bandwidth. Per-stream statistics can also be logged by calling toString() on the current stream.
Files being written are still invisible untl the write completes in the close() call, which will block until the upload is completed.
When fs.s3a.fast.upload.buffer is set to disk, all data is buffered to local hard disks prior to upload. This minimizes the amount of memory consumed, and so eliminates heap size as the limiting factor in queued uploads —exactly as the original “direct to disk” buffering.
<property> <name>fs.s3a.fast.upload.buffer</name> <value>disk</value> </property> <property> <name>fs.s3a.buffer.dir</name> <value>${hadoop.tmp.dir}/s3a</value> <description>Comma separated list of directories that will be used to buffer file uploads to.</description> </property>
This is the default buffer mechanism. The amount of data which can be buffered is limited by the amount of available disk space.
When fs.s3a.fast.upload.buffer is set to bytebuffer, all data is buffered in “Direct” ByteBuffers prior to upload. This may be faster than buffering to disk, and, if disk space is small (for example, tiny EC2 VMs), there may not be much disk space to buffer with.
The ByteBuffers are created in the memory of the JVM, but not in the Java Heap itself. The amount of data which can be buffered is limited by the Java runtime, the operating system, and, for YARN applications, the amount of memory requested for each container.
The slower the upload bandwidth to S3, the greater the risk of running out of memory —and so the more care is needed in tuning the upload settings.
<property> <name>fs.s3a.fast.upload.buffer</name> <value>bytebuffer</value> </property>
When fs.s3a.fast.upload.buffer is set to array, all data is buffered in byte arrays in the JVM’s heap prior to upload. This may be faster than buffering to disk.
The amount of data which can be buffered is limited by the available size of the JVM heap heap. The slower the write bandwidth to S3, the greater the risk of heap overflows. This risk can be mitigated by tuning the upload settings.
<property> <name>fs.s3a.fast.upload.buffer</name> <value>array</value> </property>
Both the Array and Byte buffer buffer mechanisms can consume very large amounts of memory, on-heap or off-heap respectively. The disk buffer mechanism does not use much memory up, but will consume hard disk capacity.
If there are many output streams being written to in a single process, the amount of memory or disk used is the multiple of all stream’s active memory/disk use.
Careful tuning may be needed to reduce the risk of running out memory, especially if the data is buffered in memory.
There are a number parameters which can be tuned:
The total number of threads available in the filesystem for data uploads or any other queued filesystem operation. This is set in fs.s3a.threads.max
The number of operations which can be queued for execution:, awaiting a thread: fs.s3a.max.total.tasks
The number of blocks which a single output stream can have active, that is: being uploaded by a thread, or queued in the filesystem thread queue: fs.s3a.fast.upload.active.blocks
How long an idle thread can stay in the thread pool before it is retired: fs.s3a.threads.keepalivetime
When the maximum allowed number of active blocks of a single stream is reached, no more blocks can be uploaded from that stream until one or more of those active blocks’ uploads completes. That is: a write() call which would trigger an upload of a now full datablock, will instead block until there is capacity in the queue.
How does that come together?
As the pool of threads set in fs.s3a.threads.max is shared (and intended to be used across all threads), a larger number here can allow for more parallel operations. However, as uploads require network bandwidth, adding more threads does not guarantee speedup.
The extra queue of tasks for the thread pool (fs.s3a.max.total.tasks) covers all ongoing background S3A operations (future plans include: parallelized rename operations, asynchronous directory operations).
When using memory buffering, a small value of fs.s3a.fast.upload.active.blocks limits the amount of memory which can be consumed per stream.
When using disk buffering a larger value of fs.s3a.fast.upload.active.blocks does not consume much memory. But it may result in a large number of blocks to compete with other filesystem operations.
We recommend a low value of fs.s3a.fast.upload.active.blocks; enough to start background upload without overloading other parts of the system, then experiment to see if higher values deliver more throughtput —especially from VMs running on EC2.
<property> <name>fs.s3a.fast.upload.active.blocks</name> <value>4</value> <description> Maximum Number of blocks a single output stream can have active (uploading, or queued to the central FileSystem instance's pool of queued operations. This stops a single stream overloading the shared thread pool. </description> </property> <property> <name>fs.s3a.threads.max</name> <value>10</value> <description>The total number of threads available in the filesystem for data uploads *or any other queued filesystem operation*.</description> </property> <property> <name>fs.s3a.max.total.tasks</name> <value>5</value> <description>The number of operations which can be queued for execution</description> </property> <property> <name>fs.s3a.threads.keepalivetime</name> <value>60</value> <description>Number of seconds a thread can be idle before being terminated.</description> </property>
There are two mechanisms for cleaning up after leftover multipart uploads: - Hadoop s3guard CLI commands for listing and deleting uploads by their age. Doumented in the S3Guard section. - The configuration parameter fs.s3a.multipart.purge, covered below.
If an large stream writeoperation is interrupted, there may be intermediate partitions uploaded to S3 —data which will be billed for.
These charges can be reduced by enabling fs.s3a.multipart.purge, and setting a purge time in seconds, such as 86400 seconds —24 hours. When an S3A FileSystem instance is instantiated with the purge time greater than zero, it will, on startup, delete all outstanding partition requests older than this time.
<property> <name>fs.s3a.multipart.purge</name> <value>true</value> <description>True if you want to purge existing multipart uploads that may not have been completed/aborted correctly</description> </property> <property> <name>fs.s3a.multipart.purge.age</name> <value>86400</value> <description>Minimum age in seconds of multipart uploads to purge</description> </property>
If an S3A client is instantiated with fs.s3a.multipart.purge=true, it will delete all out of date uploads in the entire bucket. That is: it will affect all multipart uploads to that bucket, from all applications.
Leaving fs.s3a.multipart.purge to its default, false, means that the client will not make any attempt to reset or change the partition rate.
The best practise for using this option is to disable multipart purges in normal use of S3A, enabling only in manual/scheduled housekeeping operations.
The S3A Filesystem client supports the notion of input policies, similar to that of the Posix fadvise() API call. This tunes the behavior of the S3A client to optimise HTTP GET requests for the different use cases.
See Improving data input performance through fadvise for the details.
S3A metrics can be monitored through Hadoop’s metrics2 framework. S3A creates its own metrics system called s3a-file-system, and each instance of the client will create its own metrics source, named with a JVM-unique numerical ID.
As a simple example, the following can be added to hadoop-metrics2.properties to write all S3A metrics to a log file every 10 seconds:
s3a-file-system.sink.my-metrics-config.class=org.apache.hadoop.metrics2.sink.FileSink s3a-file-system.sink.my-metrics-config.filename=/var/log/hadoop-yarn/s3a-metrics.out *.period=10
Lines in that file will be structured like the following:
1511208770680 s3aFileSystem.s3aFileSystem: Context=s3aFileSystem, s3aFileSystemId=892b02bb-7b30-4ffe-80ca-3a9935e1d96e, bucket=bucket, Hostname=hostname-1.hadoop.apache.com, files_created=1, files_copied=2, files_copied_bytes=10000, files_deleted=5, fake_directories_deleted=3, directories_created=3, directories_deleted=0, ignored_errors=0, op_copy_from_local_file=0, op_exists=0, op_get_file_status=15, op_glob_status=0, op_is_directory=0, op_is_file=0, op_list_files=0, op_list_located_status=0, op_list_status=3, op_mkdirs=1, op_rename=2, object_copy_requests=0, object_delete_requests=6, object_list_requests=23, object_continue_list_requests=0, object_metadata_requests=46, object_multipart_aborted=0, object_put_bytes=0, object_put_requests=4, object_put_requests_completed=4, stream_write_failures=0, stream_write_block_uploads=0, stream_write_block_uploads_committed=0, stream_write_block_uploads_aborted=0, stream_write_total_time=0, stream_write_total_data=0, s3guard_metadatastore_put_path_request=10, s3guard_metadatastore_initialization=0, object_put_requests_active=0, object_put_bytes_pending=0, stream_write_block_uploads_active=0, stream_write_block_uploads_pending=0, stream_write_block_uploads_data_pending=0, S3guard_metadatastore_put_path_latencyNumOps=0, S3guard_metadatastore_put_path_latency50thPercentileLatency=0, S3guard_metadatastore_put_path_latency75thPercentileLatency=0, S3guard_metadatastore_put_path_latency90thPercentileLatency=0, S3guard_metadatastore_put_path_latency95thPercentileLatency=0, S3guard_metadatastore_put_path_latency99thPercentileLatency=0
Depending on other configuration, metrics from other systems, contexts, etc. may also get recorded, for example the following:
1511208770680 metricssystem.MetricsSystem: Context=metricssystem, Hostname=s3a-metrics-4.gce.cloudera.com, NumActiveSources=1, NumAllSources=1, NumActiveSinks=1, NumAllSinks=0, Sink_fileNumOps=2, Sink_fileAvgTime=1.0, Sink_fileDropped=0, Sink_fileQsize=0, SnapshotNumOps=5, SnapshotAvgTime=0.0, PublishNumOps=2, PublishAvgTime=0.0, DroppedPubAll=0
Note that low-level metrics from the AWS SDK itself are not currently included in these metrics.
Hadoop’s distcp tool is often used to copy data between a Hadoop cluster and Amazon S3. See Copying Data Between a Cluster and Amazon S3 for details on S3 copying specifically.
The distcp update command tries to do incremental updates of data. It is straightforward to verify when files do not match when they are of different length, but not when they are the same size.
Distcp addresses this by comparing file checksums on the source and destination filesystems, which it tries to do even if the filesystems have incompatible checksum algorithms.
The S3A connector can provide the HTTP etag header to the caller as the checksum of the uploaded file. Doing so will break distcp operations between hdfs and s3a.
For this reason, the etag-as-checksum feature is disabled by default.
<property> <name>fs.s3a.etag.checksum.enabled</name> <value>false</value> <description> Should calls to getFileChecksum() return the etag value of the remote object. WARNING: if enabled, distcp operations between HDFS and S3 will fail unless -skipcrccheck is set. </description> </property>
If enabled, distcp between two S3 buckets can use the checksum to compare objects. Their checksums should be identical if they were either each uploaded as a single file PUT, or, if in a multipart PUT, in blocks of the same size, as configured by the value fs.s3a.multipart.size.
To disable checksum verification in distcp, use the -skipcrccheck option:
hadoop distcp -update -skipcrccheck /user/alice/datasets s3a://alice-backup/datasets


Tango is the kind of balletic dance that infuses sex and mystery in equal parts, and is traditionally performed by a man and a woman. It necessitates coordination and skill and lots of practice, like any other dance. Carlos Saura's film, "Tango," is an attempt to understand the history of tango mixed in with personal relationships and creative talents to produce a film on tango. It is not a complete success, but it is a visually luscious trip.
Miguel Angel Sola plays Mario Sanchez, a filmmaker who feels his private life is in a state of confusion. His former wife, Laura Fuentes (Cecilia Narova), is living with another man, though Mario still loves her. Laura is also a tango dancer, and is part of the troupe who will perform in Mario's theatrical presentation on the history of tango. A rich gangster, Angelo Larroca (Juan Luis Galiardo), gives Mario a new recruit, an especially attractive dancer, Elena Flores (Mia Maestro), who eventually has an affair with Mario. Naturally, Angelo loves Elena and if he discovers she is having an affair, it will result in death.
"Tango" is imminently watchable when it shows us the pure tango dances against creative, color backgrounds. There are silhouettes, flashes of color, and just about every other camera trick you can imagine from the imaginative cinematographer Vittorio Storaro, an active collaborator of Bernardo Bertolucci's. I also love the moments when we witness the preparation of the production itself, the set design and the choreography of the movements of the dancers, etc. These are the moments that count, but the human interest level is at a minimum.
I wanted to know more about the middle-aged, flamboyant dancer Carlos Nebbia (well-played by Juan Copes) and his tango desires. I also wanted to know more about Mario Sanchez and the relationships with these women. There is one effective scene where he has a dinner date with Elena and he tries to explain his feelings about men and women, and fails to make sense. There is also a hint of Mario's imagination unfolding before our eyes, like when we see him inattentive as a young boy in school or when he envisions a strikingly sexy dance in silhouette while he blows the fan to create the mood. But these are all hints of character development and nothing more.
Perhaps I expected more from director Carlos Saura, who made the flamenco masterpiece "Carmen," a film of unbridled passions in every sense of the word and strong characters. If you want tango dancing and colorful, imaginative landscapes of color and sound, "Tango" is the film for you. If you like some human involvement and a meatier story, look elsewhere.
For more reviews, check out JERRY AT THE MOVIES at http://buffs.moviething.com/buffs/faust/
E-mail me with any questions, concerns or comments at<EMAIL_ADDRESS>or at<EMAIL_ADDRESS>The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


I have a question regarding CTs and EmonTX. I have 2 CTS from an original TED energy monitor and would like to use them in a multi-circuit energy monitor I am working on.
It appears they already have a burden resistor as the label states:
TED Model: QX 200-CT
Input: 200A
MAX 600V ~ 50/60Hz
Output: 0-3V
I will be using this with an Arduino Due, so the reference voltage is 3.3V. To use this, would it be as simple as using the basic CT circuit (http://openenergymonitor.org/emon/buildingblocks/ct-sensors-interface) with no extra burden resistor and the voltage divider with the 2 10k resistors to bring the input voltage peak at 1.5V?
Re: CT Question
I'm not familiar with that CT, but like you, I would interpret that as meaning it has an internal burden resistor and that it outputs 3V (rms) when it's clamped around a wire carrying 200A (rms). That means that while measuring a 200A circuit it will be swinging from +4.24V to -4.24V so those swings would be way too large for your 3.3V system.
You could confirm that's what they mean by connecting a multimeter to the output of the CT and clamping various test loads.
Is there any chance there's a small '.' in front of the 3V, i.e. that it's 0.3V. There are a bunch of CTs with internal burden resistors that are designed to output 333mV at maximum primary current. 3V seems a big swing for any microprocessor to deal with, but I don't know what's inside a TED.
There are also some "CTs" (using the term very loosely) that attempt to output a rectified DC voltage proportional to the primary AC current, so you might want to rule that out as well. Again, a multimeter should help you characterise it further.
Re: CT Question
I'll go along with dBC's answer - you need to find out what you actually get out of your CT before you can make any rational decision. You might find that one of Robin's test sketches is useful to look at the output waveform - but start with a low current (< 20 A) just in case the output is 3 V rms. If it is 3 V, then forget the voltage divider, just add a second burden resistor in parallel with the output to bring the voltage down (remember that CTs are current sources and are quite happy working into a short circuit - it's open circuits they don't like).
Re: CT Question
Won't he still need the voltage divider to centre the swing, i.e. to stop it swinging below 0V?
Re: CT Question
Thanks for the help. @dBC I thought the same thing about the voltage, but the labels are pretty clear and neither one has a '.' in front of the 3.
I did a few test readings on a 15A circuit that already has an SCT-013-000 (with a 140 Ohm burden) on it that has a fairly steady load on it. I got .208V for the SCT-013 and .045V for the TED CT. I also tested the two mains coming in and I think the voltages were around .8V on one and .4V on the other.
Before I design further tests to characterize this CT, I think I am going to contact TED and see if they have a data sheet on this CT. That would make it very clear.
Re: CT Question
"Won't he still need the voltage divider to centre the swing, i.e. to stop it swinging below 0V?"
The way Tom phrased his question with the numbers he gave, I read it that he intended to divide down the output voltage of the CT. I think you're thinking of the bias offset that provides a centre rail of 1.65 V dc.
Tom, there's something seriously wrong with your numbers. The SCT-013-000 gives you 50 mA per 100 A, i.e. 7.5 mA at 15 A. That into a 140 Ω burden should give you 1050 mV rms. If you have the other 200 A CT that should give you 3 V rms, then that too doesn't make sense, because you'll get 0.6 V at 200 A with your numbers.
Re: CT Question
Robert,
Sorry - I didn't mean to imply the circuit was maxed out at 15A, the load there was probably around 350W or so, giving a current of almost 3 amps which would yield a voltage across the burden resistor of around 200mV. How did you calculate the value for the 200A CT? Wouldn't it just be the ratio of input amps (15 in your calculations i assume) divided by the max current (200) multiplied by the rated voltage output? If so, then I compute 225mV @ 15A and 45mV with the 3 amps current I was measuring. That number does match what I read.
Re: CT Question
When you wrote a 15 A circuit with a constant load, that meant a constant 15 A to me. You must know the current before you can do anything meaningful.
Yes, taking the current as 15 A, and if for 200 A you expect 3 V, then I expect 225 mV as you say. But I read it as you got 45 mV at 15 A, so taken together with the YHDC numbers, my suspicion was your voltage readings were way out.
If your voltage reading, burden value and YHDC CT are all accurate, your current was 2.971 A, which is close to your estimate. (And from that I assume you're on a 120 V system.)
Re: CT Question
Thanks for all the help. These lead me to two more questions:
1. Without knowing the specs for the CT, is there a way to pick the best burden resistor size to add?
2. Would using a voltage divider work? It seems to be easier that trying to figure out the "best" burden resistor to add. I was thinking of this (and choosing values for R1 & R2 to give the appropriate maximum voltage)
2. Am I correct in calculating that I will need to bring the max voltage output of the CT to about 1.16V (3.3V / 2 / 1.414)?
3. what value should I start with for ICAL in the emonlib code?
Re: CT Question
The short answers:
1. No.
2.1. Yes.
2.2. No, a little below.
3. About 200, but adjust it on test.
The long answers.
1. You need to know the CT ratio. Normally, the ratio is defined in terms of the maximum primary current and the corresponding secondary current. In the case of a CT with a built-in burden, it will be the maximum primary current and the corresponding output voltage, always in rms values. DO NOT trust a CT where the ratio is specified as a number without units, because you cannot know where it starts to saturate. Then you need to check that the voltage you will develop is within the VA rating, which is the maximum power you can get through the CT, again before saturation becomes troublesome. It's very rare for the VA rating to be specified directly for these small CTs. Often, it is implied with a recommended burden value. Sometimes, there is absolutely no information whatsoever, and you have to test it.
2.1. Putting a second burden resistor in parallel would reduce the load on the CT and therefore improve the accuracy. It's a moot point whether you'd notice the difference in practice, except that it's one fewer resistor with it's inherent inaccuracy. It would not be hard to determine the internal burden value: the CT is a current source, so if you have a fixed constant primary current, measure the voltage with and without a parallel burden resistor. With it, the current shares between both. Without it, all the current flows entirely in the internal burden, but again you know the voltage hence you have two simultaneous equations and you can get to the internal burden resistor's value. Doing the maths, its: second burden × ((unloaded voltage/loaded voltage) - 1)
2.2. You should aim a few percent at least below the maximum. Any error in the centre voltage, burden value, CT inaccuracy, supply voltage etc can all have the effect of reducing the available swing by a few percent, but the biggest factor you need to worry about is the wave shape. If your current is a perfect sine wave, your √2 is OK, but if the wave is a bit 'peaky' (crest factor > √2), then the peak value will be higher, possibly quite a bit higher, than the rms value would lead you to believe. I usually recommend aiming for not more than 1.1 V rms, which gives you a bit less than 6% for the combined effect of all the factors, worst case. It's likely that your biggest loads will distort the waveform the least, therefore I'd expect at maximum current the waveform would be quite good, so 'peakiness' won't be a major factor.
3. From the CT and AC power adaptor installation and calibration theory page in Building Blocks, the value should be the ratio (primary current) / (the voltage at that current). In your case, the voltage is the final alternating voltage into the ADC, so it will be between about 1 and 1.1 V at 200 A, therefore between about 180 and 200, or if you stay with your 1.16 V, 172.41.
Re: CT Question
1. Given he's trying to reduce the voltage out of the CT, won't that mean he's heading away from any saturation issues? If you unclamp the CT so it's not trying to generate any voltage, and measure it's resistance, you'll have a rough idea of its internal resistance (burden and secondary windings). That would be a good starting point for your parallel burden resistor calculation. Provided you stay out of any saturation zones, then it should all be nice and linear, so with a known primary load you just need to plug in 2 different parallel burdens to get you 2 points on the line. From that, you should be able to choose a value that will give you your desired maximum voltage at maximum primary current.
2.1 One other disadvantage of the divider is that it increases the source impedance of your signal, which may interact with your AtoD converter. If you go low with R1+R2 to avoid that, then R1+R2 effectively become a parallel burden resistor and you're back having to solve the problem you were trying to avoid (#1)
Re: CT Question
"1. Given he's trying to reduce the voltage out of the CT, won't that mean he's heading away from any saturation issues?" No he isn't doing that - with an external voltage divider, the CT is generating exactly the same voltage and transferring exactly the same power into its burden (given a suitably high value divider chain). So the operating point of the CT is not changed.
Measuring the resistance of the parallel combination of secondary winding and burden isn't likely to help much, the secondary is probably about 10 × the burden, so you might get within 10%. And don't expect a standard value, the burden value may well be adjusted on test to calibrate the output.
2. If the Thévenin equivalent impedance of the divider is less than 10 kΩ (from memory - check the data sheet), it won't affect the conversion. But I agree - it's better avoided and just add a parallel burden.
As I wrote, it's not difficult to deduce the value of the burden resistor with a known resistor and two voltage readings. A possibly simpler and more direct variation of the same method: measure the secondary voltage and then add parallel resistors until the voltage halves. The parallel resistance that you added is the value of the internal burden (this follows from the equation I gave above).
Re: CT Question
"1. Given he's trying to reduce the voltage out of the CT..."
What I meant to say is "Given he's trying to reduce the voltage swing into his ADC input"
As I wrote, it's not difficult to deduce the value of the burden resistor
Exactly. Somehow I completely missed that part of your answer, and found your short answer 1 pretty pessimistic. I took your short answers to mean you were favoring the voltage divider over the parallel burden resistor.
But I agree - it's better avoided and just add a parallel burden.
Good. I think we're all in violent agreement then.
Re: CT Question
Again, thanks for all the info! Didn't mean to start a fight...
I will be testing my CT as soon as I get the chance to determine the internal burden resistor vale and to really verify it is 3V. After refreshing myself on series and parallel circuits (last time I used this stuff was more years ago than I care to admit), I was going to do exactly what was suggested to determine the burden resistor value. As far as my target voltage - I was going to make sure it was well under the theoretical value I stated (thanks for the suggested values).
I will ditch the voltage divider and opt for the second burden resistor as that seems to be the best option.
Re: CT Question
Tom: Don't worry, I don't think you started a fight. Because current sources are a lot less common than voltage sources, a lot of people (I'm not saying you!) find it hard to flip their thinking around when working with CTs, that's why I spelled out in some detail (not totally complete because I didn't have the time to do diagrams etc and write out all the equations) the way to find the internal burden value. Of course when you have that, the CT current ratio is easy to determine, provided you know the primary current - which you don't need just to determine the burden value as long as you know it's constant. But that won't tell you where you lose linearity as it starts to saturate. I know that's not relevant to you as you know that, but someone reading this later might not be in that position.
dBC: "and found your short answer 1 pretty pessimistic." Too right it is, but that's the way some of the cheap Fleabay advertisers, and some not-quite-so-cheap advertisers, work. If you buy a CT or anything on the basis of an incomplete spec, it's nobody else's fault if it isn't up to the job. It's much better to know there might be a problem so that you can go to a diligent seller instead. Forewarned is forearmed, or something like that.
Re: CT Question
Robert: I meant that tongue-in-cheek about the argument. I hate when that doesn't come across in text.


Vernazza Video Player is a video player for Android phones and tablets.
It allows the user to navigate a video clip by scrolling through a list of key frames.
These key frames are listed at a fixed time interval (default is every 10 seconds).
Download the following to your device for local playback.
(Network streaming is not supported yet).
Test Video #1 (mkv) Test Video #2 (mkv)
Hi! My name is Peter Tran and I'm a software developer based in San Diego. I have a background in Linux device drivers, digital TV and settop boxes. I wanted to dive into Android development so Vernazza Video Player was born. Send me any questions or commments at twitter.com/manhdev.


New Update
Galaxy Note 10 gets its second Android 10 beta, Note 9 to receive first beta soon
Now rolling out in South Korea and India.
by Babu MohanWhat you need to know
- The second Android 10 beta update for the Galaxy Note 10 and Note 10+ has started rolling out.
- Samsung may release the first Android 10 beta build for the Galaxy Note 9 later this week.
- Both the Galaxy S10 and Note 10 series are expected to receive the stable Android 10 update before the end of the year.
Following the release of the fourth and final Android 10 beta update for the Galaxy S10 series last week, Samsung today rolled out the second Android 10 beta for the Galaxy Note 10 series. The update is currently available (via TizenHelp) for users in South Korea and India. It is soon expected to be available for download in the U.S. as well.
The second Android 10 beta build for the Galaxy Note 10 and Note 10+ brings numerous bug fixes, improved gesture interactions, improved Bluetooth stability, and a few other enhancements. You can download the latest Android 10-based One UI 2.0 beta on your Galaxy Note 10 or Note 10+ by heading over to Settings > Software update.
While the beta program for the Galaxy S10 series is expected to be closed soon, a Samsung Community Manager has claimed in a post on the company's South Korean Members Community forum that the Galaxy Note 9 will receive its first Android 10 beta later this week. The Samsung official also revealed that the Galaxy S9 and S9+ are slated to get their first Android 10 beta update next week.
Even though Samsung hasn't officially confimed anything yet, it now seems highly likely that the stable Android 10 update for the Galaxy S10 series will be rolled out before 2019 comes to an end. The Galaxy Note 10 and Note 10+ may also get the stable One UI 2.0 update soon after the rollout begins for the Galaxy S10 series.
Get More Galaxy Note 10
Samsung Galaxy Note 10/10+
- Galaxy Note 10+ Review
- Galaxy Note 10 Review
- Best Galaxy Note 10+ Cases
- Best Galaxy Note 10 Cases
- Best Galaxy Note 10+ Screen Protectors
- Best Galaxy Note 10 Screen Protectors
We may earn a commission for purchases using our links. Learn more.


Posted by The Minstrel from:<IP_ADDRESS> on October 06, 1999 at 20:39:07:
In Reply to: ATTAC v2 Helper posted by John Pritchett (EIS) on October 06, 1999 at 12:18:04:
: The new author of ATTAC (Zev Beckerman) has finished the version
that
: is compatible with TWGS/TWv3. I offered to help get the word out
about
: his new helper, but I haven't had time to put anything up on my web
site.
: Until that time, I figured I'd just post an announcement on the forum.
: If anyone has some positive feedback or constructive criticism, bug
reports
: or whatever, post them. Zev reads the forum and I'm sure he'd appreciate
: anything you have to say.
And a mighty fine helper it is too (just had to throw that one in)!
Seriously though, Zev has done an excellent job turning what was a pretty
poor helper into a good and worthwhile product. If you haven't seen
ATTAC
before or abandoned it in disgust back during the 1.0x series you really
should download and check out version 2.01.


PUT IT PLAQUE
Paisley widow blasts crematorium bosses who removed her late hubby’s pic from memorial plaque
by Chris TaylorA WIDOW has hit out at crematorium bosses who removed her late hubby’s pic from a memorial plaque.
Grieving Mary Webb, 63, was stunned when told the memento of her beloved Graham was among scores ripped off to allow the stone to be cleaned.
And she says she was “disgusted” when officials revealed she’d have to rake through photos of dead strangers to find it again.
The ordeal came a year after the retired chef died of lung cancer aged 67 and was laid to rest at Woodside cremmy in Paisley.
Mary, of the town, had placed a pic from their engagement party 41 years ago on to a tribute message on a wall in the site’s memorial garden.
She explained: “Recently a friend who was at a funeral there told me Graham’s photograph was gone. I rushed over and it was away.
“I was told it was for cleaning but they’ve not been returned.
"I phoned them and they said I could trawl through all the pictures they’d taken.
“They are families’ personal memories, so nobody has any right to go through them.
“I was also told it would cost £50 to put up a pic on the wall. I was utterly disgusted.
“Now I have nowhere to grieve my husband. I just want to kiss his photo and say goodnight.”
Funeral directors Fosters, who took over the centre in June, charge £199 for a 4in x 10in plaque.
Signs warn “personal tributes” will be removed.
A spokesman said: “It was never our intention to cause any upset.
“We’re putting in a new records system to allow staff to locate any such items for families in future.”


I'm going to be upfront. I asked a college from my old job, a trusted friend and my mum. Only my mum got back to me on this. It sucks but it is what it is, I will update this when I get around to ask others who I think would actually provide me with an answer. My personal values are pretty straight forward. Integrity, honesty, kindness, respect etc.But then again I do have a side from my teenage years which rebelled and butts heads when people are being inconsiderate or rude.
A lot of what I haven learned in my life has been through hard lessons, by experiencing myself or observing others.Something that haunted a 5 year old Steven for longer than I’d like to admit is.I stole a kids HotWheels car, he was a couple years older than me and when he found out basically blackmailed me by threatening me to give him my remaining HotWheels cars.People can tell you right from wrong, but it’s up to you if you actually take that on boarder not.Thankfully this was something that set in at a young age.
Like everyone else, I am NOT perfect. There is always room for improvement.I can have a hot head at times and not get be level headed, flustered and deflect on others.All things I try and slow myself down and work towards changing each day.
But, I like people, including and learning about others, being kind, uplifting those around me, although I have a loud voice and am tall enough to notice half was down the block I try be respectful of others space.
I come from a household with 2 parents that worked themselves to the brink of death to provide a safe home with a roof over mine and my sisters heads. This had ultimately translated to myself at 12-13 starting part time work and going right into working as soon as I left college, not wanting to be a burden. So far I have been working full time for 7 years I’ve been faced with my fair share of difficult times.
I like teams and team building, being able to know your colleges weaknesses and being able to offer help when and where it will be useful helps so much in a work environment, especially if they don’t know how to ask for help or struggle to, like myself.
Home


Getting Your Menu Item to Load Your Stage
Now that you have defined the layout of your stage, you can tell your menu item to load it instead of the master view that it is loading currently.
Steps:
- Open your menu file (menu_30_Db2authLab.xml) with an editor
- Make the changes to your document as indicated on the right
- Go to File->'Reload Menu' within the Technology Explorer, not your browser, and check your work


Binvoice-Sign Up
BINVOICE
BILL. COLLECT. CHILL.
Home
Account
Create Invoice
About Us/Help
Log Out
Sign up for Binvoice
Email address:
Password:
Name:
Business Name:
Address:
Phone Number:
Error:
Sign Up
Or log in
here


Pricing is as follows: Small: $10 Medium: $12 Large: $14 Cheese is included as a topping, and each additional topping is just $1.00
Size:
Dough:
Toppings:
Name:
Cost:


Lisa G Spear
Unafffiliated
I am not currently employed outside the home; I generally respond "Homemaker." I have not been employed outside the home over the last 10 years.
1979: Graduated Cum Laude, Marist College, Poughkeepsie, NY, BA American Studies; 1986-88 1.5 years of law school , University of Denver
80222
As a precinct leader/captain I organized volunteers within my precincts to knock on doors, drop literature and canvas registered voters. In the Article V groups I have acted as scribe as well as assisted in editing the newsletter and in presenting the groups at local assemblies of politically active individuals.
I started political activity in 1978 working as an exit pollster for the very first Marist Poll. In 1979, I interned in the office of NY State Sen Howard Nolan (Albany District). After moving to CO, I worked on several federal and state candidate campaigns, and some issue campaigns, as a volunteer, and worked as a precinct leader/captain in Denver and Arapahoe counties. I have worked as an election judge several times, the last time in 2016 as supervising judge. I even organized a candidate forum for candidates for Judicial District 18. As far as monetary donations since 2015, I have donated to George Brauchler's first campaign, and to CatholiVote.org.
Since 2015, I have been a member of the Colorado Committee of Correspondence (an Article V advocacy group) and have recently been affiliated with the State Legislators Article V Caucus. I am a past member of my parish's pastoral council. I am a dues-paying member of the Cherry Creek Republican Women, and occasionally attend their meetings.
I want to serve on the commission because I was a supporter of instituting of this commission and the reapportionment commission. I was very disappointed in our legislators in being unable to work for the good of the State in performing their duty to redraw these districts following two previous censuses. I don't believe in the concept of "safe seats", as the seats in the Federal and State governments don't belong to a party or a politician; the "seats" belong to the people, and the people deserve better than what the legislatures have done by deferring their responsibility to the courts.
I can promote consensus among the commissioners because I have very good interactive listening skills. Also, I have been a registered voter in both of the major parties in my voting lifetime, even though I choose not to affiliate with either at this time. As far as being fair and impartial, I have served on a jury in the past (regrettably, not in the recent past), and understand that I have to hear all viewpoints before making a commitment to a position.
In my professional work I was the Director of Technical Services for an international life reinsurer. Analytical skills and the ability to apply logic and reason to solve problems or support new products were critical skills for me to possess. I utilized those skills every day through project management, coordination among interested departments, and in personnel management within projects.
Female