The File System (FS) shell includes various shell-like commands that directly interact with the Hadoop Distributed File System (HDFS) as well as other file systems that Hadoop supports, such as Local FS, WebHDFS, S3 FS, and others. The FS shell is invoked by:
bin/hadoop fs <args>
All FS shell commands take path URIs as arguments. The URI format is scheme://authority/path
. For HDFS the scheme is hdfs
, and for the Local FS the scheme is file
. The scheme and authority are optional. If not specified, the default scheme specified in the configuration is used. An HDFS file or directory such as /parent/child can be specified as hdfs://namenodehost/parent/child
or simply as /parent/child
(given that your configuration is set to point to hdfs://namenodehost
).
Most of the commands in FS shell behave like corresponding Unix commands. Differences are described with each of the commands. Error information is sent to stderr and the output is sent to stdout.
If HDFS is being used, hdfs dfs
is a synonym.
Relative paths can be used. For HDFS, the current working directory is the HDFS home directory /user/<username>
that often has to be created manually. The HDFS home directory can also be implicitly accessed, e.g., when using the HDFS trash folder, the .Trash
directory in the home directory.
See the Commands Manual for generic shell options.
Usage: hadoop fs -appendToFile <localsrc> ... <dst>
Append single src, or multiple srcs from local file system to the destination file system. Also reads input from stdin and appends to destination file system.
hadoop fs -appendToFile localfile /user/hadoop/hadoopfile
hadoop fs -appendToFile localfile1 localfile2 /user/hadoop/hadoopfile
hadoop fs -appendToFile localfile hdfs://nn.example.com/hadoop/hadoopfile
hadoop fs -appendToFile - hdfs://nn.example.com/hadoop/hadoopfile
Reads the input from stdin.Exit Code:
Returns 0 on success and 1 on error.
Usage: hadoop fs -cat [-ignoreCrc] URI [URI ...]
Copies source paths to stdout.
Options
-ignoreCrc
option disables checkshum verification.Example:
hadoop fs -cat hdfs://nn1.example.com/file1 hdfs://nn2.example.com/file2
hadoop fs -cat file:///file3 /user/hadoop/file4
Exit Code:
Returns 0 on success and -1 on error.
Usage: hadoop fs -checksum [-v] URI
Returns the checksum information of a file.
Options
-v
option displays blocks size for the file.Example:
hadoop fs -checksum hdfs://nn1.example.com/file1
hadoop fs -checksum file:///etc/hosts
Usage: hadoop fs -chgrp [-R] GROUP URI [URI ...]
Change group association of files. The user must be the owner of files, or else a super-user. Additional information is in the Permissions Guide.
Options
Usage: hadoop fs -chmod [-R] <MODE[,MODE]... | OCTALMODE> URI [URI ...]
Change the permissions of files. With -R, make the change recursively through the directory structure. The user must be the owner of the file, or else a super-user. Additional information is in the Permissions Guide.
Options
Usage: hadoop fs -chown [-R] [OWNER][:[GROUP]] URI [URI ]
Change the owner of files. The user must be a super-user. Additional information is in the Permissions Guide.
Options
Identical to the -put command.
Identical to the -get command.
Usage: hadoop fs -count [-q] [-h] [-v] [-x] [-t [<storage type>]] [-u] [-e] [-s] <paths>
Count the number of directories, files and bytes under the paths that match the specified file pattern. Get the quota and the usage. The output columns with -count are: DIR_COUNT, FILE_COUNT, CONTENT_SIZE, PATHNAME
The -u and -q options control what columns the output contains. -q means show quotas, -u limits the output to show quotas and usage only.
The output columns with -count -q are: QUOTA, REMAINING_QUOTA, SPACE_QUOTA, REMAINING_SPACE_QUOTA, DIR_COUNT, FILE_COUNT, CONTENT_SIZE, PATHNAME
The output columns with -count -u are: QUOTA, REMAINING_QUOTA, SPACE_QUOTA, REMAINING_SPACE_QUOTA, PATHNAME
The -t option shows the quota and usage for each storage type. The -t option is ignored if -u or -q option is not given. The list of possible parameters that can be used in -t option(case insensitive except the parameter ""): "", “all”, “ram_disk”, “ssd”, “disk” or “archive”.
The -h option shows sizes in human readable format.
The -v option displays a header line.
The -x option excludes snapshots from the result calculation. Without the -x option (default), the result is always calculated from all INodes, including all snapshots under the given path. The -x option is ignored if -u or -q option is given.
The -e option shows the erasure coding policy for each file.
The output columns with -count -e are: DIR_COUNT, FILE_COUNT, CONTENT_SIZE, ERASURECODING_POLICY, PATHNAME
The ERASURECODING_POLICY is name of the policy for the file. If a erasure coding policy is setted on that file, it will return name of the policy. If no erasure coding policy is setted, it will return "Replicated" which means it use replication storage strategy.
The -s option shows the snapshot counts for each directory.
Example:
hadoop fs -count hdfs://nn1.example.com/file1 hdfs://nn2.example.com/file2
hadoop fs -count -q hdfs://nn1.example.com/file1
hadoop fs -count -q -h hdfs://nn1.example.com/file1
hadoop fs -count -q -h -v hdfs://nn1.example.com/file1
hadoop fs -count -u hdfs://nn1.example.com/file1
hadoop fs -count -u -h hdfs://nn1.example.com/file1
hadoop fs -count -u -h -v hdfs://nn1.example.com/file1
hadoop fs -count -e hdfs://nn1.example.com/file1
hadoop fs -count -s hdfs://nn1.example.com/file1
Exit Code:
Returns 0 on success and -1 on error.
Usage: hadoop fs -cp [-f] [-p | -p[topax]] [-t <thread count>] [-q <thread pool queue size>] URI [URI ...] <dest>
Copy files from source to destination. This command allows multiple sources as well in which case the destination must be a directory.
‘raw.*’ namespace extended attributes are preserved if (1) the source and destination filesystems support them (HDFS only), and (2) all source and destination pathnames are in the /.reserved/raw hierarchy. Determination of whether raw.* namespace xattrs are preserved is independent of the -p (preserve) flag.
Options:
-f
: Overwrite the destination if it already exists.-d
: Skip creation of temporary file with the suffix ._COPYING_
.-p
: Preserve file attributes [topx](timestamps, ownership, permission, ACL, XAttr). If -p is specified with no arg, then preserves timestamps, ownership, permission. If -pa is specified, then preserves permission also because ACL is a super-set of permission. Determination of whether raw namespace extended attributes are preserved is independent of the -p flag.-t <thread count>
: Number of threads to be used, default is 1. Useful when copying directories containing more than 1 file.-q <thread pool queue size>
: Thread pool queue size to be used, default is 1024. It takes effect only when thread count greater than 1.Example:
hadoop fs -cp /user/hadoop/file1 /user/hadoop/file2
hadoop fs -cp /user/hadoop/file1 /user/hadoop/file2 /user/hadoop/dir
hadoop fs -cp -t 5 /user/hadoop/file1 /user/hadoop/file2 /user/hadoop/dir
hadoop fs -cp -t 10 -q 2048 /user/hadoop/file1 /user/hadoop/file2 /user/hadoop/dir
Exit Code:
Returns 0 on success and -1 on error.
See HDFS Snapshots Guide.
See HDFS Snapshots Guide.
Usage: hadoop fs -df [-h] URI [URI ...]
Displays free space.
Options:
Example:
hadoop dfs -df /user/hadoop/dir1
Usage: hadoop fs -du [-s] [-h] [-v] [-x] URI [URI ...]
Displays sizes of files and directories contained in the given directory or the length of a file in case its just a file.
Options:
The du returns three columns with the following format:
size disk_space_consumed_with_all_replicas full_path_name
Example:
hadoop fs -du /user/hadoop/dir1 /user/hadoop/file1 hdfs://nn.example.com/user/hadoop/dir1
Exit Code: Returns 0 on success and -1 on error.
Usage: hadoop fs -dus <args>
Displays a summary of file lengths.
Note: This command is deprecated. Instead use hadoop fs -du -s
.
Usage: hadoop fs -expunge [-immediate] [-fs <path>]
Permanently delete files in checkpoints older than the retention threshold from trash directory, and create new checkpoint.
When checkpoint is created, recently deleted files in trash are moved under the checkpoint. Files in checkpoints older than fs.trash.interval
will be permanently deleted on the next invocation of -expunge
command.
If the file system supports the feature, users can configure to create and delete checkpoints periodically by the parameter stored as fs.trash.checkpoint.interval
(in core-site.xml). This value should be smaller or equal to fs.trash.interval
.
If the -immediate
option is passed, all files in the trash for the current user are immediately deleted, ignoring the fs.trash.interval
setting.
If the -fs
option is passed, the supplied filesystem will be expunged, rather than the default filesystem and checkpoint is created.
For example
hadoop fs -expunge --immediate -fs s3a://landsat-pds/
Refer to the HDFS Architecture guide for more information about trash feature of HDFS.
Usage: hadoop fs -find <path> ... <expression> ...
Finds all files that match the specified expression and applies selected actions to them. If no path is specified then defaults to the current working directory. If no expression is specified then defaults to -print.
The following primary expressions are recognised:
-name pattern
-iname pattern
Evaluates as true if the basename of the file matches the pattern using standard file system globbing. If -iname is used then the match is case insensitive.
-print
-print0
Always evaluates to true. Causes the current pathname to be written to standard output. If the -print0 expression is used then an ASCII NULL character is appended.
The following operators are recognised:
Logical AND operator for joining two expressions. Returns true if both child expressions return true. Implied by the juxtaposition of two expressions and so does not need to be explicitly specified. The second expression will not be applied if the first fails.
Example:
hadoop fs -find / -name test -print
Exit Code:
Returns 0 on success and -1 on error.
Usage: hadoop fs -get [-ignorecrc] [-crc] [-p] [-f] [-t <thread count>] [-q <thread pool queue size>] <src> ... <localdst>
Copy files to the local file system. Files that fail the CRC check may be copied with the -ignorecrc option. Files and CRCs may be copied using the -crc option.
Options:
-p
: Preserves access and modification times, ownership and the permissions. (assuming the permissions can be propagated across filesystems)-f
: Overwrites the destination if it already exists.-ignorecrc
: Skip CRC checks on the file(s) downloaded.-crc
: write CRC checksums for the files downloaded.-t <thread count>
: Number of threads to be used, default is 1. Useful when downloading directories containing more than 1 file.-q <thread pool queue size>
: Thread pool queue size to be used, default is 1024. It takes effect only when thread count greater than 1.Example:
hadoop fs -get /user/hadoop/file localfile
hadoop fs -get hdfs://nn.example.com/user/hadoop/file localfile
hadoop fs -get -t 10 hdfs://nn.example.com/user/hadoop/dir1 localdir
hadoop fs -get -t 10 -q 2048 hdfs://nn.example.com/user/hadoop/dir* localdir
Exit Code:
Returns 0 on success and -1 on error.
Usage: hadoop fs -getfacl [-R] <path>
Displays the Access Control Lists (ACLs) of files and directories. If a directory has a default ACL, then getfacl also displays the default ACL.
Options:
Examples:
hadoop fs -getfacl /file
hadoop fs -getfacl -R /dir
Exit Code:
Returns 0 on success and non-zero on error.
Usage: hadoop fs -getfattr [-R] -n name | -d [-e en] <path>
Displays the extended attribute names and values (if any) for a file or directory.
Options:
Examples:
hadoop fs -getfattr -d /file
hadoop fs -getfattr -R -n user.myAttr /dir
Exit Code:
Returns 0 on success and non-zero on error.
Usage: hadoop fs -getmerge [-nl] <src> <localdst>
Takes a source directory and a destination file as input and concatenates files in src into the destination local file. Optionally -nl can be set to enable adding a newline character (LF) at the end of each file. -skip-empty-file can be used to avoid unwanted newline characters in case of empty files.
Examples:
hadoop fs -getmerge -nl /src /opt/output.txt
hadoop fs -getmerge -nl /src/file1.txt /src/file2.txt /output.txt
Exit Code:
Returns 0 on success and non-zero on error.
Usage: hadoop fs -head URI
Displays first kilobyte of the file to stdout.
Example:
hadoop fs -head pathname
Exit Code: Returns 0 on success and -1 on error.
Usage: hadoop fs -help
Return usage output.
Usage: hadoop fs -ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [-e] <args>
Options:
For a file ls returns stat on the file with the following format:
permissions number_of_replicas userid groupid filesize modification_date modification_time filename
For a directory it returns list of its direct children as in Unix. A directory is listed as:
permissions userid groupid modification_date modification_time dirname
Files within a directory are order by filename by default.
Example:
hadoop fs -ls /user/hadoop/file1
hadoop fs -ls -e /ecdir
Exit Code:
Returns 0 on success and -1 on error.
Usage: hadoop fs -lsr <args>
Recursive version of ls.
Note: This command is deprecated. Instead use hadoop fs -ls -R
Usage: hadoop fs -mkdir [-p] <paths>
Takes path uri’s as argument and creates directories.
Options:
Example:
hadoop fs -mkdir /user/hadoop/dir1 /user/hadoop/dir2
hadoop fs -mkdir hdfs://nn1.example.com/user/hadoop/dir hdfs://nn2.example.com/user/hadoop/dir
Exit Code:
Returns 0 on success and -1 on error.
Usage: hadoop fs -moveFromLocal <localsrc> <dst>
Similar to put command, except that the source localsrc is deleted after it’s copied.
Usage: hadoop fs -moveToLocal [-crc] <src> <dst>
Displays a “Not implemented yet” message.
Usage: hadoop fs -mv URI [URI ...] <dest>
Moves files from source to destination. This command allows multiple sources as well in which case the destination needs to be a directory. Moving files across file systems is not permitted.
Example:
hadoop fs -mv /user/hadoop/file1 /user/hadoop/file2
hadoop fs -mv hdfs://nn.example.com/file1 hdfs://nn.example.com/file2 hdfs://nn.example.com/file3 hdfs://nn.example.com/dir1
Exit Code:
Returns 0 on success and -1 on error.
Usage: hadoop fs -put [-f] [-p] [-l] [-d] [-t <thread count>] [-q <thread pool queue size>] [ - | <localsrc> ...] <dst>
Copy single src, or multiple srcs from local file system to the destination file system. Also reads input from stdin and writes to destination file system if the source is set to “-”
Copying fails if the file already exists, unless the -f flag is given.
Options:
-p
: Preserves access and modification times, ownership and the permissions. (assuming the permissions can be propagated across filesystems)-f
: Overwrites the destination if it already exists.-l
: Allow DataNode to lazily persist the file to disk, Forces a replication factor of 1. This flag will result in reduced durability. Use with care.-d
: Skip creation of temporary file with the suffix ._COPYING_
.-t <thread count>
: Number of threads to be used, default is 1. Useful when uploading directories containing more than 1 file.-q <thread pool queue size>
: Thread pool queue size to be used, default is 1024. It takes effect only when thread count greater than 1.Examples:
hadoop fs -put localfile /user/hadoop/hadoopfile
hadoop fs -put -f localfile1 localfile2 /user/hadoop/hadoopdir
hadoop fs -put -d localfile hdfs://nn.example.com/hadoop/hadoopfile
hadoop fs -put - hdfs://nn.example.com/hadoop/hadoopfile
Reads the input from stdin.hadoop fs -put -t 5 localdir hdfs://nn.example.com/hadoop/hadoopdir
hadoop fs -put -t 10 -q 2048 localdir1 localdir2 hdfs://nn.example.com/hadoop/hadoopdir
Exit Code:
Returns 0 on success and -1 on error.
See HDFS Snapshots Guide.
Usage: hadoop fs -rm [-f] [-r |-R] [-skipTrash] [-safely] URI [URI ...]
Delete files specified as args.
If trash is enabled, file system instead moves the deleted file to a trash directory (given by FileSystem#getTrashRoot).
Currently, the trash feature is disabled by default. User can enable trash by setting a value greater than zero for parameter fs.trash.interval
(in core-site.xml).
See expunge about deletion of files in trash.
Options:
hadoop.shell.delete.limit.num.files
(in core-site.xml, default: 100). It can be used with -skipTrash to prevent accidental deletion of large directories. Delay is expected when walking over large directory recursively to count the number of files to be deleted before the confirmation.Example:
hadoop fs -rm hdfs://nn.example.com/file /user/hadoop/emptydir
Exit Code:
Returns 0 on success and -1 on error.
Usage: hadoop fs -rmdir [--ignore-fail-on-non-empty] URI [URI ...]
Delete a directory.
Options:
--ignore-fail-on-non-empty
: When using wildcards, do not fail if a directory still contains files.Example:
hadoop fs -rmdir /user/hadoop/emptydir
Usage: hadoop fs -rmr [-skipTrash] URI [URI ...]
Recursive version of delete.
Note: This command is deprecated. Instead use hadoop fs -rm -r
Usage: hadoop fs -setfacl [-R] [-b |-k -m |-x <acl_spec> <path>] |[--set <acl_spec> <path>]
Sets Access Control Lists (ACLs) of files and directories.
Options:
--set
: Fully replace the ACL, discarding all existing entries. The acl_spec must include entries for user, group, and others for compatibility with permission bits. If the ACL spec contains only access entries, then the existing default entries are retained. If the ACL spec contains only default entries, then the existing access entries are retained. If the ACL spec contains both access and default entries, then both are replaced.Examples:
hadoop fs -setfacl -m user:hadoop:rw- /file
hadoop fs -setfacl -x user:hadoop /file
hadoop fs -setfacl -b /file
hadoop fs -setfacl -k /dir
hadoop fs -setfacl --set user::rw-,user:hadoop:rw-,group::r--,other::r-- /file
hadoop fs -setfacl -R -m user:hadoop:r-x /dir
hadoop fs -setfacl -m default:user:hadoop:r-x /dir
Exit Code:
Returns 0 on success and non-zero on error.
Usage: hadoop fs -setfattr -n name [-v value] | -x name <path>
Sets an extended attribute name and value for a file or directory.
Options:
Examples:
hadoop fs -setfattr -n user.myAttr -v myValue /file
hadoop fs -setfattr -n user.noValue /file
hadoop fs -setfattr -x user.myAttr /file
Exit Code:
Returns 0 on success and non-zero on error.
Usage: hadoop fs -setrep [-R] [-w] <numReplicas> <path>
Changes the replication factor of a file. If path is a directory then the command recursively changes the replication factor of all files under the directory tree rooted at path. The EC files will be ignored when executing this command.
Options:
Example:
hadoop fs -setrep -w 3 /user/hadoop/dir1
Exit Code:
Returns 0 on success and -1 on error.
Usage: hadoop fs -stat [format] <path> ...
Print statistics about the file/directory at <path> in the specified format. Format accepts permissions in octal (%a) and symbolic (%A), filesize in bytes (%b), type (%F), group name of owner (%g), name (%n), block size (%o), replication (%r), user name of owner(%u), access date(%x, %X), and modification date (%y, %Y). %x and %y show UTC date as “yyyy-MM-dd HH:mm:ss”, and %X and %Y show milliseconds since January 1, 1970 UTC. If the format is not specified, %y is used by default.
Example:
hadoop fs -stat "type:%F perm:%a %u:%g size:%b mtime:%y atime:%x name:%n" /file
Exit Code: Returns 0 on success and -1 on error.
Usage: hadoop fs -tail [-f] URI
Displays last kilobyte of the file to stdout.
Options:
Example:
hadoop fs -tail pathname
Exit Code: Returns 0 on success and -1 on error.
Usage: hadoop fs -test -[defswrz] URI
Options:
Example:
hadoop fs -test -e filename
Usage: hadoop fs -text <src>
Takes a source file and outputs the file in text format. The allowed formats are zip and TextRecordInputStream.
Usage: hadoop fs -touch [-a] [-m] [-t TIMESTAMP] [-c] URI [URI ...]
Updates the access and modification times of the file specified by the URI to the current time. If the file does not exist, then a zero length file is created at URI with current time as the timestamp of that URI.
The timestamp format is as follows * yyyy Four digit year (e.g. 2018) * MM Two digit month of the year (e.g. 08 for month of August) * dd Two digit day of the month (e.g. 01 for first day of the month) * HH Two digit hour of the day using 24 hour notation (e.g. 23 stands for 11 pm, 11 stands for 11 am) * mm Two digit minutes of the hour * ss Two digit seconds of the minute e.g. 20180809:230000 represents August 9th 2018, 11pm
Example:
hadoop fs -touch pathname
hadoop fs -touch -m -t 20180809:230000 pathname
hadoop fs -touch -t 20180809:230000 pathname
hadoop fs -touch -a pathname
Exit Code: Returns 0 on success and -1 on error.
Usage: hadoop fs -touchz URI [URI ...]
Create a file of zero length. An error is returned if the file exists with non-zero length.
Example:
hadoop fs -touchz pathname
Exit Code: Returns 0 on success and -1 on error.
Usage: hadoop fs -truncate [-w] <length> <paths>
Truncate all files that match the specified file pattern to the specified length.
Options:
Example:
hadoop fs -truncate 55 /user/hadoop/file1 /user/hadoop/file2
hadoop fs -truncate -w 127 hdfs://nn1.example.com/user/hadoop/file1
Usage: hadoop fs -concat <target file> <source files>
Concatenate existing source files into the target file. Target file and source files should be in the same directory.
Example:
hadoop fs -concat hdfs://cluster/user/hadoop/target-file hdfs://cluster/user/hadoop/file-0 hdfs://cluster/user/hadoop/file-1
Usage: hadoop fs -usage command
Return the help for an individual command.
The Hadoop FileSystem shell works with Object Stores such as Amazon S3, Azure ABFS and Google GCS.
# Create a directory hadoop fs -mkdir s3a://bucket/datasets/ # Upload a file from the cluster filesystem hadoop fs -put /datasets/example.orc s3a://bucket/datasets/ # touch a file hadoop fs -touchz<EMAIL_ADDRESS>Unlike a normal filesystem, renaming files and directories in an object store usually takes time proportional to the size of the objects being manipulated. As many of the filesystem shell operations use renaming as the final stage in operations, skipping that stage can avoid long delays.
In particular, the put
and copyFromLocal
commands should both have the -d
options set for a direct upload.
# Upload a file from the cluster filesystem hadoop fs -put -d /datasets/example.orc s3a://bucket/datasets/ # Upload a file from under the user's home directory in the local filesystem. # Note it is the shell expanding the "~", not the hadoop fs command hadoop fs -copyFromLocal -d -f ~/datasets/devices.orc s3a://bucket/datasets/ # create a file from stdin # the special "-" source means "use stdin" echo "hello" | hadoop fs -put -d -f -<EMAIL_ADDRESS>Objects can be downloaded and viewed:
# copy a directory to the local filesystem hadoop fs -copyToLocal s3a://bucket/datasets/ # copy a file from the object store to the cluster filesystem. hadoop fs -get<EMAIL_ADDRESS>/examples # print the object hadoop fs -cat<EMAIL_ADDRESS># print the object, unzipping it if necessary hadoop fs -text<EMAIL_ADDRESS>## download log files into a local file hadoop fs -getmerge<EMAIL_ADDRESS>log.txt
Commands which list many files tend to be significantly slower than when working with HDFS or other filesystems
hadoop fs -count s3a://bucket/ hadoop fs -du s3a://bucket/
Other slow commands include find
, mv
, cp
and rm
.
Find
This can be very slow on a large store with many directories under the path supplied.
# enumerate all files in the object store's container. hadoop fs -find s3a://bucket/ -print # remember to escape the wildcards to stop the shell trying to expand them first hadoop fs -find s3a://bucket/datasets/ -name \*.txt -print
Rename
The time to rename a file depends on its size.
The time to rename a directory depends on the number and size of all files beneath that directory.
hadoop fs -mv s3a://bucket/datasets s3a://bucket/historical
If the operation is interrupted, the object store will be in an undefined state.
Copy
hadoop fs -cp s3a://bucket/datasets s3a://bucket/historical
The copy operation reads each file and then writes it back to the object store; the time to complete depends on the amount of data to copy, and the bandwidth in both directions between the local computer and the object store.
The further the computer is from the object store, the longer the copy takes
The rm
command will delete objects and directories full of objects. If the object store is eventually consistent, fs ls
commands and other accessors may briefly return the details of the now-deleted objects; this is an artifact of object stores which cannot be avoided.
If the filesystem client is configured to copy files to a trash directory, this will be in the bucket; the rm
operation will then take time proportional to the size of the data. Furthermore, the deleted files will continue to incur storage costs.
To avoid this, use the the -skipTrash
option.
hadoop fs -rm -skipTrash s3a://bucket/dataset
Data moved to the .Trash
directory can be purged using the expunge
command. As this command only works with the default filesystem, it must be configured to make the default filesystem the target object store.
hadoop fs -expunge -D fs.defaultFS=s3a://bucket/
If an object store is eventually consistent, then any operation which overwrites existing objects may not be immediately visible to all clients/queries. That is: later operations which query the same object’s status or contents may get the previous object. This can sometimes surface within the same client, while reading a single object.
Avoid having a sequence of commands which overwrite objects and then immediately work on the updated data; there is a risk that the previous data will be used instead.
Timestamps of objects and directories in Object Stores may not follow the behavior of files and directories in HDFS.
atime
access time feature is not supported by any of the object stores found in the Apache Hadoop codebase.Consult the DistCp
documentation for details on how this may affect the distcp -update
operation.
The security and permissions models of object stores are usually very different from those of a Unix-style filesystem; operations which query or manipulate permissions are generally unsupported.
Operations to which this applies include: chgrp
, chmod
, chown
, getfacl
, and setfacl
. The related attribute commands getfattr
andsetfattr
are also usually unavailable.
Filesystem commands which list permission and user/group details, usually simulate these details.
Operations which try to preserve permissions (example fs -put -p
) do not preserve permissions for this reason. (Special case: wasb://
, which preserves permissions but does not enforce them).
When interacting with read-only object stores, the permissions found in “list” and “stat” commands may indicate that the user has write access, when in fact they do not.
Object stores usually have permissions models of their own, models can be manipulated through store-specific tooling. Be aware that some of the permissions which an object store may provide (such as write-only paths, or different permissions on the root path) may be incompatible with the Hadoop filesystem clients. These tend to require full read and write access to the entire object store bucket/container into which they write data.
As an example of how permissions are mocked, here is a listing of Amazon’s public, read-only bucket of Landsat images:
$ hadoop fs -ls s3a://landsat-pds/ Found 10 items drwxrwxrwx - mapred 0 2016-09-26 12:16 s3a://landsat-pds/L8 -rw-rw-rw- 1 mapred 23764 2015-01-28 18:13 s3a://landsat-pds/index.html drwxrwxrwx - mapred 0 2016-09-26 12:16 s3a://landsat-pds/landsat-pds_stats -rw-rw-rw- 1 mapred 105 2016-08-19 18:12 s3a://landsat-pds/robots.txt -rw-rw-rw- 1 mapred 38 2016-09-26 12:16 s3a://landsat-pds/run_info.json drwxrwxrwx - mapred 0 2016-09-26 12:16 s3a://landsat-pds/runs -rw-rw-rw- 1 mapred 27458808 2016-09-26 12:16 s3a://landsat-pds/scene_list.gz drwxrwxrwx - mapred 0 2016-09-26 12:16 s3a://landsat-pds/tarq drwxrwxrwx - mapred 0 2016-09-26 12:16 s3a://landsat-pds/tarq_corrupt drwxrwxrwx - mapred 0 2016-09-26 12:16 s3a://landsat-pds/test
rwx
permissions.mapred
).-ls
operation was executed. This is because these directories are not actual objects in the store; they are simulated directories based on the existence of objects under their paths.When an attempt is made to delete one of the files, the operation fails —despite the permissions shown by the ls
command:
$ hadoop fs -rm s3a://landsat-pds/scene_list.gz rm: s3a://landsat-pds/scene_list.gz: delete on s3a://landsat-pds/scene_list.gz: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 1EF98D5957BCAB3D), S3 Extended Request ID: wi3veOXFuFqWBUCJgV3Z+NQVj9gWgZVdXlPU4KBbYMsw/gA+hyhRXcaQ+PogOsDgHh31HlTCebQ=
This demonstrates that the listed permissions cannot be taken as evidence of write access; only object manipulation can determine this.
Note that the Microsoft Azure WASB filesystem does allow permissions to be set and checked, however the permissions are not actually enforced. This feature offers the ability for a HDFS directory tree to be backed up with DistCp, with its permissions preserved, permissions which may be restored when copying the directory back into HDFS. For securing access to the data in the object store, however, Azure’s own model and tools must be used.
Here is the list of shell commands which generally have no effect —and may actually fail.
| command | limitations |
|---|---|
appendToFile |
generally unsupported |
checksum |
the usual checksum is “NONE” |
chgrp |
generally unsupported permissions model; no-op |
chmod |
generally unsupported permissions model; no-op |
chown |
generally unsupported permissions model; no-op |
createSnapshot |
generally unsupported |
deleteSnapshot |
generally unsupported |
df |
default values are normally displayed |
getfacl |
may or may not be supported |
getfattr |
generally supported |
renameSnapshot |
generally unsupported |
setfacl |
generally unsupported permissions model |
setfattr |
generally unsupported permissions model |
setrep |
has no effect |
truncate |
generally unsupported |
concat |
generally unsupported |
Different object store clients may support these commands: do consult the documentation and test against the target store.


|
Panda3D
|
This class can be used for packing a series of numeric and string data into a binary stream, according to the DC specification. More...
Public Member Functions | |
| __init__ () | |
| __init__ (const DCPacker) | |
| beginPack (const DCPackerInterface root) | |
| Begins a packing session. More... | |
| beginRepack (const DCPackerInterface root) | |
| Begins a repacking session. More... | |
| beginUnpack (const DCPackerInterface root) | |
| Begins an unpacking session. More... | |
| clearData () | |
| Empties the data in the pack buffer and unpack buffer. More... | |
| bool | endPack () |
| Finishes a packing session. More... | |
| bool | endRepack () |
| Finishes the repacking session. More... | |
| bool | endUnpack () |
| Finishes the unpacking session. More... | |
| VectorUchar | getBytes () |
| Returns the packed data buffer as a bytes object. More... | |
| const DCPackerInterface | getCurrentField () |
| Returns the field that will be referenced by the next call to pack_*() or unpack_*(). More... | |
| str | getCurrentFieldName () |
| Returns the name of the current field, if it has a name, or the empty string if the field does not have a name or there is no current field. More... | |
| const DCPackerInterface | getCurrentParent () |
| Returns the field that we left in our last call to push(): the owner of the current level of fields. More... | |
| const DCSwitchParameter | getLastSwitch () |
| Returns a pointer to the last DCSwitch instance that we have passed by and selected one case of during the pack/unpack process. More... | |
| int | getLength () |
| Returns the current length of the buffer. More... | |
| int | getNumNestedFields () |
| Returns the number of nested fields associated with the current field, if has_nested_fields() returned true. More... | |
| int | getNumUnpackedBytes () |
| Returns the number of bytes that have been unpacked so far, or after unpack_end(), the total number of bytes that were unpacked at all. More... | |
| DCPackType | getPackType () |
| Returns the type of value expected by the current field. More... | |
| str | getString () |
| Returns the packed data buffer as a string. More... | |
| int | getUnpackLength () |
| Returns the total number of bytes in the unpack data buffer. More... | |
| str | getUnpackString () |
| Returns the unpack data buffer, as a string. More... | |
| bool | hadError () |
| Returns true if there has been any error (either a pack error or a range error) since the most recent call to begin(). More... | |
| bool | hadPackError () |
| Returns true if there has been an packing error since the most recent call to begin(); in particular, this may be called after end() has returned false to determine the nature of the failure. More... | |
| bool | hadParseError () |
| Returns true if there has been an parse error since the most recent call to begin(); this can only happen if you call parse_and_pack(). More... | |
| bool | hadRangeError () |
| Returns true if there has been an range validation error since the most recent call to begin(); in particular, this may be called after end() has returned false to determine the nature of the failure. More... | |
| bool | hasNestedFields () |
| Returns true if the current field has any nested fields (and thus expects a push() . More... | |
| bool | moreNestedFields () |
| Returns true if there are more nested fields to pack or unpack in the current push sequence, false if it is time to call pop(). More... | |
| packBlob (VectorUchar value) | |
| Packs the indicated numeric or string value into the stream. More... | |
| packDefaultValue () | |
| Adds the default value for the current element into the stream. More... | |
| packDouble (double value) | |
| Packs the indicated numeric or string value into the stream. More... | |
| packInt (int value) | |
| Packs the indicated numeric or string value into the stream. More... | |
| packInt64 (int64_t value) | |
| Packs the indicated numeric or string value into the stream. More... | |
| packLiteralValue (VectorUchar value) | |
| Adds the indicated string value into the stream, representing a single pre- packed field element, or a whole group of field elements at once. More... | |
| packObject (object object) | |
| Packs the Python object of whatever type into the packer. More... | |
| packString (str value) | |
| Packs the indicated numeric or string value into the stream. More... | |
| packUint (unsigned int value) | |
| Packs the indicated numeric or string value into the stream. More... | |
| packUint64 (uint64_t value) | |
| Packs the indicated numeric or string value into the stream. More... | |
| bool | parseAndPack (Istream in) |
| Parses an object's value according to the DC file syntax (e.g. More... | |
| bool | parseAndPack (str formatted_object) |
| Parses an object's value according to the DC file syntax (e.g. More... | |
| pop () | |
| Marks the end of a nested series of fields. More... | |
| push () | |
| Marks the beginning of a nested series of fields. More... | |
| rawPackBlob (VectorUchar value) | |
| Packs the data into the buffer between packing sessions. More... | |
| rawPackFloat64 (double value) | |
| Packs the data into the buffer between packing sessions. More... | |
| rawPackInt16 (int value) | |
| Packs the data into the buffer between packing sessions. More... | |
| rawPackInt32 (int value) | |
| Packs the data into the buffer between packing sessions. More... | |
| rawPackInt64 (int64_t value) | |
| Packs the data into the buffer between packing sessions. More... | |
| rawPackInt8 (int value) | |
| Packs the data into the buffer between packing sessions. More... | |
| rawPackString (str value) | |
| Packs the data into the buffer between packing sessions. More... | |
| rawPackUint16 (unsigned int value) | |
| Packs the data into the buffer between packing sessions. More... | |
| rawPackUint32 (unsigned int value) | |
| Packs the data into the buffer between packing sessions. More... | |
| rawPackUint64 (uint64_t value) | |
| Packs the data into the buffer between packing sessions. More... | |
| rawPackUint8 (unsigned int value) | |
| Packs the data into the buffer between packing sessions. More... | |
| VectorUchar | rawUnpackBlob () |
| Unpacks the data from the buffer between unpacking sessions. More... | |
| double | rawUnpackFloat64 () |
| Unpacks the data from the buffer between unpacking sessions. More... | |
| int | rawUnpackInt16 () |
| Unpacks the data from the buffer between unpacking sessions. More... | |
| int | rawUnpackInt32 () |
| Unpacks the data from the buffer between unpacking sessions. More... | |
| int64_t | rawUnpackInt64 () |
| Unpacks the data from the buffer between unpacking sessions. More... | |
| int | rawUnpackInt8 () |
| Unpacks the data from the buffer between unpacking sessions. More... | |
| str | rawUnpackString () |
| Unpacks the data from the buffer between unpacking sessions. More... | |
| unsigned int | rawUnpackUint16 () |
| Unpacks the data from the buffer between unpacking sessions. More... | |
| unsigned int | rawUnpackUint32 () |
| Unpacks the data from the buffer between unpacking sessions. More... | |
| uint64_t | rawUnpackUint64 () |
| Unpacks the data from the buffer between unpacking sessions. More... | |
| unsigned int | rawUnpackUint8 () |
| Unpacks the data from the buffer between unpacking sessions. More... | |
| bool | seek (int seek_index) |
| Seeks to the field indentified by seek_index, which was returned by an earlier call to DCField::find_seek_index() to get the index of some nested field. More... | |
| bool | seek (str field_name) |
| Sets the current unpack (or repack) position to the named field. More... | |
| setUnpackData (VectorUchar data) | |
| Sets up the unpack_data pointer. More... | |
| str | unpackAndFormat (bool show_field_names) |
| Unpacks an object and formats its value into a syntax suitable for parsing in the dc file (e.g. More... | |
| unpackAndFormat (Ostream out, bool show_field_names) | |
| Unpacks an object and formats its value into a syntax suitable for parsing in the dc file (e.g. More... | |
| VectorUchar | unpackBlob () |
| Unpacks the current binary data value from the stream. More... | |
| double | unpackDouble () |
| Unpacks the current numeric or string value from the stream. More... | |
| int | unpackInt () |
| Unpacks the current numeric or string value from the stream. More... | |
| int64_t | unpackInt64 () |
| Unpacks the current numeric or string value from the stream. More... | |
| VectorUchar | unpackLiteralValue () |
| Returns the literal string that represents the packed value of the current field, and advances the field pointer. More... | |
| object | unpackObject () |
| Unpacks a Python object of the appropriate type from the stream for the current field. More... | |
| unpackSkip () | |
| Skips the current field without unpacking it and advances to the next field. More... | |
| str | unpackString () |
| Unpacks the current numeric or string value from the stream. More... | |
| unsigned int | unpackUint () |
| Unpacks the current numeric or string value from the stream. More... | |
| uint64_t | unpackUint64 () |
| Unpacks the current numeric or string value from the stream. More... | |
| unpackValidate () | |
| Internally unpacks the current numeric or string value and validates it against the type range limits, but does not return the value. More... | |
Static Public Member Functions | |
| static int | getNumStackElementsEverAllocated () |
| Returns the number of DCPacker::StackElement pointers ever simultaneously allocated; these are now either in active use or have been recycled into the deleted DCPacker::StackElement pool to be used again. More... | |
This class can be used for packing a series of numeric and string data into a binary stream, according to the DC specification.
See also direct/src/doc/dcPacker.txt for a more complete description and examples of using this class.
| __init__ | ( | ) |
| __init__ | ( | const | DCPacker | ) |
| beginPack | ( | const DCPackerInterface | root | ) |
Begins a packing session.
The parameter is the DC object that describes the packing format; it may be a DCParameter or DCField.
Unless you call clear_data() between sessions, multiple packing sessions will be concatenated together into the same buffer. If you wish to add bytes to the buffer between packing sessions, use append_data() or get_write_pointer().
| beginRepack | ( | const DCPackerInterface | root | ) |
Begins a repacking session.
You must have previously called set_unpack_data() to specify a buffer to unpack.
Unlike begin_pack() or begin_unpack() you may not concatenate the results of multiple begin_repack() sessions in one buffer.
Also, unlike in packing or unpacking modes, you may not walk through the fields from beginning to end, or even pack two consecutive fields at once. Instead, you must call seek() for each field you wish to modify and pack only that one field; then call seek() again to modify another field.
| beginUnpack | ( | const DCPackerInterface | root | ) |
Begins an unpacking session.
You must have previously called set_unpack_data() to specify a buffer to unpack.
If there was data left in the buffer after a previous begin_unpack() .. end_unpack() session, the new session will resume from the current point. This method may be used, therefore, to unpack a sequence of objects from the same buffer.
| clearData | ( | ) |
Empties the data in the pack buffer and unpack buffer.
This should be called between calls to begin_pack(), unless you want to concatenate all of the pack results together.
| bool endPack | ( | ) |
Finishes a packing session.
The return value is true on success, or false if there has been some error during packing.
| bool endRepack | ( | ) |
Finishes the repacking session.
The return value is true on success, or false if there has been some error during repacking (or if all fields have not been repacked).
| bool endUnpack | ( | ) |
Finishes the unpacking session.
The return value is true on success, or false if there has been some error during unpacking (or if all fields have not been unpacked).
| VectorUchar getBytes | ( | ) |
Returns the packed data buffer as a bytes object.
Also see get_data().
| const DCPackerInterface getCurrentField | ( | ) |
Returns the field that will be referenced by the next call to pack_*() or unpack_*().
This will be NULL if we have unpacked (or packed) all fields, or if it is time to call pop().
| str getCurrentFieldName | ( | ) |
Returns the name of the current field, if it has a name, or the empty string if the field does not have a name or there is no current field.
| const DCPackerInterface getCurrentParent | ( | ) |
Returns the field that we left in our last call to push(): the owner of the current level of fields.
This may be NULL at the beginning of the pack operation.
| const DCSwitchParameter getLastSwitch | ( | ) |
Returns a pointer to the last DCSwitch instance that we have passed by and selected one case of during the pack/unpack process.
Each time we encounter a new DCSwitch and select a case, this will change state.
This may be used to detect when a DCSwitch has been selected. At the moment this changes state, get_current_parent() will contain the particular SwitchCase that was selected by the switch.
| int getLength | ( | ) |
Returns the current length of the buffer.
This is the number of useful bytes stored in the buffer, not the amount of memory it takes up.
| int getNumNestedFields | ( | ) |
Returns the number of nested fields associated with the current field, if has_nested_fields() returned true.
The return value may be -1 to indicate that a variable number of nested fields are accepted by this field type (e.g. a variable-length array).
Note that this method is unreliable to determine how many fields you must traverse before you can call pop(), since particularly in the presence of a DCSwitch, it may change during traversal. Use more_nested_fields() instead.
|
static |
Returns the number of DCPacker::StackElement pointers ever simultaneously allocated; these are now either in active use or have been recycled into the deleted DCPacker::StackElement pool to be used again.
| int getNumUnpackedBytes | ( | ) |
Returns the number of bytes that have been unpacked so far, or after unpack_end(), the total number of bytes that were unpacked at all.
This can be used to validate that all of the bytes in the buffer were actually unpacked (which is not otherwise considered an error).
| DCPackType getPackType | ( | ) |
Returns the type of value expected by the current field.
See the enumerated type definition at the top of DCPackerInterface.h. If this returns one of PT_double, PT_int, PT_int64, or PT_string, then you should call the corresponding pack_double(), pack_int() function (or unpack_double(), unpack_int(), etc.) to transfer data. Otherwise, you should call push() and begin packing or unpacking the nested fields.
| str getString | ( | ) |
Returns the packed data buffer as a string.
Also see get_data().
| int getUnpackLength | ( | ) |
Returns the total number of bytes in the unpack data buffer.
This is the buffer used when unpacking; it is separate from the pack data returned by get_length(), which is filled during packing.
| str getUnpackString | ( | ) |
Returns the unpack data buffer, as a string.
This is the buffer used when unpacking; it is separate from the pack data returned by get_string(), which is filled during packing. Also see get_unpack_data().
| bool hadError | ( | ) |
Returns true if there has been any error (either a pack error or a range error) since the most recent call to begin().
If this returns true, then the matching call to end() will indicate an error (false).
| bool hadPackError | ( | ) |
Returns true if there has been an packing error since the most recent call to begin(); in particular, this may be called after end() has returned false to determine the nature of the failure.
A return value of true indicates there was a push/pop mismatch, or the push/pop structure did not match the data structure, or there were the wrong number of elements in a nested push/pop structure, or on unpack that the data stream was truncated.
| bool hadParseError | ( | ) |
Returns true if there has been an parse error since the most recent call to begin(); this can only happen if you call parse_and_pack().
| bool hadRangeError | ( | ) |
Returns true if there has been an range validation error since the most recent call to begin(); in particular, this may be called after end() has returned false to determine the nature of the failure.
A return value of true indicates a value that was packed or unpacked did not fit within the specified legal range for a parameter, or within the limits of the field size.
| bool hasNestedFields | ( | ) |
| bool moreNestedFields | ( | ) |
Returns true if there are more nested fields to pack or unpack in the current push sequence, false if it is time to call pop().
| packBlob | ( | VectorUchar | value | ) |
Packs the indicated numeric or string value into the stream.
| packDefaultValue | ( | ) |
Adds the default value for the current element into the stream.
If no default has been set for the current element, creates a sensible default.
| packDouble | ( | double | value | ) |
Packs the indicated numeric or string value into the stream.
| packInt | ( | int | value | ) |
Packs the indicated numeric or string value into the stream.
| packInt64 | ( | int64_t | value | ) |
Packs the indicated numeric or string value into the stream.
| packLiteralValue | ( | VectorUchar | value | ) |
Adds the indicated string value into the stream, representing a single pre- packed field element, or a whole group of field elements at once.
| packObject | ( | object | object | ) |
| packString | ( | str | value | ) |
Packs the indicated numeric or string value into the stream.
| packUint | ( | unsigned int | value | ) |
Packs the indicated numeric or string value into the stream.
| packUint64 | ( | uint64_t | value | ) |
Packs the indicated numeric or string value into the stream.
| bool parseAndPack | ( | Istream | in | ) |
Parses an object's value according to the DC file syntax (e.g.
as a default value string) and packs it. Returns true on success, false on a parse error.
| bool parseAndPack | ( | str | formatted_object | ) |
Parses an object's value according to the DC file syntax (e.g.
as a default value string) and packs it. Returns true on success, false on a parse error.
| pop | ( | ) |
Marks the end of a nested series of fields.
This must be called to match a previous push() only after all the expected number of nested fields have been packed. It is an error to call it too early, or too late.
| push | ( | ) |
| rawPackBlob | ( | VectorUchar | value | ) |
Packs the data into the buffer between packing sessions.
| rawPackFloat64 | ( | double | value | ) |
Packs the data into the buffer between packing sessions.
| rawPackInt16 | ( | int | value | ) |
Packs the data into the buffer between packing sessions.
| rawPackInt32 | ( | int | value | ) |
Packs the data into the buffer between packing sessions.
| rawPackInt64 | ( | int64_t | value | ) |
Packs the data into the buffer between packing sessions.
| rawPackInt8 | ( | int | value | ) |
Packs the data into the buffer between packing sessions.
| rawPackString | ( | str | value | ) |
Packs the data into the buffer between packing sessions.
| rawPackUint16 | ( | unsigned int | value | ) |
Packs the data into the buffer between packing sessions.
| rawPackUint32 | ( | unsigned int | value | ) |
Packs the data into the buffer between packing sessions.
| rawPackUint64 | ( | uint64_t | value | ) |
Packs the data into the buffer between packing sessions.
| rawPackUint8 | ( | unsigned int | value | ) |
Packs the data into the buffer between packing sessions.
| VectorUchar rawUnpackBlob | ( | ) |
Unpacks the data from the buffer between unpacking sessions.
| double rawUnpackFloat64 | ( | ) |
Unpacks the data from the buffer between unpacking sessions.
| int rawUnpackInt16 | ( | ) |
Unpacks the data from the buffer between unpacking sessions.
| int rawUnpackInt32 | ( | ) |
Unpacks the data from the buffer between unpacking sessions.
| int64_t rawUnpackInt64 | ( | ) |
Unpacks the data from the buffer between unpacking sessions.
| int rawUnpackInt8 | ( | ) |
Unpacks the data from the buffer between unpacking sessions.
| str rawUnpackString | ( | ) |
Unpacks the data from the buffer between unpacking sessions.
| unsigned int rawUnpackUint16 | ( | ) |
Unpacks the data from the buffer between unpacking sessions.
| unsigned int rawUnpackUint32 | ( | ) |
Unpacks the data from the buffer between unpacking sessions.
| uint64_t rawUnpackUint64 | ( | ) |
Unpacks the data from the buffer between unpacking sessions.
| unsigned int rawUnpackUint8 | ( | ) |
Unpacks the data from the buffer between unpacking sessions.
| bool seek | ( | int | seek_index | ) |
Seeks to the field indentified by seek_index, which was returned by an earlier call to DCField::find_seek_index() to get the index of some nested field.
Also see the version of seek() that accepts a field name.
Returns true if successful, false if the field is not known (or if the packer is in an invalid mode).
| bool seek | ( | str | field_name | ) |
Sets the current unpack (or repack) position to the named field.
In unpack mode, the next call to unpack_*() or push() will begin to read the named field. In repack mode, the next call to pack_*() or push() will modify the named field.
Returns true if successful, false if the field is not known (or if the packer is in an invalid mode).
| setUnpackData | ( | VectorUchar | data | ) |
Sets up the unpack_data pointer.
You may call this before calling the version of begin_unpack() that takes only one parameter.
| str unpackAndFormat | ( | bool | show_field_names | ) |
Unpacks an object and formats its value into a syntax suitable for parsing in the dc file (e.g.
as a default value), or as an input to parse_object.
| unpackAndFormat | ( | Ostream | out, |
| bool | show_field_names | ||
| ) |
Unpacks an object and formats its value into a syntax suitable for parsing in the dc file (e.g.
as a default value), or as an input to parse_object.
| VectorUchar unpackBlob | ( | ) |
Unpacks the current binary data value from the stream.
| double unpackDouble | ( | ) |
Unpacks the current numeric or string value from the stream.
| int unpackInt | ( | ) |
Unpacks the current numeric or string value from the stream.
| int64_t unpackInt64 | ( | ) |
Unpacks the current numeric or string value from the stream.
| VectorUchar unpackLiteralValue | ( | ) |
Returns the literal string that represents the packed value of the current field, and advances the field pointer.
| object unpackObject | ( | ) |
Unpacks a Python object of the appropriate type from the stream for the current field.
This may be an integer or a string for a simple field object; if the current field represents a list of fields it will be a tuple.
| unpackSkip | ( | ) |
Skips the current field without unpacking it and advances to the next field.
If the current field contains nested fields, skips all of them.
| str unpackString | ( | ) |
Unpacks the current numeric or string value from the stream.
| unsigned int unpackUint | ( | ) |
Unpacks the current numeric or string value from the stream.
| uint64_t unpackUint64 | ( | ) |
Unpacks the current numeric or string value from the stream.
| unpackValidate | ( | ) |
Internally unpacks the current numeric or string value and validates it against the type range limits, but does not return the value.
If the current field contains nested fields, validates all of them.


Unifile WebDAV
This is a plugin for Unifile to use WebDAV services with it.
To include it in Unifile, you first need to install it
# Node 6+
$ npm install unifile-webdav -S
# Node 8+
$ npm install unifile-webdav
Then, you'll have to register it to unifile
const Unifile = require('unifile');
const UnifileWebDAV = require('unifile-webdav');
const unifile = new Unifile();
// Register a new connector
unifile.use(new UnifileWebDAV({redirectUri: 'http://localhost:3000/callback'}));


Book Notes Hooked By Nir Eyal
Book Notes: Hooked: How to Build Habit-Forming Products by Nir Eyal
I was recently given a task to power through some books. For both personal retention and public good, I thought I’d borrow a dance move from Derek Sivers and keep public notes on them. These notes will be updated when I push to the blog and will follow along with me in the book. When I finish a book, I’ll try and do some big takeaways.
</>
“Amassing millions of users is no longer enough. Companies increasingly find that their economic value is a function of the strength of the habits they create.” pg 2
This caught my eye, but I’m up in the air about the universality of it at this point. This might be more true for companies that take percentages (ride sharing, transactions) than companies who survive on ad revenue (Facebook, Twitter). Google seems to the be an exception. Happy to be prove wrong about that (which, presumably is what this book will do.) </>
The Hook Model
Trigger (internal/ external), Action, Variable, Reward. </>
“Feedback Loops are all around us, but predictable ones don’t create desire. The unsurprising response of your fridge light turning on when you open the door doesn’t drive you to keep opening it again and again.” Pg 8
🤔 </>
“Research shows that levels of the neurotransmitter dopamine surge when the brain is expecting a reward. Introducing variability multiplies the effect, creating a focused state, which suppresses the areas of the brain associated with judgment and reason while activating parts associated with wanting and desire.” pg 8-9
Wow. The “loot box” phenomenon that’s being integrated into all aspects of games makes a lot of sense given this research. This might also be why games RPG’s are so compelling in the first place. Here’s an article from Cracked about this specifically.
“Recently, a blog reader emailed me, “If it can’t be used for evil, it’s not a superpower.”” pg 11
Whoa. </>
“Habit forming products change user behavior and create unprompted user engagement.” pg 17
I can’t help but think of this in terms of software that allow for short cutting workflows in favor of efficiency. I still reach for my Alfred snippets, and remember my Front workflows quite well. It seems like software used to create habit by simplifying process is perhaps trying to do the same thing. </>
“A business is worth the sum of its future profits. -pg 19”
A succinct summary </>
• Fostering habits creates a higher Customer Lifetime Value. pg. 19
“Users who continuously find value in a product are more likely to tell their friends about it.” pg. 21 “
For books about WOMM, I’d recommend Fizz by Ted Wright and checking out Ted on Twitter. </>
“To borrow a term from accounting, behaviors are LIFO – “Last in, first out.” In other words, the habits you’ve most recently acquired “
“Are you building a vitamin or a painkiller.” pg. 32
ie; does this solve a problem, or does it satisfy you in some other way. </>
“Emotions, particularly negative ones, are powerful internal triggers and greatly influence our daily routines. Feelings of boredom, loneliness, frustration, confusion, and indecisiveness often instigate a slight pain or irritation and prompt an almost instantaneous and often mindless action to quell the negative sensation.” pg. 48
YEP. This reminds of me of something Jason Pargin (aka Cracked’s David Wong) talks about on a couple of episodes of the Cracked podcast. How social apps are altering our brain chemistry to receive little dopamine hits from apps. The idea of these boredom triggers cycling into dopamine drops from the app use cements the process even more.
I first heard the idea from Tim Ferriss, but here’s a video talking about how to break cell phone addiction, and one of many articles talking about how boredom can be a good thing.. </>
“Products that successfully create habits sooth the user’s pain by laying claim to a particular feeling. To do so, product designers must know their users internal triggers––that is, the pain they seek to solve…The ultimate goal of a habit-forming product is to solve the user’s pain by creating an association so that the user identifies the company’s product or service as the source of relief.” pg. 51-52
• Lots of really good points about “declared preferences” vs. “revealed preferences”. What people say they want vs. what they actually do. I imagine this is what a lot of the fitness tracking industry is banking on. They don’t necessarily care if people lose weight, they want to sell them the satisfaction of feeling like they’re actively working on their weight. I suppose in this way they’re “vitamin” products.
This also ties into the idea of moral licensing (here’s Gladwell giving a topical example) and the idea that we justify bad behavior based on past good behavior. The idea being that a fitness band/tracker gives you lots of emotional triggers, AND helps you do the things you want to do (break a diet), while still providing you the comfort of licensing away bad behavior. Pretty compelling.
• Just got an alert from the Hearthstone app on my phone: “A new ranked play season has begun! See how far you can climb this month!” Quite timely.
Reading in Progress….
Best,
Watson


"""
helpers.py:
Helpers for file uplink, and downlink. This contains shared components, classes, and support architecture in order to
enable both file uplink and downlink to share the same structures. This includes the following shared objects:
1. Timeout: for managing timeouts and timeout responses
2. FileStates: managing the state of an uplink or downlinking file
3. CFDPChecksum: calculates the CFDP checksum for files
4. TransmitFile: file object for up and down
@author mstarch, and Blake A. Harriman's work
"""
import datetime
import enum
import logging
import os
import struct
import threading
[docs]class Timeout:
"""
Starts a timeout thread and will respond with a callback to a function when the timeout expires.
"""
def __init__(self):
""" Sets up needed member variables """
self.__timer = None
self.__timeout = None
self.__callback = None
self.args = ()
[docs] def setup(self, callback, timeout=5, args=()):
"""
Sets up the timeout but does not start it.
:param callback: function called when timeout expires
:param timeout: (optional) timeout duration. Default: 5 seconds
"""
self.__timeout = timeout
self.__callback = callback
self.args = args
[docs] def start(self):
""" Starts the timeout after a previous setup. """
assert self.__timer is None, "Timer already started, call restart() instead"
assert self.__callback is not None, "Setup timeout before calling start"
assert self.__timeout is not None, "Setup timeout before calling start"
self.__timer = threading.Timer(self.__timeout, self.__callback, args=self.args)
self.__timer.start()
[docs] def restart(self):
"""
Restarts the given timer first canceling and then starting it again.
"""
self.stop()
self.start()
[docs] def stop(self):
"""
Stops the timeout preventing a callback to the stored function.
"""
if self.__timer is not None:
self.__timer.cancel()
self.__timer = None
[docs]class FileStates(enum.Enum):
"""
An enumeration of states used in the file uplinker and downlinker.
"""
[docs] END_WAIT = 3 # Waiting for the handshake for CANCEL or END packet
[docs]class CFDPChecksum:
""" Class running the CFDG checksum """
def __init__(self):
""" Set initial value as zero """
self.__value = 0
[docs] def update(self, data, offset):
""" Update the checksum """
while data:
padding_len = offset % 4
calc_bytes = (
bytes([0] * padding_len) + data[: 4 - padding_len] + bytes([0, 0, 0, 0])
)
self.__value = (
self.__value + struct.unpack_from(">I", calc_bytes, 0)[0]
) & 0xFFFFFFFF
# Update pointers
data = data[4 - padding_len :]
offset = offset + (4 - padding_len)
@property
[docs] def value(self):
return self.__value
[docs]class TransmitFile:
"""
Wraps the file information needed for the uplink and downlinking processes.
"""
def __init__(self, source, destination, size=None, log_dir=None):
""" Construct the uplink file """
self.__mode = None
self.__start = None
self.__end = None
self.__source = source
self.__destination = destination
self.__size = size if size is not None else os.path.getsize(source)
self.__seek = 0
self.__state = "QUEUED"
self.__fd = None
self.__checksum = CFDPChecksum()
self.__log_dir = log_dir
self.__log_handler = None
[docs] def open(self, mode="rb"):
"""
Opens the file descriptor and prepares it for uplink/downlink
"""
assert self.__fd is None, "Must close file before attempting to reopen it"
filepath = self.__source if mode.startswith("r") else self.destination
self.__mode = mode
self.__state = "TRANSMITTING"
self.__fd = open(filepath, self.__mode)
self.__start = datetime.datetime.utcnow()
if self.__log_dir is not None:
self.__log_handler = logging.FileHandler(
os.path.join(
self.__log_dir, "{}.log".format(os.path.basename(filepath))
),
"w",
)
[docs] def read(self, chunk):
""" Read the chunk from the file """
assert self.__fd is not None, "Must open file before reading"
assert self.__mode.startswith("r"), "File opened with invalid mode for 'read'"
return self.__fd.read(chunk)
[docs] def write(self, chunk, offset):
"""
Write a chunk to the file.
:param chunk: data to write to the file
:param offset: offset to write to
"""
assert self.__fd is not None, "Must open file before writing"
assert self.__mode.startswith("w"), "File opened with invalid mode for 'write'"
self.__fd.seek(offset, 0)
self.__fd.write(chunk)
[docs] def close(self):
"""
Opens the file descriptor and prepares it for uplink
"""
if self.__log_handler is not None:
self.__log_handler.close()
self.__log_handler = None
if self.__fd is not None:
self.__fd.close()
self.__fd = None
self.__end = datetime.datetime.utcnow()
@property
[docs] def start(self):
return self.__start
@property
[docs] def end(self):
return self.__end
@property
[docs] def source(self):
return self.__source
@property
[docs] def destination(self):
return self.__destination
@property
[docs] def size(self):
return self.__size
@property
[docs] def seek(self):
return self.__seek
@seek.setter
def seek(self, seek):
self.__seek = seek
@property
[docs] def state(self):
return self.__state
@state.setter
def state(self, state):
self.__state = state
@property
[docs] def checksum(self):
return self.__checksum
@property
[docs] def log_handler(self):
return self.__log_handler
[docs]def file_to_dict(files, uplink=True):
"""
Converts files to dictionary. This creates a new list of JSONable file dictionaries.
:param files: list of TransmitFiles to convert
:return: list of dictionaries
"""
current = []
for item in files:
current.append(
{
"source": item.source,
"destination": item.destination,
"size": item.size,
"current": item.seek,
"state": item.state,
"percent": 100
if item.size == 0
else int(item.seek / item.size * 100.0),
"uplink": uplink,
"start": item.start,
"end": item.end,
}
)
return current


|
|||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
public interface SSLSessionManager
The SSLSessionManager
interface is responsible for dealing
with SSL related events such as certificates being sent from the FTP
server and randomization information.
| Field Summary | |
|---|---|
static short |
ALLOW_CERTIFICATE
Used to signal certificate acceptance. |
static short |
DENY_CERTIFICATE
Used to signal certificate rejection. |
static short |
SAVE_CERTIFICATE
Used to signal certificate acceptance and that it should be saved in the certificate key store. |
| Method Summary | |
|---|---|
boolean |
continueWithCertificateHostMismatch(SSLCertificate cert,
String actualHost,
String certHost)
If the server sends a certificate which doesn't match the hostname, this method will be called. |
boolean |
continueWithExpiredCertificate(SSLCertificate cert)
If the server sends an expired certificate, this method will be called. |
boolean |
continueWithInvalidCertificate(SSLCertificate cert)
If the server sends an invalid certificate, this method will be called. |
boolean |
continueWithoutServerCertificate()
If the server doesn't send a certificate, this method will be called. |
short |
newCertificateEncountered(SSLCertificate cert)
If the certificate sent by the FTP server is not found in the key store, this method will be called. |
void |
randomSeedGenerated()
When the random number generator is finished seeding, this method will be called. |
void |
randomSeedIsGenerating()
If the random number generator is seeding and is not ready for setting up SSL sockets, this method will be called. |
short |
replaceCertificate(SSLCertificate oldCert,
SSLCertificate newCert)
If a certificate is found in the key store and a new one that matches the same name is sent from the FTP server, this method will be called. |
void |
setCurrentCertificate(SSLCertificate cert)
When a certificate is encountered from the FTP server, this method will be called. |
| Field Detail |
|---|
static final short ALLOW_CERTIFICATE
static final short SAVE_CERTIFICATE
static final short DENY_CERTIFICATE
| Method Detail |
|---|
void setCurrentCertificate(SSLCertificate cert)
cert
- the SSL (X509) certificate.short newCertificateEncountered(SSLCertificate cert)
cert
- the SSL (X509) certificate.
ALLOW_CERTIFICATE
,
SAVE_CERTIFICATE
,
DENY_CERTIFICATE
short replaceCertificate(SSLCertificate oldCert, SSLCertificate newCert)
oldCert
- the known SSL (X509) certificate.newCert
- the new SSL (X509) certificate.
ALLOW_CERTIFICATE
,
SAVE_CERTIFICATE
,
DENY_CERTIFICATE
boolean continueWithoutServerCertificate()
boolean continueWithExpiredCertificate(SSLCertificate cert)
cert
- the certificate sent from the FTP server.
boolean continueWithInvalidCertificate(SSLCertificate cert)
cert
- the certificate sent from the FTP server.
boolean continueWithCertificateHostMismatch(SSLCertificate cert, String actualHost, String certHost)
cert
- the certificate sent from the FTP server.actualHost
- the hostname of the server.certHost
- the hostname as specified in the certificate.
void randomSeedIsGenerating()
void randomSeedGenerated()
|
|||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |


I collect Tarot decks because the art on them is interesting and I enjoy researching where it came from.
My Tarot Decks
- Aleister Crowley Thoth Tarot
- Artist: Lady Frieda Harris
- Design: Aleister Crowley
- The Book of Azathoth Tarot by Nemo
- Deviant Moon Tarot
- Artist/Designer: Patrick Valenza
- Comments: This deck has a bizarre, almost nightmarish quality to it.
- El Gran Tarot Esoterico
- Artist/Designer: Luis Pena Longo
- Everyday Tarot
- Artist: Eleanor Grosch
- Designer: Brigit Esselmont
- Ghetto Tarot
- Photographer: Alice Smeets
- Models: Atis Rezistans
- Comments: This photographic deck recreates traditional Waite-Smith imagery in a Hatian ghetto.
- The Hermetic Tarot
- Artist/Design: Godfrey Dowson
- Liber T: Tarot of Stars Eternal
- Artist: Andrea Serio
- Designer: Roberto Nigrini
- The Linestrider Tarot
- Artist/Designer: Siolo Thompson
- Lo Scarabeo
- Creators: Mark McElroy, Anna Lazzarini
- Comments: Tries to blend the two major lineages of Tarot, The Waite-Smith and The Crowley Thoth.
- Londa Tarot
- Artist/Designer: Londa Marks
- Ludy Lescot
- Creator: Patrizio Evangelisti
- The Mary-El Tarot
- Artist/Designer: Mary White
- Navigators Tarot of the Mystic Sea
- Artist/Designer: Julia Turk
- Tarot in Wonderland
- Artist: Eugene Smith
- Design: Barbara Moore
- The Tarot of Saqqara
- Artist/Designer: Donald G. Beaman
- Tarot of White Cats
- Creator: Severino Baraldi
- Viceversa Tarot
- Artist: Davide Corsi
- Design: Massimiliano Filadoro, Lunaea Weatherstone
- Comments: This is a neat deck that takes the idea of 'reversed cards' to it's extreme by making the cards double sided, one side being the light/front, the other being the dark/back of the image.
- Visconti-Sforza Tarot
- Restored And Commented by: Stuart R. Kaplan
- Comments: This is a historical reproduction of one of the earliest known Tarot decks.
- Waite-Smith (Rider-Waite)
- Artist: Pamela Colman Smith
- Design: A. E. Waite
- The Wonderland Tarot
- Artist/Designer: Chris Abbey & Morgana Abbey
Tarot decks I admire but haven't gotten around to picking up
- Ancient Tarot of Marseilles is a reproduction that inspired the Waite-Smith deck.
- Animal Os Fortuna Another animal deck. All Black & White, this uses a combination of animals and their bones to create feeling of macabre, without blood & gore
- Book of Thoth Etteilla Tarot is a reporduction of the Thoth decks, one of the first attempts at constructing an explicitly occult deck.
- Cat's Eye Tarot Designed by a feline vetrinarian uses classic Waite-Smith suits and lightly interprets the symbols in a feline context
- English Magic Tarot has this fantastic, flat imagery that looks simple on the surface but has a dozen stories.
- International Icon Tarot This is the Waite-Smith tarot, stripped of all detail, attempting to leave only the symbols.
- Kingdom Within Tarot This tarot leans heavily on the astrological and quabalistic connections for it's art, with only light connections to the Waite-Smith.
- Mystical Cats Tarot Gorgeous artwork, heavily interpreted Waite-Smith. Rather than putting cats in classic scenes, it instead imagines the feelings of the scenes and puts them in a feline context. Suits are Earth, Sky, Fire, Sea
- The Sola-Busca Tarot One of the first Tarot decks to have fully illustrated pip cards. This was likely one of the major inspirations for the Waite-Smith deck.
- The Tarot of Sweet Twilight Brilliantly colored with a surreal landscape and striking imagery.
- Thelema Tarot has beautiful images that each seem to glow.
- The Wild Unknown Tarot Pen & ink style artwork, great use of shading.


Hive Docs
Understanding Dynamic Global Properties
Maintains global state information
Intro
Dynamic Global Properties represents a set of values that are calculated during normal chain operations and reflect the current values of global blockchain properties.
The API returns an object containing information that changes every block interval such as the head block number, the total vesting fund, etc.
Sections
- Fields
confidential_sbd_supply
confidential_supply
current_aslot
current_sbd_supply
current_supply
current_witness
downvote_pool_percent
head_block_id
head_block_number
last_budget_time
last_irreversible_block_num
maximum_block_size
next_maintenance_time
num_pow_witnesses
participation_count
recent_slots_filled
required_actions_partition_percent
reverse_auction_seconds
sbd_interest_rate
sbd_start_percent
sbd_stop_percent
target_votes_per_period
time
total_pow
total_reward_fund_steem
total_reward_shares2
total_vesting_fund_steem
total_vesting_shares
virtual_supply
vote_power_reserve_rate
average_block_size
current_reserve_ratio
max_virtual_bandwidth
- Not Covered
- Example Method Call
- Example Output
average_block_size
- Removed
- Related
Removed in 0.20.6, see: #3029
Average block size is updated every block to be: average_block_size = (99 * average_block_size + new_block_size) / 100
. This property is used to update the current_reserve_ratio
to maintain approximately * 50% or less utilization of network capacity.
- Examples:
9309
confidential_sbd_supply
Total asset held in confidential balances.
- Examples:
0.000 SBD
confidential_supply
Total asset held in confidential balances.
- Examples:
0.000 STEEM
current_aslot
The current absolute slot number. Equal to the total number of slots since genesis. Also equal to the total number of missed slots plus head_block_number
.
- Examples:
24231997
current_reserve_ratio
- Removed
- Related
Removed in 0.20.6, see: #3029
Any time average_block_size <= 50% maximum_block_size
this value grows by 1 until it reaches STEEM_MAX_RESERVE_RATIO
. Any time average_block_size
is greater than 50% it falls by 1%. Upward adjustments happen once per round, downward adjustments happen every block.
- Examples:
200000000
current_sbd_supply
SBD currently in existence.
- Examples:
15478883.968 SBD
current_supply
STEEM currently in existence.
- Examples:
271546371.129 STEEM
current_witness
Account name of the current witness.
- Examples:
blocktrades
downvote_pool_percent
- Since: HF21
- Related
Used by a separate downvote pool that is a percentage of the max size of the voting mana. Downvotes will be taken from the downvote pool first, and then the upvote pool once the downvote pool has been consumed. The downvote pool will follow the same rules as the upvote pool, regenerating over five days and filling instantly and proportionally to new Steem Power and delegations.
- Examples:
2500
head_block_id
Used to implement TaPoS (Transaction as Proof of Stake). The first 4 bytes (8 hex digits) of the block ID represents the block number. E.g., 01709398
in hex is 24155032
in decimal.
- Examples:
0170939865fa4e3aa7fca8f8df35d23333fe0bee
See: RIPEMD-160 hashes
head_block_number
Block height at the head of the blockchain. This represents the latest block produced by witnesses.
- Examples:
24155032
last_budget_time
- Since: HF21
- Related
Used to compute payments for every proposal.
- Examples:
2019-11-14T20:26:57
See: blocktradesdevs/steem#7, sps_fund
last_irreversible_block_num
The latest block number that has been confirmed by two thirds of all block producers and is thus irreversible.
- Examples:
24155017
max_virtual_bandwidth
- Removed
- Related
Removed in 0.20.6, see: #3029
The maximum bandwidth the blockchain can support is max_bandwidth = maximum_block_size * STEEM_BANDWIDTH_AVERAGE_WINDOW_SECONDS / STEEM_BLOCK_INTERVAL
; The maximum virtual bandwidth is: max_bandwidth * current_reserve_ratio
- Examples:
264241152000000000000
maximum_block_size
Maximum block size is decided by the set of active witnesses which change every round. Each witness posts what they think the maximum size should be as part of their witness properties, the median size is chosen to be the maximum block size for the round.
Note: the minimum value for maximum_block_size
is defined by the protocol to prevent the network from getting stuck by witnesses attempting to set this too low.
- Examples:
65536
next_maintenance_time
- Since: HF21
- Related
Used to compute payments for every proposal.
- Examples:
2019-11-14T21:26:57
See: blocktradesdevs/steem#7, sps_fund
num_pow_witnesses
- Deprecated
- Related
The current count of how many pending POW witnesses there are, determines the difficulty of doing pow.
- Examples:
172
See: #770
participation_count
- Since: HF9
- Related
Used to compute witness participation.
- Examples:
128
recent_slots_filled
Used to compute witness participation.
- Examples:
340282366920938463463374607431768211455
required_actions_partition_percent
- Since: HF21
- Related
The size of the block that is partitioned for actions. Required actions can only be delayed if they take up more than this amount. More can be included, but are not required. Block generation should only include transactions up to maximum_block_size - required_actions_parition_size
to ensure required actions are not delayed when they should not be.
- Examples:
0
reverse_auction_seconds
- Since: HF21
- Related
Curation reverse auction duration.
- Examples:
300
See: #1874
sbd_interest_rate
This property defines the interest rate that SBD deposits receive.
- Examples:
0
sbd_start_percent
- Since: HF20
- Related
Start reducing printing of SBD at n% marketcap.
- Examples:
900
See: #2140
sbd_stop_percent
- Since: HF20
- Related
Stop printing SBD at n% marketcap.
- Examples:
1000
See: #2140
target_votes_per_period
- Since: HF23
- Related
The number of votes regenerated per day (over STEEM_VOTING_MANA_REGENERATION_SECONDS). Any user voting slower than this rate will be “wasting” voting power through spillover; any user voting faster than this rate will have their votes reduced.
- Examples:
50
time
Point in time (UTC) that the block was included in the chain.
Used to synchronize events like Hard Fork activation.
When attempting to calculate the validity of a transaction we need to lookup a past block and check its block hash and the time it occurred so we can calculate whether the current transaction is valid and at what time it should expire.
For new transactions, expirations originate from this time.
- Examples:
2018-07-14T01:19:51
total_pow
- Deprecated
- Related
The total POW accumulated, aka the sum of num_pow_witness
at the time new POW is added.
- Examples:
514415
See: #770
total_reward_fund_steem
STEEM available in the reward pool.
- Examples:
0.000 STEEM
total_reward_shares2
- Deprecated
- Related
Now deprecated way to get the number of shares in the pool.
Use condenser_api.get_reward_fund
instead.
- Examples:
0
total_vesting_fund_steem
- Deprecated
- Related
Now deprecated way to get STEEM that is invested in STEEM POWER (powered up).
Use condenser_api.get_reward_fund
instead.
- Examples:
192713261.007 STEEM
total_vesting_shares
VESTS that are invested in STEEM POWER (powered up).
- Examples:
390950506702.452773 VESTS
virtual_supply
The virtual supply is the supply of all STEEM + all SBD if all SBD were converted to STEEM at the current median price.
- Examples:
283290592.652 STEEM
vote_power_reserve_rate
The number of votes regenerated per day. Any user voting slower than this rate will be “wasting” voting power through spillover; any user voting faster than this rate will have their votes reduced.
- Examples:
10
Not Covered
Fields not covered in this recipe are:
available_account_subsidies
content_reward_percent
delegation_return_period
init_sbd_supply
pending_rewarded_vesting_shares
pending_rewarded_vesting_steem
sbd_print_rate
sps_fund_percent
sps_interval_ledger
vesting_reward_percent
Example Method Call
To retrieve the current results for condenser_api.get_dynamic_global_properties
, we can retrieve the current state information using curl
:
curl -s --data '{"jsonrpc":"2.0", "method":"condenser_api.get_dynamic_global_properties", "params":[], "id":1}' https://api.steemit.com
Example Output
{
"id":1,
"jsonrpc":"2.0",
"result":{
"head_block_number":24238248,
"head_block_id":"0171d8a833dc369abd034b0c67d8725f96df9e5b",
"time":"2018-07-16T22:41:24",
"current_witness":"xeldal",
"total_pow":514415,
"num_pow_witnesses":172,
"virtual_supply":"283434761.199 STEEM",
"current_supply":"271729171.190 STEEM",
"confidential_supply":"0.000 STEEM",
"current_sbd_supply":"15498201.173 SBD",
"confidential_sbd_supply":"0.000 SBD",
"total_vesting_fund_steem":"192913644.627 STEEM",
"total_vesting_shares":"391296886352.617261 VESTS",
"total_reward_fund_steem":"0.000 STEEM",
"init_sbd_supply":"7000000.000 TBD",
"total_reward_shares2":"0",
"pending_rewarded_vesting_shares":"379159224.860656 VESTS",
"pending_rewarded_vesting_steem":"185294.019 STEEM",
"sbd_interest_rate":0,
"sbd_print_rate":2933,
"maximum_block_size":65536,
"current_aslot":24315228,
"recent_slots_filled":"340282366920938463463374607431768211400",
"participation_count":128,
"last_irreversible_block_num":24238230,
"vote_power_reserve_rate":10
"required_actions_partition_percent":2500,
"target_votes_per_period":50,
"vote_power_reserve_rate":50,
"delegation_return_period":432000,
"reverse_auction_seconds":300,
"available_account_subsidies":137414104,
"sbd_stop_percent":1000,
"sbd_start_percent":900,
"next_maintenance_time":"2019-11-15T01:44:39",
"last_budget_time":"2019-11-15T00:44:39",
"content_reward_percent":6500,
"vesting_reward_percent":1500,
"sps_fund_percent":1000,
"sps_interval_ledger":"15.162 TBD",
"downvote_pool_percent":2500
}
}


This is the instrument driver for the Phidget BLDC Motor. The VI Tree displays all the user-callable VIs of the instrument driver in an organized table.


Symbol Index for "Measured Qualifier"
Symbol Index for diagram
"Measured Qualifier"
accuracy unit for (Association)
Category Valued Qualifier (Class)
Image Valued Qualifier (Class)
Measured Qualifier (Class)
measurement unit for (Association)
Phenomenon Qualifier (Class)
PhenQ/Value Type (Generalization Node)
precision unit for (Association)
Text Valued Qualifier (Class)
Timepoint Valued Qualifier (Class)
Unit Of Measure (Class)


Phone:<PHONE_NUMBER>
Email<EMAIL_ADDRESS>Address:
3481 Melrose Place
Santana Row, CA 90210
Hours of Operation:
Monday-Friday: 8am-10pm
Saturday: 7:30am-11pm
Sunday: 8am-9pm


{% extends "layout.html" %} {% block title %}California Emergency Alerts{% endblock %} {% block main_content %}
This alert will go to only those users registered to receive alerts about emergencies in this specific area.


Cowslip cottage has been sympathetically renovated to provide a comfortable holiday retreat with stylish furnishings in keeping with the traditional beams and cottage style windows.
Reached via a private enclosed garden it has an open-plan lounge, kitchen/diner. The lounge is equipped with flat screen TV, DVD player and a selection of books.
The kitchen includes dishwasher, fridge/freezer, microwave and hob and oven. There is also a comprehensive range of cook and tableware.
A pretty double bedroom, with fresh white linen, provides views over the lake and ample hanging and storage space.
A cosy twin bedroom provides two generous single beds and storage space.
The modern bathroom contains a walk in shower, sink, heated towel rail and w.c.
Enjoy al fresco dinning in the secluded sunny garden with comfortable patio furniture and gas bar-b-que.
Cowslip has oil central heating. Towels and bed linen are provided. All electricity, heating and WiFi are included in the tariff.
Pets are welcome at Cowslip however smoking is not permitted inside the property.


Arugam Bay Taxi Order Confirmation
Here are your transfer details for your driver.
taxisurfr
http://www.taxisurfr.info
dispatch@taxisurfr.com


We seek exciting projects that satisfy our hunger for technical challenges.
Our main focus are mobile-centric solutions that encompass:
- iOS native applications
- Android native applications
- Web solutions (AngularJS, ReactJS)
- Backend solutions (Java, Ruby, NodeJS)


Class ClientStorage
Client storage implementation
Constructor
-
class ClientStorage()
Methods
addListener
-
ClientStorage#addListener(callback)
-
get
-
ClientStorage#get(key[, isTransient])
| Arguments: |
- key (String) –
- isTransient (Boolean) – whether to use persistent storage or transient storage (defaults to false)
|
| Throws Error: | : if client storage is not supported
|
| Returns String|Boolean: |
| | the value of key or null on failure
|
Retrieves the value of key from storage
remove
-
ClientStorage#remove(key[, isTransient])
| Arguments: |
- key (String) –
- isTransient (Boolean) – whether to use persistent storage or transient storage (defaults to false)
|
| Throws Error: | : if client storage is not supported
|
Remove the key from storage
removeListener
-
ClientStorage#removeListener(callback)
-
set
-
ClientStorage#set(key, value[, isTransient])
| Arguments: |
- key (String) –
- value (Object) –
- isTransient (Boolean) – whether to use persistent storage or transient storage (defaults to false)
|
| Throws Error: | : if client storage is not supported
|
Set the value of key (add it if key doesn’t exist) into storage


@(loc: Location, feat: Feature, title: String, titleIcon: String, lowText: String, highText: String) @HistoBar(barNum: String) = {
@title @if(titleIcon != ""){}
3


Collision Polys
In order to use polygon collision testing on an object, select the object and go to the Collision section of the Edit tab. Change the following settings, if necessary (see Figure 1)
|
|
Figure 1 |
| With your object still selected, move your mouse over it in the Scene View, and click on the far left widget that appears above it - Edit this object's collision polygon (as shown in Figure 2). |
Figure 2 |
| Once you click the widget, the Collision Editor will appear (Shown in Figure 3). The main window shows your object with a graph overlay. The blue, numbered boxes are collision points which connect to each other to create the outline of your collision polygon. |
Figure 3 |
|
In the 4 corners are number boxes; these are your collision points.
There is one main rule to creating a collision polygon: the polygon must be convex. This means that
it can't have any indents in it. Some images have indents that were created during asset generation,
which means you will not always be able to create perfect collision polygons fully surrounding your
object. Simply put, you have to arrange your collision points to form a convex polygon similar to your
image. Within this scene graph, you can click and drag one of the boxes around to adjust the polygon. If you right click on the box, it will be deleted. If you click on an open area, a new collision point will be created. |
| In the top right of the editor is a list containing each of your collision points, as well as their X/Y location (Shown in Figure 4). In this local graph, each point has a range of -1.000 to 1.000. You can modify these coordinate numbers to move your collision points. This will give you slightly more precise positioning than dragging the boxes.. | Figure 4 |
| Below the list of collision points, you will see an option to create a new point (Shown in Figure 5). This has the same functionality as right clicking the scene. Once you enter your X and Y values, click the plus button to add the new point.. | Figure 5 |
| In the bottom left, you will various toggles that will aid you in creating a convex polygon (Shown in Figure 6). . | Figure 6 |
| To show off these features, let's go ahead and move Point 0 to a new location (Shown in Figure 7). What we have done is moved a collision point to an illegal position; our polygon is no longer convex.. |
Figure 7 |
|
You should now see 4 lines. The line consisting of points (3,0,1) is bent.
The new green line is a visual guide recommending you where you should have a line to make your polygon convex again.
Click the "Show Convex Hull" toggle off, which will turn this line suggestion off (Shown in Figure 8). This is a handy visual guide for correcting your polygon.. |
Figure 8 |
You might have noticed that collision point 0 has a red background, instead of blue. This is shown in both the scene and the collision point list. Any red collision points are violating the convex rule. If you toggle the "Show Convex Violations" box off, you will no longer see which points are in violation. Go ahead and turn both toggles back on. |
| Without creating new points, move 0, 1, 2 and 3 to your image's border (Shown in Figure 9). |
|
|
|
Next to your check boxes, there are 2 radio buttons.
You can only have 1 toggled at a time, as this will dictate how the next point you create
is connected to the rest of your polygon.
With the "Between Neighbors" toggle on, click on a location between points 0 and 1 (Shown in Figure 10). A new point is created between the two closest points when "Between Neighbors" is turned on. |
Figure 10 |
|
It's important to note what happened to the numbering of your points.
0 remained the same, the new point became 1, and the rest increased by 1. What is happening is the
collision editor is keeping your points in order based on clock-wise positioning.
Think of it like the numbers on a clock. Create another point between 3 and 4 (Shown in Figure 11). A new point is added and numbered 4, leaving the last point in the rotation to become #5.. |
Figure 11 |
|
Now, let's show off what the other toggle does.
Click on "New Points at End" to activate it. Click on an area between 2 and 3 to
see the results (Shown in Figure 12).
Oh no!!! The numbers are out of order, lines are crossing, there are at least 3 known convex violations...The placing of point 6 has completely borked our polygon. Right click on 6 to delete it. |
Figure 12 |
|
Let's place that in its "proper location."
Click on an area between points 0 and 5. The results are much better (Shown in Figure 13).
When "New Points a End" is toggled, the Editor is trusting you to place your collision points in clock-wise order. When I say the Editor "trusts you," it really means the Editor will not perform any reordering for you, so you should be careful. |
Figure 13 |
| In the same box as all your toggles, you will see a button that contains 2 tiny polygons (Shown in Figure 14). |
Figure 14 |
|
This handy button will automatically repair your polygon to create a convex polygon.
The benefit is less clean up work you need to perform. The downside is that some collision points you wish to keep might be deleted.
You do not have much choice which ones get deleted by this function. Activate "New Points Between Neighbors" and disable "Show Convex Hull". Create a series of points that form a concave polygon (Shown in Figure 15). |
Figure 15 |
|
Obviously, this will not work, so go ahead and click on that button
of the two polygons to let the Editor handle the repairs.
Every invalid point, which was marked in red, is deleted leaving you with a convex polygon (Shown in Figure 16). |
Figure 16 |
|
Next to your polygon repair button is another radio button consisting
of 2 ratio options (Shown in Figure 17). Your options are Square or Scene aspects.
Whenever you drag a new object onto your scene, it is always square. If you adjust its size and stretch it out, you have modified its ratio. |
Figure 17 |
|
Within the editor, you can use these toggles to view
your object in different ways. Assuming we stretched our object before editing its
collision polygon, Figure 18 shows your image if you have "Scene Aspect Ratio" toggled.
Using Scene Aspect Ratio makes the editor render the object exactly how it is in your game scene. |
Figure 18 |
| The final set of buttons is in the lower right (Shown in Figure 19). The green arrows allow you to undo and redo your latest action, just like any word or paint editor. |
Figure 19 |
|
The Display Scene Background does exactly what it describes.
If there are other objects in the scene that overlap the image you are editing, they will be
displayed in the editor as well.
Figure 20 shows the difference between having this toggle on and off. Finally, cancel will discard your changes and return back to your game editing. Save does the opposite by keeping your collision changes and returning to your main scene graph. |
Figure 20 |
When you've finished with the collision polygon, click the Save.
So, there you have it. The latest version of T2D's Collision Editor has quite a few more features than previous releases. Hopefully, these new features and this tutorial have made your collision editing a lot easier.


We visualize content!
- recontextualize revolutionary e-tailers
- repurpose dynamic networks
- utilize enterprise web-readiness
- maximize open-source channels
- orchestrate transparent functionalities
- iterate B2B initiatives
- orchestrate transparent models
- exploit e-business communities
- whiteboard global bandwidth
- facilitate transparent architectures
- scale revolutionary methodologies
- enhance scalable communities
- engineer magnetic niches
- exploit magnetic platforms
- strategize leading-edge paradigms


Dashboard
Credit
Amount
Debit
Amount
Balance
Amount
| Description | Type | Amount | Date Stamp | Tools |
|---|


7.11. EXPLAIN
Synopsis
EXPLAIN [ ( option [, ...] ) ] statement
where option can be one of:
FORMAT { TEXT | GRAPHVIZ }
TYPE { LOGICAL | DISTRIBUTED }
Description
Show the logical or distributed execution plan of a statement.
Examples
Logical plan:
presto:tiny> EXPLAIN SELECT regionkey, count(*) FROM nation GROUP BY 1;
Query Plan
----------------------------------------------------------------------------------------------------------
- Output[regionkey, _col1] => [regionkey:bigint, count:bigint]
_col1 := count
- Exchange[GATHER] => regionkey:bigint, count:bigint
- Aggregate(FINAL)[regionkey] => [regionkey:bigint, count:bigint]
count := "count"("count_8")
- Exchange[REPARTITION] => regionkey:bigint, count_8:bigint
- Aggregate(PARTIAL)[regionkey] => [regionkey:bigint, count_8:bigint]
count_8 := "count"(*)
- TableScan[tpch:tpch:nation:sf0.01, original constraint=true] => [regionkey:bigint]
LAYOUT: tpch:nation:sf0.01
regionkey := tpch:regionkey
Distributed plan:
presto:tiny> EXPLAIN (TYPE DISTRIBUTED) SELECT regionkey, count(*) FROM nation GROUP BY 1;
Query Plan
----------------------------------------------------------------------------------------------
Fragment 2 [SINGLE]
Output layout: [regionkey, count]
- Output[regionkey, _col1] => [regionkey:bigint, count:bigint]
_col1 := count
- RemoteSource[1] => [regionkey:bigint, count:bigint]
Fragment 1 [FIXED]
Output layout: [regionkey, count]
- Aggregate(FINAL)[regionkey] => [regionkey:bigint, count:bigint]
count := "count"("count_8")
- RemoteSource[0] => [regionkey:bigint, count_8:bigint]
Fragment 0 [SOURCE]
Output layout: [regionkey, count_8]
Output partitioning: [regionkey]
- Aggregate(PARTIAL)[regionkey] => [regionkey:bigint, count_8:bigint]
count_8 := "count"(*)
- TableScan[tpch:tpch:nation:sf0.01, original constraint=true] => [regionkey:bigint]
LAYOUT: tpch:nation:sf0.01
regionkey := tpch:regionkey