Object¶
Object Auditor¶
-
class
swift.obj.auditor.
AuditorWorker
(conf, logger, rcache, devices, zero_byte_only_at_fps=0)¶ Bases:
object
Walk through file system to audit objects
-
audit_all_objects
(mode='once', device_dirs=None)¶
-
create_recon_nested_dict
(top_level_key, device_list, item)¶
-
failsafe_object_audit
(location)¶ Entrypoint to object_audit, with a failsafe generic exception handler.
-
object_audit
(location)¶ Audits the given object location.
Parameters: location – an audit location (from diskfile.object_audit_location_generator)
-
record_stats
(obj_size)¶ Based on config’s object_size_stats will keep track of how many objects fall into the specified ranges. For example with the following:
object_size_stats = 10, 100, 1024
and your system has 3 objects of sizes: 5, 20, and 10000 bytes the log will look like: {“10”: 1, “100”: 1, “1024”: 0, “OVER”: 1}
-
-
class
swift.obj.auditor.
ObjectAuditor
(conf, **options)¶ Bases:
swift.common.daemon.Daemon
Audit objects.
-
audit_loop
(parent, zbo_fps, override_devices=None, **kwargs)¶ Parallel audit loop
-
clear_recon_cache
(auditor_type)¶ Clear recon cache entries
-
fork_child
(zero_byte_fps=False, **kwargs)¶ Child execution
-
run_audit
(**kwargs)¶ Run the object audit
-
run_forever
(*args, **kwargs)¶ Run the object audit until stopped.
-
run_once
(*args, **kwargs)¶ Run the object audit once
-
Object Backend¶
Disk File Interface for the Swift Object Server
The DiskFile, DiskFileWriter and DiskFileReader classes combined define the on-disk abstraction layer for supporting the object server REST API interfaces (excluding REPLICATE). Other implementations wishing to provide an alternative backend for the object server must implement the three classes. An example alternative implementation can be found in the mem_server.py and mem_diskfile.py modules along size this one.
The DiskFileManager is a reference implemenation specific class and is not part of the backend API.
The remaining methods in this module are considered implementation specific and are also not considered part of the backend API.
-
class
swift.obj.diskfile.
AuditLocation
(path, device, partition, policy)¶ Bases:
object
Represents an object location to be audited.
Other than being a bucket of data, the only useful thing this does is stringify to a filesystem path so the auditor’s logs look okay.
-
class
swift.obj.diskfile.
BaseDiskFile
(mgr, device_path, partition, account=None, container=None, obj=None, _datadir=None, policy=None, use_splice=False, pipe_size=None, use_linkat=False, **kwargs)¶ Bases:
object
Manage object files.
This specific implementation manages object files on a disk formatted with a POSIX-compliant file system that supports extended attributes as metadata on a file or directory.
Note
The arguments to the constructor are considered implementation specific. The API does not define the constructor arguments.
The following path format is used for data file locations: <devices_path/<device_dir>/<datadir>/<partdir>/<suffixdir>/<hashdir>/ <datafile>.<ext>
Parameters: - mgr – associated DiskFileManager instance
- device_path – path to the target device or drive
- partition – partition on the device in which the object lives
- account – account name for the object
- container – container name for the object
- obj – object name for the object
- _datadir – override the full datadir otherwise constructed here
- policy – the StoragePolicy instance
- use_splice – if true, use zero-copy splice() to send data
- pipe_size – size of pipe buffer used in zero-copy operations
- use_linkat – if True, use open() with linkat() to create obj file
-
account
¶
-
container
¶
-
content_length
¶
-
content_type
¶
-
content_type_timestamp
¶
-
create
(*args, **kwds)¶ Context manager to create a file. We create a temporary file first, and then return a DiskFileWriter object to encapsulate the state.
Note
An implementation is not required to perform on-disk preallocations even if the parameter is specified. But if it does and it fails, it must raise a DiskFileNoSpace exception.
Parameters: size – optional initial size of file to explicitly allocate on disk Raises: DiskFileNoSpace – if a size is specified and allocation fails
-
data_timestamp
¶
-
delete
(timestamp)¶ Delete the object.
This implementation creates a tombstone file using the given timestamp, and removes any older versions of the object file. Any file that has an older timestamp than timestamp will be deleted.
Note
An implementation is free to use or ignore the timestamp parameter.
Parameters: timestamp – timestamp to compare with each file Raises: DiskFileError – this implementation will raise the same errors as the create() method.
-
durable_timestamp
¶ Provides the timestamp of the newest data file found in the object directory.
Returns: A Timestamp instance, or None if no data file was found. Raises: DiskFileNotOpen – if the open() method has not been previously called on this instance.
-
fragments
¶
-
classmethod
from_hash_dir
(mgr, hash_dir_path, device_path, partition, policy)¶
-
get_datafile_metadata
()¶ Provide the datafile metadata for a previously opened object as a dictionary. This is metadata that was included when the object was first PUT, and does not include metadata set by any subsequent POST.
Returns: object’s datafile metadata dictionary Raises: DiskFileNotOpen – if the swift.obj.diskfile.DiskFile.open()
method was not previously invoked
-
get_metadata
()¶ Provide the metadata for a previously opened object as a dictionary.
Returns: object’s metadata dictionary Raises: DiskFileNotOpen – if the swift.obj.diskfile.DiskFile.open()
method was not previously invoked
-
get_metafile_metadata
()¶ Provide the metafile metadata for a previously opened object as a dictionary. This is metadata that was written by a POST and does not include any persistent metadata that was set by the original PUT.
Returns: object’s .meta file metadata dictionary, or None if there is no .meta file Raises: DiskFileNotOpen – if the swift.obj.diskfile.DiskFile.open()
method was not previously invoked
-
manager
¶
-
obj
¶
-
open
()¶ Open the object.
This implementation opens the data file representing the object, reads the associated metadata in the extended attributes, additionally combining metadata from fast-POST .meta files.
Note
An implementation is allowed to raise any of the following exceptions, but is only required to raise DiskFileNotExist when the object representation does not exist.
Raises: - DiskFileCollision – on name mis-match with metadata
- DiskFileNotExist – if the object does not exist
- DiskFileDeleted – if the object was previously deleted
- DiskFileQuarantined – if while reading metadata of the file some data did pass cross checks
Returns: itself for use as a context manager
-
read_metadata
()¶ Return the metadata for an object without requiring the caller to open the object first.
Returns: metadata dictionary for an object Raises: DiskFileError – this implementation will raise the same errors as the open() method.
-
reader
(keep_cache=False, _quarantine_hook=<function <lambda>>)¶ Return a
swift.common.swob.Response
class compatible “app_iter” object as defined byswift.obj.diskfile.DiskFileReader
.For this implementation, the responsibility of closing the open file is passed to the
swift.obj.diskfile.DiskFileReader
object.Parameters: - keep_cache – caller’s preference for keeping data read in the OS buffer cache
- _quarantine_hook – 1-arg callable called when obj quarantined; the arg is the reason for quarantine. Default is to ignore it. Not needed by the REST layer.
Returns: a
swift.obj.diskfile.DiskFileReader
object
-
reader_cls
= None¶
-
timestamp
¶
-
write_metadata
(metadata)¶ Write a block of metadata to an object without requiring the caller to create the object first. Supports fast-POST behavior semantics.
Parameters: metadata – dictionary of metadata to be associated with the object Raises: DiskFileError – this implementation will raise the same errors as the create() method.
-
writer_cls
= None¶
-
class
swift.obj.diskfile.
BaseDiskFileManager
(conf, logger)¶ Bases:
object
Management class for devices, providing common place for shared parameters and methods not provided by the DiskFile class (which primarily services the object server REST API layer).
The get_diskfile() method is how this implementation creates a DiskFile object.
Note
This class is reference implementation specific and not part of the pluggable on-disk backend API.
Note
TODO(portante): Not sure what the right name to recommend here, as “manager” seemed generic enough, though suggestions are welcome.
Parameters: - conf – caller provided configuration object
- logger – caller provided logger
-
cleanup_ondisk_files
(hsh_path, **kwargs)¶ Clean up on-disk files that are obsolete and gather the set of valid on-disk files for an object.
Parameters: - hsh_path – object hash path
- frag_index – if set, search for a specific fragment index .data file, otherwise accept the first valid .data file
Returns: a dict that may contain: valid on disk files keyed by their filename extension; a list of obsolete files stored under the key ‘obsolete’; a list of files remaining in the directory, reverse sorted, stored under the key ‘files’.
-
consolidate_hashes
(*args, **kwargs)¶
-
construct_dev_path
(device)¶ Construct the path to a device without checking if it is mounted.
Parameters: device – name of target device Returns: full path to the device
-
diskfile_cls
= None¶
-
get_dev_path
(device, mount_check=None)¶ Return the path to a device, first checking to see if either it is a proper mount point, or at least a directory depending on the mount_check configuration option.
Parameters: - device – name of target device
- mount_check – whether or not to check mountedness of device. Defaults to bool(self.mount_check).
Returns: full path to the device, None if the path to the device is not a proper mount point or directory.
-
get_diskfile
(device, partition, account, container, obj, policy, **kwargs)¶ Returns a BaseDiskFile instance for an object based on the object’s partition, path parts and policy.
Parameters: - device – name of target device
- partition – partition on device in which the object lives
- account – account name for the object
- container – container name for the object
- obj – object name for the object
- policy – the StoragePolicy instance
-
get_diskfile_from_audit_location
(audit_location)¶ Returns a BaseDiskFile instance for an object at the given AuditLocation.
Parameters: audit_location – object location to be audited
-
get_diskfile_from_hash
(device, partition, object_hash, policy, **kwargs)¶ Returns a DiskFile instance for an object at the given object_hash. Just in case someone thinks of refactoring, be sure DiskFileDeleted is not raised, but the DiskFile instance representing the tombstoned object is returned instead.
Parameters: - device – name of target device
- partition – partition on the device in which the object lives
- object_hash – the hash of an object path
- policy – the StoragePolicy instance
Raises: DiskFileNotExist – if the object does not exist
Returns: an instance of BaseDiskFile
-
get_hashes
(device, partition, suffixes, policy)¶ Parameters: - device – name of target device
- partition – partition name
- suffixes – a list of suffix directories to be recalculated
- policy – the StoragePolicy instance
Returns: a dictionary that maps suffix directories
-
get_ondisk_files
(files, datadir, verify=True, **kwargs)¶ Given a simple list of files names, determine the files that constitute a valid fileset i.e. a set of files that defines the state of an object, and determine the files that are obsolete and could be deleted. Note that some files may fall into neither category.
If a file is considered part of a valid fileset then its info dict will be added to the results dict, keyed by <extension>_info. Any files that are no longer required will have their info dicts added to a list stored under the key ‘obsolete’.
The results dict will always contain entries with keys ‘ts_file’, ‘data_file’ and ‘meta_file’. Their values will be the fully qualified path to a file of the corresponding type if there is such a file in the valid fileset, or None.
Parameters: - files – a list of file names.
- datadir – directory name files are from.
- verify – if True verify that the ondisk file contract has not been violated, otherwise do not verify.
Returns: - a dict that will contain keys:
ts_file -> path to a .ts file or None data_file -> path to a .data file or None meta_file -> path to a .meta file or None ctype_file -> path to a .meta file or None
- and may contain keys:
ts_info -> a file info dict for a .ts file data_info -> a file info dict for a .data file meta_info -> a file info dict for a .meta file ctype_info -> a file info dict for a .meta file which
contains the content-type value
- unexpected -> a list of file paths for unexpected
files
- possible_reclaim -> a list of file info dicts for possible
reclaimable files
obsolete -> a list of file info dicts for obsolete files
-
invalidate_hash
(*args, **kwargs)¶
-
make_on_disk_filename
(timestamp, ext=None, ctype_timestamp=None, *a, **kw)¶ Returns filename for given timestamp.
Parameters: Returns: a file name
-
object_audit_location_generator
(device_dirs=None, auditor_type='ALL')¶ Yield an AuditLocation for all objects stored under device_dirs.
Parameters: - device_dirs – directory of target device
- auditor_type – either ALL or ZBF
-
parse_on_disk_filename
(filename)¶ Parse an on disk file name.
Parameters: filename – the file name including extension Returns: a dict, with keys for timestamp, ext and ctype_timestamp: - timestamp is a
Timestamp
- ctype_timestamp is a
Timestamp
or None for .meta files, otherwise None - ext is a string, the file extension including the leading dot or the empty string if the filename has no extension.
Subclasses may override this method to add further keys to the returned dict.
Raises: DiskFileError – if any part of the filename is not able to be validated. - timestamp is a
-
pickle_async_update
(device, account, container, obj, data, timestamp, policy)¶ Write data describing a container update notification to a pickle file in the async_pending directory.
Parameters: - device – name of target device
- account – account name for the object
- container – container name for the object
- obj – object name for the object
- data – update data to be written to pickle file
- timestamp – a Timestamp
- policy – the StoragePolicy instance
-
quarantine_renamer
(*args, **kwargs)¶
-
replication_lock
(*args, **kwds)¶ A context manager that will lock on the device given, if configured to do so.
Parameters: device – name of target device Raises: ReplicationLockTimeout – If the lock on the device cannot be granted within the configured timeout.
-
yield_hashes
(device, partition, policy, suffixes=None, **kwargs)¶ Yields tuples of (full_path, hash_only, timestamps) for object information stored for the given device, partition, and (optionally) suffixes. If suffixes is None, all stored suffixes will be searched for object hashes. Note that if suffixes is not None but empty, such as [], then nothing will be yielded.
timestamps is a dict which may contain items mapping:
ts_data -> timestamp of data or tombstone file,
ts_meta -> timestamp of meta file, if one exists
- ts_ctype -> timestamp of meta file containing most recent
content-type value, if one exists
where timestamps are instances of
Timestamp
Parameters: - device – name of target device
- partition – partition name
- policy – the StoragePolicy instance
- suffixes – optional list of suffix directories to be searched
-
yield_suffixes
(device, partition, policy)¶ Yields tuples of (full_path, suffix_only) for suffixes stored on the given device and partition.
Parameters: - device – name of target device
- partition – partition name
- policy – the StoragePolicy instance
-
class
swift.obj.diskfile.
BaseDiskFileReader
(fp, data_file, obj_size, etag, disk_chunk_size, keep_cache_size, device_path, logger, quarantine_hook, use_splice, pipe_size, diskfile, keep_cache=False)¶ Bases:
object
Encapsulation of the WSGI read context for servicing GET REST API requests. Serves as the context manager object for the
swift.obj.diskfile.DiskFile
class’sswift.obj.diskfile.DiskFile.reader()
method.Note
The quarantining behavior of this method is considered implementation specific, and is not required of the API.
Note
The arguments to the constructor are considered implementation specific. The API does not define the constructor arguments.
Parameters: - fp – open file object pointer reference
- data_file – on-disk data file name for the object
- obj_size – verified on-disk size of the object
- etag – expected metadata etag value for entire file
- disk_chunk_size – size of reads from disk in bytes
- keep_cache_size – maximum object size that will be kept in cache
- device_path – on-disk device path, used when quarantining an obj
- logger – logger caller wants this object to use
- quarantine_hook – 1-arg callable called w/reason when quarantined
- use_splice – if true, use zero-copy splice() to send data
- pipe_size – size of pipe buffer used in zero-copy operations
- diskfile – the diskfile creating this DiskFileReader instance
- keep_cache – should resulting reads be kept in the buffer cache
-
app_iter_range
(start, stop)¶ Returns an iterator over the data file for range (start, stop)
-
app_iter_ranges
(ranges, content_type, boundary, size)¶ Returns an iterator over the data file for a set of ranges
-
can_zero_copy_send
()¶
-
close
()¶ Close the open file handle if present.
For this specific implementation, this method will handle quarantining the file if necessary.
-
manager
¶
-
zero_copy_send
(wsockfd)¶ Does some magic with splice() and tee() to move stuff from disk to network without ever touching userspace.
Parameters: wsockfd – file descriptor (integer) of the socket out which to send data
-
class
swift.obj.diskfile.
BaseDiskFileWriter
(name, datadir, fd, tmppath, bytes_per_sync, diskfile)¶ Bases:
object
Encapsulation of the write context for servicing PUT REST API requests. Serves as the context manager object for the
swift.obj.diskfile.DiskFile
class’sswift.obj.diskfile.DiskFile.create()
method.Note
It is the responsibility of the
swift.obj.diskfile.DiskFile.create()
method context manager to close the open file descriptor.Note
The arguments to the constructor are considered implementation specific. The API does not define the constructor arguments.
Parameters: - name – name of object from REST API
- datadir – on-disk directory object will end up in on
swift.obj.diskfile.DiskFileWriter.put()
- fd – open file descriptor of temporary file to receive data
- tmppath – full path name of the opened file descriptor
- bytes_per_sync – number bytes written between sync calls
- diskfile – the diskfile creating this DiskFileWriter instance
-
commit
(timestamp)¶ Perform any operations necessary to mark the object as durable. For replication policy type this is a no-op.
Parameters: timestamp – object put timestamp, an instance of Timestamp
-
manager
¶
-
put
(metadata)¶ Finalize writing the file on disk.
Parameters: metadata – dictionary of metadata to be associated with the object
-
put_succeeded
¶
-
write
(chunk)¶ Write a chunk of data to disk. All invocations of this method must come before invoking the :func:
For this implementation, the data is written into a temporary file.
Parameters: chunk – the chunk of data to write as a string object Returns: the total number of bytes written to an object
-
class
swift.obj.diskfile.
DiskFile
(mgr, device_path, partition, account=None, container=None, obj=None, _datadir=None, policy=None, use_splice=False, pipe_size=None, use_linkat=False, **kwargs)¶ Bases:
swift.obj.diskfile.BaseDiskFile
-
reader_cls
¶ alias of
DiskFileReader
-
writer_cls
¶ alias of
DiskFileWriter
-
-
class
swift.obj.diskfile.
DiskFileManager
(conf, logger)¶
-
class
swift.obj.diskfile.
DiskFileReader
(fp, data_file, obj_size, etag, disk_chunk_size, keep_cache_size, device_path, logger, quarantine_hook, use_splice, pipe_size, diskfile, keep_cache=False)¶
-
class
swift.obj.diskfile.
DiskFileRouter
(*args, **kwargs)¶ Bases:
object
-
policy_type_to_manager_cls
= {'erasure_coding': <class 'swift.obj.diskfile.ECDiskFileManager'>, 'replication': <class 'swift.obj.diskfile.DiskFileManager'>}¶
-
classmethod
register
(policy_type)¶ Decorator for Storage Policy implementations to register their DiskFile implementation.
-
-
class
swift.obj.diskfile.
DiskFileWriter
(name, datadir, fd, tmppath, bytes_per_sync, diskfile)¶ Bases:
swift.obj.diskfile.BaseDiskFileWriter
-
put
(metadata)¶ Finalize writing the file on disk.
Parameters: metadata – dictionary of metadata to be associated with the object
-
-
class
swift.obj.diskfile.
ECDiskFile
(*args, **kwargs)¶ Bases:
swift.obj.diskfile.BaseDiskFile
-
durable_timestamp
¶ Provides the timestamp of the newest durable file found in the object directory.
Returns: A Timestamp instance, or None if no durable file was found. Raises: DiskFileNotOpen – if the open() method has not been previously called on this instance.
-
fragments
¶ Provides information about all fragments that were found in the object directory, including fragments without a matching durable file, and including any fragment chosen to construct the opened diskfile.
Returns: A dict mapping <Timestamp instance> -> <list of frag indexes>, or None if the diskfile has not been opened or no fragments were found.
-
purge
(timestamp, frag_index)¶ Remove a tombstone file matching the specified timestamp or datafile matching the specified timestamp and fragment index from the object directory.
This provides the EC reconstructor/ssync process with a way to remove a tombstone or fragment from a handoff node after reverting it to its primary node.
The hash will be invalidated, and if empty or invalid the hsh_path will be removed on next cleanup_ondisk_files.
Parameters: - timestamp – the object timestamp, an instance of
Timestamp
- frag_index – fragment archive index, must be a whole number or None.
- timestamp – the object timestamp, an instance of
-
reader_cls
¶ alias of
ECDiskFileReader
-
writer_cls
¶ alias of
ECDiskFileWriter
-
-
class
swift.obj.diskfile.
ECDiskFileManager
(conf, logger)¶ Bases:
swift.obj.diskfile.BaseDiskFileManager
-
diskfile_cls
¶ alias of
ECDiskFile
-
make_on_disk_filename
(timestamp, ext=None, frag_index=None, ctype_timestamp=None, durable=False, *a, **kw)¶ Returns the EC specific filename for given timestamp.
Parameters: - timestamp – the object timestamp, an instance of
Timestamp
- ext – an optional string representing a file extension to be appended to the returned file name
- frag_index – a fragment archive index, used with .data extension only, must be a whole number.
- ctype_timestamp – an optional content-type timestamp, an instance
of
Timestamp
- durable – if True then include a durable marker in data filename.
Returns: a file name
Raises: DiskFileError – if ext==’.data’ and the kwarg frag_index is not a whole number
- timestamp – the object timestamp, an instance of
-
parse_on_disk_filename
(filename)¶ Returns timestamp(s) and other info extracted from a policy specific file name. For EC policy the data file name includes a fragment index and possibly a durable marker, both of which which must be stripped off to retrieve the timestamp.
Parameters: filename – the file name including extension Returns: - a dict, with keys for timestamp, frag_index, durable, ext and
- ctype_timestamp:
- timestamp is a
Timestamp
- frag_index is an int or None
- ctype_timestamp is a
Timestamp
or None for .meta files, otherwise None - ext is a string, the file extension including the leading dot or the empty string if the filename has no extension
- durable is a boolean that is True if the filename is a data file that includes a durable marker
Raises: DiskFileError – if any part of the filename is not able to be validated.
-
validate_fragment_index
(frag_index)¶ Return int representation of frag_index, or raise a DiskFileError if frag_index is not a whole number.
Parameters: frag_index – a fragment archive index
-
-
class
swift.obj.diskfile.
ECDiskFileReader
(fp, data_file, obj_size, etag, disk_chunk_size, keep_cache_size, device_path, logger, quarantine_hook, use_splice, pipe_size, diskfile, keep_cache=False)¶
-
class
swift.obj.diskfile.
ECDiskFileWriter
(name, datadir, fd, tmppath, bytes_per_sync, diskfile)¶ Bases:
swift.obj.diskfile.BaseDiskFileWriter
-
commit
(timestamp)¶ Finalize put by renaming the object data file to include a durable marker. We do this for EC policy because it requires a 2-phase put commit confirmation.
Parameters: timestamp – object put timestamp, an instance of Timestamp
Raises: DiskFileError – if the diskfile frag_index has not been set (either during initialisation or a call to put())
-
put
(metadata)¶ The only difference between this method and the replication policy DiskFileWriter method is adding the frag index to the metadata.
Parameters: metadata – dictionary of metadata to be associated with object
-
-
swift.obj.diskfile.
clear_auditor_status
(devices, auditor_type='ALL')¶
-
swift.obj.diskfile.
consolidate_hashes
(partition_dir)¶ Take what’s in hashes.pkl and hashes.invalid, combine them, write the result back to hashes.pkl, and clear out hashes.invalid.
Parameters: suffix_dir – absolute path to partition dir containing hashes.pkl and hashes.invalid Returns: the hashes, or None if there’s no hashes.pkl.
-
swift.obj.diskfile.
extract_policy
(obj_path)¶ Extracts the policy for an object (based on the name of the objects directory) given the device-relative path to the object. Returns None in the event that the path is malformed in some way.
The device-relative path is everything after the mount point; for example:
- /srv/node/d42/objects-5/30/179/
- 485dc017205a81df3af616d917c90179/1401811134.873649.data
would have device-relative path:
objects-5/30/179/485dc017205a81df3af616d917c90179/1401811134.873649.data
Parameters: obj_path – device-relative path of an object, or the full path Returns: a BaseStoragePolicy
or None
-
swift.obj.diskfile.
get_auditor_status
(datadir_path, logger, auditor_type)¶
-
swift.obj.diskfile.
invalidate_hash
(suffix_dir)¶ Invalidates the hash for a suffix_dir in the partition’s hashes file.
Parameters: suffix_dir – absolute path to suffix dir whose hash needs invalidating
-
swift.obj.diskfile.
object_audit_location_generator
(devices, mount_check=True, logger=None, device_dirs=None, auditor_type='ALL')¶ Given a devices path (e.g. “/srv/node”), yield an AuditLocation for all objects stored under that directory if device_dirs isn’t set. If device_dirs is set, only yield AuditLocation for the objects under the entries in device_dirs. The AuditLocation only knows the path to the hash directory, not to the .data file therein (if any). This is to avoid a double listdir(hash_dir); the DiskFile object will always do one, so we don’t.
Parameters: - devices – parent directory of the devices to be audited
- mount_check – flag to check if a mount check should be performed on devices
- logger – a logger object
- device_dirs – a list of directories under devices to traverse
- auditor_type – either ALL or ZBF
-
swift.obj.diskfile.
quarantine_renamer
(device_path, corrupted_file_path)¶ In the case that a file is corrupted, move it to a quarantined area to allow replication to fix it.
Params device_path: The path to the device the corrupted file is on. Params corrupted_file_path: The path to the file you want quarantined. Returns: path (str) of directory the file was moved to Raises: OSError – re-raises non errno.EEXIST / errno.ENOTEMPTY exceptions from rename
-
swift.obj.diskfile.
read_hashes
(partition_dir)¶ Read the existing hashes.pkl
Returns: a dict, the suffix hashes (if any), the key ‘valid’ will be False if hashes.pkl is corrupt, cannot be read or does not exist
-
swift.obj.diskfile.
read_metadata
(fd)¶ Helper function to read the pickled metadata from an object file.
Parameters: fd – file descriptor or filename to load the metadata from Returns: dictionary of metadata
-
swift.obj.diskfile.
strip_self
(f)¶ Wrapper to attach module level functions to base class.
-
swift.obj.diskfile.
update_auditor_status
(datadir_path, logger, partitions, auditor_type)¶
-
swift.obj.diskfile.
write_hashes
(partition_dir, hashes)¶ Write hashes to hashes.pkl
The updated key is added to hashes before it is written.
-
swift.obj.diskfile.
write_metadata
(fd, metadata, xattr_size=65536)¶ Helper function to write pickled metadata for an object file.
Parameters: - fd – file descriptor or filename to write the metadata
- metadata – metadata to write
Object Replicator¶
-
class
swift.obj.replicator.
ObjectReplicator
(conf, logger=None)¶ Bases:
swift.common.daemon.Daemon
Replicate objects.
Encapsulates most logic and data needed by the object replication process. Each call to .replicate() performs one replication pass. It’s up to the caller to do this in a loop.
-
build_replication_jobs
(policy, ips, override_devices=None, override_partitions=None)¶ Helper function for collect_jobs to build jobs for replication using replication style storage policy
-
check_ring
(object_ring)¶ Check to see if the ring has been updated :param object_ring: the ring to check
Returns: boolean indicating whether or not the ring has changed
-
collect_jobs
(override_devices=None, override_partitions=None, override_policies=None)¶ Returns a sorted list of jobs (dictionaries) that specify the partitions, nodes, etc to be rsynced.
Parameters: - override_devices – if set, only jobs on these devices will be returned
- override_partitions – if set, only jobs on these partitions will be returned
- override_policies – if set, only jobs in these storage policies will be returned
-
delete_handoff_objs
(job, delete_objs)¶
-
delete_partition
(path)¶
-
detect_lockups
()¶ In testing, the pool.waitall() call very occasionally failed to return. This is an attempt to make sure the replicator finishes its replication pass in some eventuality.
-
heartbeat
()¶ Loop that runs in the background during replication. It periodically logs progress.
-
kill_coros
()¶ Utility function that kills all coroutines currently running.
-
load_object_ring
(policy)¶ Make sure the policy’s rings are loaded.
Parameters: policy – the StoragePolicy instance Returns: appropriate ring object
-
replicate
(override_devices=None, override_partitions=None, override_policies=None)¶ Run a replication pass
-
rsync
(node, job, suffixes)¶ Uses rsync to implement the sync method. This was the first sync method in Swift.
-
run_forever
(*args, **kwargs)¶
-
run_once
(*args, **kwargs)¶
-
ssync
(node, job, suffixes, remote_check_objs=None)¶
-
stats_line
()¶ Logs various stats for the currently running replication pass.
-
sync
(node, job, suffixes, *args, **kwargs)¶ Synchronize local suffix directories from a partition with a remote node.
Parameters: - node – the “dev” entry for the remote node to sync with
- job – information about the partition being synced
- suffixes – a list of suffixes which need to be pushed
Returns: boolean and dictionary, boolean indicating success or failure
-
update
(job)¶ High-level method that replicates a single partition.
Parameters: job – a dict containing info about the partition to be replicated
-
update_deleted
(job)¶ High-level method that replicates a single partition that doesn’t belong on this node.
Parameters: job – a dict containing info about the partition to be replicated
-
-
class
swift.obj.ssync_sender.
Sender
(daemon, node, job, suffixes, remote_check_objs=None)¶ Bases:
object
Sends SSYNC requests to the object server.
These requests are eventually handled by
ssync_receiver
and full documentation about the process is there.-
connect
()¶ Establishes a connection and starts an SSYNC request with the object server.
-
disconnect
()¶ Closes down the connection to the object server once done with the SSYNC request.
-
missing_check
()¶ Handles the sender-side of the MISSING_CHECK step of a SSYNC request.
Full documentation of this can be found at
Receiver.missing_check()
.
-
readline
()¶ Reads a line from the SSYNC response body.
httplib has no readline and will block on read(x) until x is read, so we have to do the work ourselves. A bit of this is taken from Python’s httplib itself.
-
send_delete
(url_path, timestamp)¶ Sends a DELETE subrequest with the given information.
-
send_post
(url_path, df)¶
-
send_put
(url_path, df)¶ Sends a PUT subrequest for the url_path using the source df (DiskFile) and content_length.
-
updates
()¶ Handles the sender-side of the UPDATES step of an SSYNC request.
Full documentation of this can be found at
Receiver.updates()
.
-
-
swift.obj.ssync_sender.
decode_wanted
(parts)¶ Parse missing_check line parts to determine which parts of local diskfile were wanted by the receiver.
The encoder for parts is
encode_wanted()
-
swift.obj.ssync_sender.
encode_missing
(object_hash, ts_data, ts_meta=None, ts_ctype=None)¶ Returns a string representing the object hash, its data file timestamp and the delta forwards to its metafile and content-type timestamps, if non-zero, in the form:
<hash> <ts_data> [m:<hex delta to ts_meta>[,t:<hex delta to ts_ctype>]]
The decoder for this line is
decode_missing()
-
class
swift.obj.ssync_receiver.
Receiver
(app, request)¶ Bases:
object
Handles incoming SSYNC requests to the object server.
These requests come from the object-replicator daemon that uses
ssync_sender
.The number of concurrent SSYNC requests is restricted by use of a replication_semaphore and can be configured with the object-server.conf [object-server] replication_concurrency setting.
An SSYNC request is really just an HTTP conduit for sender/receiver replication communication. The overall SSYNC request should always succeed, but it will contain multiple requests within its request and response bodies. This “hack” is done so that replication concurrency can be managed.
The general process inside an SSYNC request is:
- Initialize the request: Basic request validation, mount check, acquire semaphore lock, etc..
- Missing check: Sender sends the hashes and timestamps of the object information it can send, receiver sends back the hashes it wants (doesn’t have or has an older timestamp).
- Updates: Sender sends the object information requested.
- Close down: Release semaphore lock, etc.
-
initialize_request
()¶ Basic validation of request and mount check.
This function will be called before attempting to acquire a replication semaphore lock, so contains only quick checks.
-
missing_check
()¶ Handles the receiver-side of the MISSING_CHECK step of a SSYNC request.
Receives a list of hashes and timestamps of object information the sender can provide and responds with a list of hashes desired, either because they’re missing or have an older timestamp locally.
The process is generally:
Sender sends :MISSING_CHECK: START and begins sending hash timestamp lines.
Receiver gets :MISSING_CHECK: START and begins reading the hash timestamp lines, collecting the hashes of those it desires.
Sender sends :MISSING_CHECK: END.
Receiver gets :MISSING_CHECK: END, responds with :MISSING_CHECK: START, followed by the list of <wanted_hash> specifiers it collected as being wanted (one per line), :MISSING_CHECK: END, and flushes any buffers.
Each <wanted_hash> specifier has the form <hash>[ <parts>] where <parts> is a string containing characters ‘d’ and/or ‘m’ indicating that only data or meta part of object respectively is required to be sync’d.
Sender gets :MISSING_CHECK: START and reads the list of hashes desired by the receiver until reading :MISSING_CHECK: END.
The collection and then response is so the sender doesn’t have to read while it writes to ensure network buffers don’t fill up and block everything.
-
updates
()¶ Handles the UPDATES step of an SSYNC request.
Receives a set of PUT and DELETE subrequests that will be routed to the object server itself for processing. These contain the information requested by the MISSING_CHECK step.
The PUT and DELETE subrequests are formatted pretty much exactly like regular HTTP requests, excepting the HTTP version on the first request line.
The process is generally:
- Sender sends :UPDATES: START and begins sending the PUT and DELETE subrequests.
- Receiver gets :UPDATES: START and begins routing the subrequests to the object server.
- Sender sends :UPDATES: END.
- Receiver gets :UPDATES: END and sends :UPDATES: START and :UPDATES: END (assuming no errors).
- Sender gets :UPDATES: START and :UPDATES: END.
If too many subrequests fail, as configured by replication_failure_threshold and replication_failure_ratio, the receiver will hang up the request early so as to not waste any more time.
At step 4, the receiver will send back an error if there were any failures (that didn’t cause a hangup due to the above thresholds) so the sender knows the whole was not entirely a success. This is so the sender knows if it can remove an out of place partition, for example.
-
swift.obj.ssync_receiver.
decode_missing
(line)¶ Parse a string of the form generated by
encode_missing()
and return a dict with keysobject_hash
,ts_data
,ts_meta
,ts_ctype
.The encoder for this line is
encode_missing()
-
swift.obj.ssync_receiver.
encode_wanted
(remote, local)¶ Compare a remote and local results and generate a wanted line.
Parameters: - remote – a dict, with ts_data and ts_meta keys in the form
returned by
decode_missing()
- local – a dict, possibly empty, with ts_data and ts_meta keys
in the form returned
Receiver._check_local()
The decoder for this line is
decode_wanted()
- remote – a dict, with ts_data and ts_meta keys in the form
returned by
Object Server¶
Object Server for Swift
-
class
swift.obj.server.
EventletPlungerString
¶ Bases:
str
Eventlet won’t send headers until it’s accumulated at least eventlet.wsgi.MINIMUM_CHUNK_SIZE bytes or the app iter is exhausted. If we want to send the response body behind Eventlet’s back, perhaps with some zero-copy wizardry, then we have to unclog the plumbing in eventlet.wsgi to force the headers out, so we use an EventletPlungerString to empty out all of Eventlet’s buffers.
-
class
swift.obj.server.
ObjectController
(conf, logger=None)¶ Bases:
swift.common.base_storage_server.BaseStorageServer
Implements the WSGI application for the Swift Object Server.
-
DELETE
(ctrl, *args, **kwargs)¶ Handle HTTP DELETE requests for the Swift Object Server.
-
GET
(ctrl, *args, **kwargs)¶ Handle HTTP GET requests for the Swift Object Server.
-
HEAD
(ctrl, *args, **kwargs)¶ Handle HTTP HEAD requests for the Swift Object Server.
-
POST
(ctrl, *args, **kwargs)¶ Handle HTTP POST requests for the Swift Object Server.
-
PUT
(ctrl, *args, **kwargs)¶ Handle HTTP PUT requests for the Swift Object Server.
-
REPLICATE
(ctrl, *args, **kwargs)¶ Handle REPLICATE requests for the Swift Object Server. This is used by the object replicator to get hashes for directories.
Note that the name REPLICATE is preserved for historical reasons as this verb really just returns the hashes information for the specified parameters and is used, for example, by both replication and EC.
-
SSYNC
(ctrl, *args, **kwargs)¶
-
async_update
(op, account, container, obj, host, partition, contdevice, headers_out, objdevice, policy, logger_thread_locals=None)¶ Sends or saves an async update.
Parameters: - op – operation performed (ex: ‘PUT’, or ‘DELETE’)
- account – account name for the object
- container – container name for the object
- obj – object name
- host – host that the container is on
- partition – partition that the container is on
- contdevice – device name that the container is on
- headers_out – dictionary of headers to send in the container request
- objdevice – device name that the object is in
- policy – the associated BaseStoragePolicy instance
- logger_thread_locals – The thread local values to be set on the self.logger to retain transaction logging information.
-
container_update
(op, account, container, obj, request, headers_out, objdevice, policy)¶ Update the container when objects are updated.
Parameters: - op – operation performed (ex: ‘PUT’, or ‘DELETE’)
- account – account name for the object
- container – container name for the object
- obj – object name
- request – the original request object driving the update
- headers_out – dictionary of headers to send in the container request(s)
- objdevice – device name that the object is in
- policy – the BaseStoragePolicy instance
-
delete_at_update
(op, delete_at, account, container, obj, request, objdevice, policy)¶ Update the expiring objects container when objects are updated.
Parameters: - op – operation performed (ex: ‘PUT’, or ‘DELETE’)
- delete_at – scheduled delete in UNIX seconds, int
- account – account name for the object
- container – container name for the object
- obj – object name
- request – the original request driving the update
- objdevice – device name that the object is in
- policy – the BaseStoragePolicy instance (used for tmp dir)
-
get_diskfile
(device, partition, account, container, obj, policy, **kwargs)¶ Utility method for instantiating a DiskFile object supporting a given REST API.
An implementation of the object server that wants to use a different DiskFile class would simply over-ride this method to provide that behavior.
-
server_type
= 'object-server'¶
-
setup
(conf)¶ Implementation specific setup. This method is called at the very end by the constructor to allow a specific implementation to modify existing attributes or add its own attributes.
Parameters: conf – WSGI configuration parameter
-
-
swift.obj.server.
app_factory
(global_conf, **local_conf)¶ paste.deploy app factory for creating WSGI object server apps
-
swift.obj.server.
drain
(file_like, read_size, timeout)¶ Read and discard any bytes from file_like.
Parameters: - file_like – file-like object to read from
- read_size – how big a chunk to read at a time
- timeout – how long to wait for a read (use None for no timeout)
Raises: ChunkReadTimeout – if no chunk was read in time
-
swift.obj.server.
global_conf_callback
(preloaded_app_conf, global_conf)¶ Callback for swift.common.wsgi.run_wsgi during the global_conf creation so that we can add our replication_semaphore, used to limit the number of concurrent SSYNC_REQUESTS across all workers.
Parameters: - preloaded_app_conf – The preloaded conf for the WSGI app. This conf instance will go away, so just read from it, don’t write.
- global_conf – The global conf that will eventually be passed to the app_factory function later. This conf is created before the worker subprocesses are forked, so can be useful to set up semaphores, shared memory, etc.
-
swift.obj.server.
iter_mime_headers_and_bodies
(wsgi_input, mime_boundary, read_chunk_size)¶
Object Updater¶
-
class
swift.obj.updater.
ObjectUpdater
(conf, logger=None)¶ Bases:
swift.common.daemon.Daemon
Update object information in container listings.
-
get_container_ring
()¶ Get the container ring. Load it, if it hasn’t been yet.
-
object_sweep
(device)¶ If there are async pendings on the device, walk each one and update.
Parameters: device – path to device
-
object_update
(node, part, op, obj, headers_out)¶ Perform the object update to the container
Parameters: - node – node dictionary from the container ring
- part – partition that holds the container
- op – operation performed (ex: ‘PUT’ or ‘DELETE’)
- obj – object name being updated
- headers_out – headers to send with the update
-
process_object_update
(update_path, device, policy)¶ Process the object information to be updated and update.
Parameters: - update_path – path to pickled object update file
- device – path to device
- policy – storage policy of object update
-
run_forever
(*args, **kwargs)¶ Run the updater continuously.
-
run_once
(*args, **kwargs)¶ Run the updater once.
-
-
swift.obj.updater.
random
() → x in the interval [0, 1).¶


|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
This document explains how to set up IIS to cooperate with Tomcat. Normally IIS can not execute Servlets and Java Server Pages (JSPs), configuring IIS to use the JK ISAPI redirector plugin will let IIS send servlet and JSP requests to Tomcat (and this way, serve them to clients). It is recommanded that you also read the Workers HowTo document to learn how to setup the working entities between your WebServer and Tomcat Engines.
${tomcat_home} is the root directory of tomcat. Your Tomcat installation should have the following subdirectories:
In all the examples in this document ${tomcat_home} will be c:\jakarta-tomcat . A worker is defined to be a tomcat process that accepts work from the IIS server.
The IIS-Tomcat redirector was developed and tested on:
The redirector uses ajp12 and ajp13 to send requests to the Tomcat containers. There is also an option to use Tomcat in process, more about the in-process mode can be found in the in process howto.
The ajp12 protocol is only available in Tomcat 3.2.x and 3.3.x. The ajp12 has been deprecated with Tomcat 3.3.x and you should use instead ajp13 which is the only ajp protocol known by Tomcat 4.0.x, 4.1.x and 5. Of course Tomcat 3.2.x and 3.3.x also support ajp13 protocol. Others servlet engines such as jetty have support for ajp13 protocol
A pre-built version of the ISAPI redirector server plugin, isapi_redirect.dll, is available under the win32/i386 directory of jakarta-tomcat-connectors distribution. For those using Netscape as your browser, try downloading a zip version of the file, if available. There can be problems using Netscape to download DLL files. You can also build a copy locally from the source present in jakarta-tomcat-connectors distribution. The Tomcat redirector requires three entities:
The installation includes the following parts:
In this document I will assume that isapi_redirect.dll is placed in c:\jakarta-tomcat\bin\win32\i386\isapi_redirect.dll and that you created the properties files are in c:\jakarta-tomcat\conf .
That's all, you should now start Tomcat and ask IIS to serve you the /examples context. Try http://localhost/examples/jsp/index.html for example and execute some of the JSP examples. If this does not work successfully, refer to the Troubleshooting section below for help on correcting the problem.
The examples context is useful for verifying your installation, but you will also need to add your own contexts. Adding a new context requires two operations:
Adding a context to the ISAPI redirector is simple, all you need to do is to edit your uriworkermap.properties and to add a line that looks like:
Workers and their name are defined in workers.properties, by default workers.properties comes with a single pre-configured worker named "defworker" so you can use it. As an example, if you want to add a context named "shop", the line that you should add to uriworkermap.properties will be:
A feature is present till Tomcat 3.2, where a uriworkermap.properties-auto is automatically written each time Tomcat is started. This file includes settings for each of the contexts that Tomcat will serve during its run. Each context has settings to have Tomcat handle servlet and JSP requests, but by default static content is left to be served by IIS. Each context also has a commented out setting to have Tomcat handle all requests to the context. You can rename this file (so it won't be overwritten the next time Tomcat is started) and uncomment this setting or make other customizations. You may also use this file as is in your worker_mount_file setting.
Sometimes it is better to have IIS serve the static pages (html, gif, jpeg etc.) even if these files are part of a context served by Tomcat. For example, consider the html and gif files in the examples context, there is no need to serve them from the Tomcat process, IIS will suffice. Making IIS serve static files that are part of the Tomcat contexts requires the following:
Adding a Tomcat context to IIS requires the addition of a new IIS virtual directory that covers the Tomcat context. For example adding a /example IIS virtual directory that covers the c:\jakarta-tomcat\webapps\examples directory. Configuring the redirector is somewhat harder, you will need to specify the exact URL-Path pattern(s) that you want Tomcat to handle (usually only JSP files and servlets). This requires a change to the uriworkermap.properties :
As you can see the second configuration is more explicit, it actually instruct the redirector to redirect only requests to resources under /examples/servlet/ and resources under /examples/ whose name ends with .jsp. This is similar to what is automically written to the uriworkermap.properties-auto file for each context. You can even be more explicit and provide lines such as:
that instructs the redirector to redirect request whose URL-Path equals /example/servletname to the worker named defworker.
Each servlet application (context) has a special directory named WEB-INF, this directory contains sensitive configurations data and Java classes and must be kept hidden from web users. Using the IIS management console it is possible to protect the WEB-INF directory from user access, this however requires the administrator to remember that. To avoid this need the redirector plugin automatically protects your WEB-INF directories by rejecting any request that contains WEB-INF in its URL-Path.
Sometimes you want to serve different contexts with different Tomcat processes (for example to spread the load among different machines). To achieve such goal you will need to define several workers and assign each context with its own worker. Defining workers is done in workers.properties, this file includes two types of entries:
The above example defined two workers, now we can use these workers to serve two different contexts each with its own worker :
As you can see the examples context is served by worker1 while the webpages context is served by worker2 . More informations on using and configuring workers in the Workers HowTO
The redirector was developed using Visual C++ Ver.6.0, so having this environment is a prereq if you want to perform a custom build. You should also have IIS developer SDK The steps that you need to take are:
If msdev is not in your path, enter the full path to msdev.exe. This will build both release and debug versions of the redirector plugin. An alternative will be to open the isapi workspace file (isapi.dsw) in msdev and build it using the build menu.
It is easy to have the ISAPI redirector not work the first time you try to install it. If this happens to you, here are some steps to follow to try to correct the problem. These steps aren't guaranteed to cover all possible problems, but they should help find the typical mistakes. If you make any corrections during these steps, restart the IIS service as described above in the last step of the installation, then retry the step. To enable error tracking, make sure web site activity is being logged. For PWS 4.0 make sure "Save Web Site Activity Log" is checked in the Advanced Options of the Personal Web Manager.
Note: These steps assume your
worker_mount_file
setting points to an unmodified copy of the
uriworkermap.properties
file.
Start the IIS service and Tomcat. Check for the presence of the ISAPI redirector log file you specified in the log_file setting. If not found, verify the following:
Invoke the URL http://localhost/examples/jsp/index.html in your browser. Case is important in Tomcat. The characters following "localhost" in the URL must be lower case. If the page fails to appear, stop the IIS service (required to view the IIS log file). Then examine the last line in the IIS log file in found in SYSTEM/LogFiles/W3SVC1 : If the last line contains:
then the ISAPI redirector is not recognizing that it should be handling requests for the "/examples" context. Check the following:
If the last line contains something like:
then the ISAPI redirector is recognizing that it should handle the request, but is not successful at getting Tomcat to service the request. You should check the HTTP error code following GET "/..." :
If the above settings are correct, the index.html page should appear in your browser. You should also be able to click the Execute links to execute the JSP examples.
Start the World Wide Web Publishing Service and Tomcat. Check for the presence of the ISAPI redirector log file you specified in the log_file setting. If not found, check the following:
Check the jakarta filter you added and make sure its status shows a green upward-pointing arrow. If not, check the following:
Invoke the URL http://localhost/examples/jsp/index.html in your browser. Case is important in Tomcat. The characters following "localhost" in the URL must be lower case. If the page fails to appear, examine the last line in the IIS server log file in found in SYSTEM32/LogFiles/W3SVC1. The last line should contain something like: GET "/jakarta/isapi_redirect.dll HTTP1.1", which indicates the ISAPI redirector is recognizing that it should handle the request. You should check the HTTP error code following GET "/..." :
If the above settings are correct, the index.html page should appear in your browser. You should also be able to click the Execute links to execute the JSP examples. |


| Network Working Group | J.C. Gregorio |
| INTERNET DRAFT | BitWorking, Inc |
| <draft-gregorio-03.html> | July 2003 |
| Category: Standards Track | |
| Expires: January 2004 |
This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress".
The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.
This Internet-Draft will expire in January 2004.
Copyright (C) The Internet Society (2003). All Rights Reserved.
This memo presents a technique for using XML (Extensible Markup Language) and HTTP (HyperText Transport Protocol) to edit content.
To provide feedback on this draft RFC please visit the Echo Wiki.
1
Introduction
2
Terminology
3
Scope
4
The EchoAPI Model
4.1
Introspection
4.2
Creating a new Entry request
4.2.1
Retrieving an Entry
4.3
Editing an entry
4.4
Deleting an entry
4.5
Finding an entry
4.6
Editing User Prefs
4.7
Adding Comments
4.7.1
Comment Auto-Discovery
<IP_ADDRESS>
HTML Auto-Discovery
<IP_ADDRESS>
Echo Feed Auto-Discovery
4.8
Introspection Discovery
5
Functional Specification
5.1
Formats
5.2
Actions
5.2.1
Create Entry
5.2.2
Edit Entry
5.2.3
Delete Entry
5.2.4
Get List of Entries
5.2.5
Get User Info
5.2.6
Set User Info
5.2.7
Get Categories
5.2.8
Set Categories
5.2.9
Result Codes
5.2.10
Content
6
Security Considerations
7
Revision History
§
References
§
Author's Address
§
Full Copyright Statement
EchoAPI is an application level protocol for publishing, and editing web resources. EchoAPI unifies many disparate publishing mechanisms into a single, simple, extensible protocol. The protocol at its core is the HTTP transport of an XML payload.
To provide feedback on this draft RFC please visit the Echo Wiki.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY",the and "OPTIONAL" in this document are to be interpreted as described in RFC2119.
This document covers the editing of content of a periodically updating website using the HTTP and XML. Many of the XML payloads are in Echo format, which will not be documented here.
This specification will mirror some of the functionality of the Blogger 2.0 API, which includes support for the Server Introspection API.
When editing the content of the website http://example.org/reilly, the first thing to do is find out the servers capabilites. Each server may only implement a subset of this specification, and the 'introspection' file lists all the functions that each site supports. This mirrors the Server Introspection API used by the Blogger 2.0 API.
<?xml version="1.0" encoding='iso-8859-1'?> <introspection> <edit-entry>http://example.org/reilly/</edit-entry> <user-prefs>http://example.org/reilly/prefs</user-prefs> <search-entries>http://example.org/reilly/search</search-entries> </introspection>
This site furnishes only a minimal set of functionality. A more full featured site might return:
<?xml version="1.0" encoding='iso-8859-1'?> <introspection> <edit-entry>http://example.org/reilly/</edit-entry> <user-prefs>http://example.org/reilly/prefs</user-prefs> <search-entries>http://example.org/reilly/search</search-entries> <edit-template>http://example.org/reilly/templates</edit-template> <categories>http://example.org/reilly/categories</categories> </introspection>
Where to store the introspection file on the web and how to make the URI of the introspection file easy to find will be discussed later.
This particular introspection file for http://example.org/reilly tells us that the EchoAPI implementation supports both the 'edit-entry' and the 'user-prefs' interfaces, and it also specifies the URIs to use for each of these interfaces.
Note that the example URIs given are not normative, nor are there any constraints on the URIs that can be specified. The 'edit-entry' URI could just have easily been:
http://dev.example.net/api?userid=reilly&action=edit
HTTP POST is used for content creation. The new content is formatted as an Echo Entry and then POSTed to the URI given in the <edit-entry/> element of the Introspection file.
To create a new Entry on the site the client connects to port 80 on http://example.org and sends:
POST /reilly HTTP/1.1 Content-Type: application/not-echo+xml <?xml version="1.0" encoding='iso-8859-1'?> <entry> <title>My First Entry</title> <subtitle>In which a newbie learns to blog...</subtitle> A very boring entry... <author> <name>Bob B. Bobbington</name> <homepage>http://bob.name/</homepage> <weblog>http://bob.blog/</weblog> </author> <issued>2003-02-05T12:29:29</issued> <content type="application/xhtml+xml" xml:lang="en-us">
Hello, weblog world! 2 < 4!
 </content> </entry>
This would create a new weblog Entry with the title "My First Entry".
Assuming everything goes well and the Entry is created the response might look like:
HTTP/1.1 201 Created Location: http://example.org/reilly/1
The URI of the filled in Entry is returned via the HTTP headers. If a GET is done on the URI then a filled in Entry would be returned, including the elements <link/>, <id/>, <created/>' and <modified/>.
To retreive the new Entry on the site the client connects to port 80 on http://example.org and sends:
GET /reilly/1 HTTP/1.1 Content-Type: application/not-echo+xml
This would retrieve the Entry just created.
The response might look like:
HTTP/1.1 200 Ok Content-Type: application/not-echo+xml <?xml version="1.0" encoding='iso-8859-1'?> <entry> <title>My First Entry</title> <subtitle>In which a newbie learns to blog...</subtitle> A very boring entry... <author> <name>Bob B. Bobbington</name> <homepage>http://bob.name/</homepage> <weblog>http://bob.blog/</weblog> </author> <issued>2003-02-05T12:29:29</issued> <created>2003-02-05T14:10:58Z</created> <modified>2003-02-05T14:10:58Z</modified> <link>http://example.org/reilly/2003/02/05#My_First_Entry</link> <id>urn:example.org:reilly:1</id> <content type="application/xhtml+xml" xml:lang="en-us">
Hello, weblog world! 2 < 4!
 </content> </entry>
Information has been added to the Entry, including the <link/>, <id/>, <created/> and <modified/> elements.
@@ Editors Note: Beware the Echo syntax is still in flux and the above example is not normative. @@
Editing an Entry is the same as creating one, the difference is that the Entry you POST already has 'id' filled in. The modified Echo Entry is POSTed to the same URI used to create an entry, that is, the URI stored in the <create-entry/> element of the Introspection file.
If the <content/> element is updated the transaction might look like:
POST /reilly HTTP/1.1 Content-Type: application/not-echo+xml <?xml version="1.0" encoding='iso-8859-1'?> <entry> <title>My First Entry</title> <subtitle>In which a newbie learns to blog...</subtitle> A very boring entry... <author> <name>Bob B. Bobbington</name> <homepage>http://bob.name/</homepage> <weblog>http://bob.blog/</weblog> </author> <issued>2003-02-05T12:29:29</issued> <created>2003-02-05T14:10:58Z</created> <modified>2003-02-05T14:10:58Z</modified> <link>http://example.org/reilly/2003/02/05#My_First_Entry</link> <id>urn:example.org:reilly:1</id> <content type="application/xhtml+xml" xml:lang="en-us">
A big hello to the weblog world! 2 < 4!
 </content> </entry>
The server knows this is an update instead of creation from the existence of the <id/> element.
The response from the server, on success, returns a 205 HTTP status code.
HTTP/1.1 205 Reset Content
The <content/> element now reflects the updated content, and the <modifed/> element has been updated to reflect when the change was made.
As was done after creation, a GET to the URI of the Echo Entry would return the modified Entry.
To retreive the modified Entry the client connects to port 80 on http://example.org and sends:
GET /reilly/1 HTTP/1.1 Content-Type: application/not-echo+xml
This would retrieve the recently modified Entry.
The response might look like:
HTTP/1.1 200 Ok Content-Type: application/not-echo+xml <?xml version="1.0" encoding='iso-8859-1'?> <entry> <title>My First Entry</title> <subtitle>In which a newbie learns to blog...</subtitle> A very boring entry... <author> <name>Bob B. Bobbington</name> <homepage>http://bob.name/</homepage> <weblog>http://bob.blog/</weblog> </author> <issued>2003-02-05T12:29:29</issued> <created>2003-02-05T14:10:58Z</created> <modified>2003-02-05T14:22:03Z</modified> <link>http://example.org/reilly/2003/02/05#My_First_Entry</link> <id>urn:example.org:reilly:1</id> <content type="application/xhtml+xml" xml:lang="en-us">
A big hello to the weblog world! 2 < 4!
 </content> </entry>
The <content/> element now reflects the updated content, and the <modifed/> element has been updated to reflect when the change was made.
Deleting an Entry also uses the same 'edit-entry' URI.
To delete an Entry the client connects to port 80 on the server and sends:
POST /reilly HTTP/1.1 Content-Type: application/not-echo+xml <?xml version="1.0" encoding='iso-8859-1'?> <delete id="urn:example.org:reilly:1" />
The POST handler on the server side uses the value of the 'id' attribute to determine which Entry to delete.
The response from the server upon a successful deletion is:
HTTP/1.1 200 Ok
Finding an Entry uses the 'search-entries' URI from the Introspection file. For the example Introspection file given, that is the URI http://example.org/reilly/search. The search parameters given are used to select which Entries are listed in the response.
For example, to retrieve the locations of the last 20 Entries, the client connects to port 80 on the server and sends:
POST /reilly/search HTTP/1.1 Content-Type: application/not-echo+xml <?xml version="1.0" encoding='iso-8859-1'?> <search> <num-entries>20</num-entries> </search>
The POST handler on the server side searches for Entries that match the criteria and returns a response that lists those matches.
The response from the server might look like:
HTTP/1.1 200 Ok Content-Type: application/not-echo+xml <search-results> <match title="My First Post">http://example.org/reilly/1</match> <match title="My Second Post">http://example.org/reilly/2</match> </search-results>
There will be one <match/> element for every Entry that matches the given criteria. Each <match/> element contains the URI of an Entry and an attribute 'title' which contains the title of the Entry. If the client needs the Echo Entry then it performs an HTTP GET on the given URI as outlined in section 4.2.1.
There are other search critera that can be added as child elements of <search-results/>. For example, the <all/> element, if present, requests that the server return a list of all Entries on the site. Other additional elements that can be added are <start-range/> and <endRange/>. Access to the entries is by an index, with the first Entry having an index number of 0.
For example, this request will return a list of all Entries from index 2 to index 27.
POST /reilly/search HTTP/1.1 Content-Type: application/not-echo+xml <?xml version="1.0" encoding='iso-8859-1'?> <search> <start-range>2</start-range> <end-range>27</end-range> </search>
The rest of the search parameters and their properties are covered in detail in section 5.1.
The preferences for the user are edited through the URI supplied in the <user-prefs/> element of the introspection file. To retrieve the current user preferences issue and HTTP GET on the URI.
For the website at http://example.org/reilly, the request would look like:
GET /reilly/prefs HTTP/1.1
This would return a list of user configuration parameters and their values.
The reponse would look like:
HTTP/1.1 200 OK Content-Type: application/not-echo+xml <?xml version="1.0" encoding='iso-8859-1'?> <userprefs xmlns="http://www.purl.org/EchoAPI/UserPrefs/"> <name>Reilly</name> <id>1234</id><EMAIL_ADDRESS></userprefs>
The values can then be updated, and doing an HTTP POST of the updated file back to the 'user-prefs' URI will update the user preferences.
For example, if we update our email address, from<EMAIL_ADDRESS>to<EMAIL_ADDRESS>we would then POST back to the 'user-prefs' URI:
POST /reilly/prefs HTTP/1.1 Content-Type: application/not-echo+xml <?xml version="1.0" encoding='iso-8859-1'?> <userprefs xmlns="http://www.purl.org/EchoAPI/UserPrefs/"> <name>Reilly</name> <id>1234</id><EMAIL_ADDRESS></userprefs>
Assuming the update goes successfully, the updated user preferences will be returned in the response.
The reponse would look like:
HTTP/1.1 200 OK Content-Type: application/not-echo+xml <?xml version="1.0" encoding='iso-8859-1'?> <userprefs xmlns="http://www.purl.org/EchoAPI/UserPrefs/"> <name>Reilly</name> <id>1234</id><EMAIL_ADDRESS></userprefs>
This part of the EchoAPI does not use the URIs listed in the Introspection file.
There are different interfaces now circulating for an TrackBack, Ping-Back, Post-It. All of these are systems for adding comment-like information to an Entry. The only thing missing from the mix is a way to do comments themselves. This section of the specification is intended to be a roll-up of all the above specifications and to cover comments as well.
Creating a comment is similar to creating an Entry, that is, you HTTP POST an Entry to a URI. For now we will defer discussing how to obtain the corrent URI to POST to.
For example, if "http://example.org/reilly/1/comments" is the URI that accepts comments for the first Entry on the website http://example.org/reilly, then the client would open port 80 on example.org and send:
POST /reilly/1/comments HTTP/1.1 Content-Type: application/not-echo/xml <entry> <title>My Comment</title> <author> <name>Fred F. Fobbington</name> <homepage>http://fred.name/</homepage> <weblog>http://fred.blog/</weblog> </author> <issued>2003-02-05T12:29:29</issued> <content type="application/xhtml+xml" xml:lang="en-us">
Welcome to the neighborhood!
 </content> </entry>
This would create a new comment for the first Entry on "http://example.org/reilly".
The response to this POST, assuming everything went without error, is a filled in Echo Entry, and would look like this:
HTTP/1.1 201 Created Content-Type: application/not-echo+xml <entry> <title>My Comment</title> <author> <name>Fred F. Fobbington</name> <homepage>http://fred.name/</homepage> <weblog>http://fred.blog/</weblog> </author> <issued>2003-02-05T12:29:29</issued> <created>2003-02-05T14:20:12Z</created> <modified>2003-02-05T14:20:12Z</modified> <link>http://example.org/reilly/2003/02/05#comment1</link> <id>urn:example.org:reilly:1:1</id> <content type="application/xhtml+xml" xml:lang="en-us">
Welcome to the neighborhood!
 </content> </entry>
This would create a new comment/trackback for the first Entry on "http://example.org/reilly".
Two mechanisms are available for discovering the URI that you would use to create a comment. The first is a way to put that information in HTML, the second is a way to embed that information in an Echo feed.
The <link> element has been successful in finding RSS feeds and is appropriate to use here for discovering the "commentURI" in HTML pages. The <link> tag goes in the  section of an HTML page and is used to indicate a document relationship. For a similar example of using the link tag to discover a URI, refer to the usage of of the link tag for RSS Auto-Discovery. In this case the form of the link tag used to indicate the location of the comment URI is:
<link rel="service.comment" type="application/not-echo+xml" href="url goes here" title="EchoComment">
Where href should be set to the URI that accepts POSTs for comments. Applications looking for a URI to post comments to need to parse out the headers of the web page and look for a link tag that has a relation rel of "comment" and a mime-type of "application/not-echo+xml".
There is a <comment/> element in each Entry that is used to provide the location of the URI that accepts comment POSTs. This is providing the same information as the link tag does in HTML. @@ No example yet. Needs to be specified. @@
The URI of the Introspection file is discovered using RSD. RSD, which stands for Really Simple Discovery, is used for automatic discovery of supported editing APIs.
Here is an RSD file for the website at http://example.org/reilly.
<?xml version="1.0" ?> <rsd version="0.7" xmlns="http://archipelago.phrasewise.com/rsd" > <service> <engineName>EchoAPI</engineName> <engineLink>http://purl.org/Echo/API</engineLink> <homePageLink>http://example.org/reilly</homePageLink> <apis> <api name="EchoAPI" preferred="true" location="http://example.org/reilly/introspection" blogID=""> </api> </apis> </service> </rsd>
Note that RSD also specifies the inclusion of a <link> tag included in the header of the website homepage, to aid in discovery of the RSD file.
@@ TBD @@ More formal specification of all the XML formats used in this protocol.
@@ TBD @@ A more formal specification of all the actions.
@@TBD@@ Mostly 200 for normal responses, seems that 30X codes for redirects are useful but might make implementations a little harder to implement. @@Ref CommentAPI problems returning a 303 vs 200@@
@@TBD@@ How does an Echo document change when used in these different contexts? Which parts that are required when found in a feed become optional when used in creating an Entry?
@@TBD@@ Talk here about using HTTP basic and digest authentication.
@@TBD@@ Talk here about denial of service attacks using large XML files, or the billion laughs DTD attack.
10Jul2003 - Added a link to the Wiki near the front of the document. Added a section on finding an Entry. Retrieving an Entry now broken out into it's own section. Changed the HTTP status code for a successful editing of an Entry to 205.
7Jul2003 - Entries are no longer returned from POSTs, instead they are retrieved via GET. Cleaned up figure titles, as they are rendered poorly in HTML. All content-types have been changed to application/not-echo+xml.
5Jul2003 - Renamed from EchoAPI.html to follow the more commonly used format: draft-gregorio-NN.html. Renamed all references to URL to URI. Broke out introspection into it's own section. Added the Revision History section. Added more to the warning that the example URIs are not normative.
| [RFC2119] | Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. |
| Joe Gregorio | |
| BitWorking, Inc | |
| 1002 Heathwood Dairy Rd. |
|
| Apex, NC 27502 | |
| US | |
| Phone: | +1<PHONE_NUMBER> |
| EMail: |<EMAIL_ADDRESS>|
| URI: | http://bitworking.com/ |
Copyright (C) The Internet Society (2003). All Rights Reserved.
This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.
The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

Funding for the RFC editor function is currently provided by the Internet Society.


I decided to study computer science and engineering because I thought to myself, what if I want to make a game some day. Since I can recall, games have always been with me. I am grateful to them thus I wanted to some day create one. Alas, I left that idea behind me. I dedicated my student life to gain skills in many other areas in computer science to some day land a good job and gain a stable life-style right after graduation. Then recently, I meditated on what I really want to do for my future, and I decided that I want to help people using my skills. Thus I searched for jobs where I could meet these two requirements. I landed a job at Accenture. Once I graduate, I will start there, and I hope that whatever I do can help people in some way.
The person that inspires me the most and makes me try my best in anything I do is my girlfriend Malena. She has always been there for me since first year in university. She was my best friend before we became a couple so we already knew each other very well. We’ve practically been together for 6 years and been a couple for 3 now. I have basically decided to settle down with her in the near future and for that I keep trying my best at anything I do.
In the near future I will move to Philadelphia and start working at Accenture. This will be my first official job. I am very nervous about it, but I am prepared for anything. I want to help a lot of people with the projects that I work in. Another goal I have in mind is saving a bit of money and getting married to Malena. I want to have a family with her also, thus I want to marry her.


In normal use, this panel shouldn't be needed. It can also be buggy, don't hit buttons too quickly and if it/the player breaks, refresh stuff!
Pause Song
Skip Song
Reset Playing Status


Before Change
import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
BATCH_SIZE = 128
After Change
from torchtext.utils import download_from_url, extract_archive
import io
url_base = "https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/raw/"
train_urls = ("train.de.gz", "train.en.gz")
val_urls = ("val.de.gz", "val.en.gz")
test_urls = ("test_2016_flickr.de.gz", "test_2016_flickr.en.gz")
train_filepaths = [extract_archive(download_from_url(url_base + url))[0] for url in train_urls]
val_filepaths = [extract_archive(download_from_url(url_base + url))[0] for url in val_urls]
test_filepaths = [extract_archive(download_from_url(url_base + url))[0] for url in test_urls]
de_tokenizer = get_tokenizer("spacy", language="de")


5G-EmPOWER is developed by the Future Networks (FuN) Team at CREATE-NET. The objective of the FuN team is to develop future converged broadband networks and communication infrastructures, which will connect the future Internet of people, content, cloud and things.
5G-EmPOWER is available under an Apache 2.0 License


WITNESS LIST
Land & Resource Management Committee
May 9, 2018 - 10:30 AM
Eminent Domain Interim Charge
On:
Anderson, J. David (Texas Wildlife Association)
Christian, George (Coalition for Critical Infrastructure)
Fitzsimons, Joseph (Texas and Southwestern Cattle Raisers)
Frazier, Scott (Self; Texas Farm Bureau)
Leman, Ben (Texans Against High Speed Rail)
Madsen, Kyle (Texas Department of Transportation)
Mannina, Glenn (Self)
Opiela, Eric (Self; South Texans Property Rights Assn.)
Rathgeber, Julia (Assn of Electric Companies of Texas)
Stafford, Mike (Harris County Texas)
Staples, Todd (Texas Oil & Gas Association)
Zabel, Tom (Texas Pipeline Association)
Registering, but not testifying:
On:
Beving, Rita (TX Landowners for Eminent Domain Reform)
McCulley, Hugh (BNSF Ry.)
1


PIPEGLADE
Graphical User Interfaces, The UNIX Way
Pipeglade is a helper program that displays graphical user
interfaces for other programs. It renders the GUI definition
found in a GtkBuilder file (created using
the
Glade Interface
Designer), and communicates with the main program solely
via pipes or fifos.
To have its GUI rendered by pipeglade, a program must be able to
-
send plain text commands to standard output or a named
pipe and/or
-
receive and parse simple plain text messages from standard
input or a named pipe.
Pipeglade strives for simplicity; it provides access to a
subset of the features available in GTK+ v3.2.
See below for the set of widgets accessible via pipeglade.
(Widgets
of any kind can grab focus and change visibility,
sensitivity, style,
size, and tooltip.)
WIDGET GALLERY
EXAMPLE
The shell script
clock.sh uses
pipeglade to display an analog clock on the interface
defined in the GtkBuilder
file
clock.ui. The script was
also used to generate the image below.
LICENSE


NET Tool
Redirect List
Add redirect
Pages to watch
Add page to watch
Redirect History
{{user.login}}


Open Angel Arena 5.9.0 Update
General Changes
Map changes: Added ward spots, cliffs around radiant ancients ...
Wanderer changes: He spends more time on the losing side.
Item Changes
Armor and anti-armor items:
Nerfed all armor items except Dispel Orb, Divine Reflection Shard, Martyr's Mail and Stoneskin.
All armor items now have the same scaling logic: x/x+2/x+5/x+9/x+14 or x/x+3/x+7/x+12/x+18. There are a few exceptions.
Nerfed Devastator and Solar Crest heavily.
Assault Cuirass aura is nerfed only at last level.
Damage items:
Nerfed crit, mkb, desolator and devastator.
Boots:
Nerfed movement speed bonuses on popular boots: arcane boots, tranquil boots, phase boots, guardian greaves.
Nerfed armor bonuses on phase boots and guardian greaves.
Buffed hp regen bonus on tranquil boots.
Consumable Aghanim Scepter:
Recipe: Aghanims Scepter 5 + tier 4 core + Recipe (2000)
Blade Mail:
Buffed duration to be the same at all levels: 4.5 seconds. This matches normal dota.
Crimson Guard:
Its a tier 0 item again. Vanguard not useless anymore.
Buffed all armor values.
Far Sight:
Now builds out of Ward Stack.
Lucience:
lvl 2 and 3 ms nerf; lvl 3 heal buff.
Regen Crystal:
Active hp regen buffed at all levels.
Sonic:
Negative status resistance is now the same at all levels (-25%)
Movement speed bonus slightly nerfed but the user cannot be slowed anymore.
Percentage bonus vision is now the same at all levels (50%).
Hero Changes
Abaddon:
Ult rebalanced: Nerfed cd, buffed duration without aghs, nerfed aghs range to 900.
Alchemist:
Acid Spray armor reduction nerf at OAA levels.
Chemical Rage hp regen buffed.
Anti-Mage:
Buffed blink cd talent slightly.
Axe:
Nerfed Axe Battle Hunger so its less spammable and worse in first few minutes of the game.
Nerfed Axe Culling Blade damage threshold first 3 levels. Buffed lvl 5 because its mostly useless then.
Bane:
Nerfed Brain Sap mana cost at OAA levels even more because Valve buffed his aghs again in 7.22.
Buffed Brain Sap cd (without aghs) a bit to compensate.
Fiend's Grip cd buffed to 100s at all levels.
Fiend's Grip dmg buffed.
Batrider:
Reverted Firefly cd nerf added with 7.22
Flaming Lasso dmg buffed.
Beastmaster:
Primal Roar dmg and attack speed slow buffed.
Primal Roar slow duration buffed.
Bloodseeker:
Rupture dmg buffed.
Brewmaster:
Drunken Brawler crit nerf at OAA levels.
Bristleback:
Warpath damage per stack buffed.
Broodmother:
Buffed spin web count (max number of the webs on the map) to be better early.
lvl 15 talent changed fom +45 dmg to +40 AGI.
Centaur Warrunner:
Buffed Centaur stun at OAA levels because of the existence of status resistance. Nerfed mana cost to compensate.
Stampede cooldown buffed.
Chaos Knight:
Phantasm illusion damage buffed at OAA levels.
Chatterjee:
Nerfed base dmg from 58-64 to 48-55.
Nerfed base mana regen, base movement speed by 5 and base attack speed by 10.
Nerfed Static Grip cast range
Nerfed Electric Shield damage
Cleansing Shock cannot target spell immune enemies without the lvl 25 talent.
Cleansing Shock nerfed slow, ms bonus and cd.
Lvl 15 talent: +30 MS -> +15 STR
Lvl 15 talent: +250 cast range -> +125 cast range
Lvl 25 talent: +1000 HP -> Cleansing Shock pierces spell immunity
Chen:
Hand of God heal rescaled -> this is temporary.
Clinkz:
Searing Arrows dmg nerf at OAA levels.
Clockwerk:
Hookshot dmg buffed.
Crystal Maiden:
Freezing Field damage buffed.
Dark Seer:
Ion Shell can now target allied creeps but cannot target enemies anymore. Mostly a QOL change for Dominator Dark Seer.
Vacuum cd buff because it was too long (30s at all levels).
Dazzle:
Shallow Grave cd buffed at early levels.
Bad Juju armor reduction buffed.
Death Prophet:
Spirit Siphon damage nerf because it is hp percentage dmg and its amplified by spell amp.
Disruptor:
Static storm cd and dmg buffed.
Doom:
Devour cd buff so its more useful (30s at all levels), nerfed bonus gold to compensate, also removed creep level requirement because its dumb.
Dragon Knight:
Dragon Blood bonus armor nerfed.
Dragon Blood bonus hp regen buffed.
Corrosive breath dmg from Elder Dragon Form buffed.
Earth Spirit:
Magnetize dmg buffed.
Earthshaker:
Echo Slam dmg rescaled: lvl 1 nerfed, other levels buffed.
Ember Spirit:
Flame Guard dmg at lvl 6 buffed.
Fire Remnant dmg buffed.
Enchantress:
Impetus dmg rescaled: lvl 2 and 3 buffed, lvl 5 nerfed.
Enigma:
Demonic Conversion removed creep level requirement (cannot target ancients).
Demonic Conversion buffed mana cost to match normal dota.
Black Hole cd buffed: 160/155/150/140/130 seconds.
Faceless Void:
Time Lock: Stun duration buffed because of status resistance fix.
Time Lock: Added another attack when Time Lock is procced. Cannot proc itself.
Grimstroke:
Ink Swell cast range buffed to 700.
Ink Swell max stun duration buffed -> because Valve overnerfed it.
Soul Bind duration nerfed because of the recent cooldown buff.
Sould Bind: Added scaling slow again -> this is temporary
Gyrocopter:
Flak Cannon cd buffed to 30s. Reduced number of attacks at max level.
Call Down dmg buffed.
Jakiro:
Liquid Fire dmg buffed at OAA levels.
Dual Breath Damage talent and Liquid Fire Attack Speed talent buffed.
Macropyre dmg buffed.
Keeper of the Light:
Hopefully a fix for KOTL aghs having only 4 levels.
Will o Wisp cd and number of attacks required to destroy buffed.
Kunkka:
Ghostship dmg buffed.
Legion Commander:
Duel duration buffed because of the existence of status resistance.
Duel reward dmg buffed.
Leshrac:
Diabolic Edict damage nerfed at all levels to match normal dota.
Lich:
Chain Frost cd buffed to 60s at all levels.
Chain Frost dmg buffed.
Lifestealer:
Infest damage buffed.
Lina:
Laguna Blade cd and dmg buffed.
Lion:
Finger of Death cd buffed at lvl 1 and 2.
Lone Druid:
Battle Cry and True Form bonus armor nerfed at OAA levels.
Lycan:
Shapeshift cd buffed.
Magnus:
Reverse Polarity cd and dmg buffed.
Mars:
Arena of Blood (spear) damage buffed.
Mirana:
Sacred Arrow dmg buffed at lvl 1 and 2.
Moonlight Shadow cd buffed to 100 seconds at all levels.
Morphling:
Morph cd buffed at lvl 1 and 2.
Nature's Prophet:
Wrath of Nature dmg buffed.
Night Stalker:
Crippling Fear cd set to 15 at all levels.
Nyx Assassin:
Vendetta bonus damage buffed.
Vendetta bonus ms nerfed at OAA levels.
Omniknight:
Guardian Angel cooldown now scales again - matches normal dota.
Guardian Angel bonus hp regen with Aghanims Scepter now scales.
Pangolier:
Heartpiercer chance buffed to 17% because of the status resistance fix and to match Lucky Shot proc chance in normal dota.
Rolling Thunder dmg buffed.
Phoenix:
Buffed cds of his spells (except Sun Ray).
Supernova dmg buffed.
Supernova mana cost buffed.
Puck:
Dream Coil break dmg buffed.
Pudge:
Dismember dmg buffed.
Queen of Pain:
Buffed Shadow Strike cooldown, reverted cast range buff (it was +50 anyway).
Sonic Wave dmg buffed.
Razor:
Eye of the Storm cd and dmg buffed.
Shadow Demon:
Demonic Purge duration nerf.
Shadow Fiend:
Nerfed aura (nerfed talent as well).
Silencer:
Global Silence cd buffed.
Skywrath Mage:
Not nerfed.
Slardar:
Bash duration buff because of status resistance fix.
Bash cooldown increased back to 1.65s because bash duration is longer now.
Slithereen Crush damage nerf at OAA levels.
Nerfed Slardar's ult armor reduction at all levels.
Slark:
Essence Shift duration rebalance: buffed early levels, nerfed other levels because of the +80s talent.
Shadow Dance cd buffed.
Sniper:
Headshot dmg nerf at OAA levels.
Assassinate dmg buffed.
Sohei:
Momentum slow duration buff because of the status resistance fix.
Spectre:
Desolate radius buff.
Desolate blind percentage nerfed to 40% because its annoying.
Desolate dmg talent buff.
Storm Spirit:
Electric Vortex talent buffed from +0.5 to +0.75s
Electric Vortex max level duration decreased by 0.25s
Sven:
God's Strength bonus strength buffed.
Templar Assassin:
Nerfed Meld dmg and armor reduction at OAA levels.
Psionic Trap dmg buffed.
Tidehunter:
Nerfed Gush armor reduction at OAA levels.
Ravage cd and damage buffed.
Timbersaw:
Timber Chain dmg nerf to slow down his farming and to reduce oneshot potential.
lvl 25 talent nerfed from +15% to +12% attribute reduction.
Tinker:
Rearm mana cost split percentage buffed.
Tiny:
Grow lvl 5 armor nerf because its too much.
Toss dmg at OAA levels nerfed because 3 toss charges talent triples the dmg output of this spell.
Underlord:
Highway to Hell cast range buffed to 5000 at all levels.
Vengeful Spirit:
Nerfed Wave of Terror armor reduction at OAA levels.
Venomancer:
Poison Nova dmg buffed.
Viper:
Viper Strike dmg buffed.
Visage:
Familiars not fixed.
Weaver:
Swarm cd and mana cost buff.
Geminate Attack dmg nerf at OAA levels.
Windranger:
Focus Fire cd buffed.
Winter Wyvern:
Winter's curse bonus attack speed buffed.
Wraith King:
Wraith Form aura range nerf to 900.
Zeus:
Thundergod's Wrath dmg buffed.
Previous
Home
Next


Blog : https://vcansimplify.wordpress.com/
Website: http://vighneshbirodkar.github.io/
In the 60s, Marvin Minsky assigned a couple of undergrads to spend the summer programming a computer to use a camera to identify objects in a scene. He figured they'd have the problem solved by the end of the summer. Half a century later, we're still working on it. [https://www.xkcd.com/1425/]
My Summer Project.
Create a Region Adjacency Graph and leverage it for a better segmentation.
img = io.imread('car.jpg')
labels = segmentation.slic(img, compactness=30, n_segments=400)
g = graph.rag_mean_color(img, labels)
out = graph.draw_rag(labels, g, img)
labels_new = graph.cut_threshold(labels, g, 30)
g = graph.rag_mean_color(img, labels, mode='similarity')
labels_new = graph.cut_normalized(labels, g)
def _weight_mean_color(graph, src, dst, n):
"""Callback to handle merging nodes by recomputing mean color.
The method expects that the mean color of `dst` is already computed.
Parameters
----------
graph : RAG
The graph under consideration.
src, dst : int
The vertices in `graph` to be merged.
n : int
A neighbor of `src` or `dst` or both.
Returns
-------
data : dict
A dictionary with the `"weight"` attribute set as the absolute
difference of the mean color between node `dst` and `n`.
"""
diff = graph.node[dst]['mean color'] - graph.node[n]['mean color']
diff = np.linalg.norm(diff)
return {'weight': diff}
def merge_mean_color(graph, src, dst):
"""Callback called before merging two nodes of a mean color distance graph.
This method computes the mean color of `dst`.
Parameters
----------
graph : RAG
The graph under consideration.
src, dst : int
The vertices in `graph` to be merged.
"""
graph.node[dst]['total color'] += graph.node[src]['total color']
graph.node[dst]['pixel count'] += graph.node[src]['pixel count']
graph.node[dst]['mean color'] = (graph.node[dst]['total color'] /
labels_new = graph.merge_hierarchical(labels, g, thresh=35, rag_copy=False,
in_place_merge=True, merge_func=merge_mean_color,
weight_func=_weight_mean_color)
edges = filters.sobel(gray_img)
g = graph.rag_boundary(labels, edges)
def weight_boundary(graph, src, dst, n):
"""
Handle merging of nodes of a region boundary region adjacency graph.
This function computes the `"weight"` and the count `"count"`
attributes of the edge between `n` and the node formed after
merging `src` and `dst`.
Parameters
----------
graph : RAG
The graph under consideration.
src, dst : int
The vertices in `graph` to be merged.
n : int
A neighbor of `src` or `dst` or both.
Returns
-------
data : dict
A dictionary with the "weight" and "count" attributes to be
assigned for the merged node.
"""
default = {'weight': 0.0, 'count': 0}
count_src = graph[src].get(n, default)['count']
count_dst = graph[dst].get(n, default)['count']
weight_src = graph[src].get(n, default)['weight']
weight_dst = graph[dst].get(n, default)['weight']
count = count_src + count_dst
return {
'count': count,
'weight': (count_src * weight_src + count_dst * weight_dst)/count
}
def merge_boundary(graph, src, dst):
"""Call back called before merging 2 nodes.
In this case we don't need to do any computation here.
"""
pass
labels2 = graph.merge_hierarchical(labels, g, thresh=0.1, rag_copy=False,
in_place_merge=True, merge_func=merge_boundary,
weight_func=weight_boundary)
Research Assistantship during Fall 2016 with Prof. Davi Geiger.
[Work in progress]
Rather than specifying 640 x 480 pixels, humans would describe the picture as "Red Ford GT sports car with yellow stripes parked on a road with trees in the background."


Leons ruaraka is a 15 minute drive from the airport and a 10 minute drive from the ferries. Bookings include free airport transfer, and we can help with other transfers. Private entrance, fully equipped kitchen which opens into a large living room with queen size daybed.  Downstairs is 1 large private bedroom with cold AC and king bed. Large bathroom has a beautiful garden.
Read MoreBoardering the well known two rivers
Courtesy of purchasing the premium package
Well equipped gym for both light and HEAVY workouts


| 008:063 Khan | : |
And He has united their (i.e. believers') hearts. If you had spent all that is in the earth, you could not have united their hearts, but Allah has united them. Certainly He is All- Mighty, All-Wise. |
| 008:063 Maulana | : |
And He has united their hearts. If thou hadst spent all that is in the earth, thou couldst not have united their hearts, but Allah united them. Surely he is Mighty, Wise. |
| 008:063 Pickthal | : |
And (as for the believers) hath attuned their hearts. If thou hadst spent all that is in the earth thou couldst not have attuned their hearts, but Allah hath attuned them. Lo! He is Mighty, Wise. |
| 008:063 Rashad | : |
He has reconciled the hearts (of the believers). Had you spent all the money on earth, you could not reconcile their hearts. But GOD did reconcile them. He is Almighty, Most Wise. |
| 008:063 Sarwar | : |
among whose hearts He has placed affection and unity. If you were to spend the wealth of the whole earth, you would not have been able to unite their hearts but God has been able to unite them. God is Majestic and All-wise. |
| 008:063 Shakir | : |
And united their hearts; had you spent all that is in the earth, you could not have united their hearts, but Allah united them; surely He is Mighty, Wise. |
| 008:063 Sherali | : |
And HE has put affection between their hearts. If thou hadst expended all that is in the earth, thou couldst not have put affection between their hearts, but ALLAH has put affection between them. Surely, HE is Mighty, Wise. |
| 008:063 Yusufali | : |
And (moreover) He hath put affection between their hearts: not if thou hadst spent all that is in the earth, couldst thou have produced that affection, but Allah hath done it: for He is Exalted in might, Wise. |
Concepts/Themes Covered: أركان الإسلام > التوحيد > توحيد الله تعالي > نعمه على عباده والأمر بالتحدث بها Pillars of Islam> Islamic> Oneness of Allah> His Gifts on his Worshippers and His Order to Speak about them الجهاد > (تعليمات حربية (أحكام خاصة > (تعليمات حربية (أحكام خاصة > نظام الجهاد وقانونه Jihad> Warlike Instructions\Special Rules> Warlike Instructions\Special Rules> The System of the Holy War and its Law الجهاد > الجهاد في سبيل الإسلام > الدعوة إلى الجهاد Jihad> The Call to Jihad>The Call for the Holy War الجهاد > الجهاد في سبيل الإسلام > مدح الجهاد Jihad> The Call to Jihad>Praising the Holy War العمل > العمل الصالح > البشاشة Action(Work)> The Good Deeds>Cheerfulness الإنسان والعلاقات الأخلاقية > الأخلاق الحميدة > البشاشة والوداعة Man and The Moral Relations> Good Morals>Cheerfulness and Humbleness
This tool uses eight English parallel translations for the Holy Quran imported from Mr. Clay Smith. You can also visit Qurany Concepts for browsing the Quran themes or topics. If you want to search the Holy Quran for keywords, please visit Qurany Keywords.


--- layout: post title: queries on indexes date: type: post published: false status: draft categories: - MySQL tags: [] meta: _edit_last: '2' author: login: shlomi email<EMAIL_ADDRESS>display_name: shlomi first_name: Shlomi last_name: Noach ---
a. write these queries: 1...2..3..4..5..6..
b. how would you improve indexes for these queries?
solutions:
1. SELECT Country.name, City.name FROM Country JOIN City ON (Country.CODE = City.CountryCode);
2. SELECT Country.name, City.name FROM Country JOIN City ON (Country.Capital = City.ID);
3. SELECT Country.Name FROM Country JOIN CountryLanguage on (CountryLanguage.CountryCode = Country.Code) WHERE Language='English';
4. SELECT Country.name, City.name FROM Country JOIN City ON (Country.Capital = City.ID)
WHERE City.Population > 1000000;
5. SELECT Country.name, City.name FROM Country JOIN City ON (Country.CODE = City.CountryCode)
WHERE City.Population > 1000000;
6. SELECT Country.name, City.name FROM Country JOIN City ON (Country.Capital = City.ID)
WHERE City.Population > 1000000 AND Continent='Europe';


Why stake with Hairy Staking?
If you are looking for an Australian based Staking Pool (hosted out of Sydney) with a support team with a combined IT experience of 30+ years, you can be confident that our pool will be always be ready to mint.
Our team
-
{% for person in site.staff_members %}
-
{{ person.position }}
{% endfor %}
Stake your ADA today
Pool Ticker: HAIRY
Pool ID: 43a92030961c6d68bcde62be938a53abd9d97e90101ee2bb7b5a60ab


Active ingredient
Butenafine hydrochloride 1%
Uses
- cures most jock itch
- relieves itching, burning, cracking, and scaling which accompany this condition
Warnings
For external use only
Do not use
- on nails or scalp
- in or near the mouth or the eyes
- for vaginal yeast infections
When using this product do not get into the eyes. If eye contact occurs, rinse thoroughly with water.
Stop use and ask a doctor if too much irritation occurs or irritation gets worse
Keep out of reach of children. If swallowed, get medical help or contact a Poison Control Center right away.
Directions
- adults and children 12 years and over:
- use the tip of the cap to break the seal and open the tube
- wash the affected skin with soap and water and dry completely before applying
- apply once a day to affected skin for 2 weeks or as directed by a doctor
- wash hands after each use
- children under 12 years: ask a doctor
Other information
- do not use if seal on tube is broken or not visible
- store between 20° to 25°C (68° to 77°F)
Inactive ingredients
benzyl alcohol, cetyl alcohol, diethanolamine, glycerin, glyceryl monostearate SE, polyoxyethylene (23) cetyl ether, propylene glycol dicaprylate, purified water, sodium benzoate, stearic acid, white petrolatum
Questions:?
1-800-360-3226
Distributed by MSD Consumer Care, Inc., PO Box 377, Memphis, TN 38151 USA, a subsidiary of Merck & Co.,Inc., Whitehouse Station, NJ USA.
PRINCIPAL DISPLAY PANEL - 12g Tube Carton
LOTRIMIN ULTRA®
butenafine hydrochloride cream 1% ANTIFUNGAL NET WT 12g (0.42 OZ)
MSD Consumer Care, Inc.