Source code for time_agnostic_browser.agnostic_query
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Copyright (c) 2021, Arcangelo Massari<EMAIL_ADDRESS>#
# Permission to use, copy, modify, and/or distribute this software for any purpose
# with or without fee is hereby granted, provided that the above copyright notice
# and this permission notice appear in all copies.
 IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT,
# OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE,
# DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS
# ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS
# SOFTWARE.
from typing import Set, Tuple, Dict
from rdflib.plugins.sparql.processor import prepareQuery
from rdflib.plugins.sparql.sparql import Query
from rdflib.plugins.sparql.parserutils import CompValue
from rdflib import ConjunctiveGraph, Literal
from tqdm import tqdm
from time_agnostic_browser.sparql import Sparql
from time_agnostic_browser.prov_entity import ProvEntity
from time_agnostic_browser.agnostic_entity import AgnosticEntity
from pprint import pprint
[docs]class AgnosticQuery:
def __init__(self, query:str=""):
self.query = query
if query:
prepare_query:Query = prepareQuery(self.query)
algebra:CompValue = prepare_query.algebra
if algebra.name != "SelectQuery":
raise ValueError("Only SELECT queries are allowed")
[docs] def run_the_sock(self):
dict_of_snapshots = dict()
query_times = f"""
SELECT DISTINCT ?time
WHERE {{
?snapshot <{ProvEntity.iri_generated_at_time}> ?time;
<{ProvEntity.iri_has_update_query}> ?updateQuery.
}}
"""
times = Sparql().run_select_query(query_times)
for time in times:
query_entities_at_time = f"""
SELECT DISTINCT ?s
WHERE {{
?snapshot <{ProvEntity.iri_generated_at_time}> "{time[0]}"^^<http://www.w3.org/2001/XMLSchema#dateTime>;
<{ProvEntity.iri_has_update_query}> ?updateQuery;
<{ProvEntity.iri_specialization_of}> ?s.
}}
"""
entities_at_time = Sparql().run_select_query(query_entities_at_time)
for entity_at_time in entities_at_time:
dict_of_snapshots.setdefault(time[0], set())
dict_of_snapshots[time[0]].add(entity_at_time[0])
sock = dict()
pbar_snapshot = tqdm(total=len(dict_of_snapshots))
for snapshot, set_of_entities in dict_of_snapshots.items():
graph_at_time = ConjunctiveGraph()
pbar_state = tqdm(total=len(set_of_entities))
for entity in set_of_entities:
agnostic_entity = AgnosticEntity(res=entity, related_entities_history=False)
state_at_time = agnostic_entity.get_state_at_time(time=snapshot, get_hooks=False)
for time, state in state_at_time.items():
graph_at_time += state
pbar_state.update(1)
pbar_state.close()
pbar_snapshot.update(1)
sock[snapshot] = graph_at_time
pbar_snapshot.close()
return sock
# results = AgnosticQuery().run_the_sock()
# for se, graph in results.items():
# for quad in graph.quads():
# pprint(quad)


|
robot_interfaces
|
robot_fingers
package instead (this also includes robot_interfaces
).We are using catkin as build tool (i.e. robot_interfaces
is a catkin package). While we are not really depending on any ROS packages, this means you need a basic ROS installation to build.
In the following we are using catkin_tools which need to be installed separately:
pip install catkin_tools
We are testing on Ubuntu 18.04 with ROS Melodic. Other versions may work as well but are not officially supported.
robot_interfaces
. See the documentation of the robot_fingers
package for more information.robot_interfaces
depends on several other of our packages which are organized in separate repositories. We therefore use a workspace management tool called treep which allows easy cloning of multi-repository projects.
treep can be installed via pip:
pip install treep
Clone the treep configuration containing the "ROBOT_INTERFACES" project:
git clone git@github.com:machines-in-motion/treep_machines_in_motion.git
git clone
command above or any subdirectory.Now clone the project:
treep --clone ROBOT_INTERFACES
ssh-add
first.
You should now have the following directory structure:
├── treep_machines_in_motion └── workspace └── src ├── catkin │ ├── core_robotics │ │ ├── mpi_cmake_modules │ │ ├── pybind11_catkin │ │ ├── real_time_tools │ │ ├── shared_memory │ │ ├── time_series │ │ └── yaml_cpp_catkin │ ├── examples │ │ └── ci_example │ ├── robots │ │ └── robot_interfaces │ └── tools │ ├── serialization_utils │ └── signal_handler └── not_catkin └── third_party └── pybind11
To build, cd into the workspace
directory and call
catkin build
to build the whole workspace.
With the above command Python bindings will be build for the default python version of your system (see python --version
). If you want to use a different version (e.g. python3), you can specify as follows:
catkin build -DPYTHON_EXECUTABLE=/usr/bin/python3


|
Panda3D
|
Public Member Functions | |
| __init__ () | |
| Create a patch file and initializes internal data. More... | |
| __init__ (Buffer buffer) | |
| Create patch file with buffer to patch. More... | |
| bool | apply (Filename patch_file, Filename file) |
| Patches the entire file in one call returns true on success and false on error. More... | |
| bool | apply (Filename patch_file, Filename orig_file, const Filename target_file) |
| Patches the entire file in one call returns true on success and false on error. More... | |
| bool | build (Filename file_orig, Filename file_new, Filename patch_name) |
| This implementation uses the "greedy differencing algorithm" described in the masters thesis "Differential Compression: A Generalized Solution for
Binary Files" by Randal C. More... | |
| bool | getAllowMultifile () |
| See set_allow_multifile(). More... | |
| int | getFootprintLength () |
| float | getProgress () |
| Returns a value in the range 0..1, representing the amount of progress through the patchfile, during a session. More... | |
| const HashVal | getResultHash () |
| Returns the MD5 hash for the file after the patch has been applied. More... | |
| const HashVal | getSourceHash () |
| Returns the MD5 hash for the source file. More... | |
| bool | hasSourceHash () |
| Returns true if the MD5 hash for the source file is known. More... | |
| int | initiate (const Filename patch_file, const Filename file) |
| Set up to apply the patch to the file (original file and patch are destroyed in the process). More... | |
| int | initiate (const Filename patch_file, const Filename orig_file, const Filename target_file) |
| Set up to apply the patch to the file. More... | |
| int | readHeader (const Filename patch_file) |
| Opens the patch file for reading, and gets the header information from the file but does not begin to do any real work. More... | |
| resetFootprintLength () | |
| int | run () |
| Perform one buffer's worth of patching Returns EU_ok while patching Returns EU_success when done If error happens will return one of: EU_error_abort : Patching has not been initiated EU_error_file_invalid : file is corrupted EU_error_invalid_checksum : incompatible patch file EU_error_write_file_rename : could not rename file. More... | |
| setAllowMultifile (bool allow_multifile) | |
| If this flag is set true, the Patchfile will make a special case for patching Panda Multifiles, if detected, and attempt to patch them on a subfile-by-subfile basis. More... | |
| setFootprintLength (int length) | |
Public Attributes | |
| bool | allow_multifile |
| See set_allow_multifile(). More... | |
| int | footprint_length |
| float | progress |
| Returns a value in the range 0..1, representing the amount of progress through the patchfile, during a session. More... | |
| const HashVal | result_hash |
| Returns the MD5 hash for the file after the patch has been applied. More... | |
| const HashVal | source_hash |
| Returns the MD5 hash for the source file. More... | |
| __init__ | ( | ) |
Create a patch file and initializes internal data.
| __init__ | ( | Buffer | buffer | ) |
Create patch file with buffer to patch.
Patches the entire file in one call returns true on success and false on error.
This version will delete the patch file and overwrite the original file.
Patches the entire file in one call returns true on success and false on error.
This version will not delete any files.
This implementation uses the "greedy differencing algorithm" described in the masters thesis "Differential Compression: A Generalized Solution for Binary Files" by Randal C.
Burns (p.13). For an original file of size M and a new file of size N, this algorithm is O(M) in space and O(M*N) (worst- case) in time. return false on error
| bool getAllowMultifile | ( | ) |
See set_allow_multifile().
| int getFootprintLength | ( | ) |
| float getProgress | ( | ) |
Returns a value in the range 0..1, representing the amount of progress through the patchfile, during a session.
| const HashVal getResultHash | ( | ) |
Returns the MD5 hash for the file after the patch has been applied.
| const HashVal getSourceHash | ( | ) |
Returns the MD5 hash for the source file.
| bool hasSourceHash | ( | ) |
Returns true if the MD5 hash for the source file is known.
(Some early versions of the patch file did not store this information.)
Set up to apply the patch to the file (original file and patch are destroyed in the process).
Set up to apply the patch to the file.
In this form, neither the original file nor the patch file are destroyed.
| int readHeader | ( | const Filename | patch_file | ) |
Opens the patch file for reading, and gets the header information from the file but does not begin to do any real work.
This can be used to query the data stored in the patch.
| resetFootprintLength | ( | ) |
| int run | ( | ) |
Perform one buffer's worth of patching Returns EU_ok while patching Returns EU_success when done If error happens will return one of: EU_error_abort : Patching has not been initiated EU_error_file_invalid : file is corrupted EU_error_invalid_checksum : incompatible patch file EU_error_write_file_rename : could not rename file.
| setAllowMultifile | ( | bool | allow_multifile | ) |
If this flag is set true, the Patchfile will make a special case for patching Panda Multifiles, if detected, and attempt to patch them on a subfile-by-subfile basis.
If this flag is false, the Patchfile will always patch the file on a full-file basis.
This has effect only when building patches; it is not used for applying patches.
| setFootprintLength | ( | int | length | ) |
| bool allow_multifile |
See set_allow_multifile().
| int footprint_length |
| float progress |
Returns a value in the range 0..1, representing the amount of progress through the patchfile, during a session.
| const HashVal result_hash |
Returns the MD5 hash for the file after the patch has been applied.
| const HashVal source_hash |
Returns the MD5 hash for the source file.


TV Services TV Services allow access Movistar TV multicast contents from a smart TV, an Android device or other UPnP/DLNA-capable devices. Select desired values and "Apply/Save". Enable TV Services
TV Services TV Services allow access Movistar TV multicast contents from a smart TV, an Android device or other UPnP/DLNA-capable devices. Select desired values and "Apply/Save".


class BucketHash.Bucket
extends java.lang.Object
| Modifier and Type | Class and Description |
|---|---|
private class |
BucketHash.Bucket.ListNode |
| Modifier and Type | Field and Description |
|---|---|
(package private) BucketHash.Bucket.ListNode |
header
Head of the linked list
|
| Constructor and Description |
|---|
BucketHash.Bucket()
Constructs an empty linked list.
|
| Modifier and Type | Method and Description |
|---|---|
private BucketHash.Bucket.ListNode |
find(K key)
Returns the ListNode containing the key, or null.
|
V |
get(K key)
Given a key, look for the corresponding value in the linked list.
|
int |
hashCode() |
private boolean |
isEmpty()
Tests whether the linked list is empty.
|
java.util.Set<K> |
keySet()
Returns a Set of the keys contained in this linked list.
|
V |
put(K key,
V value)
Generates a node in the linked list with the given key and value
|
V |
remove(K key)
Removes the list node containing the given key.
|
int |
size()
Returns the size of the linked list.
|
java.lang.String |
toString() |
java.util.Collection<V> |
values()
Returns a Collection of the values contained in this linked list.
|
BucketHash.Bucket.ListNode header
public int size()
public V get(K key)
key
- key of the entryprivate boolean isEmpty()
private BucketHash.Bucket.ListNode find(K key)
key
- key that used to look for the node entrypublic V put(K key, V value)
key
- with which the specified value is to be associatedvalue
- value to be associated with the specified keypublic V remove(K key)
key
- key whose node is to be removed from the linked listpublic java.util.Set<K> keySet()
public java.util.Collection<V> values()
public java.lang.String toString()
toString
in class java.lang.Object
public int hashCode()
hashCode
in class java.lang.Object


Advanced Searches:
NOTICE : This campus directory has been compiled for the use and convenience of the faculty, staff and students of the University of California, Riverside and others dealing with UCR. It is the property of the Regents of the University of California. In accordance with the California Information Practices Act, neither this directory nor the information contained herein may be used, rented, distributed, or sold for commercial purposes.


public class MainWindowController
extends java.lang.Object
implements javafx.fxml.Initializable
| Constructor and Description |
|---|
MainWindowController() |
| Modifier and Type | Method and Description |
|---|---|
void |
clearCheckOutForm()
Clears the check-out form
|
void |
clearCheckoutFormButtonPressed(javafx.event.ActionEvent event)
Clears the check-out form when button is pressed
|
void |
clearNewBookForm(javafx.event.ActionEvent event)
Clears New Book Form when button is pressed.
|
void |
clearNewStudentForm(javafx.event.ActionEvent event)
Clears Form for New Student Record when button is pressed.
|
void |
clearTableView(javafx.scene.input.MouseEvent event)
Clears TableView records.
|
void |
clearUpdateForm()
Clears Update Form.
|
void |
clearUpdateFormButton(javafx.event.ActionEvent event)
Clears Update Form when button is pressed.
|
void |
deleteUpdateButtonPressed(javafx.event.ActionEvent event)
Deletes a record form the data base.
|
void |
initialize(java.net.URL url,
java.util.ResourceBundle rb) |
void |
logoutUser(javafx.event.ActionEvent event)
Handles logout request, and return the user to the login screen.
|
void |
mouseClickComboBox(javafx.scene.input.MouseEvent event)
Process the books check-out by a student, and then it fills the combobox with them.
|
void |
mouseClickTableView(javafx.scene.input.MouseEvent event)
Populate update tab and check up tab with the information double clicked by the
mouse for faster processing of books and students.
|
void |
newBookButton(javafx.event.ActionEvent event)
Process new book record.
|
void |
newStudentButton(javafx.event.ActionEvent event)
Process new student record.
|
int |
optionsAlert(java.lang.String message)
Display an option Alert to verify user selection.
|
void |
processCheckInButtonPressed(javafx.event.ActionEvent event)
Process the transaction when a book is returned.
|
void |
processCheckOutButtonPressed(javafx.event.ActionEvent event)
Process the transaction when a student check-out a book.
|
void |
saveUpdateButtonPressed(javafx.event.ActionEvent event)
Updates student or book record on the data base, when button pressed.
|
void |
searchButtonPressed(javafx.event.ActionEvent event)
Function to run if a student ID, or an ISBN is entered in the search box.
|
void |
viewButtonPressed(javafx.event.ActionEvent event)
Process the action when the View button is pressed.
|
public void searchButtonPressed(javafx.event.ActionEvent event) throws java.io.IOException, java.sql.SQLException
event
- The event that trigger this function.java.io.IOException
- If bad data was entered.java.sql.SQLException
- If connection to data base failed.public void logoutUser(javafx.event.ActionEvent event) throws java.io.IOException
event
- The event that trigger this function.java.io.IOException
- If the fxml was not found.public void initialize(java.net.URL url, java.util.ResourceBundle rb)
initialize
in interface javafx.fxml.Initializable
public void viewButtonPressed(javafx.event.ActionEvent event) throws java.io.IOException, java.sql.SQLException
event
- The event that triggered this function.java.io.IOException
- If a file cannot be loaded.java.sql.SQLException
- If the connection to the data base failed.public void newStudentButton(javafx.event.ActionEvent event) throws java.io.IOException, java.sql.SQLException
event
- The event that triggered this function.java.io.IOException
- if a file cannot be loaded.java.sql.SQLException
- If the connection to the data base failed.public void newBookButton(javafx.event.ActionEvent event) throws java.io.IOException, java.sql.SQLException
event
- The event that triggered this function.java.io.IOException
- IOException if a file cannot be loaded.java.sql.SQLException
- SQLException If the connection to the data base failed.public void clearNewStudentForm(javafx.event.ActionEvent event)
event
- The even that triggered this function.public void clearNewBookForm(javafx.event.ActionEvent event)
event
- The even that triggered this function.public void clearUpdateFormButton(javafx.event.ActionEvent event)
event
- The even that triggered this function.public void clearUpdateForm()
public void clearTableView(javafx.scene.input.MouseEvent event) throws java.io.IOException
event
- The even that triggered this function.java.io.IOException
- In case a file cannot be loaded.public void mouseClickTableView(javafx.scene.input.MouseEvent event) throws java.io.IOException, java.sql.SQLException
event
- The even that triggered this function.java.io.IOException
- In case a file cannot be loaded.java.sql.SQLException
- If connection to the data base failed.public void mouseClickComboBox(javafx.scene.input.MouseEvent event) throws java.io.IOException, java.sql.SQLException
event
- The even that triggered this function.java.io.IOException
- In case a file cannot be loaded.java.sql.SQLException
- If connection to the data base failed.public void saveUpdateButtonPressed(javafx.event.ActionEvent event) throws java.io.IOException, java.sql.SQLException
event
- The even that triggered this function.java.io.IOException
- IOException In case a file cannot be loaded.java.sql.SQLException
- The even that triggered this function.public void deleteUpdateButtonPressed(javafx.event.ActionEvent event) throws java.io.IOException, java.sql.SQLException
event
- The even that triggered this function.java.io.IOException
- IOException In case a file cannot be loaded.java.sql.SQLException
- The even that triggered this functionpublic void processCheckOutButtonPressed(javafx.event.ActionEvent event) throws java.io.IOException, java.sql.SQLException
event
- The even that triggered this function.java.io.IOException
- IOException In case a file cannot be loaded.java.sql.SQLException
- The even that triggered this functionpublic void processCheckInButtonPressed(javafx.event.ActionEvent event) throws java.io.IOException, java.sql.SQLException
event
- The even that triggered this function.java.io.IOException
- IOException In case a file cannot be loaded.java.sql.SQLException
- The even that triggered this functionpublic void clearCheckoutFormButtonPressed(javafx.event.ActionEvent event) throws java.io.IOException, java.sql.SQLException
event
- The even that triggered this function.java.io.IOException
- IOException In case a file cannot be loaded.java.sql.SQLException
- The even that triggered this functionpublic void clearCheckOutForm()
public int optionsAlert(java.lang.String message)
message
- The message to be displayed at the window.integer
specifying if user wants to proceed or not.


Middleware¶
Account Quotas¶
account_quotas
is a middleware which blocks write requests (PUT, POST) if a
given account quota (in bytes) is exceeded while DELETE requests are still
allowed.
account_quotas
uses the x-account-meta-quota-bytes
metadata entry to
store the quota. Write requests to this metadata entry are only permitted for
resellers. There is no quota limit if x-account-meta-quota-bytes
is not
set.
The account_quotas
middleware should be added to the pipeline in your
/etc/swift/proxy-server.conf
file just after any auth middleware.
For example:
[pipeline:main]
pipeline = catch_errors cache tempauth account_quotas proxy-server
[filter:account_quotas]
use = egg:swift#account_quotas
To set the quota on an account:
swift -A http://<IP_ADDRESS>:8080/auth/v1.0 -U account:reseller -K secret post -m quota-bytes:10000
Remove the quota:
swift -A http://<IP_ADDRESS>:8080/auth/v1.0 -U account:reseller -K secret post -m quota-bytes:
The same limitations apply for the account quotas as for the container quotas.
For example, when uploading an object without a content-length header the proxy server doesn’t know the final size of the currently uploaded object and the upload will be allowed if the current account size is within the quota. Due to the eventual consistency further uploads might be possible until the account size has been updated.
-
class
swift.common.middleware.account_quotas.
AccountQuotaMiddleware
(app, *args, **kwargs)¶ Bases:
object
Account quota middleware
See above for a full description.
-
swift.common.middleware.account_quotas.
filter_factory
(global_conf, **local_conf)¶ Returns a WSGI filter app for use with paste.deploy.
Bulk Operations (Delete and Archive Auto Extraction)¶
Middleware that will perform many operations on a single request.
Extract Archive¶
Expand tar files into a Swift account. Request must be a PUT with the
query parameter ?extract-archive=format
specifying the format of archive
file. Accepted formats are tar, tar.gz, and tar.bz2.
For a PUT to the following url:
/v1/AUTH_Account/$UPLOAD_PATH?extract-archive=tar.gz
UPLOAD_PATH is where the files will be expanded to. UPLOAD_PATH can be a container, a pseudo-directory within a container, or an empty string. The destination of a file in the archive will be built as follows:
/v1/AUTH_Account/$UPLOAD_PATH/$FILE_PATH
Where FILE_PATH is the file name from the listing in the tar file.
If the UPLOAD_PATH is an empty string, containers will be auto created accordingly and files in the tar that would not map to any container (files in the base directory) will be ignored.
Only regular files will be uploaded. Empty directories, symlinks, etc will not be uploaded.
Content Type¶
If the content-type header is set in the extract-archive call, Swift will assign that content-type to all the underlying files. The bulk middleware will extract the archive file and send the internal files using PUT operations using the same headers from the original request (e.g. auth-tokens, content-Type, etc.). Notice that any middleware call that follows the bulk middleware does not know if this was a bulk request or if these were individual requests sent by the user.
In order to make Swift detect the content-type for the files based on the file extension, the content-type in the extract-archive call should not be set. Alternatively, it is possible to explicitly tell Swift to detect the content type using this header:
X-Detect-Content-Type: true
For example:
curl -X PUT http://<IP_ADDRESS>/v1/AUTH_acc/cont/$?extract-archive=tar
-T backup.tar
-H "Content-Type: application/x-tar"
-H "X-Auth-Token: xxx"
-H "X-Detect-Content-Type: true"
Assigning Metadata¶
The tar file format (1) allows for UTF-8 key/value pairs to be associated with each file in an archive. If a file has extended attributes, then tar will store those as key/value pairs. The bulk middleware can read those extended attributes and convert them to Swift object metadata. Attributes starting with “user.meta” are converted to object metadata, and “user.mime_type” is converted to Content-Type.
For example:
setfattr -n user.mime_type -v "application/python-setup" setup.py
setfattr -n user.meta.lunch -v "burger and fries" setup.py
setfattr -n user.meta.dinner -v "baked ziti" setup.py
setfattr -n user.stuff -v "whee" setup.py
Will get translated to headers:
Content-Type: application/python-setup
X-Object-Meta-Lunch: burger and fries
X-Object-Meta-Dinner: baked ziti
The bulk middleware will handle xattrs stored by both GNU and BSD tar (2).
Only xattrs user.mime_type
and user.meta.*
are processed. Other
attributes are ignored.
Notes:
(1) The POSIX 1003.1-2001 (pax) format. The default format on GNU tar 1.27.1 or later.
(2) Even with pax-format tarballs, different encoders store xattrs slightly differently; for example, GNU tar stores the xattr “user.userattribute” as pax header “SCHILY.xattr.user.userattribute”, while BSD tar (which uses libarchive) stores it as “LIBARCHIVE.xattr.user.userattribute”.
Response¶
The response from bulk operations functions differently from other Swift responses. This is because a short request body sent from the client could result in many operations on the proxy server and precautions need to be made to prevent the request from timing out due to lack of activity. To this end, the client will always receive a 200 OK response, regardless of the actual success of the call. The body of the response must be parsed to determine the actual success of the operation. In addition to this the client may receive zero or more whitespace characters prepended to the actual response body while the proxy server is completing the request.
The format of the response body defaults to text/plain but can be either
json or xml depending on the Accept
header. Acceptable formats are
text/plain
, application/json
, application/xml
, and text/xml
.
An example body is as follows:
{"Response Status": "201 Created",
"Response Body": "",
"Errors": [],
"Number Files Created": 10}
If all valid files were uploaded successfully the Response Status will be 201 Created. If any files failed to be created the response code corresponds to the subrequest’s error. Possible codes are 400, 401, 502 (on server errors), etc. In both cases the response body will specify the number of files successfully uploaded and a list of the files that failed.
There are proxy logs created for each file (which becomes a subrequest) in the tar. The subrequest’s proxy log will have a swift.source set to “EA” the log’s content length will reflect the unzipped size of the file. If double proxy-logging is used the leftmost logger will not have a swift.source set and the content length will reflect the size of the payload sent to the proxy (the unexpanded size of the tar.gz).
Bulk Delete¶
Will delete multiple objects or containers from their account with a
single request. Responds to POST requests with query parameter
?bulk-delete
set. The request url is your storage url. The Content-Type
should be set to text/plain
. The body of the POST request will be a
newline separated list of url encoded objects to delete. You can delete
10,000 (configurable) objects per request. The objects specified in the
POST request body must be URL encoded and in the form:
/container_name/obj_name
or for a container (which must be empty at time of delete):
/container_name
The response is similar to extract archive as in every response will be a 200 OK and you must parse the response body for actual results. An example response is:
{"Number Not Found": 0,
"Response Status": "200 OK",
"Response Body": "",
"Errors": [],
"Number Deleted": 6}
If all items were successfully deleted (or did not exist), the Response
Status will be 200 OK. If any failed to delete, the response code
corresponds to the subrequest’s error. Possible codes are 400, 401, 502 (on
server errors), etc. In all cases the response body will specify the number
of items successfully deleted, not found, and a list of those that failed.
The return body will be formatted in the way specified in the request’s
Accept
header. Acceptable formats are text/plain
, application/json
,
application/xml
, and text/xml
.
There are proxy logs created for each object or container (which becomes a subrequest) that is deleted. The subrequest’s proxy log will have a swift.source set to “BD” the log’s content length of 0. If double proxy-logging is used the leftmost logger will not have a swift.source set and the content length will reflect the size of the payload sent to the proxy (the list of objects/containers to be deleted).
-
swift.common.middleware.bulk.
get_response_body
(data_format, data_dict, error_list)¶ Returns a properly formatted response body according to format. Handles json and xml, otherwise will return text/plain. Note: xml response does not include xml declaration. :params data_format: resulting format :params data_dict: generated data about results. :params error_list: list of quoted filenames that failed
CatchErrors¶
-
class
swift.common.middleware.catch_errors.
CatchErrorMiddleware
(app, conf)¶ Bases:
object
Middleware that provides high-level error handling and ensures that a transaction id will be set for every request.
CNAME Lookup¶
CNAME Lookup Middleware
Middleware that translates an unknown domain in the host header to something that ends with the configured storage_domain by looking up the given domain’s CNAME record in DNS.
This middleware will continue to follow a CNAME chain in DNS until it finds a record ending in the configured storage domain or it reaches the configured maximum lookup depth. If a match is found, the environment’s Host header is rewritten and the request is passed further down the WSGI chain.
-
class
swift.common.middleware.cname_lookup.
CNAMELookupMiddleware
(app, conf)¶ Bases:
object
CNAME Lookup Middleware
See above for a full description.
Parameters: - app – The next WSGI filter or app in the paste.deploy chain.
- conf – The configuration dict for the middleware.
-
swift.common.middleware.cname_lookup.
lookup_cname
(domain)¶ Given a domain, returns its DNS CNAME mapping and DNS ttl.
Parameters: domain – domain to query on Returns: (ttl, result)
Container Quotas¶
The container_quotas
middleware implements simple quotas that can be
imposed on swift containers by a user with the ability to set container
metadata, most likely the account administrator. This can be useful for
limiting the scope of containers that are delegated to non-admin users, exposed
to formpost
uploads, or just as a self-imposed sanity check.
Any object PUT operations that exceed these quotas return a 413 response (request entity too large) with a descriptive body.
Quotas are subject to several limitations: eventual consistency, the timeliness of the cached container_info (60 second ttl by default), and it’s unable to reject chunked transfer uploads that exceed the quota (though once the quota is exceeded, new chunked transfers will be refused).
Quotas are set by adding meta values to the container, and are validated when set:
| Metadata | Use |
|---|---|
| X-Container-Meta-Quota-Bytes | Maximum size of the container, in bytes. |
| X-Container-Meta-Quota-Count | Maximum object count of the container. |
The container_quotas
middleware should be added to the pipeline in your
/etc/swift/proxy-server.conf
file just after any auth middleware.
For example:
[pipeline:main]
pipeline = catch_errors cache tempauth container_quotas proxy-server
[filter:container_quotas]
use = egg:swift#container_quotas
Container Sync Middleware¶
-
class
swift.common.middleware.container_sync.
ContainerSync
(app, conf, logger=None)¶ Bases:
object
WSGI middleware that validates an incoming container sync request using the container-sync-realms.conf style of container sync.
Cross Domain Policies¶
-
class
swift.common.middleware.crossdomain.
CrossDomainMiddleware
(app, conf, *args, **kwargs)¶ Bases:
object
Cross domain middleware used to respond to requests for cross domain policy information.
If the path is /crossdomain.xml it will respond with an xml cross domain policy document. This allows web pages hosted elsewhere to use client side technologies such as Flash, Java and Silverlight to interact with the Swift API.
To enable this middleware, add it to the pipeline in your proxy-server.conf file. It should be added before any authentication (e.g., tempauth or keystone) middleware. In this example ellipsis (...) indicate other middleware you may have chosen to use:
[pipeline:main] pipeline = ... crossdomain ... authtoken ... proxy-server
And add a filter section, such as:
[filter:crossdomain] use = egg:swift#crossdomain cross_domain_policy = <allow-access-from domain="*.example.com" /> <allow-access-from domain="www.example.com" secure="false" />
For continuation lines, put some whitespace before the continuation text. Ensure you put a completely blank line to terminate the cross_domain_policy value.
The cross_domain_policy name/value is optional. If omitted, the policy defaults as if you had specified:
cross_domain_policy = <allow-access-from domain="*" secure="false" />
-
GET
(req)¶ Returns a 200 response with cross domain policy information
-
Discoverability¶
Swift will by default provide clients with an interface providing details
about the installation. Unless disabled (i.e expose_info=false
in
Proxy Server Configuration), a GET request to /info
will return configuration
data in JSON format. An example response:
{"swift": {"version": "1.11.0"}, "staticweb": {}, "tempurl": {}}
This would signify to the client that swift version 1.11.0 is running and that staticweb and tempurl are available in this installation.
There may be administrator-only information available via /info
. To
retrieve it, one must use an HMAC-signed request, similar to TempURL.
The signature may be produced like so:
swift-temp-url GET 3600 /info secret 2>/dev/null | sed s/temp_url/swiftinfo/g
Domain Remap¶
Domain Remap Middleware
Middleware that translates container and account parts of a domain to path parameters that the proxy server understands.
container.account.storageurl/object gets translated to container.account.storageurl/path_root/account/container/object
account.storageurl/path_root/container/object gets translated to account.storageurl/path_root/account/container/object
Browsers can convert a host header to lowercase, so check that reseller prefix on the account is the correct case. This is done by comparing the items in the reseller_prefixes config option to the found prefix. If they match except for case, the item from reseller_prefixes will be used instead of the found reseller prefix. When none match, the default reseller prefix is used. When no default reseller prefix is configured, any request with an account prefix not in that list will be ignored by this middleware. reseller_prefixes defaults to ‘AUTH’.
Note that this middleware requires that container names and account names (except as described above) must be DNS-compatible. This means that the account name created in the system and the containers created by users cannot exceed 63 characters or have UTF-8 characters. These are restrictions over and above what swift requires and are not explicitly checked. Simply put, the this middleware will do a best-effort attempt to derive account and container names from elements in the domain name and put those derived values into the URL path (leaving the Host header unchanged).
Also note that using container sync with remapped domain names is not advised. With container sync, you should use the true storage end points as sync destinations.
-
class
swift.common.middleware.domain_remap.
DomainRemapMiddleware
(app, conf)¶ Bases:
object
Domain Remap Middleware
See above for a full description.
Parameters: - app – The next WSGI filter or app in the paste.deploy chain.
- conf – The configuration dict for the middleware.
Dynamic Large Objects¶
DLO support centers around a user specified filter that matches segments and concatenates them together in object listing order. Please see the DLO docs for Dynamic Large Objects further details.
Encryption¶
Encryption middleware should be deployed in conjunction with the Keymaster middleware.
Implements middleware for object encryption which comprises an instance of a
Decrypter
combined with an
instance of an Encrypter
.
-
swift.common.middleware.crypto.
filter_factory
(global_conf, **local_conf)¶ Provides a factory function for loading encryption middleware.
-
class
swift.common.middleware.crypto.encrypter.
EncInputWrapper
(crypto, keys, req, logger)¶ Bases:
object
File-like object to be swapped in for wsgi.input.
-
class
swift.common.middleware.crypto.encrypter.
Encrypter
(app, conf)¶ Bases:
object
Middleware for encrypting data and user metadata.
By default all PUT or POST’ed object data and/or metadata will be encrypted. Encryption of new data and/or metadata may be disabled by setting the
disable_encryption
option to True. However, this middleware should remain in the pipeline in order for existing encrypted data to be read.
-
swift.common.middleware.crypto.encrypter.
encrypt_header_val
(crypto, value, key)¶ Encrypt a header value using the supplied key.
Parameters: - crypto – a Crypto instance
- value – value to encrypt
- key – crypto key to use
Returns: a tuple of (encrypted value, crypto_meta) where crypto_meta is a dict of form returned by
get_crypto_meta()
Raises: ValueError – if value is empty
-
class
swift.common.middleware.crypto.decrypter.
Decrypter
(app, conf)¶ Bases:
object
Middleware for decrypting data and user metadata.
FormPost¶
FormPost Middleware
Translates a browser form post into a regular Swift object PUT.
The format of the form is:
<form action="<swift-url>" method="POST"
enctype="multipart/form-data">
<input type="hidden" name="redirect" value="<redirect-url>" />
<input type="hidden" name="max_file_size" value="<bytes>" />
<input type="hidden" name="max_file_count" value="<count>" />
<input type="hidden" name="expires" value="<unix-timestamp>" />
<input type="hidden" name="signature" value="<hmac>" />
<input type="file" name="file1" /><br />
<input type="submit" />
</form>
Optionally, if you want the uploaded files to be temporary you can set x-delete-at or x-delete-after attributes by adding one of these as a form input:
<input type="hidden" name="x_delete_at" value="<unix-timestamp>" />
<input type="hidden" name="x_delete_after" value="<seconds>" />
The <swift-url> is the URL of the Swift destination, such as:
https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix
The name of each file uploaded will be appended to the <swift-url> given. So, you can upload directly to the root of container with a url like:
https://swift-cluster.example.com/v1/AUTH_account/container/
Optionally, you can include an object prefix to better separate different users’ uploads, such as:
https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix
Note the form method must be POST and the enctype must be set as “multipart/form-data”.
The redirect attribute is the URL to redirect the browser to after the upload completes. This is an optional parameter. If you are uploading the form via an XMLHttpRequest the redirect should not be included. The URL will have status and message query parameters added to it, indicating the HTTP status code for the upload (2xx is success) and a possible message for further information if there was an error (such as “max_file_size exceeded”).
The max_file_size attribute must be included and indicates the largest single file upload that can be done, in bytes.
The max_file_count attribute must be included and indicates the
maximum number of files that can be uploaded with the form. Include
additional <input type="file" name="filexx" />
attributes if
desired.
The expires attribute is the Unix timestamp before which the form must be submitted before it is invalidated.
The signature attribute is the HMAC-SHA1 signature of the form. Here is sample code for computing the signature:
import hmac
from hashlib import sha1
from time import time
path = '/v1/account/container/object_prefix'
redirect = 'https://srv.com/some-page' # set to '' if redirect not in form
max_file_size = 104857600
max_file_count = 10
expires = int(time() + 600)
key = 'mykey'
hmac_body = '%s\n%s\n%s\n%s\n%s' % (path, redirect,
max_file_size, max_file_count, expires)
signature = hmac.new(key, hmac_body, sha1).hexdigest()
The key is the value of either the account (X-Account-Meta-Temp-URL-Key, X-Account-Meta-Temp-Url-Key-2) or the container (X-Container-Meta-Temp-URL-Key, X-Container-Meta-Temp-Url-Key-2) TempURL keys.
Be certain to use the full path, from the /v1/ onward. Note that x_delete_at and x_delete_after are not used in signature generation as they are both optional attributes.
The command line tool swift-form-signature
may be used (mostly
just when testing) to compute expires and signature.
Also note that the file attributes must be after the other attributes in order to be processed correctly. If attributes come after the file, they won’t be sent with the subrequest (there is no way to parse all the attributes on the server-side without reading the whole thing into memory – to service many requests, some with large files, there just isn’t enough memory on the server, so attributes following the file are simply ignored).
-
class
swift.common.middleware.formpost.
FormPost
(app, conf)¶ Bases:
object
FormPost Middleware
See above for a full description.
The proxy logs created for any subrequests made will have swift.source set to “FP”.
Parameters: - app – The next WSGI filter or app in the paste.deploy chain.
- conf – The configuration dict for the middleware.
-
app
= None¶ The next WSGI application/filter in the paste.deploy pipeline.
-
conf
= None¶ The filter configuration dict.
-
swift.common.middleware.formpost.
filter_factory
(global_conf, **local_conf)¶ Returns the WSGI filter for use with paste.deploy.
-
swift.common.middleware.formpost.
READ_CHUNK_SIZE
= 4096¶ The size of data to read from the form at any given time.
-
swift.common.middleware.formpost.
MAX_VALUE_LENGTH
= 4096¶ The maximum size of any attribute’s value. Any additional data will be truncated.
GateKeeper¶
The gatekeeper
middleware imposes restrictions on the headers that
may be included with requests and responses. Request headers are filtered
to remove headers that should never be generated by a client. Similarly,
response headers are filtered to remove private headers that should
never be passed to a client.
The gatekeeper
middleware must always be present in the proxy server
wsgi pipeline. It should be configured close to the start of the pipeline
specified in /etc/swift/proxy-server.conf
, immediately after catch_errors
and before any other middleware. It is essential that it is configured ahead
of all middlewares using system metadata in order that they function
correctly.
If gatekeeper
middleware is not configured in the pipeline then it will be
automatically inserted close to the start of the pipeline by the proxy server.
-
swift.common.middleware.gatekeeper.
inbound_exclusions
= ['x-account-sysmeta-', 'x-container-sysmeta-', 'x-object-sysmeta-', 'x-object-transient-sysmeta-', 'x-backend']¶ A list of python regular expressions that will be used to match against inbound request headers. Matching headers will be removed from the request.
-
swift.common.middleware.gatekeeper.
outbound_exclusions
= ['x-account-sysmeta-', 'x-container-sysmeta-', 'x-object-sysmeta-', 'x-object-transient-sysmeta-', 'x-backend']¶ A list of python regular expressions that will be used to match against outbound response headers. Matching headers will be removed from the response.
Healthcheck¶
-
class
swift.common.middleware.healthcheck.
HealthCheckMiddleware
(app, conf)¶ Bases:
object
Healthcheck middleware used for monitoring.
If the path is /healthcheck, it will respond 200 with “OK” as the body.
If the optional config parameter “disable_path” is set, and a file is present at that path, it will respond 503 with “DISABLED BY FILE” as the body.
-
DISABLED
(req)¶ Returns a 503 response with “DISABLED BY FILE” in the body.
-
GET
(req)¶ Returns a 200 response with “OK” in the body.
-
Keymaster¶
Keymaster middleware should be deployed in conjunction with the Encryption middleware.
-
class
swift.common.middleware.crypto.keymaster.
KeyMaster
(app, conf)¶ Bases:
object
Middleware for providing encryption keys.
The middleware requires its
encryption_root_secret
option to be set. This is the root secret from which encryption keys are derived. This must be set before first use to a value that is a base64 encoding of at least 32 bytes. The security of all encrypted data critically depends on this key, therefore it should be set to a high-entropy value. For example, a suitable value may be obtained by base-64 encoding a 32 byte (or longer) value generated by a cryptographically secure random number generator. Changing the root secret is likely to result in data loss.
-
class
swift.common.middleware.crypto.keymaster.
KeyMasterContext
(keymaster, account, container, obj)¶ Bases:
swift.common.wsgi.WSGIContext
The simple scheme for key derivation is as follows: every path is associated with a key, where the key is derived from the path itself in a deterministic fashion such that the key does not need to be stored. Specifically, the key for any path is an HMAC of a root key and the path itself, calculated using an SHA256 hash function:
<path_key> = HMAC_SHA256(<root_secret>, <path>)
-
fetch_crypto_keys
(*args, **kwargs)¶ Setup container and object keys based on the request path.
Keys are derived from request path. The ‘id’ entry in the results dict includes the part of the path used to derive keys. Other keymaster implementations may use a different strategy to generate keys and may include a different type of ‘id’, so callers should treat the ‘id’ as opaque keymaster-specific data.
Returns: A dict containing encryption keys for ‘object’ and ‘container’ and a key ‘id’.
-
KeystoneAuth¶
-
class
swift.common.middleware.keystoneauth.
KeystoneAuth
(app, conf)¶ Bases:
object
Swift middleware to Keystone authorization system.
In Swift’s proxy-server.conf add this keystoneauth middleware and the authtoken middleware to your pipeline. Make sure you have the authtoken middleware before the keystoneauth middleware.
The authtoken middleware will take care of validating the user and keystoneauth will authorize access.
The sample proxy-server.conf shows a sample pipeline that uses keystone.
The authtoken middleware is shipped with keystonemiddleware - it does not have any other dependencies than itself so you can either install it by copying the file directly in your python path or by installing keystonemiddleware.
If support is required for unvalidated users (as with anonymous access) or for formpost/staticweb/tempurl middleware, authtoken will need to be configured with
delay_auth_decision
set to true. See the Keystone documentation for more detail on how to configure the authtoken middleware.In proxy-server.conf you will need to have the setting account auto creation to true:
[app:proxy-server] account_autocreate = true
And add a swift authorization filter section, such as:
[filter:keystoneauth] use = egg:swift#keystoneauth operator_roles = admin, swiftoperator
The user who is able to give ACL / create Containers permissions will be the user with a role listed in the
operator_roles
setting which by default includes the admin and the swiftoperator roles.The keystoneauth middleware maps a Keystone project/tenant to an account in Swift by adding a prefix (
AUTH_
by default) to the tenant/project id.. For example, if the project id is1234
, the path is/v1/AUTH_1234
.If you need to have a different reseller_prefix to be able to mix different auth servers you can configure the option
reseller_prefix
in your keystoneauth entry like this:reseller_prefix = NEWAUTH
Don’t forget to also update the Keystone service endpoint configuration to use NEWAUTH in the path.
It is possible to have several accounts associated with the same project. This is done by listing several prefixes as shown in the following example:
reseller_prefix = AUTH, SERVICEThis means that for project id ‘1234’, the paths ‘/v1/AUTH_1234’ and ‘/v1/SERVICE_1234’ are associated with the project and are authorized using roles that a user has with that project. The core use of this feature is that it is possible to provide different rules for each account prefix. The following parameters may be prefixed with the appropriate prefix:
operator_roles service_rolesFor backward compatibility, if either of these parameters is specified without a prefix then it applies to all reseller_prefixes. Here is an example, using two prefixes:
reseller_prefix = AUTH, SERVICE # The next three lines have identical effects (since the first applies # to both prefixes). operator_roles = admin, swiftoperator AUTH_operator_roles = admin, swiftoperator SERVICE_operator_roles = admin, swiftoperator # The next line only applies to accounts with the SERVICE prefix SERVICE_operator_roles = admin, some_other_role
X-Service-Token tokens are supported by the inclusion of the service_roles configuration option. When present, this option requires that the X-Service-Token header supply a token from a user who has a role listed in service_roles. Here is an example configuration:
reseller_prefix = AUTH, SERVICE AUTH_operator_roles = admin, swiftoperator SERVICE_operator_roles = admin, swiftoperator SERVICE_service_roles = service
The keystoneauth middleware supports cross-tenant access control using the syntax
<tenant>:<user>
to specify a grantee in container Access Control Lists (ACLs). For a request to be granted by an ACL, the grantee<tenant>
must match the UUID of the tenant to which the request X-Auth-Token is scoped and the grantee<user>
must match the UUID of the user authenticated by that token.Note that names must no longer be used in cross-tenant ACLs because with the introduction of domains in keystone names are no longer globally unique.
For backwards compatibility, ACLs using names will be granted by keystoneauth when it can be established that the grantee tenant, the grantee user and the tenant being accessed are either not yet in a domain (e.g. the X-Auth-Token has been obtained via the keystone v2 API) or are all in the default domain to which legacy accounts would have been migrated. The default domain is identified by its UUID, which by default has the value
default
. This can be changed by setting thedefault_domain_id
option in the keystoneauth configuration:default_domain_id = default
The backwards compatible behavior can be disabled by setting the config option
allow_names_in_acls
to false:allow_names_in_acls = false
To enable this backwards compatibility, keystoneauth will attempt to determine the domain id of a tenant when any new account is created, and persist this as account metadata. If an account is created for a tenant using a token with reselleradmin role that is not scoped on that tenant, keystoneauth is unable to determine the domain id of the tenant; keystoneauth will assume that the tenant may not be in the default domain and therefore not match names in ACLs for that account.
By default, middleware higher in the WSGI pipeline may override auth processing, useful for middleware such as tempurl and formpost. If you know you’re not going to use such middleware and you want a bit of extra security you can disable this behaviour by setting the
allow_overrides
option tofalse
:allow_overrides = false
Parameters: - app – The next WSGI app in the pipeline
- conf – The dict of configuration values
Authorize an anonymous request.
Returns: None if authorization is granted, an error page otherwise.
-
denied_response
(req)¶ Deny WSGI Response.
Returns a standard WSGI response callable with the status of 403 or 401 depending on whether the REMOTE_USER is set or not.
-
swift.common.middleware.keystoneauth.
filter_factory
(global_conf, **local_conf)¶ Returns a WSGI filter app for use with paste.deploy.
List Endpoints¶
List endpoints for an object, account or container.
This middleware makes it possible to integrate swift with software that relies on data locality information to avoid network overhead, such as Hadoop.
Using the original API, answers requests of the form:
/endpoints/{account}/{container}/{object}
/endpoints/{account}/{container}
/endpoints/{account}
/endpoints/v1/{account}/{container}/{object}
/endpoints/v1/{account}/{container}
/endpoints/v1/{account}
with a JSON-encoded list of endpoints of the form:
http://{server}:{port}/{dev}/{part}/{acc}/{cont}/{obj}
http://{server}:{port}/{dev}/{part}/{acc}/{cont}
http://{server}:{port}/{dev}/{part}/{acc}
correspondingly, e.g.:
http://<IP_ADDRESS>:6200/sda1/2/a/c2/o1
http://<IP_ADDRESS>:6200/sda1/2/a/c2
http://<IP_ADDRESS>:6200/sda1/2/a
Using the v2 API, answers requests of the form:
/endpoints/v2/{account}/{container}/{object}
/endpoints/v2/{account}/{container}
/endpoints/v2/{account}
with a JSON-encoded dictionary containing a key ‘endpoints’ that maps to a list of endpoints having the same form as described above, and a key ‘headers’ that maps to a dictionary of headers that should be sent with a request made to the endpoints, e.g.:
{ "endpoints": {"http://<IP_ADDRESS>:6010/sda1/2/a/c3/o1",
"http://<IP_ADDRESS>:6030/sda3/2/a/c3/o1",
"http://<IP_ADDRESS>:6040/sda4/2/a/c3/o1"},
"headers": {"X-Backend-Storage-Policy-Index": "1"}}
In this example, the ‘headers’ dictionary indicates that requests to the endpoint URLs should include the header ‘X-Backend-Storage-Policy-Index: 1’ because the object’s container is using storage policy index 1.
The ‘/endpoints/’ path is customizable (‘list_endpoints_path’ configuration parameter).
Intended for consumption by third-party services living inside the cluster (as the endpoints make sense only inside the cluster behind the firewall); potentially written in a different language.
This is why it’s provided as a REST API and not just a Python API: to avoid requiring clients to write their own ring parsers in their languages, and to avoid the necessity to distribute the ring file to clients and keep it up-to-date.
Note that the call is not authenticated, which means that a proxy with this middleware enabled should not be open to an untrusted environment (everyone can query the locality data using this middleware).
-
class
swift.common.middleware.list_endpoints.
ListEndpointsMiddleware
(app, conf)¶ Bases:
object
List endpoints for an object, account or container.
See above for a full description.
Uses configuration parameter swift_dir (default /etc/swift).
Parameters: - app – The next WSGI filter or app in the paste.deploy chain.
- conf – The configuration dict for the middleware.
-
get_object_ring
(policy_idx)¶ Get the ring object to use to handle a request based on its policy.
Policy_idx: policy index as defined in swift.conf Returns: appropriate ring object
Memcache¶
-
class
swift.common.middleware.memcache.
MemcacheMiddleware
(app, conf)¶ Bases:
object
Caching middleware that manages caching in swift.
Name Check (Forbidden Character Filter)¶
Created on February 27, 2012
A filter that disallows any paths that contain defined forbidden characters or that exceed a defined length.
Place early in the proxy-server pipeline after the left-most occurrence of the
proxy-logging
middleware (if present) and before the final
proxy-logging
middleware (if present) or the proxy-serer
app itself,
e.g.:
[pipeline:main]
pipeline = catch_errors healthcheck proxy-logging name_check cache ratelimit tempauth sos proxy-logging proxy-server
[filter:name_check]
use = egg:swift#name_check
forbidden_chars = '"`<>
maximum_length = 255
There are default settings for forbidden_chars (FORBIDDEN_CHARS) and maximum_length (MAX_LENGTH)
The filter returns HTTPBadRequest if path is invalid.
@author: eamonn-otoole
Object Versioning¶
Object versioning in swift is implemented by setting a flag on the container
to tell swift to version all objects in the container. The value of the flag is
the container where the versions are stored (commonly referred to as the
“archive container”). The flag itself is one of two headers, which determines
how object DELETE
requests are handled:
X-History-Location
On
DELETE
, copy the current version of the object to the archive container, write a zero-byte “delete marker” object that notes when the delete took place, and delete the object from the versioned container. The object will no longer appear in container listings for the versioned container and future requests there will return404 Not Found
. However, the content will still be recoverable from the archive container.
X-Versions-Location
On
DELETE
, only remove the current version of the object. If any previous versions exist in the archive container, the most recent one is copied over the current version, and the copy in the archive container is deleted. As a result, if you have 5 total versions of the object, you must delete the object 5 times for that object name to start responding with404 Not Found
.
Either header may be used for the various containers within an account, but
only one may be set for any given container. Attempting to set both
simulataneously will result in a 400 Bad Request
response.
Note
It is recommended to use a different archive container for each container that is being versioned.
Note
Enabling versioning on an archive container is not recommended.
When data is PUT
into a versioned container (a container with the
versioning flag turned on), the existing data in the file is redirected to a
new object in the archive container and the data in the PUT
request is
saved as the data for the versioned object. The new object name (for the
previous version) is <archive_container>/<length><object_name>/<timestamp>
,
where length
is the 3-character zero-padded hexadecimal length of the
<object_name>
and <timestamp>
is the timestamp of when the previous
version was created.
A GET
to a versioned object will return the current version of the object
without having to do any request redirects or metadata lookups.
A POST
to a versioned object will update the object metadata as normal,
but will not create a new version of the object. In other words, new versions
are only created when the content of the object changes.
A DELETE
to a versioned object will be handled in one of two ways,
as described above.
To restore a previous version of an object, find the desired version in the
archive container then issue a COPY
with a Destination
header
indicating the original location. This will archive the current version similar
to a PUT
over the versioned object. If the client additionally wishes to
permanently delete what was the current version, it must find the newly-created
archive in the archive container and issue a separate DELETE
to it.
How to Enable Object Versioning in a Swift Cluster¶
This middleware was written as an effort to refactor parts of the proxy server,
so this functionality was already available in previous releases and every
attempt was made to maintain backwards compatibility. To allow operators to
perform a seamless upgrade, it is not required to add the middleware to the
proxy pipeline and the flag allow_versions
in the container server
configuration files are still valid, but only when using
X-Versions-Location
. In future releases, allow_versions
will be
deprecated in favor of adding this middleware to the pipeline to enable or
disable the feature.
In case the middleware is added to the proxy pipeline, you must also
set allow_versioned_writes
to True
in the middleware options
to enable the information about this middleware to be returned in a /info
request.
Note
You need to add the middleware to the proxy pipeline and set
allow_versioned_writes = True
to useX-History-Location
. Settingallow_versions = True
in the container server is not sufficient to enable the use ofX-History-Location
.
Upgrade considerations:¶
If allow_versioned_writes
is set in the filter configuration, you can leave
the allow_versions
flag in the container server configuration files
untouched. If you decide to disable or remove the allow_versions
flag, you
must re-set any existing containers that had the X-Versions-Location
flag
configured so that it can now be tracked by the versioned_writes middleware.
Clients should not use the X-History-Location
header until all proxies in
the cluster have been upgraded to a version of Swift that supports it.
Attempting to use X-History-Location
during a rolling upgrade may result
in some requests being served by proxies running old code, leading to data
loss.
Examples Using curl
with X-Versions-Location
¶
First, create a container with the X-Versions-Location
header or add the
header to an existing container. Also make sure the container referenced by
the X-Versions-Location
exists. In this example, the name of that
container is “versions”:
curl -i -XPUT -H "X-Auth-Token: <token>" -H "X-Versions-Location: versions" http://<storage_url>/container
curl -i -XPUT -H "X-Auth-Token: <token>" http://<storage_url>/versions
Create an object (the first version):
curl -i -XPUT --data-binary 1 -H "X-Auth-Token: <token>" http://<storage_url>/container/myobject
Now create a new version of that object:
curl -i -XPUT --data-binary 2 -H "X-Auth-Token: <token>" http://<storage_url>/container/myobject
See a listing of the older versions of the object:
curl -i -H "X-Auth-Token: <token>" http://<storage_url>/versions?prefix=008myobject/
Now delete the current version of the object and see that the older version is gone from ‘versions’ container and back in ‘container’ container:
curl -i -XDELETE -H "X-Auth-Token: <token>" http://<storage_url>/container/myobject
curl -i -H "X-Auth-Token: <token>" http://<storage_url>/versions?prefix=008myobject/
curl -i -XGET -H "X-Auth-Token: <token>" http://<storage_url>/container/myobject
Examples Using curl
with X-History-Location
¶
As above, create a container with the X-History-Location
header and ensure
that the container referenced by the X-History-Location
exists. In this
example, the name of that container is “versions”:
curl -i -XPUT -H "X-Auth-Token: <token>" -H "X-History-Location: versions" http://<storage_url>/container
curl -i -XPUT -H "X-Auth-Token: <token>" http://<storage_url>/versions
Create an object (the first version):
curl -i -XPUT --data-binary 1 -H "X-Auth-Token: <token>" http://<storage_url>/container/myobject
Now create a new version of that object:
curl -i -XPUT --data-binary 2 -H "X-Auth-Token: <token>" http://<storage_url>/container/myobject
Now delete the current version of the object. Subsequent requests will 404:
curl -i -XDELETE -H "X-Auth-Token: <token>" http://<storage_url>/container/myobject
curl -i -H "X-Auth-Token: <token>" http://<storage_url>/container/myobject
A listing of the older versions of the object will include both the first and second versions of the object, as well as a “delete marker” object:
curl -i -H "X-Auth-Token: <token>" http://<storage_url>/versions?prefix=008myobject/
To restore a previous version, simply COPY
it from the archive container:
curl -i -XCOPY -H "X-Auth-Token: <token>" http://<storage_url>/versions/008myobject/<timestamp> -H "Destination: container/myobject"
Note that the archive container still has all previous versions of the object, including the source for the restore:
curl -i -H "X-Auth-Token: <token>" http://<storage_url>/versions?prefix=008myobject/
To permanently delete a previous version, DELETE
it from the archive
container:
curl -i -XDELETE -H "X-Auth-Token: <token>" http://<storage_url>/versions/008myobject/<timestamp>
How to Disable Object Versioning in a Swift Cluster¶
If you want to disable all functionality, set allow_versioned_writes
to
False
in the middleware options.
Disable versioning from a container (x is any value except empty):
curl -i -XPOST -H "X-Auth-Token: <token>" -H "X-Remove-Versions-Location: x" http://<storage_url>/container
Proxy Logging¶
Logging middleware for the Swift proxy.
This serves as both the default logging implementation and an example of how to plug in your own logging format/method.
The logging format implemented below is as follows:
- client_ip remote_addr datetime request_method request_path protocol
- status_int referer user_agent auth_token bytes_recvd bytes_sent client_etag transaction_id headers request_time source log_info request_start_time request_end_time
These values are space-separated, and each is url-encoded, so that they can be separated with a simple .split()
- remote_addr is the contents of the REMOTE_ADDR environment variable, while client_ip is swift’s best guess at the end-user IP, extracted variously from the X-Forwarded-For header, X-Cluster-Ip header, or the REMOTE_ADDR environment variable.
- source (swift.source in the WSGI environment) indicates the code that generated the request, such as most middleware. (See below for more detail.)
- log_info (swift.log_info in the WSGI environment) is for additional
information that could prove quite useful, such as any x-delete-at
value or other “behind the scenes” activity that might not
otherwise be detectable from the plain log information. Code that
wishes to add additional log information should use code like
env.setdefault('swift.log_info', []).append(your_info)
so as to not disturb others’ log information. - Values that are missing (e.g. due to a header not being present) or zero are generally represented by a single hyphen (‘-‘).
The proxy-logging can be used twice in the proxy server’s pipeline when there is middleware installed that can return custom responses that don’t follow the standard pipeline to the proxy server.
For example, with staticweb, the middleware might intercept a request to /v1/AUTH_acc/cont/, make a subrequest to the proxy to retrieve /v1/AUTH_acc/cont/index.html and, in effect, respond to the client’s original request using the 2nd request’s body. In this instance the subrequest will be logged by the rightmost middleware (with a swift.source set) and the outgoing request (with body overridden) will be logged by leftmost middleware.
Requests that follow the normal pipeline (use the same wsgi environment throughout) will not be double logged because an environment variable (swift.proxy_access_log_made) is checked/set when a log is made.
All middleware making subrequests should take care to set swift.source when needed. With the doubled proxy logs, any consumer/processor of swift’s proxy logs should look at the swift.source field, the rightmost log value, to decide if this is a middleware subrequest or not. A log processor calculating bandwidth usage will want to only sum up logs with no swift.source.
-
class
swift.common.middleware.proxy_logging.
ProxyLoggingMiddleware
(app, conf, logger=None)¶ Bases:
object
Middleware that logs Swift proxy requests in the swift log format.
-
log_request
(req, status_int, bytes_received, bytes_sent, start_time, end_time, resp_headers=None)¶ Log a request.
Parameters: - req – swob.Request object for the request
- status_int – integer code for the response status
- bytes_received – bytes successfully read from the request body
- bytes_sent – bytes yielded to the WSGI server
- start_time – timestamp request started
- end_time – timestamp request completed
- resp_headers – dict of the response headers
-
Ratelimit¶
-
class
swift.common.middleware.ratelimit.
RateLimitMiddleware
(app, conf, logger=None)¶ Bases:
object
Rate limiting middleware
Rate limits requests on both an Account and Container level. Limits are configurable.
-
get_ratelimitable_key_tuples
(req, account_name, container_name=None, obj_name=None, global_ratelimit=None)¶ Returns a list of key (used in memcache), ratelimit tuples. Keys should be checked in order.
Parameters: - req – swob request
- account_name – account name from path
- container_name – container name from path
- obj_name – object name from path
- global_ratelimit – this account has an account wide ratelimit on all writes combined
-
handle_ratelimit
(req, account_name, container_name, obj_name)¶ Performs rate limiting and account white/black listing. Sleeps if necessary. If self.memcache_client is not set, immediately returns None.
Parameters: - account_name – account name from path
- container_name – container name from path
- obj_name – object name from path
-
-
swift.common.middleware.ratelimit.
filter_factory
(global_conf, **local_conf)¶ paste.deploy app factory for creating WSGI proxy apps.
-
swift.common.middleware.ratelimit.
get_maxrate
(ratelimits, size)¶ Returns number of requests allowed per second for given size.
-
swift.common.middleware.ratelimit.
interpret_conf_limits
(conf, name_prefix, info=None)¶ Parses general parms for rate limits looking for things that start with the provided name_prefix within the provided conf and returns lists for both internal use and for /info
Parameters: - conf – conf dict to parse
- name_prefix – prefix of config parms to look for
- info – set to return extra stuff for /info registration
Recon¶
-
class
swift.common.middleware.recon.
ReconMiddleware
(app, conf, *args, **kwargs)¶ Bases:
object
Recon middleware used for monitoring.
/recon/load|mem|async... will return various system metrics.
Needs to be added to the pipeline and requires a filter declaration in the object-server.conf:
[filter:recon] use = egg:swift#recon recon_cache_path = /var/cache/swift
-
get_async_info
()¶ get # of async pendings
-
get_auditor_info
(recon_type)¶ get auditor info
-
get_device_info
()¶ get devices
-
get_diskusage
()¶ get disk utilization statistics
-
get_driveaudit_error
()¶ get # of drive audit errors
-
get_expirer_info
(recon_type)¶ get expirer info
-
get_load
(openr=<built-in function open>)¶ get info from /proc/loadavg
-
get_mem
(openr=<built-in function open>)¶ get info from /proc/meminfo
-
get_mounted
(openr=<built-in function open>)¶ get ALL mounted fs from /proc/mounts
-
get_quarantine_count
()¶ get obj/container/account quarantine counts
-
get_replication_info
(recon_type)¶ get replication info
-
get_ring_md5
()¶ get all ring md5sum’s
-
get_socket_info
(openr=<built-in function open>)¶ get info from /proc/net/sockstat and sockstat6
Note: The mem value is actually kernel pages, but we return bytes allocated based on the systems page size.
-
get_swift_conf_md5
()¶ get md5 of swift.conf
-
get_time
()¶ get current time
-
get_unmounted
()¶ list unmounted (failed?) devices
-
get_updater_info
(recon_type)¶ get updater info
-
get_version
()¶ get swift version
-
Server Side Copy¶
Server side copy is a feature that enables users/clients to COPY objects between accounts and containers without the need to download and then re-upload objects, thus eliminating additional bandwidth consumption and also saving time. This may be used when renaming/moving an object which in Swift is a (COPY + DELETE) operation.
The server side copy middleware should be inserted in the pipeline after auth and before the quotas and large object middlewares. If it is not present in the pipeline in the proxy-server configuration file, it will be inserted automatically. There is no configurable option provided to turn off server side copy.
Metadata¶
- All metadata of source object is preserved during object copy.
- One can also provide additional metadata during PUT/COPY request. This will over-write any existing conflicting keys.
- Server side copy can also be used to change content-type of an existing object.
Object Copy¶
- The destination container must exist before requesting copy of the object.
- When several replicas exist, the system copies from the most recent replica. That is, the copy operation behaves as though the X-Newest header is in the request.
- The request to copy an object should have no body (i.e. content-length of the request must be zero).
There are two ways in which an object can be copied:
Send a PUT request to the new object (destination/target) with an additional header named
X-Copy-From
specifying the source object (in ‘/container/object’ format). Example:curl -i -X PUT http://<storage_url>/container1/destination_obj -H 'X-Auth-Token: <token>' -H 'X-Copy-From: /container2/source_obj' -H 'Content-Length: 0'
Send a COPY request with an existing object in URL with an additional header named
Destination
specifying the destination/target object (in ‘/container/object’ format). Example:curl -i -X COPY http://<storage_url>/container2/source_obj -H 'X-Auth-Token: <token>' -H 'Destination: /container1/destination_obj' -H 'Content-Length: 0'
Note that if the incoming request has some conditional headers (e.g. Range
,
If-Match
), the source object will be evaluated for these headers (i.e. if
PUT with both X-Copy-From
and Range
, Swift will make a partial copy to
the destination object).
Cross Account Object Copy¶
Objects can also be copied from one account to another account if the user has the necessary permissions (i.e. permission to read from container in source account and permission to write to container in destination account).
Similar to examples mentioned above, there are two ways to copy objects across accounts:
Like the example above, send PUT request to copy object but with an additional header named
X-Copy-From-Account
specifying the source account. Example:curl -i -X PUT http://<host>:<port>/v1/AUTH_test1/container/destination_obj -H 'X-Auth-Token: <token>' -H 'X-Copy-From: /container/source_obj' -H 'X-Copy-From-Account: AUTH_test2' -H 'Content-Length: 0'
Like the previous example, send a COPY request but with an additional header named
Destination-Account
specifying the name of destination account. Example:curl -i -X COPY http://<host>:<port>/v1/AUTH_test2/container/source_obj -H 'X-Auth-Token: <token>' -H 'Destination: /container/destination_obj' -H 'Destination-Account: AUTH_test1' -H 'Content-Length: 0'
Large Object Copy¶
The best option to copy a large object is to copy segments individually. To copy the manifest object of a large object, add the query parameter to the copy request:
?multipart-manifest=get
If a request is sent without the query parameter, an attempt will be made to copy the whole object but will fail if the object size is greater than 5GB.
Object Post as Copy¶
Historically, this has been a feature (and a configurable option with default set to True) in proxy server configuration. This has been moved to server side copy middleware and the default changed to False.
When object_post_as_copy
is set to true
, an incoming POST request is
morphed into a COPY request where source and destination objects are same.
This feature was necessary because of a previous behavior where POSTS would update the metadata on the object but not on the container. As a result, features like container sync would not work correctly. This is no longer the case and this option is now deprecated. It will be removed in a future release.
Static Large Objects¶
Please see the SLO docs for Static Large Objects further details.
StaticWeb¶
This StaticWeb WSGI middleware will serve container data as a static web site
with index file and error file resolution and optional file listings. This mode
is normally only active for anonymous requests. When using keystone for
authentication set delay_auth_decision = true
in the authtoken middleware
configuration in your /etc/swift/proxy-server.conf
file. If you want to
use it with authenticated requests, set the X-Web-Mode: true
header on the
request.
The staticweb
filter should be added to the pipeline in your
/etc/swift/proxy-server.conf
file just after any auth middleware. Also, the
configuration section for the staticweb
middleware itself needs to be
added. For example:
[DEFAULT]
...
[pipeline:main]
pipeline = catch_errors healthcheck proxy-logging cache ratelimit tempauth
staticweb proxy-logging proxy-server
...
[filter:staticweb]
use = egg:swift#staticweb
Any publicly readable containers (for example, X-Container-Read: .r:*
, see
ACLs for more information on this) will be checked for
X-Container-Meta-Web-Index and X-Container-Meta-Web-Error header values:
X-Container-Meta-Web-Index <index.name>
X-Container-Meta-Web-Error <error.name.suffix>
If X-Container-Meta-Web-Index is set, any <index.name> files will be served
without having to specify the <index.name> part. For instance, setting
X-Container-Meta-Web-Index: index.html
will be able to serve the object
.../pseudo/path/index.html with just .../pseudo/path or .../pseudo/path/
If X-Container-Meta-Web-Error is set, any errors (currently just 401
Unauthorized and 404 Not Found) will instead serve the
.../<status.code><error.name.suffix> object. For instance, setting
X-Container-Meta-Web-Error: error.html
will serve .../404error.html for
requests for paths not found.
For pseudo paths that have no <index.name>, this middleware can serve HTML file
listings if you set the X-Container-Meta-Web-Listings: true
metadata item
on the container.
If listings are enabled, the listings can have a custom style sheet by setting
the X-Container-Meta-Web-Listings-CSS header. For instance, setting
X-Container-Meta-Web-Listings-CSS: listing.css
will make listings link to
the .../listing.css style sheet. If you “view source” in your browser on a
listing page, you will see the well defined document structure that can be
styled.
By default, the listings will be rendered with a label of
“Listing of /v1/account/container/path”. This can be altered by
setting a X-Container-Meta-Web-Listings-Label: <label>
. For example,
if the label is set to “example.com”, a label of
“Listing of example.com/path” will be used instead.
The content-type of directory marker objects can be modified by setting
the X-Container-Meta-Web-Directory-Type
header. If the header is not set,
application/directory is used by default. Directory marker objects are
0-byte objects that represent directories to create a simulated hierarchical
structure.
Example usage of this middleware via swift
:
Make the container publicly readable:
swift post -r '.r:*' containerYou should be able to get objects directly, but no index.html resolution or listings.
Set an index file directive:
swift post -m 'web-index:index.html' containerYou should be able to hit paths that have an index.html without needing to type the index.html part.
Turn on listings:
swift post -r '.r:*,.rlistings' container swift post -m 'web-listings: true' containerNow you should see object listings for paths and pseudo paths that have no index.html.
Enable a custom listings style sheet:
swift post -m 'web-listings-css:listings.css' containerSet an error file:
swift post -m 'web-error:error.html' containerNow 401’s should load 401error.html, 404’s should load 404error.html, etc.
Set Content-Type of directory marker object:
swift post -m 'web-directory-type:text/directory' containerNow 0-byte objects with a content-type of text/directory will be treated as directories rather than objects.
-
class
swift.common.middleware.staticweb.
StaticWeb
(app, conf)¶ Bases:
object
The Static Web WSGI middleware filter; serves container data as a static web site. See staticweb for an overview.
The proxy logs created for any subrequests made will have swift.source set to “SW”.
Parameters: - app – The next WSGI application/filter in the paste.deploy pipeline.
- conf – The filter configuration dict.
-
app
= None¶ The next WSGI application/filter in the paste.deploy pipeline.
-
conf
= None¶ The filter configuration dict. Only used in tests.
-
swift.common.middleware.staticweb.
filter_factory
(global_conf, **local_conf)¶ Returns a Static Web WSGI filter for use with paste.deploy.
TempAuth¶
-
class
swift.common.middleware.tempauth.
TempAuth
(app, conf)¶ Bases:
object
Test authentication and authorization system.
Add to your pipeline in proxy-server.conf, such as:
[pipeline:main] pipeline = catch_errors cache tempauth proxy-server
Set account auto creation to true in proxy-server.conf:
[app:proxy-server] account_autocreate = true
And add a tempauth filter section, such as:
[filter:tempauth] use = egg:swift#tempauth user_admin_admin = admin .admin .reseller_admin user_test_tester = testing .admin user_test2_tester2 = testing2 .admin user_test_tester3 = testing3 # To allow accounts/users with underscores you can base64 encode them. # Here is the account "under_score" and username "a_b" (note the lack # of padding equal signs): user64_dW5kZXJfc2NvcmU_YV9i = testing4
See the proxy-server.conf-sample for more information.
Multiple Reseller Prefix Items:
The reseller prefix specifies which parts of the account namespace this middleware is responsible for managing authentication and authorization. By default, the prefix is ‘AUTH’ so accounts and tokens are prefixed by ‘AUTH_’. When a request’s token and/or path start with ‘AUTH_’, this middleware knows it is responsible.
We allow the reseller prefix to be a list. In tempauth, the first item in the list is used as the prefix for tokens and user groups. The other prefixes provide alternate accounts that user’s can access. For example if the reseller prefix list is ‘AUTH, OTHER’, a user with admin access to ‘AUTH_account’ also has admin access to ‘OTHER_account’.
Required Group:
The group .admin is normally needed to access an account (ACLs provide an additional way to access an account). You can specify the
require_group
parameter. This means that you also need the named group to access an account. If you have several reseller prefix items, prefix therequire_group
parameter with the appropriate prefix.X-Service-Token:
If an X-Service-Token is presented in the request headers, the groups derived from the token are appended to the roles derived from X-Auth-Token. If X-Auth-Token is missing or invalid, X-Service-Token is not processed.
The X-Service-Token is useful when combined with multiple reseller prefix items. In the following configuration, accounts prefixed ‘SERVICE_’ are only accessible if X-Auth-Token is from the end-user and X-Service-Token is from the
glance
user:[filter:tempauth] use = egg:swift#tempauth reseller_prefix = AUTH, SERVICE SERVICE_require_group = .service user_admin_admin = admin .admin .reseller_admin user_joeacct_joe = joepw .admin user_maryacct_mary = marypw .admin user_glance_glance = glancepw .service
The name .service is an example. Unlike .admin and .reseller_admin it is not a reserved name.
Please note that ACLs can be set on service accounts and are matched against the identity validated by X-Auth-Token. As such ACLs can grant access to a service account’s container without needing to provide a service token, just like any other cross-reseller request using ACLs.
- Account ACLs:
If a swift_owner issues a POST or PUT to the account, with the X-Account-Access-Control header set in the request, then this may allow certain types of access for additional users.
- Read-Only: Users with read-only access can list containers in the account, list objects in any container, retrieve objects, and view unprivileged account/container/object metadata.
- Read-Write: Users with read-write access can (in addition to the read-only privileges) create objects, overwrite existing objects, create new containers, and set unprivileged container/object metadata.
- Admin: Users with admin access are swift_owners and can perform any action, including viewing/setting privileged metadata (e.g. changing account ACLs).
To generate headers for setting an account ACL:
from swift.common.middleware.acl import format_acl acl_data = { 'admin': ['alice'], 'read-write': ['bob', 'carol'] } header_value = format_acl(version=2, acl_dict=acl_data)
To generate a curl command line from the above:
token=... storage_url=... python -c ' from swift.common.middleware.acl import format_acl acl_data = { 'admin': ['alice'], 'read-write': ['bob', 'carol'] } headers = {'X-Account-Access-Control': format_acl(version=2, acl_dict=acl_data)} header_str = ' '.join(["-H '%s: %s'" % (k, v) for k, v in headers.items()]) print('curl -D- -X POST -H "x-auth-token: $token" %s ' '$storage_url' % header_str) '
Parameters: - app – The next WSGI app in the pipeline
- conf – The dict of configuration values from the Paste config file
-
account_acls
(req)¶ Return a dict of ACL data from the account server via get_account_info.
Auth systems may define their own format, serialization, structure, and capabilities implemented in the ACL headers and persisted in the sysmeta data. However, auth systems are strongly encouraged to be interoperable with Tempauth.
- Account ACLs are set and retrieved via the header
- X-Account-Access-Control
- For header format and syntax, see:
Returns None if the request is authorized to continue or a standard WSGI response callable if not.
-
denied_response
(req)¶ Returns a standard WSGI response callable with the status of 403 or 401 depending on whether the REMOTE_USER is set or not.
-
extract_acl_and_report_errors
(req)¶ Return a user-readable string indicating the errors in the input ACL, or None if there are no errors.
-
get_groups
(env, token)¶ Get groups for the given token.
Parameters: - env – The current WSGI environment dictionary.
- token – Token to validate and return a group string for.
Returns: None if the token is invalid or a string containing a comma separated list of groups the authenticated user is a member of. The first group in the list is also considered a unique identifier for that user.
-
handle
(env, start_response)¶ WSGI entry point for auth requests (ones that match the self.auth_prefix). Wraps env in swob.Request object and passes it down.
Parameters: - env – WSGI environment dictionary
- start_response – WSGI callable
-
handle_get_token
(req)¶ Handles the various request for token and service end point(s) calls. There are various formats to support the various auth servers in the past. Examples:
GET <auth-prefix>/v1/<act>/auth X-Auth-User: <act>:<usr> or X-Storage-User: <usr> X-Auth-Key: <key> or X-Storage-Pass: <key> GET <auth-prefix>/auth X-Auth-User: <act>:<usr> or X-Storage-User: <act>:<usr> X-Auth-Key: <key> or X-Storage-Pass: <key> GET <auth-prefix>/v1.0 X-Auth-User: <act>:<usr> or X-Storage-User: <act>:<usr> X-Auth-Key: <key> or X-Storage-Pass: <key>
On successful authentication, the response will have X-Auth-Token and X-Storage-Token set to the token to use with Swift and X-Storage-URL set to the URL to the default Swift cluster to use.
Parameters: req – The swob.Request to process. Returns: swob.Response, 2xx on success with data set as explained above.
-
handle_request
(req)¶ Entry point for auth requests (ones that match the self.auth_prefix). Should return a WSGI-style callable (such as swob.Response).
Parameters: req – swob.Request object
-
swift.common.middleware.tempauth.
filter_factory
(global_conf, **local_conf)¶ Returns a WSGI filter app for use with paste.deploy.
TempURL¶
TempURL Middleware
Allows the creation of URLs to provide temporary access to objects.
For example, a website may wish to provide a link to download a large object in Swift, but the Swift account has no public access. The website can generate a URL that will provide GET access for a limited time to the resource. When the web browser user clicks on the link, the browser will download the object directly from Swift, obviating the need for the website to act as a proxy for the request.
If the user were to share the link with all his friends, or accidentally post it on a forum, etc. the direct access would be limited to the expiration time set when the website created the link.
Beyond that, the middleware provides the ability to create URLs, which contain signatures which are valid for all objects which share a common prefix.
Client Usage¶
To create temporary URLs, first an X-Account-Meta-Temp-URL-Key
header must be set on the Swift account. Then, an HMAC-SHA1 (RFC 2104)
signature is generated using the HTTP method to allow (GET
, PUT
,
DELETE
, etc.), the Unix timestamp the access should be allowed until,
the full path to the object, and the key set on the account.
For example, here is code generating the signature for a GET
for 60
seconds on /v1/AUTH_account/container/object
:
import hmac
from hashlib import sha1
from time import time
method = 'GET'
expires = int(time() + 60)
path = '/v1/AUTH_account/container/object'
key = 'mykey'
hmac_body = '%s\n%s\n%s' % (method, expires, path)
sig = hmac.new(key, hmac_body, sha1).hexdigest()
Be certain to use the full path, from the /v1/
onward.
Let’s say sig
ends up equaling
da39a3ee5e6b4b0d3255bfef95601890afd80709
and expires
ends up
1323479485
. Then, for example, the website could provide a link to:
https://swift-cluster.example.com/v1/AUTH_account/container/object?
temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709&
temp_url_expires=1323479485
You may also use ISO 8601 UTC timestamps with the format
"%Y-%m-%dT%H:%M:%SZ"
instead of UNIX timestamps in the URL
(but NOT in the code above for generating the signature!).
So, the latter URL could also be formulated as:
https://swift-cluster.example.com/v1/AUTH_account/container/object? temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709& temp_url_expires=2011-12-10T01:11:25Z
If a prefix-based signature with the prefix pre
is desired, set path to:
path = 'prefix:/v1/AUTH_account/container/pre'
The generated signature would be valid for all objects starting
with pre
. The middleware detects a prefix-based temporary URL by
a query parameter called temp_url_prefix
. So, if sig
and expires
would end up like above, following URL would be valid:
https://swift-cluster.example.com/v1/AUTH_account/container/pre/object?
temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709&
temp_url_expires=1323479485&
temp_url_prefix=pre
Another valid URL:
https://swift-cluster.example.com/v1/AUTH_account/container/pre/
subfolder/another_object?
temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709&
temp_url_expires=1323479485&
temp_url_prefix=pre
Any alteration of the resource path or query arguments of a temporary URL
would result in 401 Unauthorized
. Similarly, a PUT
where GET
was
the allowed method would be rejected with 401 Unauthorized
.
However, HEAD
is allowed if GET
, PUT
, or POST
is allowed.
Using this in combination with browser form post translation middleware could also allow direct-from-browser uploads to specific locations in Swift.
TempURL supports both account and container level keys. Each allows up to two
keys to be set, allowing key rotation without invalidating all existing
temporary URLs. Account keys are specified by X-Account-Meta-Temp-URL-Key
and X-Account-Meta-Temp-URL-Key-2
, while container keys are specified by
X-Container-Meta-Temp-URL-Key
and X-Container-Meta-Temp-URL-Key-2
.
Signatures are checked against account and container keys, if
present.
With GET
TempURLs, a Content-Disposition
header will be set on the
response so that browsers will interpret this as a file attachment to
be saved. The filename chosen is based on the object name, but you
can override this with a filename query parameter. Modifying the
above example:
https://swift-cluster.example.com/v1/AUTH_account/container/object?
temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709&
temp_url_expires=1323479485&filename=My+Test+File.pdf
If you do not want the object to be downloaded, you can cause
Content-Disposition: inline
to be set on the response by adding the
inline
parameter to the query string, like so:
https://swift-cluster.example.com/v1/AUTH_account/container/object?
temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709&
temp_url_expires=1323479485&inline
In some cases, the client might not able to present the content of the object,
but you still want the content able to save to local with the specific
filename. So you can cause Content-Disposition: inline; filename=...
to be
set on the response by adding the inline&filename=...
parameter to the
query string, like so:
https://swift-cluster.example.com/v1/AUTH_account/container/object?
temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709&
temp_url_expires=1323479485&inline&filename=My+Test+File.pdf
Cluster Configuration¶
This middleware understands the following configuration settings:
incoming_remove_headers
- A whitespace-delimited list of the headers to remove from
incoming requests. Names may optionally end with
*
to indicate a prefix match.incoming_allow_headers
is a list of exceptions to these removals. Default:x-timestamp
incoming_allow_headers
A whitespace-delimited list of the headers allowed as exceptions to
incoming_remove_headers
. Names may optionally end with*
to indicate a prefix match.Default: None
outgoing_remove_headers
A whitespace-delimited list of the headers to remove from outgoing responses. Names may optionally end with
*
to indicate a prefix match.outgoing_allow_headers
is a list of exceptions to these removals.Default:
x-object-meta-*
outgoing_allow_headers
A whitespace-delimited list of the headers allowed as exceptions to
outgoing_remove_headers
. Names may optionally end with*
to indicate a prefix match.Default:
x-object-meta-public-*
methods
A whitespace delimited list of request methods that are allowed to be used with a temporary URL.
Default:
GET HEAD PUT POST DELETE
-
class
swift.common.middleware.tempurl.
TempURL
(app, conf)¶ Bases:
object
WSGI Middleware to grant temporary URLs specific access to Swift resources. See the overview for more information.
The proxy logs created for any subrequests made will have swift.source set to “TU”.
Parameters: - app – The next WSGI filter or app in the paste.deploy chain.
- conf – The configuration dict for the middleware.
-
agent
= None¶ HTTP user agent to use for subrequests.
-
app
= None¶ The next WSGI application/filter in the paste.deploy pipeline.
-
conf
= None¶ The filter configuration dict.
-
incoming_allow_headers
= None¶ Headers to allow in incoming requests. Uppercase WSGI env style, like HTTP_X_MATCHES_REMOVE_PREFIX_BUT_OKAY.
-
incoming_allow_headers_startswith
= None¶ Header with match prefixes to allow in incoming requests. Uppercase WSGI env style, like HTTP_X_MATCHES_REMOVE_PREFIX_BUT_OKAY_*.
-
incoming_remove_headers
= None¶ Headers to remove from incoming requests. Uppercase WSGI env style, like HTTP_X_PRIVATE.
-
incoming_remove_headers_startswith
= None¶ Header with match prefixes to remove from incoming requests. Uppercase WSGI env style, like HTTP_X_SENSITIVE_*.
-
outgoing_allow_headers
= None¶ Headers to allow in outgoing responses. Lowercase, like x-matches-remove-prefix-but-okay.
-
outgoing_allow_headers_startswith
= None¶ Header with match prefixes to allow in outgoing responses. Lowercase, like x-matches-remove-prefix-but-okay-*.
-
outgoing_remove_headers
= None¶ Headers to remove from outgoing responses. Lowercase, like x-account-meta-temp-url-key.
-
outgoing_remove_headers_startswith
= None¶ Header with match prefixes to remove from outgoing responses. Lowercase, like x-account-meta-private-*.
-
swift.common.middleware.tempurl.
filter_factory
(global_conf, **local_conf)¶ Returns the WSGI filter for use with paste.deploy.
-
swift.common.middleware.tempurl.
DEFAULT_INCOMING_REMOVE_HEADERS
= 'x-timestamp'¶ Default headers to remove from incoming requests. Simply a whitespace delimited list of header names and names can optionally end with ‘*’ to indicate a prefix match. DEFAULT_INCOMING_ALLOW_HEADERS is a list of exceptions to these removals.
-
swift.common.middleware.tempurl.
DEFAULT_INCOMING_ALLOW_HEADERS
= ''¶ Default headers as exceptions to DEFAULT_INCOMING_REMOVE_HEADERS. Simply a whitespace delimited list of header names and names can optionally end with ‘*’ to indicate a prefix match.
-
swift.common.middleware.tempurl.
DEFAULT_OUTGOING_REMOVE_HEADERS
= 'x-object-meta-*'¶ Default headers to remove from outgoing responses. Simply a whitespace delimited list of header names and names can optionally end with ‘*’ to indicate a prefix match. DEFAULT_OUTGOING_ALLOW_HEADERS is a list of exceptions to these removals.
-
swift.common.middleware.tempurl.
DEFAULT_OUTGOING_ALLOW_HEADERS
= 'x-object-meta-public-*'¶ Default headers as exceptions to DEFAULT_OUTGOING_REMOVE_HEADERS. Simply a whitespace delimited list of header names and names can optionally end with ‘*’ to indicate a prefix match.
XProfile¶
Profiling middleware for Swift Servers.
The current implementation is based on eventlet aware profiler.(For the future, more profilers could be added in to collect more data for analysis.) Profiling all incoming requests and accumulating cpu timing statistics information for performance tuning and optimization. An mini web UI is also provided for profiling data analysis. It can be accessed from the URL as below.
Index page for browse profile data:
http://SERVER_IP:PORT/__profile__
List all profiles to return profile ids in json format:
http://SERVER_IP:PORT/__profile__/
http://SERVER_IP:PORT/__profile__/all
Retrieve specific profile data in different formats:
http://SERVER_IP:PORT/__profile__/PROFILE_ID?format=[default|json|csv|ods]
http://SERVER_IP:PORT/__profile__/current?format=[default|json|csv|ods]
http://SERVER_IP:PORT/__profile__/all?format=[default|json|csv|ods]
Retrieve metrics from specific function in json format:
http://SERVER_IP:PORT/__profile__/PROFILE_ID/NFL?format=json
http://SERVER_IP:PORT/__profile__/current/NFL?format=json
http://SERVER_IP:PORT/__profile__/all/NFL?format=json
NFL is defined by concatenation of file name, function name and the first
line number.
e.g.::
account.py:50(GETorHEAD)
or with full path:
opt/stack/swift/swift/proxy/controllers/account.py:50(GETorHEAD)
A list of URL examples:
http://localhost:8080/__profile__ (proxy server)
http://localhost:6200/__profile__/all (object server)
http://localhost:6201/__profile__/current (container server)
http://localhost:6202/__profile__/12345?format=json (account server)
The profiling middleware can be configured in paste file for WSGI servers such as proxy, account, container and object servers. Please refer to the sample configuration files in etc directory.
The profiling data is provided with four formats such as binary(by default), json, csv and odf spreadsheet which requires installing odfpy library.
sudo pip install odfpy
There’s also a simple visualization capability which is enabled by using matplotlib toolkit. it is also required to be installed if you want to use it to visualize statistic data.
sudo apt-get install python-matplotlib


(Static Line Charts With Full Size Legends)
3 Humidity Sensor Changes, Light Theme
Multiple Chart Types From the Same Data
Factory Temps Split by Station as a Line Chart, Light Theme
Factory Temps Split by Station as a Bar Chart
Factory Temps Split by Station as a Pie Chart, Light Theme
Charts Gracefully Animating Data In and Out
Factory 3 +/- Factory 1 as a Line Chart
Factory 3 +/- Factory 1 as a Bar Chart, Relative Y Axis
Line Charts Updating in Real Time
Line Charts with Multiple Series Types
Tag-Centric Hierarchy Control
Line Chart with Context Menu to Create Pie/Bar Chart
Factories 1 and 3 as a Line Chart
Factory 1 or 3 Pie Chart Generated from Line Chart Context Menu
Factory 1 or 3 Bar Chart Generated from Line Chart Context Menu
A standalone accessible grid
A heatmap with multiple aggregate expressions
A Table of Events