S3 is slower to work with than HDFS, even on virtual clusters running on Amazon EC2.
That’s because its a very different system, as you can see:
| Feature | HDFS | S3 through the S3A connector |
|---|---|---|
| communication | RPC | HTTP GET/PUT/HEAD/LIST/COPY requests |
| data locality | local storage | remote S3 servers |
| replication | multiple datanodes | asynchronous after upload |
| consistency | consistent data and listings | eventual consistent for listings, deletes and updates |
| bandwidth | best: local IO, worst: datacenter network | bandwidth between servers and S3 |
| latency | low | high, especially for “low cost” directory operations |
| rename | fast, atomic | slow faked rename through COPY & DELETE |
| delete | fast, atomic | fast for a file, slow & non-atomic for directories |
| writing | incremental | in blocks; not visible until the writer is closed |
| reading | seek() is fast | seek() is slow and expensive |
| IOPs | limited only by hardware | callers are throttled to shards in an s3 bucket |
| Security | Posix user+group; ACLs | AWS Roles and policies |
From a performance perspective, key points to remember are:
Overall, although the S3A connector makes S3 look like a file system, it isn’t, and some attempts to preserve the metaphor are “aggressively suboptimal”.
To make most efficient use of S3, care is needed.
S3Guard provides significant speedups for operations which list files a lot. This includes the setup of all queries against data: MapReduce, Hive and Spark, as well as DistCP.
Experiment with using it to see what speedup it delivers.
The S3A Filesystem client supports the notion of input policies, similar to that of the Posix fadvise() API call. This tunes the behavior of the S3A client to optimise HTTP GET requests for the different use cases.
Read through the file, possibly with some short forward seeks.
The whole document is requested in a single HTTP request; forward seeks within the readahead range are supported by skipping over the intermediate data.
This delivers maximum sequential throughput —but with very expensive backward seeks.
Applications reading a file in bulk (DistCP, any copy operations) should use sequential access, as should those reading data from gzipped .gz files. Because the “normal” fadvise policy starts off in sequential IO mode, there is rarely any need to explicit request this policy.
Optimised for random IO, specifically the Hadoop PositionedReadable operations —though seek(offset); read(byte_buffer) also benefits.
Rather than ask for the whole file, the range of the HTTP request is set to that that of the length of data desired in the read operation (Rounded up to the readahead value set in setReadahead() if necessary).
By reducing the cost of closing existing HTTP requests, this is highly efficient for file IO accessing a binary file through a series of PositionedReadable.read() and PositionedReadable.readFully() calls. Sequential reading of a file is expensive, as now many HTTP requests must be made to read through the file: there’s a delay between each GET operation.
Random IO is best for IO with seek-heavy characteristics:
The desired fadvise policy must be set in the configuration option fs.s3a.experimental.input.fadvise when the filesystem instance is created. That is: it can only be set on a per-filesystem basis, not on a per-file-read basis.
<property> <name>fs.s3a.experimental.input.fadvise</name> <value>random</value> <description> Policy for reading files. Values: 'random', 'sequential' or 'normal' </description> </property>
HDFS-2744, Extend FSDataInputStream to allow fadvise proposes adding a public API to set fadvise policies on input streams. Once implemented, this will become the supported mechanism used for configuring the input IO policy.
The normal policy starts off reading a file in sequential mode, but if the caller seeks backwards in the stream, it switches from sequential to random.
This policy essentially recognizes the initial read pattern of columnar storage formats (e.g. Apache ORC and Apache Parquet), which seek to the end of a file, read in index data and then seek backwards to selectively read columns. The first seeks may be be expensive compared to the random policy, however the overall process is much less expensive than either sequentially reading through a file with the random policy, or reading columnar data with the sequential policy.
Hadoop MapReduce, Apache Hive and Apache Spark all write their work to HDFS and similar filesystems. When using S3 as a destination, this is slow because of the way rename() is mimicked with copy and delete.
If committing output takes a long time, it is because you are using the standard FileOutputCommitter. If you are doing this on any S3 endpoint which lacks list consistency (Amazon S3 without S3Guard), this committer is at risk of losing data!
Your problem may appear to be performance, but that is a symptom of the underlying problem: the way S3A fakes rename operations means that the rename cannot be safely be used in output-commit algorithms.
Fix: Use one of the dedicated S3A Committers.
Each S3A client interacting with a single bucket, as a single user, has its own dedicated pool of open HTTP 1.1 connections alongside a pool of threads used for upload and copy operations. The default pool sizes are intended to strike a balance between performance and memory/thread use.
You can have a larger pool of (reused) HTTP connections and threads for parallel IO (especially uploads) by setting the properties
| property | meaning | default |
|---|---|---|
| fs.s3a.threads.max | Threads in the AWS transfer manager | 10 |
| fs.s3a.connection.maximum | Maximum number of HTTP connections | 10 |
We recommend using larger values for processes which perform a lot of IO: DistCp, Spark Workers and similar.
<property> <name>fs.s3a.threads.max</name> <value>20</value> </property> <property> <name>fs.s3a.connection.maximum</name> <value>20</value> </property>
Be aware, however, that processes which perform many parallel queries may consume large amounts of resources if each query is working with a different set of s3 buckets, or are acting on behalf of different users.
When uploading data, it is uploaded in blocks set by the option fs.s3a.block.size; default value “32M” for 32 Megabytes.
If a larger value is used, then more data is buffered before the upload begins:
<property> <name>fs.s3a.block.size</name> <value>128M</value> </property>
This means that fewer PUT/POST requests are made of S3 to upload data, which reduces the likelihood that S3 will throttle the client(s)
When large files are being uploaded, blocks are saved to disk and then queued for uploading, with multiple threads uploading different blocks in parallel.
The blocks can be buffered in memory by setting the option fs.s3a.fast.upload.buffer to bytebuffer, or, for on-heap storage array.
It is very easy to run out of memory when buffering to it; the option fs.s3a.fast.upload.active.blocks" exists to tune how many active blocks a single output stream writing to S3 may have queued at a time.
As the size of each buffered block is determined by the value of fs.s3a.block.size, the larger the block size, the more likely you will run out of memory.
DistCP can be slow, especially if the parameters and options for the operation are not tuned for working with S3.
To exacerbate the issue, DistCP invariably puts heavy load against the bucket being worked with, which will cause S3 to throttle requests. It will throttle: directory operations, uploads of new data, and delete operations, amongst other things
Adding more maps with the -m option does not guarantee better performance; it may just increase the amount of throttling which takes place. A smaller number of maps with a higher bandwidth per map can be more efficient.
DistCp’s -atomic option copies up data into a directory, then renames it into place, which is the where the copy takes place. This is a performance killer.
As discussed earlier, use large values for fs.s3a.threads.max and fs.s3a.connection.maximum.
Make sure that the bucket is using sequential or normal fadvise seek policies, that is, fs.s3a.experimental.fadvise is not set to random
Perform listings in parallel by setting -numListstatusThreads to a higher number. Make sure that fs.s3a.connection.maximum is equal to or greater than the value used.
If using -delete, set fs.trash.interval to 0 to avoid the deleted objects from being copied to a trash directory.
DO NOT switch fs.s3a.fast.upload.buffer to buffer in memory. If one distcp mapper runs out of memory it will fail, and that runs the risk of failing the entire job. It is safer to keep the default value, disk.
What is potentially useful is uploading in bigger blocks; this is more efficient in terms of HTTP connection use, and reduce the IOP rate against the S3 bucket/shard.
<property> <name>fs.s3a.threads.max</name> <value>20</value> </property> <property> <name>fs.s3a.connection.maximum</name> <value>30</value> <descriptiom> Make greater than both fs.s3a.threads.max and -numListstatusThreads </descriptiom> </property> <property> <name>fs.s3a.experimental.fadvise</name> <value>normal</value> </property> <property> <name>fs.s3a.block.size</name> <value>128M</value> </property> <property> <name>fs.s3a.fast.upload.buffer</name> <value>disk</value> </property> <property> <name>fs.trash.interval</name> <value>0</value> </property>
The hadoop fs -rm command can rename the file under .Trash rather than deleting it. Use -skipTrash to eliminate that step.
This can be set in the property fs.trash.interval; while the default is 0, most HDFS deployments have it set to a non-zero value to reduce the risk of data loss.
<property> <name>fs.trash.interval</name> <value>0</value> </property>
Amazon S3 uses a set of front-end servers to provide access to the underlying data. The choice of which front-end server to use is handled via load-balancing DNS service: when the IP address of an S3 bucket is looked up, the choice of which IP address to return to the client is made based on the the current load of the front-end servers.
Over time, the load across the front-end changes, so those servers considered “lightly loaded” will change. If the DNS value is cached for any length of time, your application may end up talking to an overloaded server. Or, in the case of failures, trying to talk to a server that is no longer there.
And by default, for historical security reasons in the era of applets, the DNS TTL of a JVM is “infinity”.
To work with AWS better, set the DNS time-to-live of an application which works with S3 to something lower. See AWS documentation.
An example of this is covered in HADOOP-13871.
curl -O https://landsat-pds.s3.amazonaws.com/scene_list.gz
When many requests are made of a specific S3 bucket (or shard inside it), S3 will respond with a 503 “throttled” response. Throttling can be recovered from, provided overall load decreases. Furthermore, because it is sent before any changes are made to the object store, is inherently idempotent. For this reason, the client will always attempt to retry throttled requests.
The limit of the number of times a throttled request can be retried, and the exponential interval increase between attempts, can be configured independently of the other retry limits.
<property> <name>fs.s3a.retry.throttle.limit</name> <value>20</value> <description> Number of times to retry any throttled request. </description> </property> <property> <name>fs.s3a.retry.throttle.interval</name> <value>500ms</value> <description> Interval between retry attempts on throttled requests. </description> </property>
If a client is failing due to AWSServiceThrottledException failures, increasing the interval and limit may address this. However, it it is a sign of AWS services being overloaded by the sheer number of clients and rate of requests. Spreading data across different buckets, and/or using a more balanced directory structure may be beneficial. Consult the AWS documentation.
Reading or writing data encrypted with SSE-KMS forces S3 to make calls of the AWS KMS Key Management Service, which comes with its own Request Rate Limits. These default to 1200/second for an account, across all keys and all uses of them, which, for S3 means: across all buckets with data encrypted with SSE-KMS.
If you are seeing a lot of throttling responses on a large scale operation like a distcp copy, reduce the number of processes trying to work with the bucket (for distcp: reduce the number of mappers with the -m option).
If you are reading or writing lists of files, if you can randomize the list so they are not processed in a simple sorted order, you may reduce load on a specific shard of S3 data, so potentially increase throughput.
An S3 Bucket is throttled by requests coming from all simultaneous clients. Different applications and jobs may interfere with each other: consider that when troubleshooting. Partitioning data into different buckets may help isolate load here.
If you are using data encrypted with SSE-KMS, then the will also apply: these are stricter than the S3 numbers. If you believe that you are reaching these limits, you may be able to get them increased. Consult the KMS Rate Limit documentation.
S3Guard uses DynamoDB for directory and file lookups; it is rate limited to the amount of (guaranteed) IO purchased for a table.
To see the allocated capacity of a bucket, the hadoop s3guard bucket-info s3a://bucket command will print out the allocated capacity.
If significant throttling events/rate is observed here, the pre-allocated IOPs can be increased with the hadoop s3guard set-capacity command, or through the AWS Console. Throttling events in S3Guard are noted in logs, and also in the S3A metrics s3guard_metadatastore_throttle_rate and s3guard_metadatastore_throttled.
If you are using DistCP for a large backup to/from a S3Guarded bucket, it is actually possible to increase the capacity for the duration of the operation.
Here are some best practises if you are writing applications to work with S3 or any other object store through the Hadoop APIs.
Use listFiles(path, recursive) over listStatus(path). The recursive listFiles() call can enumerate all dependents of a path in a single LIST call, irrespective of how deep the path is. In contrast, any directory tree-walk implemented in the client is issuing multiple HTTP requests to scan each directory, all the way down.
Cache the outcome of getFileStats(), rather than repeatedly ask for it. That includes using isFile(), isDirectory(), which are simply wrappers around getFileStatus().
Don’t immediately look for a file with a getFileStatus() or listing call after creating it, or try to read it immediately. This is where eventual consistency problems surface: the data may not yet be visible.
Rely on FileNotFoundException being raised if the source of an operation is missing, rather than implementing your own probe for the file before conditionally calling the operation.
Avoid any algorithm which uploads data into a temporary file and then uses rename() to commit it into place with a final path. On HDFS this offers a fast commit operation. With S3, Wasb and other object stores, you can write straight to the destination, knowing that the file isn’t visible until you close the write: the write itself is atomic.
The rename() operation may return false if the source is missing; this is a weakness in the API. Consider a check before calling rename, and if/when a new rename() call is made public, switch to it.
Keep in mind that delete(path, recursive) is a no-op if the path does not exist, so there’s no need to have a check for the path existing before you call it.
delete() is often used as a cleanup operation. With an object store this is slow, and may cause problems if the caller expects an immediate response. For example, a thread may block so long that other liveness checks start to fail. Consider spawning off an executor thread to do these background cleanup operations.


Smart Electronics
Smart Electronics
Innovative products
We have all of the latest smart phone devices in stock.
Free Downloads
Download all of the software updates you need for your devices through our website.
Security Assurance
Your personal information is secure and will be kept private.
$400
Lorem ipsum.
Lorem ipsum dolor.
Lorem ipsum.
Lorem ipsum.
$320
Lorem ipsum.
Lorem ipsum dolor.
Lorem ipsum.
Lorem ipsum.
$250
Lorem ipsum.
Lorem ipsum dolor.
Lorem ipsum.
Lorem ipsum.
PrivacyTerms and ConditionsContact Us
© 2019, Smart Electronics


Preparation time
less than 30 mins
Cooking time
1 to 2 hours
Serves
Serves 4
Preheat the oven to 220C/400F/Gas 6. Grease an ovenproof dish with butter.
Boil the eggs for 8-10 minutes in a large pan of boiling water. Remove from the pan using a slotted spoon and run under cold water until cool enough to handle.
When cool, peel the eggs and slice into rings. Set aside.
For the mashed potatoes, cook the potatoes in a separate pan of boiling, salted water for 10-15 minutes, or until tender. Drain well.
Pass the potatoes through a ricer until smooth, collecting the mash in a clean pan. Heat the mashed potatoes over a low heat and add the butter and cream, mixing until well combined. Season, to taste, with salt and freshly ground white pepper. Set aside and keep warm.
Meanwhile, boil the leeks in a pan of boiling water for 3-4 minutes, or until tender, then drain well. Stir the cooked leeks into the mashed potatoes until well combined.
For the sauce, melt the butter in a pan over a low heat. Whisk in the flour until the mixture is thick and well combined.
Gradually add the milk, whisking continuously, for 4-5 minutes, or until all of the milk has been incorporated into the mixture and the sauce is thick and smooth.
Add the mustard powder and season, to taste, with salt and freshly ground black pepper.
Stir in the cheeses until melted.
For the topping, in a large bowl, mix together all of the topping ingredients until well combined. Set aside.
Spoon the mashed potatoes and leeks into the bottom of the prepared ovenproof dish, pressing down to form an even layer.
Arrange the slices of boiled egg on top in a circular pattern, overlapping them until all of the mashed potatoes are covered.
Season, to taste, with salt and freshly ground black pepper, then pour over the cheese sauce. Sprinkle over the topping.
Place the dish onto a baking tray to catch any drips. Transfer the dish to the oven and bake for 20-25 minutes, or until the topping is crisp and golden-brown and the sauce is bubbling.
Meanwhile, heat a little oil in a non-stick frying pan over a medium to high heat. Add the bacon chops and fry for 3-4 minutes on each side, or until crisp on both sides and completely cooked through (the juices will run clear when the meat is pierced in the thickest part with a skewer).
To serve, spoon a portion of the Anglesey eggs into the centre of four serving plates. Place one bacon chop alongside each. Garnish with the flatleaf parsley.


Below are outlined some of the ways to perform typical database-related tasks with peewee.
Examples will use the following models:
import peewee
class Blog(peewee.Model):
creator = peewee.CharField()
name = peewee.CharField()
class Entry(peewee.Model):
blog = peewee.ForeignKeyField(Blog)
title = peewee.CharField()
body = peewee.TextField()
pub_date = peewee.DateTimeField()
published = peewee.BooleanField(default=True)
While it is not necessary to explicitly connect to the database before using it, managing connections explicitly is a good practice. This way if the connection fails, the exception can be caught during the “connect” step, rather than some arbitrary time later when a query is executed.
>>> database = SqliteDatabase('stats.db')
>>> database.connect()
To use this database with your models, specify it in an inner “Meta” class:
class MyModel(Model):
some_field = CharField()
class Meta:
database = database
It is possible to use multiple databases (provided that you don’t try and mix models from each):
>>> custom_db = peewee.SqliteDatabase('custom.db')
>>> class CustomModel(peewee.Model):
... whatev = peewee.CharField()
...
... class Meta:
... database = custom_db
...
>>> custom_db.connect()
>>> CustomModel.create_table()
Best practice: define a base model class that points at the database object you wish to use, and then all your models will extend it:
custom_db = peewee.SqliteDatabase('custom.db')
class CustomModel(peewee.Model):
class Meta:
database = custom_db
class Blog(CustomModel):
creator = peewee.CharField()
name = peewee.TextField()
class Entry(CustomModel):
# etc, etc
Note
Remember to specify a database in a model class (or its parent class), otherwise peewee will fall back to a default sqlite database named “peewee.db”.
Point models at an instance of PostgresqlDatabase.
psql_db = peewee.PostgresqlDatabase('my_database', user='code')
class PostgresqlModel(peewee.Model):
"""A base model that will use our MySQL database"""
class Meta:
database = psql_db
class Blog(PostgresqlModel):
creator = peewee.CharField()
# etc, etc
Point models at an instance of MySQLDatabase.
mysql_db = peewee.MySQLDatabase('my_database', user='code')
class MySQLModel(peewee.Model):
"""A base model that will use our MySQL database"""
class Meta:
database = mysql_db
class Blog(MySQLModel):
creator = peewee.CharField()
# etc, etc
# when you're ready to start querying, remember to connect
mysql_db.connect()
Some database engines may not allow a connection to be shared across threads, notably sqlite. If you would like peewee to maintain a single connection per-thread, instantiate your database with threadlocals=True:
concurrent_db = SqliteDatabase('stats.db', threadlocals=True)
The above implementation stores connection state in a thread local and will only use that connection for a given thread. Sqlite can also share a connection across threads natively, so if you would prefer to use sqlite’s native support:
native_concurrent_db = SqliteDatabase('stats.db', check_same_thread=False)
Sometimes the database information is not known until run-time, when it might be loaded from a configuration file/etc. In this case, you can “defer” the initialization of the database by passing in None as the database_name.
deferred_db = peewee.SqliteDatabase(None)
class SomeModel(peewee.Model):
class Meta:
database = deferred_db
If you try to connect or issue any queries while your database is uninitialized you will get an exception:
>>> deferred_db.connect()
Exception: Error, database not properly initialized before opening connection
To initialize your database, you simply call the init method with the database_name and any additional kwargs:
database_name = raw_input('What is the name of the db? ')
deferred_db.init(database_name)
You can use the Model.create() method on the model:
>>> Blog.create(creator='Charlie', name='My Blog')
<__main__.Blog object at 0x2529350>
This will INSERT a new row into the database. The primary key will automatically be retrieved and stored on the model instance.
Alternatively, you can build up a model instance programmatically and then save it:
>>> blog = Blog()
>>> blog.creator = 'Chuck'
>>> blog.name = 'Another blog'
>>> blog.save()
>>> blog.id
2
Once a model instance has a primary key, any attempt to re-save it will result in an UPDATE rather than another INSERT:
>>> blog.save()
>>> blog.id
2
>>> blog.save()
>>> blog.id
2
If you want to update multiple records, issue an UPDATE query. The following example will update all Entry objects, marking them as “published”, if their pub_date is less than today’s date.
>>> update_query = Entry.update(published=True).where(pub_date__lt=datetime.today())
>>> update_query.execute()
4 # <--- number of rows updated
For more information, see the documentation on UpdateQuery.
To delete a single model instance, you can use the Model.delete_instance() shortcut:
>>> blog = Blog.get(id=1)
>>> blog.delete_instance()
1 # <--- number of rows deleted
>>> Blog.get(id=1)
BlogDoesNotExist: instance matching query does not exist:
SQL: SELECT "id", "creator", "name" FROM "blog" WHERE "id" = ? LIMIT 1
PARAMS: [1]
To delete an arbitrary group of records, you can issue a DELETE query. The following will delete all Entry objects that are a year old.
>>> delete_query = Entry.delete().where(pub_date__lt=one_year_ago)
>>> delete_query.execute()
7 # <--- number of entries deleted
For more information, see the documentation on DeleteQuery.
You can use the Model.get() method to retrieve a single instance matching the given query (passed in as a mix of Q objects and keyword arguments).
This method is a shortcut that calls Model.select() with the given query, but limits the result set to 1. Additionally, if no model matches the given query, a DoesNotExist exception will be raised.
>>> Blog.get(id=1)
<__main__.Blog object at 0x25294d0>
>>> Blog.get(id=1).name
u'My Blog'
>>> Blog.get(creator='Chuck')
<__main__.Blog object at 0x2529410>
>>> Blog.get(id=1000)
BlogDoesNotExist: instance matching query does not exist:
SQL: SELECT "id", "creator", "name" FROM "blog" WHERE "id" = ? LIMIT 1
PARAMS: [1000]
For more information see notes on SelectQuery and Querying API in general.
To simply get all instances in a table, call the Model.select() method:
>>> for blog in Blog.select():
... print blog.name
...
My Blog
Another blog
When you iterate over a SelectQuery, it will automatically execute it and start returning results from the database cursor. Subsequent iterations of the same query will not hit the database as the results are cached.
Another useful note is that you can retrieve instances related by ForeignKeyField by iterating. To get all the related instances for an object, you can query the related name. Looking at the example models, we have Blogs and Entries. Entry has a foreign key to Blog, meaning that any given blog may have 0..n entries. A blog’s related entries are exposed using a SelectQuery, and can be iterated the same as any other SelectQuery:
>>> for entry in blog.entry_set:
... print entry.title
...
entry 1
entry 2
entry 3
entry 4
The entry_set attribute is just another select query and any methods available to SelectQuery are available:
>>> for entry in blog.entry_set.order_by(('pub_date', 'desc')):
... print entry.title
...
entry 4
entry 3
entry 2
entry 1
>>> for entry in Entry.select().where(blog=blog, published=True):
... print '%s: %s (%s)' % (entry.blog.name, entry.title, entry.published)
...
My Blog: Some Entry (True)
My Blog: Another Entry (True)
>>> for entry in Entry.select().where(pub_date__lt=datetime.datetime(2011, 1, 1)):
... print entry.title, entry.pub_date
...
Old entry 2010-01-01 00:00:00
You can also filter across joins:
>>> for entry in Entry.select().join(Blog).where(name='My Blog'):
... print entry.title
Old entry
Some Entry
Another Entry
If you are already familiar with Django’s ORM, you can use the “double underscore” syntax:
>>> for entry in Entry.filter(blog__name='My Blog'):
... print entry.title
Old entry
Some Entry
Another Entry
If you prefer, you can use python operators to query:
>>> for entry in Entry.select().join(Blog).where(Blog.name=='My Blog')
... print entry.title
To perform OR lookups, use the special Q object. These work in both calls to filter() and where():
>>> User.filter(Q(staff=True) | Q(superuser=True)) # get staff or superusers
To perform lookups against another column in a given row, use the F object:
>>> Employee.filter(salary__lt=F('desired_salary'))
Check the docs for more examples of querying.
>>> for e in Entry.select().order_by('pub_date'):
... print e.pub_date
...
2010-01-01 00:00:00
2011-06-07 14:08:48
2011-06-07 14:12:57
>>> for e in Entry.select().order_by(peewee.desc('pub_date')):
... print e.pub_date
...
2011-06-07 14:12:57
2011-06-07 14:08:48
2010-01-01 00:00:00
You can also order across joins. Assuming you want to order entries by the name of the blog, then by pubdate desc:
>>> qry = Entry.select().join(Blog).order_by(
... (Blog, 'name'),
... (Entry, 'pub_date', 'DESC'),
... )
>>> qry.sql()
('SELECT t1.* FROM entry AS t1 INNER JOIN blog AS t2 ON t1.blog_id = t2.id ORDER BY t2.name ASC, t1.pub_date DESC', [])
The paginate method makes it easy to grab a “page” or records – it takes two parameters, page_number, and items_per_page:
>>> for entry in Entry.select().order_by('id').paginate(2, 10):
... print entry.title
...
entry 10
entry 11
entry 12
entry 13
entry 14
entry 15
entry 16
entry 17
entry 18
entry 19
You can count the number of rows in any select query:
>>> Entry.select().count()
100
>>> Entry.select().where(id__gt=50).count()
50
To limit the amount of memory used by peewee when iterating over a lot of rows (i.e. you may be dumping data to csv), use the iterator() method on the QueryResultWrapper. This method allows you to iterate without caching each model returned, using much less memory when iterating over large result sets:
# let's assume we've got 1M stat objects to dump to csv
stats_qr = Stat.select().execute()
# our imaginary serializer class
serializer = CSVSerializer()
# loop over all the stats and serialize
for stat in stats_qr.iterator():
serializer.serialize_object(stat)
For simple queries you can see further speed improvements by using the SelectQuery.naive() query method. See the documentation for details on this optimization.
stats_query = Stat.select().naive() # note we are calling "naive()"
stats_qr = stats_query.execute()
for stat in stats_qr.iterator():
serializer.serialize_object(stat)
Use the special F object to perform an atomic update:
>>> MessageCount.update(count=F('count') + 1).where(user=some_user)
Suppose you have some blogs and want to get a list of them along with the count of entries in each. First I will show you the shortcut:
query = Blog.select().annotate(Entry)
This is equivalent to the following:
query = Blog.select({
Blog: ['*'],
Entry: [Count('id')],
}).group_by(Blog).join(Entry)
The resulting query will return Blog objects with all their normal attributes plus an additional attribute ‘count’ which will contain the number of entries. By default it uses an inner join if the foreign key is not nullable, which means blogs without entries won’t appear in the list. To remedy this, manually specify the type of join to include blogs with 0 entries:
query = Blog.select().join(Entry, 'left outer').annotate(Entry)
You can also specify a custom aggregator:
query = Blog.select().annotate(Entry, peewee.Max('pub_date', 'max_pub_date'))
Let’s assume you have a tagging application and want to find tags that have a certain number of related objects. For this example we’ll use some different models in a Many-To-Many configuration:
class Photo(Model):
image = CharField()
class Tag(Model):
name = CharField()
class PhotoTag(Model):
photo = ForeignKeyField(Photo)
tag = ForeignKeyField(Tag)
Now say we want to find tags that have at least 5 photos associated with them:
>>> Tag.select().join(PhotoTag).join(Photo).group_by(Tag).having('count(*) > 5').sql()
SELECT t1."id", t1."name"
FROM "tag" AS t1
INNER JOIN "phototag" AS t2
ON t1."id" = t2."tag_id"
INNER JOIN "photo" AS t3
ON t2."photo_id" = t3."id"
GROUP BY
t1."id", t1."name"
HAVING count(*) > 5
Suppose we want to grab the associated count and store it on the tag:
>>> Tag.select({
... Tag: ['*'],
... Photo: [Count('id', 'count')]
... }).join(PhotoTag).join(Photo).group_by(Tag).having('count(*) > 5').sql()
SELECT t1."id", t1."name", COUNT(t3."id") AS count
FROM "tag" AS t1
INNER JOIN "phototag" AS t2
ON t1."id" = t2."tag_id"
INNER JOIN "photo" AS t3
ON t2."photo_id" = t3."id"
GROUP BY
t1."id", t1."name"
HAVING count(*) > 5
Suppose you need to want to get a list of all users whose username begins with “a”. There are a couple ways to do this, but one method might be to use some SQL functions like LOWER and SUBSTR. To use arbitrary SQL functions, use the special R object to construct queries:
# select the users' id, username and the first letter of their username, lower-cased
query = User.select(['id', 'username', R('LOWER(SUBSTR(username, 1, 1))', 'first_letter')])
# now filter this list to include only users whose username begins with "a"
a_users = query.where(R('first_letter=%s', 'a'))
>>> for user in a_users:
... print user.first_letter, user.username
This same functionality could be easily exposed as part of the where clause, the only difference being that the first letter is not selected and therefore not an attribute of the model instance:
a_users = User.filter(R('LOWER(SUBSTR(username, 1, 1)) = %s', 'a'))
We can write subqueries as part of a SelectQuery, for example counting the number of entries on a blog:
entry_query = R('(SELECT COUNT(*) FROM entry WHERE entry.blog_id=blog.id)', 'entry_count')
blogs = Blog.select(['id', 'name', entry_query]).order_by(('entry_count', 'desc'))
for blog in blogs:
print blog.title, blog.entry_count
It is also possible to use subqueries as part of a where clause, for example finding blogs that have no entries:
no_entry_query = R('NOT EXISTS (SELECT * FROM entry WHERE entry.blog_id=blog.id)')
blogs = Blog.filter(no_entry_query)
for blog in blogs:
print blog.name, ' has no entries'
You can execute queries within a transaction using the transaction context manager, which will issue a commit if all goes well, or a rollback if an exception is raised:
db = SqliteDatabase(':memory:')
with db.transaction():
blog.delete_instance(recursive=True) # delete blog and associated entries
Similar to the context manager, you can decorate functions with the commit_on_success decorator:
db = SqliteDatabase(':memory:')
@db.commit_on_success
def delete_blog(blog):
blog.delete_instance(recursive=True)
By default, databases are initialized with autocommit=True, you can turn this on and off at runtime if you like. The behavior below is roughly the same as the context manager and decorator:
db.set_autocommit(False)
try:
blog.delete_instance(recursive=True)
except:
db.rollback()
raise
else:
db.commit()
finally:
db.set_autocommit(True)
If you would like to manually control every transaction, simply turn autocommit off when instantiating your database:
db = SqliteDatabase(':memory:', autocommit=False)
Blog.create(name='foo blog')
db.commit()
If you’d like to generate some models for an existing database, you can try out the database introspection tool “pwiz” that comes with peewee.
Usage:
python pwiz.py my_postgresql_database
It works with postgresql, mysql and sqlite:
python pwiz.py test.db --engine=sqlite
pwiz will generate code for:
The generated code is written to stdout.


{% block title %}Search results for {{ search_term|safe }}{% endblock%}
Search results for "{{search_term}}"
Total Products Found: {{ total }}
{% if scrapes_1 or scrapes_2 or scrapes_3 or scrapes_4 or scrapes_5 or scrapes_6 or scrapes_7 or scrapes_8 %}
{% include 'partials/_search-card.html' %}
{% else %}


--- layout: default title: Plans permalink: /plans tagline: Get a website by selecting a monthly plan below og_image: website-development stylesheet: services --- {% include intro-section.html %}
Get your business online
Publish your own content
Sell products and accept payments


--- layout: post title: ! 'Community Highlights: Yehuda Katz' categories: - activism author: Gregg Pollack published: true date: 2009-07-30 03:16:00.000000000 +01:00 ---
Over the past few months, Rails core team member Yehuda Katz has posted a series of great blog articles describing some of the process and technique he’s used while coding Rails 3 with Carl Lerche. In case you haven’t followed his blog posts, I thought I’d repost them here for your educational reading.
Rails 3: The Great Decoupling
is about decoupling components like ActionController and ActionView.
New Rails Isolation Testing
is about the creation of a new test mixin that runs each test case in its own process.
6 Steps to Refactoring Rails
is about the refactoring philosophy he’s using when coding Rails 3.
Rails Edge Architecture
is about Rails 3 Architecture including AbstractController::Base & ActionController::Http
Better Module Organization
is about cleaning up the way modules are included.
alias_method_chain in models
is about alternatives to using alias_method_chain, some of which made it into Rails 3 refactorings.
Rails 3 Extension API
is where on the new wiki Yehuda has started documenting the new extension APIs which are being added for Rails 3. There’s not a whole lot there yet, but be sure to watch this space in the coming weeks.


This construct defines a region of the program that is to be compiled into a sequence of kernels for execution on the accelerator device.
#pragma kernels [clause [[,] clause]…] new-line { structured
block }
!$acc kernels [clause [[,] clause]…]
structured block
!$acc end kernels


#include <stSlimTree.h>
Classes | |
| struct | tCluster |
Public Types | |
| typedef stSlimLogicNode< ObjectType, EvaluatorType > | tLogicNode |
Public Member Functions | |
| stSlimMSTSplitter (tLogicNode *node) | |
| ~stSlimMSTSplitter () | |
| const tLogicNode * | GetLogicNode () |
| int | Distribute (stSlimIndexNode *node0, ObjectType *&rep0, stSlimIndexNode *node1, ObjectType *&rep1, EvaluatorType *metricEvaluator) |
| int | Distribute (stSlimLeafNode *node0, ObjectType *&rep0, stSlimLeafNode *node1, ObjectType *&rep1, EvaluatorType *metricEvaluator) |
Protected Types | |
| enum | tClusterState { ALIVE, DEAD, DEATH_SENTENCE } |
| typedef stGenericMatrix< stDistance > | tDistanceMatrix |
Protected Member Functions | |
| int | FindCenter (int clus) |
| int | BuildDistanceMatrix (EvaluatorType *metricEvaluator) |
| void | PerformMST () |
| void | JoinClusters (int cluster1, int cluster2) |
Protected Attributes | |
| tLogicNode * | Node |
| tDistanceMatrix | DMat |
| struct tCluster * | Cluster |
| int * | ObjectCluster |
| int | N |
| int | Cluster0 |
| int | Cluster1 |
This class template implements the SlimTree MST split algorithm.
Documentation review.
Tests!
Definition at line 504 of file stSlimTree.h.
|
protected |
Distance matrix type.
Definition at line 548 of file stSlimTree.h.
| typedef stSlimLogicNode< ObjectType, EvaluatorType > stSlimMSTSplitter< ObjectType, EvaluatorType >::tLogicNode |
This type defines the logic node for this class.
Definition at line 509 of file stSlimTree.h.
|
protected |
| stSlimMSTSplitter< ObjectType, EvaluatorType >::stSlimMSTSplitter | ( | tLogicNode * | node | ) |
Builds a new instance of this class. It will claim the ownership of the logic node provided as input.
| stSlimMSTSplitter< ObjectType, EvaluatorType >::~stSlimMSTSplitter | ( | ) |
Disposes all associated resources.
|
protected |
Builds the distance matrix using the given metric evaluator.
| metricEvaluator | The metric evaluator. |
| int stSlimMSTSplitter< ObjectType, EvaluatorType >::Distribute | ( | stSlimIndexNode * | node0, |
| ObjectType *& | rep0, | ||
| stSlimIndexNode * | node1, | ||
| ObjectType *& | rep1, | ||
| EvaluatorType * | metricEvaluator | ||
| ) |
Distributes objects between 2 index nodes.
| int stSlimMSTSplitter< ObjectType, EvaluatorType >::Distribute | ( | stSlimLeafNode * | node0, |
| ObjectType *& | rep0, | ||
| stSlimLeafNode * | node1, | ||
| ObjectType *& | rep1, | ||
| EvaluatorType * | metricEvaluator | ||
| ) |
Distributes objects between 2 leaf nodes.
|
protected |
Returns the center of the object in the cluster clus. The.
| clus | Cluster id. |
|
inline |
Provides access to the logic node.
Definition at line 525 of file stSlimTree.h.
References stSlimMSTSplitter< ObjectType, EvaluatorType >::Node.
|
protected |
Joins 2 clusters. This method will insert custer2 into cluster1.
The state of cluster2 will change to DEATH_SENTENCE.
| cluster1 | Cluster 1. |
| cluster2 | Cluster 2. |
|
protected |
Performs the MST algorithm. This method will split the objects in 2 clusters. The result of the processing will be found at the array Cluster.
|
protected |
All clusters.
Definition at line 602 of file stSlimTree.h.
|
protected |
Name of the cluster 0.
Definition at line 617 of file stSlimTree.h.
|
protected |
Name of the cluster 1.
Definition at line 622 of file stSlimTree.h.
|
protected |
The distance matrix.
Definition at line 597 of file stSlimTree.h.
|
protected |
Total number of objects.
Definition at line 612 of file stSlimTree.h.
|
protected |
The logic node to be used as source.
Definition at line 592 of file stSlimTree.h.
Referenced by stSlimMSTSplitter< ObjectType, EvaluatorType >::GetLogicNode().
|
protected |
The names of the cluster of each object
Definition at line 607 of file stSlimTree.h.


[gecode-users] FloatVar implementation
Filip Konvička
filip.konvicka at logis.cz
Wed Apr 9 15:52:45 CEST 2008
> Hmm, I am not so sure that Guido is telling the truth here ;-) I just had a
> chat with him and he agrees ;-)
>
I hope I didn't start any fight :-)
> The kernel only knows about a base class that contains no information
> whatsoever about how the domain of a variable is implemented.
>
> What is perfectly possible is to have variables of different types in the
> system that share the same baseclass: the kernel is only concerned with what
> is in the baseclass and not with how the variables are implemented. The key
> aspect is that variable implementations are only refered to by propagators
> and branchings: they statically know the real type so that the right
> operations are performed, so no need for virtual etc.
>
> Look to the generated baseclass IntVarImpBase and how IntVarImp inherits
> from it: IntVarImpBase is what the kernel deals with, while IntVarImp is the
> real implementation of integer variables.
>
I've seen that, but what I need to understand is why you need distinct
*VarImpBases for Int, Bool, etc., why not just one. The difference I've
seen in kernel is that the classes report their own "event id" or
whatever, so I guess you use this for speeding up actor filtering.
Again, I'm just guessing and will need to look into this.
[...some time passes...]
OK, we've added float.vis and recompiled Gecode, so now it seems we have
our own bunch of ModEvents and also a FloatVarImpBase, which is cool, so
we'll come back with a status report (or more questions) soon :-)
> Having said all that, I would first attempt to get everything up for a
> single implementation.  Even if it takes 5 minutes to compile... :-))
Cheers,
Filip
More information about the gecode-users
mailing list


Plugin Information |
|---|
View Tracking SVN on the plugin site for more information. |
Lets one project track the Subversion revisions that are checked out for another project.
More than one module may be used, and the revision tracking will happen for any URL in this project that exactly matches the tracked project. For all other modules, the latest revision will be used.
When building, the TRACKING_SVN_BUILD environment variable will contain the URL of the tracked build.
It is generally useless to use SCM polling if SVN tracking is enabled. Instead, trigger this project from the tracked project.
Change Log
Version 1.1 (Jan 3, 2011)
- Fixed env var
- Link to tracked project on main page
- Add option to ignore certain URLs from tracking
Version 1.0 (Nov 28, 2009)
- Initial release


Work center
Object status
Structure name o API table extension
Data part of BAPI extension parameter
Data part of BAPI extension parameter
Data part of BAPI extension parameter
Data part of BAPI extension parameter
Structure name o API table extension
Data part of BAPI extension parameter
Data part of BAPI extension parameter
Data part of BAPI extension parameter
Data part of BAPI extension parameter
Object Type
Object key
Logical system
Notification No
Item Number in Item Record
Sequential Number for Cause
Sequential Task Number
Sequential Task Number
Object number
Partner Function
Counter for differentiation 6-digit
Partner
Object key
Consecutive Number of Activity
Sort Number for Activity
Activity Text
Code Group - Activities
Activity Code
Start Date
Start Time of Activity
End Date
End Time of Activity
Sort Number for Item
Object key
Sequential Number for Cause
Sort Number for Cause
Item Number in Item Record
Cause Text
Code Group - Causes
Cause Code
Sort Number for Item
Object key
Partner Function
Partner Function
Partner
Partner
Object key
Sequential Task Number
Sort Number for Task
Short Text for Task
Code Group - Tasks
Task Code
Function of Person Responsible for Task
Person Responsible for Task (Partner Number)
Planned Start Date
Planned Start Time for Task
Planned finish date
Planned Finish Time for Task
Person Who Completed the Task
Sort Number for Item
Object key
Item Number in Item Record
Sort Number for Item
Notification Item Short Text
Code Group - Problem
Problem or Damage Code
Code Group - Object Parts
Part of Object
Assembly
Long Material Number for Field ASSEMBLY
External GUID for ASSEMBLY Field
Version Number for ASSEMBLY Field
Start Point
End Point
Length
Unit of Measurement for Linear Data
Unit of Measurement for Linear Data in ISO Code
Type of First Offset
Value of Offset 1
Unit of Measurement for Offset 1
Unit of Measurement for Offset 1 in ISO Code
Type of Second Offset
Value of Offset 2
Unit of Measurement for Offset 2
Unit of Measurement for Offset 2 in ISO Code
Equipment Number
Functional Location
Marker for Start Point
Distance between Marker and Start Point
Marker for End Point
Length Spec for Distance between Marker and End Point
Unit for the Distance from Marker
Unit for the Distance from Marker in ISO Code
Assembly
Work center
Object status
UTC Time Stamp in Short Form (YYYYMMDDhhmmss)
Time Zone
User Name in User Master Record
Text Line
Text for Note
Message type: S Success, E Error, W Warning, I Info, A Abort
Message Class
Message Number
Message Text
Application log: log number
Application log: Internal message serial number
Message Variable
Message Variable
Message Variable
Message Variable
Parameter Name
Lines in parameter
Field in parameter
Logical system from which message originates
System status
Individual status of an object (short form)
Object status


Status : {{xlData.docfile.name}} has been uploaded!!
+++++++++++++++++====================+++++++++++++++++++
PHASE 1 : Classification
{% else %}No file uploaded to get data for :/
{% endif %}
Click here to know the genre.
[NOTE : it might take around 3-5 mins]
+++++++++++++++++====================+++++++++++++++++++
PHASE 2 : Conversion
+++++++++++++++++====================+++++++++++++++++++


Benjamin Netanyahu describes corruption indictment as 'attempted coup' – video
The Israeli prime minister said he would not resign despite being charged with bribery, fraud and breach of trust in a corruption scandal that he denounced as an 'attempted coup'. Netanyahu was indicted for bribery, fraud and breach of trust on 21 November – the first time a sitting Israeli prime minister has been charged with a crime


multi_state_year.Rmd
Sometimes one may want to collect data from multiple states over multiple years. To do this we recommend loading the following libraries:
Then we can create the different combinations of year and state in order to submit to tidyqwi.
year <- c("2008", "2009", "2010","2011")
state <- c("01","02","04","05","06","08","09","10","11","12",
"13","15","16","17","18","19","20","21","22","23",
"24","25","26","27","28","29","30","31","32","33",
"34","35","36","37","38","39","40","41","42","44",
"45","46","47","48","49","50","51","53","54","55",
"56")
argList <- list(x = state, y = year)
arguments <- cross_df(argList)
Using the multiple processing function we can submit the following:
plan("multiprocess")
qwi_data <- map2(arguments$x, arguments$y, ~
get_qwi(
states = .x,
years = .y ,
industry_level = "2",
all_groups = FALSE,
endpoint = "se",
geography = "cbsa",
processing = "multiprocess",
apikey = APIkey))
After this function returns out values we can collapse these data into a single data set.
And then add the labels for our variables if desired.


WITNESS LIST S/C on Emerging Technologies & Economic Dev. Committee April 25, 2005 - 3:00P SB 1530 FOR: Kordzik, Kelly (Texas Nanotechnology Initiative), Austin, TX Masterson, Jr., Conrad (Nanotechnology Foundation of Texas), Houston, TX ON: Butler, Bruce (Univ. of Texas Health Science Center), Houston, TX Capelli, Christoper M.D. (UT M.D. Anderson Cancer Center), Houston, TX Registering, but not testifying: For: Dennis, Hilary (Texas Medical Association), Austin, TX Kowalski, Tom (Texas Healthcare and Bioscience Institute), Austin, TX


They shake a leg at picnic to land in trouble
Video clip of Revenue staff’s dance to songs praising Jagan goes viral
by Staff ReporterA video clip showing some revenue officials in the district dancing to the numbers composed praising Chief Minister Y.S. Jagan Mohan Reddy goes viral and becomes a hot topic in the political circles.
Bhamini Tahasildar N. Narasimha Murthy, surveyor Thota Rammohana Rao, and Revenue Inspector Biddika Krishna Rao allegedly took part in the picnic organised by leaders of the YSRCP’s Pathapatnam constituency at Bhamini on Sunday.
The officials, allegedly in an inebriated condition, danced for the songs such as ‘Rayalaseema Muddubidda..Jagananna...Nee Venake Jana Prabhanjanamanna’ which were popular during the padayatra of Mr. Jagan Mohan Reddy.
Srikakulam Collector J. Nivas reportedly directed Palakonda Revenue Divisional Officer R.T.G.S. Kumar to inquire into the incident as it was against the service rules. They reportedly gave explanation to the Revenue Divisional Officer after they were summoned over the issue.
Mr. Narasimha Murthy, who earlier attended the weekly grievances programme, did not react, when some newsmen raised the issue. He, however, told the higher officials that he was invited to the picnic and could not skip it due to local political pressure.
Stating that he was not aware of the songs, Mr. Murthy said he shared his joy with locals to have cordial relations.


Design Challenges
21 August 2018
Challenge 1: Golf Ball Drop
In this challenge, we were given twelve straws and some tape and instructed to create a contraption that would catch and hold a golf ball
dropped into it from a height of six feet. In our group, we created a cradle-like structure, utilizing the bending point of the straw as
the bottom of the cradle. Since only three straws were permitted to touch the ground at once, we created legs on two sides and cut a straw
in half lengthwise to run along the bottom such that only three straws would be touching. Our greatest challenge was the limited amount of
tape, but we managed to overcome this by holding the straws together with the smallest possible bits of tape.
Our design was not particularly effective, but it was probably one of the more effective designs in the class. When the golf ball was
dropped into our cradle, it took a moment or two to bounce or roll out, but on each of the three attempts, the golf ball did not ever
remain in the cradle.
I was active throughout the entire design process, and I would say that the work done by my team during this challenge was truly
a group effort. We all contributed to all the ideas of how to construct our contraption, and everyone was active in the building
phase, as well. Personally, I think I was slightly more active in building than in designing, but I definitely contributed to the
design, too.
Challenge 2: Newspaper Towers
In this challenge, we were given a single issue of a newspaper and some tape and told to build the highest tower possible. Our team
chose to build a tower out of interlocked rolled sheets, with a base made up of a cone shape. This design was especially useful
because it required an exceptionally minimal amount of tape. Our only major challenge in this endeavor was determining when to stop
building and finding the balance between height and stability.
Our tower was the tallest in the class, with a final measurement of 88 inches tall. Our structure was very effective because of the
strength of rolled cylinders and the structural integrity that comes with having a wide base. It helped that we rolled our newspaper
diagonally in order to increase the height added with each new roll of paper, and thus decreased the amount of weight added to the
top of our tower.It also helped that we made each consecutive roll a slightly different thickness such that we could insert each new
roll into the previous one. This decreased our tape usage and added to the stability of our tower.
I was active through the entire process. This challenge was great for my team because we worked very well together and hardly had to
discuss our plans for our structure. After a few minutes of discussion at the beginning, we were all on the same page as far as what
needed done, and so we all worked together to build our very beautiful, very tall tower. The only thing my team really had any
disagreement on was towards the end when we debated whether to add more rolls to the top of our tower. I wanted to add more because
I believed our tower was strong enough to take it, but my teammates did not want to take the risk. Eventually, we compromised and
topped our tower off with a half roll.


--- layout: post title: BSkyB in Vodafone mobile TV deal date: 2005-11-01 00:00:00 categories: digital-media mobile ---
BSkyB in Vodafone mobile TV deal: "Sky Mobile TV will allow Vodafone customers to view some of BSkyB's TV content on their third generation (3G) handsets from Tuesday."
And apparently "This is not a trial on a very limited number of phones or a service available on a single make of mobile, but something that all our Vodafone live! with 3G customers can experience."
£5 a month sounds very low. I wonder how the networks will manage to support large numbers of viewers simultaneously - or have they gotten their act together with multicast now?


Graphs
Daily calorie intake Time spend on schoolSort by
Days of the week Calories high to lowFilter on feelings
About Me
I’m Chanel, I love food.... I mean I really love food. I want to lose some weight but I’ve got a sweet tooth. So in the past six weeks I’ve been documenting I few things like calories, expenses, feelings and time spend on school. I’m going to compare them, make some conclusions and hopefully... I will stop eating so much. Enjoy!
Scroll to explore
I eat the most on fridays
Or at least, it looks that way. Friday is competing with monday.
Try to sort by calories. Go to my expenses!


I'm Kev Lai.. I am a web developer which means I play with spider webs as my occupation. I like spider webs, but I'm scared of spiders. I'm still waiting for the day I win the lottery. No luck so far, so I'm still trying to live with spiders. Shit, SPIDERS!
I broke the ten-year benchmark and still hung out with my Sing Yin homies to celebrate Mid-Autumn Festival, making C4s and burning candles and stuff. I'm sure and I strongly hope these guys will be best friends for life. If you're reading this and are on that picture, bruh, I love you.
 After extensive collaboration with other world class experts like John Cheung and Yoyo Wong, and others who were generous enough to contribute their effort, I earned my reputation as a watter bottle guardian, which I'm still wearing like a badge. And between those four years, Sui Fool only almost wanted to beat me up, TWICE.
After constantly getting high on cough syrup when I was a kid, I suddenly had a vision of how people should live when they lived in space.  I had a hunch and scored the world champion and artist merit of NASA's Space Settlement Design Contest in 2009 and managed to put my name on every newspaper in Hong Kong for one day. But what I'm mostly proud of is Yahoo! showed my name on their home page when people still searched things on Yahoo! You don't get to do this anymore guys.


Please configure the CC.xml URLs, which project will be monitored and how often should the monitor refresh.


Usage examples
Map projects by name
Map harbor project to several namespaces. This will create a robot account in my-project
harbor project and sync the credentials into team-a
and team-b
’s namespace as secret central-project-token
.
kind: HarborSync
metadata:
name: my-project
spec:
type: Regex
name: "my-project" # <--- specify harbor project
robotAccountSuffix: "k8s-sync-robot" # <--- should be unique per kind: HarborSync
mapping:
- type: Translate
namespace: "team-a" # <--- target namespace
secret: "my-project-pull-token" # <--- target secret name
- type: Translate
namespace: "team-b"
secret: "my-project-pull-token"
Map projects using a regular expression
You can specify regular expressions to map a large number of projects to namespaces. This maps harbor teams with the prefix team-
. E.g. Harbor project team-frontend
maps to k8s namespace team-frontend
. The secret’s name will always be my-pull-token
. Non-existent k8s namespaces will be ignored.
kind: HarborSync
metadata:
name: team-projects
spec:
type: Regex
name: "team-(.*)" # find harbor projects matching this expression
robotAccountSuffix: "k8s-sync-robot"
mapping:
- type: Translate
namespace: "team-$1" # references capturing group from the above projectSelector.name
secret: "team-$1-pull-token" # also here
Map projects using regular expressions #2
You have one harbor project and want to deploy the pull secrets into several namespaces matching a regular expression. E.g. pull tokens for the platform-team
project should be distributed into all namespaces matching team-.*
.
Use a type: Match
on a mapping to say: hey, find namespaces using this regular expression at the namespace field rather than re-using the project name using type: Translate
.
kind: HarborSync
metadata:
name: platform-team
spec:
type: Regex
name: "platform-team"
robotAccountSuffix: "k8s-sync-robot"
mapping:
- type: Match # treat namespace as regexp
namespace: "team-.*" # if ns matches this it will receive the secret
secret: "platform-pull-token" # you can still use the capturing group from projectSelector.Name here
Mapping Projects
A mapping
defines how to lookup namespaces in the cluster. Generally there are two lookup types: Translate
and Match
.
Translate
Translate will take the Harbor project name into account when looking up namespaces. The ProjectSelector.ProjectName
can be a regular expression which holds capturing groups. The idea is to inject those capturing groups when finding namespaces.
Example:
Harbor: we have two projects, team-frontend
and team-backend
. We select them using team-(.*)
in the ProjectSelector.ProjectName
. And map them to kubernetes namespaces squad-$1
. The $1
will be replaced with frontend
and backend
respectively. In the end each namespaces will have only it’s own secret: team-frontend
will only have the secret of Harbor project team-frontend
. Namespace team-backend
will only have the secret of Harbor project team-backend
.
Match
Match doesn’t care about the ProjectSelector.ProjectName
. It will just find namespaces in the cluster that match the regular expression.
Example 1:
Harbor: we have one project, team-platform
. By setting the field ProjectMapping.Namespace
to team-.*
we deploy the robot account secret to namespaces
Example 2:
Harbor: we have two projects, team-platform
and team-operations
. By setting ProjectMapping.Namespace
to team-.*
we deploy the robot accounts of both the platform
and operations
project into the namespace. To avoid naming conflicts on the secrets we set ProjectMapping.Secret
to $1-pull-token
. The result is: All namespaces matching team-.*
will have the secrets platform-pull-token
and operations-pull-token
.
Configuring Webhook Receiver
Webhooks can be configured to notify other services whenever a Robot account is being recreated or refreshed. A POST Request is sent for every Robot account in every Project that has been (re-)created.
Example HTTP Request:
POST / HTTP/1.1
Host: localhost:1938
User-Agent: Go-http-client/1.1
Content-Length: 77
Content-Type: application/json
Accept-Encoding: gzip
{
"project": "team-foo",
"credentials": {
"name": "robot$sync-bot",
"token":"1234"
}
}
HarborSync CRD configuration:
kind: HarborSync
metadata:
name: platform-team
spec:
type: Regex
name: "team-(.*)"
robotAccountSuffix: "k8s-sync-robot"
mapping: [] # mappings are optional!
# you can specify multiple webhooks
webhook:
- endpoint: http://example.com
The only thing you can configure right now is a target endpoint for the HTTP request. Feel free to contribute or open an issue if you need more functionality.


Apache Flink Kubernetes Operator 1.5.0 Release Announcement
May 17, 2023 - Gyula Fora (@GyulaFora)The Apache Flink community is excited to announce the release of Flink Kubernetes Operator 1.5.0! The release focuses on improvements to the job autoscaler that was introduced in the previous release and general operational hardening of the operator.
We encourage you to download the release and share your feedback with the community through the Flink mailing lists or JIRA! We hope you like the new release and we’d be eager to learn about your experience with it.
Autoscaler improvements #
Algorithm improvements and better scale down behaviour #
The release contains important improvements to the core autoscaling logic. This includes improved stability of scaling decisions (leading to less parallelism oscillations) and better handling of slow or idle streams.
There are also some fixes related to output ratio computation and propagation that greatly improves the autoscaler on more complex streaming pipelines.
This version also introduces new metrics for tracking the number of scaling decisions and scaling errors together with some more Kubernetes events to improve the observability of the system.
Improved default configuration #
We have simplified and improved some default autoscaler configs for a better out-of-the-box user experience.
Some notable changes:
kubernetes.operator.job.autoscaler.metrics.window: 5m -> 10m
kubernetes.operator.job.autoscaler.target.utilization.boundary: 0.1 -> 0.4
kubernetes.operator.job.autoscaler.scale-up.grace-period: 10m -> 1h
kubernetes.operator.job.autoscaler.history.max.count: 1 -> 3
kubernetes.operator.job.autoscaler.scaling.effectiveness.detection.enabled: true -> false
kubernetes.operator.job.autoscaler.catch-up.duration: 10m -> 5m
kubernetes.operator.job.autoscaler.restart.time: 5m -> 3m
CRD Changes #
Ephemeral storage support #
Stateful streaming jobs often rely on ephemeral storage to store the working state of the pipeline. Previously it was only possible to change the ephemeral storage size through the pod template mechanism. The 1.5.0 release adds a new field to the task and jobmanager specification that allows configuring this similarly to other resources:
spec:
...
taskManager:
resource:
memory: "2048m"
cpu: 8
ephemeralStorage: "10G"
Make sure you upgrade the CRD together with the operator deployment to be able to access this feature. For more details check the docs
General operations #
Fabric8 and JOSDK version bump #
The operator have been updated to use the latest Java Operator SDK and Fabric8 versions that contain important fixes for production environments.
Health probe and canary resources #
Previous operator versions already contained a rudimentary health probe to catch simple startup errors but did not have a good mechanism to catch errors that developed during the lifetime of the running operator.
The 1.5.0 version adds two significant improvements here:
- Improved health probe to detect informer errors after startup
- Introduce canary resources for detecting general operator problems
The new canary resource feature allows users to deploy special dummy resources (canaries) into selected namespaces. The operator health probe will then monitor that these resources are reconciled in a timely manner. This allows the operator health probe to catch any slowdowns, and other general reconciliation issues not covered otherwise.
Canary FlinkDeployment:
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
name: canary
labels:
"flink.apache.org/canary": "true"
Release Resources #
The source artifacts and helm chart are available on the Downloads page of the Flink website. You can easily try out the new features shipped in the official 1.5.0 release by adding the Helm chart to your own local registry:
$ helm repo add flink-kubernetes-operator-1.5.0 https://archive.apache.org/dist/flink/flink-kubernetes-operator-1.5.0/
$ helm install flink-kubernetes-operator flink-kubernetes-operator-1.5.0/flink-kubernetes-operator --set webhook.create=false
You can also find official Kubernetes Operator Docker images of the new version on Dockerhub.
For more details, check the updated documentation and the release notes. We encourage you to download the release and share your feedback with the community through the Flink mailing lists or JIRA.
List of Contributors #
Gyula Fora, Marton Balassi, Mate Czagany, Maximilian Michels, Rafał Boniecki, Rodrigo Meneses, Tamir Sagi, Xin Hao, Xin Li, Zhanghao Chen, Zhenqiu Huang, Daren Wong, Gaurav Miglani, Peter Vary, Tan Kim, yangjf2019


In June this year we organised the first Taming the BEAST workshop, surrounded by the Swiss Alps, in Engelberg, Switzerland. The idea of holding a BEAST 2 workshop has been brewing for a while, motivated by the need for a Bayesian phylogenetics workshop that is focused on BEAST 2 and facilitates exchanges between developers and (both current and future) BEAST 2 users. During the workshop we realised that the materials we created and assembled for the workshop represent a very useful resource and that we could reach many more users by making them available to the community.
Originally we intended to simply release the tutorials and lecture slides in a single GitHub repository. However, after some discussions, we decided to build a Taming the BEAST website. The website provides a user-friendly front-end that collects tutorials, lecture slides and more in one location. To start, we have collected all of the tutorials from this year's workshop on the website (a couple are still missing, but they will be uploaded soon). Behind the scenes each tutorial lives in its own GitHub repository, where it's bundled along with all of the supporting data files and scripts needed to run the tutorial. We've also bundled example XML and output files with each tutorial. Although everything is available on a GitHub repository, there are also links to all of the files on the website.
All tutorials are available as PDF files from the website. We have started converting some of the tutorials to Markdown, which can be more easily displayed in a web browser. So far, three tutorials have been converted:
We hope that with time we will be able to convert all tutorials to Markdown. Don't worry if you want to print or download tutorials - PDFs of tutorials will always be available! (At present there is no PDF tutorial for the Structured birth-death model, but this will be uploaded shortly).
Since tutorials are stored on GitHub, it is easy for anyone in the community to contribute to Taming the BEAST. Because each tutorial is stored in its own repository the website is robust to changes in any single repository. We hope that the community will play an active role in curating the tutorials, either by updating or correcting existing tutorials, or by contributing new tutorials. For instance, someone who has developed a new BEAST 2 package can add a tutorial for the package to Taming the BEAST, where it will be accessible in a central location, along with other BEAST 2 tutorials, making it easier for BEAST 2 users to become familiar with the package.
We hope that the website will serve as a central repository that collects BEAST 2 teaching resources in one central location. The resources on the website are not only useful for individuals who want to learn how to use BEAST2, but also as a teaching aid, for workshops or even regular university courses.


"Powerful case" for inquiry into Birmingham pub bombings - Prime Minister
"I will look very hard at what we can do," he tells campaigns
by Birmingham LivePRIME Minister Boris Johnson says there is “a powerful case” for a panel-led inquiry into the Birmingham pub bombings.
And he said he would be talking to West Midlands Mayor Andy Street about what could be done.
But, speaking on BBC Radio WM, he added: “I think we would need to look at the evidence to see if it justifies it.”
Mr Johnson said: “I will look very hard at what we can do.”
He understood the grief still felt in the city following the IRA attacks in 1974 which left 21 people dead and recognised the need for closure, he added.
Mr Street has been a long-standing supporter of the Justice4the21 campaign group and has discussed the issue of the pub bombings with the past three Home Secretaries.
Last week, at the 45th memorial service, he pledged to continue to press the matter with whoever becomes Home Secretary after the general election.
Mr Street had said: “Nothing we say or do can bring those innocent victims back, but we can still try to achieve justice for their families, and for a city that has many unanswered questions.
He added: “I have come to the conclusion that the time is right for a panel-led, open public inquiry into the pub bombings.”
In April, an inquest jury ruled that the 21 people killed in explosions at The Mulberry Bush and The Tavern In The Town on November 21, 1974 were unlawfully killed, following a direction from coroner Sir Peter Thornton QC.
A botched warning call by the IRA caused or contributed to the deaths of those who were murdered in the attacks.
The 11 members of the jury panel sat through almost six weeks of evidence and heard how two massive detonations caused what one witness described as “pure carnage”, ripping apart the packed pubs on the night of November 21, killing 21 and injuring 220 more.
It found no errors in the way police responded to the explosions on the balance of the evidence provided.
But the campaign group has continued to call for a public inquiry because it believes there is further evidence which was not aired at the inquest.
Labour leader Jeremy Corbyn has already said there must be an independent inquiry into the Birmingham pub bombings.
Speaking as he launched his general election manifesto in the city, he said: “There has to be an independent inquiry into this”.
The Justice4the21 campaigners said such support could be “a game changer” but added: “We want to see politicians delivering on their promises because we have had years of empty promises.”
After Mr Johnson’s comments, campaign spokeswoman Julie Hambleton said: “Mr Johnson speaks about if it is justified - how can it not be justified?
“The Birmingham pub bombings remains the biggest unsolved mass murder of the 20th Century.”
Six men, known as the Birmingham Six, were jailed in 1975 for the bomb attacks, but their wrongful convictions were quashed by the Court of Appeal in 1991.
No-one has been since brought to justice but West Midlands Police insist the pub bombings investigation has never closed.


1. Introduction
This document extends the concepts defined in [REPORTING] to enable a class of reports which are not tied to the lifetime of any particular document. This enables network errors to be reported on, even (or especially) in cases where a document could not be loaded.
Decoupling reports from documents implies two major differences from the document-centred reporting defined in [REPORTING]: First, configuration of reporting must be done at the origin level, rather than through document response headers. Second, the reports are queued and delivered by the user agent separately from document reports.
1.1. Guarantees
This specification aims to provide a best-effort report delivery system that executes out-of-band with website activity. The user agent will be able to do a better job prioritizing and scheduling delivery of reports, as it has an overview of cross-origin activity that individual websites do not, and can deliver reports based on error conditions that would prevent a website from loading in the first place.
The delivery is not, however, guaranteed in a strict sense. We spell out a reasonable set of retry rules in the algorithms below, but it’s quite possible for a report to be dropped on the floor if things go badly.
Reporting can generate a good deal of traffic, so we allow developers to set up groups of endpoints, using a failover and load-balancing mechanism inspired by the DNS SRV record. The user agent will do its best to deliver a particular report to at most one endpoint in a group. Endpoints can be assigned weights to distribute load, with each endpoint receiving a specified fraction of reporting traffic. Endpoints can be assigned priorities, allowing developers to set up fallback collectors that are only tried when uploads to primary collectors fail.
1.2. Examples
endpoint-1
":
{ "network_reporting_endpoints": { "group": "endpoint-1", "max_age": 10886400, "endpoints": [ { "url": "https://example.com/reports", "priority": 1 }, { "url": "https://backup.com/reports", "priority": 2 } ] } }
And the following headers, which direct NEL reports to that group:
NEL: { ..., "report-to": "endpoint-1" }
2. Concepts
2.1. Endpoint groups
An endpoint group is a set of network reporting endpoints that will be used together for backup and failover purposes.
Each endpoint group has a name
, which is an ASCII
string.
Each endpoint group has an endpoints
list, which is a list of network reporting endpoints.
Each endpoint group has a subdomains
flag, which is either "include
" or
"exclude
".
Each endpoint group has a ttl
representing the number of seconds the group
remains valid for an origin.
Each endpoint group has a creation
which is the timestamp at which the group was
added to an origin.
A endpoint group is expired if its creation
plus its ttl
represents a time in the past.
2.2. Network reporting endpoints
A network reporting endpoint is an endpoint, which is extended with these additional attributes:
Each network reporting endpoint has a priority
, which is a
non-negative integer.
Each network reporting endpoint has a weight
, which is a
non-negative integer.
Each network reporting endpoint has a retry_after
, which is
either null
, or a timestamp after which delivery should be retried.
An network reporting endpoint is pending if its retry_after
is not null
, and represents a
time in the future.
2.3. Clients
A client represents a particular origin’s relationship to a set of endpoints.
Each client has an origin
,
which is an origin.
Each client has an endpoint-groups
list, which is a list of endpoint
groups, each of which MUST have a distinct name
.
(The algorithm in § 3.2 Process origin policy configuration guarantees this by keeping only
the first entry in the configuration member with a particular name.)
2.4. Failover and load balancing
The network reporting endpoints in an endpoint group that all
have the same priority
form a failover class. Failover classes allow the developer to
provide backup collectors (those with higher priority
values) that will only receive reports
if all of the primary collectors (those with lower priority
values) fail.
Developers can assign each network reporting endpoint in a failover
class a weight
, which determines how report
traffic is balanced across the failover class.
The algorithm that implements these rules is described in § 5.1 Choose an endpoint from a group.
Note: The priority
and weight
fields have the same semantics as the
corresponding fields in a DNS SRV record.
Note: Failover and load balancing is a feature that would be generally useful outside of Reporting. Reporting delegates to the [FETCH] API to actually upload reports once an endpoint has been selected. If, in the future, the Fetch API adds native support for failover and load balancing of requests, a future version of this specification will be updated to use it instead of this bespoke mechanism.
2.5. Storage
A conformant user agent MUST provide a reporting cache, which is a storage mechanism that maintains a set of endpoint groups that websites have instructed the user agent to associate with their origins, and a set of reports which are queued for delivery.
This storage mechanism is opaque, vendor-specific, and not exposed to the web, but it MUST provide the following methods which will be used in the algorithms this document defines:
3. Endpoint Delivery
A server MAY define a set of endpoint groups for an origin it controls through an origin policy manifest [ORIGIN-POLICY].
Endpoint groups are specified with the "network_reporting_endpoints"
member, which defines the endpoint groups to be associated with that
origin.
This member is defined in § 3.1 The "network_reporting_endpoints" policy item, and its processing in § 3.2 Process origin policy configuration.
3.1. The "network_reporting_endpoints" policy item
The network_reporting_endpoints
member defines the endpoint
groups to be associated with the origin.
If present, the member must be an array of objects.
Each object in the array defines a endpoint group to which reports may be delivered, and will be parsed as defined in § 3.2 Process origin policy configuration.
The following subsections define the set of known members which may be specified for each object in the array. Future versions of this document may define additional such members, and user agents MUST ignore unknown members when parsing the configuration.
3.1.1. The group
member
The OPTIONAL group
member is a
string that associates a name
with the endpoint
group.
If present, the member’s value MUST be a string. If not present, the endpoint group will be given the name
"default
".
3.1.2. The include_subdomains
member
The OPTIONAL include_subdomains
member is a boolean that enables this endpoint group for all
subdomains of the current origin’s host.
3.1.3. The max_age
member
The REQUIRED max_age
member defines the endpoint group’s lifetime, as a non-negative
integer number of seconds.
The member’s value MUST be a non-negative number.
A value of "0
" will cause the endpoint group to be removed
from the user agent’s reporting cache.
3.1.4. The endpoints
member
The REQUIRED endpoints
member defines the list of endpoints that
belong to this endpoint group.
The member’s value MUST be an array of JSON objects.
The following subsections define the initial set of known members in each JSON object in the array. Future versions of this document may define additional such members, and user agents MUST ignore unknown members when parsing the elements of the array.
3.1.5. The endpoints.url
member
The REQUIRED url
member is a
string that defines the location of the endpoint.
The member’s value MUST be a string. Moreover, the URL that the member’s value represents MUST be potentially trustworthy [SECURE-CONTEXTS]. Non-secure endpoints will be ignored.
3.1.6. The endpoints.priority
member
The OPTIONAL priority
member is
a number that defines which failover class the endpoint belongs
to.
The member’s value, if present, MUST be a non-negative integer.
3.1.7. The endpoints.weight
member
The OPTIONAL weight
member is a
number that defines load balancing for the failover class that the endpoint belongs to.
The member’s value, if present, MUST be a non-negative integer.
3.2. Process origin policy configuration
Given a map (parsed), and an origin (origin), this algorithm extracts a list of network reporting endpoints and endpoint groups for origin, and updates the reporting cache accordingly.
Note: This algorithm is called from around step 9 of Origin Policy § parse-a-string-into-an-origin-policy, and only updates the reporting cache if the response has been delivered securely.
Origin Policy monkey patching. Talk to Domenic.
-
Let groups be an empty list.
-
If parsed["network_reporting_endpoints"] exists and is a list, then for each item in parsed["network_reporting_endpoints"]:
-
If item has no member named "
max_age
", or that member’s value is not a number, skip to the next item. -
If item has no member named "
endpoints
", or that member’s value is not an array, skip to the next item. -
Let name be item’s "
group
" member’s value if present, and "default
" otherwise. -
If there is already a endpoint group in groups whose
name
is name, skip to the next item. -
Let endpoints be an empty list.
-
For each endpoint item in the value of item’s "
endpoints
" member:-
If endpoint item has no member named "
url
", or that member’s value is not a string, or if that value is not an absolute-URL string or a path-absolute-URL string, skip to the next endpoint item. -
Let endpoint url be the result of executing the URL parser on endpoint item’s "
url
" member’s value, with base URL set to response’s url. If endpoint url is failure, skip to the next endpoint item. -
If endpoint item has a member named "
priority
", whose value is not a non-negative integer, skip to the next endpoint item. -
If endpoint item has a member named "
weight
", whose value is not a non-negative integer, skip to the next endpoint item. -
Let endpoint be a new network reporting endpoint whose properties are set as follows:
-
Add endpoint to endpoints.
-
-
Let group be a new endpoint group whose properties are set as follows:
name
-
name
subdomains
-
"
include
" if item has a member named "include_subdomains
" whose value istrue
, "exclude
" otherwise. ttl
-
item’s "
max_age
" member’s value. creation
-
The current timestamp
endpoints
-
endpoints
-
Add group to groups.
-
-
Let client be a new client whose properties are set as follows:
origin
-
origin
endpoint-groups
-
groups
-
If there is already an entry in the reporting cache for origin, replace it with client. Otherwise, insert client into the reporting cache for origin.
4. Report Generation
Network reports can be generated with or without an active document. If a document is present, and can be considered the source of the report, then the report generated may be visible to reporting observers in that document.
When a user agent is to generate a network report, given a string (type), another string (endpoint group), a serializable object (data), and an optional Document (document), it must run the following steps:
-
If document is given, then
-
Let settings be document’s environment settings object.
-
Let report be the result of running Reporting API § 2.3 Queue data as type for destination with data, type, endpoint group and settings.
-
-
Otherwise, let report be the result of running Reporting API § 2.3 Queue data as type for destination with data, type, and endpoint group.
-
Append report to the reporting cache.
5. Report Delivery
Over time, various features will queue up a list of reports in the user agent’s reporting cache. The user agent will periodically grab the list of currently pending reports, and deliver them to the associated endpoints. This document does not define a schedule for the user agent to follow, and assumes that the user agent will have enough contextual information to deliver reports in a timely manner, balanced against impacting a user’s experience.
That said, a user agent SHOULD make an effort to deliver reports as soon as possible after queuing, as a report’s data might be significantly more useful in the period directly after its generation than it would be a day or a week later.
5.1. Choose an endpoint from a group
Note: This algorithm is the same as the target selection algorithm used for DNS SRV records.
Given an endpoint group (group), this algorithm chooses an arbitrary
eligible endpoint from the group, if there is one, taking into
account the priority
and weight
of the endpoints.
-
Let endpoints be a copy of group’s
endpoints
list. -
Remove every endpoint from endpoints that is pending.
-
If endpoints is empty, return
null
. -
Let priority be the minimum
priority
value of each endpoint in endpoints. -
Remove every endpoint from endpoints whose
priority
value is not equal to priority. -
If endpoints is empty, return
null
. -
Let total weight be the sum of the
weight
value of each endpoint in endpoints. -
Let weight be a random number ≥ 0 and ≤ total weight.
-
For each endpoint in endpoints:
-
It should not be possible to fall through to here, since the random number chosen earlier will be less than or equal to total weight.
5.2. Send reports
A user agent sends reports by executing the following steps:
-
Let reports be a copy of the list of queued report objects in reporting cache.
-
Let endpoint map be an empty map of network reporting endpoint objects to lists of report objects.
-
For each report in reports:
-
Let client be the entry in the reporting cache for origin.
-
If there exists an endpoint group (group) in client’s
endpoint-groups
list whosename
is report’s destination:-
Let endpoint be the result of executing § 5.1 Choose an endpoint from a group on group.
-
If endpoint is not
null
:-
Append report to endpoint map’s list of reports for endpoint.
-
Skip to the next report.
-
-
-
If origin is a tuple origin whose host is a domain:
-
For each parent domain that is a superdomain match for origin’s host [RFC6797], considering longer domains first:
-
Let parent origin be a copy of origin, with its host replaced with parent domain.
-
Let client be the entry in the reporting cache for parent origin.
-
If there exists an endpoint group (group) in client’s
endpoint-groups
list whosename
is report’s destination and whosesubdomains
flag is "include
":-
Let endpoint be the result of executing § 5.1 Choose an endpoint from a group on group.
-
If endpoint is not
null
:-
Append report to endpoint map’s list of reports for endpoint.
-
Skip to the next report.
-
-
-
Note: This algorithm ensures that more specific
subdomains
policies take precendence over less specific ones, and thatsubdomains
policies are ignored for any non-domain origins (e.g., for a request to a raw IP address). -
-
If we reach this step, the report did not match any network reporting endpoint and the user agent MAY remove report from the reporting cache directly. Depending on load, the user agent MAY instead wait for § 6.2 Garbage Collection at some point in the future.
-
For each (endpoint, reports) pair in endpoint map:
-
Let origin map be an empty map of origins to lists of report objects.
-
For each report in reports:
-
For each (origin, per-origin reports) pair in origin map, execute the following steps asynchronously:
-
Let result be the result of executing Reporting API § 3.5.2 Attempt to deliver reports to endpoint on endpoint, origin, and per-origin reports.
-
If result is "
Success
":-
Set endpoint’s
failures
to 0, and itsretry_after
tonull
. -
Remove each report in reports from the reporting cache.
Otherwise, if result is "
Remove Endpoint
":-
Remove endpoint from the reporting cache.
Note: reports remain in the reporting cache for potential delivery to other endpoints.
Otherwise (if result is "
Failure
"):-
Increment endpoint’s
failures
. -
Set endpoint’s
retry_after
to a point in the future which the user agent chooses.Note: We don’t specify a particular algorithm here, but user agents are encouraged to employ some sort of exponential backoff algorithm which increases the retry period with the number of failures, with the addition of some random jitter to ensure that temporary failures don’t lead to a crush of reports all being retried on the same schedule.
Add in a reasonable reference describing a good algorithm. Wikipedia, if nothing else.
-
-
-
Note: User agents MAY decide to attempt delivery for only a subset of the collected reports or endpoints (because, for example, sending all the reports at once would consume an unreasonable amount of bandwidth, etc). As reports are only removed from the cache when they’re successfully delivered, skipped reports will simply be delivered later.
6. Implementation Considerations
6.1. Delivery
The user agent SHOULD attempt to deliver reports as soon as possible to provide feedback to developers as quickly as possible. However, when this desire is balanced against the impact on the user, the user wins. With that in mind, the user agent MAY delay delivery of reports based on its knowledge of the user’s activities and context.
For instance, the user agent SHOULD prioritize the transmission of reporting data lower than other network traffic. The user’s explicit activities on a website should preempt reporting traffic.
The user agent MAY choose to withhold report delivery entirely until the user is on a fast, cheap network in order to prevent unnecessary data cost.
The user agent MAY choose to prioritize reports from particular origins over others (perhaps those that the user visits most often?)
6.2. Garbage Collection
Periodically, the user agent SHOULD walk through the cached reports and endpoints, and discard those that are no longer relevant. These include:
-
endpoint groups which are expired.
-
endpoint groups which have not been used in some arbitrary period of time (perhaps a ~week?)
-
reports whose attempts exceed some user-agent-defined threshold (~5 seems reasonable.)
-
reports which have not been delivered in some arbitrary period of time (perhaps ~2 days?)
7. Sample Reports
POST / HTTP/1.1 Host: example.com ... Content-Type: application/reports+json [{ "type": "csp", "age": 10, "url": "https://example.com/vulnerable-page/", "user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0", "body": { "blocked": "https://evil.com/evil.js", "directive": "script-src", "policy": "script-src 'self'; object-src 'none'", "status": 200, "referrer": "https://evil.com/" } }, { "type": "hpkp", "age": 32, "url": "https://www.example.com/", "user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0", "body": { "date-time": "2014-04-06T13:00:50Z", "hostname": "www.example.com", "port": 443, "effective-expiration-date": "2014-05-01T12:40:50Z" "include-subdomains": false, "served-certificate-chain": [ "-----BEGIN CERTIFICATE-----\n MIIEBDCCAuygAwIBAgIDAjppMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVT\n ... HFa9llF7b1cq26KqltyMdMKVvvBulRP/F/A8rLIQjcxz++iPAsbw+zOzlTvjwsto\n WHPbqCRiOwY1nQ2pM714A5AuTHhdUDqB1O6gyHA43LL5Z/qHQF1hwFGPa4NrzQU6\n yuGnBXj8ytqU0CwIPX4WecigUCAkVDNx\n -----END CERTIFICATE-----", ... ] } }, { "type": "nel", "age": 29, "url": "https://example.com/thing.js", "user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0", "body": { "referrer": "https://www.example.com/", "server-ip": "<IP_ADDRESS>", "protocol": "", "status-code": 0, "elapsed-time": 143, "age": 0, "type": "http.dns.name_not_resolved" } }]
8. Security Considerations
8.1. Capability URLs
Some URLs are valuable in and of themselves. To mitigate the possibility that such URLs will be leaked via this reporting mechanism, we strip out credential information and fragment data from the URL we store as a report’s originator. It is still possible, however, for a feature to unintentionally leak such data via a report’s body. Implementers SHOULD ensure that URLs contained in a report’s body are similarly stripped.
9. Privacy Considerations
9.1. Network Leakage
Because this reporting mechanism is out-of-band, and doesn’t rely on a page being open, it’s entirely possible for a report generated while a user is on one network to be sent while the user is on another network, even if they don’t explicitly open the page from which the report was sent.
Consider mitigations. For example, we could drop reports if we change from one network to another. <https://github.com/w3c/BackgroundSync/issues/107>
9.2. Clock Skew
Each report is delivered along with an age
property, rather than the
timestamp at which it was generated. We do this because each user’s local
clock will be skewed from the clock on the server by an arbitrary amount.
The difference between the time the report was generated and the time it
was sent will be stable, regardless of clock skew, and we can avoid the
fingerprinting risk of exposing the clock skew via this API.
9.3. Cross-origin correlation
If multiple origins all use the same reporting endpoint, that endpoint may
learn that a particular user has interacted with a certain set of websites,
as it will receive origin-tagged reports from each. This doesn’t seem worse
than the status quo ability to track the same information from cooperative
origins, and doesn’t grant any new tracking ability above and beyond what’s
possible with <img>
today.
9.4. Subdomains
This specification allows any resource on a host to declare a set of reporting endpoints for that host and each of its subdomains. This doesn’t have privacy implications in and of itself (beyond those noted in § 9.5 Clearing the reporting cache), as the reporting endpoints themselves don’t take any real action, as features will need to opt-into using these reporting endpoints explicitly. Those features certainly will have privacy implications, and should carefully consider whether they should be enabled across origin boundaries.
9.5. Clearing the reporting cache
A user agent’s reporting cache contains data about a user’s activity on the web, and user agents ought to handle this data carefully. In particular, if a user agent gives users the ability to clear their site data, browsing history, browsing cache, or similar, the user agent MUST also clear the reporting cache. Note that this includes both the pending reports themselves, as well as the endpoints to which they would be sent. Both MUST be cleared.
9.6. Disabling Reporting
Reporting is, to some extent, a question of commons. In the aggregate, it seems useful for everyone for reports to be delivered. There is direct benefit to developers, as they can fix bugs, which means there’s indirect benefit to users, as the sites they enjoy will be more stable and enjoyable. As a concrete example, Content Security Policy grants something like herd immunity to cross-site scripting attacks by alerting developers about potential holes in their sites' defenses. Fixing those bugs helps every user, even those whose user agents don’t support Content Security Policy.
The calculus, of course, depends on the nature of data that’s being delivered, and the relative maliciousness of the reporting endpoints, but that’s the value proposition in broad strokes.
That said, it can’t be the case that this general benefit be allowed to take priority over the ability of a user to individually opt-out of such a system. Sending reports costs bandwidth, and potentially could reveal some small amount of additional information above and beyond what a website can obtain in-band ([NETWORK-ERROR-LOGGING], for instance). User agents MUST allow users to disable reporting with some reasonable amount of granularity in order to maintain the priority of constituencies espoused in [HTML-DESIGN-PRINCIPLES].


| Package | Description |
|---|---|
| core | |
| gui |
| Modifier and Type | Class and Description |
|---|---|
class |
Directory
A representation of a directory in the virtual file system.
|
class |
File
A representation of a file in the virtual file system.
|
| Modifier and Type | Method and Description |
|---|---|
VNode |
VirtualFileSystem.get(java.lang.String pathName)
returns the VNode corresponding to the path name given as arguement
|
VNode |
Directory.get(java.lang.String relatifPathName)
returns the VNode at the relative path given as argument starting from the receiver.
|
VNode |
VirtualFileSystem.importFromHostFileSystem(java.lang.String importPathName)
imports a file/directory from the host file system to the current working directory of the virtual file system
|
VNode |
VirtualFileSystem.importFromHostFileSystem(java.lang.String importPathName,
java.lang.String pathName)
imports a file/directory from the host file system to the virtual file system
|
| Modifier and Type | Method and Description |
|---|---|
java.util.List<VNode> |
VirtualFileSystem.find(java.lang.String name)
search for flies/directories linked to the current working directory
|
java.util.List<VNode> |
Directory.find(java.lang.String name)
search for files/directories linked to the receiver's node in the virtual file system
|
java.util.List<VNode> |
Directory.getContentList()
returns the list of content in the directory (files/directories)
|
| Modifier and Type | Method and Description |
|---|---|
(package private) void |
Directory.addChild(VNode child) |
void |
Directory.addVNode(VNode child)
adds a VNode to the directory content list.
|
| Modifier and Type | Field and Description |
|---|---|
(package private) java.util.HashMap<VNode,javax.swing.JLabel> |
GUInterface.contents |
| Modifier and Type | Method and Description |
|---|---|
static javax.swing.tree.DefaultMutableTreeNode |
GUInterface.buildTree(VNode node) |
private void |
GUInterface.updatePropreties(VNode node) |


Civil Engineer
National Capital Reg
Sr. Manager, IT Business Operations
National Capital Reg
Area Sales Supervisor
Pasig City (National Capital Reg)
Sales Specialist
Quezon City (National Capital Reg)
MARKETING ASSISTANT
Manila City (National Capital Reg)
Online Advertiser
Laguna (Others) (Calabarzon & Mimaropa)
Sales Engineer
Pampanga (Others) (Central Luzon) - Clark
Purchasing Technical Coordinator
Laguna (Others) (Calabarzon & Mimaropa)
PURCHASING ASSISTANT MANAGER
Cavite (Calabarzon & Mimaropa)


@@include('../partials/head/head.html')
@@include('../partials/components/entry-section.html')
@@include('../partials/components/starter-box.html', {
"title": "JustCoded",
"intro-text": "Web Starter Kit"
})
@@include('../partials/scripts/common.html')


It may be that your genetic or psychological characteristics mean you are more likely to take drugs.
Some people try drugs for the first time as an experiment, because they are curious or because friends are doing it.
This is a major factor, as many people take recreational drugs for the ‘buzz’ or ‘thrill’. They make deliberate choice to pursue ‘high’.
As a destructive opioid, heroin can lead to seizures, psychosis, and hallucinations when it is abused. Heroin injections can also spread diseases such as human immunodeficiency virus and hepatitis. This drug is known to cause serious health issues when it is abused because it interferes with the brain’s receptors. Users can become physically dependent on it very quickly, requiring them to take more and more to achieve the same high, but too much heroin can lead to a fatal overdose. Individuals who take heroin always require professional rehabilitation because the symptoms of withdrawal may be life-threatening.
Accelerating the user’s bodily and mental functions, amphetamines can cause manic periods of distress in abusers. These periods are usually accompanied by extreme paranoia, inexplicable behavior, and delusions. Some amphetamine abusers become very violent and attack loved ones unintentionally and could suffer permanent physical changes in appearance, irreversible brain damage, and nerve damage. Amphetamines are harsh, dangerous drugs, and users need medical attention and safe rehabilitation in professional facilities.
PCP (phencyclidine) and LSD (lysergic acid diethylamide) are hallucinogens, which means that they make users feel, see, and hear things that are not real. While they experience hallucinations with these drugs, users lose touch with reality and enter mental states of disconnection, as if their bodies and minds are not working together or connected. Some users of PCP and LSD enter violent states of psychosis. Serious injury could occur while hallucinating on these substances, which have caused permanent neurological damage in cases of repeated abuse. Withdrawing from hallucinogens is not easy, and abusers will need professional treatment in a rehab facility.