HTTP connection (which is used to transfer HTML documents) is state-less. Let's look
at a sample interaction between a browser and a server to see what is mean by that.
| Browser | Server |
|---|---|
|
Generate an request HTTP header HTTP (as per RFC1945 for HTTP/1.0) contains a specific method such as POST or GET (other can also be used but these two are most common). Along with that, the browser includes additional HTTP information to let the server know what type it is and what the browsing capabilities are. Some browsers can also send HTTP Cookie information, but that is not supported by all browsers and may be turned off by users. Coincidentally cookies are a good way to store state, but here we will seek an alternative method due to absence of complete compliance by all browsers. |
|
|
Parses the HTTP request header and sends HTTP response header At this point the server knows what is needed and attempts to retrieve it from storage. If an error occurs then an error is sent in the header and possible error message will follow. |
|
|
Receives response HTTP header and awaits the content The response header contains useful information about what is to be sent by the server. MIME-type is mandatory and for HTML is will look like Mime-Type: text/html. Other MIME types may be used here, such as image/jpeg, audio/wav, et.al. Another useful field in the header is Content-Length that is needed for binary MIME-types and in some cases to speed up loading of text-based pages. |
|
|
Send the data of the MIME-type specified in the response HTTP header This is the actual data from the content that was requested in the request HTTP header. At the end of this the server closes connection and forgets about this session. |
|
|
Receives data and displays as needed The browser at this point is probably not connected to the server and the session is over. |
Hopefully it is clear from the above table that the connection between the server and the browser exists
only for the duration of transferring headers and the actual data. Alas, all is not lost, there are several
methods for maintaining state. Form variable and cookies and the most common. We will not discuss cookies
in this tutorial (but rather defer it to a separate one).
Form variables
There are two ways to pass information back to the server, first let's see what they look like in HTML,
we will then discuss them:
1. Let's look at use of HIDDEN form variables to maintain state:
<FORM ACTION="/cgi-bin/MyProgram.cgi" METHOD="GET"> <INPUT TYPE=TEXT NAME=Username> <INPUT TYPE=PASSWORD NAME=Password MAXLENGTH=8> <INPUT TYPE=HIDDEN NAME=State VALUE=0> <INPUT TYPE=SUBMIT NAME=Login VALUE=Login> </FORM>When the user enters the name and password and hits the Login button the following request will be sent to the server:
GET /cgi-bin/MyProgram.cgi?Username=Homer&Password=Beer&State=0&Login=Login HTTP/1.0Here we have 4 variables, one of them is state. Our CGI program will be able to generate forms from now on with the state that contains information about the user so that logging in is not done every time a connection is made. Let's look at another method, then we will see how we can code this.
<FORM ACTION="/cgi-bin/MyProgram.cgi?State=0" METHOD="POST"> <INPUT TYPE=TEXT NAME=Username> <INPUT TYPE=PASSWORD NAME=Password MAXLENGTH=8> <INPUT TYPE=SUBMIT NAME=Login VALUE=Login> </FORM>Same thing, except the form variables are submitted via a POST method and the State is submitted via GET. Here what a sample HTTP header would look like for the following FORM (omitting much of the HTTP header fields for clarity):
POST /cgi-bin/MyProgram.cgi?State=0 HTTP/1.0 Referer: http://www.myserver.com/LoginPage.html Connection: Keep-Alive User-Agent: Mozilla/4.01 [en] (WinNT; I) Content-type: application/x-www-form-urlencoded Content-length: 49 (must have a blank line here to signify end-of-header) Username=Homer&Password=Beer&State=0&Login=LoginThis may look a bit confusing, but if you look through RFC1945 must of HTTP confusion may be cleared up. We won't worry about the HTTP stuff, just that we need to know what the user is doing. Let's say after logon you have to know which user is submitting information, so that you can change them differently. We need to determine state information about the user whenever a form is submitted.
//Create state data to be saved //Assume that pccState is where you will store name and password const char *pccState = "Homer:Beer"; const BYTE *pccEncryptionKey = "MyKey"; //You can do better than this :) int iKeyLength = 5; //Keys can contain any ASCII value //Specify which method to use for encryption and encoding UNIT uMethod = ACrypto::ectXOR_Convolver|AConverto::eat6Bit; //Create a temporary area for the state, the state can have any ASCII character //This is why you have to provide the length APairItem piTest; BYTE *pbState = aMemDup(NULL, iLength); int iStateLength = strlen(pccState); memcpy(pbState, pccState, iStateLength); //Preparing the posting of data //You may want to store the user's IP here for validation DWORD dwIP = cgiOut.cgiGetIP(); piTest.piSet("State"); //This method of APairItem will encrypt and encode the data into a pair variable piTest.piSetValueChecked(pbState, iStateLength, dwIP, uMethod, pcbEncryptionKey, iKeyLength); //Cleanup, we don't need this anymore delete []pbState; //At this point you can do anything you want with the APairItem, let's add it to a form //We will assume that FORM header was already created cgiOut << ">INPUT TYPE=HIDDEN " << piTest << ">" << endl;At this point we also need a way of retrieving the State variable when a user submits the form to us. We will assume that you have created an ACGI object and parsed FORM submissions.
//Start by finding the State variable in the submission APairItem *piState = acgiOut.plGetItemByName("State"); if (piState) { const BYTE *pcbNewData; DWORD dwIP = cgiOut.cgiGetIP(); //You get this from CGI environment variables int iLength; //Variable used to get the length of the new state int iTimeout = 100; //100 seconds timeout since the State was last created //Specify which method to use for encryption and encoding, must be consistent with creation UNIT uMethod = ACrypto::ectXOR_Convolver|AConverto::eat6Bit; //Decode our State variable pcbNewData = piTest.piDecodeCheckedValueAndGetUserData(iLength, dwIP, iTimeout, uMethod, pcbKey, iKeyLength); if (pcbNewData) { //Let's just display the output, you will of course do what you need with it cgiOut << "STATE=" << (const char*)pcbNewData; cgiOut << " length=" << iLength << endl; } else { In case it was not found cgiOut << "=(null)?" << endl; } }By now you should be a bit more comfortable with state variables. The classes you should look at are APairItem, ACrypto, AConverto, and ACGI; to get a better idea of the methods available and how to use them. Also refer to samples/test/state.cpp for a functional example of using state variables.
Enjoy!
Table Of Contents


`
I am experienced in conducting qualitative and quantitative studies, leveraging HCI frameworks to study the human elements of technology, with an interest in the field of education. My work is primarily focused on bringing to light the hidden mechanisms by which underserved and underrepresented communities interact and utilize technology in their daily life, and developing tools to fit their needs.
I investigated how technology can play a role in facilitating heritage language learning within the Bangladeshi immigrant population in Canada, and highlight the various concerns immigrant parents had with regards to their and their children’s adaptation to living in a foreign society.
Developed a Virtual Reality application to assist individuals with the documentation and reliving of memories and events in an immersive environment.
As a part of my undergraduate Capstone project, I developed a software framework that can insert additional logic into a given integrated circuit netlist to accomplish security objectives defined by the circuit designer.
Worked on a team to prototype a submersible autonomous drone for coral reef monitoring. My responsibilities included adapting UAV autopilot software for underwater use, and facilitating appropriate testing enviornments during development. The project received 2nd place in the UAE National Drones for Good award in 2016
I worked on an individual project with the Ads Solutions team to plan and integrate offline compatibility of online features available to advertisers. The project involved work both on the Android application, as well as server side modifications. The project was highly successful, leading to positive metric movement.
Assisted in the design and development of an online integrated system for learning outcomes assessment and program effectiveness.
I provided both in-class and tutorial assistance in the Introduction to Computer Programming class (csc108) at the University of Toronto for four semesters with 1200+ students. I also participated and lead final and midterm exam invigilation and marking as part of my duties.
GPA: 3.93
GPA: 3.89
I have a strong interest in pursuing Human Computer Interaction and ICTD research from an educational perspective in the Middle East and North Africa, as these regions have been considerably underrepresented in HCI literature, although they are areas with rich and unique social, cultural, and political dynamics that have recently been revolutionized through the wide-spread access of mobile devices and social media.
Apart from research, my interests include keeping active by playing basketball. I have always been involved in competitive basketball communities throughout high school and university, and have captained the varsity teams at both highschool and university levels for more than 7 years, leading NYUAD's varsity basketball team to consecutive national level championships in 2014 and 2015. In my free time, I enjoy reading and following the latest trends in educational technology and policy, watching sci-fi shows, and exploring the outdoors through hiking or biking.


Paper Pen-guins |
| Overview |
Paper Pen-guins is my pencil-/pen-and-paper adaptation of the game Pingvinas, a.k.a. “Hey! That’s My Fish!” by Alvydas Jakeliunas and Günter Cornett.
| Rules |
The board is an 8x8 square grid. (Graph paper is recommended.) Each grid cell contains a number indicating point value. Eleven cells have 3 points, Twenty-one cells have 2 points, and the remaining thirty-two cells have 1 point. (These points may be placed by random process, or players may take turns, first placing eleven 3’s, then twenty-one 2’s, and finally filling remaining 1’s.)
A player-1/player-2 piece is represented as a square/circle surrounding a cell’s point value. (Although not required, players may find use of different colored pens helpful.) Players take turns placing pieces until each player has four pieces on the board. Players then take turns making moves until one player has no legal move, or both players agree on the game’s final outcome. Players then tally points that would be scored in remaining legal moves.
For a legal move:
A player chooses a piece and scores the points of that piece’s cell. (Points may be tallied to the side of the board.)
The player then crosses out that cell, forming an “X” from opposite cell corners.
Finally, if the player can, the player must “move” the piece by redrawing the piece a Chess queen’s move away (any number of spaces horizontally, vertically, or diagonally) such that the piece does not move on or beyond another piece or crossed-out cell.
| Examples |
Todd Neller,
http://cs.gettysburg.edu/~tneller
October 21st, 2009


{% extends 'base.html' %} {% block index %} {% include 'navbar.html' %} TuneFork TuneFork is literally here to change your life. Learn more Login: {% csrf_token %} account_circle Username vpn_key Password Login New to TuneFork? Create an account.
TuneFork is literally here to change your life.
New to TuneFork? Create an account.
Lorem ipsum dolor sit amet, consectetur adipisicing elit. Id amet, quisquam modi nobis nostrum iusto incidunt dolore assumenda.


LremCommand
Time complexity: O(N) (with N being the length of the list)Remove the first count occurrences of the value element from the list.If count is zero all the elements are removed. If count is negativeelements are removed from tail to head, instead to go from head to tailthat is the normal behaviour. So for example LREM with count -2 and_hello_ as value to remove against the list (a,b,c,hello,x,hello,hello) willlave the list (a,b,c,hello,x). The number of removed elements is returnedas an integer, see below for more information aboht the returned value.
Integer Reply, specifically:
The number of removed elements if the operation succeeded
-1 if the specified key does not exist
-2 if the specified key does not hold a list value


About
My name is Andy Yin, I'm a student at Rose-Hulman Institute of Technology. I'm majoring in Computer Science at the moment.
I was originally from China, I've been living in the U.S for about 6 years now. I took a gap year from college last year so I'm still counted as a sophomore at this point. When I'm not busy learning the secret of computer science, I love playing guitar and videos games. If you have any interest in discussing topics regarding programming, gaming, and guitar with me, I'll be happy to do that. I would love to meet more people and talk. It's always good to know other people who share the same interests.


job_stream.baked¶
Pre-baked templates for common distributed operations.
Members¶
-
job_stream.baked.
sweep
(variables={}, trials=0, output=None, trialsParms={}, showProgress=True)[source]¶ Generic experiment framework; wraps a user’s job_stream (realized via
job_stream.inline.Work
. The wrapped job_stream should take a parameter dictionary and return a dictionary of results to track. The numerical statistics of several invocations of the user’s code will be reported. For example:from job_stream.baked import sweep import numpy as np with sweep({ 'a': np.arange(10) }) as w: @w.job def square(id, trial, a): return { 'value': a*a + np.random.random() }
The above will print out a CSV file (in a legible, table format) that details the mean of
value
fora = 0, a = 1, ..., a = 9
. Additionally, the standard deviation and expected (95% confidence) error of the reported mean will be printed.Note
While training e.g. a machine learning model, it may be desirable to print the model’s accuracy at different points throughout the training. To accomplish this, it is recommended that the user code remember the accuracy throughout training and multiplex the column in the final answer (e.g.,
value_at_0
,value_at_1000
, etc).Parameters: - variables (any) –
Any of:
dict
:{ 'parameter name': [ 'values to try' ] }
.Tries every combination of values for the specified parameters.
Often,
values to try
will come from a function such asnumpy.linspace
, which generates a number of intermediate values on a range.Warning
When more than one parameter is specified, the number of experiments that will run will be the multiplication of the number of values to try for each; that is, all combinations are tried. This will quickly take a long time to run, so be careful.
list
:[ { 'parameter name': 'value' } ]
.Tries only the specified combinations. The dictionaries are passed as-is.
Regardless of the type passed, the arguments seen by the user’s jobs will also include
id
, a unique identifier for the combination of parameters, andtrial
, the current iteration of that combination. - trials (int) –
The number of times to try each parameter combination.
If greater than zero, this number is exact.
If zero, the number will be automatically discerned based on the standard deviations of each returned property. More specifically, the algorithm proposed by Driels et al. in 2004, “Determining the Number of Iterations for Monte Carlo Simulations of Weapon Effectiveness,” will be used to guarantee that every mean returned will be within 10% of the true value with 95% confidence.
If less than zero, the same algorithm is used as for a zero value, but the number of trials ran will not exceed
abs(trials)
. - output (str) – If
None
, results are printed to stdout and the program exits. If anything else, presumed to be the path to a CSV file where the results will be dumped. - trialsParms (dict) –
A dictionary of keys used to configure the auto-detect used to determine the number of trials needed.
- E
- The percentage of relative error between the true mean and the reported mean. For instance, if E is 0.1, then there is a 95% confidence that the true value is on the range of the estimated value * (1. +- 0.1). Setting to 0 will disable this stopping criteria. Defaults to 0.1.
- eps
- The absolute minimum estimated error. Setting to 0 will disable this stopping criteria. Defaults to 1e-2.
- min
- The minimum number of trials to run. Defaults to 3, as this is usually enough to get a stable idea of the variable’s standard deviation.
- max
- The maximum number of trials to run. Defaults to 10000. May
also be specified by setting the argument
trials
to a negative number.
- showProgress (bool) – If True (default), then print out progress indicators to stderr as work is completed.
Returns: Nothing is returned. However, both stdout and, if specified, the csv indicated by
output
will have a grid with all trial ids, parameters, and results printed. Each result column will represent the mean value; additionally, the estimated standard deviation and the 95% confidence error of the reported mean will be printed.- variables (any) –


Your current username is {{user?.username}}. It may have been auto-generated when you initially created a Dockstore account.
You can update your username only if you meet certain conditions.
You can't change your username.
warning Your username seems to be an email. We recommend changing your username to something else so that your email is
not picked up by a spambot. The @ symbol is not allowed in a username, so note that once you change your username you cannot
change it back to your email.
UsernameInvalid Username Pattern (Only alphanumeric characters and internal underscores and dashes allowed)Username is requiredUsername is too long (Max 39 characters)check Username available
clear Username not available


Throughout the years, I was never sure of the health of my actions.
With time, I learned that utopias were the best way to assess the path where I was going.
I'd always ask myself, as an unbiased spectator:
"Is my utopia essentially good?"
And, for a long time, I became overly skeptical.
My utopia was having "fuck you money", buying an island, creating some robots, find a pill for immortality, then figure out what to do.
Escaping, surviving, hiding from the world. And I knew it was bad.
But I was filled with gray days, migrating through sterile meeting rooms, lonely airports, and rotten subway stations.
No matter how far and sophisticated, everywhere was void of life.
I couldn't force myself to enjoy. There was nothing to experience.
I couldn't speak the love language of bridges, mountains, castles, museums, rivers.
But, then...
A small bird got my eyes, we started dancing. It pushed me to fly higher and higher.

There is light, and I'm not as bitter against a dark world.
I don't wanna hide - there's a world that embraces who I am.
A world that I want to experience, feel and let it heighten my mind.
My hands are not alone, cold, holding a phone.
I find warm hands that keep me going.
And, perhaps, I don't know where or when I'm going.
But, I know I'm going.
I know we are going.
Always going.
Somewhere.
Where light lives.
That's my utopia.


---
title: BIG Graph
layout: default
category: custom-layout-modes
---
Custom layout mode to replicate the Flash interface of big.dk. Similiar to Category rows, item elements are grouped by their sorting data into columns.
{% highlight javascript %}
$container.isotope({
layoutMode: 'bigGraph',
bigGraph: {
columnWidth: 45, // size of item
rowHeight: 45, // size of item
maxRows: 11, // max number of items vertically
gutterWidth: { // width of gutter, needs to match getSortData names
year: 0,
scale: 60,
program: 40,
status: 60,
title: 0
}
},
// options...
});
{% endhighlight %}
To use this layout mode, grab the $.Isotope.prototype
methods from the script at the bottom of this page's source.


How to freeze columns in Microsoft Excel to compare data throughout your spreadsheet, Business Insider - Business Insider Singapore
by Kelly Laffey, Business Insider US- You can freeze columns in Excel with a few clicks, and then unfreeze them when you no longer need to view them statically.
- Freezing a column in Excel makes that pane visible while you scroll to other parts of the worksheet.
- Visit Business Insider’s homepage for more stories.
Freezing columns in Excel is an easy way to ensure that those panes remain visible as you scroll through the rest of the document.
This allows you to easily compare the data and text in a variety of spots in the worksheet.
You can freeze multiple columns at one time, and then unfreeze them when you’re ready to view the document without having the panes locked into view.
Check out the products mentioned in this article:
Microsoft Office (From $129.99 at Best Buy)
How to freeze columns in Excel
1. In an open spreadsheet, select “View” from the top menu bar and then hit “Freeze First Column.” This will freeze column A.
2. If you want to freeze multiple columns, click in the first cell to the right of the last column that you want to freeze. Then hit “Freeze Panes” in the “View” menu.
3. Selecting a cell that’s below the first row will also freeze all prior rows. For example, if you select cell D5, you will freeze columns A, B and C; as well as rows 1, 2, 3 and 4.
4. The faint grey line that appears on the worksheet indicates that all the columns to the left are frozen. They’ll stay on the screen as you scroll throughout the document.
How to unfreeze columns in Excel
1. Once you freeze the columns, the “Freeze Panes” button turns into an “Unfreeze Panes” button.
2. To unfreeze the frames, select “View,” then select “Unfreeze Panes.”
3. You can now view your Excel document normally.
Related coverage from How To Do Everything: Tech:
-
How to make a pie chart from your spreadsheet data in Microsoft Excel in 5 easy steps
-
How to insert multiple rows in Microsoft Excel on your Mac or PC
-
How to hide and unhide columns in Excel to optimize your work in a spreadsheet
-
How to add a column in Microsoft Excel in 2 different ways
-
How to make a line graph in Microsoft Excel in 4 simple steps using data in your spreadsheet


|
|
Unlimited possibilities. Experience the 360.
Caribbean Islands
The Caribbean islands are those islands that border or are surrounded by the Caribbean Sea organized by the political entity to which each island belongs. There are thousands of islands that are part of the island countries of the broadly defined Caribbean region: Anguilla has ; Antigua-and-Barbuda has ; Aruba; Barbados, formerly 3 but Pelican Island is now absorbed into Barbados through land reclamation, 1956–1961); Belize (many); British Virgin Islands ; Cayman Islands; Cuba ; Dominica; Dominican Republic ; Grenada ; Guadeloupe ; Haiti ; Honduras; Jamaica ; Martinique; Montserrat ; Caribbean Netherlands; Puerto Rico ; Saint Barthélemy; Saint Kitts-and-Nevis ; Saint Lucia ; Saint Martin ; Saint Vincent-and-the-Grenadines; Trinidad-and-Tobago ; and United States Virgin Islands .


{%- assign articles = site.articles -%} {%- assign cheatsheets = site.cheatsheets -%} {%- assign sessions = site.sessions -%} {%- assign workshops = site.workshops -%} {%- include profile-list-exception.html author=page.title -%}


Collaborating with Netgura had lasting impact on our ability to be innovative. and execute at pace. The team there hove a strong focus on not just doing what you ask.
Flutter is Google's mobile app software development kit (SDK) made to build cross-platform applications. It's a complete environment with a framework, widgets, and tools, which gives you the opportunity to develop mobile apps efficiently.
Python is gaining a lot of popularity among Fintech, startups, cloud computing technology companies as well as data analytics projects, Machine Learning, and Internet of Thing. Python allows for rapid app development.
Python's syntax is clear and elegant, and the programming language is object-oriented. Still, Python is considered as a high-level and general-purpose programming language used not only for enhanced web development but also simple web applications.
Python a gaining a lot of popularity among Fintech, startups, cloud computing technology companies as well as data onolytics projects, Machine Learning, and InWrnet of Things. Python allows for rapid app development.
Python's syntax is clear and elegant, and the programming language is object-oriented. Still, Python is considered as a high-level and general-purpose programming language used not only for enhanced web development but also simple web applications.
The simplicity of Python means that nof only is it easy to maintain, but it can also handle quick growth. What's more, if you use the Django framework - the most popular framework for Python - you will also put yourself in good stead to scale. This is because the components that make up Django are decoupled (i.e. independent from one another), which means that they can be chopped, changed, removed, and replaced as and when your project requires larger or more specific solutions.


Why Social Distancing is Still Necessary (And How To Do It Properly)
While regulations regarding social distancing are loosening up around the world, that does not mean social distancing will soon be a thing of the past.


{%extends "adm/admin_base.html" %} {%load currency%} {%block title%}Match pending bank transaction{%endblock%} {%block layoutblock %}
Select an invoice to match payment to.
{%if matchers%}Select a matcher to match tihs payment to. Normally this should always be matched with the regexp, but in case it doesn't a manual match can be done here.
Only matchers where the amount matches exactly are included.
| Pattern | Amount | Journal Entry | Action |
|---|---|---|---|
| {{m.pattern}} | {{m.amount|format_currency}} | {{m.journalentry}} | Match this matcher |


Speak Securely
Speak Securely provide the additional encription to your messenger. Ensure that your communication is secure.
Keys

Speak Securely provide the additional encription to your messenger. Ensure that your communication is secure.


About
Originally from Florida's Treasure Coast, I moved to Portland, OR in 2014 to pursue a career in social work. Recently, I have begun studying web development and design in hopes of marrying my passion for creativity with my love of technology and the internet. Outside of school, I enjoy practicing yoga, drawing, and spending time with my cat, Buddha.
Professional Works
Git Hub
CodePen.io
Education and Skills
- Bachelors of Science in International Relations, University of Central Florida.
- Foundational understanding of HTML, CSS, and JavaScript.
- Fluent in French.
- Twelve years in the hospitality/service industry


buffer
BUbiNG is a high-speed distributed crawler. BUbiNG agents coordinate autonomously to download data from the web.
BUbiNG requires a number of features from your system.
You can specify an alternate JGroups configuration file using the JVM property {@value it.unimi.di.law.bubing.Agent#JGROUPS_CONFIGURATION_PROPERTY_NAME}.
* - nofile 65536 * - nproc 32768as a setup in limits.d (or equivalent configuration directory).
# increase TCP max buffer size settable using setsockopt() net.core.rmem_max = 8388608 net.core.wmem_max = 8388608 # increase conntrack size (or remove iptables modules) net.netfilter.nf_conntrack_max = 1048576 net.netfilter.nf_conntrack_buckets = 16386 # Wait a maximum of 5 * 2 = 10 seconds in the TIME_WAIT state after a FIN, to handle # any remaining packets in the network. net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 5 net.netfilter.nf_conntrack_tcp_timeout_time_wait = 5 # No protection from SYN flood attack (to run proxy tests) net.ipv4.tcp_syncookies = 0 # Disable packet forwarding. net.ipv4.ip_forward = 0 net.ipv6.conf.all.forwarding = 0 # Increase system file descriptor limit. Generally, set this to 64 * R, where # R is the amount of RAM in MB your box has (minus a buffer?) fs.file-max = 262144 # Send the first keepalive time after 60 seconds net.ipv4.tcp_keepalive_time = 60 # Timeout broken connections faster (amount of time to wait for FIN) net.ipv4.tcp_fin_timeout = 30 # Let the networking stack reuse TIME_WAIT connections when it thinks it's safe to do so net.ipv4.tcp_tw_reuse = 1 # Don't Log Spoofed Packets, Source Routed Packets, Redirect Packets net.ipv4.conf.all.log_martians = 0 # Make more local ports available net.ipv4.ip_local_port_range = 1024 65000
The above file is sourceable with sysctl.
allow-query { localhost; }; recursion yes; /* These must be coordinated with internal DNSJava settings. */ max-cache-size 10000; max-cache-ttl 3600; max-ncache-ttl 60;
Moreover, we run bind with the -4 option to avoid IPv6 resolution of each host and with the -U 8 option to avoid the constant logging about the maximum number of FD events.
tune2fs
utility to set the journal_data_writeback option, and to mount
the partition using the options noatime and data=writeback.
To check that your system is properly configured, it is essential that a number of trial runs are performed using the {@link it.unimi.di.law.bubing.test.NamedGraphServerHttpProxy} in random mode. For instance,
java -Xmx4G -server it.unimi.di.law.bubing.test.NamedGraphServerHttpProxy -s 100000000 -d 50 -m 3 -t 1000 -D .0001 -A1000 - &will start the proxy using 100 million hosts, average page degree 50, average depth 3, 1000 threads and a 0.01% pages being broken (connect or read timeouts). By starting BUbiNG with the configuration option proxyHost=localhost (the port, 8080, is understood) you can test that your memory and system settings are satisfactory without using any network bandwidth.
BUbiNG uses a configuration file to set up its internal values. Such values are fully documented in {@link it.unimi.di.law.bubing.StartupConfiguration}; the name of the keys are derived by reflection from the field names. Default values are annotated using the {@link it.unimi.di.law.bubing.StartupConfiguration.OptionalSpecification} annotation in {@link it.unimi.di.law.bubing.StartupConfiguration}. This is an example configuration file:
rootDir=crawl-eu maxUrlsPerSchemeAuthority=1000 parsingThreads=64 dnsThreads=50 fetchingThreads=512 fetchFilter=true scheduleFilter=( SchemeEquals(http) or SchemeEquals(https) ) and HostEndsWithOneOf(.ad\,.al\,.at\,.be\,.bg\,.ch\,.cz\,.de\,.dk\,.ee\,.es\,.eu\,.fi\,.fo\,.fr\,.gb\,.gr\,.hr\,.hu\,.ie\,.im\,.is\,.it\,.li\,.lt\,.lu\,.lv\,.mc\,.md\,.me\,.nl\,.no\,.pl\,.pt\,.ro\,.se\,.si\,.sk\,.sm\,.uk\,.va) and not PathEndsWithOneOf(.axd\,.xls\,.rar\,.sflb\,.tmb\,.pdf\,.js\,.swf\,.rss\,.kml\,.m4v\,.tif\,.avi\,.iso\,.mov\,.ppt\,.bib\,.docx\,.css\,.fits\,.png\,.gif\,.jpg\,.jpeg\,.ico\,.doc\,.wmv\,.mp3\,.mp4\,.aac\,.ogg\,.wma\,.gz\,.bz2\,.Z\,.z\,.zip) and URLShorterThan(2048) and DuplicateSegmentsLessThan(3) followFilter=true parseFilter=( ContentTypeStartsWith(text/) or PathEndsWithOneOf(.html\,.htm\,.txt) ) and not IsProbablyBinary() storeFilter=( ContentTypeStartsWith(text/) or PathEndsWithOneOf(.html\,.htm\,.txt) ) and not IsProbablyBinary() schemeAuthorityDelay=10s ipDelay=2s maxUrls=500M bloomFilterPrecision=1E-8 seed=file:/some/path/eu.seed socketTimeout=60s connectionTimeout=60s fetchDataBufferByteSize=200K cookiePolicy=compatibility cookieMaxByteSize=2000 userAgent=BUbiNG (+http://law.di.unimi.it/BUbiNG.html<EMAIL_ADDRESS>robotsExpiration=1h responseBodyMaxByteSize=2M digestAlgorithm=MurmurHash3 startPaused=false workbenchMaxByteSize=512Mi urlCacheMaxByteSize=1Gi sieveSize=64Mi parserSpec=HTMLParser(MurmurHash3) keepAliveTime=1s
Note that almost all these properties are modifiable at runtime using the JMX interface.
There are two main activities related to the preparation of configuration files:
First of all, we suggest to run as root the command sysctl -p /etc/sysctl.d/bubing (where the file bubing contains the settings discussed before) as in our experience some of the parameters turn out to be not correctly set after boot, even in included in /etc/sysctl.d.
A BUbiNG is started using the {@link it.unimi.di.law.bubing.Agent} class, as in:
java -server -Xss256K -Xms20G -XX:+UseNUMA -XX:+UseConcMarkSweepGC \ -XX:+UseTLAB -XX:+ResizeTLAB -XX:NewRatio=4 -XX:MaxTenuringThreshold=15 -XX:+CMSParallelRemarkEnabled \ -verbose:gc -Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution \ -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime \ -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1 \ -Djava.rmi.server.hostname=<IP_ADDRESS> \ -Djava.net.preferIPv4Stack=true \ -Djgroups.bind_addr=<IP_ADDRESS> \ -Dlogback.configurationFile=bubing-logback.xml \ -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false \ it.unimi.di.law.bubing.Agent -h <IP_ADDRESS> -P eu.properties -g eu agent -n 2>err >out
The JVM options are of course just an example for a 64-core, 64 GiB hardware, and should be customized. It is important, however, that the thread stack size (-Xss) is set to a small value (as BUbiNG uses thousands of threads).
The IP address specified will be
used for JGroups and JMX communication: if you have an internal high-speed interface, this is the place to use it. Otherwise,
just use the standard IP address of the machine. This crawl will use a property file named eu.properties, start an
agent with name agent part of a crawl group named eu. Logback
will be configured by the file bubing-logback.xml
(see below).
In the standard BUbiNG setup, agents in the same crawl groups coordinates autonomously using consistent hashing, so if you want to perform a multi-agent crawl you must just be sure to have properly configured your hardware and JGroups so they work together. A simple way to check that this is true is to start the crawl in pause mode, check from the logs that all agents are visible to each other, and then start the crawl.
We use Logback for logging. We suggest a separate log for messages above the WARN level, to be able to find errors quickly. The file used in the command above could be as follows (assuming you want to write your logs in /tmp):
<?xml version="1.0" encoding="UTF-8"?> <configuration> <appender name="all" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>/tmp/bubing.log</file> <encoder> <pattern>%d %r %p [%t] %logger{1} - %m%n</pattern> </encoder> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <maxHistory>100</maxHistory> <FileNamePattern>/tmp/bubing.%d{yyyy-MM-dd_HH}.log</FileNamePattern> </rollingPolicy> </appender> <appender name="warn" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>/tmp/bubing.warn</file> <encoder> <pattern>%d %r %p [%t] %logger{1} - %m%n</pattern> </encoder> <filter class="ch.qos.logback.classic.filter.ThresholdFilter"> <level>WARN</level> </filter> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <maxHistory>100</maxHistory> <FileNamePattern>/tmp/bubing.%d.warn</FileNamePattern> </rollingPolicy> </appender> <root level="INFO"> <appender-ref ref="all"/> <appender-ref ref="warn"/> </root> <logger name="net.htmlparser.jericho" level="OFF"/> <logger name="it.unimi.di.law.bubing" level="INFO"/> <logger name="it.unimi.di.law.bubing.frontier.FetchingThread$DNSJavaClientConnectionManager" level="INFO"/> <logger name="org.apache.http.impl.client" level="WARN"/> <logger name="org.apache.http.wire" level="WARN"/> </configuration>
BUbiNG's object handling deeply violates the weak generational hypothesis, that is, the idea that most allocated objects die young. During a crawl, dozens (of even hundreds) of millions of small objects have a lifetime of minutes or hours, and then disappear. Using the default settings of the garbage collector leads to a constant promotion these objects to the old generation, causing frequent full collections that decrease progressively the crawl speed.
The settings suggested above modify radically the garbage collector behavior. One third of the available memory is allocated for new objects, and we try to delay tenuring as much as possible by swapping objects between the survivor spaces the maximum possible number of times (15, as the counter has only four bits). Note that frequent minor collections have little impact on the crawler, as even during collections the operating system is filling socket buffers, flushing file buffers, handling memory mapping, etc. We also require dynamic thread-local allocation buffers, as the allocation needs of, say, fetching threads and parsing threads are very different.
It is always a good idea to log garbage collection information (as suggested above) and to examine the resulting log using a tool like Chewiebug. Ideally, you should see frequent minor collection, and small sets of objects promoted to the old generation.
The actual values of the parameter should be chosen in the following order:
The last final judge of a correct configuration of the garbage collector is the stability of the crawling speed.
Once you have decided the amount of memory that you want to use for the crawler, you must decide how to divide this memory among BUbiNG's components. Most of the memory used by BUbiNG can be divided as follows:
BUbiNG's activity requires a lot of disk I/O. Some of the I/O activity, like the flushes of the {@linkplain it.unimi.di.law.bubing.sieve.MercatorSieve}, are stop-of-the-world events and require no particular care. However, there are three activities that happen concurrently during the crawling:
Ideally, these four activities should use different disks, and, if possible, {@link it.unimi.di.law.bubing.StartupConfiguration#frontierDir} should live on solid-state storage.
There are two sources of information about the inner works of BUbiNG: JMX and the logs. Most internals of BUbiNG are modifiable at runtime using JMX, and looking at the logs reveals a lot information about what's going on. We suggest to regularly grep for ERROR and check standard error to catch problems at an early stage.
If you just want to pause a crawl, there is a JMX {@link it.unimi.di.law.bubing.Agent#pause()} operation that will suspend crawling activity until {@link it.unimi.di.law.bubing.Agent#resume()} is invoked. Alternatively, you can {@linkplain it.unimi.di.law.bubing.Agent#stop() stop the crawl}, which will cause the crawl state to be snapped to disk. At that point, the crawl can restart by launching the agent with the same configuration but without the -n option.


| Number | Feature ID | Feature Name |
| 1 | B021 | Direct SQL |
| 2 | B031 | Basic dynamic SQL |
| 3 | B032 | Extended dynamic SQL |
| 4 | B032-01 | <describe input> statement |
| 5 | B033 | Untyped SQL-invoked function arguments |
| 6 | B034 | Dynamic specification of cursor attributes |
| 7 | B041 | Extensions to embedded SQL exception declarations |
| 8 | B051 | Enhanced execution rights |
| 9 | F032 | CASCADE drop behavior |
| 10 | F033 | ALTER TABLE statement: DROP COLUMN clause |
| 11 | F034 | Extended REVOKE statement |
| 12 | F034-01 | REVOKE statement performed by other than the owner of a schema object |
| 13 | F034-02 | REVOKE statement: GRANT OPTION FOR clause |
| 14 | F034-03 | REVOKE statement to revoke a privilege that the grantee has WITH GRANT OPTION |
| 15 | F052 | Intervals and datetime arithmetic |
| 16 | F053 | OVERLAPS predicate |
| 17 | F111 | Isolation levels other than SERIALIZABLE |
| 18 | F111-01 | READ UNCOMMITTED isolation level |
| 19 | F111-02 | READ COMMITTED isolation level |
| 20 | F111-03 | REPEATABLE READ isolation level |
| 21 | F121 | Basic diagnostics management |
| 22 | F121-01 | GET DIAGNOSTICS statement |
| 23 | F121-02 | SET TRANSACTION statement: DIAGNOSTICS SIZE clause |
| 24 | F171 | Multiple schemas per user |
| 25 | F191 | Referential delete actions |
| 26 | F222 | INSERT statement: DEFAULT VALUES clause |
| 27 | F231 | Privilege tables |
| 28 | F231-01 | TABLE_PRIVILEGES view |
| 29 | F231-02 | COLUMN_PRIVILEGES view |
| 30 | F231-03 | USAGE_PRIVILEGES view |
| 31 | F251 | Domain support |
| 32 | F262 | Extended CASE expression |
| 33 | F271 | Compound character literals |
| 34 | F281 | LIKE enhancements |
| 35 | F291 | UNIQUE predicate |
| 36 | F301 | CORRESPONDING in query expressions |
| 37 | F302 | INTERSECT table operator |
| 38 | F302-01 | INTERSECT DISTINCT table operator |
| 39 | F302-02 | INTERSECT ALL table operator |
| 40 | F304 | EXCEPT ALL table operator |
| 41 | F312 | MERGE statement |
| 42 | F321 | User authorization |
| 43 | F341 | Usage tables |
| 44 | F361 | Subprogram support |
| 45 | F381 | Extended schema manipulation |
| 46 | F381-01 | ALTER TABLE statement: ALTER COLUMN clause |
| 47 | F381-02 | ALTER TABLE statement: ADD CONSTRAINT clause |
| 48 | F381-03 | ALTER TABLE statement: DROP CONSTRAINT clause |
| 49 | F391 | Long identifiers |
| 50 | F392 | Unicode escapes in identifiers |
| 51 | F393 | Unicode escapes in literals |
| 52 | F401 | Extended joined table |
| 53 | F401-01 | NATURAL JOIN |
| 54 | F401-02 | FULL OUTER JOIN |
| 55 | F401-03 | UNION JOIN |
| 56 | F401-04 | CROSS JOIN |
| 57 | F402 | Named column joins for LOBs, arrays, and multisets |
| 58 | F411 | Time zone specification |
| 59 | F421 | National character |
| 60 | F431 | Read-only scrollable cursors |
| 61 | F431-01 | FETCH with explicit NEXT |
| 62 | F431-02 | FETCH FIRST |
| 63 | F431-03 | FETCH LAST |
| 64 | F431-04 | FETCH PRIOR |
| 65 | F431-05 | FETCH ABSOLUTE |
| 66 | F431-06 | FETCH RELATIVE |
| 67 | F441 | Extended set function support |
| 68 | F442 | Mixed column references in set functions |
| 69 | F451 | Character set definition |
| 70 | F461 | Named character sets |
| 71 | F491 | Constraint management |
| 72 | F502 | Enhanced documentation tables |
| 73 | F502-01 | SQL_SIZING_PROFILES view |
| 74 | F502-02 | SQL_IMPLEMENTATION_INFO view |
| 75 | F502-03 | SQL_PACKAGES view |
| 76 | F521 | Assertions |
| 77 | F531 | Temporary tables |
| 78 | F555 | Enhanced seconds precision |
| 79 | F561 | Full value expressions |
| 80 | F571 | Truth value tests |
| 81 | F591 | Derived tables |
| 82 | F611 | Indicator data types |
| 83 | F641 | Row and table constructors |
| 84 | F651 | Catalog name qualifiers |
| 85 | F661 | Simple tables |
| 86 | F671 | Subqueries in CHECK |
| 87 | F672 | Retrospective check constraints |
| 88 | F691 | Collation and translation |
| 89 | F692 | Enhanced collation support |
| 90 | F693 | SQL-session and client module collations |
| 91 | F701 | Referential update actions |
| 92 | F711 | ALTER domain |
| 93 | F721 | Deferrable constraints |
| 94 | F731 | INSERT column privileges |
| 95 | F741 | Referential MATCH types |
| 96 | F751 | View CHECK enhancements |
| 97 | F761 | Session management |
| 98 | F771 | Connection management |
| 99 | F781 | Self-referencing operations |
| 100 | F791 | Insensitive cursors |
| 101 | F801 | Full set function |
| 102 | F813 | Extended flagging - Part 1, Subclause 8.1.4, "SQL flagger": With 'level of flagging' specified to be Core SQL Flagging and 'extent of checking' specified to be Catalog Lookup |
| 103 | F821 | Local table references |
| 104 | F831 | Full cursor update |
| 105 | F831-01 | Updateable scrollable cursors |
| 106 | F831-02 | Updateable ordered cursors |
| 107 | S023 | Basic structured types |
| 108 | S024 | Enhanced structured types |
| 109 | S025 | Final structured types |
| 110 | S026 | Self-referencing structured types |
| 111 | S027 | Create method by specific method name |
| 112 | S028 | Permutable UDT options list |
| 113 | S041 | Basic reference types |
| 114 | S043 | Enhanced reference types |
| 115 | S051 | Create table of type |
| 116 | S071 | SQL paths in function and type name resolution |
| 117 | S081 | Subtables |
| 118 | S091 | Basic array support |
| 119 | S091-01 | Arrays of built-in data types |
| 120 | S091-02 | Arrays of distinct types |
| 121 | S091-03 | Array expressions |
| 122 | S092 | Arrays of user-defined types |
| 123 | S094 | Arrays of reference types |
| 124 | S095 | Array constructors by query |
| 125 | S096 | Optional array bounds |
| 126 | S097 | Array element assignment |
| 127 | S111 | ONLY in query expressions |
| 128 | S151 | Type predicate |
| 129 | S161 | Subtype treatment |
| 130 | S162 | Subtype treatment for references |
| 131 | S201 | SQL-invoked routines on arrays |
| 132 | S201-01 | Array parameters |
| 133 | S201-02 | Array as result type of functions |
| 134 | S202 | SQL-invoked routines on multisets |
| 135 | S211 | User-defined cast functions |
| 136 | S231 | Structured type locators |
| 137 | S232 | Array locators |
| 138 | S233 | Multiset locators |
| 139 | S241 | Transform functions |
| 140 | S242 | Alter transform statement |
| 141 | S251 | User-defined orderings |
| 142 | S261 | Specific type method |
| 143 | S271 | Basic multiset support |
| 144 | S272 | Multisets of user-defined types |
| 145 | S274 | Multisets of reference types |
| 146 | S275 | Advanced multiset support |
| 147 | S281 | Nested collection types |
| 148 | T011 | Timestamp in Information Schema |
| 149 | T031 | BOOLEAN data type |
| 150 | T041 | Basic LOB data type support |
| 151 | T041-01 | BLOB data type |
| - Subclause 5.2, "<token> and <separator>": The <reserved word>s BINARY, BLOB, LARGE, and OBJECT | ||
| - Subclause 5.3, "<literal>": <binary string literal> | ||
| - Subclause 6.1, "<data type>": The BINARY LARGE OBJECT data type | ||
| - Subclause 6.28, "<string value expression>": For values of type BINARY LARGE OBJECT | ||
| - Subclause 13.6, "Data type correspondences": Type correspondences for BINARY LARGE OBJECT for all supported languages | ||
| 152 | T041-02 | CLOB data type |
| - Subclause 5.2, "<token> and <separator>": The <reserved word>s CHARACTER, CLOB, LARGE, and OBJECT | ||
| - Subclause 6.1, "<data type>": The CHARACTER LARGE OBJECT data type | ||
| - Subclause 6.28, "<string value expression>": For values of type CHARACTER LARGE OBJECT | ||
| - Subclause 13.6, "Data type correspondences": Type correspondences for CHARACTER LARGE OBJECT for all supported languages | ||
| - The automatic casting among the character types supported by subfeature E021-11 is extended to support the CHARACTER LARGE OBJECT type | ||
| 153 | T041-03 | POSITION, LENGTH, LOWER, TRIM, UPPER, and SUBSTRING functions for LOB data types |
| - Subclause 6.27, "<numeric value function>": The <position expression> for expressions of type BINARY LARGE OBJECT and CHARACTER LARGE OBJECT | ||
| - Subclause 6.27, "<numeric value function>": The <char length function> for expressions of type CHARACTER LARGE OBJECT | ||
| - Subclause 6.27, "<numeric value function>": The <octet length function> for expressions of type BINARY LARGE OBJECT and CHARACTER LARGE OBJECT | ||
| - Subclause 6.29, "<string value function>": The <fold> function for expressions of type CHARACTER LARGE OBJECT | ||
| - Subclause 6.29, "<string value function>": The <trim function> for expressions of type CHARACTER LARGE OBJECT | ||
| - Subclause 6.29, "<string value function>": The <blob trim function> | ||
| - Subclause 6.29, "<string value function>": The <character substring function> for expressions of type CHARACTER LARGE OBJECT | ||
| - Subclause 6.29, "<string value function>": The <blob substring function> | ||
| 154 | T041-04 | Concatenation of LOB data types |
| - Subclause 6.28, "<string value expression>": The <concatenation> expression for expressions of type CHARACTER LARGE OBJECT | ||
| - Subclause 6.28, "<string value expression>": The <blob concatenation> expression | ||
| 155 | T041-05 | LOB locator: non-holdable |
| - Subclause 13.3, "<externally-invoked procedure>": <locator indication> | ||
| - Subclause 14.14, "<free locator statement>" | ||
| 156 | T042 | Extended LOB data type support |
| 157 | T051 | Row types |
| 158 | T052 | MAX and MIN for row types |
| 159 | T053 | Explicit aliases for <all fields reference> |
| 160 | T061 | UCS support |
| 161 | T071 | BIGINT data type |
| 162 | T111 | Updatable joins, unions, and columns |
| 163 | T121 | WITH (excluding RECURSIVE) in query expression |
| 164 | T131 | Recursive query |
| 165 | T141 | SIMILAR predicate |
| 166 | T151 | DISTINCT predicate |
| 167 | T171 | LIKE clause in table definition |
| 168 | T172 | AS subquery clause in table definition |
| 169 | T173 | Extended LIKE clause in table definition |
| 170 | T174 | Identity columns |
| 171 | T175 | Generated columns |
| 172 | T176 | Sequence generator support |
| 173 | T191 | Referential action RESTRICT |
| 174 | T201 | Comparable data types for referential constraints |
| 175 | T211 | Basic trigger capability |
| 176 | T211-01 | Triggers activated on UPDATE, INSERT, or DELETE of one base table. |
| 177 | T211-02 | BEFORE triggers |
| 178 | T211-03 | AFTER triggers |
| 179 | T211-04 | FOR EACH ROW triggers |
| 180 | T211-05 | Ability to specify a search condition that must be True before the trigger is invoked. |
| 181 | T211-06 | Support for run-time rules for the interaction of triggers and constraints. |
| 182 | T211-07 | TRIGGER privilege |
| 183 | T211-08 | Multiple triggers for the same the event are executed in the order in which they were created in the catalog. |
| 184 | T212 | Enhanced trigger capability |
| 185 | T231 | Sensitive cursors |
| 186 | T241 | START TRANSACTION statement |
| 187 | T242 | Optional transaction modes in START TRANSACTION |
| 188 | T251 | SET TRANSACTION statement: LOCAL option |
| 189 | T261 | Chained transactions |
| 190 | T271 | Savepoints |
| 191 | T272 | Enhanced savepoint management |
| 192 | T281 | SELECT privilege with column granularity |
| 193 | T301 | Functional dependencies |
| 194 | T312 | OVERLAY function |
| 195 | T322 | Overloading of SQL-invoked functions and procedures |
| 196 | T323 | Explicit security for external routines |
| 197 | T324 | Explicit security for SQL routines |
| 198 | T325 | Qualified SQL parameter references |
| 199 | T326 | Table functions |
| 200 | T331 | Basic roles |
| 201 | T332 | Extended roles |
| 202 | T351 | Bracketed SQL comments (/*...*/ comments) |
| 203 | T431 | Extended grouping capabilities |
| 204 | T432 | Nested and concatenated GROUPING SETS |
| 205 | T433 | Multiargument GROUPING function |
| 206 | T434 | GROUP BY DISTINCT |
| 207 | T441 | ABS and MOD functions |
| 208 | T461 | Symmetric <between predicate> |
| 209 | T471 | Result sets return value |
| 210 | T491 | LATERAL derived table |
| 211 | T501 | Enhanced EXISTS predicate |
| 212 | T511 | Transaction counts |
| 213 | T551 | Optional key words for default syntax |
| 214 | T561 | Holdable locators |
| 215 | T571 | Array-returning external SQL-invoked functions |
| 216 | T572 | Multiset-returning external SQL-invoked functions |
| 217 | T581 | Regular expression substring function |
| 218 | T591 | UNIQUE constraints of possibly null columns |
| 219 | T601 | Local cursor references |
| 220 | T611 | Elementary OLAP operations |
| 221 | T612 | Advanced OLAP operations |
| 222 | T613 | Sampling |
| 223 | T621 | Enhanced numeric functions |
| 224 | T631 | IN predicate with one list element |
| 225 | T641 | Multiple column assignment |
| 226 | T651 | SQL-schema statements in SQL routines |
| 227 | T652 | SQL-dynamic statements in SQL routines |
Please send feedback to Jonathan Leffler, variously<EMAIL_ADDRESS>or<EMAIL_ADDRESS>.
@(#)$Id: sql-2003-noncore-features.html,v 1.2 2005/07/18 22:51:30 jleffler Exp $


{% extends 'base.html' %}
{% set crumbs = [(url('landings.get_involved'), _('Help Other Users')), (none, _('Help Us Write Help Articles'))] %}
{% set styles = ('landings-get-involved',) %}
{% set canonical_url = canonicalize(viewname='landings.get_involved_kb') %}
{% block below_main %}
{{ _('Help us write help articles!') }}
{{ _('Share your wisdom with the world.') }}
{% trans %}
If you like writing and teaching then the Knowledge Base is the
place for you. We need contributors that can write, edit or proof
read articles in English. Thousands of people are accessing our
Knowledge Base every week. Imagine how many will be helped by you!
{% endtrans %}
{% set user_is_contributor = is_contributor(user) %}
{% set signup_headline = _('Improve our help articles') %}
{% include 'landings/includes/register_contributor.html' %}
{{ _('One article can be viewed by 400 million users and translated into 70+ languages') }}
{{ _('How it works') }}
{% set message = _('Sign up as a volunteer') %}
{% if not user_is_contributor %}
{{ message }}
{% else %}
{{ message }}
{% endif %}


Apache Flink Code Style and Quality Guide — Common Rules
1. Copyright
Each file must include the Apache license information as a header.
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
We recommend to follow the IDE Setup Guide to get IDE tooling configured.
Warnings
- We strive for zero warnings
- Even though there are many warnings in existing code, new changes should not add any additional compiler warnings
- If it is not possible to address the warning in a sane way (in some cases when working with generics) add an annotation to suppress the warning
- When deprecating methods, check that this does not introduce additional warnings
Golden rule: Comment as much as necessary to support code understanding, but don’t add redundant information.
Think about
- What is the code doing?
- How does the code do this?
- Why is the code like that?
The code alone should explain as much as possible the “what” and the “how”
- Use JavaDocs to describe the roles of classes and the contracts of methods, in cases where the contract is not obvious or intuitive from the method name (the “what”).
- The flow of the code should give a good description of the “how”.
Think of variable and method names as part of the code documenting itself.
- It often makes reading the code easier if larger blocks that form a unit are moved into a private method with a descriptive name of what that block is doing
In-code comments help explain the “why”
- For example
// this specific code layout helps the JIT to better do this or that
- Or
// nulling out this field here means future write attempts are fail-fast
- Or
// for arguments with which this method is actually called, this seemingly naive approach works actually better than any optimized/smart version
In-code comments should not state redundant information about the “what” and “how” that is already obvious in the code itself.
JavaDocs should not state meaningless information (just to satisfy the Checkstyle checker).
Don’t:
/**
* The symbol expression.
*/
public class CommonSymbolExpression {}
Do:
/**
* An expression that wraps a single specific symbol.
* A symbol could be a unit, an alias, a variable, etc.
*/
public class CommonSymbolExpression {}
Branches and Nesting
Avoid deep nesting of scopes, by flipping the if condition and exiting early.
Don’t:
if (a) {
if (b) {
if (c) {
the main path
}
}
}
Do
if (!a) {
return ..
}
if (!b) {
return ...
}
if (!c) {
return ...
}
the main path
4. Design and Structure
While it is hard to exactly specify what constitutes a good design, there are some properties that can serve as a litmus test for a good design. If these properties are given, the chances are good that the design is going into a good direction. If these properties cannot be achieved, there is a high probability that the design is flawed.
Immutability and Eager Initialization
- Try to use immutable types where possible, especially for APIs, messages, identifiers, properties, configuration, etc.
- A good general approach is to try and make as many fields of a class
final
as possible.
- Classes that are used as keys in maps should be strictly immutable and only have
final
fields (except maybe auxiliary fields, like lazy cached hash codes).
- Eagerly initialize classes. There should be no
init()
or setup()
methods. Once the constructor completes, the object should be usable.
Nullability of the Mutable Parts
For nullability, the Flink codebase aims to follow these conventions:
- Fields, parameters, and return types are always non-null, unless indicated otherwise
- All fields, parameters and method types that can be null should be annotated with
@javax.annotation.Nullable
.
That way, you get warnings from IntelliJ about all sections where you have to reason about potential null values.
- For all mutable (non-final) fields that are not annotated, the assumption is that while the field value changes, there always is a value.
- This should be double check whether these can in fact not be null throughout the lifetime of the object.
Note: This means that @Nonnull
annotations are usually not necessary, but can be used in certain cases to override a previous annotation, or to point non-nullability out in a context where one would expect a nullable value.
Optional
is a good solution as a return type for method that may or may not have a result, so nullable return types are good candidates to be replaced with Optional
.
See also usage of Java Optional.
Avoid Code Duplication
- Whenever you are about to copy/paste some code, or reproduce a similar type of functionality in a different place, think about the ways how to refactor/reuse/abstract the changes to avoid the duplication.
- Common behavior between different specializations should be shared in a common component (or a shared superclass).
- Always use “private static final” constants instead of duplicating strings or other special values at different locations. Constants should be declared in the top member area of a class.
Design for Testability
Code that is easily testable typically has good separation of concerns and is structured to be reusable outside the original context (by being easily reusable in tests).
A good summary or problems / symptoms and recommended refactoring is in the PDF linked below.
Please note that while the examples in the PDF often use a dependency injection framework (Guice), it works in the same way without such a framework.
http://misko.hevery.com/attachments/Guide-Writing%20Testable%20Code.pdf
Here is a compact summary of the most important aspects.
Inject dependencies
Reusability becomes easier if constructors don’t create their dependencies (the objects assigned to the fields), but accept them as parameters.
- Effectively, constructors should have no
new
keyword.
- Exceptions are creating a new empty collection (
new ArrayList<>()
) or similar auxiliary fields (objects that have only primitive dependencies).
To make instantiation easy / readable, add factory methods or additional convenience constructors to construct whole object with dependencies.
In no case should it ever be required to use a reflection or a “Whitebox” util to change the fields of an object in a test, or to use PowerMock to intercept a “new” call and supply a mock.
Avoid “too many collaborators”
If you have to take a big set of other components into account during testing (“too many collaborators”), consider refactoring.
The component/class you want to test probably depends on another broad component (and its implementation), rather than on the minimal interface (abstraction) required for its work.
In that case, segregate the interfaces (factor out the minimal required interface) and supply a test stub in that case.
- For example, if testing a S3RecoverableMultiPartUploader requires actual S3 access
then the S3 access should be factored out into an interface and test should replace it by a test stub
- This naturally requires to be able to inject dependencies (see above)
⇒ Please note that these steps often require more effort in implementing tests (factoring out interfaces, creating dedicated test stubs), but make the tests more resilient to changes in other components, i.e., you do not need to touch the tests when making unrelated changes.
Write targeted tests
- Test contracts not implementations: Test that after a sequence of actions, the components are in a certain state, rather than testing that the components followed a sequence of internal state modifications.
- For example, a typical antipattern is to check whether one specific method was called as part of the test
-
A way to enforce this is to try to follow the Arrange, Act, Assert test structure when writing a unit test (https://xp123.com/articles/3a-arrange-act-assert/)
This helps to communicate the intention of the test (what is the scenario under test) rather than the mechanics of the tests. The technical bits go to a static methods at the bottom of the test class.
Example of tests in Flink that follow this pattern are:
Avoid Mockito - Use reusable test implementations
- Mockito-based tests tend to be costly to maintain in the long run by encouraging duplication of functionality and testing for implementation rather than effect
- Instead, create reusable test implementations and utilities
- That way, when some class changes, we only have to update a few test utils or mocks
We can conceptually distinguish between code that “coordinates” and code that “processes data”. Code that coordinates should always favor simplicity and cleanness. Data processing code is highly performance critical and should optimize for performance.
That means still applying the general idea of the sections above, but possibly forgoing some aspects in some place, in order to achieve higher performance.
Which code paths are Data Processing paths?
- Per-record code paths: Methods and code paths that are called for each record. Found for example in Connectors, Serializers, State Backends, Formats, Tasks, Operators, Metrics, runtime data structures, etc.
- I/O methods: Transferring messages or chunks of data in buffers. Examples are in the RPC system, Network Stack, FileSystems, Encoders / Decoders, etc.
Things that performance critical code may do that we would otherwise avoid
- Using (and reusing) mutable objects to take pressure off the GC (and sometimes help with cache locality), thus forgoing the strive for immutability.
- Using primitive types, arrays of primitive types, or MemorySegment/ByteBuffer and encoding meaning into the primitive types and byte sequences, rather than encapsulating the behavior in dedicated classes and using objects.
- Structuring the code to amortize expensive work (allocations, lookups, virtual method calls, …) across multiple records, for example by doing the work once per buffer/bundle/batch.
- Code layout optimized for the JIT rather than for readability. Examples are inlining fields from other classes (in cases where it is doubtful whether the JIT would do that optimization at runtime), or structuring code to help the JIT compiler with inlining, loop unrolling, vectorization, etc.
5. Concurrency and Threading
Most code paths should not require any concurrency. The right internal abstractions should obviate the need for concurrency in almost all cases.
- The Flink core and runtime use concurrency to provide these building blocks.
Examples are in the RPC system, Network Stack, in the Task’s mailbox model, or some predefined Source / Sink utilities.
- We are not fully there, but any new addition that introduces implements its own concurrency should be under scrutiny, unless it falls into the above category of core system building blocks.
- Contributors should reach out to committers if they feel they need to implement concurrent code to see if there is an existing abstraction/building-block, or if one should be added.
When developing a component think about threading model and synchronization points ahead.
- For example: single threaded, blocking, non-blocking, synchronous, asynchronous, multi threaded, thread pool, message queues, volatile, synchronized block/methods, mutexes, atomics, callbacks, …
- Getting those things right and thinking about them ahead is even more important than designing classes interfaces/responsibilities, since it’s much harder to change later on.
Try to avoid using threads all together if possible in any way.
- If you feel you have a case for spawning a thread, point this out in the pull request as something to be explicitly reviewed.
Be aware that using threads is in fact much harder than it initially looks
- Clean shutdown of threads is very tricky.
- Handling interruptions in a rock solid fashion (avoid both slow shutdown and live locks) requires almost a Java Wizard
- Ensuring clean error propagation out of threads in all cases needs thorough design.
- Complexity of multi-threaded application/component/class grows exponentially, with each additional synchronisation point/block/critical section. Your code initially might be easy enough to understand, but can quickly grow beyond that point.
- Proper testing of multithreaded code is basically impossible, while alternative approaches (like asynchronous code, non-blocking code, actor model with message queues) are quite easy to test.
- Usually multi-threaded code is often even less efficient compared to alternative approaches on modern hardware.
Be aware of the java.util.concurrent.CompletableFuture
- Like with other concurrent code, there should rarely be the need to use a CompletableFuture
- Completing a future would also complete on the calling thread any chained futures that are waiting for the result to be completed, unless a completion executor specified explicitly.
- This can be intentional, if the entire execution should be synchronous / single-threaded, as for example in parts of the Scheduler / ExecutionGraph.
- Flink even makes use of a “main-thread executor” to allow calling chained handlers in the same thread as a single-threaded RPC endpoint runs
- This can be unexpected, if the thread that completes the future is a sensitive thread.
- It may be better to use
CompletableFuture.supplyAsync(value, executor)
in that case, instead of future.complete(value)
when an executor is available
- When blocking on a future awaiting completion, always supply a timeout for a result instead of waiting indefinitely, and handle timeouts explicitly.
- Use
CompletableFuture.allOf()
/anyOf()
, ExecutorCompletionService
, or org.apache.flink.runtime.concurrent.FutureUtils#waitForAll
if you need to wait for: all the results/any of the results/all the results but handled by (approximate) completion order.
6. Dependencies and Modules
- Keep the dependency footprint small
- The more dependencies the harder it gets for the community to manage them as a whole.
- Dependency management includes dependency conflicts, maintaining licenses and related notices, and handling security vulnerabilities.
- Discuss whether the dependency should be shaded/relocated to avoid future conflicts.
- Don’t add a dependency for just one method
- Use Java built-in means if possible.
- If the method is Apache-licensed, you can copy the method into a Flink utility class with proper attribution.
- Declaration of dependencies
- Declare dependencies that you explicitly rely on, whether it provides classes you directly import and use or it’s something that provides a service you directly use, like Log4J.
- Transitive dependencies should only supply dependencies that are needed at runtime but that you don’t use yourself.
- [source]
- Location of classes in the Maven modules
- Whenever you create a new class, think about where to put it.
- A class might be used by multiple modules in the future and might belong into a
common
module in this case.


Use this request to set the default settings for sending SMS messages and receiving daily SMS usage reports.
You can override some of these settings for a single message when you use the publish action with the MessageAttributes.entry.N parameter. For more information, see Sending an SMS Message in the Amazon SNS Developer Guide .
See also: AWS API Documentation
set-sms-attributes
--attributes <value>
[--cli-input-json <value>]
[--generate-cli-skeleton <value>]
The default settings for sending SMS messages from your account. You can set values for the following attribute names:
MonthlySpendLimit – The maximum amount in USD that you are willing to spend each month to send SMS messages. When Amazon SNS determines that sending an SMS message would incur a cost that exceeds this limit, it stops sending SMS messages within minutes.Warning
Amazon SNS stops sending SMS messages within minutes of the limit being crossed. During that interval, if you continue to send SMS messages, you will incur costs that exceed your limit.
By default, the spend limit is set to the maximum allowed by Amazon SNS. If you want to exceed the maximum, contact AWS Support or your AWS sales representative for a service limit increase.
DeliveryStatusIAMRole – The ARN of the IAM role that allows Amazon SNS to write logs about SMS deliveries in CloudWatch Logs. For each SMS message that you send, Amazon SNS writes a log that includes the message price, the success or failure status, the reason for failure (if the message failed), the message dwell time, and other information.
DeliveryStatusSuccessSamplingRate – The percentage of successful SMS deliveries for which Amazon SNS will write logs in CloudWatch Logs. The value can be an integer from 0 - 100. For example, to write logs only for failed deliveries, set this value to 0 . To write logs for 10% of your successful deliveries, set it to 10 .
DefaultSenderID – A string, such as your business brand, that is displayed as the sender on the receiving device. Support for sender IDs varies by country. The sender ID can be 1 - 11 alphanumeric characters, and it must contain at least one letter.
DefaultSMSType – The type of SMS message that you will send by default. You can assign the following values:
- Promotional – (Default) Noncritical messages, such as marketing messages. Amazon SNS optimizes the message delivery to incur the lowest cost.
- Transactional – Critical messages that support customer transactions, such as one-time passcodes for multi-factor authentication. Amazon SNS optimizes the message delivery to achieve the highest reliability.
UsageReportS3Bucket – The name of the Amazon S3 bucket to receive daily SMS usage reports from Amazon SNS. Each day, Amazon SNS will deliver a usage report as a CSV file to the bucket. The report includes the following information for each SMS message that was successfully delivered by your account:
- Time that the message was published (in UTC)
- Message ID
- Destination phone number
- Message type
- Delivery status
- Message price (in USD)
- Part number (a message is split into multiple parts if it is too long for a single message)
- Total number of parts
To receive the report, the bucket must have a policy that allows the Amazon SNS service principle to perform the s3:PutObject and s3:GetBucketLocation actions.
For an example bucket policy and usage report, see Monitoring SMS Activity in the Amazon SNS Developer Guide .
Shorthand Syntax:
KeyName1=string,KeyName2=string
JSON Syntax:
{"string": "string"
...}
--cli-input-json (string) Performs service operation based on the JSON string provided. The JSON string follows the format provided by --generate-cli-skeleton. If other arguments are provided on the command line, the CLI values will override the JSON-provided values.
--generate-cli-skeleton (string) Prints a JSON skeleton to standard output without sending an API request. If provided with no value or the value input, prints a sample input JSON that can be used as an argument for --cli-input-json. If provided with the value output, it validates the command inputs and returns a sample output JSON for that command.


Preparation time
over 2 hours
Cooking time
less than 10 mins
Serves
Serves 4
Try this fruity dessert in the winter months, when pomegranate is in season.
For the panna cotta, soak the gelatine leaves in a bowl of cold water until softened and pliable, about 3-4 minutes.
Meanwhile, place the pomegranate juice into a small pan and gently heat, before removing from the heat.
Remove the gelatine from the water, squeezing out any excess liquid, and add the leaves to the hot pomegranate juice. Stir until dissolved.
Meanwhile, place the sugar, orange zest and 350ml/12fl oz of the cream into a saucepan and bring to a simmer, stirring to dissolve all the sugar.
Pour the pomegranate juice into the cream mixture, stirring all the time. Set aside to cool a little.
Place the remaining cream into a bowl and whisk until soft peaks form when the whisk is removed from the bowl.
Fold the whisked cream into the set-aside mixture. Pour the mixture through a sieve into a jug.
Arrange the pomegranate seeds on the base of four dariole moulds or ramekins. Place the moulds on a plate or baking sheet before pouring the mixture into each mould. Leave in the fridge for 1-2 hours, or until set.
For the pomegranate caramel, heat a frying pan until hot, add the orange segments and sear briefly. Remove the orange from the pan and set aside.
Add the pomegranate seeds, sugar and juice to the pan and cook for 2-3 minutes until thickened and the sugar has melted. Return the oranges to the pan and keep warm.
To serve, remove the moulds from the fridge and dip the base of each mould into a bowl of boiled water briefly to help loosen it.
Gently slide a knife around the edge of each panna cotta and turn out onto a plate. Spoon the pomegranate caramel around the side and serve at once.


HANGMAN for SUPERHEROS!!!
Press any Key to Start
Wins
0
Current Word
Guesses Remaining
Letters Already Guessed


Luigi's Mansion 3 Sets A New Launch Month Sales Record In The United States
It's outdone the original game
by Liam DoolanLuigi's Mansion 3 was unable to top the Japanese and UK charts, being outperformed by Persona 5 Royal, Call of Duty: Modern Warfare, Death Stranding and FIFA 20. Despite this, it still managed to become the fastest-selling physical Switch game of 2019 in the UK, surpassing The Legend of Zelda: Link's Awakening.
Over in the United States, the Nintendo exclusive has performed much the same. According to NPD Video Game Industry Analyst Mat Piscatella, Luigi's Mansion 3 was the third best-selling game during the month of October within this region, behind Call of Duty: Modern Warfare and The Outer Worlds.
Piscatella went onto explain how Luigi's latest outing was also October's best-selling Switch title and most of all – set a new launch month sales record for the Luigi's Mansion series. While no exact sales figures were provided, Mat revealed the previous best-selling entry in the series was the original Luigi's Mansion game.
Did you pick up Luigi's Mansion in October? Tell us below.
[via twitter.com]


--- title: Guides Index sections: - name: Schema - name: Queries - name: Type Definitions - name: Authorization - name: Fields - name: Mutations - name: Errors - name: Pagination - name: Relay - name: Dataloader - name: Subscriptions - name: GraphQL Pro - name: GraphQL Pro - OperationStore - name: GraphQL Pro - Defer - name: GraphQL Enterprise - Rate Limiters - name: GraphQL Enterprise - Object Cache - name: GraphQL Enterprise - Changesets - name: JavaScript Client - name: Language Tools - name: Testing - name: Other ---
{{ guide.title }} {{ guide.desc }}


This project the user will input a time and train company and it will save the information on the webpages. I used HTML, CSS, Java Script, and jQuery to set it up. I used HTML and CSS to set up input searches and background design. I used Java Script and jQuery to display responses and to let the user know if they put the right information in. I used firebase to save the user input and to display the information the user input.
This application allows the user to input a movie title and a Gif from the movie will pop. The user can also pause and play Gif just by clicking on it. I used HTML and CSS to set up the design of the application. I used JAVA and jQuery for most of the responses and function. I used the gif API from https://giphy.com/. I used a jQuery function AJAX to pull in the information necessary for the application to work.
Eventide is an app that allows user to find event near their location and to find restaurants near the event. How it works: Input your city and date, and events will populate below. Click on the event link to buy tickets and get additional information. Click the Nearby Restaurants button to show restaurants close by. Click on the link to check out the menu or book a reservation. Using HTML and CSS to design the back ground and search boxes. Used two API’s seekgeek, and Zamato. I used jQuery function AJAX to pull in information from the API. Also used firebase to save user review on the app.
This project is a game using Java script. User are given four buttons (characters) and the enemy number (total; number needed to defeat enemy). User will have to click on the buttons and are given a random number amount between 12-1 and each button will add on. The user need to match the enemy number without going over.


Class ImageSource (v2.2.2)
External image source (Google Cloud Storage or web URL image location).
Generated from protobuf message google.cloud.vision.v1.ImageSource
Methods
__construct
Constructor.
| Name | Description |
| data |
array
Optional. Data for populating the Message object. |
| ↳ gcs_image_uri |
string
Use |
| ↳ image_uri |
string
The URI of the source image. Can be either: 1. A Google Cloud Storage URI of the form |
getGcsImageUri
Use image_uri
instead.
The Google Cloud Storage URI of the form
gs://bucket_name/object_name
. Object versioning is not supported. See
Google Cloud Storage Request
URIs for more info.
Generated from protobuf field string gcs_image_uri = 1;
| Type | Description |
| string |
setGcsImageUri
Use image_uri
instead.
The Google Cloud Storage URI of the form
gs://bucket_name/object_name
. Object versioning is not supported. See
Google Cloud Storage Request
URIs for more info.
Generated from protobuf field string gcs_image_uri = 1;
| Name | Description |
| var |
string
|
| Type | Description |
| $this |
getImageUri
The URI of the source image. Can be either:
A Google Cloud Storage URI of the form
gs://bucket_name/object_name
. Object versioning is not supported. See Google Cloud Storage Request URIs for more info.A publicly-accessible image HTTP/HTTPS URL. When fetching images from HTTP/HTTPS URLs, Google cannot guarantee that the request will be completed. Your request may fail if the specified host denies the request (e.g. due to request throttling or DOS prevention), or if Google throttles requests to the site for abuse prevention. You should not depend on externally-hosted images for production applications. When both
gcs_image_uri
andimage_uri
are specified,image_uri
takes precedence.
Generated from protobuf field string image_uri = 2;
| Type | Description |
| string |
setImageUri
The URI of the source image. Can be either:
A Google Cloud Storage URI of the form
gs://bucket_name/object_name
. Object versioning is not supported. See Google Cloud Storage Request URIs for more info.A publicly-accessible image HTTP/HTTPS URL. When fetching images from HTTP/HTTPS URLs, Google cannot guarantee that the request will be completed. Your request may fail if the specified host denies the request (e.g. due to request throttling or DOS prevention), or if Google throttles requests to the site for abuse prevention. You should not depend on externally-hosted images for production applications. When both
gcs_image_uri
andimage_uri
are specified,image_uri
takes precedence.
Generated from protobuf field string image_uri = 2;
| Name | Description |
| var |
string
|
| Type | Description |
| $this |


I develop visual identities and communication styles for projects and organisations so they could be better seen, understood and remembered. I have designed in print, screen and the three dimensional environment for a number of FTSE 250 and art-based clients. I work directly for design consultancies and a small group of clients. My work and opinions have been published and exhibited in London, Birmingham and Singapore.
Work includes Branding — Logo, Visual Identity, Website and Communication. Art Exhibitions — Exhibition Visual Identity, Exhibition Space and Catalogue. Books — Coﬀee Table Books, Corporate Commemorative Books, Art Books.
London 2001 — 2009 · Singapore 2009 — 2016 · Stockholm 2016 — now
Central Saint Martins & London College of Printing (now London College of Communication) BA (Hons) Graphic and Media Design
2001-2005 <<<<<<< HEAD


--- layout: archive title: Zulip Chat Archive permalink: /stream/113489-new-members/topic/defining.20a.20predicate.html ---
Hi,
I do not have a lot of experience using computer theorem provers, and I have just started trying to learn Lean. As practice I am attempting to formalize and prove that the Principle of Induction implies the Principle of Induction from a Starting Point. So far I have:
open nat -- Principle of Induction -> Principle of Induction from a Starting Point example : ( forall P : nat -> Prop, ( ( P 0 /\ ( forall n, ( P n -> P (n + 1) ) ) ) -> ( forall n, P n ) ) ) -> ( forall Q : nat -> Prop, forall m, ( ( Q m /\ ( forall n, ( ( n >= m ) -> ( Q n -> Q (n + 1) ) ) ) ) -> ( forall n, ( ( n >= m ) -> Q n ) ) ) ) := assume a1 : forall P : nat -> Prop, ( ( P 0 /\ ( forall n, ( P n -> P (n + 1) ) ) ) -> ( forall n, P n ) ), assume Q : nat -> Prop, assume m : nat, assume a2 : ( Q m /\ ( forall n, ( ( n >= m ) -> ( Q n -> Q (n + 1) ) ) ) ), have s1 : Q m, from and.left a2, have s2 : forall n, ( ( n >= m ) -> ( Q n -> Q (n + 1) ) ), from and.right a2, have s3 : forall n, ( Q (n + m) -> Q ((n + m) + 1) ), from assume n : nat, have s4 : ( ( (n + m) >= m ) -> ( Q (n + m) -> Q ((n + m) + 1) ) ), from s2 (n + m), have s5 : (n + m) >= m, from sorry, show ( Q (n + m) -> Q ((n + m) + 1) ), from s4 s5, -- define P' n to hold if and only if Q (m + n) holds -- then P' 0 holds by s1, and forall n, ( P' n -> P' (n + 1) ) holds by s3 -- then forall n, P' n holds by a1 -- then Q (m + n) holds for all n -- then Q holds for all n >= m
I am currently trying to define a predicate P' n : nat -> Prop that holds if and only if the predicate Q holds for m + n. Is there any chance that someone could show me how this can be done?
Thank you very much for your time,
Patrick
Hi @Patrick Thomas Welcome to Lean (and Zulip).
This is certainly possible.
By the way (depending on taste) you can make your code a bit more readable by using unicode.
For example, you can type \and
or \to
to get nice symbols for /\
and ->
.
Thank you.
Are you using mathlib?
Because there must be some lemma in data/nat/basic.lean
that will tell you that (n - m) + m = n
if n ≥ m
.
No. I just read about it a couple of minutes ago actually.
Aah, it will be very helpful. There's tons of useful little facts in there.
Do you have a CS or a maths background, or both?
I did an undergraduate degree in CS and physics, and have a strong interest in math. I started studying mathematical logic about a year ago in an attempt to understand computer proof assistants.
Ok, cool. There's a nice mix of CS and math in this community.
Because there must be some lemma in
data/nat/basic.lean
that will tell you that(n - m) + m = n
ifn ≥ m
.
I'm sorry, I'm not certain how I would use this in the proof. Is this in relation to step s5?
@Patrick Thomas I would first state something like let P' : Prop := foobar,
and then have quux : Q \iff P' := xyzzy,
In that latter proof you will need some lemma from mathlib, I guess.
To define something in a proof, you can use let
. In this case let P' := λ n, Q (m + n)
will do. Here is a short proof following mathlib style. This is probably too dense for a first cut but I'm sure someone around here can unpack this a bit.
example (H : ∀ P : nat → Prop, P 0 → (∀ n, P n → P (n + 1)) → ∀ n, P n) (Q : nat → Prop) (m) (h₁ : Q m) (h₂ : ∀ n ≥ m, Q n → Q (n + 1)) (n) (mn : n ≥ m) : Q n := let P' := λ n, Q (m + n) in have P' (n - m), from H P' h₁ (λ n, h₂ _ (nat.le_add_right m n)) (n - m), by simp [P'] at this; rwa nat.add_sub_cancel' mn at this
simp [P'] at this
is using Lean's inbuilt simplifier to do a lot of dirty work involving equalities; rwa
means "rewrite, then use an assumption". Both simp
and rw
are talked about in Theorem Proving in Lean, in the chapter on tactics: https://leanprover.github.io/theorem_proving_in_lean/tactics.html
Thank you! This has helped.
{% endraw %}Last updated: Apr 24 2020 at 19:02 UTC


We are getting daily data from the sources below and they may be changed without notice.
- Countries: https://github.com/CSSEGISandData/COVID-19.git, Novel Coronavirus (COVID-19) Cases, provided by JHU CSSE
- Counties in every state in the US: https://github.com/nytimes/covid-19-data.git, New York Times
- Cities in Los Angeles County in CA in the US: http://publichealth.lacounty.gov/media/Coronavirus/locations.htm, County of Los Angeles Public Health
- Cities in Orange County in CA in the US: https://occovid19.ochealthinfo.com/coronavirus-in-oc, Orange County Health Care Agency
- Cities in Ventura County in CA in the US: https://www.vcemergency.com/, Ventura County - Office Of Emergency Services
- Cities in Santa Clara County in CA in the US: https://www.sccgov.org/sites/covid19/Pages/dashboard.aspx, Santa Clara County Public Health
- Cities in San Diego County in CA in the US: https://www.sandiegocounty.gov/content/dam/sdc/hhsa/programs/phs/Epidemiology/COVID-19%20Daily%20Update_City%20of%20Residence.pdf , San Diego County Public Health
- Most Cities in IL in the US: https://www.dph.illinois.gov/sitefiles/COVIDZip.json, Illinois Department of Public Health


{{animationExport}}
import json from time import sleep from Adafruit_8x8 import EightByEight def showFrame(frame): x = 0 grid.disp.clear() for row in frame["eight_bit"]: for y in range(0,8): if(row >> y & 1): grid.setPixel(x, y, 1) x = x+1 grid = EightByEight(address=0x70,debug=True) grid.disp.setBrightness(1) animation = json.loads('{{animationExport}}') print(animation["name"]) for frame in animation["frames"]: showFrame(frame) sleep(frame["duration"]) grid.disp.clear()


|
gtsam 4.1.1
gtsam
|
Class that implements Shonan Averaging from our ECCV'20 paper.
Note: The "basic" API uses all Rot values (Rot2 or Rot3, depending on value of d), whereas the different levels and "advanced" API at SO(p) needs Values of type SOn<Dynamic>.
The template parameter d can be 2 or 3. Both are specialized in the .cpp file.
If you use this code in your work, please consider citing our paper: Shonan Rotation Averaging, Global Optimality by Surfing SO(p)^n Frank Dellaert, David M. Rosen, Jing Wu, Robert Mahony, and Luca Carlone, European Computer Vision Conference, 2020. You can view our ECCV spotlight video at https://youtu.be/5ppaqMyHtE0
Matrix API (advanced use, debugging) | |
| Sparse | D () const |
| Sparse version of D. | |
| Matrix | denseD () const |
| Dense version of D. | |
| Sparse | Q () const |
| Sparse version of Q. | |
| Matrix | denseQ () const |
| Dense version of Q. | |
| Sparse | L () const |
| Sparse version of L. | |
| Matrix | denseL () const |
| Dense version of L. | |
| Sparse | computeLambda (const Matrix &S) const |
| Version that takes pxdN Stiefel manifold elements. | |
| Matrix | computeLambda_ (const Values &values) const |
| Dense versions of computeLambda for wrapper/testing. | |
| Matrix | computeLambda_ (const Matrix &S) const |
| Dense versions of computeLambda for wrapper/testing. | |
| Sparse | computeA (const Values &values) const |
| Compute A matrix whose Eigenvalues we will examine. | |
| Sparse | computeA (const Matrix &S) const |
| Version that takes pxdN Stiefel manifold elements. | |
| Matrix | computeA_ (const Values &values) const |
| Dense version of computeA for wrapper/testing. | |
| double | computeMinEigenValue (const Values &values, Vector *minEigenVector=nullptr) const |
| Compute minimum eigenvalue for optimality check. More... | |
| double | computeMinEigenValueAP (const Values &values, Vector *minEigenVector=nullptr) const |
| Compute minimum eigenvalue with accelerated power method. More... | |
| Values | roundSolutionS (const Matrix &S) const |
| Project pxdN Stiefel manifold matrix S to Rot3^N. | |
| Matrix | riemannianGradient (size_t p, const Values &values) const |
| Calculate the riemannian gradient of F(values) at values. | |
| Values | initializeWithDescent (size_t p, const Values &values, const Vector &minEigenVector, double minEigenValue, double gradienTolerance=1e-2, double preconditionedGradNormTolerance=1e-4) const |
| Given some values at p-1, return new values at p, by doing a line search along the descent direction, computed from the minimum eigenvector at p-1. More... | |
| static Matrix | StiefelElementMatrix (const Values &values) |
| Project to pxdN Stiefel manifold. | |
| static VectorValues | TangentVectorValues (size_t p, const Vector &v) |
| Create a VectorValues with eigenvector v_i. | |
| static Values | LiftwithDescent (size_t p, const Values &values, const Vector &minEigenVector) |
| Lift up the dimension of values in type SO(p-1) with descent direction provided by minEigenVector and return new values in type SO(p) | |
Advanced API | |
| NonlinearFactorGraph | buildGraphAt (size_t p) const |
| Build graph for SO(p) More... | |
| Values | initializeRandomlyAt (size_t p, std::mt19937 &rng) const |
| Create initial Values of type SO(p) More... | |
| Values | initializeRandomlyAt (size_t p) const |
| Version of initializeRandomlyAt with fixed random seed. | |
| double | costAt (size_t p, const Values &values) const |
| Calculate cost for SO(p) Values should be of type SO(p) | |
| Sparse | computeLambda (const Values &values) const |
| Given an estimated local minimum Yopt for the (possibly lifted) relaxation, this function computes and returns the block-diagonal elements of the corresponding Lagrange multiplier. | |
| std::pair< double, Vector > | computeMinEigenVector (const Values &values) const |
| Compute minimum eigenvalue for optimality check. More... | |
| bool | checkOptimality (const Values &values) const |
| Check optimality. More... | |
| boost::shared_ptr< LevenbergMarquardtOptimizer > | createOptimizerAt (size_t p, const Values &initial) const |
| Try to create optimizer at SO(p) More... | |
| Values | tryOptimizingAt (size_t p, const Values &initial) const |
| Try to optimize at SO(p) More... | |
| Values | projectFrom (size_t p, const Values &values) const |
| Project from SO(p) to Rot2 or Rot3 Values should be of type SO(p) | |
| Values | roundSolution (const Values &values) const |
| Project from SO(p)^N to Rot2^N or Rot3^N Values should be of type SO(p) | |
| template<class T > | |
| static Values | LiftTo (size_t p, const Values &values) |
| Lift Values of type T to SO(p) | |
Public Member Functions | |
| template<typename T > | |
| std::vector< BinaryMeasurement< T > > | maybeRobust (const std::vector< BinaryMeasurement< T > > &measurements, bool useRobustModel=false) const |
| Helper function to convert measurements to robust noise model if flag is set. More... | |
| Values | projectFrom (size_t p, const Values &values) const |
| Values | projectFrom (size_t p, const Values &values) const |
| Values | roundSolutionS (const Matrix &S) const |
| Values | roundSolutionS (const Matrix &S) const |
Standard Constructors | |
| ShonanAveraging (const Measurements &measurements, const Parameters ¶meters=Parameters()) | |
| Construct from set of relative measurements (given as BetweenFactor<Rot3> for now) NoiseModel must be isotropic. | |
Query properties | |
| size_t | nrUnknowns () const |
| Return number of unknowns. | |
| size_t | nrMeasurements () const |
| Return number of measurements. | |
| const BinaryMeasurement< Rot > & | measurement (size_t k) const |
| k^th binary measurement | |
| Measurements | makeNoiseModelRobust (const Measurements &measurements, double k=1.345) const |
| Update factors to use robust Huber loss. More... | |
| const Rot & | measured (size_t k) const |
| k^th measurement, as a Rot. | |
| const KeyVector & | keys (size_t k) const |
| Keys for k^th measurement, as a vector of Key values. | |
Basic API | |
| double | cost (const Values &values) const |
| Calculate cost for SO(3) Values should be of type Rot3. | |
| Values | initializeRandomly (std::mt19937 &rng) const |
| Initialize randomly at SO(d) More... | |
| Values | initializeRandomly () const |
| Random initialization for wrapper, fixed random seed. | |
| std::pair< Values, double > | run (const Values &initialEstimate, size_t pMin=d, size_t pMax=10) const |
| Optimize at different values of p until convergence. More... | |
Public Types | |
| using | Sparse = Eigen::SparseMatrix< double > |
| using | Parameters = ShonanAveragingParameters< d > |
| using | Rot = typename Parameters::Rot |
| using | Measurements = std::vector< BinaryMeasurement< Rot > > |
| NonlinearFactorGraph gtsam::ShonanAveraging< d >::buildGraphAt | ( | size_t | p | ) | const |
Build graph for SO(p)
| p | the dimensionality of the rotation manifold to optimize over |
| bool gtsam::ShonanAveraging< d >::checkOptimality | ( | const Values & | values | ) | const |
Check optimality.
| values | should be of type SOn |
| double gtsam::ShonanAveraging< d >::computeMinEigenValue | ( | const Values & | values, |
| Vector * | minEigenVector = nullptr |
||
| ) | const |
Compute minimum eigenvalue for optimality check.
| values | should be of type SOn |
| double gtsam::ShonanAveraging< d >::computeMinEigenValueAP | ( | const Values & | values, |
| Vector * | minEigenVector = nullptr |
||
| ) | const |
Compute minimum eigenvalue with accelerated power method.
| values | should be of type SOn |
| std::pair< double, Vector > gtsam::ShonanAveraging< d >::computeMinEigenVector | ( | const Values & | values | ) | const |
Compute minimum eigenvalue for optimality check.
| values | should be of type SOn |
| boost::shared_ptr< LevenbergMarquardtOptimizer > gtsam::ShonanAveraging< d >::createOptimizerAt | ( | size_t | p, |
| const Values & | initial | ||
| ) | const |
Try to create optimizer at SO(p)
| p | the dimensionality of the rotation manifold to optimize over |
| initial | initial SO(p) values |
| Values gtsam::ShonanAveraging< d >::initializeRandomly | ( | std::mt19937 & | rng | ) | const |
Initialize randomly at SO(d)
| rng | random number generator Example: std::mt19937 rng(42); Values initial = initializeRandomly(rng, p); |
| Values gtsam::ShonanAveraging< d >::initializeRandomlyAt | ( | size_t | p, |
| std::mt19937 & | rng | ||
| ) | const |
Create initial Values of type SO(p)
| p | the dimensionality of the rotation manifold |
| Values gtsam::ShonanAveraging< d >::initializeWithDescent | ( | size_t | p, |
| const Values & | values, | ||
| const Vector & | minEigenVector, | ||
| double | minEigenValue, | ||
| double | gradienTolerance = 1e-2 , |
||
| double | preconditionedGradNormTolerance = 1e-4 |
||
| ) | const |
Given some values at p-1, return new values at p, by doing a line search along the descent direction, computed from the minimum eigenvector at p-1.
| values | should be of type SO(p-1) |
| minEigenVector | corresponding to minEigenValue at level p-1 |
|
inline |
Update factors to use robust Huber loss.
| measurements | Vector of BinaryMeasurements. |
| k | Huber noise model threshold. |
|
inline |
Helper function to convert measurements to robust noise model if flag is set.
| T | the type of measurement, e.g. Rot3. |
| measurements | vector of BinaryMeasurements of type T. |
| useRobustModel | flag indicating whether use robust noise model instead. |
| std::pair< Values, double > gtsam::ShonanAveraging< d >::run | ( | const Values & | initialEstimate, |
| size_t | pMin = d , |
||
| size_t | pMax = 10 |
||
| ) | const |
| Values gtsam::ShonanAveraging< d >::tryOptimizingAt | ( | size_t | p, |
| const Values & | initial | ||
| ) | const |
Try to optimize at SO(p)
| p | the dimensionality of the rotation manifold to optimize over |
| initial | initial SO(p) values |