---
layout: post
status: publish
published: true
title: BizTalk 2006 Licensing Changes
author:
display_name: Tomas Restrepo
login: tomasr
email<EMAIL_ADDRESS>url: http://winterdom.com/
author_login: tomasr
author_email<EMAIL_ADDRESS>author_url: http://winterdom.com/
wordpress_id: 258
wordpress_url: http://winterdom.com/2005/11/biztalk2006licensingchanges
date: '2005-11-07 15:35:26 +0000'
date_gmt: '2005-11-07 15:35:26 +0000'
categories:
- BizTalk
tags: []
comments: []
---
With the launch of Visual Studio 2005, SQL Server 2005 and BizTalk 2006 today, the new licensing scheme for BizTalk 2006 has been announced. Scott Woodgate talks about it here.
Personally, I feel the licensing changes are great. Sure, it raises the price a little bit, but some of the more problematic restrictions have been lifted, and I am glad Microsoft has listened to their customers on this one! Some highlights of the changes:
- Everything in a box: All the adapters included in the box, not just the existing ones but also all the new ones Microsoft recently bought (i.e. Peoplesoft, Oracle, etc.). Extremely exciting!
- Two processor support for STD: Standard Edition now supports up to 2 processors (you need to buy two licenses). This is a sweet spot for many of our customers and was one piece of feedback I gave to MS on several ocassions. I'm extremely happy this was changed because it lowers the bar of entry for customers that need more throughput (or just want to take advantage of their existing dual processor servers) but don't want to pay for two ENT licenses.
- Parts of Host Integration Server going into BizTalk Server 2006 Enterprise Edition: This one took me a little bit by surprise, but I believe it is the right move; I've been thinking for a while now that Host Integration Server and BizTalk Server should really be a single product. The functionality included is DB2 database access, the transaction integrator to CICS/IMS/RPG and Host File Access.
All in all, excellent news, I think.


TL;DR - Decided to build an automated email newsletter to track the stock market and show pre-market data and historical performance of S&P 500, Nasdaq, and Dow Jones to help with my investments.
The other day I read this tweet from Jason Fried, the founder of Basecamp, saying that they are working on a new email app called Hey, and that got me thinking.
Checking my email is one of the first things I do when I wake up, in a quick scan I know if there’s something important I should pay attention to, I read the news, and I take the time to keep my inbox tidy. Email is a central tool in my daily workflow.
One thing I like to do is to subscribe to email newsletters mostly about tech like Techmeme, Ruby Weekly, Hacker News, Node Weekly and news from WSJ, Morning Brew and NYT, I signup to a bunch all the time but I’m very diligent only to keep those I’ll read.
Email has a bad rep, spam, clutter. Still, for me, it’s such a joy to receive a well-crafted email where people took the time to write something worthwhile and even more time to package that in a beautiful clean design especially in a finicky medium like email with dozens of different clients all rendering things slightly different from one another.
Being stranded at home living the “quarantine” life and with some time in my hands, I thought about building something, and my requirements were:
Watching the markets tank in March was brutal. The stock market was so volatile that it was hard to keep up with its ups and downs, so my idea was to build something around that.
There are lots of tools to track the market like screeners, tons of finance websites even Google, you can type a ticker name, and it will give you the stock price and a chart.
When you read the news or listen to podcasts about the topic they always talk about when the market open or close in terms of points like the Nasdaq composite closed down 300 points to 7500 or something like that, that’s useful for someone in finance or more versed in the markets. Still, to me, all I want to know is how much percentage it gained or lost in a given period like the day before, a week ago, six months ago, and with that, I can tell how the market is trending.
Another thing that I learned some time ago is related to Stock futures, also called pre-market, and that gives you a good indication if the market is trending up or down for the day before it opens for business.
Going back to email and the tweet from Jason Fried, I decided to build an email newsletter with information about the stock markets and data I cared about including:
The idea is for the mailing list to go out every weekday before the market opens so you can be informed enough to decide if it’s worth paying close attention to the market on any given day and make some moves.
With that in mind, I started with some good old research to figure out what I need to build this thing.
Needed
I’ve spent the first day or two signing up and playing with finance APIs
and email marketing tools until I found one that did what I want and
offered a freemium option. I ended up landing on
Sendgrid for email and
Alpha Vantage
Yahoo Finance undocumented
API
for stock market data.
It’s incredible the amount of work it takes to build even the simplest thing, besides writing code which is the fun part there are a ton of other tedious admin tasks that need to be done and configured to make everything work like:
Probably finding a name and available domain to register takes the longest in most of my side projects, and it’s the first thing I do contrary to what most people would say. I enjoy playing with names, and finding the one that feels right to me gets me psyched and in the right frame of mind to work on the project.
Some of the names I’ve considered:
The name that resonated the most with me was Bullish▲, I guess because it’s timely related to the end of one of the longest US bull markets in history, and I’ve found the perfect domain available for the small fortune of $3.88, and that’s how https://bullish.email happened.
If there’s one cool thing about tech nowadays is that pretty much everyone offers a free version of something. You can play with any cloud provider and use it gratis forever as long you are within their free tier. You can publish a site on GitHub for free, including SSL, also free; and a bunch of other stuff. It’s awesome!
Anyway, back to the project, after I hooked everything together and spent an awful lot of time designing the email in Sendgrid plus a couple more days writing trash code to test things out and then rewriting it again the proper way, I’ve finally got to a working version that was up to my standards.
The last thing to do was to decide how to schedule the code to run, prepare, and trigger Sendgrid to send emails every morning. My first choice was to go serverless with Lambda that would fit nicely and wouldn’t cost a dime, but then I realized I have a Raspberry Pi laying around, so why not use that instead?
So I’ve set up the Raspberry Pi to run a Cron job every day around 9:00 AM eastern time, which is 30 minutes before the market opens and I’ve also used a nifty little app called Cronhub to monitor the job and alert me if it doesn’t run, and that was it. A little over a week’s worth of work and that nice runner’s high kind of feeling of getting another project across the finish line.
If you want to give it a try, you can signup to get Bullish▲ in your inbox every morning at https://bullish.email; you can also check out the code on Github.
Stay healthy. Cheers ~


Project journey
Kamaal Khanum-Williams
Project Journey – to include the tasks undertaken for the project, a PDF of the submitted proposal, and a discussion of the stylistic vision for the film (Director)
Tasks undertaken
While filming there was many considerations that had to be made, due to set backs. The sound had been a major downfall to the documentary. Which stopped the progress of the film in the beginning so as the director a decision was made on the day of a shoot. The sound was not out putting leading us to go into a interview with no audio output this was a major set back as we went in without no sound check. This was a big mistake, which was later revealed after filming.
According to Artis “bad audio is an unforgiveable video sin ”(2014:186) in audio which we was well aware of but it was decided to take a chance. Which back fired tremendously as there was too many back ground noises. This led to the interview with one of our first interviewees Ann Turner being re-shot except this time having the audio fixed we proceeded with caution to avoid any more set backs.
While shooting we had the goals we wanted to achieve while filming but sticking to the exact storyboard we had created just didn’t work as plans do change which meant we started to “shoot your story with an open mind”.(Bernard:2004) Since we didn’t have much time to decide on when re shooting was possible we got what we could. Sticking to the exact storyboard would have hindered us greatly, which was the reason why there was more than one version of storyboard.
During the interviews the decision was made to have the interviewees to have their answers included in the question. As Bernard points out “ask the person being interviewed to incorporate part of your question in his or her answer”.(2004:149) The reason behind, this was due to documentary having a strong narrative that was created solely by the interviewer instead of it being a predictable set of interviews with the questions being heard on camera and they answer.
This gave the three different interviews authenticity to the narrative. The locations needed to be thoroughly checked in advance before each shoot. This decision was made to capitalise on the control we could have. Artis states “the less control you have over an interview environment the more headache you will have”. (2014:94) We found this method helped the interviews progress run much smoother, as we were able to identify the potential problems giving us more control.
Finally, aside from handling the interviews, location and sound tasks making the interviewees comfortable remained priority throughout the film by having them in a comfortable environment. As Artis points out “the best location for interviews will be a room in your subjects home or place of occupation”. (2014:94) Having the interviewee in their own environment allowed them to be more expressive verbally. Which gave us some great responses that may not have been possible if we chose the locations our self’s.
Inclusion, the project had many great aspects that lead us to getting the great piece that we did. This doesn’t mean any improvements couldn’t have been made as the director one of the main changes that could have been done would be to have more time on planning the documentary out to make sure more of the potential problems could have been identified as many was missed out i.e interview cancellations and sound checks.
Proposal
Download our Proposal : Download Proposal.
Stylistic vision for the film
Pre production this was a very pivotal part of the documentary process, as it was the first task that would set stylistic vision for the documentary. As the director of the documentary the vision was to create a piece that would be enjoyable as well as informative.
The brain storm, was to have everyone in the group put forward their ideas which was then decided which idea would be the most promising for the group as well as fit with stylistic vision. The documentary idea was chosen, to be about what a vegan / vegetarian is? Which is a subject not many of the public have much knowledge about and can be greatly informative in a documentary.
Once the decision had been made to create the documentary based on this idea, some issues had to be identified the first being what locations and in what style could this keep the audience interested. In order to achieve this background research on ideas was done. The most suitable style found was to create this piece, would be to do it in a Vlog style. Which included doing multiple days on findings around what a vegan/vegetarian is.
This idea came from the hugely successful Vlog done on YouTube with over 2 million hits. A similar Vlog style would work to a great effect, as the audience would follow the journey of the character they can relate to. We found it’s important to have a compelling narrative as Bernard suggests that it is critical to “reach people with a subject and story that grabs them”. (2004:33) Our group found this could be achieved by using the Vlog journey as the narrative to link the documentary together.
Working closely with the researcher, during pre production we established which interviewee would be the most promising for the documentary narrative. While going through interviewees that we found it was decided to make a check listed that would bring context to the documentary. One of the final stages of the preproduction of the documentary was narrowing down the strong candidates. As Artis suggests “the people you ultimately choose to speak on your topic are equally crucial to the success of your doc”.(2014:11) Otherwise we would have a some people who didn’t quite fit the purpose of the film.
Also the locations had to be carefully considered, as some places needed permission to film at. As Rabiger indicates permission “must be cleared by the relevant authority unless you care to run risk being taken to court”.(2009:376) However, we managed to get the permission from the authority before the interviews by calling or emailing them. Having the considerations completed, a loose storyboard was created which would allow the team to have space to make changes should if needed.
References:
1. Rabiger.M (2009). Directing the Documentary. Oxford: Focal Press.
2. Bernard,C.S (2004). Documentary Storytelling For Film And VideoMakers. Oxford: Focal Press.
3. Artis,Q.A (2014).


DNS tunneling
If any WAN traffic is blocked but external host names are resolved then there’s a possibility of tunneling traffic via DNS queries. You need a domain registered for this technique to work. This manual might help you with setting up your name server.


Install it to your Phone
Appkit is a PWA.
PWA stands for Progressive Web App. This means, you can save the item to the homescreen and use it as an app. These banners will be more assertive and help you convert users to installing your app easier.
Badges
Add to Home Badges are simple elements that show where your device menu is to indicate how to install the page to your home screen. These are the default interaction of most websites.
Action Sheets
The new age of showing Installation prompts for PWA's is by using Action Sheets. These are gorgeous interfaces that can deploy automatically and show the install prompt on Android or the instructions for installing on iOS.


Poképad
A collaborative teambuilder featuring a chat and easy access to smogdex sets. Create below and share the link to start building with friends!
Start Building!
A collaborative teambuilder featuring a chat and easy access to smogdex sets. Create below and share the link to start building with friends!


Metadata are everywhere around us - and they are expressed in many incompatible formats. Metadata are the Tower of Babel of the current world. Let's concentrate on the Soaplab2's contribution to the world chaos.
Soaplab2 uses two major types of metadata:
The latter one (metadata describing individual services) is the main topic of this document.
Each service has to be described. In the run time (when Soaplab2 services are serving the world) such description is also expressed in the XML format - but a service provider can (and usually does) use a simpler, more "human-readable" (and better editable) format - the ACD format (more about its origin a bit later). Soaplab2 converts (in the build, deployment time) the ACD files to its native XML files.
Here is an example of a simple ACD file for the classic HelloWorld application:
appl: HelloWorld [ documentation: "Classic greeting from the beginning of the UNIX epoch" groups: "Classic, Simple" nonemboss: "Y" executable: "echo" ] string: greeting [ additional: "Y" parameter: "Y" default: "Hello World" comment: "defaults" ] outfile: output [ additional: "Y" default: "stdout" ]The example above defines a service HelloWorld
The XML file representing the same metadata (and created by the ACD to XML converter from the file above) is much less human-readable and you do not want/need to see it (for developers: its DTD is also available).
The original purpose of Soaplab was to wrap the command-line programs as Web services. That's why on many places in this documentation we relate ACD files to the command-line options. But as Soaplab evolved it became clear that the ACD files could describe not only command-line tools but other type of programs, as well. Soaplab2 can make Web services around regular web pages (see a sub-project Gowlab) or on top of other Web services (for example, the wrappers around the Web services provided by EBI are distributed together with Soaplab2 as plug-ins).
It would be, therefore, more precise to talk about "inputs", "parameters" and/or "outputs", instead of "command-line parameters and/or options". Adapt your mind please when reading this document. By applications, we mean here a command-line program, or anything else that can be wrapped as a Web service and that needs some input data and can produce some resulting data.
Now, we can finally start with the syntactical rules...
Each application is defined by one ACD file. The ACD file names will be used as a (part of) names of the Soaplab2 services. The ACD files use the extension '.acd'.
ACD files can have comments. They start with "#" and continue to the end of the line. Empty lines are ignored.
Each application has zero or more parameters. In an ACD file, each of them is defined by a single token followed by either a colon ':' (preferred) or an equal sign '=' which in turn is followed by a second token. The definition of the parameter is delimited by a pair of square brackets '[ ]', which can span multiple lines:
token: token [ definition ]Tokens representing keywords can be abbreviated up to the point where they are not ambiguous. For example, default: can be abbreviated to def:.
The first definition in any ACD file must start with token application (usually abbreviated to appl) followed by a token representing the application name, and by attributes describing the whole application (not just an individual parameter):
appl: appname [ application attributes ]Then follow definitions of the application's parameters. The first token in these definitions is a datatype. The data type is a keyword and must be from the pre-defined set of data types. The second token is the name by which this parameter is going to be known. The definition contains parameter-specific attributes (various data types may have different set of allowed attributes):
datatype: parameter_name [ parameter attributes ]Both application and parameter attributes consist of a name (a keyword from a pre-defined set of names), followed by a colon and a value. Values can be delimited by single or double quotes. A value representing a boolean value should be either Y or N.
infile: library [ additional: "Y" information: "A custom PostScript library file" comment: "tagsepar -" ]If there is no real value (meaning: nothing separated by a space) the resulting Soaplab2 option is considered to be boolean with the value true (and the quotes are not needed), as in this example:
comment: bindataAn ACD file may have any number of such comment attributes. If you are a developer creating your own plug-in, you can use the comment attributes for passing any options or any additional metadata. Of course, if you do so, only the selected clients will be able to use such additional information - but Soaplab2 does not stop to propagate unknown options to the XML metadata.
appl: dot [ ... ]When the wrapped resource is not a command-line program (but, for example, a web page) the application name is not used (but must be present).
Here is a list of recognized application attributes (note that those known only to Soaplab2 are already shown as part of the comment attribute):
The group name, together with the ACD base file name, creates the Soaplab2 service name (using a dot as a separator and making everything lower-case). If a tool belongs to more groups, more Soaplab2 services are created (even though they are identical in their behaviour). For example:
appl: HelloWorld [ documentation: "Classic greeting from the beginning of the UNIX epoch" groups: "Classic, Simple" ... ]If the lines above are from a file helloworld.acd, two Soaplab services will be created:
classic.helloworld simple.helloworld
It is not recommended to put here the full path of the executable - it would make the ACD file non-portable. The path can be added later, as a run-time configuration property (look for the property addtopath.dir in the configuration guide for details).
An example was given above for the hellowold application where the executable was the Unix program echo.
comment: "class org.soaplab.ebi.WSJobFactory"
comment: "launcher get" comment: "launcher post"
Look for individual attribute details in the parameter attribute section.
datatype: parameter_name [ parameter attributes ]This section describes the parameter attributes that can be used for any data type (data type is here just another name for a parameter type).
Any parameter without this attribute will appear (unless an attribute template is specified) on the command line as a tagged value:
-<parameter-name> <parameter-value>With this attribute it will appear as a simple value without any qualifier (i.e. the parameter name will not appear on the resulting command line).
The above general attributes are the most used ones. There may be however situations where their combinations are unclear. This is caused by the EMBOSS legacy: EMBOSS has more ways how to define whether its parameter is optional or not. Therefore, below are the possible (useful) combinations of the these parameters, explained by examples.
-<parameter-name> <parameter-value>where the space between parameter name and parameter value can be changed to something else by the attribute tagsepar, or
If none of these cases is good enough, the attribute template can be used. It defines a string with special tokens that will be replaced by parameter name and parameter value. This attribute is similar in function to the application attribute method. The difference is that here the template is just for one parameter, while the method defines a template string for the whole command-line. Details of both are covered below.
-greeting HelloIf this attribute is specified, the parameter name is not anymore started with the minus sign. For example, having this in an ACD file:
string: greeting [ comment: "tagsepar *" ... ]and with a sent value "Hello" leads to the command line:
greeting*HelloFrom the historical syntactical reasons (legacy), use slightly different syntax when the tagsepar should be an equal sign (which is probably the most common case after the space). This ACD file:
string: greeting [ comment: "tagsepar = =" ... ]leads to the command line:
greeting=HelloYou can also use the same syntax for other separators. The first example above can be written also as:
string: greeting [ comment: "tagsepar = *" ... ]Because usually the same tag separator is used for all parameters, the tagsepar may be used also as an application attribute - and, therefore, propagated to all parameters. If in such case you need for some parameters to return to the default tag separator (a space), use there a tagsepar without any value. Here is an example with all possibilities:
appl: Tagsepar [ documentation: "Testing TAGSEPAR attribute" groups: "Testing" nonemboss: "Y" executable: "echo" comment: defaults comment: "tagsepar = =" ] # --- this prameters gets TAGSEPAR from the application level string: param1 [ default: 1 ] # --- this prameters has its own TAGSEPAR string: param2 [ comment: "tagsepar = -" default: 2 ] # --- this prameters wants to have an original (default) TAGSEPAR string: param3 [ default: 3 comment: tagsepar ] outfile: output [ additional: "Y" default: "stdout" ]
This attribute is ignored for data types boolean, infile, filelist and outfile.
string: g [ ]The user/client must send a value for this parameter under the name g. Assuming that she sent a value "hello", the command line will look like this:
-g helloThis works fine but it is hard for the client to remember the purpose of a non-descriptive parameter name g. In order to make it easier for client, one can use attribute qualifier. Once present, it is used on the command line instead of the parameter name (and the parameter name is still used to identify client value). The example above can be re-written like this:
string: greeting [ qualifier: g ]The user sends now her "hello" under the better name greeting but the command line is created as before, using the qualifier g.
By default, parameters are not hidden. Except for the data type outfile that is always hidden (which means that the clients do not specify the name of the output file that will be created on the command line - it is the Soaplab2's task to do so).
This attribute is often used for Gowlab services - where HTML forms may have some "hidden" variables.
boolean: bool_env [ information: "b4: A boolean that becomes an environment" qualifier: "b4" comment: envar ] string: str_env [ information: "str: A string that becomes an environment ENVVAR" additional: "Y" qualifier: ENVVAR default: "Ciao mundi" comment: envar comment: "defaults" ]This attribute is ignored for data type infile, filelist and outfile.
string: param [ standard: "Y" ]
string: param [ parameter: "Y" ]
string: param [ ]Note, that the following construct (a bit against the common sense) DOES NOT mean that the param is mandatory:
string: param [ additional: "N" ]The both constructs a) and b) become NOT mandatory if the definitions include a default value:
string: param [ standard: "Y" default: "a default value" ]or
string: param [ parameter: "Y" default: "a default value" ]Or, if the same definition has also additional Y.
The first purpose of this attribute is that its presence makes the parameter optional (as explained above).
The second (and the main) purpose, of course, is to tell what value will be used if the user does not send any value for a particular parameter. The question is, however, should such default value be used (e.g. should it be put on the command-line), or not? The answer is: it depends.
Soaplab2 has the following rules for that (for the situation that no value was sent by the user for a particular parameter):
Every application and each of its parameters (especially its inputs and outputs, but also other parameters) can have its meaning explicitly specified by ontology terms. Simply speaking, an ontology is a controlled vocabulary with terms pre-defined and agreed on by domain experts. Additionally to such simple vocabulary, an ontology can have (and usually has) defined relationships between its terms (saying, for example, which terms are more general and which are specific). The ontology terms do not influence how the application is invoked and what it does but can be useful, for example, for discovering which application to use for your particular data types.
In an ACD file, the ontology terms can be specified by one or more relations attribute(s). Such attribute can be used both as an application attribute or as a parameter attribute. There are two possible formats for the value of this attribute (the second one is, however, recommended):
relations: "/edam/operation/0001813 Sequence retrieval"
relations: "EDAM:0001813 operation Sequence retrieval"Both examples above say that this application (or this parameter) has semantics defined by an ontology term identifier EDAM:0001813 and that this term is defined in an ontology namespace operation. Additionally, more or less for convenience (because the same data could be obtained directly from the ontology itself by using the ontology term identifier), there is also an ontology term name Sequence retrieval.
You may use any ontology, of course. But one strong candidate is the EDAM ontology that is closely related to the EMBOSS and, therefore, all EMBOSS applications (ACD files) already are annotated by relations from this ontology.
The usual way is to build the command-line from the individual parameters, each of them can be created either as a tagged value:
-<parameter-name> <parameter-value>or as a non-tagged value (when the parameter attribute parameter is set to true):
<parameter-value>The first form can be further modified by the tagsepar attribute, resulting, for example, in this:
<parameter-name>=<parameter-value>If this is not enough, a parameter can have an attribute template that defines how the parameter should be built. The template value is a string that contains special tokens (e.g. "$$" or "&&"). The tokens will be substituted by the real parameter value (sent by a client) or by a parameter name (the latter substitution is not that crucial because you can yourself include the parameter name directly; but it helps with the maintenance of ACD files if you have the parameter name only in one place).
If a parameter does not have any value (a client sent nothing, and there is no default value), nothing is built for this parameter. In other words, the template string is not used at all (even if it has some constant text there).
Let's have a service that has one input named greeting, and a client sends there a value "I wish you luck, mate".
Without templates and with the following ACD parameter:
string: greeting [ ]we are getting a command line with exactly two elements (it is shown in the format used by the admin tool for exploring parameters, mentioned above):
Command line ------------ ( 1) -greeting ( 2) I wish you luck, mate!You can see that Soaplab2 took care and made sure that the whole sent value is not separated by white-spaces.
However, if you use a template, you have to tell whether you wish or not to wish to separate by white-spaces. Let's have now the same parameter with a template:
string: greeting [ template: "-greeting $$" ]The token "$$" is replaced by the client value. The result is now, however, different - the command line suddenly gets six elements:
Command line ------------ ( 1) -greeting ( 2) I ( 3) wish ( 4) you ( 5) luck, ( 6) mate!In order to get the same result as without a template, you need to use a slightly different token (a double-quoted $$):
string: greeting [ template: '-greeting $"$"' ]Now, we are getting the same command line with just two elements:
Command line ------------ ( 1) -greeting ( 2) I wish you luck, mate!
A template as an application attribute is specified as:
appl name [ ... comment: "method <string>" ... ]And a template as a parameter attribute is specified as:
data-type name [ ... template: "<string>" ... ]Some tokens can be abbreviated ("$$", "&&", etc.). This abbreviation cannot be used for the application attribute (because the abbreviated form does not tell which parameter is the token meant for).
| Token | Will be replaced by | Explanation |
|---|---|---|
| Parameters' values | ||
| $id | value of parameter 'id' | |
| ${id} | value of parameter 'id' | used when the construct is followed immediately by a letter (like in ${id}A) |
| $"id" | value of parameter 'id' | value will not be separated into several pieces even if it contains white-spaces |
| ${"id"} | value of parameter 'id' | ditto as above |
| $$ | value of the current element | a simpler notation for $id; cannot be used in application attribute |
| $"$" | value of the current element | a simpler notation for $"id"; cannot be used in application attribute |
| Parameters' names | ||
| &id | name of parameter 'id' | |
| &{id} | name of parameter 'id' | the same as &id but this protects from being mixed with the subsequent letters |
| && | name of the current parameter | a simpler notation for &id; cannot be used in application attribute |
appl: MedlineSRS [ documentation: "Get MEDLINE citation (in XML)" groups: "Testing" nonemboss: "Y" supplier: "http://srs.ebi.ac.uk/srsbin/cgi-bin/wgetz" comment: "method -e+[MEDLINE:'$pmid']+-ascii" ] string: pmid [ parameter: "Y" ] outfile: result [ ]The example is from Gowlab: it fetches a publication from bibliographic database Medline via SRS at EBI.
And here is a short example with a template as a parameter attribute:
string: greeting [ additional: "Y" default: "Hello World" comment: "defaults" template: "Soaplab2 sends you regards: $$" ]The result command line will be:
Program and parameters: /home/senger/soaplab2/run/echo Soaplab2 sends you regards: Hello World --- end of parametersThe same example with a slightly changed template (in order not to separate by white-spaces):
string: greeting [ additional: "Y" default: "Hello World" comment: "defaults" template: '"Soaplab2 sends you regards:" $"$"' ]Now the result command line is:
Program and parameters: /home/senger/soaplab2/run/echo Soaplab2 sends you regards: Hello World --- end of parameters
In an ACD file, every input and output is described in a separate construct that starts with a data type:
datatype: parameter_name [ parameter attributes ]There are not many data types (EMBOSS applications have much more of them - but we are not creating ACD files for EMBOSS native applications because they already come with the ACD files).
Specific attributes for this data type are:
If this attribute is not specified then both, direct and reference input are acceptable:
infile: input [ ]In the example above, a client sees (and can use) two input names (both are created from this one ACD parameter):
<parameter-name>_direct_data <parameter-name>_urlThe parameter name is extended by fixed suffixes. The _direct_data means that a user is sending directly data, the _url means that a user is sending a reference to the data. The reference should be a URL where the input data can be fetched from. Usual protocols (http, ftp) are supported.
For EMBOSS, the reference data (on the client site) has suffix _usa instead of _url (see what USA in EMBOSS means).
If this attribute has the value direct, only direct data can be sent by a client. In this case, the input name is identical with the parameter name (input in our example), no suffixes are used:
infile: input [ comment: "data direct" ]Similarly, if this attribute has the value filename, only reference data can be sent by a client. Again in this case, the input name is identical with the parameter name (input in our example), no suffixes are used:
infile: input [ comment: "data filename" ]
infile: input [ additional: "Y" default: "this is my default" comment: default_for_direct ]
#!/usr/bin/perl -w # # It copies a file (-i) (if given) to STDOUT. # Then it adds to STDOUT contents of files given by names # by the -l option. # # Usage: copy-files.pl -i <input-file> -l <list-file> # # --------------------------------------------- use strict; use warnings; use File::Copy; use File::Basename; use Getopt::Std; my %opts; getopt ('il', \%opts); exit 0 unless $opts{i} or $opts{l}; copy ($opts{i}, \*STDOUT) if $opts{i}; if ($opts{l}) { my $dir = dirname ($opts{l}); $ARGV[0] = $opts{l}; while (<>) { chomp; copy ($_, \*STDOUT) or copy ("$dir/$_", \*STDOUT) or warn "Copy of $_ failed: $!\n"; } }And here is a full ACD for the "application" above:
appl: Files [ documentation: "Copying and merging files to standard output" groups: "Testing" nonemboss: "Y" executable: "copy-files.pl" ] infile: input [ additional: "Y" qualifier: "i" ] filelist: list [ additional: "Y" qualifier: "l" ] outfile: output [ default: "stdout" ]In the filelist data type, all input data are considered direct data. If you need a list of reference data, you would need to write your own input adaptor (see the comment "input_adaptor <class-name>" above, and look for inspiration in the existing input adaptor org.soaplab.services.adaptor.InputManyFiles - the one actually used to implement the functionality of the filelist data type).
A specific attribute for this data type is:
Here is an example showing the input data types:
string: text [ additional: "Y" default: "this is a default" prompt: "An optional string with a default value" ] string: text_no_default [ standard: "Y" prompt: "A mandatory string input" ] integer: number_int [ additional: "Y" default: 42 ] float: number_float [ additional: "Y" precision: 2 default: 30.12 ] boolean: bool_false [ additional: "Y" default: false ] boolean: bool_true [ additional: "Y" default: true ] boolean: bool_no_default [ additional: "Y" ]You can try a service created from the above - the service name (in the default Soaplab2 distribution) is testing.inputtypes.
-menu F
-format jpg,svg
The specific attributes for this data type are:
Do not confuse it with attributes delimiter and codedelimiter - they both define how to write values in an ACD file but not how the resulting parameter will look like.
Each individual value can be a bit more complex: it can consist of two values, separated by attribute codedelimiter (default is colon). The first value is the one that matters - one that is accepted from client and that appears in the resulting parameter. The second value is just a better human-readable text for the first value (that may be useful for some client's GUIs).
appl: Lists [ documentation: "How to use lists" groups: "Testing" nonemboss: "Y" executable: "echo" comment: defaults ] list: format [ additional: "Y" default: "png" values: "canon; dot; fig; gd; gif; hpgl; imap; jpg; mif; mp; pcl; pic; plain; png; ps; svg" prompt: "Graphical format" comment: "separator |" ] list: menu [ default: "V" minimum: "1" maximum: "1" values: "F--fungi,I--insect,P--plant,V--vertebrate,O--other,C--Custom" delimiter: "," codedelimiter: "--" prompt: "Transcription Factor Class" information: "Select class" ] outfile: output [ additional: "Y" default: "stdout" ]If the service (created form the ACD above) is called (using the command-line client) as:
build/run/run-cmdline-client -name testing.lists -w -r -format_canon -format_fig menu Fthe resulting command line will look like:
-format canon|fig -menu F
For example, here is an ACD file with all possible streams defined. The application (a Perl script) is distributed with Soaplab2 in file run/all-streams.pl.
appl: Streams [ documentation: "Filtering stdin into stdout and stderr streams" groups: "Testing" nonemboss: "Y" executable: "all-streams.pl" ] infile: input [ additional: "Y" default: "stdin" ] outfile: std_output [ additional: "Y" default: "stdout" ] outfile: std_errors [ additional: "Y" default: "stderr" ]
You can influence where these "URL results" will be
served from by setting several run-time configuration properties (look
for properties
Once you specify an output type, you should make sure that the underlying application produces such type. Soaplab2 itself treats data as binary data if the ACD file says so - but it does not create an array of data (except for graphical EMBOSS programs that produce images on several pages). If you want them you need to create your own plug-in. An example of such plug-in (doing nothing useful) is the class class org.soaplab.samples.OutputTypesJob. And here is a complete ACD file for it:
appl: AllOutputTypes [ documentation: "Showing how a plugin can create all kinds of outputs" groups: "Plugins,Testing" nonemboss: "Y" comment: "class org.soaplab.samples.OutputTypesJobFactory" ] infile: input [ standard: "Y" help: "This input will be copied to several outputs. <p> For some of outputs, it will be even replicated (how many times, it depends on the parameter <em>count</em>)." ] integer: count [ additional: "Y" default: "3" prompt: "How many times to replicate input in the array outputs" comment: defaults ] outfile: simple_text_output [ ] outfile: simple_binary_output [ comment: bindata ] outfile: array_text_output [ comment: "output_type String[]" ] outfile: array_binary_output [ comment: bindata comment: "output_type byte[][]" ]
appl: Results [ documentation: "Testing an output adaptor" groups: "Testing" nonemboss: "Y" executable: "echo" ] string: param [ parameter: "Y" default: "this is a result" comment: defaults ] outfile: output [ additional: "Y" default: "stdout" comment: "output_adaptor org.soaplab.services.testing.TestingDataAdaptor" ]
For example, a parameter attribute extension for an output may depend on the value (sent by a client in the run-time) of an input parameter format. It would be nice to specify in the ACD file:
outfile: output [ extension: ${format} ]
If you are a client developer, you may wish to know also how the clients can access service metadata and how to benefit from them.
In any case, because this chapter is about the building, it is recommended to look first into the build guide what is the Ant tool, how to use it, and what are the built-time properties.
If you are wondering where are the ACD files for EMBOSS, they are not part of the Soaplab2 distribution but part of the EMBOSS itself. More about using them is in the EMBOSS notes.
Name your ACD files with names without spaces and other strange characters (that's because the file name become also part of the resulting Web service name).
Or, create your ACD files in your own directory and use
They are in the converter's configuration file src/etc/config/generator.config.template. Every time a converter is invoked, this file is copied into metadata/old.generator/al.Cfg.pl. During this time, Ant will substitute some data there from the following built-time properties:
This property can be used together with the built-time
property
The same configuration file contains also a name of a directory where the XML files will be generated. There is, however, no property to change it. The target directory is metadata/generated.
ant genBy the way, the task gen is also called when you install Soaplab2 (from the task install).
To convert only ACD files from some of those directories, use individual tasks:
ant gensowa ant gengowlab ant genebi ant gentestAll the gen tasks create metadata in directory metadata/generated. For each category there will be a sub-directory.
The gen... tasks can by customized by setting some built-time properties:
ant "-Dsa=helloworld dot sleep" gensowaBe aware that the converter also creates an application list. If property
The
The property
ant -Dsl=MyApps.xml gen
ant -Dsl=MySowaApps.xml gensowa ant -Dsowa.sl=MySowaApps.xml gensowaWith the general gen task, use the not-shortened properties:
ant -Dsowa.sl=MySApps.xml -Dgowlab.sl=MyGApps.xml -Debi.sl=MyEApps.xml genDefault names are:
Typical example: Create your ACD files in a directory myacd (or copy there some existing ACDs from the Soaplab2 distribution - as I did for this documentation). Then run:
ant -Dacd.dir=myacd -Dsl=MyApps.xml gensowaYou may see a similar report to this one:
_gen: [copy] Copying 1 file to /home/senger/soaplab2/metadata/old.generator [echo] /home/senger/soaplab2/metadata/old.generator/acd2xml -d -l MyApps.xml -p . -r myacd dot helloworld [acd2xml] Processing dot... [acd2xml] using myacd/dot.acd [acd2xml] (generated into module graphics) [acd2xml] Created: /home/senger/soaplab2/metadata/generated/graphics/dot_al.xml [acd2xml] Processing helloworld... [acd2xml] using myacd/helloworld.acd [acd2xml] (generated into module classic) [acd2xml] Created: /home/senger/soaplab2/metadata/generated/classic/helloworld_al.xml [acd2xml] Created: /home/senger/soaplab2/metadata/generated/MyApps.xml BUILD SUCCESSFUL Total time: 4 seconds
Once the ACD to XML converter has created metadata files, you have to
put the names of the application list files (not the names of
the individual service metadata files) in the run-time configuration
file as one or more property
base.dir = /home/senger/soaplab2 metadata.dir = ${base.dir}/metadata/generated applist = ${metadata.dir}/OtherApplications.xml applist = ${metadata.dir}/GowlabApplications.xml applist = ${metadata.dir}/EBIApplications.xml applist = ${metadata.dir}/EMBOSSApplications.xml


Invoice<EMAIL_ADDRESS>|
@Model.Date |
Southwest Parent Child Collective
6053 SW 55th Drive
Portland, OR 97221
Tax ID: 93-0748219
Billed to:
@Model.Family.Name
@Model.Family.StreetAddress
@Model.Family.City @Model.Family.State @Model.Family.Zip
@Each.Model.Parents
@Current.Email
@EndEach
| Due @Model.DueDate | |||
| Description | Rate | Quantity | Amount |
|---|---|---|---|
| @Current.Description @Current.Notes | @Current.UnitPrice | @Current.Quantity | @Current.Amount |
| Total Due @Model.DueDate | @Model.AmountDue |
Please write "CAMP 2015" in the notes field of your check. Thank you!
Send your payment to:
SWPCC
c/o Andy Sherwood
5316 SW Boundary St
Portland, OR 97221


A bug introduced in Jenkins 1.645 and 1.642.2 caused Jenkins to send anonymous usage statistics data to the Jenkins project even if the administrator opted out of it.
- Which versions are affected?
- What data is being collected?
- How is the data transferred and processed? Who can access the data?
- What are you doing with the usage data? Why are you even collecting it?
- What about private data, like names of plugins we wrote ourselves?
- What are you doing to fix this?
- What about the data you have already collected?
- What can I do about this? Is there a workaround to really disable submission of usage statistics?
- This was fixed in 1.653, why are you only notifying us now?
Which versions are affected?
The following versions of Jenkins do not properly honor the setting to opt out of usage statistics collection, resulting in data being submitted even when disabled:
- Main line: 1.645 (inclusive) to 1.652 (inclusive)
- LTS: 1.642.2 and 1.642.3
What data is being collected?
- The public instance ID
- Name and version of the servlet container
- The version of Jenkins
- For all connected nodes, including master,
- The values of the system properties
java.vm.vendor
,java.vm.name
,java.version
- The number of executors
- The operating system as shown on the "Manage Nodes" page in Jenkins, e.g.
Linux (amd64)
- The values of the system properties
- For all installed and enabled plugins,
- the short name/artifactId (e.g.
git
, oremail-ext
) - the version
- the short name/artifactId (e.g.
- For all item/job types,
- the class name (e.g.
hudson.model.FreestyleProject
) - the number of items/job of that type in your Jenkins
- the class name (e.g.
For an example of this data, see this wiki page.
We do not collect information like job names, user names, host names, IP addresses, build logs, build names or descriptions, etc.
How is the data transferred and processed? Who can access the data?
The data is encrypted, base64-encoded, and transferred via an AJAX request piggybacking on a user's browser.
The data is stored encrypted on the Jenkins project infrastructure, from where it is downloaded, decrypted, and filtered locally by one of the board members, and uploaded again. Only Jenkins board members have access to the decryption key to read the full, raw data.
Partially decrypted (see below) data was made available to the public at https://jenkins-ci.org/census/
What are you doing with the usage data? Why are you even collecting it?
We're currently providing the statistics at http://stats.jenkins-ci.org/jenkins-stats/ to everyone. They show various statistics, e.g. number of known Jenkins instances, which plugins are most popular, etc.
We're also providing mostly plugin-related statistics at http://stats.jenkins-ci.org/plugin-installation-trend/ that are used on the Jenkins wiki to indicate plugin popularity, as well as help plugin developers make choices about which versions of Jenkins to target as minimum compatible release.
Data such as operating system, servlet container and Java version help us make decisions on what platforms need to be supported by Jenkins.
In the future, we'd like to use the statistics to help tell us which versions of Jenkins/plugins are unpopular and downgraded from, to help us spot problems.
What about private data, like names of plugins we wrote ourselves?
Operating system names, servlet container names, and Java VM vendor, name, and version are always fully decrypted.
Item/job types and plugin names are only decrypted based on what's essentially a whitelist:
- In the case of job types, all types starting with
hudson.
,org.jvnet.hudson
, andorg.jenkinsci
are currently considered public. If you follow the convention to use package names corresponding to your organization (e.g. com.yourcorp.whatever), these types will not be decrypted. - In the case of plugins, only plugin names for plugins available on the Jenkins community update site are decrypted.
While the processed data still contains entries for all plugins and item types, those determined to be private are still encrypted, and not part of output such as the stats site.
Again: We do not collect information like job names, user names, host names, IP addresses, build logs, build names or descriptions, etc.
What are you doing to fix this?
As soon as we found out about this bug, we disabled third party access to the partially decrypted data at http://jenkins-ci.org/census
We also changed the decryption/filter tool to discard all data submitted by the affected Jenkins versions, to make sure we're not using data from users not upgrading Jenkins and not applying the workaround described below. We can't do anything about the encrypted data we will continue to receive, as we don't know what version it's about until we decrypt it. So we discard it in the same processing step as decryption.
We'll also release an unscheduled LTS release, 1.642.4, later this week, so users on the LTS line can upgrade to that version to get the bug fix. This LTS release will not impact the regular LTS schedule, i.e. we still plan to release 1.651.1 in two weeks.
What about the data you have already collected?
We regenerated all statistics data for January and February, applying the version filter mentioned above, to make sure we remove all traces of the affected versions' usage data from the data we have.
What can I do about this? Is there a workaround to really disable submission of usage statistics?
If you are running one of the affected versions, the best medium-term solution is to upgrade. The bug does not affect Jenkins 1.653 or newer, or Jenkins LTS 1.642.4 or newer.
In the short term, you can immediately disable submission of usage statistics to the Jenkins project by running the following script in Manage Jenkins » Script Console:
hudson.model.UsageStatistics.DISABLED = true
This will immediately disable usage data submission until you restart Jenkins. To make this permanent, change your Jenkins startup script so it passes a system property to the java process:
java -Dhudson.model.UsageStatistics.disabled=true -jar …/jenkins.war
For information how to do this when using one of the installers/packages, see the installer/package documentation here.
To verify that usage stats submission is disabled, run the following script in Manage Jenkins » Script Console and confirm the result is true:
println hudson.model.UsageStatistics.DISABLED
This was fixed in 1.653, why are you only notifying us now?
We noticed an unusual drop in usage data for the month of February. We were initially unable to determine the cause, but reverted a robustness-improving commit that changed code related to usage data submission anyway. This change went into 1.653. While again looking for the cause of the drop of reported usage data earlier this week, we determined that it was really only affecting versions that included this commit, so we took a closer look at it -- and found that it actually inverted the meaning of the usage data option: Only instances opted out of it did now submit data.


$username
$username : string
Qualtrics username
This is the email address that serves as the Qualtrics username that has access to the api
Qualtrics class
This class implements wrapper functions for all of the API methods available in the Qualtrics 2.2 API. It is meant to be used from within Laravel 4.
getResponseCountsBySurvey(string[] $params) : object
Gets the total number of responses for a survey in a given date range
$params = array(
'Format' => 'JSON', // Must be XML or JSON, defaults to JSON
'StartDate' => 'YYYY-MM-DD', // Start date of responses to include
'EndDate' => 'YYYY-MM-DD', // (optional) defaults to today's date
'SurveyID' => 'SV_9LC2rrUZT8c2EiF', // the Qualtrics ID of the survey in question.
);
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
Returns a PHP object with the resulting counts of "Auditable", "Generated" and "Deleted" responses
getUserInfo(string[] $params) : object
Gets information about a user
$params = array(
'Format' => 'JSON', // Must be XML or JSON, defaults to JSON
);
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
Information about the user in question
addRecipient(string[] $params) : object
Add a new recipient to a panel
$params = array(
'Format' => 'JSON', // Must be XML or JSON, defaults to JSON
'LibraryID' => 'GR_6G4LaoiroGxHH12', // Start date of responses to include
'PanelID' => 'ML_5yIfnFP0soZK3GJ', // Start date of responses to include
'FirstName' => 'Douglas', // Recipient's first name
'LastName' => 'Crockford', // Recipient's last name
'Email' =><EMAIL_ADDRESS>// Recipient's email address
'ExternalDataRef' => 'crockdx', // (optional)
'Language' => 'EN', // (optional) language, defaults to EN
'ED[***]' => '@dougcrockford', // (optional) embedded data, e.g. ED[twitterID]
// multiple values for ED possible
);
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
Returns true on success
createDistribution(string[] $params) : object
Creates a distribution for survey and a panel. No emails will be sent. Distribution links can be generated later to take the survey.
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
An object containing the DistributionID
createPanel(string[] $params) : object
Creates a new Panel in the Qualtrics System and returns the ID of the new panel
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
Returns the ID of the new panel
getPanel(string[] $params) : object
Gets all the panel members for the given panel
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
The panel members and any requested data
deletePanel(string[] $params) : object
Deletes a panel
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
The results of the deletion process
getPanelMemberCount(string[] $params) : object
Gets the number of panel members
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
The number of members of the panel
getPanels(string[] $params) : object
Gets all of the panels in a given library
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
The panels in the library
getRecipient(string[] $params) : object
Gets a representation of the recipient and their history
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
The representation of the recipient
importPanel(string[] $params) : object
Imports a csv file as a new panel
Imports a csv file as a new panel (optionally it can append to a previously made panel) into the database and returns the panel id. The csv file can be posted (there is an approximate 8 megabytes limit) or a url can be given to retrieve the file from a remote server. The csv file must be comma separated using " for encapsulation.
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
Contains the new PanelID, a count of imported recipients, and a count of ignored recipients
removeRecipient(string[] $params) : object
Removes a specified panel member from a specified panel
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
The representation of the recipient
sendReminder(string[] $params) : object
Sends a reminder through the Qualtrics mailer to the panel or individual as specified by the parent distribution Id
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
The email distribution ID and distribution queue id
sendSurveyToIndividual(string[] $params) : object
Sends a survey through the Qualtrics mailer to the individual specified
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
The email distribution ID and distribution queue id
sendSurveyToPanel(string[] $params) : object
Sends a survey through the Qualtrics mailer to the panel specified
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
The email distribution ID and distribution queue id
updateRecipient(string[] $params) : object
Updates the recipient’s data—any value not specified will be left alone and not updated
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
Returns true on success or WP_Error on failure
getLegacyResponseData(string[] $params) : object
Returns all of the response data for a survey in the original (legacy) data format
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
An object representing the responses for the specified panel members
getSurvey(string[] $params) : object
This request returns an xml export of the survey. NOTE: Custom response format!
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
On object representing the survey.
getSurveys(string[] $params) : object
This request returns a list of all the surveys for the user
| string[] | $params | An array of named parameters needed to complete the request (see code above) |
Thrown when one of the required parameters does not exist in the $params array
A list of surveys available to the user
parse_csv(string $csv_string, string $delimiter, boolean $skip_empty_lines, boolean $trim_fields) : array
Converts csv into 2-dimensional array
| string | $csv_string | The CSV string to be parsed |
| string | $delimiter | The delimiter to be used for parsing, defaults to comma |
| boolean | $skip_empty_lines | Whether or not to skip empty lines, defaults to true |
| boolean | $trim_fields | Whether or not to trim() fields, defaults to true |
The two-dimensional array representing the CSV
parseResponse(resource $ch, string $result) : array
Parses the data returned from a Qualtrics request.
| resource | $ch | The curl handle associated with the request |
| string | $result | The text returned from the request |
Thrown when an XML response cannot be parsed correctly
Thrown when Qualtrics returns a response with an unrecognized content-type header
Returns an array with the headers and parsed response text
request(string $request, string[] $params) : object
Private function that makes a generic Qualtrics API request.
This function takes an API function name and an array of parameters and makes an attempt to fetch the data using the Qualtrics API.
| string | $request | The name of the API function to be called |
| string[] | $params | An array which contains any necessary parameters for the API call |
Thrown when the curl library is not installed on the server
Thrown when Qualtrics returns an error with the request
Thrown when the curl request is unsuccessful, e.g. request timeout
Returns an object representing the requested data


Success!
The upload and validation of your data was successful. View a customized report based on your data in pdf format.
{# Debug views: html report, heatmap,time series plot,load duration,kW histogram,week summary,Tout vs. kWh #}
{% extends "base.html" %} {% block title %}: Explore your data{% endblock %} {% block head %}{{ super() }}{% endblock %} {# note that the head looks for the title variable. #} {% block content %} {% if building.attr.bestWBAN %} {% set bestWBAN = building.attr.bestWBAN %} {% else %} {% set bestWBAN = ['NA',0.0,'NA','NA','NA','NA','NA','NA','Unknown','NA'] %} {% endif %}


{% extends "layout/base.html" %} {% block content %}
Accurate and quick descriptions of Google Street View imagery based on machine learning algorithms and human supervision.
Find the entrance to that restaurant you're looking for. Ground truth from neighborhood signs gives more precise locations than address geocodes.
Image labels can be passed to screen readers for the visually impaired or translated into different languages. Soon, you'll be able to reverse search for all the cats found in Street View images.


Ricky the Butler
Add Command
Command
Url
Developer Window
Binding Address
Port
Log command output
Save
Path
<IP_ADDRESS>


Many commands in mitmproxy and mitmdump take a filter expression. Filter expressions consist of the following operators:
| ~a | Match asset in response: CSS, Javascript, Flash, images. |
| ~b regex | Body |
| ~bq regex | Request body |
| ~bs regex | Response body |
| ~c int | HTTP response code |
| ~d regex | Domain |
| ~e | Match error |
| ~h regex | Header |
| ~hq regex | Request header |
| ~hs regex | Response header |
| ~m regex | Method |
| ~q | Match request with no response |
| ~s | Match response |
| ~t regex | Content-type header |
| ~tq regex | Request Content-Type header |
| ~ts regex | Response Content-Type header |
| ~u regex | URL |
| ! | unary not |
| & | and |
| | | or |
| (...) | grouping |
URL containing "google.com":
google\.com
Requests whose body contains the string "test":
~q ~b test
Anything but requests with a text/html content type:
!(~q & ~t \"text/html\")


CSS Color Tutorial
home
Background Color
Hi my name is Folau
Text Color
Hi my name is Folau
Border Color
Hi my name is Folau
RGB Color
Hi my name is Folau
Hex Color
Hi my name is Folau
HSL Color
Hi my name is Folau


$1.3 Trillion Dollars Lost in Los Angeles County due to Police Violence
Los Angeles County killed 905 people between the years 2000 and 2020. 862 people were between the average working ages of 16 to 64. Combined, the years of income-potential lost to Black, Latinx, Asian and white communites amounts to $1,368,019,545.
Black and Latinx communites lost $1 trillion in income-potential between them, alone; combined they make up the bulk of total income loss, depite earning less per capita than their white and Asian counterparts.
As the national conversation about police brutality turns to defunding the police, Los Angeles County serves as a prime example of the economic impact police have on local communites. LAPD, alone, is set to take $3 billion of the city's general fund-- more than half of the city's unrestricted revenue for 2020-2021. LAPD, the largest police dept. in Los Angeles County, also kills citizens at nearly double the rate of almost any other police dept., nationwide. A look at the total years of life lost within these communites further highlights the financial indignities local law enforcement puts upon the public.
96% of those killed by LA County police officers are men. Since men make 20 to 50 cents more per dollar than women, and the graph below includes women's income in the total yearly average, the financial damage caused by these killings is a conservsative estimate.


Chris Doe
Books due in
Full Name
Chris Doe
Library Card ID
3452375
Phone
9876543210
<EMAIL_ADDRESS>Books previously borrowed
Romeo and Juliet
Annie Frank's Diary
One Day
Life of Pi
Harry Potter and The order of Pheonix
Books wishlist
Pride and Prejudice
THe priest
Into the wild
Percy Jackson
Twilight


theHarvester
A tool designed to be used in the early stages of a penetration test.
More information: home page
- Gather information on a domain using Google:
theHarvester --domain domain_name --source google
- Gather information on a domain using multiple sources:
theHarvester --domain domain_name --source google,bing,crtsh
- Change the limit of results to work with:
theHarvester --domain domain_name --source google --limit 200
- Save the output to two files in XML and HTML format:
theHarvester --domain domain_name --source google --file output_file_name
- Output all available options:
theHarvester --help


This is my website. Thanks for visiting!
I set up a ring on 3D CAD and printed it out with a 3D printer. I added the words, LJJ, which is my hero's name on the ring, but it seemed to small to see clearly.


Enter in New Store Information
Enter City Name Here:
Minimum Number of Customers:
Maximum Number of Customers:
Average Cookies Sold/Day:
Submit


Will there be money after the end of the world?
Whatever your personal fantasy of the end of civilization entails (natural disasters, pandemics, meteor strikes, nuclear war, or some other black swan theory), if humanity survives catastrophe with the memory of modern civilization, new currencies will most likely be established in the post-apocalyptic society.
Economists, doomsday preppers, crypto-anarchists, and others are already making their predictions, from gold and silver, to bartered goods, to bitcoins and bottle caps. Let's take a look at the future of cash.
Old money, new money
Along the lines of the cartoon fan theory that The Flintstones wasn't set in the Stone Age, but in the same world as the Jetsons after nuclear war, some are predicting that the post-apocalyptic world won't look much different from the premodern one.
In Belgian economist Henri Pirenne's Economic and social history of medieval Europe, he references the German economic concept of "Naturalwirtschaft," or "Natural economy," which references an economic structure similar to that of Europe in the Middle Ages, in which "the part played by money was then so small as to be almost negligible." This isn't to say that currency didn't exist, but so few people had it in meaningful enough amounts that it wasn't used as much as bartering or reciprocal trade systems.
The return to a natural economy of bartering in the post-apocalyptic future is one possibility that survivalists (also known as preppers, the folks who are stocking bunkers with canned goods and weapons right at this moment to prepare for when the Shit Hits The Fan (or SHTF), which stands for whatever their personal vision of societal collapse is) are looking forward to.
Throughout the prepper blogosphere, shopping lists abound of the types of things that will be more than worth their weight in gold (more on gold itself, later). As The Prepper Dome points out on their list "Six Kinds of Currency That Might Emerge After The Collapse,": you can’t eat gold. The list of six includes:
- Salt: citing its use to pay Roman soldiers and its multiple medical and culinary uses
- Fur: because in many countries squirrel pelts were essentially traded like dollar bills
- Tobacco: it’s portable, addictive, and the modern use of cigarettes for bartering purposes is obvious
- Knives: are portable, durable, useful, and were traded as currency in ancient China
- Alcohol: can be used for fuel, preserving food, cleaning wounds, as well as being divisible and durable
- Herbs and spices: particularly marijuana, since it can be eaten and used for medicine as well as grown for hemp fibers can be used to make rope, clothing paper, and fuel
Prepper Zine has a list of 25 barterables, including many daily hygiene items like soap and toothbrushes, as well as condoms, which are actually a survivalist staple: not only because bunker babies would definitely complicate your post-apocalypse plans, but because they have tons of non sexual uses. Other health-related items are also a great idea to stockpile, like painkillers, antibiotics, and bandages, since no matter how the infrastructure fails, it's likely that pandemics will follow.
Although The Prepper Journal predicts that ammunition will be traded like bullets for beans in the Wild West, others warn to never barter ammunition or firearms, for obvious reasons.
Silver, gold, and crypto
While the super-rich are buying land in New Zealand to ride out the apocalypse, the rest of us have to figure out whether we'll be able to get money out of banks or if we should start burying tin cans full of gold coins in our backyards.
The blog Off Grid Survival argues that cash will still be king following a crisis situation like a natural disaster, where they imagine credit and debit cards won't work, but you would still be able to throw small bills at a convenience store counter while filling your car with gas and grabbing some emergency jerky during a mass evacuation.
There are a few arguments against predicting a cash for gold economy in a world W.R.O.L. (another prepper acronym, meaning Without Rule Of Law). Forbes' "How To Doomsday Prep Like An Economist" advises that gold will be practically useless because of income effects: "A disaster bad enough to make developed nation currencies completely worthless is also going to absolutely obliterate the demand for gold because gold is a luxury and incomes will plummet." Others make the argument that while gold has more intrinsic value, silver is a better long-term investment.
According to an attendee of this year's Silver & Gold Summit in San Francisco, cryptocurrency advocates were out in full force promoting the idea that bitcoin could be the new gold. "Buy bitcoin" has recently surpassed "buy gold" in Google search results rankings, so it's clearly not only the experts who are thinking all that glitters is not.
There is still a perception amongst economists that bitcoin is for bad actors and is too unstable to be considered a serious investment. Jim Rickards, author of Currency Wars: The Making of the Next Global Crisis, was quoted in an interview for Business Insider about gold versus bitcoin characterizing it as "a utility token for criminals, terrorists, money launderers, tax evaders." He believes the price is being manipulated, that the amount of energy required to mine cryptocurrency will soon hit a wall, and that the subjective value has yet to be tested against a recession or financial panic (much less doomsday).
Goldman Sachs also says bitcoin is not the new gold, but even though Jamie Dimon, the chief executive of JP Morgan, said he would fire anyone who traded bitcoins, JP Morgan analysts have expressed the opinion that it could be.
For survivalists who believe they could still access or create enough electricity to power their own internet, bitcoin has the same appeal as gold in that it's self-banking and a finite quantity of it will exist, protecting against inflation and deflation. That is, unless quantum computing to create more advanced cryptocurrencies or asteroid mining to get more gold are features of the future.
Most bitcoin vs. gold investment comparisons (more reading from Coindesk and APMEX, who represent both sides) end up saying a diverse portfolio is best, and that there's no reason not to go for both as long as you aren't investing more than you can afford to lose. All of the above considered, do not rely on MONTAG Magazine for investment advice, because the last currency we're going to explore is totally fictional and made out of trash.
Bottle caps
In the 1950s-inspired post-nuclear world of the Fallout video games, you emerge from an underground bunker in the 22nd or 23rd century, and the currency of the land is bottle caps. Not just any bottle caps: Nuka-Cola bottle caps with 21 crimps and ridges.
According to the fans who run the Fallout Wikia and Gamepedia entry for Fallout, bottling manufacturers were lost in the Great War, a nuclear war that took place in the year 2077, and adoption of the caps happened between 2093 and 2103 in the city called the Hub. Because the facilities to produce them were lost, bottle caps made an ideal currency because they had a fixed supply, were difficult to counterfeit (though counterfeit coins do exist in the game), and were originally backed by bottles of water, an extremely valuable resource in the irradiated deserts.
When the preorder for Fallout 4 was announced, one industrious fellow who had been collecting bottle caps for over 7 years sent Bethesda Game Studios 2,240 bottle caps, 11.2 pounds of them, and while they did allow him to pay for a preorder in the game's currency, they also made it clear that they did not wish for the stunt to be repeated.
Analysis of this transaction by The Game Theorists, a YouTube channel that actually does the math on silly questions like how much fictional future video game currencies would be worth in real money, found the actual worth of 2,240 bottle caps by two different methods.
They first used calculations from the 2010 game Fallout: New Vegas, in which gold bars are sold for 10,539 caps each, and the price of gold per ounce in 1961 (based on the songs playing on the radio in the game's soundtrack), puts the price at $1.67 per bottle cap, which due to inflation would be $52 per cap today. So paying 2,240 bottle caps for a $60 video game preorder was overpaying by about $116,420.00.
However, using a second calculation, with the real world worth of bottle caps according to the value of carbon steel, it would take 57,142 caps to equal $1, so a $60 preorder should have cost 3,428,571 caps (he paid less than a tenth). The obvious calculation of how much he actually paid for the game brings the cap to dollar value at approximately 33 and 1/3 caps to $1.
So it may be impossible to predict, if our post-apocalyptic society chooses an arbitrary item like bottle caps as currency, what we should start hoarding now. It could be Starbucks plastic coffee stirrers, Keurig coffee capsules, or Bic pens. When the SHTF, the currency of the land could be bitcoin, gold, or bartered goods, but we won't know until after it happens. Maybe money won't matter at all, and all the economists, preppers, and bitcoin hoarders will be wrong. Maybe the most valuable thing we will have is each other.


Neil Ruddock and ex-wife embroiled in latest leg of 14-year family court fight
Ruddock and Sarah Ruddock, who divorced more than a decade ago, were both at a private hearing overseen by a judge in the Central Family Court in London on Tuesday
by James Rodger, https://www.facebook.com/jamesrodgerjournalistFormer footballer Neil Ruddock and his ex-wife are embroiled in the latest leg of a 14-year family court money fight following the breakdown of their marriage.
Ruddock and Sarah Ruddock, who divorced more than a decade ago, were both at a private hearing overseen by a judge in the Central Family Court in London on Tuesday.
District Judge Alun Jenkins allowed a journalist to attend the hearing but imposed limits on what could be reported.
The judge said Ruddock and his ex-wife, who still uses the name Mrs Ruddock, could be named in media reports.
But he said no detail of evidence discussed could be revealed.
Mrs Ruddock was the applicant in the case and Ruddock the respondent.
Ruddock, 51, was represented by a barrister.
Mrs Ruddock, 49, represented herself.
The case was listed by a number which indicated that the litigation began in 2005.
Ruddock, a central defender who won one England cap, played for clubs including Southampton, Tottenham, Liverpool and West Ham during a career spanning more than 12 years.
He took part in the 2019 series of the BBC's Celebrity MasterChef.


Item Pipeline¶
After an item has been scraped by a spider, it is sent to the Item Pipeline which processes it through several components that are executed sequentially.
Each item pipeline component (sometimes referred as just “Item Pipeline”) is a Python class that implements a simple method. They receive an item and perform an action over it, also deciding if the item should continue through the pipeline or be dropped and no longer processed.
Typical uses of item pipelines are:
- cleansing HTML data
- validating scraped data (checking that the items contain certain fields)
- checking for duplicates (and dropping them)
- storing the scraped item in a database
Writing your own item pipeline¶
Each item pipeline component is a Python class that must implement the following method:
-
process_item
(self, item, spider)¶ This method is called for every item pipeline component.
process_item()
must either: return a dict with data, return anItem
(or any descendant class) object, return a Twisted Deferred or raiseDropItem
exception. Dropped items are no longer processed by further pipeline components.Parameters:
Additionally, they may also implement the following methods:
-
open_spider
(self, spider)¶ This method is called when the spider is opened.
Parameters: spider ( Spider
object) – the spider which was opened
-
close_spider
(self, spider)¶ This method is called when the spider is closed.
Parameters: spider ( Spider
object) – the spider which was closed
-
from_crawler
(cls, crawler)¶ If present, this classmethod is called to create a pipeline instance from a
Crawler
. It must return a new instance of the pipeline. Crawler object provides access to all Scrapy core components like settings and signals; it is a way for pipeline to access them and hook its functionality into Scrapy.Parameters: crawler ( Crawler
object) – crawler that uses this pipeline
Item pipeline example¶
Price validation and dropping items with no prices¶
Let’s take a look at the following hypothetical pipeline that adjusts the
price
attribute for those items that do not include VAT
(price_excludes_vat
attribute), and drops those items which don’t
contain a price:
from scrapy.exceptions import DropItem
class PricePipeline(object):
vat_factor = 1.15
def process_item(self, item, spider):
if item['price']:
if item['price_excludes_vat']:
item['price'] = item['price'] * self.vat_factor
return item
else:
raise DropItem("Missing price in %s" % item)
Write items to a JSON file¶
The following pipeline stores all scraped items (from all spiders) into a
single items.jl
file, containing one item per line serialized in JSON
format:
import json
class JsonWriterPipeline(object):
def open_spider(self, spider):
self.file = open('items.jl', 'w')
def close_spider(self, spider):
self.file.close()
def process_item(self, item, spider):
line = json.dumps(dict(item)) + "\n"
self.file.write(line)
return item
Note
The purpose of JsonWriterPipeline is just to introduce how to write item pipelines. If you really want to store all scraped items into a JSON file you should use the Feed exports.
Write items to MongoDB¶
In this example we’ll write items to MongoDB using pymongo. MongoDB address and database name are specified in Scrapy settings; MongoDB collection is named after item class.
The main point of this example is to show how to use from_crawler()
method and how to clean up the resources properly.:
import pymongo
class MongoPipeline(object):
collection_name = 'scrapy_items'
def __init__(self, mongo_uri, mongo_db):
self.mongo_uri = mongo_uri
self.mongo_db = mongo_db
@classmethod
def from_crawler(cls, crawler):
return cls(
mongo_uri=crawler.settings.get('MONGO_URI'),
mongo_db=crawler.settings.get('MONGO_DATABASE', 'items')
)
def open_spider(self, spider):
self.client = pymongo.MongoClient(self.mongo_uri)
self.db = self.client[self.mongo_db]
def close_spider(self, spider):
self.client.close()
def process_item(self, item, spider):
self.db[self.collection_name].insert_one(dict(item))
return item
Take screenshot of item¶
This example demonstrates how to return Deferred from process_item()
method.
It uses Splash to render screenshot of item url. Pipeline
makes request to locally running instance of Splash. After request is downloaded
and Deferred callback fires, it saves item to a file and adds filename to an item.
import scrapy
import hashlib
from urllib.parse import quote
class ScreenshotPipeline(object):
"""Pipeline that uses Splash to render screenshot of
every Scrapy item."""
SPLASH_URL = "http://localhost:8050/render.png?url={}"
def process_item(self, item, spider):
encoded_item_url = quote(item["url"])
screenshot_url = self.SPLASH_URL.format(encoded_item_url)
request = scrapy.Request(screenshot_url)
dfd = spider.crawler.engine.download(request, spider)
dfd.addBoth(self.return_item, item)
return dfd
def return_item(self, response, item):
if response.status != 200:
# Error happened, return item.
return item
# Save screenshot to file, filename will be hash of url.
url = item["url"]
url_hash = hashlib.md5(url.encode("utf8")).hexdigest()
filename = "{}.png".format(url_hash)
with open(filename, "wb") as f:
f.write(response.body)
# Store filename in item.
item["screenshot_filename"] = filename
return item
Duplicates filter¶
A filter that looks for duplicate items, and drops those items that were already processed. Let’s say that our items have a unique id, but our spider returns multiples items with the same id:
from scrapy.exceptions import DropItem
class DuplicatesPipeline(object):
def __init__(self):
self.ids_seen = set()
def process_item(self, item, spider):
if item['id'] in self.ids_seen:
raise DropItem("Duplicate item found: %s" % item)
else:
self.ids_seen.add(item['id'])
return item
Activating an Item Pipeline component¶
To activate an Item Pipeline component you must add its class to the
ITEM_PIPELINES
setting, like in the following example:
ITEM_PIPELINES = {
'myproject.pipelines.PricePipeline': 300,
'myproject.pipelines.JsonWriterPipeline': 800,
}
The integer values you assign to classes in this setting determine the order in which they run: items go through from lower valued to higher valued classes. It’s customary to define these numbers in the 0-1000 range.


Ali Muhammad
Hey there! I'm Ali
A self taught software developer and open source contributor from pakistan.
root@ali:~$ work-in-progress
root@ali:~$
_