The Pappy Proxy¶
Introduction¶
The Pappy (Proxy Attack Proxy ProxY) Proxy is an intercepting proxy for performing web application security testing. Its features are often similar, or straight up rippoffs from Burp Suite. However, Burp Suite is neither open source nor a command line tool, thus making a proxy like Pappy inevitable. The project is still in its early stages, so there are bugs and only the bare minimum features, but it can already do some cool stuff.
Contributing¶
I am taking any and all feature requests. If you’ve used Burp and had any inconvenience with it, tell me about it and I’ll do everything in my power to make sure Pappy doesn’t have those issues. Or even better, if you want Burp to do something that it doesn’t already, let me know so that I can [STRIKEOUT:use it to stomp them into the dust] improve my project.
If you’re brave and want to try and contribute code, please let me know. Right now the codebase is kind of rough and I have refactored it a few times already, but I would be more than happy to find a stable part of the codebase that you can contribute to.
Another option is to try writing a plugin. It might be a bit easier than contributing code and plugins are extremely easy to integrate as a core feature. So you can also contribute by writing a plugin and letting me know about it. You can find out more by looking at the official plugin docs.
How to Use It¶
Installation¶
Pappy supports OS X and Linux (sorry Windows). Installation requires
pip
or some other command that can handle a setup.py
with
requirements. Once the requirements are installed, you can check that it
installed correctly by running pappy -l
to start the proxy.
$ git clone --recursive https://github.com/roglew/pappy-proxy.git
$ cd pappy-proxy
$ pip install .
Quickstart¶
Pappy projects take up an entire directory. Any generated scripts, exported responses, plugin data, etc. will be placed in the current directory so it’s good to give your project a directory of its own. To start a project, do something like:
$ mkdir test_project
$ cd test_project
$ pappy
Copying default config to directory
Proxy is listening on port 8000
pappy> exit
$ ls
data.db project_config.json
$
And that’s it! The proxy will by default be running on port 8000 and
bound to localhost (to keep the hackers out). You can modify the
port/interface in config.json
. You can list all your intercepted
requests with ls
, view a full request with vfq <reqid>
or view a
full response with vfs <reqid>
. Right now, the only command to
delete requests is filter_prune
which deletes all the requests that
aren’t in the current context (look at the sections on the
context/filter strings for more information on that).
 Pappy will use the default configuration settings and will create
a temporary data file in /tmp
to use. When you quit, the file will
be deleted. If you want to run Pappy in lite mode, run Pappy with either
-l
or --lite
.
Example:
$ pappy -l
Temporary datafile is /tmp/tmpw4mGv2
Proxy is listening on port 8000
pappy> quit
Deleting temporary datafile
$
Adding The CA Cert to Your Browser¶
In order for Pappy to view data sent using HTTPS, you need to add a
generated CA cert (certificate.crt
) to your browser. Certificates
are generated using the gencerts
command and are by default stored
in ~/.pappy/certs
. This allows Pappy to act as a CA and sign any
HTTPS certificate it wants without the browser complaining. This allows
Pappy to decrypt and modify HTTPS requests. The certificate installation
instructions are different for each browser.
Firefox¶
You can add the CA cert to Firefox by going to
Preferences -> Advanced -> View Certificates -> Authorities -> Import
and selecting the certificate.crt
file in the certs
directory.
Chrome¶
You can add the CA cert to Chrome by going to
Settings -> Show advanced settings -> HTTPS/SSL -> Manage Certificates -> Authorities -> Import
and selecting the certificate.crt
file in the certs
directory.
Safari¶
For Safari (on macs, obviously), you need to add the CA cert to your system keychain. You can do this by double clicking on the CA cert and following the prompts.
Internet Explorer¶
I didn’t search too hard for instructions on this (since Pappy doesn’t
support windows) and I don’t own a Windows machine to try this, so if
you have trouble, I’m not the one to ask. According to Google you can
double-click the cert to install it to the system, or you can do
Tools -> Content -> Certificates -> Trusted Root Certificates -> Import
.
Configuration¶
Configuration for each project is done in the config.json
file. The
file is a JSON-formatted dictionary that contains settings for the
proxy. The following fields can be used to configure the proxy:
| Key | Value |
|---|---|
data_file |
The file where requests and images will be stored |
debug_dir (optional) |
Where connection debug info should be stored. If not present, debug info is not saved to a file. |
cert_dir |
Where the CA cert and the private key for the CA cert are stored |
proxy_listeners |
A list of dicts which describe which ports the proxy will listen on. Each item is a dict with “port” and “interface” values which determine which port and interface to listen on. For example, if port=8000 and the interface is <IP_ADDRESS>, the proxy will only accept connections from localhost on port 8000. To accept connections from anywhere, set the interface to <IP_ADDRESS>. |
The following tokens will also be replaced with values:
| Token | Replaced with |
|---|---|
{DATADIR} |
The directory where Pappy’s data files are stored |
See the default config.json
for examples.
Generating Pappy’s CA Cert¶
In order to intercept and modify requests to sites that use HTTPS, you
have to generate and install CA certs to your browser. You can do this
by running the gencerts
command in Pappy. By default, certs are
stored ~/.pappy/certs
. This is also the default location that Pappy
will look for certificates (unless you specify otherwise in
config.json
.) In addition, you can give the gencerts
command an
argument to have it put the generated certs in a different directory.
| Command | Description |
|---|---|
gencerts [/path/to/put/certs/in] |
Generate a CA cert that can be added to your browser to let Pappy decrypt HTTPS traffic. Also generates the private key for that cert in the same directory. If no path is given, the certs will be placed in the default certificate location. Overwrites any existing certs. |
Browsing Recorded Requests/Responses¶
The following commands can be used to view requests and responses
| Command | Aliases | Description |
|---|---|---|
ls [a|<num> ] |
list, ls | List requests that are in the current context (see Context section). Has information like the host, target path, and status code. With no arguments, it will print the 25 most recent requests in the current context. If you pass ‘a’ or ‘all’ as an argument, it will print all the requests in the current context. If you pass a number “n” as an argument, it will print the n most recent requests in the current context. |
sm |
sm, site_map | Print a tree showing the site map. It will display all requests in the current context that did not have a 404 response. |
viq <id(s)> |
view_request_info, viq | View additional information about requests. Includes the target port, if SSL was used, applied tags, and other information. |
vfq <id(s)> |
view_full_request, vfq | [V]iew [F]ull Re[Q]uest, prints the full request including headers and data. |
vhq <id(s)> |
view_request_headers, vhq | [V]iew [H]eaders of a Re[Q]uest. Prints just the headers of a request. |
vfs <id(s)> |
view_full_response, vfs | [V]iew [F]ull Re[S]ponse, prints the full response associated with a request including headers and data. |
vhs <id(s)> |
view_response_headers, vhs | [V]iew [H]eaders of a Re[S]ponse. Prints just the headers of a response associated with a request. |
The table shown by ls
will have the following columns:
| Label | Description |
|---|---|
| ID | The request ID of that request. Used to identify the request for other commands. |
| Method | The method(/http verb) for the request |
| Host | The host that the request was sent to |
| Path | The path of the request |
| S-Code | The status code of the response |
| Req Len | The length of the data submitted |
| Rsp Len | The length of the data returned in the response |
| Time | The time in seconds it took to complete the request |
| Mngl | If the request or response were mangled with the interceptor. If the request was mangled, the column will show ‘q’. If the response was mangled, the column will show ‘s’. If both were mangled, it will show ‘q/s’. |
Tags¶
You can apply tags to a request and use filters to view specific tags. The following commands can be used to apply and remove tags to requests:
| Command | Aliases | Description |
|---|---|---|
tag <tag> [id(s)] |
tag | Apply a tag to the given requests. If no IDs are given, the tag will be applied to all in-context requests. |
untag <tag> [id(s)] |
untag | Remove a tag from the given ids. If no IDs are given, the tag is removed from every in-context request. |
clrtag <id(s)> |
clrtag | Removes all tags from the given ids. |
Request IDs¶
Request IDs are how you identify a request and every command that
involves specifying a request will take one or more request IDs. You can
see it when you run ls
. In addition, you can prepend an ID with
prefixes to get requests or responses associated with the request (for
example if you modified the request or its response with the
interceptor, you can get the unmangled versions.) Here are the valid
prefixes:
| Prefix | Description |
|---|---|
u |
If the request was mangled, prefixing the ID with u will result in the unmangled version of the request. The resulting request will not have an associated response because it was never submitted to the server. |
s |
If the response was mangled, prefixing the request ID s will result in the same request but its associated response will be the unmangled version. |
I know it sounds kind of unintuitive. Here are some example commands that will hopefully make things clearer. Suppose request 1 had its request mangled, and request 2 had its response mangled.
vfq 1
Prints the mangled version of request 1vfq u1
Prints the unmangled version of request 1rp u1
Open the repeater with the unmangled version of request 1vfs u1
Throws an error because the unmangled version was never submittedvfs s1
Throws an error because the response for request 1 was never mangledvfs 2
Prints the mangled response of request 2vfs s2
Prints the unmangled response of request 2vfq u2
Throws an error because request 2’s request was never mangledvfs u2
Throws an error because request 2’s request was never mangled
Passing Multiple Request IDs to a Command¶
Some arguments can take multiple IDs for an argument. To pass multiple IDs to a command, separate the IDs with commas (no spaces!). A few examples:
viq 1,2,u3
View information about requests 1, 2, and the unmangled version of 3gma foo 4,5,6
Generate a macro with definitions for requests 4, 5, and 6
Context¶
The context is a set of filters that define which requests are
considered “active”. Only requests in the current context are displayed
with ls
. By default, the context includes every single request that
passes through the proxy. You can limit down the current context by
applying filters. Filters apply rules such as “the response code must
equal 500” or “the host must contain google.com”. Once you apply one or
more filters, only requests/responses which pass every active filter
will be a part of the current context.
| Command | Aliases | Description |
|---|---|---|
f <filter string> |
filter, fl, f | Add a filter that limits which requests are included in the current context. See the Filter String section for how to create a filter string |
fc |
filter_clear, fc | Clears the filters and resets the context to contain all requests and responses. Ignores scope |
fu |
filter_up, fu | Removes the most recently applied filter |
fls |
filter_list, fls | Print the filters that make up the current context |
filter_prune |
filter_prune | Delete all the requests that aren’t in the current context from the data file |
Filter Strings¶
Filter strings define a condition that a request/response pair must pass to be part of the context. Most filter strings have the following format:
<field> <comparer> <value>
Where <field>
is some part of the request/response, <comparer>
is some comparison to <value>
. For example, if you wanted a filter
that only matches requests to target.org
, you could use the
following filter string:
host is target.org
field = "host"
comparer = "is"
value = "target.org"
Also if you prefix a comparer with ‘n’ it turns it into a negation.
Using the previous example, the following will match any request except
for ones where the host contains target.org
:
host nis target.org
field = "host"
comparer = "nis"
value = "target.org"
For fields that are a list of key/value pairs (headers, get params, post params, and cookies) you can use the following format:
<field> <comparer1> <value1>[ <comparer2> <value2>]
This is a little more complicated. If you don’t give comparer2/value2, the filter will pass any pair where the key or the value matches comparer1 and value1. If you do give comparer2/value2, the key must match comparer1/value1 and the value must match comparer2/value2 For example:
Filter A:
cookie contains Session
Filter B:
cookie contains Session contains 456
Filter C:
cookie ncontains Ultra
Cookie: SuperSession=abc123
Matches A and C but not B
Cookie: UltraSession=abc123456
Matches both A and B but not C
List of fields¶
| Field Name | Aliases | Description | Format |
|---|---|---|---|
| all | all | The entire request represented as one string | String |
| host | host, domain, hs, dm | The target host (ie www.target.com) | String |
| path | path, pt | The path of the url (ie /path/to/secrets.php) | String |
| body | body, data, bd, dt | The body (data section) of either the request or the response | String |
| verb | verb, vb | The HTTP verb of the request (ie GET, POST) | String |
| param | param, pm | Either the get or post parameters | Key/Value |
| header | header, hd | An HTTP header (ie User-Agent, Basic-Authorization) in the request or response | Key/Value |
| rawheaders | rawheaders, rh | The entire header section (as one string) of either the head or the response | String |
| sentcookie | sentcookie, sck | A cookie sent in a request | Key/Value |
| setcookie | setcookie, stck | A cookie set by a response | Key/Value |
| statuscode | statuscode, sc, responsecode | The response code of the response | Numeric |
| tag | tag | Any of the tags applied to the request | String |
List of comparers¶
| Field Name | Aliases | Description |
|---|---|---|
| is | is | Exact string match |
| contains | contains, ct | A contain B is true if B is a substring of A |
| containsr | containsr, ctr | A containr B is true if A matches regexp B |
| exists | exists, ex | A exists B if A is not an empty string (likely buggy) |
| Leq | Leq | A Leq B if A’s length equals B (B must be a number) |
| Lgt | Lgt | A Lgt B if A’s length is greater than B (B must be a number ) |
| Llt | Llt | A Llt B if A’s length is less than B (B must be a number) |
| eq | eq | A eq B if A = B (A and B must be a number) |
| gt | gt | A gt B if A > B (A and B must be a number) |
| lt | lt | A lt B if A < B (A and B must be a number) |
Special form filters¶
A few filters don’t conform to the field, comparer, value format. You can still negate these.
| Format | Aliases | Description |
|---|---|---|
| before | before, bf, b4 | Filters out any request that is not before the given request. Filters out any request without a time. |
| after | after, af | Filters out any request that is not before the given request. Filters out any request without a time. |
Scope¶
Scope is a set of rules to define whether Pappy should mess with a
request. You define the scope by setting the context to what you want
the scope to be and running scope_save
. The scope is saved in the
data file and is automatically restored when using the same project
directory.
Any requests which don’t match all the filters in the scope will be passed straight to the browser and will not be caught by the interceptor or recorded in the data file. This is useful to make sure you don’t accidentally do something like log in to your email through the proxy and have your plaintext username/password stored.
| Command | Aliases | Description |
|---|---|---|
scope_save |
scope_save |
Set the current context to be the scope |
sr |
scope_reset , sr |
Set the current context to the scope |
scope_delete |
scope_delete |
Clear the scope (everything’s in scope!) |
scope_list |
scope_list , sls |
List all the filters that are applied to the scope |
Built-In Filters¶
Pappy also includes some built in filters that you can apply. These are
things that you may want to filter by but may be too tedius to type out.
The fbi
command also supports tab completion.
| Filter | Description |
|---|---|
not_image |
Matches anything that isn’t an image. |
not_jscss |
Matches anything that isn’t JavaScript or CSS. |
| Command | Aliases | Description |
|---|---|---|
fbi <filter> |
builtin_filter , fbi |
Apply a built-in filter to the current context |
Interceptor¶
This feature is like Burp’s proxy with “Intercept Mode” turned on,
except it’s not turned on unless you explicitly turn it on. When the
proxy gets a request while in intercept mode, it lets you edit it before
forwarding it to the server. In addition, it can stop responses from the
server and let you edit them before they get forwarded to the browser.
When you run the command, you can pass req
and/or rsp
as
arguments to say whether you would like to intercept requests and/or
responses. Only in-scope requests/responses will be intercepted (see
Scope section).
The interceptor will use your EDITOR variable to decide which editor to
edit the request/response with. If no editor variable is set, it will
default to vi
.
To forward a request, edit it, save the file, then quit.
| Command | Aliases | Description |
|---|---|---|
ic <req,rsp>+ |
intercept , ic |
Begins interception mode. Press enter to leave interception mode and return to the command prompt. Pass in request to intercept requests, response to intercept responses, or both to intercept both. |
Intercept both requests and responses:
> ic requests responses
> ic req rsp
Intercept just requests:
> ic requests
> ic req
Intercept just responses:
> ic responses
> ic rsp
Be totally useless:
> ic
To drop a request, delete everything, save and quit.
Repeater¶
This feature is like Burp’s repeater (yes, really). You choose a request
and Pappy will open vim in a split window with your request on the left
and the original response on the right. You can make changes to the
request and then run ”:RepeaterSubmitBuffer” to submit the modified
request. The response will be displayed on the right. This command is
bound to <leader>f
by default, but you can bind it to something else
too in your vimrc (I think, dunno if vim will complain if the function
undefined which it will be for regular files). This command will submit
whatever buffer your cursor is in, so make sure it’s in the request
buffer.
When you’re done with repeater, run ”:qa!” to avoid having to save changes to nonexistent files.
| Command | Aliases | Description |
|---|---|---|
rp <id> |
repeater, rp | Open the specified request in the repeater |
| Vim Command | Keybinding | Action |
|---|---|---|
RepeaterSubmitBuffer |
f | Submit the current buffer, split the windows vertically, and show the result in the right window |
Macros¶
Macros are Pappy’s version of Burp’s intruder. You can use macros to
make automated requests through the proxy and save them to the data
file. A macro file is any python script file in the current directory
that is in the form macro_<name>.py
. An example project directory
with macros would be:
$ ls -l
-rw-r--r-- 1 scaryhacker wheel 150 Nov 26 11:17 config.json
-rw------- 1 scaryhacker wheel 2639872 Nov 26 17:18 data.db
-rw-r--r-- 1 scaryhacker wheel 471 Nov 26 18:42 macro_blank.py
-rw-r--r-- 1 scaryhacker wheel 264 Nov 26 18:49 macro_hackthensa.py
-rw-r--r-- 1 scaryhacker wheel 1261 Nov 26 18:37 macro_testgen.py
-rw-r--r-- 1 scaryhacker wheel 241 Nov 26 17:18 macro_test.py
In this case we have a blank
, hackthensa
, testgen
, and
test
macro. A macro script is any python script that defines a
run_macro(args)
function and a MACRO_NAME
variable. For example,
a simple macro would be:
### macro_print.py
MACRO_NAME = 'Print Macro'
def run_macro(args):
if args:
print "Hello, %s!" % args[0]
else:
print "Hello, Pappy!"
You can place this macro in your project directory then load and run it from Pappy. When a macro is run, arguments are passed from the command line.
You’ll need to run lma
every time you make a change to the macro in
order to reload it. In addition, any code outside of the run_macro
function will be run when it the macro gets loaded.
Generating Macros From Requests¶
You can also generate macros that have Pappy Request
objects created
with the same information as requests you’ve already made. For example:
$ pappy
Proxy is listening on port 8000
pappy> ls
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
5 GET vitaly.sexy /esr1.jpg 200 OK 0 17653 -- --
4 GET vitaly.sexy /netscape.gif 200 OK 0 1135 -- --
3 GET vitaly.sexy /construction.gif 200 OK 0 28366 -- --
2 GET vitaly.sexy /vitaly2.jpg 200 OK 0 2034003 -- --
1 GET vitaly.sexy / 200 OK 0 1201 -- --
pappy> gma sexy 1
Wrote script to macro_sexy.py
pappy> quit
$ cat macro_sexy.py
from pappyproxy.http import Request, get_request, post_request
MACRO_NAME = 'Macro 94664581'
SHORT_NAME = ''
###########
## Requests
req0 = Request((
'GET / HTTP/1.1\r\n'
'Host: vitaly.sexy\r\n'
'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0\r\n'
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n'
'Accept-Language: en-US,en;q=0.5\r\n'
'Accept-Encoding: gzip, deflate\r\n'
'Connection: keep-alive\r\n'
'Pragma: no-cache\r\n'
'Cache-Control: no-cache\r\n'
'\r\n'
))
def run_macro(args):
# Example:
# req = req0.copy() # Copy req0
# req.submit() # Submit the request to get a response
# print req.response.raw_headers # print the response headers
# req.save() # save the request to the data file
# or copy req0 into a loop and use string substitution to automate requests
pass
If you enter in a value for SHORT_NAME
, you can use it as a shortcut
to run that macro. So if in a macro you set SHORT_NAME='tm'
you can
run it by running pappy> rma tm
.
| Command | Aliases | Description |
|---|---|---|
lma [dir] |
load_macros , lma |
Load macros from a directory. If dir is not given, use the current directory (the project directory) |
rma <macro name> |
run_macro , rma |
Run a macro with the given name. You can use the shortname, filename, or long name. |
gma <name> [id(s)] |
generate_macro , gma |
Generate a macro with the given name. If request IDs are given, the macro will contain request objects that contain each request. |
rpy <id(s)> |
rpy |
Print the Python object definitions for each of the given ids |
Request Objects¶
The main method of interacting with the proxy is through Request
objects. You can submit a request with req.sumbit()
and save it to
the data file with req.save()
. The objects also have attributes
which can be used to modify the request in a high-level way. You can see
the full
documentation
for more details on using these objects.
Dict-like objects are represented with a custom class called a
RepeatableDict
. Again, look at the docs for details. For the most
part, you can interact with it like a normal dictionary, but don’t be
surprised if it’s missing some methods you would expect.
Here is a quick list of attributes that you can use with Request
objects:
| Attribute | Settable? | Data Type | Description |
|---|---|---|---|
| cookies | Yes | RepeatableDict | Cookies sent in the request |
| fragment | Yes | String | The url fragment (The text after the #) |
| full_path | No | String | The path including url params and the fragment |
| full_request | No | String | The full request including headers and data |
| headers | Yes | RepeatableDict | The headers of the request |
| host | Yes | String | The host that the request is sent to |
| is_ssl | Yes | Bool | Whether the request is/was sent over SSL |
| path | Yes | String | The document path (ie www.a.com/this/is/the/path) |
| port | Yes | Integer | The port the request is/was sent to |
| post_params | Yes | RepeatableDict | Post parameters |
| raw_data | Yes | String | The data part of the request |
| raw_headers | No | String | The text of the headers section of the request |
| reqid | Yes | Integer | The ID of the request. If set when save() is called, it replaces the request with the same id in the database |
| response | Yes | Response | The associated response for the request |
| rsptime | No | Datetime Delta | The time it took to complete the request. Set when submit() is called |
| status_line | Yes | String | The status line of the request (ie ‘GET / HTTP/1.1’) |
| time_end | Yes | Datetime | The time when the request was completed |
| time_start | Yes | Datetime | The time when the request was started |
| unmangled | Yes | Request | If the request was mangled, the unmangled version of the request |
| url | Yes | String | The URL of the request (ie ‘https://www.google.com‘) |
| url_params | Yes | RepeatableDict | The URL parameters of the request |
| verb | Yes | String | The verb used for the request (ie GET, POST, PATCH, HEAD, etc). Doesn’t have to be a valid verb. |
| version | Yes | String | The version part of the status line (ie ‘HTTP/1.1’) |
Request methods:
| Function | Description |
|---|---|
| submit() | Submit the request through the proxy. Does not save the request to the data file |
| save() | Save the request, its unmangled version, its associated response, and the unmangled version of the response to the database |
And here is a quick list of attributes that you can use with
Response
objects:
| Attribute | Settable? | Data Type | Description |
|---|---|---|---|
| cookies | Yes | RepeatableDict | Cookies set by the response |
| headers | Yes | RepeatableDict | The headers of the response |
| response_code | Yes | Integer | The response code of the response |
| response_text | Yes | String | The text associated with the response code (ie OK, NOT FOUND) |
| rspid | Yes | Integer | The response id of the response. If this is the same as another response in the database, calling save() on the associated request will replace that response in the database |
| unmangled | Yes | Response | If the response was mangled, this will refer to the unmangled version of the response. Otherwise it is None |
| version | Yes | String | The version part of the status line of the response (ie ‘HTTP/1.1’) |
| raw_headers | No | String | A text version of the headers of the response |
| status_line | Yes | String | The status line of the response |
| raw_data | Yes | String | The data portion of the response |
| full_response | No | String | The full text version of the response including headers and data |
Like I said, these interfaces are prone to change and will probably crash when you use them. If you get a traceback, send me an email so I can fix it.
Useful Functions¶
There are also a few functions which could be useful for creating
requests in macros. It’s worth pointing out that request_by_id
is
useful for passing request objects as arguments. For example, here is a
macro that lets you resubmit a request with the Google Bot user agent:
## macro_googlebot.py
from pappyproxy.http import Request, get_request, post_request, request_by_id
from pappyproxy.context import set_tag
from pappyproxy.iter import *
MACRO_NAME = 'Submit as Google'
SHORT_NAME = ''
def run_macro(args):
req = request_by_id(args[0])
req.headers['User-Agent'] = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
req.submit()
req.save()
| Function | Description |
|---|---|
| get_request(url, url_params={}) | Returns a Request object that contains a GET request to the given url with the given url params |
| post_request(url, post_params={}, url_params={}) | Returns a Request object that contains a POST request to the given url with the given url and post params |
| request_by_id(reqid) | Get a request object from its id. |
Intercepting Macros¶
Intercepting macros let you mangle requests as they pass through the
proxy. Similarly to normal macros, an intercepting macro is any python
script with an “int” prefix. For example, int_name.py
would be a
valid intercepting macro name. They are also loaded with the lma
command. An intercepting macro can define two functions:
mangle_request
or mangle_response
. Both requests only take a
Request
object as a parameter. mangle_request
returns either a
new, modified Request object to change it, or it can return the original
object to not mangle it. The mange_response
must return a
Response
(not request!) object. The request passed in to
mangle_response
will have an associated response with it. If you
want to modify the response, copy request.response
, make
modifications, then return it. If you would like to pass it through
untouched, just return request.response
.
Note, that due to twisted funkyness, you cannot save requests from
intercepting macros. Technically you can, but to do that you’ll
have to define async_mangle_request
(or response) instead of
mangle_request
(or response) then use Request.async_deep_save
which generates a deferred, then generate a deferred from
async_mangle_requests
(inline callbacks work too). If you’ve never
used twisted before, please don’t try. Twisted is hard. Plus the mangled
request will be saved before it is submitted anyways.
Confusing? Here are some example intercepting macros:
## int_cloud2butt.py
import string
MACRO_NAME = 'Cloud to Butt'
def mangle_response(request):
r = request.response.copy()
r.raw_data = string.replace(r.raw_data, 'cloud', 'butt')
r.raw_data = string.replace(r.raw_data, 'Cloud', 'Butt')
return r
## int_donothing.py
import string
MACRO_NAME = 'Do Nothing'
def mangle_request(request):
return request
def mangle_response(request):
return request.response
## int_adminplz.py
from base64 import base64encode as b64e
MACRO_NAME = 'Admin Session'
def mangle_request(request):
r = request.copy()
r.headers['Authorization'] = 'Basic %s' % b64e('Admin:Password123')
return r
In addition, you can use an init(args)
function to get arguments
from the command line. If no arguments are passed, args will be an empty
list. Here is an example macro that does a search and replace:
## int_replace.py
MACRO_NAME = 'Find and Replace'
SHORT_NAME = ''
runargs = []
def init(args):
global runargs
runargs = args
def mangle_request(request):
global runargs
if len(runargs) < 2:
return request
request.body = request.body.replace(runargs[0], runargs[1])
return request
def mangle_response(request):
global runargs
if len(runargs) < 2:
return request.response
request.response.body = request.response.body.replace(runargs[0], runargs[1])
return request.response
You can use this macro to do any search and replace that you want. For example, if you wanted to replace “Google” with “Skynet”, you can run the macro like this:
pappy> lma
Loaded "<InterceptingMacro Find and Replace (replace)>"
pappy> rim replace Google Skynet
"Find and Replace" started
pappy>
Now every site that you visit will be a little bit more accurate.
Enabling/Disabling Intercepting Macros¶
You can use the following commands to start/stop intercepting macros
| Command | Aliases | Description |
|---|---|---|
lma [dir] |
load_macros , lma |
Load macros from a directory. If dir is not given, use the current directory (the project directory) |
rim <macro name> |
run_int_macro , rim |
Run an intercepting macro. Similarly to normal macros you can use the name, short name, or file name of the macro. |
sim <macro name> [args] |
stop_int_macro , sim |
Stop an intercepting macro. If arguments are given, they will be passed to the macro’s init(args) function if it exists. |
lim |
list_int_macros , lsim |
List all enabled/disabled intercepting macros |
gima <name> |
generate_int_macro , gima |
Generate an intercepting macro with the given name. |
Logging¶
You can watch in real-time what requests are going through the proxy. Verbosisty defaults to 1 which just states when connections are made/lost and some information on what is happening. If verbosity is set to 3, it includes all the data which is sent through the proxy and processed. It will print the raw response from the server, what it decodes it to, etc. Even if you don’t run this command, all the information is stored in the dubug directory (the directory is cleared every start though!)
| Command | Description |
|---|---|
log [verbosity] |
View the log at the given verbosity. Default verbosity is 1 which just shows connections being made/lost and some other info, verbosity 3 shows full requests/responses as they pass through and are processed by the proxy |
Additional Commands and Features¶
This is a list of other random stuff you can do that isn’t categorized under anything else. These are mostly commands that I found that I needed while doing a test and just added. They likely don’t do a ton of error checking.
| Command | Aliases | Description |
|---|---|---|
dump_response <reqid> [filename] |
dump_response |
Dumps the data from the response to the given filename (useful for images, .swf, etc). If no filename is given, it uses the name given in the path. |
export <req|rsp> <reqid> |
export |
Writes either the full request or response to a file in the current directory. |
Response streaming¶
If you don’t have any intercepting macros running, Pappy will forward data to the browser as it gets it. However, if you’re trying to mangle messages/responses, Pappy will need to download the entire message first.
Plugins¶
Note that this section is a very quick overview of plugins. For a full description of how to write them, please see the official docs.
It is also possible to write plugins which are reusable across projects.
Plugins are simply Python scripts located in ~/.pappy/plugins
.
Plugins are able to create new console commands and maintain state
throughout a Pappy session. They can access the same API as macros, but
the plugin system is designed to allow you to create general purpose
commands as compared to macros which are meant to be project-specific
scripts. Still, it may not be a bad idea to try building a macro to do
something in a quick and dirty way before writing a plugin since plugins
are more complicated to write.
A simple hello world plugin could be something like:
## hello.py
import shlex
def hello_world(line):
if line:
args = shlex.split(line)
print 'Hello, %s!' % (', '.join(args))
else:
print "Hello, world!"
###############
## Plugin hooks
def load_cmds(cmd):
cmd.set_cmds({
'hello': (hello_world, None),
})
cmd.add_aliases([
('hello', 'hlo'),
('hello', 'ho'),
])
You can also create commands which support autocomplete:
import shlex
_AUTOCOMPLETE_NAMES = ['alice', 'allie', 'sarah', 'mallory', 'slagathor']
def hello_world(line):
if line:
args = shlex.split(line)
print 'Hello, %s!' % (', '.join(args))
else:
print "Hello, world!"
def complete_hello_world(text, line, begidx, endidx):
return [n for n in _AUTOCOMPLETE_NAMES if n.startswith(text)]
###############
## Plugin hooks
def load_cmds(cmd):
cmd.set_cmds({
'hello': (hello_world, complete_hello_world),
})
cmd.add_aliases([
('hello', 'hlo'),
])
Then when you run Pappy you can use the hello
command:
$ pappy -l
Temporary datafile is /tmp/tmpBOXyJ3
Proxy is listening on port 8000
pappy> ho
Hello, world!
pappy> ho foo bar baz
Hello, foo, bar, baz!
pappy> ho foo bar "baz lihtyur"
Hello, foo, bar, baz lihtyur!
pappy>
Should I Write a Plugin or a Macro?¶
A lot of the time, you can get away with writing a macro. However, you may consider writing a plugin if:
- You find yourself copying one macro to multiple projects
- You want to write a general tool that can be applied to any website
- You need to maintain state during the Pappy session
My guess is that if you need one quick thing for a project, you’re
better off writing a macro first and seeing if you end up using it in
future projects. Then if you find yourself needing it a lot, write a
plugin for it. You may also consider keeping a mine.py
plugin where
you can write out commands that you use regularly but may not be worth
creating a dedicated plugin for.
FAQ¶
I still like Burp, but Pappy looks interesting, can I use both?¶
Yes! If you don’t want to go completely over to Pappy yet, you can configure Burp to use Pappy as an upstream proxy server. That way, traffic will go through both Burp and Pappy and you can use whichever you want to do your testing.
How to have Burp forward traffic through Pappy:
- Open Burp
- Go to
Options -> Connections -> Upstream Proxy Servers
- Click
Add
- Leave
Destination Host
blank, but put<IP_ADDRESS>
inProxy Host
and8000
intoPort
(assuming you’re using the default listener) - Configure your browser to use Burp as a proxy
Why does my request have an id of --
?!?!¶
You can’t do anything with a request/response until it is decoded and
saved to disk. In between the time when a request is decoded and when
it’s saved to disk, it will have an ID of --
. So just wait a little
bit and it will get an ID you can use.
Changelog¶
The boring part of the readme
- 0.2.0
- Lots of refactoring
- Plugins
- Bugfixes probably
- Change prompt to make Pappy look more professional (but it will always be pappy time in your heart, I promise)
- Create changelog
- Add response streaming if no intercepting macros are active
- 0.1.1
- Start using sane versioning system
- Did proxy things


« Tests
/
Sampler
Object Movie Sprites
Animation
Directional
Clockwise
Non-Animated
Non-Looping
Multi-Row
Object Movie Sequences
Multi-Row
Dual-Orbit
Clockwise
Non-Looping
Direct
Camera
Panoramas
Multi-Row
Orientable Stitched
Click-Free
Non-Looping
Sprite
Animated Reels can also be controlled using events
pause
stop
or
play


Part Designer
- UX design
- UI design
- Logo design
- Research
- Making it 'pop'
Part Coder
- Front-end development
- HTML / CSS / PHP
- JavaScript
- Back-end development
- Swearing at computer
Random facts
- I drink a lot of tea
- I love anime
- I love to read novel
- I love dangdut
- I want to live on Japan
- I worship Agatha Christie
- I'm addicted to Granblue Fantasy
Random facts
- I drink a lot of tea
- I love anime
- I love to read novel
- I love dangdut
- I want to live on Japan
- I worship Agatha Christie
- I'm addicted to Granblue Fantasy


Spack in a nutshell
A 10 minutes introduction to the tool
Spack is a package manager
- Tracks metadata for installed software
- Multiple configurations of the same software
- Support for supercomputers, linux and MacOS
- Builds from sources, supports binary packaging
- Can be used for deployment, development or QA
https://spack.io
Who can profit from Spack?
- End users of HPC software
- HPC application teams
- Package developers
- User support teams at HPC centers
Spack is open source
- Started by Todd Gamblin (LLNL, Lead Developer)
- Very active and engaging community
- Dual license Apache-2.0 or MIT
- Switched from LGPL to be company friendly
- Want to get buy-in and PRs by HPC vendors
https://github.com/spack/spack
Contributions to Spack
Spack is committed to be open source
Clone it and you're ready to go!
$ git clone https://github.com/spack/spack.git
$ . spack/share/spack/setup-env.sh
$ spack install hdf5
Packages are Python classes
class Kripke(Package):
"""Kripke is a simple, scalable, 3D Sn
deterministic particle transport mini app.
"""
homepage = "https://codesign.llnl.gov/kripke.php"
url = "https://codesign.llnl.gov/kripke-1.1.tar.gz"
version('1.1', '7fe6f2b26ed983a6ce5495ab701f85bf')
version('1.0', 'f4247dde07952a5ff866b24e45b5cdd1')
variant('mpi', default=True, description='Build with MPI.')
depends_on('mpi', when="+mpi")
Easy to work with packages
$ # Create a new package in the built-in repository
$ spack create <url>
$ # Modify an existing package
$ export EDITOR='emacs -nw'
$ spack edit <package-name>
$ # Scrape for versions of an existing package
$ spack versions <package-name>
Spack fights combinatorics with combinatorics!
Complexity of a typical HPC ecosystem
Software must be easy to use in any configuration
The spec syntax describes user's needs
$ # Install a specific version by appending @
$ spack install<EMAIL_ADDRESS>$ # Specify a compiler (and optional version), with %
$ spack install<EMAIL_ADDRESS><EMAIL_ADDRESS>$ # Add special boolean compile-time options with +
$ spack install<EMAIL_ADDRESS>+szip
$ # Add custom compiler flags
$ spack install<EMAIL_ADDRESS>cppflags="-O3 -floop-block"
$ # Cross-compile on a Cray or Blue Gene/Q
$ spack install<EMAIL_ADDRESS>target=backend
Directives model allowed configurations
class Openblas(MakefilePackage):
"""OpenBLAS: An optimized BLAS library"""
homepage = 'http://www.openblas.net'
url = 'http://github.com/OpenBLAS/v0.2.19.tar.gz'
version('0.3.4', sha256='4b4b4453251')
version('0.3.3', sha256='49d88f4494a')
variant('shared', default=True,
description='Build shared libraries')
variant('ilp64', default=False,
description='64 bit integers')
conflicts('%intel@16', when='@0.2.15:0.2.19')
The concretizer fills in missing details
The abstract spec is turned into a concrete configuration
that can be installed
Each configuration is installed in its own prefix
All the software is installed in the Spack's store
The results of concretization can be queried
$ spack spec -Il hdf5+mpi ^mpich
Input spec
--------------------------------
- hdf5+mpi
- ^mpich
Concretized
--------------------------------
- wqoi56r hdf5<EMAIL_ADDRESS>arch=linux-ubuntu18.04-x86_64
- k2mhs2k ^mpich<EMAIL_ADDRESS>device=ch3 +hydra netmod=tcp +pmi+romio~verbs arch=linux-ubuntu18.04-x86_64
- rwrp443 ^findutils<EMAIL_ADDRESS>patches=84b916c0bf8c51b7e7b28417692f0ad3e7030d1f3c248ba77c42ede5c1c5d11e,bd9e4e5cc280f9753ae14956c4e4aa17fe7a210f55dd6c84aa60b12d106d47a2 arch=linux-ubuntu18.04-x86_64
- vsawnwb ^autoconf<EMAIL_ADDRESS>arch=linux-ubuntu18.04-x86_64
- i3zd457 ^m4<EMAIL_ADDRESS>patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu18.04-x86_64
- bd227q2 ^libsigsegv<EMAIL_ADDRESS>arch=linux-ubuntu18.04-x86_64
- 5pyk3t3 ^perl<EMAIL_ADDRESS>patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu18.04-x86_64
- gota243 ^gdbm<EMAIL_ADDRESS>arch=linux-ubuntu18.04-x86_64
- ltrvf62 ^readline<EMAIL_ADDRESS>arch=linux-ubuntu18.04-x86_64
- 6mbvq7i ^ncurses<EMAIL_ADDRESS>arch=linux-ubuntu18.04-x86_64
- zaevtj5 ^pkgconf<EMAIL_ADDRESS>arch=linux-ubuntu18.04-x86_64
- tcf52hq ^automake<EMAIL_ADDRESS>arch=linux-ubuntu18.04-x86_64
- rcuvdn3 ^libtool<EMAIL_ADDRESS>arch=linux-ubuntu18.04-x86_64
- 3iz4m52 ^texinfo<EMAIL_ADDRESS>arch=linux-ubuntu18.04-x86_64
[+] ivqu252 ^zlib<EMAIL_ADDRESS>arch=linux-ubuntu18.04-x86_64
Spack keeps the metadata of the software it installed!
Installed configurations are stored in a JSON file
{
"database": {
"installs": {
"ivqu252fvh7r5iar6zwx4fmeoxiykln7": {
"explicit": true,
"installation_time":<PHONE_NUMBER>.178339,
"ref_count": 0,
"installed": true,
"path": "/home/mculpo/PycharmProjects/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.2.0/zlib-1.2.11-ivqu252fvh7r5iar6zwx4fmeoxiykln7",
"spec": {
"zlib": {
"version": "1.2.11",
"arch": {
"platform": "linux",
"platform_os": "ubuntu18.04",
"target": "x86_64"
},
"compiler": {
"name": "gcc",
"version": "8.2.0"
},
"namespace": "builtin",
"parameters": {
"optimize": true,
"pic": true,
"shared": true,
"cflags": [],
"cppflags": [],
"cxxflags": [],
"fflags": [],
"ldflags": [],
"ldlibs": []
}
}
}
}
},
"version": "0.9.3"
}
}
opt/spack/.spack-db/index.json
Provenance is preserved for each configuration
$ tree $(spack location -i hdf5)/.spack
<prefix>/.spack
├── archived-files
│ └── config.log
├── build.env
├── build.out
├── repos
│ └── builtin
│ ├── packages
│ │ ├── hdf5
│ │ │ └── package.py
│ │ └── zlib
│ │ └── package.py
│ └── repo.yaml
└── spec.yaml
6 directories, 7 files
The JSON database can be regenerated based on this information
Tools are then built over the data in the DB
- Query commands
- Uninstallation procedures
- Module file generation
- Python modules activation
- ...
Query what's installed from the command line
$ spack find zlib
==> 1 installed package
-- linux-ubuntu18.04-x86_64 /<EMAIL_ADDRESS>----
<EMAIL_ADDRESS>$ spack find --start-date 'a month ago'
==> 3 installed packages
-- linux-ubuntu18.04-x86_64 /<EMAIL_ADDRESS>----
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Uninstall anything in an easy and safe way
$ spack find zlib
==> 2 installed packages
-- linux-ubuntu18.04-x86_64 /<EMAIL_ADDRESS>----
<EMAIL_ADDRESS><EMAIL_ADDRESS>$ spack uninstall<EMAIL_ADDRESS>==> The following packages will be uninstalled:
-- linux-ubuntu18.04-x86_64 /<EMAIL_ADDRESS>----
yxoie27<EMAIL_ADDRESS>==> Do you want to proceed? [y/N] y
==> Successfully uninstalled zlib<EMAIL_ADDRESS>[...] /yxoie27
Generate module files for HPC sites
- Output highly customizable using YAML files
- By default generated via post-install hooks
- Can be managed via command line afterwards
- Everything is powered by Jinja2 templates
- Support for both TCL and Lua modules
Tutorial: Spack 101 - module files
Generate module files for HPC sites
modules:
lmod:
core_compilers:
-<EMAIL_ADDRESS>hierarchy:
- mpi
- lapack
hash_length: 0
all:
suffixes:
+mpi: mpi
+openmp: openmp
...
Tutorial: Spack 101 - module files
Activate python modules on a fine-grained base
$ spack install python py-scipy py-numpy
[...]
$ # Activate only py-numpy
$ $ spack activate py-numpy
==> Activating extension py-numpy<EMAIL_ADDRESS>arch=linux-ubuntu18.04-x86_64 /o626ufb for python<EMAIL_ADDRESS>patches=123082ab3483ded78e86d7c809e98a804b3465b4683c96bd79a2fd799f572244 +pic+pythoncmd+shared~tk~ucs4 arch=linux-ubuntu18.04-x86_64 /nft4nfs
$ $(spack location -i python)/bin/python
Python 2.7.15 (default, Jan 25 2019, 09:14:19)
[GCC 8.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> import scipy
[...]
ImportError: No module named scipy
Activated modules look like they are site installed
Scrape build logs of installed configurations
$ spack log-parse $(spack location -i hdf5)/.spack/build.out
0 errors
$ spack log-parse --show=errors,warnings $(spack location -i hdf5)/.spack/build.out
0 errors
741 warnings
534 CC H5dbg.lo
535 CC H5system.lo
536 H5system.c: In function 'HDfprintf':
>> 537 H5system.c:276:45: warning: format not a string literal, argument types not checked [-Wformat-nonliteral]
538 n = fprintf(stream, format_templ, x);
539 ^~~~~~~~~~~~
[...]
Permits to investigate issues reported on
software that has already been deployed
Spack Environments
The cool new feature!
Spack's store might be inconvenient at times
$ spack find
==> 70 installed packages
-- linux-ubuntu16.04-x86_64 /<EMAIL_ADDRESS>----
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>-- linux-ubuntu16.04-x86_64 /<EMAIL_ADDRESS>----
<EMAIL_ADDRESS>-- linux-ubuntu16.04-x86_64 /<EMAIL_ADDRESS>----
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>...
It would be nice to have a terser view without throwing away what's already installed
Spack environments to the rescue!
$ spack env create my-project
==> Created environment 'my-project' in [...]/my-project
$ spack env activate my-project
$ spack env status
==> In environment my-project
$ spack find
==> In environment my-project
==> No root specs
==> 0 installed packages
An environment is a virtualized Spack instance that can be used for a specific purpose
Environments use what's available in the store
$ spack find
==> In environment my-project
==> No root specs
==> 0 installed packages
$ spack install zlib hdf5~mpi
==> zlib is already installed in [...]
==> Installing hdf5
[...]
==> Successfully installed hdf5
Fetch: 0.06s. Build: 1m 14.67s. Total: 1m 14.73s.
[+] /home/.../hdf5-1.10.4-xucyflhbo2p47n46smfsqo7p2y3hijmd
zlib
has been reused, hdf5
has been installed
in the Spack's store and then added
Users can set-up groups of packages at once
$ spack add hdf5~mpi python
==> Adding hdf5~mpi to environment my-project
==> Adding python to environment my-project
==> Adding mpich to environment my-project
$ spack find -v
==> In environment my-project
==> Root specs
hdf5~mpi mpich python
==> 0 installed packages
$ spack install
[...]
"spack install
" concretizes the root specs and builds what's missing
Each detail can be finely tuned
$ spack config get
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
# add package specs to the `specs` list
specs: [hdf5~mpi, python, mpich]
mirrors: {}
modules:
enable: []
repos: []
packages: {}
config: {}
Use "spack config edit
" to edit the file and add custom configurations
Abstract user requests are stored in spack.yaml
- It's the configuration file we have seen before
- Does not have to live inside Spack tree
- Can be bundled with anything
- Can be manipulated directly if it is in the
pwd
- Use
spack.yaml
to distribute an environment
Fully concretized specs are in spack.lock
{
"concrete_specs": {
"teneqii2xv5u6zl5r6qi3pwurc6pmypz": {
"xz": {
"version": "5.2.4",
"arch": {
"platform": "linux",
"platform_os": "ubuntu16.04",
"target": "x86_64"
},
...
Using either spack.yaml
or spack.lock
you can recreate an environment
Coming soon in Spack environments...
Associate a view with an environment
spack:
specs:
- hdf5+mpi
- bzip2
view: True
|
$ ls .spack-env/view/bin
h5clear h5diff ... h5unjam
h5copy h5dump ... ph5diff
h5debug ... ...
$ ls .spack-env/view/include
H5ACpublic.h ... hdf5.h
H5Apublic.h ... zconf.h
H5Cpublic.h ... zlib.h
|
Views are projections of the Spack combinatorial space into a single prefix


This is a vulnerable web server for showcasing CVE 2014-0160, a.k.a. Heartbleed.
In order to provide some juicy data when exploiting the vulnerability we added a simple form that creates a cookie where are stored user credentials
(We know that this is not a real case scenario but it is useful tu understand how informations are stored in the webserver)


We exchange metrics!
- grow extensible markets
- e-enable efficient infrastructures
- drive back-end infomediaries
- evolve viral channels
- implement bricks-and-clicks web-readiness
- embrace enterprise bandwidth
- syndicate dynamic synergies
- syndicate innovative web-readiness
- benchmark 24/7 functionalities


Note
Click here to download the full example code
Use Audio to Align Video Data¶
In this example, we use pd-parser
to find audio events using the same
algorithm for matching with time-stamps and rejecting misaligned
audio, but applied using the onset of an audio deflection instead of detecting
photodiode events based on their square wave shape.
# Authors: Alex Rockhill<EMAIL_ADDRESS>#
# License: BSD (3-clause)
Load in a video with audio
In this example, we’ll use audio and instead of aligning electrophysiology data, we’ll align a video. This example data is from a task where movements are played on a monitor for the participant to mirror and the video recording is synchronized by playing a pre-recorded clap. This clap sound, or a similar sound, is recommended for synchronizing audio because the onset is clear and allows good precision in synchronizing events.
Note that the commands that require ffmpeg are pre-computed and commented
out because ffmpeg must be installed to use them and it is not required by
pd-parser
.
import os
import os.path as op
import numpy as np
from scipy.io import wavfile
from subprocess import call
# from subprocess import run, PIPE, STDOUT
# import datetime
import mne
from mne.utils import _TempDir
import pd_parser
# get the data
out_dir = _TempDir()
call(['curl -L https://raw.githubusercontent.com/alexrockhill/pd-parser/'
'master/pd_parser/tests/data/test_video.mp4 '
'-o ' + op.join(out_dir, 'test_video.mp4')], shell=True, env=os.environ)
call(['curl -L https://raw.githubusercontent.com/alexrockhill/pd-parser/'
'master/pd_parser/tests/data/test_video.wav '
'-o ' + op.join(out_dir, 'test_video.wav')], shell=True, env=os.environ)
call(['curl -L https://raw.githubusercontent.com/alexrockhill/pd-parser/'
'master/pd_parser/tests/data/test_video_beh.tsv '
'-o ' + op.join(out_dir, 'test_video_beh.tsv')],
shell=True, env=os.environ)
# navigate to the example video
video_fname = op.join(out_dir, 'test_video.mp4')
audio_fname = video_fname.replace('mp4', 'wav') # pre-computed
# extract audio (requires ffmpeg)
# run(['ffmpeg', '-i', video_fname, audio_fname])
fs, data = wavfile.read(audio_fname)
data = data.mean(axis=1) # stereo audio but only need one source
info = mne.create_info(['audio'], fs, ['stim'])
raw = mne.io.RawArray(data[np.newaxis], info)
# find audio-visual time offset
offset = 0 # pre-computed value for this video
'''
result = run(['ffprobe', '-show_entries', 'stream=codec_type,start_time',
'-v', '0', '-of', 'compact=p=1:nk=0', video_fname],
stdout=PIPE, stderr=STDOUT)
output = result.stdout.decode('utf-8').split('\n')
offset = float(output[0].strip('stream|codec_type=video|start_time')) - \
float(output[1].strip('stream|codec_type=audio|start_time'))
'''
# save to disk as required by ``pd-parser``, raw needs a filename
fname = op.join(out_dir, 'sub-1_task-mytask_raw.fif')
raw.save(fname)
# navigate to corresponding behavior
behf = op.join(out_dir, 'test_video_beh.tsv')
Out:
Creating RawArray with float64 data, n_channels=1, n_times=16464896
Range : 0 ... 16464895 = 0.000 ... 343.019 secs
Ready.
Writing /var/folders/s4/y1vlkn8d70jfw7s8s03m9p540000gn/T/tmp_mne_tempdir_ran50pv8/sub-1_task-mytask_raw.fif
Closing /var/folders/s4/y1vlkn8d70jfw7s8s03m9p540000gn/T/tmp_mne_tempdir_ran50pv8/sub-1_task-mytask_raw.fif
[done]
Run the parser
Now we’ll call the main function to automatically parse the audio events.
annot, samples = pd_parser.parse_audio(fname, beh=behf,
beh_key='tone_onset_time',
audio_ch_names=['audio'], zscore=10)
Out:
Reading in /var/folders/s4/y1vlkn8d70jfw7s8s03m9p540000gn/T/tmp_mne_tempdir_ran50pv8/sub-1_task-mytask_raw.fif
Opening raw data file /var/folders/s4/y1vlkn8d70jfw7s8s03m9p540000gn/T/tmp_mne_tempdir_ran50pv8/sub-1_task-mytask_raw.fif...
Isotrak not found
Range : 0 ... 16464895 = 0.000 ... 343.019 secs
Ready.
Reading 0 ... 16464895 = 0.000 ... 343.019 secs...
Finding points where the audio is above `zscore` threshold...
17 audio candidate events found
Checking best alignments
0%| | 0/15 [00:00<?, ?it/s]
53%|#####3 | 8/15 [00:00<00:00, 76.93it/s]
100%|##########| 15/15 [00:00<00:00, 82.61it/s]
Best alignment is with the first behavioral event shifted 0.01 s relative to the first synchronization event and has errors: min -6.65 ms, q1 -3.27 ms, med -0.58 ms, q3 3.99 ms, max 11.60 ms, 0 missed events
Excluding events that have zero close synchronization events or more than one synchronization event within `max_len` time
Load the results
Finally, we’ll load the events and use them to crop the video although it requires ffmpeg so it is commented out.
print('Here are the event times: ', annot.onset)
# Crop the videos with ffmpeg
'''
from pd_parser.parse_pd import _read_tsv
beh = _read_tsv(behf)
for i in range(annot.onset.size): # skip the first video
action_time = (beh['tone_onset'][i] - beh['action_onset'][i]) / 1000
run(['ffmpeg', '-i', f'{video_fname}', '-ss',
str(datetime.timedelta(
seconds=annot.onset[i] - action_time - offset)),
'-to', str(datetime.timedelta(seconds=annot.onset[i] - offset)),
op.join(out_dir, 'movement-{}+action_type-{}.mp4'.format(
beh['movement'][i], beh['action_type'][i]))])
'''
Out:
Here are the event times: [ 19.051125 39.913 61.88575 83.5424375<PHONE_NUMBER>
126.07720833<PHONE_NUMBER>7<PHONE_NUMBER>3<PHONE_NUMBER><PHONE_NUMBER>7
250.20858333<PHONE_NUMBER>7 292.14<PHONE_NUMBER>3<PHONE_NUMBER>7]
"\nfrom pd_parser.parse_pd import _read_tsv\nbeh = _read_tsv(behf)\nfor i in range(annot.onset.size): # skip the first video\n action_time = (beh['tone_onset'][i] - beh['action_onset'][i]) / 1000\n run(['ffmpeg', '-i', f'{video_fname}', '-ss',\n str(datetime.timedelta(\n seconds=annot.onset[i] - action_time - offset)),\n '-to', str(datetime.timedelta(seconds=annot.onset[i] - offset)),\n op.join(out_dir, 'movement-{}+action_type-{}.mp4'.format(\n beh['movement'][i], beh['action_type'][i]))])\n"
Total running time of the script: ( 0 minutes 24.390 seconds)


Before Change
:param n_channel: number of saved fields
:return: empty first line of the array to be saved as csv
"""
return np.zeros([1, n_channel])
def _save_current_image(self):
"""
For all the outputs to be saved as images, go through the dictionary
After Change
:param n_channel: number of saved fields
:return: empty first line of the array to be saved as csv
return pd.DataFrame(columns=key_names)
def _save_current_image(self):
For all the outputs to be saved as images, go through the dictionary


{{this.error}}
Do you want to accept the invitation of {{fullName}} ? If you accept the invitation, you will be able to chat with him/her anonymously.


+++ title = "Alexis Bellido" slug = "alexis-bellido" date = 2007-04-06T12:00:00+10:00 tags = ["geeks", "eccentric-encounters", "locals"] locations = ["peru/lima"] draft = false summary = """
Alexis is a friend of my buddy Antonio, as well as the only Drupal developer in Peru. He's a really interesting guy: he's run his own web hosting company; he's built his own house; he's written a book about how to work from home; and he's developed web sites for a bunch of different clients. Today, I visited him at his home in Chaclacayo, and had great fun eating, drinking, chatting, and recording with him.
""" [[comments]] title = "Oh no, the recording!" date = 2007-04-13T14:24:29+10:00 author_name = "Alexis Bellido" author_url = "http://www.ventanazul.com/webzine" content = """Hey Jeremy, the pleasure was all mine.
I'm posting about our conversations on Google SOC and Drupal as soon as I get some time later this week. Keep enjoying your trip!
""" +++Alexis is a friend of my buddy Antonio, as well as the only Drupal developer in Peru. He's a really interesting guy: he's run his own web hosting company; he's built his own house; he's written a book about how to work from home; and he's developed web sites for a bunch of different clients. Today, I visited him at his home in Chaclacayo, and had great fun eating, drinking, chatting, and recording with him.
{{< galleryphoto "2007-04-06-12-00-00--alexis-author-father-tinkerer-and-drupal-guy.jpg" >}}


This example has the same functionality as Demo #8 (time-synced metadata exposing content as an overlay on the video). However, in the former demo the video is positioned with fixed width and height. This is necessary to precisely position overlay content atop the video using pixel values. The current example demonstrates that this can also be accomplished in a responsive design, with overlay content positioned using percentage values.
For additional demos see the Index of Able Player Examples.


PEDIA-workflow usage and program documentation¶
This application implements a workflow for exome variant prediction of human genetics cases on the basis of variant, feature and photographic gestalt information. In order to extend the available data, a large number of cases without exome sequencing results are used for simulation of exome simulation to train and test the PEDIA classifier.
Gene predictions are sourced from Face2Gene and Phenomizer by translating predicted Syndromes to genes. Detected pathogenic variants are cleaned using Mutalyzer and converted to VCF with Jannovar.
This document contains instructions for general installation and usage of the pipeline, but also deeper implementation details for those interested in understanding and changing the source code.


Send Images
Toggle Image Preview
Clear Image List
Reset Settings
Files Intercepted
Remove Image
Save


Click here to open the modal
You can add almost anything inside this  tag. This is simply a clickable container.
Anything can go here.
The close button  and title are optional.
If you move the close button somewhere else in the tag heirarchy, check to make sure it still works (it might). If not you'll need to modify the Javascript file, or move it someplace else.


我是世新大學資訊傳播學系一年級的學生，喜歡攝影跟運動，歡迎與我聯絡。
I'm SHU ICS first grade student, I like photograph and sport, welcome to my page.
In my free time out of school, I like to work out with my friend. I have keep this exercise for two years, because the feeling of growing with my friend is very exciting to me. When we lift heavier than our personal record, we will very happy and have a good meal.


The Group Chat (Vista, Windows 7) Sample Application demonstrates how to create a chat application based on the Peer-to-Peer Grouping and Identity Manager API. It shows a secure multiparty chat application into which you can authenticate people using a password or certificates.
Platforms supported
· Windows Vista (except Starter Edition)
· Windows 7
To Build:
· In a SDK command shell navigate to the directory containing this sample and run "nmake"; or
· Launch the .sln file in Visual Studio .NET
To Run:
· Type groupchat.exe (located in the Vista_DEBUG or WIN7_[ARCH]_DEBUG folder in the directory containing the sample)
· Just use the UI! When using the certificate model the following steps must be followed:
Creating a Peer Identity for each user
In the Group Chat application, a user can generate an identity by
selecting the "Create…" option on the "Identity" menu. The resulting
dialog box prompts the user for the friendly name (in this case, the "chat name")
that will be associated with the new identity.
Before a group can be created, an identity associated with the group creator
needs to be created.
Creating a group that will host the chat participants
When the group owner has an identity, a group is
created by selecting the "Create…" option on the "Group" menu. The resulting
dialog prompts the user for the identity of the group creator, as well as the
name of the group to be created. The newly created graph will be associated with the chosen identity.
Creating participants’ identity information files for invitations
Once the chat group has been created, chat members can solicit the group
owner for invitations to join the group. An invitation consists of an XML data
string derived from a unique identity (as created in step 1). An
invitation is thus exclusively bound to a particular identity. The group
owner uses the Peer-to-Peer Grouping API to generate an invitation from an
identity and then returns this invitation to the user requesting permission to
join the group.
Users that wish to join the chat group create user identities the same way that the group owner identity is created, using the "Create…" option on "Identity" menu. The "Identity" menu also allows the user to save this identity information to a file. This file is then sent out of band of the Peer-to-Peer infrastructure to the group owner, who creates the corresponding invitation.
Creating the invitations to join the group to chat
Once the group owner receives a file containing a user’s identity
information, they can create a file containing an invitation to join the group
using the "Create Invitation…" item on the "Group" menu. Selecting this option
opens the Create Invitation dialog and prompts the user to specify the
location of the identity information (.idt) file, as well as the destination of
the invitation information (.inv) file. The invitation file is then sent
out of band of the Peer-to-Peer infrastructure to the requesting individual.
It is important to note that invitations created with the Group Chat application demonstrate how to create invitations for the "Member" role. A group participant with a member role cannot issue invitations. Only participants with "Admin" credentials can create invitations.
Joining the Chat Group
In order to join the chat group, a user needs to have a predefined identity as
well as the invitation file that corresponds to that identity. If these
two requirements are satisfied, the "Join…" option on the "Group" menu can be
used to join the group. Selecting this option opens the Join Group dialog, which
requires the user to select his identity and location of the invitation
(.inv) file. Pressing the "OK" button initiates the join process. If this
is successful, the chat application will connect to the group and the chat user will be
online.
Whispering to other members
To carry on a private conversation with another member, simply double-click
that member in the 'Chat Members' list. The "Whisper" dialog will allow
the user to send a private message to the selected member.
Do not run groupchat.exe in an elevated command shell
The GroupChat sample uses the following ports: TCP port 3587 (used by p2p networking), UDP port 3540 (PNRP traffic: name registration), UDP port 1900 (SSDP protocol). Make sure that these ports are added to the advanced firewall inbound and outbound rules.
For example run the following in an elevated CMD shell to enable inbound TCP port number 3587
netsh advfirewall firewall add rule dir=in name="GroupChat SDK TCP 3587" localip=any remoteip=any localport=3587 remoteport=any protocol=tcp action=allow enable=yes
To test that the Peer Name Resolution Protocol works refer to the MSDN P2P Blog for the PNRP Debugging Guide Part 1


Matching Any Characters
The regular expressions thus far have matched static text only—rather anticlimactic, indeed. Next we'll look at matching unknown characters.
In regular expressions, special characters (or sets of characters) are used to identify what is to be searched for. The . character (period, or full stop) matches any one character.
Tip | If you have ever used DOS file searches, regex . is equivalent to the DOS ?. SQL users will note that the regex . is equivalent to the SQL _ (underscore). |
Therefore, searching for c.t will match cat and cot (and a bunch of other nonsensical words, too).
Here is an example:
sales1.xls
orders3.xls
sales2.xls
sales3.xls
apac1.xls
europe2.xls
na1.xls
na2.xls
sa1.xls
sales.
sales1.xls
orders3.xls
sales2.xls
sales3.xls
apac1.xls
europe2.xls
na1.xls
na2.xls
sa1.xls
Here the regex sales. is being used to find all filenames starting with sales and followed by another character. Three of the nine files match the pattern.
Tip | You'll often see the term pattern used to describe the actual regular expression. |
Note | Notice that regular expressions match patterns with string contents. Matches will not always be entire strings, but the characters that match a pattern—even if they are only part of a string. In the example used here, the regular expression did not match a filename; rather, it matched part of a filename. This distinction is important to remember when passing the results of a regular expression to some other code or application for processing. |
. matches any character, alphabetic characters, digits, and even . itself:
sales.xls
sales1.xls
orders3.xls
sales2.xls
sales3.xls
apac1.xls
europe2.xls
na1.xls
na2.xls
sa1.xls
sales.
sales.xls
sales1.xls
orders3.xls
sales2.xls
sales3.xls
apac1.xls
europe2.xls
na1.xls
na2.xls
sa1.xls
This example contains one additional file, sales.xls. The file was matched by the pattern sales. as . matches any character.
Multiple .s may be used, either together (one after the other—using .. will match any two characters next to each other) or in different locations in the pattern.
Let's look at another example using the same text. This time you need to find all files for North America (na) or South America (sa) regardless of what digit comes next:
sales1.xls
orders3.xls
sales2.xls
sales3.xls
apac1.xls
europe2.xls
na1.xls
na2.xls
sa1.xls
.a.
sales1.xls
orders3.xls
sales2.xls
sales3.xls
apac1.xls
europe2.xls
na1.xls
na2.xls
sa1.xls
The regex .a. did indeed find na1, na2, and sa1, but it also found four other matches that it was not supposed to. Why? Because the pattern matches any three characters so long as the middle one is a.
What is needed is a pattern that matches .a. followed by a period. Here is another try:
sales1.xls
orders3.xls
sales2.xls
sales3.xls
apac1.xls
europe2.xls
na1.xls
na2.xls
sa1.xls
.a..
sales1.xls
orders3.xls
sales2.xls
sales3.xls
apac1.xls
europe2.xls
na1.xls
na2.xls
sa1.xls
.a.. does not work any better than .a. did; appending a . will match any additional character (regardless of what it is). How then can you search for . when . is a special character that matches any character?
|


Normani Unloads Jaw-Dropping Lingerie Pics + Makes Huge Rihanna Savage x Fenty Announcement
by Angie DareR&B singer Normani is doing big things in 2019. The crooner has shocked the Internet with a grip of new lingerie pics and announced she’s the first-ever brand ambassador for Rihanna‘s popular Savage x Fenty line.
This week, Normani went to her social media pages to break the huge news.
Rihanna also went to her Twitter page to confirm the major power move.
Earlier this month, Chicago rap artist Dreezy flexed black girl magic in Savage x Fenty attire.
East Coast rap artist Rico Nasty also recently flexed in Savage x Fenty lingerie.


Implementation of density tracking by quadrature (DTQ) algorithms for stochastic differential equations (SDEs). DTQ algorithms numerically compute the density function of the solution of an SDE with user-specified drift and diffusion functions. The calculation does not require generation of sample paths, but instead proceeds in a deterministic fashion by repeatedly applying quadrature to the Chapman-Kolmogorov equation associated with a discrete-time approximation of the SDE. The DTQ algorithm is provably convergent. For several practical problems of interest, we have found the DTQ algorithm to be fast, accurate, and easy to use.
| Version: | 0.1 |
| Depends: | R (≥ 3.2.0) |
| Imports: | Rcpp (≥ 0.12.4), Matrix (≥ 1.2) |
| LinkingTo: | Rcpp |
| Suggests: | ggplot2 (≥ 2.1), scales (≥ 0.4.0) |
| Published: | 2016-11-22 |
| Author: | Harish S. Bhat, R. W. M. A. Madushani, Shagun Rawat |
| Maintainer: | Harish S. Bhat <hbhat at ucmerced.edu> |
| License: | GPL-2 | GPL-3 [expanded from: GPL (≥ 2)] |
| NeedsCompilation: | yes |
| Materials: | README |
| CRAN checks: | Rdtq results |
| Reference manual: | Rdtq.pdf |
| Package source: | Rdtq_0.1.tar.gz |
| Windows binaries: | r-devel: Rdtq_0.1.zip, r-release: Rdtq_0.1.zip, r-oldrel: Rdtq_0.1.zip |
| OS X binaries: | r-release: Rdtq_0.1.tgz, r-oldrel: Rdtq_0.1.tgz |
Please use the canonical form https://CRAN.R-project.org/package=Rdtq to link to this page.


Please sign here:
Your browser doesn't support signing.
Clear
Save
Copy
Your browser doesn't support signing.