Table of Contents
- Introduction
- Opening the torrent file
- Getting peers via the tracker
- Downloading from peers
- Conclusion
1. Introduction
1.1 About this guide
This guide will build a simple bittorrent client from scratch in node.js. By the end of this tutorial you should be able to use the command line to download the shared contents of a torrent file.
I had two audiences in mind for this guide. First is anyone who has learned the basics of javascript and looking for an intermediate level tutorial. Second is for web developers who may already have a good grasp of javascript but little experience working on the backend. This guide will go over things like reading/writing to file and network sockets.
I’ll be committing changes on this project using git, so you’ll be able to see how the project evolves step by step. You can find the project’s github page here. The first commit will just be a bare package.json, .gitignore file, and a sample torrent file for you to work with. I’ll also mark points throughout this guide with links to commits.
1.2 Overview of bittorrent
The bittorrent protocol has to two main parts.
Step 1: You need to send a request to something called a tracker, and the tracker will respond with a list of peers. More specifically, you tell the tracker which files you’re trying to download, and the tracker gives you the ip address of the users you download them from. Making a request to a tracker also adds your ip address to the list of users that can share that file.
Step 2: After you have the list of peer addresses, you want to connect to them directly and start downloading. This happens through an exchange of messages where they tell you what pieces they have, and you tell them which pieces you want.
You can get an idea of how this all works in the image below (credit to morehawes.co.uk):
1.3 Links and references
These are links that I referenced many times during this project:
wiki.theory.org/BitTorrentSpecification - This is an unofficial bittorrent specification but basically has everything you need to know. Detailed and very readable.
www.bittorrent.org/beps/bep_0015.html - The only thing thing you won’t find in the unofficial spec is how to form a request to a tracker that uses a UDP url. But you can find it here in this link. I’ll remind you about this link again when I cover trackers and UDP.
www.morehawes.co.uk/the-bittorrent-protocol - A very good high level explanation of the bittorrent protocol. With pictures!
www.kristenwidman.com/blog/33/how-to-write-a-bittorrent-client-part-1 - Another good high level explanation.
www.bittorrent.org/beps/bep_0020.html - BEP for peer id conventions.
2. Opening the torrent file
*Github commit #1: initial commit
Enough intro, let’s get started on the code! If you click the link above to go to our initial commit, you see that we’re basically starting from a blank slate. This is our current file structure:
You can see that I’ve added a torrent for us to work with. It contains a picture of a puppy in a cup. Let’s create a new file called index.js and start by opening the torrent file.
index.js:
readFileSync
is the easiest way to read the contents of a file.  Later on you’ll see that all our network messages are sent
and received in the form of buffers, so it’s important that you have a good
understanding of how they work. If you’re not familiar with buffers in node.js
I’ve written a mini guide on buffers here. The short story is that buffers
represent a sequence of raw bytes. If you want to read the buffer as a string
you have to specify an encoding scheme (you can see I used utf-8 above).
The output should have looked something like this:
'd8:announce43:udp://tracker.coppersurfer.tk:6969/announce10:created by13:uTorrent/187013:creation datei1462355939e8:encoding5:UTF-84:infod6:lengthi124234e4:name9:puppy.jpg12:piece lengthi16384e6:pieces160:T�k�/�_(�S\u0011h%���+]q\'B\u0018�٠:����p"�j����1-g"\u0018�s(\u001b\u000f���V��=�h�m\u0017a�nF�2���N\r�ǩ�_�\u001e"2���\'�wO���-;\u0004ע\u0017�ؑ��L&����0\u001f�D_9��\t\\��O�h,n\u001a5g�(��仑,�\\߰�%��U��\u0019��C\u0007>��df��ee'
2.1 Bencode
*Github commit #2: open torrent, get tracker url
That output probably looked fairly incomprehensible to you, and that’s because you’ve probably never heard of bencode. Bencode is data serialization format, and I don’t think I’ve seen it used anywhere outside of torrent files. But you may be familiar with JSON or XML, and bencode is essentially the same idea, it just uses a slightly different format.
Here’s the same data again in JSON:
'{"announce":"udp://tracker.coppersurfer.tk:6969/announce","created by":"uTorrent/1870","creation date":1462355939,"encoding":"UTF-8","info":{"length":124234,"name":"puppy.jpg","piece length":16384,"pieces":"T�k�/�_(�S\\u0011h%���+]q\'B\\u0018�٠:����p\\"�j����1-g\\"\\u0018�s(\\u001b\\u000f���V��=�h�m\\u0017a�nF�2���N\\r�ǩ�_�\\u001e\\"2���\'�wO���-;\\u0004ע\\u0017�ؑ��L&����0\\u001f�D_9��\\t\\\\��O�h,n\\u001a5g�(��仑,�\\\\߰�%��U��\\u0019��C\\u0007>��df��"}}'
That should look a bit more familiar (except for the pieces property, which I’ll get to later). While it’s possible to write our own bencode parser, there are many good open source libraries that can do it for us. I just googled “bencode node” and installed the first result:
npm install --save bencode
Now we can parse our torrent into data that we can interact with:
You’ll notice that many of the values in our decoded torrent object are
buffers. It’s possible to convert all buffers to strings by passing ‘utf8’ (or
any other encoding scheme) as the second arguemnt to bencode.decode
, but
since we’ll be working mostly with buffers when we use this data in the
networking portion of this project, it’s best to keep them as buffers.
3. Getting peers via the tracker
In the code above I console.log
the announce property of the torrent. For
this particular file it happens to be udp://tracker.coppersurfer.tk:6969/announce
.
The announce url is what I’ve been calling the tracker’s url, it is the
location of the torrent’s tracker.
One interesting thing you’ll notice is that instead of the usual ‘http’ in front, this url has ‘udp’. This is because instead of the http protocol, you must use the udp protocol. It used to be that all trackers used http, but nowadays nearly all new torrents are using udp. So what’s the difference between the two protocols, and why switch to udp?
3.1 Http vs udp vs tcp
The main reason that most trackers now use udp is that udp has better performance than http. Http is built on top of another protocol called tcp, which we’ll use later in the project when we start actually downloading files from peers. So what’s the difference between tcp and udp?
The main difference is that tcp guarantees that when a user sends data, the other user will recieve that data in its entirety, uncorrupted, and in the correct order – but it must create a persistent connection between users before sending data and this can make tcp much slower than udp. In the case of upd, if the data being sent is small enough (less than 512 bytes) you don’t have to worry about receiving only part of the data or receiving data out of order. However, as we’ll see shortly, it’s possible that data sent will never reach its destination, and so you sometimes end up having to resend or re-request data.
For these reasons, udp is often a good choice for trackers because they send small messages, and we use tcp for when we actually transfer files between peers because those files tend to be larger and must arrive intact.
3.2 Sending messages with UDP
In the code below, I’ve made some changes to show how we can send and receive messages via UDP. NOTE: This example may not get a response back because the message being sent isn’t formatted correctly. This is just an example to show the various modules we’ll be using.
There’s a lot going on in this code so let’s try to break it down.
-
First we require 3 more modules,
dgram
,buffer
, andurl
. These are all just from the standard library. -
I use the
url
module’s parse method on our tracker url. This lets me easily extract different parts of the url like its protocol, hostname, port, etc. -
The
dgram
module is our module for udp, and here I’m creating a new socket instance. A socket is an object through which network communication can happen. We pass the argument ‘udp4’, which means we want to use the normal 4-byte IPv4 address format (e.g. <IP_ADDRESS>). You can also pass ‘udp6’ for the newer IPv6 address format (e.g. FE80:CD00:0000:0CDE:1257:0000:211E:729C) but this format is still rarely used. -
In order to send a message through a socket, it must be in the form of a buffer, not a string or number.
Buffer.from
is an easy way to create a buffer from a string, see my mini guide for more info on buffers. -
The socket’s
send
method is used for sending messages. The first argument is the message as a buffer. The next two arguments let you send just part of the buffer as the message by specifying an offset and length of the buffer, but if you’re just sending the whole buffer you can just set the offset to 0 and the length to the whole length of the buffer. Next is the port and host of the receiver’s url. Finally the last argument is a callback for when the message has finished sending. -
Here we tell the socket how to handle incoming messages. Whenever a message comes back through the socket it will be passed to the callback function.
3.3 UDP tracker protocol and message format
Now we know how to send and receive messages through UDP, but in order to get a list of peers from the tracker, the tracker will be expecting messages to follow a specific protocol. In short you must:
- Send a connect request
- Get the connect response and extract the connection id
- Use the connection id to send an announce request - this is where we tell the tracker which files we’re interested in
- Get the announce response and extract the peers list
Since the amount of code related to the tracker is starting to get large, at this point I want to create a new file called tracker.js and move all tracker related code there.
tracker.js:
*Github commit #3: move UDP code into new file tracker.js
Here I’m exporting just the one function getPeers
which fairly closely follow
the 4 step protocol that I listed above. When we finally get the list of peers
we pass it to the callback. The other 6 functions are as
follows:
-
udpSend
is just a convenience function that mostly just callssocket.send
but lets me avoid having to set the offset and length arguments since I know I want to send the whole buffer, and sets a default callback which is just an empty function, since I mostly don’t need to do anything after sending the message (see point 5 in the previous section if you need a refresher onsocket.send
) -
respType
will check if the response was for the connect or the announce request. Since both responses come through the same socket, we want a way to distinguish them. -
to 6. These 4 methods will build and parse the connect and announce messages.
index.js has also been refactored to use our new getPeers
function.
index.js:
3.3.1 Connect messaging
Now let’s take a look at actually building the messages. Each message is a buffer with a specific format described in the BEP. Let’s take a look at the connect request first.
The BEP describes the connect request as follows:
Offset Size Name Value
0 64-bit integer connection_id 0x41727101980
8 32-bit integer action 0 // connect
12 32-bit integer transaction_id ? // random
16
This tells us that our message should start out with a 64-bit (i.e. 8 bytes) integer at index 0, and that the value should be 0x41727101980. Since we just write 8 bytes, the index of the next part is 8. Now we write 32-bit integer (4 bytes) with the value 0. This moves us up to an offset of 12 bytes, and we write a random 32-bit integer. So the total message length is 8 bytes + 4 bytes + 4bytes = 16 bytes long, and should look something like this:
<Buffer 00 00 04 17 27 10 19 80 00 00 00 00 a6 ec 6b 7d>
Let’s see how we can implement this in our tracker.js file.
tracker.js:
-
First we require the built-in crypto module to help us create a random number for our buffer. We’ll see that in action shortly.
-
Then we create a new empty buffer with a size of 16 bytes since we already know that the entire message should be 16 bytes long.
-
Here we write the the connection id, which should always be 0x41727101980 when writing the connection request. We use the method
writeUInt32BE
which writes an unsigned 32-bit integer in big-endian format (more info here). We pass the number 0x417 and an offset value of 0. And then again the number 0x27101980 at an offset of 4 bytes.You might be wondering 2 things: what’s with the 0x? and why do we have to split the number into two writes?
The 0x indicates that the number is a hexadecimal number, which can be a more conventient representation when working with bytes. Otherwise they’re basically the same as base 10 numbers.
The reason we have to write in 4 byte chunks, is that there is no method to write a 64 bit integer. Actually node.js doesn’t support precise 64-bit integers. But as you can see it’s easy to write a 64-bit hexadecimal number as a combination of two 32-bit hexadecimal numbers.
-
Next we write 0 for the action into the next 4 bytes, setting the offset at 8 bytes since just wrote an 8 byte integer. This values should always be 0 for the connection request.
-
For the final 4 bytes we generate a random 4-byte buffer using
crypto.randomBytes
which is a pretty handy way of creating a random 32-bit integer. To copy that buffer into our original buffer we use thecopy
method passing in the offset we would like to start writing at.
Parsing the response is much simpler. Here’s how the response is formatted:
Offset Size Name Value
0 32-bit integer action 0 // connect
4 32-bit integer transaction_id
8 64-bit integer connection_id
16
And here’s the parsing function:
tracker.js:
You can see I read the action and the transaction id as unsigned 32 bit
big-endian integers, passing in the offset. Then I just use the
slice
method to get the last 8 bytes. Since I can’t read a 64-bit integer it’s easier to
just leave it as a buffer.
github commit #4: add connection request and response functions
3.3.2 Announce messaging
Many of the concepts here are the same as the connect request and response. However there are some things to look out for that I’ll mention. Here’s the announce request below.
Offset Size Name Value
0 64-bit integer connection_id
8 32-bit integer action 1 // announce
12 32-bit integer transaction_id
16 20-byte string info_hash
36 20-byte string peer_id
56 64-bit integer downloaded
64 64-bit integer left
72 64-bit integer uploaded
80 32-bit integer event 0 // 0: none; 1: completed; 2: started; 3: stopped
84 32-bit integer IP address 0 // default
88 32-bit integer key ? // random
92 32-bit integer num_want -1 // default
96 16-bit integer port ? // should be betwee
98
tracker.js
util.js:
Here’s a quick list of things I want to mention about this code:
-
“peer id” is used to uniquely identify your client. I created a new file called util.js to generate an id for me. A peer id can basically be any random 20-byte string but most clients follow a convention detailed here. Basically “AT” is the name of my client (allen-torrent), and 0001 is the version number.
As you can see the id is only generated once. Normally an id is set every time the client loads and should be the same until it’s closed. We’ll we using the id again later.
-
The code for calculating “info hash” and “left” are a bit complicated so I wanted to split that out into a new file called torrent-parser.js. We’ll go over that in the next section, but for now you can just ignore those parts.
-
Notice that this function takes a
torrent
parameter now. Again, this is for the “info hash” and “left” fields, we’ll be going over that shortly. -
If you look at the BEP for this request it tells you what the offsets should be, you don’t have to count them out yourself!
-
I expect
connId
to be a buffer, since that’s how we left it in theparseConnResp
method, so I just copy it into the buffer directly. -
There are a couple spots where I need a 64-bit integer but since they’re always initialized to 0, I just used
Buffer.alloc(8)
, as this will just create an 8-byte buffer with all 0s. -
For the “num want” part I want to point out that I used writeInt32BE instead of writeUIntBE (note the U). Because the number is negative you cannot used the unsigned version.
-
Finally, for the port, the official spec says that the ports for bittorrent should be between 6881 and 6889, so I’ve decided to use a default of 6681.
Now let’s take a look at parsing the response:
Offset Size Name Value
0 32-bit integer action 1 // announce
4 32-bit integer transaction_id
8 32-bit integer interval
12 32-bit integer leechers
16 32-bit integer seeders
20 + 6 * n 32-bit integer IP address
24 + 6 * n 16-bit integer TCP port
20 + 6 * N
It’s a bit tricky because the number of addresses that come back isn’t fixed. The addresses come in groups of 6 bytes, the first 4 represent the IP address and the next 2 represent the port. So our code will need to correctly break up the addresses part of the response.
Most of this code is pretty straightforward. I think you could put the group
function outside or in a new file which would make it reusable, but for the
purposes of this tutorial I think it’s fine. Also a bit surprising that you can
call join
on a buffer. This is a bit fancy as it coerces the bytes into a
string but seems to work and I think it looks nice.
3.3.3 Info hash and torrent size
I created a new file called torrent-parser.js where I can keep all the code related getting information out of a torrent file. This means I also want to move the code for opening a torrent file here.
torrent-parser.js:
index.js:
github commit #5: add announce request and response
<IP_ADDRESS> Info Hash
Now let’s go back to when we first opened the torrent file. Remember it looked something like this:
'{"announce":"udp://tracker.coppersurfer.tk:6969/announce","created by":"uTorrent/1870","creation date":1462355939,"encoding":"UTF-8","info":{"length":124234,"name":"puppy.jpg","piece length":16384,"pieces":"T�k�/�_(�S\\u0011h%���+]q\'B\\u0018�٠:����p\\"�j���1-g\\"\\u0018�s(\\u001b\\u000f���V��=�h�m\\u0017a�nF�2���N\\r�ǩ�_�\\u001e\\"2���\'�wO���-;\\u0004ע\\u0017�ؑ��L&����0\\u001f�D_9��\\t\\\\��O�h,n\\u001a5g�(��仑,�\\\\߰�%��U��\\u0019��C\\u0007>��df��"}}'
Last time we pulled the announce property from this object. Can you see that it also has an info property? If you were take the info property and pass it through a SHA1 hashing function, you would get the info hash! You can apply a SHA1 hash easily using the built-in crypto module.
torrent-parser.js:
Why use a SHA1 hashing function?
SHA1 is one of many hashing functions but it’s the one used by bittorrent so
in our case no other hashing function will do. We want to use a hash because
it’s a compact way to uniqely identify the torrent. A hashing function returns
a fixed length buffer (in this case 20-bytes long). For example, our example
torrent would output <Buffer 11 7e 3a 66 65 e8 ff 1b 15 7e 5e c3 78 23 57 8a db 8a 71 2b>
.
Because it’s very unlikely for two inputs to output the same hash value, and because the input (the info property) contains information about every piece of the torrent’s files (more about this later), it’s a good way to uniquely identify a torrent. That’s why we must send the info hash as part of the request to the tracker, we’re saying we want the list of peers that can share this exact torrent.
<IP_ADDRESS> Torrent Size
When we were building the announce request we saw that we needed to fill out the “left” field. Actually we just send the whole size of the torrent files. Let’s see how we can calculate that.
torrent-parser.js:
github commit #6: add infoHash and size functions to torrent-parser
There are two cases we have to consider - torrents that have one file or more
than one file. If the torrent only has one file then we can get it’s size in
the torrent.info.length
property. But if it has multiple files, it will have
a torrent.info.files
property instead which is an array of objects for each
file. We have to iterate over these file objects and sum their length property.
There’s one more problem we have to consider with the file size, which is that it may be larger than a 32-bit integer. The easiest way to deal with this is to install a module to handle larger number for us. I’ve decided to use the ‘bignum’ library which is a popular one for this use case. Install it with the command:
npm install –save bignum
You can see that I write the number into a buffer using the bignum.toBuffer
function. The option {size: 8}
tells the function you want to write the
number to a buffer of size 8 bytes. This is also the buffer size required by
the announce request.
3.3.4 Finishing touches, response type and retries
Now almost all the pieces have come together for communicating with the
tracker. The last thing we have to do is write the respType
function to
identify whether a response was a connect response or an announce response.
There’s more than one way to write this method. After looking at the structure of the two response types I noticed that the connect response has an action value of 0 and the announce response has an action value of 1.
tracker.js:
github commit #7: add respType function to tracker.js
And that’s all we need to be able to run the program and get a list of peers for our torrent.
It’s possible that when you run the program it will hang. As I mentioned before it’s possible for udp messages to get dropped in transit, so if that happens just try rerunning the program. Or you could write your own function to retry after a timeout as a bonus exercise!
If you do attempt this, you should wait 2^n * 15 seconds between each request up to 8 requests total according to the BEP. This is called exponential backoff and the reason you want to do this is to balance two concerns.
-
It’s possible your message was lost in transmission, in which case you retransmit as soon as possible.
-
The message is still coming, but the network is experiencing traffic. In this case you don’t want to spam the network with even more requests.
This is a common technique for unreliable network communications.
4. Downloading from peers
Now that we’re able to get a list of peers for our files, we want to actually download the files from them. Here’s a basic overview of how this will work:
-
First you’ll want to create a tcp connection with all the peers in your list.The more peers you can get connected to the faster you can download your files.
-
After exchanging some messages with the peer as setup, you should start requesting pieces of the files you want. As we’ll see shortly, a torrent’s shared files are broken up into pieces so that you can download different parts of the files from different peers simultaneously.
-
Most likely there will be more pieces than peers, so once we’re done receiving a piece from a peer we’ll want to request the next piece we need from them. Ideally you want all the connections to be requesting different and new pieces so you’ll need to keep track of which pieces you already have and which ones you still need.
-
Finally, when you receive the pieces they’ll be stored in memory so you’ll need to write the data to your hard disk. Hopefully at this point you’ll be done!
4.1 Setup
Before we start I want to add a new file called download.js. Also, because we’re starting to have more files, I want to create a new folder called “src” and move our code there. Finally I’ve decided to remove the puppy.torrent file. Since it doesn’t have many peers, it’s not going to be that useful for this part of the project. I recommend that you find a small torrent with a lot of peers that you can play around with. Our new file structure should look like this:
I’ve also updated index.js to reflect these changes:
index.js:
Note the updated require paths. process.argv[2]
will get arguments from
the command line so you can run in the terminal something like:
node index.js /file/path/to/name-of-torrent.torrent
Also note I’m using the method exported by download.js.
github commit #8: restructure and add download.js
<a name=”tcp-connect-to-peers></a>
4.2 TCP connect to peers
Using tcp to send messages is similar to udp which we used before. In this case we use the “net” module instead of the “dgram” module. Let’s look at an example of how that would work.
You can see the tcp interface is very similar to using udp, but you have to
call the connect
method to create a connection before sending any messages.
Also it’s possible for the connection to fail, in which case we don’t want the
program to crash so we catch the error with socket.on('error', console.log)
.
This will log the error to console instead. Udp didn’t have this problem
because udp doesn’t need to create a connection.
Applying what we learned above about tcp, we can start writing our download.js file.
download.js:
We use our getPeers
method from the tracker.js file, and then for each peer
we create a tcp connection and start exchanging messages. Before we can go on
we need to know what messages we’ll be sending and receiving.
4.3 Protocol overview
Once a tcp connection is established the messages you send and receive have to follow the following protocol.
-
The first thing you want to do is let your peer know know which files you are interested in downloading from them, as well as some identifying info. If the peer doesn’t have the files you want they will close the connection, but if they do have the files they should send back a similar message as confirmation. This is called the “handshake”.
-
The most likely thing that will happen next is that the peer will let you know what pieces they have. This happens through the “have” and “bitfield” messages. Each “have” message contains a piece index as its payload. This means you will receive multiple have messages, one for each piece that your peer has.
The bitfield message serves a similar purpose, but does it in a different way. The bitfield message can tell you all the pieces that the peer has in just one message. It does this by sending a string of bits, one for each piece in the file. The index of each bit is the same as the piece index, and if they have that piece it will be set to 1, if not it will be set to 0. For example if you receive a bitfield that starts with 011001… that means they have the pieces at index 1, 2, and 5, but not the pieces at index 0, 3,and 4.
It’s possible to receive both “have” messages and a bitfield message, if which case you should combine them to get the full list of pieces.
-
Actually it’s possible to recieve another kind of message, the peer might decide they don’t want to share with you! That’s what the choke, unchoke, interested, and not interested messages are for. If you are choked, that means the peer does not want to share with you, if you are unchoked then the peer is willing to share. On the other hand, interested means you want what your peer has, whereas not interested means you don’t want what they have.
You always start out choked and not interested. So the first message you send should be the interested message. Then hopefully they will send you an unchoke message and you can move to the next step. If you receive a choke message message instead you can just let the connection drop.
-
At this point you’re ready start requesting. You can do this by sending “request” messages, which contains the index of the piece that you want (more details on this in the next section).
-
Finally you will receive a piece message, which will contain the bytes of data that you requested.
4.4 Message types
According to the spec the handshake message should be a buffer that looks like this:
handshake: <pstrlen><pstr><reserved><info_hash><peer_id>
pstrlen: string length of <pstr>, as a single raw byte
pstr: string identifier of the protocol
reserved: eight (8) reserved bytes. All current implementations use all zeroes.
peer_id: 20-byte string used as a unique ID for the client.
In version 1.0 of the BitTorrent protocol, pstrlen = 19, and pstr = "BitTorrent protocol".
Once the handshake has been established there are 10 different types of messages that can be exchanged, all following the same format:
- 4 bytes indicating the length of the message (excluding these 4 bytes)
- 1 byte for the id message
- the rest of the buffer is the message payload which varies by message
I’ll discuss these in more detail as they come up, but you can also find more info in the specs.
I’ve created a new file called message.js with some convenient functions to help build these message buffers.
message.js:
These functions are mostly straightforward, just pass a payload and then return a buffer with the appropriate length and id. Everything follows directly from the specs mentioned earlier.
github commit #9: add message.js
4.5 Grouping messages
Before going on to actually exchanging messages, there’s one more thing I need to address about tcp. You may have assumed that every time you recieve a data though a socket, it will be a single whole message. But this is not the case. Remember our code for receiving data looked like this:
socket.on('data', receivedBuffer => {
// do stuff with receivedBuffer here
})'
The problem is that the callback gets passed data as it becomes available and there’s no way to know how that data will be broken up. The socket might recieve only part of one message, or it might receive multiple messages at once. This is why every message starts with its length, to help you find the start and end of each message.
Things would be much easier for us if each time the callback was called it
would get passed a single whole message, so I want to write a function
onWhileMsg
that will do just that for us.
Like this:
download.js:
Here is the implementation of onWholeMsg
:
download.js:
How does this function work? First notice the distinction between the function
onWholeMsg
, the callback passed to onWholeMsg
, and the anonymous callback
passed to socket.on
. Next, the key to making this work is to use a closure.
Because the socket.on
callback is inside of the onWholeMsg
function, the
former will always be able to access the variables of the latter.
So every time the socket recieves data, the socket.on
callback is called. It
concats the new data with savedBuf
and as long as savedBuf
is long enough
to contain at least one whole message, it will pass it to the onWholeMsg
callback and then update savedBuf
by slicing out those messages. Basically
savedBuf
saves the pieces of incomplete messages between rounds of receiving
data from the socket.
I also have the handshake
variable in the closure. This is because the
handshake message doesn’t tell you its length as part of the message. The only
way you can tell you’re receiving a handshake message is that it’s always the
first message you’ll receive. That’s why I start with handshake
set to true,
and then the first time we receive a whole message I set it to false.
That all might have been a bit mind-bending if you’re not too familiar with
closures yet, but I encourage you to stick with it and read through the code
carefully. One thing that might help is to realize that the onWholeMsg
function is only getting called once, so the savedBuf
and handshake
variables are only initialized once. But then the socket.on
callback gets
called multiple times, each time getting and setting the same two variables.
github commit #10: add onWholeMsg function
4.6 Handshake
Now let’s code the handshake:
download.js:
-
We use the
buildHandshake
function we created in the message.js file, but this method requires the torrent object, so we now pass that in through thedownload
function. -
I didn’t want a long callback function in the
download
function so I created a new function calledmsgHandler
. This function will check what kind of message we are receiving and handle it accordingly. Here it checks if the message is a handshake response, and if so it sends the interested message and hopefully the peer will send an unchoke message. -
This function checks if the message is a handshake. Basically just checks that it’s the same length as a handshake and has pstr ‘BitTorrent protocol’.
github commit #11: add handshake handler
4.7 Pieces
After you establish the handshake your peers should tell you which pieces they have. Let’s take a moment to understand what pieces are exactly.
If you open up a torrent file, we saw that it contains data with various properties like the “announce” and “info” properties. Another property is the “piece length” property. This tells you how long a piece is in bytes. Let’s say hypothetically that you have a piece length of 1000 bytes. Then if the total size of the file(s) is 12000 bytes, that means the file should have 12 pieces. Note that the last piece might not be the full 1000 bytes. If the file were 12001 bytes large, then it would be a total of 13 pieces, where the last piece is just 1 byte large.
These pieces are indexed starting at 0, and this is how we know which piece it is that we are sending or receiving. For example, you might request the piece at index 0, that means from our previous example we want the first 1000 bytes of the file. If we ask for the piece at index 1, we want the second 1000 bytes and so on.
4.8 Handling messages
Now I want to start handling messages that aren’t the handshake message. Since these messages have a set format, I can just check their id to figure out what message it is. In order to help me do this I wrote added a function to message.js for parsing message buffers into their parts:
message.js:
If the message length isn’t greater than 4, then we know it is the keep-ahead message which has no id. If the length isn’t greater than 5 we know that it has no payload. If the id is 6, 7, 8, those messages split the pay load into index, begin, and block/length.
Now we can do this:
download.js:
github commit #12: add non-handshake message handlers
So the msgHandler
function receives a message, checks the id, and then passes
the payload, if any, to the appropriate handler function. You’ll notice we
aren’t handling all types of messages. The other messages are for sending files
rather than receiving them, so these are the messages I’ll focus on for now.
4.9 Managing connections and pieces
This is a critical point in the project because managing the connections and pieces involves a lot of interesting decisions and tradeoffs. So far we’ve mostly been following the specs directly, but from now on there are many possible solutions and I’ll be going through just one. That’s why I recommend taking some time to consider how you would implement these message handlers yourself before continuing.
Of course a big concern is efficiency. We want our downloads to finish as soon as possible. The tricky part about this is that not all peers will have all parts. Also not all peers can upload at the same rate. On top of that, it’s possible for connections to drop at any time, so we need a way of dealing with failed requests. How can we distribute the work of sharing the right pieces among all peers in order to have the fastest download speeds?
4.9.1 List of requested pieces
After some thought I decided on the following solution. First I would have a single list of all pieces that have already been requested that would get passed to each socket connection. Like this:
download.js:
The actual implementation of haveHandler
will be more detailed than this, but
you can see how the requested
list will get passed through and how it will be
used to determine whether or not a piece should be requested. You can also see
that there is just a single list that is shared by all connections. Some of you
may think that it’s inconvenient to have to pass the requested list to so many
functions. Again, there’s more than one possible solution, so I encourage you
to try you own.
This doesn’t yet account for failed requests, but I’ll be addressing that in a later section.
4.9.2 Job queue
Next I want to create a list per connection. This list will contain all the pieces that a single peer has. Why do we have to maintain this list? Why not just make a request for a piece as soon as we receive a “have” or “bitfield” message? The problem is that we would probaby end up requesting all the pieces from the very first peer we connect to and then since we don’t want to double request the same piece, none of the other peers would have pieces left to request.
Even if it’s possible to use a round-robin strategy so that each peer only gets a second piece to request after all peers have gotten at least one piece to request, there is still a problem. This strategy would lead to all peers having the same number of requests, but some peers will inevitably upload faster than others. Ideally we want the fastest peers to get more requests, rather than have multiple requests bottlenecked by the slowest peer.
A natural solution is to request just one or a few pieces from a peer at a time, and only make the next request after receiving a response. This way the faster peers will send their responses faster, “coming back” for more requests more frequently.
However, because we recieve the “have” and “bitfield” messages all at once, this means we’ll have to store the list of pieces that the peer has. This is so that we can wait until the piece’s response come back, then we can reference the list to see what to request next.
I refer to this as a job queue, because you can think of it like this: each connection has a list of pieces to request. They look at the first item on the list, and check if it’s in the list of already requested pieces or not. If not, they request the piece and wait for a response. Otherwise they discard the item and move on to the next one. When they receive a response, they move on to the next item on the list and repeat the process until the list is empty.
This list will also need to passed through to the handler functions, but it should be created per connection. Like this:
download.js:
So we only request a piece on a “have” if the queue was previously empty
(should be similar for bitfieldHandler
). From that point on we only request
pieces once we’ve received a piece response (in pieceHandler
). When we
receive a piece we can shift it out of the queue. If the piece has already been
requested, we again shift it out of the queue. You can see that happen in the
requestPiece
function which created since it is code shared by both handlers.
4.9.3 Request Failures
The last thing I want to go over before fully implementing the handler
functions is request failures. Right now we are adding the piece index to the
requested
array whenever we send a request. This way we know which pieces
have already been requested and then we can avoid the next peer from requesting
a duplicate piece.
However, it’s possible for us to request a piece but never receive it. This is
because a connection can drop at any time for whatever reason. Since we avoid
requesting pieces that have been added to the requested
array, these pieces
will never be received.
You might think we could just add pieces to the list when we receive them. But then between the time that the piece requested and received any other peer could also request that piece resulting in duplicate requests.
I’ve found that the easiest solution is to maintain two lists, one for
requested pieces and one for received pieces. We update the requested
list at
request time, and the received
list at receive time. Then whenever we have
requested all pieces but there are still pieces that we haven’t received, we
copy the received
list into the requested
list, and that will allow us to
rerequest those missing pieces.
However, it’s inconvenient to have to maintain multiple lists when just one data abstraction should be necessary. So I’ve create a new file called Pieces.js:
Pieces.js:
You can see that we need to initialize the Pieces instance with the total number of pieces, otherwise it’s not possible to know when we’re finished getting all the pieces. We assume that the array index is the same as the piece index, and then we set that index to true if it has been requested/received (all indexes start as false by default). This is a nice way to look up a piece’s status without iterating through the arrays.
This object can now be used to replace the requested
array from earlier.
4.10 Implementing the message handlers
4.10.1 Choke and unchoke
Dealing with choke and unchoke states is a little bit tricky because we don’t
want to request any pieces until we’ve been unchoked. I think the simplest way
to enforce this is to create a new object that holds both our queue
array as
well as a choked property.
Now we can add this new object and write out the choke and unchoke handlers.
download.js:
github commit #14: add choke and unchoke handlers
Here’s the new download.js file in its entirety since there’s quite a bit going on here. Fortunately most of it was covered in the previous sections. Most of these changes are simply passing our two new data structures Pieces and queue object through through to the handlers. I’ve marked all these lines with a 1.
In the previous section I changed the Pieces class so that its constructor
takes the total number of pieces as an argument. You can find this value by
looking up the torrent.info.pieces.length
property and dividing by 20. The
torrent.info.pieces
is a buffer that contains 20-byte SHA-1 hash of each
piece, and the length gives you the total number of bytes in the buffer.That’s
why we divide by 20 to get the total number of pieces.
The other big change is that we begin requesting pieces when we are unchoked
(which I’ve marked with 2). You can see that when you’re choked, you can’t make
any requests, and nothing happens. If you’re unchoked then we shift out piece
indexes until we find one that we don’t already have using the pieces.needed
method we coded earlier and request that piece. Finally we add the requested
index into pieces
and break the loop. The next request won’t happen until we
get a “piece response” which we’ll code up in the pieceHandler
function.
You might have noticed that I wrote a comment above socket.write(message.buildRequest(pieceIndex));
that something needs to be fixed. We’ll address this problem in the next
section.
4.10.2 Pieces vs. blocks
The function message.buildRequest
in the above code needs to take an object
with an index, begin, and a length property. These are the required fields for
the payload of a request message.
But what are these fields for exactly? Well index is easy, it’s the piece index
that is currently being passed in. What about begin and length?
These two properties are necessary because sometimes a piece is too big for one message. Although there is some dispute about the best size, it is typically considered to be around 2^14 (16384) bytes. This is called a “block”, where a piece consists of one or more blocks. If the piece length is greater than the length of single block, then it should be broken up into blocks with one message sent for each block.
So the “begin” field is the zero-based byte offset starting from the beginning of the piece, and the “length” is the length of the requested block. This is always going to be 2^14, except possibly the last block which might be less.
We’re going to need a number of modifications to accomodate this. But first I want to write some function in torrentParser.js that will help me determine the piece and block lengths for a given piece index.
torrent-parser.js:
In pieceLen
I had to use the bignum library to deal with the totalLength
variable because it’s in an 8 byte buffer. pieceLen
and blockLen
are
similar, they just return the piece length or block length, respectively,
unless they happen to be the very last piece or block. Then it might be shorter
than a full piece or block. Piece length can be found in the torrent file, and
block length is 2^14 bytes by convention.
Also, both the queue object and the Pieces class should be changed to deal with blocks instead of just pieces. For the queue object we just used a plain javascript object with two properties, but with the added complexity of dealing with blocks, I’ve decided to create a new Queue class.
Queue.js:
Now the queue is a list of pieceBlock
objects. I admit the terminology is getting
a little confusing but I’m referring to the object built in the queue
method
that have index, begin, and length properties. These pieceBlock
objects have the
same structure as the payload when we send a request message (check the
buildRequest
function in message.js), so we can pass them to the request
builder directly.
More generally, from now on we want to deal with these objects instead of the piece index, because it also gives us information about the block. Here’s an outline of when/where we would use these objects:
-
When we receive a “have” message, we get a piece index, but we fill the queue with these piece objects, one for each block for each piece.
-
When we deque an object we can pass it to the request builder and make a request for the associated block.
-
The Pieces class that tracks requested and received pieces should be able to add a
pieceBlock
. We’ll see this implemented next.
Note that the constructor need to get passed a torrent object now, instead of just the number of pieces.
I’ve prepended private variables with an underscore, and the other methods just
give us public methods to interact with the private _queue
variable. I’ve
left the choked
without an underscore to indicate it is a public property. If
you’re not familiar with private and public properties, it’s not that
important. The idea is that users of this module shouldn’t use the underscore
prepended properties.
Here is the Pieces class altered to handle blocks.
Pieces.js
Now the requested
and received
arrays, which used to hold the status of a
piece index, now holds an array of arrays, where the inner arrays hold the
status of a block at a give piece index. So if you wanted to find out the
status of a block at index 1 for a piece at index 7, you could look up
_requested[7][1]
and check if it’s set to true. As before all values are
initially set to false.
Also note that these methods expect to be passed a pieceBlock, the same ones we
saw in the Queue class. This means when we peek
a pieceBlock from a queue and
request it, or deque
a pieceBlock when we’ve received it, we can pass the
pieceBlock directly to addRequested
or addReceived
respectively.
Finally there’s a few changes that need to be made in download.js where we use the Queue and Pieces classes.
download.js:
github commit #15: support blocks
This code is mostly the same as before except that we’re passing and using our new Pieces and Queue classes. Also we now use the pieceBlock object instead of just the piece index.
4.10.3 Have and bitfield
These two messages do roughly the same thing, which is tell us which pieces the peer has for sharing. Let’s start with the “have” message.
download.js:
This will add the piece index into the queue. We also want to kick off the piece requesting process if this is the first item added to the queue. Why only the first? Because as we discussed before, we want to wait for the piece response to come back before requesting the next piece.
Now let’s look at bitfieldHandler
:
download.js:
github commit #16: add have and bitfield handlers
The payload here is a buffer, which you can think of as a long string of bits. If the peer has the index-0 piece, then the first bit will be a 1. If not it will be 0. If they have the index-1 piece, then the next bit will be 1, 0 if not. And so on. So we need a way to read individual bits out of the buffer.
I don’t want to dwell on this part too much. I think it’s a good exercise for the reader to try themselves. Basically repeatedly dividing by 2 and taking the remainder will convert a base-10 number to a binary number, giving you the digits of the binary number from least to most signifiant bit (right to left).
Just like haveHandler
we queue the piece index, and only make a request we
are adding the first piece in the queue.
4.10.4 Piece response handler
This is the final section before we can start downloading torrents! We just
need to write out the pieceHandler
which is for when we receive the actual
bytes of the piece we requested.
The main things we want to do in this function are:
- Add the piece to the list of received pieces.
- Write the bytes to file.
- Request the next needed piece. Or…
- If there are no more pieces to download in the queue we close the socket connection.
download.js:
I’m calling the response piece pieceResp
. It’s slightly different from a
pieceBlock because instead of the length the of piece, it contain the actual
data for the piece. But the index
and begin
properties are the same, so
you can pass it to the pieces.addReceived
method normally. We also get to use
our pieces.isDone
method here to check if we’re done. If we are we close the
socket, otherwise we request the next piece.
The only thing we need to do now is write the bytes to file. We can do this
using the fs
module by using the fs.openSync
method to open a file. This will return a file descriptor, which we pass to the
fs.write
method. Since want all the peers to be writing to the same file, creating the
file descriptor is one of the very first things we need to do:
download.js:
The openSync
method takes a file path (this includes the file name), which is
where the file will be created. Note that this exported function now takes a
second path
argument. We then use ‘w’ or write mode to create and write to
a new file. You could also change this code so that you can resume downloads
instead of overwriting and previous existing data, but we’ll keep things simple
for now. This file descriptor should get passed all the way down to the
pieceHandler
function.
The final pieceHandler
function should look like this:
download.js:
Since pieceResp.begin
only tells us the offset relative to the piece it’s in,
we have to calculate the absolute offset ourselves. Then we just write the
block of data to the file descriptor we create earlier. That’s it!
The very last thing we need to do is pass the filepath into the exported function.
index.js:
The only change we made here is to pass the torrent file’s name to the
download
function.
github commit #17: add piece handler
5. Conclusion
5.1 Improvements
Currently we can only download single files, not files in folders. This wouldn’t be hard to implement. You would just need to check the torrent object to see what the file structure looked like and where the start and end of each file was. In fact there are still many interesting problems you could work on to polish up this project. Here is a list:
- Add a graphic user interface
- Optimize for better download speeds and more efficient cpu usage. For example some clients calculate which pieces are the rarest and download those first.
- There’s also something called distributed hash tables which makes it possible to share torrents without the use of centralized trackers.
- You could write code to reconnect dropped connections
- You could look for more peers periodically.
- You could support pausing and resuming downloads.
- You could support uploading since currently our client only downloads.
- Sometimes peers are unable to connect to each other because they are behind a NAT which gives a proxy ip. You could look into NAT traversal strategies.
- It’s possible to bring bittorrent to the web using WebRTC, which creates a direct peer to peer connection.
The sky’s the limit!


“Plans to protect air and water, wilderness and wildlife, are in fact plans to protect man.”
stewart udall
ABOUT THE COUNCIL
OUR COMMITTEES
nature conservation
public awareness
educational outreach
sustainable waste management
environmental legislation
GET TO KNOW US
become a partner
if you're interested in collaborating with the AEYC, please contact us! We'd love to work with you!
follow us on Instagram
alabamayouthcouncil@gmail.com


Athrun & Aaron
小园香径独徘徊
Home
About
Resume
Page not found
The page you requested could not be found.
Similar URLs


[gecode-users] help --- how to create an object dynamically in Gecode？
Kayler Yao
risoay at gmail.com
Sat Aug 29 17:23:17 CEST 2009
hi,
Your said about using Argument arrays are right . The purpose i use
Argument arrays to do is also just as it should to be . What i want to do is
that I want to use uncertain number of Argument Arrays ,so i want to create
Argument arrays dynamically. You said that they do not have a new operator
by design ,so i can't use "new" way. I maybe create an certain number of
Argument Arrays which is much more than i want ,and it may waste some
memories. Whether there has another way to realize my thought?
welcome to talk about it.
Kayler.
2009/8/29 Christian Schulte <cschulte at kth.se>
> Hi,
>
>
>
> Argument arrays are meant for what they are called: passing arguments as is
> explained in detail in Section 4.2.2 in Modeling with Gecode. Using them for
> other purposes is a bad idea and hence they do not have a new operator by
> design!
>
>
>
> What you should do is use a different data structure and create and
> argument array just when you need it for posting a propagator or branching.
>
>
>
> Christian
>
>
>
> *From:* users-bounces at gecode.org [mailto:users-bounces at gecode.org] *On
> Behalf Of *Kayler Yao
> *Sent:* Saturday, August 29, 2009 5:20 AM
> *To:* users at gecode.org
> *Subject:* Re: [gecode-users] help --- how to create an object dynamically
> in Gecode？
>
>
>
> hi,
>
> And i find that "Gecode::Support::DynamicArray< T, A > Class Template"
> maybe can realize it. But i did it unsuccessfully.
>
> Who can give me right answer.
>
>
>
> Kayler
>
> 2009/8/29 Kayler Yao <risoay at gmail.com>
>
> hi，
>
> I want to create an object of IntArgs in my program dynamically. For
> instance " IntArgs *pt = new IntArgs(4); -----create an object of IntArgs
> with four elements" or
>
> "IntArgs *pt = new IntArgs [10];------create an array with 10 object of
> IntArgs." They all be compiled with errors. And i haven't seen the examples
> of create object dynamically in Gecode. And I want to know whether the
> Gecode support to create object of Gecode class ? And how ?
>
> Look forward to your reply!
>
>
>
> Kayler
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ps.uni-sb.de/pipermail/users/attachments/20090829/95439577/attachment.htm>
More information about the gecode-users
mailing list


...finding freelance resources.
 The Freelance Library hand delivers premium sourced resources to you every month - so you can spend less time consuming and more time creating.


By AMP Tutorials
Dogs come when they're called. Cats take a message and get back to you. --Mary Bly
The life it's like the rain => can be strong or small...
Dogs were probably the first tame animals. They have accompanied humans for some 10,000 years. Some scientists assert that all dogs, domestic and wild, share a common ancestor in the small South Asian wolf.
The life it's like the rain => can be strong or small...
This browser does not support the video element.
Rabbits can learn to follow simple voice commands and come when called by name, and are curious and playful.


|
Simbody
3.6
|
Local class that collects information about bodies. More...
Public Member Functions | |
| Body (const std::string &name, double mass, bool mustBeBaseBody, void *userRef) | |
| void | forgetGraph (MultibodyGraphMaker &graph) |
| int | getNumFragments () const |
| int | getNumSlaves () const |
| int | getNumJoints () const |
| bool | isSlave () const |
| bool | isMaster () const |
| bool | isInTree () const |
Public Attributes | |
| std::string | name |
| double | mass |
| bool | mustBeBaseBody |
| void * | userRef |
| std::vector< int > | jointsAsChild |
| std::vector< int > | jointsAsParent |
| int | level |
| int | mobilizer |
| int | master |
| std::vector< int > | slaves |
Local class that collects information about bodies.
|
inlineexplicit |
| void SimTK::MultibodyGraphMaker::Body::forgetGraph | ( | MultibodyGraphMaker & | graph | ) |
|
inline |
|
inline |
|
inline |
|
inline |
|
inline |
|
inline |
| std::string SimTK::MultibodyGraphMaker::Body::name |
| double SimTK::MultibodyGraphMaker::Body::mass |
| bool SimTK::MultibodyGraphMaker::Body::mustBeBaseBody |
| void* SimTK::MultibodyGraphMaker::Body::userRef |
| std::vector<int> SimTK::MultibodyGraphMaker::Body::jointsAsChild |
| std::vector<int> SimTK::MultibodyGraphMaker::Body::jointsAsParent |
| int SimTK::MultibodyGraphMaker::Body::level |
| int SimTK::MultibodyGraphMaker::Body::mobilizer |
| int SimTK::MultibodyGraphMaker::Body::master |
| std::vector<int> SimTK::MultibodyGraphMaker::Body::slaves |


Reverses PlantUml arrow - any set of characters containing '.' or '-' (PlantUml syntax has something like 30 different types of arrows...)


Save £2.00* on your next shop over £20 at Shortlanesend Store.
Simply bring this coupon with you and present it at the check out.
* Offer excludes lottery, tobacco and stamps.


netket.hilbert.CustomHilbert¶
-
class
netket.hilbert.
CustomHilbert
¶ A custom hilbert space.
-
__init__
(self: netket._C_netket.hilbert.CustomHilbert, graph: netket._C_netket.graph.Graph, local_states: List[float]) → None¶ Constructs a new
CustomHilbert
given a graph and a list of eigenvalues of the states.- Parameters
graph – Graph representation of sites.
local_states – Eigenvalues of the states.
Examples
Simple custom hilbert space.
>>> from netket.graph import Hypercube >>> from netket.hilbert import CustomHilbert >>> g = Hypercube(length=10,n_dim=2,pbc=True) >>> hi = CustomHilbert(graph=g, local_states=[-1232, 132, 0]) >>> print(hi.size) 100
Methods
__init__
(self, graph, local_states)Constructs a new
CustomHilbert
given a graph and a list of eigenvalues of the states.number_to_state
(*args, **kwargs)Overloaded function.
random_vals
(self, state, 1], …)Member function generating uniformely distributed local random states.
state_to_number
(self, conf, 1]])Returns index of the given many-body configuration.
states
(self)Returns an iterator over all valid configurations of the Hilbert space.
update_conf
(self, v, 1], flags.writeable], …)Member function updating a visible configuration using the information on where the local changes have been done.
Attributes
The Graph used to construct this Hilbert space.
Whether the hilbert space is discrete.
We call a Hilbert space indexable if and only if the total Hilbert space dimension can be represented by an index of type int.
Size of the local hilbert space.
List of discreet local quantum numbers.
The total dimension of the many-body Hilbert space.
The number of visible units needed to describe the system.
-
property
graph
¶ The Graph used to construct this Hilbert space.
- Type
netket.graph.Graph
-
property
is_discrete
¶ Whether the hilbert space is discrete.
- Type
bool
-
property
is_indexable
¶ We call a Hilbert space indexable if and only if the total Hilbert space dimension can be represented by an index of type int.
- Returns
Whether the Hilbert space is indexable.
- Return type
bool
-
property
local_size
¶ Size of the local hilbert space.
- Type
int
-
property
local_states
¶ List of discreet local quantum numbers.
- Type
list[float]
-
property
n_states
¶ The total dimension of the many-body Hilbert space. Throws an exception iff the space is not indexable.
- Type
int
-
number_to_state
(*args, **kwargs)¶ Overloaded function.
number_to_state(self: netket._C_netket.hilbert.Hilbert, i: int) -> numpy.ndarray[float64[m, 1]]
Returns the visible configuration corresponding to the i-th basis state for input i. i can be a single integer or an array. Throws an exception iff the space is not indexable.
number_to_state(self: netket._C_netket.hilbert.Hilbert, i: List[int]) -> numpy.ndarray[float64[m, n]]
Returns the visible configuration corresponding to the i-th basis state for input i. i can be a single integer or an array. Throws an exception iff the space is not indexable.
-
random_vals
(self: netket._C_netket.hilbert.Hilbert, state: numpy.ndarray[float64[m, 1], flags.writeable], rgen: std::__1::mersenne_twister_engine<unsigned int, 32ul, 624ul, 397ul, 31ul,<PHONE_NUMBER>u, 11ul,<PHONE_NUMBER>u, 7ul,<PHONE_NUMBER>u, 15ul,<PHONE_NUMBER>u, 18ul,<PHONE_NUMBER>u> = None) → None¶ Member function generating uniformely distributed local random states.
- Parameters
state – A reference to a visible configuration, in output this contains the random state.
rgen – The random number generator. If None, the global NetKet random number generator is used.
Examples
Test that a new random state is a possible state for the hilbert space.
>>> import netket as nk >>> import numpy as np >>> hi = nk.hilbert.Boson(n_max=3, graph=nk.graph.Hypercube(length=5, n_dim=1)) >>> rstate = np.zeros(hi.size) >>> rg = nk.utils.RandomEngine(seed=1234) >>> hi.random_vals(rstate, rg) >>> local_states = hi.local_states >>> print(rstate[0] in local_states) True
-
property
size
¶ The number of visible units needed to describe the system.
- Type
int
-
state_to_number
(self: netket._C_netket.hilbert.Hilbert, conf: numpy.ndarray[float64[m, 1]]) → int¶ - Returns index of the given many-body configuration.
Throws an exception iff the space is not indexable.
-
states
(self: netket._C_netket.hilbert.Hilbert) → netket::StateGenerator¶ - Returns an iterator over all valid configurations of the Hilbert space.
Throws an exception iff the space is not indexable.
-
update_conf
(self: netket._C_netket.hilbert.Hilbert, v: numpy.ndarray[float64[m, 1], flags.writeable], to_change: numpy.ndarray[int32], new_conf: numpy.ndarray[float64]) → None¶ Member function updating a visible configuration using the information on where the local changes have been done.
- Parameters
v – The vector of visible units to be modified.
to_change – A list of which quantum numbers will be modified.
new_conf – Contains the value that those quantum numbers should take.
-


California sues e-cigarette maker Juul over ads and sales
by MATTHEW PERRONE AP Health WriterWASHINGTON (AP) — California is suing the nation’s biggest e-cigarette maker, alleging Juul Labs deliberately targeted teenagers with its early marketing campaigns.
The lawsuit filed Monday by California’s attorney general is the latest legal action against Juul. The company faces multiple state and federal investigations into whether its early marketing efforts helped spark the current vaping craze among underage users.
Juul disputes allegations that it marketed to teens. The company points out that it has halted its advertising and removed most of its flavors from the market.
North Carolina filed the first lawsuit against the company in May.
The lawsuit from California Attorney General Xavier Becerra also alleges that Juul’s website did not previously adequately verify customers’ ages.


Psychological Dot is a blend of 12 and 18 month old golden wild ale aged in white wine barrels on to Vermont grown Purple Sage and our own wildflower honey for another 4 months. At the time of bottling in the summer of 2019 we zested Kaffir and Key Limes to deliver a mouthwatering vibrant minty, pinelike aroma with a wild juicy dry sour citrus finish. 5.8% abv.Available for online presale at 8AM on Thursday, May 21st. Purchased bottles will be available for pickup starting Friday, May 22nd. Local home deliveries will begin Friday as well. There will be an option for those of you who do not live locally to purchase & reserve your bottles as well. $17/500ml bottle. Limit 2pp.
Shop CurrentlyAvailable Beer


CASS provides basic services to implement competency-based education, training, and workforce development programs.
Define, manage, and access competencies and records.
View, create, and manage competencies and competency frameworks.
By providing a universal, consistent and persistent way to reference a given competency, CASS enables learning management systems, HR systems, learning record stores, talent management systems, and student information systems to produce competency-based credentials and transcripts that can be properly interpreted by any organization.
Manage Competency FrameworksCreate, connect, and view a user's competency profile.
By providing records and evidence of a learner’s competencies in a secure manner that protects the learner’s privacy, CASS enables learning activities to be tailored to the needs and capabilities of a learner.
View Competency ProfilesTag resources with competencies in a connected world.
By enabling learning resources to be tagged with machine-readable versions of competencies, CASS facilitates the discovery and selection of resources that can help achieve a learning goal.
Align Resources with CompetenciesDetermine competence through complex rules and relations.
By tracking assertions and relating competencies to one another, assertion processing can fill in the gaps in the learner knowledge profile and help answer the fundamental question: What does the learner know?
See Assertion ProcessingManage objects, permissions and more in your repository.
With the help of power-user tools, batch operations, adapters, and powerful search functionality, the CASS Manager will help you manage your repository and connect it to other systems.
Manage the RepositoryLearn how CASS supports your competency based efforts.
Take a high level tour or get down to the specification level of CASS, we have extensive documentation for supporting or integrating with CASS.
View DocumentationVisit the CASS Project for more information.
Join the conversation via Slack. Login or Request Account


Component: tos.chips.atm128.adc.WireAdcStreamP
configuration WireAdcStreamP
Support component for AdcReadStreamClientC.
Author:
David Gay
Provides
interface
ReadStream
<uint16_t>[uint8_t]
Uses
interface
Atm128AdcConfig
[uint8_t]
interface
Resource
[uint8_t]
Wiring


Before Change
output_profile_query = self._output_profile_transform(query)
_, idx_neighbors = self._get_region_competence(output_profile_query)
idx_neighbors = np.atleast_2d(idx_neighbors)
competences = np.sum(self.processed_dsel[idx_neighbors, :], axis=1)
return competences.astype(dtype=int)
def select(self, competences):
Select the base classifiers for the classification of the query sample.
After Change
_, idx_neighbors = self._get_region_competence(output_profile_query)
competences = np.sum(self.processed_dsel[idx_neighbors, :], axis=1, dtype=np.float)
return competences
def select(self, competences):
Select the base classifiers for the classification of the query sample.


Bilre892.github.io
ISO-TECH IDM 91E INSTRUCTION MANUAL Pdf Download. View and Download ISO-TECH IDM 91E instruction manual online. IDM 91E Multimeter pdf manual download.
There is NO VAT payable on manuals. Please contact me for current prices. We stock manuals from most manufacturers,Tektronix, Phillips, Marconi, Racal Iso-Tech 36 Ii Bedienungsanleitung.


|
Project Progress
Week 15 (21st Apr - 27th Apr 2020)
Readings:
An Empirical Study of Example Forgetting during Deep Neural Network Learning.
Do Better ImageNet Models Transfer Better?
When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks.(Camera ready version)
Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations.
Making the outline structure of the paper.
Making various timelines for the submission.
Running more experiments.
Thinking about what to do with nas meets robustness paper, as we cannot compare with them because they follow a different experimental settings.
Week 14 (14th Apr - 20th Apr 2020)
Working on ensemble NAS MAML idea.
Looking at ways to present CVPR_NASWS submission as a complete work for ICML workshop.
Looking at recently published experimental papers to extend the workshop work.
Running more experiments on different datasets to decide the trajectory of the workshop paper.
Held meetings with Gaurav regarding the same.
Week 13 (7th Apr - 13th Apr 2020)
Discussing the ideas for ensemble + NAS work.
Reading various papers related to MAML, Ensemble, searching in data space, auto augmenting data.
We decided to focus on the main goal of increasing the speed of NAS on large scale datasets. We plan to use bagging and MAML is some way to achieve this
Week 12 (31st Mar - 6 Apr 2020)
Reading Efficient Forward Architecture Search.
Main Idea: Start from a small network and add layers based on the requirement.
Allows you to start from what we already (densents, resnets are build with human expertise) have and build on top of it.
Inspired by Cascade-correlation and gradient boosting.
Proposed method Petridish can be used for both cell-search (a.k.a micro-search) and macro-search.
Reading about random forests, bagging, and other ensemble methods.
Ideating and discussing about how to combine NAS and ensemble.
Week 11 (24th Mar - 30 Mar 2020)
Reading Efficient Forward Architecture Search.
Reading about different ensemble methods watching some video lectures.
Cleaning the code and uploading it anonymously for CVPR_NASWS.
Thinking about ideas regarding future work.
Week 10 (17th Mar - 23 Mar 2020)
Running experiments for the CVPR_NASWS.
Completing the CVPR_NASWS paper and submitting it.
Meeting: Discussing about ensemble and NAS.
TODO: Read Efficient Forward Architecture Search paper
Week 9 (10th Mar - 16th Mar 2020)
Running experiments for the CVPR_NASWS.
Writing the 1st end to end submission.
Week 8 (3rd Mar - 9th Mar 2020)
Running Adversarial Experiments on CIFAR-10, Imagenet on Resnet, Densnet, VGG. Inception net, PDARTS, DARTS, NSGA net.
Writing the initial draft for CVPR_NASWS, discussed with sir, there were some changes that need to be made, started working on them.
Reading/Overviewing these papers:
Evolving Robust Neural Architectures to Defendfrom Adversarial Attacks
Understanding and Robustifying Differentiable Architecture Search
XNAS: Neural Architecture Search with Expert Advice
Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly
Searching for A Robust Neural Architecture in Four GPU Hours
Learning Resnet Blocks using Boosting Theory paper
Deep Neural Network Ensembles against Deception: Ensemble Diversity, Accuracy and Robustness
Provably Robust Boosted Decision Stumps and Trees against Adversarial Attacks
Coupled Ensembles of Neural Networks
NAS evaluation is frustratingly hard
When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks.
Improving Neural Architecture Search Image Classifiers via Ensemble Learning
AdaNet: Adaptive Structural Learning of Artificial Neural Networks
Sub-Architecture Ensemble Pruning in Neural Architecture Search
Week 7 (25th Feb - 2nd Mar 2020)
Completed the draft.
Made an outline of the stuff that needs to be done before completing the CVPR_NASWS paper:
List the papers that need to be discussed and cited for the NAS workshop submission.
Experiments:
Common adversarial attacks on 3 three standard architectures (Resnet, Densenet, VGG) and 2-3 NAS architectures (Pick from NAS eval is frustratingly hard paper)
Can we reproduce the results (that’ll make our argument better) of Improving NAS or When NAS meets robustness papers.
Can we simply make an ensemble kind of architecture from existing NAS papers.
Find 3-4 architectures, make ensemble and prune each of them.
Summarized papers for writing CVPR_NASWS:
Provably Robust Boosted Decision Stumps and Trees against Adversarial Attacks.
Deep Neural Network Ensembles against Deception: Ensemble Diversity, Accuracy and RobustnessLing Liu, Wenqi Wei, Ka-Ho Chow, Margaret Loper, Emre Gursoy, Stacey Truex, Yanzhao Wu.
Learning Resnet Blocks using Boosting Theory paper.
Ran some experiments on adversarial robustness and existing methods.
Week 6 (18th Feb - 24th Feb 2020)
Started working on the draft for the paper.
Setting up some initial experiments related to adversarial robustness and running some of the existing NAS techniques(P-DARTS)
Thought about the NAS + ensemble ideas and ideas for submission in the CVPR NAS workshop, ideas:
Ensemble models are generally more robust and accurate than the single architecture based models (cite Improving NAS Paper)
Ensembles will help leverage the power of different architectures (one architecture can have skip connections, others can have dense connections, etc)
All the existing DNN ensembles are handcrafted, using NAS to directly find ensembles might open new doors and might help in improving the performance, the robustness of neural network architectures.
Each of the sub-architectures in an ensemble can be used to learn different batches (or classes) of data. Which may be helpful in building specialized architectures for problems like few-shot, zero-shot, etc.
Show that most of the existing NAS based architectures are less robust (running 1-2 experiments on existing benchmarks) → As a result finding ensembles using NAS is an important problem.
Using ensembles can decrease the search space complexity. For example, As shown in the figure below, we can choose to fix a part of the network to be constant for all the architectures in the ensemble and do NAS to find the last few layers for each architecture.
|
|
|
Week 5 (11th Feb - 17th Feb 2020)
Thinking about ideas relevant to NAS + ensembling.
Reviewing some of the already read papers.
Reading reviews of some of the NAS papers like https:openreview.net/forum?id=HygrdpVKvr
Reading about ensembling techniques and looking at some relevant maths.
Week 4 (4th Feb - 10th Feb 2020)
Readings:
AdaNet: Adaptive Structural Learning of Artificial Neural Networks
Learning Deep ResNet Blocks Sequentially using Boosting Theory
Improving Neural Architecture Search Image Classifiers via Ensemble Learning
Todo:
We were planning to replace boosting used in Improving NAS paper with bagging and other ensembling methods, for this we'll have to first read AdaNet paper thoroughly.
Go through recent DNN+Ensembling methods survey paper.
We have two NAS + Ensembling papers, both of these are not accepted anywhere. We'll go through these thoroughly and try to find out the reasons for not getting accepted.
Write the draft for paper and submit.
Week 3 (28th Jan - 3rd Feb 2020)
Readings:
Improving Neural Architecture Search Image Classifiers via Ensemble Learning
Sub-Architecture Ensemble Pruning in Neural Architecture Search
NAS Evaluation Is Frustratingly Hard
Minutes of meeting with Vineeth Sir:
Ideas discussed:
Training a neural network by training sub-architectures separately
DNN accelerators focus on reducing the energy consumed during inference which is majorly governed by the data movement, designing DNNs that optimize the data movement based on architecture.
NAS for MAML
Treat a Resnet like architecture as Supernet + MAML
Gradient Boosting + Improving NAS paper, (Ensemble method along with NAS in general)
Todo (Next Steps):
Decide on which idea to pursue next.
Read the above mentioned two papers and see if we can replace boosting with Bagging
Read: Learning Resnet Blocks using Boosting Theory paper
Week 2 (21st Jan - 27th Jan 2020)
Readings:
Neural Architecture Search: A Survey
Best Practices for Scientific Research on Neural Architecture Search
AdversarialNAS: Adversarial Neural Architecture Search for GANs
AutoGAN: Neural Architecture Search for Generative Adversarial Networks
Evolving Robust Neural Architectures to Defend from Adversarial Attacks
Adversarial Robustness vs. Model Compression, or Both?
When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks
Adversarial Robustness of Pruned Neural Networks
Presentation:
When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks
Network Pruning via Transformable Architecture Search
Todo (Next Steps):
The problem statement we currently have seems to overlap with many existing papers.
Brainstorm and finalize the problem statement this week
Week 1 (13th Jan - 20th Jan 2020)
Building background on adversarial networks:
Readings: Intriguing Properties of Adversarial Examples
Videos Watched:
Adversarial Examples and Training, CS231n
Adversarial Machine Learning, ICLR 2019 talk
Building background on NAS:
Readings:
DARTS: Differentiable Architecture Search
Progressive Neural Architecture Search
Efficient Neural Architecture Search using Parameter Sharing
Videos Watched:
NAS SOTA review talk, MSR
Efficient Forward Architecture Search
Finding papers relevant to NAS and Adversarial attacks, found six papers.
Papers Read relevant to NAS and Adversarial attacks:
Network pruning via transformable architecture Search
When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks
Some ideas in the brainstorming session:
NAS with a budget constraint (in terms of parameters and layers)
NAS for coming up with adversarially robust architectures
NAS in transfer learning, GANs, Multi-task, and Multi-Objective learning
NAS for developing architectures based on the task at hand (One-shot, Few shot etc)
Todo (Next Steps)
We decided to work on “NAS for coming up with adversarially robust architectures” problem, but we may have to tweak the problem statement a little bit as there are a few new papers on this topic.
Read the remaining papers.
Finalise the initial idea.
|


Before Change
current_tag, next_tag = tags[i], tags[i+1]
// The scores for transitioning from current_tag to next_tag
transition_score = (
broadcast_transitions
// Choose the current_tag-th row for each input
.gather(1, current_tag.view(batch_size, 1, 1).expand(batch_size, 1, num_tags))
// Squeeze down to (batch_size, num_tags)
.squeeze(1)
// Then choose the next_tag-th column for each of those
.gather(1, next_tag.view(batch_size, 1))
// And squeeze down to (batch_size,)
.squeeze(1)
)
// The score for using current_tag
emit_score = logits[i].gather(1, current_tag.view(batch_size, 1)).squeeze(1)
After Change
// Transition from last state to "stop" state. To start with, we need to find the last tag
// for each instance.
last_tag_index = mask.sum(0).long() - 1
last_tags = tags.gather(0, last_tag_index.view(1, batch_size)).squeeze(0)
// Compute score of transitioning to `stop_tag` from each "last tag".
if self.include_start_end_transitions:
last_transition_score = self.end_transitions.index_select(0, last_tags)


DESCRIPTION
The idea of Bezel is to overcome the limitations of using different versions of the same package in the same Ruby process.
USAGE
It works like this. Let's say I wrote a library called TomsLib. Now I want to use TomsLib in my new fancy app, FancyApp. In my FancyApp namespace I have to create a reference to TomsLib.
module FancyApp
TomsLib = lib('tomslib', '1.5')
...
Now I have access to TomsLib, but it is localized to my application. If Jane comes along and wants to use a different version of TomsLib but also utilizes my FancyApp, she could do so:
module JanesProgram
TomsLib = lib('tomslib', '1.0')
FancyApp = lib('fancyapp') # use newest available
...
How does this work? When you call lib(), Bezel looks for the package in the current Gem paths (and in the future, Roll ledger) then it reads the primary package file (eg. tomslib.rb) from the package and evals it into an anonymous module.
This has a some important effects on how you write your Ruby programs:
Any reference to core/standard libraries must be referenced via '::' prefix (eg. ::Enumerable).
Core extensions are not version isolated. So avoid them when possible, or depend on highly stable standardized bases such as Ruby Facets and ActiveSupport.
Since Bezel is a completely different alternative to Ruby's normal load system, your program will require Bezel be installed by your users.
Within Bezel dependent libraries #import must be used instead of #require or #load in order to keep the code within the current module.
Despite the minor limitations and necessary practices required for its use, Bezel is highly advantageous to the developers and end-users alike in that it puts an end to the dreaded Dependency Hell.
COPYRIGHT
Copyright (c) 2009 Thomas Sawyer
Bezel is distributed under the terms of the Ruby License.
Where the UML Cats Drink!


Event listeners¶
Guacamole supports the delivery of event notifications to custom extensions. Developers can use listener extensions to integrate custom handling of events such as successful and failed authentications, and requests to connect and disconnect tunnels to desktop environments.
A listener extension could be used, for example, to record authentication attempts in an external database for security auditing or alerting. By listening to tunnel lifecycle events, a listener extension could be used to help coordinate startup and shutdown of machine resources; particularly useful in cloud environments where minimizing running-but-idle resources is an important cost savings measure.
For certain vetoable events, an event listener can even influence Guacamole’s behavior. For example, a listener can veto a successful authentication, effectively causing the authentication to be considered failed. Similarly, a listener can veto a tunnel connection, effectively preventing the tunnel from being connected to a virtual desktop resource.
Custom event listeners are packaged using the same extension mechanism used for custom authentication providers. A single listener extension can include any number of classes that implement the listener interface. A single extension module can also include any combination of authentication providers and listeners, so developers can easily combine authentication providers with listeners designed to support them.
To demonstrate the principles involved in receiving Guacamole event notifications, we will implement a simple listener extension that logs authentication events. While our approach simply writes event details to the same log used by the Guacamole web application, a listener could process these events in arbitrary ways, limited only by the imagination and ingenuity of the developer.
A Guacamole listener extension skeleton¶
For simplicity’s sake, and because this is how things are done upstream in the Guacamole project, we will use Maven to build our extension.
The bare minimum required for a Guacamole listener extension is a pom.xml
file listing guacamole-ext as a dependency, a single .java
file implementing
our stub of a listener, and a guac-manifest.json
file describing the
extension and pointing to our listener class.
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.apache.guacamole</groupId>
<artifactId>guacamole-listener-tutorial</artifactId>
<packaging>jar</packaging>
<version>1.4.0</version>
<name>guacamole-listener-tutorial</name>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<build>
<plugins>
<!-- Written for Java 8 -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.3</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<!-- Guacamole Extension API -->
<dependency>
<groupId>org.apache.guacamole</groupId>
<artifactId>guacamole-ext</artifactId>
<version>1.4.0</version>
<scope>provided</scope>
</dependency>
<!-- Slf4j API -->
<!-- This is needed only if your listener wants to
write to the Guacamole web application log -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.7</version>
<scope>provided</scope>
</dependency>
</dependencies>
</project>
Naturally, we need the actual listener extension skeleton code. While you can
put this in whatever file and package you want, for the sake of this tutorial,
we will assume you are using org.apache.guacamole.event.TutorialListener
.
For now, we won’t actually do anything other than log the fact that an event notification was received. At this point, we’re just creating the skeleton for our listener extension.
package org.apache.guacamole.event;
import org.apache.guacamole.GuacamoleException;
import org.apache.guacamole.net.event.listener.Listener;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* A Listener implementation intended to demonstrate basic use
* of Guacamole's listener extension API.
*/
public class TutorialListener implements Listener {
private static final Logger logger =
LoggerFactory.getLogger(TutorialListener.class);
@Override
public void handleEvent(Object event) throws GuacamoleException {
logger.info("received Guacamole event notification");
}
}
To conform with Maven, this skeleton file must be placed within
src/main/java/org/apache/guacamole/event
as TutorialListener.java
.
As you can see, implementing a listener is quite simple. There is a single
Listener
interface to implement. All Guacamole event notifications will be
delivered to your code by invoking the handleEvent method. We will see shortly
how to use the passed event object to get the details of the event itself.
The only remaining piece for the overall skeleton to be complete is a
guac-manifest.json
file. This file is absolutely required for all Guacamole
extensions. The guac-manifest.json
format is described in more detail in
guacamole-ext. It provides for quite a few properties, but for our listener
extension we are mainly interested in the Guacamole version sanity check (to
make sure an extension built for the API of Guacamole version X is not
accidentally used against version Y) and telling Guacamole where to find our
listener class.
The Guacamole extension format requires that guac-manifest.json
be placed in
the root directory of the extension .jar
file. To accomplish this with Maven,
we place it within the src/main/resources
directory. Maven will automatically
pick it up during the build and include it within the .jar
.
{
"guacamoleVersion" : "1.4.0",
"name" : "Tutorial Listener Extension",
"namespace" : "guac-listener-tutorial",
"listeners" : [
"org.apache.guacamole.event.TutorialListener"
]
}
Building the extension¶
Once all three of the above files are in place, the extension should build successfully even though it is just a skeleton at this point.
$ mvn package
[INFO] Scanning for projects...
[INFO] ---------------------------------------------------------------
[INFO] Building guacamole-listener-tutorial 1.4.0
[INFO] ---------------------------------------------------------------
...
[INFO] ---------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ---------------------------------------------------------------
[INFO] Total time: 1.297 s
[INFO] Finished at: 2017-10-08T13:12:39-04:00
[INFO] Final Memory: 19M/306M
[INFO] ---------------------------------------------------------------
$
Assuming you see the “BUILD SUCCESS
” message when you build the extension,
there will be a new file, target/guacamole-listener-tutorial-1.4.0.jar
, which
can be installed within Guacamole (see Installing the extension at the
end of this chapter). It should log event notifications that occur during, for
example, authentication attempts. If you changed the name or version of the
project in the pom.xml
file, the name of this new .jar
file will be
different, but it can still be found within target/
.
Handling events¶
The Guacamole Listener
interface represents a low-level event handling API. A
listener is notified of every event generated by Guacamole. The listener must
examine the event type to determine whether the event is of interest, and if so
to dispatch the event to the appropriate entry point.
The event types that can be produced by Guacamole are described in the
org.apache.guacamole.net.event
package of the guacamole-ext API. In this
package you will find several concrete event types as well as interfaces that
describe common characteristics of certain of event types. You can use any of
these types to distinguish the events received by your listener, and to examine
properties of an event of a given type.
Suppose we wish to log authentication success and failure events, while
ignoring all other event types. The AuthenticationSuccessEvent
and
AuthenticationFailureEvent
types are used to notify a listener of
authentication events. We can simply check whether a received event is of one
of these types and, if so, log an appropriate message.
package org.apache.guacamole.event;
import org.apache.guacamole.GuacamoleException;
import org.apache.guacamole.net.event.AuthenticationFailureEvent;
import org.apache.guacamole.net.event.AuthenticationSuccessEvent;
import org.apache.guacamole.net.event.listener.Listener;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* A Listener that logs authentication success and failure events.
*/
public class TutorialListener implements Listener {
private static final Logger logger =
LoggerFactory.getLogger(TutorialListener.class);
@Override
public void handleEvent(Object event) throws GuacamoleException {
if (event instanceof AuthenticationSuccessEvent) {
logger.info("successful authentication for user {}",
((AuthenticationSuccessEvent) event)
.getCredentials().getUsername());
}
else if (event instanceof AuthenticationFailureEvent) {
logger.info("failed authentication for user {}",
((AuthenticationFailureEvent) event)
.getCredentials().getUsername());
}
}
}
In our example, we use instanceof
to check for the two event types of
interest to our listener. Once we have identified an event of interest, we can
safely cast the event type to access properties of the event.
The extension is now complete and can be built as described earlier in Building the extension and installed as described below in Installing the extension.
Influencing Guacamole by event veto¶
An implementation of the handleEvent method is permitted to throw any
GuacamoleException
. For certain vetoable event types, throwing a
GuacamoleException
serves to effectively veto the action that resulted in the
event notification. See the API documentation for guacamole-ext to learn more
about vetoable event types.
As an (admittedly contrived) example, suppose we want to prevent a user named
“guacadmin” from accessing Guacamole. For whatever reason, we don’t wish to
remove or disable the auth database entry for this user. In this case we can
use a listener to block this user, preventing access to Guacamole. In the
listener, when we get an AuthenticationSuccessEvent
we can check to see if
the user is “guacadmin” and, if so, throw an exception to prevent this user
from logging in to Guacamole.
package org.apache.guacamole.event;
import org.apache.guacamole.GuacamoleException;
import org.apache.guacamole.GuacamoleSecurityException;
import org.apache.guacamole.net.event.AuthenticationFailureEvent;
import org.apache.guacamole.net.event.AuthenticationSuccessEvent;
import org.apache.guacamole.net.event.listener.Listener;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* A Listener that logs authentication success and failure events
* and prevents the "guacadmin" user from logging in by throwing
* a GuacamoleSecurityException.
*/
public class TutorialListener implements Listener {
private static final Logger logger =
LoggerFactory.getLogger(TutorialListener.class);
@Override
public void handleEvent(Object event) throws GuacamoleException {
if (event instanceof AuthenticationSuccessEvent) {
final String username = ((AuthenticationSuccessEvent) event)
.getCredentials().getUsername();
if ("guacadmin".equals(username)) {
logger.warn("user {} has been blocked", username);
throw new GuacamoleSecurityException(
"User '" + username + "' is currently blocked");
}
logger.info("successful authentication for user {}", username);
}
else if (event instanceof AuthenticationFailureEvent) {
logger.info("failed authentication for user {}",
((AuthenticationFailureEvent) event)
.getCredentials().getUsername());
}
}
}
If our Guacamole user database contains a user named “guacadmin”, and we build and install this listener extension, we will find that an attempt to log in as this user now results in a message in the UI indicating that the user is blocked. If we examine the Guacamole log, we will see the message indicating that the user was blocked. Because the successful authentication was vetoed, Guacamole sends a subsequent authentication failure notification, which we see logged as well.
Installing the extension¶
Guacamole extensions are self-contained .jar
files which are installed by
being placed within GUACAMOLE_HOME/extensions
, and this extension is no
different. As described in Configuring Guacamole, GUACAMOLE_HOME
is a
placeholder used to refer to the directory that Guacamole uses to locate its
configuration files and extensions. Typically, this will be the .guacamole
directory within the home directory of the user running Tomcat.
To install your extension, copy the
target/guacamole-listener-tutorial-1.4.0.jar
file into
GUACAMOLE_HOME/extensions
and restart Tomcat. Guacamole will automatically
load your extension, logging an informative message that it has done so:
Extension "Tutorial Listener Extension" loaded.


4
1 Then Eliphaz replied to Job. He said,
2 "Will you please allow me to say something to you?
I am not able to remain silent any longer.
3 In the past, you have instructed many people,
and you have encouraged those for whom it was difficult to trust in God.
4 In the past, when you spoke to others who were suffering, you helped them;
they were able to rejoice again because of God.
5 But now, when you yourself suffer from disasters, you become discouraged.

God never kills innocent people!
8 I have seen this happen: if farmers plant bad seeds, they do not harvest good crops;
anyone who starts trouble for others later brings trouble on himself.
9 These people die when God commands them to,
because he is very angry with them.
10 Even though wicked people may be very powerful like young lions,
God destroys them.
11 They will die like older lions that starve to death when there are no animals left to eat.
Their children will scatter from each other like young lions that separate from each other to find food.
12 I heard a message that someone came
and whispered to me.
13 He spoke to me at night when I was having a bad dream that disturbed me.
14 It caused me to be afraid and tremble;
it caused all my bones to shake.
15 A ghost glided past my face
and caused the hair on the back of my neck to stand straight up.
16 It stopped, but I could not see what it really looked like.
But I knew that there was some being in front of me,
and it said in a quiet voice,
17 'No human being can be more righteous than God.
No man can be better than God, who made him.
18 God cannot be sure that his own angels will always do what is right;
he declares that some of them have done wrong.
19 So he certainly cannot trust human beings whom he has made from dust and clay,
human beings, whom disaster crushes as easily as you can crush a moth!
20 People are sometimes in good health in the morning, but by the evening they are dead.
They are gone forever and no one even pays attention.
21 They are like tents that collapse suddenly when you pull up their stakes;
they die suddenly without ever knowing why.'"


--- layout: default title: "About me" description: "" ---
I am a Ph.D. student in the Human-Computer Interaction Institute at Carnegie Mellon University, advised by Jason Hong. My research is about building systems at the intersection of Big Data, Privacy & Security and Human-Computer Interaction.


An Auth_OpenID_Mapping maintains a mapping from arbitrary keys to arbitrary values. (This is unlike an ordinary PHP array, whose keys may be only simple scalars.)
Located in /Auth/OpenID/Message.php (line 125)
Initialize a mapping. If $classic_array is specified, its keys and values are used to populate the mapping.
Returns true if the specified value has a key in the mapping; false if not.
Deletes a key-value pair from the mapping with the specified key.
Gets a specified value from the mapping, associated with the specified key. If the key does not exist in the mapping, $default is returned instead.
Returns true if $thing is an Auth_OpenID_Mapping object; false if not.
Returns an array of (key, value) pairs in the mapping.
Returns an array of the keys in the mapping.
Returns the "length" of the mapping, or the number of keys.
Sets a key-value pair in the mapping. If the key already exists, its value is replaced with the new value.
Returns an array of values in the mapping.
Documentation generated on Tue, 21 Apr 2009 11:31:31 -0700 by phpDocumentor 1.3.2


Vidhi Joshi
Home
(current)
Timeline
(current)
Infographic
(current)
Click on the arrows below to learn more!
Previous
Next


{% extends './base.html' %} {% block content %}
Apart from being a critical driver of economic growth, foreign direct investment (FDI) is a major source of non-debt financial resource for the economic development of India. Foreign companies invest in India to take advantage of relatively lower wages, special investment privileges such as tax exemptions, etc. For a country where foreign investments are being made, it also means achieving technical know-how and generating employment. The Indian government’s favourable policy regime and robust business environment have ensured that foreign capital keeps flowing into the country. The government has taken many initiatives in recent years such as relaxing FDI norms across sectors such as defence, PSU oil refineries, telecom, power exchanges, and stock exchanges, among others.
Market sizeAccording to Department of Industrial Policy and Promotion (DIPP), the total FDI investments India received during April - September 2016 rose 30 per cent year-on-year to US$ 21.6 billion, indicating that government's effort to improve ease of doing business and relaxation in FDI norms is yielding results. Data for April - September 2016 indicates that the services sector attracted the highest FDI equity inflow of US$ 5.29 billion, followed by telecommunications – US$ 2.79 billion, and trading – US$ 1.48 billion. Most recently, the total FDI equity inflows for the month of September 2016 touched US$ 5.15 billion. During April - September 2016, India received the maximum FDI equity inflows from Mauritius (US$ 5.85 billion), followed by Singapore (US$ 4.68 billion), Japan (US$ 2.79 billion), (US$ 1.62 billion), and USA (US$ 1.44 billion). Impact investments in India is expected to grow at a compound annual growth rate (CAGR) of 20-24 per cent to touch US$ 6-8 billion by 2025, from US$ 1 billion in 2015.1