This document summarizes the key points in a paper submitted to SIGIR 2017. http://sigir-ecom.weebly.com/uploads/1/0/2/9/102947274/paper_8.pdf
Goal is to design a system that reformulates query into multiple alternative queries that enlarge recall without altering main search intent.
The approach adopted by the paper can be broken down into two parts:
For verbose queries, perform POS tagging, entities, unit of measurements.
For non-verbose queries, reformulate queries by replacing some tokens. Avoids turning specific query into broad query. Done using language model approach.
Selection of terms drop is dropped by predetermined rules. There are three categories. Tier 1 includes brands, product names, first noun phrases. Tier 2 are attributes of the core product. Tier 3 gropup are stop words, conjunctions.
Subproblem is the disambiguation of brands.
P(B|q) = \sum P(C_i|q) P(B|C_i)
Predicts probability that B is brand in context of query q.
C_i is the shopping category.
Form variations using multiple passes, which is similar to the system I currently implemented that adopts a pseudo- BFS approach.
Suggestions from earlier pass will always rank higher than the ones from the later passes, as overall queries with more original tokens are believed to be closer to the original user content.
The paper’s algorithm also prefers suggestions with more brands or nouns over suggestions with more adjectives.
Baseline tagging scoe is given by a step function over their tags. 54% of tokens are nouns. There is hence a need to rank the nouns.
Ranking nouns can be done by using the cross-entropy of clicking probabilities conditioning on their document frequencies.
P_click® = -P(click|r) log(1 - P(click|r))
r is the document frequency of the token. P(click|r) is evaluated by a large sample of null queries. P(click | r) is evaluated by a large number of null queries.
Clicking score added to tagging score using discounted cumulative gain.
Consider all reassignments of tokens of the n-gram model. To calculate the probability that one token is replaced by another token,


This is a webpage for my scripts and tutorials
Ramdisk tutorial
: A tutorial on creating a ramdisk for temporary storage on Windows


--- title: Tunneling and Onion routing to access proxy / firewall restricted internet sites tags: meta: Access proxy / firewall restricted internet sites abstract: In this post I'll explain a technique to access firewall or proxy restricted internet websites via tunneling and onion routing. This article is intended to-be-informative for network administrators and internet users. This article is just an overview and does not cover technical details thumbnail: date: 2005-11-04 ---
This article is intended for network administrators and internet users. It shows how users can bypass the firewall to get access to the restricted internet sites. This article is just an overview and does not cover technical details
Background
Most of us have been using emails since long. At least the software engineers do. Accessing your emails is not a big deal, until you are restricted. Well this does happen in many corporates. Once you are restricted to do so, what would you do? You may say: “I’ll access that from the cyber-cafe/home later in the evening”
Corporates or institutions imply these rules to increase productivity and/or maintain confidentiality.
Practically speaking, for a developer, the network or system administrators cannot deny access to all the sites. Most of us use Google, and require help from many linked sites. But a network admin can definitely identify and restrict mail sites like “mail.yahoo.com” or “gmail.com”.
The problem
How do they do these? All the sites are accessed by you through a proxy/firewall server like the Microsoft ISA server. The HTTP request strings are identified and based on these strings; the sites are blocked (and might be logged).
The following is a screenshot from a famous network-protocol analyzer (Ethereal)
There is temporary work-around to the situation like having a Remote-Desktop connection to a PC where the internet is accessing, etc. But these don’t work for long.
What if we could have a proxy over the net that will take our requests and fetch the response for us? Say “proxy.net” is one such server. We can configure this server address and ports in our browser settings and send a request to them. But will that solve our problem? No. That proxy setting will not change the HTTP request string.
The solution
We must have a way that the request is encrypted before it passes through the company’s firewall. Similarly, to have more privacy on what we are actually seeing, we also want to have the response sent to us in an encrypted form. At our end, we will encrypt the request and decrypt the response. Similarly, at the remote proxy server end, it will decrypt the request and encrypt the request. This concept is called tunneling and very well used in VPN (Virtual Private Networks).
So there is a possibility that some of you create a VPN connection to the remote network to access the internet through a completely different network (That network must have internet access). But this is not a feasible solution for everybody.
There are some web-application-proxy services that allow you to do this. “MegaProxy.com”, “Guardster.com”, etc. But your network admins can block these sites too (once they get an idea about that). You need to pay to access email through these sites.
Implementation
Tunneling is a great idea, and I started a search on how this could be put to real implementation. Initially, I started searching on how such an application can be made, but later I found many such packages that implement this.
The best and famous of those (not the most efficient) was “Tor”. It is an anonymous internet communication system that is based on onion-routing.
Tor is itself a network of virtual tunnels. Tor creates a circuit for accessing the internet. The good thing about it is that it will create new circuits. This keeps you anonymous over the internet all the time. More details about Tor can be found at: http://tor.eff.org/overview.html.en
Along with Tor, I also found JAP (Java Anon Proxy). Details about JAP can be found at: http://anon.inf.tu-dresden.de/index_en.html
I find Tor as technically superior to JAP in terms of keeping anonymity. Tor is quite difficult to install, and takes a lot of time to get a good circuit (many times I received a timeout). Comparatively JAP is very easy to use. Tor recommends the use of Privoxy to avoid DNS leaks. Well, this is important in case of true anonymous surfing, but it does not fit scenario I required. Tor and JAP are free to use. JAP may charge some fee once it gets out of research mode.
If we cannot break the rules, we can definitely bend the rules to our satisfaction. Technology has its own good and bad use. It’s up to the users on whether they use it ethically.
I am not any networking expert. Please do not hesitate to contact me to if I am incorrect at any point
Happy hacking!