Thursday, July 10, 2008

Yahoo's new Search BOSS API

Yahoo has just released a new web search API called BOSS (Build your Own Search Service) which improves on their earlier API in several ways:
  1. No daily query limits.

  2. No restrictions on how the results are displayed, ordered, or mixed in with other proprietary results.

  3. Ability to make money showing paid results.

The BOOS API is REST-based. You can receive results in either JSON or XML format, and you can get 10-50 results back per query.

There is one item that appears to be missing without explanation: the cached URL of each search result. This URL is useful to the user when the result's live URL is not responding. The old Yahoo web search API did provide this, so I'm not sure why it dropping in Boss.

One thing that makes me a little nervous about the API from a researcher's perspective is the prohibition in their Terms of Service against analyzing their search results:
You will not, will not attempt, or will not permit or take actions designed to enable other third parties to: ... perform any analysis, reverse engineering or processing of the Web Search Results
Analyzing the Yahoo search results is exactly what I did in my paper Agreeing to Disagree: Search Engines and their Public Interfaces. Well, better to do and ask forgiveness than get permission up front. ;-)

So here's a simple example in Java using the new BOSS API to search for the title of my blog "questio verum", the index status of my blog's root page, and all the pages indexed for my blog. To make this example work for you, simply put your Yahoo API key in API_KEY.

Note that this example is very similar to the Google AJAX example in Java from last month.

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLEncoder;
import org.json.JSONArray; // JSON library from http://www.json.org/java/
import org.json.JSONObject;

public class YahooQuery {

// Yahoo API key
private final String API_KEY = "Your Key Here";


public YahooQuery() {

makeQuery("questio verum");
makeQuery("url:http://frankmccown.blogspot.com/");
makeQuery("site:frankmccown.blogspot.com");
}

private void makeQuery(String query) {

System.out.println("\nQuerying for " + query);

try
{
// Convert spaces to +, etc. to make a valid URL
query = URLEncoder.encode(query, "UTF-8");

// Give me back 10 results in JSON format
URL url = new URL("http://boss.yahooapis.com/ysearch/web/v1/" + query +
"?appid=" + API_KEY + "&count=10&format=json");
URLConnection connection = url.openConnection();

String line;
StringBuilder builder = new StringBuilder();
BufferedReader reader = new BufferedReader(
new InputStreamReader(connection.getInputStream()));
while((line = reader.readLine()) != null) {
builder.append(line);
}

String response = builder.toString();

JSONObject json = new JSONObject(response);

System.out.println("\nResults:");
System.out.println("Total results = " +
json.getJSONObject("ysearchresponse")
.getString("deephits"));


System.out.println();

JSONArray ja = json.getJSONObject("ysearchresponse")
.getJSONArray("resultset_web");

System.out.println("\nResults:");
for (int i = 0; i < ja.length(); i++) {
System.out.print((i+1) + ". ");
JSONObject j = ja.getJSONObject(i);
System.out.println(j.getString("title"));
System.out.println(j.getString("url"));
}

}
catch (Exception e) {
System.err.println("Something went wrong...");
e.printStackTrace();
}
}

public static void main(String args[]) {
new YahooQuery();
}
}


Running this program produces the following results:


Querying for questio verum

Total results = 13600

Results:
1. Questio Verum
http://frankmccown.blogspot.com/
2. WikiAnswers - What does questio verum mean
http://wiki.answers.com/Q/What_does_questio_verum_mean
3. Questio Verum: URL Canonicalization
http://frankmccown.blogspot.com/2006/04/url-canonicalization.html
4. Questio Verum: WIDM 2006
http://frankmccown.blogspot.com/2006/11/widm.html
5. Questio Verum: Fav5
http://frankmccown.blogspot.com/2007/09/fav5_29.html
6. Questio Verum: Fav5
http://frankmccown.blogspot.com/2007/12/fav5.html
7. Questio Verum: August 2006
http://frankmccown.blogspot.com/2006_08_01_archive.html
8. Amazon.com: Profile for Questio Verum
http://www.amazon.com/gp/pdp/profile/A2Q6CLLQPXG55A
9. Questio Verum: JCDL 2007 - day 2
http://frankmccown.blogspot.com/2007/06/jcdl-2007-day-2.html
10. Questio Verum: OA debate - Eysenbach and Harnad
http://frankmccown.blogspot.com/2006/05/oa-debate-eysenbach-and-harnad.html


Querying for url:http://frankmccown.blogspot.com/

Total results = 1

Results:
1. Questio Verum
http://frankmccown.blogspot.com/


Querying for site:frankmccown.blogspot.com

Total results = 4080

Results:
1. Questio Verum
http://frankmccown.blogspot.com/
2. Questio Verum: OA debate - Eysenbach and Harnad
http://frankmccown.blogspot.com/2006/05/oa-debate-eysenbach-and-harnad.html
3. Questio Verum: JCDL 2007 - day 2
http://frankmccown.blogspot.com/2007/06/jcdl-2007-day-2.html
4. Questio Verum: No singles here
http://frankmccown.blogspot.com/2007/08/no-single-here.html
5. Questio Verum: Pledge Week and Insults
http://frankmccown.blogspot.com/2007/10/pledge-week-and-insults.html
6. Questio Verum: WIDM 2006
http://frankmccown.blogspot.com/2006/11/widm.html
7. Questio Verum: Fav5
http://frankmccown.blogspot.com/2007/09/fav5_29.html
8. Questio Verum: August 2006
http://frankmccown.blogspot.com/2006_08_01_archive.html
9. Questio Verum: Fav5
http://frankmccown.blogspot.com/2007/06/fav5.html
10. Questio Verum: Fav5
http://frankmccown.blogspot.com/2007/12/fav5.html


Thanks, Martin, for the head's up on this.

Update on 7/28/2008:

The missing cached URL feature is apparently coming soon.

8 comments:

  1. We recently released a java wrapper for yahoo boss to make using the service a little easier.

    It's open source and hosted on google code, link below.

    javaboss

    ReplyDelete
  2. i wanna use yahoo boss over a proxy network. my system uses proxy to connect to internet. my systems ip address is 172.16.35.38. is there any work around. rep me as soon as possible. thanks in advance.

    ReplyDelete
  3. please help me as soon as possible...

    ReplyDelete
  4. Sorry, but I can't help you with the proxy connection. You might want to search the web for an answer or post your question to the appropriate web forum.

    ReplyDelete
  5. @NavNit
    Heres the code:
    void init()
    {
    Authenticator.setDefault(new Authenticator() {
    protected PasswordAuthentication getPasswordAuthentication() {
    return new PasswordAuthentication ("user", "pwd".toCharArray());
    }
    });

    Properties systemProperties = System.getProperties();
    systemProperties.setProperty("http.proxyHost","xyz");
    systemProperties.setProperty("http.proxyPort","80");

    }

    call the init() before you make the call for URL

    ReplyDelete
  6. Hi,

    Firstly, thanks for the code, I am also doing a project on the yahoo api and this is very helpful, i seem to however be having issues with the code, it compiles with no errors however nothing gets printed!
    is there a reason this could be happening?
    Hope you can help.

    Thanks

    ReplyDelete
  7. I haven't used this API in several years. You might want to verify that your key is correct and try looking at the raw http response.

    ReplyDelete
  8. Is it possible to use the above to get more than 50 results per query?

    ReplyDelete