BOSS Hacks - an unofficial Yahoo! BOSS blog » BOSS Functionality

Searchmonkey data is now much easier in BOSS

Ted — Wed, 03 Feb 2010 17:03:40 +0000

BOSS’s integration with SearchMonkey provides great structural data. However, it’s a bit tricky to get results filled with the appropriate data. This blog post from the search team introduces the new SearchMonkey query filters: Accessing SearchMonkey Structured Objects via BOSS.

The SearchMonkey team has been encouraging developers to use our structured data to build semantic Web applications ever since we partnered with BOSS. Using the BOSS API, you can access SearchMonkey structured objects.

To restrict the result set to pages with SearchMonkey objects, just add “searchmonkey:” to your query. The result set from BOSS will only contain URLs that have objects of that type.
Accessing SearchMonkey Structured Objects via BOSS

Here’s a list of SearchMonkey filters:

searchmonkey:video – restricts the result set to videos.
searchmonkey:product – restricts the result set to products.
searchmonkey:local – restricts the result set to local businesses.
searchmonkey:event – restricts the result set to events.
searchmonkey:document – restricts the result set to presentations, spreadsheets, and similar document formats.
searchmonkey:discussion – restricts the result set to blogs and forums.
searchmonkey:game – restricts the result set to Flash games.

Here’s a sample search request for iPhone products:
http://boss.yahooapis.com/ysearch/web/v1/iphone+searchmonkey:product?appid=insert-your-appid&format=xml&start=0&count=15&view=keyterms,searchmonkey_rdf

Watch that Request Length

Ted — Tue, 15 Dec 2009 00:37:12 +0000

Yahoo! BOSS‘s sites param gives us great flexibility in creating vertical search engines. However, we are limited by the number of characters. Here are some tips in keeping the length as short as possible.

Forget passing subdirectories: foo.com/bar is considered the same as foo.com. BOSS will differentiate between subdomains bar.foo.com is not the same as foo.com. In a real-world example passing finance.yahoo.com/news will be interpreted as finance.yahoo.com. But finance.yahoo.com will give a different result than sports.yahoo.com.
Remove www from the url. This is just wasting space. There may be an exception when the site was not setup to work without the www subdomain. I doubt this would make an impact.

Yahoo BOSS to Push Around Bing (arnoldit.com)
OneRiot Confirms They’re Building Yahoo’s Real Time Search Engine (techcrunch.com)
Yahoo BOSS Might Be Bigger Than Bing (techcrunch.com)
Accessing SearchMonkey Structured Objects via BOSS (ysearchblog.com)

Dynamic “sites” Creation for Vertical Search

Ted — Mon, 15 Jun 2009 08:56:35 +0000

I recently did a presentation in London with Skills Matter about Yahoo! BOSS. The small group was filled with ideas about extending BOSS functionality. I wrote a new post for the Yahoo Developer Network that expands on some of these concepts: Make BOSS More Dynamic.

The post discusses the idea of generating the “sites” argument, which tells BOSS to limit the results to a specified list of web sites, dynamically for each query. This allows each query to determine what sites are experts and then create a result set based on those experts.

I have built a prototype and will release it this week after I have a time to clean up some of the loose ends.

Yahoo adds Delicious data to search partnership (news.cnet.com)
Yahoo BOSS Adds Support for Delicious (programmableweb.com)
Yahoo to Enable Custom Semantic Search Engines (readwriteweb.com)
Yahoo! BOSS and trainspotting (slideshare.net)
SearchMonkey and BOSS Take on ETech (ysearchblog.com)

PopGist adds discussions to search

Ted — Tue, 19 May 2009 12:36:57 +0000

AltSearchEngines has a feature on the new search engine PopGist.

PopGist improves the relevancy of the existing search engines and provides better user search experience. The prototype in popgist.com demonstrates the power of PopGist technology by utilizing the search feed provided by Yahoo BOSS. Comparing with the original Yahoo search results, the overlap in the top ten results between Google and Yahoo significantly increases by using PopGist search ranking algorithm for most of the queries tested.
PopGist improves the relevancy of major search engines – AltSearchEngines

PopGist uses BOSS, but reranks the results and adds threads of discussions to give you a more rounded result page.

Yahoo! Search Turns 5, Has Tech to Show For It (readwriteweb.com)
Are Google’s Days Numbered? (wassupblog.com)
Mse360.com – Another Search Engine (killerstartups.com)
Yahoo BOSS’ Growth (khaitan.org)
The Geek’s Search Engine? (blogs.wsj.com)

www2009 Madrid conference notes

Ted — Wed, 13 May 2009 12:13:00 +0000

I kept notes on various presentations during the www2009 conference. Unfortunately, the are rather sparce, but I wanted to do something with them before I completely forgot the context. So here are the rough notes. Think of it as liveblogging but not really live.

Socializing Big Data – Jeff Hammerbacher 4/22/09 12:20 PM

Distribution of social data
Hadoop was chosen for Facebook
Laboratory Life book influenced decision
- Use “thick descriptions” and focus on “microconcerns”
- Stud scientists doing science
Open and closed source development books
The Mythical Man-Month
The Cathedral & the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary
Moving from the age of data to the age of learning.

Hadoop is open to everyone and will play a central part in future development

Hadoop was the name of the developer’s son’s stuffed elephant toy

Written mostly in Java, inspired by Google infrastructure

plain architecture, no major hardware.
Resources: reddit, hacker news
- Banduras’ “Social learning theory”
Data sharing sites
- Data.gov, many eyes, swivel, theinfo.org, infochimps, icharts
Hadoop is for offline processing
Amazon provides Hadoop connectivity

WebNC: efficient sharing of web apps 4/22/09 12:20 PM

WebNC – presented by Andreas Girgensohn of FX Palo Alto Labaratory

share a web browser window, conference calls. Like adobe connect
Requires Firefox for the presenter
HTML, JavaScript works on iPhone, non-flash

How it works

presenter computer has a Firefox extension that grabs the screen 3x a second and breaks it into a grid of images
This is passed to the server that stores the grid and any changes.
The viewers extension rebuilds the grid to display current view and any changes.
There are some c++ logic to display native controls, such as form elements.

The Slashdot Zoo: Mining a Social Network with Negative Edges 4/22/09 12:20 PM

Jérôme Kunegis, Andreas Lommatzsch and Christian Bauckhage

social network with negative edges
multiplication rule applied to such networks
analysis at global, node, edge level
Slashdot allows you to assign a person as your friend, freak, or neutral
the enemy of my enemy is my friend
use this logic to predict trolls and popularity of freaks and friends.
Used PageRank and modifiedPageRank to predict the strength of a user

Combining multi-level audio descriptors 4/22/09 17:20 PM

Jun Wang, Xavier Amatriain and David Garcia Garzon. Chinese Academy of Sciences, Telefonica Research, David Garcia Garzon

the clam framework
started in October 2000
framework for audio and music.
Object oriented and has been documented through a Pattern Language and DSL
Can be used for rapid prototyping and real time applications

Yahoo! Pipes like interface to analyze and modify music.

Audio visualizations display chords and their frequency
Semantic web crawler is searching the web for information about a song
- The extractor is using audio fingerprinting and metadata to identify songs on
- MusicBrainz (MBID)
- It then outputs rdf statements linking local files with remote web identifiers
- It extracts high level descriptors such as editorial metadata, user comments, reviews, tags…
An xml is generated with all of the information generated by the service
Potential collaboration with annotating videos/television at realtime

Microdonations for Stopping Spam 4/22/09 12:20 PM

Sharad Goel, Jake Hofman, John Langford, David Pennock and Daniel Reeves, Yahoo! Inc.

Domain filtering has been the most straightforward approach to combating spam. Reputable sources can also be whitelisted.
- Secure hash also insures reputable messages are not altered along the way
Content filtering looks at patterns and text within the message that signals spam.
- Spammers are constantly trying to get around the filters.
Economic approaches: charge user in some way for sending a message. Charge via memory cycles, captcha, or monetary payments
- What is the amount to pay? It has to be expensive for spammers but cheap enough for users
- Computational costs are not a problem with spammers using zombie botnets
- Human cycles, via captchas, are annoying and would cause people to go elsewhere
- Early adopters are penalized for doing this as the reduction will not be visible until majority of people use it.
CentMail: users get stamps for sending messages.
- Stamps promoted via signatures to let people know about the donations
Even limited adoption would help reduce the spam proliferation. Spam filters will recognize these stamps as whitelist and will not end up in the junk filters.
Certify service defines the certification for the message that it is valid
Verify service recognizes the certificate.

Ranking for Search 4/22/09 2:20 PM

Editorial Judgements as truth

Pros

Control full process
Can calibrate judges to a consensus

Cons

Ownership of query

Position Bias

Position bias throws of clicks as a source of truth, what does no click mean?
An item that isn’t displayed cannot be clicked

Social Labeling

Social labeling game. Give people a boring task and make it fun to collect data.
Playing a game removes the no click problem. The users can also mark an item as bad.
Non-relevant content is not going to harm the brand as it is part of the game.
Preference models

Frequency

Pairwise probability model
Models pairwise interactions but not comparison set
Go Model

Compare vs. Editorial Judgments

How does the pair-wise model compare against editorial judgments
Use set of already collected editorial judgments for 60 of 427 queries.
The game data can be better than the judged values.

Relative judgments can be made more consistently than absolute judgments

Rated Aspect Summarization of Short Comments 4/22/09 12:20 PM

Yue Lu, ChengXiang Zhai and Neel Sundaresan

Opinons across the web typically have an overall rating

On Ebay there is a feedback number, short phrases, and rating of positive, negative, or neutral.

Ebay presents the percentage of feedback as positive or negative
Distributions can also be seen. Ratings over a time span is important.
How can you tell why they are good, i.e. shipping, quality, service…

Ebay wants to break this rating down into the various components. This would be generated from the large number of comments

Challenges

How to identify coherent aspects?
How to accurately rate each aspect
How to get meaningful phrases supporting the ratings?

Ebay overall approach

Aspect and discovery (fast + shipping) = positive. Head term clustering.
Combine people that say fast + shipping
Unstructured PLSA: assign high probability to modifiers i.e. fast and shipping
Structured PLSA: create configuration file for each head term and the strengths associated when combined with a modifier
- A modifier can be associated with multiple head terms, i.e. fast (shipping, delivery, communications)

Mining the Web 2.0 for Better Search 4/22/09 12:20 PM

Dr. Ricardo Baeza-Yates, Yahoo! Research, Barcelona, Spain

amount of content on the web (2007)

8-10gb user generated content/day
3-4 gb published content
private text content 3 tb

Examples

Explicit

metadata
rdf
Wikipedia = UGC
Open Data Project = UGC
Yahoo Answers = UGC
Flickr = UGC

Implicit

text = UGC
anchors and links
queries and clicks – private

Crowdsourcing – wisdom of crowds

importance of diversity, independence and decentralization – aggregating data
popularity
diversity – long tail
quality
coverage

Searchpad

Detects a trend in search queries and produces an online page for later reference

Build a semantic Search Engine with BOSS

Ted — Sun, 03 May 2009 17:56:34 +0000

What’s the quickest way to build a semantic search engine? Use BOSS and and the Yahoo! Searchmonkey query filter. You’ll be surprised how much information is available to begin working with.

The Search Monkey Query Filter

This pattern works in the basic Yahoo! search box: searchmonkeyid:com.yahoo.page.uf.{microformat}. It tells Yahoo! Search to only return web pages that include a certain microformat. We can also use this in our BOSS API requests. Here’s a list of structured data surfaced in BOSS via SearchMonkey:

hAtom -
Represents a subset of the Atom syndication format

hCalendar -
Represents calendar dates and events, using a representation of
the iCalendar
standard

hCard -
Represents people, companies, organizations, and places, using a
representation of the vCard standard

hReview -
Represents reviews of products, services, businesses, and
events

XFN – Represents
human relationships using hyperlinks

Geo -
Represents geograhic coordinates

rel-tag – Marks
up the destination of a hyperlink as an author-designated
tag

adr -
Represents address information

Dublin
Core
- Allows developers to specify document metadata

FOAF
- Friend of a Friend specifies personal profiles and social
networks

SIOC
- Specifies elements in blogs, forums and Q&A sites

Other
supported vocabularies – See the SearchMonkey
documentation for a list of other supported vocabularies

Universal BOSS API Query Operators: Apply to Web, Images and News

SearchMonkey in Your BOSS

Now we have the ability to limit our search results to sites that use a particular microformat. We can now ask for this structured data to build our robust search engine We’ll use the SearchMonkey view attributes. There are two variations that define the format of the data.

view=searchmonkey_feed (dataRSS format)
view=searchmonkey_rdf (rdf format)

The following example shows how each look.

Let’s build a semantic music search engine. We’ll start by filtering for hreview microformats to get music reviews about CocoRosie: http://boss.yahooapis.com/ysearch/web/v1/cocorosie+searchmonkeyid:com.yahoo.page.uf.hreview?appid=Your_App_Id&format=xml&start=0&count=15&view=keyterms%2Csearchmonkey_feed%2Csearchmonkey_rdf.

We can use the following sample result to begin inserting reviews as well as other structured data. You’ve got the basic foundation for building an application that presents data-rich information to your users. Explore your favorite rdf and micformat queries to come up with new ideas.

Please leave a comment if you are already using this technique in your site or have similar ideas to share with others.

Sample Result




CocoRosie... love song tekno cocorosie.
Other Info. Lifetime rating of 4.83 ...]]>

http://lrd.yahooapis.com/.../**http%3A//www.blogskins.com/info/150260
2009/04/22
blogskins.com/info/150260]]>


	skin
	Tekno
	Love Song
	download
	BlogSkins
	Skin Info
	fade
	Skin type
	bg
	follow this link
	cocorosie
	Blogger
	tear
	keep well
	Lifetime
	logged-in
	BM
	Random Skin
	Myles
	hmm




	
		
		
			screaming!tragic
			
		
		
		
		
			animate_
			
		
		
		
		
			madeline_tn
			
		
		
	
	
		
		
			Tekno Love Song
			either continue th patern or don't fade the
			background, and all will look better. but it's still lovely!
			20080421
			
			
			screaming!tragic
			
			
			
			
		
		
		
		
			Tekno Love Song
			you need stars! I think the bg shouldn't fade off.
			Other than that, it's fabulous
			20070903
			
			
				animate_
				
			
			
			
		
		
		
		
			Tekno Love Song
			hmm.. the bg shouldn't be white. should continue
			the pattern thingy... plus, the picture somehow doesn't appeal to me..
			20070713
			
				
					madeline_tn
					
				
			
			
		
		
	
	
		love
		song
		tekno
		cocorosie
		
			
				love
			
		
		
			
				song
			
		
		
			
				tekno
			
		
		
			
				cocorosie
			
		
	




	
		
	
	
		
	
	
		
	
	
		screaming!tragic
	
	
		
	
	
		
	
	
		
	
	
		
	
	
		animate_
	
	
		
	
	
		
	
	
		
	
	
		
	
	
		madeline_tn
	
	
		
	
	
		
	
	
		
	
	
		
	
	
		Tekno Love Song
	
	
		either continue th patern or
		don't fade the background, and all will look better. but it's still lovely!
		
	
	
		20080421
	
	
		
	
	
		
	
	
		screaming!tragic
	
	
		
	
	
		
	
	
		
	
	
		
	
	
		
	
	
		Tekno Love Song
	
	
		you need stars! I think
		the bg shouldn't fade off. Other than that, it's fabulous
	
	
		20070903
	
	
		
	
	
		
	
	
		animate_
	
	
		
	
	
		
	
	
		
	
	
		
	
	
		
	
	
		Tekno Love Song
	
	
	hmm.. the bg shouldn't
	be white. should continue the pattern thingy... plus, the picture somehow
	doesn't appeal to me..
	
	
		20070713
	
	
		
	
	
		
	
	
		madeline_tn
	
	
		
	
	
		
	
	
		
	
	
		
	
	
		love
	
	
		
	
	
		
	
	
		song
	
	
		
	
	
		
	
	
		tekno
	
	
		
	
	
		
	
	
		cocorosie
	


18514
BlogSkins.com Skin Info - Tekno Love Song
http://www.blogskins.com/info/150260

Troubleshooting: Watch for extra commas

Ted — Tue, 28 Apr 2009 09:51:28 +0000

I’m working on a new art search engine that features a large list of museums in the sites attribute. The url length is pushing the limits of the http char count, so I assumed my problems of spam sites infiltrating results was based on the string length.

A recent discussion on the BOSS users mailing list led me to discuss this with one of the BOSS engineers today and we discovered a more basic problem with my requests. I was sending extra commas in the string. Instead of this: moma.org,louvre.fr I had moma.org,,louvre.fr.

Sometimes you need to go back to basics to find your problems. Thanks to Mithun for helping me out.

I don’t know if this is the cause of Michael’s problems in the original mailing list request, but it is a good thing to remember.

BOSS Hacks - an unofficial Yahoo! BOSS blog » BOSS Functionality

Searchmonkey data is now much easier in BOSS

Related articles by Zemanta

Watch that Request Length

Dynamic “sites” Creation for Vertical Search

PopGist adds discussions to search

www2009 Madrid conference notes

Socializing Big Data – Jeff Hammerbacher 4/22/09 12:20 PM

WebNC: efficient sharing of web apps 4/22/09 12:20 PM

How it works

The Slashdot Zoo: Mining a Social Network with Negative Edges 4/22/09 12:20 PM

Combining multi-level audio descriptors 4/22/09 17:20 PM

Yahoo! Pipes like interface to analyze and modify music.

Microdonations for Stopping Spam 4/22/09 12:20 PM

Ranking for Search 4/22/09 2:20 PM

Position Bias

Social Labeling

Frequency

Compare vs. Editorial Judgments

Rated Aspect Summarization of Short Comments 4/22/09 12:20 PM

Challenges

Ebay overall approach

Mining the Web 2.0 for Better Search 4/22/09 12:20 PM

Examples

Crowdsourcing – wisdom of crowds

Searchpad

Related articles by Zemanta

Build a semantic Search Engine with BOSS

The Search Monkey Query Filter

SearchMonkey in Your BOSS

Sample Result

Related articles by Zemanta

Troubleshooting: Watch for extra commas