RECOMMENDATION SYSTEMS
Background
Often times we need to make decisions without personal experience of the different alternatives available. Traditionally these situations were resolved by relying on recommendations by people through word of mouth, or reviews printed in newspapers and magazines etc [1]. There has been a gradual shift from this situation to a more data digitized world.
In recent times with an explosion in the volume and variety of information available both on the web and through the introduction of new e-business services, users are getting overwhelmed. The presence of so many different choices is proving to be detrimental as opposed to productive.
It is evident that there is a need for having a system in place that can provide users with personalized recommendations in their field of interest. This is where recommendation engines come into the picture.
What is a Recommendation Engine?
Recommendation Systems are tools and techniques which are used to provide users with suggestions for items that might be of use to them [2]. Recommendation systems seeks to predict the rating a user would give to an item, in turn predicting that item which an user may be interested in.
It is important to understand that in this context an item can be anything that a user is interested in. These suggestions augment the traditional decision making process and help the user make better decisions in a variety of different processes. These choices may be deciding what to buy, what song to listen to, which book to read next etc. On the other hand it also helps answer the question who will more likely buy this book, who is more likely to listen to this song.
We can see two major entities at play here, the user himself and the item. Broadly the emphasis placed on these two entities shapes the approach towards recommending an item to an user.
On one hand we profile users by observing their behavior, compare their behaviors to similar users in the system. This way it is possible to identify items that similar users don’t’ have in common and suggest it to one another.[16]
A different approach is to group similar items based on certain qualities, then if a user buys one of the item suggest him another item from the group.
[16]
As you might see in both the cases the definition of “similar” plays a key role in the system itself. Also towards building a recommendation system we would require a great deal of information to make credible recommendations. In today’s world of supposed big data it’s easy to assume that this would be the least of troubles. In coming weeks we will look deeper into this. But recommendation systems have been around for long in various forms when there was no big data.
History and Evolution of Recommendation Systems
Let us have a look at the evolution of recommendation systems over the years.
Prior to 1992
There have always been early indications about the need for a content filtering system. Once such example going back more than two decades is the mail filtering system MAFIA. This system is an active mail filtering system for intelligent document processing. This system aimed at providing support for the automatic processing of large amounts of documents workers need to deal with [3]. These filtering systems essentially recommended (or allowed) only those documents that were not spams or what the user actually wanted.
1992 to 1998
This period saw several recommendation systems come into the forefront. The first formal recommendation system designed was Tapestry by Xerox in 1992[4]. It was intended to recommend documents drawn from newsgroups to users. The rationale behind this was to prevent users from getting flooded with documents.
In 1994 Grouplens the first rating based recommendation system was created, this was followed by the creation of the first movie based recommendation system Movielens in 1997 [5].
1999-2005
There were several breakthroughs during this period in the field of recommendation systems.
Amazon came up with its patented item based collaborative filtering technique. This was followed by another major player Pandora launching its music genome project in 2000[6]. This project aimed to mathematically describe and organize songs.
2005 – Beyond
This period marks the advent of the next generation of recommendation systems. Several big names in a variety of domains ventured into this area. Some notable examples include Facebook, Trip Advisor, and Bing etc. The next big wave was seen during this time through the introduction of context aware recommendation systems which consider external factors like time, location etc. while providing recommendations[7].
Industry apart, recommendation systems have generated significant interest in academia as well. There are dedicated conferences and workshops related to the field. Some conferences include Special Interest Group on Information Retrieval (SIGIR), User Modeling, Adaptation and Personalization (UMAP) etc [2].
Across the world universities have courses entirely focusing on recommendation systems. There are many journals covering research in this area. Some of these are – AI Communications (2008); IEEE Intelligent Systems (2007) etc.
All this is great, but why should you care? What is the impact of these systems that so much effort is being put into them.
Impacts Of Recommendation Systems
On Consumers
Recommendation engines play an important role in shaping what decision a user makes online. Studies go to show that recommendation engines account for about 10-30 percent of an online retailers sales (Schonfeld 2007)[9]. A telling example of this is that around two thirds of the movies rented on Netflix were ones the user typically would not have rented if not suggested to do so by the recommendation system.[9]
There are two different schools of thought related to the impact of recommendation systems. The first one is that recommendation systems play an important role in helping the user find out about new products, and contribute towards diversity in sales. Another school of thought is that recommendation systems only further enhance the popularity of on demand products.
Studies in this field by Ling-Wu and team focus on another interesting aspect of recommendation systems. The results of this study state that recommendations increase customer satisfaction and also increase the willingness to pay. An interesting finding was that product awareness was an important parameter that needs to be considered while providing recommendations.[11] For instance, those customers who are interested in popular products would be better served by collaborative filtering systems, and those who have more specific needs would be satisfied with content based systems.[11]
Given the abundance of information available to users, selecting the best item is an onerous task. Recommendation systems assist the user in selecting the most relevant product for them, based on their specific interests. This in turn reduces the cost per fit and the cost of processing product related information (Chen et al. 2004).[12]
A caveat with this whole system is that recommendations only have value to users if they believe that these recommendations are credible. However, since the recommendations are provided by retailers some amount of skepticism and suspicion on the customer’s side is natural. Customers fear that recommendations are being manipulated. It seems that to some extent these fears are justified, as there has been evidence of retailers manipulating the outcome of recommender systems (Flynn 2006; Mui 2006). [10][13]
The ratings and reviews of customers are a little tricky to interpret because of the subjective nature and hence can be manipulated by other users. For example, it is possible for a person to write a review even if he/she has never purchased or used the product. But the recommendations are based on credible historical data and hence it cannot be changed by anyone. The retailers might take advantage of this and change the data to suit their needs. The study conducted by Senecal and Nantel in 2004 showed that the consumers are influenced more by the recommendations than the reviews and ratings of other people.[14]
On Business
“people don’t know what they want until you show it to them.”
― Steve Jobs
Managers can choose to design recommender engines based on their sales goals and inventory levels. This kind of design will have a lower priority for consumer’s preferences. Not all firms can afford this without shedding market share. But most recommendation systems will have the consumer’s preference as its top priority.
Increase the number of items sold – This is probably the most important function for a commercial recommendation systems, i.e., to be able to increase the number of items compared to those sold without the use of recommendation engines. For example, a retailer like Best Buy would want to clear its stocks and hence lure customers towards it (Under the assumption that the firm’s interest is considered as a parameter while modeling the recommendation system).[2]
Sell more diverse items – One more major function of recommendation engines is to help the users to find items that might be difficult to purchase without a good recommendation. For example, Netflix want to make money by renting out its entire collection. But usually, the consumer are likely to just view the popular ones and not care about the low key ones. The recommendation systems come to the rescue. They suggest such low key movies to the people who are more likely to watch them.[2]
Indirect Impact on Price – The retailers can base their pricing on the usefulness of recommendations. After all these recommendations are an added feature to the consumers to make their lives easier. And hence the retailers might charge the consumers extra for this service.
While this makes it more likely for a shopper to find a product that better matches her preference, it also increases the search cost for the same shopper to find a product that fits her requirements (Stiglitz 1989). Hence the searching cost is reduced for the shoppers and a few of will be okay to pay an additional amount for this this service in order to reduce uncertainty. [8] In certain cases of recommendations, customers will be influenced by the degree of recommendation and they will be convinced that the recommended product would best suit their needs even if it is not true otherwise. The retailers can take advantage of this and charge more.
References:
[1] courses.cs.byu.edu/~cs653ta/RS_old_but_very_popular.pdf
[2] www.inf.unibz.it/~ricci/papers/intro-rec-sys-handbook.pdf
[3]http://dl.acm.org/citation.cfm?id=122222
[4] http://www.vikas.sindhwani.org/recommender.pdf
[5]http://www.hcii.cmu.edu/news/seminar/2002/02/grouplens-research-project-collaborative-fil tering-recommender-systems
[6] https://www.pandora.com/about/mgp
[7]http://ids.csom.umn.edu/faculty/gedas/nsfcareer/CARS-chapter-2010.pdf
[8] Robert Garfinkel et all, “Empirical Analysis of the Business Value of Recommender Systems”, 2007.
[9]. Schonfeld, E. ―Click here for the upsell,‖ CNNMoney.com, July 2007. [http://money.cnn.com/magazines/business2/business2_archive/2007/07/01/100117056/index.htm]
[10]. Flynn, L. J. “Like This? You’ll Hate That. (Not All Web Recommendations Are Welcome.)”New York Times, January 23, 2006. [http://www.nytimes.com/2006/01/23/technology/23recommend.html]
[11]. Ling-Ling Wu, “Recommendation Systems and Consumer Satisfaction Online: Moderating Effects of Consumer Product Awareness”, 2013.
[12]. Chen, P.-y., Wu, S.-y. and Yoon, J. “The Impact of Online Recommendations and Consumer Feedback on Sales,” Proceedings of the ICIS, 2004, pp. 711-724.
[13]. Mui, Y.Q. “Wal-Mart Blames Web Site Incident on Employee’s Error,” Washington Post,Jan. 7, 2006, pp. Financial D01.
[14]. Senecal, S. and Nantel, J. “The influence of online product recommendations on consumers’ online choices,” Journal of Retailing (80:2), 2004, pp. 159-169.
[15]. Stiglitz, J.E. “Imperfect Information in the Product Market,” In Handbook of Industrial Organization, S. R. and R. Willig (Ed.), 1, Elsevier-Science, New York, 1989.
[16]. Picture borrowed from “http://blog.comsysto.com/2013/04/03/background-of-collaborative-filtering-with-mahout/”