The Power of Recommendation Systems
Hello data science enthusiasts, in this article, I will be discussing Recommendation Systems. As a data scientist, I will explain the solutions to some of the numerous challenges that one might encounter when starting a career in this field. Similar to my previous articles, I will be discussing the topic on Medium and delving into more detailed coding on Kaggle, providing explanations for the code as we proceed.
Recommendation Systems
There is an abundance of content in today’s world, and users tend to be interested in a smaller subset of this content. Therefore, it is essential to filter the abundant content based on users’ interests. In short, Recommendation Systems are the systems that recommend products or services to users by employing various techniques.
Simple Recommender Systems
These are general recommendations made using business knowledge or simple techniques. They involve recommending the highest-rated items, trending ones, classics, and so forth within a category. This type of system is implemented without any complex algorithms, relying solely on business knowledge.
Association Rule Learning
These are product recommendations based on rules learned through association analysis, commonly known as market basket analysis. Many companies utilize this technique, especially in the e-commerce domain. The logic behind it involves using association rules-based machine learning techniques to determine the likelihood of certain items being purchased together by identifying patterns, relationships, and structures within the data. A famous example is the observation that customers buying diapers at Walmart often also buy beer. Walmart strategically places diapers and beer in close proximity in their stores, leading to a significant increase in profits.
Apriori Algorithm
The Apriori algorithm is a method for market basket analysis that uncovers product associations. It has three important metrics:
The Apriori algorithm works by calculating possible product pairs based on a predetermined support threshold value at the beginning of the step-by-step process. It then generates the final table by making eliminations at each iteration based on the defined support value.
Content-Based Recommendation
Recommendations are developed based on similarities within the content of the products. For instance, if a user reads a book of a certain category, a similar book is recommended based on its content.
To achieve this, the text is mathematically represented, often utilizing techniques such as Count Vectorization and TF-IDF (Term Frequency-Inverse Document Frequency). These methods capture key words in product descriptions, representing them as text vectors. By doing so, products with similar descriptions can be identified and recommended.
Count Vectorizer
Step 1: Place all unique terms (words) as columns and all documents (such as tweets or product titles) as rows.
Step 2: Place the frequencies of term occurrences in the cells.
TF-IDF (Term Frequency-Inverse Document Frequency)
It normalizes the frequencies of the words not only within their own documents but also across the entire dataset we are focusing on.
Step 1: Compute the count vectorizer.
Step 2: Compute Term Frequency (TF) — the frequency of the term ‘t’ in the respective document divided by the total number of terms in the document.
Step 3: Compute IDF (Inverse Document Frequency) as follows:
- 1+ loge((total number of documents +1) / (number of documents containing the term ‘t’ +1))
Step 4: Compute TF * IDF — multiply TF with IDF.
Step 5: Perform L2 normalization.
Collaborative Filtering
Item-Based Filtering
Recommendations are made based on item similarity. For instance, if a viewer enjoys a particular movie, another movie with a similar viewer rating structure is suggested.
Example: Let’s consider a scenario where a viewer likes a certain movie. Based on the similarity of the viewer rating structure, another movie that resembles the liked movie is recommended.
User-Based Filtering
Recommendations are made based on the similarities between users. For example, accessing the movies watched by a user, then accessing the movies watched by other users who have watched the same movies. The correlation of the movies watched by the other users, but not by our first user, is examined, and the one with the highest correlation is recommended.
Model-Based Filtering: Matrix Factorization
Matrix factorization is a modeling technique used to fill in the gaps when a user has not rated certain movies, while other users have. It assumes the existence of latent features (hidden factors) for users and movies and finds the weights of these latent features based on the existing data. Using these weights, predictions are made for the missing observations.
More detailed and explained codes in the link below;
