A multi-criteria decision making approach for recommending a product using sentiment analysis

Abstract

Nowadays, online platform has become a modern means of shopping among people. The reviews of products by customers have been proliferating on the online platform for a while. Since a large number of reviews are available, invariably customers read reviews before buying the product. Majority of the reviews are lengthy and repetitive, some of them even have nothing to do with the product itself. Going through the reviews before making a decision has become a tedious task. Further, the product selection is a complex decision making problem where several criteria are involved in the decision making process. Researchers have used methods like machine learning and sentiment classification to analyze the review of customers to summarize them. However, review summarization does not suggest the best/worst product. This study aims to recommend the best product based on the opinions expressed in the customers' reviews. We analyze the reviews of customers from various online platforms and use effective multi criteria decision making approach to evaluate and recommend the best suitable product. Real-time dataset from Flipkart and Amazon are used to evaluate our system's performance. Different case studies have shown that our proposed method produces a promising result which can help the user in the decision making process.

Full text

A Multi-Criteria Decision Making Approach for Recommending a Product using
Sentiment Analysis
Gaurav Kumar, Graduate Student Member, IEEE
School of Computer and Systems Sciences
Jawaharlal Nehru University
New Delhi, India.
gaurav37_scs@jnu.ac.in
Supervisor: Dr. Parimala N.
Abstract— Nowadays, online platform has become a modern
means of shopping among people. The reviews of products by
customers have been proliferating on the online platform for a
while. Since a large number of reviews are available, invariably
customers read reviews before buying the product. Majority of the
reviews are lengthy and repetitive, some of them even have nothing
to do with the product itself. Going through the reviews before
making a decision has become a tedious task. Further, the product
selection is a complex decision making problem where several
criteria are involved in the decision making process. Researchers
have used methods like machine learning and sentiment
classification to analyze the review of customers to summarize
them. However, review summarization does not suggest the
best/worst product. This study aims to recommend the best
product based on the opinions expressed in the customers’ reviews.
We analyze the reviews of customers from various online
platforms and use effective multi criteria decision making
approach to evaluate and recommend the best suitable product.
Real-time dataset from Flipkart and Amazon are used to evaluate
our system’s performance. Different case studies have shown that
our proposed method produces a promising result which can help
the user in the decision making process.
Keywords—Sentiment Analysis, Review Mining, Smartphone
Evaluation, Multi Criteria Decision Making, AHP, TOPSIS, WSM
I. INTRODUCTION
The ease of internet access and user interface of various
online platforms make customers more comfortable in sharing
their experience online. The user provides reviews and valuable
feedback about products or services they use which result in a
large number of reviews. Many consumers say that online
reviews impact their buying decisions [1]. According to a
survey report released by Pew Research Centre, 82% of U.S.
adults agree that online reviews help them decide whether or
not to purchase a product, including roughly one-third who
agree very much1.
For instance, if one looks at smartphone reviews on any
online shopping website, the number of reviews can be from a
few thousand to a few hundred thousand. The majority of the
reviews are lengthy and repetitive, and some even have nothing
to do with the product itself. Customers invariably read reviews
before buying the product. Examining all reviews can be dreary
and unbeneficial. On the contrary, if the customer reads only a
few reviews, he or she may get biased views. It would be
1 http://www.pewinternet.org/2016/12/19/online-reviews/
excellent if these reviews could be pre-processed automatically
and the best product is recommended to the user.
Most of the current work on recommender systems [2]–[7]
is based on the assumption that rating alone is not sufficient to
decide the overall quality of an item. Because the meaning of
rating is ambiguous and it varies with the knowledge,
experience, and understanding of the customers towards the
services or products they use. Some researchers [7]–[13] have
extracted valuable information from online reviews and
combined with the traditional recommendation algorithm to
enhance the accuracy of recommendation. Moreover, it has
been observed that the new buyers mostly need an overall
opinion about the product rather than fifty four-star ratings and
fifty three-star ratings [14]. Many researchers [15]–[18] have
analyzed customer reviews by using several classical machine
learning and sentiment classification approaches such as Naive
Bayes, Support Vector Machine, and Maximum Entropy
methods to provide summarized information to the new user.
Review analysis can be broadly categorized as those dealing
with
a) Summary of the review as a whole, and
b) Feature-wise summary of the review
Consider (a). Early work in this area was primarily focused
on polarity classification of reviews. Reviews were”classified
as positive or negative by looking for the occurrence of specific
sentiment phrases or words. A different source of sentiment
phrase has been proposed including manually constructed
dictionary [20], WordNet [21], and search engine hit [22].
Various machine learning algorithms have also been applied to
classify and summarize the reviews based on the polarity of
user’s sentiment [23]–[26]. All methods performed relatively
well but failed to provide high accuracy because results
suggested that customers provide mixed reviews, e.g., praising
some features of the product but criticizing others.
Consider (b). Apart from identification of features which
are important, the researchers have focused on identifying the
semantic orientation of the words which express opinions of
these features. In reference [27], authors obtain common
features using association rule mining considering noun phrases
as features. They also use various pruning techniques to filter
the unwanted features from review database and used WordNet
synonyms and antonyms in conjunction with a set of seed words

to find sentiment orientation of opinion words. In [28], the web
as a corpus is used to find the product features, and relaxation
labeling strategy is used to identify the semantic orientation of
opinion words. Reference [31] used a SentiWordNet based
algorithm, and in [29] knowledge base approach using linguistic
rules are used to distinguish the opinionated sentences and
determine the product’s features for feature wise review
summary.
We notice the following drawbacks currently facing product
recommendation when reviews are the basis for
recommendation:
1) The summary of user reviews is a polarized one –
Excellent or Poor. However, it has been argued in the research
community[30]–[32] that a positive opinion on an item does not
imply that the customer liked everything about the item and a
negative opinion does not infer that the customer disliked
everything about the item.
2) When feature-wise review summary is presented to the
user, he/she still faces the difficulty of choosing the right
product. This is because the choice of a product is a complex
decision making problem involving multiple criteria and feature
wise summary does not recommend the best/worst product.
The objective of this thesis is to evaluate products from a list of
alternatives based on costumers’ reviews and recommend the
best one to the user. In the proposed method,
1) Reviews of products are analyzed to compute the
sentiments expressed with each one of them.
2) The features with their sentiments are combined to
recommend a product. The different combinations can give rise
to multiple alternatives. To find the best product from all
feasible alternatives in the presence of multiple, usually
conflicting decision criteria, an effective Multi Criteria
Decision Making (MCDM) algorithm is used.
In other words, we have proposed to integrate sentiment
analysis with Multi Criteria Decision Making (MCDM)
approach to recommend the best product from user reviews.
The layout of the paper is as follows. In section 2, we
have described the research design and proposed solutions of
our work. The experiments are discussed in section 3. The
results of the work completed so far are discussed in section 4.
Lastly, we have concluded the paper with future scope in
section 5.
II. RESEARCH DESIGN AND PROPOSED SOLUTIONS
In this thesis, reviews of a list of products are analyzed to
recommend the best product to the user. The architecture of the
system is shown in Figure 1. The steps are explained below.
1) Crawl Review: Reviews of products are crawled from
a different e-commerce website, and pre-processing is
performed to remove HTML tag and stop words. Only certified
buyers’ reviews are considered which are most recent. As a
result, we overcome the disadvantage of not considering
reviews which may be obsolete.
2) Important Feature Selection: In a review, there are
many features of the product on which customers express their
views based on their personal experiences and satisfaction.
We do not consider all the product’s features as decision criteria
but identify a list of top features which are very essential to the
most of the users and consider them as essential decision
criteria for the product selection. To select the features, we have
proposed two different methodologies. In the first
methodology, the following steps are performed to identify
essential decision criteria:
a) In the first step, we note a list of standard features of a
product from its official website.
b) In the second step, Natural Language Toolkit (NLTK)
Parser [33] is used to parse the review sentence, and
Term-Frequency approach is applied to identify the
popular features where popular is considered as the
ones most discussed among the user community. As a
result, a count is associated with each feature.
c) In the third step, the top n features of step 2 which also
appear in step 1 are picked up. From the list of top n
features identified in step 3, x (where x<n) of them
which are essential, are picked up. To do so, a survey
is conducted to understand the relative importance of
these features, wherein thirty experts who happen to
be research scholars at our university have been asked
to give a rating on a rank scale of 1-10. The lowest
value indicates least importance and highest value
indicates the highest importance of the feature. A list
of top x essential features is picked up as decision
criteria for the analysis.
In the second methodology, we have followed a multi-pronged
approach where four different aspects are considered to identify
the essential features. These aspects are defined below:
a) Literature studies,
b) The experience of experts,
Crawl Review
Important Feature
Selection
Opinion Identification
and Strength Computation
Best Product Evaluation
using MCDM approach
Figure 1 Product Evaluation Framework

c) Customer’s reviews mining (same procedure of step of
the previous methodology)
d) A survey conducted among a target group, and
A list of top x essential features is picked up, after combining
the four steps described above, as decision criteria for the
analysis.
3) Opinion Identification and Strength Computation:
After a list of essential features obtained in the previous step,
the corresponding opinion words have been identified. We have
considered three words distance forward and backward from
feature word to identify the opinion. In this paper, only JJ word
is used as an opinion word for our research. Fuzzy string
matching concept is used to handle the issue of words variant.
After identifying the opinion word, SentiWordNet lexicon
dictionary is used to compute the strength of opinion word. It is
an extension of WordNet lexicon that provides the semantic
strength of each synset in terms of Pos(s), Neg(s) and Obj(s)
value on the scale of 0-1. The sum of these three values is
always 1.0. For each opinion word, SentiWordNet retrieves the
synset. If SentiWordNet does not find any suitable synset, the
sentiment score for this word is zero. If more than one synset
(or several synsets with differing sentiment scores) is returned,
then our system considers the very first score of synset which is
associated with an adjective tag. The very first occurrence of
synset in SentiWordNet Lexicon is said to be the most
occurring word in general [34]. We will address the precise
gloss identification of synset that returns multiple values in our
future work.
4) Best Product Evaluation: The best product is
evaluated using MCDM approach. It finds the best opinion
product from all feasible alternatives in the presence of
multiple, usually conflicting decision criteria. Here, the list of
top essential features identified in Step 2 form the attributes of
the MCDM algorithm. The opinion value computed in Step 3
populates the information matrix where different MCDM
approaches are applied. We have proposed to use two different
approaches to evaluate the best product as described below:
a) In the first approach, Weighted Sum Method (WSM) multi
criteria decision making algorithm is used to evaluate the
ranking of a list of products and the best one is
recommended to the user. Case study-1 shows its
implementation.
b) In the second approach, Technique for Order Preference by
Similarity to Ideal Solution (TOPSIS) multi criteria
decision making approach is used to evaluate the ranking
of a list of products and the best one is recommended to the
user. Case study-2 shows its implementation.
III. Experiments and Discussion
The experiment of product evaluation framework follows the
four steps outlined above. In the first step, customer’s reviews
are crawled from Flipkart and Amazon website. In the second
step, we identify the list of essential features based on
2 https://www.amazon.com/
consumer’s preferences. In the third step, we calculate the
strength of the opinion expressed on these features, and in the
last step, we apply different MCDM approaches to evaluate the
best product for the recommendation. All these processes have
been implemented on the Python platform. We have performed
our experiment on two case studies as discussed below.
A. Case Study I – Digital Camera Selection
We have taken reviews of top 10 digital cameras from
Amazon’s2 website for the case study of the first proposed
solution. Digital cameras are selected based on two criteria to
avoid biasedness. Firstly, a digital camera should have been
reviewed by at least 1,000 reviewers, i.e., it should contain at
least 1,000 reviews. Secondly, the selected digital camera
should be from a famous brand, i.e., familiar to the user
community. Based on these two criteria, top 10 digital cameras
are selected, and a dataset is created which contains 10,000
reviews, 1000 reviews of each digital camera. Some cameras
may be of the same brand. Each review contains a text review
and a title. Additional information is available but is not used in
this analysis, including date, time, author name, location, and
ratings. These review documents were then cleaned to discard
HTML tags and other additional information. Now, we conduct
the experiments in the following steps:
1) Identification of Most Important Features: The
process of identifying the most influencing criteria are followed
as discussed in section II. The first methodology discussed in
that section is implemented, and from this, the top five most
important features: Picture, Cost, Zoom, Battery, and Memory
are identified and used as decision criteria for the product
selection.
2) Opinion Identification and Strength Computation:
Once the list of top features is identified, our system identifies
the concordant opinion expressed on that feature as discussed
in the previous section. The system computes the average
sentiment score of opinion words for each feature to all five
cameras. An information matrix of the camera vs. feature is
generated. Each column in the information matrix contains
three values; each value plays a vital role in decision making
process. The first value represents aggregated positive score,
second is an aggregate negative score, and last is the total
number of users who have expressed an opinion on the
corresponding feature. Since MCDM approaches are best
applicable on decision matrix, we have derived the decision
matrix in the following way:
Case 1: Considering the higher sentiment value.
Case 2: Considering absolute difference of sentiment scores.
Most of the MCDM methods require the attributes
(features/criteria) to be assigned a weight of importance. These
weights add up to one to normalize its effect. Here, the weight
of each feature is calculated based on the number of users who
have expressed their opinion on given feature with the thought
that this is important.
3) Best Product Evaluation using Weighted Sum Method
MCDM Approach: The WSM [35] MCDM algorithm is

evaluated for both case 1 and case 2. The result indicates that
camera2 is a highest preferred choice among customers,
followed by camera3, camera4, camera1 and the least
preferred choice is camera5. To assess the performance of our
proposed system’s result, we have conducted a survey wherein
thirty experts gave their rating preference on scale 1-10 to
decide the preference order of five digital cameras. We have
applied the Borda Ranking technique [36] to evaluate the
ranking position of all cameras. We have found that Camera2
is highest rated camera by the experts; followed by camera3,
camera1, camera4, and least rated is camera5. The predicted
ranking with our proposed system remains same for the top
evaluated camera after the expert’s assessment. The important
point to note is that for the most preferred and least preferred
cameras, the result of our system matches with the survey.
However, in the middle range of preferred cameras, there is a
difference of opinion. We can consider a different multi criteria
decision making approach to improve the accuracy of the
recommended ranking. To improve the ranking results and an
in-depth understanding of the best product evaluation, we have
conducted another study with a different multi criteria decision
making approach to evaluate the best product from a list of
alternatives.
B. Case Study- 2 Smartphones Selection (Ongoing Work)
In the previous data set, we were unable to find most recent
useful reviews of digital cameras in large numbers. In this case
study, we have chosen a smartphone as a product selection
problem, which is used by the majority of the population across
the world3 and a large number of customers’ reviews are
available online. Smartphone selection depends on various
factors from personal to business usage. So, at the initial phase
of the study, we have limited this case study on Indian
customers only because India has become world second largest
smartphone users market after the United States according to
the counterpoint research report4. A list of five popular
smartphones is selected based on the report from Digit [37]
which is the most read technology magazine in India under the
fixed price range for the case study. Only first 1000 most
helpful recent reviews of certified buyers of each smartphone
are crawled from Flipkart5 and Amazon6 e-commerce website
for the analysis. Now, we conduct the experiments in the
following steps:
1) Identification of Most Important Features
(Smartphone Selection Criteria): To propose the most
desirable criteria list, we have followed the secondly proposed
methodology of decision criteria selection as discussed in
section II. Four different aspects such as in-depth literature
studies, the experiences of telecommunication experts,
customers’ reviews mining, and a survey of a smartphone to
target group are collectively combined, and top 4 essential
important features: Battery, Speed, Display, and Camera are
picked up as decision criteria for the analysis.
3 https://www.emarketer.com/Report/Worldwide-Internet-Mobile-Users-
eMarketers-Estimates-20162021/2002038
4 https://www.counterpointresearch.com/india-smartphone-market-update-q2-
2017/
2) Opinion Identification and Strength Computation: The
process of identifying opinion and computing its strength
remains same as discussed in section II. We have formulated
the information matrix which contains the aggregated strength
of opinion expressed on each criterion for all five smartphones.
This information matrix will be used as a Decision Matrix to
the input of MCDM algorithm. We have already discussed in
the previous case study that most of the MCDM methods
require the attributes (features/criteria) to be assigned a weight
of importance. To estimate the weight of importance of the
feature, we have used Analytical Hierarchy Process (AHP) [38]
multi criteria decision making approach. It uses pairwise
comparison techniques to obtain the weights of the importance
of decision criteria. We have also considered 150 decision
makers’ responses to compute the weight of importance. The
respondents are from age groups of 18-34 years. They have
been asked to give a rating to each of the factors on Likert Scale
[39] “not important at all,” “not very important,” “important,”
“quite important,” and “very important” which are the verbal
representations of the 1-5 numeric scale respectively. The
aggregated weight estimation and measuring the consistency of
decision maker’s judgment is still ongoing.
3) Best Product Evaluation using Technique for Order
Preference by Similarity to Ideal Solution (TOPSIS) MCDM
approach (ongoing): TOPSIS is based on the concept that
selected alternative should be closest to the positive ideal
solution and farthest from the negative ideal solution. The ideal
solution is the solution that maximizes the benefits and
minimizes the total cost. On the contrary, the negative ideal
solution is the solution that minimizes the benefits and
maximizes the total cost. We will apply the TOPSIS [40]
MCDM approach for the best product evaluation in the
following steps:
Step 1: In the first step, decision matrix generated in the
previous step is normalized to transform various attribute
dimensions into non-dimensional attributes, which allows
comparisons across criteria.
Step 2: A weighted normalized decision matrix is computed by
multiplying the weight of importance of criteria as obtained in
the previous step discussed above.
Step 3: A positive ideal solution and negative ideal solutions
are determined to measures the set of benefit criteria and
negative criteria.
Step 4: The separation measures are calculated using the n-
dimensional Euclidean distance.
Step 5: Relative closeness to the ideal solution is determined.
Step 6: At the final step, the preference order of five
smartphones are ranked.
To validate the results, we will manually annotate the
customers’ reviews and identify the strength of the opinion
5 https://www.flipkart.com/
6 https://www.amazon.in/

expressed in the review data set of five smartphones to validate
our ranking results.
IV. CONCLUSION AND FUTURE WORK
We have identified the list of top essential features which
influence the customer in the decision making process.
Considering these features as decision criteria, we have
evaluated the best product from a list of alternatives in different
cases and recommended the best product to the user. Sentiment
analysis techniques have been used to identify the overall
customer experience, and different MCDM approaches are
being applied to evaluate the best product for the
recommendation. Our experiment’s result is also validated
through a survey conducted among a group of experts, in which
we have found that our experiment produces the promising
result for the recommendation and is reliable for the user in
his/her decision making process. Some results are still in the
process of evaluation.
The contribution of this thesis is in evaluating the products
form a list of alternatives and recommend the best one to the
user. However, user preference has been ignored in the
recommendation process. In the next part of our thesis, we will
also be considering the target user preference on the decision
criteria and then recommending the best product from a list of
alternatives. To validate our results, we propose to manually
annotate the customers’ reviews and use this annotation for
recommending a product. Further, to measure the reliability of
our recommendation, statistical analysis using Kendal’s and
Spearman rank correlation will be performed. Our prosed
model is not limited to only e-commerce domain, but also can
be adopted in different domains such as education,
environment, agriculture, healthcare, manufacturing, tourist
and traveler; and so on to solve the complex decision making
problems to provide a better alternative.
ACKNOWLEDGMENT
I would like to thank my supervisor Prof. Parimala N. for
consistence guidance and support in formulating my ideas. I also
thank Human Resource Development Group, Council of
Scientific & Industrial Research (CSIR), Ministry of Science
and Technology, Govt. of India for funding the fellowship
(09/263(1001)/2013-EMR-1) throughout my research.
REFERENCES
[1] A. Ghose and P. G. Ipeirotis, “Estimating the helpfulness and economic
impact of product reviews: Mining text and reviewer characteristics,”
IEEE Transactions on Knowledge and Data Engineering, vol. 23, no. 10.
pp. 1498–1512, 2011.
[2] G. Adomavicius and A. Tuzhilin, “Toward the next generation of
recommender systems: A survey of the state-of-the-art and possible
extensions,” IEEE Transactions on Knowledge and Data Engineering,
vol. 17, no. 6. pp. 734–749, 2005.
[3] G. Adomavicius and Y. Kwon, “New recommendation techniques for
multicriteria rating systems,” IEEE Intell. Syst., vol. 22, no. 3, pp. 48–55,
2007.
[4] A. Koukourikos, G. Stoitsis, and P. Karampiperis, “Sentiment analysis: A
tool for rating attribution to content in recommender systems,” CEUR
Workshop Proc., vol. 896, pp. 61–70, 2012.
[5] J. Bobadilla, F. Ortega, A. Hernando, and A. Gutiérrez, “Recommender
systems survey,” Knowledge-Based Syst., vol. 46, pp. 109–132, 2013.
[6] M. Aamir and M. Bhusry, “Recommendation System: State of the Art
Approach,” Int. J. Comput. Appl., vol. 120, no. 12, pp. 975–8887, 2015.
[7] W. Wang and H. Wang, “Opinion-enhanced collaborative filtering for
recommender systems through sentiment analysis,” New Rev.
Hypermedia Multimedia., vol. 21, no. 3–4, pp. 278–300, 2015.
[8] K. Bauman, B. Liu, and A. Tuzhilin, “Estimating customer reviews in
recommender systems using sentiment analysis methods,” pp. 1–20.
[9] C. W. Leung, S. C. Chan, and F. Chung, “Integrating Collaborative
Filtering and Sentiment Analysis : A Rating Inference Approach,” ECAI
2006 Work. Recomm. Syst., pp. 62–66, 2006.
[10] L. Ferreira, N. Jakob, and I. Gurevych, “A Comparative Study of Feature
Extraction Algorithms in Customer Reviews,” Proc. 2nd IEEE Int. Conf.
Semant. Comput., pp. 144–151, 2008.
[11] F. Peleja, P. Dias, F. Martins, and J. Magalhães, “A recommender system
for the TV on the web: Integrating unrated reviews and movie ratings,”
Multimedia. Syst., vol. 19, no. 6, pp. 543–558, 2013.
[12] Y. Zhang, G. Lai, M. Zhang, Y. Zhang, Y. Liu, and S. Ma, “Explicit
Factor Models for Explainable Recommendation based on Phrase-level
Sentiment Analysis Categories and Subject Descriptors,” Sigir, pp. 83–
92, 2014.
[13] R. Dong, M. P. O’Mahony, M. Schaal, K. McCarthy, and B. Smyth,
“Combining similarity and sentiment in opinion mining for a product
recommendation,” J. Intell. Inf. Syst., vol. 46, no. 2, pp. 285–312, 2016.
[14] J. Sun, C. Long, X. Zhu, and M. Huang, “Mining Reviews for Product
Comparison and Recommendation,” Internet Services Workshop 2008,
no. 39. pp. 33–40, 2008.
[15] B. Pang, L. Lee, and S. Vaithyanathan, “Thumbs up?: Sentiment
classification using machine learning techniques,” Proc. Conf. Empir.
Methods Nat. Lang. Process., pp. 79–86, 2002.
[16] M. Osborne, “Using Maximum Entropy for Sentence Extraction,” Proc.
Work. Autom. Summary. ACL, no. July, pp. 1–8, 2002.
[17] K. Dave, K. Dave, S. Lawrence, S. Lawrence, D. M. Pennock, and D. M.
Pennock, “Mining the peanut gallery: Opinion extraction and semantic
classification of product reviews,” Proc. 12th Int. Conf. World Wide Web,
pp. 519–528, 2003.
[18] B. Pang and L. Lee, “Opinion Mining and Sentiment Analysis,” Found.
Trends® Information, vol. 1, no. 2, pp. 91–231, 2006.
[19] Q. Miao, Q. Li, and R. Dai, “AMAZING: A sentiment mining and
retrieval system,” Expert Syst. Appl., vol. 36, no. 3 PART 2, pp. 7192–
7198, 2009.
[20] S. R. Das and M. Y. Chen, “Yahoo! for Amazon: Sentiment Extraction
from Small Talk on the Web,” Manage. Sci., vol. 53, no. 9, pp. 1375–
1388, 2007.
[21] J. Kamps, M. Marx, R. J. Mokken, and M. De Rijke, “Using wordnet to
measure semantic orientation of adjectives,” ACL ’10 Proc. 48th Annu.
Meet. Assoc. Comput. Linguist., pp. 395–403, 2010.
[22] P. D. Turney, “Thumbs up or thumbs down? Semantic Orientation applied
to Unsupervised Classification of Reviews,” Proc. 40th Annu. Meet.
Assoc. Comput. Linguist., no. July, pp. 417–424, 2002.
[23] D. E. Nyaung and T. L. L. Thein, “Feature-Based Summarizing and
Ranking from Customer Reviews,” Int. J. Comput. Electr. Autom. Control
Inf. Eng., vol. 9, no. 3, pp. 691–696, 2015.
[24] D. Billsus and M. J. Pazzani, “A hybrid user model for news story
classification,” UM99 User Model., pp. 99–108, 1999.
[25] N. J. Belkin, “Helping people find what they don’t know,” Commun.
ACM, vol. 43, no. 8, pp. 58–61, 2000.
[26] A. Khan, “Sentiment classification by sentence level semantic orientation
using sentiwordnet from online reviews and Blogs,” Int. J. Comput. Sci.
Emerg. Technol., no. 4, pp. 539–552, 2011.
[27] M. Hu and B. Liu, “Mining Opinion Features in Customer Reviews,” 19th
Natl. Conf. Artifical Intell., pp. 755–760, 2004.

[28] A.-M. Popescu and O. Etzioni, “Extracting Product Features and Opinion
from Reviews,” Hum. Lang. Technol. Empir. Methods Nat. Lang.
Process. Vancouver, Br. Columbia, pp. 339–346, 2005.
[29] S. S. Htay and K. T. Lynn, “Extracting product features and opinion words
using pattern knowledge in customer reviews,” Sci. World J., vol. no. 3,
2013.
[30] P. D. Turney and M. L. Littman, “Measuring praise and criticism,” ACM
Trans. Inf. Syst., vol. 21, no. 4, pp. 315–346, 2003.
[31] W. Hu, Z. Gong, and J. Guo, “Mining Product Features from Online
Reviews,” in eBusiness Engineering ICEBE 2010 IEEE 7th International
Conference on, pp. 24–29, 2010
[32] G. Vinodhini and R. Chandrasekaran, “Sentiment Analysis and Opinion
Mining: A Survey,” Int. J. Adv. Res. Comput. Sci. Softw. Eng., vol. 2, no.
6, pp. 282–292, 2012.
[33] Bird, Steven, and Edward Loper. "NLTK: the natural language toolkit."
Proceedings of the ACL 2004 on Interactive poster and demonstration
sessions. Association for Computational Linguistics, 2004.
[34] B. Ohana and B. Tierney, “Sentiment classification of reviews using
SentiWordNet,” Sch. Comput. 9th. IT T Conf., p. 13, 2009.
[35] Triantaphyllou, Evangelos. "Multi-criteria decision making methods." In
Multi-criteria decision making methods: A comparative study, pp. 5-21.
Springer, Boston, MA, 2000.
[36] D. G. Saari, “The Optimal Ranking Method in the Borda Count,” IIASA
Collab. Pap., no. February 1985.
[37] Businesswire India. "DIGIT is the No. 1 technology magazine in India as
per IRS". The Financial Express. Retrieved 19 June 2015.
[38] Saaty, Thomas L. "Decision making with the analytic hierarchy process."
International journal of services sciences 1, no. 1, 83-98, 2008
[39] Likert R. “A technique for the measurement of attitudes,” Arch
Psychology. 1932;22(140):55.
[40] Boran, Fatih Emre, Serkan Genç, Mustafa Kurt, and Diyar Akay. "A
multi-criteria intuitionistic fuzzy group decision making for supplier
selection with TOPSIS method." Expert Systems with Applications, vol.
36, no. 8, 11363-11368, 2009.