Sentiment Analysis

Sentiment analysis is that using NLP, statistics, or machine learning methods to extract , identify or otherwise characterize the sentiment content of a text unit. It sometimes refers to as opinion mining.

Existing approaches to sentiment analysis can be grouped into three main categories: knowledge-based techniques, statistical methods, and hybrid approach.Knowledge-based techniques classify text by affect categories based on the presence of unambiguous affect words such as happy, sad , afraid and bored. Some knowledge bases not only list obvious affect words, but also assign arbitrary words a probable “affinity” to particular emotions. Statistical methods classify based on machine learning approaches such as decision tree classifier,support vector machines,neural network and naive bayes. Hybrid approaches leverage on both machine learning and elements from knowledge representation in order to detect  semantics  that are expressed in a subtle manner.

DT,KNN and NB

The decision tree will build a model based on the training dataset as soon as the training dataset is available. It is an example of eager learner. The model is then applied to testing dataset for prediction. Decision tree can be applied to data sets that have many attributes and the results are more readable. An opposite strategy is not to build a model from the training dataset; but make prediction using the training dataset when a testing record is available. It is an example of lazy learner. K-nearest  neighbor (knn) and naive Bayes (NB) classifiers are examples of lazy learner. The k-Nearest Neighbor mainly depend on the surrounding adjacent samples,and we can adopt the weighting method(The smaller of the distance, the larger of the weight) to improve. Also the calculation amount of KNN is large.Naive Bayes model originated from the classical mathematics theory, it has a solid mathematical foundation, and the clsaaification efficiency of NB is stable. For Naive Bayes,we need to know the priori probability.

Data Mining vs Social Media Analytics

Data mining Procedures include input data, data pre-processing, data mining, post-processing.

Major tasks in data pre-processing include data cleaning which means fill in missing values, smooth noisy data, identify or remove outliers and resolve inconsistencies. Data transform that involves normalization. Data reduction that is obtaining a reduced representation of the data set that is much smaller in volume but yet produces the same analytical results.

Data mining includes pattern discovery, such as association&correlation, classification, clustering, regression.

Post processing involves the evaluation, selection, interpretation, visualization of pattern.

Social media analytics is not only about data mining and statistical analysis but involves current affairs, such as people’s sharing and feelings on network which involves a lot of sentiment analysis, it provides us a new way to understand human behaviour. Social media analytics is a so interdisciplinary area that involves both humanities and technologies.

How powerful ,powerful,powerful of social network analysis ! ! !

In recent lectures, professor talked something that focus on the social network analysis, and also introduced many many formulas such as the degree centrality, betweenness centrality, closeness centrality and other things like that. At the first sight, those formulas may seem to be recondite for us, and it is no doubt that we feel them boring and useless. However, I must admit that social network analysis is so powerful, and we still have a long way to go to develop its infinite potential value.

In the blog, certainly I avoid to introduce those complicated formulas,instead,I want to share you some examples to elaborate its value. Actually,Social network analysis has been applied to many fields.

A recent news said that America has developed social network analysis technology to predict the action of terrorists. The CIA collected a large amount of personal information, including emails and telephone record, and inputted them to the computer to establish a data base, then analyzed the social actors and the relational ties between the actors. Finally identify a social scope, and find out the key actors within the scope.

Another example is that social network analysis can be applied to business forecasting. So attractive? In 2010, researchers in HP Lab find that we can know the change of people’s interests better by using Twitter, which can help us to predict the box office exactly. They collected 3 million tweet that related to a specific film within 3 months, finding that the frequency of the occurence of the film ‘s name had a strong correlation with the box office.

The above two examples shows the power of social network analysis, and related examples is countless. Just remember to try to take advantage of it, and……concentrate yourself in the class, listen to the teacher carefully!

The power of reviews

last week,my friend and I went to a restaurant located in 佐敦 to have dinner. After the meal, we submitted the pictures of the delicious food we took at that restaurant through our social applications, such as Facebook, Weibo, Wechat, and leaving our positive comments together with the photos. As a reward, the restaurant reduced a certain amount of fee for our meal(15HKD), not so much, but we were satisfied. From the perspective of merchant. through our sharing, the restaurant may be known by more and more people, and finally turn browsers into buyers because of our positive comments. It is seemed to be a win-win process.

I think behind the actions, it is the power of sharing and reviews. Customers do a lot of research online before making a buying decision. And often people trust opinions and recommendations from friends and buyers who share their interests. Based on this kind of social commerce, it allow to connect businesses with customers’ social networks, and help to support and enhance the buying and selling products and services online and in-store.
The PowerReviews is a company that focus on delivering software and network products to generate the reviews, and by syndicating the ratings and reviews to Google, it can make sure that shoppers can find you and your product. Customers rely on ratings and reviews as the authentic voice of the consumers, guiding purchases online and in-store. Many businesses now rely on PowerReviews to generate and syndicate reviews to drive traffic, increase sales, and create actionable insights.

Reviews are so powerful, please make a good use of it! Let sharing and reviews help you in your businesses.

To be a rational,sober person! ! !

We all have seen a common nature show about herding animals that when one of the animals in a herd panics and begins to break in a direction,it tends to affect the whole herd.This phenomenon can lead to fairly disastrous results such as animals injured or trapped,but at the same time it can also have positive results,like making most of the animals escaping a predator,therefore protects the herd’s survival.

  ↔

These unplanned behaviors in animals can be applied to many aspects of human culture,and these behaviors have both negative and positive consequences similar to those of herding animals.

It is commonly known that when we make a decision ,often the information you can infer from the choices of a large number of people may be more powerful than your own private information,therefore you choose to do something regardless of your own private information.In other words,people often look to others for clues on how to behave.Given a choice between two similar stores,people always choose the store that has other people in it ,representing desire to move with the “herd”.

A funny story also shows what is the herding effect vividly.It is said that an oil tycoon goes to the heaven to attend a meeting ,when he reaches the meeting place,finding that no vacant seat left,suddenly he has a brainwave and shouts that oil has been discovered in hell.After the shouting,all other oil tycoons rush to the hell sequentially and leave himself alone at the meeting place.At the moment,the tycoon himself thinks in mind with hesitation that has oil really been discovered in hell,why have all other people gone? Therefore the last tycoon himself goes to the direction of the hell hastily.

The story exhibits the tendency of individuals to mimic the actions of a larger group.And there are a couple of reasons why herd behavior happens.One is the social pressure of conformity,because most people are sociable and have a natural desire to be accepted by a group,rather than be branded as an outcast.The other reason is that the common rationale that it is unlikely that such a large group could be wrong.At these situations,you choose to follow the group.

In summary,what I want to show is that herding effect can have a big influence on us,what we should do is to be rational,sober and do not let it affect our decision making easily.

six degrees of separation

In lecture 2,what left me deep impression is the concept of six degrees of separation.

small-world phenomenon which also known as six degrees of separation is the theory that anyone on the planet can be connected to any other person on the planet through a chain of acquaintance that has no more than five intermediaries.The small world experiment was conducted by Milgram in 1967.He randomly selected people in Nebraska to send a letter located in the town near Boston.The result exhibited that it only took(on average)between five and seven intermediaries to get each letter delivered.

   This concept was popularized by John Guare’s play which is adapted into a comedy-drama film.One of the characters says that I read somewhere that everybody on this planet is separated by only six other people.(And by the way,if you have interest in that film,you can search it on Youtube.)

However,maybe the conclusion of the six degrees of separation drew from that experiment seemed doubtful.At first,letters delivered in the American network,not in the global network.Secondly,six is an average distance and many letters can not get to the destination for a couple of reasons.Well,researchers at Microsoft studied records of 30 billion electronic conversations among 180 million people in various countries.This was the first time a planetary-scale social network has been available .The database covered all the Microsoft messenger instant-messaging in June 2006,equivalent to half the world’s instant-messaging traffic at that time.The researchers looked at the minimum chain lengths it would take to connect 180 billion different pairs of users in the database.They found that the average length was 6.6 hops and 78 percent of these pairs could be connected in 7 steps or fewer.The experiment shared on a very large scale showing that this idea goes beyond folklore.

In my view,the six degrees of separation is great.It exhibits that any pair of strangers on the planet could make connections directly or indirectly through an average of 6 intermediaries. And under most circumstances ,the theory can help little more than establish that kind of connections,it can not help to fulfill personal purpose related to the person you build connections with.Just imagine,it is obviously impossible that anyone can let president Xi be your closed friend finally.

---