Semantic analysis and feedback management: I love you, I love you not

For several years now, semantic analysis has been complementing Feedback Management programs. This technology helps companies to make sense of the many comments left by their customers in satisfaction surveys.

The power of today's tools, their ever-increasing integration of AI and Big Data technologies, make them invaluable allies for quality managers and customer satisfaction directors.

In this section, we'll look at the fields of application of semantic analysis within Feedback Management programs, and the limits they impose in certain cases.

What role does semantic analysis play in a Feedback Management program?

As we mentioned in a previous article, semantic analysis can be used in a number of ways within a Feedback Management program.

Finer detection of alerts

By using semantic analysis in addition to closed questions, alert detection becomes more precise. It may be that certain satisfied customers ask to be contacted again. It's also possible that some irate customers don't fall within the limits set to trigger an alert. In these 2 cases, the commentary is a precious help in catching up with these potential detractors.

Categorizing comments

Allowing a line manager to read comments concerning his or her department or division is of paramount importance to those aiming for continuous improvement. Semantic analysis enables open comments to be matched with closed questions, and comments to be filtered by category and sub-category.

Weak and strong signal detection

Another use for semantic analysis is the detection of emerging words (or groups of words) or words with remarkable volatility. Semantic analysis tools can then alert users to behaviors that a human is technically unable to detect.

Can semantic analysis replace closed questions?

When the customer experience is short (or weak), it's tempting to limit the number of closed questions in the NPS question and supplement it with an open question. Semantics will do the rest.

Here are the main reasons why this approach is a bad idea.

Answering an open-ended question takes time

Writing a comment takes time (from ten seconds to several minutes), whereas ticking an answer to a closed question takes just a few seconds. The time saved is not necessarily to the respondent's advantage.

Limited analysis granularity

Standard semantic analysis offers 2 distinct tones per category: positive or negative. Attempts to multiply these modalities (to 4 or 5) are, in my opinion, illusory.

When we try to replace a closed question with an open one, we end up measuring the percentage of positive (or negative) tonality over time. And here, the evolution is necessarily very limited due to the low granularity of the analysis.

A closed question, containing between 4 and 10 different response modes, brings much more detail and finesse to the analysis. Score evolutions are also more tangible and more in line with operational reality.

Too few responses

The best way to get an answer is still to ask the question. The fundamental problem with semantic analysis as part of a Feedback Management program is that it is based on an open-ended question such as "Thank you for your feedback". As a result, it's unlikely that the respondent will express his or her views on all the subjects of real interest to your operations.

On average, a classic respondent expresses himself on 3 subjects (in the case of an email response, less in the case of an SMS response).

A typical semantic classification scheme contains around ten categories, each containing around ten sub-categories...let's say 100 sub-categories for simplicity's sake.

Knowing that you need 30 responses to each sub-category to be statistically correct (borderline, but correct), and even assuming that respondents express themselves uniformly across all sub-categories, you need 1,000 respondents (at 3 subjects per respondent) to start producing reliable results.

At head office level, this may be feasible...but then it becomes clear that at store or even regional level, this means that management is no longer by the week or month, but rather by the year; in the best of cases.

Inconclusion

Semantic analysis adds a great deal to a Feedback Management program. But it cannot replace it. Yes, a treemap is an interesting and undoubtedly impressive way of classifying, at a glance, the categories or sub-categories of comments received. But it's no substitute for reliable statistical analysis of survey results. And it certainly can't guide operational staff in their corrective actions.