Discussion


Note: Images may be inserted into your messages by uploading a file attachment (see "Manage Attachments"). Even though it doesn't appear when previewed, the image will appear at the end of your message once it is posted.
Register Latest Topics
 
 
 


Reply
  Author   Comment  
bpierce

Moderator
Registered:
Posts: 100
Reply with quote  #1 

For the January/February/March 2017 Visual Business Intelligence Newsletter article, titled Data Visualization Effectiveness Profile, Stephen introduces a set of criteria that can be used to evaluate and compare the merits of data visualizations: both our own visualizations, before we share them with people, and those of others, as a way to critique their effectiveness.

What are your thoughts about the article? We invite you to post your comments here.

-Bryan

Mari

Registered:
Posts: 1
Reply with quote  #2 
I really like the concept of the Data Visualization Effectiveness Profile, but find the visualization of it really misleading! First, the range within each category is subjective (Useless --> Very Useful; Ugly --> Beautiful); aligning the ends of these ranges leads the reader to compare values that aren't comparable because they don't have the same scale. Second, should categorical values be connected by a line? This leads the reader to see the shape of the 'curve' as important, but changing the sort order of the categories will change that curve's shape, and hence change the impression the reader gets. Stephen Few recommends sorting on a relevant value; what are these categories sorted on? So, I wonder, how well would these visualizations of the Data Visualization Effectiveness Profile rate on the Data Visualization Effectiveness Profile???
sfew

Moderator
Registered:
Posts: 837
Reply with quote  #3 
Mari,

The scales are indeed subjective. Each scale is supposed to be different, for it represents a different variable. As such, what I've proposed as a way of displaying a particular data visualization's effectiveness profile is a multivariate display, in many respects similar to a parallel coordinates plot. The patterns formed by the lines that connect the values are not meaningful in the same sense as the patterns formed by lines in a line graph showing a time series or in a frequency polygon showing a frequency distribution. Nevertheless, the lines are useful in a different way, for they allow us to see a visualization effectiveness profile as a single chunk of visual information, which makes it easy to compare multiple profiles.

__________________
Stephen Few
tomshanleynz

Registered:
Posts: 3
Reply with quote  #4 
Hi - I like it.

It will also be useful for assessing my own preferences for designed data viz.

Where you state "...we should shoot for data visualizations that fall within the following ranges..", this shows a very valid target range.  It would be interesting to see where others draw their own preferred ranges.  For example, a McCandless-type designer may aim for higher Aesthetics and lesser Truthfulness, which may suit their aims (eyeballs, clicks).  So the scale provides a framework to assess not only individual designs, but compare designers, and self-assess my own preferences.

T
tomshanleynz

Registered:
Posts: 3
Reply with quote  #5 
Also, its a useful companion (and comparison) to Kaiser Fung's Trifecta:
http://junkcharts.typepad.com/junk_charts/junk-charts-trifecta-checkup-the-definitive-guide.html

which focusses on

  • What is the QUESTION?
  • What does the DATA say?
  • What does the VISUAL say?

Few's framework has more to say on aesthetics aspect, whereas Fung's focusses more on the Usefulness, Completeness, Perceptibility and Truthfulness aspects.
heinzel

Registered:
Posts: 16
Reply with quote  #6 
Hi all,

It is good to see a systematic approach to evaluating visualizations. Not unlike like Mari, I did have a question when seeing the individual criteria and their scales:

The Completeness scale ranges from "No relevant data" to "All relevant data". In the description for it, the criteria is detailed as all the info needed, "but not more". That "but not more" part went missing on the scale. And this is for a reason: The scale is an attempt at representing a two-dimensional topic in a one-dimensional display. The two dimensions of Completeness as detailed in the descriptive text are "How much of the needed data is present" and "How much unneeded data is present". I don't think a one-dimensional representation can do this description justice.

I wonder if this reduction has been made as a conscious decision, in which case an asterisk explaining this might be helpful, or if it wasn't a conscious decision, in which case the follow-up question now would be: Do we keep the one-dimensional display because it fits the overall scheme? Or do we change it to 2D to reflect accurately what we're trying to measure? Or do we separate the category into two separate categories? The latter might be a good approach. Curious as to what you think, Steve!

Best,
Matthias
sfew

Moderator
Registered:
Posts: 837
Reply with quote  #7 
Hi Matthias,

I understand what you're saying about the "Completeness" scale, but I don't think it's necessary to complicate the effectiveness profile by separating completeness into two criteria: one that deals with having the essential data and one that deals with having no more than the essential data. I could have labeled the right end of the completeness scale as "All relevant data but no more," and perhaps I should have rather than shooting for conciseness. You're certainly welcome to label the scale in this manner. If you understand that the right end of the scale means "All relevant data but no more," you can indicate that a visualization includes unnecessary data by reducing its value on the completeness scale.

__________________
Stephen Few
danz

Registered:
Posts: 190
Reply with quote  #8 
Interesting article.

If I would like to be more analitic and quantify my own work, instead of nominal/ordinal scale I would prefer a simple numeric scale combined with extra quantitative information related to relevance of criteria. I would probably consider divergent scales (-10=bad, 0=neutral, 10=perfect) weighted by criteria relevance.

One criteria I feel that is missing is related to Innovation as a measurement of the ability of finding useful distinct/unique interpretation of particular set of data. This has nothing to do with finding new ways of displaying data, but with the ability of finding those attributes, metrics, correlations etc. which stand clearly out from regular interpretations. Many can copy/paste existing sales analysis (trends, targets, historical comparisons) and adapt them to new software packages. But few are able to identify and display new angles of a particular set of data.

A line graph showing sales evolution/trend over last few months of all products is always useful information. But if we extend this information by spliting the products in three groups based on degree of correlation with general trend (correlated, anti-correlated, not correlated) we might be more successful.
sfew

Moderator
Registered:
Posts: 837
Reply with quote  #9 
Hi Dan,

We're only evalulating the effectiveness of data visualizations with this tool. Evaluating the effectiveness of the analysis the produced the findings that are displayed is another matter, which is much more difficult to do. Even if we were evaluating the effectiveness of the analysis, I would rarely consider innovation a useful metric. Innovation is only needed when challenges are faced that can't be solved using existing methods. Novel findings are usually the result of skill and hard work, rarely the result of innovation.

As I mentioned in the article, you're certainly welcome to assign a quantitative scale to each of the criteria, but keep in mind that this won't reduce subjectivity.

__________________
Stephen Few
Rob_Wish

Registered:
Posts: 1
Reply with quote  #10 
Hi Steve,

Thanks for giving us all a useful tool; you've done a great job of outlining practical assessment criteria.

When developing slide presentations I try to consider the associations that the audience might have with an image. It's tricky because personal constructs vary, but it's an important consideration because those associations can shape the emotional and intellectual context in which the audience member takes in the information. If you think this is relevant in data visualization, perhaps there is some way to incorporate it into your emotive criteria.

I was surprised that you chose to connect the dots on the scales because, as I have understood your writing, the labels are nominal - they don't have a direct connection - so the shape of the line created is arbitrary depending on the sequence of the labels. I'm now afraid I haven't understood you correctly on this point. Why did you chose to connect them rather than leaving it as a dot plot?

Cheers,

Rob Wish
sfew

Moderator
Registered:
Posts: 837
Reply with quote  #11 
Hi Rob,

Many of the images that you include in your slides are photos. As such, you've chosen those particular photos because of the meanings that you expect people to associate with them. Graphs don't work in this way. They communicate specific quantitative information.

The seven criteria in the Data Visualization Effectiveness Profile do not constitute a nominal scale. Instead, they are entirely separate metrics. I chose to display the profile in a manner that is similar to a parallel coordinates plot, which also uses a line to connect the values of separate metrics. The line's purpose is to enable easy comparisons of multiple profiles. Without the lines, this would be extremely difficult.

__________________
Stephen Few
neilism

Registered:
Posts: 9
Reply with quote  #12 
As well as being useful from a personal perspective, this seems like an excellent way of evaluating visualizations from the users' perspectives.

I expect there would be some interesting patterns that would come out of it, and I guess that these would be strongly linked to both the professional background and personality traits of the user. Even 'analytical thinkers' react very differently to the graphs I've produced over the years -- engineers often seem happier with bullet graphs than accountants, for example.

Further, the various factors that Steve has identified might well load meaningfully onto the perceived Usefulness scale. This could provide guidance for designing (potentially...) more effective visualizations that would complement consideration of things like pre-attentive processing, etc..


jlbriggs

Registered:
Posts: 200
Reply with quote  #13 
Stephen - I would guess that others who have mentioned the line connecting these metrics might be thinking, like I am, of this page in your examples section:

[example17problem] 
http://www.perceptualedge.com/example17.php

I would be very interested in hearing how you think of this current display as it relates to your comments on this example and redesign. Primarily - do you see a fundamental difference between the two scenarios that changes the way they should be displayed, or has there been a change in your thinking about such a design, or is it maybe some part of both?

Although there are a number of potential design issues with the image above, the post seemed to call out the connecting lines as a significant issue.

Thanks!
sfew

Moderator
Registered:
Posts: 837
Reply with quote  #14 
jlbriggs,

The example that you referred to is different from the Data Visualization Effectiveness Profile (DVEF) in several respects. Primarily, the individual variables are differently weighted and there is a total that aggregates the individual variables. If quantitative scales and varying weights were assigned to the criteria in the DVEF and an aggregate were added, I would display it somewhat differently.

Nevertheless, my criticism of the use of lines in the example, which I wrote many years ago, conflicts with my current thinking. As an advocate of parallel coordinates plots, I can no longer criticize the use of lines to connect values related to a set of separate variables that are related to a specific entity, especially when a common purpose is to compare the multivariate profiles of multiple entities.

__________________
Stephen Few
jlbriggs

Registered:
Posts: 200
Reply with quote  #15 
Thanks for the clarification - it is helpful.
Previous Topic | Next Topic
Print
Reply

Quick Navigation:

Easily create a Forum Website with Website Toolbox.