Jennifer Neville- computer scientist

"Humans just can't look at data and see patterns in the same way than they can look at visual representations."
(links for Jennifer Neville)


Which importance do images have in your professional field?

In model graphs nodes are variables and edges are statistical dependencies among the variables

.In data mining we sometimes try to visualize the data that we are looking at as patterns. For the things that I work on specifically, we represent our data as graphs where objects, people and items in the world will be nodes in a graph. Then we show edges to indicate relationships among those people. When we learn a model of the attributes of those data we also represent the model as a graph, which is sometimes confusing because they are different. In model graphs nodes are variables and edges are statistical dependencies among the variables. We’re often looking at graphs of that nature and are trying to come up with better ways of constructing those models.

So you need images to structure data and also your models?

humans just can’t look at data in that form and see patterns in the same way than they look at visual representations

Yes, and to communicate that structures to other people as well as to ourselves. Once I had an experience: I was learning a model and I was getting very strange results just looking at the mathematical output. I was trying to debug my algorithm and everything seemed reasonable. At that point I didn’t have any visualization, I was just looking at data in a database. However, someone else in our lab was developing a visualization tools at the same time. Once that was ready I took a look at my data and it turned out that there was a corruption in my data where there were actually self-links, the sort that loop out of a node and come back. I was assuming those weren’t there and that messed up my model. But until I had the ability to see that, I couldn’t tell that there was that mistake in the data. That to me shows the importance of being able to visualize the data – it matters how familiar you are with the underlying data structures, humans just can’t look at data in that form and see patterns in the same way than they are looking at visual representations.

So you need images in the double-sense to navigate?

relational data - like navigation through friendship relationships

Yes, you can understand this type of relational data if you look for example at the web, where web pages link to each other. No, think of people and trying to navigate through friendship relationships. If you are trying to recommend your friend to someone for a job you try to navigate through those relationships. We see it as a way to start to understand how to search the data, it improves our ability to navigate.

On the other side, one of your major tools is mathematics. Do you feel the maths in your images ?


If you compare classical images like drawings we saw yesterday and computed images, do you feel a difference between the handmade images and machine made images?

machine made images are much more precise

I think the machine made images are much more precise. Even though you can be really precise with a handmade image, there is still always the sense that there’s a sort of unexpressed dimension to that image, that there could have been something else that the person had intended to put in there that you are not perceiving. But I feel with computer generated images, that’s just what the computer generated and that’s all there is. Even though there may be something that the person who was generating the image had intended to put in there, you don’t really get a sense that there is more to that picture than what you are looking at.
But let me say another thing about math because I just said “no.” I think there is a distinction, at least among computer scientists, where some people are able to see math very easily and can visualize it very easily. I am not one of those people – I much prefer presentations when they show me a visual representation of what they are trying to do because then I can immediately grasp what they are trying to do. But there are also people that just put up equations and everybody in the audience says “o.k., I know exactly what you are talking about.” I am not one of those people.

What about images in maths?

The geometry of the space doesn't come through when I just see the math equation.

If it is about geometry of the space, the math is intending to represent an image that you can’t actually draw in two dimensions. But to me that doesn’t come through when I just see the math equation. For some people math is an intuitive language to them. They see the equations and that immediately generates an image inside their brain, as if they are looking at an image. At least this is my hypothesis of what happens in their brains.
But for me, I see the math and it takes me a very long time to draw that image in my brain. When I am in a talk there is not enough time to do that. But when I am working on math myself I draw examples of what math is saying. So I can visualize it in my head. I always wanted to be that kind of person who would be able to see that right away but I don’t see it. Sometimes it’s frustrating.

A further question about some borders in math or even a border-less factor: How do you feel about infinity? How do you work with?

Infinity: The concept is that it goes on forever but in reality it’s a sort of truncated in my head.

I don’t generally work with that concept in the math that I do. But I feel the same way about that I do when people just talk about these abstract geometry systems. It’s hard for me when I can’t visualize what I am thinking about. When you work with something for a long time it becomes familiar to you. Depending on what mathematical concepts or structures you work with day to day – even if those are difficult to visualize – you eventually come up with a way to think about it in your head. I am not sure, if I worked with infinity more often, how I would feel about it. But a lot of we do with infinity is just to say well, if you are taking some series that goes on to infinity, or some procedures that will go on forever, at some point it’s going to converge, at some point it will collapse to a single number. So I generally think that there is some finite place where I can understand the concept. The concept is that it goes on forever but in reality it’s a sort of truncated in my head.

A further factor that makes me problems is that in short words the images scatter into data. In economy they use the notion of content for this phenomenon. In a certain way images equalized in a certain kind because they became content. You just mentioned that you have a certain preference for visualizations. How do you feel this process that more and more things become content?

managing automatically large amounts of data and presenting the relevant information in a meaningfull way by pictures or summaries of data

the information overload: you can’t see the forest for the trees

That’s exactly why I am in the field of data mining, because there are tons of data stores, data warehouses, computers that have huge amounts of memory. All these sciences that are storing lots and recording lots of data have the problem that there is so much of it that it becomes impossible to interpret it anymore. So the whole field of data mining is geared towards developing algorithms, visualization tools and models. They can help people interpret that those large stores of data. How I feel about that is that I don’t think that people have thought through the ramifications of their decisions to take the world in this direction. So people right now might say “I get too much spam emails, it’s annoying, I have too many files on my computer, I can’t find anything.” It’s an annoyance right now but I think in ten years, in twenty years if we keep on this trajectory we will be so far gone, that people will say “whoops, I can’t believe we let this happen and didn’t do anything." So being able to automatically manage those large amounts of data and present the relevant information to people in a meaningfull way is often to use pictures or summaries of the data. In math we have methods to do that, so people are progressing.
Because we have such an information overload, people won’t be able to communicate and we won’t be able to progress in science – it’s like the concept that you can’t see the forest for the trees, that’s how I feel about where we’re going.

This is because you need images to navigate in the forest?

navigaton in the forest without a map

I think so, yes, because if you had a map you would know how to navigate in the forest. But no one is trying to make those maps anymore. It seems like it’s not a problem because you could always climb on the tree to look over the top. But as the forest gets more and more dense, or taller, or whatever, we might not be able to do that any more.

What about the images that are more immaterial inside yourself, your imaginations?

In your imagination you don't always get a perfect image on the first try.

When we are trying to visualize information or models there are often many iterations that we go through. We try to visualize in a particular way and it doesn’t work for one reason or another but it communicates part of the information that we wanted to communicate. Then we try it again with another image to show some other aspects of the data. In a way it’s very similar to what you are doing in your own head when you’re trying to imagine something. In your imagination you don't always get a perfect image on the first try. Maybe if I view it in one way I still can't understand how this aspect works so then I visualize it in another way. I think it could be similar.

So you use images for the communication with other people?

we use images to communicate

Yes, we use them to communicate with other scientists in our lab, sometimes other people in our department and also people from companies, government, corporations--we're often analyzing their data so we use images to communicate the ideas we found back to those people.

How the web changed your conception of images? What did it change?

The reason I started in computer science was to be able to look at information on the web and understand the relationships.

I am not sure that it did. That's a reason why I became a computer scientist and started doing this. I don’t think I really thought about the issue before I became a computer scientist. The reason I started in computer science was to be able to look at information on the web and understand the relationships. It’s hard for me to think back to when the web didn’t exist and how I thought about things at that point. I am not sure I thought about them at all, so it's hard to say.

So it was the mystery of the web that brought you to computer science ?

Everything is a network.

Not so much that it was a mystery but that it was a huge amount of data that was interconnected. There is clearly semantic meaning in the way that it’s interconnected and you can’t view the data all at once. So it has motivated me to look at these very large data sets where the link structure itself is informative. I wanted to be able to learn models and extract patterns from the data not only to understand a data source that is so large that no one person will be able to look at but also to understand more about the way humans interconnect with each other. The web is just one example of a big social data set. That’s what originally motivated me. There are many other types of networks that we look at. A lot of social networks represent people that are interconnected. Also computer networks – there is the web, that is a network of web pages, but underneath is the internet which is a network of computers. And if you think of biological systems it’s a network of their genes and the interactions of proteins that those genes manufacture. Everything is a network.

How do you think about the individual being in such a network?

That’s what’s exciting about being in the real world.

Do you think, that the individual also dissolves in those networks? I am asking a little bit about the implications of the network models for your self-comprehension. Is the individual a node?

The key idea for me is that nothing is independent anymore.

Even though it is very difficult to understand things when you look at all inter-dependencies .

It depends on what level of granularity or abstraction you are looking at the world with. Definitely the individual can be a node in the network. I think the key idea for me is that nothing is independent anymore, but it never was, people only viewed it that way to try to understand it scientifically. We have made enough progress in the last few decades and now we understand simple problems when things are in isolation. But nothing in fact is in isolation in the real world. Even though it is very difficult to understand things when you look at all inter-dependencies among them – that’s when it becomes most interesting to me. Whether it's in the field of psychology or physics or astronomy or computer science or antropology – it’s all about how things are interconnected. But the difficulty is that it becomes very complex. How to visualize the model, the behavior, the function or the form of what’s happening in the world, when you can’t isolate a particular thing and then look at it - that’s what’s difficult.

Let’s have a look on music under network conditions. How do feel music as abstract formal relationships of tones?

There are things in the music that relate to each other.

Notes have relations to each other, it’s a type of sequence. Maybe you play many notes at the same time, but they are also changing over time, that’s really the same thing. But also if I just think about listening to music you can think that the instruments are relating to each other in particular ways as they are playing the music. There are things in the music that relate to each other, things being events. The music itself is trying to tell a story depending on how much imagination you add on top of the music. When I listen to it, I get the sense that it’s not that there are people in the music and things are happening to them but it’s as if those things are just happening to abstract entities in the music. I don’t actually imagine the people, but I can imagine the feelings. It doesn’t evoke a specific image to me but I feel the things that I would have felt if I had seen the image.

That what you are working on you can interpret as a certain score that you can visualize. What happens when you audify the score in a certain way?

web site diagram

audification in data mining:we would be able to use sound representations to find patterns as easily as if we had presented things to people visually. 

There is somebody in data mining that has tried to do that. They have actually tried to take activity that monitors data and encode it in an audio form so that people can hear the patterns. Actually he took web traffic on a web site and had different tones indicating how many times people were clicking on different parts of the web sites. I think he used different loudness to indicate how often people were clicking and then the deepness of the tone indicating how far down they were in the web sites, from the homepage to four/five levels deep. They did that to try to help the web designers get a sense, over the course of a day, because people’s activities levels change over the day, to get a sense of how people were using the site and where they could improve navigation. He played the recording for me and you really could tell that there was a lot of activity on the surface and then you could hear that there was maybe one person that was going down in because there was a sort of a steady “boom boom” meaning that he was deep in the website but not really active. If we thought about that more carefully we would be able to use sound representations to find patterns as easily as if we had presented things to people visually.

Final question: It’s a personal question. What’s your favorite picture at home?

My favorite is a Matisse line drawing.

My favorite is a Matisse drawing, it’s just a line drawing. It’s a black back and a white line that outlines a shape of a woman hugging her knees. I don’t remember what it is called, but that’s my favorite.

interview from 2 February 2005 at Schloß Dagstuhl

Links for Jennifer Neville:
Jennifer Neville/Knowledge Discovery Laboratory/ University of Massachusetts Amherst