IBM's 'Watson' on Jeopardy

[quantumfireball2099]

Sorry if this has been posted already. This is pretty damn amazing.

 

[Light]

This is a game changer for NLP. I've never seen something like this.

Hell, the way they explain it is language processing forms the method by which Watson creates his (its?) knowledge base, or at least how he uses it.

It never even crossed my mind that it would be so key in this way. Before now I always considered it to be a purely interface related concept.

Also, what's up with his voice? Are they doing voice synthesis or are they faking it for the benefit of the show? I'm thinking the latter, given the comically random wagers he gives. Though they could just be actually random.

Do I care that I considered the pronoun "he" before "its"? No. No I do not.

Nor do I care that I continue to use it.

 

[MineMineMine]

the size of watson physical computer makes the brain somehow even way more impressive.

and watson is pretty cool ^-^

 

[Gunboat Diplomat]

I found the match fascinating and I'm sure there will be plenty of powerful applications for Watson...

However, despite appearances, I still don't think Watson demonstrates any sort of intelligence. You can see how mechanistic it is and its lack of any sort of understanding by some of the mistakes it has made. Judging by their (admittedly basic) description of how it works and seeing how it played, it looks like it does something similar to Google's PageRank algorithm with what it believes to be the relevant words in a question. As it turns out, the results are usually correct but, unfortunately, they still don't represent intelligence...

For example, when asked to name an American city with airports named after a World War 2 hero and World War 2 battle, it guessed Toronto... which, as we all know, is not an American city. Watson didn't know this? No it didn't because, in a very real sense, Watson doesn't "know" anything! There just weren't enough associations between World War 2, airports and cities in the bodies of text used to build Watson's database...

One incident in the games that I found interesting was how weak Watson was in the "actors who direct" category compared to its human opponents. I believe what happened here was that this was a particularly easy subject for the humans. If you recognized the movies at all, and they were all famous movies, it's trivial to know who the director is and that the director was an actor was irrelevant. Watson, on the other hand, knows nothing. No question is easy (or hard) for Watson. It simply processed those questions just as it would any other. Because of this, the humans could, for that category, beat Watson in speed. For most other Jeopardy questions, they had to think harder since, for humans, they were harder questions while Watson's ranking algorithm worked at the same speed it always did which, as it turns out, is generally faster than a human's...

Finally, I don't think Watson ever made random bets. I'm sure all of the peculiar wagers it made were optimal according to some Jeopardy oriented game theory. Its bets seem weird to us because we don't analyze game theory as deeply or accurately as it does... mostly because we're not calculators...

 

[MineMineMine]

For example, when asked to name an American city with airports named after a World War 2 hero and World War 2 battle, it guessedToronto[... which, as we all know, is not an American city.

i actually didn't know that.^^



Not 'knowing' is the point of watson and they could've chosen to feed watson also some dictionaries which would've helped him in actors who direct category but beaten the point of it. (i think)

The purpose of watson is not to recall old information but to process new information. And it's different from google's pagerank because it tries to do some kind of semantical analysis of data, something which google does not.


In computer science there are generalized four categories considered:
a machine that thinks like an human
a machine that acts like an human
Systems that think rationally
Systems that act rationally

AI: A Modern Approach is the book you want to read if you are interested in AI.
The introduction is here online: http://www.cs.berkeley.edu/~russell/intro.html

 

[borrofburi]

However, despite appearances, I still don't think Watson demonstrates any sort of intelligence. You can see how mechanistic it is and its lack of any sort of understanding by some of the mistakes it has made. Judging by their (admittedly basic) description of how it works and seeing how it played, it looks like it does something similar to Google's PageRank algorithm with what it believes to be the relevant words in a question. As it turns out, the results are usually correct but, unfortunately, they still don't represent intelligence...

Intelligence is rather hard to nail down: what is it? How do you know what we do to question jeapordy answers via processes similar to page rank? We certainly sort lists in a much more pathetic way than computers do...

But I grant you that this makes Watson look like you could hold a conversation with it on Shakespeare or whatever, and you really couldn't. I also grant you that intelligence is hard, we seem to have all the components of it, but can't quite combine them right to make actual intelligence... I just don 't want to discount the impressiveness of its ability to parse jeapordy answers.

For example, when asked to name an American city with airports named after a World War 2 hero and World War 2 battle, it guessed Toronto... which, as we all know, is not an American city. Watson didn't know this? No it didn't because, in a very real sense, Watson doesn't "know" anything! There just weren't enough associations between World War 2, airports and cities in the bodies of text used to build Watson's database...

Although... we have the same problem. I couldn't answer the question either, because there isn't enough information in my brain about world war 2, airports, and cities...

One incident in the games that I found interesting was how weak Watson was in the "actors who direct" category compared to its human opponents. I believe what happened here was that this was a particularly easy subject for the humans. If you recognized the movies at all, and they were all famous movies, it's trivial to know who the director is and that the director was an actor was irrelevant.

But isn't this also a case of watson not having the database information that we humans have? We have a database that is built off our own experiences, and viewing movies is a very long time experience (as opposed to, say, learning that laguardia is in NYC (unless you've been there of course)), so it makes sense that we're likely to have this stored... Watson on the other hand didn't spend 2 hours, decide the movie was good, and want to learn more about it (perhaps in hopes of finding similarly good movies), and thus might not even have that information let alone those associations..



I'm probably reacting a bit harshly. It seems every time we have an impressive advance of technology in AI we're told "oh, that's very nice honey, but it's not very... intelligent..."

 

[Gunboat Diplomat]

For example, when asked to name an American city with airports named after a World War 2 hero and World War 2 battle, it guessedToronto[... which, as we all know, is not an American city.

i actually didn't know that.^^

Did you even watch the matches? It was surprising enough to be noteworthy!

Not 'knowing' is the point of watson and they could've chosen to feed watson also some dictionaries which would've helped him in actors who direct category but beaten the point of it. (i think)

The purpose of watson is not to recall old information but to process new information. And it's different from google's pagerank because it tries to do some kind of semantical analysis of data, something which google does not.

I disagree with your characterization here. The whole point of Watson is to answer questions from a body of knowledge. The questions posed to it are to be in a natural language but it gets its information from a database: what I think you would call "old information..."


I'm sorry for the edit but I just realized that I had more to say on the subject in regards to your post...

I'm guessing you didn't get around to actually watching any of the matches? According to the program, Watson finds a long list of words associated with the words in the question and ranks them as possible answers according to some obviously sophisticated algorithm. In this regard, it's very much like Google's PageRank...

Finally, have you seen some recent YouTube advertisements? There is one out by Google itself that seems to suggest that Google is now doing natural language processing, much like Microsoft's Bing...

 

[Gunboat Diplomat]

Intelligence is rather hard to nail down: what is it? How do you know what we do to question jeapordy answers via processes similar to page rank? We certainly sort lists in a much more pathetic way than computers do...

But I grant you that this makes Watson look like you could hold a conversation with it on Shakespeare or whatever, and you really couldn't. I also grant you that intelligence is hard, we seem to have all the components of it, but can't quite combine them right to make actual intelligence... I just don 't want to discount the impressiveness of its ability to parse jeapordy answers.

Intelligence is difficult to define but, like many other things whose definition is difficult to specify, it's often clear what it is not...

I don't mean to belittle the accomplishment that is Watson. PageRank was not easy and neither is Watson! I guess I was just disappointed by its performance when I realized it was not what I was hoping it would be...

For example, when asked to name an American city with airports named after a World War 2 hero and World War 2 battle, it guessed Toronto... which, as we all know, is not an American city. Watson didn't know this? No it didn't because, in a very real sense, Watson doesn't "know" anything! There just weren't enough associations between World War 2, airports and cities in the bodies of text used to build Watson's database...

Although... we have the same problem. I couldn't answer the question either, because there isn't enough information in my brain about world war 2, airports, and cities...

...but we don't have the same problem. Yes, neither you nor I could answer that question. That's not what disappointed me about Watson. While we couldn't answer the question, neither you nor I would have guessed Toronto as a possible answer because we know that Toronto is not an American city. Watson could not eliminate it as a possible answer because it doesn't actually think. It does some sort of statistical analysis of the relationships between words in bodies of text. It doesn't "know" anything about Toronto in the same way that we know things. It just so happened that in the text used to build its database, its algorithms didn't rank any American city higher than Toronto so that 's what it guessed, without any notion that it was already disqualified as a valid answer...

One incident in the games that I found interesting was how weak Watson was in the "actors who direct" category compared to its human opponents. I believe what happened here was that this was a particularly easy subject for the humans. If you recognized the movies at all, and they were all famous movies, it's trivial to know who the director is and that the director was an actor was irrelevant.

But isn't this also a case of watson not having the database information that we humans have? We have a database that is built off our own experiences, and viewing movies is a very long time experience (as opposed to, say, learning that laguardia is in NYC (unless you've been there of course)), so it makes sense that we're likely to have this stored... Watson on the other hand didn't spend 2 hours, decide the movie was good, and want to learn more about it (perhaps in hopes of finding similarly good movies), and thus might not even have that information let alone those associations.

I find it highly unlikely that the human players saw most of the movies in that category. However, all those movies were highly advertised and all the actors who directed those movies were very famous. This is what made them easy questions for the humans but there's no such thing as an easy (or hard) question for Watson. It crunches the numbers for that question just like it would any other...

I'm probably reacting a bit harshly. It seems every time we have an impressive advance of technology in AI we're told "oh, that's very nice honey, but it's not very... intelligent..."

You're not reacting harshly at all. I share your frustration when something amazing was done and someone else is unimpressed by it. I assure you that I recognize that Watson is a great accomplishment. I'm confident that it will have a great impact in many industries and it's certainly something I could not do myself. I guess it just wasn't what I was hoping for...

 

[borrofburi]

Ah, that makes more sense. I still think watson "thinks" more than you give it credit for. I think its algorithm for association is probably quite similar to what humans do in that it literally just associates words; which is a nice change over past NLP with its attempts at defining rigid ways to parse sentences. This is partially a natural result of jeapordy where the answer is given and relies heavily on word play.


On the topic of watson, here's a great (if a bit older) article on how it works on a slightly deeper level.... well... I couldn't find it. It was on wired.com at one point, but I can't seem to find it anymore. Oh well, it wasn't that much deeper.

 

[aaa]

Just to let you guys know... there are 7 places in the U.S. that are named Toronto
http://en.wikipedia.org/wiki/Toronto_(disambiguation)

 

[Pulsar]

I was surprised at how quickly I anthropomorphised Watson and gave 'him' a personality. I rooted for 'him' and was disappointed when 'he' gave a wrong answer. And I loved the "I'l wager 6435 dollar" bit

It would be even more impressive if the computer would use speech recognition instead of getting the questions in text format. I'm sure that will be one of their next projects.

PS. Is it me or does Watson sound a bit like NonStampCollector?