What is Star Trek? IBM’s Alchemy Language API gives a dual answer

In case you have not noticed from the multiple TV ads, for a few years now IBM has been positioning itself as a Big Data company, with its Watson platform and cloud-based services.  One of them is the Alchemy Language API, which packs together functions for text analysis and information retrieval.  As part of learning how to handle this API from R, I tried it on a news story about a sci-fi book publishing business.  Overall, the results were strong, although not without some amusing quirks…

Read more

Photo of the day – April 14, 2016

With rainy days finally over, students from the Introduction to Text Mining course (QAC 386) decided to hold the class outside, which they successfully did on the lawn near Allbritton Hall.  The topic of the day was tree parsing using openNLP package in R.

tm2

In the photo, left to right: first row: Trisha Arora ’16, Taran Carr ’16, Antonio Robayo ’16, and Jack Trowbridge ’16; second row: Grace Wong ’18, third row: Sara Eismont ’18.

The hidden power of calendars: history of Oscars told through their telecast schedules

Sundays seem natural for large TV events.  Why wouldn’t they?  NFL’s Super Bowl has been on Sundays forever. It feels like the proper order of things that the Academy Awards ceremony is also on a Sunday. Every year, somewhere near the end of February, start of March.  Yet, a simple dataset of telecast dates points out that this practice is a relatively recent phenomenon and for a long while things were quite different.  For a quick summary of the data, look at the chart below: it shows the progression of the ceremony dates from the most distant (1953) to the closest (2014).  For more details on why the changes occurred, keep reading on.

f1

Read more

How many tweets would your code crunch if it could crunch Twitter, or why holidays are bad for using Twitter’s streaming API

Twitter has emerged as a convenient source of data for those who want to explore social media. The company provides several access endpoints through APIs. There is a REST API for collecting past tweets and a streaming API for collecting tweets in real time. R has libraries for working with both. As is usual in data collection, the catchphrase is “more” – we want more tweets, ideally all that are relevant to our research question. While REST API is rate-limited (a user can submit 180 requests per 15 minutes, with each request returning 100 tweets), the streaming API holds a promise of delivering much more. The nagging question, though, is “how much?”

Read more

FiveThirtyEight’s Riddler #1 – using R to evaluate the answers

Last week a prominent data journalism blog FiveThirtyEight.com has launched The Riddler – a section dedicated to math and probability related puzzles. The deadline for submitting solutions to the first riddle is over and this post will illustrate how you can use R to evaluate potential answers without doing any analytic derivations.

Read more