Think Vitamin » APIs http://thinkvitamin.com A blog about the web Thu, 09 Sep 2010 01:38:53 +0000 en hourly 1 http://wordpress.org/?v=3.0.1 How to Get Started with the Twitter API http://thinkvitamin.com/dev/how-to-get-started-with-the-twitter-api/ http://thinkvitamin.com/dev/how-to-get-started-with-the-twitter-api/#comments Thu, 20 Aug 2009 06:52:58 +0000 Kari Pätilä http://carsonified.com/?p=2980 Kari Pätilä
Thanks to some very nice open source libraries for quite a few programming languages, interacting with the Twitter API has become exceedingly simple. In this article we’ll be looking at different ways to pull in data from Twitter. The libraries page of Twitter’s API wiki is a good place to start. For these examples I’m [...]]]>

Kevin and Gary show at FOWA London

Thanks to some very nice open source libraries for quite a few programming languages, interacting with the Twitter API has become exceedingly simple. In this article we’ll be looking at different ways to pull in data from Twitter.

The libraries page of Twitter’s API wiki is a good place to start. For these examples I’m going to use the php-twitter class, but I’ll include the requests and responses so this doesn’t turn out to be all PHP. After all, the API itself doesn’t care which language I’m using.

The php-twitter zip archive contains some nested folders and finally a file: class.twitter.php. You’ll eventually get to it, so keep opening those folders!

Getting your timeline from Twitter

With this library you don’t have to worry about data exchange formats (the default is JSON), but if you’re still stuck with PHP 4, you might have to set the $type variable to ‘xml’. But, let’s face it: if you enjoy XML you’re kind of insane, aren’t you? If your host is still using PHP 4, chances are that they’ll update you to the latest version if you ask them nicely.

The API method we’ll be using is user_timeline. You can check out the output by pointing your browser to http://twitter.com/statuses/user_timeline/karipatila.xml. Notice the “.xml” — that’s the data exchange format. You can use “json”, “rss” or “atom” there as well. That URL is the GET request the API is responding to, which could be made using any programming language.

So, using the library, this is all we need to fetch my timeline:

<?php
header('Content-type: text/html; charset=utf-8');
require_once 'class.twitter.php';
$t = new twitter;
$data = $t->userTimeline('karipatila');
?>

First we make sure the script can handle unicode characters and then we include class.twitter.php. Next we’ll set up a new instance of the php-twitter class, which can be used to pull in some data.

The script loads the timeline from Twitter and stores it into the $data variable. Along with some user information it contains the tweets themselves, which are what we’re really interested in. Let’s take a look at what kind of information we’re getting. This is one of my tweets in the $data array: 

[truncated] => 
[text] => TomTom Nordic, 69,90€: http://bit.ly/wX51t (App Store link) - wonder how expensive the car kit is?

[in_reply_to_status_id] => 
[created_at] => Mon Aug 17 07:55:51 +0000 2009
[favorited] => 
[in_reply_to_user_id] => 
[id] => 3358348095

[in_reply_to_screen_name] => 
[source] => <a href="http://www.atebits.com/" rel="nofollow">Tweetie</a>

We can use that information to make a script that lists my recent tweets. The API method statuses user_timeline returns the 20 most recent statuses posted from the authenticating user. I also wanted to link back to the accounts that are being replied to, so I’m looking for an @ symbol followed by one or more word characters and linking them to their respective Twitter profiles: 

<ul>
	<?php foreach($data as $d){ ?>
	<li><?php echo preg_replace('/(^|\s)@(\w+)/','\1<a href="http://twitter.com/\2">@\2</a>', $d->text); ?></li>

	<?php } ?>
</ul>

A short note on rate limitations; caching the output from this script will become necessary sooner or later. The technical details are beyond the scope of this article, but you might want to consider saving the output into a database or a file and checking for updates every couple of minutes or so. A limit of 150 requests per hour applies to the REST API.

Get the source for fetching a user’s timeline here.

Let’s look for treasure

For example, let’s do a search for the hashtag “#carsonified”. They have recently announced Hello App, so there should be some chatter around that subject. We can add terms to the search like this:

$data = $s->search('#carsonified OR #HelloApp');

The search API method requires the query to be URL encoded, so we replace # with %23 in the request: http://search.twitter.com/search.json?q=%23carsonified&rpp=5&page=1. The search API accepts either “json” or “atom” as the exchange format.

If we break that request down, we find it contains the following information:

q=[query string]
rpp=[results per page]
page=[page number]

The search API has virtually no rate limits, so you don’t have to worry about your app getting throttled. Note that this time we’re also creating another instance with

$s = new summize;

which refers to the search class found in the php-twitter library.

<?php
header('Content-type: text/html; charset=utf-8');
require_once 'class.twitter.php';
$t = new twitter;

$s = new summize;
$data = $s->search('#carsonified');
$data = $data->results;
?>

In this example one tweet stored in $data might contain the following information:

[text] => Just trying out Matt by #carsonified. uniquely designed site with workable UI.
[to_user_id] => 
[from_user] => _midnightshad
[id] => 3284756132
[from_user_id] => 2270963

[iso_language_code] => en
[source] => <a href="http://themattinator.com" rel="nofollow">Matt</a>
[profile_image_url] => http://a1.twimg.com/profile_images/60040548/panda_normal.jpg

[created_at] => Thu, 13 Aug 2009 11:50:01 +0000

We can use this to make a simple listing based on the results:

<?php
header('Content-type: text/html; charset=utf-8');
require_once 'class.twitter.php';

$t = new twitter;
$s = new summize;
$data = $s->search('#carsonified');
$data = $data->results;
?>
<ul>

<?php foreach($data as $d){ ?>
	<li>
		<img src="<?php echo $d->profile_image_url; ?>" alt="" />
		<?php echo preg_replace('/(^|\s)@(\w+)/','\1<a href="http://twitter.com/\2">@\2</a>', $d->text); ?>

		<em>by</em>
		<a href="http://twitter.com/<?php echo $d->from_user; ?>"><?php echo $d->from_user; ?></a>
		<?php echo $d->created_at; ?> <em>from</em> 

		<?php echo html_entity_decode($d->source); ?>
	</li>
<?php } ?>
</ul>

This script outputs the tweets containing the word #carsonified along with profile images, timestamps and links to the client used.

Screenshot of We Love Typography's listing of tweets containing the hashtag #WLT
We Love Typography uses similar formatting to pull in tweets with the hashtag #WLT.

In the following source file I’m using a function for formatting the created_at field, so instead of “Fri, 14 Aug 2009 14:24:58 +0000″ you will get something like “3 days, 17 hours ago”.

Get the source for searching Twitter here. You’ll also need the file with the time_passed function for this one.

When the library just isn’t good enough

The search API returns 50 results by default, which might not be convenient for everyone. The class php-twitter doesn’t set the results per page variable in the query strings, so let’s add that. You need to open the class.twitter.php file and replace line 844 with this:

function search( $terms=false, $rpp=false, $page=1, $callback=false )

Finally we add these three lines starting from line 854:

if( $rpp )
$qs[] = 'rpp=' . $rpp;
$qs[] = 'page=' . $page;

After these changes:

$data = $s->search('#carsonified', 5);

would only return the five latest results, and it would set the pagination to page one, whereas

$data = $s->search('#carsonified', 5, 2);

would set it to start from page two.

Know your API

The API Documentation is pretty well done, so make sure to read it. Check out the methods like searching for current or daily trends or listing your followers and go make the next big Twitter app!

]]> http://thinkvitamin.com/dev/how-to-get-started-with-the-twitter-api/feed/ 31 Go Forth and API http://thinkvitamin.com/web-apps/go-forth-and-api/ http://thinkvitamin.com/web-apps/go-forth-and-api/#comments Sun, 02 Jul 2006 07:55:41 +0000 Cameron Adams http://www.thinkvitamin.com/features/ajax/go-forth-and-api Cameron Adams
To most, the virtues of Web 2.0 are rather ephemeral; that’s always been one of its main criticisms. However, I like to think that one of the movement’s key aspects is a sense of community, an ability to create sites and applications that bring people together. One of the most powerful ways to bring people [...]]]>

To most, the virtues of Web 2.0 are rather ephemeral; that’s always been one of its main criticisms. However, I like to think that one of the movement’s key aspects is a sense of community, an ability to create sites and applications that bring people together.

One of the most powerful ways to bring people together on the Web is to give them ownership of the sites that they frequent – make them active participants in the development and growth of the product they’re using. There’s many different approaches you can take to implementing this idea, but I’m going to talk about just one of them: setting your data free.

Data sharing isn’t a new idea. It’s a concept as old as the Web, in fact, the Web is all about sharing data. However, most of the data available on the Web is designed to be consumed by humans, through a browser. It’s encapsulated by HTML pages, trapped inside Flash files, and disguised by design. There used to be a time where, if you wanted to get any real data from a site that wasn’t your own, your only recourse would be to grab the raw HTML and scrape through that tag soup for the gem that you were looking for. Fancy writing a 15-line regular expression to get the weather off Yahoo!?

However, data providers are slowly realising that offering their data in much more flexible formats can only be an advantage to them – they get used by more applications, they get seen by more eyeballs, they get more exposure.

RSS is a great example of this. Strip all the images, colour, layout, font styles and browser dependence from a Web page, and what do you have? A lightweight, versatile data format that can be easily interpreted by pretty much any software out there. Two years ago it was barely on the radar. Now your green grocer has an RSS feed that tells you when the latest shipment of fresh guava has arrived.

RSS is good, but it’s still a bit restrictive for real application development – the content provider tells you exactly what you’re going to get, and when you’re going to get it. What if, instead of getting the last 10 blog posts from your favourite incendiary author, you could get their last 10 posts that were tagged with “free beer”; or you could get their last 10 posts that were tagged with “free beer” and were written in Melbourne?

What if you could take their last 10 moblog posts that were tagged with “free beer” and written in Melbourne, get the geolocation data of each post, plot the exact position of each free beer spot on a street map and show it in relation to your current location? That’s a good night out. And APIs can let you do it.

Application Program Interfaces are, broadly speaking, tools for building software applications. An operating system API might help you build applications with a consistent interface, but Web APIs are mostly about giving you access to data.

Even Web APIs aren’t a new idea. Google’s search API has been available via SOAP since 2002, and there’s definitely older services than that. However, the recent growth in Web API availability has been fuelled by two recent developments. The first, which I’ve already mentioned, was a philosophical change in the way that data is handled. The second was the introduction of AJAX. Again, not a new idea, or even a new technology, but sometimes it’s all about timing.

Some of the more interesting JavaScript-friendly Web APIs that are available at the moment are:

The mapping APIs have received the most attention to-date, mainly through mapping “mashups” such as chicagocrime.org, or dartmaps.mackers.com. They’re also probably one of the least flexible, because you are constrained by the mapping interface that the provider gives you. When you include the Google Maps API on your page, you gain access to a number of JavaScript functions that allow you to move and zoom the map, or add your own data and visual components, but you aren’t able to manipulate any of the original data, beyond what the existing JavaScript methods allow you to do. This is perhaps just a constraint of maps themselves – how many different forms can mapping data take? A good tutorial on how to create an application with the Google Maps API is available at Hacking Maps with the Google Maps API.

Other forms of data, however, offer greater possibilities.
These web applications use AJAX to query the various non-map APIs and create new and unique ways to interact with data:

We’ll be taking a look at the how they access those APIs, so you can get right in and make your own data-sharing application.

Accessing Web APIs with AJAX

Most Web APIs are accessed via HTTP, so they’re always going to be available to server-side scripts, however, the nexus that AJAX enables between user interaction and data communication, as well as the ease with which it can be incorporated into webpages, means that the use of APIs by client-side scripts offers some of the most exciting possibilities for Web applications.

The API for Flickr has a number of methods that are specified by passing particular CGI parameters to a base script. For instance, to get a list of the most recently uploaded photos to all of Flickr, you would access their script:


http://www.flickr.com/services/rest/?method=flickr.photos.getRecent&api_key=XXXX

The “method” specifies exactly which method from the API you want to execute, and “api_key” is a way for Flickr to track who’s using what from their API. If you want to see the data that this call returns, simply sign up for your own Flickr API key, insert it into the string above, paste that string into your browser location bar, and you’ll get a nicely formatted XML file in return. Something like:

The same process applies for an XMLHttpRequest via JavaScript – you specify the URL above to your XMLHttpRequest object, and the client browser will go off and fetch it. (For a more in-depth look at creating XMLHttpRequest connections, see Remote Scripting with AJAX) Because you get XML in return, you’ll be able to use JavaScript’s in-built DOM functions to parse the data in much the same way as you do for HTML; navigating through elements, getting attributes, etc. However, as with any XMLHttpRequest, the data doesn’t necessarily have to be XML, so a Web API could send back a simple text string instead. In fact – as we’ll see later – that’s what JSON does.

XMLHttpRequest Proxies

Even though it’s theoretically possible to request this data from Flickr using XMLHttpRequest on the client-side, there’s one problem: cross site scripting. Currently, most browsers will not allow you to get data from other sites using XMLHttpRequest. Due to security restrictions that prevent cross site scripting (XSS), they will not let JavaScript access a URL that is outside of the domain that the current page is being served from. This even applies to sub-domains on the same site, for example www.example.com versus dev.example.com.

This is not an intractable problem, though. There are a couple of ways you can get around it, but they all require the server on the current domain to act as a proxy for the external data. This works because server-side scripts generally have more privileges than client-side scripts, so they are able to access any address on the Internet. With a server-side proxy, we create a script that accesses a remote server, gets the data and relays it back to the browser, making it look like the data was sent from www.example.com, when in reality it came from www.upcoming.org, or some other website which we proxy data for.

Apache Proxy

The easiest way to proxy data is to get your web server to do it for you. You can set up a proxy in Apache by making sure the mod_proxy extension is loaded and editing the httpd.conf file to include your new proxy:


ProxyPass /proxy/flickr/ http://www.flickr.com/
ProxyPassReverse /proxy/flickr/ http://www.flickr.com/

The first line forwards any requests from your server to Flickr’s, the second line handles any redirections that the Flickr server might respond with.

However, if you’re running a site off a shared server, you more than likely won’t be able to edit httpd.conf, so you can always use mod_rewrite to redirect requests by editing a .htaccess file. If I put this inside the .htaccess file for the directory /proxy/flickr/:


RewriteEngine on
RewriteRule ^(.*)$ http://www.flickr.com/$1 [P]

Anytime I make a call to a path inside http://www.example.com/proxy/flickr/, it will pass on all extra URL information to http://www.flickr.com/. So, when I reference http://www.example.com/proxy/flickr/services/rest/ it’s actually passing the request through to http://www.flickr.com/services/rest/. The [P] at the end of the RewriteRule means that the re-direction is passed through without informing the user’s browser that a change of URL has occurred.

Application Proxy

The second method to proxy data is to create a script that will do it for you – receive request URLs, create a connection, get the data, and return it to the requester. The exact code that you use to do this will differ depending upon what language you’re using, but if you are running PHP with the Curl extension installed (for creating external connections), then you could use this as your proxy script:


$remoteServer = "http://www.flickr.com";
$path = $_GET[”path”];
$proxyTarget = $remoteServer.$path;

$connection = curl_init($proxyTarget);
curl_setopt($connection, CURLOPT_HEADER, false);
curl_setopt($connection, CURLOPT_RETURNTRANSFER, true);
$data = curl_exec($connection);
curl_close($connection);

header(”Content-Type: text/xml”);
echo $data;

This script assumes that you are passing a CGI parameter called “path” that specifies what location you want to retrieve from the remote server. Note that we don’t directly pass on this parameter to the Curl connection, as we don’t want this script to become an open proxy. Instead, we specify the exact address that we are using the proxy for – www.flickr.com – and this will limit its potential for abuse. You’d call it from a browser using this URL:


http://www.example.com/proxy.php?path=/services/rest/

If you’re sending that over an XMLHttpRequest connection, you should remember to URL endcode it using escape(), so that any special characters inside the path are passed along properly.

When we initialise the Curl connection, we tell it that we don’t want it to pass on any header information, but we do want it to pass on the content of the request. Then, once the data has been received we write the appropriate header back to the client browser and also forward on the data which we retrieved from the remote server.

The downside of proxies is that you’re double handling your data every time – a request has to come to the current domain server, which then passes it along to the actual remote data server. Then the data has to go back to the current domain server before it can be sent to the client browser. This extra traffic will always introduce some delay; the amount of which depends on the network architecture and speed of the highway at any given time, so you’ll have to ask whether that delay (however minimal it is) is acceptable to your service.

JSON

Another way to avoid XSS issues with XMLHttpRequest is to not use it all. How do you do that? You use JSON.

JSON (short for JavaScript Object Notation) is an alternative format to XML. It actually stores data as an object literal – a structure that is natively available in JavaScript. So if you receive a JSON string you can use the JavaScript evaluation function eval() and it will automatically create an object structure that corresponds to the data that you want to retrieve. Although object literals are native to JavaScript, they’re farly easy for other languages to parse. Indeed, at json.org there are links to JSON parsers for over 20 different programming languages.

Using eval() on a text string that you receive from a 3rd party can be a little scary – it opens up all sorts of security holes that could be exploited in your code, so you really have to trust the data provider if you’re blindly using JSON. There’s some parsers available that have been written in JavaScript by Douglas Crockford – the creator of JSON – that will offer some protection against malicious code. These are also available at www.json.org.

Douglas calls JSON the “fat free alternative to XML”, mainly because it’s less verbose and requires less parsing than XML, particularly if you’re using it from inside JavaScript. If we translated the XML we had from Flickr earlier into a JSON structure, it might look like this: 

{
	"rsp": {
		"stat": "ok",
		"photos": {
			"page": "1",
			"pages": "10",
			"perpage": "100",
			"total": "1000",
			"photo": [
				{
					"id": "169563197",
					"owner": "98319521@N00",
					"secret": "f4002ac03f",
					"server": "67",
					"title": "turkije021",
					"ispublic": "1",
					"isfriend": "0",
					"isfamily": "0"
				},
				{
					"id": "169563188",
					"owner": "17208601@N00",
					"secret": "fbf107bffb",
					"server": "56",
					"title": "PICT0221~1",
					"ispublic": "1",
					"isfriend": "0",
					"isfamily": "0"
				}
			]
		}
	}
}

It’s entirely possible to use JSON via XMLHttpRequest – make a request to the server, receive a text string and evaluate it – but one of its real advantages is that you don’t need XMLHttpRequest in order to get it. Because JSON is a JavaScript string, if you specify a server-side script that returns JSON as the source for a