OK so I know it’s geeky as you like, and to a certain degree pointless… however, for the sake of being a geek and using Twitter too much perhaps, I decided to give this a go, and it turned out into an app that I rather enjoyed playing with (as well as writing it of course), so here it goes. If you’re a howaboutwe user and you’re a heavy Twitter — and most importantly, if you’re a geek!!! — you might find this useful. I might turn this into a fully fledged web app but I’ll have to talk to the folks at howaboutwe about how I can deal with storing user credentials and so on… Who knows, might happen one day — but if you’re not ready to wait for that long, you can customize this one and run it on your machine or a server in the cloud, or even your Raspberry Pi (yup, it works!) and have your tweets turned into dates 🙂

Before I lunge into the technical side of things, a bit of background I think might help: so I was told recently by a friend about this service called howaboutwe — for those of you who don’t know it, it’s just a dating website, where people post date ideas in the format “How about we … go for a walk to xyz” or similar; these dates appear on people’s profile and others interested in similar things will get in touch with a view to transform that idea into a date and physically meet. They have a website and an app to allow you to post these dates so you can do so on the go or from your laptop or whatever. All good so far, right? Now I don’t know where people get their date ideas from — perhaps most of them just wake up and think of it and post it, can’t really comment that much since I’ve only created an account on howaboutwe so I can play with this twitter thing 🙂 But my thinking is: I’m out with my mates, having a drink, watching a game, doing whatever I’m doing — then I realize: actually, I do enjoy doing this sort of thing, so why not post a date idea about it? Great! You pull out the app and post it — done! Trouble is, with my being an avid Twitter user, what I would do in such situations is the following:

• Send a few tweets and photos mentioning the friends I’m with
• Have more fun
• Send more tweets about what I’m doing
• Have more fun
• Tweet again
• Oh wait, this would be a good idea for a date, right, take out the app and post the date as well.

This seems to me a bit of duplication, since quite likely I would have tweeted already about the thing that I’m doing…so why not take one of those tweets and automagically transform it into a date for me? And that’s pretty much how this app started.

It’s a simple command-line app which just monitors a user’s tweets and looks for 2 things:

• a tweet beginning with the words “how about we”
• a hashtag “#howaboutwe”

In either cases, it considers that to be a marker and retrieves the tweet contents, gets rid of the “marker” and posts the text as a date idea on your howaboutwe profile! So when you’re out with your mates watching let’s say the Sharks (hopefully winning! :D) and you’re just tweeting like nuts as Pavelski and Thornton keep having a go at the opposition goal, just tweet something like “Watching Pavelski scoring from a pass made by Thornton #howaboutwe” and you’ll get a date which reads: “How about we … watching Pavelski scoring from a pass made by Thornton“. While the grammar is messed up in the above, I’m sure you’ll agree anyone who looks at that will get the idea — and the bonus is that you only sent a tweet!

With that in mind, I set off to use the twitter4j library — since I am somewhat familiar with it and it’s a nifty little library. Like I said, this is a rather crude app — which, however, might grow into something bigger. For now though, all I had to do is register an app on dev.twitter.com and retrieve the credentials needed for OAuth to authenticate with Twitter. Twitter4J supports a very easy model of configuration (see more details here: http://twitter4j.org/en/configuration.html) but all that’s needed is to store all credentials in a twitter4j.properties file and you’re ready to go!

From there on the steps are pretty straight forward:

1. Retrieve my own timeline: for these I use my own .properties file where I’m saving the ID of the last tweet checked — such that on each run of the app I don’t crawl through all of my tweets (I’ve tweeted quite a lot in my life and that would be a slow process 🙂 ) — also I don’t just rely on checking the latest 20 tweets or so and potentially miss out on some tweets prior to those. So the very first time, I’m retrieving the whole timeline and scroll through the tweets, once I find a tweet that’s meant to make it into howaboutwe (see below as well) I’m saving the ID in the howaboutwetweet.properties in the user home — this gets read every time the app starts, if present, in which case I’m simply retrieving my timeline since that tweet ID.
2. Then for each one of the tweets, scrub out either the starting “How about we” or the “#howaboutwe” hashtag and generate based on what’s left what’s going to be the contents of the date.
3. Then finally connect to howaboutwe using my username and password, and use a HTTP POST to post the date!

This last bit was probably the trickiest — so it deserves a bit more expansion perhaps: while Twitter4J makes it very easy to perform step 1, and step 2 can be easily achieved through standard string manipulation libraries, for the 3rd one I decided to take a more brutal approach and actually simulate a browser session using Apache HttpComponents. I’ve traced basically the calls from Firefox to howaboutwe and reversed engineered what’s happening, and came to this conclusions (I’ve stripped out certain bits):

1. Do a simple HTTP GET to https://www.howaboutwe.com/login — this will send back 2 important cookies it seems: _howaboutwe_session and visitor — these 2 cookies will need to be used in all sub-sequent HTTP calls to howaboutwe:

https://www.howaboutwe.com/login

GET /login HTTP/1.1
Host: www.howaboutwe.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:22.0) Gecko/20100101 Firefox/22.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://www.howaboutwe.com/me?dates=out_of_play
Connection: keep-alive
If-None-Match: “d6b9bda0359897782ba75bc783f7d95e”
Cache-Control: max-age=0

HTTP/1.1 200 OK
Server: nginx/1.2.6
Date: Fri, 28 Jun 2013 23:24:37 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Status: 200 OK
X-UA-Compatible: IE=Edge
Etag: “95c900af857b8370108b5426343350d4”
Cache-Control: max-age=0, private, must-revalidate
Set-Cookie: visitor=7f185aba19f02f98d7772081f6e4333b; path=/; expires=Sat, 28-Jun-2014 23:24:37 GMT
Set-Cookie: _howaboutwe_session=BAh7B0kiD3Nlc3Npb25faWQGOgZFRkkiJTk4NWFjZDA0MGE4ZjY0NjhhN2M5MDM1MDhjZjI2Y2FkBjsAVEkiEF9jc3JmX3Rva2VuBjsARkkiMXRzUEpuYWJ6L0JlWW5VanEyMUwxemd6RDI1b3dMdmtwczdERk1CTFloYkk9BjsARg%3D%3D–21c3a5e1332b45b064ae4443312c56a1cdfdb934; path=/; HttpOnly
X-Request-Id: 0adb85a851d5f15265df1509ed5f0447
X-Runtime: 0.051267
X-Rack-Cache: miss
Content-Encoding: gzip

2. Do a HTTP POST to the same login page, passing the 2 cookies mentioned above and also the username/password — there is a 3rd important cookie being sent back at this point called user_credentials — this needs to be sent back together with the other 2 in the last step:

https://www.howaboutwe.com/login

POST /login HTTP/1.1
Host: www.howaboutwe.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:22.0) Gecko/20100101 Firefox/22.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: https://www.howaboutwe.com/login
Cookie: visitor=0294313d1e3e86f36277a203622c6864; _howaboutwe_session=BAh7CEkiCmZsYXNoBjoGRUZvOiVBY3Rpb25EaXNwYXRjaDo6Rmxhc2g6OkZsYXNoSGFzaAk6CkB1c2VkbzoIU2V0BjoKQGhhc2h7BjoLbm90aWNlVDoMQGNsb3NlZEY6DUBmbGFzaGVzewY7CkkiQFlvdSd2ZSBzdWNjZXNzZnVsbHkgbG9nZ2VkIG91dCAtIGhvcGUgdG8gc2VlIHlvdSBiYWNrIHNvb24hBjsARjoJQG5vdzBJIg9zZXNzaW9uX2lkBjsARkkiJWUzMDJiMzc2OWI3YzNmOTU3ZWZhY2MwOWVlNjFiZmZiBjsAVEkiEF9jc3JmX3Rva2VuBjsARkkiMU1jNHp6bTRhSWRmUTFaMmhIeUVRYjJkQzQyMzE3Tit6aDVPOW5rbjkrUG89BjsARg%3D%3D–402a748c4137b95c78481f4fd401fb517ee3a017; user_ip=76.102.132.82;
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 215
utf8=%E2%9C%93&authenticity_token=Mc4zzm4aIdfQ1Z2hHyEQb2dC42317N%2Bzh5O9nkn9%2BPo%3D&user_session%5Blogin%5D=<mylogin>v&user_session%5Bpassword%5D=<mypassowrd>&user_session%5Bremember_me%5D=0&user_session%5Bremember_me%5D=1
HTTP/1.1 302 Found
Server: nginx/1.2.6
Date: Fri, 28 Jun 2013 23:12:28 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Status: 302 Found
Location: https://www.howaboutwe.com/home
X-UA-Compatible: IE=Edge
Cache-Control: no-cache
Set-Cookie: visitor=d5e3c6abc1aaa8737d1d76165196ce5e; path=/; expires=Sat, 28-Jun-2014 23:12:28 GMT
Set-Cookie: showMobilePrompt=true; path=/
Set-Cookie: user_credentials=5d2dcf45911061674ce097736ff45d29e43a44e622eed5dfa182412b972de82bd465de73d69402f1da36a5fb00f2d0a5033fbd8ff294938737f97ffd52ff1a48%3A%3A1355653; path=/; expires=Sat, 28-Sep-2013 23:12:28 GMT
Set-Cookie: _howaboutwe_session=BAh7CkkiD3Nlc3Npb25faWQGOgZFRkkiJWUzMDJiMzc2OWI3YzNmOTU3ZWZhY2MwOWVlNjFiZmZiBjsAVEkiEF9jc3JmX3Rva2VuBjsARkkiMU1jNHp6bTRhSWRmUTFaMmhIeUVRYjJkQzQyMzE3Tit6aDVPOW5rbjkrUG89BjsARkkiFXVzZXJfY3JlZGVudGlhbHMGOwBGSSIBgDVkMmRjZjQ1OTExMDYxNjc0Y2UwOTc3MzZmZjQ1ZDI5ZTQzYTQ0ZTYyMmVlZDVkZmExODI0MTJiOTcyZGU4MmJkNDY1ZGU3M2Q2OTQwMmYxZGEzNmE1ZmIwMGYyZDBhNTAzM2ZiZDhmZjI5NDkzODczN2Y5N2ZmZDUyZmYxYTQ4BjsAVEkiGHVzZXJfY3JlZGVudGlhbHNfaWQGOwBGaQOFrxRJIhNqdXN0X2xvZ2dlZF9pbgY7AEZU–5e1037c9dfde7b1e66a9a9b527a6af64767e2edc; path=/; HttpOnly
Set-Cookie: user_ip=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
X-Request-Id: aad0222baf2b29023ad0624cbce66463
X-Runtime: 0.044841
X-Rack-Cache: invalidate, pass

3. Finally, having authenticated the user send a POST to https://www.howaboutwe.com/api/users/<username>/dates and post your date idea:

POST /api/users/<your username>/dates HTTP/1.1
Host: www.howaboutwe.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:22.0) Gecko/20100101 Firefox/22.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
X-CSRF-Token: Mc4zzm4aIdfQ1Z2hHyEQb2dC42317N+zh5O9nkn9+Po=
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
X-Requested-With: XMLHttpRequest
Referer: http://www.howaboutwe.com/home
Content-Length: 572
Cookie: visitor=d5e3c6abc1aaa8737d1d76165196ce5e; _howaboutwe_session=BAh7C0kiD3Nlc3Npb25faWQGOgZFRkkiJWUzMDJiMzc2OWI3YzNmOTU3ZWZhY2MwOWVlNjFiZmZiBjsAVEkiEF9jc3JmX3Rva2VuBjsARkkiMU1jNHp6bTRhSWRmUTFaMmhIeUVRYjJkQzQyMzE3Tit6aDVPOW5rbjkrUG89BjsARkkiFXVzZXJfY3JlZGVudGlhbHMGOwBGSSIBgDVkMmRjZjQ1OTExMDYxNjc0Y2UwOTc3MzZmZjQ1ZDI5ZTQzYTQ0ZTYyMmVlZDVkZmExODI0MTJiOTcyZGU4MmJkNDY1ZGU3M2Q2OTQwMmYxZGEzNmE1ZmIwMGYyZDBhNTAzM2ZiZDhmZjI5NDkzODczN2Y5N2ZmZDUyZmYxYTQ4BjsAVEkiGHVzZXJfY3JlZGVudGlhbHNfaWQGOwBGaQOFrxRJIhRjdXJyZW50X3VzZXJfaWQGOwBGaQOFrxRJIhhyZXBvc3RhYmxlc19jb250ZXh0BjsARkkiDmRhc2hib2FyZAY7AEY%3D–c5442e54197af29c7a7cac86f94a3336df1ac6f0; user_credentials=5d2dcf45911061674ce097736ff45d29e43a44e622eed5dfa182412b972de82bd465de73d69402f1da36a5fb00f2d0a5033fbd8ff294938737f97ffd52ff1a48%3A%3A1355653;
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
proposition%5Bsource%5D=dashboard&proposition%5Bdescription%5D=<your date text>&proposition%5Bvenue_attributes%5D%5Bname%5D=&proposition%5Bvenue_attributes%5D%5Bsource_id%5D=0&proposition%5Bvenue_attributes%5D%5Bsource_name%5D=user&proposition%5Bvenue_attributes%5D%5Baddress%5D=&proposition%5Bvenue_attributes%5D%5Bzip%5D=&proposition%5Bvenue_attributes%5D%5Blat%5D=&proposition%5Bvenue_attributes%5D%5Blng%5D=&proposition%5Bvenue_attributes%5D%5Bcity%5D=&proposition%5Bvenue_attributes%5D%5Bstate%5D=&proposition%5Bvenue_attributes%5D%5Bcountry%5D=&proposition%5Bcategory_id%5D=

HTTP/1.1 200 OK
Server: nginx/1.2.6
Date: Fri, 28 Jun 2013 23:15:28 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Status: 200 OK
X-UA-Compatible: IE=Edge
Etag: “e0ce9618d722a89e3ede101ed57af7d3”
Cache-Control: max-age=0, private, must-revalidate
X-Request-Id: dacd0852ee2c5a1cbe5b1730149a1a73
X-Runtime: 0.370072
X-Rack-Cache: invalidate, pass
Content-Encoding: gzip

You can verify for yourself and trace the HTTP connection in Firefox if you use an extension like Live HTTP Headers or similar.

There is one tricky bit which comes into the above flow — you would have noticed the bit highlighted in red concerning authenticy_token — looking closer into this, this is a token generated in the page when you hit the login page first time — it’s then silently passed into the login form in order to prevent cross-site hacking and probably brute force login hacking. All one has to do though, is to actually download the HTML returned by the first HTTP GET to the login page, parse it and retrieve that — I took a more hacky approach to be honest: having noticed this is returned in a <meta name="csrf-token"...>, I’m just using a regex to find that bit in the pile of HTML returned — definitely not the best idea, however, it works 🙂

While howaboutwe seems to have some sort of API, I didn’t find out too much about it and how can you integrate it in an app, hence the brute approach — there’s quite likely a more elegant way of doing it but as you have probably figured out by now, I didn’t spend too much time on this app, so this will do.

With that in mind, you can have a look at the code I’ve put up on github — https://github.com/liviutudor/howaboutwe_link.

I’ve configured the pom so it packages everything into a single jar — using the OneJar Maven Plugin — so once you have configured everything and do a mvn clean package you will get a .one-jar.jar file which you can then launch just by typing java -jar file.one-jar.jar . Bear in mind the application executes the above steps just once then exits — so if you decide to run it, you will have to cron it (I find out running it once an hour is enough but up to you). One can look at the Twitter Streaming API and have this running continuously and receive notifications when new tweets are up and kick in the steps above — but that would be putting together a proper app 🙂

Maybe when I get more time I’ll transform this into a proper web app and allow users to sign up to the service — but I’ll have to check first with howaboutwe how to go around having to store the users passwords and usernames (in order to be able to perform the HTTP POST’s above)… Stay tuned!