ELROND #2: The Pipeline…

The second post about my final year Electronic Engineering project to identify landmarks using photographs taken from a mobile phone in seconds.

Imagine you are walking around London and you come across a building that you want to know more about. You whip out your mobile phone and open up the ELROND app. By taking a picture of the building you want to know more about, ELROND returns relevant information in just a few seconds. The name and purpose of the facility, weather it is open to the public, the opening times, the phone number of reception, a history, appropriate web links… wouldn’t that be useful? Well that’s just one potential application of my project.

It could equally be used to display information about portraits in a gallery just by holding your smartphone in front of a painting, with relevant information overlayed on the screen in real-time…

Or just as an alternative to GPS to locate you when GPS is not available… such as indoors…

My project, titled Elrond, aims to provide the backbone or infrastructure to enable such apps to be written much quicker. But how does it work? how does that picture of a building turn into information?

The diagram below gives an overview of the process.

ELROND

  1. Once the Android application has extracted the features from the image of the building using a feature detection algorithm (more on that in a later post) the extracted information is packaged into an XML format and transmitted over a data network (3G or Wi-Fi) to a Linux web server.
  2. The web server then parses the XML file into a format it can understand.
  3. Each extracted feature is then compared to all known features by Elrond, hence gathering a shortlist of which buildings this is most likely to be a picture of. Because the number of known features is likely to be in the order of millions, a neat way of searching the set needs to be used, called a KD-Tree.
  4. The searching returns a shortlist of images that most probably match the query image. The items at the top of the shortlist are the most likely, so the top 100 results are looked into in more detail. Elrond will look at all the features in both images, find ones that match, and store their locations. Then a homograph is calculated to see if there is a way to map the features from the query image to the stored image. A homograph is a matrix that describes the best way to map two sets of points to each other. If the homograph can map lots of points between the images, then that image is given a high match score.
  5. After all the homographs have been calculated, the best matching image can be determined; or no match is found, in which case Elrond cannot return any information. Assuming a reasonable match was found, Elrond now knows what the building is! Relevant information about that building is maintained on a database, and so it can be fetched.
  6. The database information is packaged into another XML file and sent to the mobile device.
  7. The mobile device interprets the XML and graphically displays the information on the screen. Voilà!

I know that some people are reading this gormless, but for those that are interested to know more I want to write more posts about how specific parts of the application work and perform. So if you have any thoughts and suggestions let me know.

The next post will be about less techy things…

Acting Debut & The FUTURE!

I’ve been part of the University of Surrey’s Student Television Station since it began a few years ago, but I’ve always focused on the technical aspects such as filming and editing… but this week, in a bizarre role-reversal between Mike and I, I presented and he edited. So I present to you my first (who knows their may be more…) foray into news presenting…

Charles Gray uncovers plans for the old bookshop to become a strip club after it’s refurbishment… or at least it should according to one student. In a MADTV EXCLUSIVE we tell you what is actually to become of our once loved book shop.

In other news, that I haven’t presented to camera, I went to London last weekend and saw some mates from way back when in Sixth form. Brought together with our love for Nando’s free whole chicken, we sat, gobbled and gobbed for hours. It’s always interesting to stay in touch and see what all your old friends have achieved with their lives. Ben, who has already graduated from an Accounting degree, has found himself working in the Pensions department at Wandsworth Council… I wonder where I’ll be in a years time? Back at Sony? Another broadcast company? McDonalds? the street? Like recycling, the possibilities are endless…

But that’s a while away yet, what about the immediate future? Well I’m off to W-hey-mouth this weekend to see the lovely Charlotte :D After that?

  • 4th December. My parents come to Guildford for Christmas dinner cooked by my housemates and I.
  • 15th December. Going to see Matthew Bourne’s Nutcraker in London!
  • 22nd December. The usual trick for Christmas commences… up to my sisters in Chesterfield for the festivities and prevailing jokes.
  • 29th December. The usual trick for New Year… Jake and I hit up London for a ‘small’ party with friends while the parents are away…
  • January. Exams, coursework, projects… back to normality, for my last year at University!!!

Some of you may have heard that I made a new years resolution for 2011 to not buy any chocolate for the whole year, and those people will also know how impossible this challenge should prove to be… well I’m still at it, but the going is getting tough with just 6 weeks to go! I started the year with a chocolate stash (below) to get me through, but this week that stash was depleted to zero. So now, as Boyzone once said, the going gets tough…

 DSCF4836

  • Happiness. 8 out of 10
  • Tiredness. 1 out of 5
  • Workload. 6.5 out of 10
  • Last Meal. A hobnob
  • Song of the day. Ed Sheeran – The A Team
  • Thought for the day. I thought love was only true in fairy-tales, meant for someone else but not for me…
  • What I’m Doing Now. Packing for the weekend

Taste the Rainbow

Settling back into the slow life is so comforting knowing that you are doing something for your future, and living the present the best way you know how.

I want to inject a bit of nostalgia into this post, knowing full well that this blog has started to show its receding hairline this year with the lacklustre modicum of posts. As the new year beckons, and lets be honest its already on the horizon, I usually turn the content switch to “past” and start idolizing the perfect moments from the year gone by. And this year will be no different in some respects, but I want to start early because it’s been such an incredible year!

January

My best memories from January lie scattered amongst many weeks spent in Switzerland, as in fact most of my memories from the year. But there’s another theme for January, random lols. If it was Sophizzle offering words of comfort to her triffle on the bus home from Tesco, Frazer and I contemplating the benefits of nipple cups, or Sophie getting a super-low voice over serious Skype discussions about housing… it was a laugh. The best moment however?

Walking around an empty Zurich airport in bare feet. Like the set of a film…

February

Another month filled with Swiss air, but also featuring a couple of trips to London to see Maroon 5, Mr Scruff and Lifehouse. And while going dressed as Sherlock Holmes into Guildford town was the best fun ever, the Lifehouse gig at the Roundhouse has a special place in my heart…

March

Sometimes life hands you fragments of perfection, that second when the world stops and you feel that despite the worries you may have, and the challenges you face, you’ve actually got it pretty good :) One particular Friday afternoon, sat outside Lugano airport in the gorgeous sunshine waiting for a plane back home after another great week at work, and the fragments just seem to fall into place. You know you’re doing the right thing and you should never stop being who you are.

April

The last one for today… April. April. Where do I start… There are two things that stand out about April. One was the house boat trip, possibly the single most hilarious, drink-fuelled, best times of my life – spent on my dad’s narrowboat with my then-future-now-current housemates. Initially a bonding weekend to get to know each other better, but once we’d got the formalities out of the way it was all about having the time of our lives. The week was so good we are still quoting from it now, in fact we have a book of quotes in our living room from that trip that goes some way in recreating some of the hilarity: everything from carnivorous geese to gooches, heart-attack sandwiches to the helm of insobriety. This is undoubtedly not the last you will hear from this trip in the run up to the new year. And as for the second thing about April, that will have to wait :)

I want to finish the post with a few pictures from the start of the year that you hopefully haven’t seen before…

DSCF4862

Even as early as January the temptation to buy chocolate (against my new years resolution!) was staring me in the face… I walked past this particular idol in the Zurich terminal building every Friday on my way back to Heathrow…

DSCF5028

Jake comes to Guildford, and we find the FREE Guildford museum to be worth every penny…

DSCF5079

Doing the speed limit in a Mercedes convertible on the Swiss motorway…

 DSCF5096

The sign outside the building next door to our hostel in Amsterdam. It was ideally located for our needs… there was a supermarket and bus stop nearby.

DSCF5173

The boat trip provided lots of time to relax and take it easy, and have cider at every possible moment.

DSC00243

Charlotte’s birthday party had a “what did you want to be when you were younger” theme… obviously I chose geek.

Next time I promise I will talk about my current housemates and give you a tour of the grounds. I just need to be writing in here when it’s light outside!

Until next time…

Charles x

ELROND #1: Feature Detection in Computer Vision

The first post about my final year Electronic Engineering project. I introduce you to the topic of feature detection and what my project is about.
Feature detection is an important problem in computer vision. Computer Vision is the study of extracting information from the real world and somehow use it as an input to computer programs. A simple example of Computer Vision would be how Gollum was created for the Lord of the Rings films…

The creation of Gollum used motion capture from real world camera shots to digitally create the character.

Feature detection is the process of computer algorithms detecting interesting, or “perceptually interesting” locations of an image. Early feature detection worked by detecting edges within an image, but there were several problems with this approach.
Edge Detection
^ from Wikipedia
One of the biggest uses of feature detection is a process known as Image Registration, where a computer can find similar images and determine that they are actually of the same building/object/face/landscape/tree/etc. And this is the basis for my project.
everyday object recognition
^ originally from http://ils.intel-research.net/ (now removed)
The premise of my project, titled Scalable Landmark Recognition on Mobile Devices, is to allow anyone with a mobile phone to take a photograph of a landmark and be told what it is, where it is and some interesting facts about it. For example, you could take a picture of the Eiffel Tower and phone app would recognise the landmark and return information about when it was built, how tall it is, how much ticket prices are today, opening times etc.

^ from http://mastersofmedia.hum.uva.nl/2011/10/10/app-review-google-goggles/
This will be a downloadable mobile app for Android phones, and it will use feature detection to find similar images in it’s database and return information about the closest match. The app will require internet access to communicate with a server that will run the query and holds the database.
The project was dubbed “ELROND” by Anne, who I lived with in second year, because it’s close to SLROMD (which is what the actual abbreviation would be) and it also has Lord of the Rings references, which I’m all for!
So far I’ve got most of the feature detection working, and it should be quick enough to identify a match from 1000’s of images in under a second… but only time will tell. I’ll explain more about how my current solution works in a later post, but for now here’s a screenshot of it working on my computer…

13 usingfile

  • On the top-right you can see the query image, this would be the image taken on the phone.
  • Below it is the image matched to it from the database.
  • The white lines are lines between features matched between the query and database image (using a process called FLANN matching).
  • The green box shows where the query image would fit onto the database image if you were to stitch them together in a panorama (called a homography).
  • The top-middle window (with the red circles) shows the features found in the query image. The features are found using the SURF algorithm.
  • The left of the image shows the code output, showing the progress of the database search and how the query image matched to the other images in the database.

Dressing up

First off I forgot to mention that last weekend in Cardiff I also was in London to see Bruno Mars:

DSC03053

Despite the long bus journey from Cardiff to London, and closures on the Victoria line, it was well worth it!

By the way, all this travelling the last year has taught me a few top travel tips to save:

  1. Check MegaTrain a few weeks (ideally 4-6) before you travel and see if they are offering your journey for less than the rail operator.
  2. Also use MegaBus if you can bare the thought, it’s a fraction of the price, and on Sundays it can actually be quicker thanks to Sunday trains stopping at every station under the sun.
  3. If you’re still drawing a blank, or would prefer to use the train, then you can still save money by using your Tesco Clubcard points to get triple discount on rail tickets! I know, epic. Just follow these instructions and you are quids in!

Anyway, back to last weekend:

halloween

I went out as a mad professor for Halloween on Friday night!

death

The house as a whole was looking pretty dapper… except Will, he looked like death :D

cocktails

On Saturday night we had a cocktail party at our house. 5 cocktails: Martini (gin), Daiquiri (rum), Cosmopolitan (vodka), Margarita (tequila) and Old Fashioned (whisky). It was a very classy evening… until we all ended up pissed and rolling around all over the place…

more cocktails

All in all a mixed weekend, but certainly one I won’t forget in a hurry :)

  • Happiness. 7 out of 10
  • Tiredness. 3 out of 5
  • Workload. 6 out of 10
  • Last Meal. Chicken Korma curry (not fish and chips!) 
  • Song of the day. Skylar Grey – Love The Way You Lie
  • Thought for the day. I must try using a head massager in the bedroom…
  • What I’m Doing Now. Working before going to see the King Blues tonight at the Roundhouse in Camden! Yes!