Open Source Investigations - building your toolkit

Dr Stephen Hill
Posted by: Dr Stephen Hill | Last modified on 16/04/2018

Open-Source Intelligence or OSINT in the eyes of many refers to a broad array of information and sources that are generally available, including information obtained from the media (newspapers, radio, television, etc.), professional and academic records (papers, conferences, professional associations, etc.), and public data (government reports, demographics, hearings, speeches, etc.).  It has been estimated that roughly 90% of valuable intelligence comes from open sources compared to traditional covert intelligence.  According to the CIA open sources often equal or surpass classified information in monitoring and analysing issues including terrorism, proliferation and counterintelligence.  To an online investigator OSINT provides access to a plethora of intelligence which if knowing where to look and how to look can be the life blood of a modern day investigation. 


To most searching the Internet is simply about using a search engine such as Google or its close rival Bing to run a key word search.  This type of search and sites such as Google are typically referred to as the surface or visible web.  In simple terms the surface web is a portion or segment of the World Wide Web that is indexable by a search engine such as Google for example.  Surface search engines typically construct a database of the Web by using programs called spiders or robots (web crawlers).  The spider gets a copy of each page and indexes it, storing useful information that will let the page be quickly retrieved again later.  Google uses a slightly more refined approach using what is commonly termed the ‘page rank algorithm’. 


Online Investigation Toolkit


When it comes to searching online there are many tools available with some being more popular than others so which are the most effective for the online investigator?  This is a difficult question as it depends on many circumstances based on the type of search being carried out.  However the following section explores some of the options available to the online investigator when exploring the world of social media.


The main surface search index tools include Google and Bing with both having the ability to filter text, images, video and offer the investigator the ability to filter by time, location, site or related pages.  Both have good advanced features and offer the ability to translate (although this is very basic).  Whilst both these search engines are good they do not cover every possible angle of searching which some investigators have come to expect.  One weakness is that by default they do not automatically search the vast wilderness of social media including everything from blogs, photos, videos, geo-location data to name but a few.


Social Media Investigations


Social Media has opened up numerous opportunities and is a key component to profiling the subject of an investigation. The pool of information about each individual can form a distinctive social signature.


Twitter, Facebook, LinkedIn to name but a few have embedded themselves in people’s lives. Posting to walls, tweets, video and image updates are emerging as a new trove of intelligence.

Social media evidence can be a valuable addition to an investigation, revealing the kind of information that, years ago, would have been difficult, if not impossible, to find. But it has to be gathered in a way that will hold up in court. Because it’s such a new source of evidence in investigations, case law is
developing rapidly. A forward-thinking investigator would be well-advised to stay on top of the latest legislation both locally and internationally.


Once the access to social media information has been secured, either through court order or simply due to public accessibility, evidence must be gathered in a way that is legal and useful.  Collecting evidence from social media sites can be challenging for several reasons.



The world of social media is constantly changing, and users can easily update and delete material that could be evidence in a case, although once a user is aware of an ongoing investigation, he or she is under an obligation to preserve social media evidence just as if it were any other type of evidence.  Deleting photos, posts and other information is akin to shredding documents and the courts have been clear about the consequences, handing out hefty fines and sanctions for spoliation.

There are a series of specialist search tools available which offer options to the online investigator especially as these tend to excel in searching social media.  The following selection of sites each has their own unique strengths whether it is looking at photos, finding user profiles or even establishing geo-social footprints – collectively they provide a powerful toolkit to the online investigator.


Useful Sites:

Echosec (

Echosec is a location-based search platform that provides public safety, security, journalism, and intelligence professionals actionable knowledge based on social media and other information.  By typing a postcode into the search bar of the online application this site will bring up the location on a digital map and when the user highlights a square of the area, around a building, for example, social media posts and images taken and published in that location start popping up beneath the map. Sites searched include Facebook, Twitter, VK, Instagram.

Geofeedia (

Geofeedia is similar to Echosec allowing the user to search the globe and draw a perimeter around any location of interest displaying all geotagged social media posts from Twitter, Facebook, Instagram, YouTube, Flickr.  Geofeedia’s visual and analytics tools make it easy to discover, monitor, and filter social data, on a real-time basis.

Social Searcher (

Social Searcher is a specialist social media search engine. It allows the user to search for content in social networks in real-time and provides deep analytics data. Users can search without logging in for publicly posted information on Twitter, Google+, Facebook, YouTube, Instagram, Tumblr, Reddit, Flickr, Dailymotion and Vimeo. Users can also save their searches and set up email alerts.

Dark Net Search

The dark net has been in the headlines over the last couple of years because of its association to nefarious activities and it being ‘hidden’ from plain site.  Websites such as Silk Road and other associated activities operated under the anonymity of the dark net.

Access to this dark net is possible through services such as TOR or I2P.  With more and more online users worried about privacy and possible intrusion into their online life, downloads of TOR have increased substantially over the last few years.  However the dark net has proved attractive to not only the typical user worried about security but also the criminal wishing to hide their identity.  The dark net serves as a platform for the cyber-criminal to become anonymous since it not only provides protection from unauthorised users but also in many cases encryption to prevent monitoring.  TOR (the onion router) is one such anonymous network with access via the TOR browser which was developed by the US Naval Research Laboratory for communicating online anonymously.

So where would you start when trying to unlock this ‘dark net’ information?  The first and most crucial part is to understand when conducting an online investigation is how can the investigator go untraced and fall under the radar – in other words how do they hide their own digital footprints?

Anonymity comes from the Greek word ‘anonymia’ referring to the state where one’s personal identity is not publicly known.  To allow this anonymity online the internet investigator should look to disguise their digital footprints by hiding their IP address. The TOR client allows this to happen by routing internet traffic through a worldwide volunteer network of servers hiding the users information and eluding any activities of monitoring. 

The onion routing is a technique for anonymous communication over a computer network.  Messages are repeatedly encrypted and sent through several network nodes called onion routers.  In the same way someone may peel the layers of an onion, each onion router removes a layer of encryption to uncover routing instructions, and sends the message to the next router where the process is repeated. This gives the internet investigator the level of protection required as this technique prevents intermediary nodes from knowing the origin, destination and contents of the message.

Go Forth and Search

There is no right or wrong way to carry out an internet search but it is a good idea to familiarise yourself with a selection (toolbox) of search engines and specialist sites to use during an online investigation.  When conducting a straight forward keyword search experiment by adding and taking away words and try using the ‘Advanced Search’ option on the chosen search engine.

Remember information is power and having access to the most accurate, up-to-date and relevant intelligence is vital to a successful online investigation.  Knowing when to stop searching is a key skill so practice good discipline by setting a time limit and remember a good investigation is based on quality not quantity.

One final point to remember – protect your ID when online by using an anonymous browser and when asked for personal details if for example accessing a Gmail account or social media profile then use details not associated to you or your organisation. Also consider using a search engine which doesn’t track you giving better privacy such as DuckDuckGo ( or StartPage (


Author: Dr Stephen Hill, Director of Data & Intelligence at Absolute Partnership Ltd and founder of Snowdrop Consulting Ltd.  Further information on OSINT and Internet investigation training can be obtained by emailing Stephen at