Skip to main content

Memex - DARPA's search engine for the Dark Web

by Mark Stockley 

Anyone who used the World Wide Web in the nineties will know that web search has come a long way. Sure, it was easy to get more search results than you knew what to do with in 1999 but it was really hard to get good ones.

What Google did better than Alta Vista, HotBot, Yahoo and the others at the dawn of the millennium was to figure out which search results were the most relevant and respected.

And so it's been ever since - search engines have become fast, simple interfaces that compete based on relevance and earn money from advertising.

Meanwhile, the methods for finding things to put in the search results have remained largely the same - you either tell the search engines your site exists or they find it by following a link on somebody else's website.

That business model has worked extremely well but there's one thing that it does not excel at - depth.

If you don't declare your site's existence and nobody links to it, it doesn't exist - in search engine land at least.

Google's stated aim may be to organize the world's information and make it universally accessible and useful but it hasn't succeeded yet. That's not just because it's difficult, it's also because Google is a business and there isn't a strong commercial imperative for it to index everything.

Estimates of how much of the web has been indexed vary wildly (I've seen figures of 0.04% and 76% so we can perhaps narrow it down to somewhere between almost none and almost all) but one thing is sure, there's enough stuff that hasn't been indexed that it's got it's own name - the Deep Web.

It's not out of the question to suggest that the part of the web that hasn't been indexed is actually bigger than the part that has.

A subset of it - the part hosted on Tor Hidden Services and referred to as the Dark Web - is very interesting to those in law enforcement.

There are all manner of people, sites and services that operate over the web that would rather not appear in your Google search results.

If you're a terrorist, paedophile, gun-runner, drug dealer, sex trafficker or serious criminal of that ilk then the shadows of the Deep Web, and particularly the Dark Web, offer a safer haven then the part occupied by, say, Naked Security or Wikipedia.

Enter Memex, brainchild of the boffins at DARPA, the US government agency that built the internet (then ARPANET).

DARPA describes Memex as a set of search tools that are better suited to government (presumably law enforcement and intelligence) use than commercial search engines.

Whereas Google and Bing are designed to be good-enough systems that work for everyone, Memex will end up powering domain-specific searches that are the very best solution for specific narrow interests (such as certain types of crime.)


Today's web searches use a centralized, one-size-fits-all approach that searches the internet with the same set of tools for all queries. While that model has been wildly successful commercially, it does not work well for many government use cases.

The goal is for users to ... quickly and thoroughly organize subsets of information based on individual interests ... and to improve the ability of military, government and commercial enterprises to find and organize mission-critical publically [sic] available information on the internet.

Although Memex will eventually have a very broad range of applications, the project's initial focus is on tackling human trafficking and slavery.

According to DARPA, human trafficking has a significant Dark Web presence in the form of forums, advertisements, job postings and hidden services(anonymous sites available via Tor).

Memex has been available to a few law enforcement agencies for about a year and has already been used with some success.

In September 2014, sex trafficker Benjamin Gaston was sentenced to a minimum of 50 years in prison having been found guilty of "Sex Trafficking, as well as Kidnapping, Criminal Sexual Act, Rape, Assault, and Sex Abuse - all in the First Degree".

Scientific American reports that Memex was in the thick of it:


A key weapon in the prosecutor's arsenal, according to the NYDA's Office: an experimental set of internet search tools the US Department of Defense is developing to help catch and lock up human traffickers.

The journal also reports that Memex is used by the New York County District Attorney's Office in every case pursued by its Human Trafficking Response Unit, and it has played a role in generating at least 20 active sex trafficking investigations.

If Memex carries on like this then we'll have to think of a new name for the Dark Web.

Comments

Popular posts from this blog

Why States Still Use Barrel Bombs

Smoke ascends after a Syrian military helicopter allegedly dropped a barrel bomb over the city of Daraya on Jan. 31.(FADI DIRANI/AFP/Getty Images) Summary Barrel bombs are not especially effective weapons. They are often poorly constructed; they fail to detonate more often than other devices constructed for a similar purpose; and their lack of precision means they can have a disproportionate effect on civilian populations. However, combatants continue to use barrel bombs in conflicts, including in recent and ongoing conflicts in Africa and the Middle East, and they are ideally suited to the requirements of resource-poor states. Analysis Barrel bombs are improvised devices that contain explosive filling and shrapnel packed into a container, often in a cylindrical shape such as a barrel. The devices continue to be dropped on towns all over Syria . Indeed, there have been several documented cases of their use in Iraq over the past months, and residents of the city of Mosul, which was re

Russia Looks East for New Oil Markets

Click to Enlarge In the final years of the Soviet Union, Soviet leader Mikhail Gorbachev began orienting his foreign policy toward Asia in response to a rising Japan. Putin has also piloted a much-touted pivot to Asia, coinciding with renewed U.S. interest in the area. A good expression of intent was Russia's hosting of the Asia-Pacific Economic Cooperation summit in 2012 in Vladivostok, near Russia's borders with China and North Korea. Although its efforts in Asia have been limited by more direct interests in Russia's periphery and in Europe, Moscow recently has been able to look more to the east. Part of this renewed interest involves finding new export markets for Russian hydrocarbons. Russia's economy relies on energy exports, particularly crude oil and natural gas exported via pipeline to the West. However, Western Europe is diversifying its energy sources as new supplies come online out of a desire to reduce its dependence on Russian energy supplies . This has

LONDON POLICE INDIRECTLY ENCOURAGE CRIMINALS TO ATTACK RUSSIAN DIPLOMATIC PROPERTY

ILLUSTRATIVE IMAGE A few days ago an unknown perpetrator trespassed on the territory of the Russian Trade Delegation in London, causing damage to the property and the vehicles belonging to the trade delegation , Russian Foreign Ministry Spokeswoman Maria Zakharova said during the September 12 press briefing. The diplomat revealed the response by the London police was discouraging. Police told that the case does not have any prospects and is likely to be closed. This was made despite the fact that the British law enforcement was provided with video surveillance tapes and detailed information shedding light on the incident. By this byehavior, British law inforcements indirectly encourage criminals to continue attacks on Russian diplomatic property in the UK. Zakharova’s statement on “Trespassing on the Russian Trade Mission premises in London” ( source ): During our briefings, we have repeatedly discussed compliance with the Vienna Convention on Diplomatic Relations, specif