chloeveltman.com > features

Moreover Technologies
3 July 2002

Gathering news from over 3,000 web-based sources

Machines are efficient things, but when it comes to achieving high levels of accuracy, human intervention is often required.

At Moreover Technologies, a San Francisco, London and New York-based information management solutions provider, technology and manpower are both used to deliver the latest news and information to Global 2000 clients with the degree of speed and accuracy on which the company's reputation has grown.

Since its foundation in 1999, the company, which won the 2001 Search Engine Watch Award for Speciality Search and which counts Deutsche Bank and McGraw-Hill among its clients, has etched a niche for itself between general search engines such as Google, and proprietary content aggregators such as Lexis-Nexis.

Moreover operates as a highly focused search engine, retrieving information from the web specifically geared to a particular client's needs. Moreover sources news from more than 3,000 web-based sources, from major newspapers and company websites to newsgroups and specialist trade publications. The information can be received via the client's e-mail, intranet system, website, extranet or desktop and is refreshed every 15 minutes.

Moreover's technological infrastructure, which operates via a New Jersey-based server farm, allows for a high degree of automation.

As well as extracting new data from each news site every 15 minutes, the system converts the http data into XML, allowing articles to be filtered under such criteria as date, time, source name and source type.

Databases store the news as it arrives, a redundancy detector eliminates multiple postings and an output engine integrates outgoing data into the client's desired delivery format. However, the system would fall apart without input from Moreover's editorial team.

Comprising 10 members (a quarter of the company's 40-strong workforce), the editorial team plays a key role in two parts of the process. Firstly, editors locate, evaluate and select news sources based on the client's requirements. Moreover provides a range of sources from which the client can pick, and the client can also ask for additional sources to be included.

"We're very proud of the quality of the news sources we harvest," says Angus Bankes, co-founder, chief technology officer and general manager of UK operations at Moreover.

"The inclusion of any particular source is client-led, but sources have to pass editorial criteria - they have to be high quality."

Having established which sources to include in the client's package, the editors create customised templates for each selected news site. Because websites change constantly and the positioning of the headline varies from site to site, manually-created templates ensure that Moreover's technology harvests only what is needed from each source and identifies the fast-changing areas of a site.

Human intervention is also integral to another part of the process. Once http sources have been converted to XML and have been checked for redundancies, Moreover's editors define rules to filter content topically and add category XML tags.

Moreover currently filters news sources across at least 360 categories, from legal and clothing industry news to joke sites and Hollywood gossip.

Mr Bankes believes that there will always be room for an editorial team at Moreover. "If you only use software, you'll never get more than 85 per cent accuracy," he says. "Automation is key, but you need human intervention to increase that figure from 85 per cent to 98 per cent."

Mr Bankes says that although software is improving, people are needed to create profiles for the software.

Another reason for employing human editors is to keep up with the fast- changing face of the internet. "Most people would accept that purely automated systems can't adapt to change," says Mr Bankes. "Because the internet moves so fast, without human intervention, you wouldn't pick up on changes for weeks."

Some people disagree. Paola di Maio, a content analyst and editor of the web-based publication Content Wire, thinks that human editing teams are less than ideal. "Human editors are high cost and high error," comments Ms Di Maio.

Unlike Mr Bankes, Ms Di Maio believes that full automation is possible. "With a good enough algorithm and appropriate filters, automation is 100 per cent possible," she says. "However, humans must do systematic quality controls."

Moreover is not the only company working in the information management sector to consider humans an integral cog in the system.

"Professional editors will always be important in story selection," says David Scott, vice president of corporate marketing at NewsEdge, a Massachusetts-based information services company whose 280-strong staff includes 40 editors.

"There are certain things which machines simply cannot do. For example, machines cannot choose the better of two stories which come out at the same time from different media outlets on the same subject. Human judgment is important."