Big Data Crawler

Crawler for large amounts of data

Automation/ Web Development

Crawlers are computer programs that automatically search the Internet, databases or other information collected according to certain criteria. Especially in the age of the digitalization of business life, they are therefore a suitable means of dealing with the exponentially growing flood of information.


How Steven Johnson wrote in his book “Where good ideas come from: A brief history of innovation” wrote, “is precisely where many people come together that most innovations arise. For this reason, many companies regularly attend trade fairs and congresses in order to get to know exhibitors personally and to take up ideas represented there that they had not previously focused on. It doesn’t matter whether you want to open up new business fields, expand your existing clientele, look for new employees or look for new technologies with which you can simplify and improve your own work. When many people come together, a potential arises that must be exploited. In order to find one’s way through the mass of events that can be attended, however, it is necessary in advance to find out as much information as possible about the type, participants and topics of the respective event. In order to make this possible, trade fair or congress operators publish lists of exhibitors, topics or lectures. These are provided meanwhile even with rudimentary filter functions, in order to make it for interested companies possible to plan their attendance or to decide at all only to a attendance. The challenge for companies is to find out which topics are offered where, which exhibitors are interesting and which lectures are offered. The analysis of offered data and its supplementation with decision-critical additional information is elementary for an efficient course of events.


The aim of the project was to develop an intelligent platform that would enable the client to prepare and analyse the information provided and to expand it with additional information relevant to him, so that he could use the information relevant to his business for his visit or decide whether or not to attend an event.


The result of this challenging task is a platform which, freely configurable to the respective company objectives, automatically searches trade fair pages for suitable exhibitions, adds arbitrarily selectable information from the Internet or other data sources to provide the user with decision aids for or against a visit, provides contact data for potential contacts of the individual exhibitors or speakers and thus shows an optimal plan for the visit of the event.


A global corporation helps other international manufacturing companies in Asia to become active as manufacturers and wants to expand its customer portfolio. For this purpose, the group is looking for all companies with German headquarters that do not yet produce in Asia, have a minimum number of employees and a minimum turnover, and a product range that could be interesting for outsourcing to Asian countries for various reasons such as quantity, price or quality.

Comments are closed.