Listen to this article
* – This article has been archived and is no longer updated by our editorial team –
Crawler2api is a data company on a mission to provide structured real-time data, fast and reliably from sources that are difficult to access for their clients. Below is our recent interview with Boris Bilsky, CEO at Crawler2api:
Q: Can you give us more insights into your services?
A: Crawler2Api is providing structured data in real-time from data sources that are difficult to access for our clients. In most cases these sources of data offer either no API or only a rate or data limited API which can’t be used for our clients specific use case. In simple terms, company A is in need of data from one or many data sources. This can be websites, mobile Apps or other publicly accessible sources. Our clients define which data is required and we built a crawler based API from the outside-in that allows for a structured flow of data between the two entities. Data can be requested in real-time or based on a preset schedule. Due to the fact that we base our APIs on web crawlers no interaction with the data source is required.
Let me give you two real-world examples that better explain what we do. Let’s assume a user visits a website offering travel services (trains, flights, etc.) in order to book his travel. He wants to experience the country he travels to by train. The rail provider in that country however does not offer an API, which means that the visited website is neither able to display the timetables or book his tickets. Crawler2Api builds the missing link between the website and the rail provider so the user is able to plan his travel on the visited website and book his tickets all in the same spot. Another example is dynamic re-pricing in today’s Ecommerce. In order to always be ahead of the game, ecommerce retailers need to know what their competition is doing and how they are pricing their products. We collect data from big retailers like Google Shopping, Ebay, Amazon, and other price comparison sites in real-time and provide them to our clients so they can gain a competitive advantage in pricing, selection, inventory and marketing.
Last but not least, a new field for us is Artificial intelligence, in particular the generation of datasets for different AI applications. The dataset is building the foundation of every AI. Without relevant data you can’t train any model or algorithm and as a result the AI won’t behave as intended. This is a rapidly growing field for us as companies in this space are facing two main challenges. One is getting good relevant data and the other is labeling that data so the AI is able to understand it. The first one we can take on, while the latter is usually left to the domain experts who are in charge of making sense of all the data. For the future, I see a lot of growth potential for Crawler2Api in this area.
Recommended: Q: What is unique about Crawler2api and how does it stand out from competition?
A: We are the only company out there that specializes in crawler-based real-time APIs. Other companies in the web crawling/web scraping space either offer crawling frameworks or web crawling services in the form of batch crawls, which means pulling batches of data from data sources over a given timeframe. Nothing happens in real-time here. The customer describes what data he needs and where he needs it from and after a predefined period the data is delivered in .xml or other formats. Complexity however increases many fold when an incoming crawling request has to be reliably answered in real-time with response times of only a couple of seconds.
Q: Who is your ideal customer and why?
A: Our ideal customer has a need for data, regardless of where the data sits. We work with companies from many different industries, from travel/mobility to entertainment and ecommerce. Data can be internal company data on our clients servers or external web data. Data can be requested in real-time or according to a certain schedule as defined by the customer. We are flexible in providing a custom solution that fulfills our customers’ needs.
Q: What are your plans for keeping Crawler2api on the forefront of technology innovation?
A: Web crawling is a very particular field of expertise. A lot of experience is required to actually create stable and high-performing APIs that can be used in real-life business scenarios. That’s why most companies trying to build these kind of APIs themselves are either struggling to come up with a stable & fast solution. Many companies just keep their fingers off it right away. Websites constantly change and so do the APIs. Compared to a batch crawl which follows a more or less fixed routine, a crawler-based API is pretty dynamic. Constant adaptations are necessary to keep the service quality high.
We are always looking into new use cases to apply our know-how and our technology. Lately we have been working a lot in the field of AI with our offer to collect training datasets for our customers AI use cases. This is a rapidly growing field since the heart of AI is and always will be the dataset with which the algorithms and statistical models are trained. Unless you have a big enough dataset which is tailored to your specifc use case, your will have a hard time providing accurate results with your AI.
Q: How would you convince the reader to start using Crawler2api?
A: Reliable scraping at scale is hard work and hiring experts, scaling server infrastructure to fetch millions of datapoints in real-time is costly. If a company decides to build the APIs themselves they should be aware that lots of niche expertise is required to create and maintain a stable and performant solution.
Activate Social Media:
