Author Archives: Eran Levy

Structuring the Dark Web!

Posted on January 24, 2018 by Eran Levy

We’ve recently launched an exciting new addition to our dark web data feed (as featured on Betanews, ProgrammableWeb, and elsewhere): now, in addition to industry-leading breadth of coverage of the TOR network, we’ll also be structuring the extracted data so that it fits into a similar JSON format as our open web data feeds. The

Continue reading

Posted in Cybersecurity, Dark Web | Leave a comment

What is the Omgili Bot, and why is it Crawling Your Website?

Posted on December 28, 2017 by Eran Levy

Hi there. If you’re reading this, it’s probably because you’ve run into Omgilibot – perhaps in your web analytics or server logs (user agent: omgili/0.5 +https://omgili.com) – and turned to Google to decide whether this crawler is a benevolent creature that should be permitted to do as it will, or something more nefarious that deserves

Continue reading

Posted in Uncategorized | Leave a comment

3 Predictions for Web Data in 2018

Posted on December 12, 2017 by Eran Levy

2017 was a turbulent year: With Donald Trump shaking up the American political system, cryptocurrencies causing riptides throughout financial markets, and advancements in artificial intelligence sparking both anticipation and anxiety in the scientific world, the passing year seems to have been dominated by a sense of uncertainty and a sea change waiting to happen at

Continue reading

Posted in Cybersecurity, Data Extraction | Leave a comment

Richer Media Analysis with Broadcast News Transcripts

Posted on November 23, 2017 by Eran Levy

After a few long nights and some very tired developers, we’re proud to introduce the Broadcast Data Feed: transcribed, structured, and machine-readable television and radio programming from 1091 US TV stations, 21 Canadian TV stations, 83 Spanish-US TV Stations and 356 Radio Stations. This new product line is available and ready for you to use,

Continue reading

Posted in API | Leave a comment

Quick Guide to News APIs

Posted on October 10, 2017 by Eran Levy

Monitoring mass media has come a long way since the days of the press-cutting agency. The bulk of today’s news is published online, while modern technology lets us store, index and query massive amounts of textual data in milliseconds. Digitization presents clear advantages for consumers, who can now read or watch the news from the

Continue reading

Posted in Data Extraction, Technology | Leave a comment

3 Ways to Use eCommerce Product Data for Market Research

Posted on September 14, 2017 by Eran Levy

The web is an invaluable source of data when it comes to competitive intelligence, research and creating a go-to-market strategy. A simple Google search will reveal incredible amounts of public information about your main competitors, and connect you to mounds of existing research, financial analysis and other information that could be pertinent to your next

Continue reading

Posted in eCommerce | Leave a comment

How to Extract Data from Websites: Scraping Tools, DIY or DaaS

Posted on September 7, 2017 by Eran Levy

This is part 2 of our guide to web data extraction. Read part 1 to learn about the questions to ask before you start, or download the complete Web Data Extraction Playbook (PDF). Now that you’ve covered both the business and technical requirements for your web data extraction project (and if you haven’t, check out

Continue reading

Posted in API, Data Extraction, Technology | Leave a comment

Crawling the Dark Web to Detect the Next Market

Posted on July 25, 2017 by Eran Levy

Over the past few days, the internet has been abuzz with talk of the recent blows dealt by law enforcement to two major dark web “marketplaces”, AlphaBay and Hansa market; and the subsequent suicide of Alexander Cazes – the Canadian programmer-turned-criminal mastermind behind AlphaBay, who ended his own life in a Thai prison while awaiting

Continue reading

Posted in Cybersecurity, News | Leave a comment