web crawling

基本解释

But all the major and legitimate Web crawling engines obey the requests in robots.txt.
不过，所有主要的合法 Web 爬虫引擎都会遵从 robots.txt 内的要求。
However the current two ways of web crawling used by focus crawler are low efficiency.
然而目前的主题爬虫所采用的两种基本抓取网页的方式效率比较低下。
kns50.chkd.cnki.net
As the key component of search strategy in topic-oriented web spider, the topic relativity judging algorithms ensure the focused web crawling process of the spider.
作为主题网络蜘蛛搜索策略的核心部分，主题相关性判断算法是网络蜘蛛能够围绕设定主题进行聚焦检索的关键。

Likewise, a web crawling robot which follows links such as Google is not effective.
FORBES: How reputation services enhance IPS
He mentions use cases like batch processing, scientific computing, video and image processing and rendering, testing and Web crawling.
FORBES: Amazon CTO Werner Vogels On The State Of The Cloud
The search engine's Web-crawling spider now keeps an eye out for viruses and malware as it combs the Web, and shows users a warning page before allowing them to access malicious sites.
FORBES: Made For Hacking