Many times you'll see a * next to Allow or Disallow which means you are either allowed or not allowed to scrape everything on the site. On the other hand, we are disallowed from scraping anything from the /scripts/subfolder. In this example we're allowed to request anything in the /pages/subfolder which means anything that starts with /pages/. The Crawl-delay tells us the number of seconds to wait before requests, so in this example we need to wait 10 seconds before making another request.Īllow gives us specific URLs we're allowed to request with bots, and vice versa for Disallow. A * means that the following rules apply to all bots (that's us). We don't really need to provide a User-agent when scraping, so User-agent: * is what we would follow. Common bots are googlebot, bingbot, and applebot, all of which you can probably guess the purpose and origin of. Some robots.txt will have many User-agents with different rules.
The User-agent field is the name of the bot and the rules that follow are what the bot should follow. Since this article is available as a Jupyter notebook, you will see how it works if you choose that format. If I'm just doing some quick tests, I'll usually start out in a Jupyter notebook because you can request a web page in one cell and have that web page available to every cell below it without making a new request. We don't want to be making a request every time our parsing or other logic doesn't work out, so we need to parse only after we've saved the page locally. Every time we scrape a website we want to attempt to make only one request per page. With this in mind, we want to be very careful with how we program scrapers to avoid crashing sites and causing damage. With a Python script that can execute thousands of requests a second if coded incorrectly, you could end up costing the website owner a lot of money and possibly bring down their site (see Denial-of-service attack (DoS)). Every time you load a web page you're making a request to a server, and when you're just a human with a browser there's not a lot of damage you can do.
0 kommentar(er)
