Scraping: When It Is and Is Not OK
What web scraping is, the rules and ethics around it, and a checklist to use before you collect anything.
Objective
By the end of this lesson, you can explain what web scraping is and use a checklist to decide whether a scraping plan is acceptable.
What scraping is
Web scraping means writing code that reads web pages and pulls data out of them. It is useful when data is shown on a site but not offered as a download or API.
Being able to scrape a page does not mean you should. The question is not "can I?" It is "should I, and on what terms?"
Try an API or download first
Before scraping, look for:
- A download link or open data portal.
- An official API.
- A contact you can email to ask for the data.
These are faster, more reliable, and respect the publisher.
The rules to check
- Terms of service. Many sites forbid scraping in their terms.
- robots.txt. A file at
site.com/robots.txtthat says which pages automated tools should not visit. Respect it. - Rate. Fast, repeated requests can slow or crash a small site. Wait between requests.
- Personal data. Collecting names, photos, or posts about people raises privacy and legal issues, even when the page is public.
- Copyright. Facts are not copyrighted, but articles, photos, and creative work usually are.
Laws on scraping differ and keep changing. When in doubt, ask permission or do not collect.
A safe practice: parse a page you control
This code reads a small made-up web page stored in a string. It uses only Python's built-in tools, so it runs here.
A national example
Judgment check
Never scrape data about minors, such as school social media pages or youth sports rosters. Never scrape private groups, even if you have access. Public is not the same as fair game.
Before you scrape anything
0 of 6 doneDiscussion
Ask. Answer. Build together.
Loading…