lesson
40 min

Scraping: When It Is and Is Not OK

What web scraping is, the rules and ethics around it, and a checklist to use before you collect anything.

Objective

By the end of this lesson, you can explain what web scraping is and use a checklist to decide whether a scraping plan is acceptable.

What scraping is

Web scraping means writing code that reads web pages and pulls data out of them. It is useful when data is shown on a site but not offered as a download or API.

Being able to scrape a page does not mean you should. The question is not "can I?" It is "should I, and on what terms?"

Try an API or download first

Before scraping, look for:

  1. A download link or open data portal.
  2. An official API.
  3. A contact you can email to ask for the data.

These are faster, more reliable, and respect the publisher.

The rules to check

  • Terms of service. Many sites forbid scraping in their terms.
  • robots.txt. A file at site.com/robots.txt that says which pages automated tools should not visit. Respect it.
  • Rate. Fast, repeated requests can slow or crash a small site. Wait between requests.
  • Personal data. Collecting names, photos, or posts about people raises privacy and legal issues, even when the page is public.
  • Copyright. Facts are not copyrighted, but articles, photos, and creative work usually are.

Laws on scraping differ and keep changing. When in doubt, ask permission or do not collect.

A safe practice: parse a page you control

This code reads a small made-up web page stored in a string. It uses only Python's built-in tools, so it runs here.

Run it. The parser finds every table cell in a small sample page. Add a third row to the HTML and run it again.
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

A national example

Judgment check

Never scrape data about minors, such as school social media pages or youth sports rosters. Never scrape private groups, even if you have access. Public is not the same as fair game.

Before you scrape anything

0 of 6 done

Ask. Answer. Build together.