84 — Web: extracting links from a page
Guide 55 fetched JSON; guide 69 matched text with regex. A web page is just text too — HTML with tags around the content. Extracting the links is one regex over the raw page, and joining them with the server's address makes the full URLs a browser would use.
Steps
Create a small page with three links. Save
page.html:<html> <head><title>my page</title></head> <body> <h1>Links</h1> <a href="story.txt">the story</a> <a href="notes.txt">notes</a> <a href="about.html">about us</a> </body> </html>Serve the folder, as in guide 55:
python3 -m http.server 8000Fetching the page is the same
urlopencall that fetched JSON; a page is text, so.decode("utf-8")gives the HTML:
run it →from urllib.request import urlopen import re base = "http://localhost:8000" html = urlopen(base + "/page.html").read().decode("utf-8")Each link lives between
href="and the next". The regexr'href="([^"]+)"'finds every one —[^"]+means "one or more characters that are not a quote", andfindallreturns just the captured parts, as in guide 69. Savelinks.nme:
run it →# links.nme — find the links on a web page. # Run: nme r links # Serve this folder first: python3 -m http.server 8000 from urllib.request import urlopen import re base = "http://localhost:8000" html = urlopen(base + "/page.html").read().decode("utf-8") links = re.findall(r'href="([^"]+)"', html) show f"found {len(links)} links:" for link in links: show base + "/" + linkThe page only knows
story.txt— a short name relative to itself. Prependingbase + "/"turns it into a full URL a browser could open: a page on the same server links with short names, and the client completes them.Run it:
nme r linksfound 3 links: http://localhost:8000/story.txt http://localhost:8000/notes.txt http://localhost:8000/about.htmlThe three
hrefs became three complete addresses. This is the same extraction step a link checker or a web crawler performs before fetching the next page.
Try it yourself
Add a link to another page that itself has links, then make the program
fetch each found link and count how many of them exist (a
404 Not Found answer means a broken link — the HTTP error appears as
an exception, guide 68 shows how to catch it). Or extract
the page's src="..." image sources with the same pattern.
What you learned
- HTML is text: fetch it with
urlopen, decode it, search it with regex. r'href="([^"]+)"'finds link targets;[^"]+stops at the closing quote.- Relative links are short names; the client prepends the server address.
- Link extraction is one regex — the first step of a link checker.