scrape a website in python

To scrape a website in Python, we can use the requests and beautifulsoup libraries. Here's an example code snippet that scrapes the title and body text of a website:

main.py
import requests
from bs4 import BeautifulSoup

# send a GET request to the website
url = 'https://www.example.com'
response = requests.get(url)

# parse the HTML content using beautifulsoup
soup = BeautifulSoup(response.content, 'html.parser')

# extract the title and body text
title = soup.find('title').get_text()
body = soup.find('body').get_text()

print('Title:', title)
print('Body:', body)
398 chars
17 lines

In this example, we first send a GET request to the website using the requests library. We then use beautifulsoup to parse the HTML content and extract the title and body text using the find function. Finally, we print out the results.

Note that web scraping may have legal and ethical implications, so it's important to be respectful of website owners and their terms of use. Always check a website's robots.txt file and consider rate limiting your requests to avoid overwhelming the server.