Get element from ResultSet: bs4.element.ResultSet.element()

Programming
, , ,
1

I am trying to select specific elements from a particular bs4.element.ResultSet.

Here’s my code:

from bs4 import BeautifulSoup
import requests
CompanyNOT = "apple"

url = "https://www.annualreports.com/Companies?search="
page_to_scrape = requests.get(url+CompanyNOT)

soup=BeautifulSoup(page_to_scrape.content,"html.parser")
urls = soup.find_all('a', href=True)

I need to extract all elements with /Company in them, like this: [/Company/apple-hospitality-reit-inc, /Company/apple-in, /Company/maui-land-pineapple-co-inc, /Company/pineapple-energy-inc]. How can I do this?

2

You can use a list comprehension to achieve this. Here’s the modified code:

from bs4 import BeautifulSoup
import requests
CompanyNOT = "apple"

url = "https://www.annualreports.com/Companies?search="
page_to_scrape = requests.get(url+CompanyNOT)

soup=BeautifulSoup(page_to_scrape.content,"html.parser")
urls = [link['href'] for link in soup.find_all('a', href=True) if '/Company' in link['href']]

print(urls)

This will give you the desired output:

['/Company/apple-hospitality-reit-inc', '/Company/apple-in', '/Company/maui-land-pineapple-co-inc', '/Company/pineapple-energy-inc']