How to iterate through webelements to extract text from HTML tags in Selenium Web Automation (Python)?

Question

I am making a reddit bot that will look for certain attributes in comments, use selenium to visit the information website, and use driver.find_elements_by... to get the value inside those tags.

Now, driver.find_elements_by... is not iterable, and there are multiple <span class="name">Lorem Ipsum</span> tags with text inside them that I want obtained. I am storing this as a variable and replying to the comment via PRAW.

Suppose that the HTML is this:

<span class="name">Lorem</span>
<span class="name">Ipsum</span>
<span class="name">Dolor</span>
<span class="name">Sit</span>
<span class="name">Amet</span>

So, how could I obtain the text from all of the <span class="name"> tags, and when I store it as a variable and reply, will it just put all the text together without spaces or will it format it with a space between each text, supposing that I write:

tags = driver.find_element_by...
comment.reply("Tags: {}".format(tags))

And if it just puts all the text together, how can I format it so that there are spaces?

Answer 1

To extract the texts e.g. Lorem, Ipsum, Dolor, Sit, Amet, etc from all of the <span> using Selenium and python you have to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

• Using CSS_SELECTOR and get_attribute("innerHTML"):

  print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "span.name")))])
  

• Using XPATH and text attribute:

  print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//span[@class='name']")))])
  

• Console Output:

  ['Lorem', 'Ipsum', 'Dolor', 'Sit', 'Amet']
  

Note: This is list of type string and you can manipulate according to your requirement.

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

---

Outro

Link to useful documentation:

• get_attribute() method Gets the given attribute or property of the element.
• text attribute returns The text of the element.
• Difference between text and innerHTML using Selenium