Parsing HTML tree in lxml : how can I retrieve the text inside the element?

Question

I'm trying to retrieve the correct text inside an element. Here is the output:

(Pdb) p etree.tostring(els[0])
'<h5 class="msg-delivered" style="padding:0;text-rendering:optimizeLegibility;line-height:1.1;margin-bottom:15px;-webkit-font-smoothing:antialiased;font-family:&quot;Open Sans&quot;, &quot;Helvetica Neue&quot;, Arial, Helvetica, sans-serif;color:#888888;vertical-align:middle;margin:0;font-size:13px;font-weight:300 !important">&#13;\n<i class="ic-icon-delivered" style="margin:0;padding:0;font-family:&quot;Open Sans&quot;, &quot;Helvetica Neue&quot;, &quot;Helvetica&quot;, Helvetica, Arial, sans-serif;text-rendering:optimizeLegibility;position:relative;background:url(https://d1s8987jlndkbs.cloudfront.net/assets/sprite-ratings-ee0696744f54df6536179c70e24217e3.png) no-repeat -12px -12px;background-size:132px 436px;display:none;vertical-align:middle;width:25px;height:25px;background-position:-16px -16px;top:0"/>&#13;\nYour order was delivered&#13;\non&#13;\n6/4&#13;\n@&#13;\n4:44 PM&#13;\n</h5>&#13;\n'
(Pdb) p els[0].text
'\r\n'

How can I get the string: "Your item was delivered on 6/4 at 4:40 PM"? I can use a regex on the etree.tostring() output, but wanted to know why the els[0].text option was not working?

Answer 1

You can try using xpath function string() which return concatenated value of all text nodes within current element :

import lxml.html
html = """<h5 class="msg-delivered" style="padding:0;text-rendering:optimizeLegibility;line-height:1.1;margin-bottom:15px;-webkit-font-smoothing:antialiased;font-family:&quot;Open Sans&quot;, &quot;Helvetica Neue&quot;, Arial, Helvetica, sans-serif;color:#888888;vertical-align:middle;margin:0;font-size:13px;font-weight:300 !important">&#13;\n<i class="ic-icon-delivered" style="margin:0;padding:0;font-family:&quot;Open Sans&quot;, &quot;Helvetica Neue&quot;, &quot;Helvetica&quot;, Helvetica, Arial, sans-serif;text-rendering:optimizeLegibility;position:relative;background:url(https://d1s8987jlndkbs.cloudfront.net/assets/sprite-ratings-ee0696744f54df6536179c70e24217e3.png) no-repeat -12px -12px;background-size:132px 436px;display:none;vertical-align:middle;width:25px;height:25px;background-position:-16px -16px;top:0"/>&#13;\nYour order was delivered&#13;\non&#13;\n6/4&#13;\n@&#13;\n4:44 PM&#13;\n</h5>"""
tree = lxml.html.etee.fromstring(html)
print(tree.xpath("string()"))

output :

'\r\n\r\nYour order was delivered\r\non\r\n6/4\r\n@\r\n4:44 PM\r\n'

Answer 2

If you want all the text, you can simply use:

els[0].text_content()

That is, given that you loaded the html with:

import lxml.html
html = """<h5 class="msg-delivered" style="padding:0;text-rendering:optimizeLegibility;line-height:1.1;margin-bottom:15px;-webkit-font-smoothing:antialiased;font-family:&quot;Open Sans&quot;, &quot;Helvetica Neue&quot;, Arial, Helvetica, sans-serif;color:#888888;vertical-align:middle;margin:0;font-size:13px;font-weight:300 !important">&#13;\n<i class="ic-icon-delivered" style="margin:0;padding:0;font-family:&quot;Open Sans&quot;, &quot;Helvetica Neue&quot;, &quot;Helvetica&quot;, Helvetica, Arial, sans-serif;text-rendering:optimizeLegibility;position:relative;background:url(https://d1s8987jlndkbs.cloudfront.net/assets/sprite-ratings-ee0696744f54df6536179c70e24217e3.png) no-repeat -12px -12px;background-size:132px 436px;display:none;vertical-align:middle;width:25px;height:25px;background-position:-16px -16px;top:0"/>&#13;\nYour order was delivered&#13;\non&#13;\n6/4&#13;\n@&#13;\n4:44 PM&#13;\n</h5>"""
tree = lxml.html.fromstring(html)

Note that you willl probably want to avoid lxml.html.etree.fromstring, and just use lxml.html.fromstring