[FIXED] Trying to scrape contents between two tags

Issue

So i'm trying to scrape this page https://www.rxlist.com/abilify-drug.htm and i'm stuck in this problem where i have to extract contents between two tags. The HTML code of the page is as follows:-

<div class="monograph_cont"><h4>What Is Abilify?</h4>
<p>Abilify (aripiprazole) is a psychotropic drug (antipsychotic) that alters brain chemical activity used to treat:</p> 
<ul>
<li>schizophrenia,</li> 
<li>mania,</li> 
<li>depression,</li> 
<li>bipolar disorders,</li> 
<li>autistic disorder, and</li> 
<li>some irritable behavior disorders.</li> 
</ul>
<p>Generic Abilify is not available in the U.S., but is available in other countries under the name aripiprazole.</p>

<h4>What Are Side Effects of Abilify?</h4>
<p>Abilify may cause serious side effects including:</p>

<ul>
    <li>severe agitation or distress,</li>
    <li>feeling restless,</li>
    <li>twitching or uncontrollable movements of your eyes, lips, tongue, face, arms or legs,</li>
    <li>mask-like appearance of the face,</li>
    <li>trouble swallowing,</li>
    <li>speech problems,</li>
    <li>seizure (convulsions),</li>
    <li>thoughts of suicide or self-harm,</li>
    <li>stiff or rigid muscles,</li>
    <li>high fever,</li>
    <li>sweating,</li>
    <li>confusion,</li>
    <li>fast or uneven heartbeats,</li>
    <li>tremors (shaking),</li>
    <li>feeling lightheaded,</li>
    <li>chills,</li>
    <li>sore throat,</li>
    <li>mouth sores,</li>
    <li>skin sores,</li>
    <li>cough,</li>
    <li>difficulty breathing,</li>
    <li>increased thirst,</li>
    <li>increased urination,</li>
    <li>dry mouth, and</li>
    <li>fruity breath odor</li>
</ul>

<p>Get medical help right away, if you have any of the symptoms listed above.</p>
<p>Common side effects of Abilify include:</p>
<ul>
    <li>dizziness</li>
    <li>lightheadedness</li>
    <li>drowsiness</li>
    <li>weakness</li>
    <li>lightheadedness</li>
    <li>nausea</li>
    <li>vomiting</li>
    <li>stomach upset</li>
    <li>tiredness</li>
    <li>excess saliva or drooling</li>
    <li>choking or trouble swallowing</li>
    <li>blurred vision</li>
    <li>headache</li>
    <li>anxiety</li>
    <li>weight gain</li>
    <li>drowsiness</li>
    <li>sleep problems (insomnia)</li>
    <li>constipation</li>
</ul>
<p>Suicidal thoughts are may occur in some patients, especially children, teens, and young adults. Tell your doctor if this occurs. Other serious side effects of Abilify include:</p>
<ul>
    <li>tardive dyskinesia (involuntary, repetitive movements)</li>
    <li>neuroleptic malignant syndrome</li>
    <li>shaking (tremors)</li>
    <li>muscle spasms</li>
    <li>fainting</li>
    <li>mental/mood changes (such as increased anxiety, depression, suicidal thoughts)</li>
    <li>trouble swallowing</li>
    <li>restlessness (especially in the legs)</li>
    <li>mask-like expression of the face</li>
    <li>seizures</li>
    <li>signs of infection (such as fever, persistent sore throat) </li>
</ul>

<p>Seek medical care or call 911 at once if you have the following serious side effects:</p>
<ul>
<li>Serious eye symptoms such as sudden vision loss, blurred vision, tunnel vision, eye pain or swelling, or seeing halos around lights;</li>
<li>Serious heart symptoms such as fast, irregular, or pounding heartbeats; fluttering in your chest; shortness of breath; and sudden dizziness, lightheadedness, or passing out;</li>
<li>Severe headache, confusion, slurred speech, arm or leg weakness, trouble walking, loss of coordination, feeling unsteady, very stiff muscles, high fever, profuse sweating, or tremors.</li>
</ul>

<p>This document does not contain all possible side effects and others may occur. Check with your physician for additional information about side effects.</p>


<h4>Dosage for Abilify</h4>
<p>Abilify is available in tablet, orally disintegrating tablets, oral solution and injectable formulations. Dosage is variable and depends on multiple factors such as the ongoing mental problem, patient age, and other factors to be determined by the prescribing doctor.</p> 

<h4>Abilify In Children</h4>
<p>Abilify has been used in the pediatric population but such use should be discussed with a pediatric specialist.</p> 

<h4>What Drugs, Substances, or Supplements Interact with Abilify?</h4>
<p>Abilify may interact with other medicines that make you sleepy such as:</p> 
<ul>
<li>cold or allergy medicine,</li> 
<li>narcotic pain medicine,</li> 
<li>sleeping pills,</li> 
<li>muscle relaxers, and</li> 
<li>medicine for seizures, depression, or anxiety),</li> 
<li>medications to treat high blood pressure or a heart condition, 
</li><li>carbamazepine,</li> 
<li>phenobarbital,</li> 
<li>phenytoin,</li> 
<li>rifabutin,</li> 
<li>rifampin,</li> 
<li>ketoconazole,</li> 
<li>itraconazole,</li> 
<li>quinidine,</li> 
<li>fluoxetine,</li> 
<li>fluvoxamine, or</li> 
<li>paroxetine</li>
</ul> 

<p>Tell your doctor all medications and supplements you use.</p> 

<h4>Abilify During Pregnancy and Breastfeeding</h4>
<p>Benefits should outweigh risks in pregnant women. Women who are breastfeeding should not take Abilify.</p>


<h4>Additional Information</h4>
<p>Our Abilify Side Effects Drug Center provides a comprehensive view of available drug information on the potential side effects when taking this medication.
</p></div>

From this i'm trying to extract the whole block of content between <h4>What Are Side Effects of Abilify?</h4> and <h4>Dosage for Abilify</h4>

I have tried this response.css('#apPage > div > div.monograph_cont > h4:nth-child(5) ~ *:not(h4)::text').getall() but it only gives me the output with the tag of <p> and not the <li> elements and plus this returns all the text after <h4>Dosage for Abilify</h4> which i dont want anyway.

Kind of new here so confused!

Solution

Probably the most elegant solution would be to use regex to isolate the section that you are targeting, and then manually add it to a scrapy.Selector and extract the text from there.

For example:

def parse(self, response):
    pattern = re.compile(r"(?<=of Abilify\?</h4>).*?(?=<h4>)", re.DOTALL)
    content = response.css("div.monograph_cont").re(pattern)
    yield {'text':scrapy.Selector(text=content[0]).xpath("//text()").getall())}

Output:

{'text': ['Abilify may cause serious side effects including:', '\r\n\r\n', '\r\n\t', 'severe agitation or distress,', '\r\n\t', 'feeling restless,', '\r\n\t', 'twitching or uncontrollable
movements of your eyes, lips, tongue, face, arms or legs,', '\r\n\t', 'mask-like appearance of the face,', '\r\n\t', 'trouble swallowing,', '\r\n\t', 'speech problems,', '\r\n\t', 'seizure
 (convulsions),', '\r\n\t', 'thoughts of suicide or self-harm,', '\r\n\t', 'stiff or rigid muscles,', '\r\n\t', 'high fever,', '\r\n\t', 'sweating,', '\r\n\t', 'confusion,', '\r\n\t', 'fas
t or uneven heartbeats,', '\r\n\t', 'tremors (shaking),', '\r\n\t', 'feeling lightheaded,', '\r\n\t', 'chills,', '\r\n\t', 'sore throat,', '\r\n\t', 'mouth sores,', '\r\n\t', 'skin sores,'
, '\r\n\t', 'cough,', '\r\n\t', 'difficulty breathing,', '\r\n\t', 'increased thirst,', '\r\n\t', 'increased urination,', '\r\n\t', 'dry mouth, and', '\r\n\t', 'fruity breath odor', '\r\n'
, '\r\n\r\n', 'Get medical help right away, if you have any of the symptoms listed above.', '\r\n', 'Common side effects of Abilify include:', '\r\n', '\r\n\t', 'dizziness', '\r\n\t', 'lig
htheadedness', '\r\n\t', 'drowsiness', '\r\n\t', 'weakness', '\r\n\t', 'lightheadedness', '\r\n\t', 'nausea', '\r\n\t', 'vomiting', '\r\n\t', 'stomach upset', '\r\n\t', 'tiredness', '\r\n\
t', 'excess saliva or drooling', '\r\n\t', 'choking or trouble swallowing', '\r\n\t', 'blurred vision', '\r\n\t', 'headache', '\r\n\t', 'anxiety', '\r\n\t', 'weight gain', '\r\n\t', 'drows
iness', '\r\n\t', 'sleep problems (insomnia)', '\r\n\t', 'constipation', '\r\n', '\r\n', 'Suicidal thoughts are may occur in some patients, especially children, teens, and young adults. Te
ll your doctor if this occurs. Other serious side effects of Abilify include:', '\r\n', '\r\n\t', 'tardive dyskinesia (involuntary, repetitive movements)', '\r\n\t', 'neuroleptic malignant
 syndrome', '\r\n\t', 'shaking (tremors)', '\r\n\t', 'muscle spasms', '\r\n\t', 'fainting', '\r\n\t', 'mental/mood changes (such as increased anxiety, depression, suicidal thoughts)', '\r\
n\t', 'trouble swallowing', '\r\n\t', 'restlessness (especially in the legs)', '\r\n\t', 'mask-like expression of the face', '\r\n\t', 'seizures', '\r\n\t', 'signs of infection (such as fe
ver, persistent sore throat) ', '\r\n', '\r\n\r\n', 'Seek medical care or call 911 at once if you have the following serious side effects:', '\r\n', '\r\n', 'Serious eye symptoms such as s
udden vision loss, blurred vision, tunnel vision, eye pain or swelling, or seeing halos around lights;', '\r\n', 'Serious heart symptoms such as fast, irregular, or pounding heartbeats; fl
uttering in your chest; shortness of breath; and sudden dizziness, lightheadedness, or passing out;', '\r\n', 'Severe headache, confusion, slurred speech, arm or leg weakness, trouble walk
ing, loss of coordination, feeling unsteady, very stiff muscles, high fever, profuse sweating, or tremors.', '\r\n', '\r\n\r\n', 'This document does not contain all possible side effects a
nd others may occur. Check with your physician for additional information about side effects.']}

Answered By - Alexander