Tuesday, December 19, 2023

[FIXED] How to scrape node's text which has  inside?

December 19, 2023 python, scrapy, web-scraping No comments

Issue

I am writing a web crawler for scraping information from job boards. I completed my first crawler, but it has some more issues to resolve.

For some companies titles I get this ПАО\xa0 as a result. This text in cyrillic, but I request and save it in UTF-8 encoding. Attribute and content inspection of node for this cases shown this text content:

ПАО
<!---->
'company's name'

This  prevents scraper to do it work well and I haven't resolved yet this issue. Have you faced with this during your experience with scraping and can you suggest a right way to handle this?

I use scrapy to process it.

UPDATE here is the code (endpoint is hidden because their /robots.txt deny all crawlers except from the key search engines)

import scrapy 

class HHSpider(scrapy.Spider):
    name = 'hh-spider'
    start_urls = [
        'https:<ENDPOINT>'
        ]

    def __init__(self):
        self.BASE_URL = 'https://hh.ru'
        self.JOB_SELECTOR = '.vacancy-serp-item-body'
        self.JOB_TITLE_SELECTOR = '.serp-item__title::text'
        self.JOB_COMPANY_SELECTOR = '.bloko-link_kind-tertiary::text'
        self.JOB_COMPANY_URL_SELECTOR = '.bloko-link_kind-tertiary::attr(href)'
        self.JOB_COMPENSATION_SELECTOR = '.bloko-header-section-2::text'
        self.NEXT_SELECTOR = '.bloko-button[data-qa="pager-next"]::attr(href)'

    def parse(self, response):
        for vacancy in response.css(self.JOB_SELECTOR): 
            yield {
                'jobTitle' : vacancy.css(self.JOB_TITLE_SELECTOR).get(),
                'compensation' : vacancy.css(self.JOB_COMPENSATION_SELECTOR).get(),
                'company' : vacancy.css(self.JOB_COMPANY_SELECTOR).get(),
                'companyUrl' : self.BASE_URL + vacancy.css(self.JOB_COMPANY_URL_SELECTOR).get()
            }

        next_page = response.css(self.NEXT_SELECTOR).get()

        if next_page is not None:
            yield scrapy.Request(response.urljoin(next_page))

Solution

An easy solution to this issue is to simply use the getall() method on your selectors instead of the get() method. This returns all of the text both before and after the comment in the form of a list. Then you can use something like ''.join([...]) in order to rejoin the text back together.

For example:

import scrapy

class HHSpider(scrapy.Spider):
    name = 'hh-spider'
    start_urls = [
        'https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_by=publication_time'
        ]

    def __init__(self):
        self.BASE_URL = 'https://hh.ru'
        self.JOB_SELECTOR = '.vacancy-serp-item-body'
        self.JOB_TITLE_SELECTOR = '.serp-item__title::text'
        self.JOB_COMPANY_SELECTOR = '.bloko-link_kind-tertiary::text'
        self.JOB_COMPANY_URL_SELECTOR = '.bloko-link_kind-tertiary::attr(href)'
        self.JOB_COMPENSATION_SELECTOR = '.bloko-header-section-2::text'
        self.NEXT_SELECTOR = '.bloko-button[data-qa="pager-next"]::attr(href)'

    def parse(self, response):
        for vacancy in response.css(self.JOB_SELECTOR):
            yield {
                'jobTitle' : vacancy.css(self.JOB_TITLE_SELECTOR).get(),
                'compensation' : ''.join(vacancy.css(self.JOB_COMPENSATION_SELECTOR).getall()),
                'company' : ''.join(vacancy.css(self.JOB_COMPANY_SELECTOR).getall()),
                'companyUrl' : self.BASE_URL + vacancy.css(self.JOB_COMPANY_URL_SELECTOR).get()
            }
        next_page = response.css(self.NEXT_SELECTOR).get()
        if next_page is not None:
            yield scrapy.Request(response.urljoin(next_page))

Partial Ouptut:

{'jobTitle': 'Tech Lead (Python)', 'compensation': '', 'company': 'Домклик', 'companyUrl': 'https://hh.ru/employer/2136954?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:23 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time>
{'jobTitle': 'Мобильный разработчик', 'compensation': '', 'company': 'GLOBALPAS', 'companyUrl': 'https://hh.ru/employer/5745172?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:23 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time>
{'jobTitle': 'Java Developer', 'compensation': '', 'company': 'Домклик', 'companyUrl': 'https://hh.ru/employer/2136954?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:23 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time>
{'jobTitle': 'Android developer', 'compensation': '80\u202f000 – 200\u202f000 ₽', 'company': 'iSimpleLab', 'companyUrl': 'https://hh.ru/employer/1081773?hhtmFrom=vacancy_search_lis
t'}
2023-09-23 07:53:23 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time>
{'jobTitle': 'Back-end TechLead (C#)', 'compensation': '', 'company': 'Municorn', 'companyUrl': 'https://hh.ru/employer/4544421?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:23 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time>
{'jobTitle': 'Android Developer', 'compensation': '170\u202f000 – 210\u202f000 ₽', 'company': 'Ювенко', 'companyUrl': 'https://hh.ru/employer/3551?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:23 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time>
{'jobTitle': 'Senior Android developer в DaGama', 'compensation': '', 'company': 'ООО\xa0ПИ БИ АЙ ФИНАНС', 'companyUrl': 'https://hh.ru/employer/9615287?hhtmFrom=vacancy_search_lis
t'}
2023-09-23 07:53:23 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time>
{'jobTitle': 'Senior/Lead Unity Games Developer (with good English)', 'compensation': '3\u202f500 – 4\u202f500 $', 'company': 'DATASAKURA', 'companyUrl': 'https://hh.ru/employer/34
81573?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:23 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time>
{'jobTitle': 'Android разработчик (Mobile Core Platform)', 'compensation': '', 'company': 'Домклик', 'companyUrl': 'https://hh.ru/employer/2136954?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:23 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time>
{'jobTitle': 'IOS разработчик (Mobile Core Platform)', 'compensation': '', 'company': 'Домклик', 'companyUrl': 'https://hh.ru/employer/2136954?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:23 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time>
{'jobTitle': 'Android разработчик', 'compensation': 'от 150\u202f000 ₽', 'company': 'ООО\xa0Ансгар', 'companyUrl': 'https://hh.ru/employer/9669362?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:23 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time>
{'jobTitle': 'Разработчик мобильных приложений (Android/IOS)', 'compensation': '', 'company': 'ООО\xa0Эрливидео', 'companyUrl': 'https://hh.ru/employer/1612009?hhtmFrom=vacancy_sea
rch_list'}
2023-09-23 07:53:23 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time>
{'jobTitle': 'Middle Android разработчик (удаленно)', 'compensation': '', 'company': 'GetTask', 'companyUrl': 'https://hh.ru/employer/4769783?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:23 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time>
{'jobTitle': 'Инженер-программист', 'compensation': 'от 150\u202f000 ₽', 'company': 'LOGISTIX', 'companyUrl': 'https://hh.ru/employer/143735?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:23 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time>
{'jobTitle': 'Software Documentation Developer', 'compensation': '150\u202f000 – 180\u202f000 ₽', 'company': 'documentat.io', 'companyUrl': 'https://hh.ru/employer/3498254?hhtmFrom
=vacancy_search_list'}
2023-09-23 07:53:23 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time>
{'jobTitle': 'Разработчик приложений с Webview (Flutter)', 'compensation': '80\u202f000 – 160\u202f000 ₽', 'company': 'Toxic Media', 'companyUrl': 'https://hh.ru/employer/4005992?h
htmFrom=vacancy_search_list'}
2023-09-23 07:53:23 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time>
{'jobTitle': 'Программист-разработчик игр и приложений', 'compensation': 'от 100\u202f000 ₽', 'company': 'ПАО\xa0Event Technologies', 'companyUrl': 'https://hh.ru/employer/2653801?
hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list> (referer: https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_by
=publication_time)
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Java разработчик (проект "Игры")', 'compensation': '', 'company': 'VK, Одноклассники', 'companyUrl': 'https://hh.ru/employer/15478?dpt=mailru-15478-od&hhtmFrom=vacanc
y_search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'IOS Developer', 'compensation': '', 'company': 'ООО\xa0Вебинар Технологии', 'companyUrl': 'https://hh.ru/employer/534148?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Android разработчик', 'compensation': '', 'company': 'Домклик', 'companyUrl': 'https://hh.ru/employer/2136954?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Разработчик мобильных приложений/Android Developer - webview', 'compensation': 'от 1\u202f000 $', 'company': 'ИП\xa0Старовойтов А.Г', 'companyUrl': 'https://hh.ru/emp
loyer/3582113?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Senior/Lead Android developer (SberBrowser)', 'compensation': '', 'company': 'Сбер для экспертов', 'companyUrl': 'https://hh.ru/employer/3529?dpt=3529-3529-prof&hhtmF
rom=vacancy_search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Android Developer (Middle+ / Senior)', 'compensation': '', 'company': 'ООО\xa0ФТК Менеджмент', 'companyUrl': 'https://hh.ru/employer/62981?hhtmFrom=vacancy_search_lis
t'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Android-разработчик ГПХ', 'compensation': '', 'company': '«РОСБАНК»', 'companyUrl': 'https://hh.ru/employer/599?dpt=rosbank-599-opyt&hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Android программист', 'compensation': '110\u202f000 – 190\u202f000 ₽', 'company': 'БИФИТ', 'companyUrl': 'https://hh.ru/employer/40912?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Android developer', 'compensation': '', 'company': 'ООО\xa01221Системс', 'companyUrl': 'https://hh.ru/employer/6067730?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Flutter Программист', 'compensation': '80\u202f000 – 180\u202f000 ₽', 'company': 'ООО\xa0Инмедикум. Онлайн', 'companyUrl': 'https://hh.ru/employer/9755671?hhtmFrom=va
cancy_search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Android разработчик, Департамент разработки Фреш', 'compensation': '', 'company': 'Ozon Информационные технологии', 'companyUrl': 'https://hh.ru/employer/2180?dpt=ozo
nru-2180-dev&hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Frontend разработчик (TypeScript - React)', 'compensation': '', 'company': 'Отклик-АДВ', 'companyUrl': 'https://hh.ru/employer/3867157?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Старший Android разработчик', 'compensation': '', 'company': 'билайн: ИТ, Data, Digital', 'companyUrl': 'https://hh.ru/employer/4934?dpt=bil-4934-it&hhtmFrom=vacancy_
search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Linux Kernel Developer', 'compensation': '', 'company': 'ООО\xa0СалютДевайсы', 'companyUrl': 'https://hh.ru/employer/4659934?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Security Software Engineer', 'compensation': '', 'company': 'ООО\xa0СалютДевайсы', 'companyUrl': 'https://hh.ru/employer/4659934?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Android разработчик (Мобильный Банк)', 'compensation': '', 'company': 'Тинькофф', 'companyUrl': 'https://hh.ru/employer/78638?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Middle+ React Native разработчик', 'compensation': '100\u202f000 – 150\u202f000 ₽', 'company': 'Elgrow', 'companyUrl': 'https://hh.ru/employer/2988566?hhtmFrom=vacanc
y_search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Android Developer', 'compensation': '', 'company': 'ООО\xa0Максилект', 'companyUrl': 'https://hh.ru/employer/2563058?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Разработчик (iOS + Android) удаленно', 'compensation': '', 'company': 'Бэнкс Софт Системс', 'companyUrl': 'https://hh.ru/employer/2575?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:24 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=1&hhtmFrom=vacancy_search_list>
{'jobTitle': 'IOS-разработчик', 'compensation': '220\u202f000 – 350\u202f000 ₽', 'company': 'АО\xa0Мидлэнд Ритейл Груп', 'companyUrl': 'https://hh.ru/employer/3370790?hhtmFrom=vaca
ncy_search_list'}
2023-09-23 07:53:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list> (referer: https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_by
=publication_time&page=1&hhtmFrom=vacancy_search_list)
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Frontend Developer (Vue.js)', 'compensation': '150\u202f000 – 200\u202f000 ₽', 'company': 'ООО\xa0Альянс Реал Эстейт', 'companyUrl': 'https://hh.ru/employer/9407275?h
htmFrom=vacancy_search_list'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Системный программист в отдел адаптации ОС Аврора', 'compensation': '', 'company': 'Открытая мобильная платформа', 'companyUrl': 'https://hh.ru/employer/2300703?hhtmF
rom=vacancy_search_list'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Senior React Native Developer в команду мобильного приложения для клиентов', 'compensation': '', 'company': 'Профи (profi.ru)', 'companyUrl': 'https://hh.ru/employer/
83639?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Lead Java разработки', 'compensation': '', 'company': 'Whoosh (ООО Вуш)', 'companyUrl': 'https://hh.ru/employer/3536822?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Senior Frontend developer', 'compensation': '', 'company': 'BestDoctor', 'companyUrl': 'https://hh.ru/employer/2730772?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Golang разработчик', 'compensation': '', 'company': 'Домклик', 'companyUrl': 'https://hh.ru/employer/2136954?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Lead Python разработчик', 'compensation': '', 'company': 'Домклик', 'companyUrl': 'https://hh.ru/employer/2136954?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Android Developer Middle, удаленно', 'compensation': 'от 230\u202f000 ₽', 'company': 'Кадровое агентство ЕВА', 'companyUrl': 'https://hh.ru/employer/2812717?hhtmFrom=
vacancy_search_list'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Senior React Native developer', 'compensation': '3\u202f000 – 3\u202f500 $', 'company': 'ООО\xa0ОнТаргет ЛАБС', 'companyUrl': 'https://hh.ru/employer/850334?hhtmFrom=
vacancy_search_list'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Senior android-разработчик', 'compensation': '', 'company': 'STARTEAM', 'companyUrl': 'https://hh.ru/employer/4844395?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Flutter разработчик', 'compensation': 'от 120\u202f000 ₽', 'company': 'ИП\xa0Кравченко Е.А.', 'companyUrl': 'https://hh.ru/employer/2551320?hhtmFrom=vacancy_search_li
st'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Программист Delphi', 'compensation': 'до 250\u202f000 ₽', 'company': 'Кадровый холдинг ПРОФИЛЬ', 'companyUrl': 'https://hh.ru/employer/1416424?hhtmFrom=vacancy_search
_list'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Android Developer', 'compensation': 'до 300\u202f000 ₽', 'company': 'Remokate', 'companyUrl': 'https://hh.ru/employer/5539742?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Андроид - разработчик', 'compensation': '', 'company': 'Телеканал Россия, ГТК', 'companyUrl': 'https://hh.ru/employer/27989?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Senior C++ разработчик в команду Core', 'compensation': '', 'company': 'Movavi', 'companyUrl': 'https://hh.ru/employer/236065?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': '1С:Эксперт по технологическим вопросам', 'compensation': '', 'company': 'ЗАО\xa01С', 'companyUrl': 'https://hh.ru/employer/882?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Senior Android Developer / Team Lead', 'compensation': '', 'company': 'ООО\xa0Макс', 'companyUrl': 'https://hh.ru/employer/6016502?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'IOS developer (middle/senior)', 'compensation': 'от 250\u202f000 ₽', 'company': 'КИБЕР-РОМ', 'companyUrl': 'https://hh.ru/employer/5599143?hhtmFrom=vacancy_search_lis
t'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Android Developer (WebView) / Android Разработчик (Java/Kotlin)', 'compensation': '150\u202f000 – 200\u202f000 ₽', 'company': 'ООО\xa0Смурфэпп', 'companyUrl': 'https:
//hh.ru/employer/3576996?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=2&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Инженер по разработке встроенного ПО мобильных устройств', 'compensation': '', 'company': 'Fplus', 'companyUrl': 'https://hh.ru/employer/6836?hhtmFrom=vacancy_search_
list'}
2023-09-23 07:53:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=3&hhtmFrom=vacancy_search_list> (referer: https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_by
=publication_time&page=2&hhtmFrom=vacancy_search_list)
2023-09-23 07:53:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=3&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Senior Android Developer', 'compensation': '', 'company': 'СИГМА', 'companyUrl': 'https://hh.ru/employer/238161?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=3&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Flutter разработчик (remote)', 'compensation': '', 'company': 'Lucy in the Sky Inc', 'companyUrl': 'https://hh.ru/employer/5259760?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=3&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Программист (embedded)', 'compensation': '', 'company': 'АО\xa0НЭПТ', 'companyUrl': 'https://hh.ru/employer/68703?hhtmFrom=vacancy_search_list'}
2023-09-23 07:53:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://hh.ru/search/vacancy?area=1&ored_clusters=true&professional_role=96&search_period=30&text=Android&order_b
y=publication_time&page=3&hhtmFrom=vacancy_search_list>
{'jobTitle': 'Разработчик Unity (Middle)', 'compensation': '100\u202f000 – 150\u202f000 ₽', 'company': 'ООО\xa0Нектар', 'companyUrl': 'https://hh.ru/employer/2979377?hhtmFrom=vacan
cy_search_list'}

Answered By - Alexander

This Answer collected from stackoverflow and tested by PythonFixing community admins, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Tuesday, December 19, 2023

[FIXED] How to scrape node's text which has  inside?

Issue

Solution

0 comments:

Post a Comment

Popular Posts

Labels