Zenno Twitter Scraper DOM Dissapearing

[allsystems]

Hi,

I am working on a Twitter Follower Scraper. Template is pretty simple, its goes to Twitter User Followers page, scrolls down page then stops, takes DOM, Regex, save to file. Problem is when I am user page with 50k followers and I am trying to scrape the followers the DOM wont load when the page is after 2.5k followers. Im not sure what issue is. I am working on 32GB dedicated server so it isnt memory. Any suggestions how to fix this?

 

[rostonix]

Does ordinary browser loads more than 2.5 followers?

 

[allsystems]

Yeh in ordinary browser and also zenno browser it loads all 20/100k followers, just does not scrape all followers

 

[rostonix]

What errors do you get?

 

[lokiys]

No errors Anton just blank DOM page.
I can confirm that this problem exist.

 

[ZennoScript]

Я парсил с одного сайта за одно взятие по 30-40 тысяч совпадений. (Не с твитера)
В твитере они то все отображаются, но по моему их там нужно подгружать, т.е. пролистать страницу вниз, чтобы все они отобразились на странице.

 

[rostonix]

Limit for DOM is 10 Mb.

You can create loop and by using action designer take necessary data from element by changing it's number of match.

 

[allsystems]

Limit for DOM is 10 Mb.

You can create loop and by using action designer take necessary data from element by changing it's number of match.

How will this work on a profile with 250k followers, the loop will be huge and ram buildup will not allow it to happen

 

[rostonix]

How will this work on a profile with 250k followers, the loop will be huge and ram buildup will not allow it to happen

These are operation with text. It doesn't use a lot of resources. But instance showing all 250K followers might crash i think. Like any other browser.