How To Increase Dom Limit

[allsystems]

Zennoposter is limited to hold 10mb in DOM. I have a laptop with 16GB of ram. How can I increase this as I want to scrape Twitter user with 200k followers. It give up after 2k.

Thanks

 

[rostonix]

You'll not be able to load page with 200K followers. Just think about it and try to load it in standard browser. It will crash I bet.
You may increase 10mb but i would not suggest to increase it more than to 150-200Mb on good machines.

 

[LexxWork]

I will make a try to read the twiter data through the fiddler. If the topic does not get lost, I'll tell you if I could do this.

 

[lokiys]

Zennoposter is limited to hold 10mb in DOM. I have a laptop with 16GB of ram. How can I increase this as I want to scrape Twitter user with 200k followers. It give up after 2k.

Thanks

Simple use mobile version of twitter

 

[LexxWork]

During the research of the full site version fiddlercore didn't handled the task because twitter uses sniff-protection. On the mobile version through the project maker, I was not able duplicate requests successfully too.
The only good solution I found when have sniffed a mobile version through Chrome and made a get queries tamplate that needs non authorized cookies and two cursors - previous and current to work. Every request contains of 20 followers.
As a result, the template can collect the full followers base. After two hours of the test the template has collected 100k followers.

 

[drvosjeca]

made a get queries tamplate that needs non authorized cookies and two cursors - previous and current to work. Every request contains of 20 followers.

I would love to see how you did this if you are willing to share.

 

[Tobbe]

I would love to see how you did this if you are willing to share.

Me too, @LexxWork

 

[LexxWork]

did you try by yourself? First try to get needed data with Chrom + Fiddler. Search for cursors in header request and body response in followers streams. So, if you've got them, the yeasy way to check the workin process to make next request manualy by the fiddler. That's all. Good luck!

 

[allsystems]

Logged in and saw this today. Would be awesome to see that template

 

[LexxWork]

realy? send me your skype contact and I'll show you.