Can you increase DOM and {-page.source-} limit to 1GB?

[Cyrix]

Currently DOM and {-page.source-} are limited to 10MB which is way too small for today's websites which use infinite auto scrolls (meaning they infinitely expand pages when page down is pressed, e.g Twitter, Google+, Pinterest, Facebook, all the big guys).

This problem was mentioned by other users before: http://zennolab.com/discussion/threads/zenno-twitter-scraper-dom-dissapearing.11719/

Because of the DOM limit, currently we have to use action designer loops to work with such pages. It consumers a whole core of CPU resource, and it takes forever.

This is a new trend in web design on the Internet, and we need to make adjustment too.
Can you please increase the limit to 1GB?
If not possible, please consider 500MB, or a number as high as other constrains allow.

Thank you very much.

 

[rostonix]

You can increase this setting: http://www.evernote.com/shard/s128/sh/5170d26d-b6c5-4476-b090-235a38b87b80/b6d49519f3946c2d5b1cfddab22b7168
We do not suggest to increase it to 500 Mb or 1 GB
Try to set a little bit higher than 10 MB and test it, increase step by step and test it.
Our main developer said that he thinks the amount of 200-300 Mb will shut down almost any PC)

 

[Cyrix]

Can you think of a work around like saving the whole page into a txt file, then regex that text file?

 

[rostonix]

You can save it , why not?

 

[Cyrix]

Can you give more detailed information?


1. What action to save a webpage into a txt file?
Is it using Browser->Download to? This one seems to save the page into a folder, not a file. How do I get that file name?

2. To regex a txt file, I need to use File->take txt to a variable. But does this variable subject to the 10MB limit when I regex it?

 

[Nick]

How about taking a "Data" - "File" action ("Save text" option) and saving a variable {-Page.Source-} there directly?

 

[Cyrix]

{-Page.Source-} has a limit of 10MB which is the problem in OP that we are looking for a work around.

 

[rostonix]

{-Page.Dom-} is not limited.
This setting http://zennolab.com/discussion/threads/can-you-increase-dom-and-page-source-limit-to-1gb.15164/#post-92212 affects limits for sharing data between processes. base.exe and ZennoPoster.exe in this case

 

[Cyrix]

But here you said DOM is limited to 10MB
http://zennolab.com/discussion/threads/zenno-twitter-scraper-dom-dissapearing.11719/

and that's the original problem we want to solve in this post.

 

[rostonix]

Settings I talked about is new. And I explained how it works)

 

[Cyrix]

You mean that setting will be out in a new version?
I couldn't find it in the latest version.
http://www.evernote.com/shard/s128/...235a38b87b80/b6d49519f3946c2d5b1cfddab22b7168

 

[lokiys]

You mean that setting will be out in a new version?
I couldn't find it in the latest version.
http://www.evernote.com/shard/s128/...235a38b87b80/b6d49519f3946c2d5b1cfddab22b7168

It is there, at least in v5.4.3.0

But if you need to scrape twitter then use mobile version

 

[Cyrix]

This is my Other Settings in ZP which is different?
If that '1' is where you mean then mine doesn't have the two arrows to increase/decrease it.
Also it should appear as 10 rather than 1. Missing the control in my ZP.

 

[lokiys]

"The size of exchange buffer between the processes" is option you have to change

 

[Cyrix]

1. Will increasing this number to 100 slow down ZP for all projects? If so by how much?

2. Will increasing this number to 100 increase memory usage for base.exe and zennoposter.exe? If so by how much?

Thanks.

 

[Cyrix]

I tested increasing that 'exchange buffer' setting from 10 to 150, but found that it did NOT increase the size limit of {-Page.Dom-}. Previous at 10, Dom starts to become empty at about 80 twitter follower pages. Now at 150, it still starts to become empty at about 80 pages.

 

[rostonix]

Did you restart sofware after changing settings?

 

[Cyrix]

Yes, restarted ZP at every 50 increment.

When the setting was at 10mb, the dom failed at a page of 10k followers.
Last night I tried 200mb, but still couldn't get Dom of a page of 10k followers (not really a big number).

 

[rostonix]

You can send us results of Daignostic.exe + your project in executable format on other PC to [email protected]