ZennoPoster Memory Usage Increases & Increases Until it Crashes

[crazyflx]

ZennoPoster Memory Usage Increases & Increases Until it Crashes - Template Attached

I made a template that analyzes the top ten results of Google for a list of given keywords. The template works 100% perfectly, with the exception that ZP crashes after awhile. The template throws no error messages while running, and successfully completes with the message "Template C:\xxx\TopTen.xml is executed successfully!".

However, ZP does not seem to be releasing the used memory upon completion. For example, lets say I'm running just one thread. If I open up Windows Task Manager to view my processes, I've got ZennoPoster.exe & instance.exe running.

Initially, upon opening and starting this particular template, ZennoPoster.exe's "Memory (private working set)" (as viewed in Windows Task Manger) usage is at about 170,000K and instance.exe is at about 56,000K.

As the template works, the amount of memory used by each obviously increases. Instance.exe typically gets up to about 67,000K just before it completes. Once the ZP returns the message "Template C:\xxx\TopTen.xml is executed successfully!" the amount of memory used by Instance.exe drops all the way back down to what it was when it started.

However, ZennoPoster.exe's memory usage DOES NOT decrease. In fact, it goes up. It continues to go up and up, all the way to well over 1,300,000K just before it crashes. It takes about an hour and a half when running just one instance before it reaches that much memory usage and crashes, and in that hour and a half span, the template completes well over 20 or so times. It's crashing because ZennoPoster.exe is out of memory.

I can even run this particular template with multiple threads with no problems, but it just reaches the point at which it crashes that much faster.

And it isn't that my PC is out of memory, it's that ZP is reaching the maximum allowed memory usage for a 32-bit program.

Other templates I've created do not have this error, and will run just fine for hours/days at a time with no issues at all. I've written and rewritten and rewritten this particular template (and let me tell you, this is not pleasant as it is a rather large/complex template) and I keep getting the same results.

I'd be happy to share this template with anybody who might be able to tell me what the issue is, or if somebody could even care to venture a guess as to what the problem is.

 

[crazyflx]

Посмотреть вложение topten.xmlПосмотреть вложение topten.xml

I've attached the XML template file so maybe somebody could take a look and see what's going on. This is incredibly frustrating, because I'm very happy with the template and how it works, but I can't "walk away" from my computer and let it run, because ZP crashes every single time.

To run the template, simply save the template into any folder, and in that folder, create a text file and name it "keywords" and in that file, have a line separated list of keywords (use 40 or 50 keywords at least).

I've attached a text file that has the correct name and list of keywords within it, so you can simply download the text file and place it in the same folder as the template, and then just run the template.Посмотреть вложение keywords.txt

 

[Hungry Bulldozer]

A quick look on the template and seems there are a few loops which are definitely increasing memory usage. You should try to split these loops on a few templates or decrease their usage on minimum.

 

[crazyflx]

A quick look on the template and seems there are a few loops which are definitely increasing memory usage. You should try to split these loops on a few templates or decrease their usage on minimum.

I definitely could see loops increasing memory usage, but shouldn't that used memory be "released" upon the template being successfully executed? I mean, each individual instance is releasing the used memory, it's just the main ZennoPoster.exe that isn't.

 

[crazyflx]

I've reduced the template to only a mere shell of its former self trying to figure this out. I've made it so the "second half" of the template isn't even called into action. Here is what the template does now:

It checks if it has created the directory "Garbage Files" in the same folder the template is running out of. If it hasn't, it creates it and makes a copy of the "keywords.txt" file and places that copy in the "Garbage Files" directory.

Then, it takes a keyword (position 0) from the file, navigates to Google.com and searches that keyword. Then, it grabs the source HTML of the search results.

Then, it parses out the URLs of the search results page, and using a counter, determines which URL it is going to be using (0 through 9 for the top ten search results).

It then compares the input keyword to the URL (using javascript) to see if the keyword/phrase is contained within the URL. It then does this for the description as well as the anchor text.

It does this for the top ten search results, each time adding the result to a string. Then the template starts over.

I've made the template so it doesn't save any results file or really accomplish anything. It is STILL using tons of memory, never releasing the previously used memory. I've added a couple of 10 second pauses, so I can see just where all the memory is being used and not being released.

I've attached the crippled template to this post. At first I thought I had programmed something incorrectly, but there is definitely a bug that is causing the ZennoPoster.exe instance to not release used memory. I'm not sure if it is because of the heavy use of javascript or what.

Посмотреть вложение topten no save.xml

You can view the template in ZP project maker, and just walk through it really quick so you can see where the 10 second pauses are, then open the template up in ZP and run it using just one thread and open up Windows Task Manager. Watch ZennoPoster.exe's memory usage increase upon the templates COMPLETION.

 

[bigcajones]

I looked at your template earlier and came up with a couple of conclusions. It looks like ZP is not something out of memory. Could be the variables in the JS. I don't know.

I haven't looked at the new template yet, but if I could make a few suggestions. In my limited practice with the JS function in ZP, you don't have to put the JS Prep step in. In some of the cases where I've used JS macros, it comes out the same without it. Also, you could try pulling all of the results from the url scrape and description into one step instead of using a loop. If you need help with this, just add -|-all to your regex step and it will give you all 10 urls. Then you can do whatever you want to them. Store them in a temp file and delete the necessary info then. Running the template in debug mode I see a lot of steps that don't really do anything new that you haven't done before, just by looking at the results. Try using the regex builder more and less of the JS, especially adding values to variables. Maybe ZP is storing them after the template executes.

I do give you credit for knowing a shitload of JS code, though.

 

[crazyflx]

It really does look like it is the use of JS that is causing the infinite increase of memory usage within ZP. Somehow ZP must not be releasing the memory used by them.

@bigcajones - I looked over my template looking for places to remove the use of JS, and I'm afraid that I simply can't (not that it can't be done, I just can't do it, haha). I did however, remove the repetitive steps. After a closer look, I saw that you were right, and I was performing a lot of tasks repeatedly. Unfortunately, this hasn't fixed the problem.

The template compares the keyword it has searched, to the URL, Root Domain, Anchor & Description of each of the top ten search results to see if the keyword is contained within each/any of them.

It does this, by removing all spaces, special characters, stop words & capitalization from both the keyword and each of the above listed items it is looking in.

It does this, so it can do a ScrapedItem.search(keywordphrase) and see if the keyword is contained within said item.

 

[crazyflx]

Just as a for instance about what I mean by not being able to remove the use of JS, the one step in my template labelled "Clean Anchor & Check for Keyword" contains this JS:

This is the anchor text of the search result it is currently working on and turns it into a JS variable -> var iu = "{-FieldData.FieldData-|-●6●3●0●4●3●3●-|-≡1≡5≡4≡1≡5≡5≡5≡5≡3≡6≡-}";
Turns above variable into all lowercase - > iu = iu.toLowerCase();
Replaces all special characters with spaces -> var iuspecialreplacedwithspaces = (iu.replace(/(<([^>]+)>)/ig,"").replace(/!/gi," ").replace(/@/gi, " ").replace(/#/gi, " ").replace(/\$/gi, " ").replace(/%/gi, " ").replace(/\^/gi, " ").replace(/&/gi, " ").replace(/\*/gi, " ").replace(/\(/gi, " ").replace(/\)/gi, " ").replace(/_/gi, " ").replace(/-/gi, " ").replace(/\+/gi, " ").replace(/\=/gi, " ").replace(/{/gi, " ").replace(/}/gi, " ").replace(/\|/gi, " ").replace(/\[/gi, " ").replace(/\]/gi, " ").replace(/\\/gi, " ").replace(/:/gi, " ").replace(/;/gi, " ").replace(/"/gi, " ").replace(/'/gi, " ").replace(/\./gi, " ").replace(/,/gi, " ").replace(/</gi, " ").replace(/>/gi, " ").replace(/\?/gi, " ").replace(/\//gi, " ").replace(/(\r\n|\n|\r)/gm,""));

Turns keyword is searched into a variable -> var keyword = "{-FieldData.FieldData-|-●2●8●5●7●8●1●-|-≡1≡0≡9≡6≡9≡4≡8≡5≡0≡5≡-}";
Makes sure keyword is all lowercase -> keyword = keyword.toLowerCase();
Replaces all special characters with spaces -> var keywordspecialreplacedwithspaces = (keyword.replace(/!/gi, " ").replace(/@/gi, " ").replace(/#/gi, " ").replace(/\$/gi, " ").replace(/%/gi, " ").replace(/\^/gi, " ").replace(/&/gi, " ").replace(/\*/gi, " ").replace(/\(/gi, " ").replace(/\)/gi, " ").replace(/_/gi, " ").replace(/-/gi, " ").replace(/\+/gi, " ").replace(/\=/gi, " ").replace(/{/gi, " ").replace(/}/gi, " ").replace(/\|/gi, " ").replace(/\[/gi, " ").replace(/\]/gi, " ").replace(/\\/gi, " ").replace(/:/gi, " ").replace(/;/gi, " ").replace(/"/gi, " ").replace(/'/gi, " ").replace(/\./gi, " ").replace(/,/gi, " ").replace(/</gi, " ").replace(/>/gi, " ").replace(/\?/gi, " ").replace(/\//gi, " ").replace(/(\r\n|\n|\r)/gm,""));

Replaces all stop words in anchor text with nothing and removes spaces -> var iunostopwords = (iuspecialreplacedwithspaces.replace(/ and /gi, "").replace(/ a /gi, "").replace(/ n /gi, "").replace(/ i /gi, "").replace(/ about /gi, "").replace(/ an /gi, "").replace(/ are /gi, "").replace(/ as /gi, "").replace(/ at /gi, "").replace(/ be /gi, "").replace(/ by /gi, "").replace(/ too /gi, "").replace(/ for /gi, "").replace(/ from /gi, "").replace(/ how /gi, "").replace(/ in /gi, "").replace(/ is /gi, "").replace(/ it /gi, "").replace(/ of /gi, "").replace(/ on /gi, "").replace(/ or /gi, "").replace(/ that /gi, "").replace(/ the /gi, "").replace(/ this /gi, "").replace(/ to /gi, "").replace(/ was /gi, "").replace(/ what /gi, "").replace(/ when /gi, "").replace(/ where /gi, "").replace(/ who /gi, "").replace(/ will /gi, "").replace(/ with /gi, "").replace(/ the /gi, "").replace(/ /gi, "").replace(/(\r\n|\n|\r)/gm,""));

Replaces all special characters in anchor text with nothing, but leaves stop words alone and removes spaces -> var iuwithstopwords = (iu.replace(/!/gi, "").replace(/@/gi, "").replace(/#/gi, "").replace(/\$/gi, "").replace(/%/gi, "").replace(/\^/gi, "").replace(/&/gi, "").replace(/\*/gi, "").replace(/\(/gi, "").replace(/\)/gi, "").replace(/_/gi, "").replace(/-/gi, "").replace(/\+/gi, "").replace(/\=/gi, "").replace(/{/gi, "").replace(/}/gi, "").replace(/|/gi, "").replace(/\[/gi, "").replace(/\]/gi, "").replace(/\\/gi, "").replace(/:/gi, "").replace(/;/gi, "").replace(/"/gi, "").replace(/'/gi, "").replace(/\./gi, "").replace(/,/gi, "").replace(/</gi, "").replace(/>/gi, "").replace(/\?/gi, "").replace(/\//gi, "").replace(/ /gi, "").replace(/(\r\n|\n|\r)/gm,""));

Removes all stop words from keyword/phrase and removes spaces-> var keywordnostopwords = (keywordspecialreplacedwithspaces.replace(/ and /gi, "").replace(/ a /gi, "").replace(/ n /gi, "").replace(/ i /gi, "").replace(/ about /gi, "").replace(/ an /gi, "").replace(/ are /gi, "").replace(/ as /gi, "").replace(/ at /gi, "").replace(/ be /gi, "").replace(/ by /gi, "").replace(/ too /gi, "").replace(/ for /gi, "").replace(/ from /gi, "").replace(/ how /gi, "").replace(/ in /gi, "").replace(/ is /gi, "").replace(/ it /gi, "").replace(/ of /gi, "").replace(/ on /gi, "").replace(/ or /gi, "").replace(/ that /gi, "").replace(/ the /gi, "").replace(/ this /gi, "").replace(/ to /gi, "").replace(/ was /gi, "").replace(/ what /gi, "").replace(/ when /gi, "").replace(/ where /gi, "").replace(/ who /gi, "").replace(/ will /gi, "").replace(/ with /gi, "").replace(/ the /gi, "").replace(/ /gi, "").replace(/(\r\n|\n|\r)/gm,""));

Replaces all special characters in keyword with nothing and removes spaces-> var keywordwithstopwords = (keyword.replace(/!/gi, "").replace(/@/gi, "").replace(/#/gi, "").replace(/\$/gi, "").replace(/%/gi, "").replace(/\^/gi, "").replace(/&/gi, "").replace(/\*/gi, "").replace(/\(/gi, "").replace(/\)/gi, "").replace(/_/gi, "").replace(/-/gi, "").replace(/\+/gi, "").replace(/\=/gi, "").replace(/{/gi, "").replace(/}/gi, "").replace(/|/gi, "").replace(/\[/gi, "").replace(/\]/gi, "").replace(/\\/gi, "").replace(/:/gi, "").replace(/;/gi, "").replace(/"/gi, "").replace(/'/gi, "").replace(/\./gi, "").replace(/,/gi, "").replace(/</gi, "").replace(/>/gi, "").replace(/\?/gi, "").replace(/\//gi, "").replace(/ /gi, "").replace(/(\r\n|\n|\r)/gm,""));

Searches anchor text with stop words removed for the keyword phrase with stop words removed - > var one = iunostopwords.search(keywordnostopwords);
Searches anchor text with stop words removed for the keyword phrase that still has stop words in it -> var two = iunostopwords.search(keywordwithstopwords);
Searches anchor text that still has stop words for the keyword phrase that has no stop words -> var three = iuwithstopwords.search(keywordnostopwords);
Searches anchor text that still has stop words for the keyword phrase that still has stop words -> var four = iuwithstopwords.search(keywordwithstopwords);

If any of the above searches returns as true, then add a "Y" to the end of the string to signify that the keyword phrase is contained within the anchor text, otherwise add a "N" to signify that it is not contained in the anchor text -> if (one >= 0) {var final = 1} else if (two >= 0) {var final = 1} else if (three >= 0) {var final = 1} else if (four >= 0) {var final = 1} else {var final = -1};var final;if ( final != -1) { var KeywordURL = ("{-FieldData.FieldData-|-●6●3●0●4●3●3●-|-≡1≡7≡8≡2≡7≡2≡4≡6≡9≡9≡-}" + "~" + "Y");} else if ( final == -1) { var KeywordURL = ("{-FieldData.FieldData-|-●6●3●0●4●3●3●-|-≡1≡7≡8≡2≡7≡2≡4≡6≡9≡9≡-}" + "~" + "N");};var KeywordURL;-}

 

[Stereomike]

I love you all, but has anyone thought of naming his branches and steps with useful and descriptive names? Don't you get mad when you debug your template? It must feel like looking at a chinese newspaper. You could have bought the Russian version if you like challenges.
Watch this

 

[jp1]

Well I've got a namecheap domain scraper sitting in front of me designed to work with different proxies and it's as simple as a butthole but it still goes up and up in memory non stop until it crashes the pc

Sure there are a few if statements but how else could it function otherwise. Sure it takes from different files and memorizes the name but how does that justify the memory gobbling. There's no javascript to be seen on the whole thing, all I do is extract a counter from a file, put it back in the file for the next thread to pick up another file later on. I've eliminated count of strings macros in it but there's little else I can do, and if I didn't have the template extracting from different files, the file would be too big. The files getting extracted from are 3mb as it is each. I've tried cleaning cache and cookies which iirc worked on other templates but no luck.

 

[CaptainObvious]

I'm not doubting what your saying, but I've had 4.0.7 run 24/7 for 6 days, with zero crashes, until i had to do a windows update restart (all my posting templates have a scrapping and loops). With ZP3 i'd dial in templates to use 80% of free memory, with ZP4 I've been leaving 50% of server memory as headroom. ZP4 does release memory (have'nt tried 4.0.8 yet), but it seems to work like a batch operation, aka slowly increase for 5 - 15 mins then sharp drop in one go.

 

[rostonix]

Well I've got a namecheap domain scraper sitting in front of me designed to work with different proxies and it's as simple as a butthole but it still goes up and up in memory non stop until it crashes the pc

Sure there are a few if statements but how else could it function otherwise. Sure it takes from different files and memorizes the name but how does that justify the memory gobbling. There's no javascript to be seen on the whole thing, all I do is extract a counter from a file, put it back in the file for the next thread to pick up another file later on. I've eliminated count of strings macros in it but there's little else I can do, and if I didn't have the template extracting from different files, the file would be too big. The files getting extracted from are 3mb as it is each. I've tried cleaning cache and cookies which iirc worked on other templates but no luck.

I'm wondering what build do you use.
Work with files was improved for ZennoPoster 4.*