A problem with multi threading.

[muchink]

Hi, I have a problem whenever I'm trying to multithread above ~30 with ZP.

I have 100 private proxies, 32GB ram, core i7 and 100mb/s internet speed.
With other programs, like SB, whenever I increase the amount of proxies and then the amount of threads, the success rate stays the same.
When I'm running the project in PM or in ZP with ~20 threads(with only 20 proxies), everything is perfect and all is working.
But above a certain amount of threads I just don't get it why the success rate drops below 10% while I'm trying to work with the best possible resources.
It seems that ZP most of the times see the pages as "ready" and tries to execute next action even though the page didn't even load yet. I usually save the html file of the page when the error happened. And I see it happening over and over with all my templates when I'm using many threads, the size of the file is 1KB, and the page is empty, meaning it didn't get to load and ZP tried to execute next action.
I can't go on now and add before every action "wait 100 seconds.." that's ridiculous.

It just makes it impossible to scale up the amount of tasks I want to get done in a certain time.

Please assist me, thanks.

 

[lokiys]

I have talked long time ago there http://zennolab.com/discussion/showthread.php?9855-Multithreading-bugs
and in some other threads about multi-thread problem but noone takes it seriously

I guess you facing with that same problem

Cheers

UPDATE:

Found that other thread as well - http://zennolab.com/discussion/showthread.php?10640-Multithread-problems

 

[lokiys]

Actually from my tests and from my view problem is that zennoposter sometimes takes just wrong action to execute.

I will try to explain.

As i know every action in template have its own action id - something like this cd607fb3-49e4-4211-90f4-301ddc96bfb4
Where problem happens in my opinion is example one thread executing template but for unclear reasons zenno taking some other thread action id and trying to execute that action and not waiting while page will load up.

Will try to explain in other way. Example we have 30 threads running and two of those threads are ready to click on Registration link but problem is that in one thread page is loaded but in other thread there need to wait till page willload up and for unclear reason zenno not waiting while page will load up but just trying to click on Registration link in page what is not loaded as there is that same action id for all threads...

If my point of view is right then this could explain also this - When you click on particular template and click show instance then you can see problem like that, when one instance will close not important with success or with error you will get other instance showed but in some cases this will NOT be that same template but could be other running template from your poster

Cheers

 

[muchink]

Wow man, seeing someone with the same problem as my really helps!
I saw your other threads with the screenshots, and thats EXACTLY what happens to me. I've got more that 100 projects, and now that I'm trying to scale it up I really got no clue what to do.
How do you overcome this problem man? do you just use low amount of threads or did you figure out other ways to bypass it?
Since right now, ZP seems really impossible to work with. any other program that will allow me to work with more than 100 threads and have 50%+ success rate would be much better than running my template on ZP with 100% success rate on 10 threads but less than 10% success rate on more than 50 threads..

Please help us guys, I really love to work with ZP and perfect my templates, but what's the point if in the end I really can't use them properly..?

Thanks.

 

[lokiys]

Only one suggestion - set one thread per base. It helps, but not much.
And I'm trying to avoid to use more than one thread per template.

I have sent this thread link directly to darkdiver - will wait some input from him It is time to resolve this bug, i will help in any possible way

 

[muchink]

Thanks man for the help.
Using 1 thread per base is a great temporary idea, I think since it will not have a connection between 2 different templates but will take way more time to load each base. However, after a while(if the templates aren't simple templates to be executed in less than 2 min), the threads will reach the maximum amount of threads that you set(for example 50) and it will remain 50 with 1 thread per base - thus will execute them faster.

I'm not sure I explained it correctly but what I mean is, ultimately when there is 1 thread per base the thread will run faster but the time to load all threads will be much longer.

But when all the threads are loaded :50
1 thread is finished (49 threads used)
1 thread is added (50 threads used)
each time only 1 will finish and another one will start right after it and the whole process will run faster (it will only work if the templates you are using aren't short and simple templates that take less than a minute to execute)

I'm not good at explaining but I hope you got me..

This is just in theory, I will try it in the next hour.
Did you try to use 1 thread per base and see the results with a big amount of threads?

Edit:
Tested doing it with 50 threads and 1 thread per base, but it still has the same problem that is solved only by using ~20 threads..

 

[muchink]

What's up? is there any update about this?
I don't get what is the point of having a good PC and internet connection with a Pro license of ZP without the ability to scale up your campaigns and use much more threads.
It is impossible to continue working with the tool since you just can't scale and do more in less time..

I really like ZP, please update this thread about this problem and hopefully the solution of using multiple threads without the problem of reduction in success rate..

Thanks.

 

[lokiys]

Hey I'm sure that developers are informed about this problem as i sent this problem directly to developers. I just think it is not possible to give clear answer in one day, as there3 was weekend before.

Lets wait a couple of days

Cheers

 

[lokiys]

Admins please move this thread to Bugs section.

Thanks

 

[muchink]

k, thanks for the update man.
What are you currently doing to overcome this man?

 

[lokiys]

k, thanks for the update man.
What are you currently doing to overcome this man?

Only one suggestion - set one thread per base. It helps, but not much.
And I'm trying to avoid to use more than one thread per template.

I have sent this thread link directly to darkdiver - will wait some input from him It is time to resolve this bug, i will help in any possible way

...

 

[muchink]

oh, yeah I thought you tried other ways. From what I tested yesterday, it doesn't help that much.. just like you said.. :\

 

[muchink]

Any update about this problem?

 

[darkdiver]

First of all we have clients who have 200+ threads simultaneously running without problems.

Some questions. What load of the CPU do you have? How you computer connected to the internet (via router or it is in the datacenter)?
If you are connected via router. It just do not have enough power to process huge amount of connections and you get wrong information about page loading. Regular router can handler approximately 300 connections.
If you CPU is overloaded it also can cause a problems. Regular

The next step is the proxies. Even if it is the server proxies. It can generate connection errors and affect busy state flag and even affect button clicking. They just do not send anything and you should click again.
For example your proxy has guaranteed 99% success connection rate for hotmail account registration you should make approximately 200 requests. This means 2 of 200 request can fail. And it fails in some critical request or JavaScript loading you action will fail. And 99% is a very optimistic value.

Ok, you are using greate PC in datacenter, but you are still failing with huge amount of threads. What do you have to do and how to avoid these problems?
You can make checks in your template for the element. Or if you know what URL should be changed you can check the Page.Url.
These tactics will help you to determine the page loading and you can click again on some buttons if your page stop loading.

 

[muchink]

Thanks for the detailed reply,

Regarding router, I probably forgot to write this, but along with a great computer I bought a great router(ASUS RT-N66U N900 ) that can run up to above 30k threads
http://www.smallnetbuilder.com/lanwan/router-charts/bar/77-max-simul-conn
So that's not the issue..

When I'm running 50 threads the CPU doesn't go above 30% so it's still ok.

About the proxies, Of course you can't expect 99% uptime, sometimes proxies have problems too, but what I don't get is why when i go from 20 proxies with 20 threads to 50+ proxies with 50+ threads, the problem begin and the % of success drops drastically, since I didn't overload the proxies and increased the amount of them as I increased the amount of threads.

About the "check for element", I'm currently using this strategy in many of my templates, but it's kind of frustrating that I every time I run a template I find more places that need to be checked before I execute them.

I did make more test with using 1 thread per base since the last time I wrote here, and it seems to work great now without that many failures of project as it was with 5 threads per base.
I really want to use ZP to the best and would love to run it smoothly without problems like the other clients who run in on 200+ threads,
To help me with that, can you tell me from your experience what other things might factor this and help increase success rate while using 200 threads?

Thanks.

 

[darkdiver]

About routers:

Regarding router, I probably forgot to write this, but along with a great computer I bought a great router(ASUS RT-N66U N900 ) that can run up to above 30k threads
http://www.smallnetbuilder.com/lanwa...max-simul-conn
So that's not the issue..

I've tested router WNR3500L it can handle 350 connection not 2000. Also I tested ASUS router It can handle approximately 700-850 connections not 30k.
I thought the main difference is that ZennoPoster creates and destroys huge amount of connections. It is not constantly existing connections. ZennoPoster we have built in tester. Please check you real situation.

To help me with that, can you tell me from your experience what other things might factor this and help increase success rate while using 200 threads?

If you are using HTTP try to switch from http to socks proxy. And if it is not critical for you set back to default 5 thread in 1 process.
Usually people works with high load on good proxies (dedicated proxies with 1 ip per proxy) with 5 thread in 1 process.

 

[muchink]

Thanks!

what impact does it have if I switch from private proxies to socks proxies?

By the way,
I've just tested and the result was ~1000 connections, does it mean that I can use 1k threads maximum, or it indicates something else?

 

[darkdiver]

what impact does it have if I switch from private proxies to socks proxies?

socks proxies creates tunnel and do not close the connection. Sometimes it helps to avoid not predictable error. But it depends from the socks quality. In any can try to switch to socks it helps for somebody.

I've just tested and the result was ~1000 connections, does it mean that I can use 1k threads maximum, or it indicates something else?

This means you system can handle 1000 connections simultaneously and nothing else.
Each zennoposter thread can open from 1 to 16 network connections. Also proxychecker need from 1 to 3 network connection per each checking thread.
Also other running programs on your computer may require network connections.

 

[smol84]

You mention to check every element is loaded or values passed to element - Checking element load is easy
but
how do you check is a click is passed to a button, for example with - he.RiseEvent("click", instance.EmulationLevel);?

 

[rostonix]

When you click on element, something on the page should change. Either URL, either other element. So you can check this.