Best method(s) to minimize/compact traffic consumption of project

[morpheus93]

Hi there,

As I'm more and more using proxies that are billed per Gigabyte I'm wondering what is the best approach to achieve minimal traffic consumption in a Zenno-Project?

I assume the least traffic is needed by a bot based on pure HTTP-Requests but that requires a lot of technical knowledge and I think this can be easily recognized by the automated sites as there is no mouse-movement, clicks and so on?

What other solutions are available to minimize the traffic usage and stay under the radar of the sites anti-bot mechanisms?

Thanks

 

[lokiys]

Hi there,

As I'm more and more using proxies that are billed per Gigabyte I'm wondering what is the best approach to achieve minimal traffic consumption in a Zenno-Project?

I assume the least traffic is needed by a bot based on pure HTTP-Requests but that requires a lot of technical knowledge and I think this can be easily recognized by the automated sites as there is no mouse-movement, clicks and so on?

What other solutions are available to minimize the traffic usage and stay under the radar of the sites anti-bot mechanisms?

Thanks

You are right HTTP request bots would eat a little traffic.
But for browser-based ones, there is not one answer. But first what comes into mind are disable pictures. It is what I have done and others do that always.
But the site can check if you load images, so this would be bad for some sites.
But you can also look in traffic monitor and block some requests there. That could include some tracking sites like block URLs what load ads on site.

 

[Aronax]

Unfortunately, the more things you disable in order to save bandwidth, the more you expose yourself (or your bot). But, you can try, as @lokiys indicated above, to disable images, ads and even javascript (although many sites will not work without javascript and I don't know for sure how much bandwidth it actually eats...but I do know that it consumes extra hardware resources)

 

[Yann]

Maybe one way would be to pretend that you already have the content, like it is browser cached. Then it could prevent you from actually downloading css, images, fonts...You could check the elements sizes of the web pages you are accessing and not downloading the biggest ones.
Otherwise if you want to actually render the content, depending on the websites you are visiting, if they use objects largely shared accross the internet, you could potentially make your own page with a lot of these elements (fonts, js llibraries and so) visit your page first without any proxy so the browser can cache those elements, and then use the proxy to visit the websites targeted.
I did not try this but might be worth a try

 

[Yann]

Also, is it possible to use chrome's lite mode otherwise?
https://support.google.com/chrome/answer/2392284?co=GENIE.Platform=Android&hl=en

 

[Yann]

Maybe there is a c# code to activate the lite mode ?

 

[morpheus93]

Maybe one way would be to pretend that you already have the content, like it is browser cached. Then it could prevent you from actually downloading css, images, fonts...You could check the elements sizes of the web pages you are accessing and not downloading the biggest ones.
Otherwise if you want to actually render the content, depending on the websites you are visiting, if they use objects largely shared accross the internet, you could potentially make your own page with a lot of these elements (fonts, js llibraries and so) visit your page first without any proxy so the browser can cache those elements, and then use the proxy to visit the websites targeted.
I did not try this but might be worth a try

@Yann Thank you for sharing your experience and for your appreciated help. The approach with caching some of the elements of a site first without proxy and then use the cached elements again and also to create a own webpage to "feed" the browser beforehand is great. I would never have thought of that.

Will try how can I implement this approach to the sites im working on and will have an eye on the Chrome Lite mode...

Thanks again

 

[Swiss Montreux]

Hi morpheus93,

Did you happen to try this out as I am looking to do the same here..?

 

[gaabs]

You are right HTTP request bots would eat a little traffic.
But for browser-based ones, there is not one answer. But first what comes into mind are disable pictures. It is what I have done and others do that always.
But the site can check if you load images, so this would be bad for some sites.
But you can also look in traffic monitor and block some requests there. That could include some tracking sites like block URLs what load ads on site.

I am facing the same problem Im trying to reduce the traffic consumption... and its not a good solution by disabling images as in Zennoposter when u disable images u cant solve captcha... so looks like a good solution but when talking about bots we always need to solve captcha which makes this an unvalid solution

also I am trying to prevent chrome to auto play videos, but I cant find a solution I'd be grateful if u guys help

also I am still trying to solve my problem where zennoposter does not show PUSH NOTIFICATIONS to allow or deny, even with the command accept on "browser notifications" setting it does not work, and I dont see any will on zennolab support to help me solving this (I already opened two posts in this forum and I already posted in two different discussion where support was active and didnt find a solution yet, all that support says me is to Set Browser Notifications and as I said, it does not work.

I hope this is the last time I ask for help about this

 

[lokiys]

I am facing the same problem Im trying to reduce the traffic consumption... and its not a good solution by disabling images as in Zennoposter when u disable images u cant solve captcha... so looks like a good solution but when talking about bots we always need to solve captcha which makes this an unvalid solution

also I am trying to prevent chrome to auto play videos, but I cant find a solution I'd be grateful if u guys help

also I am still trying to solve my problem where zennoposter does not show PUSH NOTIFICATIONS to allow or deny, even with the command accept on "browser notifications" setting it does not work, and I dont see any will on zennolab support to help me solving this (I already opened two posts in this forum and I already posted in two different discussion where support was active and didnt find a solution yet, all that support says me is to Set Browser Notifications and as I said, it does not work.

I hope this is the last time I ask for help about this

Hey. If You read through all discussion then You will see I suggested looking into traffic monitor, where You are able to block every single request from loading, so You can pick and block even one image from loading and not all.
you can block one domain. Please check that I'm sure it will help.

I can not answer this notification problem, as I never had any need for that. Sorry. Hope @VladZen can look more deeply into this.

 

[morpheus93]

I am facing the same problem Im trying to reduce the traffic consumption... and its not a good solution by disabling images as in Zennoposter when u disable images u cant solve captcha... so looks like a good solution but when talking about bots we always need to solve captcha which makes this an unvalid solution

I'm using some bots without loading images but never had a problem with captchas, they are sent to the recognition service nevertheless and i get back the answer.

 

[lokiys]

I'm using some bots without loading images but never had a problem with captchas, they are sent to the recognition service nevertheless and i get back the answer.

Yep, you are right when you develop bot with images you can then disable, and when captcha will be sent to service it will be auto-loaded and sent

 

[gaabs]

Unfortunately I just use bot as multi thread... I have to do all job manually as I get lots of requests from different pages so its impossible (for me) to create a pattern to make a bot recognize where and what to put in each form. Saying this I dont use captcha automate services I usually do it manually.
Zennoposter is used simply as a multi thread software that makes easy the "clear cookie, set proxy, set user agent etc..."
I'll check traffic monitor, thanks.

I know its out of this post, but I'd like to ask if its possible for devs to put a "timer" between each thread being running. I mean, as I do it manually, if I set few threads, the traffic is not constant (example: I set 20 threads for a link with a 0-10minutes pause variable. Within 10 minutes all 20 pages are going to be loaded, but sometimes I get 5 pages opening at same time or with a really short delay which makes traffic looks bad, and even if I handle to build 10 different modules with 1 minutes delay each one, I'll reach another problem. With all 20 threads fully loaded, I'll have to manually do my work which takes another 5 minutes for example which makes a 5 minutes gap without traffic and what makes traffic SLOW. Note: speed is different than constancy, with zenno poster I can have GREAT speed with no constancy or a useful constancy with slower speed.)

I believe adding this would be pretty easy for devs and it would be VERY helpful for me... I am breaking my mind trying to solve this by myself by trying different pauses, different threads but I cant get into a acceptable setup.

 

[Yann]

Here is a simple algorithm with 2 threads, but you can do it with more...
You can use a text file, run your script with a maximum of 2 threads, each time the thread checks the file and the number of running threads (number of "tries"), if the file is empty for today then it means it is the very first thread so it adds the current date/time and it can execute the rest of the script right away and add one more thread. If there is a date/time for today, if needs to check how long ago compared to now, and if it is less than the desired waiting time then the script needs to wait for the time difference. Once it reached the time difference or if it is more than the desired time, then it needs to check if there are still 2 threads (meaning the first one did not finish so it needs to wait longer) or if just 1 thread then it can add the current date/time to the text file, start a new thread and then continue the execution of the script.

You need to be familiar with c# to do this, you can get the task list from zennoposter like this:
var tasks = ZennoPoster.TasksList;

and then you can searh for your task and change its settings:
ZennoPoster.SetExecutionSettings(id, esElement.InnerXml);

Good luck !

 

[gaabs]

I dont know C# :/

 

[Yann]

ok then, since you said that you have to do all the job manually, why don't you manually add 1 try in zennoposter at the right time ? Like in the middle of a job so it is ready by the time you complete it...

 

[gaabs]

You mean by using scheduler? what I am doing right now is I have modules with several pauses before starting
0-3 minutes
3-6 minutes
6-9 minutes
9-12 minutes

I run 5 threads each and its the best I got so far, but its not perfect as u can see in this

The average is fine but we got some bursts, this is an generic example but sometimes it got crazy with +5 clicks within 10 seconds

 

[Yann]

I was suggesting to manually start new thread, otherwise looking at the pauses, why don't you add a little bit of padding in between, like:
0-3 minutes
4-6 minutes
7-9 minutes
10-12 minutes

Finally, I think that even without knowing c# you should be able to do something using a centralized file with date/time so each thread can check and update it to synchronize each other.