- Регистрация
- 08.07.2013
- Сообщения
- 81
- Благодарностей
- 1
- Баллы
- 8
Hi,
I have created a simple script. Basically I am using GET request to visit URLs and check for some text. I tried running 120 threads but it creates a bottleneck. I have now add the project 12 times and done 10 threads for each one. There is a loop of 4 times for each instance. The project poster is not running as fast as I expected. I have 90k URLs and it takes 2 days to go through it. This is on a dedicated server. I think it has something to do with a bottleneck somewhere. Can you help me to increase the efficiency of this project as I cant let this run for 2 days for each project when I have 20+ projects to go through.
Thanks
I have created a simple script. Basically I am using GET request to visit URLs and check for some text. I tried running 120 threads but it creates a bottleneck. I have now add the project 12 times and done 10 threads for each one. There is a loop of 4 times for each instance. The project poster is not running as fast as I expected. I have 90k URLs and it takes 2 days to go through it. This is on a dedicated server. I think it has something to do with a bottleneck somewhere. Can you help me to increase the efficiency of this project as I cant let this run for 2 days for each project when I have 20+ projects to go through.
Thanks