Very Slow after scrape 40 pages

[bluescharp]

I want to scrape many pages, url like:
http://abc.com/keywords/money&page=2
http://abc.com/keywords/money&page=3
......
http://abc.com/keywords/money&page=50
.....
http://abc.com/keywords/make&page=2
....
This site need login, so I want to use just 1 thread to scrape.

After ProjectMaker visit 40 pages, it become very slow. how to do with this?

 

[rostonix]

Try to use action Browser - Settings - Restrt instance periodically

 

[lokiys]

I would suggest different logic in your template.
First create link file:
http://abc.com/keywords/make&page=2
[URL='http://abc.com/keywords/make&page=2']http://abc.com/keywords/make&page=3
[URL='http://abc.com/keywords/make&page=2']http://abc.com/keywords/make&page=4
[URL='http://abc.com/keywords/make&page=2']http://abc.com/keywords/make&page=5[/URL][/URL][/URL]
.........
as many links you have. and then run template in multithread, you will get job bode much faster and with less memory consumption...

Cheers

 

[bluescharp]

I would suggest different logic in your template.
First create link file:
http://abc.com/keywords/make&page=2
http://abc.com/keywords/make&page=3
http://abc.com/keywords/make&page=4
http://abc.com/keywords/make&page=5
.........
as many links you have. and then run template in multithread, you will get job bode much faster and with less memory consumption...

Cheers

But this site need login. So how to set multithread using one cookie?

 

[bluescharp]

Try to use action Browser - Settings - Restrt instance periodically

You mean "Restart the instance" every time after scraping one page?

 

[rostonix]

You mean "Restart the instance" every time after scraping one page?

No. After N itarations. Before instance got stuck.

 

[lokiys]

But this site need login. So how to set multithread using one cookie?

Same way as you using one thread. No difference...