Problems about ZP template for Web Scraping

StevenZ

Новичок
Регистрация
26.08.2019
Сообщения
3
Благодарностей
0
Баллы
1
Hello, ZP Master!
I tried to create a ZP template for Web Scraping.
I found that one ZP Instance will spend a lot of RAM and create a Instance will spend some time.

I have stored the originational urls in MySQL database.

So I decide to use C# to hold the DB connection and fetch 1000 originational urls each time, then create a Instance Pool.

When there is free instance, get one url and let the instance get the DOM to extract it. After the url finished, not close the Instance, just put back to the Instance Pool.

If the originational urls is empty, fetch 1000 again until all the originational urls are finished.

Most of template is c# code with Own Code, it's difficult to debug,
especially the multi-thread part. I use ZP 5.22 and run it on Windows10.
The Instance Pool is not work until now. Not enough message in the log file.(PS: Is there any way to record my own log in the log file?), I just follow the doc like this:

private Queue<Instance> zpInstanceQueue = new Queue<Instance>();
public ZPInstancePool(int capacity)
{
for(int i =0;i< capacity; i++)
{
var url = "";
var port = 0;
var address = "";
// Gets parameters of instance
ZennoPoster.GetNewInstance(out url, out port, out address);
// Create new instance
var zpInstance = new Instance(url, port, address);

zpInstanceQueue.Enqueue(zpInstance);
}
}


https://help.zennolab.com/en/v5/zennoposter/5.11.7.0/webframe.html#topic550.html

I noticed the Requirements, I'm not sure if my win10 system cause the bug.
Requirements
Target Platforms: Desktop: Windows XP SP3 and older. Server: Windows Server 2003 and older.

Should you have any suggestion, please reply me. Thanks!
 

Кто просматривает тему: (Всего: 1, Пользователи: 0, Гости: 1)