If element found on list, do ... x

[Patrick]

Hello I am working on scraping bot, and I scrape from url1, and want add this url1 to list_scraped.txt, then move to next url2 from list_of_urls.txt, then check "if taken url2 from list_of_urls.txt are on list_scraped.txt" if yes then delete this record and take next, if no then start scraping. Wish someone could help me with this

 

[Patrick]

or after scraping remove dupes in list1, if record found is in list1 and list2, delete from list1

 

[lokiys]

Hi.
The first and easiest method would remove all duplicates BEFORE you save URLs in your list for scrapping.

If that does not work for you then you can also try to delete duplicates in list with list action.
But with URLs it can not be effective because http://scrappingUrl.com and https://scrappingUrl.com will be different URLs and will not be recognized as duplicates.
It really depends on how your URLs are formated. and how you want to dedupe them.
If by domain then you will need a small piece of C# code to strip URLs to root URL and then dedupe.

 

[Patrick]

they gonna be same 100%, but could you tell me how I could use C# to check fast if url is in both lists? does it take long if list gonna be 100k up to 1milion?, and how to delete dupes in zenno?

 

[Patrick]

I mean if url1_list1=url1_list2 then do A, if not then do B

 

[lokiys]

they gonna be same 100%, but could you tell me how I could use C# to check fast if url is in both lists? does it take long if list gonna be 100k up to 1milion?, and how to delete dupes in zenno?

Delete duplicates you can with one action List Processing

C# is programming language what You can use in zenno action Custom Code ==> C# code
You can learn and search in the forum there are many c# snippets for c#
But You can check all with a standard action. Just take one line from one list then loop through another list and compare it with if action. https://zennolab.com/wiki/en:logic