How to compare two big files?

[bfirst]

Hi again

I am working on this problem for few days but without any positive results. Please help me!

I have two files with about 2.500.000 URLs in each one. I have to compare this two lists and remove from second list all urls which are in first list.

I have tried function Delete urls in list processing. I took one record from first list and do Delete urls with specified value. It was going too slow.

Then I have created regexp with all urls from first list and do delete urls with regular expression. Same results.

At the end I have created an regexp and compare two files (not lists) to get all results which are not listed in second file. I am waiting now 9 hours and project still works.

Please help if there is any faster sollution to compare two lists? I need something like an option in scrapebox
1. Compare lists on domain level
2. Compare lists on URL level.
But scrapebox is limited to 1.000.000 URLs

 

[rostonix]

I am waiting now 9 hours and project still works.

Do you test in ZP or PM?

 

[bfirst]

in ZP. I think that it is immposible without using some type of database (MS SQL) and C#. I have tried all.

ZP is the greatest program ever for automate a lot of work but I think that I have to use another program to compare big files?

What do you think about something like that? http://www.pilotedit.com/index.php?option=com_content&view=section&layout=blog&id=6&Itemid=54

 

[rostonix]

1. Compare lists on domain level
2. Compare lists on URL level.
But scrapebox is limited to 1.000.000 URLs

I think it's possible only with C#, there are no in-built solution for this.

If you wanna compare full strings (full URLS) you can schive this via C# with this snippet (#2) http://zennolab.com/discussion/threads/search-in-list-and-tables-via-c-macro.10733/
Just add a loop there.

 

[bigcajones]

Have you tried gscraper?

 

[MarioBros]

Why you dont split your file in 3 parts to use with scrapebox?
You will need to compare each list 3 times, with the 3 differents parts, but that will work