How to Remove Duplicate Donmain?

[linda2013]

Hi guys,

I need to remove duplicate donmain. Here is an example:

This is one site, three different links:

http://www.skyscrapercity.com/forumdisplay.php?f=336
http://www.skyscrapercity.com/forumdisplay.php?f=1063
http://www.skyscrapercity.com/forumdisplay.php?f=111

The result i need is keep one of them, and delete others.

Thanks

 

[Hungry Bulldozer]

Sample template sent in skype

 

[linda2013]

Sample template sent in skype

Thanks. Yeah, i have received it. But the result is not what i need. Because it will remove the link not duplicate.

 

[linda2013]

Anyone can help me?

 

[nevadahsot]

Stuck at the first breakpoint.
Not sure whether it's caused by the bind function.( Don't know list bind very well.)

Basic outline is there. Not c#. Slowly.

 

[PapyRef]

Optimize version with C# code.
Result in file FinalUrl.txt

Test
File Dup1.txt, 3 Urls (all duplicates) => 1 Url in FinalUrl.txt
File Dup2.txt, 1692 Urls => 810 Urls in FinalUrl.txt

C# Code

System.Collections.Hashtable distinctHosts = new System.Collections.Hashtable();
var fileUrl = project.Variables["FileUrl"].Value;
int count = Convert.ToInt32(Macros.FileSystem.FileCountOfLines(fileUrl));

for(int i = 0; i < count; i++)
{
    var url = Macros.FileSystem.FileGetLine(fileUrl, i.ToString(), false);
    System.Uri uri = new Uri(url);
    if (!distinctHosts.ContainsKey(uri.Host)) { distinctHosts.Add(uri.Host, uri.ToString()); }
}

string[] uniqUrl = new string[distinctHosts.Count];
distinctHosts.Values.CopyTo(uniqUrl, 0);
return string.Join("\n", uniqUrl);