Using Regular Expression To Extract All URLs

[allsystems]

Hi Guys,

I am trying to extract all URL on a webpage. I am trying to use regex but none seem to work. Is there a good regex to extract all URLs on page?

Thanks

 

[VladZen]

Regex can be easily create in ZennoPoster. You should get source code of web page, copy it to regex constructor and put start and end of text.

 

[allsystems]

Only issue with this is that I am visiting random URLs so I dont know the format on each page which is why I need a super regex that works in Zenno and will get me 99.9% of all URLs

 

[lokiys]

Extract all hrefs and you will be fine. You can try to look also on xpath for that. Not only regex and see what works best for you...

 

[VladZen]

Only issue with this is that I am visiting random URLs so I dont know the format on each page which is why I need a super regex that works in Zenno and will get me 99.9% of all URLs

regex given in my screen is universal. It parses all href's starting with http://

 

[lokiys]

regex given in my screen is universal. It parses all href's starting with http://

In most cases in source links are not with http://
Most probably removing http:// from your regex will do better...

 

[shabbysquire]

I use this one to capture urls in format:

http://domain.com/
https://domain.com/

http://www.domain.com/
https://www.domain.com/

My regex:

(?<=https?://(?:www\.)?)(?!www\.).*?(?=['/"]|</a>)

Maybe someone can improve it?

 

[VladZen]

My regex:

(?<=https?://(?:www\.)?)(?!www\.).*?(?=['/"]|</a>)

Maybe someone can improve it?

Try this -

(?<=href=")http.*?\.com

 

[shabbysquire]

(?<=href=")http.*?\.com

The only issue is the domain ext, some are: .co.uk, .mobi, etc.

 

[VladZen]

(?<=href=")http.*?\.com

The only issue is the domain ext, some are: .co.uk, .mobi, etc.

http.*?\.\w+(?=/)

 

[CSS]

I use this one to capture urls in format:

http://domain.com/
https://domain.com/

http://www.domain.com/
https://www.domain.com/

My regex:

(?<=https?://(?:www\.)?)(?!www\.).*?(?=['/"]|</a>)

Maybe someone can improve it?

Sure, try this:

(http|ftp|https):\/\/[\w\-_]+(\.[\w\-_]+)+([\w\-\.,@?^=%&:/~\+#]*[\w\-\@?^=%&/~\+#])?

Also you can view Russian sector, there is regex list for typical tasks