WebArchiveMaster - парсер Вебархива

devas111 · 21.11.2017

Оплатил и написал в ЛС

devas111 · 23.11.2017

Подскажи плиз.
Собрал сайты по причёскам до 2014 г дроп. EN
Запустил. Всё работает, шик просто.
Но в файлах не только чистый текст.

Код:

Cute Hairs
If уου hаνе medium length hair, learn hοw tο style уουr hair іn ponytails, twist, curls, аnԁ more wіth thіѕ free beauty video series CTYPE html&gt;
&lt;html dir="ltr" lang="en-US"&gt;
&lt;head&gt;
&lt;meta
&lt;title&gt;Cute Hairs&lt;/title&gt;
&lt;link rel="canonical" href="
&lt;link rel="stylesheet" href="
&lt;link rel="stylesheet" href="
&lt;link rel="stylesheet" href="
&lt;meta name="google-site-verification" content="" /&gt;&lt;script type="text/javascript"&gt;
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-6745385-9']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('
    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
&lt;/script&gt;
&lt;meta name="google-site-verification" content="6e5RZI2SERhDhaZ6Lo9vego_ktVRmk8yqqf-yUUSqqw" /&gt;
&lt;/head&gt;
&lt;body class="home blog"&gt;
&lt;div class="amaze no-excerpt wrapper"&gt;
&lt;div class="header"&gt;&lt;div class="siteTitle"&gt;&lt;a href="
&lt;div class="menu nav"&gt;&lt;ul id="menu-site-menu" class="menu"&gt;&lt;li id="menu-item-51790" class=""&gt;&lt;a href="
&lt;li id="menu-item-51793" class=""&gt;&lt;a href="
&lt;li id="menu-item-51796" class=""&gt;&lt;a href="
&lt;/ul&gt;&lt;/div&gt;&lt;/div&gt;
&lt;a name="jump"&gt;&lt;/a&gt;&lt;div class="main"&gt;&lt;div class="content"&gt;&lt;h1 class="posttitle"&gt;How to do hairstyles for medium length hair&lt;/h1&gt;&lt;div class="adsense336"&gt;&lt;script type="text/javascript"&gt;&lt;!--
google_ad_client = "pub-5170738193212461";
google_ad_slot = "";
google_ad_width = 336;
google_ad_height = 280;
//--&gt;
&lt;/script&gt;
&lt;script type="text/javascript" src="

&lt;div class="entry"&gt;
&lt;p&gt;&lt;object width="425" height="355"&gt;&lt;param name="movie" value="
&lt;embed src="
&lt;p&gt;If &amp;#1091&amp;#959&amp;#965 h&amp;#1072&amp;#957&amp;#1077 medium length hair, learn h&amp;#959w t&amp;#959 style &amp;#1091&amp;#959&amp;#965r hair &amp;#1110n ponytails, twist, curls, &amp;#1072n&amp;#1281 more w&amp;#1110th th&amp;#1110&amp;#1109 free beauty video series &amp;#1072b&amp;#959&amp;#965t cute &amp;#1072n&amp;#1281 easy hairstyles.&lt;br /&gt;
&lt;strong&gt;Video Rating: 0 / 5&lt;/strong&gt;&lt;/p&gt;
&lt;a href="#jump" class="jumpLink"&gt;&amp;uarr; Back to Top&lt;/a&gt;&lt;/div&gt;&lt;div&gt;&lt;span class="tags"&gt;&lt;a href="
google_ad_client = "pub-5170738193212461";
google_ad_slot = "";
google_ad_width = 336;
google_ad_height = 280;
//--&gt;
&lt;/script&gt;
&lt;script type="text/javascript" src="

&lt;div class="entry"&gt;
&lt;p&gt;cute hair styles f&amp;#959r medium hair:&lt;br /&gt;
&lt;object width="425" height="355"&gt;&lt;param name="movie" value="
&lt;embed src="
&lt;p&gt;plz subscribe&lt;br /&gt;
&lt;strong&gt;Video Rating: 0 / 5&lt;/strong&gt;&lt;/p&gt;
&lt;a href="#jump" class="jumpLink"&gt;&amp;uarr; Back to Top&lt;/a&gt;&lt;/div&gt;&lt;div&gt;&lt;span class="tags"&gt;&lt;a href="
google_ad_client = "pub-5170738193212461";
google_ad_slot = "";
google_ad_width = 336;
google_ad_height = 280;
//--&gt;
&lt;/script&gt;
&lt;script type="text/javascript" src="

&lt;div class="entry"&gt;
&lt;p&gt;&lt;object width="425" height="355" classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" codebase="
&lt;p&gt;&lt;strong&gt;&lt;/strong&gt;&lt;br /&gt;
&lt;strong&gt;Video Rating: 4 / 5&lt;/strong&gt;&lt;/p&gt;
&lt;a href="#jump" class="jumpLink"&gt;&amp;uarr; Back to Top&lt;/a&gt;&lt;/div&gt;&lt;div&gt;&lt;span class="tags"&gt;&lt;a href="
google_ad_client = "pub-5170738193212461";
google_ad_slot = "";
google_ad_width = 336;
google_ad_height = 280;
//--&gt;
&lt;/script&gt;
&lt;script type="text/javascript" src="

&lt;div class="entry"&gt;
&lt;p&gt;&lt;object width="425" height="355" classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" codebase="
&lt;p&gt;&amp;nbsp;&lt;/p&gt;
&lt;a href="#jump" class="jumpLink"&gt;&amp;uarr; Back to Top&lt;/a&gt;&lt;/div&gt;&lt;div&gt;&lt;span class="tags"&gt;&lt;a href="
google_ad_client = "pub-5170738193212461";
google_ad_slot = "";
google_ad_width = 336;
google_ad_height = 280;
//--&gt;
&lt;/script&gt;
&lt;script type="text/javascript" src="

&lt;div class="entry"&gt;
&lt;p&gt;Welcome t&amp;#959 Cute Hairs portal here &amp;#1091&amp;#959&amp;#965 w&amp;#1110&amp;#406&amp;#406 see &amp;#1072&amp;#406&amp;#406 n&amp;#1077w hairstyles, &amp;#1072n&amp;#1281 lovely cute hairs videos.&lt;/p&gt;
&lt;a href="#jump" class="jumpLink"&gt;&amp;uarr; Back to Top&lt;/a&gt;&lt;/div&gt;&lt;/div&gt;
&lt;div class="sidebar"&gt;
&lt;div class="menu side recent" id="menuSideRecent"&gt;&lt;h4&gt;Recent Posts&lt;/h4&gt;&lt;ul&gt;&lt;li class="page_item"&gt;&lt;a href="
&lt;/div&gt;
&lt;div class="footer"&gt;&lt;div class="menu"&gt;&lt;ul&gt;&lt;li class="first"&gt;&lt;a href="
&lt;li class="page_item page-item-1925"&gt;&lt;a href="
&lt;li class="last page_item page-item-1926"&gt;&lt;a href="
&lt;li class="last sitemap"&gt;&lt;a href="
&lt;!-- ClickBump 5! scripts go in footer for faster sites --&gt;&lt;script type="text/javascript" src="
&lt;script type='text/javascript'&gt;jQuery(document).ready(function(){jQuery('#accordion-1').easyAccordion({autoStart:true,slideInterval:5000,slideNum:false});})&lt;/script&gt;
&lt;script type='text/javascript'&gt;jQuery(document).ready(function(){jQuery('#accordion-1').css('visibility', 'visible');});&lt;/script&gt;&lt;/body&gt;
&lt;/html&gt;
&lt;!-- Performance optimized by W3 Total Cache. Learn more:

Served from: www.cutehairstylesformediumhair.com @ 2012-03-24 20:38:43 --&gt;

devas111 · 23.11.2017

Отписал относительно спецсимволов.
?1) Как ускорить процесс? Например, база доменов по кею в итоге >100К.
Поставил мин. символов 1200.
Категории по умолчанию стояло "0"; так быстрее?
?2) Проверка на уникальность, по вашему методу, с EN сонтентом пашет? Спрашиваю, потому что нужно (другой софт) подключать. Не платил с полгода за Хрум.

Pahan8282 · 23.11.2017

Привет,до какого числа действует черная пятница?

devas111 · 28.11.2017

Не парсит.
Поковырялся. Оказывается появилась проблема на странице.
1

2

devas111 · 28.11.2017

Пишется файл 1.txt

Wayback Machine
Dear Wayback Supporter, We ask you only once a year: please help the Internet Archive today. You may not know it, but we’re an independent, non-profit website that the entire world depends on. We protect reader privacy. We never sell ads, but still need to pay for servers and staff. Most users can’t afford to donate, but we hope you can. If everyone chips in $25, we can keep this going for free. For the price of a meal, we can preserve a website forever. When I started this, people called me crazy. Collect web pages? Why? We built the Wayback Machine because open access to knowledge is fragile. We became the digital library of record because no one should be able to change the past. I think the Internet Archive is a bargain, but we need your help. If you find our site useful, chip in what you can today. Thank you. —Brewster Kahle, Founder, Internet Archive
Dear Wayback Supporter, We ask you only once a year: please help the Internet Archive today. We’re an independent, non-profit website that the entire world depends on. We protect reader privacy. We never sell ads, but still need to pay for servers and staff. If everyone chips in $25, we can keep this going for free. For the price of a meal, we can preserve a website forever. When I started this, people called me crazy. Collect web pages? Why? We built the Wayback Machine because open access to knowledge is fragile. We became the digital library of record because no one should be able to change the past. If you find our site useful, please chip in. Thank you. —Brewster Kahle, Founder, Internet Archive
Dear Wayback Supporter, We ask you only once a year: please help the Internet Archive today. We’re an independent, non-profit website that the entire world depends on. We protect reader privacy. We never sell ads, but still need to pay for servers and staff. If everyone chips in $25, we can keep this going for free. For the price of a meal, we can preserve a website forever. When I started this, people called me crazy. Collect web pages? Why? We built the Wayback Machine because open access to knowledge is fragile. We became the digital library of record because no one should be able to change the past. If you find our site useful, please chip in. Thank you. —Brewster Kahle, Founder, Internet Archive
Dear Wayback Supporter, We ask you only once a year: please help the Internet Archive today. We’re an independent, non-profit website that the entire world depends on. We never sell ads, but still need to pay for servers and staff. If everyone chips in $25, we can keep this going for free. For the price of a meal, we can preserve a website forever. When I started this, people called me crazy. Collect web pages? Why? We built the Wayback Machine because open access to knowledge is fragile. No one should be able to change the past. If you find our site useful, please chip in. Thank you. —Brewster Kahle, Founder, Internet Archive

devas111 · 29.11.2017

Убери ПЛЗ.
А то мешает работе.

GEN_13 · 30.11.2017

Оплатил на ЯД с примечанием своей почты

devas111 · 30.11.2017

Почти в каждой папке в первом файле (или под номенром 1) спаршено

Wayback Machine
Dear Wayback Supporter, We ask you only once a year: please help the Internet Archive today. You may not know it, but we’re an independent, non-profit website that the entire world depends on. We protect reader privacy. We never sell ads, but still need to pay for servers and staff. Most users can’t afford to donate, but we hope you can. If everyone chips in $25, we can keep this going for free. For the price of a meal, we can preserve a website forever. When I started this, people called me crazy. Collect web pages? Why? We built the Wayback Machine because open access to knowledge is fragile. We became the digital library of record because no one should be able to change the past. I think the Internet Archive is a bargain, but we need your help. If you find our site useful, chip in what you can today. Thank you. —Brewster Kahle, Founder, Internet Archive
Dear Wayback Supporter, We ask you only once a year: please help the Internet Archive today. We’re an independent, non-profit website that the entire world depends on. We protect reader privacy. We never sell ads, but still need to pay for servers and staff. If everyone chips in $25, we can keep this going for free. For the price of a meal, we can preserve a website forever. When I started this, people called me crazy. Collect web pages? Why? We built the Wayback Machine because open access to knowledge is fragile. We became the digital library of record because no one should be able to change the past. If you find our site useful, please chip in. Thank you. —Brewster Kahle, Founder, Internet Archive
Dear Wayback Supporter, We ask you only once a year: please help the Internet Archive today. We’re an independent, non-profit website that the entire world depends on. We protect reader privacy. We never sell ads, but still need to pay for servers and staff. If everyone chips in $25, we can keep this going for free. For the price of a meal, we can preserve a website forever. When I started this, people called me crazy. Collect web pages? Why? We built the Wayback Machine because open access to knowledge is fragile. We became the digital library of record because no one should be able to change the past. If you find our site useful, please chip in. Thank you. —Brewster Kahle, Founder, Internet Archive
Dear Wayback Supporter, We ask you only once a year: please help the Internet Archive today. We’re an independent, non-profit website that the entire world depends on. We never sell ads, but still need to pay for servers and staff. If everyone chips in $25, we can keep this going for free. For the price of a meal, we can preserve a website forever. When I started this, people called me crazy. Collect web pages? Why? We built the Wayback Machine because open access to knowledge is fragile. No one should be able to change the past. If you find our site useful, please chip in. Thank you. —Brewster Kahle, Founder, Internet Archive

devas111 · 01.12.2017

Нм. Если 60 папок и в 40-50 только файл1.txt с таким-же текстом, то это сильно притормаживает работу.
К меня не убрали рекламу ещё.

devas111 · 02.12.2017

Подскажите плиз.

На картинке видно, как зависает и приходится просто сносить и заново запускать шаблон.

PS Сейчас проверю новый шаблон по web.arch...

devas111 · 03.12.2017

Новый работает нормально.
А как сохранить свой паблик прокси из ПРОКСИЧЕКЕРА в файл My_proxys.тхт

WebArchiveMaster - парсер Вебархива

Client

Client

Client

Client

Client

Client

Вложения

Client

Client

Новичок

Client

Client

Client

Client

Client

Client

Client

Client

Client

Client

Client

Client

Client

Client

Client

Client

Client

Client

Client

Client

Client

Кто просматривает тему: (Всего: 2, Пользователи: 0, Гости: 2)