Amazon crawl: 2 versions of the page

[Trptc]

Hi everybody!

I have difficulties crawling Amazon; the page (and html) returned by amazon can be from 2 versions:
- The first is the same returned to everybody
- The second is the same as googlebot user agent

I can't reproduce easily which one will be returned when Zennoster is running.
When an instance starts it will return the same version of the page.

I use default values for my UserAgent in the project.
I had no difficulties for 2 months (first version of the page only)

I can hit the same problem even if i create a new project from scratch.
For information: I use PIA for the vpn.

does anyone know this problem?

Thank you very much!

 

[Trptc]

I found a workaround; it's ok when I delete cookies.
But I don't understand why Amazon gives me 2 versions of it's website

 

[lokiys]

Hi. Try to use up to date useragents. Or use newest zenno version where useragents can be emulated by newest versions..

 

[Trptc]

Thank you lokiys. That's what I done yesterday and it's much better

 

[MD. Shamid Islam]

Thank you

 

[russya]

I also have 2 versions of the site. But I made my own parser for everyone. The program checks which version of the site and exactly the way it goes