Is it possible to use AI to understand a screenshot of a webpage - and return click coordinates?

[zmike]

When a page is dynamic or don't have fixed location for certain buttons or checkbox.

Is it possible to use Zenno Poster to take a full screenshot, then let AI to understand the screenshot of the webpage, by calling the AI using API., then let AI to return click coordinates?

Finally let Zenno Poster to click that x, y coordinates?

 

[lokiys]

I dont see why not, but you probably have to test how your chosen AI gives you the coordinates, and if they are right and match with zenno.
But I'm sure there are much easier ways to click where you want.

1. You don't need any fixed coordinates by default as you search h element by its parameters, which it tag, id etc. Whatever where element is.
2. If for any reason you don't want to do that, you can still find element by those same parameters and get x and y coordinate points. Add topInBrowser in value for y and leftInBrowser for x coordinate point and then, using those coordinates, you can click where you want.

3. If you again for some reason can't do any of this, you can try using Search by picture feature in zenno.

 

[Alex91]

When a page is dynamic or don't have fixed location for certain buttons or checkbox.

Is it possible to use Zenno Poster to take a full screenshot, then let AI to understand the screenshot of the webpage, by calling the AI using API., then let AI to return click coordinates?

Finally let Zenno Poster to click that x, y coordinates?

Управляем ZennoBrowser с помощью AI + демонстрационный шаблон | ZennoClub - Сообщество профессионалов автоматизации

Сначала я хотел сделать управление через agent.exe, но осознал, что больше потрачу времени на объяснение нейронке в какой области нужно работать и нельзя будет использовать в этот момент компьютер. Я взял за основу идею этого репозитория и решил сделать реализацию внутри ZennoPoster, которая...

zenno.club

 

[VladZen]

You should use Search by picture option to click buttons or elements with variable position and attributes.

 

[zmike]

Thanks for your reply.