Android Automation without Intermediary. Part I.

WLDN

Client

21.05.2020

Hello, dear friends! :az:

A small introduction.
Usually I try to write bots using requests, since it is not resource-intensive and generally not difficult if the application is weakly protected.
But there are applications that take a lot of time for high-quality emulation. Therefore, I decided that some things, for example, bot for accounts registration and follower bot, should be work using emulator automation.
In this article, I will share with you my best practices for automating Android emulators. And I'll show you how it works with the Nox emulator and the Youtube application as an example. :al:

Let's get started.
Previously, I automated applications through Appium, but I didn't like bat files, cmd windows and the low flexibility of the whole system.
Therefore, I decided to do the automation directly, using only ADB and UIAutomator. For this, I used SharpAdbClient library.

I will divide the article into three parts for convenience:

Installing and configuring the necessary software.
Principle of operation. Methods overview. XPath.
Getting emulator address (ip:port), multithreading and freeze handling.

1. Installing and configuring the necessary software.

Installing UIAutomator for elements searching:

Download the archive under the article and unpack.
Install AndroidSDK. If the application asks you to install Java, then install it first (jre-8u241-windows-x64.exe).
Launch SDKManager and install needed packages.
Set path to SDK (if empty) into environment variables.
Create a shortcut for uiautomatorviewer.bat on the desktop, this file is located in "android-sdk/tools/uiautomatorviewer.bat", where you installed AndroidSDK.
Run uiautomatorviewer.bat for the test. If a window called UI Automator Viewer appears, then everything is fine.

Installing and configuring emulator and ZennoPoster:

Move libraries (.dll) from ExternalAssemblies to ZennoPoster directory.
Download and install Nox.
Launch MultiDrive and create 3 emulators for the test (for convenience in the settings you can set the phone orientation).
Add the template to ZennoPoster. Put needed paths in the settings (nox_adb.exe and nox.exe in bin folder).
Launch sequentially 3 emulators, then set 3 threads in ZennoPoster, and run for the test. Youtube automation should start.

2. Principle of operation. Methods overview. XPath.

How is it possible to link adb with an emulator without an intermediary (for example, Appium)?

ADB has many commands to control Android. For example, the command

cmd.exe:

adb shell input tap x y

allows to tap on the specified coordinates.

Accordingly, in order to tap on the needed element, you need to find out the coordinates of this element.
UIAutomator is preinstalled on the emulator by default, so you can get data of visible elements using the command:

cmd.exe:

adb shell uiautomator dump

But in this case, XML will be saved in the Android root folder, and you have to download it to your PC to view it - it is long and inconvenient, so you need to add /dev/tty to the command.
Now the command look like this:

cmd.exe:

adb shell uiautomator dump /dev/tty

Thus, you can output XML content to the log, in this case SharpAdbClient library will get the answer and you will be able to extract the needed coordinates.

Here is a short list of frequently used commands supported by adb:

cmd.exe:

adb help //List all comands
adb start-server //Start ADB server
adb kill-server //Kill ADB server
adb connect <ip:port> //Connect to device
adb devices //Show devices attached
adb reboot //Reboot device
adb install <path to .apk> //установить .apk файл
adb shell //Open or run commands in a terminal on the host Android device
adb shell input x y //Tap with x,y - coordinates
adb shell input swipe x1 y1 x2 y2 sss //Swipe, sss - speed in milliseconds
adb shell input text <string> //Send text
adb shell input keyevent <event_code> //Send event (full list of events below)
adb shell pm list packages //Show installed packages
adb shell pm uninstall <com.your.app> //Remove package
adb shell dumpsys window windows | grep -E 'mCurrentFocus|mFocusedApp' //Show current activity

All other commands can be found in the search engine, just write "adb commands list".

Now let's move on to the template.
Needed .dlls have already been added to GAC, and needed namespaces have been added to using directive and common code.

Using directive and common code:

using SharpAdbClient;
using System.Net;
using System.Xml;
using System.Windows.Forms;
using System.Diagnostics;
using System.Management;

Now SharpAdbClient library can be used for ADB commands.
I wrote methods for convenience in general code.

Let's go over the finished cubes.
Let's start by checking if adb server is running with path from settings_adb variable:

C#:

if (!AdbServer.Instance.GetStatus().IsRunning) {
AdbServer server = new AdbServer();
var result = server.StartServer(project.Variables["settings_adb"].Value, restartServerIfNewer: false);

    throw new Exception("Restart.");
}

Connect to emulator ip:port. Usually it connects automatically, but you make a little safety by command:

C#:

AdbClient.Instance.Connect(new DnsEndPoint("127.0.0.1", int.Parse(Regex.Match(project.Variables["device"].Value, "(?<=:).*").ToString())));

Create ADB object and save it in context to use further in the project:

C#:

ADB a = new ADB(project);
project.Context["ADB"] = a;

Create PackageManager object, which allow Uninstall/Install the application in the same snippet:

C#:

var a = project.Context["ADB"];
var device = a.Device();

SharpAdbClient.DeviceCommands.PackageManager manager = new SharpAdbClient.DeviceCommands.PackageManager(device);
try {
manager.UninstallPackage("com.google.android.youtube"); //Delete Package
} catch (Exception e) {}
manager.InstallPackage(project.Directory + @"\youtube.apk", reinstall: false); //Install apk

Launch app:

C#:

var a = project.Context["ADB"];
a.StartApp("com.google.android.youtube/com.google.android.apps.youtube.app.WatchWhileActivity"); //Launch app
// adb command for getting activity name
// adb shell dumpsys window windows | grep -E 'mCurrentFocus|mFocusedApp'

Waiting for element:

C#:

var a = project.Context["ADB"];
a.Wait("//node[@resource-id='' and @class='android.widget.ImageView']", 0, 10); //XPath, Index, Seconds

Tap on element:

C#:

var a = project.Context["ADB"];
a.Click("//node[@resource-id='com.google.android.youtube:id/menu_search' and @class='android.widget.TextView']", 0, 10); //XPath, Index, Seconds

Notice here for the first time you use XPath to find an element.
Learning to make your own XPath (it isn't difficult):

First, start Nox and make sure it is connected to adb.

cmd.exe:

adb devices

You should see your device in the list.

If there are no connected devices, then you have to connect the emulator yourself using the command:

cmd.exe:

adb connect 127.0.0.1:62001

You should receive message about connect. Usually the port of the first emulator is 62001, but in my case it is 62025.

If you can’t catch the port, then try searching from 62001 to 62100.

Now install and run youtube.apk from archive, and then run uiautomatorviewer.bat and dump the window.

You can move the cursor over the elements and notice that the corresponding nodes are highlighted on the right side of the window. Accordingly, in order to compose XPath, you must select needed element and take the attributes that can identify the element in the tree.
Compose XPath for the loupe (search), taking the attributes resource-id and class:

XPath:

//node[@resource-id='com.google.android.youtube:id/menu_search' and @class='android.widget.TextView']

If it happens that the element cannot be identified (attributes are not specified), then in this case you can compose a long XPath, for example, to the Home element to the left of the loupe:

XPath:

//node[@resource-id='com.google.android.youtube:id/toolbar' and @class='android.view.View']/node[@class='android.widget.TextView']

All XPath syntax can be found here.

Inputting text and pressing Enter:

C#:

var a = project.Context["ADB"];
a.Text("ZennoLab"); //text input
a.KeyEvent("66"); //Enter

0 --> "KEYCODE_UNKNOWN"
1 --> "KEYCODE_MENU"
2 --> "KEYCODE_SOFT_RIGHT"
3 --> "KEYCODE_HOME"
4 --> "KEYCODE_BACK"
5 --> "KEYCODE_CALL"
6 --> "KEYCODE_ENDCALL"
7 --> "KEYCODE_0"
8 --> "KEYCODE_1"
9 --> "KEYCODE_2"
10 --> "KEYCODE_3"
11 --> "KEYCODE_4"
12 --> "KEYCODE_5"
13 --> "KEYCODE_6"
14 --> "KEYCODE_7"
15 --> "KEYCODE_8"
16 --> "KEYCODE_9"
17 --> "KEYCODE_STAR"
18 --> "KEYCODE_POUND"
19 --> "KEYCODE_DPAD_UP"
20 --> "KEYCODE_DPAD_DOWN"
21 --> "KEYCODE_DPAD_LEFT"
22 --> "KEYCODE_DPAD_RIGHT"
23 --> "KEYCODE_DPAD_CENTER"
24 --> "KEYCODE_VOLUME_UP"
25 --> "KEYCODE_VOLUME_DOWN"
26 --> "KEYCODE_POWER"
27 --> "KEYCODE_CAMERA"
28 --> "KEYCODE_CLEAR"
29 --> "KEYCODE_A"
30 --> "KEYCODE_B"
31 --> "KEYCODE_C"
32 --> "KEYCODE_D"
33 --> "KEYCODE_E"
34 --> "KEYCODE_F"
35 --> "KEYCODE_G"
36 --> "KEYCODE_H"
37 --> "KEYCODE_I"
38 --> "KEYCODE_J"
39 --> "KEYCODE_K"
40 --> "KEYCODE_L"
41 --> "KEYCODE_M"
42 --> "KEYCODE_N"
43 --> "KEYCODE_O"
44 --> "KEYCODE_P"
45 --> "KEYCODE_Q"
46 --> "KEYCODE_R"
47 --> "KEYCODE_S"
48 --> "KEYCODE_T"
49 --> "KEYCODE_U"
50 --> "KEYCODE_V"
51 --> "KEYCODE_W"
52 --> "KEYCODE_X"
53 --> "KEYCODE_Y"
54 --> "KEYCODE_Z"
55 --> "KEYCODE_COMMA"
56 --> "KEYCODE_PERIOD"
57 --> "KEYCODE_ALT_LEFT"
58 --> "KEYCODE_ALT_RIGHT"
59 --> "KEYCODE_SHIFT_LEFT"
60 --> "KEYCODE_SHIFT_RIGHT"
61 --> "KEYCODE_TAB"
62 --> "KEYCODE_SPACE"
63 --> "KEYCODE_SYM"
64 --> "KEYCODE_EXPLORER"
65 --> "KEYCODE_ENVELOPE"
66 --> "KEYCODE_ENTER"
67 --> "KEYCODE_DEL"
68 --> "KEYCODE_GRAVE"
69 --> "KEYCODE_MINUS"
70 --> "KEYCODE_EQUALS"
71 --> "KEYCODE_LEFT_BRACKET"
72 --> "KEYCODE_RIGHT_BRACKET"
73 --> "KEYCODE_BACKSLASH"
74 --> "KEYCODE_SEMICOLON"
75 --> "KEYCODE_APOSTROPHE"
76 --> "KEYCODE_SLASH"
77 --> "KEYCODE_AT"
78 --> "KEYCODE_NUM"
79 --> "KEYCODE_HEADSETHOOK"
80 --> "KEYCODE_FOCUS"
81 --> "KEYCODE_PLUS"
82 --> "KEYCODE_MENU"
83 --> "KEYCODE_NOTIFICATION"
84 --> "KEYCODE_SEARCH"
85 --> "TAG_LAST_KEYCODE"

It happens when you need to do double tap or other actions that are not on the list.
You can use sendevent command:

cmd.exe:

adb shell sendevent /dev/input/event<x>

Get element coordinates using GetCoord method.
You can get "clean" coordinates or random point in element. In this case, use the second option:

C#:

var a = project.Context["ADB"];
string coord = a.GetCoord("//node[@resource-id='com.google.android.youtube:id/channel_avatar' and @class='android.widget.ImageView']", 0, 10, true); //XPath, Index, Seconds, random point into x1,y1 and x2,y2
string[] coords = coord.Split(new char[] {','});
project.Variables["x"].Value = coords[0];
project.Variables["y"].Value = coords[1];

Enter the command in cmd, click on the emulator and record the event:

cmd.exe:

adb shell getevent

You will get the values in HEX, convert it to DEC using any service.
Result:

Код:

/dev/input/event<N>: 1 330 1
/dev/input/event<N>: 3 58 1
/dev/input/event<N>: 3 53 <x>
/dev/input/event<N>: 3 54 <y>
/dev/input/event<N>: 0 2 0
/dev/input/event<N>: 0 0 0
/dev/input/event<N>: 0 2 0
/dev/input/event<N>: 0 0 0
/dev/input/event<N>: 1 330 0
/dev/input/event<N>: 3 58 0
/dev/input/event<N>: 3 53 0
/dev/input/event<N>: 3 54 38
/dev/input/event<N>: 0 2 0
/dev/input/event<N>: 0 0 0

Get event number using command:

C#:

var a = project.Context["ADB"];
a.Command("cat /proc/bus/input/devices", true);
project.Variables["event"].Value = Regex.Match(project.Variables["receiver"].Value, "(?<=mouse2 event).*").ToString().Trim(); //Get event №

Tap with sendevent using event № and coordinates (in this case, double tap is not needed, so I commented out the second part in the template ):

C#:

var a = project.Context["ADB"];
string evnt = project.Variables["event"].Value;
string x = project.Variables["x"].Value;
string y = project.Variables["y"].Value;
a.Command(String.Format("sendevent /dev/input/event{0} 1 330 1", evnt), false);
a.Command(String.Format("sendevent /dev/input/event{0} 3 58 1", evnt), false);
a.Command(String.Format("sendevent /dev/input/event{0} 3 53 {1}", evnt, x), false);
a.Command(String.Format("sendevent /dev/input/event{0} 3 54 {1}", evnt, y), false);
a.Command(String.Format("sendevent /dev/input/event{0} 0 2 0", evnt), false);
a.Command(String.Format("sendevent /dev/input/event{0} 0 0 0", evnt), false);
a.Command(String.Format("sendevent /dev/input/event{0} 0 2 0", evnt), false);
a.Command(String.Format("sendevent /dev/input/event{0} 0 0 0", evnt), false);
a.Command(String.Format("sendevent /dev/input/event{0} 1 330 0", evnt), false);
a.Command(String.Format("sendevent /dev/input/event{0} 3 58 0", evnt), false);
a.Command(String.Format("sendevent /dev/input/event{0} 3 53 0", evnt), false);
a.Command(String.Format("sendevent /dev/input/event{0} 3 54 38", evnt), false);
a.Command(String.Format("sendevent /dev/input/event{0} 0 2 0", evnt), false);
a.Command(String.Format("sendevent /dev/input/event{0} 0 0 0", evnt), false);

Swipe:

C#:

var a = project.Context["ADB"];
    a.Swipe("200", "600", "200", "200", "900"); //coords x1, y1, x2, y2, sss - speed in ms

You can also parse something. Parse duration of visible videos (the result will appear in the template list):

C#:

var a = project.Context["ADB"];
project.Lists["parse"].AddRange(a.Parse("//node[@resource-id='com.google.android.youtube:id/duration' and @class='android.widget.TextView']", "text", 0, 5)); //XPath, attribut for parsing, Index, Seconds

Button Back:

C#:

var a = project.Context["ADB"];
a.Back();

Button Home and Kill Process:

C#:

var a = project.Context["ADB"];
a.Home(); //Minimize all windows
a.Kill("com.google.android.youtube"); //Kill process

Reboot Android:

C#:

var a = project.Context["ADB"];
a.Reboot();

Upload file. I did 2 upload methods: standart upload

C#:

var a = project.Context["ADB"];
a.UploadFile("/storage/sdcard0/ZennoLab.txt", project.Directory + @"\ZennoLab.txt");

and from the variable.

C#:

var a = project.Context["ADB"];
a.UploadFromVar("/storage/sdcard0/ZennoLab.txt", "ZennoLab TEST");

File Download:

C#:

var a = project.Context["ADB"];
a.DownloadFile("/storage/sdcard0/ZennoLab.txt", project.Directory + @"\Download_test.txt");

Delete File:

C#:

var a = project.Context["ADB"];
a.Command("rm -rf /storage/sdcard0/ZennoLab.txt", false);

3. Getting emulator address (ip:port), multithreading and freeze handling.

To multi-threaded mode, I decided to use a locked global variable in which processes id (pid) of the emulator windows will be written via ";".
When the template starts, the pid of the first free emulator in the list is written to the global variable. If the variable did not have this pid, then another thread will not work with this pid. :bf:

PID can be found in the Task Manager (Ctrl+Shift+Esc).

Next, if you enter the command in cmd:

cmd.exe:

netstat -a -n -o

then a lot of addresses will be displayed, and in the rightmost column you can see pid.

Knowing that the ports for adb connecting to Nox starts with 620, you can take the address by pid using the command:

cmd.exe:

netstat -a -n -o | find "PID" | find "127.0.0.1" | find "620"

instead of PID enter process id (pid).

Now do the same in Zenno.
Initialize/check the existence of a global variable:

C#:

lock(SyncObject) {
    try {
        var gbVar = project.GlobalVariables["Zappium", "process"];
        return null;
    } catch (KeyNotFoundException ex) {
        string defaultValue = String.Empty;
        project.GlobalVariables.SetVariable("Zappium", "process", defaultValue);
    }
}

Get free PID using .Net library:

C#:

lock(SyncObject) {
var gbVar = project.GlobalVariables["Zappium", "process"];
Process[] processes = Process.GetProcessesByName("NoxVMHandle");
var ids = processes.Select(p => p.Id);
string process = "";
foreach(int processId in ids){
    project.SendInfoToLog(processId.ToString());
    process = processId.ToString();
    if (project.Variables["process"].Value == String.Empty && !project.GlobalVariables["Zappium", "process"].Value.ToString().Contains(process)){
        project.Variables["process"].Value = process;
        gbVar.Value = gbVar.Value + process + ";";
        }
}
}

Get the device address by substituting pid in the previously written command:

C#:

Process cmd = new Process();
cmd.StartInfo.FileName = "cmd.exe";
cmd.StartInfo.RedirectStandardInput = true;
cmd.StartInfo.RedirectStandardOutput = true;
cmd.StartInfo.CreateNoWindow = true;
cmd.StartInfo.UseShellExecute = false;
cmd.Start();

cmd.StandardInput.WriteLine(String.Format("netstat -a -n -o | find \"{0}\" | find \"127.0.0.1\" | find \"620\"",project.Variables["process"].Value));

cmd.StandardInput.Flush();
cmd.StandardInput.Close();
cmd.WaitForExit();

return "127.0.0.1:" + Regex.Match(cmd.StandardOutput.ReadToEnd(), "(?<=127.0.0.1:)62.*?(?= )");

Upon template completion, pid will be delete from the global variable to free the emulator for the following threads:

C#:

lock(SyncObject) {
    var gbVar = project.GlobalVariables["Zappium", "process"];
    gbVar.Value = gbVar.Value.ToString().Replace(project.Variables["process"].Value + ";", "");
}

Sometimes it happens that the emulator freezes, for this I use the CommandLineUtilities method in the general code to record all processes associated with the emulator, close and reopen them.
Close processes:

C#:

Process[] processes = Process.GetProcessesByName("Nox");
var ids = processes.Select(p => p.Id);
var process = Process.GetProcessById(int.Parse(project.Variables["process"].Value));

string name = Regex.Match(CommandLineUtilities.getCommandLines(process), "(?<=--comment ).*(?= --startvm)").ToString();
project.SendInfoToLog(CommandLineUtilities.getCommandLines(process));

foreach(int processId in ids){
    var nox = Process.GetProcessById(processId);
    if (Regex.Match(CommandLineUtilities.getCommandLines(nox),"(?<=-clone:).*").ToString().Contains(name)){
        project.Variables["start_cmd"].Value = CommandLineUtilities.getCommandLines(nox);
    nox.Kill();
        break;
    }
}
process.Kill();

Reopen emulator:

C#:

var proc = System.Diagnostics.Process.Start(project.Variables["settings_nox"].Value, Regex.Match(project.Variables["start_cmd"].Value, "-clone:.*").ToString());

Conclusion.
I hope my article will facilitate your work with emulators and help create a more advanced scheme for interacting with them. This is just a small part of how you can interact with emulators.
I will be glad to your votes if you liked my article and were useful for you.
Thank you for the attention! :bt: