# Tools
Run `android layout --help` and `android screen --help`.

## UI Dump
`android layout`  returns a flat JSON list of the UI elements on screen.
`android layout --diff` returns a flat JSON list of the UI elements that have changed since the last call to `layout` or `layout --diff`

Each JSON object represents a UI element in the Android app. The following properties may be present:
- `text` - any literal text the element contains
- `resourceId` - the Android resource id used to refer to the element
- `contentDesc` - a description of a UI element for use by accessibility tools
- `interactions` - the set of user interactions the element supports. May contain one or more of: `checkable`, `clickable`, `focusable`, `scrollable`, `long-clickable`, `password`
- `state` - the set of states the element is in. May contain one or more of `checked`, `focused`, `selected`
- `bounds` - the screen coordinates of the bounding rectangle of the element, in the format `[min X,min Y][max X, max Y]`
- `center` - the screen coordinates of the center of the element, in the format `[x,y]`
- `off-screen` - if true, the element is in the UI hierarchy but not visible; it may require scrolling to view.

Use `layout` as a primary means of examining an Android app. Use `layout --diff` to focus on changes and to keep your context small.
Example: When entering digits into a calculator, use `layout --diff` to output only the digit readout element.

`layout` may fail due to the app displaying a WebView or animation; in these cases, use `android screen --annotate` to inspect the app.
This failure will likely resolve after navigating away from the current screen.

## Screenshot
`android screen capture -o <file path>` saves a PNG of the current device screen to `<file path>`

Use `screen capture` as a secondary means of examining an Android app
Examples:
- Understanding the content of an on-screen image
- Looking at a `WebView` (web content does not always appear in the ui dump)
- Trying to find a UI element by its visual appearance

**IMPORTANT**: Always *VISUALLY* examine the PNG image returned from `android screen` BEFORE doing anything else.

## Annotated Screenshot
`android screen capture --annotate -o <file path>`
`android screen resolve --screen <path> --string <string>`

The `--annotate` command adds numerical labels and bounding boxes around UI elements. Use this command to locate UI elements that cannot
be located in the `layout` output.

**IMPORTANT**: When using `android –-annotate`, always *VISUALLY* examine the resulting PNG file.

To refer to these labels in input commands, use `screen resolve` to convert labels into coordinates:

`android screen resolve --screen <file path> --string "#3"` returns `<x coord of region 3> <y coord of region 3>`

To save turns, you can combine shell commands:

`adb shell input $(android screen resolve --screen screen.png --string "tap #34")`

This command taps on region #34 from `screen.png`

## Input
Use `adb shell input` for interacting with Android devices.
Refer to the `"interactions"` property of an element for what interactions can be performed on a particular element.

Interact with UI elements with their `center` coordinate or their `bounds` coordinates:
```json
{
  "key": -248568265,
  "class": "android.widget.Button",
  "bounds": "[138,9][167,38]",
  "center": "[152,23]"
}
```
To tap on this button, you would execute `adb shell input tap 152 23`. This taps the center.

```json
{
  "key": 12487234,
  "class": "com.example.ui.ScrollableList",
  "bounds": "[100,200][400,600]",
  "center": "[250,400]"
}
```
To scroll down on this list, you would execute `adb shell input swipe 250 400 250 200 500`. This swipes from the center to the top over 500ms.

# Android Interaction Rules
1. Always ensure text input fields have `"focused"` in their `"state"` list before entering text
2. If an element has `"scrollable"` in its `"interactions"` list, try scrolling it when looking for missing UI elements
3. Always scroll slowly when executing scroll inputs. In `adb shell input swipe <x1> <y1> <x2> <y2> [duration(ms)]`, the duration is the optional 5th parameter after `swipe` (the 6th argument to the `input` utility).
4. Content may take time to load; if a `layout` is missing information after you take an action, wait a few seconds, then perform `layout --diff` to see if anything changes.