BTEP has a "Using Better Terminal Emulator Pro" page on their wiki which explains most of the special shortcuts and keys. The basic breakdown of what you're looking for is:
For directional input: Either use a keyboard that supports this or use BTEP's virtual direction pad feature. You can access it by tapping the screen when the keyboard is visible, long-pressing the screen when it isn't, or pressing the Search button. The cursor should change to a
> character to indicate you're in VDP mode, and you can swipe your directional input (i.e. swipe up for the up key).
For ESC: This is usually mapped by default to the volume up key, but you can remap it to any of the available programmable shortcut keys. You can also press the Search button then press
[, since Search is mapped to the control character on devices that lack a trackball or dpad (as well as activating VDP mode) and
^[ (CTRL-[) is the ESC character.