I did rooting with Kingo root, and also i did factory reset. but i still can't use OTG When I press power+voldown, boot menu starts and immediately shows "Downloading... Do not turn of target!" then i tried Hold Power + Volume Up + Volume Down, and then i can see recovery mode, but it still not working - no command ... I still see "System update download" , still failed. what can i try more?
If you are planning on installing a custom rom you need to install a recovery to flash the the ROM. TWRP is good
To get Marshmallow on the Nexus 10 Omnirom is your best bet. You'll need to flash gapps and supersu for root too.
The nexus 10 received 5.1.1 security updates until April 2016 that you can get OTA on a stock device or install manually, but manual installs will wipe the content of the device each time.