Either your server is refusing connection or one of the routers between your phone and the server (e.g. your carrier's router) is refusing connection.
Try connecting to the server from another device (preferrably from a Wifi connection), if your connection is refused from that other device as well, then it is almost definitely a server configuration issue and this question should be asked on SuperUser or ServerFault.
Your question is way too vague to be answerable, you will need to answer these questions (hint: you can edit your question, then put a comment to notify your answerers):
There is no such thing as a device "talking" to another device. What do you mean by "talking" to the server? Is your server serving static HTML pages, CGI scripts, or is it some kind of media streaming server?
Is the IP address the server's local IP address (e.g. looks like 127.0.0.xxx or 192.168.xxx.xxx, or is it the server's public IP address?
Can you access this server from other devices? What have you tried?
How are you connecting to the serve from the Android? Is it over Wifi or over 3G?