1

Quite recently, Android Messages (com.google.android.apps.messaging) started providing reply suggestions to SMS messages I receive. The suggestions are (usually) relevant to the content of the received message: e.g., somebody texts me with a "Can I call you tomorrow?" and Messages suggest things like "Sure" or "Sounds good".

How exactly does it do that? I would be surprised if this kind of AI system could run on my phone. But does that mean that the messages are processed somewhere else? If yes, where and how?

I'm wondering because I've never given any explicit authorisation for my messages to go anywhere, and in fact, I'm using AFWAll+ to prevent Messages (and many other apps) from accessing the Internet at all.

UPDATE: As a test, I used the firewall to also block anything Google Play-related from accessing the Internet, which was the following three entries:

  • Google Play services, Google Services Framework, Google Account Manager, Google Backup Transport
  • Google Play services for Instant Apps
  • Google Play Store

Yet, the suggestions showed up again.


System details:

  • Device: Samsung Galaxy S5 Duos (SM-G900FD)
  • OS: LineageOS 14.1
  • Android version: 7.1.2
  • The phone is rooted with SuperSU.
2

After some research, it is, as I thought, Tensorflow (or to be precise, Tensorflow Lite). You can read the following articles posted by the Google Research team:

Relevant quotes:

Android Wear 2.0 which featured the first "on-device" machine learning technology for smart messaging. This enabled cloud-based technologies like Smart Reply, previously available in Gmail, Inbox and Allo, to be used directly within any application for the first time, including third-party messaging apps, without ever having to connect to the cloud.

Today, we announce TensorFlow Lite, TensorFlow’s lightweight solution for mobile and embedded devices. This framework is optimized for low-latency inference of machine learning models, with a focus on small memory footprint and fast performance.

...

This model generates reply suggestions to input conversational chat messages, with efficient inference that can be easily plugged in to your chat application to power on-device conversational intelligence.

They've also posted a source code link to TF lite's Smart Reply feature, as well as a demo app for testing out Smart Reply.

| improve this answer | |
  • Ignoring the technical aspects of those articles, I understand that no data leaves the device, so there should be no privacy concerns. That's good news! – Ratler Jul 10 '18 at 15:44
  • Absolutely, can't imagine every message going to Google's servers. What a nightmare that would be. – SSS Jul 10 '18 at 20:11
2

To my best knowledge, Google has not revealed how it processes the text messages to generate the suggested replies.

Here are a few things which Google shares: In the Messaging app, the description for Smart reply reads:

To show you suggestions, Smart reply uses your recent messages but does not store them.

Then, if you follow the Google webpage, the page doesn't reveal much other than:

Note: You might be charged for data. Check with your carrier for details.

So, at one place Google said that it uses your recent messages. Then, Google also mentions that you might be charged for data. So, I guess it is easy to summarize that the recent messages are transferred to Google servers to understand and suggest replies.

A part of this processing might also be happening locally on the device. But it is hard to imagine everything happening locally in the Google's world. The cloud and the servers are where the real algorithms reside.

Most of Google services are connected together using Google Play Services. So, even if you do not allow Messages to access the internet, Play Services can. And it won't be too hard for messages to communicate with Play Services which then connects to the internet and interpret messages to generate replies.

Note: This is my personal understanding of what I was able to read from Google website. Google has no where explicitly mentioned that it sends your messages to the cloud. They did mention that they at least do not store any messages.

| improve this answer | |
  • Here's something possibly related: How Google's Inbox does smart replies. – SSS Jul 7 '18 at 12:37
  • @singhnsk, tried blocking Google Play (see OP update for details), but suggestions still show up. – Ratler Jul 10 '18 at 14:36
  • @SSS, the basis of the processing (neural networks) is probably the same, but that article talks about Inbox by Gmail, which is a Cloud service by Google, so there's no mystery on how they would do it there. The question here is: how is the processing (by Google?) applied to my mobile-provider--related SMS messages that are blocked from Internet access. – Ratler Jul 10 '18 at 14:44
  • @Ratler yes, I know it is a cloud service -- I replied that with the assumption that there is some connection to the cloud because Smart Reply says it might charge the user for data. Anywho, I've gotten the actual answer; it's Tensorflow as I expected. I'll post the solution separately. – SSS Jul 10 '18 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.