0

I have configured experiments to test which among 10 launched apps impacts performance the most. But I can't make an objective comparison, because if I launch the experiment twice with the same configuration, I get very different performance degradation, in terms of memory occupied and launch times. This is a piece of code from one of the workload scripts. I set the seed to always generate the same events, I wipe the cache at the end of each experiment and clean up the apps before starting another, yet in two experiments with the same settings I get a memory occupancy that grows up to three times faster in one than in the other.

while true; do


adb shell monkey -p com.android.chrome -s 1 --pct-appswitch 100 --ignore-crashes --ignore-timeouts --ignore-security-exceptions --monitor-native-crashes -v -v 1 


sleep 1
adb shell monkey -p com.facebook.katana -s 1 --pct-appswitch 100 --ignore-crashes --ignore-timeouts --ignore-security-exceptions --monitor-native-crashes -v -v 1


sleep 1
adb shell monkey -p com.google.android.apps.maps -s 1 --pct-appswitch 100 --ignore-crashes --ignore-timeouts --ignore-security-exceptions --monitor-native-crashes -v -v 1 

done

Is there a way to get similar performance degradation in two experiments? Is it just me doing something wrong or is this something that can't be achieved because there are too many random factors at play?

7
  • 1
    I still don't fully understand what you want to achieve: "which among 10 launched apps impacts performance the most" what performance are you talking about and how it is measured? Do you want to develop some sort of benchmark? Do you mean performance of the app itself or how they affect a different app? Why do you use apps that heavily depend on random content from the internet (such as Facebook).
    – Robert
    May 1 at 12:26
  • The overall experiment was to be this: do 11 experiments. In the first one I would launch 10 apps, while in the other 10 experiments I would launch 9 apps by varying the app I don't launch. Then I would have to analyze the data of the first experiment and try to predict, with different techniques, the absence of which app affects performance the most and then compare it with the experimental data of the 10 experiments. But if there are other factors that degrade performance differently in each experiment, I can't compare the degradation between one experiment and another. @Robert
    – Luigi
    May 1 at 13:14
  • 1
    In my opinion this is a useless experiment, especially as it mainly tests Android and the used phone instead of the "not launched app". Because your set of apps contains apps that mainly show data received from a web server (which you don't control) you have to perform each experiment may be 100 to 1000 times and then calculate e.g. the average or examine the 95 percentile.
    – Robert
    May 1 at 13:23
  • I don't have a lot of experience with stress testing, but I do agree with you. I too had thought that it was necessary to repeat the experiments many times to get what I needed. The experiment done this way was not my idea. I was assigned it and was trying to complete it. I think it needs to be completely rethought though.
    – Luigi
    May 1 at 14:02
  • Another problem you will face is that you can't reset the "thermal state" of the device. The more the CPU/GPU performs work the more heat it produces. Even using active cooling solutions will not fully be able to reset the phone back to it's original temperature. Especially phones with a strong CPU/GPU can produce a huge amount of heat in a short time, so much that starts throttling. So for comparable results mind the environment temperature and allow the phones to cool down after a run (e.g. 30 minutes).
    – Robert
    May 1 at 14:23

0

You must log in to answer this question.

Browse other questions tagged .