Wednesday, January 16, 2013

By Tuesday afternoon I was still doing performance testing wrong. I had been looking at queueSize in PooledExecutorService in org.jboss.errai.bus.server.async.scheduling. I thought I would vary that to determine how many users can use the ErraiBus.  This is the code I was trying to test.

package org.company.firestorm.client.local;

import org.jboss.perfrunner.Axis;
import org.jboss.perfrunner.PerfRunner;
import org.jboss.perfrunner.Varying;
import org.junit.Test;
import org.junit.runner.RunWith;

@RunWith(PerfRunner.class)


public class PooledExecutorServiceTest {
     @Test
      public void testPooledExecutorService (
          @Varying(axis=Axis.X, name="Queue Size", from=0, to=100, step=10) int queueSize
          ) throws InterruptedException {
           Thread.sleep(queueSize);
       }
}
This was totally wrong. For one thing I was trying to white box test the internals of Errai instead of black box testing. Jonathan suggested I try performance test writing 0s into a file for 1 MB, 10 MB, 100 MB, and 1000 MB. It was pass 5 PM my time so I thought I would stop for the day and get something to eat. I fell asleep relatively early so I found myself up at 2 AM. I decided to do some work. I have gotten into the habit of now and then of working in the middle of the night this way. When I was in school it was common for the professors to be up at night and come in late to teach the next day. I found myself more than a few times exchanging emails with the grad adviser at 2 AM. Now that I am out of school I can work in the middle of the night without being interrupted by nurses, home health aids, and phone calls. The problem with this is Jonathan is on the irc 8 AM - 5:30 PM my time when I can ask him questions. I find myself some times having a hard time staying awake during this time because I was up early. I write emails to Jonathan of what I had been doing and answers he them when he gets in the next morning. I do chat with Jonathan on the irc during the day.

This morning I seemed to be doing something right. This is the code  I was first testing early this morning:

package org.company.firestorm.client.local;
import java.io.*;
import org.jboss.perfrunner.*;
import org.junit.Test;
import org.junit.runner.RunWith;

@RunWith(PerfRunner.class)
public class sampleTest {
    @Test
    public void numTest (
    @Varying(name = "MB", axis=Axis.X, from = 1, to = 1000, step = 10) int numMax) throws Exception{
        String content = "0";
        FileWriter fstream = new FileWriter("out.txt");
        BufferedWriter out = new BufferedWriter(fstream);
        for(int num=0; num < numMax; num++ ){

        out.write(content);
        }
        out.close();
        
   }
}
<perfrunner-org.company.firestorm.client.local.sampleTest.html><out.txt>
 
 
 
This was the result I got. I didn't get the linear up to the right line that I expected. The text file "out" did have all the 0s expected.
 
Let's draw some parallels to what we plan to eventually do with Errai performance testing:

1. I am black-box testing BufferedWriter. I didn't have to examine the inner workings of BufferedWriter in order to create this test. I just used its public API.
2. The results surprised me! This is the hallmark of a useful performance test, because it means you're about to learn something new. :-)
3. The independent variable (numMax) in this test was not used to tune the BufferedWriter I was testing. Instead, it was used to decide how much exercise to give to the BufferedWriter.
 
I then used 1 MB, 10 MB, 100 MB, and 1000 MB for the "to =  " part of the code and changed the step so the trials were tested 4 times (i.e. 0.25 MB, 2.5 MB...). This time I got more linear code that went up to the right. The "out.txt" had millions more 0s. It appears when I was measuring smaller numbers with smaller steps I was just measuring noise. The signal had not gotten over the S/N threshold. Now I must devise a performance test that would help me find the breaking point where the cost of writing to the file is detectable. I think I am going to eat and take a nap first.

No comments:

Post a Comment