Wednesday, January 23, 2013

If we make the data big enough, we would expect a big increase in the amount of time taken as we add more threads, unless you have an SSD (solid-state disk). The more files you try to write simultaneously on a regular mechanical disk, the slower the total throughput will be. This is because the disk is forced to spend more time seeking to a new track, and less time writing to the track it's already on.

So here's what I did next: I added a new variable into the mix. I added a second @Varying parameter (axis=series) for the amount of data that should be written. I tried ranging from 10MB to 100MB in increments of 10MB to what happens hoping maybe a clear trend will emerge.

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import org.jboss.perfrunner.Axis;
import org.jboss.perfrunner.PerfRunner;
import org.jboss.perfrunner.Varying;
import org.junit.Test;
import org.junit.runner.RunWith;
public class SampleTest {
    public void numTest (
    @Varying(name = "MB", axis=Axis.X, from = 1, to = 10000000,step = 250000)int numMax)throws Exception{
        String content = "0";   
        FileWriter fstream = new FileWriter("out.txt");  
        BufferedWriter out = new BufferedWriter(fstream);
        for(int num=0; num < numMax; num++){ 
     public void perThread (
     @Varying(axis=Axis.SERIES, name = "Megabyte",from = 10000000, to = 100000000, step= 10000000) final int megabyte,
    @Varying(name = "thread", axis = Axis.X, from = 1, to = 10, step = 1) final int threadCount) throws InterruptedException {
     // First define the task that needs to be done.
     Runnable task = new Runnable() {
     public void run() {
      // create temporary file with extension suffix
         File file1 = null;
     file1 = File.createTempFile("PerThreadTest", ".javatemp");
     BufferedWriter out = new BufferedWriter(new FileWriter(file1));
     for (int i = 1; i < megabyte; i++) {
        out.close() ;
        } catch (IOException e) {
            // TODO Auto-generated catch block
     // Now define the executor service that will execute the above task
     ExecutorService exec = Executors.newFixedThreadPool(threadCount);
     // Submit the task 10 times
     for (int i = 0; i < 10; i++) {
     // Finally, wait for all the submitted tasks to complete (1 hour should be way more than enough!)
     exec.awaitTermination(1, TimeUnit.SECONDS);
     // Free the threads that were created by Executors.newFixedThreadPool(threadCount) above

   I ran the test 4 more times at this weight and different weights. The results were random for outliers for different weights at a different number of threads. We think what we're seeing in the data is the expected trend: the more different files we write simultaneously, the longer it takes to write the same amount of data. There's an occasional dominating effect where one of the files takes a really long time to write (like 10x longer). We think what we're seeing is the operating system deciding (based on whatever rules it uses) to flush out the disk cache so one writer gets held back while the OS processes data from all sorts of places, including my other writers as well as other unrelated processes  but the underlying signal is there, for sure.

No comments:

Post a Comment