Is there a better way to generate 5 million csv files quickly

I would like to create 5 million csv files, I have waiting for almost 3 hours, but the program is still running. Can somebody give me some advice, how to speed up the file generation.

After these 5 million files generation complete, I have to upload them to s3 bucket.

It would be better if someone know how to generate these files through AWS, thus, we can move files to s3 bucket directly and ignore network speed issue.(Just start to learning AWS, there are lots of knowledge need to know)

The following is my code.

public class ParallelCsvGenerate implements Runnable {
    private static AtomicLong baseID = new AtomicLong(8160123456L);
    private static ThreadLocalRandom random = ThreadLocalRandom.current();
    private static ThreadLocalRandom random2 = ThreadLocalRandom.current();
    private static String filePath = "C:\\5millionfiles\\";
    private static List<String> headList = null;
    private static String csvHeader = null;
    public ParallelCsvGenerate() {
        headList = generateHeadList();
        csvHeader = String.join(",", headList);
    }


    @Override
    public void run() {
        for(int i = 0; i < 1000000; i++) {
            generateCSV();
        }s
    }


    private void generateCSV() {
        StringBuilder builder = new StringBuilder();
        builder.append(csvHeader).append(System.lineSeparator());
        for (int i = 0; i < headList.size(); i++) {
            if(i < headList.size() - 1) {
                builder.append(i % 2 == 0 ? generateRandomInteger() : generateRandomStr()).append(",");
            } else {
                builder.append(i % 2 == 0 ? generateRandomInteger() : generateRandomStr());
            }
        }


        String fileName = String.valueOf(baseID.addAndGet(1));
        File csvFile = new File(filePath + fileName + ".csv");
        FileWriter fileWriter = null;
        try {
            fileWriter = new FileWriter(csvFile);
            fileWriter.write(builder.toString());
            fileWriter.flush();
        } catch (Exception e) {
            System.err.println(e);
        } finally {
            try {
                if(fileWriter != null) {
                    fileWriter.close();
                }
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }




    private static List<String> generateHeadList() {
        List<String> headList = new ArrayList<>(20);
        String baseFiledName = "Field";
        for(int i = 1; i <=20; i++) {
            headList.add(baseFiledName + i);
        }
        return headList;
    }




    /**
     * generate a number in range of 0-50000
     * @return
     */
    private Integer generateRandomInteger() {
        return random.nextInt(0,50000);
    }




    /**
     * generate a string length is 5 - 8
     * @return
     */
    private String generateRandomStr() {
        int strLength = random2.nextInt(5, 8);
        String str="abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
        int length = str.length();
        StringBuilder builder = new StringBuilder();
        for (int i = 0; i < strLength; i++) {
            builder.append(str.charAt(random.nextInt(length)));
        }
        return builder.toString();
    }

Main

ParallelCsvGenerate generate = new ParallelCsvGenerate();


Thread a = new Thread(generate, "A");
Thread b = new Thread(generate, "B");
Thread c = new Thread(generate, "C");
Thread d = new Thread(generate, "D");
Thread e = new Thread(generate, "E");

a.run();
b.run();
c.run();
d.run();
e.run();

Thanks for your guys advice, just refactor the code, and generate 3.8million files using 2.8h, which is much better.
Refactor code:

public class ParallelCsvGenerate implements Callable<Integer> {
    private static String filePath = "C:\\5millionfiles\\";
    private static String[] header = new String[]{
            "FIELD1","FIELD2","FIELD3","FIELD4","FIELD5",
            "FIELD6","FIELD7","FIELD8","FIELD9","FIELD10",
            "FIELD11","FIELD12","FIELD13","FIELD14","FIELD15",
            "FIELD16","FIELD17","FIELD18","FIELD19","FIELD20",
    };
    private String fileName;
    public ParallelCsvGenerate(String fileName) {
        this.fileName = fileName;
    }

    @Override
    public Integer call() throws Exception {
        try {
            generateCSV();
        } catch (IOException e) {
            e.printStackTrace();
        }
        return 0;
    }

    private void generateCSV() throws IOException {

        CSVWriter writer = new CSVWriter(new FileWriter(filePath + fileName + ".csv"), CSVWriter.DEFAULT_SEPARATOR, CSVWriter.NO_QUOTE_CHARACTER);
        String[] content = new String[]{
                RandomGenerator.generateRandomInteger(),
                RandomGenerator.generateRandomStr(),
                RandomGenerator.generateRandomInteger(),
                RandomGenerator.generateRandomStr(),
                RandomGenerator.generateRandomInteger(),
                RandomGenerator.generateRandomStr(),
                RandomGenerator.generateRandomInteger(),
                RandomGenerator.generateRandomStr(),
                RandomGenerator.generateRandomInteger(),
                RandomGenerator.generateRandomStr(),
                RandomGenerator.generateRandomInteger(),
                RandomGenerator.generateRandomStr(),
                RandomGenerator.generateRandomInteger(),
                RandomGenerator.generateRandomStr(),
                RandomGenerator.generateRandomInteger(),
                RandomGenerator.generateRandomStr(),
                RandomGenerator.generateRandomInteger(),
                RandomGenerator.generateRandomStr(),
                RandomGenerator.generateRandomInteger(),
                RandomGenerator.generateRandomStr()
        };
        writer.writeNext(header);
        writer.writeNext(content);
        writer.close();
    }

}

Main

public static void main(String[] args) {
        System.out.println("Start generate");
        long start = System.currentTimeMillis();
        ThreadPoolExecutor threadPoolExecutor = new ThreadPoolExecutor(8, 8,
                0L, TimeUnit.MILLISECONDS,
                new LinkedBlockingQueue<Runnable>());
        List<ParallelCsvGenerate> taskList = new ArrayList<>(3800000);
        for(int i = 0; i < 3800000; i++) {
            taskList.add(new ParallelCsvGenerate(i+""));
        }
        try {
            List<Future<Integer>> futures = threadPoolExecutor.invokeAll(taskList);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        System.out.println("Success");
        long end = System.currentTimeMillis();
        System.out.println("Using time: " + (end-start));
    }

1. You could write directly into the file (without allocating the whole file in one StringBuilder). (I think this is the biggest time+memory bottleneck here: builder.toString())

2. You could generate each file in parallel.

3. (little tweaks:) Omit the if's inside loop.

   if(i < headList.size() - 1) is not needed, when you do a more clever loop + 1 extra iteration.

   The i % 2 == 0 can be eliminated by a better iteration (i+=2) ..and more labor inside the loop (i -> int, i + 1 -> string)

4. If applicable prefer append(char) to append(String). (Better append(',') than append(",")!)

...

You can use Fork/Join framework (java 7 and above) to make your process in parallel and use multi core of your Cpu.
I'll take an example for you.

import java.util.concurrent.ForkJoinPool;
import java.util.concurrent.ForkJoinTask;
import java.util.concurrent.RecursiveTask;
import java.util.stream.LongStream;

public class ForkJoinAdd extends RecursiveTask<Long> {

    private final long[] numbers;
    private final int start;
    private final int end;
    public static final long threshold = 10_000;

    public ForkJoinAdd(long[] numbers) {
        this(numbers, 0, numbers.length);
    }

    private ForkJoinAdd(long[] numbers, int start, int end) {
        this.numbers = numbers;
        this.start = start;
        this.end = end;
    }

    @Override
    protected Long compute() {

        int length = end - start;
        if (length <= threshold) {
            return add();
        }

        ForkJoinAdd firstTask = new ForkJoinAdd(numbers, start, start + length / 2);
        firstTask.fork(); //start asynchronously

        ForkJoinAdd secondTask = new ForkJoinAdd(numbers, start + length / 2, end);

        Long secondTaskResult = secondTask.compute();
        Long firstTaskResult = firstTask.join();

        return firstTaskResult + secondTaskResult;

    }

    private long add() {
        long result = 0;
        for (int i = start; i < end; i++) {
            result += numbers[i];
        }
        return result;
    }

    public static long startForkJoinSum(long n) {
        long[] numbers = LongStream.rangeClosed(1, n).toArray();
        ForkJoinTask<Long> task = new ForkJoinAdd(numbers);
        return new ForkJoinPool().invoke(task);
    }

}

use this example
And if you want to read more about it, Guide to the Fork/Join Framework in Java | Baeldung
and Fork/Join (The Java™ Tutorials
can help you to better understand and better design your app.
be lucky.

• Remove the for(int i = 0; i < 1000000; i++) loop from run method (leave a single generateCSV() call.
• Create 5 million ParallelCsvGenerate objects.
• Submit them to a ThreadPoolExecutor

Converted main:

public static void main(String[] args) {	
	ThreadPoolExecutor ex = (ThreadPoolExecutor) Executors.newFixedThreadPool(8);
	for(int i = 0; i < 5000000; i++) {
        ParallelCsvGenerate generate = new ParallelCsvGenerate();
    	ex.submit(generate);
	}
	ex.shutdown();
}

It takes roughly 5 minutes to complete on my laptop (4 physical cores with hyperthreading, SSD drive).

EDIT:

I've replaced FileWriter with AsynchronousFileChannel using the following code:

    Path file = Paths.get(filePath + fileName + ".csv");
    try(AsynchronousFileChannel asyncFile = AsynchronousFileChannel.open(file,
                        StandardOpenOption.WRITE,
                        StandardOpenOption.CREATE)) {

        asyncFile.write(ByteBuffer.wrap(builder.toString().getBytes()), 0);
    } catch (IOException e) {
        e.printStackTrace();
    }

to achieve 30% speedup.

I believe that the main bottleneck is the hard drive and filesystem itself. Not much more can be achieved here.