Google service account authorization error

I'm trying to use Google APIs with a service account.
Originally I used a normal project account with credentials and I managed to query the APIs. To use a service account, I modified the code to skip the browser auth flow and only use the service account credentials. It seems that the same request that used to work with the normal project credentials now give an error without clarifying much in the message. It goes as follows:

directoryService = new GoogleService().buildDirectoryService();

Users result = directoryService.users().list()
        .setCustomer("my_customer")
        .setOrderBy("email")
        .execute();

public Directory buildDirectoryService() throws GeneralSecurityException, IOException {
        final NetHttpTransport HTTP_TRANSPORT = GoogleNetHttpTransport.newTrustedTransport();

        GoogleCredential credential = GoogleCredential
                .fromStream(new FileInputStream("src/main/resources/newest_service_account_creds.json"))
                .createScoped(SCOPES);

        return new Directory.Builder(HTTP_TRANSPORT, JSON_FACTORY, credential)
                .setApplicationName(APPLICATION_NAME)
                .setHttpRequestInitializer(credential).build();
    }

Unfortunately the error message is not saying much:

Exception in thread "main" com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
{
  "code" : 400,
  "errors" : [ {
    "domain" : "global",
    "message" : "Invalid Input",
    "reason" : "invalid"
  } ],
  "message" : "Invalid Input"
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1108)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at GetDrivePermissions.getFiles(GetDrivePermissions.java:35)
at Main.main(Main.java:7)

I repeated the process two times to make sure I didn't miss anything, including creating a new project, enabling the APIs, adding a service account, adding and enabling domain-wide delegation and obtaining a credentials json file, also adding access to the app in App Access Control in the admin UI. But calling

directoryService.users().list().setCustomer("my_customer").setOrderBy("email").execute();

service still returns the above error message.

The error message refers to "Invalid input", but the request used to work before, with normal application credentials. So the problem must lay not in how I call the Directory service but with my authentication, the only difference between the working and the non-working version.

Since this worked with using the project credentials, there must be something wrong with how I use the service account, but I can't find the error even after repeating the whole process two times. Based on the guides I found about service accounts, this is supposed to work. The credential object is created and it looks proper.

What can I look into next?

Thnaks for any insight in advance.

I changed this part back to use the application's own credentials. I have another call that uses the service account credentials and that works, so this is a workaround for now.