JPanels horizontally not lining up when trying to use FlowLayout.LEFT on the components

So my GUI, I have 1 JPanel in the red box called Connection that has 2 other panels north and south. Im using FlowLayout and trying to set the Host and Port label to line up on the left side but they are off by a bit and cant seem figure out why.

public class ClientView extends JPanel {

private JComboBox<String> comboBox;
private JButton connect;
private JTextArea display;
private JTextArea text;
private JTextField host;
private JTextField text1;
private JButton send;

// Constructor for all the components of the GUI for the Client application
public ClientView() {
    // Sets the layout for the Client application and the border
    this.setLayout(new BorderLayout());
    this.setBorder(BorderFactory.createEmptyBorder(5, 5, 5, 5));
    JPanel panel = new JPanel(new FlowLayout(FlowLayout.LEFT));
    JPanel connectionPanel = new JPanel(new FlowLayout(FlowLayout.LEFT));
    JPanel clientRequestPanel = new JPanel(new FlowLayout(FlowLayout.LEFT));
    JPanel subDisplayPanel = new JPanel(new BorderLayout());
    JPanel topPanel = new JPanel(new BorderLayout());
    JScrollPane displayPanel;
    JLabel hostLabel = new JLabel("Host: ");
    JLabel portLabel = new JLabel("Port: ");
    String ports[] = new String[]{"", "8088", "65000", "65535"};
    // Makes a ComboBox and gives its requirments. Makes it the color white, editable and size
    comboBox = new JComboBox<String>(ports);
    comboBox.setBackground(Color.WHITE);
    comboBox.setEditable(true);
    
    comboBox.setMaximumRowCount(4);
    comboBox.setPreferredSize(new Dimension(110, 25));
    // Makes the Connect Buttons and gives it the color red.
    connect = new JButton("Connect");
    connect.setBackground(Color.RED);
    connect.setPreferredSize(new Dimension(110, 25));
    // Sets the host labels required dimensions of 40 and 25. Then the JTextField host with its text being localhost
    // Also makes it White, editable, size and its margin
    hostLabel.setPreferredSize(new Dimension(40, 25));
    host = new JTextField("localhost");
    host.setBackground(Color.WHITE);
    host.setEditable(true);
    host.setPreferredSize(new Dimension(490, 25));
    host.setMargin(new Insets(0, 5, 0, 0));
    // Sets the port labels required dimensions of 40 and 25
    portLabel.setPreferredSize(new Dimension(40, 25));
    // Creates the send button makes it uneditable and its dimension
    send = new JButton("Send");
    send.setEnabled(false);
    send.setPreferredSize(new Dimension(80, 25));
    // Sets the mnemonic for the letter H for the host label
    hostLabel.setDisplayedMnemonic('H');
    // Aligns the hostLabel to the left
    hostLabel.setAlignmentX(Component.LEFT_ALIGNMENT);
    // For when ALT-Key is pressed that it focus is transfered
    hostLabel.setLabelFor(host);
    // Sets the mnemonic for the letter P for the port label
    portLabel.setDisplayedMnemonic('P');
    // Aligns the portLabel to the left
    portLabel.setAlignmentX(Component.LEFT_ALIGNMENT);
    // For when ALT-Key is pressed that it focus is transfered
    portLabel.setLabelFor(comboBox);