3.2.1. Startup Issues

Virtual Accelerator cannot start

Symptoms

  • systemctl status virtual-accelerator shows issues

Hints

  • On Intel and Arm, check whether the configuration file is correct by looking at fast-path.sh config output for relevancy, and by checking config file syntactic correctness with fast-path.sh config -c. Follow the advice regarding deprecated options as it may become problematic in later versions. Take into account the WARNINGs in the output.

    Refer to the Fast Path Baseline for further details on the wizard (fast-path.sh config).

  • If you tried running the fast path and it crashed or failed along the way, some “runtime-only” files may be left unremoved. Make sure to call fast-path.sh stop before trying to start the fast path again.

    Refer to the Fast Path Baseline documentation for further details on fast-path.sh commands.

  • Look for error messages either on the console or in the logs. See rsyslog and journalctl sections for details regarding what can be found in the logs.

  • Executable paths may change between two Virtual Accelerator versions. Some shells (bash for example) keep a cache of the executable paths. After upgrading Virtual Accelerator, if some commands are not found, you may need to start a new shell.

Hugepages fragmentation

Symptoms

  • One of the following messages appears on he console or in the logs:

    No more huge pages left for fastpath initialization

EAL: Not enough memory available! Requested: <X>MB, available: <Y smaller than X>MB
PANIC in rte_eal_init(): Cannot init memory

EAL: rte_eal_common_log_init(): cannot create log_history mempool
PANIC in rte_eal_init():
Cannot init logs

Not enough physically contiguous memory to allocate the mbuf pool on this socket (0): max_seg_size=178257920, total_mem=459276288, nb_seg=35
Increase the number of huge pages, use larger huge pages, or reboot the machine
PANIC in fpn_socket_mbufpool_create():
Cannot create mbuf pool for socket 0
Hints

  • There is a problem with the available memory.

  • Add more memory.

  • Check the output from /proc/meminfo, especially the MemFree and HugePage_Free fields. See meminfo section for details.

    MemFree

    gives an indication of how much memory you may use for the fast path shared memory.

    HugePage_Free

    indicates how many huge pages are available for use by the fast path.

    Beware, if hugepages are fragmented, you need to allocate more or simply reboot, as the DPDK requires contiguous physical memory.

  • Check the output from fast-path.sh config --dump --full, optionally with the --long option. FP_MEMORY might need adjustment to reserve more hugepages, which reduces the risk of fragmentation. Refer to the Fast Path Baseline for further details on the wizard (fast-path.sh config).

  • If you tried running the fast path and it crashed or failed along the way, some “runtime-only” files may be left unremoved. Make sure to call fast-path.sh stop before trying to start the fast path again. Refer to the Fast Path Baseline documentation for further details on fast-path.sh commands.

Not enough memory

Symptoms

  • The following message appears on he console or in the logs (and subsequent commands fail with similar messages):

    /usr/bin/fast-path.sh: 435: /usr/bin/fast-path.sh: Cannot fork
/usr/bin/fast-path.sh: 668: /usr/bin/fast-path.sh: Cannot fork

  • The following message appears on he console or in the logs:

    ...
EAL:   PCI memory mapped at 0x7ffae4a40000
PMD: eth_em_dev_init(): port_id 2 vendorID=0x8086 deviceID=0x100e
Using fpn_port 0x7ffae654c000 size=150576 (0M)
Killed
//usr/bin/fast-path.sh: error starting //usr/bin//fp-rte. Check logs for details.

    At this point, the machine may have hung. Check the logs after reboot, especially if they contain something similar to:

    ...
fp-rte[5113]: Using fp_ebtables_vr_shared=0x7ffae63c2000 size=4352 (0M)
fp-rte[5113]: Using fp-tc-shared=0x7ffad976f000 size=524608 (0M)
kernel: [ 1022.485264] fp-rte invoked oom-killer: gfp_mask=0x2d2, order=0, oom_score_adj=0
kernel: [ 1022.485271] fp-rte cpuset=/ mems_allowed=0

    Note

    Look for error messages either on the console or in the logs. See rsyslog and journalctl sections for details regarding what can be found in the logs.

Hints

  • There is a problem with the available memory, the fast path process has been killed because available memory was getting too small. Typically, after hugepages allocation, the fast path tried to allocate memory and there was not enough free.

  • Add more memory.

  • Check the output from /proc/meminfo, especially the MemFree field. See meminfo section for details.

    MemFree

    estimates how much memory is free before starting the fast path.

  • Check the output from fast-path.sh config --dump --full, optionally with the --long option. FP_MEMORY and VM_MEMORY might need adjustment to reserve less hugepages. Refer to the Fast Path Baseline for further details on the wizard (fast-path.sh config).

1G hugepages problems

Symptoms

  • The following message appears on he console or in the logs:

    sh: echo: I/O error
WARNING: Can not allocate 1 hugepages for fast path
         0 pages of size 1024 MB were allocated
Hints

  • It seems you enabled the support of 1G hugepages in the kernel boot command line (hugepagesz=1G default_hugepagesz=1G). The fast path starting script failed to allocate the required amount of hugepages.

  • Ensure that enough hugepages are reserved at boot time. Check the output from fast-path.sh config --dump --full, optionally with the --long option, to get the value of FP_MEMORY representing the per-socket amount of memory needed by the fast path, and reserve the hugepages accordingly.

Invalid core-port mapping

Symptoms

  • The following message appears on he console or in the logs:

    WARNING - skipping invalid cpus [X, Y, Z, ...] in CORE_PORT_MAPPING
Hints

  • On Intel and Arm, check whether the configuration file is correct by looking at fast-path.sh config output for relevancy, and by checking config file syntactic correctness with fast-path.sh config -c. Follow the advice regarding deprecated options as it may become problematic in later versions. Take into account the WARNINGs in the output. Refer to the Fast Path Baseline for further details on the wizard (fast-path.sh config).

  • Check the output from fast-path.sh config --dump --full, optionally with the --long option. Ensure consistency between the FP_MASK and CORE_PORT_MAPPING options. The cores used for polling may not be among the ones defined as the fast path core mask. As soon as the core/port mapping is manual, the fast path core mask should be manual too.