3.2.1. Startup Issues¶

Virtual Accelerator cannot start¶

Symptoms

systemctl status virtual-accelerator shows issues

Hints

On Intel and Arm, check whether the configuration file is correct by looking at fast-path.sh config output for relevancy, and by checking config file syntactic correctness with fast-path.sh config -c. Follow the advice regarding deprecated options as it may become problematic in later versions. Take into account the WARNINGs in the output.

Refer to the Fast Path Baseline for further details on the wizard (fast-path.sh config).
If you tried running the fast path and it crashed or failed along the way, some “runtime-only” files may be left unremoved. Make sure to call fast-path.sh stop before trying to start the fast path again.

Refer to the Fast Path Baseline documentation for further details on fast-path.sh commands.
Look for error messages either on the console or in the logs. See rsyslog and journalctl sections for details regarding what can be found in the logs.
Executable paths may change between two Virtual Accelerator versions. Some shells (bash for example) keep a cache of the executable paths. After upgrading Virtual Accelerator, if some commands are not found, you may need to start a new shell.

Hugepages fragmentation¶

Symptoms

One of the following messages appears on he console or in the logs:

No more huge pages left for fastpath initialization

EAL: Not enough memory available! Requested: <X>MB, available: <Y smaller than X>MB
PANIC in rte_eal_init(): Cannot init memory

EAL: rte_eal_common_log_init(): cannot create log_history mempool
PANIC in rte_eal_init():
Cannot init logs

Not enough physically contiguous memory to allocate the mbuf pool on this socket (0): max_seg_size=178257920, total_mem=459276288, nb_seg=35
Increase the number of huge pages, use larger huge pages, or reboot the machine
PANIC in fpn_socket_mbufpool_create():
Cannot create mbuf pool for socket 0

Hints

There is a problem with the available memory.
Add more memory.
Check the output from /proc/meminfo, especially the MemFree and HugePage_Free fields. See meminfo section for details.

MemFree
gives an indication of how much memory you may use for the fast path shared memory.

HugePage_Free
indicates how many huge pages are available for use by the fast path.

Beware, if hugepages are fragmented, you need to allocate more or simply reboot, as the DPDK requires contiguous physical memory.
Check the output from fast-path.sh config --dump --full, optionally with the --long option. FP_MEMORY might need adjustment to reserve more hugepages, which reduces the risk of fragmentation. Refer to the Fast Path Baseline for further details on the wizard (fast-path.sh config).
If you tried running the fast path and it crashed or failed along the way, some “runtime-only” files may be left unremoved. Make sure to call fast-path.sh stop before trying to start the fast path again. Refer to the Fast Path Baseline documentation for further details on fast-path.sh commands.

Not enough memory¶

Symptoms

The following message appears on he console or in the logs (and subsequent commands fail with similar messages):

/usr/bin/fast-path.sh: 435: /usr/bin/fast-path.sh: Cannot fork
/usr/bin/fast-path.sh: 668: /usr/bin/fast-path.sh: Cannot fork

The following message appears on he console or in the logs:

...
EAL:   PCI memory mapped at 0x7ffae4a40000
PMD: eth_em_dev_init(): port_id 2 vendorID=0x8086 deviceID=0x100e
Using fpn_port 0x7ffae654c000 size=150576 (0M)
Killed
//usr/bin/fast-path.sh: error starting //usr/bin//fp-rte. Check logs for details.

At this point, the machine may have hung. Check the logs after reboot, especially if they contain something similar to:

...
fp-rte[5113]: Using fp_ebtables_vr_shared=0x7ffae63c2000 size=4352 (0M)
fp-rte[5113]: Using fp-tc-shared=0x7ffad976f000 size=524608 (0M)
kernel: [ 1022.485264] fp-rte invoked oom-killer: gfp_mask=0x2d2, order=0, oom_score_adj=0
kernel: [ 1022.485271] fp-rte cpuset=/ mems_allowed=0

Note

Look for error messages either on the console or in the logs. See rsyslog and journalctl sections for details regarding what can be found in the logs.

Hints

There is a problem with the available memory, the fast path process has been killed because available memory was getting too small. Typically, after hugepages allocation, the fast path tried to allocate memory and there was not enough free.
Add more memory.
Check the output from /proc/meminfo, especially the MemFree field. See meminfo section for details.

MemFree
estimates how much memory is free before starting the fast path.
Check the output from fast-path.sh config --dump --full, optionally with the --long option. FP_MEMORY and VM_MEMORY might need adjustment to reserve less hugepages. Refer to the Fast Path Baseline for further details on the wizard (fast-path.sh config).

1G hugepages problems¶

Symptoms

The following message appears on he console or in the logs:

sh: echo: I/O error
WARNING: Can not allocate 1 hugepages for fast path
         0 pages of size 1024 MB were allocated

Hints

It seems you enabled the support of 1G hugepages in the kernel boot command line (hugepagesz=1G default_hugepagesz=1G). The fast path starting script failed to allocate the required amount of hugepages.
Ensure that enough hugepages are reserved at boot time. Check the output from fast-path.sh config --dump --full, optionally with the --long option, to get the value of FP_MEMORY representing the per-socket amount of memory needed by the fast path, and reserve the hugepages accordingly.

Invalid core-port mapping¶

Symptoms

The following message appears on he console or in the logs:

WARNING - skipping invalid cpus [X, Y, Z, ...] in CORE_PORT_MAPPING

Hints

On Intel and Arm, check whether the configuration file is correct by looking at fast-path.sh config output for relevancy, and by checking config file syntactic correctness with fast-path.sh config -c. Follow the advice regarding deprecated options as it may become problematic in later versions. Take into account the WARNINGs in the output. Refer to the Fast Path Baseline for further details on the wizard (fast-path.sh config).
Check the output from fast-path.sh config --dump --full, optionally with the --long option. Ensure consistency between the FP_MASK and CORE_PORT_MAPPING options. The cores used for polling may not be among the ones defined as the fast path core mask. As soon as the core/port mapping is manual, the fast path core mask should be manual too.

fast path service fails to start¶

Note

This Problem only appears on Intel CPUs

Symptoms

The fast path service fails to start with the following error shown in the logs:

cat /var/log/fast-path.log
Starting Fast Path...
yes: standard output: Broken pipe
yes: write error
Traceback (most recent call last):
      File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
         "__main__", fname, loader, pkg_name)
      File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
         exec code in run_globals
      File "/usr/bin/fp_conf_tools/__init__.py", line 11, in <module>
         cli.main()
      File "/usr/bin/fp_conf_tools/cli.py", line 1809, in main
         machine.scan_machine(m_desc, args.input_machine_desc)
      File "/usr/bin/fp_conf_tools/machine.py", line 998, in scan_machine
         machine.read_pci_info()
      File "/usr/bin/fp_conf_tools/machine.py", line 567, in read_pci_info
         node = self._get_numa_node(bus)
      File "/usr/bin/fp_conf_tools/machine.py", line 371, in _get_numa_node
         return int(open('/sys/bus/pci/devices/%s/numa_node' % bus).read())
      IOError: [Errno 2] No such file or directory: '/sys/bus/pci/devices/0000:00:01.0

Hints

Intel VMD domain is longer than the 32-bit length of the linux design, it must be disabled from the bios.