3.2.1. Startup Issues¶
Virtual Accelerator cannot start¶
- Symptoms
systemctl status virtual-accelerator
shows issues
- Hints
On Intel and Arm, check whether the configuration file is correct by looking at
fast-path.sh config
output for relevancy, and by checking config file syntactic correctness withfast-path.sh config -c
. Follow the advice regarding deprecated options as it may become problematic in later versions. Take into account the WARNINGs in the output.Refer to the Fast Path Baseline for further details on the wizard (
fast-path.sh config
).If you tried running the fast path and it crashed or failed along the way, some “runtime-only” files may be left unremoved. Make sure to call
fast-path.sh stop
before trying to start the fast path again.Refer to the Fast Path Baseline documentation for further details on
fast-path.sh
commands.Look for error messages either on the console or in the logs. See rsyslog and journalctl sections for details regarding what can be found in the logs.
Executable paths may change between two Virtual Accelerator versions. Some shells (bash for example) keep a cache of the executable paths. After upgrading Virtual Accelerator, if some commands are not found, you may need to start a new shell.
Hugepages fragmentation¶
- Symptoms
One of the following messages appears on he console or in the logs:
No more huge pages left for fastpath initialization EAL: Not enough memory available! Requested: <X>MB, available: <Y smaller than X>MB PANIC in rte_eal_init(): Cannot init memory EAL: rte_eal_common_log_init(): cannot create log_history mempool PANIC in rte_eal_init(): Cannot init logs Not enough physically contiguous memory to allocate the mbuf pool on this socket (0): max_seg_size=178257920, total_mem=459276288, nb_seg=35 Increase the number of huge pages, use larger huge pages, or reboot the machine PANIC in fpn_socket_mbufpool_create(): Cannot create mbuf pool for socket 0
- Hints
There is a problem with the available memory.
Add more memory.
Check the output from
/proc/meminfo
, especially theMemFree
andHugePage_Free
fields. See meminfo section for details.MemFree
gives an indication of how much memory you may use for the fast path shared memory.
HugePage_Free
indicates how many huge pages are available for use by the fast path.
Beware, if hugepages are fragmented, you need to allocate more or simply reboot, as the DPDK requires contiguous physical memory.
Check the output from
fast-path.sh config --dump --full
, optionally with the--long
option.FP_MEMORY
might need adjustment to reserve more hugepages, which reduces the risk of fragmentation. Refer to the Fast Path Baseline for further details on the wizard (fast-path.sh config
).If you tried running the fast path and it crashed or failed along the way, some “runtime-only” files may be left unremoved. Make sure to call
fast-path.sh stop
before trying to start the fast path again. Refer to the Fast Path Baseline documentation for further details onfast-path.sh
commands.
Not enough memory¶
- Symptoms
The following message appears on he console or in the logs (and subsequent commands fail with similar messages):
/usr/bin/fast-path.sh: 435: /usr/bin/fast-path.sh: Cannot fork /usr/bin/fast-path.sh: 668: /usr/bin/fast-path.sh: Cannot fork
The following message appears on he console or in the logs:
... EAL: PCI memory mapped at 0x7ffae4a40000 PMD: eth_em_dev_init(): port_id 2 vendorID=0x8086 deviceID=0x100e Using fpn_port 0x7ffae654c000 size=150576 (0M) Killed //usr/bin/fast-path.sh: error starting //usr/bin//fp-rte. Check logs for details.
At this point, the machine may have hung. Check the logs after reboot, especially if they contain something similar to:
... fp-rte[5113]: Using fp_ebtables_vr_shared=0x7ffae63c2000 size=4352 (0M) fp-rte[5113]: Using fp-tc-shared=0x7ffad976f000 size=524608 (0M) kernel: [ 1022.485264] fp-rte invoked oom-killer: gfp_mask=0x2d2, order=0, oom_score_adj=0 kernel: [ 1022.485271] fp-rte cpuset=/ mems_allowed=0
Note
Look for error messages either on the console or in the logs. See rsyslog and journalctl sections for details regarding what can be found in the logs.
- Hints
There is a problem with the available memory, the fast path process has been killed because available memory was getting too small. Typically, after hugepages allocation, the fast path tried to allocate memory and there was not enough free.
Add more memory.
Check the output from
/proc/meminfo
, especially theMemFree
field. See meminfo section for details.MemFree
estimates how much memory is free before starting the fast path.
Check the output from
fast-path.sh config --dump --full
, optionally with the--long
option.FP_MEMORY
andVM_MEMORY
might need adjustment to reserve less hugepages. Refer to the Fast Path Baseline for further details on the wizard (fast-path.sh config
).
1G hugepages problems¶
- Symptoms
The following message appears on he console or in the logs:
sh: echo: I/O error WARNING: Can not allocate 1 hugepages for fast path 0 pages of size 1024 MB were allocated
- Hints
It seems you enabled the support of 1G hugepages in the kernel boot command line (
hugepagesz=1G default_hugepagesz=1G
). The fast path starting script failed to allocate the required amount of hugepages.Ensure that enough hugepages are reserved at boot time. Check the output from
fast-path.sh config --dump --full
, optionally with the--long
option, to get the value ofFP_MEMORY
representing the per-socket amount of memory needed by the fast path, and reserve the hugepages accordingly.
Invalid core-port mapping¶
- Symptoms
The following message appears on he console or in the logs:
WARNING - skipping invalid cpus [X, Y, Z, ...] in CORE_PORT_MAPPING
- Hints
On Intel and Arm, check whether the configuration file is correct by looking at
fast-path.sh config
output for relevancy, and by checking config file syntactic correctness withfast-path.sh config -c
. Follow the advice regarding deprecated options as it may become problematic in later versions. Take into account the WARNINGs in the output. Refer to the Fast Path Baseline for further details on the wizard (fast-path.sh config
).Check the output from
fast-path.sh config --dump --full
, optionally with the--long
option. Ensure consistency between the FP_MASK and CORE_PORT_MAPPING options. The cores used for polling may not be among the ones defined as the fast path core mask. As soon as the core/port mapping is manual, the fast path core mask should be manual too.
fast path service fails to start¶
Note
This Problem only appears on Intel CPUs
- Symptoms
The fast path service fails to start with the following error shown in the logs:
cat /var/log/fast-path.log Starting Fast Path... yes: standard output: Broken pipe yes: write error Traceback (most recent call last): File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/bin/fp_conf_tools/__init__.py", line 11, in <module> cli.main() File "/usr/bin/fp_conf_tools/cli.py", line 1809, in main machine.scan_machine(m_desc, args.input_machine_desc) File "/usr/bin/fp_conf_tools/machine.py", line 998, in scan_machine machine.read_pci_info() File "/usr/bin/fp_conf_tools/machine.py", line 567, in read_pci_info node = self._get_numa_node(bus) File "/usr/bin/fp_conf_tools/machine.py", line 371, in _get_numa_node return int(open('/sys/bus/pci/devices/%s/numa_node' % bus).read()) IOError: [Errno 2] No such file or directory: '/sys/bus/pci/devices/0000:00:01.0
- Hints
Intel VMD domain is longer than the 32-bit length of the linux design, it must be disabled from the bios.