# Server settings for a LXD production setup To allow your LXD server to run a large number of instances, configure the following settings to avoid hitting server limits. The `Value` column contains the suggested value for each parameter. ## `/etc/security/limits.conf` #### NOTE For users of the snap, those limits are automatically raised. | Domain | Type | Item | Value | Default | Description | |----------|--------|-----------|-------------|-----------|---------------------------------------------------------------------------------------| | `*` | soft | `nofile` | `1048576` | unset | Maximum number of open files | | `*` | hard | `nofile` | `1048576` | unset | Maximum number of open files | | `root` | soft | `nofile` | `1048576` | unset | Maximum number of open files | | `root` | hard | `nofile` | `1048576` | unset | Maximum number of open files | | `*` | soft | `memlock` | `unlimited` | unset | Maximum locked-in-memory address space (KB) | | `*` | hard | `memlock` | `unlimited` | unset | Maximum locked-in-memory address space (KB) | | `root` | soft | `memlock` | `unlimited` | unset | Maximum locked-in-memory address space (KB), only need with `bpf` syscall supervision | | `root` | hard | `memlock` | `unlimited` | unset | Maximum locked-in-memory address space (KB), only need with `bpf` syscall supervision | ## `/etc/sysctl.conf` #### NOTE Reboot the server after changing any of these parameters. `fs.aio-max-nr` Maximum number of concurrent asynchronous I/O operations [](#sysctl:fs.aio-max-nr) | **Key:** | `fs.aio-max-nr` | |--------------|-------------------| | **Type:** | integer | | **Default:** | `65536` | Suggested value: `524288` You might need to increase this limit further if you have a lot of workloads that use the AIO subsystem (for example, MySQL). `fs.inotify.max_queued_events` Upper limit on the number of events that can be queued [](#sysctl:fs.inotify.max_queued_events) | **Key:** | `fs.inotify.max_queued_events` | |--------------|----------------------------------| | **Type:** | integer | | **Default:** | `16384` | Suggested value: `1048576` This option specifies the maximum number of events that can be queued to the corresponding `inotify` instance (see [`inotify`](https://man7.org/linux/man-pages/man7/inotify.7.html) for more information). `fs.inotify.max_user_instances` Upper limit on the number of `inotify` instances [](#sysctl:fs.inotify.max_user_instances) | **Key:** | `fs.inotify.max_user_instances` | |--------------|-----------------------------------| | **Type:** | integer | | **Default:** | `128` | Suggested value: `1048576` This option specifies the maximum number of `inotify` instances that can be created per real user ID (see [`inotify`](https://man7.org/linux/man-pages/man7/inotify.7.html) for more information). `fs.inotify.max_user_watches` Upper limit on the number of watches [](#sysctl:fs.inotify.max_user_watches) | **Key:** | `fs.inotify.max_user_watches` | |--------------|---------------------------------| | **Type:** | integer | | **Default:** | `8192` | Suggested value: `1048576` This option specifies the maximum number of watches that can be created per real user ID (see [`inotify`](https://man7.org/linux/man-pages/man7/inotify.7.html) for more information). `kernel.dmesg_restrict` Whether to deny access to the messages in the kernel ring buffer [](#sysctl:kernel.dmesg_restrict) | **Key:** | `kernel.dmesg_restrict` | |--------------|---------------------------| | **Type:** | integer | | **Default:** | `0` | Suggested value: `1` Set this option to `1` to deny container access to the messages in the kernel ring buffer. Note that setting this value to `1` will also deny access to non-root users on the host system. `kernel.keys.maxbytes` Maximum size of the key ring that non-root users can use [](#sysctl:kernel.keys.maxbytes) | **Key:** | `kernel.keys.maxbytes` | |--------------|--------------------------| | **Type:** | integer | | **Default:** | `20000` | Suggested value: `2000000` `kernel.keys.maxkeys` Maximum number of keys that a non-root user can use [](#sysctl:kernel.keys.maxkeys) | **Key:** | `kernel.keys.maxkeys` | |--------------|-------------------------| | **Type:** | integer | | **Default:** | `200` | Suggested value: `2000` Set this option to a value that is higher than the number of instances. `net.core.bpf_jit_limit` Limit on the size of eBPF JIT allocations [](#sysctl:net.core.bpf_jit_limit) | **Key:** | `net.core.bpf_jit_limit` | |--------------|----------------------------| | **Type:** | integer | | **Default:** | varies | Suggested value: `1000000000` On kernels < 5.15 that are compiled with `CONFIG_BPF_JIT_ALWAYS_ON=y`, this value might limit the amount of instances that can be created. `net.ipv4.neigh.default.gc_thresh3` Maximum number of entries in the IPv4 ARP table [](#sysctl:net.ipv4.neigh.default.gc_thresh3) | **Key:** | `net.ipv4.neigh.default.gc_thresh3` | |--------------|---------------------------------------| | **Type:** | integer | | **Default:** | `1024` | Suggested value: `8192` Increase this value if you plan to create over 1024 instances. Otherwise, you will get the error `neighbour: ndisc_cache: neighbor table overflow!` when the ARP table gets full and the instances cannot get a network configuration. See [`ip-sysctl`](https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt) for more information. `net.ipv6.neigh.default.gc_thresh3` Maximum number of entries in IPv6 ARP table [](#sysctl:net.ipv6.neigh.default.gc_thresh3) | **Key:** | `net.ipv6.neigh.default.gc_thresh3` | |--------------|---------------------------------------| | **Type:** | integer | | **Default:** | `1024` | Suggested value: `8192` Increase this value if you plan to create over 1024 instances. Otherwise, you will get the error `neighbour: ndisc_cache: neighbor table overflow!` when the ARP table gets full and the instances cannot get a network configuration. See [`ip-sysctl`](https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt) for more information. `vm.max_map_count` Maximum number of memory map areas a process may have [](#sysctl:vm.max_map_count) | **Key:** | `vm.max_map_count` | |--------------|----------------------| | **Type:** | integer | | **Default:** | `65530` | Suggested value: `262144` Memory map areas are used as a side-effect of calling `malloc`, directly by `mmap` and `mprotect`, and also when loading shared libraries. ## Related topics How-to guides: - [How to benchmark performance](https://canonical.com/lxd/docs/default/howto/benchmark_performance/index.html.md#benchmark-performance) - [How to increase the network bandwidth](https://canonical.com/lxd/docs/default/howto/network_increase_bandwidth/index.html.md#network-increase-bandwidth) - [How to monitor metrics](https://canonical.com/lxd/docs/default/metrics/index.html.md#metrics) Explanation: - [Performance tuning](https://canonical.com/lxd/docs/default/explanation/performance_tuning/index.html.md#performance-tuning)