By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Amazon Linux 2 (kernel 5.10.XXX): BTF info / kernel config

0

Hi,

I have a question regarding kernel configuration and the btf information for kernel symbols on Amazon Linux 2. What I have noticed is for an AL2 and kernel version < 5.10.155-138.670.amzn2, the vmlinux file is much smaller in size and have missing BTF information.

  1. How is vmlinux file build on kernel < 5.10.155-138.670.amzn2 such that BTF information for some symbols are missing. Are the symbols being stripped or something else?
  2. What exactly changed in 5.10.157-139.675.amzn2, that btf information (/sys/kernel/btf/vmlinux) file increased by ~1MB in size and started showing additional BTF information for symbols. Please refer to example below:
  • For all kernel < 5.10.155-138.670.amzn2, (symbol do_renameat2 is missing BTF info)
# uname -r
5.10.155-138.670.amzn2.x86_64
# bpftool btf dump file /sys/kernel/btf/vmlinux | egrep 'do_renameat2'
# 
#
#cat /proc/kallsyms | egrep do_renameat2
ffffffff812fbc80 t do_renameat2
#
# ls -l /sys/kernel/btf/vmlinux
-r--r--r-- 1 root root 2968308 Jun 15 11:43 /sys/kernel/btf/vmlinux
  • For all kernel >= 5.10.157-139.675.amzn2 (BTF info exists for symbol do_renameat2)
# uname -r
5.10.157-139.675.amzn2.x86_64
#
# bpftool btf dump file /sys/kernel/btf/vmlinux | egrep 'do_renameat2'
[37323] FUNC 'do_renameat2' type_id=37322 linkage=static
#
# cat /proc/kallsyms | grep do_renameat2
ffffffffb6315a40 t do_renameat2

# ls -l /sys/kernel/btf/vmlinux
-r--r--r-- 1 root root 3976249 Jun 15 11:48 /sys/kernel/btf/vmlinux
Topics
Compute
Tags
Amazon EC2Amazon Linux
Language
English
rePost-User-rshah
asked a year ago520 views
2 Answers
1
Accepted Answer

Hello, thank you for your post.

You can find the detailed commits log for the Amazon Linux kernel 5.10 branch in the public Github:

Note the build dates for the kernel versions in question:

# rpm -q --changelog kernel-5.10.155-138.670.amzn2 |head -1
* Wed Nov 23 2022 Builder <builder@amazon.com>

# rpm -q --changelog kernel-5.10.157-139.675.amzn2.x86_64 |head -1
* Thu Dec 08 2022 Builder <builder@amazon.com>

It is not clear if one of the commits between November 25 to December 8, 2022, was responsible for fixing the issue, or if it was the kernel build process. It is possible that the following commit for the btf_encoder may have fixed the issue:

https://github.com/acmel/dwarves/commit/58a98f76ac95b1bb11920ff2b58206b2364e6b3b

However, I was unable to determine with any certainty which version of the btf_encoder was involved in building the kernel.

If you continue to notice any issues with BTF info when using the latest kernel versions, please feel free to open a support case so we can review further.

AWS
SUPPORT ENGINEER
AWS_SamM
answered 10 months ago
0

Thanks SamM for the pointers. It seems it is indeed the kernel build process. I was able to reproduce with older dwarves which is responsible for creating the BTF info. I downloaded 5.10.29 source code, build the kernel with dwarves-1.17 and dwarves-1.22.

dwarves-1.17 did not generate the BTF info for "do_rename2" and dwarves-1.22 did generate. This tells me most likely the AL2 kernels <= 5.10.155 were built without the commit you pointed.

rePost-User-rshah
answered 10 months ago
  • AWS_SamM SUPPORT ENGINEER
    10 months ago

    Hello rshah, you are welcome. I am glad to see you took it a step further and verified the differences between building the kernel with dwarves-1.17 versus dwarves-1.22. Thank you for sharing your findings.

    For additional context, I want to share a 3rd party blog post that investigates a similar issue in great depth, albeit this post focuses on a different function for which BTF info was missing in arm64 kernels: https://rhysre.net/how-an-obscure-arm64-link-option-broke-our-bpf-probe.html

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content