--- title: "LIEF v0.16.0" description: "LIEF 0.16.0: rebuilt documentation, an assembler/disassembler in LIEF Extended, dyld shared cache extraction, first mutable Rust APIs, and nanobind 2.4 bindings." canonical_url: "https://lief.re/blog/2024-12-10-lief-0-16-0/" markdown_url: "https://lief.re/blog/2024-12-10-lief-0-16-0/index.md" authors: ["Romain Thomas"] date_published: "2024-12-10T00:00:00Z" date_modified: "2024-12-10T00:00:00Z" language: "en-US" section: "blog" tags: ["release","LIEF Extended","assembler","disassembler","dyld-shared-cache","Rust","Python"] categories: [] --- # LIEF v0.16.0 > LIEF 0.16.0: rebuilt documentation, an assembler/disassembler in LIEF Extended, dyld shared cache extraction, first mutable Rust APIs, and nanobind 2.4 bindings. ## Documentation Documentation is an important aspect of LIEF and since the beginning of the project, I have spent a decent amount of time keeping comprehensive and intuitive documentation. Usually, I don't include documentation updates in the changelog but in this case, I thought it could be worth sharing this experience. LIEF is written in C++ with bindings for Python and Rust. Originally, the documentation was driven by languages API (isolated from each other) and generated by Sphinx with the [Breathe](https://breathe.readthedocs.io/en/latest/) extension to reference the C++ Doxygen domain. Recently, Rust landed in the arena. Compared to Python and C++, the Rust language embeds a built-in documentation engine to process and generate in-code documentation into HTML pages. Given the new Rust bindings and the Rust built-in documentation engine, two questions emerged: 1. Do we want to add (yet) another API page for Rust? 2. How do we reference Rust API in Sphinx? For the first point, I moved from a language-driven documentation structure to a functionality-driven structure. This changes how the documentation is consumed. Instead of looking for a language's format-specific API, you first choose the task, such as ELF processing or Dyld shared cache parsing. You can **then** access the language API you need. So instead of adding another Rust API reference page, the Rust API has been transparently integrated with the new layout. ![Documentation layout changes](https://lief.re/blog/2024-12-10-lief-0-16-0/api-diff.webp) The second point has been a bit more tricky to approach. With a reverse engineering background, I really value the **cross-reference feature** provided by Sphinx: ```rst blah blah blah :py:class:`lief.ELF.Binary` another blah: :cpp:class:`LIEF::ELF::Binary` ``` Python is a built-in domain supported by Sphinx and [Breathe](https://breathe.readthedocs.io/en/latest/) extension is doing the bridge between C++ Doxygen XML files and Sphinx. For Rust, there are some attempts to create a bridge but I decided to take another path. I created a Rust-sphinx domain that cross-references to the official or nightly Rust documentation. Basically with this custom domain, the following cross-references redirect to the official or nightly documentation: ```rst :rust:module:`lief::assembly` :rust:enum:`lief::assembly::Instructions` ``` Is translated into: ```text https://lief-rs.s3.fr-par.scw.cloud/doc/latest/lief/assembly/index.html https://lief-rs.s3.fr-par.scw.cloud/doc/latest/lief/assembly/enum.Instructions.html ``` By doing so, we can leverage Sphinx's cross-reference functionalities while still keeping the built-in Rust documentation. In addition to this Rust-specific domain, I created a `.. lief-api::` directive that can pack similar cross-language API into a single block. For instance, this directive: ```rst .. lief-api:: lief.Binary.disassemble() :rust:method:`lief::generic::Binary::disassemble [trait]` :rust:method:`lief::generic::Binary::disassemble_symbol [trait]` :rust:method:`lief::generic::Binary::disassemble_address [trait]` :rust:method:`lief::generic::Binary::disassemble_slice [trait]` :cpp:func:`LIEF::Binary::disassemble` :py:meth:`lief.Binary.disassemble` :py:meth:`lief.Binary.disassemble_from_bytes` ``` Is rendered as: ![Documentation layout changes](https://lief.re/blog/2024-12-10-lief-0-16-0/rust-domain.webp) This allows us to refer the API for different languages without being too verbose and impacting readability. Combined with Sphinx substitution, we can write: ``` This is an example that cross-reference |lief-disassemble| .. |lief-disassemble| lief-api:: lief.Binary.disassemble() :rust:method:`lief::generic::Binary::disassemble [trait]` :rust:method:`lief::generic::Binary::disassemble_symbol [trait]` :rust:method:`lief::generic::Binary::disassemble_address [trait]` :rust:method:`lief::generic::Binary::disassemble_slice [trait]` :cpp:func:`LIEF::Binary::disassemble` :py:meth:`lief.Binary.disassemble` :py:meth:`lief.Binary.disassemble_from_bytes` ``` You can go checking out this page [https://lief.re/doc/latest/formats/pe/index.html](https://lief.re/doc/latest/formats/pe/index.html) to see a concrete rendering of these changes. ## Extended Features **Public Release** The extended version is now publicly available at this address: [https://extended.lief.re](https://extended.lief.re) ## Assembler & Disassembler Adding (or not adding) a disassembler in LIEF has been a long-standing question and with the [extended](https://lief.re/doc/latest/extended/intro.html) version, I found a fair trade-off: ** LIEF core focuses on executable formats, free from any extra features that might have a significant impact on the build complexity or library size. ** On the other hand, LIEF extended provides additional functionalities that require a more complex build pipeline and increase the binary size. Among these extended functionalities, there are a [disassembler](https://lief.re/doc/latest/extended/disassembler/index.html) and an [assembler](https://lief.re/doc/latest/extended/assembler/index.html) based on the LLVM's MC layer. The disassembling API is provided at different levels: ### LIEF::Binary ```python import lief pe = lief.PE.parse("cmd.exe") for inst in pe.disassemble(0x400000): print(inst) # Instruction semantic print(inst.is_syscall) print(inst.is_memory_access) print(inst.is_call) # Instruction operands (for AArch64 and x86-64) if isinstance(inst, lief.assembly.aarch64.Instruction): for idx, operand in enumerate(inst.operands): match operand: case lief.assembly.aarch64.operands.Register(): print(f"OP[{idx}] -- REG: {operand.value}") case lief.assembly.aarch64.operands.Memory(): print(f"OP[{idx}] -- MEM: {operand.base} {operand.offset}") case lief.assembly.aarch64.operands.PCRelative(): print(f"OP[{idx}] -- PCR: {operand.value}") case lief.assembly.aarch64.operands.Immediate(): print(f"OP[{idx}] -- IMM: {operand.value}") ``` ### LIEF::dwarf::Function ```python import lief elf = lief.ELF.parse("my-dbg.elf") dwarf: lief.dwarf.DebugInfo = elf.debug_info func: lief.dwarf.Function = dwarf.find_function("my_debug_function") for inst in func.instructions: print(inst) ``` ### LIEF::dsc::DyldSharedCache ```python import lief cache = lief.dsc.load("ios-18/") for inst in cache.disassemble(0x1886f4a44): print(inst) ``` In terms of implementation, the disassembler wraps a lazy iterator that evaluates/disassembles an instruction **only** when the iterator is processed. It means that you don't pay any overhead until you access the iterator's value: ```text # O(0) inst = macho.disassemble(0x400000) inst = macho.disassemble(0x400000) # O(10) for _ in range(10): next(inst) ``` The `.end()` sentinel of the iterator is based on two properties: 1. Either a range is specified (e.g. `macho.disassemble(0x400000, /*size*/0x1000)`) and the iterator past the end of the range. 2. The instruction can't be disassembled. This kind of sentinel allows us to use this API: `macho.disassemble(0x400000)` which will disassemble (lazily) instructions at the address `0x400000` until it fails. **C++ & Rust & Python** The disassembler/assembler API is uniformly available in Rust, C++, and Python. ### Capstone? Nyxstone? [As stated in the documentation](https://lief.re/doc/latest/extended/disassembler/index.html#technical-details) the major design difference with [Capstone](https://www.capstone-engine.org/) is that LIEF uses a mainstream version of LLVM with limited patches[^llvm-patch] on the MC layer (the current version is based on LLVM `19.1.2`). The design difference with [Nyxstone](https://github.com/emproof-com/nyxstone) is that LLVM is hidden from the public API which means that it does not require to have an LLVM version pre-install on the system. Moreover, it exposes opcodes and control-flow/semantic information about the instructions. **On the other hand, LIEF does not provide a standalone API to disassemble arbitrary instructions. The disassembler engine is bound to the object from which the API is exposed.** ### Assembler In association with a disassembler, LIEF exposes a (basic) assembly API that allows generating **and patching** instructions: ```python import lief elf = lief.ELF.parse("my-android-obfuscated.so") text = elf.get_section(".text") # Disassembler syscall = [inst for inst in elf.disassemble(bytes(text)) if inst.is_syscall] # Assembler for syscall_inst in syscall: new_bytes = elf.assemble(syscall_inst.address, "nop;") # Assemble AND patch print(new_bytes.hex(", ")) ``` **Warning** In this current version, the assembler is working *pretty* well for x86/x86_64 and AArch64 but might break on other architectures. In addition, `llvm::MCFixup`** are not supported.** This can be used to patch LIEF's binary object directly at the assembly level. I plan to provide the assembly engine with LIEF Binary context. If the binary defines a function such as `call_me()` that is exported or present in the debug information, users would be able to call it at the assembly level: ```rust fn patch_with_context(macho: &mut lief::macho::Binary) { macho.assemble(0x140000090, r#" adrp x0, call_me; add x0, x0, :lo12:call_me; mov x1, 0x90; str x1, [x0]; "#r); } ``` And LIEF would handle the relocation/resolution process to instruct LLVM about the location and the definition of `call_me`. **C++ & Rust & Python** The disassembler/assembler API is seamlessly available in Rust, C++, and Python :) ## Dyld Shared Cache Initial support for processing Apple's Dyld shared cache with LIEF has been released along with an API to **deoptimize** in-cache Dylib. The API looks like this: ```python import lief cache = lief.dsc.load("ios-18.1/") for dylib in cache.libraries: print(f"0x{dylib.address:016x} {dylib.path}") # Extract the dylib as a regular lief.MachO.Binary macho: lief.MachO.Binary = dylib.get() ``` **Warning** Please note that the deoptimization feature is not working well on all the shared cache libraries. This support is going to be improved over time. One could also use this API to diff two shared caches: ```rust use lief; let ios_17 = lief::dsc::load_from_path("ios-17.7.1"); let ios_18 = lief::dsc::load_from_path("ios-18.1.1"); let libraries_17: HashSet = ios_17.libraries() .map(|lib| lib.path()) .collect(); let libraries_18: HashSet = ios_18.libraries() .map(|lib| lib.path()) .collect(); println!("{:?}", libraries_17.symmetric_difference(&libraries_18)) ``` ## Rust Rust bindings got their first **mutable** functions which are listed in the [changelog](https://lief.re/doc/latest/changelog.html). These mutable functions are limited but they allow us to make basic modifications like adding a library or patching assembly code: ```rust fn add_library(elf: &mut lief::elf::Binary) { elf.add_library("libtest.so"); elf.write("patched.elf"); } ``` ```rust fn patch_asm(elf: &mut lief::macho::Binary) { macho.assemble(0x100004090, r#" mov x0, x16; br x0; "#); macho.write("patched.macho"); } ``` In addition, the support for the `x86_64-unknown-linux-musl` target triple is now available and the minimal GLIBC version for `x86_64-unknown-linux-gnu` has been lowered to `2.28`. It means that Linux Rust bindings can now run on Debian 10, Ubuntu 19.10, ... while before it required Debian 11 or Ubuntu 20.04. The new `x86_64-unknown-linux-musl` triple can be used to generate **full static** without **any dependencies** to the `libstdc++, libc, ...`. For instance, given this code: ```rust use lief; use lief::generic::Section; fn main() { let path = std::env::args().last().unwrap(); let mut file = std::fs::File::open(path).expect("Can't open the file"); if let Some(lief::Binary::PE(pe)) = lief::Binary::from(&mut file) { for section in pe.sections() { println!( "{:20}: [0x{:016x}-0x{:016x}]", section.name(), section.virtual_address(), section.virtual_address() + section.virtual_size() as u64 ); } } } ``` We can generate a dependencies-free executable by running: ```console $ cargo build [--release] --target x86_64-unknown-linux-musl ``` ```console $ ldd target/x86_64-unknown-linux-musl/release/reader statically linked ``` ```console $ target/x86_64-unknown-linux-musl/release/reader steam.exe .text : [0x0000000000001000-0x00000000002cbe53] .rdata : [0x00000000002cc000-0x00000000003a7fa2] .data : [0x00000000003a8000-0x000000000043ada0] .rsrc : [0x000000000043b000-0x0000000000471b8c] .reloc : [0x0000000000472000-0x0000000000490c74] ``` ## Python Bindings LIEF is now using [nanobind](https://nanobind.readthedocs.io/en/latest/) v2.4.0 which improves the support for typing. Among these typing improvements, C++ enums flags are now properly inheriting from `enum.Flag` which results in a better interface with Python code. Typing stub files (`*.pyi`) are now also generated with nanobind's built-in `stubgen.py` instead for [mypy](https://github.com/python/mypy/blob/ac8957755a35a255f638c122e22c03b0e75b9a79/mypy/stubgen.py). ## Final Words Additional changes are listed in the detailed [changelog](https://lief.re/doc/latest/changelog.html). Many thanks to dornstetter and [kohnakagawa](https://github.com/kohnakagawa) for their feedback about the dyld shared cache feature. Thank you also to [Konstantin Vinogradov](https://github.com/vinogradovkonst) and [dctoralves](https://github.com/mateeuslinno) for their sponsorship. [^llvm-patch]: All the patches have been PR-submitted to the LLVM. You can check [LIEF & LLVM](https://lief.re/doc/latest/extended/intro.html#lief-extended-llvm) for the details