OpenResty Inc.

When we develop new dynamic tracing analyzers in #OpenRestyXRay, we usually start by directly copying the C/C++ code from the target programs into our Y/Y++ code, such as CPython, JVM, PHP, etc. Generally, we still need to make minor edits to minimize copying excessive runtime dependency code. Now, even these last bits of manual editing may be automated through #Claude 3 Opus. Just yesterday, we used Claude 3 Opus to rewrite a large C function, and surprisingly, the resulting code was nearly production-quality. The diff patches compared to the original code looked perfect—we were impressed. In contrast, when we assigned the same task to #ChatGPT’s GPT-4, the results were disappointing, with some obvious errors. It felt like comparing a professional’s work to that of an elementary school student. We also saw a post on Reddit where someone mentioned using Claude 3 Opus to refactor a large software project with many files, and he found it significantly better than GPT-4. He commented that ChatGPT often imagines things that don’t exist. #LLM #AI

Our CEO has been leading some engineers in our team to focus on developing #jvm deep analysis tools based on non-invasive #DynamicTracing technologies like #ebpf. We aim to release the first version this quarter. The tools are particularly suited for online production environments as they have minimal impact, unlike tools like BTrace and Alibaba's Arthas, which can significantly disrupt online #java applications and pose high risks. They also operate independently of the JVM Safepoints, providing a comprehensive analysis with no blind spots. When ready, #OpenRestyXRay will integrate these tracing tools into our AI diagnostic reasoning engine and analysis report generator. The initial release will support OpenJDK, and there is potential for future support for Oracle's proprietary JDK. This approach avoids using mechanisms like JVM's Agent for code injection, enabling a completely read-only analysis of Java services already running online, akin to taking an X-ray. The full dynamic probes at Java methods' entry and exit points are just one of many tracing features, including JIT-compiled code. Additionally, the tool will offer CPU hotspot analysis, IO blocking and lock contention analysis, memory usage analysis, GC analysis, exception analysis, thread scheduling analysis, latency analysis, targeted smart packet capture for network traffic, and tracking of Java data structure states. We can also support older systems like CentOS/RHEL 7 with 3.10 kernels and even ancient versions of JVM, as our technology does not rely on any cooperation or assistance from the JVM itself.

We wrote a new blog post, "Use C++ to Dynamic-Trace C++ Applications": https://lnkd.in/eemv8e8F It can compile real C++ code into noninvasive dynamic tracing analyzers for eBPF+, Stap+, GDB, and more! It has been a big leap since we were brought into this tracing world more than a decade ago 🙂 This opens the door to writing complex, nonintrusive dynamic tracing tools for big C++ applications like HotSpot JVM, NodeJS, and MySQL, with strictly 0 collaborations from the target process 😃 How can it be easier than writing C++ to trace C++? 😉 #eBPF #Stap

Recently we spent a day or two on #Android 14 and finally passed the full unit test suite of eBPF+. On Android 14, the ELF files compiled with the NDK toolchain have a difference of 0x1000, or 4096 bytes, between the relative memory address and the file offset, which is quite strange. Android's OS #kernel, compiled with #clang, also has more runtime checks, such as CFI. But the kernels of mainstream Linux distributions are almost all compiled with gcc, without such extra checks. To learn more about our eBPF+ and Ylang in OpenResty XRay: https://lnkd.in/gnsmzBUS #Android #eBPF #DynamicTracing

Recently we spent about ten minutes and successfully ported our eBPF+ engine to #Android 14. eBPF+ is an enhanced version of eBPF that we use in our OpenResty XRay product. The simplest two test cases for kprobes and uprobes have already passed on Android 14. The implementation of Android is almost identical to the #Linux kernel, so it was very smooth. It looks like it can also be easily compatible with older versions of Android down to Android 10. The next step is to run more eBPF+ unit tests until we run the huge test suites of our compilers, such as #Ylang, #YLua, and #YSQL. The difference in Android user space is bigger, such as using bionic libc, the header file of epoll is quite different from Linux, and it also enforces the use of PIE executable files. To learn more about our eBPF+ and Ylang: https://lnkd.in/gnsmzBUS #eBPF #DynamicTracing #FullStackAnalysis #NonInvasiveAnalysis #Android

The vision for our OpenResty XRay product is: for any customer's software business system in a production environment, including multi-cluster distributed systems, OpenResty XRay can thoroughly and transparently analyze its entire software stack. This includes hardware to the OS kernel, the network protocol stack, system software, and the business software layer. Metaphorically speaking, we aim to transform these systems from black boxes into transparent ones. It's like taking an X-ray or MRI of the production system, automatically identifying all underlying issues and causes related to performance, security, stability, or correctness. Importantly, these systems and applications require no modifications. There's no need to add any libraries or modules, no need for logging or metrics data, and even process restarts are unnecessary. All that's needed is a standalone Agent daemon of OpenResty XRay running on each machine to analyze all applications and containers. Our goal is to tackle one of the greatest challenges in the computing world: the increasing complexity of systems gradually diminishing human insight and control over software and hardware systems. #eBPF #DynamicTracing #FullStackAnalysis #RealTimeAnalysis

A netizen inquired about the oldest #Linux systems and kernels that the OpenResty XRay dynamic tracing platform can support. Our platform can support systems as old as CentOS/RHEL 6, used with the CentOS/RHEL 7's 3.10 kernels. The 2.6 kernels are just too old to support. For Ubuntu, the oldest supported version is 16.04, and for Debian, it's version 9. We can also support even older versions like Ubuntu 14.04 and Debian 8 if necessary. In contrast, the open-source eBPF toolchain usually has much higher kernel requirements, severely limiting its practicality in many enterprises. #eBPF

A major technical challenge involves automatically #generating debug symbols for Linux kernel images lacking #debuginfo. We cannot always find debug symbols for a random kernel in the wild. This task also requires addressing security mechanisms like the struct field randomization in C compilers. The AI-based automatic reverse-engineering algorithm used here is similar to the one we mentioned before, which automatically modifies kernel modules to adapt to new kernels. Once perfected, this can be integrated into our OpenResty XRay dynamic tracing product. Interestingly, Reini Urban, a core developer of the Perl 5 interpreter, commented on X (formerly Twitter) that this is an "unwinnable uphill battle," a rather pessimistic view in our humble opinion. However, our track record has been quite good. We've generally been able to achieve the technical goals we set, so please wish us the best! #eBPF #DebugSymbols #LinuxKernel

#Ylang is our universal dynamic tracing language that is compatible with C. It is employed in the OpenResty XRay product, which includes a Web #IDE. This IDE allows users to use Ylang for writing and running various dynamic tracing analysis tools. For instance, a demonstration screenshot shows the use of Ylang to create a custom dynamic tracing tool. This tool places a dynamic probe on the `ngx_http_finalize_request` function of #Nginx, printing information such as the URI, URI parameters, and response status codes of up to five requests when the probe is triggered. The screenshot displays the entered Ylang source code (seen in the top-left editing box), the configuration of the application being traced (top-right), and the output after running it online (in the black box at the bottom-left). Ylang can be executed through various backends, including eBPF+, making it suitable for online production environments. The target application being traced does not require any modifications or restarts, ensuring a 100% non-intrusive process. Compared to other open-source dynamic tracing languages like #BCC, #eBPF C, #bpftrace, D language of #dtrace, #systemtap's scripting language, and #gdb python, Ylang is significantly more concise and powerful. For more details about Ylang, refer to a series of articles I previously published: https://lnkd.in/gnsmzBUS

We have recently compiled a newsletter for the OpenResty Edge distributed gateway and private CDN software products. You can read it here: https://lnkd.in/gpqT4Wiz This newsletter summarizes the latest articles and video tutorials published about this product. #K8s #DistributedGateway #WAF #PrivateCDN