This year, the Chicago office welcomed 68 interns from tech, trading and research to work alongside our full-time colleagues on industry challenges. Our tech interns each worked on a project to solve a real business challenge, with the expectation that they would push a solution into production over the summer.

In this blog post, one of our Software Engineer interns and one of our FPGA interns provide insight into how they approached their projects, how they came up with their innovative solutions and the lessons they learned along the way.

Writing processed exchange messages more efficiently

Sabrina, Massachusetts Institute of Technology

Challenge

At a high-frequency trading firm, autotraders send a massive amount of data to exchanges. Optiver captures and processes all messages we send and receive – it’s incredibly valuable to us to understand, analyze and improve our strategies, ultimately helping us shave off nanoseconds where it matters.

It’s useful for us to be able to see these data streams live throughout the day as messages are sent back and forth. To do so, we initially had one singular requirement, that the file format needed to support live appending. As a result, Optiver chose to represent the decodes in a text-based format, which was trivially appendable.

As Optiver has scaled to more exchanges, the pipeline quickly slowed down. We took a step back and thought about the design. For efficiency, these data streams are transmitted in a binary encoding. Since most low-latency code at Optiver is written in C, we could take advantage of this, and make a very fast no-copy decoder. That was where my project came in: It aimed to make our pipeline more efficient by writing it to a custom binary file format designed by Optiver.

Approach

The binary file format takes advantage of the fact that messages sent to and received from the exchanges are usually binary streams of data, which we read into packed C structs. It’s designed in such a way that for most exchange messages, writing out the decode amounted to a memcopy, which sped up operations easily by an order of magnitude.

As part of my project, I ended up adding to:

• Message decoders to process data from the exchange
• Message writers to further process the data into a generic format
• A binary format file writer that outputs the processed data to a file
• Support to a Python C extension that reads data from these files and presents them as Pandas dataframes
• A metrics dashboard to track the full pipeline’s live performance

Working with different message types, from their arrival as binary data all the way to their presentation to researchers, was interesting and taught me a lot about the way data is handled at Optiver.

Results

The new metrics dashboard makes it easier for my team to spot issues if our pipelines are falling behind. With tens to hundreds of thousands of message PCAPs going through each exchange pipeline every second, efficiency is extremely important. The improved efficiency of the new binary format writers helps Optiver capture high-quality data as our data volume scales.

Learnings

The most unique part of my Optiver experience was how much my mentors encouraged me to take ownership of my project. Although I was initially hesitant, they consistently supported me by offering design opinions when my changes would impact other parts of the system or other people.
This taught me to consider the broader impact of my work and helped me feel more integrated into my team.

Enhancing Software to FPGA Communications

Varun, Carnegie Mellon University

Challenge

Optiver extensively uses FPGAs—custom silicon chips in our autotraders—to facilitate communication between software applications, exchanges and other FPGAs.

A challenge with FPGAs is the bottlenecks created when software applications send large bursts of data to FPGAs, causing the small buffer in the FPGA to fill up quickly. Once the buffer is full, the transfer speed slows to match the FPGA’s processing capability, which in turn makes scaling up costly and diverts resources from new features. Given their high costs and limited availability, optimizing FPGA performance is crucial.

My project focused on improving communication between software applications and FPGAs, addressing inefficiencies in the current implementation where software often sits idle, waiting for the FPGA to process large data transfers.

Approach

With this challenge, I was handed a blank slate that required a lot of research to understand FPGA communication, ideation on solutions with the help of the hardware team and collaboration with the software team.

My solution was to redesign the communication interface between the software and FPGA, removing the balanced buffer allocation between the two and increasing the software buffer space. Instead of waiting for the FPGA to process the data in full, the software would send a request to the FPGA, which would then delay the transfer until there was available room and notify the software upon completion. This shift allowed the FPGA to handle more requests without requiring a larger buffer, improving scalability.

Results

The time software spent waiting to send data to the FPGA was reduced by 88% in worst-case scenarios. Additionally, memory utilisation on the FPGA dropped by 80%, freeing up resources for other critical tasks. This solution provided a more efficient way to manage communication without compromising FPGA performance.

Learnings/reflection

This project was unique because I had full ownership of the solution, from initial research to final production. Working closely with both hardware and software teams, I gained a deep understanding of FPGA communication and Optiver’s trading systems. I feel I made a meaningful impact by addressing a real business challenge and putting a solution into production that will continue to be used at Optiver.

Ready to launch your career?

Fill out our Expression of Interest form to be notified when our 2026 internship and full-time roles open.