Memray project showcases why Bloomberg is an ‘open source first’ company

`It’s fair to say that Bloomberg and the financial services industry have been on a first-name basis for decades. Most every trader, analyst and CFO is familiar with the Bloomberg Terminal or the company’s data and news service and probably interacts with it in some way on a daily basis.

What might not be as well-known are Bloomberg’s contributions to open source software, both as a driver for its own technology and application infrastructure, as well as developing enterprise-scale tools for the broader open source community.

For example, the global finance, media and technology company has more than 3,000 engineers who write code and run data analytics using Python. Bloomberg has established a dedicated Python Infrastructure team to support its engineers and an Open Source Programs Office (OSPO) to help further its work and presence in open source.

A couple years ago, Bloomberg’s Python Infrastructure engineers were asked to find a solution to the unavailability of viable memory profilers that would meet the specific needs of Bloomberg developers. These developers were experts in Python, Linux, ELF, DWARF, and low-level systems programming tooling who frequently used native extensions written in a programming language other than Python, like C/C++. The Python ecosystem lacked a tool to help users discover where a Python program was using memory. The Python Infrastructure team went to work and, after extensive research and user studies, the team determined it was time for Bloomberg engineers to leverage their expertise and develop a new kind of memory profiler to add to its portfolio of open source tools.

Memray, an open source memory profiler for Python on Linux, was recently published by Bloomberg to fill this memory profiler gap. Considering the needs of both Bloomberg developers and other users of native extensions, Memray was designed to enable developers to analyze allocations in applications to help reduce memory usage, find memory leaks, and identify hotspots in code that cause inefficient allocations. It delivers this information transparently, showing users how the Python code and the extension code interact with each other, as well as how the Python interpreter is requesting memory. In keeping with its commitment to open source, it was important to Bloomberg to make this tool available to Python developers worldwide by creating it in the open. Memray is also extendable to provide reporting that offers a thorough and reliable view into the allocation patterns of the code specific to the data or domain problem of communities and user groups.

Memray was announced on April 27, 2022, but the community around it was quietly building since a week earlier, when a CPython developer mentioned it on Twitter. In just a couple of days, Memray rapidly attracted more than 7,000 stars on GitHub (today it has more than 8,000 stars).

“We feel this speaks not only to the need for this tool, but also to Bloomberg’s commitment to bringing awareness to and developing expertise around open source,” said Alyssa Wright of Bloomberg’s OSPO, which is based within the company’s CTO Office. “Open source is a fundamental part of Bloomberg, for both our digital infrastructure and our community-driven culture.”

Bloomberg’s journey to establish itself as an “open source first” company started a decade ago. The company sees this focus as not only a commitment to open source computing, but to a culture of collaboration and innovation as well. Bloomberg welcomes diversity in perspectives, skills, and expertise that allows different departments to work together to make products that solve real problems, all built on the foundation of open source that supports not just Bloomberg’s product innovation, but also a culture of innovation.

Bloomberg’s support of OSI is a no-brainer, according to Alyssa. “OSI is a voice for open source in every space. OSI holds the spirit of open source and is seen as a partner and an advocate for what open source stands for, even as modern development continues to evolve.”

There’s a big opportunity for companies to work collaboratively in open source to solve growing problems and impact organizational change. A few things Alyssa looks forward to seeing are:

• Recognition of the value of open source contributions and contributors within companies and ecosystems.
• Investment being made in measuring the impact of an ‘open source first’ culture and further support for the development of open source tools.
• More guidance and communication around how to “be” in an open source environment.

Check out Memray on GitHub. The community can help shape the future and direction of Memray by proposing improvements and features to add to the tool, as well as useful ways to use and analyze the large amount of data that Memray collects. It was created as open source so it can bring value to other companies and industries, allowing for the addition of new reports that adapt to problems contributors face, as well as being extendable to work on other operating systems.

Follow along with all Bloomberg’s work in open source technology and culture on the company’s blog or on Twitter by following @TechAtBloomberg.