A market data feed provides price and other information so market participants can decide whether to buy or sell securities and commodities. Raw market data feeds are generated by exchanges such as the New York Stock Exchange and organizations such as the Options Price Reporting Authority, which consolidates feeds from multiple exchanges. However, many market participants prefer to use normalized market data feeds generated by specialist market data vendors such as SpiderRock.
Markets don’t function when participants can’t access information about traded assets. A lack of information leads to a lack of liquidity. Traders need to know about the state of the market, the price of assets, and how those prices change over time, amongst other information. Market data is the totality of this information, and market data feeds are the method by which that data is delivered to market participants.
Market data feeds broadcast huge volumes of data because they track thousands of items and are updated frequently. To work with raw feeds, an organization requires network and computing resources capable of processing many gigabits of data each second. Because of this expense and complexity, many institutional and individual traders use normalized feeds from data vendors instead. These feeds reduce processing and network requirements while providing the information traders need, often alongside additional information based on analyses of the raw feed data.
How do Traders Use Market Data Feeds?
Most exchanges that handle commodities and securities trading provide information to traders via a market data feed. These include data feeds for:
- Commodities such as gold and oil
Traders typically access market feeds data via software that provides an intuitive interface for searching and comparing normalized market data. Feed data can also be accessed directly by trading software that makes buy and sell decisions based on predetermined trading algorithms, as in the case with high-frequency trading.
Feed Depth and Breadth
Feeds are often classified according to the depth and breadth of the information they contain. For example, a Level I feed, also known as top-of-book, includes information on the highest bids and offers for an asset. Level II market data, also known as depth-of-book, is a more detailed feed that contains a greater range of bids and offer prices. Top-of-book feeds provide sufficient information for many traders, but high-frequency and algorithmic trading often depends on deeper, more comprehensive market data.
Historical vs. Real-Time Market Data
In addition to the type of data, feeds also differ in timeliness. A market data feed might supply information in real-time or with a delay. A data vendor may also supply historical data that tracks an asset’s performance over several years.
- Real-time data feeds allow traders to react to the market as it evolves. Real-time feeds are often used by algorithmic traders and others seeking to exploit slight pricing differences as the market fluctuates second-by-second. Real-time data feeds are technically challenging, high volume, and expensive, but they are critical for trading strategies that depend on the lowest possible latencies.
- Delayed feeds introduce significant latencies, updating perhaps only every 15 minutes or more. Delayed feeds provide useful information for investors who don’t rely on rapid selling and buying, and they are often sufficient for organizations and individuals engaged in longer-term investment strategies. Delayed feeds are often free or available at a low cost, in contrast to the much more expensive real-time feeds.
- Historical data allows traders to access pricing and market information for a period of time in the past. For example, a trader might request stock prices for a company over the last three years. Historical data helps traders to analyze markets and identify patterns. They are also useful for backtesting algorithmic trading strategies.
How Are Market Data Feeds Delivered?
Raw market data feeds are generated by market exchanges and other trading venues for stocks, commodities, foreign currencies, options, futures, and other assets and financial instruments. Data feeds represent a market’s activity from the perspective of the provider’s internal software systems. Market activity generates data, which is stored in the feed provider’s databases and packaged into a format that can be communicated to data consumers over the network.
The raw feeds are typically broadcast with UDP, the User Datagram Protocol. UDP is a one-way communication protocol that offers lower latency and complexity than the two-way TCP protocol used by much of the internet. UDP doesn’t depend on a two-way connection between sender and recipient and is essentially a massive data firehose broadcast by the exchange and received by the consumer.
Most market feeds use UDP multicast to further simplify and accelerate data transfers. The internet’s TCP connections are unicast one-to-one connections between the sender (a web server) and the recipient (a web browser, for example). In contrast, multicast sends a single copy of the feed to network switches, which take care of copying and sending the data to other members of a multicast group.
The exchange’s feeds are received by dedicated software and infrastructure called a feed handler. The feed handler consumes the live feeds and carries out processing, including decoding, normalization, and analytics. The normalization process is critical: it transforms the feed, or multiple feeds, into an internal format designed to meet the trading requirements of end-users.
Larger organizations may then integrate the feed data into their in-house trading platforms. Alternatively, a data vendor such as SpiderRock rebroadcasts a modified feed that is smaller, less resource-intensive, and more valuable to end-users.
For example, consuming the raw Options Price Reporting Authority (OPRA) feed requires substantial cloud or colocated compute infrastructure and high-bandwidth network connectivity. For the SpiderRock normalized OPRA feed, we use a combination of field-programmable gate array (FGPA) and kernel bypass technology to process the raw feed ingested via 100GB fiber connections.
After normalization into our proprietary format, we broadcast an options feed a single consumer workstation can process. The SpiderRock feed also includes data not in the original feed, such as implied surface volatilities, greeks, and proprietary analytics.
We provide similarly normalized feeds for a comprehensive range of U.S. equities, options, and futures markets, as well as specialized greeks and volatility data fields.
How to Access Market Data Feeds
Organizations may choose to access market feeds via exchanges’ multicast UDP feeds. However, the network, infrastructure, and software demands of processing a raw feed are prohibitive. Many organizations instead choose to access a normalized feed. A reliable and technically expert data vendor allows traders to focus on the core trading activities while the vendor handles the expensive complexities of feed normalization and preliminary analysis.
SpiderRocks customers have several options for accessing our real-time market data feeds. They include:
- Raw and normalized low-latency multicast channels.
- SpiderRock’s cloud-hosted MySQL API (SRSE). SRSE is a high-throughput custom storage engine that clients can query and integrate into their trading environment.
- A C++/C# market data API, which users can integrate with client feed handlers and other software. The API converts multicast data packets into messages within the client software. It is highly optimized for the high-volume throughput of low-latency market data.
Additionally, historical data and analytics can be accessed both as flat files via SFTP or from our ElasticSearch database API.