Obtaining relevant market data to train a Long Short-Term Memory neural network will pose significant challenges. One primary hurdle in acquiring data for my training set is the cost associated with obtaining reliable data. Financial market data is often proprietary and sold by third-party providers who charge substantial fees which will make it a challenge on a limited budget. Some companies do offer free market data, but these come with limitations such as being out-of-date or less detailed.
Secondly, I have to think about obtaining relevant data for my research. This is crucial to properly training an LSTM network. Identifying and obtaining data that directly relates to each research question I ask will be very challenging. Available data may only provide broad industry indices or unrelated financial instruments. This mismatch can significantly impede my research. I will also have to consider the quality of the data that I am using. High quality data results in a high quality model which will significantly help out my project. However, acquiring clean market data is a challenge. Many endpoints return data with missing points, inconsistent formats, and strange outliers which will need to be meticulously cleaned and filled in before I am able to move forward. This time consuming process can significantly hinder my progress.
Finally, I have to consider the volume, complexity, and computing power required to gather, store, and process large time series datasets. Financial data often comes in high dimensional datasets which adds even more complexity in the preprocessing and visualization steps. All of these complicated steps and processes will take up a significant amount of my time and could shut down some ideas that may require unavailable market data.
Featured Image, Photo by Markus Spiske on Unsplash
HSF House Blogs 