The Implementation of Concave Function to Interpolate Stocks Data Using Python

Hands-on Tutorial

Overview

Volatility is a crucial characteristic that demands careful consideration from investors when dealing with stock securities. As such, investors need to employ intelligent strategies to maximize their capital gains, such as financial forecasting. Data analysis of the Jakarta Composite Index (JCI) is a prime example of this approach, with data readily accessible from Yahoo! Finance. However, one crucial aspect to consider in our analysis is the presence of missing values in the data.

Typically, the Stock Exchange operates from Monday to Friday, except on holidays. To address missing values, interpolation can be utilized, and the concave function proposed by Mittal and Goel (2012) can be applied. When there is a JCI value X on a given day and the next available value is Y with n days of missing data in between, the first missing value X1 can be approximated using the formula (X+Y)/2. This approximation can be repeated for other missing values in the dataset.

Let’s Practice with Python

To initiate the curve function, we need to first download the data from Yahoo! Finance. While you have the flexibility to select any stock, for this tutorial, we will focus on the JCI data for the year 2019. Please refer to my GitHub repository to access the data for further analysis.

# Import libraries
import pandas as pd   # Dataframe manipulation
import numpy as np    # Mathematics operation
import datetime       # Date and time
# Load the data
ihsg_data = pd.read_csv('Datasets/^JKSE.csv')
print('Dimension of JCI data:n{}'.format(ihsg_data.shape[0]),
      'rows and {}'.format(ihsg_data.shape[1]),'columns')
ihsg_data.head()

# Get the metadata of columns
ihsg_data.info()
# Check missing value
ihsg_data.isna().sum()

Upon importing the data, we define several functions that will be applied to our dataset. These functions include:

• imput_date: to input unlisted dates in our data
• return_stocks: to calculate the return on stocks

• curve_function: to interpolate the missing values using the curve function

  We can now execute the first function, return_stocks, which calculates the returns on stocks. The resulting output will be a dataset with a new column representing the calculated returns. We can utilize the pipe method on pandas to accomplish this, as illustrated below:

# Calculate the return of stocks
ihsg_data_clean = ihsg_data.pipe(return_stocks,col='Adj Close', date='Date')
print('Dimension of financial news:n{}'.format(ihsg_data_clean.shape[0]),
      'rows and {}'.format(ihsg_data_clean.shape[1]),'columns')
ihsg_data_clean.head()

Once the return_stocks function has been executed, we will obtain returns for all the listed dates in our dataset. However, we still need to address the issue of interpolating missing values for dates with blank entries. To do so, we can utilize the imput_date function to provide the dates in an interval format. The code and results for this step are as follows:

# Input the missing data on date column
ihsg_data_clean = ihsg_data_clean.pipe(imput_date,col='Date')
print('Dimension of financial news:n{}'.format(ihsg_data_clean.shape[0]),
      'rows and {}'.format(ihsg_data_clean.shape[1]),'columns')
ihsg_data_clean.head()

Certainly! It’s important to determine whether the dates with filled missing data are trading days or not. Since trading activities typically occur from Monday to Friday, we need to provide this information to the curve_function. This will enable it to make accurate interpolations based on trading days.

# Create dummy variable for deterimining free day or not
free = []
for i in range(ihsg_data_clean.shape[0]):
    if pd.isna(ihsg_data_clean.iloc[i]['Volume']):
        free.append(0)
    else:
        free.append(1)

Once the trading days are determined, we can proceed to run the main function of curve_function. This function will then interpolate the missing values in our data using the curve function, taking into account the trading days information. The codes and results are as follows.

# Interpolate the return by curve function
ihsg_data_curve = ihsg_data_clean.pipe(curve_function)
print('Dimension of financial news:n{}'.format(ihsg_data_curve.shape[0]),
      'rows and {}'.format(ihsg_data_curve.shape[1]),'columns')
ihsg_data_curve.head()

The final step is to combine the "Free" column with the table obtained from the curve_function. This will provide us with information about whether a specific date is a free day or not, which can be useful for further analysis. By filtering the data based on the "Free" column, we can easily distinguish between trading days and non-trading days. The codes and results are shown below.

# Concatenate previous-interpolated data with dummy
ihsg_data = pd.concat([ihsg_data_curve,pd.Series(free,name='Free')],axis=1)
ihsg_data.head()

Conclusion

The concave function is a crucial data preprocessing step for conducting data analysis or machine learning on stock financial data. It assumes that the pattern in daily stock data is stationary and linear. By using the concave function, an analyst can capture the availability of 365 days of stock data in a year, which can be customized as needed. This preprocessing step is essential in handling missing data in stock financial data and can provide valuable insights for further analysis or modeling purposes.

Please feel free to visit my GitHub repository to review the complete code for the concave function and its implementation in handling missing data in stock financial data. The repository contains the code files and any necessary documentation for your reference. Thank you for your interest!

References

A. Mittal, A. Goel. Stock prediction using Twitter sentiment analysis (2012), http://cs229.stanford.edu/proj2011/GoelMittalStockMarketPredictionUsingTwitterSentimentAnalysis.pdf.