From 428b70ec907a4fa6023cc706cc6a7b0bbe961ab9 Mon Sep 17 00:00:00 2001 From: didier Date: Tue, 23 Mar 2021 22:37:10 +0000 Subject: [PATCH 1/7] Add validate NN model guidelines --- .../VALIDATE-NN-model.md | 121 ++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 gamestonk_terminal/prediction_techniques/VALIDATE-NN-model.md diff --git a/gamestonk_terminal/prediction_techniques/VALIDATE-NN-model.md b/gamestonk_terminal/prediction_techniques/VALIDATE-NN-model.md new file mode 100644 index 00000000000..3e8e9c58e43 --- /dev/null +++ b/gamestonk_terminal/prediction_techniques/VALIDATE-NN-model.md @@ -0,0 +1,121 @@ +# Validate/Tweaking Neural Network Model + +1. **Select the stock that you want to attempt to predict share price** +``` +load MSFT +``` + +2. **Slice the stock price to only relevant data** + +Look at the overall share price and try to understand from when does it start behaving similarly. E.g. looking at MSFT, for me, it starts behaving similarly after 2014. Before that, the stock price is rather flat, and not representative of the current pattern. +``` +load MSFT -s 2014-01-01 +``` + +3. **Out of this training data, select which part will be used for validation** + +Since this is a timeseries, and the data has a sequence, the data at the end of the timeseries is the one to be used for validation. As a rule of thumb I usually try to "steal" the least amount of data from the training set, since the more recent the data the more important it is for training. Hence, one could set the end date of training data to be the current day - days of predictions. +``` +-e/--end : end date (format YYYY-MM-DD) of the stock - Backtesting. Default None. +``` + +4. **Pre-processing data** + +Decide whether you want to pre-process your share price data or not. I.e. do you want to use these values between 18 and 245, or perhaps normalize these between 0 and 1? The later may improve performance. +``` +-p/--pp : pre-processing data. Default normalization. +``` + +5. **Select input, prediction days and jumps in training data** + +The **input** corresponds to the number of days that our model will use for prediction. + +The more past days, the more information our model will have to make a sensible prediction. However, the more days being used will lead to a model that takes longer to train, and some days in the past may have no affect in the prediction. In order to facilitate this decision, think of it in terms of months, e.g. 40 days corresponds to roughly 2 months of data. If you think that the share price today can impact the value that it will take in 2 months, then 40 is a good value, otherwise pick a shorter value. +``` +-i/--input : number of days to use for prediction. Default 40. +``` + +The **prediction days** is the number we want to predict for. + +Personally, I recommend to set this for a really short window. This is because the bigger the window, the less reliable the prediction is. This is mainly due to the fact that by having a larger prediction horizon, more unexpected events (news, insider trading activity, rumours, ...) can occur, which may shift share price. +``` +-d/--days : prediciton days. Default 5. +``` + +The **jumps in the training data** corresponds to the number of days we want to jump between training loops. + +Let's say we start training our Neural Network from 2021-01-04, with 10 inputs (days) and 5 prediction days. If jump is set to 1, it means that the after optimizing our model for input data from 2021-01-04 to 2021-01-15, and predicting from 2021-01-18 to 2021-01-22, we will optimize the model for input data from 2021-01-05 to 2021-01-18, and predicting from 2021-01-19 to 2021-01-25. On the other hand, if we set the jump to 5, it means that we are identifying a "weekly" frequency in the data, and therefore, our next model optimization would use input data from 2021-01-11 to 2021-01-22, and predicting from 2021-01-25 to 2021-01-29. +``` +-j/--jumps : number of jumps in training data. Default 1. +``` + +6. **Select NN architecture.** TL;DR: LSTM > RNN > MLP + +**MLP**: The neurons don't have any memory. +**RNN***: The neurons can only retain really recent memory. +**LSTM** (improved RNN): The neurons have a long-term memory, hence why this is used for text prediction, since the beginning of a sentence is important to predict the ending. + +Have a look at: https://colah.github.io/posts/2015-08-Understanding-LSTMs/ + +7. **Define NN Model** + +**Number of hidden layers**: Due to the comlexity of these NN models (particularly LSTM), this value should be kept low (2, 3 or 4). Otherwise, the model will not generalize well to unseen examples (i.e. it will overfit). + +**Number of units per layers**: This is one of the hardest to know how to set, I usually set this one by trial and iteration, and understanding how the model performs. Note: Be carefuly about this iterative process, to not overfit your hyperparameters to work well in the validation data. Because then it may fail badly on the real prediction. + +**Activation function**: _tanh_ is recommended. It is proved to outperform other activation functions. See https://stats.stackexchange.com/questions/101560/tanh-activation-function-vs-sigmoid-activation-function + +**Dropout layer**: As a rule of thumb, an LSTM layer should be followed by a Dropout layer with a 20% rate. This is in order to prevent overfitting, while keeping model accuracy. + +**Dense layer**: Dense layer to represent outputs. + +``` +Long_Short_Term_Memory \ + = [ {'LSTM': + {'units':25, 'activation':'tanh', 'return_sequences':True} }, + {"Dropout": + {"rate": 0.2} }, + {'LSTM': + {'units':15, 'activation':'tanh', 'return_sequences':False} }, + {'Dense': + {'activation':'linear'} }] +``` +This model is set in [config_neural_network_models.py](/config_neural_network_models.py) + +8. **Model compilation with loss function and optimization technique** + +**Loss function**: Often chosen having the activation functions of the hidden layers in mind. See https://machinelearningmastery.com/how-to-choose-loss-functions-when-training-deep-learning-neural-networks/ +``` +-l/--loss : loss function. Default mae. +``` + +**Optimizer technique**: Adaptive Moment Estimation (_adam_) is usually the default choice for this type of problems. +``` +-o/--optimizer : optimization technique. Default adam. +``` + +9. **Training epochs** + +Select the number of epochs to train your model. The bigger number of epochs, the longer the training will take, and also increase chances of overfitting. +``` +--epochs : number of training epochs. Default 200. +``` + +10. **Model training loops** + +Select the number of loops to - using the defined model - train and predict. This allows to build confidence on the model you are building, and understand what's the median results and also the lower and upper quantiles, of 10 and 90%, respectively. +``` +--loops: number of loops to iterate and train models. Default 1. +``` + +11. **Interpret accuracy metrics** + +Interpret the following metrics: MAPE, R2, MAE, MSE and RMSE. + +12. **Advanced User** + +--batch_size: batch size for model training, should not be used unless advanced user. Default None. +--xla_cpu: if present, will enable XLA for CPU (overrides environment variables during run). +--xla_gpu: if present, will enable XLA for GPU (overrides environment variables during run). +--force_allow_gpu_growth: if true, will force TensorFlow to allow GPU memory usage to grow as needed. Otherwise will allocate 100% of available GPU memory when CUDA is set up. Default true. + From 6fe1e4d61f7723763f7d5157d34a4d55e0c92b7b Mon Sep 17 00:00:00 2001 From: didier Date: Tue, 23 Mar 2021 22:39:53 +0000 Subject: [PATCH 2/7] fix codespell --- gamestonk_terminal/prediction_techniques/VALIDATE-NN-model.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/gamestonk_terminal/prediction_techniques/VALIDATE-NN-model.md b/gamestonk_terminal/prediction_techniques/VALIDATE-NN-model.md index 3e8e9c58e43..9de87cbc269 100644 --- a/gamestonk_terminal/prediction_techniques/VALIDATE-NN-model.md +++ b/gamestonk_terminal/prediction_techniques/VALIDATE-NN-model.md @@ -59,9 +59,9 @@ Have a look at: https://colah.github.io/posts/2015-08-Understanding-LSTMs/ 7. **Define NN Model** -**Number of hidden layers**: Due to the comlexity of these NN models (particularly LSTM), this value should be kept low (2, 3 or 4). Otherwise, the model will not generalize well to unseen examples (i.e. it will overfit). +**Number of hidden layers**: Due to the complexity of these NN models (particularly LSTM), this value should be kept low (2, 3 or 4). Otherwise, the model will not generalize well to unseen examples (i.e. it will overfit). -**Number of units per layers**: This is one of the hardest to know how to set, I usually set this one by trial and iteration, and understanding how the model performs. Note: Be carefuly about this iterative process, to not overfit your hyperparameters to work well in the validation data. Because then it may fail badly on the real prediction. +**Number of units per layers**: This is one of the hardest to know how to set, I usually set this one by trial and iteration, and understanding how the model performs. Note: Be carefully about this iterative process, to not overfit your hyperparameters to work well in the validation data. Because then it may fail badly on the real prediction. **Activation function**: _tanh_ is recommended. It is proved to outperform other activation functions. See https://stats.stackexchange.com/questions/101560/tanh-activation-function-vs-sigmoid-activation-function From f7699f9ed3ac01785b11bda9f98b61065ffff4ff Mon Sep 17 00:00:00 2001 From: DidierRLopes Date: Tue, 23 Mar 2021 22:42:42 +0000 Subject: [PATCH 3/7] Update README.md --- gamestonk_terminal/prediction_techniques/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gamestonk_terminal/prediction_techniques/README.md b/gamestonk_terminal/prediction_techniques/README.md index d62e0503584..58e0a2d2eb3 100644 --- a/gamestonk_terminal/prediction_techniques/README.md +++ b/gamestonk_terminal/prediction_techniques/README.md @@ -1,6 +1,6 @@ # PREDICTION TECHNIQUES -This menu aims to predict the share price of a pre-loaded stock, and the usage of the following commands along with an example will be exploited below. +This menu aims to predict the share price of a pre-loaded stock, and the usage of the following commands along with an example will be exploited below. See [Validate NN model guidelines](VALIDATE-NN-model.md). * [sma](#sma) - simple moving average From d668b64581ea0487a652cb719c32f2923a0887b4 Mon Sep 17 00:00:00 2001 From: didier Date: Thu, 25 Mar 2021 21:26:45 +0000 Subject: [PATCH 4/7] Improve Validate NN model file --- ROADMAP.md | 2 ++ .../prediction_techniques/VALIDATE-NN-model.md | 15 ++++++++++++--- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 699d5329397..b14941c3662 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -97,10 +97,12 @@ - [x] Add several exponential smoothing techniques (@didier) - [PR #132](https://github.com/DidierRLopes/GamestonkTerminal/pull/132) - [x] Allow backtesting (@didier) - [PR #169](https://github.com/DidierRLopes/GamestonkTerminal/pull/169) - [x] Add averaging around multiple predictions (@didier) - [PR #252](https://github.com/DidierRLopes/GamestonkTerminal/pull/252) +- [ ] CNN 1D (@didier) ### Long term - [ ] Combine Sentiment Analysis with Chart data - [IS #240](https://github.com/DidierRLopes/GamestonkTerminal/issues/240) +- [ ] Cross-Validation methods (Forward Chaining, K-Fold, Group K-Fold) (@didier) --- diff --git a/gamestonk_terminal/prediction_techniques/VALIDATE-NN-model.md b/gamestonk_terminal/prediction_techniques/VALIDATE-NN-model.md index 9de87cbc269..7a7c6e6c22e 100644 --- a/gamestonk_terminal/prediction_techniques/VALIDATE-NN-model.md +++ b/gamestonk_terminal/prediction_techniques/VALIDATE-NN-model.md @@ -1,5 +1,7 @@ # Validate/Tweaking Neural Network Model +Note: Modeling a NN this way is intended for learning (fun some may say), a user guest or YOLOing on GME may be more accurate - No financial advisor tho. + 1. **Select the stock that you want to attempt to predict share price** ``` load MSFT @@ -84,12 +86,12 @@ This model is set in [config_neural_network_models.py](/config_neural_network_mo 8. **Model compilation with loss function and optimization technique** -**Loss function**: Often chosen having the activation functions of the hidden layers in mind. See https://machinelearningmastery.com/how-to-choose-loss-functions-when-training-deep-learning-neural-networks/ +**Loss function**: Often chosen having the activation functions of the hidden layers in mind. See https://machinelearningmastery.com/how-to-choose-loss-functions-when-training-deep-learning-neural-networks/, and https://www.tensorflow.org/api_docs/python/tf/keras/losses. ``` -l/--loss : loss function. Default mae. ``` -**Optimizer technique**: Adaptive Moment Estimation (_adam_) is usually the default choice for this type of problems. +**Optimizer technique**: Adaptive Moment Estimation (_adam_) is usually the default choice for this type of problems. See https://www.tensorflow.org/api_docs/python/tf/keras/optimizers. ``` -o/--optimizer : optimization technique. Default adam. ``` @@ -113,9 +115,16 @@ Select the number of loops to - using the defined model - train and predict. Thi Interpret the following metrics: MAPE, R2, MAE, MSE and RMSE. 12. **Advanced User** - +``` --batch_size: batch size for model training, should not be used unless advanced user. Default None. +``` +This can be set to `floor(log2(input_layer_size))` for much faster training if a large amount of input samples (assuming you have the RAM for it). It shouldn't have much effect on the results. + +``` --xla_cpu: if present, will enable XLA for CPU (overrides environment variables during run). --xla_gpu: if present, will enable XLA for GPU (overrides environment variables during run). --force_allow_gpu_growth: if true, will force TensorFlow to allow GPU memory usage to grow as needed. Otherwise will allocate 100% of available GPU memory when CUDA is set up. Default true. +``` +See https://www.tensorflow.org/xla +In the Roadmap: Cross-Validation (Forward Chaining, K-Fold, Group K-fold), NN to use sentiment analysis From d24a7fb03210009d83b32be6ebe80d2d2d9d6acf Mon Sep 17 00:00:00 2001 From: didier Date: Thu, 25 Mar 2021 21:31:32 +0000 Subject: [PATCH 5/7] Fune-tuning instead of fine-tuning eheh --- .../{VALIDATE-NN-model.md => FUNE-TUNING-NN-models.md} | 0 gamestonk_terminal/prediction_techniques/README.md | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) rename gamestonk_terminal/prediction_techniques/{VALIDATE-NN-model.md => FUNE-TUNING-NN-models.md} (100%) diff --git a/gamestonk_terminal/prediction_techniques/VALIDATE-NN-model.md b/gamestonk_terminal/prediction_techniques/FUNE-TUNING-NN-models.md similarity index 100% rename from gamestonk_terminal/prediction_techniques/VALIDATE-NN-model.md rename to gamestonk_terminal/prediction_techniques/FUNE-TUNING-NN-models.md diff --git a/gamestonk_terminal/prediction_techniques/README.md b/gamestonk_terminal/prediction_techniques/README.md index 58e0a2d2eb3..efd0a0f2fcd 100644 --- a/gamestonk_terminal/prediction_techniques/README.md +++ b/gamestonk_terminal/prediction_techniques/README.md @@ -1,6 +1,6 @@ # PREDICTION TECHNIQUES -This menu aims to predict the share price of a pre-loaded stock, and the usage of the following commands along with an example will be exploited below. See [Validate NN model guidelines](VALIDATE-NN-model.md). +This menu aims to predict the share price of a pre-loaded stock, and the usage of the following commands along with an example will be exploited below. See [How to fune-tuning a NN model](FUNE-TUNING-NN-models.md). * [sma](#sma) - simple moving average From 16458d307890bdee7883a9bda9d8ddf939225b8b Mon Sep 17 00:00:00 2001 From: didier Date: Thu, 25 Mar 2021 21:33:21 +0000 Subject: [PATCH 6/7] tiny improv --- .../prediction_techniques/FUNE-TUNING-NN-models.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gamestonk_terminal/prediction_techniques/FUNE-TUNING-NN-models.md b/gamestonk_terminal/prediction_techniques/FUNE-TUNING-NN-models.md index 7a7c6e6c22e..ad4ab6bab15 100644 --- a/gamestonk_terminal/prediction_techniques/FUNE-TUNING-NN-models.md +++ b/gamestonk_terminal/prediction_techniques/FUNE-TUNING-NN-models.md @@ -1,6 +1,6 @@ # Validate/Tweaking Neural Network Model -Note: Modeling a NN this way is intended for learning (fun some may say), a user guest or YOLOing on GME may be more accurate - No financial advisor tho. +Note: Modeling a NN this way is intended for learning purposes (or fun) only. A user guess or YOLOing on GME may present better returns - No financial advisor tho. 1. **Select the stock that you want to attempt to predict share price** ``` From f24c30ce44440b37890ca63acac4c26ba562d929 Mon Sep 17 00:00:00 2001 From: didier Date: Thu, 25 Mar 2021 21:41:55 +0000 Subject: [PATCH 7/7] Add export/save flag idea to README --- ROADMAP.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/ROADMAP.md b/ROADMAP.md index b14941c3662..54ababa2d78 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -245,6 +245,8 @@ ### Short term - [ ] Publish Docker +- [ ] Most commands to have export flag to save output into csv/txt file +- [ ] Most commands to have save flag to save image into png/jpg file ### Long term