| Title: | Recurrent Neural Network |
|---|---|
| Description: | Implementation of a Recurrent Neural Network architectures in native R, including Long Short-Term Memory (Hochreiter and Schmidhuber, <doi:10.1162/neco.1997.9.8.1735>), Gated Recurrent Unit (Chung et al., <arXiv:1412.3555>) and vanilla RNN. |
| Authors: | Bastiaan Quast [aut, cre] |
| Maintainer: | Bastiaan Quast <[email protected]> |
| License: | GPL-3 |
| Version: | 1.9.0 |
| Built: | 2024-11-05 05:29:02 UTC |
| Source: | https://github.com/bquast/rnn |
backpropagate the error in a model object of type gru
backprop_gru(model, a, c, j, ...)backprop_gru(model, a, c, j, ...)
model |
the output model object |
a |
the input of this learning batch |
c |
the output of this learning batch |
j |
the indexes of the sample in the current batch |
... |
argument to be passed to method |
the updated model
backpropagate the error in a model object of type rlstm
backprop_lstm(model, a, c, j, ...)backprop_lstm(model, a, c, j, ...)
model |
the output model object |
a |
the input of this learning batch |
c |
the output of this learning batch |
j |
the indexes of the sample in the current batch |
... |
argument to be passed to method |
the updated model
backpropagate the error in a model object
backprop_r(model, a, c, j, ...)backprop_r(model, a, c, j, ...)
model |
the output model object |
a |
the input of this learning batch |
c |
the output of this learning batch |
j |
the indexes of the sample in the current batch |
... |
argument to be passed to method |
the updated model
backpropagate the error in a model object of type rnn
backprop_rnn(model, a, c, j, ...)backprop_rnn(model, a, c, j, ...)
model |
the output model object |
a |
the input of this learning batch |
c |
the output of this learning batch |
j |
the indexes of the sample in the current batch |
... |
argument to be passed to method |
the updated model
Binary to Integer
bin2int(binary) b2i(binary)bin2int(binary) b2i(binary)
binary |
input binary |
integer representation
b2i(): individual Binary to Integer
clean the model for lighter output
clean_lstm(model)clean_lstm(model)
model |
the output model object |
the updated model
Initialize the weight parameters
clean_r(model)clean_r(model)
model |
the output model object |
the updated model
clean the model for lighter output
clean_rnn(model)clean_rnn(model)
model |
the output model object |
the updated model
Apply the learning rate decay to the learning rate, called in epoch_model_function
epoch_annealing(model)epoch_annealing(model)
model |
the output model object |
the updated model
Print the error adn learning rate at each epoch of the trainr learning, called in epoch_function
epoch_print(model)epoch_print(model)
model |
the output model object |
nothing
Initialize the weight parameter for a gru
init_gru(model)init_gru(model)
model |
the output model object |
the updated model
Initialize the weight parameter for a lstm
init_lstm(model)init_lstm(model)
model |
the output model object |
the updated model
Initialize the weight parameters
init_r(model)init_r(model)
model |
the output model object |
the updated model
Initialize the weight parameter for a rnn
init_rnn(model)init_rnn(model)
model |
the output model object |
the updated model
Integer to Binary
int2bin(integer, length = 8) i2b(integer, length = 8)int2bin(integer, length = 8) i2b(integer, length = 8)
integer |
input integer |
length |
binary representation length |
binary representation
i2b(): individual Integer to Binary
Apply the learning rate to the weight update, vocabulary to verify !!
loss_L1(model)loss_L1(model)
model |
the output model object |
the updated model
predict the output of a gru model
predict_gru(model, X, hidden = FALSE, real_output = T, ...)predict_gru(model, X, hidden = FALSE, real_output = T, ...)
model |
output of the trainr function |
X |
array of input values, dim 1: samples, dim 2: time, dim 3: variables (could be 1 or more, if a matrix, will be coerce to array) |
|
should the function output the hidden units states |
|
real_output |
option used when the function in called inside trainr, do not drop factor for 2 dimension array output |
... |
arguments to pass on to sigmoid function |
array or matrix of predicted values
predict the output of a lstm model
predict_lstm(model, X, hidden = FALSE, real_output = T, ...)predict_lstm(model, X, hidden = FALSE, real_output = T, ...)
model |
output of the trainr function |
X |
array of input values, dim 1: samples, dim 2: time, dim 3: variables (could be 1 or more, if a matrix, will be coerce to array) |
|
should the function output the hidden units states |
|
real_output |
option used when the function in called inside trainr, do not drop factor for 2 dimension array output |
... |
arguments to pass on to sigmoid function |
array or matrix of predicted values
predict the output of a RNN model
predict_rnn(model, X, hidden = FALSE, real_output = T, ...)predict_rnn(model, X, hidden = FALSE, real_output = T, ...)
model |
output of the trainr function |
X |
array of input values, dim 1: samples, dim 2: time, dim 3: variables (could be 1 or more, if a matrix, will be coerce to array) |
|
should the function output the hidden units states |
|
real_output |
option used when the function in called inside trainr, do not drop factor for 2 dimension array output |
... |
arguments to pass on to sigmoid function |
array or matrix of predicted values
predict the output of a RNN model
predictr(model, X, hidden = FALSE, real_output = T, ...)predictr(model, X, hidden = FALSE, real_output = T, ...)
model |
output of the trainr function |
X |
array of input values, dim 1: samples, dim 2: time, dim 3: variables (could be 1 or more, if a matrix, will be coerce to array) |
|
should the function output the hidden units states |
|
real_output |
option used when the function in called inside trainr, do not drop factor for 2 dimension array output and other actions. Let it to TRUE, the default, to let the function take care of the data. |
... |
arguments to pass on to sigmoid function |
array or matrix of predicted values
## Not run: # create training numbers X1 = sample(0:127, 10000, replace=TRUE) X2 = sample(0:127, 10000, replace=TRUE) # create training response numbers Y <- X1 + X2 # convert to binary X1 <- int2bin(X1) X2 <- int2bin(X2) Y <- int2bin(Y) # Create 3d array: dim 1: samples; dim 2: time; dim 3: variables. X <- array( c(X1,X2), dim=c(dim(X1),2) ) # train the model model <- trainr(Y=Y[,dim(Y)[2]:1], X=X[,dim(X)[2]:1,], learningrate = 1, hidden_dim = 16 ) # create test inputs A1 = int2bin( sample(0:127, 7000, replace=TRUE) ) A2 = int2bin( sample(0:127, 7000, replace=TRUE) ) # create 3d array: dim 1: samples; dim 2: time; dim 3: variables A <- array( c(A1,A2), dim=c(dim(A1),2) ) # predict B <- predictr(model, A[,dim(A)[2]:1,] ) B = B[,dim(B)[2]:1] # convert back to integers A1 <- bin2int(A1) A2 <- bin2int(A2) B <- bin2int(B) # inspect the differences table( B-(A1+A2) ) # plot the difference hist( B-(A1+A2) ) ## End(Not run)## Not run: # create training numbers X1 = sample(0:127, 10000, replace=TRUE) X2 = sample(0:127, 10000, replace=TRUE) # create training response numbers Y <- X1 + X2 # convert to binary X1 <- int2bin(X1) X2 <- int2bin(X2) Y <- int2bin(Y) # Create 3d array: dim 1: samples; dim 2: time; dim 3: variables. X <- array( c(X1,X2), dim=c(dim(X1),2) ) # train the model model <- trainr(Y=Y[,dim(Y)[2]:1], X=X[,dim(X)[2]:1,], learningrate = 1, hidden_dim = 16 ) # create test inputs A1 = int2bin( sample(0:127, 7000, replace=TRUE) ) A2 = int2bin( sample(0:127, 7000, replace=TRUE) ) # create 3d array: dim 1: samples; dim 2: time; dim 3: variables A <- array( c(A1,A2), dim=c(dim(A1),2) ) # predict B <- predictr(model, A[,dim(A)[2]:1,] ) B = B[,dim(B)[2]:1] # convert back to integers A1 <- bin2int(A1) A2 <- bin2int(A2) B <- bin2int(B) # inspect the differences table( B-(A1+A2) ) # plot the difference hist( B-(A1+A2) ) ## End(Not run)
A Recurrent Neural Network in native R, transforms numbers to binaries before adding bit by bit, teaching itself how to carry.
Bastiaan Quast [email protected]
https://qua.st/rnn/
trainr for training a model and predictr for using a model to make predictions. https://qua.st/rnn
Trains a Recurrent Neural Network.
trainr( Y, X, model = NULL, learningrate, learningrate_decay = 1, momentum = 0, hidden_dim = c(10), network_type = "rnn", numepochs = 1, sigmoid = c("logistic", "Gompertz", "tanh"), use_bias = F, batch_size = 1, seq_to_seq_unsync = F, update_rule = "sgd", epoch_function = c(epoch_print, epoch_annealing), loss_function = loss_L1, ... )trainr( Y, X, model = NULL, learningrate, learningrate_decay = 1, momentum = 0, hidden_dim = c(10), network_type = "rnn", numepochs = 1, sigmoid = c("logistic", "Gompertz", "tanh"), use_bias = F, batch_size = 1, seq_to_seq_unsync = F, update_rule = "sgd", epoch_function = c(epoch_print, epoch_annealing), loss_function = loss_L1, ... )
Y |
array of output values, dim 1: samples (must be equal to dim 1 of X), dim 2: time (must be equal to dim 2 of X), dim 3: variables (could be 1 or more, if a matrix, will be coerce to array) |
X |
array of input values, dim 1: samples, dim 2: time, dim 3: variables (could be 1 or more, if a matrix, will be coerce to array) |
model |
a model trained before, used for retraining purpose. |
learningrate |
learning rate to be applied for weight iteration |
learningrate_decay |
coefficient to apply to the learning rate at each epoch, via the epoch_annealing function |
momentum |
coefficient of the last weight iteration to keep for faster learning |
|
dimension(s) of hidden layer(s) |
|
network_type |
type of network, could be rnn, gru or lstm. gru and lstm are experimentale. |
numepochs |
number of iteration, i.e. number of time the whole dataset is presented to the network |
sigmoid |
method to be passed to the sigmoid function |
use_bias |
should the network use bias |
batch_size |
batch size: number of samples used at each weight iteration, only 1 supported for the moment |
seq_to_seq_unsync |
if TRUE, the network will be trained to backpropagate only the second half of the output error. If many to one is the target, just make Y have a time dim of 1. The X and Y data are modify at first to fit a classic learning, error are set to 0 during back propagation, input for the second part is also set to 0. |
update_rule |
rule to update the weight, "sgd", the default, is stochastic gradient descent, other available options are "adagrad" (experimentale, do not learn yet) |
epoch_function |
vector of functions to applied at each epoch loop. Use it to intereact with the objects inside the list model or to print and plot at each epoch. Should return the model. |
loss_function |
loss function, applied in each sample loop, vocabulary to verify. |
... |
Arguments to be passed to methods, to be used in user defined functions |
a model to be used by the predictr function
## Not run: # create training numbers X1 = sample(0:127, 10000, replace=TRUE) X2 = sample(0:127, 10000, replace=TRUE) # create training response numbers Y <- X1 + X2 # convert to binary X1 <- int2bin(X1, length=8) X2 <- int2bin(X2, length=8) Y <- int2bin(Y, length=8) # create 3d array: dim 1: samples; dim 2: time; dim 3: variables X <- array( c(X1,X2), dim=c(dim(X1),2) ) # train the model model <- trainr(Y=Y, X=X, learningrate = 1, hidden_dim = 16 ) ## End(Not run)## Not run: # create training numbers X1 = sample(0:127, 10000, replace=TRUE) X2 = sample(0:127, 10000, replace=TRUE) # create training response numbers Y <- X1 + X2 # convert to binary X1 <- int2bin(X1, length=8) X2 <- int2bin(X2, length=8) Y <- int2bin(Y, length=8) # create 3d array: dim 1: samples; dim 2: time; dim 3: variables X <- array( c(X1,X2), dim=c(dim(X1),2) ) # train the model model <- trainr(Y=Y, X=X, learningrate = 1, hidden_dim = 16 ) ## End(Not run)
Apply the update with adagrad, not working yet
update_adagrad(model)update_adagrad(model)
model |
the output model object |
the updated model
Apply the update
update_r(model)update_r(model)
model |
the output model object |
the updated model
Apply the update with stochastic gradient descent
update_sgd(model)update_sgd(model)
model |
the output model object |
the updated model