TODO:

~~1. Multi-Layer Neural Network Support (more than a single hidden network)~~ [Completed]

~~2. PPO Support (Unrealistic but will try!)~~ [On Hold again]

~~3. Simple DQN Support~~ [Completed]

~~4. Convolutional Neural Network~~ [Completed]

AI Algorithm for Godot 4

The goal of this project is to provide a variety of AI Algorithms in Godot 4 natively using GDscript.

https://github.com/user-attachments/assets/013f31c1-d4e4-429a-8ea1-62c760061884

Index

Simple Neural Network and Neural Net
Neural Network Advanced (Multi-Layered Neural Network)
Q-Learning Algorithm (and SARSA)
Minimax Algorithm
SDQN (Simple Deep Q-Network)
Convolutional Neural Network

Simple Neural Network and Neural Net Plugin for Godot 4

This part of the plugin allows you to create a Multi Layer Neural Network and also provides a NeuralNet by which you can easily automatically train the network (which can be found under Node2D Section in the add node window).
This plugin is intended for creating AIs that can complete a game level.

Rules to be followed if using Neural Net

If using Neural Net, the identifier or name of the variable of the Neural Network used in your code has to be nn. Like this:
```
var nn: NeuralNetwork
```
This is because the Neural Net only works when the Neural Network is named as nn.
If using Neural Net, make sure you do not assign your Neural Network Variable nn anything. All you are supposed to do is declare it like this:
```
var nn: NeuralNetwork
```
This is because the Neural Net depends on the fact that nn is not assigned anything.
When your AI or player has to be killed or removed, always use the queue_free() method. This is because the Neural Net relies on the signal emitted by the node when exiting the tree to receive the fitness and Neural Network of that node. Example:
```
Object.queue_free()
```

What each variable means and how to use them

Ai Scene: This is where you will assign the AI or Player scene by clicking on the drop down arrow on the right side, clicking quick load and selecting your scene.
Batch Size: This is the informal Batch Size of each generation. The actual batch size of each generation is emitted by the true_batch_size signal. This controls the base amount of AIs spawned.
Generation Delay: This is the time limit (in seconds) for any generation. Once a generation has lived longer than the amount specified in this, the generation is reset and the next generation comes.
Input Nodes: This is where the input nodes for the nn will be set. Input Nodes means how many different inputs will the nn receive.
Hidden Nodes: This is where the hidden nodes for the nn will be set. Hidden Nodes means how many nodes will process the data given by the input nodes. You should experiment with this amount.
Output Nodes: This is where you will set how many outputs you want to receive by the nn.
Random Population: This determines how many AIs with random nn will be spawned after the first generation (after the 0 generation). It is a good idea to set this to a value greater than 10 as it allows for more possibilities to be explored by the Neural Net.
Use Reproduction: This determines whether reproduction will also be used to create new AIs for the next generations. This enables for combination of different traits of different nns. However, you will most probably not need this as Random and Mutated Population will suffice.
Reproduced Population: If “Use Reproduction” is checked, this will determine how many AIs will be spawned with reproduced nns. Note: This value must always be greater than half of the value of Batch Size if you have checked “Use Reproduction” as true.

How to use Neural Net

Just ensure that all the variables/properties mentioned above are correctly set. The position of this node is where all the AIs will be spawned, meaning, the position of this node = position of AI when spawned.

How to use Neural Network

var nn: NeuralNetwork = NeuralNetwork.new(input_nodes, hidden_nodes, output_nodes)

Input Nodes: This is where the input nodes for the nn will be set. Input Nodes means how many different inputs will the nn receive.
Hidden Nodes: This is where the hidden nodes for the nn will be set. Hidden Nodes means how many nodes will process the data given by the input nodes. You should experiment with this amount.
Output Nodes: This is where you will set how many outputs you want to receive by the nn.
If the Neural Network depends mostly on inputs from raycasts, you can use the get_prediction_from_raycasts(optional_val: Array = []). This function returns an array of floats which are the outputs. The optional_val is optional can be used to give more custom inputs in addition to the raycasts. Example:
```
var output = nn.get_prediction_from_raycasts()

# or

var output = nn.get_prediction_from_raycasts([0, 0.4, 2])
```
You can use the predict(input_array: Array[float]) function also to get predictions. Example:
```
var output = nn.predict([0.0, 6, 0.2])
```

If you know the expected output of an input, you can use the train(input_array: Array, target_array: Array) function in a loop. Example:

for epoch in range(2000):
    nn.train([0, 1], [1])
    nn.train([1, 1], [1])
    nn.train([0, 0], [0])
    nn.train([1, 1], [0])

If you want to mutate your Neural Network, you can do so by:
```
nn = NeuralNetwork.mutate(nn)
```
where nn is your Neural Network.
If you want to copy your Neural Network, you can do so by:
```
new_nn = NeuralNetwork.copy(nn)
```
where nn is your Neural Network and new_nn is the new one to which you are copying the nn to.
If you want to reproduce your Neural Network with another, you can do so by:
```
reproduced_nn = NeuralNetwork.reproduce(nn_1, nn_2)
```
where nn_1 and nn_2 are the parent Neural Networks.

Neural Network Advanced

Note: Support for this in the Neural Net has not been implemented yet.

How to use NeuralNetworkAdvanced class

Initialising the NNA variable

var nnas: NeuralNetworkAdvanced = NeuralNetworkAdvanced.new()

Add the first layer to the network. Here you should only specify the number of nodes needed in this layer.
```
nnas.add_layer(2)
```
Add the remaining layers to the network. Here you can also specify which activation function to use. Eg:
```
nnas.add_layer(4, Activation.ARCTAN)
nnas.add_layer(6, Activation.ARCTAN)
nnas.add_layer(1, Activation.SIGMOID)
```
Currently available activation functions are listed in this file: Activation.gd
To train the network, you can call the train() function on the NNA. The first argument has to be the input array of size same as that of the first layer and the second argument has to be the output array of size same as the last layer of the network.
Note: This only runs a single train call. You need to do a lot of these to train your NNA to be accurate. Eg: Training for an XOR Gate. In the demo, you can see that this code is placed in the _physics_process function so that it is ran many times a second.
```
nnas.train([0,0], [0])
nnas.train([1,0], [1])
nnas.train([0,1], [1])
nnas.train([1,1], [0])
```
To get a prediction/output from the NNA. You have to call the predict function on the NNA. The first and only argument has to be input array for the network. It will return an array of the same size as that of the last/output layer. Eg:
```
print(nnas.predict([1,0]))
```
will return [1] when trained.

Configurable parameters in NeuralNetworkAdvanced

Learning Rate: Default value is 0.001

nnas.learning_rate = 0.001 # Or any other float

Backward Propagation Optimiser Method: Currently available methods are SGD (Scholastic Gradient Descent) [<- DEFAULT] and ADAM (Adaptive Moment Estimation). Has to be given as the argument during NeuralNetworkAdvanced.new() call.
```
var nnas := NeuralNetworkAdvanced.new(NeuralNetworkAdvanced.methods.SGD)
--or--
var nnas := NeuralNetworkAdvanced.new(NeuralNetworkAdvanced.methods.ADAM)
```

Clip Value: Only if optimiser method is chosen as SGD. Default value is INF.

nnas.clip_value = 10.0 # Or any other value you want to normalise/clip the gradients to

Use AmsGrad: Only if optimiser method is chosen as ADAM. Set to true if you want more stable training. Default value is false.
```
nnas.use_amsgrad = true
```

Additional Methods in NeuralNetworkAdvanced

copy(all: bool = false): Creates a deep copy of the neural network. If all is true, copies all properties; if false, copies only essential properties.
```
var copied_nna: NeuralNetworkAdvanced = nnas.copy()
```
to_dict(): Serializes the neural network to a dictionary. Used for saving the network state.
```
var data: Dictionary = nnas.to_dict()
```
from_dict(dict: Dictionary): Deserializes the neural network from a dictionary. Used for loading the network state.
```
nnas.from_dict(dict)
```
save(file_path: String): Saves the neural network state to a file.
```
nnas.save("res://path/to/save/file.json")
```

Note

Addition of layers should only happen once and so _ready() is an appropriate place to put them.
For more detailed ADAM hyperparameters, check Neural_Network_Advanced.gd

Q-Learning Algorithm

This algorithm implements Q-Learning algorithm using Q-Table natively in Godot.

How to use QLearning class

Initialise a QLearning variable
```
var qnet: QLearning = QLearning.new(observation_space, action_space, is_learning, not_sarsa)
```
Both the observation_space and action_space have to be natural numbers representing the possible states the agent can be in and the possible actions choices the agent can take in any given state. is_learning is a boolean value of whether the agent should be learning or not, and not_sarsa set to true will disable sarsa (on-policy). I would recommend sarsa if you want a safer route to the final path.
Get a prediction from the QLearning variable:
```
qnet.predict(current_state, reward_of_previous_state)
```
The above method returns an whole number that lies between 0 and action_space - 1. The value returned corresponds to an action the agent can take.
You can assign the returned value to variable by:
```
var action_to_do: int = qnet.predict(current_state, previous_reward)
```

Configurable Values

qnet.exploration_probability -> has to be a float value
Default Value: 1.0
The probability that the agent will take a random action or exploit the data it has learned.
Do not change unless you know what you are doing.
qnet.exploration_decreasing_decay -> has to be a float value
Default Value: 0.01
Changes how the value by which the qnet.exploration_probability decreases every qnet.decay_per_steps steps.
qnet.min_exploration_probability -> has to be a float value
Default Value: 0.01
The minimum value the exploration_probability can take.
qnet.learning_rate -> has to be a float
Default Value: 0.2
The rate at which the agent learns.
qnet.decay_per_steps -> has to be natural number
Default Value: 100
After how many steps does the qnet.exploration_probability decrease by qnet.exploration_decreasing_decay value.
qnet.is_learning -> has to be a bool value
Default Value: true
To be set to false only when the qnet.QTable.data is set manually.
print_debug_info -> has to be a bool value
Default Value: false
This can be set to true if you want to print debug info - total steps completed and current exploration probability - every qnet.decay_per_steps.

Things to keep in mind when using QLearning

The predict method of the QLearning class takes two compulsory arguments:
```
qnet.predict(current_state, previous_state_reward)
```
The current_state has to be a whole number representing the state it is currently in, while the previous_state_reward has to a float representing the reward it got for the previous action it took.

Minimax Algorithm

Alpha-Beta Pruning has been implemented!
If the AI is playing the role of the adversary, then set minimax.is_adversary to true else false.

How to use Minimax class

Initialise the Minimax class with 4 arguments:
```
var minimax: Minimax = Minimax.new(Callable(result), Callable(terminal), Callable(utility), Callable(possible_actions))
```
1. result_func: Callable: This callable argument must link to the function in your code that returns the state of the environment after a particular action is performed.
2. terminal_func: Callable: This callable argument must link to the function in your code that returns true if the game is over and false if the game can continue for a given state.
3. utility_func: Callable: This callable argument must link to the function in your code that returns the value of the given state. Currently this function only runs when the game is a terminal state. Losing states should have lesser value than winning states.
4. possible_actions_func: Callable: This callable argument must link to the function in your code that returns all the possible actions for a given state.
Every time the AI needs to perform an action, call the action(state) on the minimax variable.
```
var action_to_do: Array = minimax.action(_board)
```

Structure of the 4 arguments specified above

These functions have to be implemented by the user themselves as it is dependent on the game.

func result(state: Array, action: Array, is_adversary: bool) -> Array:
Should return the resultant state from performing the action.
func terminal(state: Array) -> bool:
Should return true if the no further action can take place, otherwise, it should return false.
func utility(state: Array, is_adversary: bool) -> float:
Should return the value of the given state. Usually positive for states in which the AI wins and negative for states in which the AI lose.
func possible_actions(state: Array) -> Array[Array]:
Should return all the possible actions that can happen in the given state. Each action is an array item inside the array that is being returned.

Look into the tictactoe demo to gain a better understanding.

SDQN (Simple Deep Q-Network)

This class implements a Simple Deep Q-Network (SDQN) for reinforcement learning in Godot using the NeuralNetworkAdvanced class.

How to use SDQN class

Initialise the SDQN class with state and action space dimensions:

var sdqn: SDQN = SDQN.new(state_space, action_space, learning_rate)

Set the Q-network for the SDQN:

var neural_network: NeuralNetworkAdvanced = NeuralNetworkAdvanced.new()
--code to change neural network properties--
sdqn.set_Q_network(neural_network)

Choose an action based on the current state using epsilon-greedy policy:
```
var action: int = sdqn.choose_action(state)
```
Add a new experience to the replay memory and train the network:
```
sdqn.add_memory(state, action, reward, next_state, done)
```

Save the SDQN model to a file:

sdqn.save("res://path/to/save/file.json")

Load the SDQN model from a file:

sdqn.load("res://path/to/save/file.json")

Configurable Parameters in SDQN

Discount Factor: Default value is 0.95

sdqn.discount_factor = 0.95  # Or any other float

Exploration Probability: Default value is 1

sdqn.exploration_probability = 1  # Or any other float

Minimum Exploration Probability: Default value is 0.01

sdqn.min_exploration_probability = 0.01  # Or any other float

Exploration Decay: Default value is 0.999

sdqn.exploration_decay = 0.999  # Or any other float

Batch Size: Default value is 196
```
sdqn.batch_size = 196  # Or any other integer
```
Batch size is the number of elements taken from memory to train on after max_steps steps have been done. Each call to add_memory is considered one step.

Max Steps: Default value is 128

sdqn.max_steps = 128  # Or any other integer

Target Update Frequency: Default value is 1024
```
sdqn.target_update_frequency = 1024  # Or any other integer
```
Target update frequency is the number of steps after which the target network is replaced with the current network.

Max Memory Size: Default value is 60 * 60 * 4

sdqn.max_memory_size = 60 * 60 * 4  # Or any other integer

Automatic Decay: Default value is false
```
sdqn.automatic_decay = false  # Or true
```
If automatic_decay is enabled, the current exploration probability and learning rate (if decaying) are multiplied with their respective decay values after max_steps steps.

Learning Rate Decay Rate: Default value is 1

sdqn.lr_decay_rate = 1  # Or any other float

Final Learning Rate: Default value is 0.0001

sdqn.final_learning_rate = 0.0001  # Or any other float

Use Multi-Threading: Default value is false. Currently not working 100% of the time.
```
sdqn.use_multi_threading = false  # Or true
```

Methods in SDQN

use_threading(): Enables multi-threading for training.
```
sdqn.use_threading()
```
set_Q_network(neural_network: NeuralNetworkAdvanced): Sets the Q-network for the SDQN.
```
sdqn.set_Q_network(neural_network)
```
set_clip_value(clip_value: float): Sets the clip value for the Q-network.
```
sdqn.set_clip_value(clip_value)
```
set_lr_value(lr: float): Sets the learning rate for the Q-network.
```
sdqn.set_lr_value(lr)
```
update_lr_linearly(): Updates the learning rate linearly.
```
sdqn.update_lr_linearly()
```
choose_action(state: Array): Chooses an action based on the current state using epsilon-greedy policy.
```
var action: int = sdqn.choose_action(state)
```
add_memory(state: Array, action: int, reward: float, next_state: Array, done: bool): Adds a new experience to the replay memory.
```
sdqn.add_memory(state, action, reward, next_state, done)
```
close_threading(): Closes the multi-threading.
```
sdqn.close_threading()
```
copy(): Copies the SDQN instance.
```
var copied_sdqn: SDQN = sdqn.copy()
```
to_dict(): Converts the SDQN instance to a dictionary.
```
var data: Dictionary = sdqn.to_dict()
```
from_dict(dict: Dictionary): Loads the SDQN instance from a dictionary.
```
sdqn.from_dict(dict)
```
save(file_path: String): Saves the SDQN instance to a file.
```
sdqn.save("res://path/to/save/file.json")
```
load(file_path: String): Loads the SDQN instance from a file.
```
sdqn.load("res://path/to/save/file.json")
```

Note

To change the clip value, learning rate, and Q-network, the respective functions (set_clip_value, set_lr_value, set_Q_network) must be called instead of directly setting the variable.
To learn more, check out the SDQN Demo available in the Demos folder.

SDQN Node

The SDQN_Node can be added to the agent node by searching for "Simple DQN" under the nodes list in the "Add Node" window. All the customizable parameters can be set using the Inspector tab to the right. If you are installing this addon from github, you will have to take the addon folder and drop it into your games main folder, then enable the addon in project settings and reload the project/editor.

You still need to call set_Q_network on the node to set the Q_network using your own NeuralNetworkAdvanced instance.

Example of referencing this node:

@onready var DQN = $"Simple DQN"

# Set the Q_network
DQN.set_q_network(your_neural_network)

# Use the DQN node
DQN.add_memory(state, action, reward, next_state, done)
var action = DQN.choose_action(state)

Convolutional Neural Network

The CNN class implements a Convolutional Neural Network (CNN) for image classification in Godot using GDScript.

Check out the cnn_demo.tscn and cnn_demo.gd to understand how everything works.

How to use CNN class

Initialize the CNN variable:
```
var cnn: CNN = CNN.new()
```
Set the learning rate and add labels:
```
cnn.learning_rate = 0.001
cnn.add_labels(["O", "X"])
```
The number of labels added should be the same as the number of nodes assigned to SoftmaxDense layer.

Add layers to the CNN:

cnn.add_layer(cnn.Layer.MutliFilterConvolutional1D.new(8, "same"))
cnn.add_layer(cnn.Layer.MultiPoolPooling.new())
cnn.add_layer(cnn.Layer.Dropout.new(0.1))

cnn.add_layer(cnn.Layer.MutliFilterConvolutional1D.new(4, "same"))
cnn.add_layer(cnn.Layer.MultiPoolPooling.new())
cnn.add_layer(cnn.Layer.Dropout.new(0.1))

cnn.add_layer(cnn.Layer.Flatten.new())

cnn.add_layer(cnn.Layer.Dense.new(64, "RELU"))
cnn.add_layer(cnn.Layer.Dropout.new(0.1))

cnn.add_layer(cnn.Layer.SoftmaxDense.new(2))

Compile the network with input dimensions and optimizer:
```
cnn.compile_network(Vector2i(28, 28), cnn.optimizers.AMSGRAD)
```
Note: The network must be compiled after adding all the layers. The input dimensions has to be the same as the number of pixels in the input image / data.
Train the network with input data and labels:
```
var loss: float = cnn.train(input_data, "O")
```
The input_data has to be a Matrix of the input that is been fed forward. The label has to match with one of the labels added using add_labels.

This function returns the training loss of that step.
Categorize new input data:
```
var prediction: String = cnn.categorise(input_data)
```
Note: The input to the CNN must be a Matrix. This is automatically handled by using the ImageHelper class.

Layer Types

MutliFilterConvolutional2D: Applies multiple convolutional filters to the input data.
- Arguments:
  - num_filters (int): Number of filters to apply.
  - padding_mode (String): Padding mode, either "same" (output size is the same as input size) or "valid" (no padding).
  - filter_shape (Vector2i): Shape of the convolutional filters.
  - stride (int): Stride of the convolution.
MultiPoolPooling: Applies max pooling to the input data.
- Arguments:
  - padding (int): Padding to apply before pooling.
  - stride (int): Stride of the pooling operation.
  - pool_size (Vector2i): Size of the pooling window.
Dropout: Applies dropout to the input data to prevent overfitting.
- Arguments:
  - rate (float): Dropout rate (fraction of inputs to drop).
Flatten: Flattens the input data to a 1D array.
Dense: Fully connected layer with specified activation function.
- Arguments:
  - output_shape (int): Number of neurons in the layer.
  - activation (String): Activation function to use (e.g., "RELU", "SIGMOID").
SoftmaxDense: Fully connected layer with softmax activation for classification.
- Arguments:
  - output_shape (int): Number of neurons in the layer. Should match the number of labels added.
BatchNormalization: Normalizes the input data to have zero mean and unit variance.
- Arguments:
  - epsilon (float): Small constant to avoid division by zero.
Note: DOES NOT WORK

Optimizers

SGD: Stochastic Gradient Descent.
SGD_MOMENTUM: Stochastic Gradient Descent with Momentum.
ADAM: Adaptive Moment Estimation.
AMSGRAD: AMSGrad variant of ADAM. (Default Optimizer)

Image Augmentations

The Augmentations class provides utility functions for augmenting images to improve the robustness of the CNN model.

Available Augmentations

rotate_random: Rotates the image by a random angle between 0 and 360 degrees.
flip_horizontal: Flips the image horizontally.
flip_vertical: Flips the image vertically.

How to use Augmentations

Add an augmentation to the CNN:

cnn.add_augmentation(Augmentations.rotate_random)

You can add multiple augmentations:

cnn.add_augmentation(Augmentations.flip_horizontal)
cnn.add_augmentation(Augmentations.flip_vertical)

The augmentations will be applied during the training process to the input data.

ImageHelper

The ImageHelper class provides utility functions for loading grayscale images from files and folders.

How to use ImageHelper class

Load a single grayscale image:

var image_data: Matrix = ImageHelper.load_grayscale_image_data("res://path/to/image.png")

Load multiple grayscale images from a folder:

var images: Array[Matrix] = ImageHelper.load_grayscale_images_from_folder("res://path/to/folder/")

Godot AI Kit

AI Algorithm for Godot 4

Index

Simple Neural Network and Neural Net Plugin for Godot 4

Rules to be followed if using Neural Net

What each variable means and how to use them

How to use Neural Net

How to use Neural Network

Neural Network Advanced

How to use NeuralNetworkAdvanced class

Configurable parameters in NeuralNetworkAdvanced

Additional Methods in NeuralNetworkAdvanced

Note

Q-Learning Algorithm

How to use QLearning class

Configurable Values

Things to keep in mind when using QLearning

Minimax Algorithm

How to use Minimax class

Structure of the 4 arguments specified above

SDQN (Simple Deep Q-Network)

How to use SDQN class

Configurable Parameters in SDQN

Methods in SDQN

Note

SDQN Node

Convolutional Neural Network

How to use CNN class

Layer Types

Optimizers

Image Augmentations

Available Augmentations

How to use Augmentations

ImageHelper

How to use ImageHelper class

Version

Engine

Category

Repository

Download

Support