AIToolbox
A library that offers tools for AI problem solving.
AIToolbox::MDP::TreeBackupLEvaluation Class Reference

This class implements off-policy evaluation via Tree Backup(lambda). More...

#include <AIToolbox/MDP/Algorithms/TreeBackupL.hpp>

Inheritance diagram for AIToolbox::MDP::TreeBackupLEvaluation:
AIToolbox::MDP::OffPolicyEvaluation< TreeBackupLEvaluation > AIToolbox::MDP::OffPolicyBase

Public Types

using Parent = OffPolicyEvaluation< TreeBackupLEvaluation >
 
- Public Types inherited from AIToolbox::MDP::OffPolicyEvaluation< TreeBackupLEvaluation >
using Parent = OffPolicyBase
 
- Public Types inherited from AIToolbox::MDP::OffPolicyBase
using Trace = std::tuple< size_t, size_t, double >
 
using Traces = std::vector< Trace >
 

Public Member Functions

 TreeBackupLEvaluation (const PolicyInterface &target, const double discount, const double alpha, const double lambda, const double tolerance)
 Basic constructor. More...
 
void setLambda (double l)
 This function sets the new lambda parameter. More...
 
double getLambda () const
 This function returns the currently set lambda parameter. More...
 
- Public Member Functions inherited from AIToolbox::MDP::OffPolicyEvaluation< TreeBackupLEvaluation >
 OffPolicyEvaluation (const PolicyInterface &target, double discount=1.0, double alpha=0.1, double tolerance=0.001)
 Basic constructor. More...
 
void stepUpdateQ (const size_t s, const size_t a, const size_t s1, const double rew)
 This function updates the internal QFunction using the discount set during construction. More...
 
- Public Member Functions inherited from AIToolbox::MDP::OffPolicyBase
 OffPolicyBase (size_t s, size_t a, double discount=1.0, double alpha=0.1, double tolerance=0.001)
 Basic construtor. More...
 
void setLearningRate (double a)
 This function sets the learning rate parameter. More...
 
double getLearningRate () const
 This function will return the current set learning rate parameter. More...
 
void setDiscount (double d)
 This function sets the new discount parameter. More...
 
double getDiscount () const
 This function returns the currently set discount parameter. More...
 
void setTolerance (double t)
 This function sets the trace cutoff parameter. More...
 
double getTolerance () const
 This function returns the currently set trace cutoff parameter. More...
 
void clearTraces ()
 This function clears the already set traces. More...
 
const TracesgetTraces () const
 This function returns the currently set traces. More...
 
void setTraces (const Traces &t)
 This function sets the currently set traces. More...
 
size_t getS () const
 This function returns the number of states on which QLearning is working. More...
 
size_t getA () const
 This function returns the number of actions on which QLearning is working. More...
 
const QFunctiongetQFunction () const
 This function returns a reference to the internal QFunction. More...
 
void setQFunction (const QFunction &qfun)
 This function allows to directly set the internal QFunction. More...
 

Additional Inherited Members

- Protected Member Functions inherited from AIToolbox::MDP::OffPolicyBase
void updateTraces (size_t s, size_t a, double error, double traceDiscount)
 This function updates the traces using the input data. More...
 
- Protected Attributes inherited from AIToolbox::MDP::OffPolicyEvaluation< TreeBackupLEvaluation >
const PolicyInterfacetarget_
 
- Protected Attributes inherited from AIToolbox::MDP::OffPolicyBase
size_t S
 
size_t A
 
double discount_
 
double alpha_
 
double tolerance_
 
QFunction q_
 
Traces traces_
 

Detailed Description

This class implements off-policy evaluation via Tree Backup(lambda).

This algorithm tries to avoid the infinite variance problem that ImportanceSampling has, by multiplying the traces by just the target policy probability. It additionally uses the lambda parameter to further tune their length.

While it succeeds in its intent, it tends to cut traces short. This happens since all actions taken by a policy have a <= 1 probability of being picked, which generally shortens the trace. While not overall a problem, this is inefficient in case the behaviour and target policies are very similar.

Member Typedef Documentation

◆ Parent

Constructor & Destructor Documentation

◆ TreeBackupLEvaluation()

AIToolbox::MDP::TreeBackupLEvaluation::TreeBackupLEvaluation ( const PolicyInterface target,
const double  discount,
const double  alpha,
const double  lambda,
const double  tolerance 
)
inline

Basic constructor.

Parameters
targetTarget policy.
discountDiscount for the problem.
alphaLearning rate parameter.
lambdaLambda trace parameter.
toleranceTrace cutoff parameter.

Member Function Documentation

◆ getLambda()

double AIToolbox::MDP::TreeBackupLEvaluation::getLambda ( ) const
inline

This function returns the currently set lambda parameter.

◆ setLambda()

void AIToolbox::MDP::TreeBackupLEvaluation::setLambda ( double  l)
inline

This function sets the new lambda parameter.

The lambda parameter must be >= 0.0 and <= 1.0, otherwise the function will throw an std::invalid_argument.

Parameters
lThe new lambda parameter.

The documentation for this class was generated from the following file: