Wasserstein Distance, Contraction Mapping, and Modern RL Theory

Classic Math Formulations And Their Impact on Modern RL

Eq1: Normal Bellman operator B
Eq2: Distributional Bellman operator Ⲧπ

What is Wasserstein Distance

Earth Mover’s distance, Image by Author
Eq3: Wasserstein metric
Eq4: Wasserstein metric

Example

Why Wasserstein Distance

Python generated examples, Image by Author

ɣ-contraction

Contraction Mapping

Eq5: Contraction Mapping

Contraction in RL

Eq6: ɣ-contraction

Proof

Image by Autho

Conclusion

References

RL | ML | ALGO TRADING | TRANSPORTATION | GAME THEORY