{ "cells": [ { "cell_type": "markdown", "metadata": { "toc": "true" }, "source": [ "

Table of Contents

\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# quick start " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Test what is the decoding change for single size training " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Resources\n", "\n", "* [*The* Reinforcement learning book from Sutton & Barto](http://incompleteideas.net/sutton/book/the-book-2nd.html)\n", "* [The REINFORCE paper from Ronald J. Williams (1992)](http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# FULL MODEL" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Packages" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# posnet without regulization \n", "\n", "%load_ext autoreload\n", "%autoreload 2\n", "\n", "import numpy as np\n", "from itertools import count\n", "import random\n", "\n", "import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", "import torch.optim as optim\n", "import torch.autograd as autograd\n", "from torch.autograd import Variable\n", "from torch.nn import init\n", "from torch.nn import DataParallel\n", "\n", "import matplotlib.mlab as mlab\n", "import matplotlib.pyplot as plt\n", "import matplotlib.animation\n", "from IPython.display import HTML\n", "\n", "import POMDPgame_bars\n", "from POMDPgame_bars import*\n", "\n", "import POMDPgame_basic\n", "from POMDPgame_basic import*\n", "\n", "import POMDPgame_holes\n", "from POMDPgame_holes import*\n", "\n", "\n", "import RNN\n", "from RNN import *\n", "\n", "import navigation2\n", "from navigation2 import*\n", "\n", "import Nets\n", "from Nets import*\n", "\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Qnetwork\n", "\n", "To select actions we take maximum of Q value, corresponding to certain move." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For the liquid state approach to work, you need a lot of neurons as surplus or enough hidden to hidden connectivity to make it have an effect." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## POMDP RNN Game\n", "\n", "In this game , we use a new reward function determined by game, if the agent achieves the goal before 50, reward is 1. If time pass 50 reward is 0.5, once time pass 100 agent gets a reward of -0.5 . Practically, this is found to be easier to learn than the rewards as a continous function of time. Tf the agent learns to search in a efficient way, the largest possible way for search is to firstly arrive at corner then goes to the goal, which, takes about 50 steps, it is reasonble to make 50 and 100 as milestone thing. Also in principe as the game doesn't have a timer , it is not if it can use a reward as funtion of time" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "3 condition for ending , when pass time limit, game over" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For weight update, it seems to be better do it after episode, as it makes non-sense evaluate strategy during episode, but a the end. Also, it is much quicker. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A programming of MDP here, hidden state is as state of enviroment" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQQAAAECCAYAAAAYUakXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAACG1JREFUeJzt3U+opXd9x/HPtyZOMLiIxIRQRSSG\ngpsO5RILgkREiW6ii9JmlYUwFpqNu+x06UayKpaRhmRjbDfBLMQaZpONiiMEO4WWpJK2Y0Kmkk0p\nNMb47SJnvo5xxjve8+e5957XCy7nnGeemfP9zYU3z3Puc86t7g5AkvzR0gMAx4cgAEMQgCEIwBAE\nYAgCMBYNQlU9WFX/VlUvVdVjS86ybVX1clX9c1W9UFUXl55nk6rqiaq6UlWXrtn2vqp6rqpeXN3e\nseSMm3KDtX61qn6++t6+UFWfW3LGdSwWhKp6V5K/TfLZJB9N8nBVfXSpeXbkk919trsPlh5kw55M\n8uA7tj2W5EJ335fkwurxafBkfnetSfL46nt7tru/u+OZNmbJI4T7k7zU3T/r7l8m+XaShxachyPq\n7ueTvP6OzQ8leWp1/6kkn9/pUFtyg7WeGksG4Y+T/Nc1jy+vtp1WneT7VfWTqjq39DA7cHd3v5ok\nq9u7Fp5n2x6tqp+uTilO7OnRkkGo62w7zddRf7y7/yxvnyL9TVV9YumB2JhvJLk3ydkkryb5+rLj\nHN2SQbic5IPXPP5AklcWmmXruvuV1e2VJM/k7VOm0+y1qronSVa3VxaeZ2u6+7Xufqu7f53kmznB\n39slg/DjJPdV1Yer6t1J/irJswvOszVVdXtVvffq/SSfSXLp9/+tE+/ZJI+s7j+S5DsLzrJVV8O3\n8oWc4O/tLUs9cXf/qqoeTfJPSd6V5Inu/pel5tmyu5M8U1XJ2//n3+ru7y070uZU1dNJHkhyZ1Vd\nTvKVJF9L8o9V9cUk/5nkL5abcHNusNYHqups3j7lfTnJlxYbcE3l7c/AVa5UBIYgAEMQgCEIwBAE\nYByLIOzJpbxJ9met+7LO5HSt9VgEIcmp+Q+9Cfuy1n1ZZ3KK1npcggAcAzu9MOnddaZvy+2/s/3N\nvJFbc2ZncyxpX9a6L+tMTsZa/y//m1/2G9d7Q+Fv2emly7fl9nysPrXLpwSS/Kgv3NR+a50y7NNH\noME+OHIQ9vQj0OBUW+cIwUegwSmzThD27SPQ4NRb50XFm/oItNVFG+eS5La8Z42nA7ZtnSOEm/oI\ntO4+390H3X1w3H80A/tunSDszUegwb448inDnn0EGuyFtS5MWv2GmhP7W2qA3+a9DMAQBGAIAjAE\nARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGDv91OVte+nxP196BDiyj3z5h0uP4AgB+A1BAIYgAEMQ\ngCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIAhCMAQBGAIAjAE\nARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAIQjAuGWdv1xVLyf5nyRvJflVdx9s\nYihgGWsFYeWT3f2LDfw7wMKcMgBj3SB0ku9X1U+q6twmBgKWs+4pw8e7+5WquivJc1X1r939/LU7\nrEJxLkluy3vWfDpgm9Y6QujuV1a3V5I8k+T+6+xzvrsPuvvg1pxZ5+mALTtyEKrq9qp679X7ST6T\n5NKmBgN2b51ThruTPFNVV/+db3X39zYyFbCIIwehu3+W5E83OAuwMD92BIYgAEMQgCEIwBAEYAgC\nMAQBGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIAhCMAQBGAIAjAEARiCAAxBAIYg\nAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAccvSA5xW//6Xf/cH7X/vP/z1liaBm+cIARiCAAxB\nAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhvcybIn3JnASOUIAhiAA49AgVNUTVXWlqi5ds+19VfVc\nVb24ur1ju2MCu3AzRwhPJnnwHdseS3Khu+9LcmH1GDjhDg1Cdz+f5PV3bH4oyVOr+08l+fyG5wIW\ncNTXEO7u7leTZHV71412rKpzVXWxqi6+mTeO+HTALmz9RcXuPt/dB919cGvObPvpgDUcNQivVdU9\nSbK6vbK5kYClHDUIzyZ5ZHX/kSTf2cw4wJJu5seOTyf5QZI/qarLVfXFJF9L8umqejHJp1ePgRPu\n0EuXu/vhG/zRpzY8C7AwVyoCQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIAhCMAQBGAIAjAEARiC\nAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIAh\nCMAQBGAIAjAEARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAA49AgVNUTVXWlqi5ds+2r\nVfXzqnph9fW57Y4J7MLNHCE8meTB62x/vLvPrr6+u9mxgCUcGoTufj7J6zuYBVjYOq8hPFpVP12d\nUtyxsYmAxRw1CN9Icm+Ss0leTfL1G+1YVeeq6mJVXXwzbxzx6YBdOFIQuvu17n6ru3+d5JtJ7v89\n+57v7oPuPrg1Z446J7ADRwpCVd1zzcMvJLl0o32Bk+OWw3aoqqeTPJDkzqq6nOQrSR6oqrNJOsnL\nSb60xRmBHTk0CN398HU2//0WZgEW5kpFYAgCMAQBGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggCM\nQ9/cdJJ85Ms/XHoEONEcIQBDEIAhCMAQBGAIAjAEARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIA\nDEEAhiAAQxCAIQjAEARgCAIwBAEYggCM6u7dPVnVfyf5j+v80Z1JfrGzQZa1L2vdl3UmJ2OtH+ru\n9x+2006DcMMhqi5298HSc+zCvqx1X9aZnK61OmUAhiAA47gE4fzSA+zQvqx1X9aZnKK1HovXEIDj\n4bgcIQDHgCAAQxCAIQjAEARg/D+FDrP+euJ/tgAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQQAAAECCAYAAAAYUakXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAACPpJREFUeJzt3U+opfV9x/HPt6lRIlkYrCJtKcVK\nIZsO5WIDgWIICTYbk0VpXbkITAp10527ZJlNcFVSDBHdxLYbiYuQRmbjpi2dgKRTaNEG204Up8FN\nKdQY8+3C4zczZiZz554/zz33vF4wnHMej3O+zz3w5nnO/J5zq7sDkCS/svQAwOkhCMAQBGAIAjAE\nARiCAIxFg1BVD1fVv1XVq1X1xJKzbFtVvVZV/1xVL1fVxaXn2aSqerqqrlTVpau2fayqXqyqV1a3\ndy0546bcYF+/UlU/Wr23L1fV55accR2LBaGqPpTkL5P8UZKPJ3m0qj6+1Dw78qnuPtfdR0sPsmHP\nJHn4A9ueSHKhux9IcmH1+Cx4Jr+4r0ny5Oq9Pdfd39nxTBuz5BHCg0le7e4fdvdPkvx1kkcWnIcT\n6u6Xkrz1gc2PJHl2df/ZJJ/f6VBbcoN9PTOWDMKvJ/mvqx5fXm07qzrJ96rq+1V1fulhduDe7n4j\nSVa39yw8z7Y9XlU/WJ1S7O3p0ZJBqOtsO8vrqD/Z3b+f906R/ryq/nDpgdiYrye5P8m5JG8k+dqy\n45zckkG4nOQ3r3r8G0leX2iWrevu11e3V5I8n/dOmc6yN6vqviRZ3V5ZeJ6t6e43u/vd7v5Zkm9k\nj9/bJYPwT0keqKrfrqoPJ/nTJC8sOM/WVNWdVfXR9+8n+WySS7/8/9p7LyR5bHX/sSTfXnCWrXo/\nfCtfyB6/t7+61At390+r6vEkf5fkQ0me7u5/WWqeLbs3yfNVlbz3M/9Wd3932ZE2p6qeS/JQkrur\n6nKSLyf5apK/raovJvnPJH+83ISbc4N9faiqzuW9U97XknxpsQHXVC5/Bt5npSIwBAEYggAMQQCG\nIADjVAThQJbyJjmcfT2U/UzO1r6eiiAkOTM/0GM4lH09lP1MztC+npYgAKfAThcmfbhu7zty5y9s\nfydv57bcvrM5lnQo+3oo+5nsx77+X/43P+m3r3dB4TV2unT5jtyZP6hP7/IlgST/2BeO9by1ThkO\n6SvQ4BCcOAgH+hVocKatc4TgK9DgjFknCIf2FWhw5q3zoeKxvgJttWjjfJLckY+s8XLAtq1zhHCs\nr0Dr7qe6+6i7j077P83AoVsnCAfzFWhwKE58ynBgX4EGB2GthUmr31Czt7+lBriWaxmAIQjAEARg\nCAIwBAEYggAMQQCGIABDEIAhCMAQBGAIAjB2+q3L2/bqk59YegQ4sd/5i39YegRHCMDPCQIwBAEY\nggAMQQCGIABDEIAhCMAQBGAIAjDO1NLlbbrVZaWWUS/D+7QeRwjAEARgCAIwBAEYggAMQQCGIABD\nEIAhCMAQBGAIAjBcy3BM1rzvB+/TehwhAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAIQjAcC0D\np5rfs7BbjhCAIQjAWOuUoapeS/I/Sd5N8tPuPtrEUMAyNvEZwqe6+8cb+HuAhTllAMa6Qegk36uq\n71fV+U0MBCxn3VOGT3b361V1T5IXq+pfu/ulq5+wCsX5JLkjH1nz5YBtWusIobtfX91eSfJ8kgev\n85ynuvuou49uy+3rvBywZScOQlXdWVUfff9+ks8mubSpwYDdW+eU4d4kz1fV+3/Pt7r7uxuZCljE\niYPQ3T9M8nsbnAVYmGsZONVcm7Bb1iEAQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIAhCMBwLQPc\ngrP+eyIcIQBDEIAhCMAQBGAIAjAEARiCAAxBAIYgAEMQgGHp8gE468ttr7btfd3nn81xOEIAhiAA\nQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIDhWoYt+fc/+atbev79f/NnW5rk9K2/v5XrDVxrsFuO\nEIAhCMAQBGAIAjAEARiCAAxBAIYgAEMQgCEIwBAEYLiWYUu2eW3Cvv+ehdM2Dz/nCAEYggCMmwah\nqp6uqitVdemqbR+rqher6pXV7V3bHRPYheMcITyT5OEPbHsiyYXufiDJhdVjYM/dNAjd/VKStz6w\n+ZEkz67uP5vk8xueC1jAST9DuLe730iS1e09N3piVZ2vqotVdfGdvH3ClwN2YesfKnb3U9191N1H\nt+X2bb8csIaTBuHNqrovSVa3VzY3ErCUkwbhhSSPre4/luTbmxkHWNJx/tnxuSR/n+R3q+pyVX0x\nyVeTfKaqXknymdVjYM/ddOlydz96g//06Q3PAizMtQx76LRdC7DP11bs8+zbYOkyMAQBGIIADEEA\nhiAAQxCAIQjAEARgCAIwBAEYli6ztn1ezrvPs2+DIwRgCAIwBAEYggAMQQCGIABDEIAhCMAQBGAI\nAjAEARiuZTgmX9d9Nnlfr+UIARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhmsZjuk0rWG3\n/n5z/Gyu5QgBGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggAMQQCGaxn20L6vv7+VazFO276e9etI\nHCEA46ZBqKqnq+pKVV26attXqupHVfXy6s/ntjsmsAvHOUJ4JsnD19n+ZHefW/35zmbHApZw0yB0\n90tJ3trBLMDC1vkM4fGq+sHqlOKujU0ELOakQfh6kvuTnEvyRpKv3eiJVXW+qi5W1cV38vYJXw7Y\nhRMFobvf7O53u/tnSb6R5MFf8tynuvuou49uy+0nnRPYgRMFoaruu+rhF5JcutFzgf1x04VJVfVc\nkoeS3F1Vl5N8OclDVXUuSSd5LcmXtjgjsCM3DUJ3P3qdzd/cwizAwqxUBIZrGdi5fVvff7V9nv04\nHCEAQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIBxppYu3+pXZAPXcoQADEEAhiAAQxCAIQjAEARg\nCAIwBAEYggAMQQCGIABDEIAhCMAQBGAIAjAEARiCAAxBAIYgAEMQgCEIwBAEYAgCMKq7d/diVf+d\n5D+u85/uTvLjnQ2yrEPZ10PZz2Q/9vW3uvvXbvaknQbhhkNUXezuo6Xn2IVD2ddD2c/kbO2rUwZg\nCAIwTksQnlp6gB06lH09lP1MztC+norPEIDT4bQcIQCngCAAQxCAIQjAEARg/D/5avmjXQ1IFwAA\nAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQQAAAECCAYAAAAYUakXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAACIBJREFUeJzt3U+opXd9x/HPtyZOMLiIpAmhikgM\nBTcd5JIKgkREiW6ii9JmlYUwFszGXXa6dCNZiWXEkGyMuglmIdYwm2y0dIRgp9CSVKIdEzKVbKTQ\nGOO3i5z5OiYzznjPn+fee14vuJxznjkz5/ubC2+e59znPLe6OwBJ8hdLDwAcHYIADEEAhiAAQxCA\nIQjAWDQIVXV/Vf1nVb1QVY8sOcu2VdWLVfVvVfVcVZ1fep5NqqrHqupSVV24Ytt7quqZqnp+dXvb\nkjNuyjXW+pWq+tXqe/tcVX1myRnXsVgQquodSb6e5NNJPpTkwar60FLz7MjHu/t0dx8sPciGPZ7k\n/rdseyTJue6+J8m51eOT4PG8fa1J8ujqe3u6u3+w45k2Zsk9hHuTvNDdP+/u3yb5TpIHFpyHQ+ru\nZ5O8+pbNDyR5YnX/iSSf3elQW3KNtZ4YSwbhr5L89xWPL662nVSd5EdV9dOqOrP0MDtwZ3e/nCSr\n2zsWnmfbHq6qn60OKY7t4dGSQairbDvJ51F/tLs/nDcPkb5YVR9beiA25htJ7k5yOsnLSb627DiH\nt2QQLiZ53xWP35vkpYVm2brufml1eynJU3nzkOkke6Wq7kqS1e2lhefZmu5+pbvf6O7fJ/lmjvH3\ndskg/GuSe6rqA1X1ziT/kOTpBefZmqq6tareffl+kk8lufCn/9ax93SSh1b3H0ry/QVn2arL4Vv5\nXI7x9/ampV64u39XVQ8n+eck70jyWHf/+1LzbNmdSZ6qquTN//Nvd/cPlx1pc6rqyST3Jbm9qi4m\n+XKSryb5XlV9Pskvk/zdchNuzjXWel9Vnc6bh7wvJvnCYgOuqXz8GbjMmYrAEARgCAIwBAEYggCM\nIxGEPTmVN8n+rHVf1pmcrLUeiSAkOTH/oTdgX9a6L+tMTtBaj0oQgCNgpycmvbNO9S259W3bX89r\nuTmndjbHkvZlrfuyzuR4rPX/8r/5bb92tQ8U/pGdnrp8S27N39YndvmSQJJ/6XM39Ly1Dhn26RJo\nsA8OHYQ9vQQanGjr7CG4BBqcMOsEYd8ugQYn3jpvKt7QJdBWJ22cSZJb8q41Xg7YtnX2EG7oEmjd\nfba7D7r74Kj/aAb23TpB2JtLoMG+OPQhw55dAg32wlonJq1+Q82x/S01wB/zWQZgCAIwBAEYggAM\nQQCGIABDEIAhCMAQBGAIAjAEARiCAIydXnV521549CNLjwCH9sEv/WTpEewhAH8gCMAQBGAIAjAE\nARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggAMQQCGIABD\nEIAhCMAQBGAIAjAEARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGDet85er6sUkv0nyRpLfdffBJoYC\nlrFWEFY+3t2/3sC/AyzMIQMw1g1CJ/lRVf20qs5sYiBgOeseMny0u1+qqjuSPFNV/9Hdz175hFUo\nziTJLXnXmi8HbNNaewjd/dLq9lKSp5Lce5XnnO3ug+4+uDmn1nk5YMsOHYSqurWq3n35fpJPJbmw\nqcGA3VvnkOHOJE9V1eV/59vd/cONTAUs4tBB6O6fJ/mbDc4CLMyPHYEhCMAQBGAIAjAEARiCAAxB\nAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIAhCMAQ\nBGAIAjAEARiCAAxBAIYgAEMQgCEIwBAEYNy09AAn1X/9/T/9Wc+/+7v/uKVJ4MbZQwCGIABDEIAh\nCMAQBGAIAjAEARiCAAxBAIYgAEMQgOGzDFviswkcR/YQgCEIwLhuEKrqsaq6VFUXrtj2nqp6pqqe\nX93ett0xgV24kT2Ex5Pc/5ZtjyQ51933JDm3egwcc9cNQnc/m+TVt2x+IMkTq/tPJPnshucCFnDY\n9xDu7O6Xk2R1e8e1nlhVZ6rqfFWdfz2vHfLlgF3Y+puK3X22uw+6++DmnNr2ywFrOGwQXqmqu5Jk\ndXtpcyMBSzlsEJ5O8tDq/kNJvr+ZcYAl3ciPHZ9M8uMkf11VF6vq80m+muSTVfV8kk+uHgPH3HVP\nXe7uB6/xR5/Y8CzAwpypCAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAIQjAEARgCAIw\nBAEYggAMQQCGIABDEIAhCMAQBGAIAjAEARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAA\nQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIAhCMAQBGAIAjBuWnqApXzwSz9ZegSOgBce/cjSIxwp\n9hCAcd0gVNVjVXWpqi5cse0rVfWrqnpu9fWZ7Y4J7MKN7CE8nuT+q2x/tLtPr75+sNmxgCVcNwjd\n/WySV3cwC7Cwdd5DeLiqfrY6pLhtYxMBizlsEL6R5O4kp5O8nORr13piVZ2pqvNVdf71vHbIlwN2\n4VBB6O5XuvuN7v59km8mufdPPPdsdx9098HNOXXYOYEdOFQQququKx5+LsmFaz0XOD6ue2JSVT2Z\n5L4kt1fVxSRfTnJfVZ1O0kleTPKFLc4I7Mh1g9DdD15l87e2MAuwMGcqAmNvP8vgHHZ4O3sIwBAE\nYAgCMAQBGIIADEEAhiAAQxCAIQjAEARgnKhTl11aHdZjDwEYggAMQQCGIABDEIAhCMAQBGAIAjAE\nARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAIQjAEARgVHfv7sWq/ifJL67yR7cn\n+fXOBlnWvqx1X9aZHI+1vr+7//J6T9ppEK45RNX57j5Yeo5d2Je17ss6k5O1VocMwBAEYByVIJxd\neoAd2pe17ss6kxO01iPxHgJwNByVPQTgCBAEYAgCMAQBGIIAjP8H6kW4pR0J7KIAAAAASUVORK5C\nYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQQAAAECCAYAAAAYUakXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAC39JREFUeJzt3X/sXXV9x/Hna7S0FiXANo1SMnAS\nNkPcMN9sqItbrCYMCfjHkmHG0k2TZsk2kZgohD/M/luicZpskTSAkklwCeIkRB0NasySQVZ+hBXK\nBH8MqtWymKnBDNr43h/f23dK09LunnvPuaTPR9J87709fN+f75f22XPuPd97UlVIEsAvTb0ASavD\nIEhqBkFSMwiSmkGQ1AyCpLYSQUhyWZL/TPJUkutHnn1ekq8n2ZvksSTXjjn/iHWcluThJPdMMPus\nJHcmeWL2fXjLyPOvm33v9yS5I8nmJc+7NcmBJHuOeOycJLuSPDn7ePbI8z82+/4/muSLSc5a1vyX\nMnkQkpwG/APwh8AbgfcmeeOISzgEfKiqfhO4FPjLkecfdi2wd4K5AJ8CvlpVvwH81pjrSHIu8AFg\nraouBk4Drl7y2M8Clx312PXAfVV1IXDf7P6Y83cBF1fVm4BvATcscf5xTR4E4HeAp6rqO1X1AvB5\n4KqxhlfV/qp6aHb7Z6z/ZTh3rPkASbYC7wZuHnPubPaZwNuBWwCq6oWq+p+Rl7EBeEWSDcAW4AfL\nHFZV3wR+fNTDVwG3zW7fBrxnzPlVdW9VHZrdvR/Yuqz5L2UVgnAu8MwR9/cx8l/Iw5KcD1wCPDDy\n6E8CHwZ+MfJcgNcDzwKfmR2y3JzkjLGGV9X3gY8DTwP7gZ9U1b1jzT/Ca6pq/2xN+4FXT7CGw94H\nfGWKwasQhBzjsdHPp07ySuALwAer6qcjzr0COFBVD4418ygbgDcDn66qS4DnWO7u8ovMjtWvAi4A\nXgeckeSaseavmiQ3sn4Ye/sU81chCPuA8464v5Ul7zIeLclG1mNwe1XdNeZs4G3AlUm+x/rh0juS\nfG7E+fuAfVV1eK/oTtYDMZZ3At+tqmer6iBwF/DWEecf9qMkrwWYfTww9gKSbAeuAP6kJvoho1UI\nwr8DFya5IMnprD+hdPdYw5OE9ePnvVX1ibHmHlZVN1TV1qo6n/Wv/WtVNdq/kFX1Q+CZJBfNHtoG\nPD7WfNYPFS5NsmX2/2Ib0zy5ejewfXZ7O/ClMYcnuQz4CHBlVf18zNkvUlWT/wIuZ/2Z1W8DN448\n+/dYP0R5FHhk9uvyib4PfwDcM8Hc3wZ2z74H/wycPfL8vwGeAPYA/whsWvK8O1h/vuIg63tI7wd+\nmfVXF56cfTxn5PlPsf5c2uE/gzeN/eegqshsgZK0EocMklaEQZDUDIKkZhAkNYMgqa1UEJLscP6p\nOf9U/tpXYf5hKxUEYOpvivNPzdnOn1m1IEia0KgnJp2eTbWZ4/8g3UGeZyObRluP81dn/qn8tY8x\n/395jhfq+WP9IOGLbFjaCo5hM2fwu9k25khJwAN130lt5yGDpGYQJLVBQZjyzVElLd7cQViBN0eV\ntGBD9hAmfXNUSYs3JAgr8+aokhZjyMuOJ/XmqLNTMncAbGbLgHGSlm3IHsJJvTlqVe2sqrWqWpvy\nxA9JJzYkCJO+OaqkxZv7kKGqDiX5K+BfWL/81q1V9djCViZpdINOXa6qLwNfXtBaJE3MMxUlNYMg\nqRkESc0gSGoGQVIzCJKaQZDUDIKkZhAkNYMgqRkESc0gSGoGQVIzCJKaQZDUDIKkZhAkNYMgqY16\n9eepPfV3l069BJ3C3nDd/VMv4YTcQ5DUDIKkZhAkNYMgqQ25HPx5Sb6eZG+Sx5Jcu8iFSRrfkFcZ\nDgEfqqqHkrwKeDDJrqp6fEFrkzSyufcQqmp/VT00u/0zYC9eDl56WVvIcwhJzgcuAR5YxOeTNI3B\nJyYleSXwBeCDVfXTY/z+DmAHwGa2DB0naYkG7SEk2ch6DG6vqruOtU1V7ayqtapa28imIeMkLdmQ\nVxkC3ALsrapPLG5JkqYyZA/hbcCfAu9I8sjs1+ULWpekCcz9HEJV/SuQBa5F0sQ8U1FSMwiSmkGQ\n1AyCpGYQJDWDIKkZBEnNIEhqBkFSMwiSmkGQ1AyCpGYQJDWDIKkZBEnNIEhqBkFSMwiSmkGQ1AyC\npGYQJDWDIKkZBEltcBCSnJbk4ST3LGJBkqaziD2Ea1m/FLykl7mhF3vdCrwbuHkxy5E0paF7CJ8E\nPgz8YgFrkTSxIVd/vgI4UFUPnmC7HUl2J9l9kOfnHSdpBEOv/nxlku8Bn2f9KtCfO3qjqtpZVWtV\ntbaRTQPGSVq2uYNQVTdU1daqOh+4GvhaVV2zsJVJGp3nIUhqGxbxSarqG8A3FvG5JE3HPQRJzSBI\nagZBUjMIkppBkNQMgqRmECQ1gyCpGQRJzSBIagZBUjMIkppBkNQMgqRmECQ1gyCpGQRJzSBIagZB\nUjMIkppBkNQMgqRmECS1oVd/PivJnUmeSLI3yVsWtTBJ4xt6oZZPAV+tqj9KcjqwZQFrkjSRuYOQ\n5Ezg7cCfAVTVC8ALi1mWpCkMOWR4PfAs8JkkDye5OckZC1qXpAkMCcIG4M3Ap6vqEuA54PqjN0qy\nI8nuJLsP8vyAcZKWbUgQ9gH7quqB2f07WQ/Ei1TVzqpaq6q1jWwaME7Sss0dhKr6IfBMkotmD20D\nHl/IqiRNYuirDH8N3D57heE7wJ8PX5KkqQwKQlU9AqwtaC2SJuaZipKaQZDUhj6HoP+Hb//xTYP+\n+1//p79Y0EqkY3MPQVIzCJKaQZDUDIKkZhAkNYMgqRkESc0gSGoGQVIzCJKaQZDUDIKkZhAkNYMg\nqRkESc33QxiR72egVecegqRmECQ1gyCpGQRJbVAQklyX5LEke5LckWTzohYmaXxzByHJucAHgLWq\nuhg4Dbh6UQuTNL6hhwwbgFck2QBsAX4wfEmSpjLkYq/fBz4OPA3sB35SVfcuamGSxjfkkOFs4Crg\nAuB1wBlJrjnGdjuS7E6y+yDPz79SSUs35JDhncB3q+rZqjoI3AW89eiNqmpnVa1V1dpGNg0YJ2nZ\nhgThaeDSJFuSBNgG7F3MsiRNYchzCA8AdwIPAf8x+1w7F7QuSRMY9MNNVfVR4KMLWoukiXmmoqRm\nECQ1gyCpGQRJzSBIagZBUjMIkppBkNQMgqRmECQ1gyCpGQRJzSBIagZBUjMIkppBkNQMgqRmECQ1\ngyCpGQRJzSBIagZBUjMIktoJg5Dk1iQHkuw54rFzkuxK8uTs49nLXaakMZzMHsJngcuOeux64L6q\nuhC4b3Zf0svcCYNQVd8EfnzUw1cBt81u3wa8Z8HrkjSBeZ9DeE1V7QeYfXz14pYkaSqDru14MpLs\nAHYAbGbLssdJGmDePYQfJXktwOzjgeNtWFU7q2qtqtY2smnOcZLGMG8Q7ga2z25vB760mOVImtLJ\nvOx4B/BvwEVJ9iV5P/C3wLuSPAm8a3Zf0svcCZ9DqKr3Hue3ti14LZIm5pmKkppBkNQMgqRmECQ1\ngyCpGQRJzSBIagZBUjMIkppBkNQMgqRmECQ1gyCpGQRJzSBIagZBUjMIkppBkNSW/jbsq+QN190/\n9RKkleYegqRmECQ1gyCpzXs5+I8leSLJo0m+mOSs5S5T0hjmvRz8LuDiqnoT8C3ghgWvS9IE5roc\nfFXdW1WHZnfvB7YuYW2SRraI5xDeB3xlAZ9H0sQGnYeQ5EbgEHD7S2zj5eCll4m5g5BkO3AFsK2q\n6njbVdVOYCfAmTnnuNtJmt5cQUhyGfAR4Per6ueLXZKkqcx7Ofi/B14F7ErySJKblrxOSSOY93Lw\ntyxhLZIm5pmKkppBkNQMgqRmECQ1gyCpGQRJzSBIagZBUjMIkppBkNQMgqRmECQ1gyCpGQRJzSBI\nagZBUjMIkppBkNQMgqSWl3gH9cUPS54F/uslNvkV4L9HWo7zV2v+qfy1jzH/16rqV0+00ahBOJEk\nu6tqzfmn3vxT+WtfhfmHecggqRkESW3VgrDT+afs/FP5a1+F+cCKPYcgaVqrtocgaUIGQVIzCJKa\nQZDUDIKk9n9qIVxyXTQ+QgAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAb4AAAECCAYAAACBhbLnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAADU9JREFUeJzt3W2MZXddB/Dvz+7SSiGhFdo0FEVL\noxAjixmxSY0pIGRFYiGRaKOmL0gWEkiA4EPhDWhCgi+g+MJgCtT2BY+RhzamUZoFU02kWmCFkmIK\nWLF005UgAU0sFH6+mLNhWGZmZ2fuzL2H/+eTTO45/3vunm/+2bnfOefch+ruAMAofmzZAQDgICk+\nAIai+AAYiuIDYCiKD4ChKD4AhrK04quqo1X1b1X1xaq6YVk5dqOqHqiqz1XViaq6Z9l5tlJVN1fV\nqaq6d8PYxVV1Z1XdP91etMyMW9ki+5uq6qvTvJ+oqhcuM+NmquopVfWJqrqvqj5fVa+exld63rfJ\nPYc5v6Cq/rmq/nXK/ifT+E9X1d3TnH+gqh6z7Kxn2ib7LVX17xvm/ciys26mqs6rqs9U1d9M6ys/\n58mSiq+qzkvyF0l+PckzklxXVc9YRpY9eE53H+nutWUH2cYtSY6eMXZDkuPdfWWS49P6KrolP5w9\nSW6c5v1Id99xwJl24tEkr+vupye5Kskrp//bqz7vW+VOVn/OH0ny3O5+ZpIjSY5W1VVJ/izr2a9M\n8t9JXrbEjFvZKnuS/OGGeT+xvIjbenWS+zasz2HOl3bE9+wkX+zuL3f3t5O8P8m1S8ryI6u770ry\n9TOGr01y67R8a5IXH2ioHdoi+8rr7pPd/elp+VtZf1J4clZ83rfJvfJ63f9Mq4enn07y3CR/PY2v\n3Jwn22ZfeVV1eZLfSPKuab0ygzlPlld8T07ynxvWH8xMfskmneRjVfWpqjq27DDn6NLuPpmsP9kl\nuWTJec7Vq6rqs9Op0JU6XXimqnpqkmcluTszmvczciczmPPplNuJJKeS3JnkS0m+0d2PTpus7HPM\nmdm7+/S8v3ma9xur6vwlRtzK25P8UZLvTes/kZnM+bKKrzYZm8VfOZOru/sXs36q9pVV9avLDjSI\ndyS5IuunhE4meety42ytqh6X5ENJXtPd31x2np3aJPcs5ry7v9vdR5JcnvUzSk/fbLODTbUzZ2av\nqp9P8vokP5fkl5JcnOSPlxjxh1TVi5Kc6u5PbRzeZNOVnPNlFd+DSZ6yYf3yJA8tKcs56+6HpttT\nST6S9V+0uXi4qi5Lkun21JLz7Fh3Pzw9SXwvyTuzovNeVYezXh7v6e4PT8MrP++b5Z7LnJ/W3d9I\n8vdZv075hKo6NN218s8xG7IfnU49d3c/kuSvsnrzfnWS36yqB7J+qeq5WT8CnMWcL6v4/iXJldMr\ngB6T5HeS3L6kLOekqi6sqsefXk7ygiT3bv+olXJ7kuun5euT3LbELOfkdHFMXpIVnPfpOse7k9zX\n3W/bcNdKz/tWuWcy50+qqidMyz+e5Neyfo3yE0l+a9ps5eY82TL7Fzb8kVRZv062UvPe3a/v7su7\n+6lZf/7+eHf/bmYw50lSy/p2hull0W9Pcl6Sm7v7zUsJco6q6meyfpSXJIeSvHdVs1fV+5Jck+SJ\nSR5O8sYkH03ywSQ/meQrSV7a3Sv3IpItsl+T9VNuneSBJC8/fd1sVVTVryT5hySfy/evfbwh69fL\nVnbet8l9XVZ/zn8h6y+kOC/rf8x/sLv/dPpdfX/WTxV+JsnvTUdQK2Ob7B9P8qSsnz48keQVG14E\ns1Kq6pokf9DdL5rDnCdLLD4AWAaf3ALAUBQfAENRfAAMRfEBMBTFB8BQll58M/zIryTzzZ3MN/tc\ncyfzzT7X3Ml8s881dzKf7EsvviSzmKhNzDV3Mt/sc82dzDf7XHMn880+19zJTLKvQvEBwIE50Dew\nP6bO7wty4Q+MfSeP5HBW8YPHtzfX3Ml8s881dzLf7HPNncw3+1xzJ8vP/n/533y7H9nsw7J/wKGz\nbbCdqjqa5M+z/nE77+rut2y3/QW5ML9cz9vLLgFgU3f38R1tt+tTnT8i36IOwGD2co3Pt6gDMDt7\nKb65f4s6AAPayzW+HX3b7vS+jmNJckEeu4fdAcDe7eWIb0ffot7dN3X3WnevzfWVSgD86NhL8c32\nW9QBGNeuT3V296NV9aokf5fvf4v65xeWDAD2wZ7ex9fddyS5Y0FZAGDf+cgyAIai+AAYiuIDYCiK\nD4ChKD4AhqL4ABiK4gNgKIoPgKHs6Q3sy/TFG69adgQAFuBpr/3kge7PER8AQ1F8AAxF8QEwFMUH\nwFAUHwBDUXwADEXxATAUxQfAUBQfAENRfAAMRfEBMBTFB8BQFB8AQ1F8AAxF8QEwFMUHwFAUHwBD\nUXwADEXxATAUxQfAUBQfAENRfAAMRfEBMBTFB8BQFB8AQ1F8AAxF8QEwFMUHwFAO7eXBVfVAkm8l\n+W6SR7t7bRGhAGC/7Kn4Js/p7q8t4N8BgH3nVCcAQ9lr8XWSj1XVp6rq2CICAcB+2uupzqu7+6Gq\nuiTJnVX1he6+a+MGUyEeS5IL8tg97g4A9mZPR3zd/dB0eyrJR5I8e5Ntburute5eO5zz97I7ANiz\nXRdfVV1YVY8/vZzkBUnuXVQwANgPeznVeWmSj1TV6X/nvd39twtJBQD7ZNfF191fTvLMBWYBgH3n\n7QwADEXxATAUxQfAUBQfAENRfAAMRfEBMBTFB8BQFB8AQ1F8AAxF8QEwFMUHwFAUHwBDUXwADEXx\nATAUxQfAUBQfAENRfAAMRfEBMBTFB8BQFB8AQ1F8AAxF8QEwFMUHwFAUHwBDUXwADEXxATAUxQfA\nUBQfAENRfAAMRfEBMBTFB8BQFB8AQ1F8AAxF8QEwFMUHwFAUHwBDUXwADEXxATCUsxZfVd1cVaeq\n6t4NYxdX1Z1Vdf90e9H+xgSAxdjJEd8tSY6eMXZDkuPdfWWS49M6AKy8sxZfd9+V5OtnDF+b5NZp\n+dYkL15wLgDYF7u9xndpd59Mkun2ksVFAoD9c2i/d1BVx5IcS5IL8tj93h0AbGu3R3wPV9VlSTLd\nntpqw+6+qbvXunvtcM7f5e4AYDF2W3y3J7l+Wr4+yW2LiQMA+2snb2d4X5J/SvKzVfVgVb0syVuS\nPL+q7k/y/GkdAFbeWa/xdfd1W9z1vAVnYUm+9Nt/eaD7u+IDrzjQ/QFs5JNbABiK4gNgKIoPgKEo\nPgCGovgAGIriA2Aoig+AoSg+AIai+AAYiuIDYCiKD4ChKD4AhqL4ABiK4gNgKIoPgKEoPgCGovgA\nGIriA2Aoig+AoSg+AIai+AAYiuIDYCiKD4ChKD4AhqL4ABiK4gNgKIoPgKEcWnYAlu+KD7xi2REA\nDowjPgCGovgAGIriA2Aoig+AoSg+AIai+AAYiuIDYCiKD4ChKD4AhnLW4quqm6vqVFXdu2HsTVX1\n1ao6Mf28cH9jAsBi7OSI75YkRzcZv7G7j0w/dyw2FgDsj7MWX3ffleTrB5AFAPbdXq7xvaqqPjud\nCr1oYYkAYB/ttvjekeSKJEeSnEzy1q02rKpjVXVPVd3znTyyy90BwGLsqvi6++Hu/m53fy/JO5M8\ne5ttb+rute5eO5zzd5sTABZiV8VXVZdtWH1Jknu32hYAVslZv4i2qt6X5JokT6yqB5O8Mck1VXUk\nSSd5IMnL9zEjACzMWYuvu6/bZPjd+5AFAPadT24BYCiKD4ChKD4AhqL4ABiK4gNgKIoPgKEoPgCG\novgAGIriA2Aoig+AoSg+AIai+AAYiuIDYCiKD4ChKD4AhqL4ABiK4gNgKIoPgKEoPgCGovgAGIri\nA2Aoig+AoSg+AIai+AAYiuIDYCiKD4ChKD4AhqL4ABiK4gNgKIoPgKEoPgCGovgAGIriA2Aoig+A\noSg+AIai+AAYiuIDYCiKD4ChnLX4quopVfWJqrqvqj5fVa+exi+uqjur6v7p9qL9jwsAe7OTI75H\nk7yuu5+e5Kokr6yqZyS5Icnx7r4yyfFpHQBW2lmLr7tPdvenp+VvJbkvyZOTXJvk1mmzW5O8eL9C\nAsCinNM1vqp6apJnJbk7yaXdfTJZL8cklyw6HAAs2o6Lr6oel+RDSV7T3d88h8cdq6p7quqe7+SR\n3WQEgIXZUfFV1eGsl957uvvD0/DDVXXZdP9lSU5t9tjuvqm717p77XDOX0RmANi1nbyqs5K8O8l9\n3f22DXfdnuT6afn6JLctPh4ALNahHWxzdZLfT/K5qjoxjb0hyVuSfLCqXpbkK0leuj8RAWBxzlp8\n3f2PSWqLu5+32DgAsL98cgsAQ1F8AAxF8QEwFMUHwFAUHwBDUXwADEXxATAUxQfAUHbyyS0r6Wmv\n/eSyIwAwQ474ABiK4gNgKIoPgKEoPgCGovgAGIriA2Aoig+AoSg+AIai+AAYiuIDYCiKD4ChKD4A\nhqL4ABiK4gNgKIoPgKEoPgCGovgAGIriA2Ao1d0Ht7Oq/0ryH2cMPzHJ1w4sxOLMNXcy3+xzzZ3M\nN/tccyfzzT7X3Mnys/9Udz/pbBsdaPFtGqDqnu5eW2qIXZhr7mS+2eeaO5lv9rnmTuabfa65k/lk\nd6oTgKEoPgCGsgrFd9OyA+zSXHMn880+19zJfLPPNXcy3+xzzZ3MJPvSr/EBwEFahSM+ADgwig+A\noSg+AIai+AAYiuIDYCj/D+mwlg8A0jkhAAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQQAAAGhCAYAAACH2cpXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAADJdJREFUeJzt3X+s3Xddx/HXWzo2GRI2ccvciCAs\nCCEyTINTjEEnZiJh+IdRoqaJJNUEIhASHfKH+h+JCvqHgVSYLIpTww9ZCOKWSrKYCLEgweLQDUQY\n1BVCFIJx6+DtH/f0nbK0u+Xec7/nzj4eyc0553u/t+/POW2f/X7PPfe0ujsASfJtm14AsH8IAjAE\nARiCAAxBAIYgAGNfBKGqbqyqf62qe6vq5oVnP7mqPlhVd1fVJ6rqVUvOP2Mdj6mqf6qq921g9hOr\n6p1V9cnV4/BDC89/zeqxP15Vt1XVJXs875aqOllVx8/YdnlV3VlV96wuL1t4/u+uHv+PV9V7quqJ\nezX/kWw8CFX1mCR/lOSnkjwrycuq6lkLLuGhJK/t7mcmuT7JKxaef9qrkty9gblJ8odJPtDd35fk\nOUuuo6quTvJrSQ5297OTPCbJz+/x2LcnufFh225OcrS7r01ydHV7yfl3Jnl2d39/kn9L8ro9nH9O\nGw9Ckuclube7P93dDyb5iyQ3LTW8u09090dX17+arb8MVy81P0mq6pokP53krUvOXc1+QpIfTfK2\nJOnuB7v7vxZexoEk315VB5I8LskX9nJYd9+V5MsP23xTkltX129N8tIl53f3Hd390Ormh5Jcs1fz\nH8l+CMLVST53xu37svBfyNOq6ilJnpvkwwuP/oMkv57kGwvPTZLvTfLFJH+yOmV5a1VdutTw7v58\nkt9L8tkkJ5L8d3ffsdT8M1zZ3SdWazqR5IoNrOG0X07yN5sYvB+CUGfZtvjrqavq8UneleTV3f2V\nBee+OMnJ7v7IUjMf5kCSH0jy5u5+bpKvZW8Pl7/J6lz9piRPTfLdSS6tql9cav5+U1Wvz9Zp7Ds2\nMX8/BOG+JE8+4/Y12eNDxoerqouyFYN3dPe7l5yd5PlJXlJVn8nW6dKPV9WfLTj/viT3dffpo6J3\nZisQS/mJJP/e3V/s7lNJ3p3khxecf9r9VXVVkqwuTy69gKo6lOTFSX6hN/RDRvshCP+Y5NqqempV\nPTZbTyjdvtTwqqpsnT/f3d1vXGruad39uu6+prufkq37/nfdvdi/kN39n0k+V1XPWG26Icm/LDU/\nW6cK11fV41a/FzdkM0+u3p7k0Or6oSTvXXJ4Vd2Y5DeSvKS7/2fJ2d+kuzf+keRF2Xpm9VNJXr/w\n7B/J1inKx5N8bPXxog09Di9I8r4NzL0uybHVY/DXSS5beP7vJPlkkuNJ/jTJxXs877ZsPV9xKltH\nSC9P8p3Z+u7CPavLyxeef2+2nks7/WfwLUv/Oeju1GqBAPvilAHYJwQBGIIADEEAhiAAY18FoaoO\nm39hzr+Q7/t+mH/avgpCkk0/KOZfmLPNX9lvQQA2aNEXJj22Lu5Lcu4fpDuVB3JRLl5sPebvn/kX\n8n1fYv7/5mt5sB842w8SfpMDe7aCs7gkl+YH64YlRwJJPtxHz2s/pwzAEARg7CoIm3xzVGD9dhyE\nffDmqMCa7eYIYaNvjgqs326CsG/eHBVYj9182/G83hx19ZLMw0lySR63i3HAXtvNEcJ5vTlqdx/p\n7oPdfXCTL/wAtrebIGz0zVGB9dvxKUN3P1RVr0zyt9n677du6e5PrG1lwOJ29dLl7n5/kvevaS3A\nhnmlIjAEARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYi/7v\nz5t275uu3/QSuIA9/TUf2vQStuUIARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAAQxCA\nIQjAEARgCAIwBAEYggAMQQCGIABDEIAhCMAQBGAIAjAEARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQB\nGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIAhCMAQBGAIAjAEARiCAAxBAIYgAOPA\nbr64qj6T5KtJvp7koe4+uI5FAZuxqyCs/Fh3f2kNvw6wYU4ZgLHbIHSSO6rqI1V1+Gw7VNXhqjpW\nVcdO5YFdjgP20m5PGZ7f3V+oqiuS3FlVn+zuu87cobuPJDmSJE+oy3uX84A9tKsjhO7+wuryZJL3\nJHneOhYFbMaOg1BVl1bVd5y+nuQnkxxf18KA5e3mlOHKJO+pqtO/zp939wfWsipgI3YchO7+dJLn\nrHEtwIb5tiMwBAEYggAMQQCGIABDEIAhCMAQBGAIAjAEARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQB\nGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIAhCMAQBGAIAjAEARiCAAxBAIYgAEMQ\ngCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIAhCMAQBGAIAjAE\nARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAIQjA2DYIVXVLVZ2squNnbLu8qu6s\nqntWl5ft7TKBJZzPEcLbk9z4sG03Jzna3dcmObq6DTzKbRuE7r4ryZcftvmmJLeurt+a5KVrXhew\nATt9DuHK7j6RJKvLK861Y1UdrqpjVXXsVB7Y4ThgCXv+pGJ3H+nug9198KJcvNfjgF3YaRDur6qr\nkmR1eXJ9SwI2ZadBuD3JodX1Q0neu57lAJt0Pt92vC3JPyR5RlXdV1UvT/KGJC+sqnuSvHB1G3iU\nO7DdDt39snN86oY1rwXYMK9UBIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAIQjAEARgCAIw\nBAEYggAMQQDGtu+YxPp86ufesquvf9pf/uqaVgJn5wgBGIIADEEAhiAAQxCAIQjAEARgCAIwBAEY\nggAMQQCGIABDEIAhCMAQBGB4P4QFeT8D9jtHCMAQBGAIAjAEARiCAAxBAIYgAEMQgCEIwBAEYAgC\nMAQBGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIAhCMAQBGAIAjAEARiCAAxBAIYg\nAEMQgCEIwBAEYAgCMAQBGNsGoapuqaqTVXX8jG2/XVWfr6qPrT5etLfLBJZwPkcIb09y41m2v6m7\nr1t9vH+9ywI2YdsgdPddSb68wFqADdvNcwivrKqPr04pLlvbioCN2WkQ3pzkaUmuS3Iiye+fa8eq\nOlxVx6rq2Kk8sMNxwBJ2FITuvr+7v97d30jyx0me9wj7Hunug9198KJcvNN1AgvYURCq6qozbv5M\nkuPn2hd49Diw3Q5VdVuSFyR5UlXdl+S3krygqq5L0kk+k+RX9nCNwEK2DUJ3v+wsm9+2B2sBNswr\nFYEhCMAQBGAIAjAEARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAIQjAEARgCAIw\nBAEYggAMQQCGIABDEIAhCMAQBGAIAjAEARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAA\nQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIAhCMAQBGAIAjAEARiCAAxBAIYgAEMQgCEIwBAEYAgC\nMAQBGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggAMQQDGtkGoqidX1Qer6u6q+kRVvWq1/fKqurOq\n7lldXrb3ywX20vkcITyU5LXd/cwk1yd5RVU9K8nNSY5297VJjq5uA49i2wahu09090dX17+a5O4k\nVye5Kcmtq91uTfLSvVoksIxv6TmEqnpKkucm+XCSK7v7RLIVjSRXrHtxwLLOOwhV9fgk70ry6u7+\nyrfwdYer6lhVHTuVB3ayRmAh5xWEqrooWzF4R3e/e7X5/qq6avX5q5KcPNvXdveR7j7Y3QcvysXr\nWDOwR87nuwyV5G1J7u7uN57xqduTHFpdP5TkvetfHrCkA+exz/OT/FKSf66qj622/WaSNyT5q6p6\neZLPJvnZvVkisJRtg9Ddf5+kzvHpG9a7HGCTvFIRGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggAM\nQQCGIABDEIAhCMAQBGAIAjAEARiCAIzzeU/F/zee/poPbXoJsK85QgCGIABDEIAhCMAQBGAIAjAE\nARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggAMQQCGIABD\nEIAhCMAQBGAIAjAEARjV3csNq/pikv94hF2elORLCy3H/P01/0K+70vM/57u/q7tdlo0CNupqmPd\nfdD8C2/+hXzf98P805wyAEMQgLHfgnDE/At2/oV83/fD/CT77DkEYLP22xECsEGCAAxBAIYgAEMQ\ngPF//EEXZRzIeIUAAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQQAAAECCAYAAAAYUakXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAACG1JREFUeJzt3U+opXd9x/HPtyZOMLiIxIRQRSSG\ngpsO5RILgkREiW6ii9JmlYUwFpqNu+x06UayKpaRhmRjbDfBLMQaZpONiiMEO4WWpJK2Y0Kmkk0p\nNMb47SJnvo5xxjve8+e5957XCy7nnGeemfP9zYU3z3Puc86t7g5AkvzR0gMAx4cgAEMQgCEIwBAE\nYAgCMBYNQlU9WFX/VlUvVdVjS86ybVX1clX9c1W9UFUXl55nk6rqiaq6UlWXrtn2vqp6rqpeXN3e\nseSMm3KDtX61qn6++t6+UFWfW3LGdSwWhKp6V5K/TfLZJB9N8nBVfXSpeXbkk919trsPlh5kw55M\n8uA7tj2W5EJ335fkwurxafBkfnetSfL46nt7tru/u+OZNmbJI4T7k7zU3T/r7l8m+XaShxachyPq\n7ueTvP6OzQ8leWp1/6kkn9/pUFtyg7WeGksG4Y+T/Nc1jy+vtp1WneT7VfWTqjq39DA7cHd3v5ok\nq9u7Fp5n2x6tqp+uTilO7OnRkkGo62w7zddRf7y7/yxvnyL9TVV9YumB2JhvJLk3ydkkryb5+rLj\nHN2SQbic5IPXPP5AklcWmmXruvuV1e2VJM/k7VOm0+y1qronSVa3VxaeZ2u6+7Xufqu7f53kmznB\n39slg/DjJPdV1Yer6t1J/irJswvOszVVdXtVvffq/SSfSXLp9/+tE+/ZJI+s7j+S5DsLzrJVV8O3\n8oWc4O/tLUs9cXf/qqoeTfJPSd6V5Inu/pel5tmyu5M8U1XJ2//n3+ru7y070uZU1dNJHkhyZ1Vd\nTvKVJF9L8o9V9cUk/5nkL5abcHNusNYHqups3j7lfTnJlxYbcE3l7c/AVa5UBIYgAEMQgCEIwBAE\nYByLIOzJpbxJ9met+7LO5HSt9VgEIcmp+Q+9Cfuy1n1ZZ3KK1npcggAcAzu9MOnddaZvy+2/s/3N\nvJFbc2ZncyxpX9a6L+tMTsZa/y//m1/2G9d7Q+Fv2emly7fl9nysPrXLpwSS/Kgv3NR+a50y7NNH\noME+OHIQ9vQj0OBUW+cIwUegwSmzThD27SPQ4NRb50XFm/oItNVFG+eS5La8Z42nA7ZtnSOEm/oI\ntO4+390H3X1w3H80A/tunSDszUegwb448inDnn0EGuyFtS5MWv2GmhP7W2qA3+a9DMAQBGAIAjAE\nARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGDv91OVte+nxP196BDiyj3z5h0uP4AgB+A1BAIYgAEMQ\ngCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIAhCMAQBGAIAjAE\nARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAIQjAuGWdv1xVLyf5nyRvJflVdx9s\nYihgGWsFYeWT3f2LDfw7wMKcMgBj3SB0ku9X1U+q6twmBgKWs+4pw8e7+5WquivJc1X1r939/LU7\nrEJxLkluy3vWfDpgm9Y6QujuV1a3V5I8k+T+6+xzvrsPuvvg1pxZ5+mALTtyEKrq9qp679X7ST6T\n5NKmBgN2b51ThruTPFNVV/+db3X39zYyFbCIIwehu3+W5E83OAuwMD92BIYgAEMQgCEIwBAEYAgC\nMAQBGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIAhCMAQBGAIAjAEARiCAAxBAIYg\nAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAccvSA5xW//6Xf/cH7X/vP/z1liaBm+cIARiCAAxB\nAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhvcybIn3JnASOUIAhiAA49AgVNUTVXWlqi5ds+19VfVc\nVb24ur1ju2MCu3AzRwhPJnnwHdseS3Khu+9LcmH1GDjhDg1Cdz+f5PV3bH4oyVOr+08l+fyG5wIW\ncNTXEO7u7leTZHV71412rKpzVXWxqi6+mTeO+HTALmz9RcXuPt/dB919cGvObPvpgDUcNQivVdU9\nSbK6vbK5kYClHDUIzyZ5ZHX/kSTf2cw4wJJu5seOTyf5QZI/qarLVfXFJF9L8umqejHJp1ePgRPu\n0EuXu/vhG/zRpzY8C7AwVyoCQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIAhCMAQBGAIAjAEARiC\nAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggAMQQCGIABDEIAh\nCMAQBGAIAjAEARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIADEEAhiAA49AgVNUTVXWlqi5ds+2r\nVfXzqnph9fW57Y4J7MLNHCE8meTB62x/vLvPrr6+u9mxgCUcGoTufj7J6zuYBVjYOq8hPFpVP12d\nUtyxsYmAxRw1CN9Icm+Ss0leTfL1G+1YVeeq6mJVXXwzbxzx6YBdOFIQuvu17n6ru3+d5JtJ7v89\n+57v7oPuPrg1Z446J7ADRwpCVd1zzcMvJLl0o32Bk+OWw3aoqqeTPJDkzqq6nOQrSR6oqrNJOsnL\nSb60xRmBHTk0CN398HU2//0WZgEW5kpFYAgCMAQBGIIADEEAhiAAQxCAIQjAEARgCAIwBAEYggCM\nQ9/cdJJ85Ms/XHoEONEcIQBDEIAhCMAQBGAIAjAEARiCAAxBAIYgAEMQgCEIwBAEYAgCMAQBGIIA\nDEEAhiAAQxCAIQjAEARgCAIwBAEYggCM6u7dPVnVfyf5j+v80Z1JfrGzQZa1L2vdl3UmJ2OtH+ru\n9x+2006DcMMhqi5298HSc+zCvqx1X9aZnK61OmUAhiAA47gE4fzSA+zQvqx1X9aZnKK1HovXEIDj\n4bgcIQDHgCAAQxCAIQjAEARg/D+FDrP+euJ/tgAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "basicgame = CreateGame(GameBasic, holes = 0)\n", "basicgame.reset(size = 15, reward_control=0)\n", "plt.matshow(basicgame.grid.grid)\n", "\n", "holegame = CreateGame(GameHole, holes = 50)\n", "holegame.reset(size = 15, reward_control=0)\n", "plt.matshow(holegame.grid.grid)\n", "\n", "bargame = CreateGame(GameBar, holes = 0)\n", "bargame.reset(size = 15, reward_control=0)\n", "plt.matshow(bargame.grid.grid)\n", "\n", "scalegame = CreateGame(GameScale, holes = 0)\n", "scalegame.reset(reward_control=0)\n", "plt.matshow(scalegame.grid.grid)\n", "\n", "scalegame_x = CreateGame(GameScale_x, holes = 0)\n", "scalegame_x.reset(reward_control=0)\n", "plt.matshow(scalegame_x.grid.grid)\n", "\n", "scalegame_y = CreateGame(GameScale_y, holes = 0)\n", "scalegame_y.reset(reward_control=0)\n", "plt.matshow(scalegame_y.grid.grid)\n", "\n", "Imp_game = CreateGame(GameImplicit, holes = 0)\n", "Imp_game.reset(size = 15, reward_control=0)\n", "plt.matshow(Imp_game.grid.grid)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Think of how to make this traning stable, adding exploration noise , intenral noise or environmental variability like multiple mazes " ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "hole\n", "start [15]\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/home/tie/anaconda3/lib/python3.6/site-packages/torch/tensor.py:255: UserWarning: non-inplace resize is deprecated\n", " warnings.warn(\"non-inplace resize is deprecated\")\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "clear session data 4 6882639872\n", "hole performance -0.30549347518727066\n", "rewards -0.30549347518727066\n", "clear session data 4 7134629888\n", "hole performance -0.04719085298340009\n", "rewards -0.04719085298340009\n", "clear session data 4 7165169664\n", "hole performance -0.003306028937732741\n", "rewards -0.003306028937732741\n", "clear session data 4 7255347200\n", "hole performance -0.060857040602234014\n", "rewards -0.060857040602234014\n", "clear session data 4 7316471808\n", "hole performance -0.029471395242929077\n", "rewards -0.029471395242929077\n", "clear session data 4 7385210880\n", "hole performance -0.08639659253297566\n", "rewards -0.08639659253297566\n", "clear session data 4 7385210880\n", "hole performance -0.07652659325929134\n", "rewards -0.07652659325929134\n", "clear session data 4 7323127808\n", "hole performance -0.10052376880430683\n", "rewards -0.10052376880430683\n", "clear session data 4 7440461824\n", "hole performance -0.10355726174800597\n", "rewards -0.10355726174800597\n", "clear session data 4 7440461824\n", "hole performance -0.06432802950204308\n", "rewards -0.06432802950204308\n", "clear session data 4 7427772416\n", "hole performance -0.08326856533442487\n", "rewards -0.08326856533442487\n", "clear session data 4 7458172928\n", "hole performance -0.07389053284861088\n", "rewards -0.07389053284861088\n", "clear session data 4 7463366656\n", "hole performance -0.13303553208585076\n", "rewards -0.13303553208585076\n", "clear session data 4 7555534848\n", "hole performance -0.13947490199041618\n", "rewards -0.13947490199041618\n", "clear session data 4 7617150976\n", "hole performance -0.14295032651349232\n", "rewards -0.14295032651349232\n", "clear session data 4 7617150976\n", "hole performance -0.041259649385812336\n", "rewards -0.041259649385812336\n", "clear session data 4 7525105664\n", "hole performance -0.06079085420944941\n", "rewards -0.06079085420944941\n", "clear session data 4 7617536000\n", "hole performance -0.010411106340943554\n", "rewards -0.010411106340943554\n", "clear session data 4 7617536000\n", "hole performance -0.05655833957688923\n", "rewards -0.05655833957688923\n", "clear session data 4 7555923968\n", "hole performance 0.14588375523336325\n", "rewards 0.14588375523336325\n", "clear session data 4 7617339392\n", "hole performance 0.14935074486629774\n", "rewards 0.14935074486629774\n", "clear session data 4 7617396736\n", "hole performance -0.030206655023529427\n", "rewards -0.030206655023529427\n", "clear session data 4 7555514368\n", "hole performance -0.034523422574340656\n", "rewards -0.034523422574340656\n", "clear session data 4 7617191936\n", "hole performance 0.005684687613419365\n", "rewards 0.005684687613419365\n", "clear session data 4 7617249280\n", "hole performance -0.16567958062107185\n", "rewards -0.16567958062107185\n", "clear session data 4 7586361344\n", "hole performance -0.08212550028643972\n", "rewards -0.08212550028643972\n", "clear session data 4 7586549760\n", "hole performance 0.025617060593551233\n", "rewards 0.025617060593551233\n", "clear session data 4 7524950016\n", "hole performance -0.013389990614329272\n", "rewards -0.013389990614329272\n", "clear session data 4 7617110016\n", "hole performance -0.035219099707822527\n", "rewards -0.035219099707822527\n", "clear session data 4 7648051200\n", "hole performance -0.12210334105735338\n", "rewards -0.12210334105735338\n", "basic\n", "start [15]\n", "clear session data 4 7049420800\n", "basic performance -0.865442093713578\n", "rewards -0.865442093713578\n", "clear session data 4 7049420800\n", "basic performance -0.6288014812660097\n", "rewards -0.6288014812660097\n", "clear session data 4 7049420800\n", "basic performance -0.2206808557836729\n", "rewards -0.2206808557836729\n", "clear session data 4 7049420800\n", "basic performance -0.25620410786030573\n", "rewards -0.25620410786030573\n", "clear session data 4 7049420800\n", "basic performance -0.1569091798765519\n", "rewards -0.1569091798765519\n", "clear session data 4 7049420800\n", "basic performance -0.46932198986408236\n", "rewards -0.46932198986408236\n", "clear session data 4 7049420800\n", "basic performance 0.18426792038792109\n", "rewards 0.18426792038792109\n", "clear session data 4 7049420800\n", "basic performance -0.2132383782868442\n", "rewards -0.2132383782868442\n", "clear session data 4 7049420800\n", "basic performance -0.036862099506663644\n", "rewards -0.036862099506663644\n", "clear session data 4 7049420800\n", "basic performance 0.3778357881169353\n", "rewards 0.3778357881169353\n", "bar\n", "start [15]\n", "clear session data 4 6813061120\n", "bar performance -0.5135416745107738\n", "rewards -0.5135416745107738\n", "clear session data 4 6813061120\n", "bar performance -0.16834136764715604\n", "rewards -0.16834136764715604\n", "clear session data 4 6813061120\n", "bar performance -0.7971767698564812\n", "rewards -0.7971767698564812\n", "clear session data 4 6813061120\n", "bar performance -0.6134736996037535\n", "rewards -0.6134736996037535\n", "clear session data 4 6813061120\n", "bar performance -0.5037897850736044\n", "rewards -0.5037897850736044\n", "clear session data 4 6813061120\n", "bar performance -0.24688211536249996\n", "rewards -0.24688211536249996\n", "clear session data 4 6813061120\n", "bar performance -0.024244994397714714\n", "rewards -0.024244994397714714\n", "clear session data 4 6813061120\n", "bar performance 0.03553008552277742\n", "rewards 0.03553008552277742\n", "clear session data 4 6813061120\n", "bar performance -0.20372785407786648\n", "rewards -0.20372785407786648\n", "clear session data 4 6813061120\n", "bar performance 0.19563542883353513\n", "rewards 0.19563542883353513\n", "clear session data 4 6813061120\n", "bar performance 0.2066240247932706\n", "rewards 0.2066240247932706\n", "clear session data 4 6813061120\n", "bar performance 0.11408753338231337\n", "rewards 0.11408753338231337\n", "clear session data 4 6813061120\n", "bar performance 0.5365131345641323\n", "rewards 0.5365131345641323\n", "clear session data 4 6813061120\n", "bar performance 0.6047402235809236\n", "rewards 0.6047402235809236\n", "clear session data 4 6813061120\n", "bar performance 0.5921636933123515\n", "rewards 0.5921636933123515\n", "clear session data 4 6813061120\n", "bar performance 0.5922034906291751\n", "rewards 0.5922034906291751\n", "clear session data 4 6813061120\n", "bar performance 0.26041429034536595\n", "rewards 0.26041429034536595\n", "clear session data 4 6813061120\n", "bar performance 0.28733425683514796\n", "rewards 0.28733425683514796\n", "clear session data 4 6813061120\n", "bar performance 0.2144606863814732\n", "rewards 0.2144606863814732\n", "clear session data 4 6813061120\n", "bar performance 0.352644205206822\n", "rewards 0.352644205206822\n", "clear session data 4 6813061120\n", "bar performance 0.5432476696578383\n", "rewards 0.5432476696578383\n", "clear session data 4 6813061120\n", "bar performance 0.503270138975416\n", "rewards 0.503270138975416\n", "clear session data 4 6813061120\n", "bar performance 0.2878670797941316\n", "rewards 0.2878670797941316\n", "clear session data 4 6813061120\n", "bar performance 0.35058438058331787\n", "rewards 0.35058438058331787\n", "clear session data 4 6813061120\n", "bar performance 0.3377100522762133\n", "rewards 0.3377100522762133\n", "clear session data 4 6813061120\n", "bar performance 0.2239125555152457\n", "rewards 0.2239125555152457\n", "clear session data 4 6813061120\n", "bar performance 0.3511505515593682\n", "rewards 0.3511505515593682\n", "clear session data 4 6813061120\n", "bar performance 0.3178682037189939\n", "rewards 0.3178682037189939\n", "clear session data 4 6813061120\n", "bar performance 0.630794709967904\n", "rewards 0.630794709967904\n", "clear session data 4 6813061120\n", "bar performance 0.47376760482705266\n", "rewards 0.47376760482705266\n", "scale\n", "start [10 20 30 40 50]\n", "clear session data 4 6816047104\n", "scale performance -0.828125\n", "rewards -0.828125\n", "clear session data 4 6816047104\n", "scale performance -0.06638377564046691\n", "rewards -0.06638377564046691\n", "clear session data 4 6816858112\n", "scale performance -0.20433591337430537\n", "rewards -0.20433591337430537\n", "clear session data 4 6817669120\n", "scale performance -0.19337435080764453\n", "rewards -0.19337435080764453\n", "clear session data 4 6817669120\n", "scale performance -0.35797678591784826\n", "rewards -0.35797678591784826\n", "clear session data 4 6817669120\n", "scale performance -0.2358774590505886\n", "rewards -0.2358774590505886\n", "clear session data 4 6817669120\n", "scale performance -0.30931552250052874\n", "rewards -0.30931552250052874\n", "clear session data 4 6817669120\n", "scale performance -0.29975177816034415\n", "rewards -0.29975177816034415\n", "clear session data 4 6817669120\n", "scale performance -0.38035055340436813\n", "rewards -0.38035055340436813\n", "clear session data 4 6817669120\n", "scale performance -0.08291976750530476\n", "rewards -0.08291976750530476\n", "scale_x\n", "start [15]\n", "clear session data 4 6817669120\n", "scale_x performance -0.8612801277840341\n", "rewards -0.8612801277840341\n", "clear session data 4 6817669120\n", "scale_x performance -0.8329111428768043\n", "rewards -0.8329111428768043\n", "clear session data 4 6817669120\n", "scale_x performance -0.9754941239316239\n", "rewards -0.9754941239316239\n", "clear session data 4 6817669120\n", "scale_x performance -0.9247159090909091\n", "rewards -0.9247159090909091\n", "clear session data 4 6817669120\n", "scale_x performance -0.8677273300036656\n", "rewards -0.8677273300036656\n", "clear session data 4 6817669120\n", "scale_x performance -0.9038953553756185\n", "rewards -0.9038953553756185\n", "clear session data 4 6817669120\n", "scale_x performance -0.6188459541749033\n", "rewards -0.6188459541749033\n", "clear session data 4 6817669120\n", "scale_x performance -0.8894316515668768\n", "rewards -0.8894316515668768\n", "clear session data 4 6817669120\n", "scale_x performance -0.8964040763999646\n", "rewards -0.8964040763999646\n", "clear session data 4 6817669120\n", "scale_x performance -0.96875\n", "rewards -0.96875\n", "clear session data 4 6817669120\n", "scale_x performance -0.890625\n", "rewards -0.890625\n", "clear session data 4 6817669120\n", "scale_x performance -0.8333333333333333\n", "rewards -0.8333333333333333\n", "clear session data 4 6817669120\n", "scale_x performance -0.7286124718573344\n", "rewards -0.7286124718573344\n", "clear session data 4 6817669120\n", "scale_x performance -0.9630580357142857\n", "rewards -0.9630580357142857\n", "clear session data 4 6817669120\n", "scale_x performance -0.8599759615384616\n", "rewards -0.8599759615384616\n", "clear session data 4 6817669120\n", "scale_x performance -0.8654209109095805\n", "rewards -0.8654209109095805\n", "clear session data 4 6817669120\n", "scale_x performance -0.8816220238095238\n", "rewards -0.8816220238095238\n", "clear session data 4 6817669120\n", "scale_x performance -0.8849105695363981\n", "rewards -0.8849105695363981\n", "clear session data 4 6817669120\n", "scale_x performance -0.3977872870191519\n", "rewards -0.3977872870191519\n", "clear session data 4 6817669120\n", "scale_x performance -0.565849005810246\n", "rewards -0.565849005810246\n", "clear session data 4 6817669120\n", "scale_x performance -0.7069954938646393\n", "rewards -0.7069954938646393\n", "clear session data 4 6817669120\n", "scale_x performance -0.7668509239296926\n", "rewards -0.7668509239296926\n", "clear session data 4 6817669120\n", "scale_x performance -0.9033937411813361\n", "rewards -0.9033937411813361\n", "clear session data 4 6817669120\n", "scale_x performance -0.8011455583836098\n", "rewards -0.8011455583836098\n", "clear session data 4 6817669120\n", "scale_x performance -0.8053386021082007\n", "rewards -0.8053386021082007\n", "clear session data 4 6817669120\n", "scale_x performance -0.594463134826219\n", "rewards -0.594463134826219\n", "clear session data 4 6817669120\n", "scale_x performance -0.7515867271435783\n", "rewards -0.7515867271435783\n", "clear session data 4 6817669120\n", "scale_x performance -0.7568932550684571\n", "rewards -0.7568932550684571\n", "clear session data 4 6817669120\n", "scale_x performance -0.7041706159624839\n", "rewards -0.7041706159624839\n", "clear session data 4 6817669120\n", "scale_x performance -0.4590420151406552\n", "rewards -0.4590420151406552\n", "scale_y\n", "start [15]\n", "clear session data 4 6817669120\n", "scale_y performance -0.9868259803921569\n", "rewards -0.9868259803921569\n", "clear session data 4 6817669120\n", "scale_y performance -1.0\n", "rewards -1.0\n", "clear session data 4 6817669120\n", "scale_y performance -0.953125\n", "rewards -0.953125\n", "clear session data 4 6817669120\n", "scale_y performance -1.0\n", "rewards -1.0\n", "clear session data 4 6817669120\n", "scale_y performance -0.6080542588865403\n", "rewards -0.6080542588865403\n", "clear session data 4 6817669120\n", "scale_y performance -0.9894622093023255\n", "rewards -0.9894622093023255\n", "clear session data 4 6817669120\n", "scale_y performance -0.8913077080488147\n", "rewards -0.8913077080488147\n", "clear session data 4 6817669120\n", "scale_y performance -0.7328854723427122\n", "rewards -0.7328854723427122\n", "clear session data 4 6817669120\n", "scale_y performance -0.5968754097705327\n", "rewards -0.5968754097705327\n", "clear session data 4 6817669120\n", "scale_y performance -1.0\n", "rewards -1.0\n", "clear session data 4 6817669120\n", "scale_y performance -1.0\n", "rewards -1.0\n", "clear session data 4 6817669120\n", "scale_y performance -0.9420454545454545\n", "rewards -0.9420454545454545\n", "clear session data 4 6817669120\n", "scale_y performance -0.953125\n", "rewards -0.953125\n", "clear session data 4 6817669120\n", "scale_y performance -1.0\n", "rewards -1.0\n", "clear session data 4 6817669120\n", "scale_y performance -0.9375\n", "rewards -0.9375\n", "clear session data 4 6817669120\n", "scale_y performance -0.7600458099341109\n", "rewards -0.7600458099341109\n", "clear session data 4 6817669120\n", "scale_y performance -0.96875\n", "rewards -0.96875\n", "clear session data 4 6817669120\n", "scale_y performance -0.9215973829740675\n", "rewards -0.9215973829740675\n", "clear session data 4 6817669120\n", "scale_y performance -0.9375\n", "rewards -0.9375\n", "clear session data 4 6817669120\n", "scale_y performance -0.9256544132360105\n", "rewards -0.9256544132360105\n", "clear session data 4 6817669120\n", "scale_y performance -0.9906791460396039\n", "rewards -0.9906791460396039\n", "clear session data 4 6817669120\n", "scale_y performance -0.953125\n", "rewards -0.953125\n", "clear session data 4 6817669120\n", "scale_y performance -0.953125\n", "rewards -0.953125\n", "clear session data 4 6817669120\n", "scale_y performance -0.984375\n", "rewards -0.984375\n", "clear session data 4 6817669120\n", "scale_y performance -0.9805797383922383\n", "rewards -0.9805797383922383\n", "clear session data 4 6817669120\n", "scale_y performance -0.5241947127508527\n", "rewards -0.5241947127508527\n", "clear session data 4 6817669120\n", "scale_y performance -0.9276868344320487\n", "rewards -0.9276868344320487\n", "clear session data 4 6817669120\n", "scale_y performance -0.96875\n", "rewards -0.96875\n", "clear session data 4 6817669120\n", "scale_y performance -0.8633298356340422\n", "rewards -0.8633298356340422\n", "clear session data 4 6817669120\n", "scale_y performance -0.984375\n", "rewards -0.984375\n", "implicit\n", "start [15]\n", "clear session data 4 6817669120\n", "implicit performance -0.6517055016772534\n", "rewards -0.6517055016772534\n", "clear session data 4 6817632256\n", "implicit performance -0.5974943957002237\n", "rewards -0.5974943957002237\n", "clear session data 4 6817632256\n", "implicit performance -0.8111242044313857\n", "rewards -0.8111242044313857\n", "clear session data 4 6817632256\n", "implicit performance -0.7107899912587412\n", "rewards -0.7107899912587412\n", "clear session data 4 6817632256\n", "implicit performance -0.5695808418790114\n", "rewards -0.5695808418790114\n", "clear session data 4 6817632256\n", "implicit performance -0.12688552439085052\n", "rewards -0.12688552439085052\n", "clear session data 4 6817632256\n", "implicit performance -0.3974396298560647\n", "rewards -0.3974396298560647\n", "clear session data 4 6817632256\n", "implicit performance -0.5996073193485858\n", "rewards -0.5996073193485858\n", "clear session data 4 6817632256\n", "implicit performance -0.20721049500456148\n", "rewards -0.20721049500456148\n", "clear session data 4 6817632256\n", "implicit performance -0.10005889108357485\n", "rewards -0.10005889108357485\n", "clear session data 4 6817632256\n", "implicit performance -0.2873360258796684\n", "rewards -0.2873360258796684\n", "clear session data 4 6817632256\n", "implicit performance -0.6747513042630819\n", "rewards -0.6747513042630819\n", "clear session data 4 6817632256\n", "implicit performance -0.5642625787925529\n", "rewards -0.5642625787925529\n", "clear session data 4 6817632256\n", "implicit performance -0.44270337383307024\n", "rewards -0.44270337383307024\n", "clear session data 4 6817632256\n", "implicit performance -0.3495699064627672\n", "rewards -0.3495699064627672\n", "clear session data 4 6817632256\n", "implicit performance -0.3048038242877458\n", "rewards -0.3048038242877458\n", "clear session data 4 6817632256\n", "implicit performance -0.4292933490149046\n", "rewards -0.4292933490149046\n", "clear session data 4 6817632256\n", "implicit performance -0.2601094717255633\n", "rewards -0.2601094717255633\n", "clear session data 4 6817632256\n", "implicit performance -0.27443938893116504\n", "rewards -0.27443938893116504\n", "clear session data 4 6817632256\n", "implicit performance -0.3105321072270742\n", "rewards -0.3105321072270742\n", "clear session data 4 6817632256\n", "implicit performance -0.20679905780581748\n", "rewards -0.20679905780581748\n", "clear session data 4 6817632256\n", "implicit performance -0.27993006319957614\n", "rewards -0.27993006319957614\n", "clear session data 4 6817632256\n", "implicit performance -0.053170445143041\n", "rewards -0.053170445143041\n", "clear session data 4 6817632256\n", "implicit performance 0.42426194658587146\n", "rewards 0.42426194658587146\n", "clear session data 4 6817632256\n", "implicit performance 0.11907195944388782\n", "rewards 0.11907195944388782\n", "clear session data 4 6817632256\n", "implicit performance 0.3651093001936023\n", "rewards 0.3651093001936023\n", "clear session data 4 6817632256\n", "implicit performance 0.18193851197455171\n", "rewards 0.18193851197455171\n", "clear session data 4 6817632256\n", "implicit performance 0.0924479694942105\n", "rewards 0.0924479694942105\n", "clear session data 4 6817632256\n", "implicit performance 0.32388655868929556\n", "rewards 0.32388655868929556\n", "clear session data 4 6817632256\n", "implicit performance 0.536404329121745\n", "rewards 0.536404329121745\n" ] } ], "source": [ "# moving bar train\n", "trial = 399\n", "tasks = ['hole', 'basic', 'bar', 'scale', 'scale_x', 'scale_y', 'implicit']\n", "episodes = [30, 10, 30, 10, 30, 30, 30]\n", "# iterations = [1, 1, 1, 1, 1, 1, 1]\n", "for n, task in zip(episodes, tasks):\n", " Task = MultipleTasks(task = task, weight_write = 'weights_cpu_pos1/rnn_1515tanh512_checkpoint{}'.format(trial))\n", " weight_read = Task.weight\n", " weight_write = 'weights_' + task + '/rnn_1515tanh512_checkpoint{}'.format(trial)\n", " if task == 'scale':\n", " size_train = np.arange(10, 51, 10)\n", " else:\n", " size_train = [15]\n", " if task == 'hole':\n", " iterations = 5\n", " epochs = 100\n", " else:\n", " iterations = 5\n", " epochs = 10\n", " print ('start', size_train)\n", " Task.qlearn(task, weight_read, weight_write, episodes = n, noise = 0, size_train = size_train, size_test=[15], iterations = iterations\n", " , epochs = epochs)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "hole\n", "start [15]\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/home/tie/anaconda3/lib/python3.6/site-packages/torch/tensor.py:255: UserWarning: non-inplace resize is deprecated\n", " warnings.warn(\"non-inplace resize is deprecated\")\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "clear session data 4 5793517568\n", "hole performance -0.4542971737760023\n", "clear session data 4 6121148416\n", "hole performance -0.3489718985231542\n", "clear session data 4 6121443328\n", "hole performance -0.35768147436589764\n", "clear session data 4 6152392704\n", "hole performance -0.17569389390767368\n", "clear session data 4 6152392704\n", "hole performance 0.0007932174016826998\n", "clear session data 4 6121197568\n", "hole performance -0.14465832859872152\n", "clear session data 4 6157987840\n", "hole performance -0.05567686645963735\n", "clear session data 4 6157987840\n", "hole performance 0.11421016070099081\n", "clear session data 4 6158249984\n", "hole performance 0.2411646391828226\n", "clear session data 4 6157725696\n", "hole performance 0.10114841336128276\n", "clear session data 4 6157991936\n", "hole performance 0.2370491177567558\n", "clear session data 4 6183071744\n", "hole performance 0.15651696065519094\n" ] } ], "source": [ "# moving bar train\n", "trial = 300\n", "tasks = ['hole', 'basic', 'bar', 'scale', 'scale_x', 'scale_y', 'implicit']\n", "episodes = [30, 10, 30, 10, 30, 30, 30]\n", "# iterations = [1, 1, 1, 1, 1, 1, 1]\n", "for n, task in zip(episodes, tasks):\n", " Task = MultipleTasks(task = task, weight_write = 'weights_cpu_pos2/rnn_1515tanh512_checkpoint{}'.format(trial))\n", " weight_read = Task.weight\n", " weight_write = 'weights_' + task + '/rnn_1515tanh512_checkpoint{}'.format(trial)\n", " if task == 'scale':\n", " size_train = np.arange(10, 51, 10)\n", " else:\n", " size_train = [15]\n", " if task == 'hole':\n", " iterations = 5\n", " epochs = 100\n", " else:\n", " iterations = 5\n", " epochs = 10\n", " print ('start', size_train)\n", " Task.qlearn(task, weight_read, weight_write, episodes = n, noise = 0, size_train = size_train, size_test=[15], iterations = iterations\n", " , epochs = epochs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### measure decoding " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Decoding accuracy decreases, there are two possible reasons: \n", "1, decoding has bias, information of position is only refleted by decoding, if there is certain bias , it is not trustable\n", "2, it is the real measure of information, so there are less spatial information about space, or an effect decoupling/disentanglement of position. This can be due the network succeed at finding an even lower diemnsion object which gives successful performance. The representation deceases its information to input in a sense. \n", "\n", "This can be linked to receptive field by looking at how space is represented. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.3" }, "latex_envs": { "LaTeX_envs_menu_present": true, "autoclose": false, "autocomplete": true, "bibliofile": "biblio.bib", "cite_by": "apalike", "current_citInitial": 1, "eqLabelWithNumbers": true, "eqNumInitial": 1, "hotkeys": { "equation": "Ctrl-E", "itemize": "Ctrl-I" }, "labels_anchors": false, "latex_user_defs": false, "report_style_numbering": false, "user_envs_cfg": false }, "toc": { "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": true, "toc_position": { "height": "759px", "left": "0px", "right": "1228px", "top": "67px", "width": "212px" }, "toc_section_display": "block", "toc_window_display": true }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 4 }