<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Python on Nansen Li's Blog
李楠森的博客</title><link>https://nansenli.com/tags/python/</link><description>Recent content in Python on Nansen Li's Blog
李楠森的博客</description><generator>Hugo -- gohugo.io</generator><language>en</language><copyright>Nansen Li 🌈 （李楠森）</copyright><lastBuildDate>Sun, 11 May 2025 21:02:53 +0100</lastBuildDate><atom:link href="https://nansenli.com/tags/python/index.xml" rel="self" type="application/rss+xml"/><item><title>Learning Python uv Package</title><link>https://nansenli.com/post/2025/05/pythonuv/</link><pubDate>Sun, 11 May 2025 21:02:53 +0100</pubDate><guid>https://nansenli.com/post/2025/05/pythonuv/</guid><description>&lt;p>Recently, I&amp;rsquo;ve been looking into MCP-related content, and many sources mention using uv to manage Python package dependencies. After learning more about it, uv appears to be a very useful tool. Here&amp;rsquo;s a summary of its functionality.&lt;/p>
&lt;h3 id="what-exactly-is-uv">What exactly is uv?
&lt;/h3>&lt;p>First of all, uv is a package written in Rust for Python. After installation, you can use the uv command.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-javascript" data-lang="javascript">&lt;span class="line">&lt;span class="cl">&lt;span class="nx">pip&lt;/span> &lt;span class="nx">install&lt;/span> &lt;span class="nx">uv&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h3 id="what-can-uv-do">What can uv do?
&lt;/h3>&lt;ol>
&lt;li>It can replace pip&lt;/li>
&lt;/ol>
&lt;p>Simply add uv before your regular pip commands. But why use uv instead of pip? The answer is speed - according to statistics, it&amp;rsquo;s 77 times faster than pip.&lt;/p>
&lt;ol start="2">
&lt;li>It can replace venv or virtualenv&lt;/li>
&lt;/ol>
&lt;p>You need one less package, and the commands are cleaner. uv is much faster than &lt;code>venv&lt;/code> and &lt;code>virtualenv&lt;/code> when creating virtual environments and installing packages.&lt;/p>
&lt;ol start="3">
&lt;li>It can replace poetry&lt;/li>
&lt;/ol>
&lt;p>You might ask, what is poetry? Poetry is a tool for more convenient management of Python dependencies. So why not just use pip + requirements? The answer is that it&amp;rsquo;s cumbersome. Additionally, requirements files cannot lock the dependencies of dependencies. For example, if I lock package A to a specific version, but A depends on B, C, and D, this often leads to the &amp;ldquo;works on my machine but not on others&amp;rdquo; problem. Now, with uv, poetry can also be replaced.&lt;/p>
&lt;ol start="4">
&lt;li>It can replace pipx&lt;/li>
&lt;/ol>
&lt;p>Another question: what is pipx? pipx is a tool for installing Python utilities without polluting your environment.&lt;/p>
&lt;p>For example, if I need to install a command-line tool, I would run:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-javascript" data-lang="javascript">&lt;span class="line">&lt;span class="cl">&lt;span class="nx">pip&lt;/span> &lt;span class="nx">install&lt;/span> &lt;span class="nx">httpie&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>But this might pollute my environment, and uninstalling could have side effects. Now, I can run:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-javascript" data-lang="javascript">&lt;span class="line">&lt;span class="cl">&lt;span class="nx">pipx&lt;/span> &lt;span class="nx">install&lt;/span> &lt;span class="nx">httpie&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>This won&amp;rsquo;t pollute the environment.&lt;/p>
&lt;p>However, this command has also been replaced. Now you can run:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-javascript" data-lang="javascript">&lt;span class="line">&lt;span class="cl">&lt;span class="nx">uv&lt;/span> &lt;span class="nx">tool&lt;/span> &lt;span class="nx">install&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>To summarize uv in one sentence: Python dependency management&amp;rsquo;s &amp;ldquo;next-generation de facto standard.&amp;rdquo;&lt;/p></description></item><item><title>A Discussion on Python Decorators</title><link>https://nansenli.com/post/jianshu/python/python-%E8%A3%85%E9%A5%B0%E5%99%A8%E6%BC%AB%E8%B0%88/</link><pubDate>Tue, 04 Sep 2018 08:54:00 +0000</pubDate><guid>https://nansenli.com/post/jianshu/python/python-%E8%A3%85%E9%A5%B0%E5%99%A8%E6%BC%AB%E8%B0%88/</guid><description>&lt;p>Reference: &lt;a class="link" href="http://python.jobbole.com/81683/" target="_blank" rel="noopener"
>http://python.jobbole.com/81683/&lt;/a>&lt;/p>
&lt;ul>
&lt;li>
&lt;p>Python&amp;rsquo;s Namespace
Any variable is created within its own namespace. If your own namespace doesn&amp;rsquo;t contain this name, a new one will be created, rather than overriding the upper namespace. When accessing, if your namespace doesn&amp;rsquo;t contain it, the upper namespace will be accessed.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Python&amp;rsquo;s Closures
Any function is an object. You can create a callable object. If you create a callable function that uses a local variable in its internal process, this involves a closure: a function nested and defined in a non-global scope can remember the enclosing namespace where it was defined.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;p>In simple terms: a child function can use local variables from its parent function, and this behavior is called a closure.&lt;/p>
&lt;ul>
&lt;li>
&lt;p>Closure Details in C++ vs Python
In most languages, a parent function&amp;rsquo;s local variables end their lifecycle when the parent function exits. When lambda functions or function objects reference these local variables, problems can arise. If a lambda or function object can remember the current namespace during its definition, that&amp;rsquo;s a closure. In C++, local variable lifecycles are the same as their parent function, so what to do? Use the mutable modifier, which makes the variable&amp;rsquo;s lifecycle as long as the function object or lambda expression, otherwise capturing these variables might cause problems. Python captures all objects in the closure, preventing their release until the function object is collected.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Decorators
Sometimes we need to modify functions in similar ways. For example, we might want to calculate a function&amp;rsquo;s execution time by adding timestamps at the beginning and end, then printing at the end. Or we might need to validate parameters and return values. We could call validation functions at the start and end of each function.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;p>Decorators can accomplish this. For example, if we have a function:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">def func():
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> time.sleep(1)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>We can use another function that returns a function object with closure to achieve this:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;span class="lnt">6
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-python" data-lang="python">&lt;span class="line">&lt;span class="cl">&lt;span class="k">def&lt;/span> &lt;span class="nf">addTimePrint&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">func&lt;/span>&lt;span class="p">):&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">def&lt;/span> &lt;span class="nf">retFuncObj&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="o">*&lt;/span>&lt;span class="n">args&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="o">**&lt;/span>&lt;span class="n">kwargs&lt;/span>&lt;span class="p">):&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">starttime&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">time&lt;/span>&lt;span class="p">()&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">func&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="o">*&lt;/span>&lt;span class="n">args&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="o">**&lt;/span>&lt;span class="n">kwargs&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">print&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">time&lt;/span>&lt;span class="p">()&lt;/span> &lt;span class="o">-&lt;/span> &lt;span class="n">starttime&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">return&lt;/span> &lt;span class="n">retFuncObj&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>Then we need to replace the original func function:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-python" data-lang="python">&lt;span class="line">&lt;span class="cl">&lt;span class="n">func&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">addTimePrint&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">func&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>Python provides syntactic sugar: after defining addTimePrint, use @addTimePrint to complete both function definition and decoration:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">@addTimePrint
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">def func():
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> time.sleep(1)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div></description></item><item><title>TensorFlow: A Simple Introduction Case Study</title><link>https://nansenli.com/post/jianshu/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/tensorflow-%E4%B8%80%E4%B8%AA%E7%AE%80%E5%8D%95%E7%9A%84%E5%85%A5%E9%97%A8%E7%94%A8%E4%BE%8B/</link><pubDate>Wed, 09 Aug 2017 07:08:00 +0000</pubDate><guid>https://nansenli.com/post/jianshu/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/tensorflow-%E4%B8%80%E4%B8%AA%E7%AE%80%E5%8D%95%E7%9A%84%E5%85%A5%E9%97%A8%E7%94%A8%E4%BE%8B/</guid><description>&lt;h2 id="case-study">Case Study
&lt;/h2>&lt;p>Suppose I have a series of x-y data points with a linear relationship between x and y. How would we fit a straight line to this data?&lt;/p>
&lt;p>In the program below, train_x is a series of numbers between -1 and 1, and train_y is twice x plus 10, with a random number between 0 and 1 added.
Next, we build the model. X and Y are tensor placeholders waiting to be initialized. During optimization, the XY values in the model will continuously change to data from train_x and train_y, and then the optimizer will optimize by changing the slope w and intercept b in a direction that reduces error. Through iteration, w and b will eventually make the model fit the data.&lt;/p>
&lt;p>After the model is built, we start running it. First, we open a session, and we must remember to initialize all variables. Next, we iterate through all the data 10 times. In each iteration, we input a coordinate, calculate the error, and use gradient descent to correct w and b. Finally, we output the calculated values of w and b.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-gdscript3" data-lang="gdscript3">&lt;span class="line">&lt;span class="cl">&lt;span class="n">import&lt;/span> &lt;span class="n">tensorflow&lt;/span> &lt;span class="n">as&lt;/span> &lt;span class="n">tf&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">import&lt;/span> &lt;span class="n">numpy&lt;/span> &lt;span class="n">as&lt;/span> &lt;span class="n">np&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">train_x&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">np&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">linspace&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="o">-&lt;/span>&lt;span class="mi">1&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="mi">1&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="mi">101&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">train_y&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="mi">2&lt;/span> &lt;span class="o">*&lt;/span> &lt;span class="n">train_x&lt;/span> &lt;span class="o">+&lt;/span> &lt;span class="n">np&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">random&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">rand&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">train_x&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">shape&lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="mi">0&lt;/span>&lt;span class="p">])&lt;/span> &lt;span class="o">+&lt;/span> &lt;span class="mi">10&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">X&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">placeholder&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;float&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">Y&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">placeholder&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;float&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">w&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">Variable&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="mf">0.0&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">name&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;w&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">b&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">Variable&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="mf">0.0&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">name&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;b&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">loss&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">square&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">Y&lt;/span> &lt;span class="o">-&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">multiply&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">X&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="n">w&lt;/span>&lt;span class="p">)&lt;/span> &lt;span class="o">-&lt;/span> &lt;span class="n">b&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">train_op&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">train&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">GradientDescentOptimizer&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="mf">0.01&lt;/span>&lt;span class="p">)&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">minimize&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">loss&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">with&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">Session&lt;/span>&lt;span class="p">()&lt;/span> &lt;span class="n">as&lt;/span> &lt;span class="n">session&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">session&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">run&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">global_variables_initializer&lt;/span>&lt;span class="p">())&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">for&lt;/span> &lt;span class="n">i&lt;/span> &lt;span class="ow">in&lt;/span> &lt;span class="nb">range&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="mi">10&lt;/span>&lt;span class="p">):&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">for&lt;/span> &lt;span class="n">x&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="n">y&lt;/span> &lt;span class="ow">in&lt;/span> &lt;span class="n">zip&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">train_x&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">train_y&lt;/span>&lt;span class="p">):&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">session&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">run&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">train_op&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">feed_dict&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="p">{&lt;/span>&lt;span class="n">X&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="n">x&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">Y&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="n">y&lt;/span>&lt;span class="p">})&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">print&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;w: &amp;#34;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">session&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">run&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">w&lt;/span>&lt;span class="p">))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">print&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;b: &amp;#34;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">session&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">run&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">b&lt;/span>&lt;span class="p">))&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>Running results:
&lt;img src="http://otwwkzjm5.bkt.clouddn.com/17-8-9/99539186.jpg"
loading="lazy"
alt="Results"
>&lt;/p></description></item><item><title>Using TensorFlow for a Kaggle Task — Titanic: Machine Learning from Disaster</title><link>https://nansenli.com/post/jianshu/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/%E4%BD%BF%E7%94%A8tensorflow%E5%AE%8C%E6%88%90kaggle%E4%BB%BB%E5%8A%A1%E6%B3%B0%E5%9D%A6%E5%B0%BC%E5%85%8B%E5%8F%B7titanic--machine-learning-from-disaster/</link><pubDate>Fri, 04 Aug 2017 10:08:00 +0000</pubDate><guid>https://nansenli.com/post/jianshu/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/%E4%BD%BF%E7%94%A8tensorflow%E5%AE%8C%E6%88%90kaggle%E4%BB%BB%E5%8A%A1%E6%B3%B0%E5%9D%A6%E5%B0%BC%E5%85%8B%E5%8F%B7titanic--machine-learning-from-disaster/</guid><description>&lt;h2 id="import-necessary-libraries">Import Necessary Libraries
&lt;/h2>&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">import csv
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">import tensorflow as tf
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">import numpy as np
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">import random
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">import sys
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">import pandas as pd
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">from pandas import DataFrame
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">from __future__ import absolute_import
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">from __future__ import division
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">from __future__ import print_function
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="read-source-file-and-display">Read Source File and Display
&lt;/h2>&lt;p>In this section, we&amp;rsquo;ll work with basic CSV operations and display the results.
We&amp;rsquo;ll read the train.csv file downloaded from Kaggle and show its contents.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">trainFilePath = &amp;#39;./train.csv&amp;#39;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">trainSize = 0
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">def testCSV(filePath):
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> with open(filePath, &amp;#39;rb&amp;#39;) as trainFile:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> global trainSize
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> csvReader = csv.reader(trainFile)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> dataList = [data for data in csvReader]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> df = DataFrame(dataList[1:], columns=dataList[0])
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> trainSize = len(df)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(df)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(&amp;#34;trainSize&amp;#34;, trainSize)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">testCSV(trainFilePath)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="read-source-file-extract-data-and-build-neural-network">Read Source File, Extract Data, and Build Neural Network
&lt;/h2>&lt;p>In this section, we&amp;rsquo;ll extract gender, class, ticket fare, and SibSp from the source file to fit the survival probability.
Then we&amp;rsquo;ll build a 5-layer neural network with 3 hidden layers containing 4-10-20-10-2 neurons respectively.
Finally, we&amp;rsquo;ll execute the reading function.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;span class="lnt">21
&lt;/span>&lt;span class="lnt">22
&lt;/span>&lt;span class="lnt">23
&lt;/span>&lt;span class="lnt">24
&lt;/span>&lt;span class="lnt">25
&lt;/span>&lt;span class="lnt">26
&lt;/span>&lt;span class="lnt">27
&lt;/span>&lt;span class="lnt">28
&lt;/span>&lt;span class="lnt">29
&lt;/span>&lt;span class="lnt">30
&lt;/span>&lt;span class="lnt">31
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">def readTrainDataCSV(filePath):
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> global trainData, targetData, classifier
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> with open(filePath, &amp;#39;rb&amp;#39;) as trainFile:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> csvReader = csv.reader(trainFile)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> dataList = [data for data in csvReader]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> dataSize = len(dataList) - 1
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> trainData = np.ndarray((dataSize, 4), dtype=np.float32)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> targetData = np.ndarray((dataSize, 1), dtype=np.int32)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> trainDataFrame = DataFrame(dataList[1:], columns=dataList[0])
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> trainDataFrame_fliter = trainDataFrame.loc[:,[&amp;#39;Pclass&amp;#39;,&amp;#39;Sex&amp;#39;,&amp;#39;SibSp&amp;#39;,&amp;#39;Fare&amp;#39;,&amp;#39;Survived&amp;#39;]]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> for i in range(dataSize):
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> thisData = np.array(trainDataFrame_fliter.iloc[i])
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Pclass,Sex,SibSp,Fare,Survived = thisData
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Pclass = float(Pclass)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Sex = 0 if Sex == &amp;#39;female&amp;#39; else 1
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> SibSp = float(SibSp)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Fare = float(Fare)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Survived = int(Survived)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(Pclass,Sex,SibSp,Fare,Survived)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> trainData[i,:] = [Pclass,Sex,SibSp,Fare]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> targetData[i,:] = [Survived]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(thisData)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(trainData)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(targetData)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> feature_columns = [tf.contrib.layers.real_valued_column(&amp;#34;&amp;#34;, dimension=4)]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> classifier = tf.contrib.learn.DNNClassifier(feature_columns=feature_columns,
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> hidden_units=[10, 20, 10],
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> n_classes=2)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># model_dir=&amp;#34;/tmp/titanic_model&amp;#34;)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">readTrainDataCSV(trainFilePath)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="create-input-data">Create Input Data
&lt;/h2>&lt;p>We&amp;rsquo;ll wrap the training data and labels into a tuple and return it.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;span class="lnt">6
&lt;/span>&lt;span class="lnt">7
&lt;/span>&lt;span class="lnt">8
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-gdscript3" data-lang="gdscript3">&lt;span class="line">&lt;span class="cl">&lt;span class="k">def&lt;/span> &lt;span class="nf">get_train_inputs&lt;/span>&lt;span class="p">():&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">x&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">constant&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">trainData&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">y&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">constant&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">targetData&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">print&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">x&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">print&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">y&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">return&lt;/span> &lt;span class="n">x&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">y&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">get_train_inputs&lt;/span>&lt;span class="p">()&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="train-the-model">Train the Model
&lt;/h2>&lt;p>Now we start training the neural network.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">def train():
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> classifier.fit(input_fn=get_train_inputs, steps=2000)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">train()
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="check-accuracy">Check Accuracy
&lt;/h2>&lt;p>We use the entire dataset to check accuracy. Note that we should use a validation set for this task, but since this is just for demonstration purposes, we&amp;rsquo;ll skip that step.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">accuracy_score = classifier.evaluate(input_fn=get_train_inputs,
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> steps=1)[&amp;#34;accuracy&amp;#34;]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">print(&amp;#34;accuracy:&amp;#34;,accuracy_score)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="read-test-set-and-output-results">Read Test Set and Output Results
&lt;/h2>&lt;p>In this section, we&amp;rsquo;ll read the test data from Kaggle and output the results to a file, which will ultimately be submitted to the official website.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;span class="lnt">21
&lt;/span>&lt;span class="lnt">22
&lt;/span>&lt;span class="lnt">23
&lt;/span>&lt;span class="lnt">24
&lt;/span>&lt;span class="lnt">25
&lt;/span>&lt;span class="lnt">26
&lt;/span>&lt;span class="lnt">27
&lt;/span>&lt;span class="lnt">28
&lt;/span>&lt;span class="lnt">29
&lt;/span>&lt;span class="lnt">30
&lt;/span>&lt;span class="lnt">31
&lt;/span>&lt;span class="lnt">32
&lt;/span>&lt;span class="lnt">33
&lt;/span>&lt;span class="lnt">34
&lt;/span>&lt;span class="lnt">35
&lt;/span>&lt;span class="lnt">36
&lt;/span>&lt;span class="lnt">37
&lt;/span>&lt;span class="lnt">38
&lt;/span>&lt;span class="lnt">39
&lt;/span>&lt;span class="lnt">40
&lt;/span>&lt;span class="lnt">41
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">testFilePath = &amp;#39;./test.csv&amp;#39;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">def readTestDataCSV(filePath):
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> global testData, PassengerIdStart
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> with open(filePath, &amp;#39;rb&amp;#39;) as testFile:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> csvReader = csv.reader(testFile)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> dataList = [data for data in csvReader]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> dataSize = len(dataList)-1
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> trainDataFrame = DataFrame(dataList[1:], columns=dataList[0])
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> trainDataFrame_fliter = trainDataFrame.loc[:,[&amp;#39;Pclass&amp;#39;,&amp;#39;Sex&amp;#39;,&amp;#39;SibSp&amp;#39;,&amp;#39;Fare&amp;#39;]]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> testData = np.ndarray((dataSize, 4), dtype=np.float32)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> PassengerIdStart = trainDataFrame[&amp;#39;PassengerId&amp;#39;][0]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> PassengerIdStart = int(PassengerIdStart)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(&amp;#39;PassengerId&amp;#39;,PassengerIdStart)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> for i in range(dataSize):
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> thisData = np.array(trainDataFrame_fliter.iloc[i])
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Pclass,Sex,SibSp,Fare = thisData
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Pclass = float(Pclass)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Sex = 0 if Sex == &amp;#39;female&amp;#39; else 1
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> SibSp = float(SibSp)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Fare = 0 if Fare==&amp;#39;&amp;#39; else float(Fare)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(Pclass,Sex,SibSp,Fare)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> testData[i,:] = [Pclass,Sex,SibSp,Fare]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(thisData)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(testData)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">def testData_samples():
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> return testData
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">readTestDataCSV(testFilePath)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">predictions = list(classifier.predict(input_fn=testData_samples))
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">print(predictions)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">with open(&amp;#39;predictions.csv&amp;#39;, &amp;#39;wb&amp;#39;) as csvfile:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> writer = csv.writer(csvfile, dialect=&amp;#39;excel&amp;#39;)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> writer.writerow([&amp;#39;PassengerId&amp;#39;,&amp;#39;Survived&amp;#39;])
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> PassengerId = PassengerIdStart
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> for i in predictions:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> writer.writerow([PassengerId, i])
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> PassengerId += 1
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>Finally, using only 4 features, we achieved an accuracy of 75%. The next goal is to utilize the other available data.&lt;/p></description></item><item><title>Building a Personal Blog with Flask</title><link>https://nansenli.com/post/jianshu/%E7%BD%91%E7%BB%9C/%E5%9F%BA%E4%BA%8Eflask%E7%9A%84%E4%B8%AA%E4%BA%BA%E5%8D%9A%E5%AE%A2%E6%90%AD%E5%BB%BA/</link><pubDate>Sun, 30 Jul 2017 17:38:00 +0000</pubDate><guid>https://nansenli.com/post/jianshu/%E7%BD%91%E7%BB%9C/%E5%9F%BA%E4%BA%8Eflask%E7%9A%84%E4%B8%AA%E4%BA%BA%E5%8D%9A%E5%AE%A2%E6%90%AD%E5%BB%BA/</guid><description>&lt;h2 id="introduction">Introduction
&lt;/h2>&lt;p>The source code can be run directly, supports markdown syntax, supports email registration, and any issues can be referenced in &lt;a class="link" href="https://book.douban.com/subject/26274202/" target="_blank" rel="noopener"
>《Flask Web Development: Web Application Development in Action with Python》&lt;/a>.&lt;/p>
&lt;p>Blog website: &lt;a class="link" href="http://www.unrealblue.cc" target="_blank" rel="noopener"
>http://www.unrealblue.cc&lt;/a>
Project repository: &lt;a class="link" href="https://github.com/linanwx/unrealblue-blog" target="_blank" rel="noopener"
>https://github.com/linanwx/unrealblue-blog&lt;/a>&lt;/p>
&lt;h2 id="preview">Preview
&lt;/h2>&lt;p>&lt;img src="https://nansenli.com/img/rescued/a0027251f6b8.jpg"
loading="lazy"
alt="image.png"
>&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/ee8b6a7c6286.jpg"
loading="lazy"
alt="markdown syntax"
>&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/2482752767fe.jpg"
loading="lazy"
alt="login"
>&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/a827700f36d3.jpg"
loading="lazy"
alt="homepage"
>&lt;/p>
&lt;h2 id="deployment-process">Deployment Process
&lt;/h2>&lt;p>First, to verify the program functionality, you need to deploy it on your local machine. After that, you can use nginx as a reverse proxy tool to expose the port, so that others can access your blog on the public network. Then follow the same steps on the server. The specific process is as follows:&lt;/p>
&lt;ol>
&lt;li>Install the virtualenv Python virtual environment with &lt;code>pip install virtualenv&lt;/code> or &lt;code>pip3 install virtualenv&lt;/code>. Then use virtualenv to create a venv environment in a suitable directory, for example under this project with &lt;code>virtualenv venv&lt;/code>.&lt;/li>
&lt;li>Activate the virtualenv environment by running the activate script in the venv directory: &lt;code>. venv/bin/activate&lt;/code>. Note the position of the dot. After this, you will see the (venv) marker at the beginning of the command line.&lt;/li>
&lt;li>Install all modules listed in requirements.txt within the virtual environment: &lt;code>pip3 install -r requirements.txt&lt;/code>. If installation is too slow, you may need to configure a domestic pip source. See the pip official page for how to change the pip source.&lt;/li>
&lt;li>Import environment variables by creating an &lt;code>env&lt;/code> file in the project directory with the following fields:&lt;/li>
&lt;/ol>
&lt;ul>
&lt;li>&lt;code>MAIL_USERNAME=email@example.com&lt;/code> (Email used by the server to send verification codes, an email account with smtp service enabled. The program uses QQ email by default, modify the config file to use other types of email)&lt;/li>
&lt;li>&lt;code>MAIL_PASSWORD=password&lt;/code> (Password for the above email, note that QQ email uses a special 16-digit password)&lt;/li>
&lt;li>&lt;code>FLASK_ADMIN=admin@example.com&lt;/code> (After the server is running, an account created with this email will be the administrator)&lt;/li>
&lt;li>&lt;code>MAIL_SERVER=smtp.qq.com&lt;/code> (Email server address)&lt;/li>
&lt;li>&lt;code>FLASKY_MAIL_SENDER=example@foxmail.com&lt;/code> (Sender&amp;rsquo;s email)&lt;/li>
&lt;/ul>
&lt;ol start="5">
&lt;li>Set up database migration by entering the following commands:&lt;/li>
&lt;/ol>
&lt;ul>
&lt;li>&lt;code>python manager.py db init&lt;/code> (The init command creates a migration repository, which will add a migrations folder)&lt;/li>
&lt;li>&lt;code>python manager.py db migrate -m &amp;quot;initial migration&amp;quot;&lt;/code> (The migrate command is used to automatically create migration scripts)&lt;/li>
&lt;li>&lt;code>python manager.py db upgrade&lt;/code> (Update the database. The first time you use this command, it will create a new database called data-dev.sqlite)&lt;/li>
&lt;/ul>
&lt;ol start="6">
&lt;li>Deploy the program: &lt;code>python manager.py deploy&lt;/code>&lt;/li>
&lt;li>Run the program locally: &lt;code>python manager.py runserver&lt;/code>. Open http://127.0.0.1:5000 to view it, press Ctrl+C to exit the program.&lt;/li>
&lt;li>If running on a server and you want to preserve data, you can copy the migrations folder and the data-dev.sqlite database to the server, then use &lt;code>./venv/bin/gunicorn -w 4 -b 127.0.0.1:8080 manager:app&lt;/code>. At this point, you should be able to see the webpage on port 8080, and this port is exposed to the external network. Enter the server address in your local browser, and you will be able to see the blog.&lt;/li>
&lt;/ol></description></item><item><title>Deep Learning Beginner's Study Notes One (Udacity)</title><link>https://nansenli.com/post/jianshu/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E9%9B%B6%E5%9F%BA%E7%A1%80%E5%AD%A6%E4%B9%A0%E7%AC%94%E8%AE%B0%E4%B8%80%E4%BC%98%E8%BE%BE%E5%AD%A6%E5%9F%8E/</link><pubDate>Wed, 19 Jul 2017 07:42:00 +0000</pubDate><guid>https://nansenli.com/post/jianshu/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E9%9B%B6%E5%9F%BA%E7%A1%80%E5%AD%A6%E4%B9%A0%E7%AC%94%E8%AE%B0%E4%B8%80%E4%BC%98%E8%BE%BE%E5%AD%A6%E5%9F%8E/</guid><description>&lt;h2 id="introduction">Introduction
&lt;/h2>&lt;p>On a whim, I decided to learn machine learning. These are notes from my learning process.&lt;/p>
&lt;h2 id="preparation">Preparation
&lt;/h2>&lt;p>I made these preparations:&lt;/p>
&lt;ul>
&lt;li>A MacBook with Python environment set up, and numpy and matplotlib installed&lt;/li>
&lt;li>Registered for Udacity&amp;rsquo;s free &amp;ldquo;Deep Learning&amp;rdquo; course (in collaboration with Google)&lt;/li>
&lt;li>Studied Liao Xuefeng&amp;rsquo;s Python introductory tutorial&lt;/li>
&lt;li>Spent two days roughly browsing through &amp;ldquo;Machine Learning in Action&amp;rdquo;&lt;/li>
&lt;/ul>
&lt;p>Learning these fundamentals should be sufficient for the upcoming Udacity course.&lt;/p>
&lt;h2 id="course-one-from-machine-learning-to-deep-learning">Course One: From Machine Learning to Deep Learning
&lt;/h2>&lt;p>&lt;img src="https://nansenli.com/img/rescued/71c43f961173.png"
loading="lazy"
alt="Introduction"
>&lt;/p>
&lt;p>Sections 1-8 mainly introduce the current state of deep learning and related knowledge.&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/ff643a3a1ec4.png"
loading="lazy"
alt="image.png"
>&lt;/p>
&lt;p>Sections 9-12 introduce the softmax model.&lt;/p>
&lt;p>After a rough review of &amp;ldquo;Machine Learning in Action,&amp;rdquo; I learned that machine learning consists of several classification and clustering algorithms. On the surface, machine learning appears to be a collection of classification and clustering algorithms. Among these algorithms, one called logistic regression classification was introduced.&lt;/p>
&lt;p>In sections 9-12, the focus is on the classifier model—logistic regression, using the softmax function as the classification function.&lt;/p>
&lt;ul>
&lt;li>What is the softmax function?&lt;/li>
&lt;/ul>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/cb4b58f90872.jpg"
loading="lazy"
alt="softmax"
>&lt;/p>
&lt;p>This image illustrates what a softmax function is. For each number z in the original sequence, we calculate exp(z), and the proportion of each new number&amp;rsquo;s magnitude becomes the softmax probability for that number.&lt;/p>
&lt;ul>
&lt;li>Properties&lt;/li>
&lt;/ul>
&lt;p>If the inputs are scaled up proportionally, the classifier&amp;rsquo;s results become more polarized and confident. If the inputs are scaled down proportionally, the classifier&amp;rsquo;s results tend toward the average and lack confidence.&lt;/p>
&lt;ul>
&lt;li>Algorithm&lt;/li>
&lt;/ul>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;span class="lnt">6
&lt;/span>&lt;span class="lnt">7
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-python" data-lang="python">&lt;span class="line">&lt;span class="cl">&lt;span class="kn">import&lt;/span> &lt;span class="nn">numpy&lt;/span> &lt;span class="k">as&lt;/span> &lt;span class="nn">np&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">def&lt;/span> &lt;span class="nf">softmax&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">x&lt;/span>&lt;span class="p">):&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="s2">&amp;#34;&amp;#34;&amp;#34;Compute softmax values for each sets of scores in x.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">expList&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="p">[&lt;/span>&lt;span class="n">np&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">exp&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">i&lt;/span>&lt;span class="p">)&lt;/span> &lt;span class="k">for&lt;/span> &lt;span class="n">i&lt;/span> &lt;span class="ow">in&lt;/span> &lt;span class="n">x&lt;/span>&lt;span class="p">]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">expSum&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="nb">sum&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">expList&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">x&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="p">[&lt;/span>&lt;span class="n">i&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="n">expSum&lt;/span> &lt;span class="k">for&lt;/span> &lt;span class="n">i&lt;/span> &lt;span class="ow">in&lt;/span> &lt;span class="n">expList&lt;/span>&lt;span class="p">]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">return&lt;/span> &lt;span class="n">np&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">array&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">x&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>&lt;img src="https://nansenli.com/img/rescued/8ba75ea46168.png"
loading="lazy"
alt="image.png"
>&lt;/p>
&lt;p>Sections 13-14 mainly discuss One-Hot encoding. After the softmax function provides a sequence of probability values, how do we determine the classification? For example, a sequence where the highest probability is 1 and others are 0 is called One-Hot encoding. This type of encoding has already determined the classification.&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/d0050afa0519.png"
loading="lazy"
alt="Cross-entropy"
>&lt;/p>
&lt;p>Sections 15-16 cover cross-entropy. Softmax can calculate a probability sequence, and OneHot is a determined classification. So how do we calculate the distance from a probability sequence to a specific classification? We use cross-entropy to measure this distance.&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/7a800c91612a.png"
loading="lazy"
alt="image.png"
>&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/a1e862b4bd40.png"
loading="lazy"
alt="image.png"
>&lt;/p>
&lt;p>Sections 17-20 explain how to use this classifier. Section 18 specifically discusses why special initial data is needed.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;span class="lnt">6
&lt;/span>&lt;span class="lnt">7
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">sum = 1000000000
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">for i in range(1000000):
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> sum += 0.000001
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sum -= 1000000000
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">print(sum)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>The result of running this code is not 1. If we change sum to a very small number, like 1, instead of 1000000000, we find that the error becomes smaller. Based on this reason, we want our initial data to always have a mean of 0 and consistent variance in all directions. For example, for a grayscale image with pixel values from 0-255, we need to subtract 128 and then divide by 128, so that each number is between -1 and 1. Such initial data is more suitable for training.&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/34f543ea3c1e.png"
loading="lazy"
alt="image.png"
>&lt;/p>
&lt;p>This way, we can proceed with training. Reviewing the video content: xi is the training data matrix, w is a random weight matrix. For performance reasons, random values are taken from a normal distribution with an axis of 0 and very small variance. Then we calculate the probability sequence and the distance to the target. Then we compute the average distance to all targets. Our goal is to make this distance smaller, so we optimize the weight matrix along the direction of gradient descent while optimizing the intercept b. We repeat this process continuously until we reach a local optimum.&lt;/p>
&lt;ul>
&lt;li>Installing Docker&lt;/li>
&lt;/ul>
&lt;p>&lt;a class="link" href="https://www.docker-cn.com/community-edition#/download" target="_blank" rel="noopener"
>https://www.docker-cn.com/community-edition#/download&lt;/a>&lt;/p>
&lt;p>Configure the official Chinese mirror.&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/1a9fbd212a79.png"
loading="lazy"
alt="image.png"
>&lt;/p>
&lt;ul>
&lt;li>Installing Jupyter Notebook&lt;/li>
&lt;/ul>
&lt;p>&lt;code>$ pip3 install jupyter&lt;/code>
&lt;code>$ jupyter notebook&lt;/code>
You can now use the jupyter notebook command to open a Jupyter editor.&lt;/p>
&lt;ul>
&lt;li>Setting up TensorFlow environment&lt;/li>
&lt;/ul>
&lt;p>&lt;code>$ docker run -it -p 8888:8888 tensorflow/tensorflow&lt;/code>
Running this command will automatically download the TensorFlow image, provided that the repository mirror is set to a Chinese mirror; otherwise, the download will be very slow. After running the command, you&amp;rsquo;ll be prompted to open a webpage. When you open this URL, you&amp;rsquo;ll see the TensorFlow Jupyter editing environment, assuming Jupyter Notebook is installed correctly.&lt;/p>
&lt;ul>
&lt;li>Mounting Docker&amp;rsquo;s file directory
We need to import the official assignments. Close the container, reopen it, and use &lt;code>-v host_directory:container_directory&lt;/code> for mounting.
&lt;code>docker run -v /Users/hahaha/tensorflow/:/notebooks -it -p 8888:8888 tensorflow/tensorflow&lt;/code>&lt;/li>
&lt;/ul>
&lt;p>Where /Users/hahaha/tensorflow/ is a folder on my Mac, and notebooks is the default Jupyter editing directory in TensorFlow.&lt;/p>
&lt;p>Paste the first assignment file, 1_notmnist.ipynb, into the mounted directory on the host. This file can be found here: &lt;a class="link" href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/udacity/1_notmnist.ipynb" target="_blank" rel="noopener"
>1_notmnist.ipynb&lt;/a>&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/7faf184ae282.png"
loading="lazy"
alt="Assignment One Content"
>&lt;/p>
&lt;h2 id="assignment-code-segment-one">Assignment Code Segment One
&lt;/h2>&lt;p>First, run the import statements in the first code segment. There should be no errors. If you see red error output, it means these imports were not successful.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;span class="lnt">21
&lt;/span>&lt;span class="lnt">22
&lt;/span>&lt;span class="lnt">23
&lt;/span>&lt;span class="lnt">24
&lt;/span>&lt;span class="lnt">25
&lt;/span>&lt;span class="lnt">26
&lt;/span>&lt;span class="lnt">27
&lt;/span>&lt;span class="lnt">28
&lt;/span>&lt;span class="lnt">29
&lt;/span>&lt;span class="lnt">30
&lt;/span>&lt;span class="lnt">31
&lt;/span>&lt;span class="lnt">32
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-gdscript3" data-lang="gdscript3">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># These are all the modules we&amp;#39;ll be using later. Make sure you can import them&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># before proceeding further.&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">from&lt;/span> &lt;span class="n">__future__&lt;/span> &lt;span class="n">import&lt;/span> &lt;span class="n">print_function&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># print function&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">import&lt;/span> &lt;span class="n">matplotlib&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">pyplot&lt;/span> &lt;span class="n">as&lt;/span> &lt;span class="n">plt&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># plotting tool&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">import&lt;/span> &lt;span class="n">numpy&lt;/span> &lt;span class="n">as&lt;/span> &lt;span class="n">np&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># matrix calculations&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">import&lt;/span> &lt;span class="n">os&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># file paths&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">import&lt;/span> &lt;span class="n">sys&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># file output&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">import&lt;/span> &lt;span class="n">tarfile&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># decompression&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">from&lt;/span> &lt;span class="n">IPython&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">display&lt;/span> &lt;span class="n">import&lt;/span> &lt;span class="n">display&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="ne">Image&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># display images&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">from&lt;/span> &lt;span class="n">scipy&lt;/span> &lt;span class="n">import&lt;/span> &lt;span class="n">ndimage&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># image processing&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">from&lt;/span> &lt;span class="n">sklearn&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">linear_model&lt;/span> &lt;span class="n">import&lt;/span> &lt;span class="n">LogisticRegression&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># logistic regression module for linear models&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">from&lt;/span> &lt;span class="n">six&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">moves&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">urllib&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">request&lt;/span> &lt;span class="n">import&lt;/span> &lt;span class="n">urlretrieve&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># url handling&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">from&lt;/span> &lt;span class="n">six&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">moves&lt;/span> &lt;span class="n">import&lt;/span> &lt;span class="n">cPickle&lt;/span> &lt;span class="n">as&lt;/span> &lt;span class="n">pickle&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># data processing&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Config the matplotlib backend as plotting inline in IPython&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">%&lt;/span>&lt;span class="n">matplotlib&lt;/span> &lt;span class="n">inline&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># matplotlib is the most famous Python chart plotting extension library,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># it supports outputting various formats of graphical images, and can use various GUI interface libraries to display charts interactively.&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Using the %matplotlib command can embed matplotlib charts directly into the Notebook,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># or display charts using a specified interface library, it has a parameter specifying how matplotlib charts are displayed.&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># inline indicates embedding charts in the Notebook.&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="assignment-code-segment-two">Assignment Code Segment Two
&lt;/h2>&lt;p>Next is the second code segment, which will download letter sets for training and testing, approximately 300MB in size. After successful download, you can see these two files in the mounted directory.&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/66a1b044606a.png"
loading="lazy"
alt="Assignment"
>&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;span class="lnt">21
&lt;/span>&lt;span class="lnt">22
&lt;/span>&lt;span class="lnt">23
&lt;/span>&lt;span class="lnt">24
&lt;/span>&lt;span class="lnt">25
&lt;/span>&lt;span class="lnt">26
&lt;/span>&lt;span class="lnt">27
&lt;/span>&lt;span class="lnt">28
&lt;/span>&lt;span class="lnt">29
&lt;/span>&lt;span class="lnt">30
&lt;/span>&lt;span class="lnt">31
&lt;/span>&lt;span class="lnt">32
&lt;/span>&lt;span class="lnt">33
&lt;/span>&lt;span class="lnt">34
&lt;/span>&lt;span class="lnt">35
&lt;/span>&lt;span class="lnt">36
&lt;/span>&lt;span class="lnt">37
&lt;/span>&lt;span class="lnt">38
&lt;/span>&lt;span class="lnt">39
&lt;/span>&lt;span class="lnt">40
&lt;/span>&lt;span class="lnt">41
&lt;/span>&lt;span class="lnt">42
&lt;/span>&lt;span class="lnt">43
&lt;/span>&lt;span class="lnt">44
&lt;/span>&lt;span class="lnt">45
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-gdscript3" data-lang="gdscript3">&lt;span class="line">&lt;span class="cl">&lt;span class="n">url&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s1">&amp;#39;https://commondatastorage.googleapis.com/books1000/&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">last_percent_reported&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">None&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">data_root&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s1">&amp;#39;.&amp;#39;&lt;/span> &lt;span class="c1"># Change me to store data elsewhere&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">def&lt;/span> &lt;span class="nf">download_progress_hook&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">count&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">blockSize&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">totalSize&lt;/span>&lt;span class="p">):&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="s2">&amp;#34;&amp;#34;&amp;#34;A hook to report the progress of a download. This is mostly intended for users with
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> slow internet connections. Reports every 5&lt;/span>&lt;span class="si">% c&lt;/span>&lt;span class="s2">hange in download progress.
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Hook function to display download progress in real-time&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">global&lt;/span> &lt;span class="n">last_percent_reported&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">percent&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="ne">int&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">count&lt;/span> &lt;span class="o">*&lt;/span> &lt;span class="n">blockSize&lt;/span> &lt;span class="o">*&lt;/span> &lt;span class="mi">100&lt;/span> &lt;span class="o">/&lt;/span> &lt;span class="n">totalSize&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">if&lt;/span> &lt;span class="n">last_percent_reported&lt;/span> &lt;span class="o">!=&lt;/span> &lt;span class="n">percent&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">if&lt;/span> &lt;span class="n">percent&lt;/span> &lt;span class="o">%&lt;/span> &lt;span class="mi">5&lt;/span> &lt;span class="o">==&lt;/span> &lt;span class="mi">0&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">sys&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">stdout&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">write&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="si">%s%%&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="o">%&lt;/span> &lt;span class="n">percent&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">sys&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">stdout&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">flush&lt;/span>&lt;span class="p">()&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">else&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">sys&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">stdout&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">write&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;.&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">sys&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">stdout&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">flush&lt;/span>&lt;span class="p">()&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">last_percent_reported&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">percent&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">def&lt;/span> &lt;span class="nf">maybe_download&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">filename&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">expected_bytes&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">force&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="n">False&lt;/span>&lt;span class="p">):&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="s2">&amp;#34;&amp;#34;&amp;#34;Download a file if not present, and make sure it&amp;#39;s the right size.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">dest_filename&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">os&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">path&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">join&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">data_root&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">filename&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># data_root is the current directory, add the filename to it, set as the location to save the file&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">if&lt;/span> &lt;span class="n">force&lt;/span> &lt;span class="ow">or&lt;/span> &lt;span class="ow">not&lt;/span> &lt;span class="n">os&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">path&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">exists&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">dest_filename&lt;/span>&lt;span class="p">):&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># force is to force download, ignoring already downloaded files&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">print&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;Attempting to download:&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">filename&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">filename&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">_&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">urlretrieve&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">url&lt;/span> &lt;span class="o">+&lt;/span> &lt;span class="n">filename&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">dest_filename&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">reporthook&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="n">download_progress_hook&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Use urlretrieve to download the file, with the hook attached&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">print&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;&lt;/span>&lt;span class="se">\n&lt;/span>&lt;span class="s1">Download Complete!&amp;#39;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">statinfo&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">os&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">stat&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">dest_filename&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Get information about the downloaded file&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">if&lt;/span> &lt;span class="n">statinfo&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">st_size&lt;/span> &lt;span class="o">==&lt;/span> &lt;span class="n">expected_bytes&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Correct size&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">print&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;Found and verified&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">dest_filename&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">else&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Wrong size, prompt user to use a browser to download&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">raise&lt;/span> &lt;span class="n">Exception&lt;/span>&lt;span class="p">(&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="s1">&amp;#39;Failed to verify &amp;#39;&lt;/span> &lt;span class="o">+&lt;/span> &lt;span class="n">dest_filename&lt;/span> &lt;span class="o">+&lt;/span> &lt;span class="s1">&amp;#39;. Can you get to it with a browser?&amp;#39;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">return&lt;/span> &lt;span class="n">dest_filename&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">train_filename&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">maybe_download&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;notMNIST_large.tar.gz&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="mi">247336696&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">test_filename&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">maybe_download&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;notMNIST_small.tar.gz&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="mi">8458043&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="assignment-code-segment-three">Assignment Code Segment Three
&lt;/h2>&lt;p>Extracting use cases&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;span class="lnt">21
&lt;/span>&lt;span class="lnt">22
&lt;/span>&lt;span class="lnt">23
&lt;/span>&lt;span class="lnt">24
&lt;/span>&lt;span class="lnt">25
&lt;/span>&lt;span class="lnt">26
&lt;/span>&lt;span class="lnt">27
&lt;/span>&lt;span class="lnt">28
&lt;/span>&lt;span class="lnt">29
&lt;/span>&lt;span class="lnt">30
&lt;/span>&lt;span class="lnt">31
&lt;/span>&lt;span class="lnt">32
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">num_classes = 10
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Total number of digits
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">np.random.seed(133)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Initialize random seed
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">def maybe_extract(filename, force=False):
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Assuming already extracted
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> root = os.path.splitext(os.path.splitext(filename)[0])[0] # remove .tar.gz
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># splitext(filename)[0] removes one suffix, used twice to remove both suffixes, i.e., remove the .tar.gz suffix
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> if os.path.isdir(root) and not force:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> # You may override by setting force=True.
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># If already extracted, don&amp;#39;t extract again
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(&amp;#39;%s already present - Skipping extraction of %s.&amp;#39; % (root, filename))
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> else:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(&amp;#39;Extracting data for %s. This may take a while. Please wait.&amp;#39; % root)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> tar = tarfile.open(filename)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> sys.stdout.flush()
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> tar.extractall(data_root)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> tar.close()
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Extract to the current directory
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> data_folders = [
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> os.path.join(root, d) for d in sorted(os.listdir(root))
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> if os.path.isdir(os.path.join(root, d))]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> if len(data_folders) != num_classes:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> raise Exception(
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &amp;#39;Expected %d folders, one per class. Found %d instead.&amp;#39; % (
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> num_classes, len(data_folders)))
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(data_folders)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Check if the number of extracted directories matches expectations, and print the extracted directories
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> return data_folders
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">train_folders = maybe_extract(train_filename)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">test_folders = maybe_extract(test_filename)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="question-one">Question One
&lt;/h2>&lt;p>Write code to display information about the extracted file contents&lt;/p>
&lt;ul>
&lt;li>Reference answer&lt;/li>
&lt;/ul>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;span class="lnt">21
&lt;/span>&lt;span class="lnt">22
&lt;/span>&lt;span class="lnt">23
&lt;/span>&lt;span class="lnt">24
&lt;/span>&lt;span class="lnt">25
&lt;/span>&lt;span class="lnt">26
&lt;/span>&lt;span class="lnt">27
&lt;/span>&lt;span class="lnt">28
&lt;/span>&lt;span class="lnt">29
&lt;/span>&lt;span class="lnt">30
&lt;/span>&lt;span class="lnt">31
&lt;/span>&lt;span class="lnt">32
&lt;/span>&lt;span class="lnt">33
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">import random
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">import matplotlib.image as mpimg
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">def plot_samples(data_folders, sample_size, title=None):
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> fig = plt.figure()
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Create empty figure
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> if title: fig.suptitle(title, fontsize=16, fontweight=&amp;#39;bold&amp;#39;)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Add title
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> for folder in data_folders:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Loop through each letter
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> image_files = os.listdir(folder)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> image_sample = random.sample(image_files, sample_size)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Randomly select a certain number of images from that letter
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> for image in image_sample:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> image_file = os.path.join(folder, image)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> ax = fig.add_subplot(len(data_folders), sample_size, sample_size * data_folders.index(folder) +
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> image_sample.index(image) + 1)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Create a subplot
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> image = mpimg.imread(image_file)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Load subplot image
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> ax.imshow(image)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Display subplot image
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> ax.set_axis_off()
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Turn off subplot coordinate lines
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> fig.set_size_inches(18.5, 10.5)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Set the display size of the image
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> plt.show()
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">plot_samples(train_folders, 20, &amp;#39;Train&amp;#39;)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">plot_samples(test_folders, 20, &amp;#39;Test&amp;#39;)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>Running results:&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/94887f5b6eab.png"
loading="lazy"
alt="Training.png"
>&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/3930a2cfe6b0.png"
loading="lazy"
alt="Testing.png"
>&lt;/p>
&lt;p>As we can see, some of the training data has issues.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Assignment Code Segment Four
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">After this, we need to normalize the data, which means transforming each image pixel from 0~255 to -1.0~1.0, and persisting it to a file.
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>image_size = 28 # Pixel width and height.
pixel_depth = 255.0 # Number of levels per pixel.&lt;/p>
&lt;h1 id="image-width-height-and-pixel-depth">Image width, height and pixel depth
&lt;/h1>&lt;p>def load_letter(folder, min_num_images):
&amp;ldquo;&amp;ldquo;&amp;ldquo;Load the data for a single letter label.&amp;rdquo;&amp;rdquo;&amp;rdquo;&lt;/p>
&lt;h1 id="process-files-in-a-folder-belonging-to-one-letter">Process files in a folder belonging to one letter
&lt;/h1>&lt;p>image_files = os.listdir(folder)&lt;/p>
&lt;h1 id="list-all-files-in-that-directory">List all files in that directory
&lt;/h1>&lt;p>dataset = np.ndarray(shape=(len(image_files), image_size, image_size),
dtype=np.float32)&lt;/p>
&lt;h1 id="create-a-dataset-with-length-equal-to-number-of-files-width-and-height-of-28">Create a dataset with length equal to number of files, width and height of 28
&lt;/h1>&lt;p>print(folder)&lt;/p>
&lt;h1 id="print-directory">Print directory
&lt;/h1>&lt;p>num_images = 0&lt;/p>
&lt;h1 id="initialize-num_images">Initialize num_images
&lt;/h1>&lt;p>for image in image_files:&lt;/p>
&lt;h1 id="process-each-file">Process each file
&lt;/h1>&lt;pre>&lt;code>image_file = os.path.join(folder, image)
&lt;/code>&lt;/pre>
&lt;h1 id="get-complete-file-path">Get complete file path
&lt;/h1>&lt;pre>&lt;code>try:
image_data = (ndimage.imread(image_file).astype(float) -
pixel_depth / 2) / pixel_depth
&lt;/code>&lt;/pre>
&lt;h1 id="read-in-the-image-and-normalize-it">Read in the image and normalize it
&lt;/h1>&lt;pre>&lt;code> if image_data.shape != (image_size, image_size):
&lt;/code>&lt;/pre>
&lt;h1 id="check-image-width-and-height">Check image width and height
&lt;/h1>&lt;pre>&lt;code> raise Exception('Unexpected image shape: %s' % str(image_data.shape))
dataset[num_images, :, :] = image_data
&lt;/code>&lt;/pre>
&lt;h1 id="read-into-the-dataset">Read into the dataset
&lt;/h1>&lt;pre>&lt;code> num_images = num_images + 1
&lt;/code>&lt;/pre>
&lt;h1 id="increment-image-number">Increment image number
&lt;/h1>&lt;pre>&lt;code>except IOError as e:
&lt;/code>&lt;/pre>
&lt;h1 id="if-file-cant-be-read-skip-it">If file can&amp;rsquo;t be read, skip it
&lt;/h1>&lt;pre>&lt;code> print('Could not read:', image_file, ':', e, '- it\'s ok, skipping.')
&lt;/code>&lt;/pre>
&lt;p>dataset = dataset[0:num_images, :, :]&lt;/p>
&lt;h1 id="if-fewer-files-were-read-than-the-minimum-required">If fewer files were read than the minimum required
&lt;/h1>&lt;p>if num_images &amp;lt; min_num_images:
raise Exception(&amp;lsquo;Many fewer images than expected: %d &amp;lt; %d&amp;rsquo; %
(num_images, min_num_images))&lt;/p>
&lt;h1 id="display-number-of-missing-files">Display number of missing files
&lt;/h1>&lt;p>print(&amp;lsquo;Full dataset tensor:&amp;rsquo;, dataset.shape)&lt;/p>
&lt;h1 id="display-file-count-image-width-and-height">Display file count, image width and height
&lt;/h1>&lt;p>print(&amp;lsquo;Mean:&amp;rsquo;, np.mean(dataset))&lt;/p>
&lt;h1 id="mean-value">Mean value
&lt;/h1>&lt;p>print(&amp;lsquo;Standard deviation:&amp;rsquo;, np.std(dataset))&lt;/p>
&lt;h1 id="standard-deviation">Standard deviation
&lt;/h1>&lt;p>return dataset&lt;/p>
&lt;p>def maybe_pickle(data_folders, min_num_images_per_class, force=False):
dataset_names = []
for folder in data_folders:&lt;/p>
&lt;h1 id="process-each-letter-folder">Process each letter folder
&lt;/h1>&lt;pre>&lt;code>set_filename = folder + '.pickle'
&lt;/code>&lt;/pre>
&lt;h1 id="set-output-file">Set output file
&lt;/h1>&lt;pre>&lt;code>dataset_names.append(set_filename)
&lt;/code>&lt;/pre>
&lt;h1 id="set-processed-folders">Set processed folders
&lt;/h1>&lt;pre>&lt;code>if os.path.exists(set_filename) and not force:
# You may override by setting force=True.
&lt;/code>&lt;/pre>
&lt;h1 id="check-if-processed-file-already-exists">Check if processed file already exists
&lt;/h1>&lt;pre>&lt;code> print('%s already present - Skipping pickling.' % set_filename)
else:
print('Pickling %s.' % set_filename)
dataset = load_letter(folder, min_num_images_per_class)
&lt;/code>&lt;/pre>
&lt;h1 id="normalize-all-images-in-this-folder">Normalize all images in this folder
&lt;/h1>&lt;pre>&lt;code> try:
with open(set_filename, 'wb') as f:
pickle.dump(dataset, f, pickle.HIGHEST_PROTOCOL)
&lt;/code>&lt;/pre>
&lt;h1 id="persist-data-save-to-disk-instead-of-keeping-in-memory">Persist data, save to disk instead of keeping in memory
&lt;/h1>&lt;pre>&lt;code> except Exception as e:
print('Unable to save data to', set_filename, ':', e)
&lt;/code>&lt;/pre>
&lt;p>return dataset_names&lt;/p>
&lt;p>train_datasets = maybe_pickle(train_folders, 45000)
test_datasets = maybe_pickle(test_folders, 1800)&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Question Two
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Display processed images
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">- Reference answer
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>def plot_samples_2(data_folders, sample_size, title=None):
fig = plt.figure()&lt;/p>
&lt;h1 id="create-empty-figure">Create empty figure
&lt;/h1>&lt;pre>&lt;code>if title: fig.suptitle(title, fontsize=16, fontweight='bold')
&lt;/code>&lt;/pre>
&lt;h1 id="add-title">Add title
&lt;/h1>&lt;pre>&lt;code>for folder in data_folders:
&lt;/code>&lt;/pre>
&lt;h1 id="loop-through-each-letter">Loop through each letter
&lt;/h1>&lt;pre>&lt;code> with open(folder, 'rb') as pk_f:
data = pickle.load(pk_f)
for index, image in enumerate(data):
if index &amp;lt; sample_size :
&lt;/code>&lt;/pre>
&lt;h1 id="randomly-select-a-certain-number-of-images-from-that-letter">Randomly select a certain number of images from that letter
&lt;/h1>&lt;pre>&lt;code> ax = fig.add_subplot(len(data_folders), sample_size, sample_size * data_folders.index(folder) +
index + 1)
&lt;/code>&lt;/pre>
&lt;h1 id="load-subplot-image">Load subplot image
&lt;/h1>&lt;pre>&lt;code> ax.imshow(image)
&lt;/code>&lt;/pre>
&lt;h1 id="display-subplot-image">Display subplot image
&lt;/h1>&lt;pre>&lt;code> ax.set_axis_off()
&lt;/code>&lt;/pre>
&lt;h1 id="turn-off-subplot-coordinate-lines">Turn off subplot coordinate lines
&lt;/h1>&lt;pre>&lt;code>fig.set_size_inches(18.5, 10.5)
&lt;/code>&lt;/pre>
&lt;h1 id="set-the-display-size-of-the-image">Set the display size of the image
&lt;/h1>&lt;pre>&lt;code>plt.show()
&lt;/code>&lt;/pre>
&lt;p>plot_samples_2(train_datasets, 20, &amp;lsquo;Train&amp;rsquo;)
plot_samples_2(test_datasets, 20, &amp;lsquo;Test&amp;rsquo;)&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">![image.png](/img/rescued/c09ea5ab69e9.png)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">![image.png](/img/rescued/d4363496653b.png)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Question Three
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Check if the number of files under each letter is similar.
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">- Reference answer
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>file_path = &amp;rsquo;notMNIST_large/{0}.pickle&amp;rsquo;
for ele in &amp;lsquo;ABCDEFJHIJ&amp;rsquo;:
with open(file_path.format(ele), &amp;lsquo;rb&amp;rsquo;) as pk_f:&lt;/p>
&lt;h1 id="loop-through-each-directory">Loop through each directory
&lt;/h1>&lt;pre>&lt;code> dat = pickle.load(pk_f)
&lt;/code>&lt;/pre>
&lt;h1 id="load-the-persisted-file-in-this-directory">Load the persisted file in this directory
&lt;/h1>&lt;pre>&lt;code>print('number of pictures in {}.pickle = '.format(ele), dat.shape[0])
&lt;/code>&lt;/pre>
&lt;h1 id="print-relevant-information">Print relevant information
&lt;/h1>&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-gdscript3" data-lang="gdscript3">&lt;span class="line">&lt;span class="cl">&lt;span class="n">Results&lt;/span> &lt;span class="n">show&lt;/span> &lt;span class="n">that&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">numbers&lt;/span> &lt;span class="n">are&lt;/span> &lt;span class="n">basically&lt;/span> &lt;span class="n">consistent&lt;/span>&lt;span class="o">.&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">!&lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="n">Question&lt;/span> &lt;span class="mi">3&lt;/span> &lt;span class="n">Result&lt;/span>&lt;span class="p">](&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="n">img&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="n">rescued&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="mi">31&lt;/span>&lt;span class="n">bcb36f8c25&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">png&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">## Code Segment—Data Splitting&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">Data&lt;/span> &lt;span class="n">cannot&lt;/span> &lt;span class="n">be&lt;/span> &lt;span class="n">loaded&lt;/span> &lt;span class="n">all&lt;/span> &lt;span class="n">at&lt;/span> &lt;span class="n">once&lt;/span> &lt;span class="n">into&lt;/span> &lt;span class="n">memory&lt;/span>&lt;span class="o">.&lt;/span> &lt;span class="n">This&lt;/span> &lt;span class="n">code&lt;/span> &lt;span class="n">segment&lt;/span> &lt;span class="n">splits&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">data&lt;/span>&lt;span class="o">.&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>def make_arrays(nb_rows, img_size):
if nb_rows:
dataset = np.ndarray((nb_rows, img_size, img_size), dtype=np.float32)&lt;/p>
&lt;h1 id="create-an-empty-set-data-type-is-a-matrix-with-rows-length-img_size-width-img_size-height-data-type-is-32-bit-float">Create an empty set, data type is a matrix with rows length, img_size width, img_size height, data type is 32-bit float
&lt;/h1>&lt;pre>&lt;code>labels = np.ndarray(nb_rows, dtype=np.int32)
&lt;/code>&lt;/pre>
&lt;h1 id="create-a-label-data-type-is-32-bit-integer-length-is-rows">Create a label, data type is 32-bit integer, length is rows
&lt;/h1>&lt;p>else:
dataset, labels = None, None
return dataset, labels&lt;/p>
&lt;h1 id="return-created-data-type">Return created data type
&lt;/h1>&lt;p>def merge_datasets(pickle_files, train_size, valid_size=0):
num_classes = len(pickle_files)&lt;/p>
&lt;h1 id="number-of-categories-to-process">Number of categories to process
&lt;/h1>&lt;p>valid_dataset, valid_labels = make_arrays(valid_size, image_size)&lt;/p>
&lt;h1 id="build-validation-dataset-length-is-validation-length">Build validation dataset, length is validation length
&lt;/h1>&lt;p>train_dataset, train_labels = make_arrays(train_size, image_size)&lt;/p>
&lt;h1 id="build-training-dataset-length-is-training-length">Build training dataset, length is training length
&lt;/h1>&lt;p>vsize_per_class = valid_size // num_classes
tsize_per_class = train_size // num_classes&lt;/p>
&lt;h1 id="calculate-average-length-for-each-category-with-given-training-and-validation-lengths">Calculate average length for each category with given training and validation lengths
&lt;/h1>&lt;p>start_v, start_t = 0, 0&lt;/p>
&lt;h1 id="initialize-indices-start_v-is-the-start-of-validation-data-start_t-is-the-start-of-training-data">Initialize indices, start_v is the start of validation data, start_t is the start of training data
&lt;/h1>&lt;p>end_v, end_t = vsize_per_class, tsize_per_class&lt;/p>
&lt;h1 id="initialize-indices-end_v-is-the-end-of-validation-data-end_t-is-the-end-of-training-data">Initialize indices, end_v is the end of validation data, end_t is the end of training data
&lt;/h1>&lt;p>end_l = vsize_per_class + tsize_per_class&lt;/p>
&lt;h1 id="initialize-indices-end_l-is-the-end-of-the-letter-set-equal-to-length-of-validation-data-for-each-category--length-of-training-data">Initialize indices, end_l is the end of the letter set, equal to length of validation data for each category + length of training data
&lt;/h1>&lt;p>for label, pickle_file in enumerate(pickle_files):&lt;/p>
&lt;h1 id="loop-through-each-pickle_file">Loop through each pickle_file
&lt;/h1>&lt;pre>&lt;code>try:
with open(pickle_file, 'rb') as f:
&lt;/code>&lt;/pre>
&lt;h1 id="open-this-persistence-file">Open this persistence file
&lt;/h1>&lt;pre>&lt;code> letter_set = pickle.load(f)
&lt;/code>&lt;/pre>
&lt;h1 id="load-dataset">Load dataset
&lt;/h1>&lt;pre>&lt;code> # let's shuffle the letters to have random validation and training set
np.random.shuffle(letter_set)
&lt;/code>&lt;/pre>
&lt;h1 id="shuffle-the-dataset">Shuffle the dataset
&lt;/h1>&lt;pre>&lt;code> if valid_dataset is not None:
&lt;/code>&lt;/pre>
&lt;h1 id="if-not-a-test-set-update-the-test-set-otherwise-valid_dataset-is-not-updated">If not a test set, update the test set, otherwise valid_dataset is not updated
&lt;/h1>&lt;pre>&lt;code> valid_letter = letter_set[:vsize_per_class, :, :]
&lt;/code>&lt;/pre>
&lt;h1 id="numpy-slicing-----httpbrieflyxme2015python-modulenumpy-array-split">numpy slicing &lt;a class="link" href="http://brieflyx.me/2015/python-module/numpy-array-split/" target="_blank" rel="noopener"
>http://brieflyx.me/2015/python-module/numpy-array-split/&lt;/a>
&lt;/h1>&lt;h1 id="select-data-of-valid-data-per-class-count-from-shuffled-data-for-processing-put-into-valid_letter">Select data of &amp;lsquo;valid data per class&amp;rsquo; count from shuffled data for processing, put into valid_letter
&lt;/h1>&lt;pre>&lt;code> valid_dataset[start_v:end_v, :, :] = valid_letter
&lt;/code>&lt;/pre>
&lt;h1 id="put-this-data-into-valid_dataset">Put this data into valid_dataset
&lt;/h1>&lt;pre>&lt;code> valid_labels[start_v:end_v] = label
&lt;/code>&lt;/pre>
&lt;h1 id="mark-label-should-be-one-of-09">Mark label should be one of 0~9
&lt;/h1>&lt;pre>&lt;code> start_v += vsize_per_class
end_v += vsize_per_class
&lt;/code>&lt;/pre>
&lt;h1 id="update-indices">Update indices
&lt;/h1>&lt;h1 id="at-the-end-of-the-loop-valid_dataset-should-be-data-with-total-length-valid_size-valid_labels-is-the-label-at-the-corresponding-position">At the end of the loop, valid_dataset should be data with total length valid_size, valid_labels is the label at the corresponding position
&lt;/h1>&lt;pre>&lt;code> train_letter = letter_set[vsize_per_class:end_l, :, :]
&lt;/code>&lt;/pre>
&lt;h1 id="other-random-elements-except-valid-part-length-is-end_l---vsize_per_class--tsize_per_class">Other random elements except valid part, length is end_l - vsize_per_class = tsize_per_class
&lt;/h1>&lt;pre>&lt;code> train_dataset[start_t:end_t, :, :] = train_letter
&lt;/code>&lt;/pre>
&lt;h1 id="at-the-end-of-the-loop-train_dataset-should-be-data-with-total-length-train_size">At the end of the loop, train_dataset should be data with total length train_size
&lt;/h1>&lt;h1 id="heading">
&lt;/h1>&lt;pre>&lt;code> train_labels[start_t:end_t] = label
start_t += tsize_per_class
end_t += tsize_per_class
&lt;/code>&lt;/pre>
&lt;h1 id="update-indices-1">Update indices
&lt;/h1>&lt;pre>&lt;code>except Exception as e:
print('Unable to process data from', pickle_file, ':', e)
raise
&lt;/code>&lt;/pre>
&lt;p>return valid_dataset, valid_labels, train_dataset, train_labels&lt;/p>
&lt;p>train_size = 200000
valid_size = 10000
test_size = 10000&lt;/p>
&lt;p>valid_dataset, valid_labels, train_dataset, train_labels = merge_datasets(
train_datasets, train_size, valid_size)
_, _, test_dataset, test_labels = merge_datasets(test_datasets, test_size)&lt;/p>
&lt;p>print(&amp;lsquo;Training:&amp;rsquo;, train_dataset.shape, train_labels.shape)
print(&amp;lsquo;Validation:&amp;rsquo;, valid_dataset.shape, valid_labels.shape)
print(&amp;lsquo;Testing:&amp;rsquo;, test_dataset.shape, test_labels.shape)&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Code Segment—Shuffling Data
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Introduction to the permutation function: http://www.jianshu.com/p/f0eb10acaa2d
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>def randomize(dataset, labels):&lt;/p>
&lt;h1 id="labelsshape0-is-the-length-of-labels">labels.shape[0] is the length of labels
&lt;/h1>&lt;p>permutation = np.random.permutation(labels.shape[0])&lt;/p>
&lt;h1 id="randomly-select-a-shuffled-set-of-this-many-numbers">Randomly select a shuffled set of this many numbers
&lt;/h1>&lt;p>print(labels.shape[0])
shuffled_dataset = dataset[permutation,:,:]&lt;/p>
&lt;h1 id="shuffle-data">Shuffle data
&lt;/h1>&lt;p>shuffled_labels = labels[permutation]&lt;/p>
&lt;h1 id="shuffle-labels">Shuffle labels
&lt;/h1>&lt;p>return shuffled_dataset, shuffled_labels
train_dataset, train_labels = randomize(train_dataset, train_labels)
test_dataset, test_labels = randomize(test_dataset, test_labels)
valid_dataset, valid_labels = randomize(valid_dataset, valid_labels)&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Question Four
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Verify if the shuffled data is correct
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">- Reference answer
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>import random
def plot_sample_3(dataset, labels, title):
fig = plt.figure()
plt.suptitle(title, fontsize=16, fontweight=&amp;lsquo;bold&amp;rsquo;)&lt;/p>
&lt;h1 id="set-title-style">Set title style
&lt;/h1>&lt;pre>&lt;code>items = random.sample(range(len(labels)), 200)
&lt;/code>&lt;/pre>
&lt;h1 id="shuffle-the-sequential-sequence-of-labels-length">Shuffle the sequential sequence of labels length
&lt;/h1>&lt;pre>&lt;code>for i, item in enumerate(items):
&lt;/code>&lt;/pre>
&lt;h1 id="randomly-pick-one">Randomly pick one
&lt;/h1>&lt;pre>&lt;code> plt.subplot(10, 20, i + 1)
&lt;/code>&lt;/pre>
&lt;h1 id="draw-subplot">Draw subplot
&lt;/h1>&lt;pre>&lt;code> plt.axis('off')
&lt;/code>&lt;/pre>
&lt;h1 id="turn-off-coordinate-axes">Turn off coordinate axes
&lt;/h1>&lt;pre>&lt;code> plt.title(chr(ord('A') + labels[item]))
&lt;/code>&lt;/pre>
&lt;h1 id="add-title-1">Add title
&lt;/h1>&lt;pre>&lt;code> plt.imshow(dataset[item])
&lt;/code>&lt;/pre>
&lt;h1 id="display-subplot-at-corresponding-position">Display subplot at corresponding position
&lt;/h1>&lt;pre>&lt;code>fig.set_size_inches(18.5, 10.5)
plt.show()
&lt;/code>&lt;/pre>
&lt;h1 id="display-image">Display image
&lt;/h1>&lt;p>plot_sample_3(train_dataset, train_labels, &amp;rsquo;train dataset suffled&amp;rsquo;)
plot_sample_3(valid_dataset, valid_labels, &amp;lsquo;valid dataset suffled&amp;rsquo;)
plot_sample_3(test_dataset, test_labels, &amp;rsquo;test dataset suffled&amp;rsquo;)&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;span class="lnt">6
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">![Question 4](/img/rescued/2041fe44668c.png)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Similar two figures omitted
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Code Segment—Saving Data
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>pickle_file = os.path.join(data_root, &amp;rsquo;notMNIST.pickle&amp;rsquo;)&lt;/p>
&lt;h1 id="output-file-path">Output file path
&lt;/h1>&lt;p>try:
f = open(pickle_file, &amp;lsquo;wb&amp;rsquo;)&lt;/p>
&lt;h1 id="open-this-file">Open this file
&lt;/h1>&lt;p>save = {
&amp;rsquo;train_dataset&amp;rsquo;: train_dataset,
&amp;rsquo;train_labels&amp;rsquo;: train_labels,
&amp;lsquo;valid_dataset&amp;rsquo;: valid_dataset,
&amp;lsquo;valid_labels&amp;rsquo;: valid_labels,
&amp;rsquo;test_dataset&amp;rsquo;: test_dataset,
&amp;rsquo;test_labels&amp;rsquo;: test_labels,
}&lt;/p>
&lt;h1 id="write-a-dictionary-string-ndarray">Write a dictionary string-ndarray
&lt;/h1>&lt;p>pickle.dump(save, f, pickle.HIGHEST_PROTOCOL)
f.close()
except Exception as e:
print(&amp;lsquo;Unable to save data to&amp;rsquo;, pickle_file, &amp;lsquo;:&amp;rsquo;, e)
raise&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Code Segment—Displaying Saved Data Size
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>statinfo = os.stat(pickle_file)
print(&amp;lsquo;Compressed pickle size:&amp;rsquo;, statinfo.st_size)&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-gdscript3" data-lang="gdscript3">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">## Question Five&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">Google&lt;/span> &lt;span class="n">translation&lt;/span> &lt;span class="n">of&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">question&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">By&lt;/span> &lt;span class="n">construction&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">this&lt;/span> &lt;span class="n">dataset&lt;/span> &lt;span class="n">may&lt;/span> &lt;span class="n">contain&lt;/span> &lt;span class="n">a&lt;/span> &lt;span class="n">lot&lt;/span> &lt;span class="n">of&lt;/span> &lt;span class="n">overlapping&lt;/span> &lt;span class="n">samples&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">including&lt;/span> &lt;span class="ow">in&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">validation&lt;/span> &lt;span class="ow">and&lt;/span> &lt;span class="n">test&lt;/span> &lt;span class="n">sets&lt;/span>&lt;span class="o">.&lt;/span> &lt;span class="n">Overlap&lt;/span> &lt;span class="n">between&lt;/span> &lt;span class="n">training&lt;/span> &lt;span class="ow">and&lt;/span> &lt;span class="n">test&lt;/span> &lt;span class="n">can&lt;/span> &lt;span class="n">skew&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">results&lt;/span> &lt;span class="k">if&lt;/span> &lt;span class="n">you&lt;/span> &lt;span class="n">expect&lt;/span> &lt;span class="n">to&lt;/span> &lt;span class="n">use&lt;/span> &lt;span class="n">your&lt;/span> &lt;span class="n">model&lt;/span> &lt;span class="ow">in&lt;/span> &lt;span class="n">an&lt;/span> &lt;span class="n">environment&lt;/span> &lt;span class="n">where&lt;/span> &lt;span class="n">there&lt;/span> &lt;span class="n">is&lt;/span> &lt;span class="n">never&lt;/span> &lt;span class="n">an&lt;/span> &lt;span class="n">overlap&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">but&lt;/span> &lt;span class="ow">in&lt;/span> &lt;span class="n">practice&lt;/span> &lt;span class="n">this&lt;/span> &lt;span class="n">doesn&lt;/span>&lt;span class="s1">&amp;#39;t usually matter. Measure how much overlap there is between training, validation, and test samples.&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">Optional&lt;/span> &lt;span class="n">question&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">What&lt;/span> &lt;span class="n">about&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">duplicates&lt;/span> &lt;span class="n">between&lt;/span> &lt;span class="n">datasets&lt;/span>&lt;span class="err">?&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="n">For&lt;/span> &lt;span class="n">instance&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">same&lt;/span> &lt;span class="n">letter&lt;/span> &lt;span class="n">images&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">Create&lt;/span> &lt;span class="n">a&lt;/span> &lt;span class="n">sanitized&lt;/span> &lt;span class="n">validation&lt;/span> &lt;span class="ow">and&lt;/span> &lt;span class="n">test&lt;/span> &lt;span class="n">set&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="ow">and&lt;/span> &lt;span class="n">compare&lt;/span> &lt;span class="n">your&lt;/span> &lt;span class="n">accuracy&lt;/span> &lt;span class="n">on&lt;/span> &lt;span class="n">those&lt;/span> &lt;span class="n">versus&lt;/span> &lt;span class="n">your&lt;/span> &lt;span class="n">accuracy&lt;/span> &lt;span class="n">on&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">original&lt;/span> &lt;span class="n">sets&lt;/span>&lt;span class="o">.&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">The&lt;/span> &lt;span class="n">basic&lt;/span> &lt;span class="n">idea&lt;/span> &lt;span class="n">is&lt;/span> &lt;span class="n">that&lt;/span> &lt;span class="n">training&lt;/span> &lt;span class="n">data&lt;/span> &lt;span class="n">should&lt;/span> &lt;span class="ow">not&lt;/span> &lt;span class="n">overlap&lt;/span> &lt;span class="n">with&lt;/span> &lt;span class="n">testing&lt;/span> &lt;span class="n">data&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">otherwise&lt;/span> &lt;span class="n">it&lt;/span> &lt;span class="n">leads&lt;/span> &lt;span class="n">to&lt;/span> &lt;span class="n">inaccurate&lt;/span> &lt;span class="n">accuracy&lt;/span>&lt;span class="o">.&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="ne">Reference&lt;/span> &lt;span class="n">code&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">-&lt;/span> &lt;span class="n">Just&lt;/span> &lt;span class="n">check&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">number&lt;/span> &lt;span class="n">of&lt;/span> &lt;span class="n">duplicate&lt;/span> &lt;span class="n">images&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>import hashlib&lt;/p>
&lt;p>pickle_file = os.path.join(&amp;rsquo;.&amp;rsquo;, &amp;rsquo;notMNIST.pickle&amp;rsquo;)
try:
with open(pickle_file, &amp;lsquo;rb&amp;rsquo;) as f:
data = pickle.load(f)
except Exception as e:
print(&amp;lsquo;Unable to open data from&amp;rsquo;, pickle_file, &amp;lsquo;:&amp;rsquo;, e)
raise&lt;/p>
&lt;h1 id="after-saving-the-data-if-the-kernel-crashed-you-can-read-directly-from-local-without-rerunning-previous-code">After saving the data, if the kernel crashed, you can read directly from local without rerunning previous code
&lt;/h1>&lt;h1 id="if-theres-an-error-you-can-search-for-the-exception-online">If there&amp;rsquo;s an error, you can search for the exception online
&lt;/h1>&lt;p>def calcOverlap(sourceSet, targetSet, description):
sourceSetMd5 = np.array([hashlib.md5(img).hexdigest() for img in sourceSet])&lt;/p>
&lt;h1 id="build-an-md5-table">Build an md5 table
&lt;/h1>&lt;pre>&lt;code>targetSetMd5 = np.array([hashlib.md5(img).hexdigest() for img in targetSet])
&lt;/code>&lt;/pre>
&lt;h1 id="build-an-md5-table-1">Build an md5 table
&lt;/h1>&lt;pre>&lt;code>overlap = np.intersect1d(sourceSetMd5, targetSetMd5, assume_unique=False)
&lt;/code>&lt;/pre>
&lt;h1 id="deduplicate">Deduplicate
&lt;/h1>&lt;pre>&lt;code>print(description)
print(&amp;quot;overlap&amp;quot;,overlap.shape[0], &amp;quot;from&amp;quot;,sourceSetMd5.shape[0],&amp;quot;to&amp;quot;, targetSetMd5.shape[0])
print(&amp;quot;rate&amp;quot;,overlap.shape[0]*100.0/sourceSetMd5.shape[0],&amp;quot;% and&amp;quot;, overlap.shape[0]*100.0/targetSetMd5.shape[0],&amp;quot;%&amp;quot;)
&lt;/code>&lt;/pre>
&lt;h1 id="print-overlap-count">Print overlap count
&lt;/h1>&lt;p>calcOverlap(data[&amp;rsquo;train_dataset&amp;rsquo;], data[&amp;lsquo;valid_dataset&amp;rsquo;], &amp;ldquo;train_dataset &amp;amp; valid_dataset&amp;rdquo;)
calcOverlap(data[&amp;rsquo;train_dataset&amp;rsquo;], data[&amp;rsquo;test_dataset&amp;rsquo;], &amp;ldquo;train_dataset &amp;amp; test_dataset&amp;rdquo;)
calcOverlap(data[&amp;rsquo;test_dataset&amp;rsquo;], data[&amp;lsquo;valid_dataset&amp;rsquo;], &amp;ldquo;test_dataset &amp;amp; valid_dataset&amp;rdquo;)&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">![Running result](/img/rescued/fc3c3db78e52.png)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">- Remove duplicate image resources
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">To be updated
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Question Six
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Use logistic regression to train the model and test it
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">- Reference code
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>import random
def disp_sample_dataset(dataset, labels,trueLabels, title=None):&lt;/p>
&lt;h1 id="display-training-results">Display training results
&lt;/h1>&lt;pre>&lt;code>fig = plt.figure()
if title: fig.suptitle(title, fontsize=16, fontweight='bold')
&lt;/code>&lt;/pre>
&lt;h1 id="set-title-style-1">Set title style
&lt;/h1>&lt;pre>&lt;code>items = random.sample(range(len(labels)), 200)
&lt;/code>&lt;/pre>
&lt;h1 id="randomly-select-a-series-of-images">Randomly select a series of images
&lt;/h1>&lt;pre>&lt;code>for i, item in enumerate(items):
plt.subplot(10, 20, i + 1)
&lt;/code>&lt;/pre>
&lt;h1 id="set-a-subplot">Set a subplot
&lt;/h1>&lt;pre>&lt;code> plt.axis('off')
&lt;/code>&lt;/pre>
&lt;h1 id="turn-off-coordinate-lines">Turn off coordinate lines
&lt;/h1>&lt;pre>&lt;code> lab = str(chr(ord('A') + labels[item]))
trueLab = str(chr(ord('A') + trueLabels[item]))
if lab == trueLab:
plt.title( lab )
else:
plt.title(lab + &amp;quot; but &amp;quot; + trueLab)
&lt;/code>&lt;/pre>
&lt;h1 id="add-title-2">Add title
&lt;/h1>&lt;pre>&lt;code> plt.imshow(dataset[item])
&lt;/code>&lt;/pre>
&lt;h1 id="display-this-image">Display this image
&lt;/h1>&lt;pre>&lt;code>fig.set_size_inches(18.5, 10.5)
plt.show()
&lt;/code>&lt;/pre>
&lt;p>def train_and_predict(train_dataset, train_labels, test_dataset, test_labels ,sample_size):
regr = LogisticRegression()&lt;/p>
&lt;h1 id="generate-trainer">Generate trainer
&lt;/h1>&lt;pre>&lt;code>X_train = train_dataset[:sample_size].reshape(sample_size, 784)
&lt;/code>&lt;/pre>
&lt;h1 id="choose-amount-of-data-to-train-based-on-sample_size">Choose amount of data to train based on sample_size
&lt;/h1>&lt;h1 id="compress-2d-vector-to-1d-vector">Compress 2D vector to 1D vector
&lt;/h1>&lt;pre>&lt;code>y_train = train_labels[:sample_size]
&lt;/code>&lt;/pre>
&lt;h1 id="extract-training-data">Extract training data
&lt;/h1>&lt;pre>&lt;code>regr.fit(X_train, y_train)
&lt;/code>&lt;/pre>
&lt;h1 id="train-data">Train data
&lt;/h1>&lt;pre>&lt;code>X_test = test_dataset.reshape(test_dataset.shape[0], 28 * 28)
&lt;/code>&lt;/pre>
&lt;h1 id="compress-test-data-to-1d-vector">Compress test data to 1D vector
&lt;/h1>&lt;pre>&lt;code>y_test = test_labels
&lt;/code>&lt;/pre>
&lt;h1 id="true-labels-corresponding-to-test-data">True labels corresponding to test data
&lt;/h1>&lt;pre>&lt;code>pred_labels = regr.predict(X_test)
&lt;/code>&lt;/pre>
&lt;h1 id="generate-prediction-data">Generate prediction data
&lt;/h1>&lt;pre>&lt;code>print('Accuracy:', regr.score(X_test, y_test), 'when sample_size=', sample_size)
disp_sample_dataset(test_dataset, pred_labels, test_labels, 'sample_size=' + str(sample_size))
&lt;/code>&lt;/pre>
&lt;p>train_and_predict(data[&amp;rsquo;train_dataset&amp;rsquo;],data[&amp;rsquo;train_labels&amp;rsquo;],data[&amp;rsquo;test_dataset&amp;rsquo;],data[&amp;rsquo;test_labels&amp;rsquo;], 1000)&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;span class="lnt">21
&lt;/span>&lt;span class="lnt">22
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">![image.png](/img/rescued/abdb5d761390.png)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Model Performance
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Sections 22~27 discuss model performance-related knowledge. We usually hope that the model&amp;#39;s performance can reach 100%, which is obviously impossible. Also, in order to improve the accuracy of the training set, the model may overfit. At this point, we should follow two points:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">- Don&amp;#39;t use all training data at once, but use it in blocks, train a portion each time
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">- When model parameter changes cause 30 or more cases to change from error to correct, then this parameter change is effective.
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">![Model Performance](/img/rescued/a8e154206ab8.png)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Stochastic Gradient Descent
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Sections 29~31 explain what stochastic gradient descent is.
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">During training, to make the model move in the optimal direction, we need to calculate the derivative at that point. 1. The calculation of derivatives is quite large, so we need to randomly select a subset of samples to calculate derivatives, to substitute for the real derivative. This is stochastic gradient descent. 2. To reduce the randomness of random selection, we use momentum inertia to reduce randomness. 3. To make the model stable in later stages, we reduce the learning step size.
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">End of Course One
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&amp;gt; Reference for assignment code
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&amp;gt; http://www.hankcs.com/ml/notmnist.html
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div></description></item></channel></rss>