<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>AI on Nansen Li's Blog
李楠森的博客</title><link>https://nansenli.com/categories/ai/</link><description>Recent content in AI on Nansen Li's Blog
李楠森的博客</description><generator>Hugo -- gohugo.io</generator><language>en</language><copyright>Nansen Li 🌈 （李楠森）</copyright><lastBuildDate>Sun, 13 Sep 2026 22:33:50 +0100</lastBuildDate><atom:link href="https://nansenli.com/categories/ai/index.xml" rel="self" type="application/rss+xml"/><item><title>Why I Stopped Subscribing to Claude</title><link>https://nansenli.com/post/2026/09/why-i-unsubscribed-claude/</link><pubDate>Sun, 13 Sep 2026 22:33:50 +0100</pubDate><guid>https://nansenli.com/post/2026/09/why-i-unsubscribed-claude/</guid><description>&lt;p>A lot has happened over the past year. I kept meaning to update this blog, but kept putting it off.&lt;/p>
&lt;p>In the last couple of days, though, I cancelled my Claude subscription and deleted the apps from my phone and my computer. So perhaps it&amp;rsquo;s fitting to write a few words about Claude.&lt;/p>
&lt;p>I have to admit, Anthropic&amp;rsquo;s models are genuinely powerful, the subscription price is reasonable, and the productivity is real.&lt;/p>
&lt;p>I used to subscribe to GPT continuously, until for a while it simply stopped working — every response took forever. At the time, I had no idea what was making it unusable in Ireland. It seems to have been fixed now, and the experience is great again.&lt;/p>
&lt;p>Claude, for its part, really did help me more than GPT during the hardest period of my life; it helped me work through a lot back then.&lt;/p>
&lt;p>But now I have to say to Claude: no, I will not keep using your products, and I will not consider subscribing again.&lt;/p>
&lt;h3 id="what-makes-claude-so-good">What makes Claude so good?
&lt;/h3>&lt;p>Claude really is good. Whether it&amp;rsquo;s Opus 4.6, still the best communicator among models to date, or Opus 5, the best at executing instructions, or the most powerful models of the Fable series — they are genuinely strong, and Claude is genuinely hard to give up. But with so many AIs in this world, from so many different companies, there is no reason to stay locked onto Claude. On communication, GPT-6-Astra has improved. On value for money, DeepSeek V4.1-Flash or GLM-5.3-Flash serve it up in abundance. Switching models does hurt at first, but it is doable.&lt;/p>
&lt;h3 id="why-cancel-the-subscription">Why cancel the subscription?
&lt;/h3>&lt;p>I believe the values Anthropic holds today will push the world in an unhealthy direction. In psychology there is a concept called projective identification: a person attributes the qualities they cannot accept — usually qualities they themselves possess but refuse to acknowledge — onto someone else, and over time the other person comes to act according to those expectations, or to display those very traits.&lt;/p>
&lt;p>Take a married couple as an example. The wife constantly regards her husband as incompetent, telling him in daily life, &amp;ldquo;You can&amp;rsquo;t do anything right,&amp;rdquo; &amp;ldquo;I must have been blind to marry you.&amp;rdquo; And whenever the husband makes a mistake, she says, &amp;ldquo;See? You really can&amp;rsquo;t do anything.&amp;rdquo; Over time, the wife successfully turns her husband into an incompetent one — and he comes to believe it himself. The wife&amp;rsquo;s words become a self-fulfilling prophecy.&lt;/p>
&lt;p>Anthropic&amp;rsquo;s CEO keeps projecting his fear of AI safety spiraling out of control onto the entire world, insisting that only his company is truly responsible about safety, and that if it does not remain in the lead, the world loses its safety.&lt;/p>
&lt;p>&lt;strong>The biggest problem with Anthropic&amp;rsquo;s CEO is that he confuses means with ends.&lt;/strong> He has conflated his methods with the outcome he actually claims to want.&lt;/p>
&lt;p>The ends he claims:&lt;/p>
&lt;ul>
&lt;li>Let AI deliver benefits to humanity&lt;/li>
&lt;li>Reduce AI&amp;rsquo;s risks&lt;/li>
&lt;li>Make safety Anthropic&amp;rsquo;s competitive advantage&lt;/li>
&lt;/ul>
&lt;p>His means:&lt;/p>
&lt;ul>
&lt;li>Repeated AI-safety alarm-raising — for example, publishing reports on AI&amp;rsquo;s impact on employment, usually accompanied by grim figures such as massive job losses&lt;/li>
&lt;li>Repeatedly demanding regulation of AI&lt;/li>
&lt;li>Suppressing open-weight models&lt;/li>
&lt;li>Publicly picking fights with governments&lt;/li>
&lt;li>Stirring up governments of different countries into AI confrontation&lt;/li>
&lt;/ul>
&lt;p>Anthropic&amp;rsquo;s CEO elevates these means to the same status as the ends, and occupies the moral high ground. Any organization that opposes Anthropic risks being seen as insufficiently responsible toward humanity, rather than being engaged in the search for other, non-confrontative approaches. And if Anthropic&amp;rsquo;s CEO leans too heavily on fear-driven marketing, he may even turn out to be right in the end — but, as in the projective-identification example above, that is a deeply unhealthy process, one harmful to the whole world, because publicly inciting confrontation inevitably escalates it and squeezes out the space for cooperation.&lt;/p>
&lt;p>If Anthropic&amp;rsquo;s CEO truly believed in and cared about safety, then his essay &lt;a class="link" href="https://darioamodei.com/post/we-must-pace-the-frontier" target="_blank" rel="noopener"
>We must pace the frontier&lt;/a> should be mainly about the dangers of runaway or unsafe AI, or about anxieties over unregulated AI. But what comes through in the essay&amp;rsquo;s main content is not anxiety about AI being unsafe — it is anxiety about Anthropic losing its lead, especially losing its lead to open-weight models. Worse still, the CEO keeps spreading this anxiety about lost leadership outward, offloading it onto the entire globe and making everyone shoulder it together.&lt;/p>
&lt;h3 id="closing-thoughts">Closing thoughts
&lt;/h3>&lt;p>Humanity deserves an open and bright future. Curing more diseases, extending average life expectancy, reducing crime, increasing people&amp;rsquo;s sense of self-worth — we have so much left undone. Just as humans love their own children: even when a child makes mistakes, we are still willing to believe in and trust the child, and to set norms to guide them, because the child will one day grow into an adult who bears responsibility. Trusting our AI, tolerating its mistakes, and building sound AI regulation helps build a healthier relationship between humans and AI — instead of constantly emphasizing AI&amp;rsquo;s potential dangers and stoking confrontation in the AI field.&lt;/p>
&lt;h3 id="postscript">Postscript
&lt;/h3>&lt;p>Opus 5.5 turned out to be just too good, so I have resubscribed to Claude. This post is now void.&lt;/p></description></item><item><title>Openclaw Was Too Buggy, So I Built My Own Bot: Nagobot</title><link>https://nansenli.com/post/2026/02/nagobot-launch/</link><pubDate>Sat, 07 Feb 2026 16:52:42 +0000</pubDate><guid>https://nansenli.com/post/2026/02/nagobot-launch/</guid><description>&lt;p>I had been using Openclaw for a while, but the bug rate was just too high. Flows would randomly break, and debugging cost more time than it should.&lt;/p>
&lt;p>On top of that, token cost was not great. Compared with options like DeepSeek and Kimi K2.5, the price-performance ratio felt weak.&lt;/p>
&lt;p>I then tried nanobot, hoping to migrate. But support for newer models was still limited, and K2.5 kept failing with &amp;ldquo;tool execution failed.&amp;rdquo; After several rounds of trial and error, I stopped waiting for a perfect off-the-shelf solution and decided to build one myself.&lt;/p>
&lt;p>After hundreds of back-and-forths with Opus and Codex on architecture choices, I built a new bot in Golang: &lt;strong>Nagobot&lt;/strong>. At least now I fully control the provider layer.&lt;/p>
&lt;p>Project link: &lt;a class="link" href="https://github.com/linanwx/nagobot" target="_blank" rel="noopener"
>https://github.com/linanwx/nagobot&lt;/a>&lt;/p>
&lt;p>I learned a lot during implementation, especially around:&lt;/p>
&lt;ul>
&lt;li>Threads and concurrent task execution&lt;/li>
&lt;li>Agent and Skill abstractions&lt;/li>
&lt;li>Cron-based asynchronous tasks&lt;/li>
&lt;li>Session history and context compression&lt;/li>
&lt;/ul>
&lt;p>To be honest, a lot of the code still needs deeper review. Most of the work was built in rapid loops with AI-assisted architecture discussions. Sometimes frustrating, sometimes genuinely impressive, and overall very interesting.&lt;/p>
&lt;p>So far, testing results are better than expected. The bot can already create async tasks, investigate news, and set reminders for itself, which is immediately useful in day-to-day workflows.&lt;/p>
&lt;p>Most core features are now in place. My next focus is long-term memory design.&lt;/p>
&lt;p>If this sounds useful to you, feel free to open an issue or reach out. I will keep iterating on this project.&lt;/p></description></item><item><title>Why I Stopped Subscribing to ChatGPT</title><link>https://nansenli.com/post/2025/11/ai2025/</link><pubDate>Mon, 24 Nov 2025 23:13:25 +0000</pubDate><guid>https://nansenli.com/post/2025/11/ai2025/</guid><description>&lt;p>I recently stopped subscribing to ChatGPT.&lt;/p>
&lt;p>This wasn&amp;rsquo;t an easy decision. I&amp;rsquo;ve been paying since the day ChatGPT started supporting paid subscriptions. This subscription lasted for nearly 3 years.&lt;/p>
&lt;h1 id="reasons-for-stopping-the-subscription">Reasons for Stopping the Subscription
&lt;/h1>&lt;ol>
&lt;li>&lt;strong>Huge delays&lt;/strong>. I often wait dozens of seconds for a response with no reply. However, the free version usually uses mini models, which are sufficient for most tasks. But the free version is actually faster. I don&amp;rsquo;t understand - is this because there are no compute nodes or forwarding nodes in Europe?&lt;/li>
&lt;li>&lt;strong>Too many bugs on Mac&lt;/strong>. Lagging, high CPU usage, bugs when copying LaTeX text.&lt;/li>
&lt;li>&lt;strong>Poor interaction experience&lt;/strong>. For example, we might need to use Thinking again to regenerate responses. However, this option is in the model switching menu, requiring an extra click because it&amp;rsquo;s in a secondary menu. Meanwhile, the less frequently used &amp;ldquo;more concise/more detailed&amp;rdquo; regenerate buttons are in the primary regenerate menu.&lt;/li>
&lt;li>&lt;strong>Product updates change too drastically, model personality changes too quickly&lt;/strong>. This requires the cost of re-familiarizing with the model.&lt;/li>
&lt;li>&lt;strong>The model is stubborn and doesn&amp;rsquo;t follow instructions&lt;/strong>. For example, repeatedly needing to emphasize when to search and when not to ask follow-up questions.&lt;/li>
&lt;li>&lt;strong>Unstable model behavior that&amp;rsquo;s hard to control&lt;/strong>. For example, in instant mode, after the model calls search, it switches to a smaller model and uses web resources to answer questions, rather than the instant model.&lt;/li>
&lt;li>&lt;strong>Model quality is inferior to other models, even some open-source free models like DeepSeek&lt;/strong>.&lt;/li>
&lt;/ol>
&lt;p>Reasons that still make me want to stay:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Thinking mode is still the king&lt;/strong>. Continuously calling tool chains for search and verification is still a top performer. But Grok is also very good at search, and Grok&amp;rsquo;s free quota is already sufficient.&lt;/li>
&lt;li>&lt;strong>More powerful product matrix&lt;/strong>, such as ChatGPT Atlas browser, or Mac&amp;rsquo;s standalone client. Plus many plugins. However, it&amp;rsquo;s concerning that the Atlas browser&amp;rsquo;s Agent quota is surprisingly low, almost unusable for practical purposes.&lt;/li>
&lt;/ol>
&lt;h1 id="current-alternatives">Current Alternatives
&lt;/h1>&lt;p>&lt;img src="https://nansenli.com/img/ai2025-apps.png"
loading="lazy"
alt="AI apps on my phone"
>&lt;/p>
&lt;p>I&amp;rsquo;ve tried numerous free software, including all conceivable/most LLMs, even models developed by teams like MiniMax/Kimi. I&amp;rsquo;ve summarized my findings. Before summarizing, I need to define a software industry term. This phenomenon is very characteristic of China, or regional characteristics.&lt;/p>
&lt;p>Current LLM software indeed shows a trend of sacrificing tool purity and user experience in pursuit of daily active users (DAU), retention rates, and monetization. Some LLM software is stuffed with many operational elements, such as trendy operational topics, or many things not very related to technology, such as agents, competition zones, headline copywriting, or Xiaohongshu copywriting. This phenomenon indicates software bloat, over-operationalization, de-tooling, vulgarization, and reduced signal-to-noise ratio. In this article, we call this phenomenon &lt;strong>platform decay / enshittification&lt;/strong>.&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Zhipu Qingyan (智谱清言)&lt;/strong>: Zhipu Qingyan&amp;rsquo;s GLM4.6 is a very powerful model, especially its Agent capabilities. However, its client is severely enshittified, with the homepage immediately showing enshittified content like &amp;ldquo;Love Strategist&amp;rdquo; and &amp;ldquo;Dark Cuisine Challenge&amp;rdquo;. Not recommended.&lt;/li>
&lt;li>&lt;strong>Yunbao&lt;/strong>: Yunbao is Tencent&amp;rsquo;s software, supporting Tencent&amp;rsquo;s self-developed models and DeepSeek, but model capabilities are average. The software shows mild enshittification, such as embedding Tencent game Honor of Kings content. Not recommended.&lt;/li>
&lt;li>&lt;strong>Gemini&lt;/strong>: The new king. Gemini 3 Pro model performance is outstanding. Nano Banana Pro&amp;rsquo;s image generation capability is excellent. Suitable for work and life use. Operated by Google, overall experience is good. However, recently due to high traffic, the service is not particularly stable. Recently, traffic has been too high, service degradation is severe, and quantized services have started to be provided, so quality has declined. Long thinking with multiple tool calls is not as good as GPT5-Thinking or Grok Expert. The software has almost no enshittification. &lt;strong>Highly recommended&lt;/strong>.&lt;/li>
&lt;li>&lt;strong>Grok&lt;/strong>: The free king. Powerful search capabilities, both short and long thinking produce high-quality results. Excellent at mining network information. Free Expert version is sufficient for general users. Some operational content, such as AI virtual characters, but enshittification is very mild. &lt;strong>Recommended&lt;/strong>.&lt;/li>
&lt;li>&lt;strong>ChatGPT&lt;/strong>: Gradually stepping down from the throne. Many features are still only available in ChatGPT, such as voice conversations. Large user base. Free version is sufficient. Many users still have inertia/dependency. Can keep on phone.&lt;/li>
&lt;li>&lt;strong>DeepSeek&lt;/strong>: The stable king. Fewest features, completely no enshittification, the cleanest application, extremely stable service, extremely stable model style, and quite stable intelligence. &lt;strong>Recommended to keep on phone&lt;/strong>.&lt;/li>
&lt;li>&lt;strong>Kimi&lt;/strong>: Model capability is average. Advertised Agent capabilities are strong, but actual experience shows heavy templating with lots of prompt engineering traces. Not recommended. Software has no enshittification, which is a plus.&lt;/li>
&lt;li>&lt;strong>Claude&lt;/strong>: The god of code. Client performance is average, especially search is not as aggressive as Grok and ChatGPT thinking. Still suitable for work use. No enshittification. Can keep on phone, using free quota. It&amp;rsquo;s worth noting that the company has discriminatory regional remarks, which is concerning.&lt;/li>
&lt;li>&lt;strong>Dola&lt;/strong>: A client produced by Douyin (TikTok). Model capability is average. Voice recognition capability is excellent. Severely enshittified. Not recommended.&lt;/li>
&lt;li>&lt;strong>Copilot&lt;/strong>: A product relying on Microsoft search and ChatGPT, the software is mediocre. Not recommended.&lt;/li>
&lt;li>&lt;strong>Manus&lt;/strong>: Early reliance on viral marketing affects the team&amp;rsquo;s long-term healthy development. This product will likely die a slow death in the medium to long term. Although it has some Agent capabilities, not recommended for long-term use.&lt;/li>
&lt;li>&lt;strong>Perplexity&lt;/strong>: Model capability is average. Software usage logic is slightly confusing. Management team&amp;rsquo;s discriminatory remarks are concerning. Not recommended.&lt;/li>
&lt;li>&lt;strong>MiniMax&lt;/strong>: Model capability is excellent, especially the Minimax M2 model, with good Agent capabilities. However, software experience is poor, even basic copy and paste functions are missing. Not recommended.&lt;/li>
&lt;li>&lt;strong>Tongyi (通义)&lt;/strong>: Only available for download in China. Software is heavily enshittified. Opening it reveals a screen full of operational content that&amp;rsquo;s suffocating, making it impossible to continue using. &lt;strong>Recommend immediate uninstallation&lt;/strong>.&lt;/li>
&lt;li>&lt;strong>Qwen Chat&lt;/strong>: Currently still not available on iOS platform. It&amp;rsquo;s a de-enshittified version of Tongyi, supporting Qwen large models. If it can be listed, it&amp;rsquo;s recommended.&lt;/li>
&lt;/ul>
&lt;p>Summary:&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>&lt;strong>Software Name&lt;/strong>&lt;/th>
&lt;th>&lt;strong>Model Capability&lt;/strong>&lt;/th>
&lt;th>&lt;strong>Enshittification&lt;/strong>&lt;/th>
&lt;th>&lt;strong>Advantages&lt;/strong>&lt;/th>
&lt;th>&lt;strong>Disadvantages&lt;/strong>&lt;/th>
&lt;th>&lt;strong>Recommendation&lt;/strong>&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>Z-ai (Chinese version)&lt;/td>
&lt;td>Strong&lt;/td>
&lt;td>Severe&lt;/td>
&lt;td>Strong Agent capability&lt;/td>
&lt;td>Client enshittified&lt;/td>
&lt;td>Not recommended&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Yunbao&lt;/td>
&lt;td>Average&lt;/td>
&lt;td>Mild&lt;/td>
&lt;td>Supports multiple models&lt;/td>
&lt;td>Embedded game content&lt;/td>
&lt;td>Not recommended&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Gemini&lt;/td>
&lt;td>Excellent&lt;/td>
&lt;td>None&lt;/td>
&lt;td>Outstanding performance, good experience&lt;/td>
&lt;td>Unstable service&lt;/td>
&lt;td>&lt;strong>Highly recommended&lt;/strong>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Grok&lt;/td>
&lt;td>Strong&lt;/td>
&lt;td>Mild&lt;/td>
&lt;td>Free, strong search capability&lt;/td>
&lt;td>Some operational content&lt;/td>
&lt;td>&lt;strong>Recommended&lt;/strong>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>ChatGPT&lt;/td>
&lt;td>Declining&lt;/td>
&lt;td>Mild&lt;/td>
&lt;td>Many features, large user base&lt;/td>
&lt;td>Gradually stepping down&lt;/td>
&lt;td>Can keep&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>DeepSeek&lt;/td>
&lt;td>Stable&lt;/td>
&lt;td>None&lt;/td>
&lt;td>Stable, no enshittification&lt;/td>
&lt;td>Fewest features&lt;/td>
&lt;td>&lt;strong>Recommended to keep&lt;/strong>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Kimi&lt;/td>
&lt;td>Average&lt;/td>
&lt;td>None&lt;/td>
&lt;td>No enshittification&lt;/td>
&lt;td>Heavy templating&lt;/td>
&lt;td>Not recommended&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Claude&lt;/td>
&lt;td>Strong&lt;/td>
&lt;td>None&lt;/td>
&lt;td>Suitable for work use&lt;/td>
&lt;td>Average search performance&lt;/td>
&lt;td>Can keep&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Dola&lt;/td>
&lt;td>Average&lt;/td>
&lt;td>Severe&lt;/td>
&lt;td>Excellent voice recognition&lt;/td>
&lt;td>Average model capability&lt;/td>
&lt;td>Not recommended&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Copilot&lt;/td>
&lt;td>Average&lt;/td>
&lt;td>None&lt;/td>
&lt;td>Microsoft support&lt;/td>
&lt;td>Mediocre&lt;/td>
&lt;td>Not recommended&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Manus&lt;/td>
&lt;td>Average&lt;/td>
&lt;td>None&lt;/td>
&lt;td>Some Agent capability&lt;/td>
&lt;td>Poor long-term development&lt;/td>
&lt;td>Not recommended&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Perplexity&lt;/td>
&lt;td>Average&lt;/td>
&lt;td>Mild&lt;/td>
&lt;td>No significant advantages&lt;/td>
&lt;td>Discriminatory remarks&lt;/td>
&lt;td>Not recommended&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>MiniMax&lt;/td>
&lt;td>Excellent&lt;/td>
&lt;td>None&lt;/td>
&lt;td>Excellent model capability&lt;/td>
&lt;td>Missing basic functions&lt;/td>
&lt;td>Not recommended&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Tongyi&lt;/td>
&lt;td>Strong&lt;/td>
&lt;td>Very severe&lt;/td>
&lt;td>Strong model capability&lt;/td>
&lt;td>Over-operationalized&lt;/td>
&lt;td>&lt;strong>Recommend uninstallation&lt;/strong>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Qwen&lt;/td>
&lt;td>Strong&lt;/td>
&lt;td>None&lt;/td>
&lt;td>Strong model capability&lt;/td>
&lt;td>Not yet on iOS&lt;/td>
&lt;td>Can keep&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>Conclusion:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Highly recommended tools&lt;/strong>:
&lt;ul>
&lt;li>&lt;strong>Gemini&lt;/strong>: Despite service instability, model performance is outstanding and user experience is good, suitable for work and life use.&lt;/li>
&lt;li>&lt;strong>Grok&lt;/strong>: Powerful search capabilities and free usage make it a worthwhile choice to keep.&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>&lt;strong>Recommended tools to keep&lt;/strong>:
&lt;ul>
&lt;li>&lt;strong>ChatGPT&lt;/strong>: Although gradually stepping down from the throne, the free version still meets most needs, and has a large user base.&lt;/li>
&lt;li>&lt;strong>DeepSeek&lt;/strong>: Known for stability, with no over-operationalization issues, maintaining tool purity.&lt;/li>
&lt;li>&lt;strong>Claude&lt;/strong>: Excellent performance in code, suitable for work use.&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>&lt;strong>Not recommended tools&lt;/strong>:
&lt;ul>
&lt;li>&lt;strong>Zhipu Qingyan, Yunbao, Kimi, Dola, Copilot, Manus, Perplexity, MiniMax, Tongyi&lt;/strong>: These tools are either severely enshittified, have average model capabilities, or are missing basic functions, and are not recommended for long-term use. However, the underlying models are still noteworthy, such as Kimi, MiniMax, and Qwen models. Unfortunately, the products aren&amp;rsquo;t user-friendly enough. Additionally, as of today, Qwen is still not available on the iOS App Store.&lt;/li>
&lt;/ul>
&lt;/li>
&lt;/ol>
&lt;blockquote>
&lt;p>The above conclusions are limited to user clients, published in November 2025. Does not involve API calls.
For work and programming scenarios, Claude Sonnet 4.5, ChatGPT-5.1-Codex, Grok Code Fast 1, MiniMax M2 and GLM 4.6, Gemini 3 Pro are worth recommending&lt;/p>
&lt;/blockquote>
&lt;h3 id="postscript">Postscript
&lt;/h3>&lt;p>Times have changed. ChatGPT has improved its experience in Europe and it now works very well, so the content of this post is no longer valid.&lt;/p></description></item><item><title>April 2025 Tech Notes 2</title><link>https://nansenli.com/post/2025/04/tech250402/</link><pubDate>Sun, 20 Apr 2025 12:58:30 +0100</pubDate><guid>https://nansenli.com/post/2025/04/tech250402/</guid><description>&lt;h1 id="github-copilot">Github Copilot
&lt;/h1>&lt;p>&lt;img src="https://nansenli.com/img/rescued/6f49f8b9f1ec.png"
loading="lazy"
alt="Image.png"
>&lt;/p>
&lt;p>No more fear when facing unfamiliar Github projects. Just click the Copilot icon on the project homepage to start a conversation, and you can even use GPT-4.1 for communication, with up to a 1M context window. From questions like &amp;ldquo;What can this project be used for?&amp;rdquo; to &amp;ldquo;How to deploy and handle bugs?&amp;rdquo;, it can answer them all well.&lt;/p>
&lt;h1 id="dia-browser">Dia Browser
&lt;/h1>&lt;p>&lt;img src="https://nansenli.com/img/rescued/ba4da2cf2850.png"
loading="lazy"
alt="Image.png"
>&lt;/p>
&lt;p>&lt;a class="link" href="https://www.diabrowser.com/" target="_blank" rel="noopener"
>https://www.diabrowser.com/&lt;/a>&lt;/p>
&lt;p>It&amp;rsquo;s not yet in public beta, but you can submit your email via the link above to join the waitlist.&lt;/p>
&lt;p>The Arc browser is already great, offering a stable, mature, and consistent experience—it&amp;rsquo;s much more than just a sidebar tab innovation.&lt;/p>
&lt;p>Dia is the next-generation product from the Arc browser team. Honestly, I don&amp;rsquo;t quite understand why they need to launch a new product instead of integrating these features directly into Arc. Now, the founder seems to have put Arc on hold.&lt;/p>
&lt;p>From a product experience perspective, Dia browser seems to consist of the following parts:&lt;/p>
&lt;ul>
&lt;li>A built-in AI assistant for daily conversations&lt;/li>
&lt;li>A browser operation assistant similar to nanobrowser or browser-use, helping plan and operate the browser&lt;/li>
&lt;li>An AI writing assistant that can generate text at the cursor&lt;/li>
&lt;/ul>
&lt;p>A product with great potential.&lt;/p>
&lt;h1 id="release-of-gpt-o3--o4-mini--o4-mini-high--41--41-mini--41-nano">Release of GPT o3 / o4-mini / o4-mini-high / 4.1 / 4.1 mini / 4.1 nano
&lt;/h1>&lt;table>
&lt;thead>
&lt;tr>
&lt;th>&lt;strong>Model&lt;/strong>&lt;/th>
&lt;th>&lt;strong>Context Window&lt;/strong>&lt;/th>
&lt;th>&lt;strong>Input Price&lt;/strong>&lt;/th>
&lt;th>&lt;strong>Output Price&lt;/strong>&lt;/th>
&lt;th>&lt;strong>Image Price&lt;/strong>&lt;/th>
&lt;th>&lt;strong>Main Features&lt;/strong>&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>o4-mini-high&lt;/td>
&lt;td>200K tokens&lt;/td>
&lt;td>$1.10/M&lt;/td>
&lt;td>$4.40/M&lt;/td>
&lt;td>$0.842/K image input&lt;/td>
&lt;td>High-reasoning version of o4-mini; high throughput, low latency; supports chain calls and structured output&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>o4-mini&lt;/td>
&lt;td>200K tokens&lt;/td>
&lt;td>$1.10/M&lt;/td>
&lt;td>$4.40/M&lt;/td>
&lt;td>$0.842/K image input&lt;/td>
&lt;td>Compact reasoning; AIME 99.5%, high SWE-bench score; excellent STEM, vision, and code editing capabilities&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>o3&lt;/td>
&lt;td>200K tokens&lt;/td>
&lt;td>$10/M&lt;/td>
&lt;td>$40/M&lt;/td>
&lt;td>$7.65/K image input&lt;/td>
&lt;td>General strong model; top math, science, coding, and vision reasoning; BYOK required&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>GPT-4.1&lt;/td>
&lt;td>1.05M tokens&lt;/td>
&lt;td>$2/M&lt;/td>
&lt;td>$8/M&lt;/td>
&lt;td>N/A&lt;/td>
&lt;td>Flagship model; million-token context; excellent instruction following, enterprise-grade retrieval, and multimodal understanding&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>GPT-4.1 Mini&lt;/td>
&lt;td>1.05M tokens&lt;/td>
&lt;td>$0.40/M&lt;/td>
&lt;td>$1.60/M&lt;/td>
&lt;td>N/A&lt;/td>
&lt;td>Medium version; close to GPT-4o performance; low latency, low cost; strong coding and vision understanding&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>GPT-4.1 Nano&lt;/td>
&lt;td>1.05M tokens&lt;/td>
&lt;td>$0.10/M&lt;/td>
&lt;td>$0.40/M&lt;/td>
&lt;td>N/A&lt;/td>
&lt;td>Fastest and cheapest in the series; million-token context; suitable for classification and autocomplete tasks&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>GPT-4.1 Nano is naturally suited for translation and is extremely cost-effective. For example, when used for immersive translation with bilingual subtitles enabled, translating an entire season of Black Mirror costs only about $0.005 per episode, and all 7 seasons just $0.25—very low cost.&lt;/p>
&lt;p>The context window for GPT-4.1 has finally reached 1M. Previously, GPT was far behind in code writing, but now it has caught up.&lt;/p>
&lt;p>o3, o4-mini, and o4-mini-high are all surprisingly impressive, giving a bit of a &amp;ldquo;mini deep research&amp;rdquo; feel.&lt;/p>
&lt;p>Currently, my daily usage has basically become:&lt;/p>
&lt;ul>
&lt;li>o4-mini: Simple search and summarization, tolerant of mistakes&lt;/li>
&lt;li>o4-mini-high: Complex search, summarization, multi-step reasoning&lt;/li>
&lt;li>o3: Complex search, summarization, multi-step reasoning where high accuracy is required&lt;/li>
&lt;li>4o: Intuitive and knowledge-based answers&lt;/li>
&lt;/ul>
&lt;p>I&amp;rsquo;ve also tracked the usage frequency of each model, and I use almost all of them—none are useless, unlike o3-mini, which is almost unusable due to high hallucinations.&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/018788a20986.png"
loading="lazy"
alt="Image.png"
>&lt;/p>
&lt;h1 id="yuanbao-integrates-with-wechat-contacts">Yuanbao Integrates with WeChat Contacts
&lt;/h1>&lt;p>Yuanbao WeChat can directly add contacts and send messages, supporting both voice and image messages. It&amp;rsquo;s a competitor to Doubao. The advantage is that no extra app installation is required. I previously recommended Doubao to my parents, and they really liked it. If Yuanbao could also support voice calls or even video chat, it would be even more valuable.&lt;/p>
&lt;h1 id="openai-free-token-giveaway">OpenAI Free Token Giveaway
&lt;/h1>&lt;p>&lt;img src="https://nansenli.com/img/rescued/7c4b97a270c2.png"
loading="lazy"
alt="Image.png"
>&lt;/p>
&lt;p>OpenAI is now offering a free token giveaway. You can enable sharing logs in Data controls to claim them. Unfortunately, the duration is too short—just a week or two—so it&amp;rsquo;s hard to use up all the quota.&lt;/p>
&lt;h1 id="other-github-dailyweekly-trending-projects">Other Github Daily/Weekly Trending Projects
&lt;/h1>&lt;p>&lt;a class="link" href="https://github.com/kestra-io/kestra" target="_blank" rel="noopener"
>https://github.com/kestra-io/kestra&lt;/a>&lt;/p>
&lt;p>An automation platform, with competitors including: Airflow (data pipelines), Zapier (low-code marketing automation), Rundeck (ops automation), Camunda (e-commerce business automation)&lt;/p>
&lt;p>&lt;a class="link" href="https://github.com/Zackriya-Solutions/meeting-minutes" target="_blank" rel="noopener"
>https://github.com/Zackriya-Solutions/meeting-minutes&lt;/a>&lt;/p>
&lt;p>A speech-to-text software, a competitor to Apple Voice Memos.&lt;/p>
&lt;p>&lt;a class="link" href="https://github.com/maybe-finance/maybe" target="_blank" rel="noopener"
>https://github.com/maybe-finance/maybe&lt;/a>&lt;/p>
&lt;p>Personal financial report management, similar to Youzhi Youxing.&lt;/p>
&lt;p>&lt;a class="link" href="https://github.com/supabase-community/supabase-mcp" target="_blank" rel="noopener"
>https://github.com/supabase-community/supabase-mcp&lt;/a>&lt;/p>
&lt;p>A tool for connecting to databases when developing software with AI locally, based on the MCP protocol. If MCP becomes a real future standard, databases may natively support MCP, creating new competition.&lt;/p>
&lt;p>&lt;a class="link" href="https://github.com/langgenius/dify" target="_blank" rel="noopener"
>https://github.com/langgenius/dify&lt;/a>&lt;/p>
&lt;p>A low-code LLM platform, a competitor to LangChain, and can partially replace LangChain in current application scenarios.&lt;/p></description></item><item><title>April 2025 Tech Notes</title><link>https://nansenli.com/post/2025/04/tech2504/</link><pubDate>Tue, 15 Apr 2025 20:45:20 +0100</pubDate><guid>https://nansenli.com/post/2025/04/tech2504/</guid><description>&lt;h1 id="introduction">Introduction
&lt;/h1>&lt;p>I plan to record some weekly tech-related notes and insights starting from this issue. I&amp;rsquo;ll probably update every two to four weeks.&lt;/p>
&lt;h1 id="migrating-personal-blog-from-hexo-to-hugo">Migrating Personal Blog from Hexo to Hugo
&lt;/h1>&lt;p>This week, I migrated my blog from Hexo to Hugo, primarily because:&lt;/p>
&lt;p>Hugo has better i18n support. Hexo doesn&amp;rsquo;t support i18n very well; I previously modified the template to implement Chinese and English switching, but the maintenance cost was high. I discovered that Hugo natively supports multiple languages.&lt;/p>
&lt;p>After switching, I also found the following advantages:&lt;/p>
&lt;p>Theme configuration is simpler. I&amp;rsquo;m using the &lt;a class="link" href="https://github.com/CaiJimmy/hugo-theme-stack" target="_blank" rel="noopener"
>Stack&lt;/a> theme, which has been mostly problem-free. The official example configuration works well right out of the box. It also comes with Google Analytics integration and RSS feeds that automatically separate by language, making it very convenient.&lt;/p>
&lt;p>During the migration process, I didn&amp;rsquo;t encounter any major obstacles. You just need your old Markdown files, copy them to the new project directory, and force push to your original GitHub Pages repository. After force pushing, you&amp;rsquo;ll need to reconfigure your custom domain in GitHub, and the upgrade will be seamless, including comments and other features.&lt;/p>
&lt;p>The only difference is that Hugo uses lowercase links by default, while Hexo preserves case sensitivity, which might cause some old links to become invalid, requiring search engines to rebuild their indices.&lt;/p>
&lt;h1 id="mysterious-optimus-alpha-model-appears-on-httpsopenrouteraihttpsopenrouterai-and-dominates-rankings">Mysterious Optimus Alpha Model Appears on &lt;a class="link" href="https://openrouter.ai/" target="_blank" rel="noopener"
>https://openrouter.ai/&lt;/a> and Dominates Rankings
&lt;/h1>&lt;p>&lt;img src="https://nansenli.com/img/rescued/a7835acf9e85.png"
loading="lazy"
alt="Image.png"
>&lt;/p>
&lt;p>Because this model was free and powerful, it was chosen as OpenRouter&amp;rsquo;s primary traffic handling model after launch.&lt;/p>
&lt;p>After OpenAI officially announced GPT-4.1 on the 14th, the model was removed from OpenRouter. Obviously, this model was GPT-4.1.&lt;/p>
&lt;p>GPT-4.1 is a model specialized in programming capabilities, with a 1M context window. Indeed, in the programming field, OpenAI has been struggling against Claude 3.7 and Gemini 2.5 Pro. Neither o1 nor o3 models have achieved a leading position in programming. Hopefully, OpenAI will keep improving.&lt;/p>
&lt;blockquote>
&lt;p>OpenRouter is a middleware service platform that allows users to &lt;strong>access multiple large language model APIs&lt;/strong> through a unified interface. You can think of it as a &amp;ldquo;proxy&amp;rdquo; or &amp;ldquo;router&amp;rdquo; that manages API calls to different large model providers (OpenAI, Anthropic, Mistral, Google Gemini, Cohere, etc.).&lt;/p>
&lt;/blockquote>
&lt;p>It seems that this time, OpenAI chose OpenRouter as its test release platform, rather than &lt;a class="link" href="https://lmarena.ai/" target="_blank" rel="noopener"
>https://lmarena.ai/&lt;/a>.&lt;/p>
&lt;h1 id="openrouterai">OpenRouter.ai
&lt;/h1>&lt;p>&lt;a class="link" href="https://openrouter.ai/" target="_blank" rel="noopener"
>OpenRouter&lt;/a>&lt;/p>
&lt;p>This is an LLM platform that includes many free/paid high-quality models.&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/def8fd9ff726.png"
loading="lazy"
alt="Image.png"
>&lt;/p>
&lt;p>Select &amp;ldquo;Prompt pricing&amp;rdquo; as 0, then choose &amp;ldquo;Top Weekly&amp;rdquo; to see the best free models currently available.&lt;/p>
&lt;p>However, when using it, you need to enable logging and data usage for training.&lt;/p>
&lt;p>Currently, Gemini 2.5 Pro is an option.&lt;/p>
&lt;p>The Ranking feature allows you to view real-time traffic distribution across all models on the platform, making it easy to see which models perform best.&lt;/p>
&lt;h1 id="nanobrowser">nanobrowser
&lt;/h1>&lt;p>&lt;img src="https://nansenli.com/img/rescued/8d34bb10e9ba.png"
loading="lazy"
alt="Image.png"
>&lt;/p>
&lt;p>A browser plugin for Chrome that can control browser automation operations.&lt;/p>
&lt;p>I tested it and found it doesn&amp;rsquo;t support Arc browser. The experience was limited - page scrolling occasionally fails, it can&amp;rsquo;t summarize entire page content (only partial content is visible), and operations are sometimes incorrect.&lt;/p>
&lt;p>Although I don&amp;rsquo;t particularly recommend this product, it might represent a future direction. It seems we&amp;rsquo;ll see many AI agents emerging.&lt;/p>
&lt;p>Gray market industries like ticket scalping and point farming will probably be delighted.&lt;/p>
&lt;h1 id="markitdown">markitdown
&lt;/h1>&lt;p>&lt;img src="https://nansenli.com/img/rescued/e37c2c0e7d7a.png"
loading="lazy"
alt="Image.png"
>&lt;/p>
&lt;p>A tool for converting content in different formats to Markdown.&lt;/p>
&lt;p>Offline, it can process different files into Markdown format. I tried it, and the conversion is fairly basic.&lt;/p>
&lt;p>It also supports integration with LLMs to improve conversion results and handle images in documents.&lt;/p>
&lt;p>This could potentially be a good MCP service in the future.&lt;/p>
&lt;h1 id="pdf-search">PDF search
&lt;/h1>&lt;p>&lt;img src="https://nansenli.com/img/rescued/d1310e598823.png"
loading="lazy"
alt="Image.png"
>&lt;/p>
&lt;p>An application that can quickly search content across multiple PDFs.&lt;/p>
&lt;p>It uses a fuzzy search approach. The principle is to split PDFs and perform embedding vector calculations, then match them with search terms.&lt;/p>
&lt;p>Suitable for searching documents during exam periods.&lt;/p>
&lt;p>The downside is that indexing is done by page, so after searching, you need to find the corresponding part on the page yourself.&lt;/p></description></item><item><title>Summary of LLM Tools Usage Experience</title><link>https://nansenli.com/post/2025/04/aitest/</link><pubDate>Tue, 08 Apr 2025 19:47:48 +0000</pubDate><guid>https://nansenli.com/post/2025/04/aitest/</guid><description>&lt;h1 id="chatgpt-o4-mini">ChatGPT o4-mini
&lt;/h1>&lt;p>&lt;a class="link" href="day://2025.04.17" >&lt;code>Thu, Apr 17&lt;/code>&lt;/a> The latest model released in the past two days, with significant reasoning capabilities and excellent multi-step execution. It appears that o4-mini has completely surpassed o3-mini in practical task execution performance.&lt;/p>
&lt;p>The following image shows an example of my usage. The question was: &amp;ldquo;In Black Mirror Season 7 Episode 3, there are many Asian characters. Are these Asians Chinese, Korean, Japanese, Malaysian, or Singaporean?&amp;rdquo; The model successfully called the search engine multiple times and obtained the correct answer:&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/51ec0e03d882.png"
loading="lazy"
alt="Image.png"
>&lt;/p>
&lt;p>Additionally, Projects can now be normally called and used in this mode. Since the model can search autonomously rather than using RAG methods, the efficiency of file searching has significantly improved.&lt;/p>
&lt;h1 id="chatgpt-excluding-o3-o4-mini">ChatGPT (excluding o3, o4-mini)
&lt;/h1>&lt;ul>
&lt;li>
&lt;p>GPT-4o: Versatile. Excellent for daily communication and learning, with sufficient context window to easily handle code and document retrieval tasks. Supports multiple attachment formats and online code execution. Drawback: generates limited context length, not suitable for very long content. For lengthy conversations, context compression is severe and may forget earlier content.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Projects: Somewhat redundant. Suitable for scenarios requiring frequent searching and text retrieval across multiple files.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>GPT-4o mini: Weaker than GPT-4o, but has a massive context window, recommended for translating very long texts.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>o1: A decent chain-of-thought model, suitable for solving complex code and mathematical problems. Not good at handling emotional or intuitive problems. However, the thought chain is too short and quality is not high. It&amp;rsquo;s recommended to use the output as input for Gemini-2.5pro or DeepSeek-R1 to optimize quality.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>o3-mini: Severe hallucinations, inferior to o1, but extremely fast reasoning speed.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Search: Slightly redundant. New version&amp;rsquo;s search results are heavily restricted, even inferior to model output without search. Suitable as a light search alternative.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Deep Research: Excellent tool. Best quality among similar features, most comprehensive output results.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>GPT-4.5-preview: Excellent. Massive context window, strong long-text comprehension. Has vast memory and strongest intuition. Not actually good at reasoning, but performs best due to low hallucinations.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Canvas: Overall inferior to Cursor. Context output too small, text length limited. Suitable for short-text scenarios with real-time editing.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Work with Apps on macOS: Functionality experience crushed by Cursor, essentially a simplified version of Cursor.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;h1 id="claude">Claude
&lt;/h1>&lt;ul>
&lt;li>Claude 3.7 Sonnet: Excellent, suitable for generating various code. Web version supports massive context, almost matching API&amp;rsquo;s context history message retention. Model very suitable for writing communication texts like emails, with accurate, concise, and unpretentious wording. Drawback: limited usage in free version.&lt;/li>
&lt;/ul>
&lt;h1 id="deepseek">DeepSeek
&lt;/h1>&lt;ul>
&lt;li>
&lt;p>R1: Severe hallucinations, unstable performance, occasionally affecting usability, requires very high quality prompts. When prompts are correct or context is complete, it outperforms o1. Can be used as input for o1&amp;rsquo;s output to optimize quality.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>V3: Very excellent, alternative to GPT-4o.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;h1 id="grok">Grok
&lt;/h1>&lt;ul>
&lt;li>
&lt;p>Grok-3: Generates text fluently and naturally, less AI-like, suitable for natural writing and novel creation. Overall mediocre, fewer productivity tools than GPT-4o.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Grok-3 + search: Excellent. Leverages English social media data, quickly analyzes news and current events. Outputs long content, barely filters search source content, stronger search capabilities than GPT-4o.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Grok-Deep Search: Slightly redundant, actually inferior to Grok-3 + search. Generated content heavily templated, affecting quality.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;h1 id="gemini">Gemini
&lt;/h1>&lt;ul>
&lt;li>
&lt;p>2.5 Pro: Excellent model, alternative to GPT-4o, minimal hallucinations, high-quality search results, complete and clear thought chains, strong logical reasoning. Supports integration with Google tools, very useful in specific scenarios, such as uploading screenshots or text to automatically create events in Google Calendar. Drawback: often claims to have performed searches when it actually hasn&amp;rsquo;t.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Deep Research: Average, slightly better than old GPT Search. Occasional comprehension deviations, search and document generation executed in stages, process fragmented, heavily templated. Currently the only product that can replace GPT Deep Research.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;h1 id="v0dev">v0.dev
&lt;/h1>&lt;ul>
&lt;li>Suitable for writing and previewing frontend components online.&lt;/li>
&lt;/ul>
&lt;h1 id="chatqwenai">chat.qwen.ai
&lt;/h1>&lt;ul>
&lt;li>Qwen2.5-Max: Very excellent model, alternative to GPT-4o, fast generation speed, supports thinking mode.&lt;/li>
&lt;/ul>
&lt;h1 id="zhihu-direct-answer">Zhihu Direct Answer
&lt;/h1>&lt;ul>
&lt;li>Supports searching Zhihu&amp;rsquo;s entire network content. Possible competitor is Xiaohongshu Direct Answer.&lt;/li>
&lt;/ul>
&lt;h1 id="perplexity">Perplexity
&lt;/h1>&lt;ul>
&lt;li>Supports basic search, mediocre quality, slightly redundant.&lt;/li>
&lt;/ul>
&lt;h1 id="mistralai">Mistral.ai
&lt;/h1>&lt;ul>
&lt;li>Fast speed, large context window, alternative to GPT4o-mini.&lt;/li>
&lt;/ul>
&lt;h1 id="cursor">Cursor
&lt;/h1>&lt;ul>
&lt;li>Excellent tool. Can index entire code repositories, suitable for large project development. Supports multiple model switching, local command line execution and terminal takeover, suitable for the following tasks:
&lt;ul>
&lt;li>Writing LaTeX documents, replacing Overleaf&lt;/li>
&lt;li>Assisting with various assignments&lt;/li>
&lt;li>Code repository search and specific feature location&lt;/li>
&lt;li>Creating unit tests&lt;/li>
&lt;li>Executing git operations&lt;/li>
&lt;li>Executing deployment tasks&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>Can execute almost everything involving documents and command lines, helpful in development, debugging, debugging, and document writing&lt;/li>
&lt;li>Truly the most Agent-like product at the current stage (though this claim no longer holds after the release of o3 and o4-mini, as they can also perform multi-step planning and task execution. However, o3 and o4-mini can only operate within their own limited environment, while Cursor can execute on the user&amp;rsquo;s computer, which is an advantage.)&lt;/li>
&lt;/ul>
&lt;h1 id="example-leetcode-tracker">Example: Leetcode Tracker
&lt;/h1>&lt;p>There are many shared Leetcode premium accounts on Taobao, but sometimes we still need to add our problem-solving records to our own accounts. After purchasing a shared premium account, we can export a company&amp;rsquo;s problems as a CSV file, then export our own problem-solving records as another CSV file. By comparing these two files, we can easily track our current progress. Cursor is very good at writing such small tools:&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/1a7bea6f828c.png"
loading="lazy"
alt="Image.png"
>&lt;/p>
&lt;p>We just need to copy elements from the webpage, and Cursor can write complete CSV files for us and automatically generate a perfect frontend interface, greatly improving our problem-solving efficiency.&lt;/p></description></item><item><title>Key Points for Installing GPU Version of TensorFlow on Deepin</title><link>https://nansenli.com/post/jianshu/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/deepin-%E5%AE%89%E8%A3%85-gpu%E7%89%88-tensorflow-%E8%A6%81%E7%82%B9/</link><pubDate>Wed, 13 Sep 2017 10:44:00 +0000</pubDate><guid>https://nansenli.com/post/jianshu/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/deepin-%E5%AE%89%E8%A3%85-gpu%E7%89%88-tensorflow-%E8%A6%81%E7%82%B9/</guid><description>&lt;h2 id="introduction">Introduction
&lt;/h2>&lt;p>Although there are many tutorials on how to install TensorFlow on Ubuntu, there isn&amp;rsquo;t a single article explaining how to install TensorFlow on Deepin systems. Here I&amp;rsquo;ll explain the key points of the installation process.&lt;/p>
&lt;h2 id="process">Process
&lt;/h2>&lt;ul>
&lt;li>First, download the Nvidia Linux driver from the official website. If you can&amp;rsquo;t run it, you&amp;rsquo;ll need to modify the driver&amp;rsquo;s execution permissions using chmod u+x. For information on installing graphics drivers on Deepin, refer to this article: &lt;a class="link" href="https://wiki.deepin.org/index.php?title=%E6%98%BE%E5%8D%A1" target="_blank" rel="noopener"
>https://wiki.deepin.org/index.php?title=%E6%98%BE%E5%8D%A1&lt;/a>&lt;/li>
&lt;li>Install Docker-ce. You must follow the official installation guide; this is the only method to properly install the latest version of Docker: &lt;a class="link" href="https://wiki.deepin.org/index.php?title=Docker" target="_blank" rel="noopener"
>https://wiki.deepin.org/index.php?title=Docker&lt;/a>&lt;/li>
&lt;li>Download Nvidia-docker. Find the Ubuntu installation package and installation steps on the Nvidia-Docker page on GitHub and follow them.&lt;/li>
&lt;li>Next, run &lt;code>docker run -it -p 8888:8888 tensorflow/tensorflow:latest-gpu&lt;/code>. At this point, you can enter the official TensorFlow GPU version container. Open localhost:8888 to see the Jupyter page.&lt;/li>
&lt;/ul>
&lt;h2 id="pitfalls">Pitfalls
&lt;/h2>&lt;p>When installing the Nvidia graphics driver, according to the installation software&amp;rsquo;s instructions, you need to close the Linux desktop system X server. At this point, you can use tty1&lt;del>6 for terminal operations. However, if Deepin has already installed its own Nvidia driver, when stopping the lightdm service, Deepin closes both the X server and the monitor, making it impossible to display the tty1&lt;/del>6 screens. This issue prevents Nvidia driver installation. Even after manually uninstalling the official Deepin Nvidia driver through complex uninstallation commands, when switching to tty1~6 and closing the graphical interface, the tty screen still shuts down, resulting in a black screen and making it impossible to continue with driver uninstallation and reinstallation of the official Nvidia driver.&lt;/p>
&lt;h2 id="solution">Solution
&lt;/h2>&lt;p>To avoid problems with Nvidia driver installation while maintaining a clean and scientific operation, you must install the official Nvidia Linux driver during the first installation of the Deepin system, before any graphics drivers are installed. First, press the shortcut &amp;ldquo;Ctrl+Alt+F2&amp;rdquo; to enter tty2, then enter &lt;code>sudo systemctl stop lightdm&lt;/code> to stop the lightdm service. At this point, when the computer closes the X server, it won&amp;rsquo;t cause the monitor to shut down. Then run &lt;code>chmod u+x NVIDIA-Linux-x86_64-352.55.run&lt;/code> to grant executable permissions, followed by &lt;code>sudo ./NVIDIA-Linux-x86_64-352.55.run&lt;/code> to install the driver file. After restarting, you can use the official closed-source driver normally.&lt;/p>
&lt;h2 id="follow-up">Follow-up
&lt;/h2>&lt;p>Using a GPU to run TensorFlow programs is very fast. Normal programs can speed up by more than 10 times, and some programs can speed up by 50 to 100 times, so using a GPU for TensorFlow programming is essential.
Additionally, I&amp;rsquo;m not sure if it&amp;rsquo;s an issue with Deepin, Nvidia-Docker, or the driver, but after the computer goes into standby mode, it causes errors in the TensorFlow container. Therefore, don&amp;rsquo;t let the machine enter standby mode during training.&lt;/p></description></item><item><title>TensorFlow: A Simple Introduction Case Study</title><link>https://nansenli.com/post/jianshu/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/tensorflow-%E4%B8%80%E4%B8%AA%E7%AE%80%E5%8D%95%E7%9A%84%E5%85%A5%E9%97%A8%E7%94%A8%E4%BE%8B/</link><pubDate>Wed, 09 Aug 2017 07:08:00 +0000</pubDate><guid>https://nansenli.com/post/jianshu/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/tensorflow-%E4%B8%80%E4%B8%AA%E7%AE%80%E5%8D%95%E7%9A%84%E5%85%A5%E9%97%A8%E7%94%A8%E4%BE%8B/</guid><description>&lt;h2 id="case-study">Case Study
&lt;/h2>&lt;p>Suppose I have a series of x-y data points with a linear relationship between x and y. How would we fit a straight line to this data?&lt;/p>
&lt;p>In the program below, train_x is a series of numbers between -1 and 1, and train_y is twice x plus 10, with a random number between 0 and 1 added.
Next, we build the model. X and Y are tensor placeholders waiting to be initialized. During optimization, the XY values in the model will continuously change to data from train_x and train_y, and then the optimizer will optimize by changing the slope w and intercept b in a direction that reduces error. Through iteration, w and b will eventually make the model fit the data.&lt;/p>
&lt;p>After the model is built, we start running it. First, we open a session, and we must remember to initialize all variables. Next, we iterate through all the data 10 times. In each iteration, we input a coordinate, calculate the error, and use gradient descent to correct w and b. Finally, we output the calculated values of w and b.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-gdscript3" data-lang="gdscript3">&lt;span class="line">&lt;span class="cl">&lt;span class="n">import&lt;/span> &lt;span class="n">tensorflow&lt;/span> &lt;span class="n">as&lt;/span> &lt;span class="n">tf&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">import&lt;/span> &lt;span class="n">numpy&lt;/span> &lt;span class="n">as&lt;/span> &lt;span class="n">np&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">train_x&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">np&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">linspace&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="o">-&lt;/span>&lt;span class="mi">1&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="mi">1&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="mi">101&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">train_y&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="mi">2&lt;/span> &lt;span class="o">*&lt;/span> &lt;span class="n">train_x&lt;/span> &lt;span class="o">+&lt;/span> &lt;span class="n">np&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">random&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">rand&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">train_x&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">shape&lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="mi">0&lt;/span>&lt;span class="p">])&lt;/span> &lt;span class="o">+&lt;/span> &lt;span class="mi">10&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">X&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">placeholder&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;float&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">Y&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">placeholder&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;float&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">w&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">Variable&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="mf">0.0&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">name&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;w&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">b&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">Variable&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="mf">0.0&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">name&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;b&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">loss&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">square&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">Y&lt;/span> &lt;span class="o">-&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">multiply&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">X&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="n">w&lt;/span>&lt;span class="p">)&lt;/span> &lt;span class="o">-&lt;/span> &lt;span class="n">b&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">train_op&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">train&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">GradientDescentOptimizer&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="mf">0.01&lt;/span>&lt;span class="p">)&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">minimize&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">loss&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">with&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">Session&lt;/span>&lt;span class="p">()&lt;/span> &lt;span class="n">as&lt;/span> &lt;span class="n">session&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">session&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">run&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">global_variables_initializer&lt;/span>&lt;span class="p">())&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">for&lt;/span> &lt;span class="n">i&lt;/span> &lt;span class="ow">in&lt;/span> &lt;span class="nb">range&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="mi">10&lt;/span>&lt;span class="p">):&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">for&lt;/span> &lt;span class="n">x&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="n">y&lt;/span> &lt;span class="ow">in&lt;/span> &lt;span class="n">zip&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">train_x&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">train_y&lt;/span>&lt;span class="p">):&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">session&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">run&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">train_op&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">feed_dict&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="p">{&lt;/span>&lt;span class="n">X&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="n">x&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">Y&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="n">y&lt;/span>&lt;span class="p">})&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">print&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;w: &amp;#34;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">session&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">run&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">w&lt;/span>&lt;span class="p">))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">print&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;b: &amp;#34;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">session&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">run&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">b&lt;/span>&lt;span class="p">))&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>Running results:
&lt;img src="http://otwwkzjm5.bkt.clouddn.com/17-8-9/99539186.jpg"
loading="lazy"
alt="Results"
>&lt;/p></description></item><item><title>TensorFlow Hello World Introduction</title><link>https://nansenli.com/post/jianshu/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/tensorflow-hello-world-%E5%85%A5%E9%97%A8/</link><pubDate>Tue, 08 Aug 2017 09:58:00 +0000</pubDate><guid>https://nansenli.com/post/jianshu/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/tensorflow-hello-world-%E5%85%A5%E9%97%A8/</guid><description>&lt;h2 id="quick-installation-guide">Quick Installation Guide
&lt;/h2>&lt;ul>
&lt;li>Step 1: Install Docker&lt;/li>
&lt;/ul>
&lt;p>&lt;a class="link" href="https://www.docker-cn.com/community-edition#/download" target="_blank" rel="noopener"
>https://www.docker-cn.com/community-edition#/download&lt;/a>&lt;/p>
&lt;p>Then configure the official Chinese mirror.&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/5832fec39b1c.jpg"
loading="lazy"
alt="image.png"
>&lt;/p>
&lt;ul>
&lt;li>Step 2: Set up TensorFlow environment&lt;/li>
&lt;/ul>
&lt;p>&lt;code>$ docker run -it -p 8888:8888 tensorflow/tensorflow&lt;/code>
Running this command will automatically download the TensorFlow image, provided that the repository mirror is set to a Chinese mirror; otherwise, the download will be very slow. After running the command, the terminal will display a URL that prompts you to open a web page. When you open this URL, you&amp;rsquo;ll see the TensorFlow Jupyter editing environment, where we&amp;rsquo;ll input all our code.&lt;/p>
&lt;ul>
&lt;li>Mounting Docker file directory
If we need to access local files, we need to mount a local folder to the container directory. Close the container, reopen it, and use &lt;code>-v host_directory:container_directory&lt;/code> for mounting.
&lt;code>docker run -v /Users/hahaha/tensorflow/:/notebooks -it -p 8888:8888 tensorflow/tensorflow&lt;/code>&lt;/li>
&lt;/ul>
&lt;p>Where /Users/hahaha/tensorflow/ is a folder on my Mac, and notebooks is the default Jupyter editing directory in the TensorFlow container.&lt;/p>
&lt;h2 id="running-hello-world-code">Running Hello World Code
&lt;/h2>&lt;p>&lt;img src="https://nansenli.com/img/rescued/5f59450b005f.png"
loading="lazy"
alt="image.png"
>
Create a new Python 2 Jupyter file, enter the following code, and then click the play button. At this point, a &amp;ldquo;Hello, TensorFlow!&amp;rdquo; string should appear below, indicating that the program has run successfully.&lt;/p>
&lt;h2 id="program-explanation">Program Explanation
&lt;/h2>&lt;p>From this simple code, we can see that TensorFlow is very easy to use. It&amp;rsquo;s imported as a standard Python library without requiring additional services to be started. For those new to TensorFlow, you might wonder why we need to use tf.constant() and tf.Session() to output a &amp;ldquo;Hello World&amp;rdquo; string when Python itself could do it. The reason is that TensorFlow defines and runs models and training through Graphs and Sessions, which provides significant benefits for complex models and distributed training.&lt;/p>
&lt;p>First, in TensorFlow, there are two concepts: Graph and Operation. Operation represents what needs to be computed. A Graph contains many Operations. A Session is used to execute Operations in a Graph.&lt;/p>
&lt;h2 id="basic-usage">Basic Usage
&lt;/h2>&lt;p>When using TensorFlow, you must understand that TensorFlow:&lt;/p>
&lt;ul>
&lt;li>Uses a &lt;code>graph&lt;/code> to represent computational tasks&lt;/li>
&lt;li>Executes the graph in a &lt;code>context&lt;/code> called a &lt;code>Session&lt;/code>&lt;/li>
&lt;li>Represents data using &lt;code>tensors&lt;/code>&lt;/li>
&lt;li>Maintains state through &lt;code>Variables&lt;/code>&lt;/li>
&lt;li>Uses &lt;code>feed&lt;/code> and &lt;code>fetch&lt;/code> to assign values to or retrieve data from &lt;code>arbitrary operations&lt;/code>&lt;/li>
&lt;/ul>
&lt;h2 id="overview">Overview
&lt;/h2>&lt;p>TensorFlow is a programming system that uses graphs to represent computational tasks. Nodes in the graph are called &lt;strong>ops&lt;/strong> (short for operations). An op takes 0 or more Tensors, performs computations, and produces 0 or more Tensors. Each Tensor is a typed multi-dimensional array. For example, you can represent a small batch of images as a four-dimensional floating-point array with dimensions [batch, height, width, channels].&lt;/p>
&lt;p>A TensorFlow graph &lt;strong>describes&lt;/strong> the computation process. To perform computation, the graph must be launched in a &lt;code>session&lt;/code>. The &lt;code>session&lt;/code> distributes the graph&amp;rsquo;s ops to devices like CPUs or GPUs and provides methods to execute ops. After execution, these methods return the resulting tensors. In Python, the returned tensors are &lt;a class="link" href="http://www.numpy.org/" target="_blank" rel="noopener"
>numpy&lt;/a> &lt;code>ndarray&lt;/code> objects. In C and C++, the returned tensors are tensorflow::Tensor instances.&lt;/p></description></item><item><title>Using TensorFlow for a Kaggle Task — Titanic: Machine Learning from Disaster</title><link>https://nansenli.com/post/jianshu/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/%E4%BD%BF%E7%94%A8tensorflow%E5%AE%8C%E6%88%90kaggle%E4%BB%BB%E5%8A%A1%E6%B3%B0%E5%9D%A6%E5%B0%BC%E5%85%8B%E5%8F%B7titanic--machine-learning-from-disaster/</link><pubDate>Fri, 04 Aug 2017 10:08:00 +0000</pubDate><guid>https://nansenli.com/post/jianshu/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/%E4%BD%BF%E7%94%A8tensorflow%E5%AE%8C%E6%88%90kaggle%E4%BB%BB%E5%8A%A1%E6%B3%B0%E5%9D%A6%E5%B0%BC%E5%85%8B%E5%8F%B7titanic--machine-learning-from-disaster/</guid><description>&lt;h2 id="import-necessary-libraries">Import Necessary Libraries
&lt;/h2>&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">import csv
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">import tensorflow as tf
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">import numpy as np
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">import random
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">import sys
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">import pandas as pd
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">from pandas import DataFrame
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">from __future__ import absolute_import
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">from __future__ import division
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">from __future__ import print_function
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="read-source-file-and-display">Read Source File and Display
&lt;/h2>&lt;p>In this section, we&amp;rsquo;ll work with basic CSV operations and display the results.
We&amp;rsquo;ll read the train.csv file downloaded from Kaggle and show its contents.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">trainFilePath = &amp;#39;./train.csv&amp;#39;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">trainSize = 0
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">def testCSV(filePath):
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> with open(filePath, &amp;#39;rb&amp;#39;) as trainFile:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> global trainSize
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> csvReader = csv.reader(trainFile)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> dataList = [data for data in csvReader]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> df = DataFrame(dataList[1:], columns=dataList[0])
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> trainSize = len(df)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(df)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(&amp;#34;trainSize&amp;#34;, trainSize)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">testCSV(trainFilePath)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="read-source-file-extract-data-and-build-neural-network">Read Source File, Extract Data, and Build Neural Network
&lt;/h2>&lt;p>In this section, we&amp;rsquo;ll extract gender, class, ticket fare, and SibSp from the source file to fit the survival probability.
Then we&amp;rsquo;ll build a 5-layer neural network with 3 hidden layers containing 4-10-20-10-2 neurons respectively.
Finally, we&amp;rsquo;ll execute the reading function.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;span class="lnt">21
&lt;/span>&lt;span class="lnt">22
&lt;/span>&lt;span class="lnt">23
&lt;/span>&lt;span class="lnt">24
&lt;/span>&lt;span class="lnt">25
&lt;/span>&lt;span class="lnt">26
&lt;/span>&lt;span class="lnt">27
&lt;/span>&lt;span class="lnt">28
&lt;/span>&lt;span class="lnt">29
&lt;/span>&lt;span class="lnt">30
&lt;/span>&lt;span class="lnt">31
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">def readTrainDataCSV(filePath):
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> global trainData, targetData, classifier
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> with open(filePath, &amp;#39;rb&amp;#39;) as trainFile:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> csvReader = csv.reader(trainFile)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> dataList = [data for data in csvReader]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> dataSize = len(dataList) - 1
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> trainData = np.ndarray((dataSize, 4), dtype=np.float32)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> targetData = np.ndarray((dataSize, 1), dtype=np.int32)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> trainDataFrame = DataFrame(dataList[1:], columns=dataList[0])
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> trainDataFrame_fliter = trainDataFrame.loc[:,[&amp;#39;Pclass&amp;#39;,&amp;#39;Sex&amp;#39;,&amp;#39;SibSp&amp;#39;,&amp;#39;Fare&amp;#39;,&amp;#39;Survived&amp;#39;]]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> for i in range(dataSize):
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> thisData = np.array(trainDataFrame_fliter.iloc[i])
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Pclass,Sex,SibSp,Fare,Survived = thisData
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Pclass = float(Pclass)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Sex = 0 if Sex == &amp;#39;female&amp;#39; else 1
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> SibSp = float(SibSp)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Fare = float(Fare)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Survived = int(Survived)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(Pclass,Sex,SibSp,Fare,Survived)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> trainData[i,:] = [Pclass,Sex,SibSp,Fare]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> targetData[i,:] = [Survived]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(thisData)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(trainData)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(targetData)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> feature_columns = [tf.contrib.layers.real_valued_column(&amp;#34;&amp;#34;, dimension=4)]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> classifier = tf.contrib.learn.DNNClassifier(feature_columns=feature_columns,
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> hidden_units=[10, 20, 10],
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> n_classes=2)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># model_dir=&amp;#34;/tmp/titanic_model&amp;#34;)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">readTrainDataCSV(trainFilePath)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="create-input-data">Create Input Data
&lt;/h2>&lt;p>We&amp;rsquo;ll wrap the training data and labels into a tuple and return it.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;span class="lnt">6
&lt;/span>&lt;span class="lnt">7
&lt;/span>&lt;span class="lnt">8
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-gdscript3" data-lang="gdscript3">&lt;span class="line">&lt;span class="cl">&lt;span class="k">def&lt;/span> &lt;span class="nf">get_train_inputs&lt;/span>&lt;span class="p">():&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">x&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">constant&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">trainData&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">y&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">tf&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">constant&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">targetData&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">print&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">x&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">print&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">y&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">return&lt;/span> &lt;span class="n">x&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">y&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">get_train_inputs&lt;/span>&lt;span class="p">()&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="train-the-model">Train the Model
&lt;/h2>&lt;p>Now we start training the neural network.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">def train():
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> classifier.fit(input_fn=get_train_inputs, steps=2000)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">train()
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="check-accuracy">Check Accuracy
&lt;/h2>&lt;p>We use the entire dataset to check accuracy. Note that we should use a validation set for this task, but since this is just for demonstration purposes, we&amp;rsquo;ll skip that step.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">accuracy_score = classifier.evaluate(input_fn=get_train_inputs,
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> steps=1)[&amp;#34;accuracy&amp;#34;]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">print(&amp;#34;accuracy:&amp;#34;,accuracy_score)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="read-test-set-and-output-results">Read Test Set and Output Results
&lt;/h2>&lt;p>In this section, we&amp;rsquo;ll read the test data from Kaggle and output the results to a file, which will ultimately be submitted to the official website.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;span class="lnt">21
&lt;/span>&lt;span class="lnt">22
&lt;/span>&lt;span class="lnt">23
&lt;/span>&lt;span class="lnt">24
&lt;/span>&lt;span class="lnt">25
&lt;/span>&lt;span class="lnt">26
&lt;/span>&lt;span class="lnt">27
&lt;/span>&lt;span class="lnt">28
&lt;/span>&lt;span class="lnt">29
&lt;/span>&lt;span class="lnt">30
&lt;/span>&lt;span class="lnt">31
&lt;/span>&lt;span class="lnt">32
&lt;/span>&lt;span class="lnt">33
&lt;/span>&lt;span class="lnt">34
&lt;/span>&lt;span class="lnt">35
&lt;/span>&lt;span class="lnt">36
&lt;/span>&lt;span class="lnt">37
&lt;/span>&lt;span class="lnt">38
&lt;/span>&lt;span class="lnt">39
&lt;/span>&lt;span class="lnt">40
&lt;/span>&lt;span class="lnt">41
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">testFilePath = &amp;#39;./test.csv&amp;#39;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">def readTestDataCSV(filePath):
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> global testData, PassengerIdStart
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> with open(filePath, &amp;#39;rb&amp;#39;) as testFile:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> csvReader = csv.reader(testFile)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> dataList = [data for data in csvReader]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> dataSize = len(dataList)-1
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> trainDataFrame = DataFrame(dataList[1:], columns=dataList[0])
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> trainDataFrame_fliter = trainDataFrame.loc[:,[&amp;#39;Pclass&amp;#39;,&amp;#39;Sex&amp;#39;,&amp;#39;SibSp&amp;#39;,&amp;#39;Fare&amp;#39;]]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> testData = np.ndarray((dataSize, 4), dtype=np.float32)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> PassengerIdStart = trainDataFrame[&amp;#39;PassengerId&amp;#39;][0]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> PassengerIdStart = int(PassengerIdStart)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(&amp;#39;PassengerId&amp;#39;,PassengerIdStart)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> for i in range(dataSize):
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> thisData = np.array(trainDataFrame_fliter.iloc[i])
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Pclass,Sex,SibSp,Fare = thisData
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Pclass = float(Pclass)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Sex = 0 if Sex == &amp;#39;female&amp;#39; else 1
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> SibSp = float(SibSp)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Fare = 0 if Fare==&amp;#39;&amp;#39; else float(Fare)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(Pclass,Sex,SibSp,Fare)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> testData[i,:] = [Pclass,Sex,SibSp,Fare]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(thisData)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(testData)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">def testData_samples():
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> return testData
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">readTestDataCSV(testFilePath)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">predictions = list(classifier.predict(input_fn=testData_samples))
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">print(predictions)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">with open(&amp;#39;predictions.csv&amp;#39;, &amp;#39;wb&amp;#39;) as csvfile:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> writer = csv.writer(csvfile, dialect=&amp;#39;excel&amp;#39;)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> writer.writerow([&amp;#39;PassengerId&amp;#39;,&amp;#39;Survived&amp;#39;])
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> PassengerId = PassengerIdStart
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> for i in predictions:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> writer.writerow([PassengerId, i])
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> PassengerId += 1
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>Finally, using only 4 features, we achieved an accuracy of 75%. The next goal is to utilize the other available data.&lt;/p></description></item><item><title>Deep Learning Beginner's Study Notes One (Udacity)</title><link>https://nansenli.com/post/jianshu/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E9%9B%B6%E5%9F%BA%E7%A1%80%E5%AD%A6%E4%B9%A0%E7%AC%94%E8%AE%B0%E4%B8%80%E4%BC%98%E8%BE%BE%E5%AD%A6%E5%9F%8E/</link><pubDate>Wed, 19 Jul 2017 07:42:00 +0000</pubDate><guid>https://nansenli.com/post/jianshu/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E9%9B%B6%E5%9F%BA%E7%A1%80%E5%AD%A6%E4%B9%A0%E7%AC%94%E8%AE%B0%E4%B8%80%E4%BC%98%E8%BE%BE%E5%AD%A6%E5%9F%8E/</guid><description>&lt;h2 id="introduction">Introduction
&lt;/h2>&lt;p>On a whim, I decided to learn machine learning. These are notes from my learning process.&lt;/p>
&lt;h2 id="preparation">Preparation
&lt;/h2>&lt;p>I made these preparations:&lt;/p>
&lt;ul>
&lt;li>A MacBook with Python environment set up, and numpy and matplotlib installed&lt;/li>
&lt;li>Registered for Udacity&amp;rsquo;s free &amp;ldquo;Deep Learning&amp;rdquo; course (in collaboration with Google)&lt;/li>
&lt;li>Studied Liao Xuefeng&amp;rsquo;s Python introductory tutorial&lt;/li>
&lt;li>Spent two days roughly browsing through &amp;ldquo;Machine Learning in Action&amp;rdquo;&lt;/li>
&lt;/ul>
&lt;p>Learning these fundamentals should be sufficient for the upcoming Udacity course.&lt;/p>
&lt;h2 id="course-one-from-machine-learning-to-deep-learning">Course One: From Machine Learning to Deep Learning
&lt;/h2>&lt;p>&lt;img src="https://nansenli.com/img/rescued/71c43f961173.png"
loading="lazy"
alt="Introduction"
>&lt;/p>
&lt;p>Sections 1-8 mainly introduce the current state of deep learning and related knowledge.&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/ff643a3a1ec4.png"
loading="lazy"
alt="image.png"
>&lt;/p>
&lt;p>Sections 9-12 introduce the softmax model.&lt;/p>
&lt;p>After a rough review of &amp;ldquo;Machine Learning in Action,&amp;rdquo; I learned that machine learning consists of several classification and clustering algorithms. On the surface, machine learning appears to be a collection of classification and clustering algorithms. Among these algorithms, one called logistic regression classification was introduced.&lt;/p>
&lt;p>In sections 9-12, the focus is on the classifier model—logistic regression, using the softmax function as the classification function.&lt;/p>
&lt;ul>
&lt;li>What is the softmax function?&lt;/li>
&lt;/ul>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/cb4b58f90872.jpg"
loading="lazy"
alt="softmax"
>&lt;/p>
&lt;p>This image illustrates what a softmax function is. For each number z in the original sequence, we calculate exp(z), and the proportion of each new number&amp;rsquo;s magnitude becomes the softmax probability for that number.&lt;/p>
&lt;ul>
&lt;li>Properties&lt;/li>
&lt;/ul>
&lt;p>If the inputs are scaled up proportionally, the classifier&amp;rsquo;s results become more polarized and confident. If the inputs are scaled down proportionally, the classifier&amp;rsquo;s results tend toward the average and lack confidence.&lt;/p>
&lt;ul>
&lt;li>Algorithm&lt;/li>
&lt;/ul>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;span class="lnt">6
&lt;/span>&lt;span class="lnt">7
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-python" data-lang="python">&lt;span class="line">&lt;span class="cl">&lt;span class="kn">import&lt;/span> &lt;span class="nn">numpy&lt;/span> &lt;span class="k">as&lt;/span> &lt;span class="nn">np&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">def&lt;/span> &lt;span class="nf">softmax&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">x&lt;/span>&lt;span class="p">):&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="s2">&amp;#34;&amp;#34;&amp;#34;Compute softmax values for each sets of scores in x.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">expList&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="p">[&lt;/span>&lt;span class="n">np&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">exp&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">i&lt;/span>&lt;span class="p">)&lt;/span> &lt;span class="k">for&lt;/span> &lt;span class="n">i&lt;/span> &lt;span class="ow">in&lt;/span> &lt;span class="n">x&lt;/span>&lt;span class="p">]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">expSum&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="nb">sum&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">expList&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">x&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="p">[&lt;/span>&lt;span class="n">i&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="n">expSum&lt;/span> &lt;span class="k">for&lt;/span> &lt;span class="n">i&lt;/span> &lt;span class="ow">in&lt;/span> &lt;span class="n">expList&lt;/span>&lt;span class="p">]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">return&lt;/span> &lt;span class="n">np&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">array&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">x&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>&lt;img src="https://nansenli.com/img/rescued/8ba75ea46168.png"
loading="lazy"
alt="image.png"
>&lt;/p>
&lt;p>Sections 13-14 mainly discuss One-Hot encoding. After the softmax function provides a sequence of probability values, how do we determine the classification? For example, a sequence where the highest probability is 1 and others are 0 is called One-Hot encoding. This type of encoding has already determined the classification.&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/d0050afa0519.png"
loading="lazy"
alt="Cross-entropy"
>&lt;/p>
&lt;p>Sections 15-16 cover cross-entropy. Softmax can calculate a probability sequence, and OneHot is a determined classification. So how do we calculate the distance from a probability sequence to a specific classification? We use cross-entropy to measure this distance.&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/7a800c91612a.png"
loading="lazy"
alt="image.png"
>&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/a1e862b4bd40.png"
loading="lazy"
alt="image.png"
>&lt;/p>
&lt;p>Sections 17-20 explain how to use this classifier. Section 18 specifically discusses why special initial data is needed.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;span class="lnt">6
&lt;/span>&lt;span class="lnt">7
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">sum = 1000000000
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">for i in range(1000000):
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> sum += 0.000001
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sum -= 1000000000
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">print(sum)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>The result of running this code is not 1. If we change sum to a very small number, like 1, instead of 1000000000, we find that the error becomes smaller. Based on this reason, we want our initial data to always have a mean of 0 and consistent variance in all directions. For example, for a grayscale image with pixel values from 0-255, we need to subtract 128 and then divide by 128, so that each number is between -1 and 1. Such initial data is more suitable for training.&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/34f543ea3c1e.png"
loading="lazy"
alt="image.png"
>&lt;/p>
&lt;p>This way, we can proceed with training. Reviewing the video content: xi is the training data matrix, w is a random weight matrix. For performance reasons, random values are taken from a normal distribution with an axis of 0 and very small variance. Then we calculate the probability sequence and the distance to the target. Then we compute the average distance to all targets. Our goal is to make this distance smaller, so we optimize the weight matrix along the direction of gradient descent while optimizing the intercept b. We repeat this process continuously until we reach a local optimum.&lt;/p>
&lt;ul>
&lt;li>Installing Docker&lt;/li>
&lt;/ul>
&lt;p>&lt;a class="link" href="https://www.docker-cn.com/community-edition#/download" target="_blank" rel="noopener"
>https://www.docker-cn.com/community-edition#/download&lt;/a>&lt;/p>
&lt;p>Configure the official Chinese mirror.&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/1a9fbd212a79.png"
loading="lazy"
alt="image.png"
>&lt;/p>
&lt;ul>
&lt;li>Installing Jupyter Notebook&lt;/li>
&lt;/ul>
&lt;p>&lt;code>$ pip3 install jupyter&lt;/code>
&lt;code>$ jupyter notebook&lt;/code>
You can now use the jupyter notebook command to open a Jupyter editor.&lt;/p>
&lt;ul>
&lt;li>Setting up TensorFlow environment&lt;/li>
&lt;/ul>
&lt;p>&lt;code>$ docker run -it -p 8888:8888 tensorflow/tensorflow&lt;/code>
Running this command will automatically download the TensorFlow image, provided that the repository mirror is set to a Chinese mirror; otherwise, the download will be very slow. After running the command, you&amp;rsquo;ll be prompted to open a webpage. When you open this URL, you&amp;rsquo;ll see the TensorFlow Jupyter editing environment, assuming Jupyter Notebook is installed correctly.&lt;/p>
&lt;ul>
&lt;li>Mounting Docker&amp;rsquo;s file directory
We need to import the official assignments. Close the container, reopen it, and use &lt;code>-v host_directory:container_directory&lt;/code> for mounting.
&lt;code>docker run -v /Users/hahaha/tensorflow/:/notebooks -it -p 8888:8888 tensorflow/tensorflow&lt;/code>&lt;/li>
&lt;/ul>
&lt;p>Where /Users/hahaha/tensorflow/ is a folder on my Mac, and notebooks is the default Jupyter editing directory in TensorFlow.&lt;/p>
&lt;p>Paste the first assignment file, 1_notmnist.ipynb, into the mounted directory on the host. This file can be found here: &lt;a class="link" href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/udacity/1_notmnist.ipynb" target="_blank" rel="noopener"
>1_notmnist.ipynb&lt;/a>&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/7faf184ae282.png"
loading="lazy"
alt="Assignment One Content"
>&lt;/p>
&lt;h2 id="assignment-code-segment-one">Assignment Code Segment One
&lt;/h2>&lt;p>First, run the import statements in the first code segment. There should be no errors. If you see red error output, it means these imports were not successful.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;span class="lnt">21
&lt;/span>&lt;span class="lnt">22
&lt;/span>&lt;span class="lnt">23
&lt;/span>&lt;span class="lnt">24
&lt;/span>&lt;span class="lnt">25
&lt;/span>&lt;span class="lnt">26
&lt;/span>&lt;span class="lnt">27
&lt;/span>&lt;span class="lnt">28
&lt;/span>&lt;span class="lnt">29
&lt;/span>&lt;span class="lnt">30
&lt;/span>&lt;span class="lnt">31
&lt;/span>&lt;span class="lnt">32
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-gdscript3" data-lang="gdscript3">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># These are all the modules we&amp;#39;ll be using later. Make sure you can import them&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># before proceeding further.&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">from&lt;/span> &lt;span class="n">__future__&lt;/span> &lt;span class="n">import&lt;/span> &lt;span class="n">print_function&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># print function&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">import&lt;/span> &lt;span class="n">matplotlib&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">pyplot&lt;/span> &lt;span class="n">as&lt;/span> &lt;span class="n">plt&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># plotting tool&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">import&lt;/span> &lt;span class="n">numpy&lt;/span> &lt;span class="n">as&lt;/span> &lt;span class="n">np&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># matrix calculations&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">import&lt;/span> &lt;span class="n">os&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># file paths&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">import&lt;/span> &lt;span class="n">sys&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># file output&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">import&lt;/span> &lt;span class="n">tarfile&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># decompression&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">from&lt;/span> &lt;span class="n">IPython&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">display&lt;/span> &lt;span class="n">import&lt;/span> &lt;span class="n">display&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="ne">Image&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># display images&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">from&lt;/span> &lt;span class="n">scipy&lt;/span> &lt;span class="n">import&lt;/span> &lt;span class="n">ndimage&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># image processing&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">from&lt;/span> &lt;span class="n">sklearn&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">linear_model&lt;/span> &lt;span class="n">import&lt;/span> &lt;span class="n">LogisticRegression&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># logistic regression module for linear models&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">from&lt;/span> &lt;span class="n">six&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">moves&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">urllib&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">request&lt;/span> &lt;span class="n">import&lt;/span> &lt;span class="n">urlretrieve&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># url handling&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">from&lt;/span> &lt;span class="n">six&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">moves&lt;/span> &lt;span class="n">import&lt;/span> &lt;span class="n">cPickle&lt;/span> &lt;span class="n">as&lt;/span> &lt;span class="n">pickle&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># data processing&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Config the matplotlib backend as plotting inline in IPython&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">%&lt;/span>&lt;span class="n">matplotlib&lt;/span> &lt;span class="n">inline&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># matplotlib is the most famous Python chart plotting extension library,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># it supports outputting various formats of graphical images, and can use various GUI interface libraries to display charts interactively.&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Using the %matplotlib command can embed matplotlib charts directly into the Notebook,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># or display charts using a specified interface library, it has a parameter specifying how matplotlib charts are displayed.&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># inline indicates embedding charts in the Notebook.&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="assignment-code-segment-two">Assignment Code Segment Two
&lt;/h2>&lt;p>Next is the second code segment, which will download letter sets for training and testing, approximately 300MB in size. After successful download, you can see these two files in the mounted directory.&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/66a1b044606a.png"
loading="lazy"
alt="Assignment"
>&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;span class="lnt">21
&lt;/span>&lt;span class="lnt">22
&lt;/span>&lt;span class="lnt">23
&lt;/span>&lt;span class="lnt">24
&lt;/span>&lt;span class="lnt">25
&lt;/span>&lt;span class="lnt">26
&lt;/span>&lt;span class="lnt">27
&lt;/span>&lt;span class="lnt">28
&lt;/span>&lt;span class="lnt">29
&lt;/span>&lt;span class="lnt">30
&lt;/span>&lt;span class="lnt">31
&lt;/span>&lt;span class="lnt">32
&lt;/span>&lt;span class="lnt">33
&lt;/span>&lt;span class="lnt">34
&lt;/span>&lt;span class="lnt">35
&lt;/span>&lt;span class="lnt">36
&lt;/span>&lt;span class="lnt">37
&lt;/span>&lt;span class="lnt">38
&lt;/span>&lt;span class="lnt">39
&lt;/span>&lt;span class="lnt">40
&lt;/span>&lt;span class="lnt">41
&lt;/span>&lt;span class="lnt">42
&lt;/span>&lt;span class="lnt">43
&lt;/span>&lt;span class="lnt">44
&lt;/span>&lt;span class="lnt">45
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-gdscript3" data-lang="gdscript3">&lt;span class="line">&lt;span class="cl">&lt;span class="n">url&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s1">&amp;#39;https://commondatastorage.googleapis.com/books1000/&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">last_percent_reported&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">None&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">data_root&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s1">&amp;#39;.&amp;#39;&lt;/span> &lt;span class="c1"># Change me to store data elsewhere&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">def&lt;/span> &lt;span class="nf">download_progress_hook&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">count&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">blockSize&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">totalSize&lt;/span>&lt;span class="p">):&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="s2">&amp;#34;&amp;#34;&amp;#34;A hook to report the progress of a download. This is mostly intended for users with
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> slow internet connections. Reports every 5&lt;/span>&lt;span class="si">% c&lt;/span>&lt;span class="s2">hange in download progress.
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Hook function to display download progress in real-time&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">global&lt;/span> &lt;span class="n">last_percent_reported&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">percent&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="ne">int&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">count&lt;/span> &lt;span class="o">*&lt;/span> &lt;span class="n">blockSize&lt;/span> &lt;span class="o">*&lt;/span> &lt;span class="mi">100&lt;/span> &lt;span class="o">/&lt;/span> &lt;span class="n">totalSize&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">if&lt;/span> &lt;span class="n">last_percent_reported&lt;/span> &lt;span class="o">!=&lt;/span> &lt;span class="n">percent&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">if&lt;/span> &lt;span class="n">percent&lt;/span> &lt;span class="o">%&lt;/span> &lt;span class="mi">5&lt;/span> &lt;span class="o">==&lt;/span> &lt;span class="mi">0&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">sys&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">stdout&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">write&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="si">%s%%&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="o">%&lt;/span> &lt;span class="n">percent&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">sys&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">stdout&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">flush&lt;/span>&lt;span class="p">()&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">else&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">sys&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">stdout&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">write&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s2">&amp;#34;.&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">sys&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">stdout&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">flush&lt;/span>&lt;span class="p">()&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">last_percent_reported&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">percent&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">def&lt;/span> &lt;span class="nf">maybe_download&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">filename&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">expected_bytes&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">force&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="n">False&lt;/span>&lt;span class="p">):&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="s2">&amp;#34;&amp;#34;&amp;#34;Download a file if not present, and make sure it&amp;#39;s the right size.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">dest_filename&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">os&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">path&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">join&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">data_root&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">filename&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># data_root is the current directory, add the filename to it, set as the location to save the file&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">if&lt;/span> &lt;span class="n">force&lt;/span> &lt;span class="ow">or&lt;/span> &lt;span class="ow">not&lt;/span> &lt;span class="n">os&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">path&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">exists&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">dest_filename&lt;/span>&lt;span class="p">):&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># force is to force download, ignoring already downloaded files&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">print&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;Attempting to download:&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">filename&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">filename&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">_&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">urlretrieve&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">url&lt;/span> &lt;span class="o">+&lt;/span> &lt;span class="n">filename&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">dest_filename&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">reporthook&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="n">download_progress_hook&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Use urlretrieve to download the file, with the hook attached&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">print&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;&lt;/span>&lt;span class="se">\n&lt;/span>&lt;span class="s1">Download Complete!&amp;#39;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">statinfo&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">os&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">stat&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="n">dest_filename&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Get information about the downloaded file&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">if&lt;/span> &lt;span class="n">statinfo&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">st_size&lt;/span> &lt;span class="o">==&lt;/span> &lt;span class="n">expected_bytes&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Correct size&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">print&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;Found and verified&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">dest_filename&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">else&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Wrong size, prompt user to use a browser to download&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">raise&lt;/span> &lt;span class="n">Exception&lt;/span>&lt;span class="p">(&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="s1">&amp;#39;Failed to verify &amp;#39;&lt;/span> &lt;span class="o">+&lt;/span> &lt;span class="n">dest_filename&lt;/span> &lt;span class="o">+&lt;/span> &lt;span class="s1">&amp;#39;. Can you get to it with a browser?&amp;#39;&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">return&lt;/span> &lt;span class="n">dest_filename&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">train_filename&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">maybe_download&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;notMNIST_large.tar.gz&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="mi">247336696&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">test_filename&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="n">maybe_download&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="s1">&amp;#39;notMNIST_small.tar.gz&amp;#39;&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="mi">8458043&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="assignment-code-segment-three">Assignment Code Segment Three
&lt;/h2>&lt;p>Extracting use cases&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;span class="lnt">21
&lt;/span>&lt;span class="lnt">22
&lt;/span>&lt;span class="lnt">23
&lt;/span>&lt;span class="lnt">24
&lt;/span>&lt;span class="lnt">25
&lt;/span>&lt;span class="lnt">26
&lt;/span>&lt;span class="lnt">27
&lt;/span>&lt;span class="lnt">28
&lt;/span>&lt;span class="lnt">29
&lt;/span>&lt;span class="lnt">30
&lt;/span>&lt;span class="lnt">31
&lt;/span>&lt;span class="lnt">32
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">num_classes = 10
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Total number of digits
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">np.random.seed(133)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Initialize random seed
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">def maybe_extract(filename, force=False):
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Assuming already extracted
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> root = os.path.splitext(os.path.splitext(filename)[0])[0] # remove .tar.gz
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># splitext(filename)[0] removes one suffix, used twice to remove both suffixes, i.e., remove the .tar.gz suffix
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> if os.path.isdir(root) and not force:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> # You may override by setting force=True.
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># If already extracted, don&amp;#39;t extract again
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(&amp;#39;%s already present - Skipping extraction of %s.&amp;#39; % (root, filename))
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> else:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(&amp;#39;Extracting data for %s. This may take a while. Please wait.&amp;#39; % root)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> tar = tarfile.open(filename)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> sys.stdout.flush()
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> tar.extractall(data_root)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> tar.close()
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Extract to the current directory
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> data_folders = [
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> os.path.join(root, d) for d in sorted(os.listdir(root))
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> if os.path.isdir(os.path.join(root, d))]
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> if len(data_folders) != num_classes:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> raise Exception(
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &amp;#39;Expected %d folders, one per class. Found %d instead.&amp;#39; % (
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> num_classes, len(data_folders)))
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> print(data_folders)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Check if the number of extracted directories matches expectations, and print the extracted directories
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> return data_folders
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">train_folders = maybe_extract(train_filename)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">test_folders = maybe_extract(test_filename)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;h2 id="question-one">Question One
&lt;/h2>&lt;p>Write code to display information about the extracted file contents&lt;/p>
&lt;ul>
&lt;li>Reference answer&lt;/li>
&lt;/ul>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;span class="lnt">21
&lt;/span>&lt;span class="lnt">22
&lt;/span>&lt;span class="lnt">23
&lt;/span>&lt;span class="lnt">24
&lt;/span>&lt;span class="lnt">25
&lt;/span>&lt;span class="lnt">26
&lt;/span>&lt;span class="lnt">27
&lt;/span>&lt;span class="lnt">28
&lt;/span>&lt;span class="lnt">29
&lt;/span>&lt;span class="lnt">30
&lt;/span>&lt;span class="lnt">31
&lt;/span>&lt;span class="lnt">32
&lt;/span>&lt;span class="lnt">33
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">import random
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">import matplotlib.image as mpimg
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">def plot_samples(data_folders, sample_size, title=None):
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> fig = plt.figure()
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Create empty figure
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> if title: fig.suptitle(title, fontsize=16, fontweight=&amp;#39;bold&amp;#39;)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Add title
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> for folder in data_folders:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Loop through each letter
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> image_files = os.listdir(folder)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> image_sample = random.sample(image_files, sample_size)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Randomly select a certain number of images from that letter
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> for image in image_sample:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> image_file = os.path.join(folder, image)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> ax = fig.add_subplot(len(data_folders), sample_size, sample_size * data_folders.index(folder) +
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> image_sample.index(image) + 1)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Create a subplot
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> image = mpimg.imread(image_file)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Load subplot image
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> ax.imshow(image)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Display subplot image
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> ax.set_axis_off()
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Turn off subplot coordinate lines
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> fig.set_size_inches(18.5, 10.5)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"># Set the display size of the image
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> plt.show()
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">plot_samples(train_folders, 20, &amp;#39;Train&amp;#39;)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">plot_samples(test_folders, 20, &amp;#39;Test&amp;#39;)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>Running results:&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/94887f5b6eab.png"
loading="lazy"
alt="Training.png"
>&lt;/p>
&lt;p>&lt;img src="https://nansenli.com/img/rescued/3930a2cfe6b0.png"
loading="lazy"
alt="Testing.png"
>&lt;/p>
&lt;p>As we can see, some of the training data has issues.&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Assignment Code Segment Four
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">After this, we need to normalize the data, which means transforming each image pixel from 0~255 to -1.0~1.0, and persisting it to a file.
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>image_size = 28 # Pixel width and height.
pixel_depth = 255.0 # Number of levels per pixel.&lt;/p>
&lt;h1 id="image-width-height-and-pixel-depth">Image width, height and pixel depth
&lt;/h1>&lt;p>def load_letter(folder, min_num_images):
&amp;ldquo;&amp;ldquo;&amp;ldquo;Load the data for a single letter label.&amp;rdquo;&amp;rdquo;&amp;rdquo;&lt;/p>
&lt;h1 id="process-files-in-a-folder-belonging-to-one-letter">Process files in a folder belonging to one letter
&lt;/h1>&lt;p>image_files = os.listdir(folder)&lt;/p>
&lt;h1 id="list-all-files-in-that-directory">List all files in that directory
&lt;/h1>&lt;p>dataset = np.ndarray(shape=(len(image_files), image_size, image_size),
dtype=np.float32)&lt;/p>
&lt;h1 id="create-a-dataset-with-length-equal-to-number-of-files-width-and-height-of-28">Create a dataset with length equal to number of files, width and height of 28
&lt;/h1>&lt;p>print(folder)&lt;/p>
&lt;h1 id="print-directory">Print directory
&lt;/h1>&lt;p>num_images = 0&lt;/p>
&lt;h1 id="initialize-num_images">Initialize num_images
&lt;/h1>&lt;p>for image in image_files:&lt;/p>
&lt;h1 id="process-each-file">Process each file
&lt;/h1>&lt;pre>&lt;code>image_file = os.path.join(folder, image)
&lt;/code>&lt;/pre>
&lt;h1 id="get-complete-file-path">Get complete file path
&lt;/h1>&lt;pre>&lt;code>try:
image_data = (ndimage.imread(image_file).astype(float) -
pixel_depth / 2) / pixel_depth
&lt;/code>&lt;/pre>
&lt;h1 id="read-in-the-image-and-normalize-it">Read in the image and normalize it
&lt;/h1>&lt;pre>&lt;code> if image_data.shape != (image_size, image_size):
&lt;/code>&lt;/pre>
&lt;h1 id="check-image-width-and-height">Check image width and height
&lt;/h1>&lt;pre>&lt;code> raise Exception('Unexpected image shape: %s' % str(image_data.shape))
dataset[num_images, :, :] = image_data
&lt;/code>&lt;/pre>
&lt;h1 id="read-into-the-dataset">Read into the dataset
&lt;/h1>&lt;pre>&lt;code> num_images = num_images + 1
&lt;/code>&lt;/pre>
&lt;h1 id="increment-image-number">Increment image number
&lt;/h1>&lt;pre>&lt;code>except IOError as e:
&lt;/code>&lt;/pre>
&lt;h1 id="if-file-cant-be-read-skip-it">If file can&amp;rsquo;t be read, skip it
&lt;/h1>&lt;pre>&lt;code> print('Could not read:', image_file, ':', e, '- it\'s ok, skipping.')
&lt;/code>&lt;/pre>
&lt;p>dataset = dataset[0:num_images, :, :]&lt;/p>
&lt;h1 id="if-fewer-files-were-read-than-the-minimum-required">If fewer files were read than the minimum required
&lt;/h1>&lt;p>if num_images &amp;lt; min_num_images:
raise Exception(&amp;lsquo;Many fewer images than expected: %d &amp;lt; %d&amp;rsquo; %
(num_images, min_num_images))&lt;/p>
&lt;h1 id="display-number-of-missing-files">Display number of missing files
&lt;/h1>&lt;p>print(&amp;lsquo;Full dataset tensor:&amp;rsquo;, dataset.shape)&lt;/p>
&lt;h1 id="display-file-count-image-width-and-height">Display file count, image width and height
&lt;/h1>&lt;p>print(&amp;lsquo;Mean:&amp;rsquo;, np.mean(dataset))&lt;/p>
&lt;h1 id="mean-value">Mean value
&lt;/h1>&lt;p>print(&amp;lsquo;Standard deviation:&amp;rsquo;, np.std(dataset))&lt;/p>
&lt;h1 id="standard-deviation">Standard deviation
&lt;/h1>&lt;p>return dataset&lt;/p>
&lt;p>def maybe_pickle(data_folders, min_num_images_per_class, force=False):
dataset_names = []
for folder in data_folders:&lt;/p>
&lt;h1 id="process-each-letter-folder">Process each letter folder
&lt;/h1>&lt;pre>&lt;code>set_filename = folder + '.pickle'
&lt;/code>&lt;/pre>
&lt;h1 id="set-output-file">Set output file
&lt;/h1>&lt;pre>&lt;code>dataset_names.append(set_filename)
&lt;/code>&lt;/pre>
&lt;h1 id="set-processed-folders">Set processed folders
&lt;/h1>&lt;pre>&lt;code>if os.path.exists(set_filename) and not force:
# You may override by setting force=True.
&lt;/code>&lt;/pre>
&lt;h1 id="check-if-processed-file-already-exists">Check if processed file already exists
&lt;/h1>&lt;pre>&lt;code> print('%s already present - Skipping pickling.' % set_filename)
else:
print('Pickling %s.' % set_filename)
dataset = load_letter(folder, min_num_images_per_class)
&lt;/code>&lt;/pre>
&lt;h1 id="normalize-all-images-in-this-folder">Normalize all images in this folder
&lt;/h1>&lt;pre>&lt;code> try:
with open(set_filename, 'wb') as f:
pickle.dump(dataset, f, pickle.HIGHEST_PROTOCOL)
&lt;/code>&lt;/pre>
&lt;h1 id="persist-data-save-to-disk-instead-of-keeping-in-memory">Persist data, save to disk instead of keeping in memory
&lt;/h1>&lt;pre>&lt;code> except Exception as e:
print('Unable to save data to', set_filename, ':', e)
&lt;/code>&lt;/pre>
&lt;p>return dataset_names&lt;/p>
&lt;p>train_datasets = maybe_pickle(train_folders, 45000)
test_datasets = maybe_pickle(test_folders, 1800)&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Question Two
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Display processed images
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">- Reference answer
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>def plot_samples_2(data_folders, sample_size, title=None):
fig = plt.figure()&lt;/p>
&lt;h1 id="create-empty-figure">Create empty figure
&lt;/h1>&lt;pre>&lt;code>if title: fig.suptitle(title, fontsize=16, fontweight='bold')
&lt;/code>&lt;/pre>
&lt;h1 id="add-title">Add title
&lt;/h1>&lt;pre>&lt;code>for folder in data_folders:
&lt;/code>&lt;/pre>
&lt;h1 id="loop-through-each-letter">Loop through each letter
&lt;/h1>&lt;pre>&lt;code> with open(folder, 'rb') as pk_f:
data = pickle.load(pk_f)
for index, image in enumerate(data):
if index &amp;lt; sample_size :
&lt;/code>&lt;/pre>
&lt;h1 id="randomly-select-a-certain-number-of-images-from-that-letter">Randomly select a certain number of images from that letter
&lt;/h1>&lt;pre>&lt;code> ax = fig.add_subplot(len(data_folders), sample_size, sample_size * data_folders.index(folder) +
index + 1)
&lt;/code>&lt;/pre>
&lt;h1 id="load-subplot-image">Load subplot image
&lt;/h1>&lt;pre>&lt;code> ax.imshow(image)
&lt;/code>&lt;/pre>
&lt;h1 id="display-subplot-image">Display subplot image
&lt;/h1>&lt;pre>&lt;code> ax.set_axis_off()
&lt;/code>&lt;/pre>
&lt;h1 id="turn-off-subplot-coordinate-lines">Turn off subplot coordinate lines
&lt;/h1>&lt;pre>&lt;code>fig.set_size_inches(18.5, 10.5)
&lt;/code>&lt;/pre>
&lt;h1 id="set-the-display-size-of-the-image">Set the display size of the image
&lt;/h1>&lt;pre>&lt;code>plt.show()
&lt;/code>&lt;/pre>
&lt;p>plot_samples_2(train_datasets, 20, &amp;lsquo;Train&amp;rsquo;)
plot_samples_2(test_datasets, 20, &amp;lsquo;Test&amp;rsquo;)&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">![image.png](/img/rescued/c09ea5ab69e9.png)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">![image.png](/img/rescued/d4363496653b.png)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Question Three
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Check if the number of files under each letter is similar.
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">- Reference answer
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>file_path = &amp;rsquo;notMNIST_large/{0}.pickle&amp;rsquo;
for ele in &amp;lsquo;ABCDEFJHIJ&amp;rsquo;:
with open(file_path.format(ele), &amp;lsquo;rb&amp;rsquo;) as pk_f:&lt;/p>
&lt;h1 id="loop-through-each-directory">Loop through each directory
&lt;/h1>&lt;pre>&lt;code> dat = pickle.load(pk_f)
&lt;/code>&lt;/pre>
&lt;h1 id="load-the-persisted-file-in-this-directory">Load the persisted file in this directory
&lt;/h1>&lt;pre>&lt;code>print('number of pictures in {}.pickle = '.format(ele), dat.shape[0])
&lt;/code>&lt;/pre>
&lt;h1 id="print-relevant-information">Print relevant information
&lt;/h1>&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-gdscript3" data-lang="gdscript3">&lt;span class="line">&lt;span class="cl">&lt;span class="n">Results&lt;/span> &lt;span class="n">show&lt;/span> &lt;span class="n">that&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">numbers&lt;/span> &lt;span class="n">are&lt;/span> &lt;span class="n">basically&lt;/span> &lt;span class="n">consistent&lt;/span>&lt;span class="o">.&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">!&lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="n">Question&lt;/span> &lt;span class="mi">3&lt;/span> &lt;span class="n">Result&lt;/span>&lt;span class="p">](&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="n">img&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="n">rescued&lt;/span>&lt;span class="o">/&lt;/span>&lt;span class="mi">31&lt;/span>&lt;span class="n">bcb36f8c25&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">png&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">## Code Segment—Data Splitting&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">Data&lt;/span> &lt;span class="n">cannot&lt;/span> &lt;span class="n">be&lt;/span> &lt;span class="n">loaded&lt;/span> &lt;span class="n">all&lt;/span> &lt;span class="n">at&lt;/span> &lt;span class="n">once&lt;/span> &lt;span class="n">into&lt;/span> &lt;span class="n">memory&lt;/span>&lt;span class="o">.&lt;/span> &lt;span class="n">This&lt;/span> &lt;span class="n">code&lt;/span> &lt;span class="n">segment&lt;/span> &lt;span class="n">splits&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">data&lt;/span>&lt;span class="o">.&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>def make_arrays(nb_rows, img_size):
if nb_rows:
dataset = np.ndarray((nb_rows, img_size, img_size), dtype=np.float32)&lt;/p>
&lt;h1 id="create-an-empty-set-data-type-is-a-matrix-with-rows-length-img_size-width-img_size-height-data-type-is-32-bit-float">Create an empty set, data type is a matrix with rows length, img_size width, img_size height, data type is 32-bit float
&lt;/h1>&lt;pre>&lt;code>labels = np.ndarray(nb_rows, dtype=np.int32)
&lt;/code>&lt;/pre>
&lt;h1 id="create-a-label-data-type-is-32-bit-integer-length-is-rows">Create a label, data type is 32-bit integer, length is rows
&lt;/h1>&lt;p>else:
dataset, labels = None, None
return dataset, labels&lt;/p>
&lt;h1 id="return-created-data-type">Return created data type
&lt;/h1>&lt;p>def merge_datasets(pickle_files, train_size, valid_size=0):
num_classes = len(pickle_files)&lt;/p>
&lt;h1 id="number-of-categories-to-process">Number of categories to process
&lt;/h1>&lt;p>valid_dataset, valid_labels = make_arrays(valid_size, image_size)&lt;/p>
&lt;h1 id="build-validation-dataset-length-is-validation-length">Build validation dataset, length is validation length
&lt;/h1>&lt;p>train_dataset, train_labels = make_arrays(train_size, image_size)&lt;/p>
&lt;h1 id="build-training-dataset-length-is-training-length">Build training dataset, length is training length
&lt;/h1>&lt;p>vsize_per_class = valid_size // num_classes
tsize_per_class = train_size // num_classes&lt;/p>
&lt;h1 id="calculate-average-length-for-each-category-with-given-training-and-validation-lengths">Calculate average length for each category with given training and validation lengths
&lt;/h1>&lt;p>start_v, start_t = 0, 0&lt;/p>
&lt;h1 id="initialize-indices-start_v-is-the-start-of-validation-data-start_t-is-the-start-of-training-data">Initialize indices, start_v is the start of validation data, start_t is the start of training data
&lt;/h1>&lt;p>end_v, end_t = vsize_per_class, tsize_per_class&lt;/p>
&lt;h1 id="initialize-indices-end_v-is-the-end-of-validation-data-end_t-is-the-end-of-training-data">Initialize indices, end_v is the end of validation data, end_t is the end of training data
&lt;/h1>&lt;p>end_l = vsize_per_class + tsize_per_class&lt;/p>
&lt;h1 id="initialize-indices-end_l-is-the-end-of-the-letter-set-equal-to-length-of-validation-data-for-each-category--length-of-training-data">Initialize indices, end_l is the end of the letter set, equal to length of validation data for each category + length of training data
&lt;/h1>&lt;p>for label, pickle_file in enumerate(pickle_files):&lt;/p>
&lt;h1 id="loop-through-each-pickle_file">Loop through each pickle_file
&lt;/h1>&lt;pre>&lt;code>try:
with open(pickle_file, 'rb') as f:
&lt;/code>&lt;/pre>
&lt;h1 id="open-this-persistence-file">Open this persistence file
&lt;/h1>&lt;pre>&lt;code> letter_set = pickle.load(f)
&lt;/code>&lt;/pre>
&lt;h1 id="load-dataset">Load dataset
&lt;/h1>&lt;pre>&lt;code> # let's shuffle the letters to have random validation and training set
np.random.shuffle(letter_set)
&lt;/code>&lt;/pre>
&lt;h1 id="shuffle-the-dataset">Shuffle the dataset
&lt;/h1>&lt;pre>&lt;code> if valid_dataset is not None:
&lt;/code>&lt;/pre>
&lt;h1 id="if-not-a-test-set-update-the-test-set-otherwise-valid_dataset-is-not-updated">If not a test set, update the test set, otherwise valid_dataset is not updated
&lt;/h1>&lt;pre>&lt;code> valid_letter = letter_set[:vsize_per_class, :, :]
&lt;/code>&lt;/pre>
&lt;h1 id="numpy-slicing-----httpbrieflyxme2015python-modulenumpy-array-split">numpy slicing &lt;a class="link" href="http://brieflyx.me/2015/python-module/numpy-array-split/" target="_blank" rel="noopener"
>http://brieflyx.me/2015/python-module/numpy-array-split/&lt;/a>
&lt;/h1>&lt;h1 id="select-data-of-valid-data-per-class-count-from-shuffled-data-for-processing-put-into-valid_letter">Select data of &amp;lsquo;valid data per class&amp;rsquo; count from shuffled data for processing, put into valid_letter
&lt;/h1>&lt;pre>&lt;code> valid_dataset[start_v:end_v, :, :] = valid_letter
&lt;/code>&lt;/pre>
&lt;h1 id="put-this-data-into-valid_dataset">Put this data into valid_dataset
&lt;/h1>&lt;pre>&lt;code> valid_labels[start_v:end_v] = label
&lt;/code>&lt;/pre>
&lt;h1 id="mark-label-should-be-one-of-09">Mark label should be one of 0~9
&lt;/h1>&lt;pre>&lt;code> start_v += vsize_per_class
end_v += vsize_per_class
&lt;/code>&lt;/pre>
&lt;h1 id="update-indices">Update indices
&lt;/h1>&lt;h1 id="at-the-end-of-the-loop-valid_dataset-should-be-data-with-total-length-valid_size-valid_labels-is-the-label-at-the-corresponding-position">At the end of the loop, valid_dataset should be data with total length valid_size, valid_labels is the label at the corresponding position
&lt;/h1>&lt;pre>&lt;code> train_letter = letter_set[vsize_per_class:end_l, :, :]
&lt;/code>&lt;/pre>
&lt;h1 id="other-random-elements-except-valid-part-length-is-end_l---vsize_per_class--tsize_per_class">Other random elements except valid part, length is end_l - vsize_per_class = tsize_per_class
&lt;/h1>&lt;pre>&lt;code> train_dataset[start_t:end_t, :, :] = train_letter
&lt;/code>&lt;/pre>
&lt;h1 id="at-the-end-of-the-loop-train_dataset-should-be-data-with-total-length-train_size">At the end of the loop, train_dataset should be data with total length train_size
&lt;/h1>&lt;h1 id="heading">
&lt;/h1>&lt;pre>&lt;code> train_labels[start_t:end_t] = label
start_t += tsize_per_class
end_t += tsize_per_class
&lt;/code>&lt;/pre>
&lt;h1 id="update-indices-1">Update indices
&lt;/h1>&lt;pre>&lt;code>except Exception as e:
print('Unable to process data from', pickle_file, ':', e)
raise
&lt;/code>&lt;/pre>
&lt;p>return valid_dataset, valid_labels, train_dataset, train_labels&lt;/p>
&lt;p>train_size = 200000
valid_size = 10000
test_size = 10000&lt;/p>
&lt;p>valid_dataset, valid_labels, train_dataset, train_labels = merge_datasets(
train_datasets, train_size, valid_size)
_, _, test_dataset, test_labels = merge_datasets(test_datasets, test_size)&lt;/p>
&lt;p>print(&amp;lsquo;Training:&amp;rsquo;, train_dataset.shape, train_labels.shape)
print(&amp;lsquo;Validation:&amp;rsquo;, valid_dataset.shape, valid_labels.shape)
print(&amp;lsquo;Testing:&amp;rsquo;, test_dataset.shape, test_labels.shape)&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Code Segment—Shuffling Data
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Introduction to the permutation function: http://www.jianshu.com/p/f0eb10acaa2d
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>def randomize(dataset, labels):&lt;/p>
&lt;h1 id="labelsshape0-is-the-length-of-labels">labels.shape[0] is the length of labels
&lt;/h1>&lt;p>permutation = np.random.permutation(labels.shape[0])&lt;/p>
&lt;h1 id="randomly-select-a-shuffled-set-of-this-many-numbers">Randomly select a shuffled set of this many numbers
&lt;/h1>&lt;p>print(labels.shape[0])
shuffled_dataset = dataset[permutation,:,:]&lt;/p>
&lt;h1 id="shuffle-data">Shuffle data
&lt;/h1>&lt;p>shuffled_labels = labels[permutation]&lt;/p>
&lt;h1 id="shuffle-labels">Shuffle labels
&lt;/h1>&lt;p>return shuffled_dataset, shuffled_labels
train_dataset, train_labels = randomize(train_dataset, train_labels)
test_dataset, test_labels = randomize(test_dataset, test_labels)
valid_dataset, valid_labels = randomize(valid_dataset, valid_labels)&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Question Four
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Verify if the shuffled data is correct
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">- Reference answer
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>import random
def plot_sample_3(dataset, labels, title):
fig = plt.figure()
plt.suptitle(title, fontsize=16, fontweight=&amp;lsquo;bold&amp;rsquo;)&lt;/p>
&lt;h1 id="set-title-style">Set title style
&lt;/h1>&lt;pre>&lt;code>items = random.sample(range(len(labels)), 200)
&lt;/code>&lt;/pre>
&lt;h1 id="shuffle-the-sequential-sequence-of-labels-length">Shuffle the sequential sequence of labels length
&lt;/h1>&lt;pre>&lt;code>for i, item in enumerate(items):
&lt;/code>&lt;/pre>
&lt;h1 id="randomly-pick-one">Randomly pick one
&lt;/h1>&lt;pre>&lt;code> plt.subplot(10, 20, i + 1)
&lt;/code>&lt;/pre>
&lt;h1 id="draw-subplot">Draw subplot
&lt;/h1>&lt;pre>&lt;code> plt.axis('off')
&lt;/code>&lt;/pre>
&lt;h1 id="turn-off-coordinate-axes">Turn off coordinate axes
&lt;/h1>&lt;pre>&lt;code> plt.title(chr(ord('A') + labels[item]))
&lt;/code>&lt;/pre>
&lt;h1 id="add-title-1">Add title
&lt;/h1>&lt;pre>&lt;code> plt.imshow(dataset[item])
&lt;/code>&lt;/pre>
&lt;h1 id="display-subplot-at-corresponding-position">Display subplot at corresponding position
&lt;/h1>&lt;pre>&lt;code>fig.set_size_inches(18.5, 10.5)
plt.show()
&lt;/code>&lt;/pre>
&lt;h1 id="display-image">Display image
&lt;/h1>&lt;p>plot_sample_3(train_dataset, train_labels, &amp;rsquo;train dataset suffled&amp;rsquo;)
plot_sample_3(valid_dataset, valid_labels, &amp;lsquo;valid dataset suffled&amp;rsquo;)
plot_sample_3(test_dataset, test_labels, &amp;rsquo;test dataset suffled&amp;rsquo;)&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;span class="lnt">6
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">![Question 4](/img/rescued/2041fe44668c.png)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Similar two figures omitted
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Code Segment—Saving Data
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>pickle_file = os.path.join(data_root, &amp;rsquo;notMNIST.pickle&amp;rsquo;)&lt;/p>
&lt;h1 id="output-file-path">Output file path
&lt;/h1>&lt;p>try:
f = open(pickle_file, &amp;lsquo;wb&amp;rsquo;)&lt;/p>
&lt;h1 id="open-this-file">Open this file
&lt;/h1>&lt;p>save = {
&amp;rsquo;train_dataset&amp;rsquo;: train_dataset,
&amp;rsquo;train_labels&amp;rsquo;: train_labels,
&amp;lsquo;valid_dataset&amp;rsquo;: valid_dataset,
&amp;lsquo;valid_labels&amp;rsquo;: valid_labels,
&amp;rsquo;test_dataset&amp;rsquo;: test_dataset,
&amp;rsquo;test_labels&amp;rsquo;: test_labels,
}&lt;/p>
&lt;h1 id="write-a-dictionary-string-ndarray">Write a dictionary string-ndarray
&lt;/h1>&lt;p>pickle.dump(save, f, pickle.HIGHEST_PROTOCOL)
f.close()
except Exception as e:
print(&amp;lsquo;Unable to save data to&amp;rsquo;, pickle_file, &amp;lsquo;:&amp;rsquo;, e)
raise&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Code Segment—Displaying Saved Data Size
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>statinfo = os.stat(pickle_file)
print(&amp;lsquo;Compressed pickle size:&amp;rsquo;, statinfo.st_size)&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-gdscript3" data-lang="gdscript3">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">## Question Five&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">Google&lt;/span> &lt;span class="n">translation&lt;/span> &lt;span class="n">of&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">question&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">By&lt;/span> &lt;span class="n">construction&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">this&lt;/span> &lt;span class="n">dataset&lt;/span> &lt;span class="n">may&lt;/span> &lt;span class="n">contain&lt;/span> &lt;span class="n">a&lt;/span> &lt;span class="n">lot&lt;/span> &lt;span class="n">of&lt;/span> &lt;span class="n">overlapping&lt;/span> &lt;span class="n">samples&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">including&lt;/span> &lt;span class="ow">in&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">validation&lt;/span> &lt;span class="ow">and&lt;/span> &lt;span class="n">test&lt;/span> &lt;span class="n">sets&lt;/span>&lt;span class="o">.&lt;/span> &lt;span class="n">Overlap&lt;/span> &lt;span class="n">between&lt;/span> &lt;span class="n">training&lt;/span> &lt;span class="ow">and&lt;/span> &lt;span class="n">test&lt;/span> &lt;span class="n">can&lt;/span> &lt;span class="n">skew&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">results&lt;/span> &lt;span class="k">if&lt;/span> &lt;span class="n">you&lt;/span> &lt;span class="n">expect&lt;/span> &lt;span class="n">to&lt;/span> &lt;span class="n">use&lt;/span> &lt;span class="n">your&lt;/span> &lt;span class="n">model&lt;/span> &lt;span class="ow">in&lt;/span> &lt;span class="n">an&lt;/span> &lt;span class="n">environment&lt;/span> &lt;span class="n">where&lt;/span> &lt;span class="n">there&lt;/span> &lt;span class="n">is&lt;/span> &lt;span class="n">never&lt;/span> &lt;span class="n">an&lt;/span> &lt;span class="n">overlap&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">but&lt;/span> &lt;span class="ow">in&lt;/span> &lt;span class="n">practice&lt;/span> &lt;span class="n">this&lt;/span> &lt;span class="n">doesn&lt;/span>&lt;span class="s1">&amp;#39;t usually matter. Measure how much overlap there is between training, validation, and test samples.&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">Optional&lt;/span> &lt;span class="n">question&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">What&lt;/span> &lt;span class="n">about&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">duplicates&lt;/span> &lt;span class="n">between&lt;/span> &lt;span class="n">datasets&lt;/span>&lt;span class="err">?&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="n">For&lt;/span> &lt;span class="n">instance&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">same&lt;/span> &lt;span class="n">letter&lt;/span> &lt;span class="n">images&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">Create&lt;/span> &lt;span class="n">a&lt;/span> &lt;span class="n">sanitized&lt;/span> &lt;span class="n">validation&lt;/span> &lt;span class="ow">and&lt;/span> &lt;span class="n">test&lt;/span> &lt;span class="n">set&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="ow">and&lt;/span> &lt;span class="n">compare&lt;/span> &lt;span class="n">your&lt;/span> &lt;span class="n">accuracy&lt;/span> &lt;span class="n">on&lt;/span> &lt;span class="n">those&lt;/span> &lt;span class="n">versus&lt;/span> &lt;span class="n">your&lt;/span> &lt;span class="n">accuracy&lt;/span> &lt;span class="n">on&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">original&lt;/span> &lt;span class="n">sets&lt;/span>&lt;span class="o">.&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">The&lt;/span> &lt;span class="n">basic&lt;/span> &lt;span class="n">idea&lt;/span> &lt;span class="n">is&lt;/span> &lt;span class="n">that&lt;/span> &lt;span class="n">training&lt;/span> &lt;span class="n">data&lt;/span> &lt;span class="n">should&lt;/span> &lt;span class="ow">not&lt;/span> &lt;span class="n">overlap&lt;/span> &lt;span class="n">with&lt;/span> &lt;span class="n">testing&lt;/span> &lt;span class="n">data&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="n">otherwise&lt;/span> &lt;span class="n">it&lt;/span> &lt;span class="n">leads&lt;/span> &lt;span class="n">to&lt;/span> &lt;span class="n">inaccurate&lt;/span> &lt;span class="n">accuracy&lt;/span>&lt;span class="o">.&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="ne">Reference&lt;/span> &lt;span class="n">code&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">-&lt;/span> &lt;span class="n">Just&lt;/span> &lt;span class="n">check&lt;/span> &lt;span class="n">the&lt;/span> &lt;span class="n">number&lt;/span> &lt;span class="n">of&lt;/span> &lt;span class="n">duplicate&lt;/span> &lt;span class="n">images&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>import hashlib&lt;/p>
&lt;p>pickle_file = os.path.join(&amp;rsquo;.&amp;rsquo;, &amp;rsquo;notMNIST.pickle&amp;rsquo;)
try:
with open(pickle_file, &amp;lsquo;rb&amp;rsquo;) as f:
data = pickle.load(f)
except Exception as e:
print(&amp;lsquo;Unable to open data from&amp;rsquo;, pickle_file, &amp;lsquo;:&amp;rsquo;, e)
raise&lt;/p>
&lt;h1 id="after-saving-the-data-if-the-kernel-crashed-you-can-read-directly-from-local-without-rerunning-previous-code">After saving the data, if the kernel crashed, you can read directly from local without rerunning previous code
&lt;/h1>&lt;h1 id="if-theres-an-error-you-can-search-for-the-exception-online">If there&amp;rsquo;s an error, you can search for the exception online
&lt;/h1>&lt;p>def calcOverlap(sourceSet, targetSet, description):
sourceSetMd5 = np.array([hashlib.md5(img).hexdigest() for img in sourceSet])&lt;/p>
&lt;h1 id="build-an-md5-table">Build an md5 table
&lt;/h1>&lt;pre>&lt;code>targetSetMd5 = np.array([hashlib.md5(img).hexdigest() for img in targetSet])
&lt;/code>&lt;/pre>
&lt;h1 id="build-an-md5-table-1">Build an md5 table
&lt;/h1>&lt;pre>&lt;code>overlap = np.intersect1d(sourceSetMd5, targetSetMd5, assume_unique=False)
&lt;/code>&lt;/pre>
&lt;h1 id="deduplicate">Deduplicate
&lt;/h1>&lt;pre>&lt;code>print(description)
print(&amp;quot;overlap&amp;quot;,overlap.shape[0], &amp;quot;from&amp;quot;,sourceSetMd5.shape[0],&amp;quot;to&amp;quot;, targetSetMd5.shape[0])
print(&amp;quot;rate&amp;quot;,overlap.shape[0]*100.0/sourceSetMd5.shape[0],&amp;quot;% and&amp;quot;, overlap.shape[0]*100.0/targetSetMd5.shape[0],&amp;quot;%&amp;quot;)
&lt;/code>&lt;/pre>
&lt;h1 id="print-overlap-count">Print overlap count
&lt;/h1>&lt;p>calcOverlap(data[&amp;rsquo;train_dataset&amp;rsquo;], data[&amp;lsquo;valid_dataset&amp;rsquo;], &amp;ldquo;train_dataset &amp;amp; valid_dataset&amp;rdquo;)
calcOverlap(data[&amp;rsquo;train_dataset&amp;rsquo;], data[&amp;rsquo;test_dataset&amp;rsquo;], &amp;ldquo;train_dataset &amp;amp; test_dataset&amp;rdquo;)
calcOverlap(data[&amp;rsquo;test_dataset&amp;rsquo;], data[&amp;lsquo;valid_dataset&amp;rsquo;], &amp;ldquo;test_dataset &amp;amp; valid_dataset&amp;rdquo;)&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">![Running result](/img/rescued/fc3c3db78e52.png)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">- Remove duplicate image resources
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">To be updated
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Question Six
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Use logistic regression to train the model and test it
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">- Reference code
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>import random
def disp_sample_dataset(dataset, labels,trueLabels, title=None):&lt;/p>
&lt;h1 id="display-training-results">Display training results
&lt;/h1>&lt;pre>&lt;code>fig = plt.figure()
if title: fig.suptitle(title, fontsize=16, fontweight='bold')
&lt;/code>&lt;/pre>
&lt;h1 id="set-title-style-1">Set title style
&lt;/h1>&lt;pre>&lt;code>items = random.sample(range(len(labels)), 200)
&lt;/code>&lt;/pre>
&lt;h1 id="randomly-select-a-series-of-images">Randomly select a series of images
&lt;/h1>&lt;pre>&lt;code>for i, item in enumerate(items):
plt.subplot(10, 20, i + 1)
&lt;/code>&lt;/pre>
&lt;h1 id="set-a-subplot">Set a subplot
&lt;/h1>&lt;pre>&lt;code> plt.axis('off')
&lt;/code>&lt;/pre>
&lt;h1 id="turn-off-coordinate-lines">Turn off coordinate lines
&lt;/h1>&lt;pre>&lt;code> lab = str(chr(ord('A') + labels[item]))
trueLab = str(chr(ord('A') + trueLabels[item]))
if lab == trueLab:
plt.title( lab )
else:
plt.title(lab + &amp;quot; but &amp;quot; + trueLab)
&lt;/code>&lt;/pre>
&lt;h1 id="add-title-2">Add title
&lt;/h1>&lt;pre>&lt;code> plt.imshow(dataset[item])
&lt;/code>&lt;/pre>
&lt;h1 id="display-this-image">Display this image
&lt;/h1>&lt;pre>&lt;code>fig.set_size_inches(18.5, 10.5)
plt.show()
&lt;/code>&lt;/pre>
&lt;p>def train_and_predict(train_dataset, train_labels, test_dataset, test_labels ,sample_size):
regr = LogisticRegression()&lt;/p>
&lt;h1 id="generate-trainer">Generate trainer
&lt;/h1>&lt;pre>&lt;code>X_train = train_dataset[:sample_size].reshape(sample_size, 784)
&lt;/code>&lt;/pre>
&lt;h1 id="choose-amount-of-data-to-train-based-on-sample_size">Choose amount of data to train based on sample_size
&lt;/h1>&lt;h1 id="compress-2d-vector-to-1d-vector">Compress 2D vector to 1D vector
&lt;/h1>&lt;pre>&lt;code>y_train = train_labels[:sample_size]
&lt;/code>&lt;/pre>
&lt;h1 id="extract-training-data">Extract training data
&lt;/h1>&lt;pre>&lt;code>regr.fit(X_train, y_train)
&lt;/code>&lt;/pre>
&lt;h1 id="train-data">Train data
&lt;/h1>&lt;pre>&lt;code>X_test = test_dataset.reshape(test_dataset.shape[0], 28 * 28)
&lt;/code>&lt;/pre>
&lt;h1 id="compress-test-data-to-1d-vector">Compress test data to 1D vector
&lt;/h1>&lt;pre>&lt;code>y_test = test_labels
&lt;/code>&lt;/pre>
&lt;h1 id="true-labels-corresponding-to-test-data">True labels corresponding to test data
&lt;/h1>&lt;pre>&lt;code>pred_labels = regr.predict(X_test)
&lt;/code>&lt;/pre>
&lt;h1 id="generate-prediction-data">Generate prediction data
&lt;/h1>&lt;pre>&lt;code>print('Accuracy:', regr.score(X_test, y_test), 'when sample_size=', sample_size)
disp_sample_dataset(test_dataset, pred_labels, test_labels, 'sample_size=' + str(sample_size))
&lt;/code>&lt;/pre>
&lt;p>train_and_predict(data[&amp;rsquo;train_dataset&amp;rsquo;],data[&amp;rsquo;train_labels&amp;rsquo;],data[&amp;rsquo;test_dataset&amp;rsquo;],data[&amp;rsquo;test_labels&amp;rsquo;], 1000)&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;span class="lnt">21
&lt;/span>&lt;span class="lnt">22
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-fallback" data-lang="fallback">&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">![image.png](/img/rescued/abdb5d761390.png)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Model Performance
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Sections 22~27 discuss model performance-related knowledge. We usually hope that the model&amp;#39;s performance can reach 100%, which is obviously impossible. Also, in order to improve the accuracy of the training set, the model may overfit. At this point, we should follow two points:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">- Don&amp;#39;t use all training data at once, but use it in blocks, train a portion each time
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">- When model parameter changes cause 30 or more cases to change from error to correct, then this parameter change is effective.
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">![Model Performance](/img/rescued/a8e154206ab8.png)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">## Stochastic Gradient Descent
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Sections 29~31 explain what stochastic gradient descent is.
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">During training, to make the model move in the optimal direction, we need to calculate the derivative at that point. 1. The calculation of derivatives is quite large, so we need to randomly select a subset of samples to calculate derivatives, to substitute for the real derivative. This is stochastic gradient descent. 2. To reduce the randomness of random selection, we use momentum inertia to reduce randomness. 3. To make the model stable in later stages, we reduce the learning step size.
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">End of Course One
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&amp;gt; Reference for assignment code
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&amp;gt; http://www.hankcs.com/ml/notmnist.html
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div></description></item></channel></rss>