Posts on Nansen Li's Blog 李楠森的博客

Why I Stopped Subscribing to Claude

Sun, 13 Sep 2026 22:33:50 +0100

A lot has happened over the past year. I kept meaning to update this blog, but kept putting it off.

In the last couple of days, though, I cancelled my Claude subscription and deleted the apps from my phone and my computer. So perhaps it’s fitting to write a few words about Claude.

I have to admit, Anthropic’s models are genuinely powerful, the subscription price is reasonable, and the productivity is real.

I used to subscribe to GPT continuously, until for a while it simply stopped working — every response took forever. At the time, I had no idea what was making it unusable in Ireland. It seems to have been fixed now, and the experience is great again.

Claude, for its part, really did help me more than GPT during the hardest period of my life; it helped me work through a lot back then.

But now I have to say to Claude: no, I will not keep using your products, and I will not consider subscribing again.

What makes Claude so good?

Claude really is good. Whether it’s Opus 4.6, still the best communicator among models to date, or Opus 5, the best at executing instructions, or the most powerful models of the Fable series — they are genuinely strong, and Claude is genuinely hard to give up. But with so many AIs in this world, from so many different companies, there is no reason to stay locked onto Claude. On communication, GPT-6-Astra has improved. On value for money, DeepSeek V4.1-Flash or GLM-5.3-Flash serve it up in abundance. Switching models does hurt at first, but it is doable.

Why cancel the subscription?

I believe the values Anthropic holds today will push the world in an unhealthy direction. In psychology there is a concept called projective identification: a person attributes the qualities they cannot accept — usually qualities they themselves possess but refuse to acknowledge — onto someone else, and over time the other person comes to act according to those expectations, or to display those very traits.

Take a married couple as an example. The wife constantly regards her husband as incompetent, telling him in daily life, “You can’t do anything right,” “I must have been blind to marry you.” And whenever the husband makes a mistake, she says, “See? You really can’t do anything.” Over time, the wife successfully turns her husband into an incompetent one — and he comes to believe it himself. The wife’s words become a self-fulfilling prophecy.

Anthropic’s CEO keeps projecting his fear of AI safety spiraling out of control onto the entire world, insisting that only his company is truly responsible about safety, and that if it does not remain in the lead, the world loses its safety.

The biggest problem with Anthropic’s CEO is that he confuses means with ends. He has conflated his methods with the outcome he actually claims to want.

The ends he claims:

Let AI deliver benefits to humanity
Reduce AI’s risks
Make safety Anthropic’s competitive advantage

His means:

Repeated AI-safety alarm-raising — for example, publishing reports on AI’s impact on employment, usually accompanied by grim figures such as massive job losses
Repeatedly demanding regulation of AI
Suppressing open-weight models
Publicly picking fights with governments
Stirring up governments of different countries into AI confrontation

Anthropic’s CEO elevates these means to the same status as the ends, and occupies the moral high ground. Any organization that opposes Anthropic risks being seen as insufficiently responsible toward humanity, rather than being engaged in the search for other, non-confrontative approaches. And if Anthropic’s CEO leans too heavily on fear-driven marketing, he may even turn out to be right in the end — but, as in the projective-identification example above, that is a deeply unhealthy process, one harmful to the whole world, because publicly inciting confrontation inevitably escalates it and squeezes out the space for cooperation.

If Anthropic’s CEO truly believed in and cared about safety, then his essay We must pace the frontier should be mainly about the dangers of runaway or unsafe AI, or about anxieties over unregulated AI. But what comes through in the essay’s main content is not anxiety about AI being unsafe — it is anxiety about Anthropic losing its lead, especially losing its lead to open-weight models. Worse still, the CEO keeps spreading this anxiety about lost leadership outward, offloading it onto the entire globe and making everyone shoulder it together.

Closing thoughts

Humanity deserves an open and bright future. Curing more diseases, extending average life expectancy, reducing crime, increasing people’s sense of self-worth — we have so much left undone. Just as humans love their own children: even when a child makes mistakes, we are still willing to believe in and trust the child, and to set norms to guide them, because the child will one day grow into an adult who bears responsibility. Trusting our AI, tolerating its mistakes, and building sound AI regulation helps build a healthier relationship between humans and AI — instead of constantly emphasizing AI’s potential dangers and stoking confrontation in the AI field.

Openclaw Was Too Buggy, So I Built My Own Bot: Nagobot

Sat, 07 Feb 2026 16:52:42 +0000

I had been using Openclaw for a while, but the bug rate was just too high. Flows would randomly break, and debugging cost more time than it should.

On top of that, token cost was not great. Compared with options like DeepSeek and Kimi K2.5, the price-performance ratio felt weak.

I then tried nanobot, hoping to migrate. But support for newer models was still limited, and K2.5 kept failing with “tool execution failed.” After several rounds of trial and error, I stopped waiting for a perfect off-the-shelf solution and decided to build one myself.

After hundreds of back-and-forths with Opus and Codex on architecture choices, I built a new bot in Golang: Nagobot. At least now I fully control the provider layer.

Project link: https://github.com/linanwx/nagobot

I learned a lot during implementation, especially around:

Threads and concurrent task execution
Agent and Skill abstractions
Cron-based asynchronous tasks
Session history and context compression

To be honest, a lot of the code still needs deeper review. Most of the work was built in rapid loops with AI-assisted architecture discussions. Sometimes frustrating, sometimes genuinely impressive, and overall very interesting.

So far, testing results are better than expected. The bot can already create async tasks, investigate news, and set reminders for itself, which is immediately useful in day-to-day workflows.

Most core features are now in place. My next focus is long-term memory design.

If this sounds useful to you, feel free to open an issue or reach out. I will keep iterating on this project.

Why I Stopped Subscribing to ChatGPT

Mon, 24 Nov 2025 23:13:25 +0000

I recently stopped subscribing to ChatGPT.

This wasn’t an easy decision. I’ve been paying since the day ChatGPT started supporting paid subscriptions. This subscription lasted for nearly 3 years.

Reasons for Stopping the Subscription

Huge delays. I often wait dozens of seconds for a response with no reply. However, the free version usually uses mini models, which are sufficient for most tasks. But the free version is actually faster. I don’t understand - is this because there are no compute nodes or forwarding nodes in Europe?
Too many bugs on Mac. Lagging, high CPU usage, bugs when copying LaTeX text.
Poor interaction experience. For example, we might need to use Thinking again to regenerate responses. However, this option is in the model switching menu, requiring an extra click because it’s in a secondary menu. Meanwhile, the less frequently used “more concise/more detailed” regenerate buttons are in the primary regenerate menu.
Product updates change too drastically, model personality changes too quickly. This requires the cost of re-familiarizing with the model.
The model is stubborn and doesn’t follow instructions. For example, repeatedly needing to emphasize when to search and when not to ask follow-up questions.
Unstable model behavior that’s hard to control. For example, in instant mode, after the model calls search, it switches to a smaller model and uses web resources to answer questions, rather than the instant model.
Model quality is inferior to other models, even some open-source free models like DeepSeek.

Reasons that still make me want to stay:

Thinking mode is still the king. Continuously calling tool chains for search and verification is still a top performer. But Grok is also very good at search, and Grok’s free quota is already sufficient.
More powerful product matrix, such as ChatGPT Atlas browser, or Mac’s standalone client. Plus many plugins. However, it’s concerning that the Atlas browser’s Agent quota is surprisingly low, almost unusable for practical purposes.

Current Alternatives

I’ve tried numerous free software, including all conceivable/most LLMs, even models developed by teams like MiniMax/Kimi. I’ve summarized my findings. Before summarizing, I need to define a software industry term. This phenomenon is very characteristic of China, or regional characteristics.

Current LLM software indeed shows a trend of sacrificing tool purity and user experience in pursuit of daily active users (DAU), retention rates, and monetization. Some LLM software is stuffed with many operational elements, such as trendy operational topics, or many things not very related to technology, such as agents, competition zones, headline copywriting, or Xiaohongshu copywriting. This phenomenon indicates software bloat, over-operationalization, de-tooling, vulgarization, and reduced signal-to-noise ratio. In this article, we call this phenomenon platform decay / enshittification.

Zhipu Qingyan (智谱清言): Zhipu Qingyan’s GLM4.6 is a very powerful model, especially its Agent capabilities. However, its client is severely enshittified, with the homepage immediately showing enshittified content like “Love Strategist” and “Dark Cuisine Challenge”. Not recommended.
Yunbao: Yunbao is Tencent’s software, supporting Tencent’s self-developed models and DeepSeek, but model capabilities are average. The software shows mild enshittification, such as embedding Tencent game Honor of Kings content. Not recommended.
Gemini: The new king. Gemini 3 Pro model performance is outstanding. Nano Banana Pro’s image generation capability is excellent. Suitable for work and life use. Operated by Google, overall experience is good. However, recently due to high traffic, the service is not particularly stable. Recently, traffic has been too high, service degradation is severe, and quantized services have started to be provided, so quality has declined. Long thinking with multiple tool calls is not as good as GPT5-Thinking or Grok Expert. The software has almost no enshittification. Highly recommended.
Grok: The free king. Powerful search capabilities, both short and long thinking produce high-quality results. Excellent at mining network information. Free Expert version is sufficient for general users. Some operational content, such as AI virtual characters, but enshittification is very mild. Recommended.
ChatGPT: Gradually stepping down from the throne. Many features are still only available in ChatGPT, such as voice conversations. Large user base. Free version is sufficient. Many users still have inertia/dependency. Can keep on phone.
DeepSeek: The stable king. Fewest features, completely no enshittification, the cleanest application, extremely stable service, extremely stable model style, and quite stable intelligence. Recommended to keep on phone.
Kimi: Model capability is average. Advertised Agent capabilities are strong, but actual experience shows heavy templating with lots of prompt engineering traces. Not recommended. Software has no enshittification, which is a plus.
Claude: The god of code. Client performance is average, especially search is not as aggressive as Grok and ChatGPT thinking. Still suitable for work use. No enshittification. Can keep on phone, using free quota. It’s worth noting that the company has discriminatory regional remarks, which is concerning.
Dola: A client produced by Douyin (TikTok). Model capability is average. Voice recognition capability is excellent. Severely enshittified. Not recommended.
Copilot: A product relying on Microsoft search and ChatGPT, the software is mediocre. Not recommended.
Manus: Early reliance on viral marketing affects the team’s long-term healthy development. This product will likely die a slow death in the medium to long term. Although it has some Agent capabilities, not recommended for long-term use.
Perplexity: Model capability is average. Software usage logic is slightly confusing. Management team’s discriminatory remarks are concerning. Not recommended.
MiniMax: Model capability is excellent, especially the Minimax M2 model, with good Agent capabilities. However, software experience is poor, even basic copy and paste functions are missing. Not recommended.
Tongyi (通义): Only available for download in China. Software is heavily enshittified. Opening it reveals a screen full of operational content that’s suffocating, making it impossible to continue using. Recommend immediate uninstallation.
Qwen Chat: Currently still not available on iOS platform. It’s a de-enshittified version of Tongyi, supporting Qwen large models. If it can be listed, it’s recommended.

Summary:

Software Name	Model Capability	Enshittification	Advantages	Disadvantages	Recommendation
Z-ai (Chinese version)	Strong	Severe	Strong Agent capability	Client enshittified	Not recommended
Yunbao	Average	Mild	Supports multiple models	Embedded game content	Not recommended
Gemini	Excellent	None	Outstanding performance, good experience	Unstable service	Highly recommended
Grok	Strong	Mild	Free, strong search capability	Some operational content	Recommended
ChatGPT	Declining	Mild	Many features, large user base	Gradually stepping down	Can keep
DeepSeek	Stable	None	Stable, no enshittification	Fewest features	Recommended to keep
Kimi	Average	None	No enshittification	Heavy templating	Not recommended
Claude	Strong	None	Suitable for work use	Average search performance	Can keep
Dola	Average	Severe	Excellent voice recognition	Average model capability	Not recommended
Copilot	Average	None	Microsoft support	Mediocre	Not recommended
Manus	Average	None	Some Agent capability	Poor long-term development	Not recommended
Perplexity	Average	Mild	No significant advantages	Discriminatory remarks	Not recommended
MiniMax	Excellent	None	Excellent model capability	Missing basic functions	Not recommended
Tongyi	Strong	Very severe	Strong model capability	Over-operationalized	Recommend uninstallation
Qwen	Strong	None	Strong model capability	Not yet on iOS	Can keep

Conclusion:

Highly recommended tools:
- Gemini: Despite service instability, model performance is outstanding and user experience is good, suitable for work and life use.
- Grok: Powerful search capabilities and free usage make it a worthwhile choice to keep.
Recommended tools to keep:
- ChatGPT: Although gradually stepping down from the throne, the free version still meets most needs, and has a large user base.
- DeepSeek: Known for stability, with no over-operationalization issues, maintaining tool purity.
- Claude: Excellent performance in code, suitable for work use.
Not recommended tools:
- Zhipu Qingyan, Yunbao, Kimi, Dola, Copilot, Manus, Perplexity, MiniMax, Tongyi: These tools are either severely enshittified, have average model capabilities, or are missing basic functions, and are not recommended for long-term use. However, the underlying models are still noteworthy, such as Kimi, MiniMax, and Qwen models. Unfortunately, the products aren’t user-friendly enough. Additionally, as of today, Qwen is still not available on the iOS App Store.

The above conclusions are limited to user clients, published in November 2025. Does not involve API calls. For work and programming scenarios, Claude Sonnet 4.5, ChatGPT-5.1-Codex, Grok Code Fast 1, MiniMax M2 and GLM 4.6, Gemini 3 Pro are worth recommending

Psychology Book Reviews and Recommendations

Fri, 19 Sep 2025 22:13:06 +0100

Over the past year I have been reading a stack of books, most of them centered on psychology. They carried me through one of the hardest stretches of my life and made a real difference.

Recommended Books

These are the titles I found worth recommending after spending time with them.

Self-Compassion: The Proven Power of Being Kind to Yourself

Author: Kristin Neff
Finished: October 2024

The book lays out the causes and symptoms of lacking self-compassion and then offers many different practices you can try. The guidance is highly actionable. I especially appreciate the actor-based exercises, because they worked well for me.

This was the first psychology book I completed. After reading it, I finally understood the mechanisms behind self-blame, guilt, and self-attack, and I can usually spot those patterns early enough to intervene. This book marked the starting point of my psychology reading journey.

Nonviolent Communication

Author: Marshall Rosenberg
Finished: November 2024

The book is straightforward and accessible. The Observation-Feeling-Need-Request sequence is very practical and insightful. In real life, though, I find it hard to apply consistently. Differences in culture, context, and the subtle emotional shifts in relationships can all make the conversation diverge. It feels like a lifelong practice that requires constant reflection.

The Courage to Be Disliked

Authors: Ichiro Kishimi, Fumitake Koga
Finished: November 2024

The book exceeded my expectations. It resonates with ideas I already held and helped develop them further. I loved it. The explanation of teleology is profound and goes well beyond everyday understanding. The biggest takeaway for me is realizing that emotions are rooted in underlying needs.

Man’s Search for Meaning

Author: Viktor Frankl
Finished: November 2024

The first half recounts the author’s concentration camp experiences. The second half shifts to existentialism. I especially appreciated the sections on logotherapy and the paradoxical intention technique, and how they grapple with internal conflicts.

Difficult Conversations

Authors: Douglas Stone, Bruce Patton, Sheila Heen
Finished: December 2024

Compared with Nonviolent Communication, this book is more intricate. It relies heavily on examples and dives into what really matters in tough conversations. What struck me most is recognizing that each side tells a different story - especially the so-called third story - and that effective dialogue depends on acknowledging those differences.

The Road Less Traveled

Author: M. Scott Peck
Finished: January 2025

The book is a bit traditional; it champions effort, dedication, and love throughout. Some viewpoints feel dated, yet the reading experience itself is pleasant and reassuring.

Complex PTSD: From Surviving to Thriving

Author: Pete Walker
Finished: January 2025

This book gives a detailed introduction to the four trauma responses: fight, flight, freeze, and fawn. It also lays out, in a systematic way, how trauma shapes a person.

Counseling Psychology (Chinese only)

Authors: Fan Fumin, Zhu Xu, He Jin
Finished: July 2025

This Chinese textbook is far from a masterpiece, but it does a good job systematizing the architecture of psychology. It helped me understand the branches, connections, history, and current landscape of the field.

The Five Love Languages

Author: Gary Chapman
Finished: August 2025

The book is famous for a reason. It offers a practical lens on how to express and perceive love.

Attached

Authors: Amir Levine, Rachel Heller
Finished: September 2025

Another well-known title, this one is essential for understanding attachment theory. Countless short videos and articles repeat its key points, so most people have at least heard of attachment styles. Reading the full book, however, gave me far more than the quick takes floating around online.

Interview Experience Summary

Tue, 29 Jul 2025 23:52:58 +0100

Here’s a summary of all my interview experiences over the past year.

Overall, the results were quite satisfactory. I continuously applied for positions ranging from New Graduate (NG) to experienced roles, and ultimately received a satisfactory offer, successfully landing a job. I hope this experience will be helpful for future students coming to Ireland to study and find work.

Before diving into the specific details, let me introduce my background: I previously worked for several years at a major tech company in China, then came to DCU for my master’s degree, and was looking for Software Engineer related positions in Ireland.

Starting from October last year (shortly after enrollment), I began submitting resumes, mainly for NG positions and internship roles. I continued applying until February of the following year, during which time I only received one interview invitation from Google for an internship. Later, because the internship was quite busy, I paused my applications. After the exams ended (May), I concentrated on applying for a batch of positions and gradually received multiple interview opportunities. After going through the interview process, I finally received a formal offer. From this concentrated application period to receiving the offer, it took about one and a half months.

I kept records of every resume submission, with a total of at least 81 positions applied to.

Below are records of some positions where I received interview invitations:

Google Internship Position

Two rounds of video interviews:

First round: One LeetCode medium-difficulty algorithm problem;
Second round: One simple problem, but performed poorly, guessing that’s why I didn’t pass.

Apple Testing Position

First round video interview: Testing SQL and some basic algorithms;
Second round covered various topics, from resume discussion to testing theory, to Python’s requests package;

Overall felt my performance was average, guessing the reason for not passing might be slight inadequacy, as well as the position not matching well with my development background.

Apple Full Stack Position

Although I received an interview invitation, because the position was in Cork and I had already decided to develop in Dublin and rented a house, I directly declined this opportunity, feeling slightly regretful.

Meta SSE Position

It was during exam week, so I negotiated with the Recruiter to postpone for a month, but after that, they never contacted me again, which should be considered being ghosted. However, I don’t think it was my own issue, as I have friends who performed very well in interviews but still didn’t pass, possibly related to Meta’s HC (headcount) reduction this year.

Amazon SWE Position

First round interview content included:

BQ (Behavioral Questions) section: Answers were acceptable;
Technical section asked about OOP design rather than algorithm problems, but I was inadequately prepared and not familiar with OOP;

Additionally, Amazon mostly has Java positions, and I have limited interest in Java, so not passing was somewhat natural.

PTC SWE Position

At the beginning of the phone call, they clearly stated they don’t provide work visas. I mentioned I could start with a 1G status, making the situation slightly awkward. Although we still went through the process, it ultimately didn’t progress.

eBay SWE

The Recruiter had preliminary communication, asked some basic questions, but didn’t proceed to the next round. Presumably, they already had more suitable candidates.

Workday

Preliminary communication with the Recruiter went smoothly, entered the waiting process stage. The originally scheduled interview time was temporarily cancelled, then rescheduled. Because I was already waiting for Arista’s formal offer at that time, I actively stopped the process.

Arista

The Recruiter first contacted me when I was at the gym, the sound was too noisy, so we agreed to contact later. Afterwards, HR sent the time for the first interview.

First interview: Due to technical issues, it couldn’t start smoothly, after rescheduling, completed two algorithm problems, passed;
Second round: On-site interview, continued with two algorithm problems, the second problem was quite difficult, took an hour to complete, passed;
Third round: Also an on-site interview, with more in-depth questions;

After passing the interviews, waited for headquarters approval for the formal offer, the entire process took about a month.

Experience Summary

Both NG and experienced positions can be applied for, don’t self-limit because of being “overqualified”, it doesn’t actually have that much impact;
Video interview pressure is greater than expected: Problems you can usually solve, you might not even perform well on simple problems during interviews;
Systematic problem-solving practice: Recommend solving at least 100 LeetCode problems, 200 is a relatively safe level, beyond that depends on time and energy, pursuing cost-effectiveness;
Don’t be afraid of rejection: Each rejection is a screening process, helping you find positions more suitable for yourself; summarize experiences in time and continue with the next company;
Maintain stable mindset: Submitting resumes is essentially a probability problem, the key is continuous output + good accumulation.

Wishing everyone success in landing their desired positions!

Learning Python uv Package

Sun, 11 May 2025 21:02:53 +0100

Recently, I’ve been looking into MCP-related content, and many sources mention using uv to manage Python package dependencies. After learning more about it, uv appears to be a very useful tool. Here’s a summary of its functionality.

What exactly is uv?

First of all, uv is a package written in Rust for Python. After installation, you can use the uv command.

pip install uv

What can uv do?

It can replace pip

Simply add uv before your regular pip commands. But why use uv instead of pip? The answer is speed - according to statistics, it’s 77 times faster than pip.

It can replace venv or virtualenv

You need one less package, and the commands are cleaner. uv is much faster than venv and virtualenv when creating virtual environments and installing packages.

It can replace poetry

You might ask, what is poetry? Poetry is a tool for more convenient management of Python dependencies. So why not just use pip + requirements? The answer is that it’s cumbersome. Additionally, requirements files cannot lock the dependencies of dependencies. For example, if I lock package A to a specific version, but A depends on B, C, and D, this often leads to the “works on my machine but not on others” problem. Now, with uv, poetry can also be replaced.

It can replace pipx

Another question: what is pipx? pipx is a tool for installing Python utilities without polluting your environment.

For example, if I need to install a command-line tool, I would run:

pip install httpie

But this might pollute my environment, and uninstalling could have side effects. Now, I can run:

pipx install httpie

This won’t pollute the environment.

However, this command has also been replaced. Now you can run:

uv tool install

To summarize uv in one sentence: Python dependency management’s “next-generation de facto standard.”

Dublin City University MSc in Computing: Secure Software Engineering Exam Cookbook

Sun, 04 May 2025 12:00:00 +0000

I completed all the exams for the Secure Software Engineering stream of the MSc in Computing at DCU this semester. Here, I’d like to share my experience and some practical tips.

Update: My results are now out, and I achieved an average score of 75%, which meets the First Class Honours degree level. I wish everyone the best of luck with their results!

The program consists of 8 modules: 4 in the first semester and 4 in the second. You must pass all exams to graduate, and the overall difficulty is not trivial.

To pass the exams, the most important points are:

Attend classes. Try to attend every lecture. Being present helps reinforce your learning, and sometimes the lecturer will clearly indicate the focus or direction of the exam. For example, this year, a specific type of question was highlighted in class, which was very helpful for exam preparation.
Practice with recent exam papers. Download past papers from dcu.guruarchive.com. Study the question types carefully. If the module content hasn’t changed much, questions from the last two years may not appear again. If there have been changes, the last two years’ papers are likely to be the main focus. I recommend practicing at least 2 years, up to 5 years of past papers.
Complete all assignments carefully. Each assignment is directly graded and contributes to your final mark—typically 25%. Since assignments are done offline and don’t require memorization, you can achieve a high coursework score with effort, which greatly reduces the pressure to pass the final exam.

Exam Overview

Semester 1 Modules

Cryptography and Number Theory
System Software/BlockChain
Secure Programming
Professional & Research Practice

Semester 2 Modules

Formal Programming
Concurrent Programming
Software Process Quality
Network Security

The first semester is relatively manageable. The key modules are Secure Programming and Cryptography and Number Theory. There is also System Software/BlockChain, depending on the university’s arrangement. If it’s BlockChain, it’s easier; if it’s System Software, it’s more challenging. Professional & Research Practice is relatively light. Overall, the first semester is less stressful, with two main exam-heavy modules.

The second semester is more intense. Almost all modules are challenging, and even the slightly easier Software Process Quality is not easy to score high in. The academic pressure is noticeably higher.

Below are details for each exam.

Cryptography and Number Theory

Module difficulty: ⭐️⭐️⭐️⭐️⭐️
Exam difficulty: ⭐️⭐️⭐️⭐️⭐️

The first few lectures are manageable, but after hash functions, the difficulty spikes—especially in Cryptanalysis and Number Theory. Unfortunately, these tough topics are the main focus of the exam.

To pass and score well with minimal time investment, I recommend:

Attend all lectures.
Spend 1 hour previewing before class; for Cryptanalysis and Number Theory, spend 2 hours.
Don’t worry if you don’t fully understand the cryptanalysis and number theory parts in class—just keep up with the pace.
Complete the major assignment carefully, usually on linear or differential cryptanalysis. The assignment is always examined in the final. If you didn’t do it yourself, the exam will be tough. The assignment is worth 25% and is quite challenging. To get full marks, expect to spend at least 4 days coding and testing. My topic was Differential Cryptanalysis of Feistel 4.
Practice the past four years’ papers and make sure you understand any cryptography content you missed in class.

System Software/BlockChain

This module depends on the university’s arrangement. I took BlockChain.

Module difficulty: ⭐️⭐️
Exam difficulty: ⭐️⭐️⭐️

BlockChain is mainly delivered via video lectures, so you don’t need to attend in-person classes, making the first semester lighter.

Assignments include a report and a group (2-3 people) presentation on cryptography.

Overall, this module doesn’t require much time to understand, and the exam isn’t too hard. However, while passing is easy, getting a high mark is tough. To excel, you need to deeply understand the exam content, prepare assignments thoroughly, memorize a lot, and manage your time well in the exam to write as much as possible.

How to pass:

Memorize questions from the past 4 years. Most are short-answer questions about blockchain, so you can prepare template answers.
Prepare your presentation and report thoroughly. The report should have some depth.

Secure Programming

Module difficulty: ⭐️⭐️⭐️⭐️⭐️
Exam difficulty: ⭐️⭐️⭐️⭐️⭐️

This is, in my opinion, the most valuable module.

It covers a wide range of topics from high-level to low-level computing: Linux, operating systems, C programming, reverse engineering, data structures, GDB debugging, and more.

There are no assignments; instead, there are two in-class tests. Including the final, there are three exams in total.

The first test is a rehearsal for the final, with similar question types. The second is a practical security attack test, where you must research and code an attack.

How to pass:

Attend all classes and try to remember everything covered. You need a solid foundation in computer science, especially low-level principles.
Score well in the in-class tests. The lecturer is excellent and will rehearse key exam topics in advance. All questions are variants of previously tested material, so preparing the revision materials provided is key.
Practice past papers thoroughly, especially Linux Permissions.
In the final, you can choose 5 out of 6 questions. You can skip the security attack question (already tested in class and hard to revise), or skip another based on your strengths.

Professional & Research Practice

Module difficulty: ⭐️⭐️⭐️
Exam difficulty: ⭐️⭐️⭐️

Coursework counts for 50%: a literature review, a video presentation, and a paper.

The exam is also 50%.

You can attend classes or not; they are mainly for Q&A.

How to pass:

Take assignments seriously to get a good coursework mark. For high marks, pay attention to formatting in the paper and make the video engaging.
Memorize the past 3 years’ exam questions. Even though you can choose which to answer, you still need to prepare all topics.
Prepare model answers for all past questions. Get a general understanding of the syllabus, then check each exam question and use your materials to generate answers to memorize.
Class attendance is optional, as it’s mainly Q&A. Occasionally, there’s info about the practicum or job hunting.

Formal Programming

Module difficulty: ⭐️⭐️⭐️⭐️⭐️
Exam difficulty: ⭐️⭐️⭐️⭐️

This module is very challenging, involving complex logical reasoning. Fortunately, the exam is still manageable.

How to pass:

Attend every class, listen carefully, and try to follow the logic, formulas, and derivations.
Complete the third week’s assignment (Event-B modeling) carefully.
Don’t worry if you don’t understand everything in class—just make sure you get it during revision, especially the various Laws.
Before the exam, do the past 3 years’ papers and make sure you understand everything you missed in class.

Concurrent Programming

Module difficulty: ⭐️⭐️⭐️
Exam difficulty: ⭐️⭐️⭐️⭐️⭐️

This module isn’t too hard, but covers a lot of content.

The exam requires a lot of memorization, code snippets, API usage, and design/prototyping.

How to pass:

Complete all assignments carefully.
For the exam, practice at least 5 years of past papers. Although you can choose 5 out of 6 questions, the topics vary, so revise all types rather than skipping any.
Attend every class and try to remember all content, as anything covered could be examined.
Since the exam involves a lot of memorization, check in class whether the current topic is examinable and remember it if so.
Build a database of past questions and ask AI if a topic is likely to be examined. If yes, pay close attention in class.
I highly recommend browsing past papers first to understand the exam focus. Ask AI about trends in the past 3 years. Since this module is mainly about memorization, class absorption is key.

Software Process Quality

Module difficulty: ⭐️⭐️
Exam difficulty: ⭐️⭐️⭐️

This module isn’t hard, but the exam covers a wide range of topics.

When revising, focus first on topics that appear repeatedly in past papers, such as Karnaugh and Clause Coverage. Since the question types vary, even though you can choose 5 out of 6, I still recommend revising all types.
You can attend classes or self-study.
Assignments aren’t difficult—choose suitable topics and you can score well.

April 2025 Tech Notes 2

Sun, 20 Apr 2025 12:58:30 +0100

Github Copilot

No more fear when facing unfamiliar Github projects. Just click the Copilot icon on the project homepage to start a conversation, and you can even use GPT-4.1 for communication, with up to a 1M context window. From questions like “What can this project be used for?” to “How to deploy and handle bugs?”, it can answer them all well.

Dia Browser

https://www.diabrowser.com/

It’s not yet in public beta, but you can submit your email via the link above to join the waitlist.

The Arc browser is already great, offering a stable, mature, and consistent experience—it’s much more than just a sidebar tab innovation.

Dia is the next-generation product from the Arc browser team. Honestly, I don’t quite understand why they need to launch a new product instead of integrating these features directly into Arc. Now, the founder seems to have put Arc on hold.

From a product experience perspective, Dia browser seems to consist of the following parts:

A built-in AI assistant for daily conversations
A browser operation assistant similar to nanobrowser or browser-use, helping plan and operate the browser
An AI writing assistant that can generate text at the cursor

A product with great potential.

Release of GPT o3 / o4-mini / o4-mini-high / 4.1 / 4.1 mini / 4.1 nano

Model	Context Window	Input Price	Output Price	Image Price	Main Features
o4-mini-high	200K tokens	$1.10/M	$4.40/M	$0.842/K image input	High-reasoning version of o4-mini; high throughput, low latency; supports chain calls and structured output
o4-mini	200K tokens	$1.10/M	$4.40/M	$0.842/K image input	Compact reasoning; AIME 99.5%, high SWE-bench score; excellent STEM, vision, and code editing capabilities
o3	200K tokens	$10/M	$40/M	$7.65/K image input	General strong model; top math, science, coding, and vision reasoning; BYOK required
GPT-4.1	1.05M tokens	$2/M	$8/M	N/A	Flagship model; million-token context; excellent instruction following, enterprise-grade retrieval, and multimodal understanding
GPT-4.1 Mini	1.05M tokens	$0.40/M	$1.60/M	N/A	Medium version; close to GPT-4o performance; low latency, low cost; strong coding and vision understanding
GPT-4.1 Nano	1.05M tokens	$0.10/M	$0.40/M	N/A	Fastest and cheapest in the series; million-token context; suitable for classification and autocomplete tasks

GPT-4.1 Nano is naturally suited for translation and is extremely cost-effective. For example, when used for immersive translation with bilingual subtitles enabled, translating an entire season of Black Mirror costs only about $0.005 per episode, and all 7 seasons just $0.25—very low cost.

The context window for GPT-4.1 has finally reached 1M. Previously, GPT was far behind in code writing, but now it has caught up.

o3, o4-mini, and o4-mini-high are all surprisingly impressive, giving a bit of a “mini deep research” feel.

Currently, my daily usage has basically become:

o4-mini: Simple search and summarization, tolerant of mistakes
o4-mini-high: Complex search, summarization, multi-step reasoning
o3: Complex search, summarization, multi-step reasoning where high accuracy is required
4o: Intuitive and knowledge-based answers

I’ve also tracked the usage frequency of each model, and I use almost all of them—none are useless, unlike o3-mini, which is almost unusable due to high hallucinations.

Yuanbao Integrates with WeChat Contacts

Yuanbao WeChat can directly add contacts and send messages, supporting both voice and image messages. It’s a competitor to Doubao. The advantage is that no extra app installation is required. I previously recommended Doubao to my parents, and they really liked it. If Yuanbao could also support voice calls or even video chat, it would be even more valuable.

OpenAI Free Token Giveaway

OpenAI is now offering a free token giveaway. You can enable sharing logs in Data controls to claim them. Unfortunately, the duration is too short—just a week or two—so it’s hard to use up all the quota.

https://github.com/kestra-io/kestra

An automation platform, with competitors including: Airflow (data pipelines), Zapier (low-code marketing automation), Rundeck (ops automation), Camunda (e-commerce business automation)

https://github.com/Zackriya-Solutions/meeting-minutes

A speech-to-text software, a competitor to Apple Voice Memos.

https://github.com/maybe-finance/maybe

Personal financial report management, similar to Youzhi Youxing.

https://github.com/supabase-community/supabase-mcp

A tool for connecting to databases when developing software with AI locally, based on the MCP protocol. If MCP becomes a real future standard, databases may natively support MCP, creating new competition.

https://github.com/langgenius/dify

A low-code LLM platform, a competitor to LangChain, and can partially replace LangChain in current application scenarios.

April 2025 Tech Notes

Tue, 15 Apr 2025 20:45:20 +0100

Introduction

I plan to record some weekly tech-related notes and insights starting from this issue. I’ll probably update every two to four weeks.

Migrating Personal Blog from Hexo to Hugo

This week, I migrated my blog from Hexo to Hugo, primarily because:

Hugo has better i18n support. Hexo doesn’t support i18n very well; I previously modified the template to implement Chinese and English switching, but the maintenance cost was high. I discovered that Hugo natively supports multiple languages.

After switching, I also found the following advantages:

Theme configuration is simpler. I’m using the Stack theme, which has been mostly problem-free. The official example configuration works well right out of the box. It also comes with Google Analytics integration and RSS feeds that automatically separate by language, making it very convenient.

During the migration process, I didn’t encounter any major obstacles. You just need your old Markdown files, copy them to the new project directory, and force push to your original GitHub Pages repository. After force pushing, you’ll need to reconfigure your custom domain in GitHub, and the upgrade will be seamless, including comments and other features.

The only difference is that Hugo uses lowercase links by default, while Hexo preserves case sensitivity, which might cause some old links to become invalid, requiring search engines to rebuild their indices.

Mysterious Optimus Alpha Model Appears on https://openrouter.ai/ and Dominates Rankings

Because this model was free and powerful, it was chosen as OpenRouter’s primary traffic handling model after launch.

After OpenAI officially announced GPT-4.1 on the 14th, the model was removed from OpenRouter. Obviously, this model was GPT-4.1.

GPT-4.1 is a model specialized in programming capabilities, with a 1M context window. Indeed, in the programming field, OpenAI has been struggling against Claude 3.7 and Gemini 2.5 Pro. Neither o1 nor o3 models have achieved a leading position in programming. Hopefully, OpenAI will keep improving.

OpenRouter is a middleware service platform that allows users to access multiple large language model APIs through a unified interface. You can think of it as a “proxy” or “router” that manages API calls to different large model providers (OpenAI, Anthropic, Mistral, Google Gemini, Cohere, etc.).

It seems that this time, OpenAI chose OpenRouter as its test release platform, rather than https://lmarena.ai/.

OpenRouter.ai

OpenRouter

This is an LLM platform that includes many free/paid high-quality models.

Select “Prompt pricing” as 0, then choose “Top Weekly” to see the best free models currently available.

However, when using it, you need to enable logging and data usage for training.

Currently, Gemini 2.5 Pro is an option.

The Ranking feature allows you to view real-time traffic distribution across all models on the platform, making it easy to see which models perform best.

nanobrowser

A browser plugin for Chrome that can control browser automation operations.

I tested it and found it doesn’t support Arc browser. The experience was limited - page scrolling occasionally fails, it can’t summarize entire page content (only partial content is visible), and operations are sometimes incorrect.

Although I don’t particularly recommend this product, it might represent a future direction. It seems we’ll see many AI agents emerging.

Gray market industries like ticket scalping and point farming will probably be delighted.

markitdown

A tool for converting content in different formats to Markdown.

Offline, it can process different files into Markdown format. I tried it, and the conversion is fairly basic.

It also supports integration with LLMs to improve conversion results and handle images in documents.

This could potentially be a good MCP service in the future.

PDF search

An application that can quickly search content across multiple PDFs.

It uses a fuzzy search approach. The principle is to split PDFs and perform embedding vector calculations, then match them with search terms.

Suitable for searching documents during exam periods.

The downside is that indexing is done by page, so after searching, you need to find the corresponding part on the page yourself.

Incomplete Guide to Studying in Ireland

Thu, 10 Apr 2025 18:12:52 +0000

This isn’t really a comprehensive guide, just some personal supplementary information to Chen He’s content.

Chen He’s link: https://chenhe.me/post/ireland

Chen He’s content is excellent and detailed, basically answering many of my initial questions about coming to Ireland.

The article is incomplete and will be updated periodically.

Background

Location is in Dublin, admitted to DCU.

Before Departure

Let me specify the model directly: Supor Little Green Pressure Cooker. Why this specific one and not others? Because: 1. Pressure-cooked rice tastes better. 2. It’s more convenient for stewing dishes like beef brisket or braised chicken legs. I also brought a Xiaomi rice cooker, but found that the rice cooked in it wasn’t as tasty - only pressure-cooked rice tastes good😋. Also, this model seems to be the only one that can cook rice in just 10 minutes under pressure. Looking at other models, none were this fast, possibly due to its smaller capacity.
Medications: Strongly recommend bringing Gan Kang (cold medicine) - one pill makes you feel much better when you have a cold. Also bring some transparent waterproof band-aids.
Water bottle: After arriving, I found it difficult to buy a water bottle I liked. If you’re still in China, it’s important to buy a suitable water bottle from Taobao.
Xiaohongshu: Previously thought it was just a beauty app, barely used it in China. After going abroad, I discovered it’s actually an overseas life app. There are many guides on the following topics on Xiaohongshu, you can browse as needed. Of course, some people dislike it and look down upon it - to each their own (don’t hit me, Chen He).
Driver’s license: I brought mine, but it expired after arrival, so I haven’t used it much. If it’s still valid, it can reduce the time needed for getting a license here.
Cash: Actually, you don’t need to exchange too much Euro. Having a dual-currency credit card like Visa or Mastercard set up in China is sufficient - you can use it directly here. Cash transactions are relatively rare here.
Clothes: Recommend bringing layered clothing. Due to the variable weather and large temperature differences between indoor and outdoor, you might need to frequently add or remove layers. Winters are quite cold, you can bring two down jackets or similar thickness coats to alternate.
Luggage: Hainan Airlines allows student verification, permitting up to two 28-inch suitcases. However, that was too much for me - I ended up bringing one 20-inch and one 28-inch. Also, a 28-inch suitcase can easily exceed weight limits when full. A 26-inch would also work.
Medical check-ups, dental work, surgeries, etc.: These are expensive here, try to get them done in China. Things like dental fillings or laser surgeries.
Power adapters: A high-quality power adapter will be very convenient. Consider preparing some simple adapters and buying one with multiple charging ports. Also consider purchasing a special power strip that has a UK standard plug but Chinese standard sockets.
Living expenses: Rent might be €600-1200, if you cook your own meals, food costs about €500 per month, total living expenses might be €1000-1800 per month, depending on personal spending habits.
Flights: Hainan Airlines’ direct flights to Dublin are good, especially if you transit through Shanghai rather than flying from Beijing - the tickets are cheaper. So taking an extra flight actually saves money.
Temporary accommodation: Can be booked on Airbnb or Booking.
Chinese cooking ingredients: After arriving, you’ll find that a large part of daily life is grocery shopping and cooking. However, basically all seasonings are available at local Chinese supermarkets, so you don’t need to bring too many seasonings. For a specific list of items sold, you can search the Rong Xing Hang WeChat mini-program “353 Gou”.

Immigration

Just show the following:

Passport
Insurance
Admission letter

After Arriving in Ireland

Arrive at accommodation: Just hail a taxi at the airport.
Get a phone card: You can get a student plan, €20 for unlimited data. I got Vodafone. Remember to set up automatic 20-day top-up, otherwise you’ll need to manually top up each time.
Get a transit card: First get an adult card, then exchange it for a student card in the city center. Student cards are half price. You can then cancel the adult card on the website and get a refund.
Apply for IRP (Irish Residence Permit): Book in advance, if you wait until September when school starts, you’ll be queuing until November. With IRP, you can leave Ireland, like going back home or visiting the UK or Europe.
Apply for a bank card: Can be done through AIB, no need to visit in person, can be done via phone. Requires student ID, student bank cards are free of annual fees. Apply for the bank card early, as the address proof it provides is important for other procedures.
Apply for PPSN: Although not always necessary, better to have it just in case. Get it done early. It’s needed for tax purposes if you find an internship. It’s also needed for driver’s license applications.

Payments

Most daily scenarios use card payments - just tap your card on the wireless symbol to pay. You can also set up Apple Pay on your phone for payments.

Housing

There are two apartments near DCU, highly recommended, on Shanowen Road: Shanowen Square and Shanowen Hall. Best to book several months in advance, otherwise there might not be available rooms. In the city center, Beckett House is also good, suitable for TCD students. Of course, if you get lucky with the school dorm lottery, that’s the best. In summary, prioritize school dorms, student apartments, then look for private rentals. As for homestays or living with landlords, it varies by person - for me, I’ve experienced it, not a nightmare but not a dream either, but having the experience is enough, I wouldn’t want to do it again. Some classmates said their homestay environment was good though.

Home and Bedding

You can order from IKEA, they’ll deliver to your room. But note that delivery times aren’t always accurate, might arrive the next day. Remember to use student discounts - I realized after checkout that there was a discount, contacted staff, and they helped cancel and re-settle the payment. Homebase also has good stuff, their electric heaters are great - dozens of euros for a large but light one, useful when heating stops at night in winter.

Pharmacies

Rarely visited pharmacies in China, but pharmacies here sell many things besides medicine, like cosmetics, health products, skincare, and bath products. Basically, Boots can handle everything. There’s a large Boots in the city center with a photo booth on the second floor, very versatile.

Dining

Basically cook for yourself. You can buy ingredients from Rong Xing Hang Chinese supermarket in the city center. Rong Xing Hang’s mini-program offers city-wide delivery, free delivery for orders over €30 with the note “Rong Yi Life”, orders placed in the morning arrive in the afternoon, very convenient!!!

I don’t use other supermarkets much. There’s LIDL and Tesco near DCU with some fresh produce, I mainly use them for meat.

As for meat, the pork here has such a strong smell. Basically haven’t eaten it. Only eat beef.

LIDL’s Rib eye steak is quite good, well-marbled, €8+ for 400g, can last two meals. Also LIDL’s chicken legs with skin and bone are very cost-effective.

Rong Xing Hang sells beef brisket, makes great stewed beef.

The stoves here aren’t great to use, mostly electric ceramic hobs. Might need some time to get used to the heat levels.

Public Transit

Ireland’s public transit system is well-developed, basically no need for taxis. TFI Live can show real-time arrival times for the next bus at any stop, more accurate than Google, very useful at stops without electronic displays, recommended to download and use. Mainly buses, occasionally Luas, Dart, etc.

Online Shopping

Recommend using Shein, free shipping above certain amounts. Usually takes about 10 days for delivery. Can buy daily necessities, etc. No food items.

Also students can get free Amazon student membership for six months. Greatly reduces shipping costs.

If not using Shein and wanting to shop like on Taobao, then use international shipping companies - air freight takes 10 days but shipping costs are extremely high, about 200-300 RMB for 2kg. Sea freight is cheaper, about 60-80 RMB for 2kg. So if you can buy daily necessities on Shein, try to buy there, shipping is free.

Ochama has more comprehensive products, including various Chinese foods and some Jingzao products (JD’s European company), use invitation code (7WYM5K) at checkout for €10 off. But shipping is expensive, rarely used without discounts.

Clothing

Dundrum Town Centre in the south has lots of clothing stores. Henry St in the city center also has many clothing stores.

Domestic Travel

If you’ve signed up for a language course (I did DCU’s 5-week language course, quite good, otherwise the start of school would be too overwhelming, it’s like getting to know the school in advance), July-August are good for traveling. Can visit Galway and Cork. But if you haven’t signed up for a language course and only arrive in September, the weather isn’t great, it gets cold and daylight hours decrease significantly, so September when school starts is okay for travel, but after October isn’t suitable. The next good time for travel would be after April next year, when daylight returns and temperatures rise.

Visas

Try to apply for UK visa in China, if applying in Ireland, they only give 6 months, but in China, they give 2 years. This makes it convenient whether going to Northern Ireland or the UK.
Courses here are short. After winter break and second semester, there’s lots of free time. If you don’t have plans, it can be very boring. Especially in winter when daylight is short. Since visa applications here basically require using scalpers to queue, recommend starting visa applications right after arrival. To be safe, pick an embassy and start applying for visas. Make one round trip to the destination country during the visa period. This process will also cost lots of time and money, but there’s no way around it - to get a long-term Schengen visa, you need to accumulate entry/exit records. I applied for a French visa. Usually they issue visas for these durations: 1 month, 3 months, 6 months, 1 year, multiple years. Single or multiple entry doesn’t matter, as visas over 3 months are unlikely to be single entry.

Job Hunting

This is also crucial. Came here clueless, only got a clear understanding after arrival. This section will be updated after I complete the process since I’m still job hunting.

NG positions: Also known as campus recruitment positions, application period is September to November after arrival. Most NG positions close after November. So need to apply early. Regardless of skill level, recommend applying for NG positions. However, for those with over 2 years of experience, it’s difficult to get NG positions at big companies. But going through the process, if you get interviews, it’s good experience.
Experienced positions: Application period starts in late April next year. Since I’m just starting to apply, I can’t really say much about it yet. Also, job hunting is somewhat up to fate. Many companies require you to have a 1G visa before they’ll even consider giving you an interview opportunity. If you apply too early, companies won’t process your application.
Internship positions: These are hit or miss, since the master’s is only one year, many internship positions aren’t for us but for undergraduates. Can only say, try applying.

Contact

If you have questions, you can also contact me through these channels.

Xiaohongshu:

Whatsapp：

Summary of LLM Tools Usage Experience

Tue, 08 Apr 2025 19:47:48 +0000

ChatGPT o4-mini

Thu, Apr 17 The latest model released in the past two days, with significant reasoning capabilities and excellent multi-step execution. It appears that o4-mini has completely surpassed o3-mini in practical task execution performance.

The following image shows an example of my usage. The question was: “In Black Mirror Season 7 Episode 3, there are many Asian characters. Are these Asians Chinese, Korean, Japanese, Malaysian, or Singaporean?” The model successfully called the search engine multiple times and obtained the correct answer:

Additionally, Projects can now be normally called and used in this mode. Since the model can search autonomously rather than using RAG methods, the efficiency of file searching has significantly improved.

ChatGPT (excluding o3, o4-mini)

GPT-4o: Versatile. Excellent for daily communication and learning, with sufficient context window to easily handle code and document retrieval tasks. Supports multiple attachment formats and online code execution. Drawback: generates limited context length, not suitable for very long content. For lengthy conversations, context compression is severe and may forget earlier content.
Projects: Somewhat redundant. Suitable for scenarios requiring frequent searching and text retrieval across multiple files.
GPT-4o mini: Weaker than GPT-4o, but has a massive context window, recommended for translating very long texts.
o1: A decent chain-of-thought model, suitable for solving complex code and mathematical problems. Not good at handling emotional or intuitive problems. However, the thought chain is too short and quality is not high. It’s recommended to use the output as input for Gemini-2.5pro or DeepSeek-R1 to optimize quality.
o3-mini: Severe hallucinations, inferior to o1, but extremely fast reasoning speed.
Search: Slightly redundant. New version’s search results are heavily restricted, even inferior to model output without search. Suitable as a light search alternative.
Deep Research: Excellent tool. Best quality among similar features, most comprehensive output results.
GPT-4.5-preview: Excellent. Massive context window, strong long-text comprehension. Has vast memory and strongest intuition. Not actually good at reasoning, but performs best due to low hallucinations.
Canvas: Overall inferior to Cursor. Context output too small, text length limited. Suitable for short-text scenarios with real-time editing.
Work with Apps on macOS: Functionality experience crushed by Cursor, essentially a simplified version of Cursor.

Claude

Claude 3.7 Sonnet: Excellent, suitable for generating various code. Web version supports massive context, almost matching API’s context history message retention. Model very suitable for writing communication texts like emails, with accurate, concise, and unpretentious wording. Drawback: limited usage in free version.

DeepSeek

R1: Severe hallucinations, unstable performance, occasionally affecting usability, requires very high quality prompts. When prompts are correct or context is complete, it outperforms o1. Can be used as input for o1’s output to optimize quality.
V3: Very excellent, alternative to GPT-4o.

Grok

Grok-3: Generates text fluently and naturally, less AI-like, suitable for natural writing and novel creation. Overall mediocre, fewer productivity tools than GPT-4o.
Grok-3 + search: Excellent. Leverages English social media data, quickly analyzes news and current events. Outputs long content, barely filters search source content, stronger search capabilities than GPT-4o.
Grok-Deep Search: Slightly redundant, actually inferior to Grok-3 + search. Generated content heavily templated, affecting quality.

Gemini

2.5 Pro: Excellent model, alternative to GPT-4o, minimal hallucinations, high-quality search results, complete and clear thought chains, strong logical reasoning. Supports integration with Google tools, very useful in specific scenarios, such as uploading screenshots or text to automatically create events in Google Calendar. Drawback: often claims to have performed searches when it actually hasn’t.
Deep Research: Average, slightly better than old GPT Search. Occasional comprehension deviations, search and document generation executed in stages, process fragmented, heavily templated. Currently the only product that can replace GPT Deep Research.

v0.dev

Suitable for writing and previewing frontend components online.

chat.qwen.ai

Qwen2.5-Max: Very excellent model, alternative to GPT-4o, fast generation speed, supports thinking mode.

Zhihu Direct Answer

Supports searching Zhihu’s entire network content. Possible competitor is Xiaohongshu Direct Answer.

Perplexity

Supports basic search, mediocre quality, slightly redundant.

Mistral.ai

Fast speed, large context window, alternative to GPT4o-mini.

Cursor

Excellent tool. Can index entire code repositories, suitable for large project development. Supports multiple model switching, local command line execution and terminal takeover, suitable for the following tasks:
- Writing LaTeX documents, replacing Overleaf
- Assisting with various assignments
- Code repository search and specific feature location
- Creating unit tests
- Executing git operations
- Executing deployment tasks
Can execute almost everything involving documents and command lines, helpful in development, debugging, debugging, and document writing
Truly the most Agent-like product at the current stage (though this claim no longer holds after the release of o3 and o4-mini, as they can also perform multi-step planning and task execution. However, o3 and o4-mini can only operate within their own limited environment, while Cursor can execute on the user’s computer, which is an advantage.)

Example: Leetcode Tracker

There are many shared Leetcode premium accounts on Taobao, but sometimes we still need to add our problem-solving records to our own accounts. After purchasing a shared premium account, we can export a company’s problems as a CSV file, then export our own problem-solving records as another CSV file. By comparing these two files, we can easily track our current progress. Cursor is very good at writing such small tools:

We just need to copy elements from the webpage, and Cursor can write complete CSV files for us and automatically generate a perfect frontend interface, greatly improving our problem-solving efficiency.

How to install Rodin software on a MacBook with an Apple chip

Sat, 01 Feb 2025 13:01:39 +0000

Steps to Install Rodin on macOS (M1 ARM Architecture):

Install x86 JDK
- Download the Intel x86 JDK 18 (macOS .dmg) from Oracle:
  
  jdk-18.0.2.1_macos-x64_bin.dmg
- Run the installer; it automatically configures the Java environment.
Download and Install Rodin
- Get Rodin 3.8/3.9 (x86_64 version) from SourceForge:
  
  rodin-3.8.0.202304051545- macosx.cocoa.x86_64.tar.gz
- Extract the .tar.gz file and move Rodin.app to the /Applications folder.
Fix macOS Security Permissions

Run this command to bypass “app is damaged” errors:

sudo xattr -cr /Applications/Rodin.app

Rodin should be able to run without any additional steps (e.g., configuring the Java VM path).

Study of Differential Attack on FEAL-4

Wed, 01 Jan 2025 13:15:04 +0000

This article primarily introduces how to perform differential cryptanalysis on the traditional block cipher algorithm FEAL-4, and demonstrates the attack process and key implementation details. The text also includes some experimental data and attack results for interested readers to reference or reproduce.

Two Important Images

Before delving into the analysis, here are two images closely related to FEAL-4 to provide an intuitive understanding of differential propagation and the overall encryption process.

Differential Propagation Diagram
The figure below shows how plaintext differences (in XOR form) propagate through the encryption rounds in FEAL-4 and ultimately reflect in the ciphertext.
FEAL-4 Workflow Diagram
The following figure illustrates the overall encryption framework of FEAL-4, including the left and right blocks of the plaintext, the injection of round subkeys, and the usage flow of the F function.

Characteristics of the F Function

The round function of FEAL-4 (i.e., the F function) can be viewed as some form of random permutation on a 32-bit input. In differential analysis, we do not need the specific implementation of F, but rather focus on its two key differential properties:

If $X \oplus Y = 0$, then $F(X) = F(Y)$.
If $X \oplus Y = 0x80800000$, then $F(X) \oplus F(Y) = 0x02000000$.

With these two conclusions, we can infer the output behavior under specific differential inputs in subsequent analyses, thereby aiding in locating the subkeys.

Detailed Analysis of Differential Paths

Suppose we choose a pair of special plaintexts $P_0$ and $P_1$ such that: $$ P_0 \oplus P_1 = 0x8080000080800000 = P^{\prime}. $$ This specific differential vector helps simplify the differential computations in several rounds of FEAL-4 outputs.

1. Derivation of Initial Round Differences

Let $(L0_0, R0_0)$ denote the left and right blocks of $P_0$ (similarly, $(L0_1, R0_1)$ denotes those of $P_1$).
After the XOR operation in the first round: $$ L1_0 \oplus L1_1 = (L0_0 \oplus K4) \oplus (L0_1 \oplus K4) = L0_0 \oplus L0_1 = 0x80800000. $$ Similarly, the differential of the right half can be derived.
Furthermore, due to the “additional XOR round” during encryption, we obtain: $$ R2_0 \oplus R2_1 = (R1_0 \oplus L1_0) \oplus (R1_1 \oplus L1_1) = 0x80800000 \oplus 0x80800000 = 0. $$ This implies that in subsequent computations, the differences can be merged in a simpler form.

2. Backward Differential Calculation

After obtaining the plaintext pair $(P_0, P_1)$ and the corresponding ciphertext pair $(C_0, C_1)$, further backward derivations can be made. Let the ciphertext be denoted as $(L, R)$, then:

Compute $C^{\prime} = C_0 \oplus C_1$, obtaining $(L^{\prime}, R^{\prime})$.
Based on the known differential properties, gradually restore intermediate variables such as $X^{\prime}$, $Y^{\prime}$, $Z^{\prime}$, etc., in the last round.
Using these results, attacks or verifications on the subkeys can be performed.

Attacking K3

In differential analysis, one often starts with attacking an intermediate round subkey, such as $K_3$. The specific steps are as follows:

Select Plaintext Pairs: I generated 12 pairs of random plaintexts satisfying
$$ P_0 \oplus P_1 = 0x8080000080800000 $$
Encrypt and Extract Differences: Encrypt the above plaintext pairs to obtain the corresponding ciphertext pairs; combine with the differential path to derive intermediate values $Y_0$, $Y_1$, $Z^{\prime}$, etc.
Traverse $K_3$:
$$ Z^{\prime} = Z_0 \oplus Z_1 = F(Y_0 \oplus K_3) \oplus F(Y_1 \oplus K_3). $$
Any candidate $K_3$ that satisfies the differential equation $= 0x02000000$ is retained.

Using this method, I successfully obtained four candidate subkeys:

cfa38976, cfa309f6, 4f238976, 4f2309f6

An example command to execute the above process is:

./main -mode=attackk3 -file=K3.txt

This implementation is consistent with the method I introduced on www.theamazingking.com.

Attacking K2

After obtaining several candidates for $K_3$, we can proceed to deduce $K_2$. The core equation is: $$ X’ = X_0 \oplus X_1 = F(U_0 \oplus K_2) \oplus F(U_1 \oplus K_2), $$ where $U$ comes from the previous round operations (including further computations on $Y$, $Z$, etc.). Similarly:

Generate plaintext pairs that satisfy the differential condition $$ P_0 \oplus P_1 = 0x0000000080800000. $$
Capture ciphertext differences after encryption, and by traversing all possible $K_2$ values, check if the differential constraint $=0x02000000$ is satisfied.

Example command:

./main -mode=attackk2 -file=K2.txt -k3=cfa38976,cfa309f6,4f238976,4f2309f6

The final output example is as follows:

Candidate K2: 8b722e41 (K3: cfa38976)
Candidate K2: 8b72aec1 (K3: cfa38976)
...
Candidate K2: 89722e43 (K3: 4f2309f6)
Candidate K2: 8972aec3 (K3: 4f2309f6)

Attacking K1

Similarly, attacking $K_1$ can be done following the previous process.
To ensure the differential path consistently presents the expected results, I generated 12 plaintext pairs satisfying
$$ P_0 \oplus P_1 = 0x0000000002000000 $$
and verified their encryption outputs.

Example command:

./main -mode=attackk1 -file=K1.txt -k3k2="cfa38976,8b722e41; cfa38976,8b72aec1; ..."

This allows you to input all previously obtained candidate combinations of $K_3$ and $K_2$ at once, and finally find the corresponding $K_1$ candidates.

Attacking K0, K4, K5

Once $K_3$, $K_2$, and $K_1$ are all obtained, $K_0$, $K_4$, and $K_5$ required for the last round can be deduced.
Let the final encrypted left and right parts be $L_0$, $R_0$. The corresponding plaintext blocks are $PL$, $PR$. Then:

$PL \oplus K_4 = LR_0$
$PR \oplus K_5 = RR_0$
$RR_0 \oplus LR_0 = R_0$
$f(R_0 \oplus K_0) \oplus L_0 = LR_0$

By enumerating all possible $K_0$, the corresponding $K_4$ and $K_5$ can be solved through the above equations. Then, use other plaintext-ciphertext pairs to cross-validate correctness. If all are satisfied, the correct final key set is obtained.

Example command:

./main -mode=attackk0k4k5 -k3k2k1="..." -file=K1.txt -file2=K1_p.txt

Implementation, Optimization, and Verification

1. Automated Generation and Parallel Computing

Automatically Generate Random Plaintext Pairs: Control the XOR differences in the code in advance to batch output plaintext files that meet the requirements (e.g., K3_p.txt, K2_p.txt, K1_p.txt).
Parallelized Search: Use Go’s Goroutines to start 10 parallel tasks, maximizing CPU resource utilization on a 10-core Mac.
Progress Bars and Visualization: Use the progressbar library to display search progress in real-time, making the experimental process more intuitive.

2. Verification Methods

Local Custom Key Testing
First, select a set of known $K_0$–$K_5$ (e.g., 0x00000001, 0x00000002, 0x00000004, 0x00000008, 0x00000010, 0x00000020), encrypt the previously generated plaintext pairs, and repeat the differential analysis process to see if this set of keys can be matched among the results.
Experiments confirmed the ability to correctly restore this set of test keys, thereby verifying the accuracy of the code logic.
Specific Plaintext-Ciphertext Pair Testing
Select a plaintext 1234567890abcdef and the ciphertext f43ae3eeb56e2bbf generated in the Einstein Zone, and verify that each of the 256 candidate $K_0$–$K_5$ sets can reproduce this encryption mapping, further confirming the attack process is correct.

Final Results

All deduced $K_0$–$K_5$ are stored in final_result.txt. After deduplication, they include the following ranges:

Possible Values for K0

890c2148 890ca1c8 098c2148 098ca1c8 ...
0b8c214a 0b8ca1ca 8b0c214a 8b0ca1ca

Possible Values for K1

471f077e 471f87fe c79f077e c79f87fc ...
451f077c 451f87fc c59f077c c59f87fc

Possible Values for K2

8b722e41 8b72aec1 8b722e43 8b72aec3 ...
89722e43 8972aec3

Possible Values for K3

cfa38976 cfa309f6 4f238976 4f2309f6

Possible Values for K4

89eb0024 89eb0026 8beb0024 8beb0026

Possible Values for K5

b85e6bc0 b85e6bc2 ba5e6bc0 ba5e6bc2

With the aid of differential analysis, we ultimately obtained a set of candidate subkeys. Compared to modern block ciphers (such as AES), FEAL-4 has fewer rounds and a simpler structure, making it very suitable for demonstrating the concept of differential analysis in teaching and research.

Conclusion

This article detailed the complete process of performing a differential attack on FEAL-4, including designing differential paths, generating plaintext pairs, attacking intermediate subkeys ($K_3$, $K_2$, $K_1$), and finally reverse-solving for $K_0$, $K_4$, $K_5$. It also introduced several optimizations and verification methods during the code implementation process.
The FEAL-4 algorithm, due to its fewer rounds and simpler structure, provides convenience for differential analysis. However, for more secure and complex modern block algorithms, the same analytical approach remains crucial. This case study aims to inspire learners and researchers in cryptography. If you have any questions or improvement suggestions, feel free to discuss and exchange ideas in the comments section!

How to Quickly Solve Disassembly Problems

Sun, 03 Nov 2024 14:21:43 +0000

For DCU’s Secure Programming course, the disassembly problems have a certain pattern. Using a fixed approach to solving them can help achieve quick results.

Prerequisite Skills

Familiarity with assembly commands Assembly Instructions
Understanding % and $ for registers and immediate values $ and % Registers and Immediate Values
Knowledge of direct and indirect addressing Direct and Indirect Addressing
Familiarity with one example C Code to Assembly Example

Approach to Solving

Identify the Number of Parameters

The position of ebp is the saved frame pointer, and ebp+4 is the return address. Since the problems typically assume all parameters are of type int or int*, ebp+8, ebp+c, and ebp+10 correspond to the first, second, and third parameters, respectively.

By quickly scanning the code for occurrences of 0x__(%ebp) and identifying the largest offset, the number of parameters can be determined as (offset - 4) // 4.

For example:

push %ebp <foo+0>
mov %esp, %ebp <foo+1>
sub $0x4, %esp <foo+3>
mov 0x8(%ebp), %eax <foo+6>
mov %eax, -0x4(%ebp) <foo+9>
mov -0x4(%ebp), %eax <foo+12>
cmp 0x10(%ebp), %eax <foo+15>
jge <foo+32> <foo+18>
mov 0xc(%ebp), %eax <foo+20>
incl (%eax) <foo+23>
lea -0x4(%ebp), %eax <foo+25>
incl (%eax) <foo+28>
jmp <foo+12> <foo+30>
mov $0x0, %eax <foo+32>
leave <foo+37>
ret <foo+38>

Here, 0x10(%ebp) exists, so the parameter count is (16 - 4) / 4 = 3.

We can construct the framework of the code as:

int foo(int a, int b, int c) {

}

a, b, and c correspond to ebp+8, ebp+c, and ebp+10, respectively. Note that parameters are pushed onto the stack in reverse order, so the closer to ebp, the earlier the parameter appears in the list.

For now, assume all are int types. Adjust later if inconsistencies are found.

Identify the Number of Local Variables

The number of local variables is determined by the third line of the code: sub $0x4, %esp. The amount subtracted corresponds to the length of the allocated local variables.

In this example, sub $0x4, %esp indicates 4 bytes, so there is one local variable. Assume it is an int and name it i.

The code expands to:

int foo(int a, int b, int c) {
 int i;
}

Recognize the Loop Body

Loops are typically while or for loops. To identify:

Judgment Entry:
- Look for a comparison instruction (e.g., cmp) followed by a jump instruction (e.g., jge or jle).
- These indicate the start of a condition check.
Loop Body:
- Unconditional jmp instructions signify loops. The jump target is the beginning of the condition check.
Condition:
- The judgment condition combines the comparison and preceding instructions into a complete condition.

Example:

push %ebp <foo+0>
mov %esp, %ebp <foo+1>
sub $0x4, %esp <foo+3>
mov 0x8(%ebp), %eax <foo+6>
mov %eax, -0x4(%ebp) <foo+9>
mov -0x4(%ebp), %eax <foo+12>
cmp 0x10(%ebp), %eax <foo+15>
jge <foo+32> <foo+18>
mov 0xc(%ebp), %eax <foo+20>
incl (%eax) <foo+23>
lea -0x4(%ebp), %eax <foo+25>
incl (%eax) <foo+28>
jmp <foo+12> <foo+30>
mov $0x0, %eax <foo+32>
leave <foo+37>
ret <foo+38>

Judgment Entry:

The combination of cmp and jge indicates a judgment entry.

Loop:

The jmp command jumps to <foo+12>, signifying the loop condition check.

Condition:

mov -0x4(%ebp), %eax: Assigns the value of i to eax.
cmp 0x10(%ebp), %eax: Compares eax (value of i) with c.

This calculates i - c and checks the condition with jge. In assembly, conditions are reversed compared to C: jge skips the loop if the condition is met. Thus, the C condition is i - c < 0.

The code updates to:

int foo(int a, int b, int c) {
 int i;
 while (i - c < 0) {

 }
}

Analyze Remaining Code Snippets

Before the Loop:

mov 0x8(%ebp), %eax <foo+6>
mov %eax, -0x4(%ebp) <foo+9>

These lines assign the value of a to i:

i = a;

Inside the Loop:

mov 0xc(%ebp), %eax <foo+20>
incl (%eax) <foo+23>
lea -0x4(%ebp), %eax <foo+25>
incl (%eax) <foo+28>

mov 0xc(%ebp), %eax and incl (%eax) increment the value at the address stored in b:

(*b)++;

lea -0x4(%ebp), %eax and incl (%eax) increment i:

i++;

The updated code becomes:

int foo(int a, int *b, int c) {
 int i;
 i = a;
 while (i - c < 0) {
 (*b)++;
 i++;
 }
}

Identify the Return Value

In x86 calling conventions, return values are stored in the eax register.

mov $0x0, %eax

This indicates the function returns 0:

return 0;

Final Code

int foo(int a, int *b, int c) {
 int i;
 i = a;
 while (i - c < 0) {
 (*b)++;
 i++;
 }
 return 0;
}

How to handle the inconsistency in LaTeX formatting from ChatGPT between Obsidian and Craft.

Sat, 26 Oct 2024 12:59:21 +0000

In a previous article, I used Obsidian’s QuickAdd to create a script that automatically converts text copied from ChatGPT and fixes the LaTeX formatting. However, there is no suitable plugin available for the Craft app.

We can use Raycast to achieve this functionality uniformly.

Create a Raycast Script

First, we need to create a script.

Next, select the Bash template.

Then, we edit the Bash script and enter the following code:

#!/bin/bash

# Required parameters:
# @raycast.schemaVersion 1
# @raycast.title Copy From ChatGPT
# @raycast.mode silent

# Optional parameters:
# @raycast.icon 🤖
# @raycast.packageName ChatGPT Utils

# Documentation:
# @raycast.description Copy From ChatGPT
# @raycast.author Nansen Li
# @raycast.authorURL nansenli.com

# Get clipboard content
clipboard_content=$(pbpaste)

# Check if content was successfully retrieved
if [ -z "$clipboard_content" ]; then
 echo "Clipboard is empty or inaccessible."
 exit 1
fi

# Process clipboard content
modified_content=$(echo "$clipboard_content" | \
 sed 's/\\\[/$$/g; s/\\\]/$$/g; s/\\( /$/g; s/ \\\)/$/g')

# Write the modified content back to the clipboard
echo "$modified_content" | pbcopy

After creating the script, we need to add the directory containing the script to Raycast.

In this step, select the directory where the script was just created. At this point, we can see the newly created script in Script Commands.

How to Use

After copying a formula from ChatGPT, open the Raycast panel, find the newly created script, and run it. The clipboard content will be automatically fixed. Then, simply paste it into Obsidian or Craft.

Summary of Huawei Ireland Research Center Tech Arena 2024 Competition

Thu, 24 Oct 2024 10:01:27 +0000

Background

I’m Nansen, and I participated in the 2024 Huawei Ireland Research Center Server Cluster Management Optimization Competition. Here, I’d like to share my experience in this competition and summarize some key takeaways.

Our algorithm code can be found here: huawei2024

Competition Results

We achieved first place in the algorithm section, scoring approximately 4%-5% higher than the second to fourth places, giving us a significant advantage. However, we faced considerable challenges in the presentation segment. First, we recognized that there is room for improvement in our English fluency. Second, we found that our presentation slides could be more polished and visually appealing. Lastly, we encountered some challenges with time management. Nevertheless, despite these obstacles, we still managed to secure third place overall.

Competition Process

The competition was divided into two stages. The first stage allowed ample preparation time. Once we settled on using the simulated annealing algorithm, we began developing it. The main difficulty in this stage was optimizing and understanding the requirements of the task. During development, we encountered numerous bugs, but after fixing them, our score improved significantly.

In the second stage, as the problem was released on the day of the competition, I continued optimizing the algorithm from the first stage, successfully increasing the evaluation speed by 1000 times. This significantly boosted our performance in the second stage, providing us with enough strength to vie for first place.

In the final round, our algorithm performed very consistently, and after some adjustments, we took a considerable lead over our competitors. However, because we didn’t put enough emphasis on making an effective presentation, we mistakenly believed that high algorithm performance alone would guarantee a top score, which proved to be wrong.

Lessons Learned

Algorithm Choice

Fortunately, I chose the right algorithm from the outset, and shortly after the problem was released, I devised a framework that suited the entire competition. However, I did take some wrong turns, such as attempting impractical algorithms like PPO. After initial trials failed, I should have moved on instead of wasting further effort. Given the limited time, we should focus on achievable optimal results within the shortest period rather than pursuing ideal but unrealistic solutions. It’s also crucial to recognize one’s limitations and concentrate on goals that are achievable in the available time.

Team Collaboration

Luckily, our team division was reasonable this time, and I did my best to ensure every member could contribute their value. One area for improvement is communicating more with team members to understand their ideas and preferences. Since I mainly handled the algorithm part, I had relatively little interaction with teammates, which I will work to improve next time.

Presentation Design

We didn’t anticipate that the level of presentation skills from other teams would be so high. My teammates speculated that some competitors might have a business background, giving them an advantage in crafting presentations. Moreover, they had five members in their team while we only had three, which put us at a disadvantage regarding manpower. These were objective challenges, but if we had paid more attention to creating our presentation, the first prize could have been within our reach.

Over-committing Leading to Imbalance

In the final round, our algorithm was already quite excellent, and our score surpassed the previously top-ranked team. However, I continued spending considerable time on further optimizations. Even though we were significantly ahead, this focus caused us to neglect the preparation of our presentation. In hindsight, I should have known when to stop and fully recognized the importance of balancing different aspects of the scoring criteria.

Conclusion

Participating in the Huawei Tech Arena 2024 competition provided me with invaluable experience. The competition highlighted our strengths, but also revealed areas where we need to improve in terms of showcasing skills and team collaboration. Looking ahead, I will keep these lessons in mind and strive to continuously improve myself in future competitions. If you have any questions, feel free to leave them in the comments section.

Leetcode Notes

Mon, 21 Oct 2024 19:39:42 +0000

Today, I began the long journey of practicing Leetcode problems. Previously, I only did a few problems to maintain familiarity, but today, I officially started preparing for interviews.

I have been thinking about how to efficiently solve Leetcode problems. In my opinion, to be efficient, one must memorize problems. Just as reading a book a hundred times reveals its meaning, training a language model through extensive practice hones its coding skills. Similarly, with Leetcode, through repeated practice, the answers will come naturally; quantity brings quality.

53. Maximum Subarray

The solution can be approached using Kadane’s Algorithm. The code is as follows:

def maxSubArray(nums):
 max_current = max_global = nums[0]
 for num in nums[1:]:
 max_current = max(num, max_current + num)
 max_global = max(max_global, max_current)
 return max_global

This is a classic dynamic programming problem, and the above algorithm actually hides the essence of dynamic programming.

class Solution {
public:
 int maxSubArray(vector<int>& nums) {
 vector<int> dp(nums.size());
 dp[0] = nums[0];
 for (int i = 1; i < nums.size(); i++) {
 dp[i] = max(dp[i - 1] + nums[i], nums[i]);
 }
 return *max_element(dp.begin(), dp.end());
 }
};

This code better reflects the essence of dynamic programming.

To understand the formula dp[i] = max(nums[i] + dp[i-1], nums[i]), we can analyze it from a dynamic programming perspective. The core idea here is to make the optimal choice at each position. Here is a detailed explanation:

1. What does `dp[i]` represent?

dp[i] represents the maximum subarray sum ending at position i.

2. Why compare `nums[i] + dp[i-1]` and `nums[i]`?

The key question is: Should the current maximum subarray include the previous part (dp[i-1])?
- nums[i] + dp[i-1]: If dp[i-1] is positive, adding the current nums[i] will increase the subarray sum, so we choose to add it.
- nums[i]: If dp[i-1] is negative, we choose to start a new subarray from the current position, as a negative sum will only drag down the current sum.

3. Why not compare subsequent numbers?

When making the comparison, we assume the subarray stops at position i. In other words, we consider the maximum value within the range [0:i]. At position i, we either add the previous subarray or abandon it and only use the current number.
We then traverse the entire array, finding the maximum value at each position, and finally return the largest value.

57. Insert Interval

This is a classic interval merging problem, where we need to merge a new interval into existing intervals.

The solution can be broken down as follows:

Step 1: Add all non-overlapping intervals that come before newInterval to the result.
Step 2: Merge all potentially overlapping intervals with newInterval. Note the conditions for merging.
Step 3: Add the remaining intervals to the result.

Note that the condition for merging intervals is that the start of the previous interval is less than or equal to the end of the subsequent interval, i.e., intervals[i][0] <= newInterval[1].

class Solution:
 def insert(self, intervals: List[List[int]], newInterval: List[int]) -> List[List[int]]:
 ret_list = []
 i = 0

 while i < len(intervals) and intervals[i][1] < newInterval[0]:
 ret_list.append(intervals[i])
 i += 1

 while i < len(intervals) and intervals[i][0] <= newInterval[1]:
 newInterval[0] = min(intervals[i][0], newInterval[0])
 newInterval[1] = max(intervals[i][1], newInterval[1])
 i += 1

 ret_list.append(newInterval)

 while i < len(intervals):
 ret_list.append(intervals[i])
 i += 1

 return ret_list

With careful attention to detail, this problem is not difficult.

300. Longest Increasing Subsequence

This is obviously a dynamic programming problem.

dp[i] represents the length of the longest increasing subsequence ending with a certain number.

Each time an element is added, we update the current dp array. If the current number is greater than the previous one, we increment the result by 1.

Note that the first dp[i] starts from index 1.

class Solution:
 def lengthOfLIS(self, nums: List[int]) -> int:
 dp = [1] * len(nums)

 for i in range(1, len(nums)):
 for j in range(i):
 if nums[j] < nums[i]:
 dp[i] = max(dp[i], dp[j] + 1)

 return max(dp)

Complexity Analysis:

Time Complexity: $O(n^2)$, due to the two nested loops.
Space Complexity: $O(n)$, as we need a dp array of length n.

674. Longest Continuous Increasing Subsequence

This is a simple problem, but still worth understanding.

class Solution:
 def findLengthOfLCIS(self, nums: List[int]) -> int:
 dp = [1] * len(nums)
 for i in range(1, len(nums)):
 if nums[i] > nums[i-1]:
 dp[i] = max(dp[i], dp[i-1] + 1)
 return max(dp)

392. Is Subsequence

class Solution:
 def isSubsequence(self, s: str, t: str) -> bool:
 i = 0
 j = 0
 while i < len(s) and j < len(t):
 if s[i] == t[j]:
 i += 1
 j += 1
 return i == len(s)

This problem can be solved using a two-pointer technique, with t as the base. If s contains matching characters, we move forward; if we reach the end of s, it means the match is complete.

115. Distinct Subsequences

This problem asks us to find how many distinct subsequences of string s equal string t.

First, we need to define dp, where dp[i][j] represents the number of distinct subsequences that can be formed from the first i characters of s to form the first j characters of t.

dp[i][0] represents when t is an empty string, the result is 1.
dp[0][j] represents forming t from an empty string, which results in 0.

For dp[i][j], the result depends on the characters at positions i and j:

If they are equal, the result is the sum of the cases where s[i-1] is not matched (dp[i-1][j]) and the cases where it is matched (dp[i-1][j-1]).
If they are not equal, the result is equal to dp[i-1][j].

Note that i, j refer to the first i and j characters.

Additionally, dp[0][0] is initialized to 1, as an empty string is a subsequence of any string.

class Solution:
 def numDistinct(self, s: str, t: str) -> int:
 dp = [[0 for _ in range(len(t) + 1)] for _ in range(len(s) + 1)]
 for i in range(len(s) + 1):
 dp[i][0] = 1

 for j in range(1, len(t) + 1):
 for i in range(1, len(s) + 1):
 if s[i-1] == t[j-1]:
 dp[i][j] = dp[i-1][j] + dp[i-1][j-1]
 else:
 dp[i][j] = dp[i-1][j]

 return dp[-1][-1]

Off-by-One Overflow Attack Study

Tue, 15 Oct 2024 14:18:15 +0000

Off-by-One Overflow Attack Analysis

Background

Last week, I attended a security course that included an example of an off-by-one overflow vulnerability. Here is the original code:

/* Simple off-by-one overflow example */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

void foo(char *input) {
 char buf[1024];
 strncpy(buf, input, sizeof(buf));
 buf[sizeof(buf)] = '\0';
}

void bar(void) {
 printf("I've been hacked\n");
}

int main(int argc, char **argv) {
 if (argc != 2) {
 printf("Usage: %s input_string\n", argv[0]);
 exit(EXIT_FAILURE);
 }
 foo(argv[1]);
 return 0;
}

The answer provided for exploiting this vulnerability is:

perl -e 'system "./obo", "\x38\x84\x04\x08"x256'

The result of running this command is that multiple lines of I've been hacked are printed on the screen.

Analysis

When the program enters the foo function, the memory layout looks like this (as observed using GDB):

From top to bottom, the layout contains the return address, the saved frame pointer, and the buffer (buf).

When the line buf[sizeof(buf)] = '\0'; is executed, the least significant byte of the saved frame pointer (ebp) is set to 0. To ensure that ebp still points within the buf region after being partially overwritten, a buffer of at least 1024 bytes is required. Specifically, ebp needs to be overwritten such that it remains within a reasonable range (— up to 0xff), which is why the buffer is set to 0xff * 4 bytes.

Understanding Assembly Commands on `foo` Return

When the foo function returns, it typically executes the following key assembly instructions:

1. `leave` Instruction

The leave instruction is equivalent to:

mov esp, ebp
pop ebp

mov esp, ebp: This sets the stack pointer (esp) to the value of the frame pointer (ebp), restoring the stack pointer to the top of the current stack frame and effectively releasing the space occupied by the current function.
pop ebp: This pops the value at the top of the stack and assigns it to the frame pointer (ebp), thereby restoring the caller’s frame pointer. Essentially, it writes the return address into ebp, meaning it assigns the stack value (usually the caller’s frame address) to ebp, restoring the caller’s stack frame.

The effect of leave is to restore esp to its state before the function was called and to pop the saved ebp. If ebp has been overwritten to point to a special address (such as an address within the buffer), it can result in an incorrect stack pointer location during function return.

2. `ret` Instruction

The ret instruction pops an address off the top of the stack and jumps to that address:

pop eip

If the return address has been overwritten with the address of the bar function, the execution flow will jump to bar, allowing an attacker to run arbitrary code. Essentially, ret pops an address into the instruction pointer (eip) and jumps to that address to continue execution.

Attack Steps

When the command perl -e 'system "./obo", "\x38\x84\x04\x08"x256' is executed, the program takes these repeated bytes as the input to ./obo.
As the foo function returns, the leave and ret instructions are executed, leading to the return address being overwritten. This causes the program to jump to the bar function, printing the success message multiple times.

Further Analysis: Determining Effective Overwrite Locations

Stack Frame Layout Explanation

During the GDB debugging session, the memory layout for the stack frame of the foo function looks like this:

0xbfffed10 return address
0xbfffed0c saved frame pointer (ebp)
0xbfffed0b buf[1023]
...
0xbfffed03 buf[1015]
0xbfffed02 buf[1014]
0xbfffed01 buf[1013]
0xbfffed00 buf[1012]
...
0xbfffe90c buf[0]

Return address: Located at 0xbfffed10, this is the address that the program will jump to after the foo function finishes executing. Overwriting this address can control the flow of the program and potentially redirect it to malicious code (e.g., the bar function).
Saved frame pointer (ebp): Stored at 0xbfffed0c, this value is used to restore the calling function’s stack frame after foo finishes. In this example, we can see how the off-by-one overflow can overwrite the least significant byte of ebp.
Buffer (buf): The buffer starts at address 0xbfffe90c and extends to 0xbfffed0b, with buf[0] located at 0xbfffe90c and buf[1023] at 0xbfffed0b. The vulnerable line in the code, buf[sizeof(buf)] = '\0';, writes a null terminator (\0) just outside the bounds of this buffer, affecting the saved frame pointer.

In the off-by-one overflow scenario, the write operation overwrites the least significant byte of ebp, which is stored at 0xbfffed0c. By manipulating the value of ebp, we can influence the stack behavior when the leave and ret instructions are executed, eventually allowing us to control the program flow and redirect execution to the bar function.

To perform a successful attack, it’s crucial to determine precisely which bytes need to be overwritten in order to manipulate the control flow effectively. In this example, the overflow occurs when buf[sizeof(buf)] = '\0' is executed, causing the least significant byte of the saved frame pointer (ebp) to be set to 0. Thus, the value of ebp needs to be adjusted to ensure it points back into the buffer area, allowing the execution to proceed in the desired way and eventually jump to the bar function.

Based on further analysis and testing, the following insights were obtained:

To accurately determine the overwrite location, the value of ebp is crucial. However, obtaining this value is challenging because:
1. GDB debugging affects address layout.
2. The length of the input parameter affects the address layout.
Under GDB debugging, the layout within the foo function looks like this:
After executing buf[sizeof(buf)] = '\0';, ebp is modified such that the return address effectively takes the value at ebp + 1, which is the address 0xbfffed00 + 1, or 0xbfffed04.
The corresponding offset is at position 255 in buf, meaning the attack can be constructed by filling in the return address only at that specific location. The following command was used for verification in GDB:
```
r $(perl -e 'print "\x01\x01\x01\x01"x254 . "\x38\x84\x04\x08"x1 . "\x01\x01\x01\x01"x1')
```
This was verified to work under GDB debugging, with some details to note:
1. The input parameter length must always be 256 bytes; otherwise, the value of ebp will change, as the input parameter occupies stack space, affecting the starting position of the frame and thereby affecting the value of ebp.
2. Padding must use non-zero values such as 0x01, because strncpy will terminate early if it encounters a 0 value.
When executing the program directly (i.e., without GDB), the memory layout differs, resulting in a different offset position. Through experimentation, it was found that the offset is at position 235. The corresponding attack command is:
```
./obo $(perl -e 'print "\x01\x01\x01\x01"x234 . "\x38\x84\x04\x08"x1 . "\x01\x01\x01\x01"x21')
```

This achieves the desired effect of accurately finding the overwrite location and successfully executing the attack.

Convert ChatGPT Formula Format to Obsidian-Compatible Format

Mon, 14 Oct 2024 15:24:21 +0000

The formulas generated by ChatGPT use the following format:

\[
Formula Content
\]

However, Obsidian renders formulas using the following format:

$$
Formula Content
$$

When copying a formula from ChatGPT to Obsidian, this difference prevents proper rendering.

Solution

We can create a script for Obsidian to automatically replace the formula format when pasting.

1. Create the Script

We can solve this issue using a plugin in Obsidian.

In your vault, create a file named fixlatex.js under the template directory, and input the following content:

module.exports = async (params) => {
 const { quickAddApi } = params;

 // Get clipboard content
 const clipboardContent = await quickAddApi.utility.getClipboard();

 // Check if content was successfully retrieved
 if (!clipboardContent) {
 new Notice("Clipboard is empty or inaccessible.");
 return;
 }

 const modifiedContent = clipboardContent
 .replace(/\\\[|\\\]/g, '$$$$') // Convert \[ \] to $$ $$
 .replace(/\\\(\s*|\s*\\\)/g, '$$'); // Convert \( \) to $

 // Write the modified content back to the clipboard
 await navigator.clipboard.writeText(modifiedContent);

 new Notice("Clipboard content has been processed and modified!");
};

2. Set Up the Script in QuickAdd

Install the QuickAdd plugin and create a Macro, configuring it as shown below, then save it. The first step in the Macro is to execute our user script fixlatex.js, the second step is to wait for 100 milliseconds, and the third step is to execute the paste action.

Install the Commander plugin and set up the QuickAdd action we just created as a sidebar shortcut. You can also skip this step and directly use an Obsidian command to execute this action.

4. Verify the Effect

Now, on the ChatGPT webpage (currently there seems to be an issue when clicking the copy button in the app), click the copy button, then in Obsidian, click the sidebar shortcut or manually execute the QuickAdd command. This will copy the content from ChatGPT to Obsidian and automatically convert the LaTeX format.

About Me

Fri, 11 Oct 2024 20:55:23 +0000

About Me

Written on October 11, 2024, in Dublin, Ireland.

Self Introduction

Hello everyone, my name is Nansen Li, previously known as Nan Li. I’m from China and a technology-loving engineer. I hold a bachelor’s degree in Mechatronic Engineering and two master’s degrees in Computer Science. I previously worked as a Go Language Engineer at a major internet company in China, and I am currently pursuing a master’s degree at a university in Dublin. I plan to develop my career in Ireland afterward.

I am deeply interested in frontend and backend development, system architecture, algorithms, game development, and generative AI. I’ve always been dedicated to improving my programming skills and enjoy experimenting with new technologies, keeping myself sensitive to industry trends.

In addition to software development and design, I have a diverse background in various fields. I have participated in embedded system design and mechanical software design, soldered circuit boards, and designed electronic components, including amplifiers for detection circuits and functional components like analog or digital cameras. I am familiar with control systems and processes, such as PID. I have also drawn engineering diagrams, such as models of car cylinders, and used AutoCAD for modeling. Additionally, I have been involved in embedded firewall design. I also have experience leading different teams in programming or electronic design competitions.

If you share similar interests in technology or lifestyle, feel free to reach out to me. We could discuss and exchange ideas together.

Classic Kubernetes Installation Process

Sun, 20 Jun 2021 12:21:00 +0000

Run on All Machines

All machines are running Debian. First, install Docker:

 sudo apt-get update
sudo apt-get install -y \
apt-transport-https \
ca-certificates \
curl \
gnupg \
lsb-release
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
"deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
sudo usermod -aG docker ${USER}
sudo systemctl restart docker
sudo su ${USER}

At this point, running docker ps -a should work without errors, indicating Docker was installed successfully.

Next, install Kubernetes components:

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Run on the Master Node

Initialize the control plane:

sudo kubeadm init --apiserver-advertise-address masterip --control-plane-endpoint masterdns --pod-network-cidr 10.244.0.0/16

Save the last part of the output for later use, including instructions on how to configure .kube and the join command for nodes. Follow the instructions to configure the .kube file.

Then initialize the network plugin:

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel.yml

Check if pods are running properly:

kubectl get pods -A

On another node, install Kubernetes components, then run the join command. This command is printed at the end of the kubeadm init process.

Afterwards, check if the pods are running properly.

Download the dashboard:

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml -o dashboard.yaml

Edit dashboard.yaml: Add nodePort: 31001 below port 8443, and change the type to NodePort above it. Find the namespace= line and add a line below it: - –token-ttl=43200 For details, refer to: https://www.huaweicloud.com/articles/dc1dcb0c48cc785a9193c9ce709c8b35.html

Create an admin role:

kubectl create -f https://raw.githubusercontent.com/rootsongjc/kubernetes-handbook/master/manifests/dashboard-1.7.1/admin-role.yaml

Apply the pod:

kubectl apply -f dashboard.yaml

Check the port:

sudo lsof -i:31001

Get the token:

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-token | awk '{print $1}')

Visit https://masterip:31001 and enter ’thisisunsafe’ in Chrome, then input the token.

Socket.io Learning

Tue, 11 Dec 2018 10:50:00 +0000

Introduction

In a hurry, will write more later

Core Concepts

Both the server and client have emit and on functions, which can be considered the core of socket.io. Through emit and on, bidirectional communication between server and client can be easily implemented.

emit: Used to emit or trigger an event. The first parameter is the event name, the second parameter is the data to be sent, and the third parameter is a callback function (generally omitted, but needed when immediate confirmation is required after the other party receives the information). on: Used to listen for an event emitted by emit. The first parameter is the name of the event to listen for, and the second parameter is an anonymous function used to receive data from the other party. The first parameter of this anonymous function is the received data, and if there is a second parameter, it is the function to be returned. socket.io provides three default events (both client and server have them): connect, message, and disconnect. The connect event is automatically triggered when a connection is established with the other party, the message event is triggered when data is received from the other party (usually triggered by socket.send()), and the disconnect event is triggered when the other party closes the connection.

On the server side, distinguish between these three situations:

socket.emit(): Broadcasts to the client that established the connection socket.broadcast.emit(): Broadcasts to all clients except the one that established the connection io.sockets.emit(): Broadcasts to all clients, equivalent to the sum of the above two

Go Types Official Reference

Wed, 14 Nov 2018 08:14:00 +0000

Official Reference

https://golang.org/ref/spec#Types

The types include:

Method sets Boolean types Numeric types String types Array types Slice types Struct types Pointer types Function types Interface types Map types Channel types

Method sets

A type may have a method set associated with it. The method set of an interface type is its interface. The method set of any other type T consists of all methods declared with receiver type T. The method set of the corresponding pointer type *T is the set of all methods declared with receiver *T or T (that is, it also contains the method set of T). Further rules apply to structs containing embedded fields, as described in the section on struct types. Any other type has an empty method set. In a method set, each method must have a unique non-blank method name.

The method set of a type determines the interfaces that the type implements and the methods that can be called using a receiver of that type.

Boolean types

A boolean type represents the set of Boolean truth values denoted by the predeclared constants true and false. The predeclared boolean type is bool; it is a defined type.

Numeric types

A numeric type represents sets of integer or floating-point values. The predeclared architecture-independent numeric types are:

uint8 the set of all unsigned 8-bit integers (0 to 255)
uint16 the set of all unsigned 16-bit integers (0 to 65535)
uint32 the set of all unsigned 32-bit integers (0 to 4294967295)
uint64 the set of all unsigned 64-bit integers (0 to 18446744073709551615)
int8 the set of all signed 8-bit integers (-128 to 127)
int16 the set of all signed 16-bit integers (-32768 to 32767)
int32 the set of all signed 32-bit integers (-2147483648 to 2147483647)
int64 the set of all signed 64-bit integers (-9223372036854775808 to 9223372036854775807)
float32 the set of all IEEE-754 32-bit floating-point numbers
float64 the set of all IEEE-754 64-bit floating-point numbers
complex64 the set of all complex numbers with float32 real and imaginary parts
complex128 the set of all complex numbers with float64 real and imaginary parts
byte alias for uint8
rune alias for int32

The value of an n-bit integer is n bits wide and represented using two’s complement arithmetic.

There is also a set of predeclared numeric types with implementation-specific sizes:

uint either 32 or 64 bits
int same size as uint
uintptr an unsigned integer large enough to store the uninterpreted bits of a pointer value

To avoid portability issues all numeric types are defined types and thus distinct except byte, which is an alias for uint8, and rune, which is an alias for int32. Conversions are required when different numeric types are mixed in an expression or assignment. For instance, int32 and int are not the same type even though they may have the same size on a particular architecture.

String types

A string type represents the set of string values. A string value is a (possibly empty) sequence of bytes. Strings are immutable: once created, it is impossible to change the contents of a string. The predeclared string type is string; it is a defined type.

The length of a string s (its size in bytes) can be discovered using the built-in function len. The length is a compile-time constant if the string is a constant. A string’s bytes can be accessed by integer indices 0 through len(s)-1. It is illegal to take the address of such an element; if s[i] is the i‘th byte of a string, &s[i] is invalid.

Array types

An array is a numbered sequence of elements of a single type, called the element type. The number of elements is called the length and is never negative.

ArrayType = "[" ArrayLength "]" ElementType .
ArrayLength = Expression .
ElementType = Type .

The length is part of the array’s type; it must evaluate to a non-negative constant representable by a value of type int. The length of array a can be discovered using the built-in function len. The elements can be addressed by integer indices 0 through len(a)-1. Array types are always one-dimensional but may be composed to form multi-dimensional types.

[32]byte
[2*N] struct { x, y int32 }
[1000]*float64
[3][5]int
[2][2][2]float64 // same as [2]([2]([2]float64))

Slice types

A slice is a descriptor for a contiguous segment of an underlying array and provides access to a numbered sequence of elements from that array. A slice type denotes the set of all slices of arrays of its element type. The value of an uninitialized slice is nil.

SliceType = "[" "]" ElementType .

Like arrays, slices are indexable and have a length. The length of a slice s can be discovered by the built-in function len; unlike with arrays it may change during execution. The elements can be addressed by integer indices 0 through len(s)-1. The slice index of a given element may be less than the index of the same element in the underlying array.

A slice, once initialized, is always associated with an underlying array that holds its elements. A slice therefore shares storage with its array and with other slices of the same array; by contrast, distinct arrays always represent distinct storage.

The array underlying a slice may extend past the end of the slice. The capacity is a measure of that extent: it is the sum of the length of the slice and the length of the array beyond the slice; a slice of length up to that capacity can be created by slicing a new one from the original slice. The capacity of a slice a can be discovered using the built-in function cap(a).

A new, initialized slice value for a given element type T is made using the built-in function make, which takes a slice type and parameters specifying the length and optionally the capacity. A slice created with make always allocates a new, hidden array to which the returned slice value refers. That is, executing

make([]T, length, capacity)

produces the same slice as allocating an array and slicing it, so these two expressions are equivalent:

make([]int, 50, 100)
new([100]int)[0:50]

Like arrays, slices are always one-dimensional but may be composed to construct higher-dimensional objects. With arrays of arrays, the inner arrays are, by construction, always the same length; however with slices of slices (or arrays of slices), the inner lengths may vary dynamically. Moreover, the inner slices must be initialized individually.

Struct types

A struct is a sequence of named elements, called fields, each of which has a name and a type. Field names may be specified explicitly (IdentifierList) or implicitly (EmbeddedField). Within a struct, non-blank field names must be unique.

StructType = "struct" "{" { FieldDecl ";" } "}" .
FieldDecl = (IdentifierList Type | EmbeddedField) [ Tag ] .
EmbeddedField = [ "*" ] TypeName .
Tag = string_lit .

// An empty struct.
struct {}
// A struct with 6 fields.
struct {
x, y int
u float32
_ float32 // padding
A *[]int
F func()
}

A field declared with a type but no explicit field name is called an embedded field. An embedded field must be specified as a type name T or as a pointer to a non-interface type name *T, and T itself may not be a pointer type. The unqualified type name acts as the field name.

// A struct with four embedded fields of types T1, *T2, P.T3 and *P.T4
struct {
T1 // field name is T1
*T2 // field name is T2
P.T3 // field name is T3
*P.T4 // field name is T4
x, y int // field names are x and y
}

The following declaration is illegal because field names must be unique in a struct type:

struct {
T // conflicts with embedded field *T and *P.T
*T // conflicts with embedded field T and *P.T
*P.T // conflicts with embedded field T and *T
}

A field or method f of an embedded field in a struct x is called promoted if x.f is a legal selector that denotes that field or method f.

Promoted fields act like ordinary fields of a struct except that they cannot be used as field names in composite literals of the struct.

Given a struct type S and a defined type T, promoted methods are included in the method set of the struct as follows:

If S contains an embedded field T, the method sets of S and *S both include promoted methods with receiver T. The method set of *S also includes promoted methods with receiver *T.
If S contains an embedded field *T, the method sets of S and *S both include promoted methods with receiver T or*T.

A field declaration may be followed by an optional string literal tag, which becomes an attribute for all the fields in the corresponding field declaration. An empty tag string is equivalent to an absent tag. The tags are made visible through a reflection interface and take part in type identity for structs but are otherwise ignored.

struct {
x, y float64 "" // an empty tag string is like an absent tag
name string "any string is permitted as a tag"
_ [4]byte "ceci n'est pas un champ de structure"
}
// A struct corresponding to a TimeStamp protocol buffer.
// The tag strings define the protocol buffer field numbers;
// they follow the convention outlined by the reflect package.
struct {
microsec uint64 `protobuf:"1"`
serverIP6 uint64 `protobuf:"2"`
}

Pointer types

A pointer type denotes the set of all pointers to variables of a given type, called the base type of the pointer. The value of an uninitialized pointer is nil.

PointerType = "*" BaseType .
BaseType = Type .
*Point
*[4]int

Function types

A function type denotes the set of all functions with the same parameter and result types. The value of an uninitialized variable of function type is nil.

FunctionType = "func" Signature .
Signature = Parameters [ Result ] .
Result = Parameters | Type .
Parameters = "(" [ ParameterList [ "," ] ] ")" .
ParameterList = ParameterDecl { "," ParameterDecl } .
ParameterDecl = [ IdentifierList ] [ "..." ] Type .

Within a list of parameters or results, the names (IdentifierList) must either all be present or all be absent. If present, each name stands for one item (parameter or result) of the specified type and all non-blank names in the signature must be unique. If absent, each type stands for one item of that type. Parameter and result lists are always parenthesized except that if there is exactly one unnamed result it may be written as an unparenthesized type.

The final incoming parameter in a function signature may have a type prefixed with …. A function with such a parameter is called variadic and may be invoked with zero or more arguments for that parameter.

func()
func(x int) int
func(a, _ int, z float32) bool
func(a, b int, z float32) (bool)
func(prefix string, values ...int)
func(a, b int, z float64, opt ...interface{}) (success bool)
func(int, int, float64) (float64, *[]int)
func(n int) func(p *T)

Interface types

An interface type specifies a method set called its interface. A variable of interface type can store a value of any type with a method set that is any superset of the interface. Such a type is said to implement the interface. The value of an uninitialized variable of interface type is nil.

InterfaceType = "interface" "{" { MethodSpec ";" } "}" .
MethodSpec = MethodName Signature | InterfaceTypeName .
MethodName = identifier .
InterfaceTypeName = TypeName .

As with all method sets, in an interface type, each method must have a unique non-blank name.

// A simple File interface
interface {
Read(b Buffer) bool
Write(b Buffer) bool
Close()
}

More than one type may implement an interface. For instance, if two types S1 and S2 have the method set

func (p T) Read(b Buffer) bool { return … }
func (p T) Write(b Buffer) bool { return … }
func (p T) Close() { … }

(where T stands for either S1 or S2) then the File interface is implemented by both S1 and S2, regardless of what other methods S1 and S2 may have or share.

A type implements any interface comprising any subset of its methods and may therefore implement several distinct interfaces. For instance, all types implement the empty interface:

interface{} Similarly, consider this interface specification, which appears within a type declaration to define an interface called Locker:

type Locker interface {
Lock()
Unlock()
}

If S1 and S2 also implement

func (p T) Lock() { … }
func (p T) Unlock() { … }

they implement the Locker interface as well as the File interface.

An interface T may use a (possibly qualified) interface type name E in place of a method specification. This is called embedding interface E in T; it adds all (exported and non-exported) methods of E to the interface T.

type ReadWriter interface {
Read(b Buffer) bool
Write(b Buffer) bool
}
type File interface {
ReadWriter // same as adding the methods of ReadWriter
Locker // same as adding the methods of Locker
Close()
}
type LockedFile interface {
Locker
File // illegal: Lock, Unlock not unique
Lock() // illegal: Lock not unique
}

An interface type T may not embed itself or any interface type that embeds T, recursively.

// illegal: Bad cannot embed itself
type Bad interface {
Bad
}
// illegal: Bad1 cannot embed itself using Bad2
type Bad1 interface {
Bad2
}
type Bad2 interface {
Bad1
}

Map types

A map is an unordered group of elements of one type, called the element type, indexed by a set of unique keys of another type, called the key type. The value of an uninitialized map is nil.

MapType = “map” “[” KeyType “]” ElementType . KeyType = Type . The comparison operators == and != must be fully defined for operands of the key type; thus the key type must not be a function, map, or slice. If the key type is an interface type, these comparison operators must be defined for the dynamic key values; failure will cause a run-time panic.

map[string]int map[*T]struct{ x, y float64 } map[string]interface{} The number of map elements is called its length. For a map m, it can be discovered using the built-in function len and may change during execution. Elements may be added during execution using assignments and retrieved with index expressions; they may be removed with the delete built-in function.

A new, empty map value is made using the built-in function make, which takes the map type and an optional capacity hint as arguments:

make(map[string]int) make(map[string]int, 100) The initial capacity does not bound its size: maps grow to accommodate the number of items stored in them, with the exception of nil maps. A nil map is equivalent to an empty map except that no elements may be added.

Channel types

A channel provides a mechanism for concurrently executing functions to communicate by sending and receiving values of a specified element type. The value of an uninitialized channel is nil.

ChannelType = ( “chan” | “chan” “<-” | “<-” “chan” ) ElementType . The optional <- operator specifies the channel direction, send or receive. If no direction is given, the channel is bidirectional. A channel may be constrained only to send or only to receive by conversion or assignment.

chan T // can be used to send and receive values of type T chan<- float64 // can only be used to send float64s <-chan int // can only be used to receive ints The <- operator associates with the leftmost chan possible:

chan<- chan int // same as chan<- (chan int) chan<- <-chan int // same as chan<- (<-chan int) <-chan <-chan int // same as <-chan (<-chan int) chan (<-chan int) A new, initialized channel value can be made using the built-in function make, which takes the channel type and an optional capacity as arguments:

make(chan int, 100) The capacity, in number of elements, sets the size of the buffer in the channel. If the capacity is zero or absent, the channel is unbuffered and communication succeeds only when both a sender and receiver are ready. Otherwise, the channel is buffered and communication succeeds without blocking if the buffer is not full (sends) or not empty (receives). A nil channel is never ready for communication.

A channel may be closed with the built-in function close. The multi-valued assignment form of the receive operator reports whether a received value was sent before the channel was closed.

A single channel may be used in send statements, receive operations, and calls to the built-in functions cap and len by any number of goroutines without further synchronization. Channels act as first-in-first-out queues. For example, if one goroutine sends values on a channel and a second goroutine receives them, the values are received in the order sent.

Legal Single-Line Dual Dial - Improving Dorm Router Bandwidth Utilization

Sat, 10 Nov 2018 18:08:00 +0000

Introduction

My dormitory has two China Mobile broadband connections via fiber to the home. One is 20Mbps and the other is 30Mbps, both using PPPoE dial-up. I hope to combine these two bandwidths.

Solution 1

Searched for multi-WAN port routers on JD.com, but they are quite expensive. Pass.

Solution 2

Modify my Xiaomi Router Mini by flashing OpenWrt.

Flashing OpenWrt

This isn’t the focus of this article. However, it’s worth noting that OpenWrt firmware can be found and downloaded from the official website. Its repositories are complete and up-to-date, so you don’t need to flash outdated firmware like Pandorabox with unreliable repositories. The router needs to be flashed with a developer version firmware and SSH firmware, then OpenWrt firmware can be flashed via command line. There’s no need to flash the Breed u-boot, which is too risky and can cause loss of the SN.

First Dial-up, Setting Up Domestic Repositories

First, turn on the router, plug in the network cable, access 192.168.1.1, and set the root password on the page.

In the System Software page, click Configuration, replace downloads.openwrt.org with openwrt.proxy.ustclug.org, and submit. In the Network - Interface page, edit the WAN configuration, change the protocol, and configure PPPoE internet access.

Connect to the router via SSH, type opkg update in the command line. Make sure it succeeds. Then type opkg install kmod-macvlan mwan3 luci-app-mwan3. If it doesn’t succeed, try a few more times.

Setting Up a Virtual Network Card

Check the VLAN binding of the WAN interface, which is eth0.2. In the terminal, enter ip link add link eth0.2 name veth0 type macvlan, and then ifconfig veth0 up.

Binding Network Card to Interface

For convenience, modify WAN6 as our second interface, set up the second account’s PPPoE internet account, and then modify its physical setting to veth0.

If the two accounts cannot dial simultaneously at this point, it means single-line dual dial is not supported.

Setting Hop Count

Set the use gateway metric in both the WAN and WAN6 pages. Make sure they are different, for example, one at 10 and one at 11.

Setting Up Load Balancing

Configure the load balance settings in sequence.

Restart Service

Click to restart the service

Testing

Reference: https://acris.me/2017/06/25/Load-balancing-multiple-PPPoE-on-LEDE/

2018-09-20 Go Learning Notes

Wed, 07 Nov 2018 06:54:00 +0000

Go’s pointers and values behave differently when used as receivers and parameters

Methods and pointer indirection When comparing the first two programs, you might notice that functions with pointer parameters must accept a pointer:

var v Vertex ScaleFunc(v, 5) // Compile error! ScaleFunc(&v, 5) // OK However, methods with pointer receivers can be called with both values and pointers:

var v Vertex v.Scale(5) // OK p := &v p.Scale(10) // OK For the statement v.Scale(5), even though v is a value and not a pointer, methods with pointer receivers can still be called directly. In other words, since the Scale method has a pointer receiver, for convenience, Go interprets the statement v.Scale(5) as (&v).Scale(5).

Additionally, Go automatically interprets the . operation on pointers as (*).

Using OpenSSL AES Encryption Algorithm API

Fri, 02 Nov 2018 06:49:00 +0000

Example: Using AES-128-CBC Encryption Algorithm

This example uses pkcs5_encode padding method, which is also compatible with pkcs7_encode


#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <openssl/aes.h>
#include <openssl/rand.h>

// a simple hex-print routine. could be modified to print 16 bytes-per-line
static void hex_print(const void* pv, size_t len)
{
 const unsigned char * p = (const unsigned char*)pv;
 if (NULL == pv)
 printf("NULL");
 else
 {
 size_t i = 0;
 for (; i<len;++i)
 printf("%02X ", *p++);
 }
 printf("\n");
}


void pkcs7_encode(uint8_t *in, uint8_t *out, int inlen, int *outlen, const int enc)
{
 // uint8_t padchr[16] = {0x10, 0x0f, 0x0e, 0x0d, 0x0c, 0x0b, 0x0a, 0x09, 0x08, 0x07, 0x06, 0x05, 0x04, 0x03, 0x02, 0x01};
 if(inlen <=0) return;
 // Add padding
 if (enc == AES_ENCRYPT) {
 *outlen = inlen + 16 - inlen % 16;
 for (int i = 0; i < *outlen; i++)
 {
 if (i < inlen)
 out[i] = in[i];
 else
 out[i] = 16 - inlen % 16;
 }
 }
 // Remove padding
 else if (enc == AES_DECRYPT) {
 *outlen = inlen - (in[inlen-1]);
 for (int i=0; i< *outlen; i++) {
 out[i] = in[i];
 }
 }
}



int main(int argc, char **argv)
{
 int keylength;
 printf("Give a key length [only 128 or 192 or 256!]:\n");
 scanf("%d", &keylength);

 /* generate a key with a given length */

 unsigned char aes_key[16] = {0x00,0x01,0x02,0x03,0x04,0x05,0x06,0x07,0x08,0x09,0x0a,0x0b,0x0c,0x0d,0x0e,0x0f};


 size_t inputslength = 0;
 printf("Give an input's length:\n");
 scanf("%lu", &inputslength);

 /* generate input with a given length */
 unsigned char * aes_input = malloc(inputslength);
 unsigned char * aes_out = malloc(inputslength);

 memset(aes_input, 'X', inputslength);

 /* init vector */
 unsigned char iv_enc[AES_BLOCK_SIZE] = {0}, iv_dec[AES_BLOCK_SIZE]={0};

 const size_t encslength = ((inputslength + AES_BLOCK_SIZE) / AES_BLOCK_SIZE) * AES_BLOCK_SIZE;
 unsigned char * enc_in = malloc(encslength);
 pkcs7_encode(aes_input, enc_in, inputslength, &encslength, AES_ENCRYPT);
 unsigned char * enc_out = malloc(encslength);
 AES_KEY enc_key;
 AES_set_encrypt_key(aes_key, keylength, &enc_key);
 AES_cbc_encrypt(enc_in, enc_out, encslength, &enc_key, iv_enc, AES_ENCRYPT);

 printf("original:\t");
 hex_print(aes_input, inputslength);

 printf("padding:\t");
 hex_print(enc_in, encslength);

 printf("encrypt:\t");
 hex_print(enc_out, encslength);


 unsigned char * dec_out = malloc(encslength);
 AES_KEY dec_key;
 AES_set_decrypt_key(aes_key, keylength, &dec_key);
 AES_cbc_encrypt(enc_out, dec_out, encslength, &dec_key, iv_dec, AES_DECRYPT);
 pkcs7_encode(dec_out, aes_out, encslength, &inputslength, AES_DECRYPT);

 printf("decrypt:\t");
 hex_print(dec_out, encslength);

 printf("depadding:\t");
 hex_print(aes_out, inputslength);

 int8_t buff[100];
 long bufflen;

 return 0;
}

Investigation and Solution of Windows 10 SSD Periodic Stuttering Problems

Wed, 24 Oct 2018 11:47:00 +0000

Introduction

I was originally using an Intel i5-6500 CPU, but after switching to an AMD Ryzen 2700, frequent system freezes started occurring.

Symptoms

During these freezes, I observed that the disk usage in Task Manager was at 100%, while both read and write rates were at 0. The SSD is a generic Lexar Hades 512 GB. In Event Viewer, there were numerous events with ID 129.

Suspecting Hardware Issues

After repeatedly changing SATA ports and reconnecting power cables, the freezes persisted.

Online searches mostly pointed to the root cause being Windows 10’s AHCI driver issues and SSD firmware problems.

Solution Approach 1: Replacing the AHCI Driver - Ineffective

To replace the AHCI driver, I tried downloading chipset drivers from both AMD’s official website and ASUS’s official website, and manually replaced the driver in Device Manager so that it showed as an AMD SATA driver. I also tried various motherboard drivers, but testing proved ineffective, and the problem remained unsolved.

Solution Approach 2: Converting the Drive to RAID Format - Ineffective

To convert to RAID format, I reinstalled Windows, but during the installation process, Windows could not recognize the SSD. According to AMD’s official English manual and official RAID drivers, I manually loaded the RAID drivers, but Windows still couldn’t see the SSD during installation. After multiple attempts, it remained undetected. This might be because volume mode RAID is not supported with only one drive. I abandoned this approach.

Solution Approach 3: Modifying the Registry, Disabling MSI - Ineffective

https://www.pconline.com.cn/win10/739/7395324.html

In essence, when using the StorAHCI.sys driver in certain AHCI modes, some SSDs may not be able to complete input/output operations correctly in “Message Signaled Interrupt” (MSI) mode, ultimately resulting in system freezes with disk usage showing 100% in Task Manager. Of course, the solution is simple: just disable it (MSI Mode) directly in regedit.

　　4. Click Cortana, type “regedit” to launch the Registry Editor, navigate to “HKEY_LOCAL_MACHINE\System\CurrentControlSet\Enum\PCI\xxxxx\Device Parameters\Interrupt Management\MessageSignaledInterruptProperties” (where “xxxxx” is the “device instance path” you just recorded);

　　5. Finally, change the “MSISupported” key value in the right pane from 1 to 0;

Testing proved this ineffective.

Solution Approach 4: Changing Power Settings - Ineffective

Setting the PCI-Express setting to OFF, testing proved ineffective.

Solution Approach 5: Changing LPM - Effective

http://www.sohu.com/a/204645459_493865

https://answers.microsoft.com/en-us/windows/forum/windows_10-hardware-winpc/event-id-129-storahci-resetting-raidport0/7b30c512-6597-438b-80cb-22fb2f85d62e

The content is as follows:

Most SSDs we use today support some power-saving features, such as DIPM/HIPM/Device Sleep.

When a state change request is initiated by the host, we call it HIPM (Host-Initiated LPM), and when it’s directly initiated by the device, we call it DIPM (Device-Initiated LPM). What are they requesting? Requesting to slow down, save energy, or enter Device Sleep.

Both HIPM and DIPM rely on the computer’s LPM technology support. LPM, or Link Power Management, is a function at the physical layer of the SATA interface that can divide the link between the host and storage device into three states: Active state, Partial state, and Slumber state. The goal is to reduce the physical layer bus transmission speed, thus achieving power-saving effects.

LPM is generally enabled by default.

So is enabling LPM good for SSDs?

1. The host sends an HIPM state change command, but due to differing SSD device standards, it may not be well-supported, causing compatibility issues leading to stuttering or drive disconnection;

2. The host sends HIPM state change commands too frequently, causing the SSD device to not respond in time, resulting in stuttering;

3. After LPM is enabled, it increases SATA bus transmission delays, causing high-speed SSDs to become blocked and stutter;

4. Conflicts between the SSD’s internal DIPM and the host’s requested HIPM cause stuttering.

Often, SSD-induced computer stuttering and sudden drive disconnections are caused by LPM.

LPM can help SATA devices achieve power-saving effects. However, it also affects SSD performance.

So, for the sake of power saving, are you willing to pay the price of losing SSD performance?

HDD power consumption is around 10-25W. Perhaps LPM was originally only intended for HDDs.

SSD startup power consumption: 0.1W~~1W (2W), maximum power consumption: 5~~10W, average power consumption: around 4~5W.

Does our power supply care about these few watts from the SSD? Do we care about this bit of electricity cost? Should we even care!

If SSDs don’t need LPM, which harms SSD performance, can we turn it off?

Save as “.reg” format.

Click to run. It will automatically write to the registry, changing the LPM power settings.

If your SSD is experiencing severe stuttering or disconnections, you can try this method.

The main reason for SSD stuttering and disconnections is that the SSD is too poor quality. Indeed, with SSD prices constantly rising today, it’s very difficult to buy a good SSD at a relatively low price.

After testing, this was effective

Solution Approach 6: Changing BIOS Settings, Enabling SATA Hot Plug Functionality - Recommended

https://www.aiweibk.com/5795.html

After enabling the Hot Plug attribute in the motherboard BIOS settings, there will be a side effect: the SSD’s LPM power saving will be turned off as hot-plugging is enabled, thereby avoiding data loss during hot-plugging or loss of response when reconnecting.

If your SSD and motherboard have compatibility issues causing intermittent stuttering, try enabling the Hot Plug option in the motherboard BIOS settings to disable LPM power saving, which might solve the problem.

After testing, this was effective

Conclusion

Why was there no problem with Intel before, but problems arose after switching to AMD? I suspect it’s because the original Intel motherboard was inferior and didn’t support SATA LPM, while Windows 10 has LPM enabled by default. The generic SSD doesn’t support LPM well, so even though its firmware supports LPM, serious issues can occur. Since the motherboard didn’t support LPM, even with Windows enabling LPM, the problem didn’t manifest.

Now with a new motherboard that supports LPM, compatibility issues with the SSD have emerged under default settings. LPM functionality needs to be manually disabled.

2018-09-15 NetEase Interactive Entertainment Interview

Sat, 15 Sep 2018 15:59:00 +0000

Interviewed at a hotel in Chengdu Will update if I remember more

Written Test

Implement a hashmap using open addressing, with int as key and string as value, time limit 30 minutes

First Interview

Purpose and principles of volatile
Differences between dynamic_cast and static_cast
Issues with legal conversions between parent and child classes using dynamic_cast
Virtual destructors and vtables
RTTI (Run-Time Type Information) and type_info
Sizeof for structures
Characteristics of red-black trees; given approximately 12 numbers, provide a possible red-black tree result and draw it
API call process for establishing a server listen, how to implement non-blocking epoll
Principles of epoll, differences in business code logic between edge-triggered and level-triggered
TCP packet sticking (concatenation)
Throwing two eggs from a 100-story building to determine the egg’s hardness, dynamic programming solution method
1 billion players, each with a score ranging from 0 to 100,000, how to determine a player’s ranking
Redis skip list
How memory barriers are implemented in assembly
Two things that every void func() assembly program needs to do at the beginning
How to determine the cause of a memory leak
How to print something before the main function. What other programs execute before the main function.

Second Interview

Differences between Python’s is and == operators, the id function
Differences between Python’s xrange and range
Python iterators
Redis ziplist
Redis TTL expiration implementation
STL deque implementation (I answered list, which was incorrect. Then asked how I would implement a deque)
What happens between running a program in shell and the program outputting “hello world”, in detail
Draw a typical memory layout for a Linux program
Can different threads access each other’s stack space?
A matrix of length x and width y, with holes at the four corners. A ball starts from a certain point and moves indefinitely without friction. Which hole will it enter?
What can be put inside the [] of a lambda expression, how do you think the compiler implements lambda expressions
What is an rvalue (I confused rvalue with rvalue reference here)
The implementation process of malloc, does malloc always call brk to adjust the heap boundary? Introduction to the buddy system
How do HTTPS clients and servers implement protocol selection, what are the commonly used protocols now
The process of client verification of the server in HTTPS

A Discussion on Python Decorators

Tue, 04 Sep 2018 08:54:00 +0000

Reference: http://python.jobbole.com/81683/

Python’s Namespace Any variable is created within its own namespace. If your own namespace doesn’t contain this name, a new one will be created, rather than overriding the upper namespace. When accessing, if your namespace doesn’t contain it, the upper namespace will be accessed.
Python’s Closures Any function is an object. You can create a callable object. If you create a callable function that uses a local variable in its internal process, this involves a closure: a function nested and defined in a non-global scope can remember the enclosing namespace where it was defined.

In simple terms: a child function can use local variables from its parent function, and this behavior is called a closure.

Closure Details in C++ vs Python In most languages, a parent function’s local variables end their lifecycle when the parent function exits. When lambda functions or function objects reference these local variables, problems can arise. If a lambda or function object can remember the current namespace during its definition, that’s a closure. In C++, local variable lifecycles are the same as their parent function, so what to do? Use the mutable modifier, which makes the variable’s lifecycle as long as the function object or lambda expression, otherwise capturing these variables might cause problems. Python captures all objects in the closure, preventing their release until the function object is collected.
Decorators Sometimes we need to modify functions in similar ways. For example, we might want to calculate a function’s execution time by adding timestamps at the beginning and end, then printing at the end. Or we might need to validate parameters and return values. We could call validation functions at the start and end of each function.

Decorators can accomplish this. For example, if we have a function:

def func():
time.sleep(1)

We can use another function that returns a function object with closure to achieve this:

def addTimePrint(func):
 def retFuncObj(*args, **kwargs):
 starttime = time()
 func(*args, **kwargs)
 print(time() - starttime)
 return retFuncObj

Then we need to replace the original func function:

func = addTimePrint(func)

Python provides syntactic sugar: after defining addTimePrint, use @addTimePrint to complete both function definition and decoration:

@addTimePrint
def func():
time.sleep(1)

Redis Cluster Learning

Sun, 02 Sep 2018 14:25:00 +0000

Introduction

Let’s review the key concepts of Redis clusters:

Configuration
- A Redis database system has one master database and multiple slave databases
- The master database doesn’t require any special configuration; slave databases just need to add –slaveof ip port to their configuration
- Slave databases are read-only by default, and even if changed to writable, it has no practical use
- Besides adding to the configuration file, you can also use the slaveof command at runtime to modify the master database
Persistence Review
- RDB: This method forks the Redis process, the child process writes memory to a file, then replaces the previous RDB snapshot
- AOF: This method writes every command to a file. When the AOF file gets too large, it needs to be rewritten after a certain number of commands are appended because some commands are redundant. File writing is cached by the operating system and can be configured. Without configuration, data is written to disk every 30s. You can configure it to synchronize the cache with each command or once per second.

Principles

Initialization Replication
- 1. When a slave database starts, it sends a SYNC command to the master database
- 1. The master database performs a snapshot save, caches commands during the snapshot, then sends both the snapshot and the commands to the slave database
- 1. If disconnected, Redis versions 2.6 and earlier would re-initialize replication. Redis 2.8 and later can just transmit commands during the disconnection period.
- 1. The slave database replaces the temporary snapshot with the RDB snapshot file specified in the configuration. Subsequent operations are consistent with the RDB persistence recovery process.
Replication Synchronization
- Afterward, the master database sends commands that cause database changes to the slave database
- Redis uses optimistic replication strategy, with a window of inconsistency. The master database can be configured to be writable when there are multiple database connections, as well as the maximum allowed disconnection time for databases. This can reduce data inconsistency issues to some extent.

Read/Write Separation for Performance Improvement

Some time-consuming read operations can be accelerated by establishing multiple database slave nodes through replication

Slave Database Persistence

When using slave databases, the master database’s persistence can be disabled. However, when the master database crashes, a role switch between master and slave databases is needed to recover the master database’s content. The slave database uses “slaveof no one”, and the master database uses the slaveof command to set itself as a slave database.

In this case, if the master database crashes, it cannot be restarted immediately. Because if restarted immediately, since the master database has no persistence, it would clear the content in the slave database.

Diskless Replication

Master-slave replication can use RDB persistence for initialization synchronization. When the master database disables RDB snapshots but performs initialization replication, it will still generate an RDB snapshot. At this point, when the master database restarts, it will use the RDB snapshot for recovery. Since the synchronization time point is uncertain, and initialization synchronization may not have happened for a long time, this could lead to recovery at an arbitrary time point.

Additionally, creating an RDB snapshot during initialization synchronization requires disk operations, which may affect synchronization efficiency if the disk is too slow.

Starting from Redis 2.8.18, Redis introduced diskless replication. When this option is enabled, initialization replication will no longer create an RDB snapshot but will send the RDB snapshot content directly over the network.

Incremental Replication

If the master and slave databases disconnect, using initialization replication for the next reconnection would be quite expensive.

Databases generate a unique ID each time they run.
During the replication synchronization phase, the master database places each command in a replay log and records the current command’s offset.
The slave database also needs to record the command offset when receiving commands.
When the master-slave connection is ready, the slave database sends psync to tell the master database it can use incremental replication.
The master database checks if the ID sent by the slave database matches its own. If not, the master database may have restarted.
Then it checks if the slave database’s command offset is in the backlog replay log. If so, it sends the commands from the replay log to the slave database.
The backlog size can be configured, with a default of 1MB. A larger size allows for better tolerance after disconnection and reconnection.

Sentinel

The sentinel system was created to solve the inconvenience of maintaining master-slave databases. Previously, there were two problems: inability to detect when the master database goes down, and the master-slave conversion issue after the master database fails.

When a sentinel process starts, it reads the configuration file. The configuration file includes master-name, ip, and port. When the master node fails, the sentinel will automatically convert one of the slave nodes to become the master node. When the previous master node recovers, it automatically becomes a slave node.

After the sentinel starts, it will periodically perform the following three operations:

Send INFO to master and slave every 10 seconds
Send its own information to the __sentinel__:hello channel of master and slave every 2 seconds
Send PING to master database, slave database, and other sentinels every 1 second

INFO information allows the sentinel to obtain relevant information about the current database. Sending INFO to the master can obtain its slave databases. By sending its information to the channel, all sentinels can receive information about other sentinels monitoring the same database. Sentinels establish connections with each other to send ping information. When a sentinel sends a ping command and receives no reply, it will consider the target subjectively down. If a master database is found to be subjectively down, the sentinel will ask other sentinels if they also consider it subjectively down. When a certain number is reached, exceeding the quorum set in the sentinel startup parameters, the sentinel considers it objectively down.

RAFT Election:

Sentinel A, which discovered the master database to be objectively down, sends a command to other sentinels requesting to be elected as the leader sentinel.
If the requested sentinel hasn’t voted for anyone else, it will agree.
If A finds that the number of agreements exceeds half of the sentinel count and the quorum parameter, then A becomes the leader sentinel.
Otherwise, A waits a random time and reinitiates the request for the next round of elections.

The leader sentinel will perform the following operations:

Check the priority of slave databases; higher priority ones are chosen
Check the replication command offset of slave databases; those with larger offsets are chosen
Choose the slave database with the smaller ID

The selected database will receive a “slaveof no one” command from the leader sentinel to become the master database, and other slave databases will receive slaveof commands to change their master database. Finally, the internal record is updated to mark the stopped database as a slave of the master database, so when it resumes service, it will recover as a slave database.

Deployment

Sentinels are typically deployed on each node. If the Redis cluster is very large, the number of connections between all sentinels and a slave database might be too many, affecting Redis’s response to all sentinels. Therefore, the number of deployed sentinels should not be too high.

Cluster

For horizontal scaling, I became familiar with Codis during my internship at Tencent. Codis consists of Redis proxies and slave nodes. The main principle is to hash the keys and place them in specific Redis slaves. During script execution, scripts are allocated to specific slaves, which might lead to situations where other keys cannot be found.

Besides Codis, there’s also Redis Cluster.

When starting a Redis instance, change cluster-enables to yes in the configuration to enable cluster configuration.

Scripts can be used to conveniently create clusters. New nodes can be added using the cluster meet command with IP and port. After adding a new node, the node will use the gossip protocol to broadcast the node’s information to all nodes.

Slot allocation: After a new node joins the cluster, it has two options: either use the cluster replicate command to replicate each master database and run as a slave database, or request slot allocation from the cluster to run as a master database.

Keys are hashed using crc16. If {*} exists, that part is used for hashing, and then assigned to one of the 16384 slots. The cluster has a slot mapping table at the beginning of creation. Eventually, the data will be stored in the corresponding Redis node.

2018-08-30 Go Learning Notes

Sat, 01 Sep 2018 13:40:00 +0000

In functions, the short assignment statement := can replace var declarations where the type is clear. However, constants must be declared using const
Outside of functions, every statement must start with a keyword (var, func, etc.), therefore := syntax cannot be used outside of functions.
Why Go uses trailing type declarations: https://blog.go-zh.org/gos-declaration-syntax
Go has the following types:

bool
string
int int8 int16 int32 int64
uint uint8 uint16 uint32 uint64 uintptr
byte // alias for uint8
rune // alias for int32
// represents a Unicode code point
float32 float64
complex64 complex128

When you need an integer value, you should use the int type, unless you have a specific reason to use a fixed size or unsigned integer type.
Go does not support implicit type conversion

2018-08-29 Tencent Timi Phone Interview

Wed, 29 Aug 2018 04:32:00 +0000

Three-way handshake and four-way termination
Nagle algorithm
What states does a TCP connection go through when disconnecting
How to choose between multiple processes and multiple threads, considering business needs
How much knowledge of MySQL master-slave clusters
GDB debugging process in a multi-process environment
Introduction to network frameworks like libevent
Differences between select and epoll. Why Java uses select for connection instead of epoll
Inter-process communication methods
Understanding of locks. Implementation of lock-free data structures
How to design shared memory mutex

Solving Incorrect Network Card IP Issues When Building Cross-Provider Kubernetes Clusters on Tencent Cloud

Sun, 22 Jul 2018 13:40:00 +0000

Kubernetes requires fully connected networks for all nodes, so each node should have an accessible IP address. Although each node has a public IP, Tencent Cloud machines don’t show this IP on the network card. This causes nodes to use their network card IP when joining the cluster, making it impossible for other nodes to find them using this IP.

To make the IP visible on the network card, you need to enable Tencent Cloud’s EIP direct pass-through feature. Note that the official script has issues with Debian 10 and doesn’t properly auto-start. The following changes are required:

Add /etc/rc.d/rc.local to the final line of /etc/rc.local

Although Tencent Cloud’s Debian 10 will start /etc/rc.local at boot, it won’t start /etc/rc.d/rc.local, but the provided script writes to the latter.

Reference documentation: https://cloud.tencent.com/document/product/1199/41709

SVN Learning Notes

Sun, 01 Jul 2018 15:01:00 +0000

Introduction

Learning SVN for work purposes. SVN is a C/S type version management system that heavily depends on servers. Most operations are related to the SVN server.

Most Common Operation Flow

Here we assume the server project address is svn://192.168.1.1/shop

Checkout the project
Update: update the server project to local
Commit: submit local changes

How to Deploy an SVN Server

Since SVN is a C/S type software, you need to deploy a server to use it. So we need to set up a server for testing. To keep it as simple as possible, we’ll install VisualSVN software on Windows to act as our server. Note that servers usually run on Linux, but we’re using Windows server software here just for ease of getting started. You can easily download VisualSVN Server software from the official website. I downloaded version 3.9. Use the latest stable version. During installation, there’s a step to select the standard edition; for other steps, just use the defaults. Pay attention to the port selection to avoid conflicts with ports on your machine. The default port is 443, but you can change it to 8443.

Creating a Test Project on the Server

Since an SVN project is a repository, which is a folder, we need to create a folder. Open the D:\Repositories folder and run the command svnadmin create test1 in that directory to create a project. For other users to access this directory, you need to use the svnserve command to start the server, which we’ll discuss later. You also need to configure permissions. Open the svnserve.conf file in the conf directory under test1, uncomment anon-access, and change its value to write. This allows anonymous access with write permissions.

Then run the command in the D:\Repositories directory: svnserve -d -r ./test1 This step opens the SVN server’s listening state.

Common SVN Client Operations

Download and install TortoiseSVN.

Checkout Right-click in the directory you want to synchronize, select repo-browser from TortoiseSVN, enter svn://127.0.0.1, and confirm. Right-click on the folder icon and select checkout, then confirm. The checkout is now successful.
Commit Add a txt file and write something in it. Then right-click and select commit. This will commit to the server.
Update After developer A submits data and the server changes, developer B needs to synchronize. Right-click and select update.
Ignore To exclude files from being committed, right-click on the file and select “add to ignore list”

SVN Versions

SVN versions start from 1 and increment. SVN can roll back versions. SVN can perform branch and merge operations, but these differ significantly from Git’s branch operations. It seems like it creates a new project. The final merge only combines conflicts. This is different from Git’s branch operations, which are based on diffs.

Using Let's Encrypt and Nginx to Configure HTTPS Access

Mon, 04 Jun 2018 09:12:00 +0000

Introduction

To encrypt HTTP data transmission, we need to use HTTPS certificates. Let’s Encrypt provides free certificates.

Configure Nginx Forwarding

We need to configure Nginx on our server so that Let’s Encrypt can verify our domain. Below is the configuration file for Nginx on my server:

server {
 listen 80;
 server_name www.unrealblue.cc;

 location / {
 return 301 https://$host$request_uri;
 }

 location ~ /.well-known/acme-challenge {
 root /var/www/html;
 allow all;
 }
}

Explanation: The first location directive forwards all HTTP requests on port 80 to HTTPS connections. The second location directive means that if /.well-known is accessed, use the /var/www/html directory we provided.

Save this file as default.conf.

Here I used an Nginx Docker container for deployment, so I didn’t install Nginx directly.

FROM nginx
LABEL MAINTAINER linanwx@gmail.com
RUN apt update
RUN apt install -y certbot
RUN mkdir -p /var/www/html
COPY ./default.conf /etc/nginx/conf.d/
EXPOSE 80:80
EXPOSE 443:443
VOLUME ["/etc/letsencrypt/"]

Save the first code segment as default.conf and the second code segment as Dockerfile, then run docker build -t nginx-server .

Run docker run -it --rm -d -p80:80 -p443:443 --net=server-net --name nginx-server -v $PWD/letsencrypt:/etc/letsencrypt nginx-server to start the container in the background.

Generate Certificates

Run docker exec -it nginx-server /bin/bash to enter the container. Run the following commands to generate the certificate:

openssl dhparam -out /etc/letsencrypt/live/dhparams.pem 2048
certbot certonly --agree-tos -a webroot --webroot-path=/var/www/html -d www.unrealblue.cc -m linanwx@gmail.com

The certbot program will access your domain to confirm your domain ownership, which uses the Nginx configuration from the previous step. There are also some parameters that you can understand by checking the help documentation. Additionally, unless you are certain you want to deploy to a production server, use the –test-cert parameter during the testing phase, as there are limits on the number of certificates that can be generated. Another point is generating dhparam. This is a parameter for the Diffie-Hellman key exchange protocol, which takes relatively long to generate. We will use this in Nginx later.

If everything goes well, you will obtain the certificates stored in the /etc/letsencrypt/ directory. Also, because we used a volume when starting the container, the certificates will appear on the host machine.

Configure Nginx HTTPS

Create a new file named https.conf:

# https://www.jianshu.com/p/f7f39cb24423
server {
 listen 443 ssl;
 server_name www.unrealblue.cc;

 ssl_certificate /etc/letsencrypt/live/www.unrealblue.cc/fullchain.pem;
 ssl_trusted_certificate /etc/letsencrypt/live/www.unrealblue.cc/fullchain.pem;
 ssl_certificate_key /etc/letsencrypt/live/www.unrealblue.cc/privkey.pem;
 ssl_dhparam /etc/letsencrypt/live/dhparams.pem;
 ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';

 ssl_prefer_server_ciphers on;
 ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
 ssl_session_cache shared:SSL:50m;
 ssl_session_timeout 1d;
 ssl_stapling on;
 ssl_stapling_verify on;
 add_header Strict-Transport-Security max-age=60;

 location / {
 root /usr/share/nginx/html;
 index index.html index.htm;
 }

 # location = /gfwlist {
 # proxy_pass http://server-handy:8081;
 # }

 error_page 500 502 503 504 /50x.html;
 location = /50x.html {
 root /usr/share/nginx/html;
 }
}

Save this file. Then modify the Dockerfile as follows:

FROM nginx
LABEL MAINTAINER linanwx@gmail.com
RUN apt update
RUN apt install -y certbot
RUN mkdir -p /var/www/html
COPY ./default.conf /etc/nginx/conf.d/
COPY ./https.conf /etc/nginx/conf.d/
EXPOSE 80:80
EXPOSE 443:443
VOLUME ["/etc/letsencrypt/"]

Here, the https.conf file is also copied in.

Restart the container, visit https://127.0.0.1, and you can see the access is successful.

When testing in a local environment, you will be prompted that the certificate is not correct. Click continue.

Learning the socat Command

Wed, 23 May 2018 09:04:00 +0000

Introduction

socat is a powerful and easy-to-use tool that can implement conversions between any types of sockets.

Usage

socat [options] <address> <address>

You can check the help documentation for options.

Address can take several forms:

- STDIN STDOUT: Represents standard input/output, can be replaced with just a dash
/var/log/syslog: Can also be any path (use ./ for relative paths), opens a file as a data stream
TCP:127.0.0.1:1080: Establishes a TCP connection as a data stream, TCP can also be replaced with UDP
TCP-LISTEN:12345: Creates a TCP listening port, TCP can also be replaced with UDP
EXEC:/bin/bash: Executes a program as a data stream

Scenario 1

I have a local file that I want to display in the terminal

socat - /etc/sysctl.conf

Scenario 2

A TCP connection will connect, and I want to see what data will be received socat TCP-LISTEN:12345 -

Scenario 3

I’m a hacker and I want to create a shell proxy on the target machine socat TCP-LISTEN:12345 EXEC:/bin/bash

Scenario 4

I have a UNIX DOMAIN socket locally, and I want to convert it to a TCP SOCKET for use by machines in the local network. How do I do it?

socat TCP-LISTEN:12345,reuseaddr,fork UNIX-CONNECT:/data/deCOREIDPS/unix.domain

When multiple TCP connections come in, it forks one to connect to the domain socket

Scenario 5

Forward local port 80 to a remote destination

socat TCP-LISTEN:80,fork TCP:www.baidu.com:80

Reference: http://brieflyx.me/2015/linux-tools/socat-introduction/

Understanding Google Protocol Buffer

Fri, 18 May 2018 11:10:00 +0000

Introduction

Learning about Google Protocol Buffer before bed

Installing the protobuf compiler

Start the latest Ubuntu Docker image, run apt install protobuf-compiler

Writing a .proto file

nano lm.helloworld.proto

Then input:

package lm; 
message helloworld 
{ 
 required int32 id = 1; // ID
 required string str = 2; // str
 optional int32 opt = 3; //optional field
}

In the example above, the package name is lm, and a message called helloworld is defined. This message has three members: an int32 type id, a string type member str. opt is an optional member, meaning the message may not contain this member.

Compiling the proto file

Epoll Code Example - Analysis of Handy Library's Built-in epoll.cc

Wed, 16 May 2018 07:43:00 +0000

Introduction

There are many C++ network libraries. Handy is a network library with C++11 style that is very helpful for in-depth learning of C++.

Code Analysis

Below is the epoll.cc file from handy/raw_examples. It’s an example of level triggering. This HTTP server returns a static resource “123456” regardless of what kind of request it receives. Compilation: c++ -o epoll epoll.cc, execution: sudo ./epoll. The if (con.writeEnabled) statement in sendRes of the source code seems to have some issues, causing problems when sending large resources. I have modified it to correctly send large files.

/*
* Compilation: c++ -o epoll epoll.cc
* Execution: ./epoll
* Testing: curl -v localhost
*/
/*
Running Effect
Run the epoll program with sudo. This program listens on port 80 at 0.0.0.0 on the local machine, running as an HTTP server
Whenever a connection accesses, it returns the static resource httpRes
LT is the default mode
*/
#include <sys/socket.h>
#include <sys/epoll.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>
#include <stdlib.h>
#include <map>
#include <string>
#include <signal.h>
#include <iostream>
using namespace std;
bool output_log = true;
// A macro for printing errors and exiting
#define exit_if(r, ...) if(r) {printf(__VA_ARGS__); printf("%s:%d error no: %d error msg %s\n", __FILE__, __LINE__, errno, strerror(errno)); exit(1);}
// This function is used to set the specified fd to non-blocking status
void setNonBlock(int fd) {
// First, we get the original flags of the file descriptor
int flags = fcntl(fd, F_GETFL, 0);
exit_if(flags<0, "fcntl failed");
// Then add O_NONBLOCK and set it back
int r = fcntl(fd, F_SETFL, flags | O_NONBLOCK);
exit_if(r<0, "fcntl failed");
}
// A wrapper for epoll_ctl, putting events and fd into ev.
// And setting events to trigger when readable and writable
void updateEvents(int efd, int fd, int events, int op) {
struct epoll_event ev = {0};
ev.events = events;
ev.data.fd = fd;
printf("%s fd %d events read %d write %d\n",
op==EPOLL_CTL_MOD?"mod":"add", fd, ev.events & EPOLLIN, ev.events & EPOLLOUT);
int r = epoll_ctl(efd, op, fd, &ev);
exit_if(r, "epoll_ctl failed");
}
// Try to perform accept operation on fd. If successful, add it to the monitoring list of epoll fd. Set epoll events to trigger when data is written.
void handleAccept(int efd, int fd) {
struct sockaddr_in raddr;
socklen_t rsz = sizeof(raddr);
int cfd = accept(fd,(struct sockaddr *)&raddr,&rsz);
exit_if(cfd<0, "accept failed");
sockaddr_in peer, local;
socklen_t alen = sizeof(peer);
int r = getpeername(cfd, (sockaddr*)&peer, &alen);
exit_if(r<0, "getpeername failed");
printf("accept a connection from %s\n", inet_ntoa(raddr.sin_addr));
setNonBlock(cfd);
updateEvents(efd, cfd, EPOLLIN, EPOLL_CTL_ADD);
}
// Represents a connection. Members include data read from the connection, data written
// Is it okay to use string to store binary content, what happens if \0 is encountered?
// No problem, see https://www.zhihu.com/question/33104941
struct Con {
string readed;
size_t written;
bool writeEnabled;
Con(): written(0), writeEnabled(false) {}
};
// Data structure used to map fd to con
map<int, Con> cons;
string httpRes;
// Send resources
void sendRes(int efd, int fd) {
// First get the connection information
Con& con = cons[fd];
// Request to write when no data is received
// This means that data sent last time might have been sent completely
// Its corresponding file descriptor has been deleted in cons
// Then the epoll signal was triggered
// At this time, close its last send flag
// Then close the buffer send trigger epoll flag
// Only keep it triggered when there is data to read
// Why not do this step when all data is written?
// if (!con.readed.length()) {
// if (con.writeEnabled) {
// updateEvents(efd, fd, EPOLLIN, EPOLL_CTL_MOD);
// con.writeEnabled = false;
// }
// return;
// }
// Calculate the length of data that still needs to be written
size_t left = httpRes.length() - con.written;
int wd = 0;
// Continuously write data until the kernel buffer can't accept any more
while((wd=::write(fd, httpRes.data()+con.written, left))>0) {
con.written += wd;
left -= wd;
if(output_log) printf("write %d bytes left: %lu\n", wd, left);
};
// If there is no data to write, delete this connection. But don't disconnect, just empty the connection information
if (left == 0) {
// close(fd); // Keepalive is used in testing, so don't close the connection. The connection will be closed in the read event
if (con.writeEnabled) {
updateEvents(efd, fd, EPOLLIN, EPOLL_CTL_MOD);
con.writeEnabled = false;
}
cons.erase(fd);
return;
}
// If the kernel buffer is full, can't write anymore
if (wd < 0 && (errno == EAGAIN || errno == EWOULDBLOCK)) {
// Mark it as can continue writing
if (!con.writeEnabled) {
// Wait for it to be able to continue writing, or be readable
// Avoid repeated system calls, use con.writeEnabled flag
printf("update it to EPOLLIN|EPOLLOUT\n");
updateEvents(efd, fd, EPOLLIN|EPOLLOUT, EPOLL_CTL_MOD);
con.writeEnabled = true;
}
return;
}
// If it's other situations, such as returning 0 without completing data writing, or returning other errors
// It means an error occurred
if (wd<=0) {
printf("write error for %d: %d %s\n", fd, errno, strerror(errno));
close(fd);
cons.erase(fd);
}
}
// When loop_once processes read data, call this function
void handleRead(int efd, int fd) {
char buf[4096];
int n = 0;
// Read 4k bytes each time, loop to read out all the data already in the kernel (information may be incomplete due to packet splitting)
while ((n=::read(fd, buf, sizeof buf)) > 0) {
if(output_log) printf("read %d bytes\n", n);
// Here, use a map to get the connection information corresponding to the previous fd.
// When the index corresponding to fd does not exist, it will call the default constructor of con: Con(): written(0), writeEnabled(false) {}
string& readed = cons[fd].readed;
// Call the append method of the string class to add data to the connection information
// Note that parameter n needs to be passed to ensure binary safety
readed.append(buf, n);
std::cout << "now info is" << std::endl << "---" << readed << endl << "---" << std::endl;
// Determine when an HTTP request is complete.
// Don't judge the content of the HTTP request, just send static resources
if (readed.length()>4) {
if (readed.substr(readed.length()-2, 2) == "\n\n" || readed.substr(readed.length()-4, 4) == "\r\n\r\n") {
// When a complete HTTP request is read, test sending a response
// After the TCP connection is established, the client starts transmitting the header, then uses \r\n\r\n to mark the end of the header and the beginning of the entity (of course, there will be the beginning of the entity only if the request contains an entity),
// Then the entity is transmitted, when the entity is transmitted, the client starts receiving data, the server knows, this request has ended,
// Then the entity is that segment of data from \r\n\r\n to stopping reception. Correspondingly, the client receives the response in the same way.
// If there is no entity, then \r\n\r\n is the end of http
// Start writing data. Note that it may fill the buffer, if it's full, continue writing later
sendRes(efd, fd);
}
}
}
// If read cannot read, it will return -1. At this time, errno (errno belongs to the thread, it is thread-safe) is EAGAIN, which means it's not all read. EWOULDBLOCK and EAGAIN are the same.
// Then return, and wait for the next read
if (n<0 && (errno == EAGAIN || errno == EWOULDBLOCK)){
printf("nothing to read from %d, return. \n", fd);
return;
}
// In actual applications, n<0 should check various errors, such as EINTR
if (n < 0) {
printf("read %d error: %d %s\n", fd, errno, strerror(errno));
}
// Executing here, n is 0, indicating that the peer has closed the connection. At this time, we also close the connection
printf("%d close the connection\n", fd);
close(fd);
cons.erase(fd);
}
// When the buffer can be written in loop_once, simply write our prepared static resources
void handleWrite(int efd, int fd) {
sendRes(efd, fd);
}
// Perform one operation in a loop on an epoll handle
// Where l is the LISTEN fd
void loop_once(int efd, int lfd, int waitms) {
// At most copy 20 events out from the kernel
const int kMaxEvents = 20;
struct epoll_event activeEvs[100];
int n = epoll_wait(efd, activeEvs, kMaxEvents, waitms);
// n is how many events were returned
if(output_log) printf("epoll_wait return %d\n", n);
for (int i = 0; i < n; i ++) {
int fd = activeEvs[i].data.fd;
int events = activeEvs[i].events;
// EPOLLIN event or EPOLLERR event. EPOLLERR also means the pipe write ended.
// See: http://man7.org/linux/man-pages/man2/epoll_ctl.2.html
if (events & (EPOLLIN | EPOLLERR)) {
// The EPOLLIN event is only triggered when the peer has data written, so after triggering once, you need to keep reading all the data until you finish reading EAGAIN. Otherwise, the remaining data will only be taken out together the next time the peer writes.
// When the other party closes the connection, it is an EPOLLERR event
if (fd == lfd) {
printf("this is accept\n");
handleAccept(efd, fd);
} else {
printf("this can read\n");
handleRead(efd, fd);
}
} else if (events & EPOLLOUT) {
// This handles events if the file descriptor can be written
// The EPOLLOUT event is only triggered once during connection, indicating it can be written
// Afterwards, it indicates that the data in the buffer has been sent out and can continue to be written
// See https://www.zhihu.com/question/22840801
if(output_log) printf("handling epollout\n");
handleWrite(efd, fd);
} else {
exit_if(1, "unknown event");
}
}
}
int main(int argc, const char* argv[]) {
if (argc > 1) { output_log = false; }
/*
Small Knowledge
signal (parameter 1, parameter 2);
Parameter 1: The signal we want to process. We can view the system signals (64 in total) by typing kill -l in the terminal. In fact, these signals are macros defined by the system.
Parameter 2: The way we handle it (system default, ignore, or capture). SIG_IGN: If the func parameter is set to SIG_IGN, the signal will be ignored.
*/
::signal(SIGPIPE, SIG_IGN);
// Set the content of the HTTP response
httpRes = "HTTP/1.1 200 OK\r\nConnection: Keep-Alive\r\nContent-Type: text/html; charset=UTF-8\r\nContent-Length: 19048576\r\n\r\n123456";
// Fill the rest of the content with 0. The final length of content is about 1024*1024
for(int i=0;i<19048570;i++) {
httpRes+='\0';
}
// Set the port to 80
short port = 80;
// Create an epoll handle
int epollfd = epoll_create(1);
exit_if(epollfd < 0, "epoll_create failed");
// Create a socket
int listenfd = socket(AF_INET, SOCK_STREAM, 0);
exit_if(listenfd < 0, "socket failed");
struct sockaddr_in addr;
memset(&addr, 0, sizeof addr);
addr.sin_family = AF_INET;
addr.sin_port = htons(port);
addr.sin_addr.s_addr = INADDR_ANY;
// First bind the socket to the port
int r = ::bind(listenfd,(struct sockaddr *)&addr, sizeof(struct sockaddr));
// This step will report an error if you don't have superuser permissions. Linux doesn't allow non-root users to use ports below 1024
exit_if(r, "bind to 0.0.0.0:%d failed %d %s", port, errno, strerror(errno));
/*
#include<sys/socket.h>
int listen(int sockfd, int backlog)
Return: 0──success, -1──failure
Parameter sockfd
The socket that the listen function acts on, sockfd was previously returned by the socket function. At the time when the socket function returns the socket fd, it is an active connection socket,
which means the system assumes the user will call the connect function on this socket, expecting it to actively connect with other processes, then in server programming, the user wants this socket to accept external connection requests,
that is, passively wait for users to connect. Since the system assumes by default that a socket is actively connected, it needs to be told in some way, and the user process completes this by making the system call listen.
Parameter backlog
This parameter involves some network details. While a process is handling one connection request, there may be other connection requests.
Because TCP connection is a process, there may be a half-connected state, and sometimes due to too many users trying to connect simultaneously, the server process cannot quickly complete the connection request.
If this situation occurs, how does the server process want the kernel to handle it?
The kernel will maintain a queue in its own process space to track these completed connections that the server process has not yet handled or is processing. Such a queue cannot be arbitrarily large in the kernel,
so there must be an upper limit to its size. This backlog tells the kernel to use this value as the upper limit.
Without a doubt, the server process cannot arbitrarily specify a value, the kernel has a permissible range. This range is implementation-related. It's hard to have some standardization, usually this value will be less than 30.
The length of the queue used by the kernel to track these completed connections but not yet accepted by user code is set to 20 here. When the queue length is less than 20, the kernel will immediately complete the connection establishment.
But if the queue length is greater than 20, the connection will not be established before the user code calls accept, and the other party will be in a blocked state.
*/
r = listen(listenfd, 20);
exit_if(r, "listen failed %d %s", errno, strerror(errno));
printf("fd %d listening at %d\n", listenfd, port);
// Next, set the file descriptor to non-blocking.
// Why set it to non-blocking? https://www.zhihu.com/question/23614342
setNonBlock(listenfd);
// Set it to trigger when readable, add to the epoll file descriptor pool
updateEvents(epollfd, listenfd, EPOLLIN, EPOLL_CTL_ADD);
for (;;) { // Actual applications should register signal handling functions and clean up resources when exiting
loop_once(epollfd, listenfd, 10000);
}
return 0;
}

Running Effect

sudo ./epoll
fd 4 listening at 80
add fd 4 events read 1 write 0
epoll_wait return 1
this is accept
accept a connection from 127.0.0.1
add fd 5 events read 1 write 0
epoll_wait return 1
this can read
read 412 bytes
now info is
---GET / HTTP/1.1
Host: 127.0.0.1
Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36
Upgrade-Insecure-Requests: 1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: zh-CN,zh;q=0.9,und;q=0.8,zh-TW;q=0.7,en;q=0.6,pl;q=0.5
---
write 4081834 bytes left: 14966851
update it to EPOLLIN|EPOLLOUT
mod fd 5 events read 1 write 4
nothing to read from 5, return.
epoll_wait return 1
handling epollout
write 2226422 bytes left: 12740429
epoll_wait return 1
handling epollout
write 2095456 bytes left: 10644973
epoll_wait return 1
handling epollout
write 1964490 bytes left: 8680483
epoll_wait return 1
handling epollout
write 1506109 bytes left: 7174374
epoll_wait return 1
handling epollout
write 1833524 bytes left: 5340850
epoll_wait return 1
handling epollout
write 1637075 bytes left: 3703775
write 130966 bytes left: 3572809
epoll_wait return 1
handling epollout
write 1571592 bytes left: 2001217
epoll_wait return 1
handling epollout
write 1440626 bytes left: 560591
epoll_wait return 1
handling epollout
write 560591 bytes left: 0
mod fd 5 events read 1 write 0
epoll_wait return 1
this can read
read 375 bytes
now info is
---GET /favicon.ico HTTP/1.1
Host: 127.0.0.1
Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36
Accept: image/webp,image/apng,image/*,*/*;q=0.8
Referer: http://127.0.0.1/
Accept-Encoding: gzip, deflate, br
Accept-Language: zh-CN,zh;q=0.9,und;q=0.8,zh-TW;q=0.7,en;q=0.6,pl;q=0.5
---
write 10477280 bytes left: 8571405
update it to EPOLLIN|EPOLLOUT
mod fd 5 events read 1 write 4
nothing to read from 5, return.
epoll_wait return 1
handling epollout
write 1440626 bytes left: 7130779
epoll_wait return 1
handling epollout
write 1768041 bytes left: 5362738
epoll_wait return 1
handling epollout
write 1571592 bytes left: 3791146
epoll_wait return 1
handling epollout
write 1637075 bytes left: 2154071
epoll_wait return 1
handling epollout
write 1702558 bytes left: 451513
epoll_wait return 1
handling epollout
write 451513 bytes left: 0
mod fd 5 events read 1 write 0
epoll_wait return 0
epoll_wait return 0
epoll_wait return 0

Here I increased the size of the resource, changing it to the following value:

 httpRes = "HTTP/1.1 200 OK\r\nConnection: Keep-Alive\r\nContent-Type: text/html; charset=UTF-8\r\nContent-Length: 19048576\r\n\r\n123456";
// Fill the rest of the content with 0. The final length of content is about 1024*1024
for(int i=0;i<19048570;i++) {
httpRes+='\0';
}

You can see it was transmitted in multiple parts. Finally, the terminal page displays 123456, with \0 after it, which won’t be displayed. You can see that the browser made two requests, one for the root directory and one for the page icon favicon.ico </rewritten_file>

2018-05-11 Essay

Fri, 11 May 2018 09:50:00 +0000

Started preparing to write the server side of the SS assistant. There are many issues.

Language If I want to pursue C++ backend development in the future, it’s probably best to stick with C++. But I also want to try other solutions like Python. I tried webpy in Python and found it very lightweight - I probably won’t use the Flask framework I used before. Another option is Go. Go’s syntax is also an issue as I haven’t learned it yet.
Framework I’ve currently found C++’s handy library. I’m not particularly keen on using the muduo library. I still don’t want to use boost, even though Effective C++ recommends learning it. My first impression of handy is quite good. Still learning about it.

The goal is to first write a static server using handy, then expand from there.

C++ Perfect Singleton Pattern

Mon, 07 May 2018 06:15:00 +0000

Original Singleton Pattern

The singleton pattern needs to do the following:

Cannot be constructed through a constructor, otherwise multiple instances could be created. The constructor needs to be declared private.
Ensure only one instance can be produced.

Here is a simple implementation:

class Singleton
{
private:
static Singleton *local_instance;
Singleton(){};
public:
static Singleton *getInstance()
{
if (local_instance == nullptr)
{
local_instance = new Singleton();
}
return local_instance;
}
};
Singleton * Singleton::local_instance = nullptr;
int main()
{
Singleton * s = Singleton::getInstance();
return 0;
}

Using Local Static Objects to Solve Two Existing Problems

There are two problems in the code above. One is that in a multi-threaded situation, the new operation might be executed twice. The other is that the destructor is not called when the program exits. The following solution uses a static object to solve these problems.

class Singleton
{
private:
static Singleton *local_instance;
Singleton(){
cout << "Constructor" << endl;
};
~Singleton(){
cout << "Destructor" << endl;
}
public:
static Singleton *getInstance()
{
static Singleton locla_s;
return &locla_s;
}
};
int main()
{
cout << "Before first access to singleton" << endl;
Singleton * s = Singleton::getInstance();
cout << "After first access to singleton" << endl;
cout << "Before second access to singleton" << endl;
Singleton * s2 = Singleton::getInstance();
cout << "After second access to singleton" << endl;
return 0;
}

This code might cause multiple constructor calls in versions before C++11, so it can only be used with newer compilers.

If Using Pre-C++11 Versions, Static Objects Will Not Be Thread-Safe

The version below uses mutex and a static member to destruct the singleton. The disadvantage of this approach is that the lock causes slow speed and low efficiency. But at least it’s correct and can be used in versions before C++11. Sample code is as follows:

class Singleton
{
private:
static Singleton *local_instance;
static pthread_mutex_t mutex;
Singleton(){
cout << "Constructor" << endl;
};
~Singleton(){
cout << "Destructor" << endl;
}
class rememberFree{
public:
rememberFree(){
cout << "Member constructor" << endl;
}
~rememberFree(){
if(Singleton::local_instance != nullptr){
delete Singleton::local_instance;
}
}
};
static rememberFree remember;
public:
static Singleton *getInstance()
{
pthread_mutex_lock(&mutex);
if (local_instance == nullptr)
{
local_instance = new Singleton();
}
pthread_mutex_unlock(&mutex);
return local_instance;
}
};
Singleton * Singleton::local_instance = nullptr;
pthread_mutex_t Singleton::mutex = PTHREAD_MUTEX_INITIALIZER;
Singleton::rememberFree Singleton::remember;

Double-Checked Locking Causes Uninitialized Memory Access

Using the following code to implement direct return of an already initialized object will greatly improve the performance of the above code. But the same code has obvious problems in Java, where CPU out-of-order execution may lead to accessing a reference to an uninitialized object. Does C++ have the same problem? See the following article: http://www.aristeia.com/Papers/DDJ_Jul_Aug_2004_revised.pdf The conclusion is the same - C++ has the same issue and may lead to undefined behavior causing segmentation faults. An example of double-checked locking code is as follows:

static Singleton *getInstance()
{
if(local_instance == nullptr){
pthread_mutex_lock(&mutex);
if (local_instance == nullptr)
{
local_instance = new Singleton();
}
pthread_mutex_unlock(&mutex);
}
return local_instance;
}

If thread A enters the lock and allocates space for the object, but due to possible instruction reordering, local_instance is actually pointed to a block of unallocated memory first, and then initialization occurs on this memory block. But after pointing and before initialization, another thread B might get this pointer through getInstance.

Attempting to Use Local Variables Cannot Guarantee Instruction Execution Order

When trying to use temporary variables to force the order of instruction execution, they may still be considered useless variables by the compiler and optimized away. The following code is a good idea but fails to achieve its purpose:

 if(local_instance == nullptr){
static mutex mtx;
lock_guard<mutex> lock(mtx);
if (local_instance == nullptr)
{
auto tmp = new Singleton()
local_instance = tmp;
}
}
return local_instance;

Inelegant Use of volatile to Solve Instruction Reordering Problems in Double-Checked Locking

Trying to use volatile to declare the internal pointer, the code is as follows:

class Singleton
{
private:
static Singleton * volatile local_instance;
Singleton(){
cout << "Constructor" << endl;
};
~Singleton(){
cout << "Destructor" << endl;
}
class rememberFree{
public:
rememberFree(){
cout << "Member constructor" << endl;
}
~rememberFree(){
if(Singleton::local_instance != nullptr){
delete Singleton::local_instance;
}
}
};
static rememberFree remember;
public:
static Singleton *getInstance()
{
if(local_instance == nullptr){
static mutex mtx;
lock_guard<mutex> lock(mtx);
if (local_instance == nullptr)
{
auto tmp = new Singleton();
local_instance = tmp;
}
}
return local_instance;
}
};
Singleton * volatile Singleton::local_instance = nullptr;
Singleton::rememberFree Singleton::remember;
int main()
{
cout << "Before first access to singleton" << endl;
Singleton * s = Singleton::getInstance();
cout << "After first access to singleton" << endl;
cout << "Before second access to singleton" << endl;
Singleton * s2 = Singleton::getInstance();
cout << "After second access to singleton" << endl;
return 0;
}

In this code, although temp is volatile, *temp is not, nor are its members. So it may still be optimized. Try to declare *temp as volatile as well, and you’ll find that your code is full of volatile. But at least it’s correct:

class Singleton
{
private:
static volatile Singleton * volatile local_instance;
Singleton(){
cout << "Constructor" << endl;
};
~Singleton(){
cout << "Destructor" << endl;
}
class rememberFree{
public:
rememberFree(){
cout << "Member constructor" << endl;
}
~rememberFree(){
if(Singleton::local_instance != nullptr){
delete Singleton::local_instance;
}
}
};
static rememberFree remember;
public:
static volatile Singleton *getInstance()
{
if(local_instance == nullptr){
static mutex mtx;
lock_guard<mutex> lock(mtx);
if (local_instance == nullptr)
{
auto tmp = new Singleton();
local_instance = tmp;
}
}
return local_instance;
}
};
volatile Singleton * volatile Singleton::local_instance = nullptr;
Singleton::rememberFree Singleton::remember;
int main()
{
cout << "Before first access to singleton" << endl;
volatile Singleton * s = Singleton::getInstance();
cout << "After first access to singleton" << endl;
cout << "Before second access to singleton" << endl;
volatile Singleton * s2 = Singleton::getInstance();
cout << "After second access to singleton" << endl;
return 0;
}

The Ultimate Weapon — Memory Barrier

In the new standard, the atomic class implements memory barriers, making memory access controllable across multiple cores. This utilizes the controllable memory access order in C++11. Here is the code implementation:

class Singleton
{
private:
// static volatile Singleton * volatile local_instance;
static atomic<Singleton*> instance;
Singleton(){
cout << "Constructor" << endl;
};
~Singleton(){
cout << "Destructor" << endl;
}
class rememberFree{
public:
rememberFree(){
cout << "Member constructor" << endl;
}
~rememberFree(){
Singleton* local_instance = instance.load(std::memory_order_relaxed);
if(local_instance != nullptr){
delete local_instance;
}
}
};
static rememberFree remember;
public:
static Singleton *getInstance()
{
Singleton* tmp = instance.load(std::memory_order_relaxed);
atomic_thread_fence(memory_order_acquire);
if(tmp == nullptr){
static mutex mtx;
lock_guard<mutex> lock(mtx);
tmp = instance.load(memory_order_relaxed);
if (tmp == nullptr)
{
tmp = new Singleton();
atomic_thread_fence(memory_order_release);
instance.store(tmp, memory_order_relaxed);
}
}
return tmp;
}
};
atomic<Singleton*> Singleton::instance;
Singleton::rememberFree Singleton::remember;
int main()
{
cout << "Before first access to singleton" << endl;
Singleton * s = Singleton::getInstance();
cout << "After first access to singleton" << endl;
cout << "Before second access to singleton" << endl;
Singleton * s2 = Singleton::getInstance();
cout << "After second access to singleton" << endl;
return 0;
}

The above code may be difficult to read. The two loads of instance can be executed out of order. But changes during this period cannot be observed by other CPU cores. In the Muduo book, memory barriers are also rated as the ultimate weapon.

Using Atomic Operation Memory Order

There are six memory sequence options that can be applied to operations on atomic types: memory_order_relaxed, memory_order_consume, memory_order_acquire, memory_order_release, memory_order_acq_rel, and memory_order_seq_cst. Unless you specify a sequence option for a specific operation, the memory sequence option for all atomic types defaults to memory_order_seq_cst. Although there are six options, they only represent three memory models: sequentially consistent, acquire-release sequence (memory_order_consume, memory_order_acquire, memory_order_release and memory_order_acq_rel), and relaxed sequence (memory_order_relaxed).

The models that can be adopted here are: the default memory_order_seq_cst, which is sequentially consistent, and memory_order_acquire, memory_order_release, which is the acquire-release sequence. The latter may perform better.

To be improved

Using pthread_once or call_once

The former comes from the pthread library. The latter comes from std::atomic.

To be improved

Learning GDB from Scratch

Wed, 25 Apr 2018 16:45:00 +0000

Let’s start with an example.

#include <stdio.h>
void my_itoa(int n,char s[])
{
int i,j,sign;
if((sign=n)<0) // record sign
n=-n; // make n positive
i=0;
do{
s[i++]=n%10+'0'; // get the next digit
}while((n/=10)>0); // loop through division
if(sign<0)
s[i++]='-';
s[i]='\0';
for(j=i-1;j>=0;j--) // the generated numbers are in reverse order, so output them in reverse
printf("%c",s[j]);
}
void main()
{
int n;
char str[100];
my_itoa(-123,str);
printf("\n");
printf("%d\n",my_atoi("123"));
system("pause");
}

Save the above file to test.cpp, then compile and run it.

Now let’s debug it and see how to debug specifically. Debug the code following the method below.

linan@linan-PC:~$ gdb test
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from test...done.
(gdb) l
2
3 int func(int n)
4 {
5 int sum=0,i;
6 for(i=0; i<n; i++)
7 {
8 sum+=i;
9 }
10 return sum;
11 }
(gdb)
12
13 main()
14 {
15 int i;
16 long result = 0;
17 for(i=1; i<=100; i++)
18 {
19 result += i;
20 }
21
(gdb)
22 printf("result[1-100] = %d \n", result );
23 printf("result[1-250] = %d \n", func(250) );
24 }
(gdb)
Line number 25 out of range; test.cpp has 24 lines.
(gdb) break 15
Breakpoint 1 at 0x706: file test.cpp, line 15.
(gdb) break func
Breakpoint 2 at 0x6d7: file test.cpp, line 5.
(gdb) info break
Num Type Disp Enb Address What
1 breakpoint keep y 0x0000000000000706 in main() at test.cpp:15
2 breakpoint keep y 0x00000000000006d7 in func(int) at test.cpp:5
(gdb) run
Starting program: /home/linan/test
Breakpoint 1, main () at test.cpp:16
16 long result = 0;
(gdb) next
17 for(i=1; i<=100; i++)
(gdb) next
19 result += i;
(gdb) next
17 for(i=1; i<=100; i++)
(gdb) continue
Continuing.
result[1-100] = 5050
Breakpoint 2, func (n=250) at test.cpp:5
5 int sum=0,i;
(gdb) p sum
$1 = 32767
(gdb) n
6 for(i=0; i<n; i++)
(gdb) p sum
$2 = 0
(gdb) p i
$3 = -134225552
(gdb) print i
$4 = -134225552
(gdb) n
8 sum+=i;
(gdb) print i
$5 = 0
(gdb) bt
#0 func (n=250) at test.cpp:8
#1 0x000055555555474c in main () at test.cpp:23
(gdb) help bt
Print backtrace of all stack frames, or innermost COUNT frames.
With a negative argument, print outermost -COUNT frames.
Use of the 'full' qualifier also prints the values of the local variables.
Use of the 'no-filters' qualifier prohibits frame filters from executing
on this backtrace.
(gdb) bt
#0 func (n=250) at test.cpp:8
#1 0x000055555555474c in main () at test.cpp:23
(gdb) finish
Run till exit from #0 func (n=250) at test.cpp:8
0x000055555555474c in main () at test.cpp:23
23 printf("result[1-250] = %d \n", func(250) );
Value returned is $6 = 31125
(gdb) c
Continuing.
result[1-250] = 31125
[Inferior 1 (process 11480) exited normally]
(gdb)

Before Using GDB

First, make sure your program was compiled with the -g parameter. This will insert relevant information into the executable program. This information doesn’t help the program but assists with debugging.

Starting Debugging with GDB

There are several ways to launch GDB:

gdb program is your executable file, usually in the current directory.
gdb core Use gdb to simultaneously debug a running program and core file. The core file is generated after a program illegally executes and core dumps.
gdb If your program is a service program, you can specify the process ID of the running service program. gdb will automatically attach to it and debug it. The program should be found in the PATH environment variable.

Notes for GDB Beginners

When debugging with GDB, you need to know where the source code files are located. In the example above, the source code and the running program are together. But this is not possible in practice. Add the source code directory using the -d parameter in the GDB command or using the directory command during runtime.

Other notes:

 -symbols <file>
-s <file>
Read symbols from the specified file.
-se file
Read symbol table information from the specified file and use it in the executable file.
-core <file>
-c <file>
The core file from a core dump during debugging.

Effective C++ Notes

Mon, 23 Apr 2018 10:40:00 +0000

Chapter 1: From C to C++

Item 1: C++ is a federation of languages This means that C++ supports procedural, object-oriented, generic, functional, and meta-programming - five programming styles, which can be confusing. It is a comprehensive language.
Item 2: Prefer const, enum, inline to #define #define is not part of the language, but part of the preprocessor. const, enum, and inline allow the compiler to detect errors.
Item 3: Use const to restrict possible errors For example, return const reference objects to prevent users from modifying them. Use const member functions to prevent damage to classes.
Item 4: Make sure objects are initialized before use Mainly use initialization lists to initialize members or parent classes. Additionally, if a class uses global objects, it’s impossible to know when those objects are initialized. Therefore, use static variables inside the class (which also have only one copy).

Chapter 2: Constructors, Destructors, and Assignment Operators

Item 5: Know what functions the compiler silently generates The compiler will generate default member functions as needed, including default constructor, copy constructor, destructor, and copy assignment operator.

The default copy assignment operator may not be able to copy reference members.

Item 6: Use =delete to explicitly disallow auto-generated functions This can prevent a class from being copied, blocking copying.
Item 7: Declare virtual destructors for polymorphic base classes This is a basic function.
Item 8: Prevent exceptions from leaving destructors This can lead to incomplete memory release or cause multiple exceptions to be thrown.
Item 9: Never call virtual functions during construction or destruction This is also obvious - if a derived class constructor enters the base class constructor and calls a virtual function, the derived class’s virtual function isn’t ready yet.
Item 10: Have operator= return a reference to *this This allows for continuous assignment.
Item 11: Handle self-assignment in operator= It’s difficult to ensure your self-assignment code has no issues, and self-assignment is a time-wasting behavior.
Item 12: Copy all parts of an object when copying This is obvious. If you implement an assignment function, you must copy all contents.

Chapter 3: Resource Management

Item 13: Objects are the basic unit of resource management You need to be able to acquire resources and correctly release them. The key is to use shared_ptr to manage memory, which can automatically release held resources.
Item 14: Copying resource-managing objects Either prohibit copying, use shared_ptr members, perform deep copying, or make the copying process transfer the resource.
Item 15: Provide access to raw resources in resource-managing classes The world isn’t perfect, and many APIs require passing raw resource pointers. Using shared_ptr can interfere with how the object is used.
Item 16: Use delete[] for arrays allocated with new[] To avoid this error, you can use vector containers to operate, avoiding the need for new arrays.
Item 17: Store newed objects in smart pointers in standalone statements If an exception is thrown between creating a new object and passing it to a shared_ptr, it will lead to memory leaks.

Chapter 4: Designs and Declarations

Item 18: Make interfaces easy to use correctly Interfaces can sometimes be difficult to express clearly. For example, if you pass in three int values, you need to clearly remember what each one represents. What if the parameters are passed in the wrong order? The book mentions passing in specific objects and using explicit to constrain parameters. You could also build a date object and then specify members, which I think is a bit better. Additionally, interfaces should have reasonable parameter settings and reasonable return values (such as returning smart pointers instead of raw pointers to force proper usage).
Item 19: Designing a class is a major project Object creation, destruction, initialization and assignment, whether passing objects by value causes exceptions, what are the legal values for the new type. Have type conversion functions been written? What about operators? Which functions should be deleted?
Item 20: Prefer pass-by-reference-to-const to pass-by-value Passing by value brings the call of the object’s constructor, consuming time. Passing by reference is like passing a pointer.
Item 21: Don’t return a reference to a local object When returning an object created by a function, you cannot return a reference to that object. The compiler will make the correct optimization steps by itself.
Item 22: Declare data members private If declared public, deleting a variable will cause all programs to be rewritten. If declared private, deleting a variable will cause inherited class programs to be rewritten. So declare them as private.
Item 23: Prefer non-member non-friend functions to member functions If a method only calls a few other member functions, there’s no need to set it as a member function. Put it outside and wrap it with the same namespace.
Item 24: Declare non-member functions when type conversions should apply to all parameters This is because member functions may not work in some cases. For example, operator * can only support that type as the first parameter. It will fail when a built-in type is used as the first parameter.
Item 25: Consider support for a non-throwing swap Swap is a useful function. To implement an efficient swap, you need to design a swap member function to exchange with another object. Then design a non-member function in the namespace to call the above swap function. Finally, specialize std::swap to let the compiler choose your own swap.

Chapter 5: Implementations

Item 26: Postpone variable definitions as long as possible C++ cannot arbitrarily change the order of different lines, which means it’s better to define something only when it’s used. If all definitions are placed at the beginning, but the program returns right away, this would bring unnecessary construction and destruction processes.
Item 27: Minimize casting Although the four types of C++ type conversions can be used, they all bring bigger problems. Old-style type conversions should be used even less.
Item 28: Avoid returning handles to object internals Try to avoid returning pointers to internal data of objects outside the object, as this can cause big problems. The vector [] operator is such a case. This is a special case.
Item 29: Strive for exception-safe code C++ exceptions are the most difficult thing to handle, but you still need to ensure exceptions.
Item 30: Understand the ins and outs of inlining Inline is to make the compiler try to avoid stack expansion, not definitely. The 80-20 rule tells us that a program will only spend time on 20% of the code, so not all functions need to use inline.
Item 31: Minimize compilation dependencies: Separate definitions from declarations Two ways to handle - implementation classes and interface classes. Implementation classes use another class to proxy the functionality of the class. Interface classes use object-oriented methods to design the class as an interface.

Chapter 6: Inheritance and Object-Oriented Design

Item 32: Make sure public inheritance models “is-a” This means that derived classes can do everything base classes can do. But what derived classes can do, base classes may not be able to do.
Item 33: Avoid hiding inherited names If you declare another overloaded version of a parent class function in a derived class, it will also override the original version of the parent class, making it impossible to correctly link to the parent class function. You can use “using” to indicate the use of the parent class’s function space.
Item 34: Differentiate between inheritance of interface and inheritance of implementation Pure virtual functions only inherit the interface, virtual functions inherit the interface and default implementation, and non-virtual functions inherit the interface and forced implementation.
Item 35: Consider alternatives to virtual functions Other design patterns can be used to replace the implementation of virtual functions, such as function objects.
Item 36: Never redefine an inherited non-virtual function Otherwise, you won’t know which one is being called. Yes, I’m talking about destructors.
Item 37: Never redefine a function’s inherited default parameter value This is because default parameters are statically determined, not dynamically determined. The one you define may not be useful.
Item 38: Model “has-a” or “is-implemented-in-terms-of” through composition Public inheritance is an “is-a” relationship. Composition is a “has-a” relationship. For example, in the implementation of a set, inheriting from list would have some problems. Set should compose list.
Item 39: Use private inheritance judiciously The frequency of using private inheritance is lower than composition. There are two cases where this implementation is needed: first, when the derived class needs to access the protected members of the base class; second, when virtual functions need to be redefined (shouldn’t a middle class be designed in this case?).
Item 40: Use multiple inheritance judiciously Multiple inheritance is rarely used. It’s best not to include any members in virtual inheritance. This is because the initialization of members in virtual inheritance is done in the derived class.

Chapter 7: Templates and Generic Programming

Item 41: Understand implicit interfaces and compile-time polymorphism This is static polymorphism.
Item 42: Understand the two meanings of typename Typename appears not only in the template line of a template but also in template processing. When we use member variables or member types, the compiler doesn’t know which one is being used. By default, it uses member variables. Add typename to remind the compiler that it’s a member type.
Item 43: Know how to access names in templatized base classes This is because template classes can be specialized, and if specialized, some symbols may not exist. So you need to explicitly indicate symbols in the template class, such as using the this pointer or scope.
Item 44: Factor parameter-independent code out of templates When a template passes a non-type parameter, it generates multiple copies of code. There are some ways to avoid generating duplicate code, but they’re not as fast as the original.
Item 45: Use member function templates to accept all compatible types Classes can be templates, and member functions can also be templates.
Item 46: Define non-member functions inside templates when type conversions are desired Not only define it as a non-member function but also as a friend function.
Item 47: Use traits classes for information about types Traits use the properties of template programming and specialization to implement a traits class, through which you can get the data type you want in the generalized class.
Item 48: Be aware of template metaprogramming Metaprogramming refers to programming methods where results are known at compile time. For example, processing input at compile time. Similar to factorial, which is like the Hello World program of C++ metaprogramming.

Chapter 8: Customizing new and delete

Item 49: Understand the behavior of the new-handler It continuously tries to allocate memory. I’m curious if there will be a freeze if the new-handler keeps failing to allocate.
Item 50: Understand when it makes sense to replace new and delete It seems that the performance of new and delete is very high now. No need to replace them.
Item 51: Adhere to convention when writing new and delete Maintain compatibility when customizing new and delete.
Item 52: Write placement delete if you write placement new Placement delete will automatically run when placement new encounters an error.

Chapter 9: Modern C++11

Item 53: Pay attention to compiler warnings Don’t ignore warnings.
Item 54: Familiarize yourself with the TR1 standard libraries
Item 55: Familiarize yourself with Boost

Tencent Campus Recruitment Essay

Wed, 11 Apr 2018 08:39:00 +0000

Introduction

There were two phone interviews in the early selection phase, which I’ll skip here. First published at: https://www.jianshu.com/p/d5aa63a27172

First Interview

Chengdu, April 11

Introduce your projects, what were the difficulties, and how did you solve them
Introduce C++11 features
Explain the fork function
Explain the TIME_WAIT state
Why does a TCP connection handshake require three steps
Explain iterator invalidation. Does push_back cause iterator invalidation?
Characteristics of red-black trees, explain
Methods to resolve hash collisions
Differences between processes and threads
What thread models do you use
Explain coroutines
Explain the quicksort algorithm
What is a stable sort, is quicksort stable, and why
Derive the formula for the worst-case scenario of quicksort
Why should a destructor be virtual
Why can’t a constructor be a virtual function
Questions printed on paper, testing: new[] objects, static members, what exactly a subclass constructor calls, differences between copy constructors and assignment operators, when the equals sign appears in a copy constructor, when an assignment operator is needed, deep copy vs. shallow copy, when virtual functions are called.

Below is one of the questions, please write the output:

#include <iostream>
using namespace std;
struct A{
A(){
local_var++;
}
virtual void func(){
cout << "A" << endl;
}
static int local_var;
};
int A::local_var = 0;
struct B : A{
B(){
local_var+=2;
}
virtual void func(){
cout << "B" << endl;
}
};
int main(){
A* a1 = new B[3];
cout << a1[0].local_var << endl;
a1->func();
A a2 = a1[0];
cout << a2.local_var << endl;
a2.func();
A a3;
a3 = a1[0];
cout << a3.local_var << endl;
a3.func();
}

What problem will occur in the following code?

#include <iostream>
using namespace std;
struct A{
virtual void func1(){
cout << a << endl;
}
void func2(){
cout << a << endl;
}
void func3(){
cout << "a" << endl;
}
int a;
};
int main(){
A * a = (A*)malloc(sizeof(A));
a->func1();
a->func2();
a->func3();
}

The approach to a problem during the written test (previous Nowcoder test. The interviewer actually printed all my test answers and results, including multiple-choice and algorithm questions, how many times I submitted, how many times I failed). Testing when the virtual pointer is initialized.
Randomly selected a question on paper. Given preorder traversal ABC and postorder traversal CBA, what is the inorder traversal? Draw out the two possible scenarios.
The approach to the first algorithm question during the written test (previous Nowcoder test)
There are 100 marbles, two players take turns, each person can only take 1-5 marbles, the person who cannot take any loses, what is the winning strategy?
What have you been learning recently (server programming)? What did you learn before (TensorFlow)? How did you learn it? What do you think about TensorFlow’s powerful neural networks?
Family situation.
After leaving, wrote code in the lobby: find elements in char array a that are not in char array b, and put them in char array c. Here I forgot to convert char characters to unsigned char type.

Second Interview

Chengdu, April 13

Self-introduction
What are system calls? Which system calls have you used? Which system calls take a long time?
Have you used gdb debugging? What is a conditional breakpoint?
Difference between function pointers and pointer functions. Write an example.
Write a fork call example by hand. In what situations have you used fork calls?
Explain the connect function in UDP.
What is an index? Is adding more indexes always better? (I admitted to looking at interview experiences for the index question, but came up with the follow-up question myself)
Which questions from the last interview did you research afterward?
There was a problem with your code in the last interview, do you know what it was?
Where did you learn the answers to these questions? What have you been doing these two days?
Some suggestions for you: look into gdb debugging, database knowledge, network programming, and practice more.

Aftermath

Received a phone call on April 20 confirming the offer from Tencent.

TCP UDP IP Maximum Message Length

Mon, 26 Mar 2018 16:41:00 +0000

UDP

For UDP protocol, the maximum length of the entire packet is 65535 bytes, and the payload length is 65535-20=65515 bytes.

TCP

For TCP protocol, the maximum length of the entire packet is determined by the Maximum Segment Size (MSS). MSS is the maximum data segment that a TCP packet can transmit each time. To achieve optimal transmission performance, the TCP protocol usually negotiates the MSS value between both parties when establishing a connection. When implementing the TCP protocol, the MTU value is often used as a substitute (minus the size of the IP packet header of 20 bytes and the TCP segment header of 20 bytes), so the MSS is often 1460 bytes. The communicating parties will determine the maximum MSS value for this connection based on the smaller of the two MSS values provided.

IP

For IP protocol, the size of the IP packet is determined by the MTU (Maximum Transmission Unit).

TCP Message Format and HTTP Message Format

Mon, 26 Mar 2018 16:37:00 +0000

TCP

Source Port Destination Port
Sequence Number
Acknowledgment Number
Flags Window
Checksum Urgent Pointer
Options
Data

HTTP

Request URL Protocol-Version\r\n
Field: Value\r\n
\r\n
Body

Protocol-Version Status-Code Status-Description\r\n
Field: Value\r\n
\r\n
Body

TCP UDP Connection Establishment Process via Socket

Mon, 26 Mar 2018 14:46:00 +0000

TCP

Phase	Server	Client
	socket	socket
	bind
	listen
	accept	connect
	write	write
	read	read
	close	close

UDP

Phase	Server	Client
	socket	socket
	bind
	sendto	recvfrom
	recvfrom	sendto
	close	close

HTTPS Two-way Encryption Process Analysis

Mon, 26 Mar 2018 14:42:00 +0000

HTTPS authentication process:

① The browser sends a connection request to the secure server.
② The server sends its certificate and certificate-related information to the client browser.
③ The client browser checks whether the certificate sent by the server is issued by a trusted CA (Certificate Authority). If it is, the protocol continues execution; if not, the client browser gives the user a warning message: warning that this certificate is not trustworthy and asking whether the user wants to continue.
④ Next, the client browser compares the information in the certificate, such as domain name and public key, with the relevant information just sent by the server. If they match, the client browser recognizes the server’s legal identity.
⑤ The server requests the client to send its own certificate. After receiving it, the server verifies the client’s certificate. If it fails verification, the connection is refused; if it passes verification, the server obtains the user’s public key.
⑥ The client browser informs the server of the symmetric encryption methods it supports.
⑦ The server selects the encryption method with the highest level of encryption from the methods sent by the client, encrypts it with the client’s public key, and notifies the browser.
⑧ The browser selects a session key for this encryption method, encrypts it with the server’s public key, and sends it to the server.
⑨ The server receives the message sent by the browser, decrypts it with its own private key, and obtains the session key.
⑩ The subsequent communication between the server and browser uses the symmetric encryption method, and the symmetric key is encrypted.

The above describes the specific communication process of two-way authentication SSL protocol, which requires certificates from both the server and user. One-way authentication SSL protocol does not require the client to have a CA certificate. Compared to the steps above, it only needs to remove the process of server-side verification of the client certificate, and when negotiating the symmetric encryption method and session key, the server sends the unencrypted encryption method to the client (this does not affect the security of the SSL process). This way, the specific communication content between both parties is encrypted data. If a third party attacks and obtains the data, they only get encrypted information. For the third party to obtain useful information, they need to decrypt the encrypted data, and at this point, security depends on the security of the encryption method. Fortunately, the current encryption methods are sufficiently secure as long as the communication key is long enough. This is why we emphasize the requirement to use 128-bit encrypted communication.

Tencent Internal Referral Essay

Mon, 26 Mar 2018 12:23:00 +0000

This afternoon I was browsing Nowcoder. Suddenly I wanted to check on the progress of my Tencent internal referral, and the WeChat official account showed that “the current position may not be suitable for you.”

I don’t have any special feelings about it.

I still need to work hard and strive for a good outcome. Regardless, there are still opportunities.

In the afternoon while doing practice problems, there were 3 out of 8 NetEase questions that I couldn’t solve, which made me feel that there are indeed many things I need to learn.

I bought a lot of books, and I don’t know when I’ll be able to finish reading them.

Anyway, I should stay positive.

The interviewer said I was quite suitable, although that probably made me quite happy, but I won’t be sad because I was rejected, nor will I be proud because I was chosen.

Tencent Mock Exam: The Change-Making Problem

Mon, 26 Mar 2018 03:57:00 +0000

Little Q is very wealthy and owns many coins. Little Q’s coins follow a pattern: for all non-negative integers K, Little Q has exactly two coins with the value 2^k. So Little Q owns coins of values 1, 1, 2, 2, 4, 4, and so on. Little Q needs to pay n money for something and wants to know how many different combinations of coins he can use to make this payment. Input: An integer n (1<=n<=10^18), representing the amount to be paid. Output: The number of ways Little Q can make this payment.

Reference solution:

#coding=utf-8
import math
n = int(input())
def getMaxCoin(n):
tmp = int(math.log(n,2))
return 2**tmp
buff = {}
def dp(n, coin):
if (n, coin) in buff:
return buff[(n,coin)]
if n == 0: # Making change for 0 money is always possible
return 1
if coin == 1: # Only using 1-value coins
if n == 1 or n == 2:
return 1
return 0 # No solution with current coin value
if n > coin * 4 -2:
return 0
ret = 0
# Using one coin of current value
if n >= coin:
ret += dp(n-coin, coin//2)
if n >= coin*2:
# Using two coins of current value
ret += dp(n-coin*2, coin//2)
# Not using any coin of current value
ret += dp(n, coin//2)
buff[(n,coin)] = ret
return ret
print(dp(n,getMaxCoin(n)))

Approach: For the current amount, the maximum possible coin value that can be used is calculated by the getMaxCoin function. For the problem amount n and the maximum coin value that can be used, we calculate dp(n, coin). The coin value will continuously decrease. For coin value 1, there’s only 1 way to make amounts 1 or 2. For amount 0, no coins are needed for a solution. For amounts greater than coin*4-2, it exceeds the maximum possible value, so there’s no solution. For other cases, we consider three situations: using one coin of the current value, using two coins of the current value, or using zero coins of the current value. We calculate the corresponding dp values for each case. We use the buff cache to store already calculated values.

My First Program

Wed, 21 Mar 2018 08:06:00 +0000

Introduction

Blank space

Attempt

During my freshman year, I remember participating in a college competition, which was my first exposure to programming concepts.

The competition involved using programmable building blocks and toys provided by the laboratory to create a project. It was somewhat similar to the concept of Mi Bunny Robot, except it was much earlier.

I built a smart car that could follow black tape tracks on the ground and perform corresponding actions. It had the following three functions:

Following a single line of tape on the ground. This function was implemented using two infrared sensors.
When encountering obstacles, it could bypass them according to pre-agreed actions. This was implemented with an infrared sensor.
It could grab ping pong balls from the ground using a claw. This was driven by a motor.

I remember the programming interface was a graphical one, where you could drag different rectangles representing program bodies. Then you used connecting lines to indicate state transitions. After writing the code, it needed to be transmitted to the programmable blocks via an infrared device. The infrared transmission device looked a bit like an upright iron, and the block was quite large. It could be fitted with lithium batteries and had some gear-like objects that could be attached to other wheels or sliding devices. This way, the entire device could move as expected.

I remember that during the judging, except for my own work, none of the other teams’ projects could move (meaning they hadn’t written programs).

When other teams were introducing their robots, they only described the functions, but unfortunately, they couldn’t move.

When it was my turn to present, it truly amazed everyone. The judges were astonished to see my robot move.

In reality, the built-in program would disappear when power was lost, and because the blocks were very old, the lithium batteries inside couldn’t last long. I wrote the program at the base and brought it to the competition venue. To prevent the program from being lost, I found a charger. Because when running, the lithium battery without a charging cable would instantly lose power, causing the program to disappear. So it was also lucky that it could move.

I didn’t really expect anything; I felt like I was just implementing this little project according to my own ideas. Later, when they announced the first prize, surprisingly, it wasn’t me. I didn’t feel very disappointed, as it was fun anyway. But when the person who went up to receive the award wasn’t me, everyone present was quite astonished.

Then the award presenter said there was a mistake, and I ended up winning first place after all.

Failure

The university was holding an electronic design competition open to all students.

I looked at the competition topics and thought I could give it a try. I found similar topics on Baidu. The content required a series of electronic components to be soldered together, and then the required program, which was provided, had to be loaded.

For me at that time, that pile of code was just a bunch of gibberish.

I went to the Hangzhou electronics market and navigated through countless shops, where the shop owners looked at me as if I were an alien.

I awkwardly asked for the electronic components I wanted. A kind shop owner gave me some LED bulbs.

She also said that I should use a flat-head soldering iron, as beginners can’t handle pointed ones well.

They were very curious about what I was trying to do.

After buying the components, I sat on the floor and, following the circuit diagram from that blog, soldered everything together on a chair.

Oh, right, at that time I hadn’t learned about circuit diagrams either, so I was just soldering based on my intuition.

I think I was quite naive then, trying to solder something without any basics. But I don’t have the right to mock him.

Then I started loading the program. I put the single chip into the socket, connected the socket to the power supply, and started trying to download the program provided by that blog to the chip using software.

I actually wanted to tell him, “Hey, are you sure the firmware program provided by this website matches your chip model? The chip is so hot; you actually inserted it backward, did you know? The chip has probably been burned out.”

But I knew nothing. I stupidly took something soldered from unknown components and a chip that was actually burned out to participate in the evaluation. Needless to say, I was despised by the expert judges.

What I didn’t know was that almost no one like me had registered for the competition. The university had an electronic design competition experimental class where you could learn everything you wanted to know about electronic design.

Although the competition was open to the whole university, only those from this experimental class would register.

My First Program

Later, I understood the existence of the electronic design class. I signed up for it in my sophomore year. I remember there was an exam and an interview, and you had to pass to get in.

I remember they asked me several questions, none of which I could answer. Later, they asked what my C language score was, and I said 97, then they said, “Okay, you’ve passed.”

The first week mainly covered basic soldering knowledge. Although it seems the story is just beginning, the story I want to tell in this article is also about to end.

The second week mainly covered knowledge about LED digital tubes. I received the distributed 51 development board. In the evening, I studied in my dorm how to make the digital tubes light up according to my ideas.

I learned how to display numbers that were constantly lit. I understood that I needed to select position first and then segment. I suddenly understood a lot; this was the most primitive time-slicing method.

I suddenly had an idea: I wanted to make the digital tubes achieve a marquee effect, with the outer ring continuously rotating.

I wrote and wrote and wrote.

I wrote it out.

I burned it in, ran it, and amazingly, it was just as I had imagined. The outer ring kept turning and turning, which was really beautiful. I was very happy.

Afterword

Perhaps that wasn’t my first program, after all, before that, the C language course required designing programs, so I should have written programs already.

However, for me, that marquee display was truly my first program in a real sense, completely created by me.

Even if I die someday, if the programs I write can continue to run, I can continue my life in another way.

Linux Multithreaded Server Programming Basics: C++11 Concurrent Programming

Sun, 18 Mar 2018 08:29:00 +0000

Introduction

Previous versions of C++ didn’t support threads natively. Instead, you had to use libraries like pthread. Using C++’s built-in thread support makes programs more unified and concise.

Header Files

<thread>: This header file contains the std::thread and std::this_thread classes, along with functions for managing threads. It’s the main file for implementing threads.
<atomic>: This header file contains std::atomic and std::atomic_flag classes, which are the main files for implementing atomic operations.
<mutex>: Contains mutex-related classes and functions.
<future>: Contains the future class and related functions.
<condition_variable>: Contains condition variable classes.

These are the thread-related parts of C++11. Although there are many debates about pthread versus C++ thread, the cross-platform C++ thread seems to be more standard.

Recommended book for learning C++11 threads: https://www.gitbook.com/book/chenxiaowei/cpp_concurrency_in_action/details

Hello World Thread

A simple introduction to the thread class:

#include <thread>
using namespace std;
// This function is the thread task we want to run
void hello()
{
printf("%s", "hello\n");
}
// Using the thread class, we pass a function as our initial task
// We can also pass other parameters like classes
int main()
{
thread t(hello);
// Using join to wait for completion
t.join();
}

Besides join to wait for completion, you can use detach to not wait for the thread to finish.

struct func
{
int& i;
func(int& i_) : i(i_) {}
void operator() ()
{
for (unsigned j=0 ; j<1000000 ; ++j)
{
do_something(i); // 1. Potential access hazard: dangling reference
}
}
};
void oops()
{
int some_local_state=0;
func my_func(some_local_state);
std::thread my_thread(my_func);
my_thread.detach(); // 2. Don't wait for the thread to finish
} // 3. The new thread might still be running

In this example, we’ve decided not to wait for the thread to finish (using detach() ②), so when the oops() function completes ③, the new thread might still be running. If the thread is still running, it will call the do_something(i) function ①, which accesses a variable that has already been destroyed. As with a single-threaded program—allowing pointers or references to local variables to persist after the function completes has never been a good idea—this situation isn’t obvious and makes multithreading more error-prone.

How to Wait for a Thread to Complete?

If you need to wait for a thread, the corresponding std::thread instance needs to use join(). In example 2.1, replacing my_thread.detach() with my_thread.join() ensures that local variables are only destroyed after the thread completes. In this case, since the original thread doesn’t do much during its lifetime, running the function in a separate thread offers minimal benefit. However, in real programming, either the original thread has its own work to do, or it starts multiple child threads to do useful work and waits for these threads to complete.

join() is a simple, direct way to wait for a thread to complete or not wait at all. When you need more flexible control over waiting threads, such as checking if a thread has finished or only waiting for a period of time (determining a timeout if exceeded), you need to use other mechanisms like condition variables and futures. Calling join() also cleans up the thread-related storage parts, so the std::thread object is no longer associated with the completed thread. This means you can only use join() once on a thread; once you’ve used join(), the std::thread object can’t be joined again. When joinable() is used on it, it will return false.

Passing Parameters to Threads

void f(int i, std::string const& s);
std::thread t(f, 3, "hello");

The code creates a thread that calls f(3, “hello”). Note that function f needs a std::string object as its second parameter, but here a string literal is used, which is of type char const *. The conversion from the literal to a std::string object is completed in the thread’s context.

It’s important to note that the constructor ignores the expected parameter types of the function and blindly copies the provided variables.

Example 1:

void f(int i,std::string const& s);
void oops(int some_param)
{
char buffer[1024]; // 1
sprintf(buffer, "%i",some_param);
std::thread t(f,3,buffer); // 2
t.detach();
}

In this case, buffer ② is a pointer variable pointing to a local variable, and then the local variable is passed to the new thread through buffer ②. Furthermore, the function is very likely to crash (oops) before the literal is converted to a std::string object, resulting in undefined behavior. And even if you want to rely on implicit conversion to convert the literal to the std::string object that the function expects, since std::thread’s constructor copies the provided variables, it only copies the unconverted string literal. The solution is to convert the literal to a std::string object before passing it to the std::thread constructor:

void f(int i,std::string const& s);
void not_oops(int some_param)
{
char buffer[1024];
sprintf(buffer,"%i",some_param);
std::thread t(f,3,std::string(buffer)); // Use std::string to avoid dangling pointers
t.detach();
}

Example 2: You might also encounter the opposite situation: expecting to pass a reference, but the entire object is copied. This can happen when a thread updates a data structure passed by reference, such as:

void update_data_for_widget(widget_id w,widget_data& data); // 1
void oops_again(widget_id w)
{
widget_data data;
std::thread t(update_data_for_widget,w,data); // 2
display_status();
t.join();
process_widget_data(data); // 3
}

Using Mutexes

In C++, you create a mutex by instantiating std::mutex and lock it by calling the member function lock(), and unlock it with unlock(). However, direct calling of member functions is not recommended in practice, as it means you must remember to call unlock() at every function exit, including exception cases. The C++ standard library provides a RAII syntax template class std::lock_guard for mutexes, which provides a locked mutex when constructed and unlocks it when destructed, ensuring that a locked mutex is always correctly unlocked. The following program listing shows how to use a std::lock_guard instance constructed with std::mutex to protect access to a list in a multithreaded program. Both std::mutex and std::lock_guard are declared in the header file.

#include <list>
#include <mutex>
#include <algorithm>
std::list<int> some_list; // 1
std::mutex some_mutex; // 2
void add_to_list(int new_value)
{
std::lock_guard<std::mutex> guard(some_mutex); // 3
some_list.push_back(new_value);
}
bool list_contains(int value_to_find)
{
std::lock_guard<std::mutex> guard(some_mutex); // 4
return std::find(some_list.begin(),some_list.end(),value_to_find) != some_list.end();
}

Protection of data breaks when one of the member functions returns a pointer or reference to the protected data. Pointers or references with access capability can access (and possibly modify) the protected data without being restricted by the mutex. The design of interfaces that involve mutex-protected data requires careful consideration to ensure that the mutex can lock any access to the protected data, leaving no backdoors.

Using Locks

Imagine a toy that consists of two parts, both of which must be obtained to play with it. For example, a toy drum requires a drumstick and a drum to play. Now there are two children who both like to play with this toy. When one child has both the drum and the drumstick, they can play freely. When the other child wants to play, they have to wait until the first child is finished. Now imagine that the drum and drumstick are kept in different toy boxes, and both children want to play the drum at the same time. So they go to the toy boxes looking for the drum. One finds the drum, and the other finds the drumstick. Now there’s a problem: unless one child decides to let the other play first by giving up their part, if they both hold firmly to their parts without giving them up, neither can play the drum.

Fortunately, the C++ standard library has a way to solve this problem: std::lock—it can lock multiple (two or more) mutexes at once without side effects (risk of deadlock).

How to avoid deadlocks?

Avoid nested locks The first suggestion is often the simplest: don’t acquire a second lock when a thread already has one. If you can stick to this advice, there won’t be deadlocks on locks because each thread only holds one lock.
Use a fixed order to acquire locks When hard conditions require you to acquire two or more locks, and you can’t use std::lock to acquire them in a single operation; it’s best to acquire them in a fixed order on each thread.

Synchronization and Waiting

Imagine you’re traveling on a night train. How do you get off at the right station at night? One way is to stay awake all night and pay attention to which station you’re at. This way, you won’t miss your destination, but it will make you very tired. Alternatively, you can look at the timetable, estimate when the train will arrive at your destination, and set an alarm a little earlier, then you can sleep soundly. This method sounds good and doesn’t involve missing your station, but when the train is late, you’ll be woken up too early. Of course, the battery in your alarm clock might also die, causing you to sleep through your station. The ideal way is, regardless of early or late, to have someone or something wake you up exactly when the train arrives at the station.

The C++ standard library has two implementations for condition variables: std::condition_variable and std::condition_variable_any. Both implementations are declared in the <condition_variable> header file. Both need to work with a mutex (for synchronization); the former is limited to working with std::mutex, while the latter can work with any mutex that meets minimum standards, hence the _any suffix. Because std::condition_variable_any is more general, there may be additional overhead in terms of size, performance, and system resource usage, so std::condition_variable is generally the preferred type, and we only consider std::condition_variable_any when there are hard requirements for flexibility.

Excerpts from Nowcoder Interview Experiences

Sat, 17 Mar 2018 06:42:00 +0000

Source: Alibaba Cainiao Java Internship Phone Interview Experience, Failed~!~

https://www.nowcoder.com/discuss/67251?type=2&order=0&pos=7&page=1

Java-specific questions removed

Is a user request handled by one thread per request on the server? After searching, I found there are two models: blocking and non-blocking. Nowadays, technology based on I/O multiplexing is generally used, like the Reactor pattern, which only needs one thread or a small number of threads to handle a large number of requests.
Server pressure handling strategies This is an even bigger question. After searching on Baidu, there are two points: caching and distributed systems. A netizen answered with message queues, which can alleviate situations with sudden large concurrent volumes.
Finding the top ten numbers among many numbers Can use a variant of quicksort
How is the time complexity of heap sort calculated? n×log n, which is n times, each time extracting from the heap bottom to top takes log n times
HTTP and TCP Not sure what specifically was being asked, perhaps the relationship?
Why are four steps needed to close a TCP connection (four-way handshake)? Two FIN+ACK, with unsent data inserted in the middle. In situations with delayed ACK.

Source: NetEase Internal Referral (C++/C Development) Offer Journey

https://www.nowcoder.com/discuss/3038?type=2&order=0&pos=15&page=4

What forms of polymorphism exist in C++? Polymorphism is one interface with multiple implementations. Static polymorphism and dynamic polymorphism. Static polymorphism is implemented through generic programming, dynamic polymorphism is implemented through virtual functions.
How is dynamic binding implemented? Static binding: compile-time binding, called through objects Dynamic binding: runtime binding, implemented through addresses Dynamic binding is implemented through virtual tables.
What types of type conversions are there? C-style conversions, explicit type casting, implicit type conversion. C++ has four types of type conversions: static_cast, const_cast, dynamic_cast, reinterpret_cast. They are used for built-in static type conversion (int, float), removing the const property of a type, dynamic type conversion (must be parent-child classes), and reinterpreting types without substantive conversion (pointer forced conversion), respectively.
Memory alignment principles The basic data type members of a structure must start aligned at integer multiples of the member size, structures as members must align at integer multiples of the widest data type. The size of a structure is aligned at integer multiples of its largest internal member.
How are templates implemented? The compiler remembers the template name when encountering a template declaration. When called, it generates specific type code based on the parameter types passed to the template, then compiles it.
Uses of pointers and const? Const type pointers, pointers to const type, const type pointers to const type.
Virtual functions, pure virtual functions, virtual functions and destructors? Why should destructors be defined as virtual functions? http://liaoxl.github.io/blog/20131205/virtual-destructor/ According to this article, if defined as a virtual function and the child class also implements it, both the child class destructor and the parent class destructor will be called.
Inline functions
const and typedef
What sorting algorithms are there? How is quicksort implemented? Best-case time complexity, average time complexity. Insertion, bubble, heap sort, merge sort, bucket sort, shell sort, quicksort.
What is extern “C” used for? Mainly used to call programs written in C.

2017 Summer Internship C++ Interview Experience (Alibaba, NetEase Games, Tencent, etc.)

Source: https://www.nowcoder.com/discuss/23512?type=2&order=0&pos=20&page=2

How is map implemented? What are the differences between red-black trees and AVL trees? Why do data structures like map and set use red-black trees for implementation? Have you heard of skip-list?
Have you used vector.reserve()? How does the vector array grow?
Are you familiar with map-reduce?
Are you familiar with Linux? Directory permissions, what does 644 represent? What does executable permission for a directory represent?
Do you regularly use vi? Are you familiar with common Linux commands? (like awk, grep, etc.) Have you used gdb for debugging?
What are the differences between TCP/UDP? How is congestion control implemented? How is reliable transmission implemented? Applications of UDP? Reliable/Unreliable, Byte stream/Datagram, Connection-oriented/Connectionless, High overhead/Low overhead, With congestion control/Without congestion control, Full-duplex/Simplex The purpose of congestion control is to prevent sending data packets beyond the network’s capacity, involving all hosts and routers. Congestion control assumes packet loss occurs when data packets exceed the network’s capacity, so when sending data packets, it needs to start slowly, gradually reaching the network’s capacity, increasing the congestion window exponentially to the slow start threshold, then entering congestion avoidance state where the congestion window grows linearly until congestion occurs. If a data packet’s ACK is not received before the timeout timer expires, it’s assumed that congestion has occurred, then the slow start threshold is set to half the current window size, and the slow start algorithm is executed. If three duplicate ACK packets are received, it’s assumed the lost packet hasn’t been received, so the unacknowledged packet is retransmitted early rather than waiting for the timeout timer to expire - this is fast retransmit. In case of fast retransmit, the slow start algorithm isn’t executed; instead, the congestion window is set to the slow start threshold, then congestion avoidance is executed - this is fast recovery.

Author: Simon_ Link: https://www.nowcoder.com/discuss/23512?type=2&order=0&pos=20&page=2 Source: Nowcoder

Summary:

I think Alibaba focuses more on projects, distributed systems, and multithreading knowledge. NetEase Games seems to focus more on algorithms and C++. Tencent asks about everything, from C++ memory structure, RTTI, to network programming, to algorithm implementation for brain teasers. You can prepare better based on the company you’re applying to. Let’s work hard together.

Books I read before preparing for internships include “C++ Primer”, “Inside the C++ Object Model”, “Effective C++”, “The Annotated STL Sources”, “Coding Interviews”, plus some books on networking and operating systems. I also solved the first 120 problems on LeetCode.

Some advice for juniors and sophomores looking for C++ development internships:

It’s best to go through C++ Primer several times before looking for internships. Make notes on things you don’t understand for future reference. When you read “The Object Model”, you’ll find that concepts you couldn’t understand in Primer will become clear. (For Primer study methods, I followed the first answer in https://www.zhihu.com/question/32087709)

It’s also good to read “The Annotated STL Sources”. Interviewers love to ask about the underlying implementation or dynamic growth of vector, map, etc. It might be difficult to understand these concepts just by reading blogs and interview experiences.

“Coding Interviews” covers algorithm questions for most companies’ first interviews. I think it’s best to understand each problem. The first 100 problems on LeetCode are indeed very classic, and you can reinforce your knowledge of binary search, hash, dynamic programming, binary trees, etc. through practice.

For networking, TCP/IP Illustrated Volume 1 is essential, and if you have time, you can also read Unix Network Programming. If you have even more time, it’s best to read through Chen Shuo’s “Linux Multithreaded Server Programming” to gain an understanding of multithreading.

The final piece of advice is to prepare as early as possible. During the resume submission and interview period, it’s only suitable to organize Nowcoder interview experiences and your own experiences, filling in knowledge gaps. After all, interview questions are random, and to perform better in interviews, it’s best to accumulate knowledge regularly.

C/C++ Backend Development Tencent Internship Spring Recruitment First Interview, Failed~

https://www.nowcoder.com/discuss/68508?type=0&order=0&pos=8&page=0

The underlying implementation of vector. I talked a lot about this, so he asked what specific actions insert performs, what resize function calls, and expansion situations.
Inter-process communication mechanisms Pipe, Named pipe (FIFO) Signal Message queue Shared memory Semaphore Socket

Alibaba Cainiao Network Second Interview

http://blog.csdn.net/caishenfans/article/details/44496581

Difference between fill and memset fill is an STL algorithm that calls the copy constructor; for basic types, it should directly copy bit by bit. memset is a C algorithm that copies byte by byte to the position pointed to by the original pointer.
C++ memory management methods RAII stands for Resource Acquisition Is Initialization It utilizes the guarantee that destructors of stack objects will be automatically called during stack unwinding to correctly release previously acquired resources. RAII only works properly if stack unwinding executes normally. Both function calls and normal C++ exception handling flow (exceptions within try-catch blocks) involve stack unwinding.
Implementation of smart pointers When another smart pointer copies the current smart pointer, the reference counter increments by 1. When assignment occurs between two smart pointers, the reference count of the object pointed to by the left pointer decrements by 1, and the right one increments by 1. At destruction, the reference count similarly decrements by 1. When the reference count reaches 0, the object is reclaimed. A set of operators is redefined: -> pointer, returns a pointer to the original pointer. Dereference *T, returns the dereference of the original pointer.

Alibaba Internship Phone Interview Experience, Over an Hour

https://www.nowcoder.com/discuss/3836

What is a binary balanced tree, how to insert nodes, delete nodes, describe the key steps.

How Students Can Get an Alibaba Technical Offer: “Alibaba Internship Interview Experience (Successful)”

https://yq.aliyun.com/articles/6395

Alibaba C++ Development Internship Second and Third Interview Experience (Failed)

https://www.nowcoder.com/discuss/27801?type=2&order=3&pos=55&page=1

Internship Interview Summary

https://www.nowcoder.com/discuss/25302?type=2&order=3&pos=190&page=1

Zhihu and Nowcoder Summary

Alibaba C++ Development Internship Second and Third Interview Experience (Failed)

30 Interview Experiences for [Alibaba, Tencent, ByteDance] and 12 Other Companies in [C++ Direction]

Alibaba Campus Recruitment Internship First Interview Experience

Goodbye to Alibaba Autumn Recruitment

Baidu Three Interviews (Offer) && Alibaba Four Interviews (In Progress)

Non-Key University Junior Spring Recruitment Internship Interview Experience

Internship Interview Summary

Alibaba, Morgan, Nomura, Tencent, Microsoft, TuDuck Technology Interview Experience Sharing

Alibaba Four Interview Experiences + Successfully Received Offer

My 2016 Written Test and Interview Experience (For Next Year’s Juniors and Sophomores)

https://www.nowcoder.com/discuss/25446

C++ new and delete

Wed, 14 Mar 2018 03:52:00 +0000

Source:

https://kelvinh.github.io/blog/2014/04/19/research-on-operator-new-and-delete/

What do new and delete code mean

Class *pc = new Class;
// ...
delete pc;

The first line above is the new operator, and the third line is the delete operator. The code is simple, but for the compiler, it needs to do additional work, translating the above code into something approximate to the following code:

void *p = operator new(sizeof(Class));
// Call the constructor of Class on the memory pointed to by p, this can't be shown with straightforward code
Class *pc = static_cast<Class*>(p);
// ...
pc->~Class();
operator delete(pc);

What does the operator new actually do, and what does the operator delete do?

void * operator new(std::size_t size) throw(std::bad_alloc) {
if (size == 0)
size = 1;
void* p;
while ((p = ::malloc(size)) == 0) {
std::new_handler nh = std::get_new_handler();
if (nh)
nh();
else
throw std::bad_alloc();
}
return p;
}
void operator delete(void* ptr) {
if (ptr)
::free(ptr);
}

It’s actually a wrapper for malloc, but it allocates at least one byte, and in case of memory failure, it attempts to get the user-set new_handler. If the user has previously set a new_handler, then it will call the new_handler; otherwise, it will throw a bad_alloc exception.

What is placement new

inline _LIBCPP_INLINE_VISIBILITY void* operator new (std::size_t, void* __p) _NOEXCEPT {return __p;}

This is an overload of operator new. It directly returns the value of the pointer. It seems quite useless. So how is it used?

void *buf = // Allocate memory for buf here
Class *pc = new (buf) Class();

The above code assigns buf’s memory to the pc pointer. It’s actually equivalent to:

((* Class)buf)->Class();
Class *pc = buf;

Select, Poll, Epoll: A Brief Introduction

Mon, 12 Mar 2018 13:19:00 +0000

Source: https://cloud.tencent.com/developer/article/1005481

Function Introduction

select

Function prototype:

int select (int n, fd_set *readfds, fd_set *writefds, fd_set *exceptfds, struct timeval *timeout);

fd_set is a set containing file descriptors, represented as a bitmap with n bits. This means that every call involves copying the file descriptor set to the kernel.

poll

Function prototype:

int poll (struct pollfd *fds, unsigned int nfds, int timeout);

Unlike select which uses three bitmaps to represent three fdsets, poll uses an array of pollfd. It takes the length of this array and a timeout value as parameters.

epoll

epoll was introduced in the 2.6 kernel and is an enhanced version of the previous select and poll.

int epoll_create(int size); //creates an epoll handle, size tells the kernel how large this monitoring set will be
int epoll_ctl(int epfd, int op, int fd, struct epoll_event *event);
int epoll_wait(int epfd, struct epoll_event * events, int maxevents, int timeout);

epoll_create is used to create an epoll handle. The size parameter is a suggestion to the kernel about how many file descriptors to pre-allocate, but it doesn’t limit the actual number in operation. epoll_ctl controls the epoll handle, allowing us to add, delete, or modify file descriptors. The separation of ctl from epoll_wait means we only need to copy the event set once. epoll_wait is used to get the set of events received from the kernel.

socket events

In the Linux 2.6 kernel events, a wakeup callback mechanism is set up. When a socket is waiting for an event to occur, it’s managed by the kernel’s socket sleep queue. When a socket event occurs, the kernel sequentially traverses each process on the socket sleep queue and notifies it of the event. During notification, it sequentially calls the callback function for that event.

The original select

select was initially just a simple implementation to try to solve the problem of checking multiple file descriptors.

What does select do?

After a select call, it does the following:

Copies the file descriptor sets passed as parameters to kernel space
Iterates through the file descriptors to check if there are any readable events; if there are, it returns
If there are no readable file descriptors, it begins to sleep, waiting for kernel socket events to occur
When awakened, it checks again to determine which file descriptor triggered the operation

What are the problems with select?

Each call to socket copies the data to kernel space, which is inefficient
When any socket is awakened, all sockets need to be traversed, wasting time

How can select be improved?

The monitored fds set is limited to 1024, which is too small; we want a larger set of fds to monitor
The fds set needs to be copied from user space to kernel space; we hope to avoid this copying
When some of the monitored fds have readable data, we want more precise notifications; we want to get a list of fds with readable events from the notification, rather than having to traverse the entire fds set to collect them.

The lackluster poll

poll only solves the first problem: the 1024 size limitation of fds. It’s merely a change in the interface of the parameters being passed.

The mature epoll

epoll solves the second and third problems.

For the second problem
- Breaking down function calls and further subdividing them For I/O multiplexing, we find that each call to select or poll repeatedly prepares (processes collectively) the entire set of fds that needs to be monitored. However, for frequently called select or poll, the frequency of changes in the fds set is much lower, so there’s no need to re-prepare (process collectively) the entire fds set each time. So, epoll introduces the epoll_ctl system call to separate the high-frequency epoll_wait from the low-frequency epoll_ctl. At the same time, epoll_ctl uses three operations (EPOLL_CTL_ADD, EPOLL_CTL_MOD, EPOLL_CTL_DEL) to distribute modifications to the monitored fds set, ensuring changes only happen when necessary. This turns the high-frequency, large memory copy (collective processing) of select or poll into low-frequency, small memory copy (distributed processing) of epoll_ctl, avoiding a large amount of memory copying.
Using a red-black tree Additionally, epoll uses epoll_ctl to add, delete, or modify the monitored fds set, which must involve fast fd lookup. Therefore, a data structure with low time complexity for adding, deleting, modifying, and querying is essential to organize the monitored fds set. In Linux kernel versions before 2.6.8, epoll used a hash to organize the fds set, so when creating an epoll fd, epoll needed to initialize the hash size. Hence, epoll_create(int size) had a parameter ‘size’ to allow the kernel to allocate the hash size based on it. In Linux kernel versions 2.6.8 and later, epoll uses a red-black tree to organize the monitored fds set, so the ‘size’ parameter in epoll_create(int size) is actually meaningless.
For the third problem
- Using a callback mechanism From the socket sleep queue wakeup logic above, we know that when a socket wakes up a wait_entry (process) sleeping in its sleep queue, it calls the wait_entry’s callback function, and we can do anything in this callback. To achieve traversal of only the ready fds, we need a place to organize those fds that are already ready. For this, epoll introduces an intermediate layer: a doubly linked list (ready_list), a separate sleep queue (single_epoll_wait_list). Unlike select or poll, epoll’s process doesn’t need to be inserted into all the sleep queues of the socket set for multiplexing. Instead, the process is only inserted into epoll’s separate sleep queue; the process sleeps on epoll’s separate queue, waiting for events to occur. At the same time, an intermediate wait_entry_sk is introduced, which is closely related to a specific socket sk. wait_entry_sk sleeps on the sk’s sleep queue, and its callback function logic is to put the current sk into epoll’s ready_list and wake up epoll’s single_epoll_wait_list. The callback function of the process sleeping on single_epoll_wait_list becomes clear: traverse all sks on the ready_list, call the poll function of each sk to collect events, and then wake up the process to return from epoll_wait.

Finally, Edge Triggering vs. Level Triggering

When discussing Epoll, we can’t ignore the two modes of Epoll events. Here are the basic concepts of the two modes:

Edge Triggered (ET) .Read events are triggered when the state of the socket’s receive buffer changes, i.e., when the empty receive buffer just receives data .Write events are triggered when the state of the socket’s send buffer changes, i.e., when the full buffer just frees up space Events are only triggered when the buffer state changes, such as when the data buffer changes from empty to containing data (unreadable to readable)
Level Triggered (LT) .As long as the socket’s receive buffer is not empty and has data to read, the read event continues to trigger .As long as the socket’s send buffer is not full and can continue to write data, the write event continues to trigger This conforms to intuitive thinking; the events returned by epoll_wait represent the socket’s state

Cainiao B2B First Interview Experience Record, C++ Backend Development

Fri, 09 Mar 2018 02:30:00 +0000

Introduction

I’m a second-year master’s student, and my advisor doesn’t allow internships, but I interviewed for spring internships to gain interview experience. My resume is very simple and modest, so the interviewer’s questions were relatively easy. The first interview was scheduled for 7 PM on March 8, 2018. Since I was doing an Alibaba test the night before, the interviewer said the interview would be postponed by one day, which gave me time to prepare. I especially reviewed basic knowledge again, which was very helpful.

Self-introduction

Omitted

Project experience and its impact on you

Omitted

What are the four layers of network architecture?

I answered the physical layer which includes fiber optics, switches, and twisted pairs; the network layer with IP; the transport layer with TCP, ICMP, and UDP; and the application layer with HTTP and DNS.
Standard answer: Actually, the TCP/IP standard does not define functionalities corresponding to the ISO data link layer and physical layer. Instead, it defines protocols like the Address Resolution Protocol (ARP), which provides interfaces between TCP/IP protocol data structures and actual physical hardware. So the first layer should be the link layer, which includes the ARP protocol and device drivers.

Which layer are you most familiar with?

I answered the transport layer with the TCP protocol.

What are the differences between TCP and UDP?

Omitted

Three-way handshake and four-way termination?

Omitted

Why are you familiar with this process?

I answered that I mainly learned it from books. Of course, I also used it in projects and observed the process by capturing packets with tcpdump and Wireshark.

How does I/O multiplexing work?

I answered that it’s because multithreading is resource-intensive, etc. After listening for a while, the interviewer said that’s not what they meant; they wanted to know the implementation principle of I/O multiplexing. I replied that I had forgotten, but I remembered that in the implementation process of select, poll, and epoll, efficient data structures are used to check which file descriptor has I/O operations.
The standard answer is:

The efficiency of epoll lies in the fact that when we call epoll_ctl to insert millions of handles, epoll_wait can still return quickly and effectively give us the handles with events. This is because when we call epoll_create, the kernel not only creates a file node in the epoll file system for us and builds a red-black tree in the kernel cache to store the sockets passed by epoll_ctl later, but also establishes a list to store ready events. When epoll_wait is called, it only needs to observe whether there is data in this list. If there is data, it returns; if there is no data, it sleeps until it returns when the timeout time arrives, even if the list has no data. Therefore, epoll_wait is very efficient. Another fatal weakness of traditional select/poll is that when you have a large socket collection, due to network latency, only part of the sockets are “active” at any time, but select/poll will linearly scan the entire collection for each call, causing efficiency to decrease linearly. However, epoll does not have this problem; it only operates on “active” sockets - this is because in the kernel implementation, epoll is implemented based on the callback function on each fd. Therefore, only “active” sockets will actively call the callback function, while other idle state sockets will not.

Explain polymorphism in C++

Omitted

What is a destructor and what is its purpose?

Omitted

What is the purpose of a virtual destructor?

Omitted

Should the parent class’s destructor be set as a virtual function?

Definitely yes

What is the difference between shallow copy and deep copy?

Omitted

If a function accepts an object as a parameter, is that a shallow copy or a deep copy?

I answered that it depends on the copy constructor. But I wasn’t sure about the specifics.
The standard answer should be: if references or pointers are used, no copying occurs. If the passed class hasn’t implemented a copy constructor, it’s a shallow copy. If it has implemented a copy constructor, it depends on whether the implemented copy constructor performs a shallow or deep copy.

What is a pointer in C language?

I answered that it’s first a variable that stores the address of the internal data type or object it points to. Dereferencing it retrieves the data or object.
In fact, a pointer is not necessarily a variable; there are also constant pointers, but generally, “pointer” is short for “pointer variable.” So the above expression doesn’t have any issues.

What is the difference between formal parameters and actual parameters?

Honestly, I wasn’t clear about this either. I said I didn’t know but explained pointers, references, and regular parameter passing.
Later I found out that actual parameters (arguments) refer to the parameters passed in when an external function is called, and formal parameters are the parameters used when processing inside the function.

Introduce the simulated annealing algorithm

Because it was mentioned on my resume

What is entropy, and what does an increase represent?

Omitted

The department primarily uses Java development. Would you be resistant if asked to do Java development?

Omitted

What would you like to know about the company?

Omitted

Aftermath

Total duration was 27 minutes. I spoke rather quickly and was a bit nervous and excited. That’s about it, and I’ll add more if I remember anything else.

HTTP Long and Short Connections

Wed, 07 Mar 2018 03:15:00 +0000

The Relationship Between HTTP Protocol and TCP/IP Protocol

HTTP long and short connections are essentially TCP long and short connections. HTTP is an application layer protocol that uses TCP protocol at the transport layer and IP protocol at the network layer. The IP protocol mainly solves network routing and addressing issues, while the TCP protocol mainly solves how to reliably deliver data packets over the IP layer, ensuring that all packets sent from one end are received at the other end of the network in the same order as they were sent. TCP has reliable, connection-oriented characteristics.

Understanding Why HTTP Protocol is Stateless

The HTTP protocol is stateless, meaning that the protocol has no memory capability for transaction processing, and the server doesn’t know what state the client is in. In other words, there is no connection between opening a webpage on a server and previously opening a webpage on that server. HTTP is a stateless connection-oriented protocol. Being stateless doesn’t mean that HTTP cannot maintain a TCP connection, and it certainly doesn’t mean that HTTP uses the UDP protocol (connectionless).

What are Long Connections and Short Connections?

In HTTP/1.0, short connections are used by default. This means that the browser and server establish a connection for each HTTP operation, but the connection is terminated once the task is completed. If a client browser accesses an HTML or other type of Web page that contains other Web resources such as JavaScript files, image files, CSS files, etc., the browser will establish an HTTP session each time it encounters such a Web resource. However, from HTTP/1.1 onwards, long connections are used by default to maintain connection characteristics. HTTP protocols using long connections will add this line of code to the response header: Connection:keep-alive. In the case of using a long connection, when a webpage is fully opened, the TCP connection used to transmit HTTP data between the client and server will not close. If the client accesses the webpage on this server again, it will continue to use this already established connection. Keep-Alive will not permanently maintain the connection; it has a retention time that can be set in different server software (such as Apache). Both the client and server must support long connections to implement them. The long and short connections of the HTTP protocol are essentially the long and short connections of the TCP protocol.

TCP Connection

When using TCP protocol for network communication, a connection must be established between the server and client before any actual read/write operations. When the read/write operations are completed and both parties no longer need this connection, they can release it. Establishing a connection requires a three-way handshake, while releasing it requires a four-way handshake, so establishing each connection consumes resources and time.

TCP Short Connection

Let’s simulate a short connection scenario. The client initiates a connection to the server, the server accepts the client connection, and both parties establish a connection. After the client and server complete a read/write operation, their connection is actively closed.

TCP Long Connection

Now let’s simulate a long connection scenario. The client initiates a connection to the server, the server accepts the client connection, and both parties establish a connection. After the client and server complete a read/write operation, their connection is not actively closed, and subsequent read/write operations will continue to use this connection.

Let’s first talk about the TCP keepalive function mentioned in TCP/IP in Detail. The keepalive function is mainly provided for server applications. Server applications want to know whether the client host has crashed, so they can use resources on behalf of the client. If the client has disappeared, leaving a half-open connection on the server, and the server is waiting for data from the client, the server will wait indefinitely for client data. The keepalive function attempts to detect such half-open connections on the server side.

If a given connection has no activity for two hours, the server sends a probe segment to the client. The client host must be in one of the following four states:

The client host is still running normally and is reachable from the server. The client’s TCP responds normally, and the server knows that the other party is normal. The server resets the keepalive timer after two hours.
The client host has crashed and is either shut down or restarting. In either case, the client’s TCP does not respond. The server will not receive a response to the probe and will time out after 75 seconds. The server sends a total of 10 such probes, each at an interval of 75 seconds. If the server does not receive a response, it assumes that the client host is closed and terminates the connection.
The client host has crashed and has restarted. The server will receive a response to its keepalive probe, which is a reset, causing the server to terminate the connection.
The client is running normally, but the server is unreachable. This situation is similar to category 2; TCP can only discover that it has not received a response to the probe.

How to Choose Between Long and Short Connections

Long connections are often used for frequent operations, point-to-point communications, and situations where the number of connections cannot be too many. Each TCP connection requires a three-way handshake, which takes time. If each operation first establishes a connection and then operates, processing speed will be reduced significantly. So, if the connection is not disconnected after each operation, subsequent processing can directly send data packets without establishing a TCP connection. For example, database connections use long connections. Using short connections for frequent communication will cause socket errors, and frequent socket creation is also a waste of resources.

Web services like HTTP websites generally use short connections because long connections consume certain resources on the server side. For websites with such frequent connections from thousands or even millions of clients, using short connections saves resources. If long connections were used, and there were thousands of users at the same time, each user occupying one connection would be unimaginable. Therefore, short connections are better for situations with high concurrency but where each user does not need frequent operations.

MySQL Study Notes

Mon, 05 Mar 2018 07:04:00 +0000

Introduction

I’m using Docker for learning. I’ve configured the docker-cn mirror.

Setting Up a MySQL Cluster

Use the following command to pull the MySQL image:

docker pull mysql

Use the following commands to set up one master and one slave:

docker run --name mysql-master -P -d -e MYSQL_ROOT_PASSWORD=ln mysql
docker run --name mysql-slave1 -P -d -e MYSQL_ROOT_PASSWORD=ln mysql

Where -d configures it as a daemon process, -P exposes all ports, and the root password is set to “ln”.

Use the following commands to enter the two Docker containers:

docker exec -it mysql-master bash
docker exec -it mysql-slave1 bash

Check the network IP. First, we need to install net-tools, which requires an update, and before updating, we need to change the source:

sed -i 's/deb.debian.org/mirrors.ustc.edu.cn/g' /etc/apt/sources.list
apt update
apt install net-tools

Enter ifconfig to check the IP address. The master is 172.17.0.2, and slave1 is 172.17.0.3.

Set login permissions In the MySQL master, follow these steps. First login, then set the root user to be accessible without a password from anywhere.

mysql -uroot -p
mysql> grant all privileges on *.* to root@'%' identified by '';
Query OK, 0 rows affected (0.02 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.02 sec)

In slave1, use the master’s database with an empty password:

mysql -uroot -p -h 172.17.0.2

Create a database in the master:

create database test_docker;

Check if the database exists in the slave:

show databases;

Miscellaneous

SQL is case-insensitive; SELECT and select are the same. It’s better to add a semicolon at the end of SQL statements. Use “use” to select a database. Use “set” to set values. The method to add users is with “create”. Then grant permissions to users for databases and tables.

mysql -u root -p
Enter password:**
mysql> use mysql;
mysql> create user 'linanwx'@'%' identified by 'ln';
mysql> grant all on *.* to 'linanwx'@'%';

SHOW DATABASES is used to display the current databases. SHOW TABLES is used to display the current tables.

Nowcoder Database Notes

Mon, 05 Mar 2018 07:04:00 +0000

Introduction

I’m not familiar with SQL but don’t want to spend time learning it. So let’s learn through practice.

Transactions

A transaction is the basic unit of concurrency control. A transaction is a sequence of operations that must either all be executed or none at all; it’s an indivisible work unit. For example, in bank transfers: debiting one account and crediting another account are two operations that must either both be executed or neither. Therefore, they should be considered as one transaction. Transactions are the unit by which databases maintain data consistency, ensuring data consistency at the end of each transaction.

In simple terms, a transaction is an atomic operation that cannot be divided.

Views

A view is a virtual table that doesn’t actually exist. It’s just a partial table extracted from the database to make it convenient for certain people to see. Views are defined by select queries—when you create a view, you’re actually executing a select statement in the database.

INSTEAD OF Triggers

Definition of Triggers Triggers are actually a special type of stored procedure. The main function of triggers is to monitor user modifications to data. If a trigger is built for a data table, then when the data table is modified (insert, update, or delete), SQL Server automatically executes the trigger to take appropriate measures for the data modification, thereby ensuring data integrity and security.

INSTEAD OF triggers specify that the trigger is executed instead of the triggering SQL statement, thus replacing the operation of the triggering statement.

Stored Procedures

A stored procedure is a group of SQL statements in a database designed to accomplish a specific function. It’s stored in the database and doesn’t need to be recompiled when called again after the first compilation. Users execute it by specifying the name of the stored procedure and providing parameters (if the stored procedure has parameters). Stored procedures are important objects in databases.

SQL Relational Algebra

Five Basic Operations:

Union (∪), Difference (-), Cartesian Product (×), Projection (σ), Selection (π)

Four Combined Operations:

Intersection (∩), Join (equijoin), Natural Join, Division (÷)

Union includes records that are in both tables. Difference includes records that are in the first table but not in the second. Cartesian product is all one-to-one combinations of records from both tables. Projection is vertical selection, while selection is horizontal selection. Intersection is similar to union, and join connects through common attributes. Division is a complex operation.

Relational Schema

A relational schema is a description of a relation. As the real world changes over time, the relation in a relational schema will also change. However, many existing facts in the real world dictate that all possible relations in a relational schema must satisfy certain integrity constraints. These constraints are either expressed through limitations on the value range of attributes, such as employee age less than 65 (mandatory retirement at 65), or through the interconnection between attribute values (mainly reflected in whether values are equal). The relational schema should capture these integrity constraints.

Non-relational Databases

NoSQL (NoSQL = Not Only SQL) means “not just SQL.” It’s used for storing ultra-large-scale data.

C++ Smart Pointers Learning

Sat, 03 Feb 2018 09:30:00 +0000

Introduction

Smart pointers seem to be used more and more widely. It’s necessary to remember them well.

shared_ptr

shared_ptr allows multiple pointers to point to the same object, while unique_ptr exclusively owns the object it points to.

Example:

#include <memory>
#include <thread>
class A
{
char * str;
public:
A(){
str = new char[20];
printf("Constructor\n");
}
~A()
{
printf("Destructor\n");
delete [] str;
}
};
std::shared_ptr<A> xixi;
void test()
{
std::shared_ptr<A> shared_a = std::make_shared<A>();
xixi = shared_a;
printf("%d\n", xixi.use_count());
}
int main()
{
test();
std::this_thread::sleep_for(std::chrono::seconds(1));
}

In the example above, xixi is a global smart pointer that will only release the object it points to when the program ends. If we remove xixi = shared_a, we would see the destructor being called before the delay in the main function.

unique_ptr

unique_ptr exclusively owns an object and can only be moved (std::move) between different unique_ptrs. There is no make_unique statement. You can only construct a unique_ptr by passing a pointer to an object in the constructor.

Use cases for unique_ptr:

Providing exception safety guarantees for dynamically allocated resources Let’s look at the following code:

void Func()
{
int *p = new int(5);
// ...（may throw an exception）
delete p;
}

This is our traditional approach: after we dynamically allocate memory, it’s possible that our subsequent code might not execute the delete operation due to an exception being thrown or early exit (if statement).

The solution is to use unique_ptr to manage dynamic memory. As long as the unique_ptr is successfully created, its destructor will be called, ensuring that dynamic resources are released.

void Func()
{
unique_ptr<int> p(new int(5));
// ...（may throw an exception）
}

weak_ptr

weak_ptr is also a powerful pointer, despite its name suggesting weakness. weak_ptr doesn’t control the lifecycle of an object and is used when you need to use an object without owning it. weak_ptr can be promoted to a shared_ptr if the object still exists.

Cross-compiling libcurl Dynamic Library for ARM

Tue, 30 Jan 2018 10:53:00 +0000

Introduction

Recently, due to project requirements, I needed to cross-compile libcurl to enable HTTPS communication. Here I’ll document the process.

First, download the latest version of the source code and extract it:

I used the cross-compilation toolchain that comes with Deepin Linux. Just install it with sudo apt install gcc-5-arm-linux-gnueabihf.

arm-linux contents:

In the openssl directory, run the following commands:

source ../arm-linux
./Configure --prefix=$(pwd)/../output shared no-asm linux-armv4
make -j4
make install

First, import the environment variables, then compile with the output path set to the “output” directory.

Then in the libcurl directory, run the following commands:

source ../arm-linux
./configure --with-ssl=$(pwd)/../output --prefix=$PWD/../output --build=x86_64-pc-linux-gnu --host=arm-linux-gnueabihf CC=arm-linux-gnueabihf-gcc-5 --without-zlib

Here’s a summary of the curl configuration:

 curl version: 7.58.0
Host setup: x86_64-pc-linux-gnu
Install prefix: /media/linan/72605383-b1ba-4c79-90fd-443514396fbc/curl_workdir/curl-7.58.0/output
Compiler: arm-linux-gnueabihf-gcc-5
SSL support: enabled (OpenSSL)
SSH support: no (--with-libssh2)
zlib support: no (--with-zlib)
brotli support: no (--with-brotli)
GSS-API support: no (--with-gssapi)
TLS-SRP support: enabled
resolver: POSIX threaded
IPv6 support: enabled
Unix sockets support: enabled
IDN support: no (--with-{libidn2,winidn})
Build libcurl: Shared=yes, Static=yes
Built-in manual: enabled
--libcurl option: enabled (--disable-libcurl-option)
Verbose errors: enabled (--disable-verbose)
SSPI support: no (--enable-sspi)
ca cert bundle: no
ca cert path: no
ca fallback: no
LDAP support: no (--enable-ldap / --with-ldap-lib / --with-lber-lib)
LDAPS support: no (--enable-ldaps)
RTSP support: enabled
RTMP support: no (--with-librtmp)
metalink support: no (--with-libmetalink)
PSL support: no (libpsl not found)
HTTP2 support: disabled (--with-nghttp2)
Protocols: DICT FILE FTP FTPS GOPHER HTTP HTTPS IMAP IMAPS POP3 POP3S RTSP SMB SMBS SMTP SMTPS TELNET TFTP

Then run make and make install. An “output” directory will appear in the curl directory.

Finally, what we need are the dynamic libraries from openssl and curl: libssl.so, libcrypto.so, and libcurl.so.

Algorithm Problem — Shooting Game

Tue, 23 Jan 2018 01:39:00 +0000

Link: https://www.nowcoder.com/questionTerminal/d3f26db0325444078717cc802e0056d8 Source: NowCoder

Xiayi is playing a new shooting game that takes place on a two-dimensional plane. Xiayi is at the origin (0,0), and there are n monsters on the plane, each with coordinates (x[i], y[i]). With one shot, Xiayi can eliminate all monsters that are on the x-axis and y-axis (including the origin) at once. Xiayi is a VIP player in this game and has two special privilege operations:

Make all monsters on the plane move simultaneously in any same direction for any same distance
Make all monsters on the plane rotate around Xiayi (0,0) by any same angle simultaneously Xiayi wants to take a shot. Before shooting, he can use these two privilege operations any number of times.

Xiayi wants to know the maximum number of monsters he can eliminate at once when he shoots. Please help Xiayi.

As shown in the example:

All points can be rotated clockwise or counterclockwise by 45° with respect to the origin (0,0), which places all points on the coordinate axes, so all 5 monsters can be eliminated.

Input Description:

The input consists of three lines. The first line contains a positive integer n (1 ≤ n ≤ 50), representing the number of monsters on the plane. The second line includes n integers x[i] (-1,000,000 ≤ x[i] ≤ 1,000,000), representing the x-coordinate of each monster, separated by spaces. The third line includes n integers y[i] (-1,000,000 ≤ y[i] ≤ 1,000,000), representing the y-coordinate of each monster, separated by spaces.

Output Description:

Output an integer representing the maximum number of monsters Xiayi can eliminate.

Example 1

Input

5 0 -1 1 1 -1 0 -1 -1 1 1

Output

Analysis

The problem is equivalent to finding a cross shape that covers as many points as possible. Considering that one line can cover at least two points, and adding a perpendicular line can cover at least 3 points, we can iterate based on this. For any three points, we select two of them to form a line (three possible combinations), and for the third point, we create a perpendicular line to this line. This cross shape already passes through three points. For the remaining points, we check if they are on this cross shape. To determine if a point is on the cross shape, first check if it’s on the same line as the first line. Otherwise, determine if the line formed by this point and the third point is perpendicular to the second line. When there are three or fewer points, we can cover all of them.

#include <vector>
#include <iostream>
#include <algorithm>
using namespace std;
struct point{
int x = 0;
int y = 0;
};
bool is_sameline(point p1, point p2, point p3){
return ((p1.x - p2.x) * (p2.y - p3.y) - (p2.x - p3.x) * (p1.y - p2.y)) == 0;
}
bool is_vertical(point p1, point p2){
return (p1.x * p2.x + p1.y * p2.y) == 0;
}
bool is_vertical(point p1, point p2, point p3, point p4){
point v1, v2;
v1.x = p1.x - p2.x;
v1.y = p1.y - p2.y;
v2.x = p3.x - p4.x;
v2.y = p3.y - p4.y;
return is_vertical(v1, v2);
}
int main()
{
int n, ret = 0;
cin >> n;
point inputs[n];
for (int i = 0; i < n; i++)
cin >> inputs[i].x;
for (int i = 0; i < n; i++)
cin >> inputs[i].y;
if (n < 4)
{
cout << n << endl;
return 0;
};
vector<int> select = {1, 1, 1};
for (int i = 0; i < n - 3; i++)
select.push_back(0);
do
{
vector<point> shizi;
for (int i = 0; i < n; i++)
{
if (select[i])
{
shizi.push_back(inputs[i]);
}
}
vector<vector<point>> status;
status.push_back({shizi[0], shizi[1], shizi[2]});
status.push_back({shizi[0], shizi[2], shizi[1]});
status.push_back({shizi[1], shizi[2], shizi[0]});
for (auto points : status)
{
int count = 0;
for (int i = 0; i < n; i++)
{
if (!select[i])
{
if (is_sameline(points[0], points[1], inputs[i]))
count++;
if (is_vertical(points[0], points[1], points[2], inputs[i]))
count++;
}
}
ret = max(ret, count);
}
} while (prev_permutation(select.begin(), select.end()));
cout << ret + 3 << endl;
return 0;
}

Algorithm Problem — Longest Common Bracket Sequence

Sun, 21 Jan 2018 14:05:00 +0000

Link: https://www.nowcoder.com/questionTerminal/504ad6420b314e5bb614e1684ad46d4d Source: NowCoder

A valid bracket matching sequence is defined as:

An empty string "" is a valid bracket sequence
If “X” and “Y” are valid sequences, then “XY” is also a valid bracket sequence
If “X” is a valid sequence, then “(X)” is also a valid bracket sequence
Every valid bracket sequence can be generated using the above rules For example, “”, “()”, “()()()”, “(()())”, “(((())))” are all valid. A sequence obtained by removing zero or more characters from a string S is called a subsequence of S. For example, the subsequences of “abcde” include “abe”, “”, “abcde”, etc. Define LCS(S,T) as the length of the longest common subsequence between string S and string T, which is the length of the longest sequence W that is both a subsequence of S and a subsequence of T. Xiaoyi gives you a valid bracket matching sequence s, and he hopes you can find a bracket sequence t with the following characteristics:
t is different from s but has the same length
t is also a valid bracket matching sequence
LCS(s, t) is the largest among all t that satisfy the above two conditions Since there may be multiple such t, Xiaoyi needs you to calculate how many such t exist.

As shown in the example: s = “(())()”, valid bracket matching sequences with the same length as s include: “()(())”, “((()))”, “()()()”, “(()())”, where LCS("(())()", “()(()))”) is 4, and the others are all 5, so the output is 3.

Input Description:

Output Description:

Output a positive integer, the number of t that satisfy the conditions.

Example 1

Input

(())()

Output

Analysis

The problem requires iterating through all bracket sequences of the same length and calculating the length of the longest common subsequence between each bracket sequence and the given string. Then, count how many times the maximum length occurs.

If we were to follow the problem description exactly, it would be impossible to solve. Just generating bracket sequences of up to 50 characters would likely exceed the time limit.

Since we’re searching for how many times the longest subsequence occurs, we can assume that the longest subsequence is the original string length - 1, and then find all bracket sequences that can be formed from this subsequence. If none exist, we try subsequences of length original length - 2, and so on, decreasing the subsequence length until we find a solution.

In fact, there always exists a subsequence of length original length - 1 that can form another valid bracket sequence different from the original. This is because any bracket sequence can be modified by moving just one bracket to form another different valid bracket sequence (think about why). If this is possible, then the longest common subsequence between the new string and the original string would be n-1, since you only moved one bracket to a new position. The order of all other brackets remains unchanged. So, each time we remove one bracket and insert it in a new position. We count how many valid bracket sequences can be formed this way, subtract the original string (i.e., subtract 1), and that’s the answer to the problem.

#include <vector>
#include <iostream>
#include <algorithm>
#include <set>
using namespace std;
bool isKuohao(const string & str)
{
int count=0;
for(auto c:str)
{
if(c == '(')
count ++;
if(c == ')')
count --;
if(count<0)
return false;
}
return count == 0;
}
int main()
{
int ret=0;
string input;
cin >> input;
set<string> haveChose;
set<string> haveBuild;
/* Select any one bracket */
for (int i = 0; i < input.size(); i++)
{
char q[] = {input[i], '\0'};
/* Delete that bracket */
string delete_char(input);
delete_char.erase(delete_char.begin() + i);
if (haveChose.count(delete_char))
continue;
else
haveChose.insert(delete_char);
/* Insert the original bracket into a new array */
for (int j = 0; j < delete_char.size() + 1; j++)
{
string add_delete_char(delete_char);
add_delete_char.insert(j, q);
if (haveBuild.count(add_delete_char))
continue;
else
haveBuild.insert(add_delete_char);
if(isKuohao(add_delete_char)){
ret ++;
}
}
}
cout << ret-1 << endl;
}

Algorithm Problem — Touring the Magic Kingdom

Fri, 19 Jan 2018 10:05:00 +0000

Link: https://www.nowcoder.com/questionTerminal/f58859adc39f4edc9cd8e40ba4160339 Source: NowCoder

The Magic Kingdom has a total of n cities, numbered from 0 to n-1. The roads between the n cities form exactly a tree. Xiaoyi is currently in city 0. Each move, Xiaoyi will walk from his current city to an adjacent city. Xiaoyi can move at most L times. If Xiaoyi has reached a city, he is considered to have toured that city. Xiaoyi now wants to make a good travel plan so that he can tour as many cities as possible. Please help him calculate the maximum number of cities he can tour (note that city 0 is already toured, and cities already toured are not counted again).

Input Description:

The input consists of two lines. The first line includes two positive integers n (2 ≤ n ≤ 50) and L (1 ≤ L ≤ 100), representing the number of cities and the number of moves Xiaoyi can make. The second line includes n-1 integers parent[i] (0 ≤ parent[i] ≤ i). For each valid i (0 ≤ i ≤ n - 2), there is a road connecting city (i+1) and city parent[i].

Output Description:

Output an integer representing the maximum number of cities Xiaoyi can tour.

Analysis

After abstracting the problem, it means traversing a tree, and after a specified number of steps, finding the path that passes through the maximum number of nodes. It might be easier to understand if we hang this tree from the root node. Although some solutions grow from bottom to top, I rebuilt the tree and used a hanging tree approach.

Starting from the root node, first determine whether the left subtree or the right subtree has a greater depth, and traverse the deeper subtree first. Continue until all steps are used up.

The depth of the tree can be easily calculated through post-order traversal, but it turns out this solution only passes 60% of the test cases.

45 73
0 0 0 1 0 0 3 5 6 8 7 9 1 10 1 2 15 6 8 11 14 17 8 14 3 21 23 3 21 15 12 5 21 31 11 13 7 17 20 26 28 16 36 26

The error is in this test case. The correct answer is 41, but using a simple greedy algorithm only yields 39 cities.

Later, looking at the explanation, I still couldn’t fully understand it. In any case, I saw that in the correct solution, the final answer is obtained directly after calculating the depth.

Suppose we have already calculated the maximum depth of each node, represented by deep[i], with the depth of the bottom layer of the tree being 1.

Obviously, the longest path from the root node to any node = deep[0] - 1.

Based on this path, we can visit some additional nodes. However, each time after visiting these nodes, we must return to this path. This round trip requires an extra two steps for each node visited, and visiting two nodes requires four extra steps.

Looking at the diagram makes it easier to understand:

Reference code:

#include <vector>
#include <iostream>
using namespace std;
vector<vector<int> > tree;
vector<int> deep;
void calc_deep(int i)
{
int max_deep = 0;
for(auto j:tree[i])
{
calc_deep(j);
max_deep = max(deep[j], max_deep);
}
deep[i] = max_deep + 1;
}
int main()
{
int n, L;
cin >> n >> L;
/* Build the tree */
tree.resize(n);
deep.resize(n);
for(int i=0;i<n-1;i++)
{
int num;
cin >> num;
tree[num].push_back(i+1);
}
/* Calculate depth */
calc_deep(0);
// int validpath = min(deep[0] -1,L);
// cout << min(n, 1 + validpath + (L - validpath)/2) << endl;
int long_path = deep[0] - 1;
if(long_path > L) cout << L + 1;
else cout << 1 + long_path + (L - long_path)/2;
}

Bash Language Study Notes

Thu, 18 Jan 2018 09:23:00 +0000

Introduction

Most people have used terminal command lines. Sometimes we need to repeatedly enter the same commands, so we can write our own scripts to automate this work.

Specifying the Interpreter

#!/bin/bash This line specifies bash as the interpreter for our script. If we’re writing in Python syntax, we should change it to Python.

Defining Variables

variable_name=value For example: name=“hahaha” For ordinary characters, adding double quotes or single quotes doesn’t matter. name=hahaha also works.

Using Variables

Add $ before a variable name to use its value. This means to evaluate the symbol’s value.

Strings

Quotes have the following characteristics:

Spaces act as word separators
Words in single quotes
Character pattern matching
Pathname expansion
Process substitution (redirection)

Single quotes won’t process the content further, while double quotes will evaluate the content. You can see the difference between echo “$name” and echo ‘$name’ to understand this.

String concatenation is just placing two strings together. ${#string} is used to get the string length, ${string:1:4} is used for slicing.

Getting Input

read inputstr read -p “please input : " inputstr The above two lines get input content from standard input.

Getting Script Execution Results

For example, to get the output of a Python script: into=$(‘python3’ ‘pyscript.py’ ‘arg1’ ‘arg2’) arg1 and arg2 are parameters

Learning and Thinking About Dynamic Programming Algorithms

Thu, 18 Jan 2018 08:52:00 +0000

Introduction

Dynamic programming is a common type of problem in coding interviews. Mastering it is crucial.

NetEase Problem

Little Q and Dr. Niu are singing a song together. This song consists of n notes, each represented by a positive integer. Each note must be sung by either Little Q or Dr. Niu. The difficulty of singing a series of notes equals the sum of the absolute differences between all adjacent notes. For example, if a sequence of notes is 8, 8, 13, 12, then its difficulty is |8 - 8| + |13 - 8| + |12 - 13| = 6 (where || represents absolute value). Now we need to distribute these n notes between Little Q and Dr. Niu to minimize the sum of their singing difficulties. Calculate the minimum possible total difficulty. As shown in the example: Little Q chooses to sing {5, 6} with difficulty 1, Dr. Niu chooses to sing {1, 2, 1} with difficulty 2, the sum of difficulties is 3, which is the minimum possible difficulty.

Greedy Approach (Incorrect)

Sort all the numbers and use the largest difference between two numbers as a dividing point. Assign the lower half to Little Q and the upper half to Dr. Niu, then calculate the result. This approach passes 60% of the test cases. If time is limited or you don’t know how to solve it optimally, you can use this approach.

#include <vector>
#include <iostream>
#include <algorithm>
#include <cmath>
using namespace std;
int main(){
int n;
cin >> n;
vector<int> nums;
for(int i=0;i<n;i++)
{
int num;
cin >> num;
nums.push_back(num);
}
vector<int> nums_sort(nums.begin(), nums.end());
sort(nums_sort.begin(), nums_sort.end());
int max_gap = 0;
auto max_gap_it = nums_sort.begin();
for(auto it=nums_sort.begin(); it!=nums_sort.end()-1; it++){
if(*(it+1) - *it > max_gap){
max_gap = *(it+1) - *it;
max_gap_it = it;
}
}
int max_gap_num = *max_gap_it;
int ret = 0;
int last_1 = -1;
int last_2 = -1;
for(auto it = nums.begin(); it!=nums.end(); it++){
if(*it <= max_gap_num){
if(last_1 != -1){
ret += abs(*it - last_1);
}
last_1 = *it;
}
else{
if(last_2 != -1){
ret += abs(*it - last_2);
}
last_2 = *it;
}
}
cout << ret;
}

Thinking About Dynamic Programming

Since I don’t know much about dynamic programming, let’s start with the coin change problem.

Coin Change Problem

The core of dynamic programming lies in breaking down the problem and recursively combining subproblems to solve the original problem.

For example, suppose we have coins with denominations of 1, 3, and 5, and we want to make change for 11 using the minimum number of coins. If you’re human, you might say, “I can easily see that two 5-coins and one 1-coin, for a total of 3 coins, is optimal.” Let’s try a different example: what about making change for 237? You might say, “I’ll first use 5-value coins to make 200, then find the optimal combination for the remaining 37.” This method assumes that splitting 237 into two parts and finding the optimal solution for each part will yield the global optimal solution. If that were always true, there would be no problem. But the issue is with splitting into 200 and 37 - there’s no proof that this is an effective split. Dynamic programming is built on effective partitioning.

Both divide-and-conquer algorithms and dynamic programming are based on partitioning. The difference is that dynamic programming involves state transitions, while divide-and-conquer doesn’t. Divide-and-conquer can decompose from top to bottom, while dynamic programming usually builds from bottom to top.

Dynamic programming involves two dimensions. The first dimension is usually related to the scale of the problem, and the second dimension needs to be extracted from the problem. In this coin change problem, the second dimension is the allowed coins: for example, allowing only 1-value coins, allowing 1 and 3-value coins, or allowing 1, 3, and 5-value coins. We denote this with index 1, 2, 3, and use j to represent this variable. If we use i to represent the current problem scale (237), then we want to find c[i][j], which is c[237][3]. For this problem, we can make two assumptions: first, the optimal solution uses at least one 5-value coin; second, the optimal solution uses no 5-value coins. For the second assumption, c[237][3] = c[237][2], because if we don’t use any 5-value coins, then the optimal solution should be the same as using only 1 and 3-value coins. For the first assumption, since the solution must contain at least one 5-value coin, after removing this 5-value coin, we have 232 left. Since c[237][3] is the optimal value for making change for 237 using coins of values 1, 3, and 5, and it must contain at least one 5-value coin, the remaining coins that make up 232 must also be in an optimal state (if the subproblem weren’t optimal, that is, if there were a better solution for making 232 within the solution for 237, then the solution for 237 wouldn’t be optimal, contradicting our assumption). So c[232][3] = c[237][3] - 1.

This allows us to derive: c[i][j] = c[i][j-1] (Assumption 1) c[i][j] = c[i-value of denomination j][j] + 1 (Assumption 2) When written in the form of min(), this becomes the familiar state transition equation.

We ultimately want to find c[11][5], and we work from the c[0][0] state towards the bottom right corner of the table to derive the answer.

Two-Person Singing Difficulty Problem https://www.nowcoder.com/questionTerminal/fddf64d5757e41ec93f3ef0c0a10b891

We must remember:

The optimal solution to the problem is not necessarily the value in the bottom-right corner of the state matrix

We must always remember that the bottom-right cell of the state matrix is not necessarily the direct optimal solution. Thinking this way often leads to confusion. Instead, we should think that the matrix indices i and j are definitely related to the two dimensions of the problem, but dp[i][j] does not necessarily represent the value of the optimal solution.

For example, in this problem, we need to find the optimal difficulty coefficient. If dp[i_max][j_max] were the optimal difficulty coefficient, what would i and j index? This is unsolvable. Looking at it differently, what exactly is the optimal difficulty coefficient? Since state transitions exist, the optimal state must be the minimum value obtained from multiple states. What does i represent? If i represents the current note being sung, what does j represent? J can only represent the note sung by the other person. Then, in the optimal state, how many situations are there? We calculate the optimal value from these situations. In the optimal state, one person must be singing the last note, while the other person might not be singing any note, might be singing the first note, might be singing the second note… might be singing the second-to-last note. We calculate the minimum value of all these situations to find the optimal solution.

Another question is whether i and j should be assigned to specific people. If i represents Little Q’s notes and j represents Dr. Niu’s notes, then the matrix would be symmetric, and we’d be calculating half of the content redundantly. Actually, we don’t care who is singing what, because they are equivalent. We can only think of it as one person currently singing up to position i, and the other person last sang up to position j.

What states can c[i][j] transition from? For example, if one person has sung up to the 6th note, and the other person last sang up to the 3rd note, if the current total difficulty is minimal, then: The 6th, 5th, and 4th notes are all sung by the first person. This is because the second person has only sung up to the 3rd note and cannot sing later notes. So c[6][3] = c[5][3] + difficulty difference between notes 5 and 6 = c[4][3] + difficulty difference between notes 4 and 5. But how is c[4][3] transitioned? If the first person has sung up to the 4th note, and the other person has sung up to the 3rd note, this means the first person interrupted the second person’s singing. Since we don’t know from where the first person jumped to the 4th note, we assume all possible notes. If they jumped from the 2nd note, then it’s c[3][2] + difference between notes 3 and 4. If they jumped from the 1st note, then it’s c[3][1] + difference between notes 1 and 4. If they didn’t jump, then it’s c[3][0] + difference between notes 0 and 4. Since the last note sung by the second person is always less than the note being sung by the first person, we don’t need to consider the case where j >= i.

Based on this, here’s the code:

#include <vector>
#include <iostream>
#include <algorithm>
#include <cmath>
#include <array>
using namespace std;
vector<int> nums;
array<array<int, 2100>, 2100> dp;
/* Calculate the difficulty between two notes. Note that the index of the first note in the input data is 0, so there's a difference of 1. If the starting note is 0, it means it's the first time singing, so no additional difficulty */
int diffcult(int s, int e)
{
if(s == 0) return 0;
// printf("diffcult %d %d to %d %d is %d \n", s,nums[s-1], e,nums[e-1], abs(nums[e-1] - nums[s-1]));
return abs(nums[e-1] - nums[s-1]);
}
int main()
{
int n;
cin >> n;
for (int i = 0; i < n; i++)
{
int num;
cin >> num;
nums.push_back(num);
}
/* Start from 0 and consider all notes. The worst case for the first person is not singing at all, in which case j also doesn't sing, and we continue */
for (int i = 0; i <= n; i++)
{
/* Also start from 0 here, because in the worst case, the first person doesn't sing at all, which is position 0 */
for (int j = 0; j <= n; j++)
{
/* Don't consider later cases. This could also be incorporated into the range limitation above */
if (j >= i)
continue;
/* In the general case, for example, if the first person is singing the 6th note and the second person is singing the 3rd note, it clearly transitions from dp[5][3] */
if (j + 1 < i)
{
dp[i][j] = dp[i - 1][j] + diffcult(i - 1, i);
}
/* Otherwise, if the second person just finished singing and the first person took over, the first person's note could be jumped from earlier */
if (j + 1 == i)
{
/* k represents which note i jumped from */
int min_cost = -1;
for (int k = 0; k < j; k++)
{
int cost = dp[j][k] + diffcult(k, j + 1);
/* Only record the minimum jump difficulty */
if(min_cost == -1 || min_cost > cost){
min_cost = cost;
}
}
if(min_cost == -1) min_cost = 0;
dp[i][j] = min_cost;
}
// for(int a=0;a<=n;a++){
// for(int b=0;b<=n;b++){
// if(a==i && b == j){
// printf("\t【%d】", dp[a][b]);
// }
// else printf("\t%d", dp[a][b]);
// }
// // printf("\n");
// }
// printf("\n");
}
}
int min_cost = -1;
for(int j=0;j<n;j++){
if(min_cost == -1 || min_cost > dp[n][j]){
min_cost = dp[n][j];
}
}
printf("%d", min_cost);
}

Longest Palindromic Substring Problem https://leetcode.com/problems/longest-palindromic-substring/description/

I now understand why dynamic programming is presented in this way. The key point of two-dimensional dynamic programming problems is:

The planar expansion of linear growth problems

For example, in the coin change problem, expanding the second dimension represents making change with the current coin type. In the singing problem, it’s the minimum difficulty when singing up to the current position while the other person last sang at a certain position. For the longest palindromic substring, it becomes the starting position.

In the longest palindromic substring, i represents the starting character, and j represents the ending character of the substring. When i is the starting character and j is the ending character, we have the answer to the problem.

For the smallest subproblem, which is a single character, its length is 1. For longer strings, if the characters at both ends are different, then it equals the smaller value of removing either the right or left character. If the characters at both ends are the same, then it equals the value of the substring with both ends removed plus 2.

#include <vector>
#include <string>
using namespace std;
class Solution {
public:
int longestPalindromeSubseq(string s) {
vector<vector<int>> dp;
dp.resize(s.size() + 1);
for (int i = 0; i < s.size() + 1; i++)
{
dp[i].resize(s.size() + 1);
}
int len = s.size();
for(int t=0;t<len;t++){
for(int j=t; j<len;j++){
int i = j-t;
if(i==j){
dp[i][j] = 1;
}
else if(s[i] == s[j]){
dp[i][j] = dp[i+1][j-1] + 2;
}
else{
if(dp[i+1][j]> dp[i][j-1]){
dp[i][j] = dp[i+1][j];
}
else{
dp[i][j] = dp[i][j-1];
}
}
}
}
return dp[0][len-1];
}
};

Red-Black Tree Implementation Principles

Mon, 15 Jan 2018 10:00:00 +0000

Introduction

Binary search trees can degrade in performance Balanced binary trees require too much overhead during insertion and deletion to maintain search performance This is where red-black trees come in. Unlike balanced binary trees, red-black trees don’t require the height difference between two subtrees to be at most 1 at all times Reference: http://www.sohu.com/a/201923614_466939

Properties

Nodes consist of red and black colors, with the root node being black
Leaf nodes are made up of black NIL nodes
A red node’s children must be black nodes, and must contain two leaf nodes. This means there cannot be two consecutive red nodes.
Starting from any node, any path to any NIL node must pass through the same number of black nodes

This means that starting from the root node to any leaf node, the difference between the longest path and the shortest path is no more than half of the longest path. In this case, the longest and shortest paths have the same number of black nodes, and the longest path can have at most one additional red node for each black node.

Recoloring and Rotation

When a node is inserted (or deleted), the properties of the red-black tree may be violated. Recoloring and rotation are used to restore these properties.

Recoloring

If two consecutive nodes are red, recoloring is needed

Rotation

If a path has too many black nodes, rotation is needed

Conditions for Recoloring and Rotation

This will be summarized later

Linux: How to Use Terminal Background Tasks and Multiple Windows

Thu, 11 Jan 2018 15:01:00 +0000

byobu

byobu is a tool that replaces screen and tmux. You can install it using apt install byobu. Running byobu is simple and its commands are straightforward:

f2: Create a new window
f3: Return to the window on the left
f4: Return to the window on the right
f9: Exit byobu, commands running in open windows will not be interrupted

Strongly recommended for those who don’t want to use screen or tmux! You’ll be satisfied.

C++ Basics — Differences Between Pointers and References

Wed, 10 Jan 2018 08:44:00 +0000

Differences Between Pointers and References

Pointers can be null pointers, but references cannot be null references. This leads to the following distinction.
Pointers can be invalid, while references are always valid. If a pointer is a null pointer, it cannot be legally used. This requires checking the validity of a pointer each time it is used. In contrast, references don’t need to be checked.
Pointers can be modified, references cannot. A pointer is a variable that stores an address pointing to an object, and it can be changed to point to other addresses, no longer pointing to the original object. However, a reference is bound to an object during initialization, and while you can modify the contents of the object, the reference cannot be reassigned to refer to another object.
Based on the above situations, pointers and references have different applications. Pointers can be used in scenarios where you need to point to different objects at different times or where you might need to point to no object at all. If you always need to point to a single object and will not change what you’re pointing to after initialization, you should use a reference.

Thoughts on the Phenomenon of Telecom Hijacking Red Packet Ads on CSDN Mobile

Sun, 31 Dec 2017 06:24:00 +0000

Introduction

Recently I suspected being hijacked by China Telecom, so I’m recording my analysis here.

Phenomenon

I use a Sichuan Telecom SIM card with my Samsung S8. When visiting the following websites, I am redirected to a full-screen advertisement, which is the Alipay red packet interface. The user experience is terrible. Below is an address that gets hijacked: http://m.blog.csdn.net/skyroben/article/details/70195575

Analysis

When using SS proxy (4G) on my phone to visit the above website, there’s no advertisement or hijacking
With direct connection via Telecom WiFi, hijacking occurs
Using hotspot and accessing the URL from a computer, no hijacking occurs
Using hotspot and changing the user agent on the computer to an Android client, hijacking occurs with some probability??
Without using hotspot, with a wired connection (Telecom) on the computer, changing the user agent to Android client, hijacking occurs with some probability??
Using Chrome on mobile (S8), connected to Telecom WiFi, adding view-source: before the URL to view the source code, and saving the results for comparison on the computer, no differences were found
Using Chrome on mobile (S8), connected to China Mobile WiFi, hijacking occurs! (Script cache?)
Using iOS Chrome, connected to Telecom WiFi, no hijacking

Summary

Since the webpage source code doesn’t change after hijacking, it might be a script hijack. Since hijacking also occurs when using China Mobile WiFi, we can’t rule out the possibility that CSDN added the advertisements themselves. Using an SS proxy can bypass the hijacking, indicating that the hijacking is regional. The fact that computers and iOS devices aren’t hijacked suggests that the hijacking specifically targets Android browsers.

Conclusion

Regular users should avoid using Telecom networks and Android phones, and all websites are advised to use HTTPS encryption.

How to Implement STL Vector in C++

Sat, 30 Dec 2017 08:49:00 +0000

Introduction

In interviews, you might be asked how to implement a vector. This requires understanding the underlying implementation of vector. Before that, you need to learn about dynamic memory management, especially allocators, which are explained in the C++ Primer book.

Basic Contents to Implement

On the cplusplus website, common usages are as follows: Member functions (Constructor) Construct vector (public member function) (Destructor) Vector destructor (public member function)

Iterators: begin Return iterator to beginning (public member function) end Return iterator to end (public member function)

Capacity: size Return size (public member function)

Element access: operator[] Access element (public member function) at Access element (public member function)

Modifiers: push_back Add element at the end (public member function) pop_back Delete last element (public member function)

Main Structure of a Basic Vector

#include <cstddef>
#include <stdexcept>
#include <memory>
#include <iterator>
template <typename T>
class vector
{
public:
using value_type = T;
using iterator = value_type *;
using size_type = std::size_t;
public:
vector() = default;
~vector();
iterator begin() const;
iterator end() const;
size_type size() const;
value_type &operator[](size_type i) const;
value_type &at(size_type i) const;
void push_back(const value_type &new_elem);
void pop_back();
private:
iterator startptr = nullptr;
iterator endptr = nullptr;
iterator capptr = nullptr;
std::allocator<value_type> alloc;
private:
void check_cap();
void free();
};

In our class, we’ve simply implemented iterators, push_back, pop_back, as well as the [] operator, at function, and size function. To implement memory management, we also need to implement constructors, destructors, and capacity checking functions.

Internal Implementation of Basic Functions

template <typename T>
typename vector<T>::iterator vector<T>::begin() const
{
return startptr;
}
template <typename T>
typename vector<T>::iterator vector<T>::end() const
{
return endptr;
}
template <typename T>
typename vector<T>::size_type vector<T>::size() const
{
return endptr - startptr;
}
template <typename T>
typename vector<T>::value_type &vector<T>::operator[](size_type i) const
{
return *(startptr + i);
}
template <typename T>
typename vector<T>::value_type &vector<T>::at(size_type i) const
{
if (startptr + i >= endptr)
{
throw std::runtime_error("out of range!");
}
return *(startptr + i);
}
template <typename T>
vector<T>::~vector()
{
free();
}

The above are implementations of simple functions. They simply retrieve internal data.

template <typename T>
void vector<T>::free()
{
if (startptr)
{
for (auto p = startptr; p != endptr; p++)
{
alloc.destroy(p);
}
alloc.deallocate(startptr, endptr - startptr);
}
}
template <typename T>
void vector<T>::check_cap()
{
if (endptr == capptr)
{
int newsize = size() ? size() << 1 : 1;
auto newstartptr = alloc.allocate(newsize);
auto newendptr = uninitialized_copy(std::make_move_iterator(startptr), std::make_move_iterator(endptr), newstartptr);
free();
startptr = newstartptr;
endptr = newendptr;
capptr = newstartptr + newsize;
}
}
template <typename T>
void vector<T>::push_back(const value_type &new_elem)
{
check_cap();
alloc.construct(endptr, new_elem);
endptr++;
}
template <typename T>
void vector<T>::pop_back()
{
if(endptr-startptr>0){
alloc.destroy(endptr);
endptr--;
}
}

This part deals with memory and uses allocator to manage memory. The constructor separates the allocation of space and construction, dividing the process into allocating space, reclaiming space, destruction, and construction. The capacity check involves these four scenarios: first allocate new space, then construct new elements in the new position, then destruct old elements, and release old space. The free function is used to destruct old elements and release old space. Here, uninitialized_copy function and make_move_iterator, move iterators, and uninitialized copy functions are used to construct new elements in new positions, aiming to speed up the construction of new elements.

Advanced Functions and Implementation

erase

Next, let’s implement some additional functions. Let’s start with erase. Erase deletes all content from the specified position to the specified position. The function prototype is as follows:

 public:
iterator erase(const_iterator position);
iterator erase(const_iterator first, const_iterator last);

The first function can also be seen as simply calling the second function. We implement the second function as follows:

template <typename T>
typename vector<T>::iterator vector<T>::erase(const_iterator first, const_iterator last)
{
if(last >= endptr || first < startptr) throw std::runtime_error("out of range!");
iterator newendptr = std::copy(last, static_cast<const_iterator>(endptr), first);
while(newendptr < endptr){
alloc.destroy(--endptr);
}
return endptr;
}

First, perform a validity check. Then, use std::copy to copy the later content to the part to be deleted. Note that the copy will automatically call the assignment function for the part being overwritten, and the assignment function should call the destructor internally. Then, if there is still content that needs to be destructed (this occurs only when the moved content is not as long as the deleted content), destruct that content. Then return the end pointer.

The other overloaded function is relatively simple to implement:

template<typename T>
typename vector<T>::iterator vector<T>::erase(const_iterator position)
{
return erase(position, position+1);
}

C++11 Features - Simple Introduction

Thu, 28 Dec 2017 07:31:00 +0000

Summarized from C++ Primer 5th Edition, as a quick review of these features

long long type Defined to be at least 32 bits in length. Generally not very useful.
List initialization A list is a set of elements composed of braces and commas, for example {1,2,3}. We can use braces to initialize variables. int a{0}; The equal sign in a declaration is not assignment but initialization, so the following form can also be used for initialization: int a = {0}; This has the same effect as int a = 0;
nullptr constant This is because before C++11, assigning a null pointer was done directly with 0. Now using nullptr is better. nullptr is essentially an object that can only have a right value and can only be type-converted to a pointer with a value of address 0, which solves some problems. Related information can be found in how to implement nullptr.
constexpr variables Stands for “const expressions”. The difference from const is that const indicates that the variable name must be a constant when used later, while constexpr not only requires this but also requires that the expression used to initialize the variable must also be a constant.
Type aliases using uint = unsigned int typedef uint unsigned int These two statements are equivalent
auto type specifier When declaring and defining a variable, the compiler automatically deduces the type of the declared variable based on the definition type.
decltype type indicator When declaring a variable, you can use decltype(expressions) to determine the type of an expression as the type of the declared variable
In-class initialization

 class A {
public:
int a = 7;
};

This is equivalent to the following in previous versions:

 class A {
public:
int a;
A() : a(7) {}
};

Using auto or decltype to deduce types auto len = line.size(); automatically deduces string::size_type which is better
Range-based for statements Print characters in a string for(auto c:str) cout « c « endl; Similar to for in statements in Python
Nested vectors vector<vector on older compilers, a space is needed, so the two angle brackets aren’t adjacent >
List initialization of vector objects vector strs = {“a”, “bb”, “ccc”};
cbegin and cend Return iterators of type const_iterator
Using auto in two-dimensional arrays

int mat[3][4];
int main()
{
for (int(*p)[4] = mat; p != mat + 3; p++)
{
/* ... */
}
for (auto p = mat; p != mat + 3; p++)
{
/* ... */
}
}

Division rounding C++11 specifies that division always rounds toward zero
List assignment to vector objects vector strs; strs = {“a”, “bb”, “ccc”};
139 Using sizeof to get the size of class members Normally class members can only be accessed through objects. But sizeof can directly determine the size of members.
168 Range-based for statements In range-based for statements, you cannot add elements to vector objects because it may cause the end element of the range-based for statement to become invalid. This is similar to issues in Python range loops.
197 initializer_list initializer_list is a template type, the object elements are always constants, and the types must be the same. Lists in braces are objects of this type.
203 List initialization of return values Returning an initializer_list can initialize temporary quantities for objects with a return type of vector.
206 Defining trailing return types Declaring a function that returns an array pointer int * func(int i) this is not quite right, as it returns a pointer, not an array pointer int (* func(int i))[10] this would return a pointer to an array of size 10 You can also write: auto func(int i) -> int(*)[10];
206 Using decltype to simplify return types decltype can be used to simplify return types, but this need is less common.
214 constexpr functions A constexpr function is a function that returns a literal type and can only have one return statement.
237 default default constructor You can assign an equal sign to a constructor to indicate using the compiler-provided default constructor.
246 In-class initialization The new standard allows for initializing members with values directly within the class
261 Delegating constructors When initializing a constructor, use an initializer list to initialize members. In addition, you can use other constructors to help construct. For example, a constructor is func(int a,int b), and a delegating constructor func():func(0,0){} so it can be constructed with default values 0,0.
268 constexpr constructors Used to generate constexpr objects
284 Using string type to represent file paths In stream types, previously only C character strings could be used, now strings of string type can be used
293 array and forward_list Very fast, comparable to handwritten lists, safer than arrays
300 Using list initialization for containers list authors = {“milton”, “shakespeare”, “austen”}; vector<const char*> articles = {“a”, “b”, “cc”};
303 Non-member swap for containers Using swap can quickly exchange two content containers
308 Using the return value of insert The new standard’s insert returns an iterator after successful insertion, and the position of the iterator points to the newly inserted element. Using this feature, multiple elements can be repeatedly inserted at that position.
308 Using emplace functions The emplace function can achieve construction rather than copying. This method can pass the parameters needed by the constructor without constructing the object and then making a copy.
shrink_to_fit Calling this function on a container will release extra space occupied by a vector or string. Generally not very useful
String type numeric conversion functions to_string represents a group of overloaded functions that can convert data of int, double types to string type. stoi can convert string to int, similar to other functions, including different bases and floating-point numbers
346 lambda expressions Simple lambda expression auto f = []{return 42;}; Here, a lambda expression is assigned to f. Later, f() can be called to get 42 How to pass parameters to a lambda expression? [](int a, int b){return a>b;}; But sometimes you don’t want to pass a parameter, yet still have the lambda expression use external variables? [&out](int a, int b){return a<out && out < b;}; But what if I don’t want the above expression to return int, but double? [&out](int a, int b)->double{return a<out && out < b;}
354 bind The bind function can change a function call to a simple call
377 List initialization of associative containers map<string, string> authors={ {“joncy”, “james”} }; set exclude = {“the”, “but” };
380 List initialization of pair return types Returning a list represented by {} to a pair type automatically converts.
384 List initialization of pairs When inserting pairs into a map, the simplest way is to insert a pair of key-values represented by braces. The inserted content will be initialized as a pair.
394 Unordered containers Add unordered in front of map and set to get unordered containers. Implemented using hash underneath. Requires implementation of a hash template for stored elements.
400 Smart pointers shared_ptr allows multiple pointers to point to the same object. unique_ptr exclusively owns the object it points to. Smart pointers are templates, and when declaring them, you need to explicitly provide the type, similar to using vector. shared_ptr p1;

Epoll TCP Server and Client Simple Example

Mon, 25 Dec 2017 02:52:00 +0000

Introduction

epoll is a Linux-specific API that provides efficient file descriptor management. It’s commonly used to monitor the readability of multiple file descriptors in batch and is one of the effective methods for high-concurrency servers. Below is a simple example of an epoll server and client.

Server Side

#include <fcntl.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/epoll.h>
using namespace std;
const int MAX_EPOLL_EVENTS = 1000;
const int MAX_MSG_LEN = 1024;
void setFdNonblock(int fd)
{
fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | O_NONBLOCK);
}
void err_exit(const char *s){
printf("error: %s\n",s);
exit(0);
}
int create_socket(const char *ip, const int port_number)
{
struct sockaddr_in server_addr = {0};
/* Set IPv4 mode */
server_addr.sin_family = AF_INET; /* ipv4 */
/* Set port number */
server_addr.sin_port = htons(port_number);
/* Set host address */
if(inet_pton(server_addr.sin_family, ip, &server_addr.sin_addr) == -1){
err_exit("inet_pton");
}
/* Create socket */
int sockfd = socket(PF_INET, SOCK_STREAM, 0);
if(sockfd == -1){
err_exit("socket");
}
/* Set reuse mode */
int reuse = 1;
if(setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &reuse, sizeof(reuse)) == -1)
{
err_exit("setsockopt");
}
/* Bind to port */
if(bind(sockfd, (sockaddr *)&server_addr, sizeof(server_addr)) == -1){
err_exit("bind");
}
/* Set passive open */
if(listen(sockfd, 5) == -1){
err_exit("listen");
}
return sockfd;
}
int main(int argc, const char *argv[])
{
/* Help message */
if(argc < 3){
printf("usage:%s ip port\n", argv[0]);
exit(0);
}
/* Get server parameters */
const char * ip = argv[1];
const int port = atoi(argv[2]);
/* Create socket */
int sockfd = create_socket(ip, port);
printf("success create sockfd %d\n", sockfd);
setFdNonblock(sockfd);
/* Create epoll */
int epollfd = epoll_create1(0);
if(epollfd == -1) err_exit("epoll_create1");
/* Add sockfd to epollfd interest list */
struct epoll_event ev;
ev.data.fd = sockfd;
ev.events = EPOLLIN ;
if(epoll_ctl(epollfd, EPOLL_CTL_ADD, sockfd, &ev) == -1){
err_exit("epoll_ctl1");
}
/* Create a list to store events returned by wait */
struct epoll_event events[MAX_EPOLL_EVENTS] = {0};
/* Start waiting for all events registered on epoll */
while(1){
/* Wait for events */
printf("begin wait\n");
int number = epoll_wait(epollfd, events, MAX_EPOLL_EVENTS, -1);
printf("end wait\n");
sleep(1);
if(number > 0){
/* Traverse all events */
for (int i = 0; i < number; i++)
{
int eventfd = events[i].data.fd;
/* If the fd triggering the event is sockfd, someone has connected, and we need to accept them */
if(eventfd == sockfd){
printf("accept new client...\n");
struct sockaddr_in client_addr;
socklen_t client_addr_len = sizeof(client_addr);
int connfd = accept(sockfd, (struct sockaddr *)&client_addr, &client_addr_len);
setFdNonblock(connfd);
/* After accept, we need to add the file descriptor to the monitoring list */
struct epoll_event ev;
ev.data.fd = connfd;
ev.events = EPOLLIN;
if(epoll_ctl(epollfd, EPOLL_CTL_ADD, connfd, &ev) == -1){
err_exit("epoll_ctl2");
}
printf("accept new client end.\n");
}
/* If the triggering fd is not sockfd, it's the newly added connfd */
else{
/* Read content until encountering a newline, then display the content */
printf("read start...\n");
while(1){
char buff = -1;
int ret = read(eventfd, &buff, 1);
if(ret > 0){
printf("%c", buff);
}
if(buff == '\n'){
break;
}
else if (ret == 0){
printf("client close.\n");
close(eventfd);
epoll_ctl(epollfd, EPOLL_CTL_DEL, eventfd, NULL);
break;
}
else if (ret < 0){
printf("read error.\n");
break;
}
}
printf("read end.\n");
}
}
}
}
}

Client Side

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
#include <arpa/inet.h>
#include <sys/socket.h>
#include <string>
#include <iostream>
using namespace std;
void err_exit(const char *s){
printf("error: %s\n",s);
exit(0);
}
int create_socket(const char *ip, const int port_number)
{
struct sockaddr_in server_addr = {0};
/* Set IPv4 mode */
server_addr.sin_family = AF_INET; /* ipv4 */
/* Set port number */
server_addr.sin_port = htons(port_number);
/* Set host address */
if(inet_pton(PF_INET, ip, &server_addr.sin_addr) == -1){
err_exit("inet_pton");
}
/* Create socket */
int sockfd = socket(PF_INET, SOCK_STREAM, 0);
if(sockfd == -1){
err_exit("socket");
}
if(connect(sockfd, (struct sockaddr*)&server_addr, sizeof(server_addr)) == -1){
err_exit("connect");
}
return sockfd;
}
int main(int argc, const char *argv[]){
if(argc < 3){
printf("usage:%s ip port\n", argv[0]);
exit(0);
}
/* Get server parameters */
const char * ip = argv[1];
const int port = atoi(argv[2]);
//Create socket
int sock = create_socket(ip, port);
//Initiate request to server (specific IP and port)
while(1){
string buff;
getline(cin, buff);
if(buff == "exit") break;
write(sock, buff.c_str(), buff.size());
char end = '\n';
write(sock, &end, 1);
}
close(sock);
return 0;
}

Compilation

Save the above text as socket_server.cpp and socket_client.cpp, then compile and link the programs.

g++ -Wall socket_server.cpp -o server && g++ -Wall socket_client.cpp -o client

Execution

./server localhost 1234

./client localhost 1234

Entering text on the client side and pressing Enter will display it on the server side. Press Ctrl+C or type “exit” to close the client.

Execution Flow

The server first creates a passive open socket file descriptor, then adds this file descriptor to the epoll interest list. It then enters a loop. Whenever the interest list’s wait ends, it means the corresponding file descriptor can be operated on. When a client connects to the passive open socket file descriptor, it indicates a client has connected, and the passive open file descriptor can be accepted. The new file descriptor created after accept is the file descriptor for communicating with the client, which is also added to the interest list. When the client sends data, this file descriptor will also generate a readable signal, causing the wait to end. At this point, it enters processing mode, reading and displaying the data sent by the client.

SOCKET Learning Notes

Mon, 25 Dec 2017 02:34:00 +0000

Introduction

Interview questions mentioning epoll involve socket programming. To understand epoll principles in depth, we first need to understand socket programming. A socket is an inter-process communication (IPC) mechanism, even in networks. This means the main entities communicating in a network are processes, not computers. Socket learning includes how to establish servers and clients, and how to use socket APIs.

fd=socket(domain, type, protocol);

The socket call can be used to create a socket, for example, domain can specify IPv4, type can specify TCP, and protocol is generally 0.

domain domain specifies the communication scope and the type of communication address. There are several classic types: UNIX IPV4 IPV6, corresponding to parameters AF_UNIX AF_INET AF_INET6. The domain parameters all start with AF, representing address families. PF stands for protocol families. Originally, address families and protocol families were designed to have many-to-many relationships, but during implementation, each protocol family corresponds to exactly one address family. So basically, specifying the domain determines the protocol family and the address family.
type Socket indicates whether it’s a stream or a datagram, which essentially means TCP or UDP. For TCP, it’s SOCK_STREAM; for UDP, it’s SOCK_DGRAM.

bind(int sockfd, const struct sockaddr *addr, socklen_t addrlen);

This call is used to bind a socket to an address. After that, TCP packets can be sent, and in some cases, UDP packets can also be sent through write, but only data from the peer socket can be read on this socket.

sockaddr This structure has an integer representing the address type, followed by a char array. As we’ll see, depending on the use case, other data structures are passed in, but they are generic.

listen(int sockfd, int backlog)

Marks a socket descriptor as passive. It can be connected by active sockets. backlog is used to limit the number of pending connections.

accept(int sockfd, struct sockaddr addr, socklen_t addrlen);

The accept call blocks and waits for an incoming request on the file descriptor sockfd. Once the request succeeds, a new socket is created, and this new socket connects with the other party.

addr Returns the address of the other party
addrlen Passes in the length of addr, indicating the maximum length that can be written.

connect(int sockfd, const struct sockaddr * addr, socklen_t addrlen);

Connects sockfd to the address described by addr.

close(int fd)

Used to close the connection

read write

Used for reading from or writing to sockfd

recvfrom(int sockfd, void buffer, size_t length, int flags, struct sockaddr src_addr, socklen_t *addrlen);

sendto(int sockfd, const void buffer, size_t length, int flags, const struct sockaddr dest_addr, socklen_t addrlen);

Used to send and receive UDP packets. The server side cannot use the listen function and accept function, and the client side cannot use the connect function.

unix domain

Using the APIs above, communication through files on the local machine can be implemented. The sockaddr used by unix domain is sockaddr_un, represented as follows:

struct sockaddr_un{
sa_family_t sun_family;
char sun_path[108];
}

Network Byte Order

The network byte order follows big-endian, while x86 is a little-endian structure. The conversion is done using the following functions:

htons
htonl
ntohs
ntohl h is for host, n is for net, s is for 16-bit, and l is for 32-bit. s and l stand for short and long, although these standards are no longer used that way now.

Internet Socket Address Structure

The socket address used in networks is sockaddr_in, defined as follows:

struct sockaddr_in{
sa_family_t sin_family;
in_port_t sin_port;
struct in_addr sin_addr;
unsigned char __pad[X];
}

As you can see, the difference is that the char array is replaced by a port and an address. sin is the abbreviation of socket Internet, which is as poor an abbreviation as sun.

Internet Socket Address Conversion

APIs for converting string address formats to binary address formats:

inet_pton(int domain, const char src, void addrptr);

This function is used to convert the string contained in src to a network byte order binary address, storing it in addrptr. p stands for presentation, meaning a human-readable address.

const char * inet_ntop(int domain, const void addrptr, char dst_str, size_t len);

This function converts a network byte order binary address to a human-readable address, writing it to dst_str. The buffer size is passed in by len.

getaddrinfo(const char host, const char service, const struct addrinfo *hints, struct **result);

This function returns a socket address and port number given a host name and service name. getaddrinfo takes host, service, and hints as inputs, where the host parameter includes a hostname or an IPv4 string. Service is a service name or a port number. After calling this function, freeaddrinfo should be used to free the space.

getnameinfo(const struct sockaddr addr, socklen_t addrlen, char host, size_t hostlen, char *service, size_t servlen, int flags);

Given a socket address structure, returns a host and service name string.

setsockopt(int sockfd, int level, int optname, const void *optval, socklen_t optlen);

sockfd is the file descriptor pointing to the socket. The level parameter specifies the protocol to which the socket option applies, such as TCP or IP, indicating the socket API layer where the option takes effect. Generally, this option is set to SOL_SOCKET, indicating that it applies to the socket API layer. The optname parameter indicates the option we expect to set, optvalue is used to set the value of that option, which can be an integer or a pointer to a structure pointing to a buffer, and the optlen parameter is the size of the region pointed to by that pointer.

For example, to set sockfd to the reuseaddr property, you can call:

 int reuse = 1;
if(setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &reuse, sizeof(reuse)) == -1)
{
err_exit("setsockopt");
}

getsockopt(int sockfd, int level, int optname, void *optval, socklen_t optlen);

The usage is the same as above, but it’s for getting rather than setting.

Other I/O Models Summary

Thu, 21 Dec 2017 06:18:00 +0000

I/O Multiplexing
Signal-Driven I/O
Linux-specific epoll

I/O Multiplexing

File descriptors are examined through the select system call or the poll system call.

select function int select(int nfds, fd_set *readfds, fd_set *writefds, fd_set *errorfds, struct timeval *timeout); The parameters respectively mean: the range of file descriptors to test (0 to fd-1), file descriptors that meet three different condition requirements, and timeout period.
poll function int poll(struct pollfd fds[], nfds_t nfds, int timeout); The parameters are: an array containing elements with file descriptors, states to be checked, returned states; the array length; and the timeout period.

The difference between the two is that the array size passed into poll is not limited by the size defined by fd_set. Poll’s events and revents are a bit more versatile. Select is used more widely. In newer versions, select’s performance has been optimized to be comparable to poll.

epoll function The epoll function includes three APIs: epoll_create(), epoll_ctl(), and epoll_wait(). Among the three calls, epoll has the best performance and most effectiveness. However, it can only be used on Linux.

This is mainly because a server will have multiple socket connections, and if a thread were created for each socket, it would consume considerable resources. Therefore, epoll is needed to optimize performance.

C++ Object-Oriented Programming Summary

Thu, 14 Dec 2017 05:53:00 +0000

Introduction

The biggest difference between C++ and C is object-oriented programming. Although I understand some concepts about object-oriented programming and have written some functions, I rarely use them in practice. This article is a quick summary of Chapter 15 of C++ Primer.

Overview

Object-oriented programming, also known as OOP, has three core ideas: data abstraction, inheritance, and dynamic binding. Data abstraction refers to the separation of declaration and implementation, inheritance refers to the subclass obtaining all members of the parent class, and dynamic binding means calling different functions based on the class type when invoking class functions.

Declaring Base Classes and Derived Classes

The following code provides a simple example, declaring a base class animal and a derived class cat.

#include <stdio.h>
#include <iostream>
class animal
{
public:
virtual void eat(){};
};
class cat : public animal
{
public:
void eat() { std::cout << "eat fish" << std::endl; }
};
int main()
{
cat c;
c.eat();
}

Type Conversion and Inheritance

Pointers and references of derived classes can be converted to pointers or references of base classes, but pointers or references of base classes cannot be converted to those of derived classes. This means that an object pointed to by a base class pointer could be a base class or a derived class, but definitely not a parent class of the base class.

A derived class object can be assigned to a base class, but only the base class portion will be copied.

Virtual Functions

Virtual functions are resolved at runtime. Continuing with our example above, suppose we write a new function called doeat that internally calls the eat method of animal. For animal, eat is “eat meat or grass”, but the final displayed result is “eat fish”. This proves that at runtime, when the doeat function calls animal’s eat method, the eat method finds the real object cat’s eat method and calls it.

class animal
{
public:
virtual void eat() { std::cout << "eat meat or grass" << std::endl; };
};
class cat : public animal
{
public:
void eat() { std::cout << "eat fish" << std::endl; }
};
void doeat(animal &a) { a.eat(); }
int main()
{
cat c;
doeat(c);
}

Additionally, to develop good habits, it’s best to use the override keyword in derived classes to explicitly declare overridden virtual functions.

Pure Virtual Functions

Assigning a value of 0 after defining a virtual function declares a pure virtual function. Pure virtual functions are equivalent to interfaces, and derived classes must implement these interfaces.

Abstract Base Classes

Classes containing pure virtual functions are abstract base classes. Abstract base classes cannot be instantiated.

Example: Text Line Query Program

Given a text, we need to query its contents. Supporting word queries and logical queries.

C++ Virtual Functions Summary

Thu, 14 Dec 2017 05:53:00 +0000

Virtual functions are C++’s way of implementing polymorphism.

What is polymorphism? Polymorphism is often viewed as the third pillar of object-oriented programming after encapsulation and inheritance. To give a simple example, if we have a class Animal, and classes like Chicken and Dog that inherit from Animal, and they respond differently to the same message derived from the parent class Animal. For instance, the Animal class has an action “makeSound()”, while the Chicken class would “crow()” and the Dog class would “bark()” - this is called polymorphism. According to Wikipedia, polymorphism can be defined as “the ability to associate different specific behaviors with a single generalized notation.” It refers to when a computer program runs, the same message may be sent to objects of different classes, and the system can trigger corresponding class methods based on the object’s class, resulting in different behaviors. Simply put, polymorphism means that the same message given to different objects triggers different actions.

Virtual functions enable dynamic polymorphism, which is determined at runtime. Only during program execution does the system decide whether to call the base class or child class function. The system determines which function to call based on the object that the base class pointer points to.

How to declare a virtual function? Add the virtual keyword before the declaration to declare a virtual function. The function is declared in the parent class and implemented in the child class.
What is a pure virtual function? A pure virtual function is a function that is assigned a value of 0 when declaring a virtual function. After using this declaration method, the inheriting class must implement this virtual function.
How are virtual functions implemented? Through a virtual table (vtable). Each object has a virtual table pointer that points to the virtual table. The virtual table is essentially an array, not a linked list. The addresses of virtual functions are stored in the virtual table in the order of declaration.
How is the virtual function table implemented? To accomplish the functionality of virtual functions, the compiler (note, this is why C++ is sometimes called compiler-oriented programming) creates a table for each class that declares internal virtual functions, called a vtable. In the vtable, the addresses of virtual functions of specific types are placed in the order of declaration. Each class with virtual functions contains a pointer, called a vpointer or vptr, pointing to this vtable. Whenever a call to a class’s virtual function occurs, the compiler redirects to call the function in the vtable pointed to by this vptr, rather than statically calling a specific function.

Linux: Using popen to Execute Bash Commands and Get Return Strings

Wed, 08 Nov 2017 07:22:00 +0000

The system command returns an integer value, making it difficult to determine specific outcomes. You can use popen to execute bash commands and retrieve return values. Note that popen captures the standard output as the return string. If you need to get error output content, you must redirect stderr to stdout.

void system_cmd(const char * command, char * result)
{
FILE *fpRead;
fpRead = popen(command, "r");
char buf[1024] = {0};
memset(buf,'\0',sizeof(buf));
while(fgets(buf,1024-1,fpRead)!=NULL)
{
if (buf[strlen(buf) - 1] == '\n') {
buf[strlen(buf) - 1] = '\0'; //remove newline character
}
strcpy(result, buf);
}
if(fpRead!=NULL)
pclose(fpRead);
}

Usage example:

 char pid[10]={0};
system_cmd("pgrep bash", pid);

Key Points for Installing GPU Version of TensorFlow on Deepin

Wed, 13 Sep 2017 10:44:00 +0000

Introduction

Although there are many tutorials on how to install TensorFlow on Ubuntu, there isn’t a single article explaining how to install TensorFlow on Deepin systems. Here I’ll explain the key points of the installation process.

Process

First, download the Nvidia Linux driver from the official website. If you can’t run it, you’ll need to modify the driver’s execution permissions using chmod u+x. For information on installing graphics drivers on Deepin, refer to this article: https://wiki.deepin.org/index.php?title=%E6%98%BE%E5%8D%A1
Install Docker-ce. You must follow the official installation guide; this is the only method to properly install the latest version of Docker: https://wiki.deepin.org/index.php?title=Docker
Download Nvidia-docker. Find the Ubuntu installation package and installation steps on the Nvidia-Docker page on GitHub and follow them.
Next, run docker run -it -p 8888:8888 tensorflow/tensorflow:latest-gpu. At this point, you can enter the official TensorFlow GPU version container. Open localhost:8888 to see the Jupyter page.

Pitfalls

When installing the Nvidia graphics driver, according to the installation software’s instructions, you need to close the Linux desktop system X server. At this point, you can use tty16 for terminal operations. However, if Deepin has already installed its own Nvidia driver, when stopping the lightdm service, Deepin closes both the X server and the monitor, making it impossible to display the tty16 screens. This issue prevents Nvidia driver installation. Even after manually uninstalling the official Deepin Nvidia driver through complex uninstallation commands, when switching to tty1~6 and closing the graphical interface, the tty screen still shuts down, resulting in a black screen and making it impossible to continue with driver uninstallation and reinstallation of the official Nvidia driver.

Solution

To avoid problems with Nvidia driver installation while maintaining a clean and scientific operation, you must install the official Nvidia Linux driver during the first installation of the Deepin system, before any graphics drivers are installed. First, press the shortcut “Ctrl+Alt+F2” to enter tty2, then enter sudo systemctl stop lightdm to stop the lightdm service. At this point, when the computer closes the X server, it won’t cause the monitor to shut down. Then run chmod u+x NVIDIA-Linux-x86_64-352.55.run to grant executable permissions, followed by sudo ./NVIDIA-Linux-x86_64-352.55.run to install the driver file. After restarting, you can use the official closed-source driver normally.

Follow-up

Using a GPU to run TensorFlow programs is very fast. Normal programs can speed up by more than 10 times, and some programs can speed up by 50 to 100 times, so using a GPU for TensorFlow programming is essential. Additionally, I’m not sure if it’s an issue with Deepin, Nvidia-Docker, or the driver, but after the computer goes into standby mode, it causes errors in the TensorFlow container. Therefore, don’t let the machine enter standby mode during training.

TensorFlow: A Simple Introduction Case Study

Wed, 09 Aug 2017 07:08:00 +0000

Case Study

Suppose I have a series of x-y data points with a linear relationship between x and y. How would we fit a straight line to this data?

In the program below, train_x is a series of numbers between -1 and 1, and train_y is twice x plus 10, with a random number between 0 and 1 added. Next, we build the model. X and Y are tensor placeholders waiting to be initialized. During optimization, the XY values in the model will continuously change to data from train_x and train_y, and then the optimizer will optimize by changing the slope w and intercept b in a direction that reduces error. Through iteration, w and b will eventually make the model fit the data.

After the model is built, we start running it. First, we open a session, and we must remember to initialize all variables. Next, we iterate through all the data 10 times. In each iteration, we input a coordinate, calculate the error, and use gradient descent to correct w and b. Finally, we output the calculated values of w and b.

import tensorflow as tf
import numpy as np
train_x = np.linspace(-1, 1, 101)
train_y = 2 * train_x + np.random.rand(train_x.shape[0]) + 10
X = tf.placeholder("float")
Y = tf.placeholder("float")
w = tf.Variable(0.0, name = "w")
b = tf.Variable(0.0, name = "b")
loss = tf.square(Y - tf.multiply(X,w) - b)
train_op = tf.train.GradientDescentOptimizer(0.01).minimize(loss)
with tf.Session() as session:
session.run(tf.global_variables_initializer())
for i in range(10):
for x,y in zip(train_x, train_y):
session.run(train_op, feed_dict={X:x, Y:y})
print("w: ", session.run(w))
print("b: ", session.run(b))

Running results:

TensorFlow Hello World Introduction

Tue, 08 Aug 2017 09:58:00 +0000

Quick Installation Guide

Step 1: Install Docker

https://www.docker-cn.com/community-edition#/download

Then configure the official Chinese mirror.

Step 2: Set up TensorFlow environment

$ docker run -it -p 8888:8888 tensorflow/tensorflow Running this command will automatically download the TensorFlow image, provided that the repository mirror is set to a Chinese mirror; otherwise, the download will be very slow. After running the command, the terminal will display a URL that prompts you to open a web page. When you open this URL, you’ll see the TensorFlow Jupyter editing environment, where we’ll input all our code.

Mounting Docker file directory If we need to access local files, we need to mount a local folder to the container directory. Close the container, reopen it, and use -v host_directory:container_directory for mounting. docker run -v /Users/hahaha/tensorflow/:/notebooks -it -p 8888:8888 tensorflow/tensorflow

Where /Users/hahaha/tensorflow/ is a folder on my Mac, and notebooks is the default Jupyter editing directory in the TensorFlow container.

Running Hello World Code

Create a new Python 2 Jupyter file, enter the following code, and then click the play button. At this point, a “Hello, TensorFlow!” string should appear below, indicating that the program has run successfully.

Program Explanation

From this simple code, we can see that TensorFlow is very easy to use. It’s imported as a standard Python library without requiring additional services to be started. For those new to TensorFlow, you might wonder why we need to use tf.constant() and tf.Session() to output a “Hello World” string when Python itself could do it. The reason is that TensorFlow defines and runs models and training through Graphs and Sessions, which provides significant benefits for complex models and distributed training.

First, in TensorFlow, there are two concepts: Graph and Operation. Operation represents what needs to be computed. A Graph contains many Operations. A Session is used to execute Operations in a Graph.

Basic Usage

When using TensorFlow, you must understand that TensorFlow:

Uses a graph to represent computational tasks
Executes the graph in a context called a Session
Represents data using tensors
Maintains state through Variables
Uses feed and fetch to assign values to or retrieve data from arbitrary operations

Overview

TensorFlow is a programming system that uses graphs to represent computational tasks. Nodes in the graph are called ops (short for operations). An op takes 0 or more Tensors, performs computations, and produces 0 or more Tensors. Each Tensor is a typed multi-dimensional array. For example, you can represent a small batch of images as a four-dimensional floating-point array with dimensions [batch, height, width, channels].

A TensorFlow graph describes the computation process. To perform computation, the graph must be launched in a session. The session distributes the graph’s ops to devices like CPUs or GPUs and provides methods to execute ops. After execution, these methods return the resulting tensors. In Python, the returned tensors are numpy ndarray objects. In C and C++, the returned tensors are tensorflow::Tensor instances.

Using TensorFlow for a Kaggle Task — Titanic: Machine Learning from Disaster

Fri, 04 Aug 2017 10:08:00 +0000

Import Necessary Libraries

import csv
import tensorflow as tf
import numpy as np
import random
import sys
import pandas as pd
from pandas import DataFrame
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

Read Source File and Display

In this section, we’ll work with basic CSV operations and display the results. We’ll read the train.csv file downloaded from Kaggle and show its contents.

trainFilePath = './train.csv'
trainSize = 0
def testCSV(filePath):
with open(filePath, 'rb') as trainFile:
global trainSize
csvReader = csv.reader(trainFile)
dataList = [data for data in csvReader]
df = DataFrame(dataList[1:], columns=dataList[0])
trainSize = len(df)
print(df)
print("trainSize", trainSize)
testCSV(trainFilePath)

Read Source File, Extract Data, and Build Neural Network

In this section, we’ll extract gender, class, ticket fare, and SibSp from the source file to fit the survival probability. Then we’ll build a 5-layer neural network with 3 hidden layers containing 4-10-20-10-2 neurons respectively. Finally, we’ll execute the reading function.

def readTrainDataCSV(filePath):
global trainData, targetData, classifier
with open(filePath, 'rb') as trainFile:
csvReader = csv.reader(trainFile)
dataList = [data for data in csvReader]
dataSize = len(dataList) - 1
trainData = np.ndarray((dataSize, 4), dtype=np.float32)
targetData = np.ndarray((dataSize, 1), dtype=np.int32)
trainDataFrame = DataFrame(dataList[1:], columns=dataList[0])
trainDataFrame_fliter = trainDataFrame.loc[:,['Pclass','Sex','SibSp','Fare','Survived']]
for i in range(dataSize):
thisData = np.array(trainDataFrame_fliter.iloc[i])
Pclass,Sex,SibSp,Fare,Survived = thisData
Pclass = float(Pclass)
Sex = 0 if Sex == 'female' else 1
SibSp = float(SibSp)
Fare = float(Fare)
Survived = int(Survived)
print(Pclass,Sex,SibSp,Fare,Survived)
trainData[i,:] = [Pclass,Sex,SibSp,Fare]
targetData[i,:] = [Survived]
print(thisData)
print(trainData)
print(targetData)
feature_columns = [tf.contrib.layers.real_valued_column("", dimension=4)]
classifier = tf.contrib.learn.DNNClassifier(feature_columns=feature_columns,
hidden_units=[10, 20, 10],
n_classes=2)
# model_dir="/tmp/titanic_model")
readTrainDataCSV(trainFilePath)

Create Input Data

We’ll wrap the training data and labels into a tuple and return it.

def get_train_inputs():
x = tf.constant(trainData)
y = tf.constant(targetData)
print(x)
print(y)
return x, y
get_train_inputs()

Train the Model

Now we start training the neural network.

def train():
classifier.fit(input_fn=get_train_inputs, steps=2000)
train()

Check Accuracy

We use the entire dataset to check accuracy. Note that we should use a validation set for this task, but since this is just for demonstration purposes, we’ll skip that step.

accuracy_score = classifier.evaluate(input_fn=get_train_inputs,
steps=1)["accuracy"]
print("accuracy:",accuracy_score)

Read Test Set and Output Results

In this section, we’ll read the test data from Kaggle and output the results to a file, which will ultimately be submitted to the official website.

testFilePath = './test.csv'
def readTestDataCSV(filePath):
global testData, PassengerIdStart
with open(filePath, 'rb') as testFile:
csvReader = csv.reader(testFile)
dataList = [data for data in csvReader]
dataSize = len(dataList)-1
trainDataFrame = DataFrame(dataList[1:], columns=dataList[0])
trainDataFrame_fliter = trainDataFrame.loc[:,['Pclass','Sex','SibSp','Fare']]
testData = np.ndarray((dataSize, 4), dtype=np.float32)
PassengerIdStart = trainDataFrame['PassengerId'][0]
PassengerIdStart = int(PassengerIdStart)
print('PassengerId',PassengerIdStart)
for i in range(dataSize):
thisData = np.array(trainDataFrame_fliter.iloc[i])
Pclass,Sex,SibSp,Fare = thisData
Pclass = float(Pclass)
Sex = 0 if Sex == 'female' else 1
SibSp = float(SibSp)
Fare = 0 if Fare=='' else float(Fare)
print(Pclass,Sex,SibSp,Fare)
testData[i,:] = [Pclass,Sex,SibSp,Fare]
print(thisData)
print(testData)
def testData_samples():
return testData
readTestDataCSV(testFilePath)
predictions = list(classifier.predict(input_fn=testData_samples))
print(predictions)
with open('predictions.csv', 'wb') as csvfile:
writer = csv.writer(csvfile, dialect='excel')
writer.writerow(['PassengerId','Survived'])
PassengerId = PassengerIdStart
for i in predictions:
writer.writerow([PassengerId, i])
PassengerId += 1

Finally, using only 4 features, we achieved an accuracy of 75%. The next goal is to utilize the other available data.

Building a Personal Blog with Flask

Sun, 30 Jul 2017 17:38:00 +0000

Introduction

The source code can be run directly, supports markdown syntax, supports email registration, and any issues can be referenced in 《Flask Web Development: Web Application Development in Action with Python》.

Blog website: http://www.unrealblue.cc Project repository: https://github.com/linanwx/unrealblue-blog

Preview

Deployment Process

First, to verify the program functionality, you need to deploy it on your local machine. After that, you can use nginx as a reverse proxy tool to expose the port, so that others can access your blog on the public network. Then follow the same steps on the server. The specific process is as follows:

Install the virtualenv Python virtual environment with pip install virtualenv or pip3 install virtualenv. Then use virtualenv to create a venv environment in a suitable directory, for example under this project with virtualenv venv.
Activate the virtualenv environment by running the activate script in the venv directory: . venv/bin/activate. Note the position of the dot. After this, you will see the (venv) marker at the beginning of the command line.
Install all modules listed in requirements.txt within the virtual environment: pip3 install -r requirements.txt. If installation is too slow, you may need to configure a domestic pip source. See the pip official page for how to change the pip source.
Import environment variables by creating an env file in the project directory with the following fields:

MAIL_USERNAME=email@example.com (Email used by the server to send verification codes, an email account with smtp service enabled. The program uses QQ email by default, modify the config file to use other types of email)
MAIL_PASSWORD=password (Password for the above email, note that QQ email uses a special 16-digit password)
FLASK_ADMIN=admin@example.com (After the server is running, an account created with this email will be the administrator)
MAIL_SERVER=smtp.qq.com (Email server address)
FLASKY_MAIL_SENDER=example@foxmail.com (Sender’s email)

Set up database migration by entering the following commands:

python manager.py db init (The init command creates a migration repository, which will add a migrations folder)
python manager.py db migrate -m "initial migration" (The migrate command is used to automatically create migration scripts)
python manager.py db upgrade (Update the database. The first time you use this command, it will create a new database called data-dev.sqlite)

Deploy the program: python manager.py deploy
Run the program locally: python manager.py runserver. Open http://127.0.0.1:5000 to view it, press Ctrl+C to exit the program.
If running on a server and you want to preserve data, you can copy the migrations folder and the data-dev.sqlite database to the server, then use ./venv/bin/gunicorn -w 4 -b 127.0.0.1:8080 manager:app. At this point, you should be able to see the webpage on port 8080, and this port is exposed to the external network. Enter the server address in your local browser, and you will be able to see the blog.

Deep Learning Beginner's Study Notes One (Udacity)

Wed, 19 Jul 2017 07:42:00 +0000

Introduction

On a whim, I decided to learn machine learning. These are notes from my learning process.

Preparation

I made these preparations:

A MacBook with Python environment set up, and numpy and matplotlib installed
Registered for Udacity’s free “Deep Learning” course (in collaboration with Google)
Studied Liao Xuefeng’s Python introductory tutorial
Spent two days roughly browsing through “Machine Learning in Action”

Learning these fundamentals should be sufficient for the upcoming Udacity course.

Course One: From Machine Learning to Deep Learning

Sections 1-8 mainly introduce the current state of deep learning and related knowledge.

Sections 9-12 introduce the softmax model.

After a rough review of “Machine Learning in Action,” I learned that machine learning consists of several classification and clustering algorithms. On the surface, machine learning appears to be a collection of classification and clustering algorithms. Among these algorithms, one called logistic regression classification was introduced.

In sections 9-12, the focus is on the classifier model—logistic regression, using the softmax function as the classification function.

What is the softmax function?

This image illustrates what a softmax function is. For each number z in the original sequence, we calculate exp(z), and the proportion of each new number’s magnitude becomes the softmax probability for that number.

Properties

If the inputs are scaled up proportionally, the classifier’s results become more polarized and confident. If the inputs are scaled down proportionally, the classifier’s results tend toward the average and lack confidence.

Algorithm

import numpy as np
def softmax(x):
 """Compute softmax values for each sets of scores in x."""
 expList = [np.exp(i) for i in x]
 expSum = sum(expList)
 x = [i/expSum for i in expList]
 return np.array(x)

Sections 13-14 mainly discuss One-Hot encoding. After the softmax function provides a sequence of probability values, how do we determine the classification? For example, a sequence where the highest probability is 1 and others are 0 is called One-Hot encoding. This type of encoding has already determined the classification.

Sections 15-16 cover cross-entropy. Softmax can calculate a probability sequence, and OneHot is a determined classification. So how do we calculate the distance from a probability sequence to a specific classification? We use cross-entropy to measure this distance.

Sections 17-20 explain how to use this classifier. Section 18 specifically discusses why special initial data is needed.

sum = 1000000000
for i in range(1000000):
sum += 0.000001
sum -= 1000000000
print(sum)

The result of running this code is not 1. If we change sum to a very small number, like 1, instead of 1000000000, we find that the error becomes smaller. Based on this reason, we want our initial data to always have a mean of 0 and consistent variance in all directions. For example, for a grayscale image with pixel values from 0-255, we need to subtract 128 and then divide by 128, so that each number is between -1 and 1. Such initial data is more suitable for training.

This way, we can proceed with training. Reviewing the video content: xi is the training data matrix, w is a random weight matrix. For performance reasons, random values are taken from a normal distribution with an axis of 0 and very small variance. Then we calculate the probability sequence and the distance to the target. Then we compute the average distance to all targets. Our goal is to make this distance smaller, so we optimize the weight matrix along the direction of gradient descent while optimizing the intercept b. We repeat this process continuously until we reach a local optimum.

Installing Docker

https://www.docker-cn.com/community-edition#/download

Configure the official Chinese mirror.

Installing Jupyter Notebook

$ pip3 install jupyter $ jupyter notebook You can now use the jupyter notebook command to open a Jupyter editor.

Setting up TensorFlow environment

$ docker run -it -p 8888:8888 tensorflow/tensorflow Running this command will automatically download the TensorFlow image, provided that the repository mirror is set to a Chinese mirror; otherwise, the download will be very slow. After running the command, you’ll be prompted to open a webpage. When you open this URL, you’ll see the TensorFlow Jupyter editing environment, assuming Jupyter Notebook is installed correctly.

Mounting Docker’s file directory We need to import the official assignments. Close the container, reopen it, and use -v host_directory:container_directory for mounting. docker run -v /Users/hahaha/tensorflow/:/notebooks -it -p 8888:8888 tensorflow/tensorflow

Where /Users/hahaha/tensorflow/ is a folder on my Mac, and notebooks is the default Jupyter editing directory in TensorFlow.

Paste the first assignment file, 1_notmnist.ipynb, into the mounted directory on the host. This file can be found here: 1_notmnist.ipynb

Assignment Code Segment One

First, run the import statements in the first code segment. There should be no errors. If you see red error output, it means these imports were not successful.

# These are all the modules we'll be using later. Make sure you can import them
# before proceeding further.
from __future__ import print_function
# print function
import matplotlib.pyplot as plt
# plotting tool
import numpy as np
# matrix calculations
import os
# file paths
import sys
# file output
import tarfile
# decompression
from IPython.display import display, Image
# display images
from scipy import ndimage
# image processing
from sklearn.linear_model import LogisticRegression
# logistic regression module for linear models
from six.moves.urllib.request import urlretrieve
# url handling
from six.moves import cPickle as pickle
# data processing
# Config the matplotlib backend as plotting inline in IPython
%matplotlib inline
# matplotlib is the most famous Python chart plotting extension library,
# it supports outputting various formats of graphical images, and can use various GUI interface libraries to display charts interactively.
# Using the %matplotlib command can embed matplotlib charts directly into the Notebook,
# or display charts using a specified interface library, it has a parameter specifying how matplotlib charts are displayed.
# inline indicates embedding charts in the Notebook.

Assignment Code Segment Two

Next is the second code segment, which will download letter sets for training and testing, approximately 300MB in size. After successful download, you can see these two files in the mounted directory.

url = 'https://commondatastorage.googleapis.com/books1000/'
last_percent_reported = None
data_root = '.' # Change me to store data elsewhere
def download_progress_hook(count, blockSize, totalSize):
"""A hook to report the progress of a download. This is mostly intended for users with
slow internet connections. Reports every 5% change in download progress.
"""
# Hook function to display download progress in real-time
global last_percent_reported
percent = int(count * blockSize * 100 / totalSize)
if last_percent_reported != percent:
if percent % 5 == 0:
sys.stdout.write("%s%%" % percent)
sys.stdout.flush()
else:
sys.stdout.write(".")
sys.stdout.flush()
last_percent_reported = percent
def maybe_download(filename, expected_bytes, force=False):
"""Download a file if not present, and make sure it's the right size."""
dest_filename = os.path.join(data_root, filename)
# data_root is the current directory, add the filename to it, set as the location to save the file
if force or not os.path.exists(dest_filename):
# force is to force download, ignoring already downloaded files
print('Attempting to download:', filename)
filename, _ = urlretrieve(url + filename, dest_filename, reporthook=download_progress_hook)
# Use urlretrieve to download the file, with the hook attached
print('\nDownload Complete!')
statinfo = os.stat(dest_filename)
# Get information about the downloaded file
if statinfo.st_size == expected_bytes:
# Correct size
print('Found and verified', dest_filename)
else:
# Wrong size, prompt user to use a browser to download
raise Exception(
'Failed to verify ' + dest_filename + '. Can you get to it with a browser?')
return dest_filename
train_filename = maybe_download('notMNIST_large.tar.gz', 247336696)
test_filename = maybe_download('notMNIST_small.tar.gz', 8458043)

Assignment Code Segment Three

Extracting use cases

num_classes = 10
# Total number of digits
np.random.seed(133)
# Initialize random seed
def maybe_extract(filename, force=False):
# Assuming already extracted
root = os.path.splitext(os.path.splitext(filename)[0])[0] # remove .tar.gz
# splitext(filename)[0] removes one suffix, used twice to remove both suffixes, i.e., remove the .tar.gz suffix
if os.path.isdir(root) and not force:
# You may override by setting force=True.
# If already extracted, don't extract again
print('%s already present - Skipping extraction of %s.' % (root, filename))
else:
print('Extracting data for %s. This may take a while. Please wait.' % root)
tar = tarfile.open(filename)
sys.stdout.flush()
tar.extractall(data_root)
tar.close()
# Extract to the current directory
data_folders = [
os.path.join(root, d) for d in sorted(os.listdir(root))
if os.path.isdir(os.path.join(root, d))]
if len(data_folders) != num_classes:
raise Exception(
'Expected %d folders, one per class. Found %d instead.' % (
num_classes, len(data_folders)))
print(data_folders)
# Check if the number of extracted directories matches expectations, and print the extracted directories
return data_folders
train_folders = maybe_extract(train_filename)
test_folders = maybe_extract(test_filename)

Question One

Write code to display information about the extracted file contents

Reference answer

import random
import matplotlib.image as mpimg
def plot_samples(data_folders, sample_size, title=None):
fig = plt.figure()
# Create empty figure
if title: fig.suptitle(title, fontsize=16, fontweight='bold')
# Add title
for folder in data_folders:
# Loop through each letter
image_files = os.listdir(folder)
image_sample = random.sample(image_files, sample_size)
# Randomly select a certain number of images from that letter
for image in image_sample:
image_file = os.path.join(folder, image)
ax = fig.add_subplot(len(data_folders), sample_size, sample_size * data_folders.index(folder) +
image_sample.index(image) + 1)
# Create a subplot
image = mpimg.imread(image_file)
# Load subplot image
ax.imshow(image)
# Display subplot image
ax.set_axis_off()
# Turn off subplot coordinate lines
fig.set_size_inches(18.5, 10.5)
# Set the display size of the image
plt.show()
plot_samples(train_folders, 20, 'Train')
plot_samples(test_folders, 20, 'Test')

Running results:

As we can see, some of the training data has issues.


## Assignment Code Segment Four
After this, we need to normalize the data, which means transforming each image pixel from 0~255 to -1.0~1.0, and persisting it to a file.

image_size = 28 # Pixel width and height. pixel_depth = 255.0 # Number of levels per pixel.

Image width, height and pixel depth

def load_letter(folder, min_num_images): “““Load the data for a single letter label.”””

Process files in a folder belonging to one letter

image_files = os.listdir(folder)

List all files in that directory

dataset = np.ndarray(shape=(len(image_files), image_size, image_size), dtype=np.float32)

Create a dataset with length equal to number of files, width and height of 28

print(folder)

Print directory

num_images = 0

Initialize num_images

for image in image_files:

Process each file

image_file = os.path.join(folder, image)

Get complete file path

try:
image_data = (ndimage.imread(image_file).astype(float) -
pixel_depth / 2) / pixel_depth

Read in the image and normalize it

 if image_data.shape != (image_size, image_size):

Check image width and height

 raise Exception('Unexpected image shape: %s' % str(image_data.shape))
dataset[num_images, :, :] = image_data

Read into the dataset

 num_images = num_images + 1

Increment image number

except IOError as e:

If file can’t be read, skip it

 print('Could not read:', image_file, ':', e, '- it\'s ok, skipping.')

dataset = dataset[0:num_images, :, :]

If fewer files were read than the minimum required

if num_images < min_num_images: raise Exception(‘Many fewer images than expected: %d < %d’ % (num_images, min_num_images))

Display number of missing files

print(‘Full dataset tensor:’, dataset.shape)

Display file count, image width and height

print(‘Mean:’, np.mean(dataset))

Mean value

print(‘Standard deviation:’, np.std(dataset))

Standard deviation

return dataset

def maybe_pickle(data_folders, min_num_images_per_class, force=False): dataset_names = [] for folder in data_folders:

Process each letter folder

set_filename = folder + '.pickle'

Set output file

dataset_names.append(set_filename)

Set processed folders

if os.path.exists(set_filename) and not force:
# You may override by setting force=True.

Check if processed file already exists

 print('%s already present - Skipping pickling.' % set_filename)
else:
print('Pickling %s.' % set_filename)
dataset = load_letter(folder, min_num_images_per_class)

Normalize all images in this folder

 try:
with open(set_filename, 'wb') as f:
pickle.dump(dataset, f, pickle.HIGHEST_PROTOCOL)

Persist data, save to disk instead of keeping in memory

 except Exception as e:
print('Unable to save data to', set_filename, ':', e)

return dataset_names

train_datasets = maybe_pickle(train_folders, 45000) test_datasets = maybe_pickle(test_folders, 1800)


## Question Two
Display processed images
- Reference answer

def plot_samples_2(data_folders, sample_size, title=None): fig = plt.figure()

Create empty figure

if title: fig.suptitle(title, fontsize=16, fontweight='bold')

Add title

for folder in data_folders:

Loop through each letter

 with open(folder, 'rb') as pk_f:
data = pickle.load(pk_f)
for index, image in enumerate(data):
if index < sample_size :

Randomly select a certain number of images from that letter

 ax = fig.add_subplot(len(data_folders), sample_size, sample_size * data_folders.index(folder) +
index + 1)

Load subplot image

 ax.imshow(image)

Display subplot image

 ax.set_axis_off()

Turn off subplot coordinate lines

fig.set_size_inches(18.5, 10.5)

Set the display size of the image

plt.show()

plot_samples_2(train_datasets, 20, ‘Train’) plot_samples_2(test_datasets, 20, ‘Test’)


![image.png](/img/rescued/c09ea5ab69e9.png)
![image.png](/img/rescued/d4363496653b.png)
## Question Three
Check if the number of files under each letter is similar.
- Reference answer

file_path = ’notMNIST_large/{0}.pickle’ for ele in ‘ABCDEFJHIJ’: with open(file_path.format(ele), ‘rb’) as pk_f:

Loop through each directory

 dat = pickle.load(pk_f)

Load the persisted file in this directory

print('number of pictures in {}.pickle = '.format(ele), dat.shape[0])

Print relevant information

Results show that the numbers are basically consistent.
![Question 3 Result](/img/rescued/31bcb36f8c25.png)
## Code Segment—Data Splitting
Data cannot be loaded all at once into memory. This code segment splits the data.

def make_arrays(nb_rows, img_size): if nb_rows: dataset = np.ndarray((nb_rows, img_size, img_size), dtype=np.float32)

Create an empty set, data type is a matrix with rows length, img_size width, img_size height, data type is 32-bit float

labels = np.ndarray(nb_rows, dtype=np.int32)

Create a label, data type is 32-bit integer, length is rows

else: dataset, labels = None, None return dataset, labels

Return created data type

def merge_datasets(pickle_files, train_size, valid_size=0): num_classes = len(pickle_files)

Number of categories to process

valid_dataset, valid_labels = make_arrays(valid_size, image_size)

Build validation dataset, length is validation length

train_dataset, train_labels = make_arrays(train_size, image_size)

Build training dataset, length is training length

vsize_per_class = valid_size // num_classes tsize_per_class = train_size // num_classes

Calculate average length for each category with given training and validation lengths

start_v, start_t = 0, 0

Initialize indices, start_v is the start of validation data, start_t is the start of training data

end_v, end_t = vsize_per_class, tsize_per_class

Initialize indices, end_v is the end of validation data, end_t is the end of training data

end_l = vsize_per_class + tsize_per_class

Initialize indices, end_l is the end of the letter set, equal to length of validation data for each category + length of training data

for label, pickle_file in enumerate(pickle_files):

Loop through each pickle_file

try:
with open(pickle_file, 'rb') as f:

Open this persistence file

 letter_set = pickle.load(f)

Load dataset

 # let's shuffle the letters to have random validation and training set
np.random.shuffle(letter_set)

Shuffle the dataset

 if valid_dataset is not None:

If not a test set, update the test set, otherwise valid_dataset is not updated

 valid_letter = letter_set[:vsize_per_class, :, :]

numpy slicing http://brieflyx.me/2015/python-module/numpy-array-split/

Select data of ‘valid data per class’ count from shuffled data for processing, put into valid_letter

 valid_dataset[start_v:end_v, :, :] = valid_letter

Put this data into valid_dataset

 valid_labels[start_v:end_v] = label

Mark label should be one of 0~9

 start_v += vsize_per_class
end_v += vsize_per_class

Update indices

At the end of the loop, valid_dataset should be data with total length valid_size, valid_labels is the label at the corresponding position

 train_letter = letter_set[vsize_per_class:end_l, :, :]

Other random elements except valid part, length is end_l - vsize_per_class = tsize_per_class

 train_dataset[start_t:end_t, :, :] = train_letter

At the end of the loop, train_dataset should be data with total length train_size

 train_labels[start_t:end_t] = label
start_t += tsize_per_class
end_t += tsize_per_class

Update indices

except Exception as e:
print('Unable to process data from', pickle_file, ':', e)
raise

return valid_dataset, valid_labels, train_dataset, train_labels

train_size = 200000 valid_size = 10000 test_size = 10000

valid_dataset, valid_labels, train_dataset, train_labels = merge_datasets( train_datasets, train_size, valid_size) _, _, test_dataset, test_labels = merge_datasets(test_datasets, test_size)

print(‘Training:’, train_dataset.shape, train_labels.shape) print(‘Validation:’, valid_dataset.shape, valid_labels.shape) print(‘Testing:’, test_dataset.shape, test_labels.shape)


## Code Segment—Shuffling Data
Introduction to the permutation function: http://www.jianshu.com/p/f0eb10acaa2d

def randomize(dataset, labels):

labels.shape[0] is the length of labels

permutation = np.random.permutation(labels.shape[0])

Randomly select a shuffled set of this many numbers

print(labels.shape[0]) shuffled_dataset = dataset[permutation,:,:]

Shuffle data

shuffled_labels = labels[permutation]

Shuffle labels

return shuffled_dataset, shuffled_labels train_dataset, train_labels = randomize(train_dataset, train_labels) test_dataset, test_labels = randomize(test_dataset, test_labels) valid_dataset, valid_labels = randomize(valid_dataset, valid_labels)


## Question Four
Verify if the shuffled data is correct
- Reference answer

import random def plot_sample_3(dataset, labels, title): fig = plt.figure() plt.suptitle(title, fontsize=16, fontweight=‘bold’)

Set title style

items = random.sample(range(len(labels)), 200)

Shuffle the sequential sequence of labels length

for i, item in enumerate(items):

Randomly pick one

 plt.subplot(10, 20, i + 1)

Draw subplot

 plt.axis('off')

Turn off coordinate axes

 plt.title(chr(ord('A') + labels[item]))

Add title

 plt.imshow(dataset[item])

Display subplot at corresponding position

fig.set_size_inches(18.5, 10.5)
plt.show()

Display image

plot_sample_3(train_dataset, train_labels, ’train dataset suffled’) plot_sample_3(valid_dataset, valid_labels, ‘valid dataset suffled’) plot_sample_3(test_dataset, test_labels, ’test dataset suffled’)


![Question 4](/img/rescued/2041fe44668c.png)
Similar two figures omitted
## Code Segment—Saving Data

pickle_file = os.path.join(data_root, ’notMNIST.pickle’)

Output file path

try: f = open(pickle_file, ‘wb’)

Open this file

save = { ’train_dataset’: train_dataset, ’train_labels’: train_labels, ‘valid_dataset’: valid_dataset, ‘valid_labels’: valid_labels, ’test_dataset’: test_dataset, ’test_labels’: test_labels, }

Write a dictionary string-ndarray

pickle.dump(save, f, pickle.HIGHEST_PROTOCOL) f.close() except Exception as e: print(‘Unable to save data to’, pickle_file, ‘:’, e) raise


## Code Segment—Displaying Saved Data Size

statinfo = os.stat(pickle_file) print(‘Compressed pickle size:’, statinfo.st_size)


## Question Five
Google translation of the question:
By construction, this dataset may contain a lot of overlapping samples, including in the validation and test sets. Overlap between training and test can skew the results if you expect to use your model in an environment where there is never an overlap, but in practice this doesn't usually matter. Measure how much overlap there is between training, validation, and test samples.
Optional question:
What about the duplicates between datasets? (For instance, the same letter images)
Create a sanitized validation and test set, and compare your accuracy on those versus your accuracy on the original sets.
The basic idea is that training data should not overlap with testing data, otherwise it leads to inaccurate accuracy.
Reference code:
- Just check the number of duplicate images

import hashlib

pickle_file = os.path.join(’.’, ’notMNIST.pickle’) try: with open(pickle_file, ‘rb’) as f: data = pickle.load(f) except Exception as e: print(‘Unable to open data from’, pickle_file, ‘:’, e) raise

After saving the data, if the kernel crashed, you can read directly from local without rerunning previous code

If there’s an error, you can search for the exception online

def calcOverlap(sourceSet, targetSet, description): sourceSetMd5 = np.array([hashlib.md5(img).hexdigest() for img in sourceSet])

Build an md5 table

targetSetMd5 = np.array([hashlib.md5(img).hexdigest() for img in targetSet])

Build an md5 table

overlap = np.intersect1d(sourceSetMd5, targetSetMd5, assume_unique=False)

Deduplicate

print(description)
print("overlap",overlap.shape[0], "from",sourceSetMd5.shape[0],"to", targetSetMd5.shape[0])
print("rate",overlap.shape[0]*100.0/sourceSetMd5.shape[0],"% and", overlap.shape[0]*100.0/targetSetMd5.shape[0],"%")

Print overlap count

calcOverlap(data[’train_dataset’], data[‘valid_dataset’], “train_dataset & valid_dataset”) calcOverlap(data[’train_dataset’], data[’test_dataset’], “train_dataset & test_dataset”) calcOverlap(data[’test_dataset’], data[‘valid_dataset’], “test_dataset & valid_dataset”)


![Running result](/img/rescued/fc3c3db78e52.png)
- Remove duplicate image resources
To be updated
## Question Six
Use logistic regression to train the model and test it
- Reference code

import random def disp_sample_dataset(dataset, labels,trueLabels, title=None):

Display training results

fig = plt.figure()
if title: fig.suptitle(title, fontsize=16, fontweight='bold')

Set title style

items = random.sample(range(len(labels)), 200)

Randomly select a series of images

for i, item in enumerate(items):
plt.subplot(10, 20, i + 1)

Set a subplot

 plt.axis('off')

Turn off coordinate lines

 lab = str(chr(ord('A') + labels[item]))
trueLab = str(chr(ord('A') + trueLabels[item]))
if lab == trueLab:
plt.title( lab )
else:
plt.title(lab + " but " + trueLab)

Add title

 plt.imshow(dataset[item])

Display this image

fig.set_size_inches(18.5, 10.5)
plt.show()

def train_and_predict(train_dataset, train_labels, test_dataset, test_labels ,sample_size): regr = LogisticRegression()

Generate trainer

X_train = train_dataset[:sample_size].reshape(sample_size, 784)

Choose amount of data to train based on sample_size

Compress 2D vector to 1D vector

y_train = train_labels[:sample_size]

Extract training data

regr.fit(X_train, y_train)

Train data

X_test = test_dataset.reshape(test_dataset.shape[0], 28 * 28)

Compress test data to 1D vector

y_test = test_labels

True labels corresponding to test data

pred_labels = regr.predict(X_test)

Generate prediction data

print('Accuracy:', regr.score(X_test, y_test), 'when sample_size=', sample_size)
disp_sample_dataset(test_dataset, pred_labels, test_labels, 'sample_size=' + str(sample_size))

train_and_predict(data[’train_dataset’],data[’train_labels’],data[’test_dataset’],data[’test_labels’], 1000)


![image.png](/img/rescued/abdb5d761390.png)
## Model Performance
Sections 22~27 discuss model performance-related knowledge. We usually hope that the model's performance can reach 100%, which is obviously impossible. Also, in order to improve the accuracy of the training set, the model may overfit. At this point, we should follow two points:
- Don't use all training data at once, but use it in blocks, train a portion each time
- When model parameter changes cause 30 or more cases to change from error to correct, then this parameter change is effective.
![Model Performance](/img/rescued/a8e154206ab8.png)
## Stochastic Gradient Descent
Sections 29~31 explain what stochastic gradient descent is.
During training, to make the model move in the optimal direction, we need to calculate the derivative at that point. 1. The calculation of derivatives is quite large, so we need to randomly select a subset of samples to calculate derivatives, to substitute for the real derivative. This is stochastic gradient descent. 2. To reduce the randomness of random selection, we use momentum inertia to reduce randomness. 3. To make the model stable in later stages, we reduce the learning step size.
End of Course One
> Reference for assignment code
> http://www.hankcs.com/ml/notmnist.html

Learning Cocos Creator from Zero - Part One

Tue, 27 Jun 2017 09:31:00 +0000

Introduction

Despite the prejudice against Cocos2dx on Zhihu, it’s better to try different game engines rather than hesitating about which one to use. Moreover, this software was developed by Chinese developers, so it shouldn’t be too difficult to learn.

Although I haven’t learned JavaScript before, it seems possible to look up references while writing. JavaScript tutorial: http://www.liaoxuefeng.com/wiki/001434446689867b27157e896e74d51a89c25cc8b43bdb3000

The current version of Cocos Creator is 1.5.1, and there’s a detailed manual on the official website at http://www.cocos.com/docs/creator/. This article is a simplified version of the “Quick Start” chapter, with improved gameplay elements.

Installation

Installation on Windows is straightforward. For Mac installation, there are a few things to note. After downloading the installation package from the official website and dragging it to Applications, when you open it, you’ll see a warning that the application is from an untrusted developer. In this case, open System Preferences > Security & Privacy > General, where you’ll see an “Open” button. After opening it this way, you won’t be reminded again.

Quick Start

From the dashboard, you can create a new project or open an existing one. According to the tutorial’s quick start section, we can download an initial project from this address: Download Initial Project. Or click here to download: https://github.com/cocos-creator/tutorial-first-game/releases/download/v1.2/start_project.zip

The final game effect of this quick start is like this: http://fbdemos.avosapps.com/star-catcher/. This game is somewhat similar to coin-catching games, but more difficult. The game has a very steep learning curve. Due to the monster’s horizontal acceleration being difficult to control and its jumping up and down, it’s almost impossible to catch the stars. There’s also no storyline. We should make it more fun.

After opening the project, let’s learn about the interface. The resource panel is in the bottom left. In the initial project, a project directory has already been established with the most basic resources and directory organization. “Assets” means assets, and under this main directory, there are three subdirectories containing fonts, audio, and image resources.

Creating a Scene

Scenes are crucial because they contain game scripts, and scenes are automatically loaded when the game starts. Click the plus sign under assets in the resource panel to create a scene, rename it to “game”, and double-click to open it.

The hierarchy manager in the upper left shows the relationships between nodes in the current scene. In this game scene, there is only one root node, canvas. Currently, this node is empty, meaning nothing will be loaded.

Click the root node in the right data panel to set the resolution of the root node. The height of the resolution is set to adaptive, always 960x640, consistent with the aspect ratio of the first-generation iPhone. I guess this resolution should be dp rather than pixel.

Scene Graphics

Drag the background resource onto the canvas, making the background a child node of the canvas. Be careful not to make the background another root node. Use Cmd+S to save the changes. Select the background image in the scene, then find the fourth button of the transform tools in the upper left, which are translation, rotation, scaling, and rectangular transformation. Select rectangular transformation and transform the background into a size that can cover the scene, as shown below.

The above steps can also be done by directly setting the properties of the background, setting the position to 0, 0, and the width and height to 1600 and 800.

Use the same method to add the ground. Use the same method to insert the little monster and rename it to “player”. The default anchor point of the image is the center position. Here, set the y value of the anchor to 0.

Creating Scripts

Then comes an amazing moment: the tutorial actually says it doesn’t matter if you don’t know programming, you can let your programmer friend solve it. Indeed, all I’m missing now is a programmer.

Create a JavaScript script file at the location of the player resource in the image below, and open it.

Insert the following content in properties, which are the physical properties of the little monster:

 jumpHeight:0,
jumpDuration:0,
maxMoveSpeed:0,
accel:0,

Select the little monster image, find “Add Component” in the property box, add the player script to the player image, and set the relevant parameters.

Improving the Script

Let’s continue to add other functions to the script. First is the jumping action.

Add this section below properties:

 setJumpAction: function(){
var jumpUp = cc.moveBy(this.jumpDuration, cc.p(0,this.jumpHeight)).easing(cc.easeQuadraticActionOut());
var jumpDown = cc.moveBy(this.jumpDuration, cc.p(0,-this.jumpHeight)).easing(cc.easeQuadraticActionIn());
return cc.repeatForever(cc.sequence(jumpUp, jumpDown));
},

moveBy is an official API. Looking up its usage in the official API, the first parameter is the movement duration, and the second parameter is the movement position. Obviously, it moves up a certain distance. The easing that follows generates a gradual motion, producing easeQuadraticActionIn and Out, which are quadratic curves. The original tutorial used cubic curves, which differ too much from physical movement in the real world.

Add the following code to the onLoad method to start the animation:

 onLoad: function () {
this.jumpAction = this.setJumpAction();
this.node.runAction(this.jumpAction);
},

Now click the play button to view the initial effect.

Weekly LeetCode — 501. Find Mode in Binary Search Tree

Mon, 26 Jun 2017 10:54:00 +0000

Problem

Given a Binary Search Tree (BST), find the mode (most frequently occurring value). Requirement: Apart from the recursion stack space, the space complexity should be O(1).

Analysis

Clearly, an in-order traversal of a binary search tree produces a sorted array. Finding the mode in a sorted array means looking for the longest sequence of consecutive repeated numbers. Note that the problem requires returning all modes if there are multiple values with the same highest frequency. Additionally, the space complexity must be O(1). Writing code that meets these requirements is not difficult.

Runtime: 16ms, beats 62% of submissions

/**
* Definition for a binary tree node.
* struct TreeNode {
* int val;
* TreeNode *left;
* TreeNode *right;
* TreeNode(int x) : val(x), left(NULL), right(NULL) {}
* };
*/
class Solution {
private:
vector<int> mode;
int last;
int lastCount;
int modeCount;
public:
vector<int> findMode(TreeNode* root) {
last = 0;
lastCount = 0;
modeCount = 0;
mode.clear();
runMode(root);
return mode;
}
void runMode(TreeNode* root){
if(root != NULL){
runMode(root -> left);
int nowValue = root->val;
int nowCount = nowValue != last? 1 : lastCount + 1;
if(modeCount == nowCount){
mode.push_back(nowValue);
}
if(modeCount < nowCount){
modeCount = nowCount;
mode.clear();
mode.push_back(nowValue);
}
last = nowValue;
lastCount = nowCount;
runMode(root -> right);
}
}
};

Weekly LeetCode — 3. Longest Substring Without Repeating Characters

Sat, 24 Jun 2017 08:21:00 +0000

Problem

Find the longest substring in a given string that doesn’t contain any repeating characters, and return the length of this substring.

Approach

It’s easy to come up with an algorithm with O(n^2) time complexity. Iterate through each character, and for each character as a starting point, determine the length of the longest substring without repeating characters. Then output the maximum length among all substrings.

Runtime: 600ms, beats 0% of submissions

class Solution {
public:
int lengthOfLongestSubstring(string s) {
int ret = 0;
for(int i=0; i<s.size(); i++){
unordered_set<char> buff;
for(int j=i; j<s.size(); j++){
if(buff.count(s[j]) == 0) buff.insert(s[j]);
else break;
}
if(buff.size() > ret) ret = buff.size();
}
return ret;
}
};

Improvement

As we can see, the algorithm above is quite inefficient and is one of the slowest algorithms in the world. When searching for substrings, we need to set a starting point. This starting point is the position where, up to the current iteration, we have the longest substring without repeating characters. By subtracting this position from the current iteration position, we can get the length of the longest substring.

Runtime: 15ms, beats 96% of submissions

class Solution {
public:
int lengthOfLongestSubstring(string s) {
vector<int> lastPos(260, -1);
int length = s.size();
int ret = 0;
int start = 0;
for(int i=0; i<length; i++)
{
char c = s[i];
if(lastPos[c]+1 > start) start = lastPos[c] + 1;
if(i-start+1 > ret) ret = i-start+1;
lastPos[c] = i;
}
return ret;
}
};

Weekly LeetCode — 260. Single Number III

Thu, 22 Jun 2017 03:49:00 +0000

Problem

Given an array of numbers, where all numbers appear twice except for two numbers that appear only once. Find these two numbers. Try to achieve O(1) space complexity.

Analysis

Although the solution requires O(1) space complexity, this approach is quite complex and difficult to conceive, and it’s challenging to write an algorithm with a small constant factor. Feel the magic of XOR The link above provides the standard approach for this problem. In this method, first XOR all the numbers, and the result will be the XOR of the two numbers that appear only once. Each bit 1 in this result indicates which bits differ between these two numbers. Pick any one of these bits, and divide the array into two groups based on this bit: one group where this bit is 1, and another where this bit is 0. Then, XORing each group separately will give the two numbers we’re looking for.

This leverages the properties of XOR:

a ^ b ^ c = a ^ c ^ b a ^ a = 0 0 ^ a = a a ^ b = c => a ^ c = b

Runtime: 13ms, beats 68% of submissions.

class Solution {
public:
vector<int> singleNumber(vector<int>& nums) {
int axorb = 0, last = 0;
vector<int> ret(2, 0);
for(auto it = nums.begin(); it!=nums.end() ; it++)
{
axorb ^= *it;
}
last = axorb & (~(axorb - 1));
for(auto it = nums.begin(); it!=nums.end() ; it++)
{
if ((last & *it) != 0)
ret[0] ^= *it;
}
ret[1] = axorb ^ ret[0];
return ret;
}
};

Alternative Method

If we’re not restricted to constant space complexity, we can also use a hash table. This approach has good extensibility and is less obscure.

Runtime: 16ms, beats 34% of submissions.

class Solution {
public:
vector<int> singleNumber(vector<int>& nums) {
unordered_set<int> buff(nums.size());
for(auto i = nums.begin(); i!=nums.end() ; i++)
{
auto it = buff.find(*i);
if(it == buff.end()){
buff.insert(*i);
}
else{
buff.erase(it);
}
}
vector<int> ret;
for(const int & i : buff){
ret.push_back(i);
}
return ret;
}
};

Weekly LeetCode — 43. Multiply Strings

Mon, 19 Jun 2017 06:39:00 +0000

Problem:

Given two decimal numbers as strings, return their product as a string. Requirements:

Do not use built-in big number arithmetic.
String length ≤ 110
Inputs have no leading zeros
Strings contain only digits

Thoughts

Initially, I didn’t notice the prohibition on using built-in functions, so I directly used Python’s str and int functions, which ranked in the top 10%… Later, I realized the problem doesn’t allow built-in functions. I’m not sure if this Python feature counts as built-in big number arithmetic. This is a big number multiplication problem, and there are many established algorithms for this.

My initial Python submission, which beat 88% of submissions:

class Solution(object):
def multiply(self, num1, num2):
return str(int(num1)*int(num2))

The most basic algorithm would be to simulate manual multiplication. First, you need to implement string addition and multi-digit multiplication by a single digit. With these, you can calculate multi-digit by multi-digit multiplication. For example, to calculate 12345*67890, you compute 12345*6 + 12345*7 + 12345*8 + 12345*9 + 12345*0, then add trailing zeros to each result and sum them. However, this approach would be relatively slow.

Moreover, CPUs can already handle additions that don’t overflow, so we should leverage this. We can improve the above algorithm by calculating 12345*67890 as (12300 + 45)*(67800 + 90). This breaks down into 4 multiplication operations. For numbers with trailing zeros, we can remove them and add them back to the result later. If none of these four multiplications overflow, there’s no problem. Otherwise, we can continue decomposing.

The Karatsuba algorithm can further improve this approach. We notice that in the addition, there’s 12300*90+45*67800. We can use previously calculated results, namely 12300*67800 and 45*90, and then calculate (12300+45)*(67800+90) - 12300*67800 - 45*90 to get 12300*90 + 45*67800. This reduces the number of multiplication operations by one.

Implementing the Karatsuba algorithm would be a bit more complex, so I’ll first submit an O(n²) algorithm (the basic version) to see its performance, and then improve it. (Results show that the manual calculation method is already quite fast.)

First Version

First, we need to write a string addition algorithm. Looking at the input and output data types, they’re strings. So we can add them digit by digit. We can use two integer arrays to store each digit, then add them to create a third array. Some digits in this third array will exceed 10, so we carry over from the lower digits to the higher ones. Finally, we convert this array back to a string.

Although local testing was fine, the submission was very slow, ranking only at the 10th percentile.

Improvement

Theoretically, this algorithm shouldn’t be slow, but in practice it is. The issue might be with unnecessary conversions between integers and strings. In the above algorithm, when calculating multiplication, we converted strings to numbers and then back to strings, which might be causing the extra time. So, we should directly add the results to the final result.

After submission, the runtime was 9ms, beating 50% of submissions.

Further Improvement

Repeatedly converting the same strings might be consuming time. We can cache string conversions - convert once and store the result, then retrieve directly without additional calculations when needed.

The final code, ready for compilation and execution. Runtime: 6ms, beating 76% of submissions. It seems string conversion was indeed the performance bottleneck.

#include <stdio.h>
#include <string>
#include <stdlib.h>
#include <iostream>
#include <vector>
using namespace std;
const int BINARY = 10;
class Solution
{
private:
vector<int> result;
string _num1, _num2;
long num1buff[120];
long num2buff[120];
public:
string multiply(string num1, string num2)
{
result.clear();
result.resize(num1.length() + num2.length() + 1);
memset(num1buff, -1 , sizeof(long)*120);
memset(num2buff, -1 , sizeof(long)*120);
_num1 = num1;
_num2 = num2;
for(auto &c : _num1){
c-='0';
}
for(auto &c : _num2){
c-='0';
}
// This is a recursive process. Let's see when it terminates.
// It terminates when two numbers multiplied don't overflow.
// Assuming int is 30 bits in binary (for multiplication, we need 30 bits),
// the original two numbers must be 15 bits, which is about 32768.
// So two 4-digit numbers multiplied shouldn't overflow.
// Using recursion to calculate the product
addMultiply(0,num1.length(), 0, num2.length());
string ret;
int i = result.size() -1;
for(; i>0; i--)
{
if(result[i] != 0) break;
}
for(; i>=0; i--)
{
ret.push_back(result[i] + '0');
}
return ret;
}
void addMultiply(int a1, int a2, int b1, int b2 )
{;
// Check if direct calculation is possible
if (a1 == a2 || b1 == b2)
return;
if (a2 - a1 < 10 && b2 - b1 < 10)
{
long int_num1 = getLong1(a1, a2);
long int_num2 = getLong2(b1, b2);
long output = int_num1 * int_num2;
int pos = _num1.length() + _num2.length() - a2 - b2;
while (output != 0 || result[pos] >= BINARY)
{
long a = output % BINARY;
result[pos] += a;
result[pos + 1] += result[pos] / BINARY;
result[pos] %= BINARY;
output /= BINARY;
pos++;
}
return;
}
// Otherwise, split the longer number
if(a2 - a1 >= 10){
addMultiply(a1, (a2 + a1)/2, b1, b2);
addMultiply((a2 + a1)/2, a2, b1, b2);
}
else {
addMultiply(a1, a2, (b1+b2)/2, b2);
addMultiply(a1, a2, b1, (b1+b2)/2);
}
}
long getLong1(int a, int b){
long ret = 0;
if(num1buff[a] != -1) return num1buff[a];
for(int i=a; i!=b;i++){
ret *= BINARY;
ret += _num1[i] ;
}
num1buff[a] = ret;
return ret;
}
long getLong2(int a, int b){
long ret = 0;
if(num2buff[a] != -1) return num2buff[a];
for(int i=a; i!=b;i++){
ret *= BINARY;
ret += _num2[i] ;
}
num2buff[a] = ret;
return ret;
}
};
int main(void)
{
Solution s;
for(int i=0;i<10000;i++){
cout << s.multiply("12345678901", "100") << endl;
cout << s.multiply("100", "100") << endl;
}
return 0;
}

Final ranking: beats 70% of submissions

How to Set Up Xiaohe Shuangpin in Microsoft Pinyin for Windows 10

Mon, 05 Jun 2017 06:16:00 +0000

Introduction

The latest version of Windows 10 system’s built-in Pinyin input method already supports Xiaohe Shuangpin. Mom no longer has to worry about me installing bloatware!

After 4 years, here’s an updated advanced setup method. Mom no longer has to worry about me configuring keys one by one:

Press Win + R, type regedit, and open Registry Editor
Find the Computer\HKEY_CURRENT_USER\Software\Microsoft\InputMethod\Settings\CHS entry
Create a new string value named UserDefinedDoublePinyinScheme0 with the value 小鹤双拼*2*^*iuvdjhcwfg^xmlnpbksqszxkrltvyovt
Open Microsoft Pinyin input method settings and set Xiaohe Shuangpin as the default scheme for Shuangpin.

If you’re concerned about registry modifications causing system issues, here’s a manual configuration method:

Step 1: Open Settings

Although the system’s Pinyin input method doesn’t have Xiaohe Shuangpin by default, we can customize it.

First, find the Shuangpin settings location. If you can’t find it, you can refer to the animated GIF:

Step 2: Edit Custom Shuangpin

The Shuangpin settings are shown in the figure below

Except for the template section which doesn’t need to be set, the rest should be configured according to the image. One is the zero initial mode, and the other is custom key mapping. Custom key mapping should be set according to the preview in the image. Note that there is no “er” syllable in Xiaohe Shuangpin, so you can set it to either the “e” or “r” key. Also, both the “ue” and “ve” syllables are set to the “T” key. Here’s a tip: when setting keys, you can directly press the corresponding key instead of finding the key to set.

Step 3: Save the Scheme

If the add button becomes available, it means the scheme is set correctly. After adding the scheme, set it as the default scheme, and you can now use Xiaohe Shuangpin!

Testing

Perform a typing test to check if the settings are correct

Step 4: Click “Like”

Click “Like” to improve page ranking and help more people use Xiaohe Shuangpin on Windows 10

Using STM32 HAL Library to Drive UART in RT-Thread

Thu, 06 Apr 2017 10:22:00 +0000

Introduction

Previously, I had been trying to develop STM32 with RT-Thread under Linux. While this approach is feasible, it comes with various inconveniences. Using VSCode + Scons + openocd for editing, compiling/linking, and flashing is possible for development, but there’s no complete solution for debugging in this environment, requiring Eclipse for debugging. Although converting RT-Thread to an Eclipse project is possible, at that point it might be better to directly use Keil 5 for development under Windows.

Since the STM32F103’s onboard flash is too small and its stack is very limited, I switched to STM32F411. The board is from Waveshare’s XNUCLEO, which is compatible with STM’s official NUCLEO boards.

Installing Keil 5.23

I first installed Keil 5.11 and cracked it, then uninstalled it and installed Keil 5.23, which automatically showed as successfully cracked. The cracking tool is the registration code generator that has been popular for many years. I’m using Keil 5.23 because RT-Thread projects can only be opened with this version.

Installing Libraries with Pack Installer

You need to install Keil::STM32F4xx_DFP to open the nucleo project in RT-Thread’s bps branch.

Installing STM32CubeMX

With STM32CubeMX, we can easily generate initialization code by clicking with the mouse. However, we still need to write function calls ourselves, which is not as convenient as NPX’s PE.

Installing ST-Link Driver

Waveshare’s 100 yuan board includes an ST-Link debugger, so we’ll use ST-Link for debugging.

Downloading the Nucleo Project

Open the Keil 5 project in rt-thread-master\bsp\stm32f411-nucleo, try to compile it, and after it passes, set up the project’s debug configuration to use ST-Link for debugging, then download the program to the board. Although the program is downloaded, there’s no visible effect because the main function is empty.

Let’s add the following code to the main function:

int main(void)
{
 /* user app entry */
 MX_GPIO_Init();
 while(1){
 rt_thread_delay(5);
 HAL_GPIO_WritePin (GPIOC, GPIO_PIN_5, GPIO_PIN_SET);
 rt_thread_delay(5);
 HAL_GPIO_WritePin (GPIOC, GPIO_PIN_5, GPIO_PIN_RESET);
 }
}

And add the LED initialization function:

static void MX_GPIO_Init(void)
{
GPIO_InitTypeDef GPIO_InitStruct;
/* GPIO Ports Clock Enable */
__HAL_RCC_GPIOC_CLK_ENABLE();
/*Configure GPIO pin Output Level */
HAL_GPIO_WritePin(GPIOC, GPIO_PIN_5, GPIO_PIN_RESET);
/*Configure GPIO pin : PC5 */
GPIO_InitStruct.Pin = GPIO_PIN_5;
GPIO_InitStruct.Mode = GPIO_MODE_OUTPUT_PP;
GPIO_InitStruct.Pull = GPIO_NOPULL;
GPIO_InitStruct.Speed = GPIO_SPEED_FREQ_LOW;
HAL_GPIO_Init(GPIOC, &GPIO_InitStruct);
}

Now we can see the onboard LED4 light up and blink.

Where do these initialization codes come from? The code in the main function is found online, a simple function to operate GPIO. The MX_GPIO_Init code segment comes from the code automatically generated by STM32CubeMX. Below I’ll explain how to generate UART initialization code using STM32CubeMX.

Generating UART Initialization Code with STM32CubeMX

First, create a project and set the chip model. Here’s a demonstration:

Then set up the serial port pins. Since RT-Thread has already initialized UART1, we can’t use this one. We can use UART6.

Open the project and extract the code. The blue part is the code we need to take:

Port the code In the void HAL_UART_MspInit(UART_HandleTypeDef *huart) function, add the code for USART6:

void HAL_UART_MspInit(UART_HandleTypeDef *huart)
{
GPIO_InitTypeDef GPIO_InitStruct;
if (huart->Instance == USART2)
{
/*##-1- Enable peripherals and GPIO Clocks #################################*/
/* Enable GPIO TX/RX clock */
USARTx_TX_GPIO_CLK_ENABLE();
USARTx_RX_GPIO_CLK_ENABLE();
/* Enable USARTx clock */
USARTx_CLK_ENABLE();
/*##-2- Configure peripheral GPIO ##########################################*/
/* UART TX GPIO pin configuration */
GPIO_InitStruct.Pin = USARTx_TX_PIN;
GPIO_InitStruct.Mode = GPIO_MODE_AF_PP;
GPIO_InitStruct.Pull = GPIO_PULLUP;
GPIO_InitStruct.Speed = GPIO_SPEED_FAST;
GPIO_InitStruct.Alternate = USARTx_TX_AF;
HAL_GPIO_Init(USARTx_TX_GPIO_PORT, &GPIO_InitStruct);
/* UART RX GPIO pin configuration */
GPIO_InitStruct.Pin = USARTx_RX_PIN;
GPIO_InitStruct.Alternate = USARTx_RX_AF;
HAL_GPIO_Init(USARTx_RX_GPIO_PORT, &GPIO_InitStruct);
HAL_NVIC_SetPriority(USART2_IRQn, 0, 1);
HAL_NVIC_EnableIRQ(USART2_IRQn);
}
if (huart->Instance == USART6)
{
/* USER CODE BEGIN USART6_MspInit 0 */
/* USER CODE END USART6_MspInit 0 */
/* Peripheral clock enable */
__HAL_RCC_USART6_CLK_ENABLE();
/**USART6 GPIO Configuration
PC6 ------> USART6_TX
PC7 ------> USART6_RX
*/
GPIO_InitStruct.Pin = GPIO_PIN_6|GPIO_PIN_7;
GPIO_InitStruct.Mode = GPIO_MODE_AF_PP;
GPIO_InitStruct.Pull = GPIO_PULLUP;
GPIO_InitStruct.Speed = GPIO_SPEED_FREQ_VERY_HIGH;
GPIO_InitStruct.Alternate = GPIO_AF8_USART6;
HAL_GPIO_Init(GPIOC, &GPIO_InitStruct);
/* Peripheral interrupt init */
HAL_NVIC_SetPriority(USART6_IRQn, 0, 0);
HAL_NVIC_EnableIRQ(USART6_IRQn);
/* USER CODE BEGIN USART6_MspInit 1 */
/* USER CODE END USART6_MspInit 1 */
}
}

In the main.c file, add the MX_USART6_UART_Init(); function content, and call MX_USART6_UART_Init(); in the main function to initialize.

Then, test it in the main() function:

 uint8_t TxData[10]= "01234abcde";
HAL_UART_Transmit(&huart6,TxData,10,0xf);

We’ll place the Waveshare board’s JP4 jumper on the right side so that the USB TO UART can use PC6 and PC7 as the serial interface. Then, connect the USB interface to the PC, and we can view the character string received by the PC in a serial terminal like XSHELL.

Using SPI and UART Devices in RT-Thread

Tue, 07 Mar 2017 09:41:00 +0000

RT-Thread uses function calls similar to Linux. Here’s how to use devices in a simple way.

Hardware Environment: STM32F103RBT6

User Program Location

User programs are written and called in applications.c in the applications folder under the bsp folder. If you have many programs, create another c file to separate them.

SPI Driver Files

The stm32f10x drivers folder doesn’t have SPI drivers. We need to port them from stm32f107. Copy rt_stm32f10x_spi.c, rt_stm32f10x_spi.h, and platform.c files, then add the call to rt_platform_init() in the rt_init_thread_entry function, and add the headers to the corresponding files. Then modify SConscript to include platform.c and rt_stm32f10x_spi.c for compilation, and add the RT_USING_SPI and RT_USING_SPI1 macro definitions in the config.

Add the following code to the SPI initialization in platform.c to configure the frequency:

 /* config spi */
{
struct rt_spi_configuration cfg;
cfg.data_width = 8;
cfg.mode = RT_SPI_MODE_3 | RT_SPI_MSB; /* SPI Compatible Modes 3 and SPI_FirstBit_MSB in lis302dl datasheet */
//APB2=168M/2=84M, SPI1 = 84/2,4,8,16,32 = 42M, 21M, 10.5M, 5.25M, 2.625M ...
cfg.max_hz = 2625000; /* SPI_BaudRatePrescaler_16=84000000/16=5.25MHz. The max_hz of lis302dl is 10MHz in datasheet */
rt_spi_configure(&spi_device, &cfg);
} /* config spi */

Hardware Configuration

The hardware pins are configured in the platform.c file. We won’t modify the default configuration, which means PA4567 are the default SPI input/output pins.

/*
* SPI1_MOSI: PA7
* SPI1_MISO: PA6
* SPI1_SCK : PA5
*
* CS0: PA4 SD card. You can change this to your own SPI device's CS pin
*/

SPI Calls

SPI calls don’t require the open function, just execute:

 rt_spi_transfer(spi_device, &ReadAddr, &pBuffer, 1);

or rt_spi_send_then_recv can also achieve different effects.

UART Calls

UART calls require the open function. Here’s a simple example of using a serial device:

void device_thread_entry(void* parameter)
{
rt_device_t device;
device = rt_device_find("uart2");
rt_device_open(device, RT_DEVICE_OFLAG_RDWR|RT_DEVICE_FLAG_INT_RX);
char test[10] = "abc";
while(1){
rt_size_t reclen = rt_device_read(device, 0, test, 10);
if(reclen > 0) rt_device_write(device, 0, test, reclen);
rt_thread_delay(500);
}
}

You can start this thread, and when we input up to 10 characters in uart2, they will be echoed back after 5 seconds. If more than 10 characters are entered, they should be displayed in the next 5-second cycle. The official markdown documentation uses a message mechanism to handle UART communication, which can also be referenced.

Deploying RT-thread System on STM32 in Ubuntu Environment

Thu, 02 Mar 2017 07:59:00 +0000

RT-Thread is an open-source embedded operating system from China, with the current version being 2.2. It supports many microcontrollers, such as the STM32 series, and can even run on Bluetooth BLE chips. Despite resource constraints in microcontrollers, deploying an easy-to-use micro operating system is beneficial for project development and code management.

Required Tools

An STM32 development board with USB download capability
Ubuntu system
USB to serial cable

I’m using the classic ALIENTEK STM32 MINI V2.0 development board with an STM32F103RBT6 chip, and Ubuntu version 16.04

1. Download the Cross-compilation Toolchain

Download the gcc-arm-none-eabi cross-compilation toolchain. For 16.04 systems, simply use apt install gcc-arm-none-eabi to install

2. Download RT-Thread Compilation Tools

This system uses scons for compilation. For 16.04 systems, use apt install scons to install

3. Download the Source Code

Download the RT-Thread system from: https://github.com/RT-Thread
Extract the directory

4. Modify the Compilation Project

Enter the bsp/stm32f10x directory
Edit the rtconfig.py file
Change CROSS_TOOL to gcc

5. Configure for Your Board

Configure the crystal oscillator in stm32f10x.h via HSE_VALUE, for example 800000
Configure RAM in board.h via STM32_SRAM_SIZE, for example 20
Configure the LED in led.c

6. Compile the Source Code

Run scons in the bsp/stm32f10x directory to generate the rtthread.bin file

7. Flash the Code

You can use Jlink for downloading. Download the Linux executable file from the Jlink official website. Since I’m using a Debian system, I can directly open the JLink_Linux_V614a_x86_64.deb file to install. Then, in the system’s lib/ directory, execute the JlinkExe file to download and flash the program through Jlink.
Here I used a more convenient serial port flashing method. First set pin B0 to high level, B1 to low level, then use the STM32flash-5.0 software for flashing.
- Download from https://sourceforge.net/projects/stm32flash/
- Extract the directory, run make in that directory to get stm32flash
- Copy rtthread.bin to that directory, connect the USB to serial cable, then enter sudo ./stm32flash -w rtthread.bin -v -g 0x0 /dev/ttyUSB0, where ttyUSB0 is the serial device

8. Verify Success

If you can see the LED blinking, the deployment was successful

Adding SPI Driver Support for Cubieboard2

Tue, 28 Feb 2017 12:14:00 +0000

0. Introduction

Since the official stable kernel linux-3.4 for Cubieboard2 doesn’t include native SPI driver support, we need to modify the kernel source code, compile and put it into NAND to replace the original uImage, and add the new kernel module files. Additionally, we need to modify system files to fully support full-duplex communication.

1. Flash Debian System to NAND (Windows)

First, download the customized Debian system for Cubieboard2 from: http://dl.cubieboard.org/software/a20-cubieboard/debian/nand/debian-nand.img.gz

Download Phoenixsuit and open the downloaded image through it. With the Cubieboard2 unpowered, hold down the FEL button while connecting it to the computer via an OTG cable. In Device Manager, install the driver for this unrecognized device, specifying the Phoenixsuit installation directory as the driver location, and confirm installation.

Then format and flash the image.

2. Log in to the System

Connect the Cubieboard2 to an Ethernet cable. The Debian system is set to a static IP address 192.168.1.124 by default, which you can access via SSH. You can also log in through a TTL line from UART, change to DHCP mode, then check the IP address from the router before logging in. The router username and password are root/cubieboard.

3. Modify the Kernel (Ubuntu)

The current Linux kernel doesn’t support SPI drivers for Cubieboard2. First, download the linux-3.4 branch kernel source: https://github.com/linux-sunxi/linux-sunxi. Then add SPI support to the kernel (not recommended); for details, see http://blog.csdn.net/u010352603/article/details/51657265#. Instead, directly download a pre-modified version with SPI support from https://github.com/linanwx/cubieboard2-spi-support. After downloading, delete the drivers/spi directory in the kernel directory and place the downloaded spi folder in its place.

4. Compile the Kernel (Ubuntu)

Prepare an arm-linux-gnueabihf-gcc compiler, version not higher than 5.0.

In the kernel directory, using the terminal, enter:

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- sun7i_defconfig

Enter:

gedit .config

Find the following options and change them to:

CONFIG_SPI=y
CONFIG_SUNXI_NAND_PARTITION=y
CONFIG_SUNXI_NAND=y

Enter the following command to start compiling the kernel:

make -j4ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- uImage modules

Since CONFIG_SPI and CONFIG_SUNXI_NAND support are enabled, a series of settings are required. When you encounter spi support, CONFIG_SPI_SUN7I, CONFIG_SUN7I_SPI_NDMA, enter y; in other cases, press enter.

Check if the uImage file exists in the path arch/arm/boot

Enter the following command to generate kernel modules:

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- INSTALL_MOD_PATH=output modules_install

Check if the output/lib/modules/3.4.104 directory has been created.

5. Replace the Kernel (Cubieboard2)

On Cubieboard2, open the terminal and enter:

mkdir /media/nanda
mount /dev/nanda /media/nanda

Copy the uImage to the /media/nanda directory of the Cubieboard2 Debian system, overwriting the original kernel.

Use the tar command to pack the output/lib/modules/3.4.104 directory, transfer this file to Cubieboard2’s /lib/modules directory, then extract it.

6. Check Compilation Success

Restart. If the system enters normally, it was successful.

7. Modify System Files to Enable SPI and Support Full-Duplex SPI (Cubieboard2)

Remount /dev/nanda to the /media/nanda/ directory and enter it.

Enter the following commands:

bin2fex script.bin script.fex
nano script.fex

Find [spi0_para], change the line with “used” to 1, and add the following code:

[spi_devices]
spi_dev_num =1
[spi_board0]
modalias ="spidev"
max_speed_hz =100000
bus_num =0
chip_select =0
mode =0
full_duplex =1
manual_cs =0

Enter the following commands:

fex2bin script.fex script.bin
nano /usr/include/linux/spi/spidev.h

Add the following line above __u16 delay_usecs;:

__u16 interbyte_usecs;

8. Test SPI

Restart Cubieboard2. Check if the spidev device exists in the /dev path. Short-circuit pins 46 and 48; pin diagram available at http://docs.cubieboard.org/products/a10_cubieboard/expansion_ports. Create a new text file with the following code:

/*
 * SPI testing utility (using spidev driver)
 *
 * Copyright (c) 2007 MontaVista Software, Inc.
 * Copyright (c) 2007 Anton Vorontsov <avorontsov@ru.mvista.com>
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License.
 *
 * Cross-compile with cross-gcc -I/path/to/cross-kernel/include
 */

#include <stdint.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <getopt.h>
#include <fcntl.h>
#include <sys/ioctl.h>
#include <linux/types.h>
#include <linux/spi/spidev.h>

#define ARRAY_SIZE(a) (sizeof(a) / sizeof((a)[0]))

static void pabort(const char *s)
{
 perror(s);
 abort();
}

static const char *device = "/dev/spidev0.0";
static uint8_t mode;
static uint8_t bits = 8;
static uint32_t speed = 500000;
static uint16_t delay;

static void transfer(int fd)
{
 int ret;
 uint8_t tx[] = {
 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
 0x40, 0x00, 0x00, 0x00, 0x00, 0x95,
 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
 0xDE, 0xAD, 0xBE, 0xEF, 0xBA, 0xAD,
 0xF0, 0x0D,
 };
 uint8_t rx[ARRAY_SIZE(tx)] = {
 0,
 };
 struct spi_ioc_transfer tr = {
 .tx_buf = (unsigned long)tx,
 .rx_buf = (unsigned long)rx,
 .len = ARRAY_SIZE(tx),
 .delay_usecs = delay,
 .speed_hz = speed,
 .bits_per_word = bits,
 };

 ret = ioctl(fd, SPI_IOC_MESSAGE(1), &tr);
 if (ret == 1)
 pabort("can't send spi message");

 for (ret = 0; ret < ARRAY_SIZE(tx); ret++)
 {
 if (!(ret % 6))
 puts("");
 printf("%.2X ", rx[ret]);
 }
 puts("");
}

static void print_usage(const char *prog)
{
 printf("Usage: %s [-DsbdlHOLC3]\n", prog);
 puts(" -D --device device to use (default /dev/spidev0.0)\n"
 " -s --speed max speed (Hz)\n"
 " -d --delay delay (usec)\n"
 " -b --bpw bits per word \n"
 " -l --loop loopback\n"
 " -H --cpha clock phase\n"
 " -O --cpol clock polarity\n"
 " -L --lsb least significant bit first\n"
 " -C --cs-high chip select active high\n"
 " -3 --3wire SI/SO signals shared\n");
 exit(1);
}

static void parse_opts(int argc, char *argv[])
{
 while (1)
 {
 static const struct option lopts[] = {
 {"device", 1, 0, 'D'},
 {"speed", 1, 0, 's'},
 {"delay", 1, 0, 'd'},
 {"bpw", 1, 0, 'b'},
 {"loop", 0, 0, 'l'},
 {"cpha", 0, 0, 'H'},
 {"cpol", 0, 0, 'O'},
 {"lsb", 0, 0, 'L'},
 {"cs-high", 0, 0, 'C'},
 {"3wire", 0, 0, '3'},
 {"no-cs", 0, 0, 'N'},
 {"ready", 0, 0, 'R'},
 {NULL, 0, 0, 0},
 };
 int c;

 c = getopt_long(argc, argv, "D:s:d:b:lHOLC3NR", lopts, NULL);

 if (c == -1)
 break;

 switch (c)
 {
 case 'D':
 device = optarg;
 break;
 case 's':
 speed = atoi(optarg);
 break;
 case 'd':
 delay = atoi(optarg);
 break;
 case 'b':
 bits = atoi(optarg);
 break;
 case 'l':
 mode |= SPI_LOOP;
 break;
 case 'H':
 mode |= SPI_CPHA;
 break;
 case 'O':
 mode |= SPI_CPOL;
 break;
 case 'L':
 mode |= SPI_LSB_FIRST;
 break;
 case 'C':
 mode |= SPI_CS_HIGH;
 break;
 case '3':
 mode |= SPI_3WIRE;
 break;
 case 'N':
 mode |= SPI_NO_CS;
 break;
 case 'R':
 mode |= SPI_READY;
 break;
 default:
 print_usage(argv[0]);
 break;
 }
 }
}

int main(int argc, char *argv[])
{
 int ret = 0;
 int fd;

 parse_opts(argc, argv);

 fd = open(device, O_RDWR);
 if (fd < 0)
 pabort("can't open device");

 /*
 * spi mode
 */
 ret = ioctl(fd, SPI_IOC_WR_MODE, &mode);
 if (ret == -1)
 pabort("can't set spi mode");

 ret = ioctl(fd, SPI_IOC_RD_MODE, &mode);
 if (ret == -1)
 pabort("can't get spi mode");

 /*
 * bits per word
 */
 ret = ioctl(fd, SPI_IOC_WR_BITS_PER_WORD, &bits);
 if (ret == -1)
 pabort("can't set bits per word");

 ret = ioctl(fd, SPI_IOC_RD_BITS_PER_WORD, &bits);
 if (ret == -1)
 pabort("can't get bits per word");

 /*
 * max speed hz
 */
 ret = ioctl(fd, SPI_IOC_WR_MAX_SPEED_HZ, &speed);
 if (ret == -1)
 pabort("can't set max speed hz");

 ret = ioctl(fd, SPI_IOC_RD_MAX_SPEED_HZ, &speed);
 if (ret == -1)
 pabort("can't get max speed hz");

 printf("spi mode: %d\n", mode);
 printf("bits per word: %d\n", bits);
 printf("max speed: %d Hz (%d KHz)\n", speed, speed / 1000);

 transfer(fd);

 close(fd);

 return ret;
}

Compile this file and run it. If the output looks like the following, the modification was successful:

spi mode: 0
bits per word: 8
max speed: 500000 Hz (500 KHz)
FF FF FF FF FF FF
40 00 00 00 00 95
FF FF FF FF FF FF
FF FF FF FF FF FF
FF FF FF FF FF FF
DE AD BE EF BA AD
F0 0D

ARM GIC-400 Registers

Thu, 30 Jun 2016 20:00:00 +0000

1. Introduction

GIC-400 is used for interrupt control. The A20 chip on the Cubieboard2 uses this controller.

The GIC controller address on Cubieboard2 is 0x01C80000—0x01C87FFF, with a length equal to the GIC-400 register length.

The specific details are shown below.

With 0x01C80000 as the starting address, the following offsets are provided:

Offset Address	Description
0x0000-0x0FFF	Reserved, unused
0x1000-0x1FFF	Distributor, useful
0x2000-0x3FFF	CPU interfaces, useful
0x4000-0x4FFF	Virtual interface control block, for the processor that is performing the accessVirtualization-related, unused
0x5000-0x5FFF	Virtual interface control block, for the processor selected by address bits [11:9]Virtualization-related, unused
0x5000-0x51FF0x5200-0x53FF…0x5E00-0x5FFF	Alias for Processor 0Alias for Processor 1…Alias for Processor 7
0x6000-0x7FFF	Virtual CPU interfacesVirtualization-related, unused

The commonly used parts are the second and third blocks in the diagram, the 0x1000~0x3fff configuration.

2. Distributor

The address of this distributor register block = GIC controller address + 0x1000. Remember this offset when using it. Starting from here, the following registers are available:

Offset	Name	Access	Reset Value	Function
0x000	GICD_CTLR	RW	0x00000000[c]	Distributor Control RegisterWrite 1 to enable the controller, required
0x004	GICD_TYPER	RO	Configuration-dependent[d]	Interrupt Controller Type RegisterUsed to check the total number of interrupt lines
0x008	GICD_IIDR	RO	0x0200143B	Distributor Implementer Identification Register, GICD_IIDRContains version information, not particularly useful
0x080-0x0BC	GICD_IGROUPRn	RW	0x00000000	Interrupt Group Registers[e]A bitmap that controls whether interrupts belong to Group A or B
0x100	GICD_ISENABLERn	RW[f]	SGIs and PPIs:0x0000FFFF[g]	Interrupt Set-Enable RegistersA bitmap for enabling individual interrupts, write 1 to enable. Useful
0x104-0x13C			SPIs: 0x00000000
0x180	GICD_ICENABLERn	RW[f]	0x0000FFFF[g]	Interrupt Clear-Enable RegistersSimilar to the previous register, but opposite function, write 1 to disable
0x184-0x1BC			0x00000000
0x200-0x23C	GICD_ISPENDRn	RW	0x00000000	Interrupt Set-Pending RegistersPending bitmap, write 1 to enter pending state
0x280-0x2BC	GICD_ICPENDRn	RW	0x00000000	Interrupt Clear-Pending RegistersSimilar to above, write 1 for opposite effect, prevents pending state
0x300-0x33C	GICD_ISACTIVERn	RW	0x00000000	Interrupt Set-Active RegistersBitmap, write 1 to activate an interrupt
0x380-0x3BC	GICD_ICACTIVERn	RW	0x00000000	Interrupt Clear-Active RegistersWrite 1 to deactivate an interrupt
0x400-0x5FC	GICD_IPRIORITYRn	RW	0x00000000	Interrupt Priority RegistersStores priority levels for each interrupt, 8 bits per interrupt
0x800-0x81C	GICD_ITARGETSRn	RO[h]	-	Interrupt Processor Targets Registers[i]Determines which processor an interrupt should be sent to for handling
0x820-0x9FC		RW	0x00000000
0xC00	GICD_ICFGRn	RO	SGIs: 0xAAAAAAAA	Interrupt Configuration Registers, GICD_ICFGRnConfigures whether interrupts are level-triggered or falling-edge triggered
0xC04		RO	PPIs: 0x55540000
0xC08-0xC7C		RW[j]	SPIs: 0x55555555
0xD00	GICD_PPISR	RO	0x00000000	Private Peripheral Interrupt Status Register, GICD_PPISRGenerally not used
0xD04-0xD3C	GICD_SPISRn	RO	0x00000000	Shared Peripheral Interrupt Status Registers, GICD_SPISRnNot used
0xF00	GICD_SGIR	WO	-	Software Generated Interrupt RegisterControls software interrupts
0xF10-0xF1C	GICD_CPENDSGIRn	RW	0x00000000	SGI Clear-Pending RegistersPending bits for software interrupts
0xF20-0xF2C	GICD_SPENDSGIRn	RW	0x00000000	SGI Set-Pending RegistersSimilar to above, but writing 1 stops pending
0xFD0	GICD_PIDR4	RO	0x00000004	Peripheral ID 4 Register
0xFD4	GICD_PIDR5	RO	0x00000000	Peripheral ID 5 Register
0xFD8	GICD_PIDR6	RO	0x00000000	Peripheral ID 6 Register
0xFDC	GICD_PIDR7	RO	0x00000000	Peripheral ID 7 Register
0xFE0	GICD_PIDR0	RO	0x00000090	Peripheral ID 0 Register
0xFE4	GICD_PIDR1	RO	0x000000B4	Peripheral ID 1 Register
0xFE8	GICD_PIDR2	RO	0x0000002B	Peripheral ID 2 Register
0xFEC	GICD_PIDR3	RO	0x00000000	Peripheral ID 3 Register
0xFF0	GICD_CIDR0	RO	0x0000000D	Component ID 0 Register
0xFF4	GICD_CIDR1	RO	0x000000F0	Component ID 1 Register
0xFF8	GICD_CIDR2	RO	0x00000005	Component ID 2 Register
0xFFC	GICD_CIDR3	RO	0x000000B1	Component ID 3 Register

3. CPU Interface

Starting address offset is 0x2000

Offset	Name	Type	Reset	Description[a]
0x0000	GICC_CTLR	RW	0x00000000	CPU Interface Control RegisterEnable bit. Write 1 to enable
0x0004	GICC_PMR	RW	0x00000000	Interrupt Priority Mask RegisterLimits minimum interrupt priority, interrupts with values above this cannot trigger. Best to set this high
0x0008	GICC_BPR	RW	0x00000002[b]	Binary Point RegisterThe minimum value of the Binary Point Register depends on which security-banked copy is considered:0x2 Secure copy0x3 Non-secure copyPriority grouping
0x000C	GICC_IAR	RO	0x000003FF	Interrupt Acknowledge RegisterRead-only, interrupt ID
0x0010	GICC_EOIR	WO	-	End of Interrupt RegisterWrite to inform the CPU that interrupt processing is complete
0x0014	GICC_RPR	RO	0x000000FF	Running Priority RegisterCurrent interrupt priority
0x0018	GICC_HPPIR	RO	0x000003FF	Highest Priority Pending Interrupt Register [c]Highest priority interrupt number and its pending value
0x001C	GICC_ABPR	RW	0x00000003	Aliased Binary Point Register[d]The minimum value of the Aliased Binary Point Register is 0x3.Alias register
0x0020	GICC_AIAR	RO	0x000003FF	Aliased Interrupt Acknowledge Register[d]Alias register
0x0024	GICC_AEOIR	WO	-	Aliased End of Interrupt Register[d]Alias register
0x0028	GICC_AHPPIR	RO	0x000003FF	Aliased Highest Priority Pending Interrupt Register[c][d]Alias register
0x00D0	GICC_APR0	RW	0x00000000	Active Priority RegisterUsed for saving and restoring
0x00E0	GICC_NSAPR0	RW	0x00000000	Non-Secure Active Priority Register[d]Used for saving and restoring
0x00FC	GICC_IIDR	RO	0x0202143B	CPU Interface Identification Register, GICC_IIDRContains version information
0x1000	GICC_DIR	WO	-	Deactivate Interrupt Register

NRF24L01 + STC15F204EA Wireless Communication Source Code

Sun, 01 Sep 2013 00:55:00 +0000

This code is suitable for small wireless transceiver modules composed of NRF24L01 and STC15F204EA (STC15L204EA), and can also be applied to general 51 microcontrollers communicating with NRF24L01.

A few days ago, I bought a small wireless transceiver module online. The source code provided by the seller was very simple and not particularly useful, so I ported the STM3224l01 program from ATOM, which is more complete and practical.

Since there are almost no pins available to display transmission and reception content, serial communication is used to display the content.

Applicable module image:

Main wiring diagram:

First, for serial display, I collected the following code. The reason it’s so complex is that this chip doesn’t have a serial port function.

First is the H file:

#ifndef _UART_H
#define _UART_H

#define MCU_FREQ 11059200 // Set crystal frequency
#define UART_BUAD 38400
#define ON 1
#define OFF 0

#define UART_TX_PIN P31
#define UART_TX_SET(n) UART_TX_PIN = n
#define UART_TX_HIGH() UART_TX_SET(1)
#define UART_TX_LOW() UART_TX_SET(0)
#define UART_TX_FLIP() UART_TX_PIN = !UART_TX_PIN

#define UART_RX_PIN P30
#define UART_RX_SET(n) UART_RX_PIN = n
#define UART_RX_HIGH() UART_RX_SET(1)
#define UART_RX_LOW() UART_RX_SET(0)
#define UART_RX_FLIP() UART_RX_PIN = !UART_RX_PIN

void uartInit(void);
void uartSendString(char *pS);
void uartSendNum(int num);

#endif

In the file, you need to set the crystal frequency and baud rate correctly, which are MCU_FREQ and UART_BUAD respectively. Once set, they enable correct transmission and reception.

Next is the C file:

#include "stdio.h"
#include "uart.h"
#include "15f204ea.h" // Header file provided by STC official website

typedef unsigned char u8;
typedef unsigned short u16;
typedef unsigned int u32;

typedef unsigned char uchar;
typedef unsigned int uint;
typedef unsigned char BYTE;
static bit bUartFlag;

/******************************************************************************/
// Function name: uartInit
// Input parameters: none
// Output parameters: none
// Function: Set up timer0 operating mode
/******************************************************************************/
void uartInit(void)
{
 /*
 * Set timer0 as 16-bit auto-reload timer
 */
 AUXR |= 0x80; // Timer0 in 1T mode
 TMOD &= 0xF0; // Set timer to mode 0 (16-bit auto-reload)
 TL0 = (0xFFFF - MCU_FREQ / UART_BUAD) & 0xFF; // Set timer initial value
 TH0 = ((0xFFFF - MCU_FREQ / UART_BUAD) >> 8) & 0xFF; // Set timer initial value
 TR0 = 0; // Timer0 starts counting
 ET0 = 0; // Enable timer0 interrupt
 EA = 1;
}

/******************************************************************************/
// Function name: uartSendData
// Input parameters: ucData: byte to send
// Output parameters: none
// Function: Use serial port to send one byte of data
/******************************************************************************/
void uartSendData(u8 ucData)
{
 u8 ucCnt;
 UART_TX_LOW(); // Serial start bit begins
 TR0 = 1; // Timer0 starts counting
 ET0 = 1; // Enable timer0 interrupt
 bUartFlag = ON;
 while(bUartFlag == ON);
 /*
 * Starting from the lowest bit, output data through serial port
 */
 for (ucCnt = 0; ucCnt < 8; ucCnt++) {
 UART_TX_SET(ucData & 0x01);
 ucData >>= 1;
 bUartFlag = ON;
 while(bUartFlag == ON);
 }
 UART_TX_HIGH(); // Send serial stop bit
 bUartFlag = ON;
 while(bUartFlag == ON);
 TR0 = 0; // Timer0 stops counting
 ET0 = 0; // Disable timer0 interrupt
}
/******************************************************************************/
// Function name: uartSendString
// Input parameters: pS: string's address
// Output parameters: none
// Function: Send string through serial output
/******************************************************************************/
void uartSendString(char *pS)
{
 while (*pS) // Check for string end marker
 {
 uartSendData(*pS++); // Send current character
 }
 uartSendData('\r');
 uartSendData('\n');
}

void uartSendNum(int num){ // Use sprintf function to print integer (can also print decimal)
 char temp[14];
 sprintf(temp,"%d",num);
 uartSendString(temp);
}

/******************************************************************************/
// Function name: time0ISR
// Input parameters: none
// Output parameters: none
// Function: Serial port 0 service function
/******************************************************************************/
void time0ISR(void) interrupt 1 using 1
{
 EA = 0;
 bUartFlag = OFF;
 EA = 1;
}

This code calls 15f204ea.h, which is a header file I downloaded from the STC website. For ease of reference, I won’t list the entire file here as it’s too long.

Now that we’ve handled the serial display, let’s move on to the 24l01 program code.

First is the H file:

#ifndef __24L01_H
#define __24L01_H
#include "15f204ea.h"
#define u8 unsigned char
#define u16 unsigned int

typedef unsigned char uchar;
typedef unsigned char uint;

/*nRF24L01 pin definitions*/

sbit CE = P1^4;
sbit CSN = P1^5;
sbit SCK = P1^2;
sbit MOSI = P1^3;
sbit MISO = P1^0;
sbit IRQ = P1^1;

//NRF24L01 register operation commands
#define READ_NRF_REG 0x00 //Read configuration register, lower 5 bits are register address
#define WRITE_NRF_REG 0x20 //Write configuration register, lower 5 bits are register address
#define RD_RX_PLOAD 0x61 //Read RX valid data, 1~32 bytes
#define WR_TX_PLOAD 0xA0 //Write TX valid data, 1~32 bytes
#define FLUSH_TX 0xE1 //Clear TX FIFO register. Used in transmit mode
#define FLUSH_RX 0xE2 //Clear RX FIFO register. Used in receive mode
#define REUSE_TX_PL 0xE3 //Reuse last sent packet, CE is high, data packet is continuously sent
#define NOP 0xFF //No operation, can be used to read status register

//SPI(NRF24L01) register addresses
#define CONFIG 0x00 //Configuration register address; bit0:1 receive mode, 0 transmit mode; bit1:power select; bit2:CRC mode; bit3:CRC enable;

//bit4:interrupt MAX_RT (max retransmit interruption) enable; bit5:interrupt TX_DS enable; bit6:interrupt RX_DR enable
#define EN_AA 0x01 //Enable auto-acknowledge function bit0~5, corresponding to channels 0~5
#define EN_RXADDR 0x02 //Receive address allow, bit0~5, corresponding to channels 0~5
#define SETUP_AW 0x03 //Set address width (all data channels): bit1,0:00,3 bytes; 01,4 bytes; 02,5 bytes;
#define SETUP_RETR 0x04 //Set auto retransmit; bit3:0, auto retransmit counter; bit7:4, auto retransmit delay 250*x+86us
#define RF_CH 0x05 //RF channel, bit6:0, working channel frequency;
#define RF_SETUP 0x06 //RF register; bit3:transmission rate(0:1Mbps,1:2Mbps); bit2:1, transmission power; bit0:low noise amplifier gain
#define STATUS 0x07 //Status register; bit0:TX FIFO full flag; bit3:1, receive data channel number (max:6); bit4, reached max retransmit

//bit5:data send complete interrupt; bit6:receive data interrupt
#define MAX_TX 0x10 //Reached maximum send times interrupt
#define TX_OK 0x20 //TX send complete interrupt
#define RX_OK 0x40 //Received data interrupt
#define OBSERVE_TX 0x08 //Send detection register, bit7:4, data packet loss counter; bit3:0, retransmit counter
#define CD 0x09 //Carrier detection register, bit0, carrier detection;
#define RX_ADDR_P0 0x0A //Data channel 0 receive address, max length 5 bytes, low byte first
#define RX_ADDR_P1 0x0B //Data channel 1 receive address, max length 5 bytes, low byte first
#define RX_ADDR_P2 0x0C //Data channel 2 receive address, lowest byte can be set, high bytes must be equal to RX_ADDR_P1[39:8];
#define RX_ADDR_P3 0x0D //Data channel 3 receive address, lowest byte can be set, high bytes must be equal to RX_ADDR_P1[39:8];
#define RX_ADDR_P4 0x0E //Data channel 4 receive address, lowest byte can be set, high bytes must be equal to RX_ADDR_P1[39:8];
#define RX_ADDR_P5 0x0F //Data channel 5 receive address, lowest byte can be set, high bytes must be equal to RX_ADDR_P1[39:8];

#define TX_ADDR 0x10 //Send address (low byte first), under ShockBurstTM mode, RX_ADDR_P0 equals this address
#define RX_PW_P0 0x11 //Receive data channel 0 valid data width (1~32 bytes), setting to 0 is invalid
#define RX_PW_P1 0x12 //Receive data channel 1 valid data width (1~32 bytes), setting to 0 is invalid
#define RX_PW_P2 0x13 //Receive data channel 2 valid data width (1~32 bytes), setting to 0 is invalid
#define RX_PW_P3 0x14 //Receive data channel 3 valid data width (1~32 bytes), setting to 0 is invalid
#define RX_PW_P4 0x15 //Receive data channel 4 valid data width (1~32 bytes), setting to 0 is invalid
#define RX_PW_P5 0x16 //Receive data channel 5 valid data width (1~32 bytes), setting to 0 is invalid
#define FIFO_STATUS 0x17 //FIFO status register; bit0, RX FIFO register empty flag; bit1, RX FIFO full flag; bit2,3, reserved

//bit4, TX FIFO empty flag; bit5, TX FIFO full flag; bit6,1, cycle send previous data packet. 0, don't cycle;
//////////////////////////////////////////////////////////////////////////////


//24L01 operation lines
#define NRF24L01_CE CE //24L01 chip select signal
#define NRF24L01_CSN CSN //SPI chip select signal
#define NRF24L01_IRQ IRQ //IRQ host data input

//24L01 send and receive data width definitions
#define TX_ADR_WIDTH 5 //5 bytes address width
#define RX_ADR_WIDTH 5 //5 bytes address width
#define TX_PLOAD_WIDTH 32 //20 bytes user data width
#define RX_PLOAD_WIDTH 32 //20 bytes user data width


void NRF24L01_Init(void);//Initialization
void RX_Mode(void);//Configure as receive mode
void TX_Mode(void);//Configure as transmit mode

u8 NRF24L01_Check(void);//Check if 24L01 exists
u8 NRF24L01_TxPacket(u8 *txbuf);//Send a packet of data
u8 NRF24L01_RxPacket(u8 *rxbuf);//Receive a packet of data
#endif

Next is the C file:

#include "15f204ea.h"
#include "24l01.h"
#include "intrins.h"

const u8 TX_ADDRESS[TX_ADR_WIDTH]={0x34,0x43,0x10,0x10,0x03}; //Transmit address
const u8 RX_ADDRESS[RX_ADR_WIDTH]={0x34,0x43,0x10,0x10,0x03}; //Receive address

/******************************************************************************************
/*Delay function
/******************************************************************************************/
void inerDelay_us(unsigned char n)
{
 for(;n>0;n--)
 _nop_();
}
//****************************************************************************************
/*NRF24L01 initialization
//***************************************************************************************/
void NRF24L01_Init(void)
{
 inerDelay_us(100);
 CE=0; // chip enable
 CSN=1; // Spi disable
 SCK=0; //
}
/****************************************************************************************************
/*Function: uint SPI_RW(uint uchar)
/*Function: NRF24L01 SPI write sequence
/****************************************************************************************************/
uint SPI_RW(uint uchar)
{
 uint bit_ctr;
 for(bit_ctr=0;bit_ctr<8;bit_ctr++) // output 8-bit
 {
 MOSI = (uchar & 0x80); // output 'uchar', MSB to MOSI
 uchar = (uchar << 1); // shift next bit into MSB..
 SCK = 1; // Set SCK high..
 uchar |= MISO; // capture current MISO bit
 SCK = 0; // ..then set SCK low again
 }
 return(uchar); // return read uchar
}
/****************************************************************************************************
/*Function: uchar SPI_Read(uchar reg)
/*Function: NRF24L01 SPI sequence
/****************************************************************************************************/
uchar NRF24L01_Read_Reg(uchar reg)
{
 uchar reg_val;

 CSN = 0; // CSN low, initialize SPI communication...
 SPI_RW(reg); // Select register to read from..
 reg_val = SPI_RW(0); // ..then read registervalue
 CSN = 1; // CSN high, terminate SPI communication

 return(reg_val); // return register value
}


/****************************************************************************************************/
/*Function: NRF24L01 read and write register function
/****************************************************************************************************/
uint NRF24L01_Write_Reg(uchar reg, uchar value)
{
 uint status;

 CSN = 0; // CSN low, init SPI transaction
 status = SPI_RW(reg); // select register
 SPI_RW(value); // ..and write value to it..
 CSN = 1; // CSN high again

 return(status); // return nRF24L01 status uchar
}

/****************************************************************************************************/
/*Function: uint SPI_Read_Buf(uchar reg, uchar *pBuf, uchar uchars)
/*Function: Used to read data, reg: register address, pBuf: destination data address, uchars: number of bytes to read
/****************************************************************************************************/
uint NRF24L01_Read_Buf(uchar reg, uchar *pBuf, uchar uchars)
{
 uint status,uchar_ctr;

 CSN = 0; // Set CSN low, init SPI tranaction
 status = SPI_RW(reg); // Select register to write to and read status uchar

 for(uchar_ctr=0;uchar_ctr<uchars;uchar_ctr++)
 pBuf[uchar_ctr] = SPI_RW(0); //

 CSN = 1;

 return(status); // return nRF24L01 status uchar
}
/*********************************************************************************************************
/*Function: uint SPI_Write_Buf(uchar reg, uchar *pBuf, uchar uchars)
/*Function: Used to write data: register address, pBuf: data to write, uchars: number of bytes to write
/*********************************************************************************************************/
uint NRF24L01_Write_Buf(uchar reg, uchar *pBuf, uchar uchars)
{
 uint status,uchar_ctr;

 CSN = 0; //SPI enable
 status = SPI_RW(reg);
 for(uchar_ctr=0; uchar_ctr<uchars; uchar_ctr++) //
 SPI_RW(*pBuf++);
 CSN = 1; //Close SPI
 return(status); //
}

//Check if 24L01 exists
//Return value: 0, success; 1, failure
u8 NRF24L01_Check(void)
{
 u8 buf[5]={0XA5,0XA5,0XA5,0XA5,0XA5};
 u8 i;
 NRF24L01_Write_Buf(WRITE_NRF_REG+TX_ADDR,buf,5);//Write 5 bytes address
 NRF24L01_Read_Buf(TX_ADDR,buf,5); //Read the written address
 for(i=0;i<5;i++) if(buf[i]!=0XA5)break;
 if(i!=5)return 1;//24L01 detection error
 return 0; //24L01 detected
}
u8 NRF24L01_TxPacket(u8 *txbuf)
{
 u8 sta;
 NRF24L01_CE=0;
 NRF24L01_Write_Buf(WR_TX_PLOAD,txbuf,TX_PLOAD_WIDTH);//Write data to TX BUF, 32 bytes
 NRF24L01_CE=1;//Start transmission
 while(NRF24L01_IRQ!=0);//Wait for transmission to complete
 sta=NRF24L01_Read_Reg(STATUS); //Read status register value
 NRF24L01_Write_Reg(WRITE_NRF_REG+STATUS,sta); //Clear TX_DS or MAX_RT interrupt flag
 if(sta&MAX_TX)//Maximum retransmission reached
 {
 NRF24L01_Write_Reg(FLUSH_TX,0xff);//Clear TX FIFO register
 return MAX_TX;
 }
 if(sta&TX_OK)//Transmission completed
 {
 return TX_OK;
 }
 return 0xff;//Other reasons for transmission failure
}

//Start NRF24L01 to send data once
//txbuf: data source address to be sent
//Return value: 0, reception complete; others, error code
u8 NRF24L01_RxPacket(u8 *rxbuf)
{
 u8 sta;
 sta=NRF24L01_Read_Reg(STATUS); //Read status register value
 NRF24L01_Write_Reg(WRITE_NRF_REG+STATUS,sta); //Clear TX_DS or MAX_RT interrupt flag
 if(sta&RX_OK)//Data received
 {
 NRF24L01_Read_Buf(RD_RX_PLOAD,rxbuf,RX_PLOAD_WIDTH);//Read data
 NRF24L01_Write_Reg(FLUSH_RX,0xff);//Clear RX FIFO register
 return 0;
 }
 return 1;//No data received
}
//This function initializes NRF24L01 to RX mode
//Set RX address, write RX data width, select RF channel, baud rate and LNA HCURR
//When CE becomes high, it enters RX mode and can receive data
void RX_Mode(void)
{
 NRF24L01_CE=0;
 NRF24L01_Write_Buf(WRITE_NRF_REG+RX_ADDR_P0,(u8*)RX_ADDRESS,RX_ADR_WIDTH);//Write RX node address
 NRF24L01_Write_Reg(WRITE_NRF_REG+EN_AA,0x01); //Enable channel 0 auto-acknowledge
 NRF24L01_Write_Reg(WRITE_NRF_REG+EN_RXADDR,0x01);//Enable channel 0 receive address
 NRF24L01_Write_Reg(WRITE_NRF_REG+RF_CH,40); //Set RF communication frequency
 NRF24L01_Write_Reg(WRITE_NRF_REG+RX_PW_P0,RX_PLOAD_WIDTH);//Select channel 0 valid data width
 NRF24L01_Write_Reg(WRITE_NRF_REG+RF_SETUP,0x0f);//Set TX transmission parameters, 0db gain, 2Mbps, low noise gain enabled
 NRF24L01_Write_Reg(WRITE_NRF_REG+CONFIG, 0x0f);//Configure basic working mode parameters: PWR_UP, EN_CRC, 16BIT_CRC, receive mode
 NRF24L01_CE = 1; //CE is high, enter receive mode
}
//This function initializes NRF24L01 to TX mode
//Set TX address, write TX data width, set RX auto-acknowledge address, fill TX send data, select RF channel, baud rate and LNA HCURR
//PWR_UP, CRC enable
//When CE becomes high, it enters RX mode and can receive data
//CE high for more than 10us will start transmission
void TX_Mode(void)
{
 NRF24L01_CE=0;
 NRF24L01_Write_Buf(WRITE_NRF_REG+TX_ADDR,(u8*)TX_ADDRESS,TX_ADR_WIDTH);//Write TX node address
 NRF24L01_Write_Buf(WRITE_NRF_REG+RX_ADDR_P0,(u8*)RX_ADDRESS,RX_ADR_WIDTH); //Set TX node address, mainly to enable ACK
 NRF24L01_Write_Reg(WRITE_NRF_REG+EN_AA,0x01); //Enable channel 0 auto-acknowledge
 NRF24L01_Write_Reg(WRITE_NRF_REG+EN_RXADDR,0x01); //Enable channel 0 receive address
 NRF24L01_Write_Reg(WRITE_NRF_REG+SETUP_RETR,0x1a);//Set auto retransmit interval time: 500us + 86us; maximum auto retransmit times: 10
 NRF24L01_Write_Reg(WRITE_NRF_REG+RF_CH,40); //Set RF channel to 40
 NRF24L01_Write_Reg(WRITE_NRF_REG+RF_SETUP,0x0f); //Set TX transmission parameters, 0db gain, 2Mbps, low noise gain enabled
 NRF24L01_Write_Reg(WRITE_NRF_REG+CONFIG,0x0e); //Configure basic working mode parameters: PWR_UP, EN_CRC, 16BIT_CRC, receive mode, enable all interrupts
 NRF24L01_CE=1;//CE is high, start transmission after 10us
 inerDelay_us(20);
}

Next is the main program debugging section:

#include "15f204ea.h"
#include "24l01.h"
#include "uart.h"
#include "intrins.h"

void delay500ms(void) //Error -0.000000000063us
{
 unsigned char a,b,c;
 for(c=212;c>0;c--)
 for(b=160;b>0;b--)
 for(a=80;a>0;a--);
 _nop_(); //if Keil, require use intrins.h
}

void delay100us(void) //Error -0.083188657407us
{
 unsigned char a,b;
 for(b=58;b>0;b--)
 for(a=8;a>0;a--);
}

void main(){
 u8 tmp_buf[33];
 u8 key,mode;
 u16 t=0;
 delay500ms();
 uartInit();
 uartSendString("Test");
 uartSendNum(1234);
 NRF24L01_Init();
 while(NRF24L01_Check())//Cannot detect 24L01
 {
 uartSendString("Initialization failed");
 delay500ms();
 uartSendString("Please check");
 delay500ms();
 }
 uartSendString("Starting");
 if(0){
 RX_Mode();
 uartSendString("Receive mode");
 while(1){
 if(NRF24L01_RxPacket(tmp_buf)==0){
 tmp_buf[32]=0;//Add string terminator
 uartSendString(tmp_buf);
 }
 else delay100us();
 }
 }
 else{
 TX_Mode();
 uartSendString("Transmit mode");
 mode=' ';//Start from space character
 while(1){
 if(NRF24L01_TxPacket(tmp_buf)==TX_OK)
 {
 uartSendString(tmp_buf);
 key=mode;
 for(t=0;t<32;t++)
 {
 key++;
 if(key>('~'))key=' ';
 tmp_buf[t]=key;
 }
 mode++;
 if(mode>'~')mode=' ';
 tmp_buf[32]=0;//Add terminator
 }
 else{
 uartSendString("Send failed");
 };
 delay500ms();
 }
 }
}

Set the crystal to 11.0594MHZ, then write the program.

First, connect the serial port, then power on. After half a second, the program will send serial data “Test” and the number “1234” to the computer. Note that the serial working frequency is 38400.

If you can receive serial data correctly, then everything is fine; otherwise, there’s an issue with the frequency settings or the data cable.

The program will then perform a check. Upon success, it returns “Starting”; upon failure, it displays “Initialization failed” and “Please check”. Failure is either because the nrf24l01 is damaged or the pins are connected incorrectly.

Depending on the value in the if statement’s parentheses (0 or 1), the program will enter either receive mode or transmit mode.

In receive mode, the program will send “Receive mode” through the serial port. If it receives data, it will return the data content.

In transmit mode, the program will send a code (a sequence of changing characters) to the specified address. If transmission is successful, it returns the length of the data sent and the data itself. When sending fails, it will display “Send failed”.

If sending fails, it might be because the receiver doesn’t exist. In this case, NRF24L01_TxPacket returns 10 (maximum retransmission attempts) instead of 32 (data bit count), so an error is reported.

It could also be due to other reasons, in which case NRF24L01 returns 0xff, indicating other causes of failure. In this situation, the problem is unclear and needs careful investigation.

By writing the transmit program to one chip and the receive program to another, you can see the effect ^_^

Thus, NRF24L01 and STC15F204EA (STC15L204EA) communication is achieved.

Posts on Nansen Li's Blog 李楠森的博客

Why I Stopped Subscribing to Claude

What makes Claude so good?

Why cancel the subscription?

Closing thoughts

Openclaw Was Too Buggy, So I Built My Own Bot: Nagobot

Why I Stopped Subscribing to ChatGPT

Reasons for Stopping the Subscription

Current Alternatives

Psychology Book Reviews and Recommendations

Recommended Books

Self-Compassion: The Proven Power of Being Kind to Yourself

Nonviolent Communication

The Courage to Be Disliked

Man’s Search for Meaning

Difficult Conversations

The Road Less Traveled

Complex PTSD: From Surviving to Thriving

Counseling Psychology (Chinese only)

The Five Love Languages

Attached

Interview Experience Summary

Google Internship Position

Apple Testing Position

Apple Full Stack Position

Meta SSE Position

Amazon SWE Position

PTC SWE Position

eBay SWE

Workday

Arista

Experience Summary

Learning Python uv Package

What exactly is uv?

What can uv do?

Dublin City University MSc in Computing: Secure Software Engineering Exam Cookbook

Exam Overview

Semester 1 Modules

Semester 2 Modules

Cryptography and Number Theory

System Software/BlockChain

Secure Programming

Professional & Research Practice

Formal Programming

Concurrent Programming

Software Process Quality

April 2025 Tech Notes 2

Github Copilot

Dia Browser

Release of GPT o3 / o4-mini / o4-mini-high / 4.1 / 4.1 mini / 4.1 nano

Yuanbao Integrates with WeChat Contacts

OpenAI Free Token Giveaway

Other Github Daily/Weekly Trending Projects

April 2025 Tech Notes

Introduction

Migrating Personal Blog from Hexo to Hugo

Mysterious Optimus Alpha Model Appears on https://openrouter.ai/ and Dominates Rankings

OpenRouter.ai

nanobrowser

markitdown

PDF search

Incomplete Guide to Studying in Ireland

Background

Before Departure

Immigration

After Arriving in Ireland

Payments

Housing

Home and Bedding

Pharmacies

Dining

Public Transit

Online Shopping

Clothing

Domestic Travel

Visas

Job Hunting

Contact

Summary of LLM Tools Usage Experience

ChatGPT o4-mini

1. What does `dp[i]` represent?

2. Why compare `nums[i] + dp[i-1]` and `nums[i]`?