The 101 of ELF Binaries on Linux: Understanding and Analysis
Executable and Linkable Format 可執行檔與可連結格式
An extensive dive into ELF files: for security incident response, development, and better understanding
廣泛的深入 ELF 檔案: 為了資安事件響應、程式開發與更好的理解
We often don’t realize the craftsmanship of others, as we conceive them as normal. One of these things is the usage of common tools, like ps and ls.Even though the commands might be perceived as simple, under the hood there is more to it: ELF binaries. Let’s have an introduction into the world of this common file format for Linux and UNIX-based systems.
我們通常不理解其他的技術，把他們想的跟喝水一樣普通。其一便是一般工具的使用方法，像是 ps 與 ls。甚至該指令可能是被認為簡單的，其實還有更多隱藏在 ELF 二進位格式中。讓我們向世界介紹這個在 Linux 與 基於 UNIX 系統上普遍可見的檔案格式。
Why learn the details of ELF ? 為什麼要了解 ELF 的細節?
Before diving into more technical details, it might be good to explain why understanding of the ELF format is useful. As a starter, it helps to learn the inner workings of our operating system. When somehing goes wrong, we might better understand what happened (or why). Then there is the value in being able to research ELF files, e.g. after a security breach (incident response, malware research, forencis). Last but not least, for a better understanding while developing. Even if you program in a high-level language like Golang, you still might benefit from knowing what happens behind the scenes.
在更深入到技術細節之前，最好先解釋一下為什麼了解 ELF 格式是有用的，當作開頭，可以幫助學習我們作業系統裡面的運作原理。當某些東西出錯了，我們或許較能知道發生了什麼事情(或是為什麼)。這就是研究 ELF 檔案的價值所在，例如在一波資安攻擊之後(事件響應，惡意軟體研究，鑑識)。最後而同樣重要的是，能在開發時有更好的理解力。甚至當你在使用高級語言如 Golang，你仍可能受益於知曉該場景背後發生了些什麼事情。
From source to process 從原始碼到行程
So whatever operating system we run, it needs to translate common functions to the language of the CPU, also known as machine code. A function could be something basic like opening a file on disk or showing something on the screen. Instead of talking directly to the CPU, we use a programming language, using internal functions. A compiler then translates these functions into object code. This object code is then linked into a full program, by using a linked tool. The result is a binary file, which then can be executed on that specific platform and CPU type.
所以無論我們運行任何作業系統，它需要將一般功能翻譯成 CPU 的語言，也被稱為機器語言。一個函式可被拆分成基本的指令像是在硬碟上開啟一個檔案或是顯示一些東西在螢幕上。儘管可以直接對 CPU 下達指令，我們使用一種程式語言，使用內部的函式。然後使用編譯器翻譯這些函式成為目的碼(編按: object code)。藉由使用連結工具這個物件碼將被連結成一個完整的程式。結果是一個可以執行在特定平台與 CPU 架構下的二進制檔案。
Before you start 在開始之前
This blog post will share a lot of commands. Don’t run them on production systems. Better do it on a test machine. If you like to test commands, copy an existing binary and use that. Additionally, we have provided a small C program, which can you compile. After all, trying out is the best way to learn and compare results.
這個部落格文章將分享一堆指令。請勿在正式環境執行。最好是在測試環境上。如果你喜歡測試指令，複製一個已存在的二進制檔並使用他們。此外，我們提共了一個小型的 C 語言程式，你可以自行嘗試編譯。總之，嘗試是最好的學習方式與比對結果。
Not Just Executables 並非只是可執行檔
A common misconception is that ELF files are just for executables. We already have seen they can be used for partial pieces (object code). Another example includes shared libraries, and even core dumps (those core or a.out files). ELF is also used for the kernel and kernel moduels on Linux machines.
一種常見的誤解是 ELF 檔案只會是可執行檔。我們已經看到他們可以被用來部份利用(object code). 另外一個例子包含 shared libries，而甚至 core dump (core 或 a.out 檔)。在 Linux 機器上 ELF 也被用在核心與核心模組。
Structure 結構
Due to the extensible design of ELF files, the structure differs per file. An ELF file consists of:
因為可擴充設計的緣故，每個檔案的結構不同。一個 ELF 檔案的組成有：
- ELF header (ELF 標頭)
- File data (檔案資料)
With the readelf command we can look at the structure of a file and it will look something like this:
可以使用 readelf 指令我們可以看到檔案的結構可以看到如下的畫面：
ELF header / ELF 標頭
As can be seen in this screenshot, the ELF header starts with some magic. While this might look fuzzy at first, it is a partial representation of the header data itself. The first 4 hexdecimal pieces define that this is an ELF file (45=E,4c=L,46=F), prefixed with the 7f value.
可由截圖看到，ELF 標頭從一些 magic 開始。或許一開始看起來會有點混論，那是一個表示標頭資料的部份。頭四個 16 進制部份定義了這是一個 ELF 檔(45=E,4c=L,46=F)，並由 7f 做為前置標記。
This ELF header is mandatory and ensures that data is correctly interpreted during linking or execution. To better understand the inner working of an ELF file, it is useful to know the file used. It is actually easier than it looks.
這個 ELF 標頭是強制的並且可以確保資料正確的在連結或是執行階段被直譯。為了更好的了解 ELF 內部運作，了解使用到的檔案是有幫助的，實際上那比看上去的簡單。
Class / 類別
After the ELF type declaration, there is a Class field defined. This value determines if the file is meant for a 32 (=1) or 64 (=2) bit architecture. The magic shows a 2, which is displayed by the readelf command as an ELF64 file. In other words, an ELF file using 64 bit architecture. Not surprising, as this particular machine contains a modern CPU.
在 ELF 類型宣告之後，有定義一個類別的欄位。這個值決定這個檔案是 32 或是 64 位元架構。magic 顯示一個 2，被 readelf 指令顯示為 ELF64 的檔案。換句話說，一個 ELF 檔案使用 64 位元架構。不意外的，這個特定機器包含一個現代 CPU。
Data 資料
Next there is a data field. It knows two options: 01 for LSB (Least Significant Bit), also known as little-endian. The there is the value 02, for MSB (Most Significant Bit, big-endian). This particular value helps to interpret the remaining objects correctly within the file. This is important, as different types of processors deal differently with the incoming instructions and data structures. In this case LSB is used, which is common for AMD64 type processors.
接下來有一個資料欄位。它有兩個選項： 01 表示 LSB(Least Significant Bit)，也被稱為 little-endian。這個特定值幫助正確的直譯在檔案中剩下的物件。這是非常重要的，不同的處理器使用不同的方式處理接收到的指令與資料結構。這個案例使用 LSB，是普遍 AMD64 處理器的型態。
The effect of LSB becomes visible when using hexdump on a binary file like /bin/ps.
當對一個二進位檔如 /bin/ps 使用 hexdump 則效果是顯而易見的。
$ hexdump -n 16 /bin/ps
We can see that the value pairs are different, which is caused by the right interpretation of the byte order.
我們可以看到一對值是不同的，因為是由右開始解釋的位元組順序。
Version 版本
Next in line is another “01” in the magic, which is the version number. Currently, there is only 1 version type: currently, which is the value “01”. So nothing interesting to remember.
下一個要講的是在 magic 裡面的另外一個 “01”，表示的是版本號。目前只有一種版本類型: 就是 “01” 這個值。所以沒啥有趣的東東好記的。
OS/ABI and ABI version 作業系統/應用二進為介面與應用二進為介面版本
Each operating system has a big overlap in common functions. In addition, each of them has specific ones, or at least minor differences between them. To ensure the right functions are used, an application binary interface (ABI), is defined. This way the operating system and applications both know what to expect and functions are correctly forwarded. These two fields describe what ABI is used and the related version. For Linux systems this is the SystemV.
每個作業系統在常見的函式上常有重疊。另外，他們每一個都有一個特例，或是之間存在微小的差異。為了確保使用正確的函式，才定義了一個應用二進位介面 (ABI)。如此一來作業系統與應用程式便可以預期甚麼樣的事情及功能會被傳遞過來。這兩個欄位描述了 ABI 被使用及其相關的版本。對 Linux 系統來說就是 SystemV。
Machine 機器
In the header we can also find the expected machine type(AMD64)
在標頭我們也可以找到預期的機器類型 (AMD64)
Type 型態
The type field tells us what the purpose of the file is.
類型欄位這個檔案的用途為何
Usually it is:
它通常是:
- DYN(Shared object file), for libraries
- EXEC (Executable file), for binaries
-
REL(Relocatable file), before linked into an executable file
- DYN (共享物件檔)，函式庫
- EXEC (可執行檔)，二進位檔案
- REL (可重定位檔)，連結進一個執行檔前的檔案
Machine / 機器
While some of the fields could already be displayed via the magic value of the readelf output, there is more. For example for waht specific processor type the file is. Using hexdump we can see the real values.
某些欄位可以透過 readelf 輸出顯示其魔術值，但其實還有更多。舉例來說: 該檔案的指定處理器類型是哪一個。使用 hexdump 我們可以看到真實的資料。
7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 .ELF………… 02 00 3e 00 01 00 00 00 a8 2b 40 00 00 00 00 00<EMAIL_ADDRESS>40 00 00 00 00 00 00 00 30 65 01 00 00 00 00 00 @…….0e…… 00 00 00 00 40 00 38 00 09 00 40 00 1c 00 1b 00 …<EMAIL_ADDRESS>(Output created with hexdump -C -n 64 /bin/ps) (透過 hexdump -C -n 64 /bin/ps 製造的輸出)
The highlighted field above is what defines the machine type. The value 3e is 62 in decimal, which equals to AMD64. To get an idea of all machine types, have a look at this ELF header file.
上面粗體的欄位就是定義了機器類型。3e 就是十進位的 62，表示的是 AMD64。若想知道更多機器類型，瞧一瞧這個吧ELF header file。
With all these fields clarified, it is time to look at where the real magic happens and move into the next headers!
透過這幾個欄位的闡述，該是時候看看真正魔法並進到後面的標頭了!
File data / 檔案資料
Besides the ELF header, ELF files consist of:
除了 ELF 標頭之外，ELF 檔案的組成為:
- Program Headers or Segments(9) (程式標頭 或 區段)
- Section Headers or Sections(28) (節區標頭 或 節區)
- Data (資料)
Before we dive into these headers, it is good to know that ELF has two complementary “views”. One for used for the linker to allow execution (segments), one for categorizing instructins and data (sections). So depending on the goal, the related header types are used. Let’s start with program headers, which we find on ELF binaries.
在我們深入這些標頭之前，了解到 ELF 有兩個互補的”視圖”是再好不過了。一個是為了可以讓連結器執行 (區段)，一個用來分類指令(編按: 原文typo，應為 instruction)與資料 (節區)，基於這個目標，相關的標頭類型將被使用到。讓我們從 ELF 二進位檔中找到的程式標頭們開始吧。
Program headers / 程式標頭
An ELF file consists of zero or more segments, and describe how to create a process/memory image for runtime execution. When the kernel sees these segments, it uses them to map them into virtual address space, using the mmap(2) system call. In other words, it converts predefined instructions into a memory image. If your ELF file is a normal binary, it requires these program headrs, otherwise it won’t run. And it uses these headers, with the underlying data structure, to form a process. This process is similar for shared libraries.
一個 ELF 檔案由零個或更多區段組成，並且描述如何創造一個執行時期的行程/記憶體映像。當核心看到這些區段，它使用系統呼叫 mmap(2) 將他們映射到虛擬位址空間，換句話來說，它轉換預先定義的指令到記憶體映像中。如果你的 ELF 檔案是一個普通的二進位檔，它將需求這些程式標頭，否則無法運行。並且它使用這些標頭，與底層資料結構來形成一個程序。這個程序與共享函式庫類似。
An overview of program heders in an ELF binary
一個 ELF 二進位檔中的程式標頭概觀
We see in this example that there are 9 program headers. When looking at it for the first time, it hard to understand what happens here. So let’s go into a few details.
我們看到這個範例有 9 個程式標頭。當第一次看到的時候，它(編按: 原文應該少了 is)讓人難以理解這在搞啥。所以讓我進入到一點細節部分。
GNU_EH_FRAME
This is a sorted queue, used by the GNU C (gcc), to store exeception handlers. So when something goes wrong, it can use this part to deal correctly with it.
被 GNU C (gcc) 所使用的是一個排序過的佇列，來存放意外處理器。所以當某東西出錯了，它可以使用這個部分來處理修正錯誤。
GNU_STACK
This header is used to sotre stack information. The stack is a buffer, or scratch place, where items are stored, like local variables. This will occur with LIFO(Last In, First Out), similar to putting boxes on top of each other. When a process function is started a block is reserved. When the funtion is finished, it will be marked as free again. Now the interesting part is that a stack shouldn’t be executable, as this might introduce security vulnerabilities. By manipulation of memory, one could refer to this executable stack and run intended instructions.
這個標頭被用來存放堆疊資訊。堆疊是一個緩衝區，或是一個存放龐雜物品的地方，像是區域變數。這裡的運作是 LIFO(後進先出)，類似於把一個盒子放置於其他盒子之上。當一個程序函式開始執行，就會有一個區塊被保留。當函式結束該區塊將再次被標記為自由。如今有趣的部分是，堆疊不該為可執行，當作是介紹安全弱點。透過操作記憶體，其他地方可能參考到這個可執行的堆疊並故意運行一些指令。
If the GNU_STACK segment is not available, then usasally an executable stack is used. The scanelf and execstack tools are two examples to show the stack details.
如果 GNU_STACK 區段並非可用的，則通常可執行的堆疊會被使用。
scanelf
與execstack
是兩個可以呈現堆疊詳細資訊的工具。
# scanelf -e /bin/ps
TYPE STK/REL/PTL FILE
ET_EXEC RW- R-- RW- /bin/ps
# execstack -q /bin/ps
- /bin/ps
Commands to see program headers
可以看程式標頭的指令
- dumpelf (pax-utils)
- elfls -S /bin/ps
- eu-readelf -program-headers /bin/ps
Sections / 節區
Section headers / 節區標頭
The section headres define all the sections in the file. As said, this “view” is used for linking and relocation.
節區標頭定義了在檔案中的所有節區。就像前面提到的，這個”視圖”視被用來連結與重定位的。
Sections can be found in an ELF binary after the GNU C compiler transformed C code into assembly, followed by the GNU assembler, which creates objects of it.
節區可以在 GNU C 編譯器將 C 程式碼轉換成組合語言且被後面的 GNU 組譯器，在 ELF 二進位檔中發現，
As the image above shows, a segment can have 0 or more sections. For executable files there are four main sections: .text, .data, .rodata, and .bss. Each of these sections are loaded with different access rights, which can be seen with readelf -S.
如上面的圖所示，一個區段可以擁有 0 或更多個節區。以可執行檔來說有主要四個節區: .text、.data、.rodata與 .bss。這每一個節區都被載入且擁有不同的存取權限，可以用 readelf -S 來查看。
.text / 程式碼區段
Contains executable code. It will be packed into a segment with read and execute access rights. It is only loaded once, as the contents will not change. This can be seen with the objdump utility.
包含可執行程式碼。程式碼將被包裝到一個區段中，有可讀可執行的權限並且它只會被載入一次，因為程式碼內容不會改變。可以使用 objdump 工具查看。
12 .text 0000a3e9<PHONE_NUMBER>402120<PHONE_NUMBER>402120 00002120 24
CONTENTS, ALLOC, LOAD, **READONLY, CODE
.data / 資料區段
Initialized data, with read/write access rights.
初始化後的資料，有可讀可寫的權限。
.rodata / 唯讀資料區段
Initialized data, with read access rights only (=A).
初始化後的資料，只有可讀的權限。
.bss / 位初始化資料區段
Uninitialized data, with read/write access rights (=WA)
未初始化的資料，有可讀可寫的權限
[24] .data PROGBITS<PHONE_NUMBER>6172e0 000172e0
0000000000000100<PHONE_NUMBER>000000 WA 0 0 8
[25] .bss NOBITS<PHONE_NUMBER>6173e0 000173e0
0000000000021110<PHONE_NUMBER>000000 WA 0 0 32
Commands to see section and headers
用來看節區與標頭的指令
- dumpelf
- elfls -p /bin/ps
- eu-readelf -section-headers /bin/ps
- rreadelf -S /bin/ps
- objudmp -h /bin/ps
Section groups / 節區群組
Some sections can be grouped, as they form a whole, or in other words be a dependency. Newer linkers support this functionality. Still this is not common to find that often:
某一些節區可以被群組起來成為一個整體，換句話說成為從屬關係，新一點的連結器才有支援這個功能。這仍然不是這麼常見:
# readelf -g /bin/ps
There are no section groups in this file.
While this might not be looking very interesting, it shows a clear benefit of reasearching the ELF toolkits which are available, for analysis. For this reason, an overview of tools and their primary goal have been included at the end of this article.
這看起或許並不有趣，可用的 ELF 工具組在研究與分析方面有顯著的益處，基於這個原因，一些工具與他們主要的用途概述將會在文末附上。
Static VS Dynamic / 靜態 VS 動態
Another thing to mention before closing an introduction on the subject of ELF is static and dynamic binaries. For optimization purposes we often see that binaries are “dynamic”, which means it needs external components to run correctly. Often these external components are normal libraries, which contain common functions, like opening files or createing a network socket. Static binaries on the other hand have all libraries included, which make them bigger, yet more portable (e.g. using them on another system).
另外一個在結束介紹 ELF 是靜態和動態二進位的主題前要提及的，為了最佳化的目的，我們通常將二進位檔視為”動態”，代表的是它需要額外的部件來正確運行。
If you want to check if a file is statically or dynamically compiled, use the file command. If it shows something like:
如果你要確認一個檔案是否為動態或是靜態編譯的，使用
file
命令。如果它顯示如:
$ file /bin/ps
/bin/ps: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=2053194ca4ee8754c695f5a7a7cff2fb8fdd297e, stripped
To determine what external libraries are being used, simply use the ldd on the same binary:
要確認額外函式庫有那些被使用到，簡單的使用
ldd
在同一個二進位檔上:
$ ldd /bin/ps
linux-vdso.so.1 => (0x00007ffe5ef0d000)
libprocps.so.3 => /lib/x86_64-linux-gnu/libprocps.so.3 (0x00007f8959711000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f895934c000)
/lib64/ld-linux-x86-64.so.2 (0x00007f8959935000)
Tip: To see underlying dependencie, it might be better to use the lddtree utility instead.
小技巧: 要看到底層依賴關係，使用 lddtree 工具來代替。
What Did We Learn ? / 我們學到了什麼 ?
ELF files are for execution, or for linking. Depending on one of these goals, it contains the required segemnts or sections. Segments are viewed by the kernel and mapped into memory (using mmap). Sections are viewed by the linker to create executable code or shared objects.
ELF 檔案是用來執行或是連結的。基於其中一個目標，而包含要求的區段或是節區。區段是核心的觀點並映射到記憶體中(使用 mmap)。節區是連結器的觀點用來創造可執行程式碼或是分享物件。
The ELF file type is very flexible and provides support for multiple CPU types, machine architectures, and operating systems. It is also very extensible: each file is differently constructed, depending on the required parts.
ELF 檔案類型是非常彈性且提供許多支援的 CPU 類型、機器架構與作業系統。同時它具有可擴展性: 基於不同的要求，每個檔案有不同的構造。
Headers form an important part of the file, describing exactly the contents of and ELF file. By using the right tools, you can gain a basic understanding on the purpose of the file. From there on, you can further “interrogate” the binaries by determining the related functions it uses, or strings stored in the file. A great start for those who are into malware research, or want to know better how processes behave (or not behave!).
標頭形成一個檔案重要的部分，精確的描述 ELF 檔案的內容。透過使用對的工具你可以獲得對檔案的目的有一個基本的了解。從現在開始，你可以進一步的”質問”二進位檔取決於，它使用的相關函式或是儲存在檔案你的字串。對於惡意軟體研究有興趣的人是一個好的開始，或是想要對程序行為有更好的了解 (或是程序不乖!)。(編按: behave 有乖巧的意味，這裡作者玩了個雙關)
Packages / 套裝軟體
Most Linux systems will already have the the binutils package installed. Other packages might help with showing much more details. Having the right toolkit might simplify your work, especially when doing analysis or learning more about ELF files. So we have collected a list of packages and the related utilities in it.
多數的 Linux 系統將已經安裝了
binutils
套裝軟體(編按: 原文多打一個 the)。其他套裝軟體對顯示更多詳細資訊可能會有幫助，擁有隊的工具箱可以簡化你做的工。特別是在分析或是學習更多有關 ELF 檔案的時候。所以我們蒐集了一個套裝軟體清單與其相關的工具。
elfutils
- /usr/bin/eu-addr2line
- /usr/bin/eu-ar – alternative to ar, to create, manipulate archive files
- /usr/bin/eu-elfcmp
- /usr/bin/eu-elflint – compliance check against gABI and psABI specifications
- /usr/bin/eu-findtextrel – find text relocations
- /usr/bin/eu-ld – combining object and archive files
- /usr/bin/eu-make-debug-archive
- /usr/bin/eu-nm – display symbols from object/executable files
- /usr/bin/eu-objdump – show information of object files
- /usr/bin/eu-ranlib – create index for archives for performance
- /usr/bin/eu-readelf – human-readable display of ELF files
- /usr/bin/eu-size – display size of each section (text, data, bss, etc)
- /usr/bin/eu-stack – show the stack of a running process, or coredump
- /usr/bin/eu-strings – display textual strings (similar to strings utility)
- /usr/bin/eu-strip – strip ELF file from symbol tables
- /usr/bin/eu-unstrip – add symbols and debug information to stripped binary
Notes: the elfutils package is a great start, as it contains most utilities to perform analysis.
註: elfutils 包是一個好的開始，因為它包含了最多用來執行分析的功能
elfkickers
- /usr/bin/ebfc – compiler for Brainfuck programming language
- /usr/bin/elfls – shows program headers and section headers with flags
- /usr/bin/elftoc – converts a binary into a C program
- /usr/bin/infect – tool to inject a dropper, which creates setuid file in /tmp
- /usr/bin/objres – creates an object from ordinary or binary data
- /usr/bin/rebind – changes bindings/visibility of symbols in ELF file
- /usr/bin/sstrip – strips unneeded components from ELF file
Notes: the author of the ELFKickers package focuses on mainipulation of ELF files, which might be great to learn more when you find malformed ELF binaries.
註: ELFKickers 的作者專注在操作 ELF 檔岸上，或許對想要學習畸形 ELF 二進位檔有幫助
pax-utils
- /usr/bin/dumpelf – dump internal ELF structure
- /usr/bin/lddtree – like ldd, with levels to show dependencies
- /usr/bin/pspax – list ELF/PaX information about running processes
- /usr/bin/scanelf – wide range of information, including PaX details
- /usr/bin/scanmacho – shows details for Mach-O binaries (Mac OS X)
- /usr/bin/symtree – displays a leveled output for symbols
Notes: Several of the utilities in this package can scan recursively in a whole directory. Ideal for mass-analysis of a directory. The forcus of the tools is to gather PaX details. Besides ELF support, some details regarding Mach-O binaries can be extracted as well.
註: 在這個套裝軟體中數個功能可以遞迴的掃描一整個目錄。用於對一個目錄的大量分析。這個工具聚焦的點在於集合 PaX 細節。除了 ELF 的支援，一些關於 Mach-O 二進位的細節也可以被萃取。
Example outputs
範例輸出
scanelf -a /bin/ps
TYPE PAX PERM ENDIAN STK/REL/PTL TEXTREL RPATH BIND FILE
ET_EXEC PeMRxS 0755 LE RW- R-- RW- - - LAZY /bin/ps
prelink
- /usr/bin/execstack – display or change if stack is executable
- /usr/bin/prelink – remaps/relocates calls in ELF files, to speed up process
Example / 範例
If you want to create a binary yourself, simply create a small C program, and compile it. Here is an example, which opens /tmp/test.txt, reads the contents into a buffer and displays it. Make usre to create the related /tmp/test.txt file.
如果你想要自己創造一個二進位檔，簡單的創建一個小型的 C 程式，並編譯它。這裡是一個範例，可以打開 /tmp/test.txt，將內容讀進一個緩衝區並顯示。記得要確認創建 /tmp/test.txt 檔案。
#include <stdio.h>
int main(int argc, char **argv)
{
FILE *fp;
char buff\[255\];
fp = fopen("/tmp/test.txt", "r");
fgets(buff, 255, fp);
printf("%s\\n", buff);
fclose(fp);
return 0;
}
This program can be compiled with: gcc -o test test.c
這個程式用
gcc -o test test.c
來編譯
More sources / 更多資料來源
If you like to know more, a good source would be to follow WikiPedias Executable and Linkable Format(ELF) page. Another good in-depth document: ELF_Format and the document authored by Brian Raiter (ELFkikers). For those who love to read sources, have a look at documented ELF structure header file from Apple. And them finally, if you really wnat to know how a binary works, test it with a disassembler tool like Hopper for Linux.
如果你傾向於知道更多，一個好的來源包括維基百科 Executable and Linkable Format (ELF)頁面。另外一個詳盡的文件:作者與 ELFkickers 相同的著作 ELF Format。對那些愛上閱讀原始資料，看一看這個從 Apple 的文件 ELF structure header file。最後，如果你真的想要知道一個二進位檔案是如何運作的，使用像是 Hopper for Linux 的反組譯工具來做測試。


Lately there have been a number of kerfuffles in the tech sector that have highlighted the industry's lack of valuing and respecting the diversity within: GitHub had a rather public issue with sexism; Mozilla's CEO stepped down due to his stance (of non-support) on marriage equality; RadiumOne's CEO and Chairman avoided jail and managed to get 45 felony charges dropped (battery of a partner.) There are a variety of other charming instances that one could highlight that would further exemplify the lack of - one may argue humanity - withing the sector.
Addressing the multitude of gender inequality incidents of late nine women with fairly prominent positions in Tech published an open letter to the industry entitled An Open Letter on Feminism In Tech which lays out the variety of ways in which devaluing women is the accepted norm within the sector. Perhaps the most blatant of all - gender inequality - is a pervasive problem that has not, despite some honest efforts in various areas, seemingly improved in the past few decades... well, really ever. Unlike other industries where gender inequality is mostly "just" a question of pay inequality the culture of "brogramming" within Tech takes sexism to new heights.
Brogrammer (Urban Dictionary): A programmer who breaks the usual expectations of quiet nerdiness and opts instead for the usual trappings of a frat-boy: popped collars, bad beer, and calling everybody "bro". Despised by everyone, especially other programmers.
The homophobic issue is, at times, a more subtle one - oftentimes because being gay (unlike being female) isn't always obvious. A recent HuffPo article laid out the almost concurrent issues of Brendan Eich's homophobic stance and the L.A. Clippers' Don Sterling's racist one. In what was an interesting turn of events the NBA took (almost) immediate sanctions against Sterling while Mozilla did nothing with Eich. Eich, I believe, just happened to be caught in the recent turning of the tide on the marriage equality front and the free market had enough muscle to force him out. Were his appointment to have happened a year or three ago he may well still be the company's CEO. While one hopes this bodes well for gay rights I'm still quite skeptical that the tide has turned for this segment of the population - never mind within tech with it's "brogrammer" culture.
Because a more inclusive Tech community means that the products they're producing by extension are more inclusive makes this a pivotal area for the industry to address. The rise of brogrammer culture is disturbing to say the least and with its continued rise in popularity it's merely a matter of time before the slight downward trend on the gender equality front (measured by the simple metric of percentage of women in programmer/developer/engineer positions) becomes a flat out exodus.
It's really up to you. Fortunately I'm enrolled at an coding school that has chosen to deliberately focus on Engineering Empathy. But at the end of the day it's up to each individual in the workplace - regardless of sector - to make a difference. I jut believe that this is the most important challenge facing tech at this time and I hope that I can help be an instrument of change in as much by what I don't do as by what I do. (No problem of my ever trying to emulate a brogrammer but there are plenty of ways I am sure I can/need/will work on myself that will hopefully have a positive impact on my surroundings - including my professional ones.)


Learning how to filter in the search bar
Name
course
position
Vinod
MCS
Full Stack
Bahadur
BBA
Hr manager
Thapa
BCS
Front end
Arjun
MCS
Full stack
Ramesh
MCS
full stack
Ram
MCA
Full stack
Amrit
BCS
Full stack
Naran
MCS
Full stack


Stagecast for Enterprise
The audience activation tool for larger teams
Collect valuable audience data
and convert participants into customers
Schedule a Demo
Don't have an account?
Sign up
Log in to your account
Email
Password
Log In


Releases Archive
- Hadoop Common
- HADOOP-10433 - Key management server (beta)
- HADOOP-10607 - Credential provider (beta)
- Hadoop HDFS
- Heterogeneous Storage Tiers - Phase 2
- HDFS-5682 - Application APIs for heterogeneous storage
- HDFS-7228 - SSD storage tier
- HDFS-5851 - Memory as a storage tier (beta)
- HDFS-6584 - Support for Archival Storage
- HDFS-6134 - Transparent data at rest encryption (beta)
- HDFS-2856 - Operating secure DataNode without requiring root access
- HDFS-6740 - Hot swap drive: support add/remove data node volumes without restarting data node (beta)
- HDFS-6606 - AES support for faster wire encryption
- Hadoop YARN
- YARN-896 - Support for long running services in YARN
- YARN-913 - Service Registry for applications
- YARN-666 - Support for rolling upgrades
- YARN-556 - Work-preserving restarts of ResourceManager
- YARN-1336 - Container-preserving restart of NodeManager
- YARN-796 - Support node labels during scheduling
- YARN-1051 - Support for time-based resource reservations in Capacity Scheduler (beta)
- YARN-1964 - Support running of applications natively in Docker containers (alpha)
- Authentication improvements when using an HTTP proxy server.
- A new Hadoop Metrics sink that allows writing directly to Graphite.
- Specification for Hadoop Compatible Filesystem effort.
- Support for POSIX-style filesystem extended attributes.
- OfflineImageViewer to browse an fsimage via the WebHDFS API.
- Supportability improvements and bug fixes to the NFS gateway.
- Modernized web UIs (HTML5 and Javascript) for HDFS daemons.
- YARN’s REST APIs support submitting and killing applications.
- Kerberos integration for the YARN’s timeline store.
- FairScheduler allows creating user queues at runtime under any specified parent queue.
- CVE-2014-0229: Add privilege checks to HDFS admin sub-commands refreshNamenodes, deleteBlockPool and shutdownDatanode.
- Support for Access Control Lists in HDFS
- Native support for Rolling Upgrades in HDFS
- Usage of protocol-buffers for HDFS FSImage for smooth operational upgrades
- Complete HTTPS support in HDFS
- Support for Automatic Failover of the YARN ResourceManager
- Enhanced support for new applications on YARN with Application History Server and Application Timeline Server
- Support for strong SLAs in YARN CapacityScheduler via Preemption
- Support for Heterogeneous Storage hierarchy in HDFS.
- In-memory cache for HDFS data with centralized administration and management.
- Simplified distribution of MapReduce binaries via HDFS in YARN Distributed Cache.
- YARN - A general purpose resource management system for Hadoop to allow MapReduce and other other data processing frameworks and services
- High Availability for HDFS
- HDFS Federation
- HDFS Snapshots
- NFSv3 access to data in HDFS
- Support for running Hadoop on Microsoft Windows
- Binary Compatibility for MapReduce applications built on hadoop-1.x
- Substantial amount of integration testing with rest of projects in the ecosystem
- HDFS - The HDFS community decided to push the symlinks feature out to a future 2.3.0 release and is currently disabled.
- YARN/MapReduce - Users need to change ShuffleHandler service name from mapreduce.shuffle to mapreduce_shuffle.
Release 2.5.2 available
2014 Nov 19
Apache Hadoop 2.5.2 is a point release in the 2.5.x release line, and fixes a few critical issues in 2.5.1
Release 2.6.0 available
2014 Nov 18
Apache Hadoop 2.6.0 contains a number of significant enhancements such as:
Please see the Hadoop 2.6.0 Release Notes for details.
Release 2.5.1 available
2014 Sep 12
Apache Hadoop 2.5.1 is a point release in the 2.5.x release line, and fixes a few release issues with 2.5.0.
Release 2.5.0 available
2014 Aug 11
Apache Hadoop 2.5.0 is a minor release in the 2.x release line.
The release includes the following major features and improvements:
Users are encouraged to try out 2.5.0
Please see the Hadoop 2.5.0 Release Notes for details.
Release 2.4.1 available
2014 Jun 30
Apache Hadoop 2.4.1 is a bug-fix release for the stable 2.4.x line.
There is also a security bug fix in this minor release.
Users are encouraged to immediately move to 2.4.1.
Please see the Hadoop 2.4.1 Release Notes for details.
Release 0.23.11 available
2014 Jun 27
A point release for the 0.23.X line. Bug fixes to continue stabilization.
Please see the Hadoop 0.23.11 Release Notes for details.
Release 2.4.0 available
2014 Apr 7
Apache Hadoop 2.4.0 contains a number of significant enhancements such as:
Please see the Hadoop 2.4.0 Release Notes for details.
Release 2.3.0 available
2014 Feb 20
Apache Hadoop 2.3.0 contains a number of significant enhancements such as:
Please see the Hadoop 2.3.0 Release Notes for details.
Release 0.23.10 available
2013 Dec 11
A point release for the 0.23.X line. Bug fixes to continue stabilization.
Please see the Hadoop 0.23.10 Release Notes for details.
Release 2.2.0 available
2013 Oct 15
Apache Hadoop 2.2.0 is the GA release of Apache Hadoop 2.x.
Users are encouraged to immediately move to 2.2.0 since this release is significantly more stable and is guaranteed to remain compatible in terms of both APIs and protocols.
To recap, this release has a number of significant highlights compared to Hadoop 1.x:
A couple of important points to note while upgrading to hadoop-2.2.0:
Please see the Hadoop 2.2.0 Release Notes for details.


The best way to learn music composition online
- Interactivity: Anyway that we can make learning interactive, we do.
- Composer Focused: We focus on compositional techniques, not theory or analysis. Everything is meant to help you write.
- Free: Absolutely free, forever.
Get Started
Blog Posts Optimized for Learning
Ways to Improve Your Rhythmic Content for Melodies
August 31, 2018 | Nick Mazuk
Rhythm is one of the least looked at aspects in melody writing because there are so many things to learn.
However, mastering rhythm in the context of a melody can take your melodies to the next level.
I’ve already talked about specific techniques to improve your rhythmic variety in melodies, but today I’m going to show you how get used to applying those ideas in your melodies.
Here are 2 specific practice techniques that I use to get better rhythm in melodies.


--- layout: article title: Details about our next game Burn Tactics author: matt ---
Update: we had to change plans, so Burn Tactics is no longer under development.
We recently started developing our next game, tentatively called Burn Tactics. The concept is a turn-based medieval fantasy strategy game where players control a vengeful dragon who's intent on burning down villages and seeking havoc on the kingdom.
While we're still very early in development, we've already made wireframes for every screen in the game and have figured out how the interface will flow in its entirety. As you can see from the included images, we'll be periodically uploading (really rough and ugly!) screenshots of our progress. We even have some game code written; we're using the impact HTML5 game engine and are so far really liking it.
We've had probably a dozen ideas for what to work on next including puzzle games, RTS games, or even a sequel to Onslaught! Arena. Among all these directions we could go, we decided to work on Burn Tactics for the following reasons:
Though many players are enjoying Onslaught! Arena, we've also heard that some of them are turned off by certain features, including:
Burn Tactics is a concept that shouldn't have these problems, so hopefully it'll appeal to a larger audience of gamers.
As you may have read before, we both have bad habits of letting our game concepts explode into unclimbable mountains. Given that, our mantra for Burn Tactics is:
While we would love to get Onslaught! Arena onto mobile platforms such as Android and iPad, there are many hurdles and in some cases, brick walls. Conversely, Burn Tactics is being designed from the ground up to make it feasible to get onto mobile/touch devices. This is great because releasing Onslaught! Arena onto these devices has been one of our most common requests.
We've got some issues we're looking into fixing and we're also trying to get Onslaught! Arena into the new Mac OSX App Store.


I'm a web developer and I love programming in JavaScript.
Those are my skills.
You can find me in Twitter


No information about you or your team members' usage of this site is shared with anyone else. While this site doesn't intentionally collect personally identifiable information, any such information shared will similarly not be used except in support of issues with this site.


6. Statistics and Linear Algebra Preliminaries¶
Chinese proverb
If you only know yourself, but not your opponent, you may win or may lose. If you know neither yourself nor your enemy, you will always endanger yourself – idiom, from Sunzi’s Art of War
6.1. Notations¶
m : the number of the samples
n : the number of the features
: i-th label
: i-th predicted label
: the mean of .
: the label vector.
: the predicted label vector.
6.2. Linear Algebra Preliminaries¶
Since I have documented the Linear Algebra Preliminaries in my Prelim Exam note for Numerical Analysis, the interested reader is referred to [Feng2014] for more details (Figure. Linear Algebra Preliminaries).
6.3. Measurement Formula¶
6.3.1. Mean absolute error¶
In statistics, MAE (Mean absolute error) is a measure of difference between two continuous variables. The Mean Absolute Error is given by:
6.3.2. Mean squared error¶
In statistics, the MSE (Mean Squared Error) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors or deviations—that is, the difference between the estimator and what is estimated.
6.3.3. Root Mean squared error¶
6.3.4. Total sum of squares¶
In statistical data analysis the TSS (Total Sum of Squares) is a quantity that appears as part of a standard way of presenting results of such analyses. It is defined as being the sum, over all observations, of the squared differences of each observation from the overall mean.
6.3.5. Explained Sum of Squares¶
In statistics, the ESS (Explained sum of squares), alternatively known as the model sum of squares or sum of squares due to regression.
The ESS is the sum of the squares of the differences of the predicted values and the mean value of the response variable which is given by:
6.3.6. Residual Sum of Squares¶
In statistics, RSS (Residual sum of squares), also known as the sum of squared residuals (SSR) or the sum of squared errors of prediction (SSE), is the sum of the squares of residuals which is given by:
6.3.7. Coefficient of determination ¶
Note
In general (), total sum of squares = explained sum of squares + residual sum of squares, i.e.:
More details can be found at Partitioning in the general ordinary least squares model.
6.5. Statistical Tests¶
6.5.1. Correlational Test¶
Pearson correlation: Tests for the strength of the association between two continuous variables.
Spearman correlation: Tests for the strength of the association between two ordinal variables (does not rely on the assumption of normal distributed data).
Chi-square: Tests for the strength of the association between two categorical variables.
6.5.2. Comparison of Means test¶
Paired T-test: Tests for difference between two related variables.
Independent T-test: Tests for difference between two independent variables.
ANOVA: Tests the difference between group means after any other variance in the outcome variable is accounted for.
6.5.3. Non-parametric Test¶
Wilcoxon rank-sum test: Tests for difference between two independent variables - takes into account magnitude and direction of difference.
Wilcoxon sign-rank test: Tests for difference between two related variables - takes into account magnitude and direction of difference.
Sign test: Tests if two related variables are different – ignores magnitude of change, only takes into account direction.


Character Name:
Age:
Height:
Eyes:
Family:
Canidae
Felidae
Reptilia
Mustelidae
Avialae
Species:
Dog
Fox
Wolf
Coyote
Jackal
Morphism:
Plantigrade
Digitigrade
Atypical Patterning
Hemi
How did that get there!?
Hybrid
Lateralism
Microism
Taurism
Twin-Tailed


Bacon ipsum dolor amet bacon sausage flank turducken, tail filet mignon landjaeger jerky rump tenderloin salami pork.
Jowl strip steak sirloin cupim, ribeye spare ribs chicken.
Bacon ipsum dolor amet bacon sausage flank turducken, tail filet mignon landjaeger jerky rump tenderloin salami pork.
Bacon ipsum dolor amet bacon sausage flank turducken, tail filet mignon landjaeger jerky rump tenderloin salami pork.
Jowl strip steak sirloin cupim, ribeye spare ribs chicken.
Bacon ipsum dolor amet bacon sausage flank turducken, tail filet mignon landjaeger jerky rump tenderloin salami pork.
Jowl strip steak sirloin cupim, ribeye spare ribs chicken.
Bacon ipsum dolor amet bacon sausage flank turducken, tail filet mignon landjaeger jerky rump tenderloin salami pork.
Jowl strip steak sirloin cupim, ribeye spare ribs chicken.
Bacon ipsum dolor amet bacon sausage flank turducken, tail filet mignon landjaeger jerky rump tenderloin salami pork.
Jowl strip steak sirloin cupim, ribeye spare ribs chicken.
Summary: the design's central point is information-carrying image set and user is guided trough it.


[Me]: I’ll just swap out that new modem from Comcast real quick. Shouldn’t take too long.
[narrator voice]: It took too long.
4/3/2018, 9:22:22 PM
Favs: 6
Retweets: 0
link--- layout: tweet ---
[Me]: I’ll just swap out that new modem from Comcast real quick. Shouldn’t take too long.
[narrator voice]: It took too long.
4/3/2018, 9:22:22 PM
Favs: 6
Retweets: 0
link


Intent Out, Reality In
Many organizations have an org chart that looks something like this:
At the top is the CEO and the board. Below that are the other C-level executives, then the VPs and Directors, and at the bottom, the individual contributors.
People draw charts like this because it feels satisfying to put the people with more power higher up. It gives us all a warm feeling of safety to know that the people at the “top” have a clear view of the terrain, and are steering the ship in the best direction possible.
It’s clear, when looking at a chart like this, that being higher “up” the chart is better. The best engineer becomes the king of their team, and then the best team-king becomes the King of Engineering (aka CTO).
I think a better metaphor is something like this:
Leadership sits in the middle. They don’t have a “high level view”. In fact, their view of reality is the worst of anyone. They do have the best view of the interactions between the teams, and the highest degree of access to confidential information internal to the organization. They maintain the culture and facilitate the inter-team activities.
(Note that the board is locked in the center with the CEO, but without even a connection to anything else.)
Individual contributors in any given team have extremely detailed, timely, and accurate information about their section of reality. Engineers know the reality of what the software can do. Sales and support know what customers are asking for. However, they tend to have little insight into what things look like on the other side of the org. When they turn around and look back in at the org, they see their manager, and maybe a few interactions in neighboring teams. They necessarily rely on the management infrastructure to inform them about the broader organization.
A healthy organization moves from its center, and responds to input from its edges. This requires two extremely important communication flows:
- Intent flows out.
- Reality flows in.
When in a leadership position, people will follow you. They can’t help it. It’s how we’re wired. This is very dangerous.
When a leader starts telling their organization what to do, this destroys intent-based leadership. If the leader is wrong about the facts on the ground (which of course they often will be, since they can’t see it!), then this will lead to two types of disasters:
- My leader is foolish, and I am enlightened (losing faith in the organization)
- My leader is enlightened, and I am foolish (losing faith in oneself)
Both of these are disasters because they break the “intent out, reality in” model. People at the edges will stop sharing their experience if they stop trusting it, because they don’t want to let on that they don’t know what they’re doing. If they stop trusting the organization as a whole, then it’s only a matter of time before they’re out, and in the meantime, they’re not going to be working at full potential.
Intent-based leadership is like muscle memory. If your neocortex had to direct every single body part individually, you’d never be able to talk or walk, let alone ride a bike or drive a car. Instead, your neocortex says “I want a snack.
This requires a high degree of trust and practice. Practice leads to competence, and competence over time builds trust. When trust is lost, the information flows get blocked. This leads to an increasing failure rate, blame games start, and it gets harder and harder to regain footing.
(Of course, in reality, organizations don’t exist in 2-dimensional space, and lateral communication is extremely important. But avoiding balkanization is a subject for another day. The goal of this post is to make a point about leadership and hierarchy.)
The best book I’ve found about intent-based leadership is Turn The Ship Around by David Marquet. I highly recommend it.


@
Enter your feedback:
{% from "_formhelpers.html" import render_field %}
{{ render_field(form.moviereview, cols='40', rows='10') }}


This script will encode javascript to make it more difficult for people to read and/or steal. Just follow the directions below.
Add your javascript code to the textarea box above.
Select the "Code Key" you wish to use.
Click the "Encode" button.
Select all the text that appears in the box below and paste it into your HTML page where you would want the script to be. The pasted code should appear all on one line in your editor, unless you have word-wrap on. Do not add any linebreaks (by pressing "Enter") or the script may not work.


Threading Example¶
Scripting style¶
Start with working code that is clear, simple, and runs top to bottom. This is easy to develop and test incrementally.
counter = 0
print('Starting up')
for i in range(10):
counter += 1
print('The count is %d' % counter)
print('---------------')
print('Finishing up')
That gives us the obvious output:
Starting up
The count is 1
---------------
The count is 2
---------------
The count is 3
---------------
The count is 4
---------------
The count is 5
---------------
The count is 6
---------------
The count is 7
---------------
The count is 8
---------------
The count is 9
---------------
The count is 10
---------------
Finishing up
Note
Get your app tested and debugged in a singled threaded mode first before you start threading. Threading NEVER makes debugging easier.
Function style¶
A next step in development is to factor re-usable code into functions.
counter = 0
def worker():
'My job is to increment the counter and print the current count'
global counter
counter += 1
print('The count is %d' % counter)
print('---------------')
print('Starting up')
for i in range(10):
worker()
print('Finishing up')
Multi-threading is easy!¶
It is just a matter of launching a few worker threads.
import threading
counter = 0
def worker():
'My job is to increment the counter and print the current count'
global counter
counter += 1
print('The count is %d' % counter)
print('---------------')
print('Starting up')
for i in range(10):
threading.Thread(target=worker).start()
print('Finishing up')
Testing proves the code is correct!¶
A simple test run compares perfectly to the original output:
$ python3.6 threading_multi1.py
Starting up
The count is 1
---------------
The count is 2
---------------
The count is 3
---------------
The count is 4
---------------
The count is 5
---------------
The count is 6
---------------
The count is 7
---------------
The count is 8
---------------
The count is 9
---------------
The count is 10
---------------
Finishing up
Can you spot the race conditions?¶
Most people spot the “counter increment” race condition, but most don’t immediately see the “print function” race condition.
Note
Testing cannot prove the absence of errors. It is still useful, don’t rely on it. Many interest racing conditions don’t reveal themselves in test environments.
Why didn’t testing reveal the flaws?
What can we do to improve the effectiveness of testing?
Fuzzing¶
Fuzzing is a technique for amplifying race conditions.
import threading, time, random
##########################################################################################
# Fuzzing is a technique for amplifying race condition errors to make them more visible
FUZZ = True
def fuzz():
if FUZZ:
time.sleep(random.random())
###########################################################################################
counter = 0
def worker():
'My job is to increment the counter and print the current count'
global counter
fuzz()
oldcnt = counter
fuzz()
counter = oldcnt + 1
fuzz()
print('The count is %d' % counter, end='')
fuzz()
print()
fuzz()
print('---------------', end='')
fuzz()
print()
fuzz()
print('Starting up')
fuzz()
for i in range(10):
threading.Thread(target=worker).start()
fuzz()
print('Finishing up')
fuzz()
This technique is limited to relatively small blocks of code and is imperfect in that is can’t prove the absence of errors.
Still, fuzzed tests do reveal the presence of errors:
Starting up
The count is 1The count is 2The count is 2The count is 2
---------------The count is 3
---------------The count is 4
---------------
---------------The count is 4
The count is 5------------------------------
Finishing up
The count is 5
------------------------------
The count is 6---------------
---------------
David Baron at Mozilla’s San Francisco Office¶
More Careful Threading with Queues¶
Interestingly, the rules for threading are just for computing and programming. The physical world is full of concurrency as well. Many of these techniques has physical analogs that are useful for managing people and projects.
Note
RR 1000
ALL shared resources SHALL be run in EXACTLY ONE thread. ALL communication with that thread SHALL be done using an atomic message queue: typically the Queue module, email, message queues like RabbitMQ or ZeroMQ, interesting you can communicate via a database as well.
Resources that need this technique: global variables, user input, output devices, files, sockets, etc.
Some resources that already have locks inside (thread-safe): logging module, decimal module (thread local variables), databases (reader locks and writer locks), email (this is an atomic message queue).
Note
RR 1001
One category of sequencing problems is to make sure that step A and step B happen sequentially. The solution is to put both in the same thread where all actions proceed sequentially.
Note
RR 1002
To implement a “barrier” that waits for parallel threads to complete, just join() all of the threads.
Note
RR 1003
You can’t wait on daemon threads to complete (they are infinite loops). Instead, you join() on the queue itself. It waits until all the requested tasks are marked as being done.
Note
RR 1004
Sometimes you need a global variable to communicate between functions. Global variables work great for this purpose in a single threaded program. In multi-threaded code, it mutable global state is a disaster. The better solution is to use a threading.local() that is global WITHIN a thread but not without.
Note
RR 1005
Never try to kill a thread from something external to that thread. You never know if that thread is holding a lock. Python doesn’t provide a direct mechanism for kill threads externally; however, you can do it using ctypes, but that is a recipe for a deadlock.
Applying all the rules¶
import threading, time, random, queue
##########################################################################################
# Fuzzing is a technique for amplifying race condition errors to make them more visible
FUZZ = True
def fuzz():
if FUZZ:
time.sleep(random.random())
###########################################################################################
counter = 0
counter_queue = queue.Queue()
def counter_manager():
'I have EXCLUSIVE rights to update the counter variable'
global counter
while True:
increment = counter_queue.get()
fuzz()
oldcnt = counter
fuzz()
counter = oldcnt + increment
fuzz()
print_queue.put([
'The count is %d' % counter,
'---------------'])
fuzz()
counter_queue.task_done()
t = threading.Thread(target=counter_manager)
t.daemon = True
t.start()
del t
###########################################################################################
print_queue = queue.Queue()
def print_manager():
'I have EXCLUSIVE rights to call the "print" keyword'
while True:
job = print_queue.get()
fuzz()
for line in job:
print(line, end='')
fuzz()
print()
fuzz()
print_queue.task_done()
fuzz()
t = threading.Thread(target=print_manager)
t.daemon = True
t.start()
del t
###########################################################################################
def worker():
'My job is to increment the counter and print the current count'
counter_queue.put(1)
fuzz()
print_queue.put(['Starting up'])
fuzz()
worker_threads = []
for i in range(10):
t = threading.Thread(target=worker)
worker_threads.append(t)
t.start()
fuzz()
for t in worker_threads:
fuzz()
t.join()
counter_queue.join()
fuzz()
print_queue.put(['Finishing up'])
fuzz()
print_queue.join()
fuzz()
Cleaned-up code without fuzzing¶
import threading, queue
###########################################################################################
counter = 0
counter_queue = queue.Queue()
def counter_manager():
'I have EXCLUSIVE rights to update the counter variable'
global counter
while True:
increment = counter_queue.get()
counter += increment
print_queue.put([
'The count is %d' % counter,
'---------------'])
counter_queue.task_done()
t = threading.Thread(target=counter_manager)
t.daemon = True
t.start()
del t
###########################################################################################
print_queue = queue.Queue()
def print_manager():
'I have EXCLUSIVE rights to call the "print" keyword'
while True:
job = print_queue.get()
for line in job:
print(line)
print_queue.task_done()
t = threading.Thread(target=print_manager)
t.daemon = True
t.start()
del t
###########################################################################################
def worker():
'My job is to increment the counter and print the current count'
counter_queue.put(1)
print_queue.put(['Starting up'])
worker_threads = []
for i in range(10):
t = threading.Thread(target=worker)
worker_threads.append(t)
t.start()
for t in worker_threads:
t.join()
counter_queue.join()
print_queue.put(['Finishing up'])
print_queue.join()
Careful Threading with locks¶
import threading, time, random
##########################################################################################
# Fuzzing is a technique for amplifying race condition errors to make them more visible
FUZZ = True
def fuzz():
if FUZZ:
time.sleep(random.random())
###########################################################################################
counter_lock = threading.Lock()
printer_lock = threading.Lock()
counter = 0
def worker():
'My job is to increment the counter and print the current count'
global counter
with counter_lock:
oldcnt = counter
fuzz()
counter = oldcnt + 1
fuzz()
with printer_lock:
print('The count is %d' % counter, end='')
fuzz()
print()
fuzz()
print('---------------', end='')
fuzz()
print()
fuzz()
with printer_lock:
print('Starting up', end='')
fuzz()
print()
fuzz()
worker_threads = []
for i in range(10):
t = threading.Thread(target=worker)
worker_threads.append(t)
t.start()
fuzz()
for t in worker_threads:
t.join()
fuzz()
with printer_lock:
print('Finishing up', end='')
fuzz()
print()
fuzz()
Let’s see how well the runs:
Starting up
The count is 1
---------------
The count is 2
---------------
The count is 3
---------------
The count is 4
---------------
The count is 5
---------------
The count is 6
---------------
The count is 7
---------------
The count is 8
---------------
The count is 9
---------------
The count is 10
---------------
Finishing up
It is perfect!
Cleaned-up code without fuzzing¶
Now, let’s clean it up:
import threading, time, random
counter_lock = threading.Lock()
printer_lock = threading.Lock()
counter = 0
def worker():
'My job is to increment the counter and print the current count'
global counter
with counter_lock:
counter += 1
with printer_lock:
print('The count is %d' % counter)
print('---------------')
with printer_lock:
print('Starting up')
worker_threads = []
for i in range(10):
t = threading.Thread(target=worker)
worker_threads.append(t)
t.start()
for t in worker_threads:
t.join()
with printer_lock:
print('Finishing up')
Results:
- It is perfect!
- It is beautiful.
- It is simpler than using queues.
Notes on Locks¶
Note
RR 1005
Locks don’t lock anything. They are just flags and can be ignored. It is a cooperative tool, not an enforced tool.
Note
RR 1006
In general, locks should be considered a low level primitive that is difficult to reason about in non-trivial examples. For more complex applications, you’re almost always better off with using atomic message queues.
Note
RR 1007
The more locks you are acquire at one time, the more you lose the advantages of concurrency.
Dining Philosophers¶
The rules given above help you reliably create multi-threaded code is the underlying data flow is a DAG (directed acyclic graph).
When the control flow or data flow is circular, the problem can be much harder. At that point, more formal design and verification techniques are warranted. Otherwise, it can be quite difficult in complex applications to avoid deadlock, have thread starvation, or to have unfair solutions.


Introduction¶
What is protlearn?¶
protlearn is a feature extraction tool for protein sequences. It is a freely available Python package that allows the user to efficiently extract amino acid sequence features from proteins or peptides, which can then be used for a variety of downstream machine learning tasks.
Package Contents¶
protlearn is currenty comprised of three main modules:
preprocessing
feature extraction
dimensionality reduction
The Preprocessing section allows the user to prepare the datasets according to specific needs, such as filtering out sequences containing unnatural amino acids or converting the alphabetical strings to integers. The Feature Extraction section can then be used to compute amino acid sequence features from the dataset, such as amino acid composition or AAIndex-based physicochemical properties. In total, protlearn currently provides 21 such features. Finally, Dimensionality Reduction & Feature Selection methods are provided to reduce the dimensionality of the computed features, which reduces redundancy and alleviates the computational burden for the classifiers.
Input Formats¶
protlearn was designed to handle both single and multiple sequence inputs in the following formats:
string (single)
.fasta (single)
list of strings (multiple)
.fasta (multiple)


As a form of clean and renewable energy, wind power has an increasing share in electricity generation both in U.S. and the world. Since energy generation of a turbine is proportional to its rotor area, large wind turbines have become more popular recently. However, large wind turbines are susceptible to large-amplitude vibrations that can lead to their catastrophic failure. Determining the conditions that can lead to failure is not trivial, since large turbines have complicated dynamics due to nonlinear effects, such as unsteady winds, structural nonlinearities and cyclically varying parameters.
We are creating dynamical models to identify the conditions that lead to large blade deformations and turbine failure. We also do parametric studies to relate these conditions to structural parameters of turbine parts and turbine speeds. These studies are essential for developing design guidelines for more reliable systems and avoiding critical operation conditions through control schemes.


Defines a render strategy that is defined by a sequence of textures.
More...
#include <SimpleSequenceRenderStrategy.h>
|
|
int | mFrames = 1 |
| |
|
int | mCurIdx = 0 |
| |
|
bool | mRepeated = true |
| |
|
std::vector< sf::Sprite > | mSequence |
| |
|
int | mCurFrame = 0 |
| |
|
bool | mIsFinished = false |
| |
|
bool | mCanRender |
| |
Defines a render strategy that is defined by a sequence of textures.
◆ SimpleSequenceRenderStrategy()
| game::SimpleSequenceRenderStrategy::SimpleSequenceRenderStrategy |
( |
GameBase * |
game, |
|
|
std::vector< sf::Texture * > |
sequence, |
|
|
int |
frames = 1 , |
|
|
int |
startAt = 0 , |
|
|
bool |
repeat = true |
|
) |
| |
Constructs a new Simple Sequence Render Strategy.
- Parameters
-
| game | the game |
| sequence | the sequence of textures |
| frames | the frames per texture |
| startAt | the start frame |
| repeat | wether sequence is repeated |
◆ ~SimpleSequenceRenderStrategy()
| game::SimpleSequenceRenderStrategy::~SimpleSequenceRenderStrategy |
( |
| ) |
|
|
virtual |
Destroys the Simple Sequence Render Strategy.
◆ IsFinished()
| bool game::SimpleSequenceRenderStrategy::IsFinished |
( |
| ) |
|
|
virtual |
Wether the sequence is finished.
- Returns
- bool sequence finished
◆ OnRender()
| void game::SimpleSequenceRenderStrategy::OnRender |
( |
sf::RenderTarget * |
target, |
|
|
sf::RenderStates |
states |
|
) |
| |
|
virtual |
Renders the object.
- Parameters
-
| target | the render target |
| states | the render states |
Implements game::RenderStrategy.
◆ SetSize() [1/2]
| void game::SimpleSequenceRenderStrategy::SetSize |
( |
int |
idx, |
|
|
sf::Vector2f |
size |
|
) |
| |
|
virtual |
Sets the Size of a texture.
- Parameters
-
| idx | texture index |
| size | the new size |
◆ SetSize() [2/2]
| void game::SimpleSequenceRenderStrategy::SetSize |
( |
sf::Vector2f |
size | ) |
|
|
virtual |
Sets the Size of all textures.
- Parameters
-
The documentation for this class was generated from the following files:


Defines a SVGPathSeg command that draws a linear closing line from the end of the prior segment to the starting point of the path data.
Members
The SVGPathSegClosePath object has these types of members:
Properties
The SVGPathSegClosePath object has these properties.
| Property | Description |
|---|---|
|
Gets the type of the path segment. | |
|
Gets the type of the path segment, specified by the corresponding one-character command name. |
Standards information
- Scalable Vector Graphics: Paths, Section 8.5.2
Remarks
Note In addition to the attributes, properties, events, methods, and styles listed above, SVG elements also inherit core HTML attributes, properties, events, methods, and styles.
The closepath command (Z or z) ends the current subpath and causes an automatic straight line to be drawn from the current point to the initial point of the current subpath. If you follow a "closepath" command immediately by a "moveto" command, the "moveto" command identifies the start point of the next subpath. If you follow a "closepath" command immediately by any other command, the next subpath starts at the same initial point as the current subpath.
When a subpath ends in a "closepath" command, the implementation of 'stroke-linejoin' and 'stroke-linecap' differs from when you manually close a subpath through a "lineto" command. With a "closepath" command, the end of the final segment of the subpath is joined with the start of the initial segment of the subpath by using the current value of 'stroke-linejoin'. If you manually close the subpath through a "lineto" command, the start of the first segment and the end of the last segment are each capped by using the current value of 'stroke-linecap' instead of being joined. At the end of the command, the new current point is set to the initial point of the current subpath.
Build date: 1/13/2014


You should also read the discussion on editing ldmd.conf when configuring an LDM installation.
The LDM configuration-file, ldmd.conf, provides information to the top-level ldmd and contains the following types of entries:
This file normally resides in the etc/ subdirectory of the LDM installation.
An EXEC entry tells the top-level ldmd to execute a program on startup. The programs most typically executed are pqact and rtstats (for IDD gateway LDMs).
The syntax of an EXEC entry is
EXEC "command"
where the quotation marks are mandatory if command contains whitespace (including newlines). Escape quotation marks in command with a backslash (\"). Each pair of backslashes is replaced with a single backslash.
The command strings of EXEC entries are executed via the execvp() function when the top-level ldmd program is started.
A REQUEST entry tells the top-level ldmd to start a downstream LDM child process.
The syntax of a REQUEST entry is
REQUEST feedtype prodIdEre
hostId[:port]
When the top-level ldmd is started, it creates a downstream LDM child process for every REQUEST entry. The downstream LDM connects to the upstream LDM on the specified host. The specified port is used if it exists; otherwise, the default port is used. The downstream LDM requests the specified class of data-products. If there are multiple request-entries with the same feedtype and prodIdEre combination, then the downstream LDM corresponding to the first such entry in the file will use the primary transfer-mode and all others will use the alternate transfer-mode.
The behavior of the LDM is unspecified if multiple REQUEST entries exist for the same upstream LDM and the feedtypes in the entries are not disjoint.
An ALLOW entry tells the top-level ldmd what downstream LDMs are allowed to connect and what class of data-product they are allowed to receive.
The syntax of an ALLOW entry is
ALLOW feedtype hostIdEre
OK_pattern [NOT_pattern]]
where the square brackets denote optional fields and are not part of the syntax.
When a downstream LDM connects, its host identifier is checked against the set of ALLOW entries. If nothing matches the host identifier, then the downstream LDM is disconnected. The feedtype of the matching entry specifies the type of data-products the downstream LDM is allowed to receive. If the host identifier and feedtype in the incoming request match multiple ALLOW entries and the feedtypes of the entries are not disjoint, then the behavior of the upstream LDM is determined by the first matching entry.
Besides matching the data-product selection-criteria specified by the downstream LDM, the data-product identifier must also match the optional OK_pattern and not match the optional NOT_pattern in order to be transmitted. The default for the OK-pattern is ".*" and the default for the NOT-pattern is to not perform that test. These two patterns allow an upstream LDM to fine-tune the products that a downstream LDM receives. For example, the following entry will allow a downstream LDM on host ldm.downstream.site to receive all data-products except those that begin with "TEST"
ALLOW ANY ^ldm\.downstream\.site$ .* ^TEST
The configuration-file should always have the following entries:
ALLOW ANY ^(localhost|loopback)|(127\.0\.0\.1\.?$) ALLOW ANY ^([a-z].*\.unidata\.ucar\.edu\.?$)
The first entry allows the LDM to request data from itself and is necessary for proper operation. The second entry allows the Unidata Program Center to monitor the LDM.
An ACCEPT entry tells the top-level ldmd what unrequested upstream LDMs are allowed to connect and what class of data-product they are allowed to send.
The syntax of an ACCEPT entry is
ACCEPT feedtype prodIdEre
hostIdEre
When an upstream LDM connects using the HIYA message, its host identifier is checked against the set of ACCEPT entries. If nothing matches the host identifier, then the upstream LDM is disconnected. The feedtype and prodIdEre of the first matching entry specifies the type of data-products the upstream LDM is allowed to send.
This entry was primarily used for receiving (unREQUESTed) radar data from WSI Corporation and is now primarily used to receive statistics from the rtstats(1) utility, which are formatted as data-products.
An INCLUDE entry tells the top-level ldmd to temporarily suspend processing of the current configuration-file and to process another configuration-file instead. Processing of the current configuration-file resumes when the end of the other configuration-file is reached.
The syntax of an INCLUDE entry is
INCLUDE pathname
or
INCLUDE "pathname with spaces"
A non-absolute pathname is relative to the directory that contains the configuration-file being processed.


--- notes: | The background for `user-overview` is transparent since the parent (`user-section`) has a background-image ---


{% load staticfiles %}
If you are looking at this page you are probably worried about the stock market, don't be. It's dumb. Trust this guy. He is much smarter than me.
As it turns out humans are very good at constructing narritives to explain purely statistical (ie. non causal) events. That's why it feels so much better to hear that the market is going down because traders are 'profit taking' than the market is regressing to the mean.
Enjoy some numbers and graphs!


Posted on: Tuesday, October 1, 2019
Reading time: 0 minutes and 57 seconds.
Before we start, lets take a look at the grammar of a guard statement for the Swift language.
Grammar of a Guard Statement
guard-statement –> guard condition-list else code-block
This was a conversation I had with a co-worker a couple of days back. I asked him, what is this guard in Swift? I have not been programming in Swift since iOS development made a shift from Objective C to Swift and I have not seen a guard statement used in any other languages I programmed in. It seems that guards are everywhere in Swift, but my coworker said that guards are something he never used. This is a very interesting notion. At the end of the day, each programmer has his or her own style. It is not wrong to not use guards at all but if so, why are guards everywhere?
(In fact, guards were only added in Swift 2.0)
Let’s try to understand why guards are everywhere in Swift. Semantically, guards are just a methodology to practice defensive programming and the notion where you need to return early.
The main reason guards are so dominant in Swift is due to the fact there are Optionals.


| 1 | /* | |
| 2 | * Licensed to the Apache Software Foundation (ASF) under one or more | |
| 3 | * contributor license agreements. See the NOTICE file distributed with | |
| 4 | * this work for additional information regarding copyright ownership. | |
| 5 | * The ASF licenses this file to You under the Apache License, Version 2.0 | |
| 6 | * (the "License"); you may not use this file except in compliance with | |
| 7 | * the License. You may obtain a copy of the License at | |
| 8 | * | |
| 9 | * http://www.apache.org/licenses/LICENSE-2.0 | |
| 10 | * | |
| 11 | * Unless required by applicable law or agreed to in writing, software | |
| 12 | * distributed under the License is distributed on an "AS IS" BASIS, | |
| 13 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | |
| 14 | * See the License for the specific language governing permissions and | |
| 15 | * limitations under the License. | |
| 16 | */ | |
| 17 | package org.apache.commons.collections4; | |
| 18 | ||
| 19 | import java.util.LinkedList; | |
| 20 | import java.util.Queue; | |
| 21 | ||
| 22 | import org.apache.commons.collections4.queue.PredicatedQueue; | |
| 23 | import org.apache.commons.collections4.queue.TransformedQueue; | |
| 24 | import org.apache.commons.collections4.queue.UnmodifiableQueue; | |
| 25 | ||
| 26 | /** | |
| 27 | * Provides utility methods and decorators for {@link Queue} instances. | |
| 28 | * | |
| 29 | * @since 4.0 | |
| 30 | * @version $Id: QueueUtils.java 1686855 2015-06-22 13:00:27Z tn $ | |
| 31 | */ | |
| 32 | public class QueueUtils { | |
| 33 | ||
| 34 | /** | |
| 35 | * An empty unmodifiable queue. | |
| 36 | */ | |
| 37 | @SuppressWarnings("rawtypes") // OK, empty queue is compatible with any type | |
| 38 | public static final Queue EMPTY_QUEUE = UnmodifiableQueue.unmodifiableQueue(new LinkedList<Object>()); | |
| 39 | ||
| 40 | /** | |
| 41 | * QueueUtils should not normally be instantiated. | |
| 42 | */ | |
| 43 | private QueueUtils() {} | |
| 44 | ||
| 45 | //----------------------------------------------------------------------- | |
| 46 | ||
| 47 | /** | |
| 48 | * Returns an unmodifiable queue backed by the given queue. | |
| 49 | * | |
| 50 | * @param <E> the type of the elements in the queue | |
| 51 | * @param queue the queue to make unmodifiable, must not be null | |
| 52 | * @return an unmodifiable queue backed by that queue | |
| 53 | * @throws NullPointerException if the queue is null | |
| 54 | */ | |
| 55 | public static <E> Queue<E> unmodifiableQueue(final Queue<? extends E> queue) { | |
| 56 |
1
1. unmodifiableQueue : mutated return of Object value for org/apache/commons/collections4/QueueUtils::unmodifiableQueue to ( if (x != null) null else throw new RuntimeException ) → KILLED |
return UnmodifiableQueue.unmodifiableQueue(queue); |
| 57 | } | |
| 58 | ||
| 59 | /** | |
| 60 | * Returns a predicated (validating) queue backed by the given queue. | |
| 61 | *
 | |
| 62 | * Only objects that pass the test in the given predicate can be added to the queue. | |
| 63 | * Trying to add an invalid object results in an IllegalArgumentException. | |
| 64 | * It is important not to use the original queue after invoking this method, | |
| 65 | * as it is a backdoor for adding invalid objects. | |
| 66 | * | |
| 67 | * @param <E> the type of the elements in the queue | |
| 68 | * @param queue the queue to predicate, must not be null | |
| 69 | * @param predicate the predicate used to evaluate new elements, must not be null | |
| 70 | * @return a predicated queue | |
| 71 | * @throws NullPointerException if the queue or predicate is null | |
| 72 | */ | |
| 73 | public static <E> Queue<E> predicatedQueue(final Queue<E> queue, final Predicate<? super E> predicate) { | |
| 74 |
1
1. predicatedQueue : mutated return of Object value for org/apache/commons/collections4/QueueUtils::predicatedQueue to ( if (x != null) null else throw new RuntimeException ) → KILLED |
return PredicatedQueue.predicatedQueue(queue, predicate); |
| 75 | } | |
| 76 | ||
| 77 | /** | |
| 78 | * Returns a transformed queue backed by the given queue. | |
| 79 | *
 | |
| 80 | * Each object is passed through the transformer as it is added to the | |
| 81 | * Queue. It is important not to use the original queue after invoking this | |
| 82 | * method, as it is a backdoor for adding untransformed objects. | |
| 83 | *
 | |
| 84 | * Existing entries in the specified queue will not be transformed. | |
| 85 | * If you want that behaviour, see {@link TransformedQueue#transformedQueue}. | |
| 86 | * | |
| 87 | * @param <E> the type of the elements in the queue | |
| 88 | * @param queue the queue to predicate, must not be null | |
| 89 | * @param transformer the transformer for the queue, must not be null | |
| 90 | * @return a transformed queue backed by the given queue | |
| 91 | * @throws NullPointerException if the queue or transformer is null | |
| 92 | */ | |
| 93 | public static <E> Queue<E> transformingQueue(final Queue<E> queue, | |
| 94 | final Transformer<? super E, ? extends E> transformer) { | |
| 95 |
1
1. transformingQueue : mutated return of Object value for org/apache/commons/collections4/QueueUtils::transformingQueue to ( if (x != null) null else throw new RuntimeException ) → KILLED |
return TransformedQueue.transformingQueue(queue, transformer); |
| 96 | } | |
| 97 | ||
| 98 | /** | |
| 99 | * Get an empty Queue. | |
| 100 | * | |
| 101 | * @param <E> the type of the elements in the queue | |
| 102 | * @return an empty {@link Queue} | |
| 103 | */ | |
| 104 | @SuppressWarnings("unchecked") // OK, empty queue is compatible with any type | |
| 105 | public static <E> Queue<E> emptyQueue() { | |
| 106 |
1
1. emptyQueue : mutated return of Object value for org/apache/commons/collections4/QueueUtils::emptyQueue to ( if (x != null) null else throw new RuntimeException ) → KILLED |
return (Queue<E>) EMPTY_QUEUE; |
| 107 | } | |
| 108 | } | |
Mutations | ||
| 56 |
1.1 |
|
| 74 |
1.1 |
|
| 95 |
1.1 |
|
| 106 |
1.1 |


We teach our students front- and backend development for the web, making them full-stack developers as they finish our course.
We do this with our group of experienced developers who closely support our students in their process. Besides coding we train our students in project-management, becoming independent coders and teach them about the working culture in Dutch companies.
Visit us at Hack Your Future.


Please enable JavaScript!
THOUGHT PROVOKING
QUESTION OF THE DAY
Please enable JavaScript!
Journal in Day One


"Hold on"said Kito "do we have to stop these beast in violence" "what choice do we have said alesha" "i wanted to tell you earlier but we need to warn the kingdom first"said Kito "these monsters aren't just conquering our home just for pleasure there more that meets the eye,i'll try to make this quick"kito took out some potion and showed the merpeople a little blast from the past. Long ago merfolk and sea monsters roamed the seas together in harmony,but each race saw each other different form another,so a war broke between them,the merfolk reigned victorious,and the sea monsters were exiled to the undersea,and as years went by sea monsters were discriminated and feared apon,don't you see"said kito "these guys grew up to hate us and we grew up to fear them,what if we can make peace with the sea monsters and things can go to the way they used to be" the merpeople thought about this "but what if the sea monsters don't cooperate"asked alesha
If you chose on page 11 to have alesha tell kito her problem


My Webpage
First Name
Last Name
Age
University
Felipe
Olivera
20
UTFPR
My objective on creating this page is to be able to not only notice my progress on web development but better yet be able to show others what i'm current capable off.
For more material:
Github


< This Week >
SLA
≈ to previous week
Customer Satisfaction
↑ 5% from previous week
billing
account setup
checkout
trial
login
1024 tickets
↑ 31% from last week
↑ 15% from last week
↑ 10% from last week
↓ 5% from last week
↓ 2% from last week
↓ 1% from last week
Name: John Doe Email<EMAIL_ADDRESS>If you want to send it to different emails, separate them by comma.


We can easily create shortcut in Visual Studio by going to Tools -> Options. We search for what we want and by pressing the shortcut as shown
To use Entity framework we have to install it from Nuget Package Manager as shown . Then at Package Manager Console we write :
install-package EntityFramework
With code first workflow we write first our code classes, properties etc which is our model and then the db context which is a class with the database and derives from DbContext from Entity namespace.
public class PlutoContext : DbContext { }
Here we are gonna have a some Db sets. A Db set is a collection of objects that represent the tables in the database. With the below code we can access the Courses table in the database.
public virtual DbSetCourses { get; set; }
Go to application configuration file.
<connectionStrings>
<add name="DefaultConnection" connectionString="data source=Giant-PC\SQLExpress; initial catalog=PlutoCodeFirst; integrated security=SSPI" providerName="System.Data.SqlClient"/>
</connectionStrings>
If our name is different than the DbContext class we need to specify it as a constructor.
public PlutoContext() :base("name=DefaultConnection") { }
We must enable the migrations at console.
First check application for errors in Debug before migation.
PM> enable-migrations
Then we add the migration and give it a name.
PM> add-migration InitialModelWe see the migration folder created. If entity framework since a property names Id will automatically figure out that's an identity column and a primary key. Also enity framework will create the relation with foreign key if we have a class in a class.
public class Course { public int Id { get; set; } public string Title { get; set; } public string Description { get; set; } public CourseLevel Level { get; set; } public float FullPrice { get; set; } public Author Author { get; set; } public IListAnd it createdTags { get; set; } }
CreateTable( "dbo.Courses", c => new { Id = c.Int(nullable: false, identity: true), Title = c.String(), Description = c.String(), Level = c.Int(nullable: false), FullPrice = c.Single(nullable: false), Author_Id = c.Int(), }) .PrimaryKey(t => t.Id) .ForeignKey("dbo.Authors", t => t.Author_Id) .Index(t => t.Author_Id);
Finally we update the migartion to the database .
PM> update-database
We can use code first workflow alos with an existing database.
Existing DB -> Model (make changes) -> Migration -> Existing DB
Then code first from database.
Then specify the connection String
Choose all tables except migration history since we don t want a migration history table to be created.
* * * * * * * * * * * * * * *
Migrations as we saw are used when we make changes to our model so they can applied to database.
PM> enable-migrations
PM> add-migration NameOfMigrationIf you run the migration with an empty database it going to sync with this model. If we have already the database we have to use a switch which is going to tell to entity framework that we all have these in our database so
PM> add-migration InitialModel -IgnoreChanges -ForceYou can only run one migration at a time.
PM> update-database
To delete a migration right click at the class created at migrations in solution explorer and delete. This applies only if you haven't update the database.
public virtual DbSet<Category> Categories { get; set; }If the migration exists use -Force
This is a method that lets you create directly a query for your database.
public partial class AddDataToCategories : DbMigration { public override void Up() { Sql("INSERT INTO Categories (Name) VALUES ('Web Development')"); Sql("INSERT INTO Categories (Name) VALUES ('NET Framework')"); } public override void Down() { } }To clean package manager console type cls
You go to the class you want to add the new property and then you create a new migration. You can see the changes in the migration.
public partial class AddedCategoryToCoursesTable : DbMigration { public override void Up() { AddColumn("dbo.Courses", "Category_Id", c => c.Int()); CreateIndex("dbo.Courses", "Category_Id"); AddForeignKey("dbo.Courses", "Category_Id", "dbo.Categories", "Id"); } public override void Down() { DropForeignKey("dbo.Courses", "Category_Id", "dbo.Categories"); DropIndex("dbo.Courses", new[] { "Category_Id" }); DropColumn("dbo.Courses", "Category_Id"); } }Finally again you update-database
PM> update-database
If you add a DateTime property it must allow null. Since C# Datetime is a struct and can't be nullable to make it nullable you add ?
public DateTime? Datepublished { get; set; }
If you want to change a property you press F2 while you have chosen the name of it and then apply. Then add-migration MigrationName. Finally again you update-database.
BE CAREFULL! The below code will drop the table and create new string one and also have a nullable value which doesn't make sense to have a nullable title. So whatever data we have are going to be lost.
public override void Up() { AddColumn("dbo.Courses", "Name", c => c.String()); DropColumn("dbo.Courses", "Title"); }
Make nullable false :
public override void Up() { AddColumn("dbo.Courses", "Name", c => c.String(nullable:false)); DropColumn("dbo.Courses", "Title"); }
The first argument is the table,second old name, third new name,forth argument optional.
public override void Up() { RenameColumn("dbo.Courses","Title","Name") }
After we add this name column we can populated it with the data we have.
public override void Up() { AddColumn("dbo.Courses", "Name", c => c.String(nullable:false)); SQL("UPDATE Courses SET Name = Title"); DropColumn("dbo.Courses", "Title"); }
Always make the reverse to Down method also if you need to Downgrade your database and not only make changes to Up().
public override void Down() { AddColumn("dbo.Courses", "Title", c => c.String(nullable:false)); SQL("UPDATE Courses SET Title = Name"); DropColumn("dbo.Courses", "Name"); }
First we delete the property from the class. Then we add the migration and update database. Here how the migration wii look :
namespace CodeFirstExistingDatabase.Migrations { using System; using System.Data.Entity.Migrations; public partial class DeleteDatePublishedColumnFromCoursesTable : DbMigration { public override void Up() { DropColumn("dbo.Courses", "Datepublished"); } public override void Down() { AddColumn("dbo.Courses", "Datepublished", c => c.DateTime()); } } }
Let's suppoe we will delete the Category class.
namespace CodeFirstExistingDatabase { using System; using System.Collections.Generic; using System.ComponentModel.DataAnnotations; using System.ComponentModel.DataAnnotations.Schema; using System.Data.Entity.Spatial; [Table("Courses")] public partial class Course { [System.Diagnostics.CodeAnalysis.SuppressMessage("Microsoft.Usage", "CA2214:DoNotCallOverridableMethodsInConstructors")] public Course() { Tags = new HashSet(); } public int Id { get; set; } public string Name { get; set; } public string Description { get; set; } public Category Category { get; set; } public int Level { get; set; } public float FullPrice { get; set; } public int? Author_Id { get; set; } public virtual Author Author { get; set; } [System.Diagnostics.CodeAnalysis.SuppressMessage("Microsoft.Usage", "CA2227:CollectionPropertiesShouldBeReadOnly")] public virtual ICollection Tags { get; set; } } }
public partial class DeleteCategoryColumnFromCoursesTable : DbMigration { public override void Up() { DropForeignKey("dbo.Courses", "Category_Id", "dbo.Categories"); DropIndex("dbo.Courses", new[] { "Category_Id" }); DropColumn("dbo.Courses", "Category_Id"); } public override void Down() { AddColumn("dbo.Courses", "Category_Id", c => c.Int()); CreateIndex("dbo.Courses", "Category_Id"); AddForeignKey("dbo.Courses", "Category_Id", "dbo.Categories", "Id"); } }
PM> add-migration DeleteCategoryTable
SQL("CREATE TABLE...")
namespace CodeFirstExistingDatabase.Migrations { using System; using System.Data.Entity.Migrations; public partial class DeleteCategoryTable : DbMigration { public override void Up() { CreateTable( "dbo._Categories", c => new { Id = c.Int(nullable: false, identity: true), Name = c.String(), }) .PrimaryKey(t => t.Id); Sql("INSERT INTO _Categories (Name) SELECT Name FROM Categories "); DropTable("dbo.Categories"); } public override void Down() { CreateTable( "dbo.Categories", c => new { Id = c.Int(nullable: false, identity: true), Name = c.String(), }) .PrimaryKey(t => t.Id); Sql("INSERT INTO Categories (Name) SELECT Name FROM _Categories "); DropTable("dbo._Categories"); } } }
If we have made a migration that we didn't have to do we must create another migration to fix it. Don't delete the existing migration.
Imagine we had some migrations from beginning and we want to return to version 2 to fix something. The last migration for v2 is Migration F. start -> Migration A, Migration B, Migration C, v1-> Migration D -> Migration E, Migration F, v2 -> Migration G -> Migration H now
We must tell entity framework to go where we want
pm>Update-Database -TargetMigration:FNow entity framework will run the Down methods to reverse what we have done and go where we want. Imagine that we want to go to v2 which was in the picture.
We can see the migration history also at Management Studio
We specify the target migration and entity framework reverts:
PM> update-database -TargetMigration:DeleteDatePublishedColumnFromCoursesTable Specify the '-Verbose' flag to view the SQL statements being applied to the target database. Reverting migrations: [201801201647299_DeleteCategoryTable,<PHONE_NUMBER>36594_DeleteCategoryColumnFromCoursesTable]. Reverting explicit migration:<PHONE_NUMBER>47299_DeleteCategoryTable. Reverting explicit migration:<PHONE_NUMBER>36594_DeleteCategoryColumnFromCoursesTable.After we make a change we go to the latest version again by running update-database.
To add data you want in the database add an empty migration and write SQL method to populate it with data.
When we enable-migrations visual studio automatically creates a Configuration.cs. We can use this method for dummy data to our database.
protected override void Seed(CodeFirstExistingDatabase.PlutoContext context) { context.Authors.AddOrUpdate(a => a.Name, new Author { Name = "Author 1", Courses = new CollectionWe update the database and check the message we get() { new Course() {Name = "Course for author 1",Description = "Description" } } } ); }
PM> update-database Specify the '-Verbose' flag to view the SQL statements being applied to the target database. No pending explicit migrations. Running Seed method.We see we have the data in our database.
PM> update-database -Script - SourceMigration:0
In a real world scenario we might need to change the range of migrations included in the SQL script.
PM> update-database -Script - SourceMigration:Migr1 -TargetMigration:Migr2


Write a function called count_pets
that consumes a list of strings of pet types owned and returns the number of each type of pet as a dictionary. Use the Dictionary Counting pattern. Call your function with the following list to test it.
Notice the list is written somewhat sloppy. The capitalization is inconsistent. Your function needs to be able to fix this so that "Dog"
and "dog"
go into the same dictionary entry.
This problem requires us to write a function that uses the Dictionary Counting pattern to create a dictionary that maps pet type to number of pets of that type. The problem also says the words in the list can have inconsistent capitalization; our function has to treat two of the same words as one key in the dictionary.
Once we understand the problem, it helps to think of the possible cases we can test. If we do these first, we can run our code as we go to make sure we're catching every possibility.
Here is a basic case with a list of strings with consistent capitalization.
{% filter highlight_python_code(False) %} pets1 = ["dog", "cat", "snake", "dog", "lizard", "cat", "cat"] count_pets(pets1) # => "{"DOG": 2, "CAT": 3, "SNAKE": 1, "LIZARD": 1] {% endfilter %}The case given in the problem, where the capitalization is inconsistent
{% filter highlight_python_code(False) %} pets2 = ["Dog", "cat", "Cat", "Snake", "mouse", "snake", "dog", "dog"] count_pets(pets2) # => "{"DOG": 3, "CAT": 2, "SNAKE": 2, "MOUSE": 1] {% endfilter %}A case with an empty list
{% filter highlight_python_code(False) %} count_pets([]) # => "{}" {% endfilter %}Our function should be called count_pets
and takes one parameter. The parameter holds a list of strings representing types of pets so good names would be pets
or list_of_pets
.
According to the Dictionary Counting pattern, we need to create an empty dictionary to hold entries.
{% filter highlight_python_code(False) %} pets_count = {} {% endfilter %}Next in the Dictionary Counting pattern, we check each entry in the pets list using a for
loop. This for
loop will look at every string in the list, one at a time. The item
variable from the pattern is the variable that will hold each value in the list as your loop repeats. Since we have a pets
list, we should name our item variable pet
.
Since the strings in the list could have inconsistent capitalization, we need a way to "normalize" the data. In programming, "normalize" usually means to take data that is in some way inconsistent and change it in a way that makes it easier to process.
The easiest way to normalize strings that have inconsistent capitalization is to change them all to the same capitalization. This can be done using the upper
or lower
methods that return the string in all caps and all lowercase, respectively. Since "Dogs".upper()
is "DOGS"
and "dogs".upper()"
is also "DOGS"
, these two strings can now be seen as the same.
It doesn't matter which of these we use, so we'll use upper
. Because strings are immutable in Python, we need to assign the result of the method call back to the string variable.
In the Dictionary Counting Pattern, there is a test inside the loop to determine if the potential key is already present in the dictionary (using the dictionary membership operator in
). If the dictionary already contains a key matching the current pet, we increment the value of the counter value for that key.
Otherwise if the key doesn't exist, we create it with an initial value of 1:
{% filter highlight_python_code(False) %} else: pets_count[pet] = 1 {% endfilter %}Notice we use pet as the key. This makes sure that we're not adding "dog" and "Dog" as separate keys. Both string will become a single key "DOG".
Once our loop finishes, all we have left to do is return the resulting dictionary.
{% filter highlight_python_code(False) %} return pets_count {% endfilter %}With all the pieces put together, we get our solution, which passes the tests we wrote when run!
{% filter highlight_python_code(False) %} def count_pets(pets): pets_count = {} for pet in pets: pet = pet.upper() if pet in pets_count: pets_count[pet] = pets_count[pet] + 1 else: pets_count[pet] = 1 return pets_count # Tests pets1 = ["dog", "cat", "snake", "dog", "lizard", "cat", "cat"] count_pets(pets1) # => "{"DOG": 2, "CAT": 3, "SNAKE": 1, "LIZARD": 1] pets2 = ["Dog", "cat", "Cat", "Snake", "mouse", "snake", "dog", "dog"] count_pets(pets2) # => "{"DOG": 3, "CAT": 2, "SNAKE": 2, "MOUSE": 1] count_pets([]) # => "{}" {% endfilter %}