diff --git a/.gitignore b/.gitignore index a39a2a97..5c7999b5 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,4 @@ /test cert* .*.sw* +/docs diff --git a/README.md b/README.md index 50d13c9d..3fb3e5f8 100644 --- a/README.md +++ b/README.md @@ -54,6 +54,11 @@ Section 2: Singularity Setup git clone https://github.com/underground-software/singularity.git ``` +0. Create an empty `docs` folder + ```sh + mkdir docs + ``` + 0. Build the containers. ```sh @@ -97,7 +102,14 @@ Section 2: Singularity Setup podman-compose down ``` -Section 3: Development Configuration for Live Editiing +Section 3: Adding web content + +By default no static web content is included with the repo. It goes in the docs folder. +Markdown files with the `.md` extension will be converted to html automatically. +Other static content will be served as is. You can edit `index.md` in docs to set the +homepage that shows up when you visit the website without specifying a path. + +Section 4: Development Configuration for Live Editiing -- By default, your edits to the web content in the repo are not reflected on the live website until you rebuild the container. @@ -111,7 +123,7 @@ export COMPOSE_FILE="container-compose.yml:container-compose-dev.yml" For security reasons, be sure to `unset COMPOSE_FILE` before production deployment. -Section 4: Production Deployment +Section 5: Production Deployment -- To publish an instance of singularity on the internet, you must configure the hostname. diff --git a/container-compose-dev.yml b/container-compose-dev.yml index b0307857..0784fbd6 100644 --- a/container-compose-dev.yml +++ b/container-compose-dev.yml @@ -8,7 +8,7 @@ services: target: /etc/nginx/include.d read_only: true - type: bind - source: ./kdlp.underground.software + source: ./docs target: /var/www/html read_only: true orbit: @@ -16,6 +16,6 @@ services: - label:disable volumes: - type: bind - source: ./kdlp.underground.software + source: ./docs target: /usr/local/share/orbit/docs read_only: true diff --git a/container-compose.yml b/container-compose.yml index fd9fbcab..76f6eacf 100644 --- a/container-compose.yml +++ b/container-compose.yml @@ -4,7 +4,7 @@ services: context: extenginx additional_contexts: - nginx_snippet_source=./nginx_snippets - - nginx_webroot_content=./kdlp.underground.software + - nginx_webroot_content=./docs dockerfile: Containerfile target: nginx args: @@ -38,7 +38,7 @@ services: dockerfile: Containerfile additional_contexts: - orbit_singularity_git_dir=./.git - - orbit_docs_source=./kdlp.underground.software + - orbit_docs_source=./docs - orbit_repos_source=./cgit_repos target: orbit args: diff --git a/kdlp.underground.software/README.md b/kdlp.underground.software/README.md deleted file mode 100644 index 10651735..00000000 --- a/kdlp.underground.software/README.md +++ /dev/null @@ -1 +0,0 @@ -default root for website diff --git a/kdlp.underground.software/assets/android-chrome-192x192.png b/kdlp.underground.software/assets/android-chrome-192x192.png deleted file mode 100644 index 190b1cbd..00000000 Binary files a/kdlp.underground.software/assets/android-chrome-192x192.png and /dev/null differ diff --git a/kdlp.underground.software/assets/android-chrome-512x512.png b/kdlp.underground.software/assets/android-chrome-512x512.png deleted file mode 100644 index eaea5749..00000000 Binary files a/kdlp.underground.software/assets/android-chrome-512x512.png and /dev/null differ diff --git a/kdlp.underground.software/assets/apple-touch-icon.png b/kdlp.underground.software/assets/apple-touch-icon.png deleted file mode 100644 index f6ca2acc..00000000 Binary files a/kdlp.underground.software/assets/apple-touch-icon.png and /dev/null differ diff --git a/kdlp.underground.software/assets/favicon.ico b/kdlp.underground.software/assets/favicon.ico deleted file mode 100644 index b78acde7..00000000 Binary files a/kdlp.underground.software/assets/favicon.ico and /dev/null differ diff --git a/kdlp.underground.software/assets/site.webmanifest b/kdlp.underground.software/assets/site.webmanifest deleted file mode 100644 index 5a2bf231..00000000 --- a/kdlp.underground.software/assets/site.webmanifest +++ /dev/null @@ -1,19 +0,0 @@ -{ - "name": "", - "short_name": "", - "icons": [ - { - "src": "/assets/android-chrome-192x192.png", - "sizes": "192x192", - "type": "image/png" - }, - { - "src": "/assets/android-chrome-512x512.png", - "sizes": "512x512", - "type": "image/png" - } - ], - "theme_color": "#ffffff", - "background_color": "#ffffff", - "display": "standalone" -} diff --git a/kdlp.underground.software/assets/style.css b/kdlp.underground.software/assets/style.css deleted file mode 100644 index e12c2ecf..00000000 --- a/kdlp.underground.software/assets/style.css +++ /dev/null @@ -1,465 +0,0 @@ -/* -A template for near-instantaneous deployment of websites -Copyright (C) 2019 Joel Savitz and Chris Odom - -This program is free software: you can redistribute it and/or modify -it under the terms of the GNU General Public License as published by -the Free Software Foundation, either version 3 of the License, or -(at your option) any later version. - -This program is distributed in the hope that it will be useful, -but WITHOUT ANY WARRANTY; without even the implied warranty of -MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -GNU General Public License for more details. - -You should have received a copy of the GNU General Public License -along with this program. If not, see . -*/ - -/* Basic elements */ -html, body { - font-family: courier, mono; - font-size: 15px; -} - -body { - margin: .5rem 20%; -} - -a { - color: #2db8d4; -} - -code { - font: inherit; - background: #eeeeee; -} - -pre > code { - max-width: 100%; - display: block; - width: fit-content; -} - -pre { - font: inherit; - width: fit-content; - margin: 1em; - padding: .2em; - background: #eeeeee; - white-space: pre-wrap; - word-wrap: break-word; -} - -a:visited { - color: #005d70; -} - -img { - display: block; - margin: auto; - max-width:100%; -} - -h1 { - font-size: 2rem; - margin: 1rem 0 1rem 0; -} - -h2 { - font-size: 1.5rem; - margin: 1rem 0 1rem 0; -} - -h3 { - font-size: 1.4rem; - margin: 1rem 0 1rem 0; -} - -h4 { - font-size: 1.3rem; - margin: 1rem 0 1rem 0; -} - -h5 { - font-size: 1.2rem; - margin: 1rem 0 1rem 0; -} - -input, select { - border: none; - border-bottom: 2px solid #e83d2a; - border-radius: 0; - background-color: #ffffcc; - font: inherit; -} - -button { - border: 2px solid #e83d2a; - border-radius: 0; - background-color: #eeeeee; - font: inherit; -} - -/* Login/logout/register page */ -#login, #register { - margin: 1em 0; -} - -#login label, -#register label { - font-size: 1.5em; - margin: .5em; -} - -#login input, -#register input { - font-size: 1.4em; - margin: 1em; -} - -#login button, -#register button { - margin-left: 2em; -} - -#register button { - margin-bottom: .5rem; -} - -.logout-info { - margin-top: .5rem; -} - -.logout_buttons { - margin-bottom: .5rem; -} - -/* Title + Navbar */ -img.logo { - width: 8em; - height: auto; - margin: 0 1rem 0 0; -} - -div.logo { - display: flex; - align-items: center; - vertical-align: middle; - margin-bottom: .5rem; -} - -h1.title { - margin: auto; -} - -hr { - background: #e83d2a; - color: #e83d2a; - height: 2px; - margin: 0; -} - -.nav { - display: flex; - flex-wrap: wrap; - text-align: center; - box-sizing: border-box; -} - -a.nav { - padding: 0.5em; - font-size: 1.3em; - border-left: 1px solid black; - background-color: #eeeeee; - color: #005d70; - text-decoration: none; -} - -a.nav:last-child { - border-right: 1px solid black; -} - -/* Tables */ -table { - min-width: 100%; - /**/ - border-collapse: collapse; - /**/ -} - -table, th, td { - border: 1px solid; - padding: 0.5em; -} - - /* faq page table needs a lil spacing from the
*/ -table + hr { - margin-top: .5rem; -} - -/* Lists */ -@counter-style hex { - system: alphabetic; - symbols: '0' '1' '2' '3' '4' '5' '6' '7' '8' '9' 'A' 'B' 'C' 'D' 'E' 'F'; -} - -ol { - margin-left: 1em; - counter-reset: count 16; -} - -ol > li { - counter-increment: count; -} - -ol > li::marker { - content: '0x' counter(count, hex) ': '; -} - -li > ul { - margin-left: -1em; -} - -li > p { - /* Some list items have raw text, others hold `

`s that hold text? - this fixes inconsistent padding */ - margin: 0; - padding: 0; -} - -li > p > img { - margin: 1rem auto; -} - -li { - padding-top: .4rem; -} - -/* Screen size adjustments */ -@media only screen and (max-width: 1365px) { - body { - margin: .5rem 10%; - } -} - -@media only screen and (max-width: 720px) { - body { - margin: .5rem 2%; - } -} - - /* Makes the course calendar completely visible, but hard to navigate */ -@media only screen and (max-width: 470px) { - table { - font-size: .75rem; - } -} - -@media only screen and (max-width: 370px) { - body { - margin: .5rem 1%; - } - h1.title { - font-size: 1.8rem; - } -} - -/* Cgit */ -#cgit { - margin: 1rem 0; -} - -#cgit .logo { - display: none; -} - /* Search bar */ -#cgit > table > tbody > tr > td > form { - float: right; -} - - /* Tabs (e.g. `summary` `refs` `log` `tree` ... links when in a repo, - `index` link on main page.) */ -#cgit > table.tabs > tbody > tr > td:first-child { - justify-content: space-between; - display: flex; - border-width: 0; -} - -#cgit > table.tabs > tbody > tr > td:first-child a { - padding: 0 2px; -} - -#cgit > table.tabs > tbody .form { - width: 40%; -} - -#cgit > .footer { - display: flex; - justify-content: center; - flex-wrap: wrap; - color: #ccc; -} - -#cgit .footer a:first-child { - margin-left: .5em; -} - - /* remove border between columns in header & tabs */ -#cgit > table.tabs > tbody > tr > td , #cgit > table#header > tbody > tr > td { - border-width: 1px 0; -} - - /* fix double border between header & tabs */ -#cgit > #header, #cgit .sub { - border-bottom: 0; -} - -#cgit > .content { - margin: 1rem 0; -} - -#cgit > .content > .commit-info, -#cgit > .content > .commit-msg, -#cgit > .content > .cgit-panel, -#cgit > .content > .diffstat-summary { - margin-bottom: 1rem; -} - - /* Decorations next to commit messages */ -#cgit .decoration > a { - margin: 0 .5em; - padding: 0 .25em; - border: solid 1px; - color: #000; - text-decoration: none; -} - /* HEAD decoration */ -#cgit .deco { - background-color: #ff8888; - border-color: #770000; -} - /* Branch decoration */ -#cgit .branch-deco { - background-color: #88ff88; - border-color: #007700; -} - -/* `diff options` panel */ -#cgit .cgit-panel { - display: flex; - flex-direction: column; - float: right; - margin-left: 1rem; -} - -#cgit .cgit-panel > b { - display:flex; - justify-content: center; - margin-bottom: .5rem; -} - -#cgit .cgit-panel tr:last-child, #cgit .cgit-panel tr:first-child { - display: none; -} - -#cgit .cgit-panel .label, #cgit .cgit-panel .ctrl { - border: 0; -} - -#cgit .cgit-panel .ctrl { - float: right; -} - -/* diff statistics, general commit overview + diff page stuff */ -#cgit .commit-info, -#cgit .diffstat { - min-width: 0; -} - - /* Make columns in diffstat panel take up as little space as necessary */ -#cgit .diffstat .mode, -#cgit .list .ls-mode, -#cgit .diffstat .right { - width: 1%; - white-space: nowrap; -} - -#cgit .diffstat .graph td { - padding: 0; - margin: 0; - height: .5rem; - border-width: 0; -} - -#cgit tr > .add { - background-color: #5c5; -} - -#cgit .rem { - background-color: #c55; -} - -#cgit .commit-subject { - font-weight: bold; - font-size: 1.1rem; -} - -#cgit .commit-subject, -#cgit .diffstat { - margin-bottom: .2rem; -} - -#cgit .commit-msg { - padding: 2px; - background-color: #eeeeee; - white-space: break-spaces; - text-align: left; - border: 1px solid black; -} - -#cgit .diff .add { - color: #5c5; -} - -#cgit .diff .hunk { - color: blue; -} - -#cgit .diff .del { - color: #c55; -} - -#cgit .linenumbers, -#cgit .lines { - background-color: #eeeeee; -} - -#cgit .linenumbers a, -#cgit .lines code { - font-size: 15px; - font-family: courier, mono; -} - -/* hide the 'plain' button from 'tree' view */ -#cgit .list > tbody > tr > td:last-child > .button:last-child { - display: none; -} - -#cgit ul.pager { - display: flex; - justify-content: center; -} - -#cgit ul.pager li { - padding: 0 - 10px; -} - -#cgit ul.pager li::marker { - content: none; -} - -#cgit .content > a { - display: none; -} diff --git a/kdlp.underground.software/course/fall2022/index.md b/kdlp.underground.software/course/fall2022/index.md deleted file mode 100644 index 97e0578e..00000000 --- a/kdlp.underground.software/course/fall2022/index.md +++ /dev/null @@ -1,45 +0,0 @@ -### Fall 2022 Course - -The semester has begun. To audit this course and gain access to our materials, request to join [this](https://groups.google.com/g/kdlp-fall22) mailing list. We also recommend joining our [discord](https://discord.gg/CFpuxqYgHM). - -#### Required Materials - -A linux environment to which you have root access and ideally physical access. - -#### Recommended Hardware - -Raspberry Pi 4B - -#### Recommended Books - -[Linux Device Drivers, 3rd Edition (a.k.a. LDD3)](https://lwn.net/Kernel/LDD3/) - -#### Course schedule - -Each number on the chart below is a course session scheduled for 5:00 - 6:15 PM Eastern USA time. - -|# week|S|M|T|W|R|F|S| -|--|--|--|--|--|--|--|--| -|0 4 Sept||||0||| -|1 11 Sept||1||2||| -|2 18 Sept||3||4||| -|3 25 Sept||5||6|| -|4 2 Oct||7||8|| -|5 9 Oct|||9|10|| -|6 16 Oct||11||12|| -|6 23 Oct||13||14|| -|7 30 Oct||15||16|| -|8 6 Nov||17||18|| -|9 13 Nov||19||20|| -|10 13 Nov||19||20|| -|11 20 Nov||21|||| -|12 27 Nov||22||23|| -|13 4 Dec||24||25|| -|14 11 Dec||26||| -|15 18 Dec||||||| - ---- - -##### SECRET LOGIN [🔐](/login) - ---- diff --git a/kdlp.underground.software/course/fall2023/assignments/E0.md b/kdlp.underground.software/course/fall2023/assignments/E0.md deleted file mode 100644 index 18186cd1..00000000 --- a/kdlp.underground.software/course/fall2023/assignments/E0.md +++ /dev/null @@ -1,101 +0,0 @@ -### E0 - The Big Upgrade 🎢 - -Build, install, and boot the latest Linux kernel fresh off Linus Torvalds’ tree. - -#### Outcomes: -* Set up a kernel development environment. -* Get comfortable with building the kernel. - -#### What to submit: - -* Make a copy of `template.txt` in the assignment directory renamed to `firstname_lastname.txt`. - Edit it as nescessary (don't forget to include the output of `uname -a` and `dmesg | head` and `whoami` - on the system running your newly compiled Linux kernel.) and make a commit out of it. -* Generate your patches. -* Don’t forget a cover letter. -* send them to exercise0@kdlp.underground.software - -#### Procedure: - -0. Install dependencies: - - * On Fedora, you can use: - `sudo dnf builddep kernel` - * This is roughly equivalent to: - - sudo dnf -y install gcc flex make bison openssl openssl-devel elfutils-libelf-devel ncurses-devel bc git tar dwarves - -0. Install ccache to speed up subsequent kernel builds: - - `sudo dnf -y install ccache` - - * *Be sure to restart your terminal after installing ccache so that it gets added to your path* - -0. Clone the latest (v6.5) Linux kernel release from: . - - git clone --depth=1 --branch v6.5 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git - -0. Change the current directory to the kernel tree - - cd linux - -0. Copy the kernel config for your running kernel to your kernel source tree - - cp /boot/config-$(uname -r) .config - -0. Set any new config options to the defaults - - make olddefconfig - -0. Set a custom version tag for your kernel build. Edit "Makefile" and add something cool to the "EXTRAVERSION" variable. We suggest it begin with a '.' character. It must NOT contain spaces. -0. You can print the new version string to test it by: - - make kernelversion - -0. Now, build the kernel. The `-j` option will invoke a parallel build using the number of threads specified. The `$(nproc)` will resolve to the output of the `nproc` command, and the `nproc` command will return the number of processor cores on your VM. - - make -j $(nproc) - -0. Next, install the kernel modules for your newly build kernel onto your system. The directories you will be writing to are owned by root, so you need superuser permission (granted by wrapping your command with `sudo`) in order to complete this step. - - sudo make -j $(nproc) modules_install - -0. Finally, install the new kernel into your system's `/boot` directory. The same superuser considerations from the previous step apply. - - sudo make install - - -0. Shutdown your VM - - sudo shutdown now - -0. If you were running your VM in headless mode, boot your VM in normal mode (so you can see the console output, including the pre-OS bootloader). - * When you power on your system, you should see a menu that looks roughly like the following: - * As soon as it appears, hit the down arrow key to stop the countdown that will automatically boot the first selected menu entry in a couple of seconds. - - grub bootloader image from https://jfearn.fedorapeople.org/fdocs/en-US/Documentation/0.1/html/Fedora_Multiboot_Guide/GRUB-runtime.html - -0. Use the arrow keys to select your newly build kernel with your custom tag and hit enter when it is highlighted. - -0. If you would like to set your new kernel as the default (i.e. it will boot automatically without selecting it at the menu), then use the following command to locate the index associated with your new kernel - - sudo grubby --info ALL - -0. Then, use this number to set the default to that index - - sudo grubby --set-default-index - -0. Finally, when you have booted into your new kernel and logged in, run the following commands and save their output for the assignment submission. - - uname -a - - dmesg | head - - whoami - -Seems easy enough right? 😉 - - -Expect the compilation to take a little while Make sure you have 5-10 GB of free space on the partition where you are storing the kernel tree and a bit of extra space on whatever partition /boot resides in to store the new kernel image and initramfs. - -[Submission Guidelines](../policies/submission_guidelines.md) diff --git a/kdlp.underground.software/course/fall2023/assignments/E1.md b/kdlp.underground.software/course/fall2023/assignments/E1.md deleted file mode 100644 index 9ff11011..00000000 --- a/kdlp.underground.software/course/fall2023/assignments/E1.md +++ /dev/null @@ -1,61 +0,0 @@ -### E1 - Scavenger Hunt 🔎 - -We will assign you a syscall and give you our specially designed program. -Your task is to trace invocations of that syscall by the program within -userspace and kernelspace and document your observations. - -#### Outcomes: - -* Strengthen understanding of the interaction between the kernel and userspace -* Learn how to monitor the syscalls being executed by a program -* Understand how C code translates into system calls and vice versa -* Build awareness of and basic familiarity with the linux BPF subsystem via `bpftrace` -* Gain experience with tracing based debugging - -#### What to submit - -* A cover letter following the assignment submission guidelines (as always) -* A patch which adds a directory named `firstname_lastname` to the E1 folder of the class repo with a copy of `questions.txt`. -* A patch that includes two logs with the full output from strace while running the program. Once as your regular user and once as running as root using `sudo`. Make sure you label which is which -* A patch that adds your `*.bp` `bpftrace` program, and a file that shows the expected output of it successfully tracing the syscall within the program -* A patch that includes the changes to your copy of `questions.txt` made by answering the questions -* A patch that adds the `log.txt` file in your folder with the output from all the times you ran the program - -#### Procedure: - -0. Create a `firstname_lastname` folder named for yourself within the e1 directory and cd into it. -0. Copy the provided questions.txt file into your folder (`cp ../questions.txt .`). -0. You can make the commit for your first patch at this point. -0. Play with the provided program. You can run it by typing `../program_amd64` (or `../program_arm64` if you are running fedora aarch64 on an apple silicon mac) - * It will create a `log.txt` file in your folder, look at what values it printed there -0. Examine the source code for the provided program [here](https://kdlp.underground.software/cgit/e1_demo/tree/program.c) - * Look for the places where it makes your assigned system call -0. Run `strace` on the provided program - * Locate the lines corresponding to call(s) to your syscall in the output. -0. Answer question 0 in `questions.txt` -0. Read the manual page for your system call (type `man 2 [syscall name]` e.g. `man 2 execve`) -0. Answer question 1 in `questions.txt` -0. Find the syscall definition within the kernel source - * Suggested: make use of `git grep` or [elixir](https://elixir.bootlin.com/) or [cscope](http://cscope.sourceforge.net/) -0. Answer question 2 in `questions.txt` -0. Pull the full history of the linux repository with `git pull --unshallow` (this will take a little while to download) -0. Run git blame on the file containing the definition of the syscall and scroll to the lines containing the definition - * Find the line or lines that most recently changed. -0. Answer question 3 in `questions.txt` -0. Identify the probe you will use for tracing by searching for kprobes that match your syscall with `sudo bpftrace -l` -0. Write the bpf script that inserts a probe and prints the syscall arguments, the kernelspace stack, and the userspace stack each time the syscall is executed by the program. - * Ensure that the probe is only triggered on instances of the system call caused by running our command. -0. Capture the necessary output, and create commits out of your changes as described in "what to submit" -0. Generate patches out of your commits. Don't forget the cover letter -0. Submit your patches to exercise1@kdlp.underground.software - -#### Tools of interest - -* `strace` -* `man` -* `cscope` -* `git-blame` -* `git-grep [ -n ]` -* `bpftrace` and other BPF tooling - -[Submission Guidelines](../policies/submission_guidelines.md) diff --git a/kdlp.underground.software/course/fall2023/assignments/E2.md b/kdlp.underground.software/course/fall2023/assignments/E2.md deleted file mode 100644 index b257ed73..00000000 --- a/kdlp.underground.software/course/fall2023/assignments/E2.md +++ /dev/null @@ -1,46 +0,0 @@ -### E2 - Your first kernel module 🍿 - -You will specify, implement, and test a simple kernel module that creates a `kdlp` entry in the Linux kernel's -[`/proc`](https://docs.kernel.org/filesystems/proc.html) -filesystem with a basic read only interface that returns a unchanging message that includes the student's name. - -#### Outcomes: -* Get familiar with the internals of Linux's [`/proc`](https://docs.kernel.org/filesystems/proc.html) filesystem. -* Learn how to specify the behavior of a kernel module (/ software in general) with a guided example. -* Get familiar with software testing, how to write good tests, and how to develop code using tests as a guide. -* Write your first kernel module! - -#### What to submit: - -* A patch which adds a directory named `firstname_listname` to the E2 folder of the class repo with a filled out copy of the `specification.txt` file. -* A patch which adds your testing program and makefile. -* A patch which adds your kernel module and modifies the makefile to add support for compiling it. -* Don't forget your cover letter (as always). - -#### Procedure: - -0. Create your named folder in the `e2` directory, change directory into it, and copy the specification template into your folder (`cp ../specification.txt .`). -0. Edit your copy of `specification.txt` and fill in the blanks to fully specify the expected behavior of your module. -0. At this point you can `git add` the specification file and make your first commit. -0. Write a testing program and makefile that exercises all the functionality your driver will have according to the specification by performing operations on the file that it will create in `/proc`. - * Be sure to write tests for all the behavior (existence of the file, file permissions, behavior of reading, writing, seeking, etc.). - * Make sure to also test error conditions to make sure strange user input to your driver won't crash the system, e.g. a null or invalid pointer. - * You should have at least 10 distinct tests that the driver can pass or fail. - * When run, the program should provide information about which tests succeeded or failed. - * If a test failed, include in the program output how the behavior differed from what was expected. - * At this point, you have nothing to test. In order to validate your testing program, temporarily change the hard-coded path of the file it opens. - * Instead of opening a file in `/proc`, you can have it open a regular file whose contents/owner/permissions you can change manually to see which tests pass or fail. - * Don't forget to change this back before committing your code (or write your program / makefile in such a way that what file is used can be overridden from the default in `/proc` for testing). -0. At this point you can `git add` your makefile and testing program and make your second commit. -0. Implement the driver. - * Update your makefile so it can build/clean, load/unload, and test your kernel driver. - * See the example kernel module from [the slides](https://kdlp.underground.software/course/slides/writing_modules.html) for information about how to create a basic module and build it. - * The kernel code in [`fs/proc/cmdline.c`](https://elixir.bootlin.com/linux/v6.5/source/fs/proc/cmdline.c) and [`kernel/configs.c`](https://elixir.bootlin.com/linux/v6.5/source/kernel/configs.c) - contain examples that create simple `/proc` entries that providing access read-only data. - * Use the newer, better `proc_create_single` API from the `/proc/cmdline` code. - * However, reference the code for `/proc/config.gz` for information on how to remove a `/proc` entry. - * Hint: You can do this with three functions: init, exit, and one that generates the data. - * Don't overcomplicate things. -0. Submit your patches to `exercise2@kdlp.underground.software` by the due date. - -[Submission Guidelines](../policies/submission_guidelines.md) diff --git a/kdlp.underground.software/course/fall2023/assignments/F0.md b/kdlp.underground.software/course/fall2023/assignments/F0.md deleted file mode 100644 index 92c34758..00000000 --- a/kdlp.underground.software/course/fall2023/assignments/F0.md +++ /dev/null @@ -1,65 +0,0 @@ -## F0 - Specification 📐 - -You will conceive of an idea for a novel character driver with the objective of designing something that is creative and non-trivial and will provide for an entertaining demonstration as part of someone's final presentation. - -With the knowledge that it is one of your colleagues who will be responsible for actually implementing your idea, you will write a specification of the behavior of each syscall that is precise and detailed enough for them to implement it correctly. - -To further guide your colleague in implementing a driver that complies with your creative vision, you must write a userspace program that executes a suite of tests that a correct driver should pass. - -### Outcomes: - -* Validate your conceptual understanding of character devices -* Demonstrate your creative capacity and vision -* Simulate teamwork and communication aspects of working on a real-world coding team -* Practice clarity in technical writing by clearly specifying the intended behavior of code to colleagues -* Further develop your ability to rigorously test code - -### What to submit: - -* A patch which adds a directory named `firstname_lastname` to the F0 folder of the class repo containing your specification. -* A patch which adds your test program and makefile. - - * Make sure to have compiler warnings enabled (at least `-Wall` but ideally `-Wextra -Wpedantic` or even `-Weverything` if you use clang) and that your code doesn't have any warnings or errors. - -* Don’t forget your cover letter. - -* Submit your patches to `final0@kdlp.underground.software` - -### Procedure: - -0. Brainstorm some ideas with an eye for how the behavior can be implemented in a real character driver. -Some entertaining ideas may be more feasible on paper than in practice, so get started early so that you don't have to pick the first idea you can come up with right before the due date. -As a source of inspiration, here are some categories of ideas that have proved viable in the past with examples of what previous students did. - * A cipher or code for encrypting and decrypting text e.g. Vigenère or Pig Latin. - * A simple game, like tic-tac-toe. - * A tool such as a unit converter. - * A simulator for some process e.g. ordering food at a drive-thru window, or a Tamagotchi-pet, or interpreting a simple programming language. - -0. The most important system calls to figure out will be `read`, `write` and `ioctl`. Read and write are best at handling strings of text, while if you want to process numbers, ioctl is probably a better option. Ioctl can also be a catch all way to signal that the user wishes for something to happen and different commands can be passed to indicate different actions. - -0. Once you have designed the core functionality of the driver, consider how those functions will (or will not) interact with the file position value that the kernel stores for each open file and how `lseek` will fit into that picture. What if any types of repositioning should be supported? How will the other functions handle a file position that might be out of bounds? - -0. Finally you have to think about the life-cycle of a given file and the driver overall. Does some the state persist the whole time the driver is loaded? If so, what initial condition should it be in when the driver is first loaded. Does anything get reset or created in response to a call to `open`? If so do things need to be cleaned up in `release` when the file is closed? - -0. Write a plain-text specification for the `open`, `read`, `write`, `close`, `ioctl`, and `lseek` syscalls. - * You must describe how a user can correctly invoke each syscall and what behavior or information it accepts and/or returns. - * You must describe the behavior and `errno` values returned for all possible failure modes that come to mind. - * You must describe with as much precision as possible the state of the system following a successful invocation of any of the features. - - -0. You must test all of the system calls that your driver supports and write at least 25 tests in total. For the main system calls `read`, `write` and `ioctl`, you should have at least 5-7 tests. - * Be sure to write tests that cover all of the behavior you specified, including but not limited to: - * Simple tests that try to isolate a specific behavior and verify it is correct. - * More complex integration tests that verify that different system calls can work together. - * All failure modes that happen within the logic of the driver or as a result of user behavior (i.e. not ones resulting from extreme system conditions beyond the driver such as lack of resources). - * Get creative when considering the incorrect usage of your API. Come up with some exotic input data and insane parameters. - * Don't forget to test for simple memory errors like usage of a null or invalid pointer. - * When run, the program should provide information about which tests succeeded or failed. - * If a test failed, include in the output of the testing program how the behavior differed from what was expected. - * Your test program should gracefully handle errors, e.g. it should not crash if a test fails or if the character driver's device file cannot be found. - * Program output should be easy to understand and clear. -0. You will not have a driver implementation to run your test program against, however you can still verify that it is performing the right operations by pointing it at a dummy file like `/dev/null` or `/dev/zero` and verifying that it is making the right system calls with `strace`. - * Be careful while writing your tests and document them well so that the person implementing the driver can understand your thinking. - * Mistakes happen, and it may be that the tests are not 100% correct. The person implementing the driver will have some leeway to fix bugs in the tests while trying not to change the spirit of what is being tested. We will not take off points for honest mistakes, but we do reserve the right to take points off if the tests are completely incoherent and prove impossible to work with. - -[Submission Guidelines](../policies/submission_guidelines.md) diff --git a/kdlp.underground.software/course/fall2023/assignments/F1.md b/kdlp.underground.software/course/fall2023/assignments/F1.md deleted file mode 100644 index 5caa455f..00000000 --- a/kdlp.underground.software/course/fall2023/assignments/F1.md +++ /dev/null @@ -1,66 +0,0 @@ -## F1 - Contributing to Open Source Software 🤝 - -You will receive a specification and tests created per -[F0](F0.md) -by one of your colleagues. - -You must implement the character device from spec as a Linux kernel module. - -Your colleague's tests will pass when run against your implementation. - -### Outcomes: - -* Put your knowledge of character devices into practice -* Demonstrate your reading comprehension of technical topics -* Simulate real software engineering teamwork and communication -* Discern between your own errors and those of others while working with imperfect information - -### What to submit: - -* The `f1` folder will contain a directory with your name. - * Within you will find your given spec, tests, and a makefile to build them. -* If there are any bugs in the tests, your first patch fixes them. -* Your next patch (the first if you determine the tests to be bug-free) adds your module. - * The module code and its makefile will be created in a folder named `module` within your named directory. -* Your final patch adds additional tests to the provided program. -* Don’t forget your cover letter. -* Submit your patches to `final1@kdlp.underground.software` - * Your initial submission is due by 11:59 PM the day before you present. - * There will be no peer review. - * Your final submission is due by 11:59 PM on the last day of presentations. - -### Procedure: - -0. Locate and examine your assigned spec, tests, and makefile. - * The `f1` directory within the assignments repository contains named subdirectories for each student. - * The provided makefile should build the tests as the default target. - * However, initially there is nothing to test. -0. Carefully review the spec and examine each of the test cases. - * It will be helpful to begin by associating each test case with the corresponding part of the spec. - * If you find any issues, inconsistencies, or ambiguities in the spec or tests, then: - * For simple problems, fix the issue yourself as you see fit. You may take some creative liberty but do your best to maintain the intent of the original author. - * For more puzzling concerns, communicate with the original author and/or post in the #questions channel on KDLP Matrix. - * Think about whether the given test file tests all cases. - * This will give you a head start later when you are writing additional tests. - * If you modify the spec, you may need to adjust some tests to reflect these updates. - * Your first patch will consist of any adjustments you make to the spec or tests. - * If you do not need to make any changes, then your series won't include this patch. -0. Using the now thoroughly reviewed spec, implement the character device it describes. - * First, create a subdirectory within your named folder called `module`. - * Use the kernel's -[miscdevice API](https://www.kernel.org/doc/html/v6.5/driver-api/misc_devices.html) -to register and de-register your character device. This is standard practice for simple kernel drivers implemented as character devices. - * Implement one syscall at a time and focus only on the tests pertaining to that syscall. - * Assume an arbitrary number of processes will be accessing your character device at the same time. - * Assume that any of the threads in these processes may be interrupted after any line of code for an arbitrary amount of time. - * Code designed to run in these conditions must be -[reentrant](https://en.wikipedia.org/wiki/Reentrancy_(computing)). - * To properly implement this concurrent behavior, you must enforce mutual exclusion around any access to shared data. - * As always, no memory leaks, no use-after-free, no buffer overflows. This is especially important in the kernel. - * Once all the tests pass, make your next commit. -0. Create 5 new tests - * Try to add new tests for untested edge cases, areas of the spec that you may have changed, or more complicated integrations of multiple syscalls in a row. - * Your driver added by the previous patch must pass these tests without further modification. - * Your next patch adds these tests to the existing file. - -[Submission Guidelines](../policies/submission_guidelines.md) diff --git a/kdlp.underground.software/course/fall2023/assignments/P0.md b/kdlp.underground.software/course/fall2023/assignments/P0.md deleted file mode 100644 index f56aa929..00000000 --- a/kdlp.underground.software/course/fall2023/assignments/P0.md +++ /dev/null @@ -1,80 +0,0 @@ -## P0 - Build a Shell 🐚 - -The objective of this assignment is to build a UNIX style shell. - -### Outcomes: - -* Demonstrate your knowledge of C, specifically working with strings. -* Get an introduction to system calls. -* Make sure the course is a good fit for you. - -### What to submit: - -* A patch which adds a directory named firstname_lastname to the assignment repo. - * The first patch should create this directory and add your Makefile. - * Make sure to have compiler warnings enabled (at least -Wall but ideally -Wextra -Wpedantic or even -Weverything if you use clang) and that your code doesn’t have any warnings or errors. -* The subsequent patches should add your shell. Each patch that completes a level should indicate that fact clearly. You can use as many patches as you'd like, just be sure to describe the purpose of each patch. Your program should compile after each patch is individually applied (you can use `git commit --amend` to fix mistakes). -* Don't forget a cover letter. -* Submit your patches to `programming0@kdlp.underground.software` - -### Procedure: - -0. This assignment will guide you through the process of building a shell by iteratively improving your code and adding more features, starting with a very simple program. At the end you should have a shell that mostly works. -0. You must make at least one commit per level completed, and label clearly at which commit you think you completed a particular level. -0. You must pass all ten levels listed below. If you want to go further for an extra challenge, see the list of ideas at the bottom. - -### Levels: - -0. **lvl 0**: - * The shell prints a prompt, informing the user about which absolute path they are currently in (see `man 3 getcwd`) followed by a $ and a space (e.g. `/your/current/directory$ `). - * The shell then prints a new line and exits without any user interaction. -0. **lvl 1:** - * The shell reads lines of user input, but doesn't need to do anything with them. It just prints a new prompt before each line. - * The shell loops until it gets EOF from user input (ctrl+d). - * The shell exits with code 0. -0. **lvl 2:** - * If the user types any text, the shell prints "Unrecognized command" (but does not exit the loop). - * However, if the user just hits enter without typing anything, no error message is printed. -0. **lvl 3:** - * The shell splits the line of input into pieces delimited by whitespace characters (see `man 3 isspace`). - * Instead of just printing "Unrecognized command" the shell shall include the name of the program in the error message (e.g. if the user types `cat shell.c`, the shell prints "Unrecognized command: cat"). -0. **lvl 4:** - * The shell supports a few builtin commands (exit, cd and exec). If the first piece of the input is not "exit" or "cd" or "exec" it will still print the unrecognized command message, otherwise: - * exit: takes no arguments (prints error if they are provided) and closes the shell (return value of 0) - * cd: takes exactly one argument (otherwise it prints usage info) and changes the working directory of the shell process (see `man 2 chdir`). If chdir does not accept the path provided, an error message including a description of the errno is printed. The new working directory is reflected in the prompt. - * exec: takes at least one argument (maybe more) and replaces the shell with an instance of the command specified (see `man 3 execv`). If execution fails, an error message is printed including a description of the errno, and the shell continues. Otherwise, having been replaced, the shell does not return. -0. **lvl 5:** - * The shell supports running executable files as commands within child processes. If the first piece of the input looks like a path (starts with . or /) a child process is created (see `man 2 fork`) and the command specified by the first argument is executed within the child using the provided arguments (see `man 3 execv`) similar to the exec builtin. If executing the command fails, the child process prints an error message including a description of the errno (don't forget to exit the child process). The shell waits for the child to finish running before printing the next prompt (see `man 2 waitpid`). -0. **lvl 6:** - * In the case that the user types something that isn't a path or a builtin, before printing the unrecognized command error, the shell checks whether a file with that name exists in each of the directories listed in the `PATH` environment variable in order (see `man 3 getenv` and `man 2 stat`). If a file, with that name is found, the search can stop and that file is executed with arguments in a child process. Only if no file is found in any of the directories, should the unrecognized command error be printed (NOTE: you must do the path searching manually, and cannot rely on a member of the exec family that does path searching automatically e.g. `execvp`). -0. **lvl 7:** - * Before processing the entered commands, the shell performs home directory substitution on the pieces (command name or arguments) that start with a `~`. - * The shell determines a username string by taking a substring of the piece after the `~` until the end of the string or the first `/`, whichever comes first. - * If the username string is empty, then the `~` is replaced with the value of the `HOME` environment variable (see `man 3 getenv`). - * If the username string is not empty, the shell attempts to locate the the user with that username and, if successful, replaces the `~` and the username substring with their home directory (see `man 3 getpwnam`). - * If `getpwnam` does not locate such a user, the shell leaves the piece unmodified. -0. **lvl 8:** - * As the shell is processing the commands and arguments, if it finds a `<` or `>`, it skips any whitespace characters and attempts to treat the next piece of the input as a filename for redirection. If there is no filename before the end of the string, an error message is printed. - * A command can have more than one redirection (even more than one of the same type). If there are multiple of the same redirections, then the right-most one takes precedence. - * A redirection beginning with `<` causes the shell to open the corresponding file and replace the stdin of the child process with the file descriptor of the open file (see `man 2 dup`). - * A redirection beginning with `>` behaves similarly, but it replaces the stdout of the process. -0. **lvl 9:** - * The shell supports the `|` (pipe) operator to chain multiple commands and their inputs and outputs. - * Each command separated by a `|` is spawned as its own child process. The shell can handle more than one pipe. - * The shell creates a unidirectional pipe (see `man 2 pipe`) for each `|` and redirects the stdout from the left command to the writing end of the pipe and redirects the stdin of the right command to the reading end of the pipe. - * Any file redirections specified by the user take precedence over any implied redirections of the `|`. - -### Bonus - -If you were inspired by completing all ten required level and you want to go further (for fun and bragging rights), feel free to add some or all of these features. - -* Add support for `;` to put multiple commands on one line and the `#` for comments. -* Add support for the `&&` and `||` operators to chain commands based on their return codes. -* Add support for special variables like `$?`, `$#`, `$*`, `$$`. -* Add support for reading environment variables using `$NAME` (i.e. that would be replaced with `getenv("NAME")`). -* Add support for modifying environment variables using an `export` builtin. -* Add support for background jobs (`jobs`, `fg`, `bg` builtins), and `&` for spawning a job in the background. -* Add support for the `source` builtin to run commands from a file. -* Add support for control flow operators (e.g. loops, conditionals, selection statements). - -[Submission Guidelines](../policies/submission_guidelines.md) diff --git a/kdlp.underground.software/course/fall2023/assignments/P1.md b/kdlp.underground.software/course/fall2023/assignments/P1.md deleted file mode 100644 index 82086fcf..00000000 --- a/kdlp.underground.software/course/fall2023/assignments/P1.md +++ /dev/null @@ -1,114 +0,0 @@ -### P1 - Add a new syscall 🤫 - -Add a new system call to the kernel - -#### Outcomes: - -* Get comfortable with the concept of a syscall from both kernelspace and userspace - * Learn how userspace programs make system calls - * Learn how those calls are handled by the kernel -* Understand how C relates to assembly language - * Write a C program to call your system call - * Write the same program in the assembly language of your platform - -#### What to submit: - -* A patch which adds a directory named `firstname_lastname` to the assignment repo. - * The first patch should add your kernel patch *as a .patch file* to our repo. - * Generate a single patch from your commit to the Linux kernel using the command `git format-patch -1` within your linux repo - * Take the `.patch` file outputted, and put it in your named directory (e.g. `cp 0001*.patch ~/ILKD_Assignments/p1/your_name/` - * Add *the email patch itself* to the staged git files and commit this. -* The second patch adds your C program to the named directory - * Include a Makefile that can compile it - * Make sure to have compiler warnings enabled (at least `-Wall` but ideally `-Wextra` and `-Wpedantic` too, or even `-Weverything` if you use clang) - * Make sure that your code doesn't have any warnings or errors. -* The third patch adds your assembly program to your named directory. - * Edit the Makefile to add rules for assembling and linking your assembly program using `as` and `ld` -* The fourth patch adds the output of running `strace` on your assembly program as it uses the new syscall. -* Don't forget a cover letter. -* Submit your patches to programming1@kdlp.underground.software - -#### Procedure: -0. Start by adding a new syscall to your copy of the linux kernel. - 0. Create a new `.c` file in the `kernel` directory within the linux repo named `kdlp.c` - 0. Using the appropriately numbered `SYSCALL_DEFINE` macro, define a syscall entry function for a new `kdlp` syscall in the `kdlp.c` file that behaves as follows: - * Takes two arguments, an appropriately marked pointer to a buffer from userspace of type `char *`, an integer specifying the size of the buffer of type `size_t`. - * Formats a message into a local buffer that includes at least the student's name and the name of the current task that is running on the CPU (see the `get_task_comm` macro). - * Determines the amount of data to send - equal to the lesser of the size of the kernel message and how much space the user provided (see `min` macro). - * Copies that many bytes from the local message buffer into the user pointer (see the `copy_to_user` function). - * Returns a negative error code of `-EFAULT` (errno 14, bad address) if the copy is not successful. - * Otherwise, returns the amount of data that was copied. - 0. The syscall implementation should take care to prevent any possibility of a buffer overflow. - 0. Modify the Makefile for the kernel subdirectory in `kernel/Makefile` to add `kdlp.o` to the list of objects that are always compiled (add it to the end of the `obj-y` variable). - 0. Add an `asmlinkage` declaration of `sys_kdlp` to the bottom of `include/linux/syscalls.h` (take inspiration from the numerous examples of other syscalls in that file). - 0. Add an entry to the syscall table for your architecture. - * On x86-64, this is in a special table format located at `arch/x86/entry/syscalls/syscall_64.tbl` for historical reasons. - * You need to pick a number for it, put it in order with the next highest number after the last `common` entry in the table - * Your syscall will be `common` (i.e. shared between 32 and 64 bit userspace programs transparently by the kernel) - * The name is `kdlp` and the entry point is `sys_kdlp` - * Take inspiration from the other nearby syscalls - * On aarch64 the list comes from the modern shared table located in `include/uapi/asm-generic/unistd.h`. - * Find where `__NR_syscalls` is defined and increment the value. - * Just above that line, add a `#define` for `__NR_kdlp` with the next syscall number that is free - * add an invocation of the `__SYSCALL` macro with the `__NR_kdlp` number and and the `sys_kdlp` entry point. - * Take inspiration from the other nearby syscalls. - 0. Update the extra version tag in the main kernel Makefile so you can tell this kernel that has your syscall apart from the other kernel you compiled. - 0. Update the config using `make oldconfig` (this should only take a few seconds, and shouldn't require you to answer any questions). - 0. Compile your new kernel with `make -j $(nproc)` - 0. If the command finishes, make sure it was successful: `echo $?` should output 0. - * If it does not, re run `make` without `-j $(nproc)` to get a better error message. - * Fix whatever issue you see, and re run `make -j $(nproc)`. - * Repeat this process of checking the result and fixing any errors until it compiles successfully. - 0. Install your new kernel and its modules `sudo make -j $(nproc) modules_install install` -0. Reboot your vm and pick the new kernel. - * If all is well it should boot successfully. - * Congrats, you just wrote your first kernel code! -0. Make a patch out of your kernel code. - * Find all untracked files, and files you modified by looking at the output of `git status` within your linux kernel repository. - * Add them to the staging area with `git add` and then run `git commit -s` and provide a message. - * Format the resulting patch as a `.patch` file with `git format-patch -1`. - * You don't need `--rfc` or `-v1` or `--cover-letter` because this `.patch` is not getting emailed directly - * Copy the `.patch` file into your named folder within the `p1` folder in the ILKD Assignments Repo. - * You can make your first commit for the assignment at this time. -0. Write a C program that invokes the new syscall. - * Use the `syscall(2)` function from the C standard library to invoke your syscall. - * Pass the number you gave it in the table as the first argument and then pass the rest of the arguments. - * Check for any errors and print an appropriate message if they occur. - * Otherwise, print the message you received from the kernel using the `write(2)` syscall. - * You can now make your second commit for the assignment at this time. -0. Write an assembly language program that behaves exactly the same as your C program with the following minor differences because you are not using the C standard library: - * Your program starts at a label named `_start` instead of a function named `main` - * you need to explicitly call the `exit` system call to terminate your program (trying to return from `_start` will segfault) - * You explicitly invoke the system calls by following the conventions of the linux syscall ABI (application binary interface) - * For x86-64: - * Arguments go into specific registers in order (`%rdi`, `%rsi`, `%rdx`, `%r10`, `%r8`, `%r9`) - * The syscall number goes into `%rax` - * The syscall is invoked using the `syscall` instruction - * The return value is in `%rax` and is either a non-negative return value, or a negative error number (errno variable is a C library concept) - * For aarch64: - * Arguments go into specific registers in order (`x0`, `x1`, `x2`, `x3`, `x4`, `x5`) - * The syscall number goes into `x8` - * The syscall is invoked using the `svc #0` instruction - * The return value is in `x0` and is either a non-negative return value, or or negative error number (errno variable is a C library concept) - * Make sure to update Makefile to assemble and link your program - * You can now make your third commit for the assignment at this time. -0. Test your C program and assembly program. - * They should both print a message including your name and the name of executable file when you try them on the kernel you just compiled. - * Try running `strace` on your assembly program. Capture the output into a file using `2> strace_log` on the end of the command and add it to the repo as your fourth patch. - * You can also test the error case by booting into a kernel without your syscall (i.e. the regular upstream kernel you compiled or the one packaged by fedora) - * the kernel will return `-ENOSYS` to indicate that the syscall is not implemented. - * both programs should print some sort of error message and exit early with a nonzero code. - -The following pages will be of interest to a student: - -* - -Conventions: - - 0. Assembly files that must be pre-processed use the extension `.S` - - 0. Assembly files that are ready to be assembled use the extension `.s` - -Refer to the [Linux kernel documentation about adding syscalls](https://www.kernel.org/doc/html/latest/process/adding-syscalls.html) for further guidance. - -[Submission Guidelines](../policies/submission_guidelines.md) diff --git a/kdlp.underground.software/course/fall2023/assignments/P2.md b/kdlp.underground.software/course/fall2023/assignments/P2.md deleted file mode 100644 index 0ae0149b..00000000 --- a/kdlp.underground.software/course/fall2023/assignments/P2.md +++ /dev/null @@ -1,36 +0,0 @@ -### P2 - Capture the Flag 🏁 - -Solve a puzzle and uncover the secret message by demonstrating your knowledge of Linux. - -#### Outcomes: - -* Learn about the system calls commonly implemented by character devices and how they interact with the file position stored in the kernel -* Get familiar with how userspace syscalls translate in running kernel code within a driver. -* Have fun solving a custom puzzle - -#### What to submit: - -* A patch that adds a copy of the ctf driver code and makefile to your named directory. - * There will already be a folder with your name in the p2 folder. - * You should copy `ctf.c` and `Makefile` from the p2 directory into to your named directory. - * Add the files to git and make your first commit. -* A patch which adds your solution program and changes the makefile to compile it. - * Write a C program that performs the right sequence of operations on `/dev/ctf` so that the messages from the driver in `dmesg` match the provided output in `dmesg_log`. - * Each successful operation will return one byte of the secret message. Your C program should collect those bytes and print them at the end. - * If the arguments passed to the operation are incorrect, the returned byte will be nonsense. - * If the operation would leave the device in an invalid state (i.e. f_pos outside of the range 0 through 256), the operation will return an error. -* Don't forget your cover letter. - * Make sure to include the secret message you obtained in your cover letter. - -### Procedure: -0. Copy the driver code and makefile into your folder. - * Examine the driver code to understand how information flows from one function to another. - * Where do the values printed to the kernel ring buffer come from? - * How can you reverse that process to determine the arguments you need to pass to the syscalls? -0. Write your C program to perform the operations and gather the secret message. - * Initially, it should just open `/dev/ctf` and verify that it got a valid file descriptor then close it. - * Between those two operations, insert code that calls `read`/`write`/`lseek`/`ioctl` on the file descriptor based on the contents of `dmesg_log`. - * The buffer argument for calls to `read` and `write` can be `NULL` since the driver does not actually read or write data from it, the size is the only important argument. -0. Submit your patches to programming2@kdlp.underground.software - -[Submission Guidelines](../policies/submission_guidelines.md) diff --git a/kdlp.underground.software/course/fall2023/assignments/course_application.md b/kdlp.underground.software/course/fall2023/assignments/course_application.md deleted file mode 100644 index 8e0f8974..00000000 --- a/kdlp.underground.software/course/fall2023/assignments/course_application.md +++ /dev/null @@ -1,34 +0,0 @@ -## Course Application - -WELCOME APPLICANTS! - -Please thoroughly read and then complete the following instructions: - -Send an email to kdlp-application@googlegroups.com satisfying the following conditions: - -a) The subject of the email contains your full legal name followed by the text "-- [SESSION] Application for Introduction to Linux Kernel Development", where [SESSION] is replaced by the appropriate string, i.e. "Fall" or "Spring" followed by the current year. - -b) If you are enrolled at UMass Lowell, then send this email must be sent from your @student.uml.edu address. - -c) The email should state your current student and/or occupational status, i.e. if you are enrolled in a school, name the school, your course of study, your level of progress through this course of study, and if you are employed, name your employer (or business, if you own one), and your job title. - -d) The email should contain your response to the following question: - - What does the following C language expression evaluate to? - - "KDLP"[4] - - Explain your answer in fewer than 5 complete English sentences. - -e) If you have a resume, KDLP allows that it be attached to this email as a PDF file or text file. No other document in any other format may be attached to this email. - -f) The email must contain your discord username that you will use to complete step 2 - -g) If you prefer to use a different email to send and receive email patches on our mailing list than the one you are using to send this message, you should specify this address. KDLP will provide instructions on the usage of gmail with our systems, but it is not required that all students use gmail. - -h) Include your time zone. - -i) Include a line certifying the [Developer Certificate of Origin](https://bssw.io/items/the-developer-certificate-of-origin) - -By submitting this email, you acknowledge that you have read and agree to the course policies listed [here](course_policies.html) - diff --git a/kdlp.underground.software/course/fall2023/assignments/final_pres_guide.md b/kdlp.underground.software/course/fall2023/assignments/final_pres_guide.md deleted file mode 100644 index cc99fe11..00000000 --- a/kdlp.underground.software/course/fall2023/assignments/final_pres_guide.md +++ /dev/null @@ -1,54 +0,0 @@ -### Final Presentation Guidelines 🎯 🏹 - -This is your opportunity to demonstrate your skills, -reflect on what you've learned in the course, -and give the instructors feedback. - -### Outcomes: - -* Practice your soft skills -* Reflect on what you have learned -* Demonstrate your technical prowess -* Connect your work to the bigger picture - -### What to present: - -- Course reflection and feedback (10 mins) - - In your own words, what did you learn in this class? - - Try and pick 2-3 important topics - - How did your work on specific assignments build your understanding? - - How does your work fit into the big picture of...? - - Your academic career - - Your software engineering skill development - - Your understanding of open source sofware - - What assignments did you particularly enjoy or not enjoy? - - Explain why your felt that way - - For those you enjoyed, how could you go further? - - For those you did not enjoy, how could we improve? - - Pick an assignment you struggled with: - - How did you overcome your obstacles? - - How will your approach differ if you did it again? - - Pick and assignment you excelled at: - - What made your work exceptional? - - Do you have any advice for future students? - - You must present this section with slides -- Demonstration your work on F1 (10 mins) - - You must present your initial submission unmodified - - You must build, load, test, and unload the module - - You must show the key features of the module - - You must discuss your design choices - - You must present this section from your live terminal/editor -- Questions and live coding challenge (10-15 mins) - - We will spend a couple of minutes on questions about your demo - - Prepare to defend your engineering choices - - Expect to explain the thread and memory safety of your code - - We will give you a small coding problem, such as: - - Fixing a bug in your module - - Adding a small feature to your module -- Total presentation length must be 30-35 minutes - -### Tips: - -- Refer back to your assignment cover letters. -- Do more than summarize assignment descriptions. Synthesize insight in your own words. -- If your initial submission is incomplete, don't panic. You can discuss what's broken and how you will fix it. diff --git a/kdlp.underground.software/course/fall2023/assignments/index.md b/kdlp.underground.software/course/fall2023/assignments/index.md deleted file mode 100644 index 3174f43e..00000000 --- a/kdlp.underground.software/course/fall2023/assignments/index.md +++ /dev/null @@ -1,28 +0,0 @@ -## Pre-requisites -[A(-1)](course_application.md) - -## Assignments - -[A0](A0.md) - -[A1](A1.md) - -[A2](A2.md) - -[A3](A3.md) - -[A4](A4.md) - -[A5](A5.md) - -[A6](A6.md) - -[Fin1](fin1.md) - -[Fin2](fin2.md) - -## Presentations - -[P0](P0.md) - -[P1](P1.md) diff --git a/kdlp.underground.software/course/fall2023/assignments/mid_pres_guide.md b/kdlp.underground.software/course/fall2023/assignments/mid_pres_guide.md deleted file mode 100644 index 128891c6..00000000 --- a/kdlp.underground.software/course/fall2023/assignments/mid_pres_guide.md +++ /dev/null @@ -1,34 +0,0 @@ -**KDLP Fall 2023 Midterm Presentation Guidelines** - -* Overview: - * Short presentation about research into a topic of your choice live during class - * Propose topic and outline ahead of time - * Plenty of room to be creative 🙂 -* Example midpoint presentation topics (pick one of these or propose your own): - * What is the OOM killer? - * Why are there multiple versions of some syscalls with numbers at the end? What issue were they added to address? - * pipe, pipe2, pipe3 - * dup, dup2, dup3 - * etc. - * Why do some syscalls have a version ending in `at`? What issue were they added to address? - * open vs openat - * fstat vs fstatat - * rename vs renameat - * symlink vs symlinkat - * etc - * What does .config do? How does it interact with the build process for the kernel? - * What is physical and virtual memory? Why is there a distinction between the two? - * What is the history of git? How does it fit into the bigger picture of version control software? - * Or pick your own! -* Select a topic and submit your proposal with a short outline as soon as possible. The hard due date is at 23:59 Eastern on 10/31/2023 -* Presentation requirements: - * 10 ± 1 minutes long - * Must include some sort of visual component: - * Slideshow - * Diagrams - * Etc. - * Must include some sort of live demo in the terminal - * Must include discussion of some snippet(s) kernel code that is relevant to your topic - * Present live during class time in the class Jitsi - * Share your screen for terminal + visual elements - * There will be two presentations per day on L16, L17, L18, L19, and L20 diff --git a/kdlp.underground.software/course/fall2023/assignments/setup.md b/kdlp.underground.software/course/fall2023/assignments/setup.md deleted file mode 100644 index aac55ede..00000000 --- a/kdlp.underground.software/course/fall2023/assignments/setup.md +++ /dev/null @@ -1,213 +0,0 @@ -# Introduction and course setup - -Welcome to the class! Setting up a Linux environment is required for the course, and that is your first task should you choose to accept it. -Use this as an opportunity to introduce yourself, get your feet wet with the email patch submission process we will use for this class. - -### Outcomes: -* Set up your Linux environment for the rest of the class. -* Get familiar with git and git patches. -* Get familiar with how to submit to the mailing list (`git send-email`). -* Get familiar with the peer review process, which we will use for the rest of the course. - -### Procedure: - -#### Set up your VM and Linux environment - -##### Get the appropriate software - -* Start downloading the standard ISO image for Fedora for your architecture. - * You probably want the ISO for x86\_64, unless you have an M1 Mac or you are running Linux on ARM, in which case you want the aarch64 ISO. -* This may take a while depending on your internet speed. . - -* Download and install hypervisor software for your operating system - * Windows - [VirtualBox](https://www.virtualbox.org/wiki/Downloads): - * Click the download link for Windows hosts. - * MacOS - [UTM](https://mac.getutm.app/): - * You don't need to buy it from the App Store for (they charge $10 for it there), - there is a free download that is identical but you have to install it yourself (press the download button at the above link). - * Linux - [virt-manager](https://virt-manager.org/): - * You can most likely get it from your system's package manager. - -##### Figure out how many resources to give your VM -* Depending on how powerful your machine is, you can give it more or less. - * Generally, leave at least half of the cores and RAM for your host system. - * The more you can give the VM the better. - * For the class you will need at least 30G of hard-disk space to hold the VM. -* Balance it how you want. Feel free to reach out for specific advice, depending on the specs of your machine. - -##### Set up the VM -* Once the ISO image has finished downloading, you can begin the VM set-up process. -* Depending on the software you downloaded, the exact procedure will vary. - However, you want to create a new virtual machine guest and set the resources to the values you decided to allocate. - Most likely just follow the set-up wizard of your software and it will guide you. - Make sure to create a virtual hard drive for the machine (at least 30G is suggested). -* Attach the ISO image to the machine, this will allow it to boot into the Fedora live environment. -* Start the VM for the first time and it will boot the Fedora ISO image. - -##### Install Fedora to your virtual machine -* Once it boots you will see a grub menu with two options "test this media and install fedora" and "install fedora" - * You can select either one, testing the media will verify that your ISO downloaded correctly, but it will take longer. -* Once the actual operating system boots, the anaconda installer should open automatically. -* Mostly follow the on-screen instructions to install fedora on to your VM's harddrive. -* However, there may be some confusing steps. If you get stuck, there are numerous tutorials online on how to proceed based on the VM software you are using. -* There are two settings you have to adjust before you can start the installation, the root password, and the installation destination. -* Click on root password and setup the root account with a password you will remember then click done. -* Next, this particular step where you have to set the "Installation Destination", depicted below, is especially confusing. - The installation destination icon will have a red warning triangle, to proceed, click on the icon and it will open the screen below. Just click "Done", - you don't need to change anything---it just wants you to confirm the default hard-drive and partitioning selections it has made. - -fedora confusion image - -* Once the installation has finished, Fedora will tell you to reboot. Shut down the machine and remove the ISO image from where you attached it to the VM - (once this is done and you confirmed the installation was successful, you can remove the ISO image from your host machine to free up disk space). -* Boot the VM and you should be greeted with a Fedora login prompt, use the username "root" and then enter the root password you selected during installation - (it won't look like you are typing any characters when you enter the password, but they are being received, and hiding them is just a security measure). -* If you logged in correctly, you should be greeted with a command prompt that looks something like: - -``` -[root@localhost ~]# _ -``` - -If you see this, you have completed the basic installation of Fedora correctly. Congrats! - -##### Post Install Setup -* Set a hostname for the machine: - * Login with your root account. - * Pick a hostname for the machine (e.g. `joels-fedora-vm`) - * Run the command `echo hostname > /etc/hostname` where you replace *hostname* with the name you picked. - * Run the command `hostname --file /etc/hostname` to update it for this boot without needing to restart the machine. - * Log out and the login prompt will include the new hostname before the word login (e.g. `joels-fedora-vm login: _` -* Run a software update: - * Login with your root account. - * Run the command `dnf update -y`, (this may take a while to complete.) - * While you are waiting you can switch to a different console and continue following the steps. - * Press `ctrl+alt+f2` (on some keyboards you might need a function key to press f2). - * You can check back on the update any time by pressing `ctrl+alt+f1`. -* Create yourself a non-root account with sudo permissions. - * Login to the other console as root. - * Pick a username (e.g. `jsavitz`) for your account. - * Run the command `useradd username` where *username* is the username you picked. - * Run the command `usermod -aG wheel username` to give the new user permission to run commands as the superuser (root). - * Run `passwd username` and select a password. - * Log out of root (type `exit` or hit `ctrl+d`). - * Try logging in as your new user with the username and password you selected. - * Verify that you have root access via running `sudo whoami`, you should be prompted for a password - and when you enter your password the `whoami` command should execute and output `root`. - -#### Configure your account for the class email system and send your first email patches - -* Login as your non root user account -* Install `git` and `git-email`. Run `sudo dnf install -y git git-email`. -* Configure your `.gitconfig`, which lives in `~/.gitconfig`, using a text editor (e.g. `nano ~/.gitconfig` or `vi ~/.gitconfig`). - * You can pick your default editor by adding: 

[core]
-        editor = nano # Or which ever editor you prefer
-
- * Set your identity and account information by adding: 
[user]
-		name = Your Name Here
-		email = YOUR_USERNAME@kdlp.underground.software
-[sendemail]
-		smtpUser = YOUR_USERNAME
-		smtpPass = YOUR_PASSWORD
-		smtpserver = kdlp.underground.software
-		smtpserverport = 465
-		smtpencryption = ssl
-
- * Make sure to fill in the fields with your username and password (the ones you use to log into the website) -* Clone [the assignment git repository](https://kdlp.underground.software/cgit/ILKD_assignments/): Run `git clone https://kdlp.underground.software/cgit/ILKD_assignments/`. -* Change directory into the `introductions` folder within the repository. -* There are already introductions from the instructors in there, read them to see an example of what to do. -* Add a file named `firstname_lastname.txt` (e.g. `joel_savitz.txt`), containing an introduction about yourself - * The content can be whatever you want, whether it be why you are taking this class, your favorite ice cream flavor, or a fun fact about yourself. -* Run `uname -a >> firstname_lastname.txt` to append a line with information about the VM environment you set up to the end of your introduction file. -* Make a commit out of your changes. - * By default git will not be tracking changes to newly created files. Add your file to the list that git is tracking with `git add firstname_lastname.txt`. - * Make a commit to save this version of the repository so it can be shared. Run `git commit -s`. - * the `-s` flag makes git include the `Signed-off-by` DCO line for you automatically. - * Git will open an instance of your preferred text editor to let you input a message for the commit. - * Put a title containing a short summay of what you did on the first line (e.g. `Introductions: Added introduction for so and so`). - * Press enter twice and write a more detailed explanation that will act as the body of the commit. - * There should already be a signed off by DCO line for your account at the bottom. If not, add one. - * Save your changes and exit the editor to finish the commit. - * Check to make sure you see the commit and it looks good by running `git log -p`. - * Your new commit should be the top most one, and you should see the title, message, DCO, and difference view containing the changes. - * If there is more output than can fit on one screen, git will open a scrolling view that you can maneuver up and down within using the arrow keys. Press `q` to go back to the terminal. -* Create a patch series with a cover letter out of your commit. - * Run the command `git format-patch -1 --cover-letter -v1 --rfc` - * `-1` means include the most recent 1 commit - * `--cover-letter` means generate a cover letter for the whole patch series with a summary - * `-v1` means mark this as the first version of this patch set - * `--rfc` means mark these patches as a request for comment. This is required for all intial submissions since they will be recieving peer feedback. - * Git will generate two `.patch` files for you. - * Edit the file `v1-0000-cover-letter.patch` and write your cover letter. - * Follow the directions in the assignment submission guidelines - * Don't forget to add the DCO `Signed-off-by` line at the end of they body. -* Send your patches to the class mailing list. - * Run the command `git send-email --to=introductions@kdlp.underground.software v1*.patch` - * Each assignment will have its own special address to send submissions to. In this case it is `introductions@kdlp.underground.software` - * the expression `v1*.patch` will be exapanded by the shell into all file names matchin that pattern (any file whose name starts with `v1` and ends with `.patch`) - * Git send email will prompt you to ask whether it should send the emails. Type `a` and hit enter to send them all. - * If it is successful, the output should end with `Result: 250` which indicates that the server accepted your emails. - * If it does not work. Do not hesistate to reach out and we can help with troubleshooting. - -#### Check your work -* While Logged in as your non root user account -* Install `mutt` a terminal program for viewing email. Run `sudo dnf install -y mutt` -* Configure your `.muttrc` which lives in `~/.muttrc`, using a text editor (e.g. `nano ~/.muttrc` or `vi ~/.muttrc`). - * Set your identity and account information by adding: 
set realname="Your Name Here"
-set my_username="YOUR_USERNAME"
-set my_password="YOUR_PASSWORD"
-
- * Fill in your credentials that you use for the website and your full name. - * Set some sane defaults, and configure the server addresses by adding: 
set spoolfile=
-set record=
-set folder=
-set sort=threads
-set from="$my_username@kdlp.underground.software"
-set smtp_url="smtps://$my_username:$my_password@kdlp.underground.software:465"
-push "<change-folder>pops://$my_username:$my_password@kdlp.underground.software:995"\n
-
- * Copy that block in verbatim, mutt will substitute the variables you set in the previous declarations for you. -* Open the email list by running the `mutt` command. You can quit by pressing `q` or hitting `ctrl+c` -* You will see a list of email threads navigate up and down with the arrow keys or `j` and `k` -* You should see a welcome to the email system message at the top and then all subsequent emails in chronological order. -* Press enter to view which message you have highlighted and press `q` to exit back to the main screen. Press the spacebar to scroll to the next page within the email. -* You should see your patch series including your introduction in the list of messages. - * If you do, congratulations, you successfully completed the setup. You can shut down the VM and go do something else :) - * If not, feel free to reach out with questions we are happy to help. -* Once you are finished, you can shut down the vm with the command `sudo poweroff`. - * Before running it, you should check back in on the updates running in `tty1`. Press `ctrl+alt+f1` and make sure that the update command is done and you are back in a shell. - * If it is finished, switch back to tty2 (`ctrl+alt+f2`) and run the poweroff command. - -#### Optional VM Configuration - -* Set up SSH access: - * While logged in to the machine on the tty, run the command `ip -br a | grep UP`, read the ip address listed there. - * On your host machine, open a terminal and try running `ssh username@ip-address` where *username* is the username you picked earlier and the *ip-address* is the one you just found. - * The first time you connect to a new server via ssh, ssh will not know the server and will say it cannot establish the authenticity of the machine you are connecting to. - * It will ask if you are sure you want to continue connecting. Type `yes` and hit enter. - * You will be prompted for your password. This is the password for your account on the VM. - * After you enter it you will be able to access your vm from your host machine. - * You will be accessing a shell exactly like the tty you get by logging in within the vm window. - * You can log out by typing `exit` and hitting enter, or pressing `ctrl+d` just like the tty. - * If you want to obviate the need to enter the password every time you log in, you can set up SSH-keys. - * Open a terminal on your host machine (`cmd.exe` on windows) - * If you have never set up SSH-keys (you might have done it already if you use ssh for github or gitlab): - * Run the command `ssh-keygen` in a terminal on your host machine and accept the default values. - * If you already have SSH-keys or you just created them using the step above: - * If you are not on Windows, run `ssh-copy-id username@ip-address` to copy your keys to the VM. - * If you are on Windows, run `type $env:USERPROFILE\.ssh\id_rsa.pub | ssh username@ip-address "cat >> .ssh/authorized_keys"` - * You will be prompted for your vm password one final time, enter it, and the command should finish. - * Try logging in with `ssh username@ip-address` and you should not need to enter your password ever again. -* Make the VM headless: - * Power down the machine with `sudo poweroff` - * Open the VM Settings and disable the graphical output, the exact settings depends on the VM software you are using. - * Most of the time you will be able to keep the graphics off, since you already set up SSH access. - * This will lighten the load on your computer since it won't have to render the GUI and will make the VM run faster. - - - -### What to submit: -* Generate a patchset from your commit with a cover letter. -* Send the patchset to introductions@kdlp.underground.software. - -[Submission Guidelines](../policies/submission_guidelines.md) diff --git a/kdlp.underground.software/course/fall2023/index.md b/kdlp.underground.software/course/fall2023/index.md deleted file mode 100644 index 14b494b0..00000000 --- a/kdlp.underground.software/course/fall2023/index.md +++ /dev/null @@ -1,52 +0,0 @@ -## Fall 2023 Course - -### Course Summary and Purpose - -We aim to introduce students to the concepts, tools, and framework necessary for independent work on the Linux kernel. - -### Course description - -Linux is the most widely used operating system in the world. The core software component of the Linux operating system is the kernel. A couple of its roles include managing hardware interactions, virtualizing system resources, and enforcing security constraints. In effect, Linux kernel powers almost all of the world’s top supercomputers, android phones, and an innumerable variety of other computers. This course will introduce students to Linux kernel development by focusing on device driver development. This will give students hands-on experience working with internal Linux kernel APIs and provide an overview of some of the core features and components of the kernel. Gaining an understanding of the inner workings of the operating system and how to make changes to it will give students an invaluable perspective on how their computers work behind the scenes that will reveal a new layer of understanding to apply to any future software engineering practice. - -### Course objectives - -Students will: - -* Stretch their C programing skills to the limit - -* Create, develop, and test interesting and creative Linux kernel modules - -* Develop the ability to autonomously participate in the open source software community - -* Learn how to give and recieve feedback on code patches - -* Demonstrate the ability to explain your code and your knowledge of Linux in your own words - -### Required Materials - -* A computer that can run a virtual machine to give you access to a Linux environment. - -* Patience, determination, and an ability to focus on a potentially frustrating task for sustained periods of time. - -### Course Schedule - -|#| week starting (Sunday date) |S|M|T|W|R|F|S|Assignment Due| -|--|--|--|--|--|--|--|--|--|--| -|00| 03 Sep||| ||[L00](lectures/L00.md)|||| -|01| 10 Sep|||[L01](lectures/L01.md)||[L02](lectures/L02.md)|||[setup](assignments/setup.md)| -|02| 17 Sep|||[L03](lectures/L03.md)||[L04](lectures/L04.md)|||[E0](assignments/E0.md) | -|03| 24 Sep|||[L05](lectures/L05.md)||[L06](lectures/L06.md)|||[P0](assignments/P0.md) | -|04| 01 Oct|||[L07](lectures/L07.md)||[L08](lectures/L08.md)|||[E1](assignments/E1.md) | -|05| 08 Oct|||[L09](lectures/L09.md)||[L10](lectures/L10.md)||| | -|06| 15 Oct|||[L11](lectures/L11.md)||[L12](lectures/L12.md)|||[P1](assignments/P1.md) | -|07| 22 Oct|||[L13](lectures/L13.md)||[L14](lectures/L14.md)|||[E2](assignments/E2.md) | -|08| 29 Oct|||[L15](lectures/L15.md)||[L16](lectures/L16.md)|||[Midpoint Presentation](assignments/mid_pres_guide.md) | -|09| 05 Nov|||[L17](lectures/L17.md)||[L18](lectures/L18.md)|||[P2](assignments/P2.md) | -|10| 12 Nov|||[L19](lectures/L19.md)||[L20](lectures/L20.md)||| | -|11| 19 Nov|||[L21](lectures/L21.md)|| ||| | -|12| 26 Nov|||[L22](lectures/L22.md)||[L23](lectures/L24.md)||| | -|13| 03 Dec|||[L24](lectures/L24.md)||[L25](lectures/L25.md)|||[F0](assignments/F0.md) | -|14| 10 Dec|||[L26](lectures/L26.md)||[L27](lectures/L27.md)|||[F1](assignments/F1.md) | -|15| 17 Dec|||[L28](lectures/L28.md)|| |||[Final Presentation](assignments/final_pres_guide.md)| - -[Course Policies](policies/course_policies.md) | [Submission Guidelines](policies/submission_guidelines.md) | [Peer Review Process](policies/peer_review.md) diff --git a/kdlp.underground.software/course/fall2023/lectures/L00.md b/kdlp.underground.software/course/fall2023/lectures/L00.md deleted file mode 100644 index b702d345..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L00.md +++ /dev/null @@ -1,80 +0,0 @@ -# Lecture 0 - 7 September 2023 - -## Topics covered: - -* Introductions -* Course format -* Course policies -* Breakdown of grading -* Overview of assignments -* Next steps - -### Introductions - -See [this](https://kdlp.underground.software/who.md) page - -### Course format - -Flipped classroom style. Watch lectures at home and interact in class. - -### Course policies - -See [this](https://kdlp.underground.software/course/fall2023/policies/course_policies.md) page - -#### Breakdown of grading - -The formatting was incorrect on the course page during class. It has since been fixed, but it has been replicated here for your convinience: - -Student grades will be weighted according to the following scheme: - -| Category | Percentage | -|--|--| -| Programs | 30% | -| Exercises | 20% | -| Midpoint Presentation | 15% | -| Final Assignment + Presentation | 30% | -| Participation | 5% | - -Students will be given a letter grade according to the following this scheme: - -|Letter grade|Percentage range| -|--|--| -|A |90 ~ 100| -|A-| 85 ~ 89.99| -|B+| 80 ~ 84.99| -|B |75 ~ 79.99| -|B-| 70 ~ 74.99| -|C+| 65 ~ 69.99| -|C |60 ~ 64.99| -|C-| 55 ~ 59.99| -|D+| 50 ~ 54.99| -|D |40 ~ 49.99| -|F |below 40| - -A (4.0), A- (3.7), B+ (3.3), B (3.0), B- (2.7), C+ (2.3), C (2.0), C- (1.7), D+ (1.3), D (1.0), F (0.0) - -### Overview of assignments - -See the table at the bottom of [this page](https://kdlp.underground.software/course/fall2023/index.md). - -### Before next class - -* Watch the vidoes in [this playlist](https://www.youtube.com/playlist?list=PLD7h3bLB5mqtPe-I1z2j4-Xao_5yK_N2H) (approx 5m total) to learn how to get access to your course account and join the matrix server to ask questions. - -* Read this assignment page: [setup](https://kdlp.underground.software/course/fall2023/assignments/setup.md) - -* Read the [submission guidelines](https://kdlp.underground.software/course/fall2023/policies/submission_guidelines.md) - -* Watch one of the following playlists (approx 10m total) to learn how to set up a virtual machine and install a linux environment in it suitable for this class - * Windows users follow [this tutorial](https://www.youtube.com/playlist?list=PLD7h3bLB5mqsfHZkNlyYG177Y6oKP63Vh) - * Linux users follow [this tutorial](https://www.youtube.com/playlist?list=PLD7h3bLB5mquEdrZJ6xG3izW8l30WqIXs) - * Mac Users on apple silicon follow [this tutorial](https://www.youtube.com/playlist?list=PLD7h3bLB5mquN8-yTutftOWS8jQipZvOZ) - * Mac Users on intel macs, follow the windows tutorial, but download and install the mac version of virtual box instead - -* Watch [this tutorial](https://youtu.be/Ub62duJEPhQ) after you finish the installation of your VM to learn how to configure it, and get step by step instructions for completing the introduction assignment. - -### The introduction assignment is due at 11:59PM next tuesday (12 September 2023) - -Please get in touch if you are struggling with anything or have any questions. -If you can use matrix it is preferred since any question that you ask and get answered benefits the whole class, -but the instructors can also be reached via email: [Charlie](mailto:cmirabil@redhat.com) [Joel](mailto:jsavitz@redhat.com) diff --git a/kdlp.underground.software/course/fall2023/lectures/L01.md b/kdlp.underground.software/course/fall2023/lectures/L01.md deleted file mode 100644 index ea25ea82..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L01.md +++ /dev/null @@ -1,55 +0,0 @@ -# Lecture 1 - 12 September 2023 - -## Topics covered: - -* Questions on setup -* Introduction to E0 & P0 -* C system programing interactive demo - - -### Questions on setup - -* For clarification: git repositories are read-only for students - -* If this setup or C programing is overwhelming, this course may not be right for you - -### Introduction to E0 & P0 - -See each assignment page for more details. - -**[E0](https://kdlp.underground.software/course/fall2023/assignments/E0.md) is due at 11:59PM next tuesday (19 September 2023)** - -See [this video](https://www.youtube.com/watch?v=zYB72Rnz3TA) for more guidance on how to complete E0. - -**[P0](https://kdlp.underground.software/course/fall2023/assignments/P0.md) is due at 11:59PM a week from next Tuesday (26 September 2023)** - -See [this video](https://www.youtube.com/watch?v=JqljsuVaUIU) for more guidance on how to complete P0. - -### C system programing interactive demo - -0. the most basic C program possible: `main(){}` - -1. hello world - -2. What is a system call? - - * Synchronously ask Linux to do something for you as a user program - - * Student examples: open, read, write, close - - * Use these with file descriptors to do file I/O without `FILE *` - - * Use `man 2 xxx` to learn about system call `xxx` - -3. Working with processes - - * [execve](https://man7.org/linux/man-pages/man2/execve.2.html): transform one process into another - - * [fork](https://man7.org/linux/man-pages/man2/fork.2.html): split one process in two - - * [wait](https://man7.org/linux/man-pages/man2/wait.2.html): wait for a child process - -### tl;dr - -** GET STARTED ON [P0](https://kdlp.underground.software/course/fall2023/assignments/P0.md) EARLY!! ** - diff --git a/kdlp.underground.software/course/fall2023/lectures/L02.md b/kdlp.underground.software/course/fall2023/lectures/L02.md deleted file mode 100644 index b9e723fe..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L02.md +++ /dev/null @@ -1,80 +0,0 @@ -# Lecture 2 - 14 September 2023 - -## Topics covered: - -* Course roster is stabilizing -* Recent homework submissions -* Introduction to git - -### Course roster is stabilizing - -* End of UMass Lowell Fall 2023 add drop period is approaching: September 19 for both -[undgraduates](https://www.uml.edu/registrar/calendars/2023-fall-undergrad.aspx) and -[graduates](https://www.uml.edu/registrar/calendars/2023-fall-grad.aspx) - - * Subsequent withdrawal from the course is not as painless - -### Recent homework submissions - -* Nine (9) people made an initial submission on Tuesday (12 September 2023), which is excellent - -* In general, key information for students to get started is in the lecture notes and students -should read them carefully - -* If any student has not registered on the KDLP website, please do so ASAP - - * Contact the KDLP team if there are any issues with setting up your credentials - -* This registration provides students their credentials for: - - * The course dashboard for assignment feedback - - * The course chat system -- a self-hosted [matrix](https://matrix.org) instance - - * The course email system for assignment submission and review - -### Introduction to git - -* We used [this](https://kdlp.underground.software/course/slides/git.html) slide deck - -* Git is distributed [version control](https://en.wikipedia.org/wiki/Version_control) software - -* [Git](https://git-scm.com/) is not [GitHub](https://github.com) - - * GitHub is one implementation of an interface for git - - * There are variously featured alternatives, such as [GitLab](https://gitlab.com/), [Bitbucket](https://bitbucket.org/), and [cgit](https://git.zx2c4.com/cgit/) - - * The KDLP team maintain a custom-themed cgit instance [here](https://kdlp.underground.software/cgit) - -* Git is built on a [tree-like data structure](https://en.wikipedia.org/wiki/Tree_(data_structure)) that contains the entire change history of a project - -* **Git proficiency is of the most useful and valuable software engineering skills a computer science student can learn in preparation to enter the industry** - -* Charlie did a demo in the terminal. Here's a rough outline of the various git commands he covered: - - * `git clone`: Cloning the [ILKD_assignments](https://kdlp.underground.software/cgit/ILKD_assignments/) repository - - * `git commit`: Committing new local changes to the repository - - * `git merge`: Combining two change histories into one - - * `git reset`: Undoing previous changes, and going nuclear with `--hard` - - * `git rebase`: Rewriting the git history - - * (single commit rewrite cases can be handled with `git commit -amend`) - - * When things don't go right, you may have to resolve merge conflicts by manually editing source files and re-committing - - * This should not be something you have to do for this course, however for anyone who is interested, here is an article on [merge conflicts](https://css-tricks.com/merge-conflicts-what-they-are-and-how-to-deal-with-them/) - -### Important Reminders - -**[E0](https://kdlp.underground.software/course/fall2023/assignments/E0.md) is due at 11:59PM next tuesday (19 September 2023)** - -See [this video](https://www.youtube.com/watch?v=zYB72Rnz3TA) for more guidance on how to complete E0. - -**[P0](https://kdlp.underground.software/course/fall2023/assignments/P0.md) is due at 11:59PM a week from next Tuesday (26 September 2023)** - -See [this video](https://www.youtube.com/watch?v=JqljsuVaUIU) for more guidance on how to complete P0. diff --git a/kdlp.underground.software/course/fall2023/lectures/L03.md b/kdlp.underground.software/course/fall2023/lectures/L03.md deleted file mode 100644 index 51737987..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L03.md +++ /dev/null @@ -1,57 +0,0 @@ -# Lecture 3 - 19 September 2023 - -## Topics covered: - -* Today is add drop date -* Continue syscall demo -* Live student submission review - -### Today is add drop date - -* Just a friendly reminder - -### Continue Syscall Demo - -* Charlie demonstrated several system calls needed to complete `P0`, with a focus on dealing with file descriptors - -* Take note that the `struct stat` for a file contains a `mode` field that specifies file permissions - -* Use the [`dup(2)`](https://man7.org/linux/man-pages/man2/dup.2.html) family to create unnamed pipes on your system - -* When working with C strings in the kernel, we recommend passing around a pointer and a length pair rather than relying on null-termination - - * Once must take extra precaution to avoid buffer overflows in the kernel - * While in userspace a buffer overflow is harmful, in the kernel it can be catastrophic to a system, and in a production environment, devastating to an organization - - * We will spend more time discussing security concerns later on - -* Note that file descriptors are integers that index into a table of file descriptions. - -* [`dup(2)`](https://man7.org/linux/man-pages/man2/dup.2.html) uses the file descriptor table to connect two file descriptors to each other. - -* While `dup()` takes an fd to duplicate, `dup2()` takes a second existing fd to overwrite as the other end of the pipe. - - -### Live student submission review - -* Be sure to submit all patches when you run `git send-email` - -* Be sure to locally `git am` your patchset on a repo after running `git checkout -b test origin/master`. This will do the same thing that we will do to test your patchset. - -* Be sure to include the DCO with your cover letter. - -## Before next class - -* Watch [this video](https://youtu.be/dcxi578qOvI) (25m) that gives a brief overview of the who why where and what of linux, and discusses some of the pieces that make up a functioning system and how they fit together. - -* Watch [this video](https://youtu.be/qcue__Z2n6w) (25m) that explains and demonstrates some of the most common commands used on the commandline that you will want to be familiar with for the rest of the semester. - -### Important Reminders - -**[E0](https://kdlp.underground.software/course/fall2023/assignments/E0.md) is due at 11:59PM the day of this lecture (19 September 2023)** - -See [this video](https://www.youtube.com/watch?v=zYB72Rnz3TA) for more guidance on how to complete E0. - -**[P0](https://kdlp.underground.software/course/fall2023/assignments/P0.md) is due at 11:59PM next Tuesday (26 September 2023)** - -See [this video](https://www.youtube.com/watch?v=JqljsuVaUIU) for more guidance on how to complete P0. diff --git a/kdlp.underground.software/course/fall2023/lectures/L04.md b/kdlp.underground.software/course/fall2023/lectures/L04.md deleted file mode 100644 index 074a060b..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L04.md +++ /dev/null @@ -1,86 +0,0 @@ -# Lecture 4 - 21 September 2023 - -## Topics covered: - -* Miscommunication regarding these notes -* Questions on P0 -* Linux crash course and minimal distro - -### A time saver for vim users - -Append these two lines to your `~/.vimrc` to auto-highlight whitespace errors in your editor: - -``` -:highlight ExtraWhitespace ctermbg=red guibg=red -:match ExtraWhitespace /\s\+$/ -``` - -### Miscommunication regarding these notes - -* Reading through these lecture notes and following any instructions given there is essential for students to succeed in this course. -* We apologize for the lack of emphasis on this -* We did things a bit differently in the past -* As such, we will relax our expectations of what videos you have watched so far. -* If you have not done so, please go through all of [L0](L00.md), [L1](L01.md), [L2](L02.md), [L3](L03.md) -* We will announce lecture notes and any subsequent errata via the course matrix server -* In case anyone is having trouble finding a good client, try [element](https://element.io/). It's got mobile, web, and desktop apps. - -### Questions on P0 - -* How do students make the commits while writing the code to later generate the patches? - * Commit early and commit often when developing your shell - * When in doubt about whether to commit, just do it - * When your repo is in a state where your program builds and correctly implements stage `N`, make sure to make a commit and put a note about completing `N` in the log - * Later, when you are preparing your work for submission, you will squash everything between these commits into single commits implementing a stage - * Note that squash here is a technical term referring to the [`git-rebase`](https://git-scm.com/docs/git-rebase) option - * You will run `git format-patch ...` on the branch containing only these squashed commits - * This is a great opportunity to practice `git rebase` -* What functions can we use? - * When P1 specifies a syscall for a scenario, you must use that - * Beyond that, feel free to use anything provided by libc -* Do we need to check for memory leaks? - * Yes - * We will deduct credit from assignments for any memory leaks - * Use [valgrind](https://valgrind.org/) to find the problems - * Memory leaks are not as severe as build or runtime failures, which merit no credit - * Try to fix these, but don't let this stop you from submitting if you are very close to the deadline - -### Linux crash course and minimal distro - -* We started with the [Linux Crash Course](https://kdlp.underground.software/course/slides/linux_crash_course.html) slide deck. -* First, we gave the general what, where, why, and who of Linux - * Free and open source operating system kernel and ecosystem - * Running on many systems large and small, fast and slow, distributed and centralized - * Many pre-packaged combinations of system components capable of running user applications are available - * We call one of these a Linux distribution, or distro for short. - * Some examples are Fedora, Ubuntu, Arch Linux, RHEL, Puppy, and TAILS -* Having motivated our discussion of system components, we proceeded to discuss each one briefly - * We quickly went through the bootloader, kernel, C standard library, shared libraries, storage configuration, and filesystem hierarchy layout - * As we were about to proceed to the demo, we didn't go into great detail - * Take note of the availability of manpages: run `man man` and read through the description. -* After finishing the slide deck, we build a minimal Linux distribution live - * Using a ruthlessly minimal `.config` file, we built the kernel in noticably less time than it took to clone, even with the `--depth=1` option. Barely minutes. - * Then, we tested this kernel with `qemu-system`, generating a kernel panic as expected - * This is because the kernel attempts to start an inital userspace process and no such thing exists on the system - * On many Linux systems, systemd is this first "init" process, and it is given PID `1` - * We then created a root filesystem and a "hello world" init process - * This native assembly simply prints hello world - * Take note that the entry point is `_start` and not `main` - * Normally this is hidden as `libc` does various init things between `_start` and the user-defined `main` - * We used `strace` to see how our little app talks to the kernel - * In order to pass the app binary into the kernel, we use `cpio` to create an archive usable as an inital ram disk for our system, which allows the kernel to run the app as an init process - * This is generally refered to by the abbreviation `initrd`. - * `initrd` is an area of RAM that provides a storage device interface to the rest of the kernel. -* Since there were no questions, we attempted to make our minimal Linux distro interactive - * We implemented a parameter allowing a user to specify a string to print after "hello" instead of "world" - * The feature started working right in time for the end of class - -## Before next class - -* Make sure you have carefully read through [L0](L00.md), [L1](L01.md), [L2](L02.md), and [L3](L03.md]) and seen all the provided video content. - -### Important Reminders - -**[P0](https://kdlp.underground.software/course/fall2023/assignments/P0.md) is due at 11:59PM next Tuesday (26 September 2023)** - -See [this video](https://www.youtube.com/watch?v=JqljsuVaUIU) for more guidance on how to complete P0. diff --git a/kdlp.underground.software/course/fall2023/lectures/L05.md b/kdlp.underground.software/course/fall2023/lectures/L05.md deleted file mode 100644 index 974fbc9b..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L05.md +++ /dev/null @@ -1,110 +0,0 @@ -# Lecture 5 - 26 September 2023 - -## Topics covered: - -* Questions on P0 -* kernelspace vs userspace - -### Questions on P0 - -* To clarify, you are to submit the following email patch files: - * Email patch file 0 is your cover letter - * Patch 1 adds your Makefile, creating `p0/firstname_lastname/Makefile` - * Each subsequent patch `N` implement step `N-2` listed on - [P0](/course/fall2023/assignments/P0.md). E.g, patch 5 implements step 3 - - -### Kernelspace vs userspace - -#### Fundamental difference: CPU privilege level at a given time - -* On a Linux system, when the CPU is executing code in a fully privileged mode, we say that the CPU is executing the code in kernelspace - -* On a Linux system, When the CPU is executing code at a restricted privilege level, we say that the CPU is executing the code in userspace - -#### What does CPU privilege enable? -* Set behavior on [trap](https://en.wikipedia.org/wiki/Interrupt#Terminology) -* Set behavior on [interrupt](https://en.wikipedia.org/wiki/Interrupt) - * A quick explanation of [both](https://stackoverflow.com/questions/3149175/what-is-the-difference-between-trap-and-interrupt) -* Set virtual-physical address mappings, i.e. configure [page tables](https://en.wikipedia.org/wiki/Page_table) -* Execute privileged instructions - * See [this table](https://www.felixcloutier.com/x86/) as an x86 instruction reference - * We discussed [RDMSR](https://www.felixcloutier.com/x86/rdmsr) as an example - * Unpriveleged execution of this instruction [raises](https://www.felixcloutier.com/x86/rdmsr#protected-mode-exceptions) the `#GP(0)` [CPU exception](https://wiki.osdev.org/Exceptions). - * This is a [General Protection Exception](https://wiki.osdev.org/Exceptions#General_Protection_Fault) - * There are three types of CPU exceptions: [faults](https://wiki.osdev.org/Exceptions#Faults), [traps](https://wiki.osdev.org/Exceptions#Traps), and [aborts](https://wiki.osdev.org/Exceptions#Aborts). - * The website linked in the previous bulled, wiki.osdev.org, has a huge about of information on this subject for the interested reader -* We experimented bit of assembly code to test out the exception handling codepaths of the Linux kernel. - -#### Userspace Demo - -Here's a short AT&T-style x86 assembly file we can use to generate a binary that will attempt to execute a privileged instruction: - -``` -global _start ; declare the _start symbol to have exernal linkage for visibility of linker -_start: ; the true entry point for an x86 executable program - rdmsr ; execute the RDMSR instruction -``` - -Build the object file `rdmsr.o` from [`rdmsr.src`](https://kdlp.underground.software/cgit/priv_rdmsr_demo/tree/rdmsr.src) with: - -`as -o rdmsr.o rdmsr.src` - -Create the linked executable binary `rdmsr` from `rdmsr.o` with: - -`ld -o rdmsr rdmsr.o`. - -Invocation of this binary by `./rdmsr` should trigger a protection fault. - -More information on the [`#UD` Invalid Opcode](https://wiki.osdev.org/Exceptions#Invalid_Opcode) exception. - -#### Kernelspace Demo - -With a small kernel module, we can get Linux to run the same instruction in kernelspace: - -``` -#include -#include -MODULE_LICENSE("GPL"); -static int priv_demo_init(void) { - /* arbitrary poison values */ - int result_lower_32 = -0xAF, result_upper_32 = -0xBF; - pr_info("EDX:EAX := MSR[ECX];"); - asm ( "rdmsr" - : "=r" (result_upper_32), "=r" (result_lower_32) : : ); - pr_info("rdmsr: EDX=0x%x, EAX=0x%x\n", - result_lower_32, result_upper_32); - return 0; -} -static void priv_demo_exit(void) { - pr_info("rdmsr exiting"); -} -module_init(priv_demo_init); -module_exit(priv_demo_exit) -``` - -We can build this with the same Makefile as shown [here on the E2 page](https://kdlp.underground.software/course/fall2023/assignments/E2.md). - -#### Fully Automated demo - -We created fully automated demo of privileged and unprivileged instruction execution. -To acquire and run this demo, enter your VM and run `git clone https://kdlp.underground.software/cgit/priv_rdmsr_demo/` and run `make` inside the directory. - -## Before next class - -* Watch this video about [strace and syscalls](https://youtu.be/-2jstkyudnQ) - -* Read through the [E1](E1.md) assignment page and check out this [completed demo](https://kdlp.underground.software/cgit/e1_demo/) - -### Important Reminders - -|[P0](/course/fall2023/assignments/P0.md) requirement| Deadline Time | Deadline Date | -|--|--|--| -|**initial submission**|11:59PM|Tuesday 26 September 2023| -|**peer review**|11:59PM|Wednesday 27 September 2023| -|**final submission**|11:59PM|Thursday 28 September 2023| - -
-
- -# *ALL P0 SUBMISSIONS GO TO: [programming0@kdlp.underground.software](mailto:programming0@kdlp.underground.software)* diff --git a/kdlp.underground.software/course/fall2023/lectures/L06.md b/kdlp.underground.software/course/fall2023/lectures/L06.md deleted file mode 100644 index 68a2716b..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L06.md +++ /dev/null @@ -1,50 +0,0 @@ -# Lecture 6 - 28 September 2023 - -## Topics covered: - -* P0 status -* continued: kernelspace vs userspace demo - -### P0 Status - -We received 6 initial submissions by the initial P0 deadline and assigned peer reviews. - -### Kernelspace vs userspace demo - -We took another look at the demo we posted on [L05](L05.md) after class. - -That demo can be found -[here](https://kdlp.underground.software/cgit/priv_rdmsr_demo/) and obtained by running: - - git clone https://kdlp.underground.software/cgit/priv_rdmsr_demo - -Ensure that you are comfortable with some of the introductory details -we discussed in [L05](L05.md). - -Recall from [L05](L05.md) that a trap is a type of CPU exception. - -We browsed the source for the Linux implementation of trap handling to understand the codepath that executes when the user executes the "UD2" instruction and prints a message to the kernel ring buffer (`dmesg`). - -Th address of the handler for this exception is defined in -[arch/x86/kernel/traps.c](https://elixir.bootlin.com/linux/v6.5.5/source/arch/x86/kernel/traps.c), as -[`exc_invalid_op`](https://elixir.bootlin.com/linux/v6.5.5/source/arch/x86/kernel/traps.c#L336). -Elsewhere, the corresponding row of the -[IDT](https://wiki.osdev.org/IDT) -is set to this address, so when the exception is generated, -[`handle_invalid_op`](https://elixir.bootlin.com/linux/v6.5.5/source/arch/x86/kernel/traps.c#L292) is called. - -If you are interested in the IDT then may also be interested in the -[GDT](https://wiki.osdev.org/GDT). - -Linux implements a lot of x86-specific IDT related code in -[arch/x86/kernel/idt.c](https://elixir.bootlin.com/linux/v6.5.5/source/arch/x86/kernel/idt.c). - -## Before next class - -* Watch this video introducing [bpftrace](https://youtu.be/T2n1Eg76zCY) - -* Get started on [E1](https://kdlp.underground.software/course/fall2023/assignments/E1.md) and check out this [completed demo](https://kdlp.underground.software/cgit/e1_demo/) - -### Important Reminders - -Initial submission for [E1](https://kdlp.underground.software/course/fall2023/assignments/E1.md) is due Tuesday 3 October 2023. We suggest that you get started early. diff --git a/kdlp.underground.software/course/fall2023/lectures/L07.md b/kdlp.underground.software/course/fall2023/lectures/L07.md deleted file mode 100644 index b6a861fc..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L07.md +++ /dev/null @@ -1,425 +0,0 @@ -# Lecture 7 - 3 October 2023 - -## Topics covered: - -* Basics of kernelspace -* Tracing with ftrace -* Syscalls in kernel source -* Tracing with bpftrace - -### Basics of kernelspace - -To begin, we used parts of the [Kernel Modules and Device Drivers](https://kdlp.underground.software/course/slides/modules_drivers.html) slide deck. - -* The slides are a little bit out of sync with how we have re-arranged the course and we have not yet reached device driver development. - -* The last three slides were the most relevant, however students may be interested in taking a look at the rest of it. - -* The kernel uses a small, fixed-size stack, compared to the larger, extendable stack used by userspace programs. - -* The C library, itself being a userspace program, is not available in kernelspace. Instead, many -- but importantly not all -- are implemented within the kernel. - -* For example, the IEEE754 floating point storage type that we all know and love from userspace C programming is entirely banned from the kernel. - -* The reason is that when CPU switches between kernelspace and userspace, it has to save and restore it's execution state to remember where it left off, and saving and restoring the floating point registers is considered to be too much overhead. - -* The kernel uses a different range of the address space than userspace. On x86_64 systems, the virtual address space is generally split in half - -#### The most important takeaway: kernel code is **reentrant** - -* Definition: A computer program is considered **reentrant** if and only if multiple concurrent executions of the same program always run correctly. - -* Further information can be found on the [reentrancy](https://en.wikipedia.org/wiki/Reentrancy_(computing)) Wikipedia page. - -* Assume that *any* line of code in the kernel can be running at *any* time with *any* number of concurrent executions of the same code. - -### Tracing with ftrace - -Ftrace is mounted as tracefs file system, usually at `/sys/kernel/debug/tracing` - -The kernel source has some -[documentation](https://www.kernel.org/doc/Documentation/trace/ftrace.txt) - -The demo in class was done approximately like so: - -0. Ensure tracing is off and wipe the buffer: - - echo -n > trace - echo 0 > tracing_on - -0. Select the function_graph tracer instead of nop tracer - - echo function_graph > current_tracer - -0. Turn tracing on and back off again - - echo 1 > tracing_on - echo 0 > tracing_on - -0. Take a look at the output - - {one of: less,nano,vim} trace - -The events recorded in this file all took place between enable and disabling tracing. There should be a large number of lines in this file. - -### Syscalls in kernel source - -This file defines the `SYSCALL_DEFINE*` macros: - -[/include/linux/syscalls.h](https://elixir.bootlin.com/linux/latest/source/include/linux/syscalls.h) - -This will quickly locate the syscall named `$SYSCALLNAME` in the kernel: - - git grep '^SYSCALL_DEFINE.($SYSCALLNAME' - -e.g: `^SYSCALL_DEFINE.(read'` - -### Tracing with bpftrace - -bpftrace is a scripting language that implements a frontend for the BPF Linux subsystem. - -[BPF](https://en.wikipedia.org/wiki/Berkeley_Packet_Filter) (formerly eBPF) is a Linux subsystem that implements a virtual machine to quickly insert verified BPF bytecode programs while the kernel is running. - -The bpftrace syntax is based on -[awk](https://en.wikipedia.org/wiki/AWK), -a small scripting language for rapidly processing text files using very simple programs. - -Here's a complete hello world executable script in awk: - - #!/bin/awk -f - - BEGIN { printf("hello, World!\n"); } - -You can also run this directly in the shell: - - [root@kdlp ~]# awk 'BEGIN { printf("hello, World!\n"); }' - -This also valid bpftrace (you must `dnf install bpftrace`): - - [root@kdlp ~]# bpftrace -e 'BEGIN { printf("hello, World!\n"); }' - Attaching 1 probe... - hello, World! - -Use this if you run into problems with bpftrace: -[Great Reference Guide](https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md) - -Buy this to take your BPF knowledge to the next level: -[The BPF Bible: how to become one with the kernel](https://www.amazon.com/Performance-Tools-Addison-Wesley-Professional-Computing/dp/0136554822) - - -**Cscope Interlude** - -With the vim cscope plugin and cscope installed on your system: - -Run: `cscope -R -k` in the root of your kernel tree to create your cscope database. - - -Within vim you can use `cs find g struct cs find t 'struct whatever {'` to find the definition of that structure. You can also do this without vim by directly running `cscope` in your kernel tree. - - -**A Quick Example** - -List all available kprobes that trace functions containing the characters xxx: - - bpftrace -l k:*xxx* - -On every invocation of the `xxx` kernel function, print the command name string and the pid for the current task on the CPU (see: -[task_struct](https://elixir.bootlin.com/linux/latest/source/include/linux/sched.h#L738)). - - bpftrace -e 'k:xxx { printf("%s/%d\n", comm, pid) ; }' - -**A Final Example** - -First, we'll need a little program to trace: - -``` -[root@kdlp ~]# cat > foo < x -cat x -EOF -[root@kdlp bpftest]# chmod +x foo - -``` - -Trigger on the read system call and print its arguments (sans user buffer), the kernel stack, and the userspace stack. - - bpftrace -e 'kprobe:__x64_sys_read /comm=="foo"/ { printf("hello read(%d, %s, %zu) %s/%d [%s] [%s]\n",arg0, str(arg1), arg2, comm, pid, kstack, ustack); }' - -This is unwieldy at the interactive shell. Let's put it in an executable file with some formatting: - -``` -[root@kdlp bpftest]# cat > kprobe_read.bp < tracepoint_read.bp <fd, str(args->buf), args->count, comm, pid, kstack, ustack); -} -EOF - -[root@kdlp bpftest]# chmod +x tracepoint_read.bp -[root@kdlp bpftest]# ./tracepoint_read.bp -Attaching 1 probe... -``` - -Finally, we invoke `./foo` in a separate shell on the same system to generate something like the following: - -``` -hello read(3, , 832) foo/38364 [] [ - __GI___read_nocancel+8 - 0x7f89f99a4af7 - 0x7f89f999e78d - 0x7f89f999d523 - 0x7f89f999ebe8 - 0x7f89f99ba10e - 0x7f89f99b6c16 - 0x7f89f99b83de - 0x7f89f99b7208 -] -hello read(3, ����, 832) foo/38364 [] [ - 0x7f89f99bd478 - 0x7f89f99a4af7 - 0x7f89f999e78d - 0x7f89f999d523 - 0x7f89f999ebe8 - 0x7f89f99ba10e - 0x7f89f99b6c16 - 0x7f89f99b83de - 0x7f89f99b7208 -] -hello read(3, , 4096) foo/38364 [] [ - 0x7f89f98891e8 - 0x7f89f9808fb9 - 0x7f89f97fb4ca - 0x7f89f98058ae - 0x7f89f97bccf8 - 0x7f89f97bd26a - 0x7f89f97b69c9 - 0x7f89f97b5f7d - 0x556ebbbad4b5 - 0x556ebbb3a703 - 0x7f89f97aab8a - 0x7f89f97aac4b - 0x556ebbb3c255 -] -hello read(3, # Locale name alias data base. -# Copyright (C) 1996-2023 Free S, 4096) foo/38364 [] [ - 0x7f89f98891e8 - 0x7f89f9808fb9 - 0x7f89f97fb4ca - 0x7f89f98058ae - 0x7f89f97bccf8 - 0x7f89f97bd26a - 0x7f89f97b69c9 - 0x7f89f97b5f7d - 0x556ebbbad4b5 - 0x556ebbb3a703 - 0x7f89f97aab8a - 0x7f89f97aac4b - 0x556ebbb3c255 -] -hello read(3, (�!, 80) foo/38364 [] [ - 0x7f89f98840c1 - 0x7f89f97aab8a - 0x7f89f97aac4b - 0x556ebbb3c255 -] -hello read(255, , 30) foo/38364 [] [ - 0x7f89f98840c1 - 0x556ebbb96628 - 0x556ebbb3f018 - 0x556ebbc185bb - 0x556ebbb43d01 - 0x556ebbb47940 - 0x556ebbb47b0c - 0x556ebbb47d2e - 0x556ebbb3b522 - 0x7f89f97aab8a - 0x7f89f97aac4b - 0x556ebbb3c255 -] -hello read(255, #!/bin/bash - -echo x > x -cat x -, 30) foo/38364 [] [ - 0x7f89f98840c1 - 0x556ebbb96628 - 0x556ebbb3f018 - 0x556ebbc185bb - 0x556ebbb43d01 - 0x556ebbb47940 - 0x556ebbb47b0c - 0x556ebbb47d2e - 0x556ebbb3b522 - 0x7f89f97aab8a - 0x7f89f97aac4b - 0x556ebbb3c255 -] - -``` - -An important advantage of the tracepoint probe type is interface stability as systemcalls do not change, but unfortunately, the kernel stack is not available to the tracepoint-type probe. There is no one-size-fits-all solution as each probe type has its pros and cons. - - -For a much more detailed demo by the bpftrace grandmaster himself (Brendan Gregg), check out this -[Kernel analysis with bpftrace](https://lwn.net/Articles/793749/) -article from -[Linux Weekly News](https://lwn.net/). -Some of the information may be out of date as the article dates to mid-2019, it's definitely worth a read for anyone who wants to learn more. - -### Important Reminders - -Final submission for [E1](https://kdlp.underground.software/course/fall2023/assignments/E1.md) is due Thursday 5 October 2023 by 11:59 PM diff --git a/kdlp.underground.software/course/fall2023/lectures/L08.md b/kdlp.underground.software/course/fall2023/lectures/L08.md deleted file mode 100644 index 500525ae..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L08.md +++ /dev/null @@ -1,231 +0,0 @@ -# Lecture 8 - 5 October 2023 - -## IMPORTANT ANNOUNCEMENT -
errerewerewrerewwwe... the following is a test of the emergency broadcast system. errrrrwerwererwerwe ... ....
 - -The instructors are willing to spend more time helping students who take initiative and are less patient when asked questions at 11 PM on a due date for which the answers are *very* easily available by a web search. If you have made it this far in computer science, then you should have at least moderate proficiency in searching the internet for the solution to your computer problems. - -**BEFORE** you contact an instructor about a problem you are having, the expectation is that: - -* You have thoroughly read the assignment instructions -* You have proofread your command or code for typos -* If something has failed, you have read the error messsage and you have it ready to send to the instructor -* You read any relevant man pages or documentation -* You have searched the web for solutions - -**IF** you are able to solve your own problem, but you think that: - -* The directions are incorrect -* The directions could be made clearer -* There is a bug in our course infrastructure -* You have an idea for a feature that could improve the course quality - -Then, we invite you to contact us and we will appreciate your participation and take it into account. But if you are interested in contributing your change to our open source project that is the entire course, we will guide you through the process of making, testing, and submitting the change. Student who go this extra mile and complete the process will be given extra credit in our course and public recognition for their contribution. - -Keep in mind that this policy covers _anything_ in the course, including code, scripts, process, curriculum, formatting, style, security, design, testing, automation, devops, site reliability, IT, or any other aspect that you can imagine and convince us is a part of the course. - -**HOWEVER**, if you are still stuck and you have tried a couple of solutions, include _ALL_ of the following in your request for help: - -* What you were doing when the error occurred, that is to say, what specific steps did you take that we can take to see the same error. -* Any input or output remotely relevant to the error, include as applicable all of the following: - * The command you ran and the output - * Your complete source code Makefile - * Any error messages - * Any dmesg or system log messages that seem relevant to the error -* Any references or resources you consulted during your troubleshooting, e.g. manpages, websites, repositories, documentation -* What you tried to do to fix your problem -* Your best educated guess or guesses as to what's wrong. - -The more complete this information is when you ask for help, the more generous and flexible the instructors will be in their effort to help you. - -You will get out of this course what you put into it. - -And now back to your regularly scheduled programming. - -## Topics covered: - -* Assignment questions -* C compilation process deep dive -* Crossing CPU privilege boundaries - * Exceptions review - * Syscalls: end-to-end - -### Assignment Questions - -A student asked us to clarify the -[P1](https://kdlp.underground.software/course/fall2023/assignments/P1.md) -due date. - -P1 is **NOT** due next Tuesday. We want to give you plenty of time to do P1 since it's a bit more involved than E1. As a reminder, P assignments are the most -[heavily weighted](https://kdlp.underground.software/course/fall2023/policies/course_policies.md) -component of the course outside of the final assignment and presentation combo. - -As always, we encourage you to get started early so you have plenty of time to take breaks when you run into problems and later return to your work with a fresh mind. - -Next, a student asked when their grades will be available. E0 grades are available upon request via Matrix DM to one of the instructors. P0 grades are not yet available and any new information will be shared as soon as it is ready. - -The instructors wish to congratulate our students. This group delivered the all-time strongest average performance on the kernel compilation assignment! Nice job! - -### C compilation process deep dive: how a C program becomes an executable - -Take this C source file `hello.c`: - -``` -#include - -int main(void) -{ - puts("Hello, world!"); - return 0; -} -``` - -You should be familiar with the following invocation: - - gcc hello.c -o hello - -This is the most basic way to compile a C source file into an binary file you can execute. - -Most of you are also familiar with breaking this process into two steps: - -**Compilation**: - - gcc -c hello.c -o hello.o - -**Linking**: - - gcc hello.o -o hello - - -This has advantages for large projects because the compilation can be done in parallel, and as you edit the code, only the files that you change need to be recompiled. - -While *two* steps is enough for practical purposes (e.g. decreasing build time), it is not the full story. -In reality, the C compiler performs least _four_ distinct processes behind the scenes: preprocessing, compilation, assembly, and linking. - -The command `gcc -c hello.c -o hello.o` encompasses the preprocessing, compilation, and assembly steps, while the command `gcc hello.o -o hello` encompasses the linking step. - -We can invoke each step manually like so: - -0. Preprocessing - - cpp hello.c -o hello.i - -The -[C preprocessor](https://en.wikipedia.org/wiki/C_preprocessor) -removes comments, collapses whitespace, and resolves macros. - -The output is traditionally given the suffix `.i` which stands for intermediate. - -0. Compilation - - cc -S hello.i -o hello.s - -This is where C language constructs like variables, types and control-flow are flattened into undifferentiated data and code. - -After this point, we have no way to tell with certainty that this assembly output came from C program input. A compiler for a different language could plausibly generate identical assembly output. - -0. Assembly - - as hello.s -o hello.o - -The instructions are replaced with their machine code equivalents. This part is reversible, but -the assembler also rips out the last remnants of structure leftover from the original C program source. Static data and functions lose their names and are referred to by only their address, and any exported or imported variables and functions become generic entries in a symbol table. All other labels (e.g. the target of a jump within the same function) are gone without a trace. - -After this step, the output is no longer human readable text. - -0. Linking - - ld hello.o -l:crt1.o -lc -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o hello - -Even though _our_ functions have been compiled into machine code that our CPU could in theory execute, -there is still work do be done. The linker collects the dependencies of our program -(the C startup runtime `-l:crt1.o` that provides the `_start` symbol and the C standard library `-lc` which provides the `puts` symbol) and bundles them into one file. -The linker makes connections between object files, by cross-referencing their symbol tables -to resolve previously unresolved symbols with their now known locations. - -In reality symbol resolution is an instance of -the classic engineering trade-off between -execution speed and memory footprint. -Our C program, like most, is at least partially -[dynamically](https://en.wikipedia.org/wiki/Dynamic_linker) -linked at runtime (`-dynamic-linker /lib64/ld-linux-x86-64.so.2`). - -The output is an executable ELF file that the kernel loader can load into memory and execute on a CPU. - - -### Crossing CPU privilege boundaries - -In previous lectures, we've discussed what -CPU privilege levels are and what they mean. - -This lecture will round off that discussion -by analyzing in detail the final mechanism -for crossing into a privileged execution mode: -syscalls. - -#### Exceptions review - -We've previously demonstrated how our attempts -to execute an invalid instruction or a privileged -instruction while in user mode causes a CPU -exception. At the hardware level, the CPU -immediately switches it's privilege mode to kernel mode -and jumps at corresponding kernel function installed -at boot to handle handle the exception. In this case, -the handler function prints an error message to the -kernel ring buffer and kills our program. - -More information about exceptions can be found in -[L05](L05.md) -and -[L06](L06.md). - - -To give a couple more examples: -Software conditions such as dividing by zero or accessing an unaligned address trigger CPU exceptions. Hardware can of course also interrupt CPU execution by changing voltage on CPU pins. Finally, attempting to access a pointer -([virtual memory address](https://en.wikipedia.org/wiki/Virtual_memory)) for which the -[memory-mapping unit](https://en.wikipedia.org/wiki/Memory_management_unit) does not have a corresponding physical address triggers an -[page fault](https://wiki.osdev.org/Exceptions#Page_Fault) -exception which the kernel may resolve by setting up appropriate mapping (e.g. memory that was lazily allocated or swapped to disk), or by sending the program the `SIGSEV` signal otherwise known as a "Segmentation Fault". - -### Syscalls: end-to-end - -All previously discussed exceptions were just that: exceptional. - -The kernel handled them behind the scenes invisibly to the user or they were the result -of a bug in which case the kernel sent the program a fatal signal (e.g. `SIGILL`, `SIGSEGV`, and `SIGFPE`) - -Today we demonstrate the mechanism by which your code can intentionally call a system function running in kernel mode; in other words, a system call. - -We want to know how the userspace invocation ultimately connects back to the `SYSCALL_DEFINE` macro that you hunted down, traced, and wrote a history report on during -[E1](https://kdlp.underground.software/course/fall2023/assignments/E1.md) - -Let's follow the path of a syscall on x86. - -First of all, a user program invokes the -[syscall](https://www.felixcloutier.com/x86/syscall) -instruction. - -As described therein, the CPU immediately elevates -privilege level and jumps to the address in the LSTAR MSR register. - -Who set the value of this `LSTAR` Model Specific Register? - -In the -[syscall_init](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/kernel/cpu/common.c#L2054) -function, we find this -[line](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/kernel/cpu/common.c#L2057): - - wrmsrl(MSR_LSTAR, (unsigned long)entry_SYSCALL_64); - -Which sets set the value of LSTAR to the address of the -[entry_SYSCALL_64](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/entry_64.S#L87) -function. - -This is not C code :) - -The `.S` suffix indicates this is not just regular assembly code, but code that must be run through `cpp` in order to resolve macros in the file. The kernel uses these macros to make it easier to write the code correctly and facilitate interoperability with the rest of the kernel. - -At this time, the kernel is executing in a privileged mode, but all of the registers refer to userspace data. - -To be continued... diff --git a/kdlp.underground.software/course/fall2023/lectures/L09.md b/kdlp.underground.software/course/fall2023/lectures/L09.md deleted file mode 100644 index 3779a10e..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L09.md +++ /dev/null @@ -1,92 +0,0 @@ -# Lecture 9 - 10 October 2023 - -## Topics covered: - -* Syscalls: end to end (cont. from L08) - -### Syscalls: end-to-end - -At the end of L08 we had discussed the syscall process up when the processor jumps to the -[entry_SYSCALL_64](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/entry_64.S#L87) -function within `entry_64.S` which is written in assembly. As documented [here](https://www.felixcloutier.com/x86/syscall), -for performance reasons the only actions taken by the `syscall` instruction besides elevating privilege to ring 0 are: - -* Saving the return address before jumping to the kernel handler: - * The current instruction pointer `rip` is copied into `rcx` - * The address of the kernel handler is loaded from the `LSTAR` model specific register into `rip` -* Saving the current processor flags before reseting them to a known value - * The current flags `RFLAGS` are copied into `r11` - * The flags register is adjusted with a bitmask from the `FMASK` model specific register - -Within the `entry_SYSCALL_64` handler function, it is the responsibility of the kernel to save any -other userspace state that it wishes to restore when the syscall returns. -In the case of the linux kernel, all normal CPU registers should be saved. - -However, this presents a problem because essentially all cpu instructions involve manipulating the -values stored in the cpu registers. We want to save the data somewhere in memory, but we can't even -load a fixed pointer into a register to move data into that memory location because that would clobber -one of the values we need to save. - -Fortunately the designers of the CPU built an escape hatch in for this exact problem: the `swapgs` instruction. - -On x86 `gs` is a special type of register called a "segment" register. Segment registers were historically added to -facilitate easier access to more than 64K of memory on Intel's 16bit 8086 cpu (general purpose registers could store -16 bit pointers and segmentation could fill in the correct higher order bits to determine the full virtual address -depending on what your instruction was doing with the pointer (using it to access code, or data, or the stack, etc.). - -Segmentation is no longer a concern on 64 bit systems where the registers can easily store pointers to orders of magnitude -more virtual addresses than any computer could have physical ram, but the segmentation registers still exist on modern CPUs -and they have picked up a new function: storing a pointer for accessing thread local data. The `gs` register holds a pointer -to a block of memory reserved for thread specific data and any instruction can access this pointer by setting the `gs` prefix -on a memory access and providing the desired offset into the thread specific data as the "address". The CPU will add the base -address of the thread specific data from `gs` to the offset supplied in the instruction and the thread local data will be accessed. - -The special `swapgs` instruction ([line 91](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/entry_64.S#L91)) allows -ring 0 (kernel) code to atomically swap the value of `gs` with a well known value previously established by the kernel in a model -specific register that will hold a pointer to per cpu data while saving the old `gs` value from userspace into a different MSR -so it can be restored later. - -The handler code can then use scratch space allocated in the per cpu block to save the userspace stack pointer -([line 93](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/entry_64.S#L93)) and replace `rsp` with a pointer to -kernel stack ([line 95](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/entry_64.S#L95)). Once that has been completed, - the rest of the registers can be saved by just pushing them onto the kernel stack. The values are pushed in a specific order -([lines 100-109](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/entry_64.S#L100)) to make the overall footprint of the -data on the stack match the layout of a [`struct pt_regs`](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/include/asm/ptrace.h#L59). - -This means that after all the pushing, `rsp` is a valid `struct pt_regs *` pointer. It can be copied into `rdi` -([line 112](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/entry_64.S#L112)) to be the first argument along with -the syscall number in `rax` into `rsi` ([line 114](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/entry_64.S#L114)) -to become the second argument when it calls the C function [`do_syscall_64`](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/common.c#L73). - -From this point things are more simple, the kernel attempts to interpret the syscall number as a 64 bit syscall by calling -[`do_syscall_x64`](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/common.c#L40) and we can assume this is successful -if we are calling from 64 bit code. - -The meat of that function is verifying that the syscall number is in range ([line 48](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/common.c#L48)) -and then looking up the function pointer for the corresponding syscall number in an array then calling it and saving the return value into the entry for the ax register in -the `struct pt_regs` that will be restored when the kernel code returns ([line 50](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/common.c#L50)). - -The [`sys_call_table`](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/syscall_64.c#L16) array in generated using a technique called an -[X macro](https://en.wikipedia.org/wiki/X_macro). During the kernel build process, a header file is generated using -[the syscall table](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/syscalls/syscall_64.tbl) -that invokes a macro `__SYSCALL` (that is not defined within the header) once for each syscall with arguments of its number and entry point. - -The `__SYSCALL` macro can be given whatever definition the user wants and then the header can be included to programmatically generate invocations -of that specific version of the macro for each syscall. Within [`arch/x86/entry/syscall_64.c`](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/syscall_64.c) -the `__SYSCALL` macro is defined twice, the first time ([lines 10-12](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/syscall_64.c#L10)) -it uses the syscall name in the argument to form a declaration for a function named `__x64_sys_something` that takes a `const struct pt_regs *` argument, -and the second time ([lines 14-18](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/syscall_64.c#L14)) -it fills the `sys_call_table` variable will pointers to each of those functions in the right order. - -These wrapper functions are defined as part of the [`SYSCALL_DEFINE` macro](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/include/asm/syscall_wrapper.h#L228). -The [`__X64_SYS_STUBx`](https://elixir.bootlin.com/linuxv/v6.5/source/arch/x86/include/asm/syscall_wrapper.h#L96) macro generates -a function named `__x64_sys_whatever` that takes a `struct pt_regs` whose body just calls another wrapper starting the `__se_` with the real syscall args. -These are obtained by the [`SC_X86_64_REGS_TO_ARGS`](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/include/asm/syscall_wrapper.h#L56) -macro which converts the list of arguments into accessing `regs->register` for each register in the appropriate order. - -The `__se_` wrapper (short for sign extension) has to do with 32 bit compatibility and is generated in place by the SYSCALL_DEFINE macro -([line 233](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/include/asm/syscall_wrapper.h#L233)), and finally it calls into -a function with the prefix `__do_` that is the function whose header is right at the end of the wrapper ([line 240](https://elixir.bootlin.com/linux/v6.5/source/arch/x86/include/asm/syscall_wrapper.h#L240)) -and whose body is supplied by the code within the curly braces that follows a given `SYSCALL_DEFINE` invocation. - -At this point it is running the code in that block and the syscall has officially begun :) diff --git a/kdlp.underground.software/course/fall2023/lectures/L10.md b/kdlp.underground.software/course/fall2023/lectures/L10.md deleted file mode 100644 index 77f05026..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L10.md +++ /dev/null @@ -1,234 +0,0 @@ -# Lecture 10 - 12 October 2023 - -## Topics covered: - -* Assembly programming - -Because of the overlap with the UMass Lowell Career Fair, this lecture was recorded. -[Here is the recording](https://youtu.be/9s12he7g-0c). - -### Assembly code written in class: - -``` -.text -.globl _start -_start: - #write(1, QUESTION, sizeof(QUESTION) - 1); - mov $1, %rdi #stdout fileno - lea question, %rsi #pointer to string - mov $question_len, %rdx #length - mov $1, %rax #no. for write - syscall #do it! - cmp $0, %rax #check return value - jl error #if negative, error out - - #read(0, buffer, sizeof(buffer)); - mov $0, %rdi #stdin fileno - lea buffer, %rsi #pointer to buffer - mov $buffer_len, %rdx #length - mov $0, %rax #no. for read - syscall #do it! - push %rax #save return value - cmp $1, %rax #check return value - jle error #if <= 1, error out - - #write(1, MESSAGE, sizeof(MESSAGE) - 1); - mov $1, %rdi #stdout fileno - lea hellomsg, %rsi #pointer to string - mov $hello_len, %rdx #length - mov $1, %rax #no. for write - syscall #do it! - cmp $0, %rax #check return value - jl error #if negative, error out - - #write(1, buffer, (size_t)len); - mov $1, %rdi #stdout fileno - lea buffer, %rsi #pointer to buffer - pop %rdx #saved length - mov $1, %rax #no. for write - syscall #do it! - cmp $0, %rax #check return value - jl error #if <= 1, error out - - mov $0, %rdi #exit status of 0 - mov $60, %rax #no. for exit - syscall #do it! -error: - mov $2, %rdi #stderr fileno - lea errormsg, %rsi #pointer to string - mov $error_len, %rdx #length - mov $1, %rax #no. for write - syscall #do it! - mov $1, %rdi #exit status of 1 - mov $60, %rax #no. for exit - syscall #do it! - -.data -question: -.ascii "What is your name?\n" -.equ question_len, . - question -errormsg: -.ascii "error!\n" -.equ error_len, . - errormsg -buffer: -.equ buffer_len, 100 -.space buffer_len, 0 -hellomsg: -.ascii "Hello, " -.equ hello_len, . - hellomsg -``` - -### Similar prewritten example for both x86-64 and aarch64: - -#### x86-64: - -``` -#include - -#define STDIN_FILENO 0 -#define STDOUT_FILENO 1 - -.globl _start //make _start a global symbol so linker can find it -_start: //_start is entry point for all executibles - mov %rax, $SYS_write //%rax holds syscall number, 1 represents `write` - mov %rdi, $STDOUT_FILENO //%rdi holds first syscall arg, 1 represents `stdout` - lea %rsi, prompt //%rsi holds second arg, =prompt gets address if prompt string from data section - mov %rdx, $prompt_len //%rdx holds third arg, prompt_len is macro that expands to calculated size - syscall //perform a system call - cmp %rdi, $0 //check if return is negative - jl .out //if it is, exit program early with exit code based on return value - mov %rax, $SYS_read //0 represents `read` - mov %rdi, $STDIN_FILENO //0 represents `stdin` - ldr %rsi, =buffer //read into buffer - mov %rdx, $buffer_len //at most buffer_len bytes - syscall //perform syscall - cmp %rdi, $0 //check for error as above - jl .out - mov %rcx, %rdi //save returned length to only print that many bytes - mov %rax, $SYS_write //back to writing, send "Hello, " to stdout - mov %rdi, $STDOUT_FILENO - ldr %rsi, =msg - mov %rdx, $msg_len - syscall - cmp %rdi, $0 //check for error - jl .out - mov %rdi, $1 //need to set %rdi back to 1 because it was replaced with return code of last call - ldr %rsi, =buffer //whatever they input - mov %rdx, %rcx //and however long it was - syscall //send that - cmp %rdi, $0 //check for errors - jl .out - mov %rdi, $0 //if there was not an error, set return code to 0 -.out: //otherwise we were sent here and %rdi already contains error code to return - mov %rax, $SYS_exit //60 represents exit - syscall //exit program - //exit syscall does not return, so _start function does not need to return to caller - -.data //data section for strings -prompt: .ascii "What is your name? " -.equ prompt_len, .-prompt //.equ makes a new macro, `.` represents current location in binary, and subtracting the value of prompt gives how many bytes prompt contained -buffer: .space 64 -.equ buffer_len, .-buffer -msg: .ascii "Hello, " -.equ msg_len, .-msg - -.data -message: - .ascii "Hello, World!\n" - len = . - message -.text -.global _start -_start: - mov $1, %rdi - mov $message, %rsi - mov $len, %rdx - mov $1, %rax - syscall - mov $13, %rdi - mov $60, %rax - syscall -``` -#### aarch64: - -``` -#include - -#define STDIN_FILENO 0 -#define STDOUT_FILENO 1 - -.globl _start //make _start a global symbol so linker can find it -_start: //_start is entry point for all executibles - mov x8, #SYS_write //x8 holds syscall number, 64 represents `write` - mov x0, #STDOUT_FILENO //x0 holds first syscall arg, 1 represents `stdout` - ldr x1, =prompt //x1 holds second arg, =prompt gets address if prompt string from data section - mov x2, #prompt_len //x2 holds third arg, prompt_len is macro that expands to calculated size - svc #0 //perform a system call - cmp x0, #0 //check if return is negative - b.lt .out //if it is, exit program early with exit code based on return value - mov x8, #SYS_read //63 represents `read` - mov x0, #STDIN_FILENO //0 represents `stdin` - ldr x1, =buffer //read into buffer - mov x2, #buffer_len //at most buffer_len bytes - svc #0 //perform syscall - cmp x0, #0 //check for error as above - b.lt .out - mov x3, x0 //save returned length to only print that many bytes - mov x8, #SYS_write //back to writing, send "Hello, " to stdout - mov x0, #STDOUT_FILENO - ldr x1, =msg - mov x2, #msg_len - svc #0 - cmp x0, #0 //check for error - b.lt .out - mov x0, #1 //need to set x0 back to 1 because it was replaced with return code of last call - ldr x1, =buffer //whatever they input - mov x2, x3 //and however long it was - svc #0 //send that - cmp x0, #0 //check for errors - b.lt .out - mov x0, #0 //if there was not an error, set return code to 0 -.out: //otherwise we were sent here and x0 already contains error code to return - mov x8, #SYS_exit //93 represents exit - svc #0 //exit program - //exit syscall does not return, so _start function does not need to return to caller - -.data //data section for strings -prompt: .ascii "What is your name? " -.equ prompt_len, .-prompt //.equ makes a new macro, `.` represents current location in binary, and subtracting the value of prompt gives how many bytes prompt contained -buffer: .space 64 -.equ buffer_len, .-buffer -msg: .ascii "Hello, " -.equ msg_len, .-msg -``` - -### Example makefile for assembly with preproccessing - -``` -.PHONY: all clean -all:asm_hello - -asm_hello: asm_hello.o - ld -o asm_hello asm_hello.o - -asm_hello.o: asm_hello.s - as asm_hello.s -o asm_hello.o - -asm_hello.s: asm_hello.S - cpp asm_hello.S -o asm_hello.s - -clean: - -rm asm_hello.s asm_hello.o asm_hello -``` - -### Important Reminders - -|[P1](/course/fall2023/assignments/P1.md) requirement| Deadline Time | Deadline Date | -|--|--|--| -|**initial submission**|11:59PM|Tuesday 17 October 2023| -|**peer review**|11:59PM|Wednesday 18 October 2023| -|**final submission**|11:59PM|Thursday 19 October 2023| - -
-
- -# *ALL P1 SUBMISSIONS GO TO: [programming1@kdlp.underground.software](mailto:programming1@kdlp.underground.software)* diff --git a/kdlp.underground.software/course/fall2023/lectures/L11.md b/kdlp.underground.software/course/fall2023/lectures/L11.md deleted file mode 100644 index 6db4e7ae..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L11.md +++ /dev/null @@ -1,215 +0,0 @@ -# Lecture 11 - 17 October 2023 - -## Topics Covered: -* FAQ for first patch of P1 -* Everything is a file (in Linux) -* `/proc` filesystem -* A `/proc`tical example - -### FAQ for first patch of P1: - -**Q:** What goes in the first patch? - -**A:** From Joel's 14 October 2023 post in Matrix: -> We are asking you to do something a little bit weird, -> but I think this will clarify: You are asked to make -> changes to the Linux kernel to implement your syscall. -> Once you have tested these changes -> (or earlier to back them up), -> you make a commit containing the all of these local changes -> to Linux and then generate an email patch for this new commit. -> This is the `.patch` file we are referring to in the instructions. -> Then, you copy the `.patch` file from its original location -> to your (new) `firstname_lastname` directory in the P1 -> directory of the `ILKD_assignments` repo. -> The commit for the first patch of your homework submission will -> consist of adding this `.patch` file to your named directory. - -**Q:** When I generate the `.patch` file for my first commit, -and try to locally apply it, `git am` complains about whitespace errors! -Is this OK? - -**A:** Yes, as long as it is in the line above the git version that has the -two dashes and space. This is an artifact of `git format-patch`. -You are welcome to remove this extraneous space from the `.patch` file -before making your first commit, -however we will not penalize you if you leave it in. - -**Q:** Why goes `git format-patch` have this behavior? - -**A:** The upstream git source code prints the space -[here](https://github.com/git/git/blob/2e8e77cbac8ac17f94eee2087187fa1718e38b14/builtin/log.c#L1205). -Via extensive digging with `git blame`, -we traced the origin of this behavior back to -[this commit](https://github.com/git/git/commit/a004d3f70f1c074f2d9bd55e7a925ff5916ebbeb) -from 2005: -```diff -$ curl https://github.com/git/git/commit/a004d3f70f1c074f2d9bd55e7a925ff5916ebbeb.patch -From a004d3f70f1c074f2d9bd55e7a925ff5916ebbeb Mon Sep 17 00:00:00 2001 -From: Junio C Hamano -Date: Tue, 29 Nov 2005 13:51:27 -0800 -Subject: [PATCH] format-patch: do not abuse 3-dash marker line. - -Before GIT version at the end of output we used a 3-dash marker; -but 3-dash marker is special and should not be overused. -Instead, use "-- " which is a standard practice in e-mails to -signal the beginning of trailing garbage. - -Signed-off-by: Junio C Hamano ---- - git-format-patch.sh | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -diff --git a/git-format-patch.sh b/git-format-patch.sh -index a26d46dba0b1f9..4cd38f34efd84e 100755 ---- a/git-format-patch.sh -+++ b/git-format-patch.sh -@@ -239,7 +239,7 @@ Date: '"$ad" - git-diff-tree -p $diff_opts "$commit" | git-apply --stat --summary - echo - git-diff-tree -p $diff_opts "$commit" -- echo "---" -+ echo "-- " - echo "@@GIT_VERSION@@" - - case "$mbox" in -``` - -Back then, -`git format-patch` was still just a shell script. -We speculate that the space was added to maintain -compatibility with existing code containing the -hardcoded expectation of three characters. - -### Everything is a file (in Linux) - -This elegant design principle dates back to -the beginning of (Unix) time, a.k.a. -[the 70s](https://en.wikipedia.org/wiki/January 1, 1970). -However, this simple principle is an -oversimplification - consider the existence -of directories. -In reality, the slogan -["Everything is a file"](https://en.wikipedia.org/wiki/Everything_is_a_file) -is a convenient shorthand for the more accurate -but less catchy notion that (almost) all -resources available to a process on a -[Unix-like](https://en.wikipedia.org/wiki/Unix-like) -operating system can be referenced by a -[file descriptor](https://en.wikipedia.org/wiki/File_descriptor). - - -We will continue to investigate -this concept throughout the course, -but today we focus on one example: `/proc`. - -### The process filesystem: `/proc` - -Unlike some of the more esoteric resources that -can be referred to by a file descriptor, -the entries found in the `/proc` directory on -any Linux system are in fact real files. - -However, they are not entirely like other files: -they are transient. -That is to say, these files are not stored -on any long-term storage -media, e.g. a hard drive. -These files don't need long term storage because -they provide access to information that only -exists at runtime. - -Instead of reading the directory -structure and contents from a storage medium, -the kernel creates the files in `/proc` at runtime -and synthesizes their contents on demand. - -Specifically, the kernel creates a directory for each -running process on the system named after its pid. -In addition, the kernel provides a "magic" symlink -named `self` -whose target depends on which process is looking. -Any process that examines the symlink -sees it resolve to the folder that corresponds to -the calling process's pid. - -This directory contains information about running processes. -For a complete list of the contents, refer to the kernel -[documentation](https://docs.kernel.org/filesystems/proc.html) and the -[manpage](https://man7.org/linux/man-pages/man5/proc.5.html). - - -Unfortunately, -`/proc` also contains many -miscellaneous files that were added -before the community developed `/sys`. -They are still present to preserve -backwards compatibility. - -### A `/proc`tical example - -In bash, `$$` is a -[special variable](https://www.gnu.org/software/bash/manual/html_node/Special-Parameters.html) -that expands to the pid of the bash process. - -For example: - - $ echo $$ - 1337 - -This means we can use `$$` when building a path -to reference the `/proc` subdirectory corresponding -to the running bash process. -In P1, the systemcall used the -`get_task_comm` kernel macro to find the name -of the running program. -`/proc` also provides userspace access to this -information. Here is an example: - - $ cat /proc/$$/comm - bash - -We can also discover the absolute path of the -executable invoked to start the process by -traversing another "magic" symlink named `exe`: - - $ readlink /proc/$$/exe - /usr/bin/bash - -If we replace `$$` with `self`, -we are now referring to the child process -the shell created by `fork`ing itself -and `exec`ing the user command: - - $ cat /proc/self/comm - cat - - $ readlink /proc/self/exe - /usr/bin/readlink - -Another useful entry in `/proc` for -a given process is the `fd` directory, -which contains magic symlinks to all file -descriptors owned by the process: - - $ ls -l /proc/self/fd - ... 0 -> /dev/pts/0 - ... 1 -> /dev/pts/0 - ... 2 -> /dev/pts/0 - ... 3 -> /proc/128523/fd - -As expected, the first three entries are -`stdin`, `stdout`, and `stderr` -which are connected to our terminal. -We can also see how the `ls` program opens -its own subdirectory in `/proc` by following -the "magic" `/proc/self` symlink. - -## Watch before next class - -* [Modules and Drivers](https://www.youtube.com/watch?v=pYbR3lyYOgY) -* [Writing Kernel Modules](https://www.youtube.com/watch?v=B-Yw6SLFlQg) - - - - diff --git a/kdlp.underground.software/course/fall2023/lectures/L12.md b/kdlp.underground.software/course/fall2023/lectures/L12.md deleted file mode 100644 index 0556a8c9..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L12.md +++ /dev/null @@ -1,6 +0,0 @@ -# Lecture 12 - 19 October 2023 - -## Topics Covered: -* non-permission mode bits (i.e. setuid setgit and sticky) -* Using procfs to avoid race condition -* Writing Basic Linux Kernel Module diff --git a/kdlp.underground.software/course/fall2023/lectures/L13.md b/kdlp.underground.software/course/fall2023/lectures/L13.md deleted file mode 100644 index e68062e2..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L13.md +++ /dev/null @@ -1,6 +0,0 @@ -# Lecture 13 - 24 October 2023 - -## Topics Covered: -* midpoint presentation announced -* introduction to 3 main kernel ram psuedo-filesystems `/proc`, `/dev`, and `/sys` -* introduction to special device files, block and char diff --git a/kdlp.underground.software/course/fall2023/lectures/L14.md b/kdlp.underground.software/course/fall2023/lectures/L14.md deleted file mode 100644 index ab8c7405..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L14.md +++ /dev/null @@ -1,6 +0,0 @@ -# Lecture 14 - 26 October 2023 - -## Topics Covered: -* Background: Privelege escalation and course email infrastructure -* Chroot system call -* Dynamic linking diff --git a/kdlp.underground.software/course/fall2023/lectures/L15.md b/kdlp.underground.software/course/fall2023/lectures/L15.md deleted file mode 100644 index c9a68571..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L15.md +++ /dev/null @@ -1,6 +0,0 @@ -# Lecture 15 - 31 October 2023 - -## Topics Covered: -* Background: C syntax -* Character devices provided by `drivers/char/mem.c` -* `struct file_operations` diff --git a/kdlp.underground.software/course/fall2023/lectures/L16.md b/kdlp.underground.software/course/fall2023/lectures/L16.md deleted file mode 100644 index 6a6281a6..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L16.md +++ /dev/null @@ -1,5 +0,0 @@ -# Lecture 16 - 2 November 2023 - -## Topics Covered: -* Mid point presentations -* kernel `dev_t` datatype diff --git a/kdlp.underground.software/course/fall2023/lectures/L17.md b/kdlp.underground.software/course/fall2023/lectures/L17.md deleted file mode 100644 index d2408b28..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L17.md +++ /dev/null @@ -1,1385 +0,0 @@ -# Lecture 17 - 7 November 2023 - - -## Topics Covered: -* P2 (lack of) questions -* Translating a user program into kernel code - - -### P2 (lack of) questions - -We opened the floor for any and all questions about P2, -but we had no takers. -I hope everyone had fun with this one! -This is the first semester we are running this assignment. - -We may unveil some of the mathematical structure -of this assignment later this week in L18. - -### Translating a user program into kernel code - -We begin with a simple but non-trivial user program: - -```c -$ cat user_code.c -#include -#include -#include -#include -#include -#include - -struct example -{ - char *message; - size_t size; -}; - -static struct example *example_create(const char *msg) -{ - struct example *ex = malloc(sizeof *ex); - if(!ex) - goto out; - ex->size = strlen(msg); - ex->message = strdup(msg); - if(!ex->message) - goto out_free; - return ex; -out_free: - free(ex); - ex = NULL; -out: - return ex; -} - -static void example_destroy(struct example *ex) -{ - free(ex->message); - free(ex); -} - -static bool example_update_message(struct example *ex, const char *msg) -{ - size_t size = strlen(msg); - char *data = strdup(msg); - if(!data) - return false; - free(ex->message); - ex->message = data; - ex->size = size; - return true; -} - -static char *example_get_message(struct example *ex) -{ - return ex->message; -} - -int main(void) -{ - struct example *ex = example_create("hello"); - if(!ex) - err(1, "unable to allocate memory"); - printf("%s\n", example_get_message(ex)); - if(!example_update_message(ex, "goodbye")) { - int temperrno = errno; - example_destroy(ex); - errno = temperrno; - err(1, "unable to update"); - } - printf("%s\n", example_get_message(ex)); - example_destroy(ex); - return 0; -} - -``` - -Before we proceed, let's note a few key features. - -**Data flow** - -The program works with structured data, primarily in the form of `struct example`: - -```c -struct example -{ - char *message; - size_t size; -}; -``` - -This pair of elements represents a simple byte string and its size. -Take note that both the data structure itself -and the memory located at `message` -can be allocated either statically or dynamically, -and we take care to ensure that these two layers are handled appropriately. - -Our typical userspace entry point, -the `main` function, -declares a pointer to one of these `struct example` types -and then immediately assigns the return value of a constructor-style -function `example_create()`, -whose job is to encapsulate the finer details of allocation and initialization. - -In good style, `main` is responsible for cleaning up its own mess, -and this task is executed right before `main` returns back to the C library -at the bottom of the function by invocation of `example_destroy()`. -When implementing a more complex program, we may pass a pointer to our -local reference in order to zero the value to avoid subsequent misuse by the caller, -i.e. a dangling pointer, however this is unnecessary complexity for this simple example -and it suffices to simply ensure that our program does not leak memory. -Usage of the userspace tool `valgrind` will validate this property of our program, -but we do admit for a short-lived program such as this example whose memory is cleaned up -by the kernel at termination, the fuss and rigor around memory leaks appears pedantic beyond -the practice of good habits. Though practice is reason enough, -we will soon find ourselves in kernelspace where there is no one to clean up after us. -In the kernel, a memory leak will persist until reboot and in the meantime will clog the tubes of the -[memory allocator](https://lwn.net/Articles/229984/). - - -**Control Flow** - -Our example program implements a control flow that should -not raise the eyebrows of a C programmer with beyond novice-level skill. -We don't do anything fancy with the entry point, -and we don't create any threads. -We invoke a constructor to allocate our memory in fairly standard form, -using the old reliable `malloc` function from the `` section -of the trusty C library. During instantiation, we make a couple of calls -to the `` section in the form of `strlen()` and `strdup()`, -both of which assume as a precondition a nicely null-terminated input string -as the `msg` parameter. Likewise, we perform the same operations -in `example_update_message()`, assuming the same precondition. - -Each call to `malloc()` pairs with a corresponding call to `free()`, -both at the level of the allocated message and the data structure itself, -and in just the same pattern our `example_create()` constructor function pairs -with our `example_destroy()` destructor function. -The `example_get_message()` implements a getter and `example_update_message()` -implements a setter. The complexity of the latter is due to the need to duplicate the -byte-string `msg` argument and free the now-junk memory residing at the -address contained in `ex->message`. - -**Error Flow** - -A careful reader of our example may take alarm at a particular feature. -We too have heard these rumors, that the C `goto` statement is considered -["harmful"](https://homepages.cwi.nl/~storm/teaching/reader/Dijkstra68.pdf). -Despite these tall tales, we inform you with confidence that while there -are many paths to correct code, the -[shortest path](https://en.wikipedia.org/wiki/Dijkstra%27s_algorithm) -to readability and maintainability is often by use of this fearsome little keyword. -For one, correct usage of `goto` and error case labeling as seen in our example -eliminates the need for repetitive code and unnecessary indentation. -As it is -[written](https://www.kernel.org/doc/Documentation/process/coding-style.rst): -"if you need more than 3 levels of indentation, you're screwed anyway and you should fix your program". -We will not elaborate any -[further](https://www.cs.utexas.edu/users/EWD/transcriptions/EWD10xx/EWD1009.html). - -Next, note our usage of `err()` from ``. -This handy tool lets us perform the work of `perror()` and `exit` -with a single invocation. -We first pass the return code we would have handed to `exit` -and then we specify the string snatched from the jaws of `perror()`. - -The final point worth noting is our usage of `temperrno`. -Think of this as if we were "pushing" the value of `errno` -at that instance onto the stack, -like we would do at the assembly level for a register -before a jump or call into a section of code that may clobber said register. -Usage of the C library function `free()` -in our call to `example_destroy()` may overwrite the previous value of `errno`, -but this is the relevant value to report in the context -of cleaning up after a failed call to `example_update_message()`. - - -**System Flow** - -The program sends the following text to `stdout` when run: - -``` -$ ./user_example -hello -goodbye -``` - -Other than this, the program does not interact with the system -in any manner worth noting. - -#### Leaving Kansas - -Now that we have analyzed the user code with -excruciatingly thorough exposition, -let us turn to the primary task at hand. - -In order to satisfy what we assume to be our reader's -ravenous appetite for kernel module code and alleviate -the all-too-familiar pangs of hunger for privileged execution, -we'll begin by dropping the complete `diff -up` output between -the above program and its kernel equivalent: - -```diff -$ diff -Naup user_code.c kernel_code.c ---- user_code.c 2023-11-07 23:30:25.792075105 -0500 -+++ kernel_code.c 2023-11-07 23:30:16.628563819 -0500 -@@ -1,9 +1,6 @@ --#include --#include --#include --#include --#include --#include -+#include -+#include -+#include - - struct example - { -@@ -13,16 +10,16 @@ struct example - - static struct example *example_create(const char *msg) - { -- struct example *ex = malloc(sizeof *ex); -+ struct example *ex = kmalloc(sizeof *ex, GFP_KERNEL); - if(!ex) - goto out; - ex->size = strlen(msg); -- ex->message = strdup(msg); -+ ex->message = kstrdup(msg, GFP_KERNEL); - if(!ex->message) - goto out_free; - return ex; - out_free: -- free(ex); -+ kfree(ex); - ex = NULL; - out: - return ex; -@@ -30,17 +27,17 @@ out: - - static void example_destroy(struct example *ex) - { -- free(ex->message); -- free(ex); -+ kfree(ex->message); -+ kfree(ex); - } - - static bool example_update_message(struct example *ex, const char *msg) - { - size_t size = strlen(msg); -- char *data = strdup(msg); -+ char *data = kstrdup(msg, GFP_KERNEL); - if(!data) - return false; -- free(ex->message); -+ kfree(ex->message); - ex->message = data; - ex->size = size; - return true; -@@ -51,20 +48,39 @@ static char *example_get_message(struct - return ex->message; - } - --int main(void) -+int example_init(void) - { -+ int ret = -ENOMEM; -+ const char *msg; - struct example *ex = example_create("hello"); -+ msg = KERN_ERR "unable to allocate memory"; - if(!ex) -- err(1, "unable to allocate memory"); -- printf("%s\n", example_get_message(ex)); -- if(!example_update_message(ex, "goodbye")) { -- int temperrno = errno; -- example_destroy(ex); -- errno = temperrno; -- err(1, "unable to update"); -- } -- printf("%s\n", example_get_message(ex)); -+ goto out; -+ -+ pr_info("%s\n", example_get_message(ex)); -+ -+ msg = KERN_ERR "unable to update\n"; -+ if(!example_update_message(ex, "goodbye")) -+ goto out_free; -+ -+ pr_info("%s\n", example_get_message(ex)); -+ -+ ret = 0; -+ msg = NULL; -+out_free: - example_destroy(ex); -- return 0; -+out: -+ if(msg) -+ printk(msg); -+ return ret; -+} -+ -+void example_exit(void) -+{ - } - -+module_init(example_init); -+module_exit(example_exit); -+ -+MODULE_LICENSE("GPL"); -+ - -``` - -The length of this `diff` output exceeds the length of the original user program. -We will proceed with an explanation of each change. - -#### Welcome to Oz - -The transition to writing kernel code is a shift to another plane of reality. -Previous assumptions about what a C program looks like may no longer hold, -and the reader may encounter strange looking constructs and ludicrously -deep layers of macro invocations, generating the sense of a fever dream. -When all appears to be lost, bear in mind one key point: -There is no escape from the kernel. -The kernel has been running since the CPU exited the bootloader -and only a semi-magical illusion has hidden this raw truth from your eyes. -Today, we lift this curse from the reader, -revealing, as the scales fall from their eyes, -the vibrant glory of kernel module code, -and forever dispelling the last remnant -of prestidigitation from their mental model of the computer. -Magic no more! The entirety of the machine, -software and hardware stack united as one, -lies bare before the attentive reader, -and nothing, save polynomial time factoring of large numbers, -remains beyond reach. - -Well then, lets get started. - -**Switch to kernel headers** - -First off, the C standard library is not available -within the kernel, so we discard the inclusion -of the header files that provide C library -declarations: - -``` --#include --#include --#include --#include --#include --#include -+#include -+#include -+#include -``` - -Instead, we include headers declaring -Linux kernel API entry points. -These paths are relative to the `include` -directory within the kernel repository. - -The first, -[](https://elixir.bootlin.com/linux/latest/source/include/linux/module.h), -provides the basic building blocks for a kernel module, -such as `#define`s of the `module_init()` and `module_exit()` macros we encounter later on. -Importantly, this file also `#define`s the mandatory `MODULE_LICENSE()` macro, -which we will return to at the end, as well as `printk()` and the associated macros. - -Next, we include -[](https://elixir.bootlin.com/linux/latest/source/include/linux/string.h) -to replace some of the functionality we accessed via the C library's `string.h`. -Some of the functions retain their familiar names, like `strlen()`, while others -like `kstrdup()` take on new names and new arguments. - -Finally, in order to allocate and free memory, we include -[](https://elixir.bootlin.com/linux/latest/source/include/linux/slab.h), -which gives us the duo of `kmalloc()` and `kfree()`, -second cousins of the familiar userspace versions. - -That's all for the `#include`s. -Here we can briefly note that the `struct example` we defined in userspace -is perfectly suitable for usage in kernelspace, so we skip right over it. - -**Memory allocation with a twist** - -``` - struct example - { -@@ -13,16 +10,16 @@ struct example - - static struct example *example_create(const char *msg) - { -``` - -Now, we arrive at our first usage of `kmalloc()`. -Like userspace `malloc()`, -this function takes a number of bytes to allocate -as its first argument, but `kmalloc()` takes a mysterious -second argument. In fact, this is the same argument -passed as the mysterious second argument to `kstrdup()`. -Luckily for the simplicity of this paragraph, `kfree()` -works exactly like `free()`. - -``` -- struct example *ex = malloc(sizeof *ex); -+ struct example *ex = kmalloc(sizeof *ex, GFP_KERNEL); - if(!ex) - goto out; - ex->size = strlen(msg); -- ex->message = strdup(msg); -+ ex->message = kstrdup(msg, GFP_KERNEL); - if(!ex->message) - goto out_free; - return ex; - out_free: -- free(ex); -+ kfree(ex); - ex = NULL; - out: - return ex; -@@ -30,17 +27,17 @@ out: - - static void example_destroy(struct example *ex) - { -- free(ex->message); -- free(ex); -+ kfree(ex->message); -+ kfree(ex); - } - - static bool example_update_message(struct example *ex, const char *msg) - { - size_t size = strlen(msg); -- char *data = strdup(msg); -+ char *data = kstrdup(msg, GFP_KERNEL); - if(!data) - return false; -- free(ex->message); -+ kfree(ex->message); - ex->message = data; - ex->size = size; - return true; -@@ -51,20 +48,39 @@ static char *example_get_message(struct - return ex->message; - } -``` - -The changes to the three functions `example_init()`, -`example_destroy()`, and `example_update_message()` are -all limited to these three substitutions, two of which introduce -this mysterious second `GFP_KERNEL` argument. -We will pause here to discuss this in more depth before getting into -the real funky stuff. - -We find the declaration of kmalloc in the latter half of -[`include/linux/slab.h`](https://elixir.bootlin.com/linux/v6.6/source/include/linux/slab.h#L590) -and the included comment provides us with far more articulate explication than we could muster. - -We include a snippet of the -[Linux `v6.6` kmalloc comment](https://elixir.bootlin.com/linux/v6.6/source/include/linux/slab.h#L548) -verbatim: - -``` - * The @flags argument may be one of the GFP flags defined at - * include/linux/gfp_types.h and described at - * :ref:`Documentation/core-api/mm-api.rst ` - * - * The recommended usage of the @flags is described at - * :ref:`Documentation/core-api/memory-allocation.rst ` - * - * Below is a brief outline of the most useful GFP flags - * - * %GFP_KERNEL - * Allocate normal kernel ram. May sleep. - * - * %GFP_NOWAIT - * Allocation will not sleep. - * - * %GFP_ATOMIC - * Allocation will not sleep. May use emergency pools. - * - * Also it is possible to set different flags by OR'ing - * in one or more of the following additional @flags: - * - * %__GFP_ZERO - * Zero the allocated memory before returning. Also see kzalloc(). - * - * %__GFP_HIGH - * This allocation has high priority and may use emergency pools. - * - * %__GFP_NOFAIL - * Indicate that this allocation is in no way allowed to fail - * (think twice before using). - * - * %__GFP_NORETRY - * If memory is not immediately available, - * then give up at once. - * - * %__GFP_NOWARN - * If allocation fails, don't issue any warnings. - * - * %__GFP_RETRY_MAYFAIL - * Try really hard to succeed the allocation but fail - * eventually. -``` - - -The curious reader should feel free to pursue any rabbit hole -referenced within that comment. - -The signature of `kmalloc()` itself is quite simple when the funny business is hidden: - - void *kmalloc(size_t size, gfp_t flags) - -The second argument is a `typedef`ed wrapper for what is really nothing more than a fancy -[`unsigned int`](https://elixir.bootlin.com/linux/v6.6/source/include/linux/types.h#L154), -but, in good style, -these implementation details are hidden from us -unless we search for them. -Essentially, this second `flags` argument is used to specify additional options -to the memory allocator. -One could easily implement such a compact bit-flags argument in userspace, -and certainly many of our readers have done so, -but we understand the confusion a novice kernel programmer may encounter -when forced to select options from a menu of foreign-language items in order -to perform a task as apparently simple as memory allocation. - -Let us back up a couple of steps and motivate this complexity. -As we noted in our discussion of the userspace program in the "Data Flow" section, -there is no other process within a system who will come save the kernel. -Without expanding the scope of our analysis -[beyond a single system](https://en.wikipedia.org/wiki/Virtual_machine) -or into the realm of -[exotic hardware](https://en.wikipedia.org/wiki/Intel_Active_Management_Technology), -we must operate under the knowledge that -the kernel is the sovereign and absolute monarch of a computer system -from the time that the bootloader kindly requests that the CPU jump into the kernel code -to the time the computer is either reset or physically destroyed. -While this absolute authority grants the CPU the enjoyment of maximally privileged execution, -this absolute responsibility yokes the CPU with the burden of maximally privileged execution. - -When we write kernel code, in this case a kernel module that allocates and frees memory, -we can't just blindly type up some half-baked garbage willy-nilly -and grind out a compile/valgrind/debug loop until all the errors are ironed out. -Certainly -[there are tools](https://docs.kernel.org/dev-tools/kmemleak.html) -for searching the kernel for memory leaks, -but the instrumentation of the kernel is not nearly as trivial -as the runtime instrumentation performed by `valgrind`. -To zoom into our particular context, take a closer look at the three `GFP_*` flags -in the `kmalloc()` comment which are not prefixed by a double underscore ("dunder"): - -``` - * %GFP_KERNEL - * Allocate normal kernel ram. May sleep. - * - * %GFP_NOWAIT - * Allocation will not sleep. - * - * %GFP_ATOMIC - * Allocation will not sleep. May use emergency pools. -``` - -We briefly note a -[(non-standards compliant)](https://stackoverflow.com/questions/73542215/leading-underscores-in-linux-kernel-programming) -design choice: -identifiers that begin with an underscore -are more "internal" than those without one, -and those two are are extra internal. -While internal is doing a lot of heavy lifting -in that sentence, the context of each usage clarifies the details. -A less "internal" API function may be -[exported as a symbol](https://docs.kernel.org/core-api/symbol-namespaces.html) -to the rest of the kernel, -while a more "internal" identifier may provide an entry point -to a kernel function that skips certain locking steps, -or in case of -[`_copy_from_user()`](https://elixir.bootlin.com/linux/v6.6/source/include/linux/uaccess.h#L143), -[permissions and protection checks](https://unix.stackexchange.com/questions/674962/why-are-copy-from-user-and-copy-to-user-needed-when-the-kernel-is-mappe). -In the case of the `GFP_*` flags above, -the dunder versions are declared as such -to hint to kernel engineers that these flags -are generally not used directly like the non-dunder versions. - -As can be validated by a `ctrl+f`, -our kernel module uses `GFP_KERNEL`. -This is because we are running in the context of -a user process and therefore it's ok if the -codepath of the allocation includes a sleep or two -before returning to the caller. -We may even schedule out and switch processes multiple times -before the allocation spits out the needed valid memory address. -However, the CPU may be executing code in a context -where sleep is not only undesirable, -but theoretically terminal for the entire system. -One example of such a context is within the -[top-half or bottom-half](https://static.lwn.net/images/pdf/LDD3/ch10.pdf#page=18) -of an interrupt handler. - -The crucial topic of kernel context -deserves its own thorough treatment, -so we will only briefly touch upon it here. -The essential difference for our purpose -is that kernel code can sleep in user context, -while it cannot sleep in atomic or interrupt context. -In process context, we have a process associated with -the running kernel thread, though the immediate business -of the kernel may not be directly relevant to that particular -userspace process. -These kernel threads can copy data to or from userspace memory, -send signals to the current process, -and generally muck around with the -[`struct task_struct`](https://elixir.bootlin.com/linux/v6.6/source/include/linux/sched.h#L743) -found by dereferencing the address the `current` macro resolves to. -On the other hand, -a kernel thread running in interrupt or atomic context -is not associated with any userspace process. -Though `current` will point to the process whose execution -this kernel thread is interrupting, -this thread must accomplish its business as soon as possible. -It cannot sleep at all, -so any memory allocation must return immediately. -The `GFP_NOWAIT` flag requests this behavior with less urgency, -however the `GFP_ATOMIC` flag marks the allocation request with -a huge, red, bold exclamation mark attached, -and requests to be fed with the emergency reserves in the case of low memory. -This is sane, as we would like something like our keyboard to be able to -send interrupts that are immediately received and processed, -even when the bloated closed-source -software we run by choice or by force decides to consume all of our system resources. - -tl;dr just use `GFP_KERNEL` unless you have a good reason not to. - - -At last, we move on to the changes to our classical userspace entry point. - -**Entry to the other side** - -``` --int main(void) -+int example_init(void) -``` - -This change simply renames `main` to `example_init`. -Do not take this for any sort of magic -as this is nothing but a naming convention -whose purpose will be discussed near the bottom -of this diff analysis. -We could just as well call our module initialization function `main`, -but this would be confusing. -The demotion of this function -from the known entry point styled `main` -sets our footing loose from -that familiar foundation -of the userspace coding environment, -and we will return to this concern -near the bottom of this diff analysis. - -``` -+ int ret = -ENOMEM; -``` - -While the classic idiom of a -print to standard error and -nonzero-argument invocation of the exit syscall -consolidated with the `err()` API call suited our needs -quite satisfactorily back in Kansas, -we will find this exit strategy -falls flat on its face here in Oz. -To begin with, this exit strategy -relies on the invocation of a system call, -that is to say, -an explicit invocation of the kernel by userspace code, -and more specifically, a request for the kernel -to terminate the calling process. -As we are already executing in kernel mode, -there is no need to invoke ourselves, -and we certainly don't wish to commit suicide -on behalf of anyone in the failure case, -least of all on behalf of the kernel itself. -Instead, as the userspace integral file descriptor -is to the kernelspace `struct file`, -the thread-local userspace integral errno variable is -to the kernelspace negative integral errno value. -Though the specific reason for the convention of negativity -is unimportant and perhaps -[unknowable](https://stackoverflow.com/questions/1848729/why-return-a-negative-errno-e-g-return-eio), -one should take note of the convention itself. -We default to the negated out-of-memory errno value of -[-ENOMEM](https://elixir.bootlin.com/linux/v6.6/source/include/uapi/asm-generic/errno-base.h#L16) -as the return code for our function since -that is the only error we check for. -Once we confirm that we are in fact able -to allocate the necessary memory, -we set this value to zero. -One may frequently see code -that defaults the value of the return code to zero. -A careful treatment of that flamewar -is beyond the scope of this section. - -When one of these errno return values is propagated -all the way back to userspace in the context -of a systemcall, -the userspace caller will then be able to -access this value via the thread-local errno variable. - -Keep in mind that a thread-local variable in userspace -corresponds to a per-task variable from the perspective of kernelspace. -A process ID in kernelspace, known as a `pid`, corresponds one-to-one -with a userspace thread ID, known as a `tid`. -Confusingly, a userspace process is identified by -the more common usage of the same term "process ID" or `pid`, -which contains one or more threads, each identified by -a unique thread ID, or `tid`. -When a userspace process contains but a single thread, -the `pid` and the `tid` are the same, -and the kernelspace `pid` refers to the `struct task_struct` -representing the single userspace thread. -For a multi-threaded userspace process, -a userspace `pid` is associated with multiple `tid` values, -and each of these userspace `tid` values corresponds -one-to-one with a kernelspace `pid` value and a representative -`struct task_struct` as the Linux implementation of -the more general concept of a -[Process control block](https://en.wikipedia.org/wiki/Process_control_block). -These threads are grouped together logically, -and so as one might expect, the kernel refers to -the collection of kernelspace `pid` values grouped -under a single userspace `pid` value as userspace `tid`s -by the term "Thread-group ID", abbreviated as `tgid`. - -To summarize: - -|Concept|Userspace name|Kernelspace name| -|--|--|--| -|Single thread|`tid`|`pid`| -|Logical Process|`pid`|`tgid`| - -**Buffering with style** - -```c -+ const char *msg; - struct example *ex = example_create("hello"); -+ msg = KERN_ERR "unable to allocate memory"; -``` - -Though this construct appears strange at first glance, -we will quickly demystify this last assignment -with a quick exposition of C string syntax. -[Section 6.4.5](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3096.pdf#page=85) -of the C standard defines -the syntax of a string literal. -As a C-literate reader should expect, -a string literal is defined to be a series of characters -from a slight restriction of the character set called "s-chars" -in between terminating double quote characters. -Optionally, the string may be prefixed by what the standard terms an "encoding prefix" -but the details of that are not important here. -To quote the 1 April 2023 working draft, an "s-char" is: -"any member of the source character set except -the double-quote ", backslash \, or new-line character". -[Section 5.1.1.2](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3096.pdf#page=29) -specifies the order of precedence for translation stages during compilation, -and we see that item 6 clearly states that: -"Adjacent string literal tokens are concatenated." -Therefore, by process of elimination and -before even looking up the definition of `KERN_ERR`, -we know that `KERN_ERR` must be a string literal -because this code compiles and we have no other option. -That covers the syntactic mystery, -but it does not explain the semantics of this statement. - -Allow us one more quick detour that will be necessary just below. -[Section 6.4.4.4](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3096.pdf#page=82) -of the standard specifies various character constants, -including the encoding prefixes we mention just above. -We see that an "octal-escape-sequence" is a valid "escape-sequence", -and that it is specified with one, two, or three octal digits following a backslash. -The "octal-escape-sequence" is the only one -which is implemented with no character between the backslash -and the value itself. -For example, one begins a hexadecimal escape sequence using "\x", -and a -[universal character name](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3096.pdf#page=75) -using a "u" or "U". - - -Let us turn to the -[definition](https://elixir.bootlin.com/linux/v6.6/source/include/linux/kern_levels.h#L11) -of this symbol in the `kern_levels.h` header, -whose brief 39 lines we will include in their entirety from the v6.6 source: - - -```c -$ cat include/linux/kern_levels.h -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef __KERN_LEVELS_H__ -#define __KERN_LEVELS_H__ - -#define KERN_SOH "\001" /* ASCII Start Of Header */ -#define KERN_SOH_ASCII '\001' - -#define KERN_EMERG KERN_SOH "0" /* system is unusable */ -#define KERN_ALERT KERN_SOH "1" /* action must be taken immediately */ -#define KERN_CRIT KERN_SOH "2" /* critical conditions */ -#define KERN_ERR KERN_SOH "3" /* error conditions */ -#define KERN_WARNING KERN_SOH "4" /* warning conditions */ -#define KERN_NOTICE KERN_SOH "5" /* normal but significant condition */ -#define KERN_INFO KERN_SOH "6" /* informational */ -#define KERN_DEBUG KERN_SOH "7" /* debug-level messages */ - -#define KERN_DEFAULT "" /* the default kernel loglevel */ - -/* - * Annotation for a "continued" line of log printout (only done after a - * line that had no enclosing \n). Only to be used by core/arch code - * during early bootup (a continued line is not SMP-safe otherwise). - */ -#define KERN_CONT KERN_SOH "c" - -/* integer equivalents of KERN_ */ -#define LOGLEVEL_SCHED -2 /* Deferred messages from sched code - * are set to this special level */ -#define LOGLEVEL_DEFAULT -1 /* default (or last) loglevel */ -#define LOGLEVEL_EMERG 0 /* system is unusable */ -#define LOGLEVEL_ALERT 1 /* action must be taken immediately */ -#define LOGLEVEL_CRIT 2 /* critical conditions */ -#define LOGLEVEL_ERR 3 /* error conditions */ -#define LOGLEVEL_WARNING 4 /* warning conditions */ -#define LOGLEVEL_NOTICE 5 /* normal but significant condition */ -#define LOGLEVEL_INFO 6 /* informational */ -#define LOGLEVEL_DEBUG 7 /* debug-level messages */ - -#endif -``` - -By examination of the above header, -we observe the resolved value of `KERN_ERR` -to be a string literal containing two bytes, -the "\001" octal escape sequence which represents -"start of heading" in the -[ASCII](https://man7.org/linux/man-pages/man7/ascii.7.html) -standard, followed by the ASCII character literal "3", -which can just as easily be represented using "\063", -however the kernel chooses to be readable. - -This may be obvious by this point, -but these bytes are used to specify -the relatively -[well-documented](https://www.kernel.org/doc/html/latest/core-api/printk-basics.html) -kernel logging level. -The usage of usage of the `KERN_*` prefix before a string literal -is generally done within the parenthesis of a `printk()` invocation, -such as the one we use later in the code, -however we assign the resulting string value to a local `char *` variable -to demonstrate what is really going on -and dispel any illusions the reader may hold. -We believe this more verbose, -multi-step usage is less likely to trigger -that part of the trained C programmer's brain -which says that there is a comma missing. - -Though direct usage of `printk()` is acceptable, -we recommend the usage of the `pr_*` macros -described in the -[printk documentation](https://www.kernel.org/doc/html/latest/core-api/printk-basics.html), -as these helpful wrappers will prevent -one's attempted kernel build -from generating strange-looking macro-resolution errors -in the case one makes a typo. -Usage of `KERN_ERROR` is such an example. -We believe it will be easier to spot the error -when one attempts to build kernel code containing -the alternative equivalent mistaken usage of `pr_error()` -in place of the correct `pr_err()`. -In addition, the code is cleaner and shorter when -using the `pr_*` family of functions, -and you can define customized wrappers on a per-file basis -by redefining `pr_fmt()`, which we will explain below. - -``` - if(!ex) -- err(1, "unable to allocate memory"); -- printf("%s\n", example_get_message(ex)); -- if(!example_update_message(ex, "goodbye")) { -- int temperrno = errno; -- example_destroy(ex); -- errno = temperrno; -- err(1, "unable to update"); -- } -- printf("%s\n", example_get_message(ex)); -+ goto out; -``` - -With our handy `goto` statement, -we can dispose of all that mid-function -error handling code and consolidate the codepaths -of this function to flow through a single exit point. - -``` -+ -+ pr_info("%s\n", example_get_message(ex)); -+ -+ msg = KERN_ERR "unable to update\n"; -+ if(!example_update_message(ex, "goodbye")) -+ goto out_free; -+ -+ pr_info("%s\n", example_get_message(ex)); -``` - -Here, we make use of the `pr_info()` macro helper -to do exactly what `printk()` would have done, -but without having to include that strange looking -syntax prefixing the format string with a macro -separated by nothing but whitespace. -Actually, as we mention above, -the `pr_*` family provides one extra feature -that we do not use but we feel is worth a quick discussion. - -The -[definition](https://elixir.bootlin.com/linux/v6.6/source/include/linux/printk.h#L528) -of `pr_info()` -passes the format string wrapped with yet another macro, -this being `pr_fmt`. -As the -[API documentation](https://www.kernel.org/doc/html/latest/core-api/printk-basics.html#c.pr_fmt) tells us, -we can define a custom format to be used each time -a `pr_*` macro is subsequently invoked in that translation unit. -The example given in the documentation is -yet another macro, `KBUILD_MODNAME`, -Which is resolved at build time by -[Kbuild](https://docs.kernel.org/kbuild/kbuild.html), -the Linux kernel's bespoke build system, -and a flag set by -[scripts/Makefile.lib](https://elixir.bootlin.com/linux/v6.6/source/scripts/Makefile.lib#L126) -is passed to the compiler, defining this value appropriately in each context. -This is common, but one may use any string they like, -or leave out the definition entirely, as we do in this module. - -These two invocations of `pr_info()` -are the kernelspace replacements -for the two `printf()` calls back in our userspace code, -and here too, the success of these two calls results in -the strings "hello" and "goodbye" appearing in some external buffer. - -**Three clean exits** - -``` -+ ret = 0; -``` -The value of `ret` before this assignment is `-ENOMEM`, -so we must clear the error and set the return value -to 0, which indicates success. - -``` -+ msg = NULL; -``` - -As the `msg` variable contains an error message, -we set it to `NULL` to skip the invocation of `printk()` just below. - -``` -+out_free: - example_destroy(ex); -- return 0; -+out: -+ if(msg) -+ printk(msg); -+ return ret; -+} -``` -Finally, we conclude the definition of `example_init()` -by overlapping three exit cases together -using the `goto` statements defined earlier and the two labels -we define just above. -This is less complex than it may seem, -and as you may notice, we only use one level of indentation. - -First, the success case. -If all goes right, the CPU arrives at the code following -the `out_free` label, continues right along -after invocation of `example_destroy()`, -moves right past `out`, and jumps past the `printk()` -due to the `NULL` value of `msg` set just above. -We return with the value of `ret` set to `0`, -which is also taken care of just above, -and that's that for an error-free execution of `example_init()`. - -Second, our first call to `kmalloc()` to allocate memory -for a `struct example` may fail. -Then, our error-checking conditional leaves us -on the `goto out;` line just following, -and right away, the CPU is then executing just below the `out` label. -At this point, the value of `msg` is -the string "unable to allocate memory" -prefixed by "\001" "3", a.k.a `KERN_ERR`. -As this value is in fact not `NULL`, -we pass it to `printk()` -and we expect to see this string show up in our kernel ring buffer. -As always, we can check this with `dmesg`. -To conclude, we return the value of `ret`, -which is unmodified since its initialization and declaration -and therefore is `-ENOMEM`, which is correct. - -Third and finally, -we may succeed in allocating memory -for a `struct example`, -but then fail somewhere in `example_update_message()`, -which is indicated by a logically false, i.e. `0` return value -from the conditional wrapped invocation. -We can inspect this short function -and see that this failure can only happen in a single case, -and that case is also failed allocation, -but this detail is not important here. -What is important in the context of handling this error in the caller -is that we are responsible for `free`ing the memory -we allocated just before this to store our `struct example`. -If we were to simply return to the caller of `example_init()` right now, -not only would we lose the syntactically clean unified exit path, -we would generate a memory leak. -We also want to print the contents of the string data -at `msg`'s address to the kernel ring buffer, -and for whatever remains of brevity in this example, -we don't bother modifying the contained string. -Therefore, we jump over the second `pr_info()` invocation -and the assignment of appropriate success-case values -to `ret` and `msg`, -and immediately invoke `example_destroy()` on the address -we obtained from the initial and successful call to `kmalloc()`. -This closes the loop in terms of allocation -and prevents the module from leaking memory. -Do not forget that the severity of a memory leakage in the kernel -is almost always far greater than in a user program, -especially a short-lived one. -As you may recall from the exposition above, -should we modify our example user program program above -to leak memory, which can be implemented by the removal -of one or more calls to `free()`, we can easily debug -the issue with `valgrind`, -and regardless, -the kernel will clean up our mess -upon termination of the process and its threads. -In the kernel, every similar memory leak -will persist until the system is reset. -We emphasize this to illustrate the importance -of correctly managing the memory of kernel code -even in the more subtle codepaths such as this third case. -Once we free the memory at `ex`, -the non-`NULL` value of `msg` triggers the call to `printk()` -just as in the second case, -and finally, -we return `-ENOMEM`, -also just like the second case. - -``` -+void example_exit(void) -+{ - } -``` -We define this empty function because we need -to give a callable address with a particular type signature -to the kernel's module subsystem. -This is explained just below. - -**The final plumbing** - -``` -+module_init(example_init); -+module_exit(example_exit); -``` - -In order to properly explain -these two macro invocations, -we first need to take a step back -and talk about the bigger picture. - -We are translating a C program designed -to compile into an executable binary file -that creates a single thread and interacts with Linux from userspace -into a C program designed to -compile into the Linux implementation of a -[loadable kernel module](https://en.wikipedia.org/wiki/Loadable_kernel_module) -that interacts with the kernel API -and expects to run on a CPU in privileged mode. -To build and run this code, -we first need to write an idiomatic makefile -and make sure the files necessary to build -modules specifically for the running or target kernel -are present in their expected locations. -When all of this is in place, -we can build a "kernel object" file, -whose filename is canonically but meaninglessly -suffixed with ".ko". -Using a utility like -[`insmod`](https://git.kernel.org/pub/scm/utils/kernel/kmod/kmod.git/tree/tools/insmod.c) -we can pass this kernel object -to either the -[`init_module(2)` or `finit_module(2)`](https://man7.org/linux/man-pages/man2/init_module.2.html) -syscall, though in practice the `insmod` and `modprobe` utilities from -[kmod](https://git.kernel.org/pub/scm/utils/kernel/kmod/kmod.git) exclusively invoke the -[latter](https://git.kernel.org/pub/scm/utils/kernel/kmod/kmod.git/tree/shared/missing.h#n36) -due to an engineering preference for working with file descriptors. - -This syscall loads the module into kernel memory, -and if needed, relocates symbols and initializes module parameters. -After this, the kernel invokes the module's `init` function. -Now as we have made abundantly clear by now, -the `main` function that the C standard so generously specifies in -[section 5.1.2.2.1](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3096.pdf#page=31) -is not relevant to a Linux kernel module. -Instead of using a pre-defined name as our entry point, -we simply set the module's init function -to the address of a function of our choice -with the only constraint being the type signature, -which must be `int (*)(void)`. -Within the definition of the intuitively-named -[struct module](https://elixir.bootlin.com/linux/v6.6/source/include/linux/module.h#L402), -we find a member named -[`init`](https://elixir.bootlin.com/linux/v6.6/source/include/linux/module.h#L458), -with just the type signature we expect. -This `init` member holds the address of the init function -defined by a given module and -this `struct module` is the in-kernel representation -of a Linux kernel module. -Likewise, when we wish to unload a kernel module, -we use a tool such as -[`rmmod`](https://git.kernel.org/pub/scm/utils/kernel/kmod/kmod.git/tree/tools/rmmod.c) -or the -[removal mode](https://git.kernel.org/pub/scm/utils/kernel/kmod/kmod.git/tree/tools/modprobe.c#n114) -of modprobe, -which passes the name of the module into the kernel by way of the -[`delete_module(2)`](https://man7.org/linux/man-pages/man2/delete_module.2.html) -syscall. -After checking whether the supplied name refers to -an extant loaded kernel module -with no outstanding references -held by other modules, -the kernel checks whether an `exit` function -is defined for the module. -If so, it is invoked before the module is unloaded. -The address of this `exit` function, -just like `init`, -is stored within a module's `struct module` -as a member helpfully named -[`exit`](https://elixir.bootlin.com/linux/v6.6/source/include/linux/module.h#L568). -While specification of a module `init` function is mandatory, -specification of an `exit` function is not. -If we don't ever need or want to unload the module, -then `exit` will never be called, -so we can exclude it entirely. -Since we do in fact wish to be able to unload our module -but we don't have anything to cleanup at unload time, -we simply define a dummy function and set `exit` to its address. - -We now return to the point -from which we took a step back, -namely, the usage of the `module_init` and `module_exit` -seen just above. -This is the method we use -to set the `init` and `exit` members -of the soon-to-be-generated `struct module` -that will be packaged into the kernel object file -by the kernel build system. -The two macros are -[defined](https://elixir.bootlin.com/linux/latest/source/include/linux/module.h#L130) -one right after the other. -The first part of the definition may initially bamboozle the reader, -however when we take away the semantically irrelevant -[`static` storage class specifier](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3096.pdf#page=116), -the -[`inline` function specifier](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3096.pdf#page=141), -and -[`unused` function attribute](https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-unused-function-attribute) -hiding right behind the -[`__maybe_unused` macro](https://elixir.bootlin.com/linux/v4.14/source/include/linux/compiler-gcc.h#L128), -we find the definition of a dummy function named `__inittest` -which takes no arguments and returns a value of type `initcall_t`. -The sole statement in the function body returns the address -of our candidate to be set as the module's `init` function. -The `unused` attribute hints at the true intent of this function. -The compiler will throw an error if the type signature of our chosen function -differs from that of -[initcall_t](https://elixir.bootlin.com/linux/v6.6/source/include/linux/init.h#L118). -This dummy function implements that compliance check at compile time, -saving all users the headache of debugging the runtime consequences -of a module author mistakenly using something exotic and noncompliant. -The final line is the business end of the macro definition. -We declare a function named `init_module` with the same type signature as `initcall_t`. -Then, we utilize the -[alias function attribute](https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-alias-function-attribute) -to bind the address of our `init` function to the `init_module` symbol. -One of the artifacts generated and used by a kernel module build -is a file given the same filename as the primary C program source, -but with a `.mod.c` extension replacing the simple `.c`. -This file contains the following snippet, or something similar: - -```c -__visible struct module __this_module -__section(".gnu.linkonce.this_module") = { - .name = KBUILD_MODNAME, - .init = init_module, -#ifdef CONFIG_MODULE_UNLOAD - .exit = cleanup_module, -#endif - .arch = MODULE_ARCH_INIT, -}; -``` - -Thus the `init` function of our choosing is set as the module's `init` function -and made available to the kernel on a module load via its inclusion in this -generated `struct module`. -The `cleanup_module` function is similarly generated -by the `module_exit` macro. -This is how the kernel knows about the `example_init` and `example_exit` -functions in our kernel module example. - - -``` -+MODULE_LICENSE("GPL"); - -``` - -This line is required. -If we attempt to build our module without it, -we find that the modpost stage of the kernel build system -[complains and explodes](https://elixir.bootlin.com/linux/v6.6/source/scripts/mod/modpost.c#L1731). -This suicide stems from a failed check for a string -beginning with "`license=`" -in the module binary. -This string is emitted by the -[definition of `__MODULE_INFO`](https://elixir.bootlin.com/linux/v6.6/source/include/linux/moduleparam.h#L23) -which is the powerhouse of the -[`MODULE_LICENSE` macro](https://elixir.bootlin.com/linux/v6.6/source/include/linux/module.h#L230). -We use the string `"GPL"` to refer to the -[GNU General Public License](https://en.wikipedia.org/wiki/GNU_General_Public_License), -specifically -[version 2](https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html). -Our usage of this license designation in the module source code -assigns this free software license to our code. -This allows any individual or company to ensure -that their usage of our module or any other -complies with their legal and philosophical constraints. - -#### The Emerald City - -We have reached the end -of the yellow brick road -of our diff. -Although everything you need -to generate the final kernel driver is above, -we include the final result right here -for emphasis and ease. - -```c -$ cat kernel_code.c -#include -#include -#include - -struct example -{ - char *message; - size_t size; -}; - -static struct example *example_create(const char *msg) -{ - struct example *ex = kmalloc(sizeof *ex, GFP_KERNEL); - if(!ex) - goto out; - ex->size = strlen(msg); - ex->message = kstrdup(msg, GFP_KERNEL); - if(!ex->message) - goto out_free; - return ex; -out_free: - kfree(ex); - ex = NULL; -out: - return ex; -} - -static void example_destroy(struct example *ex) -{ - kfree(ex->message); - kfree(ex); -} - -static bool example_update_message(struct example *ex, const char *msg) -{ - size_t size = strlen(msg); - char *data = kstrdup(msg, GFP_KERNEL); - if(!data) - return false; - kfree(ex->message); - ex->message = data; - ex->size = size; - return true; -} - -static char *example_get_message(struct example *ex) -{ - return ex->message; -} - -int example_init(void) -{ - int ret = -ENOMEM; - const char *msg; - struct example *ex = example_create("hello"); - msg = KERN_ERR "unable to allocate memory"; - if(!ex) - goto out; - - pr_info("%s\n", example_get_message(ex)); - - msg = KERN_ERR "unable to update\n"; - if(!example_update_message(ex, "goodbye")) - goto out_free; - - pr_info("%s\n", example_get_message(ex)); - - ret = 0; - msg = NULL; -out_free: - example_destroy(ex); -out: - if(msg) - printk(msg); - return ret; -} - -void example_exit(void) -{ -} - -module_init(example_init); -module_exit(example_exit); - -MODULE_LICENSE("GPL"); -``` - -For further convenience, -we include an idiomatic makefile -which will build the above kernel module -on a properly configured system. - -```Makefile -$ cat Makefile -obj-m += kernel_code.o - -.PHONY: build clean load unload - -build: - make -C /lib/modules/$(shell uname -r)/build modules M=$(shell pwd) -clean: - make -C /lib/modules/$(shell uname -r)/build clean M=$(shell pwd) -load: - sudo insmod kernel_code.ko -unload: - -sudo rmmod kernel_code -``` diff --git a/kdlp.underground.software/course/fall2023/lectures/L18.md b/kdlp.underground.software/course/fall2023/lectures/L18.md deleted file mode 100644 index 49fdca31..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L18.md +++ /dev/null @@ -1,6 +0,0 @@ -# Lecture 18 - 9 November 2023 - -## Topics Covered: -* Mid point presentation -* Relation between file description and file descriptor -* traditional char dev creation boilerplate diff --git a/kdlp.underground.software/course/fall2023/lectures/L19.md b/kdlp.underground.software/course/fall2023/lectures/L19.md deleted file mode 100644 index 1fddb058..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L19.md +++ /dev/null @@ -1,4 +0,0 @@ -# Lecture 18 - 9 November 2023 - -## Topics Covered: -* Mid point presentations diff --git a/kdlp.underground.software/course/fall2023/lectures/L20.md b/kdlp.underground.software/course/fall2023/lectures/L20.md deleted file mode 100644 index da2f0759..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L20.md +++ /dev/null @@ -1,6 +0,0 @@ -# Lecture 20 - 16 November 2023 - -## Topics Covered: -* Mid point presentations -* Converting legacy char device to misc device -* `container_of` diff --git a/kdlp.underground.software/course/fall2023/lectures/L21.md b/kdlp.underground.software/course/fall2023/lectures/L21.md deleted file mode 100644 index c867d82b..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L21.md +++ /dev/null @@ -1,5 +0,0 @@ -# Lecture 21 - 21 November 2023 - -## Topics Covered: -* Mid point presentations -* Happy thanksgiving diff --git a/kdlp.underground.software/course/fall2023/lectures/L22.md b/kdlp.underground.software/course/fall2023/lectures/L22.md deleted file mode 100644 index 32217290..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L22.md +++ /dev/null @@ -1,4 +0,0 @@ -# Lecture 22 - 28 November 2023 - -## Topics Covered: -* userspace locking diff --git a/kdlp.underground.software/course/fall2023/lectures/L23.md b/kdlp.underground.software/course/fall2023/lectures/L23.md deleted file mode 100644 index 2ee48e44..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L23.md +++ /dev/null @@ -1,4 +0,0 @@ -# Lecture 23 - 30 November 2023 - -## Topics Covered: -* kernel locking diff --git a/kdlp.underground.software/course/fall2023/lectures/L24.md b/kdlp.underground.software/course/fall2023/lectures/L24.md deleted file mode 100644 index 936e32d7..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L24.md +++ /dev/null @@ -1,4 +0,0 @@ -# Lecture 24 - 5 December 2023 - -## Topics Covered: -* live module debugging diff --git a/kdlp.underground.software/course/fall2023/lectures/L25.md b/kdlp.underground.software/course/fall2023/lectures/L25.md deleted file mode 100644 index c1b344e0..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L25.md +++ /dev/null @@ -1,4 +0,0 @@ -# Lecture 25 - 7 December 2023 - -## Topics Covered: -* live module debugging cont diff --git a/kdlp.underground.software/course/fall2023/lectures/L26.md b/kdlp.underground.software/course/fall2023/lectures/L26.md deleted file mode 100644 index ca68d3af..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L26.md +++ /dev/null @@ -1,4 +0,0 @@ -# Lecture 26 - 12 December 2023 - -## Topics Covered: -* Guest lecture - Memory Barriers diff --git a/kdlp.underground.software/course/fall2023/lectures/L27.md b/kdlp.underground.software/course/fall2023/lectures/L27.md deleted file mode 100644 index f80ff0d0..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L27.md +++ /dev/null @@ -1,4 +0,0 @@ -# Lecture 27 - 14 December 2023 - -## Topics Covered: -* Final Presentations diff --git a/kdlp.underground.software/course/fall2023/lectures/L28.md b/kdlp.underground.software/course/fall2023/lectures/L28.md deleted file mode 100644 index a3205f89..00000000 --- a/kdlp.underground.software/course/fall2023/lectures/L28.md +++ /dev/null @@ -1,4 +0,0 @@ -# Lecture 28 - 19 December 2023 - -## Topics Covered: -* Final Presentations diff --git a/kdlp.underground.software/course/fall2023/policies/course_policies.md b/kdlp.underground.software/course/fall2023/policies/course_policies.md deleted file mode 100644 index 07ac09ce..00000000 --- a/kdlp.underground.software/course/fall2023/policies/course_policies.md +++ /dev/null @@ -1,98 +0,0 @@ -## Course Policies - -**Assignment Procedures:** - -* Coding Assignments: - * Assignments must be submitted in the format of plaintext unless explicitly specified. - * Assignments must be submitted in the form of email code patches, and furthermore: - * Submissions must conform to all Linux kernel coding standards that are sane in a particular context. - * Submissions must not contain whitespace errors - * The instructors will provide detailed guidelines for automated checking of these requirements, and will gladly explain them upon request to any interested student - * Therefore the instructors reserve the right to refuse to grade any submission that does not conform to these standards. - * Students will participate in peer review for these assignments, as explained in the section below. - * Any code submission that fails to compile with zero warnings and zero errors will receive a zero. - -* Presentation Assignments: - * Students will be required to complete two (2) presentations throughout the course - * Students must deliver their presentations over a live video call - * Presentations must include original visual content, such as slides - * Students are required to include live demonstrations of their code by sharing their text editor and/or terminal during the presentation - -* Instructors reserve the right to change due dates within reason - * In such a case, we will provide sufficient advance notice to students - -**Peer Review Procedure (only for coding assignments)** - -* Students must participate in the class mailing list for submissions by giving other students feedback on their work. -* This is a part of each assignment grade and the tripartite scheme is as follows: - * The review process spans 2 days. - * Part 1: Initial submission - * The student makes their submission to the mailing list using git-send-email. - * This submission is due on the listed assignment "due date", see the [course schedule](course_spring2023.html). - * If the initial submission is late, the student will get a zero on the entire assignment. - * Part 2: Peer review - * A student is given two other student names (call them student A and student B). - * The student is assigned to review student A's and student B's submissions. - * If the student approves of a submission, then the student will reply to the approved email with what we call an "ack". - * An "ack" consists of a single line containing the following: Acked-by: Firstname Lastname . - * If the student sees problems with a submission, the student will reply to the problematic email with their feedback. - * In parallel, other students have been assigned the student's submission and the student should recieve feedback from two other students. - * These reviews are due one day past the initial submission, a late or no submission results in a zero for the review section of the assignment. - * Reviews are graded based on how many issues a student missed. - * The student receives 20% off for each unique issue not spotted with max penalty of 100%. - * Part 3: Final submission - * The student, if canny, will act on the feedback from the received reviews. - * Regardless of whether the student made changes to their initial submission, they must make a final submission. - * This is due two days after the initial submission, and due one day after receiving the reviews. - * Overall assignment grading scheme (percentages are relative to the total assignment grade): - * The initial submission is purely for the peer-review process. It is crucial that the initial submission is made on time, otherwise, as stated above, the student will recieve a zero on the entire assignment. - * Review 1: 10% - * Review 2: 10% - * Final Submission: 80% - -**Late Work Policy:** - -* There is no late work! - * Students receive a zero for assignments not submitted on time. -* One exception (a once per semester get out of jail free card): - * Students may choose to opt out of the peer review process. - * By bypassing the ack requirement (both giving and receiving) students receive the two days that would be used for that process as an extension before they make their first and only final submission. - * This means that you do you not get a chance to receive early feedback from your peers and so you get what you get for a final grade. - -**Mentorship Graduation:** - -* Students and mentees will be given the status of “Graduated” if we determine that their performance has been suitable to the degree that we would recommend as a strong candidate for an internship doing basic Linux kernel engineering work. - -**Grading Policies:** - -* Automatic Course Failure (Grade of F) - * In the case that a student fails to complete a task by a deadline with no prior notice, an instructor will reach out to you via email and/or direct message. If the student fails to respond to this message within seven (7) days of it’s receipt, the instructors reserve the right to give the student a failing grade (F) for the course. -* Attendance is... required. -* Student grades will be weighted according to the following scheme: - -| Category | Percentage | -|--|--| -| Programs | 30% | -| Exercises | 20% | -| Midpoint Presentation | 15% | -| Final Assignment + Final Presentation | 30% | -| Participation | 5% | - - -Students will be given a letter grade according to the following this scheme: - -|Letter grade|Percentage range| -|--|--| -|A |90 ~ 100| -|A-| 85 ~ 89.99| -|B+| 80 ~ 84.99| -|B |75 ~ 79.99| -|B-| 70 ~ 74.99| -|C+| 65 ~ 69.99| -|C |60 ~ 64.99| -|C-| 55 ~ 59.99| -|D+| 50 ~ 54.99| -|D |40 ~ 49.99| -|F |below 40| - -######A (4.0), A- (3.7), B+ (3.3), B (3.0), B- (2.7), C+ (2.3), C (2.0), C- (1.7), D+ (1.3), D (1.0), F (0.0) diff --git a/kdlp.underground.software/course/fall2023/policies/index.md b/kdlp.underground.software/course/fall2023/policies/index.md deleted file mode 100644 index 9904b79e..00000000 --- a/kdlp.underground.software/course/fall2023/policies/index.md +++ /dev/null @@ -1,10 +0,0 @@ - -## Latest Policy Documents - -[Course Policies](course_policies.md) - -[Submission Guidelines](submission_guidelines.md) - -[Peer Review Process](peer_review.md) - -[Presentation Guidelines](presentation_guidelines.md) diff --git a/kdlp.underground.software/course/fall2023/policies/peer_review.md b/kdlp.underground.software/course/fall2023/policies/peer_review.md deleted file mode 100644 index d453a05c..00000000 --- a/kdlp.underground.software/course/fall2023/policies/peer_review.md +++ /dev/null @@ -1,15 +0,0 @@ -### How to submit a code review - -* You will be assigned two students to peer review shortly after you complete your own initial submission. -* For each student, which you are assigned, you will do the following: - * Locate their submissions and download all of the patches in the patchset (besides the cover letter). - * Make a new branch off of master/main and switch over to that branch to apply the patches. - * Make sure each patch applies cleanly IN ORDER (a.k.a no corrupt patches, whitespace errors, etc.) - * You can run checkpatch.pl on the patches, ignore any errors pertaining to maintainers. - * Make sure the code compiles without warnings or errors. - * Make sure the program runs without crashing. - * Make sure that the output is correct (passes all tests, if applicable). - * If there are any problems with the submission, document said issues and report them in your email reply to the student who you are reviewing. - * If you determine that there are no issues with the submission, inform the recipient. - * If it turns out that there were issues with the submission that you missed, points will be deducted from your overall assignment grade. - * Refer [here](course_policies.md) for the grading policy regarding peer reviewing. diff --git a/kdlp.underground.software/course/fall2023/policies/presentation_guidelines.md b/kdlp.underground.software/course/fall2023/policies/presentation_guidelines.md deleted file mode 100644 index c91ba47d..00000000 --- a/kdlp.underground.software/course/fall2023/policies/presentation_guidelines.md +++ /dev/null @@ -1,43 +0,0 @@ -### Presentation Guidelines 🎯 🏹 - -Duration: Tentative till further notice. Midterm will be a bit shorter, final a bit longer, neither longer than 20m - -

- -Content: - -What have you learned so far by doing the assignments? - -- How did the assignments relate to what we learned in class in your own words - -What challenges have you faced and overcome working on these tasks? - -- Your assignment write ups should be useful here - -Assignments you think you did an exceptional job on - -- Why do you think so? Maybe demonstrate your competence/understanding - -What assignments did you enjoy or not enjoy and any feedback - -

- -Suggestions on what to prepare: - -- Visual demonstrations if you think they are helpful - -Live example of how your project functions - -- Could do something in the terminal live - -Maybe show your program’s inner workings - -- Explain your thought process - -

- -The presentation should demonstrate that you understand your work, you are able to explain what you did and why it matters to others, and you are able to show your thought process when you are faced with a problem. - -An example of a previous presentation can be found here. - -These are general guidelines for what we are looking for in a presentation. You do not need to follow it strictly but you should try to cover all of the areas at least somewhat. diff --git a/kdlp.underground.software/course/fall2023/policies/submission_guidelines.md b/kdlp.underground.software/course/fall2023/policies/submission_guidelines.md deleted file mode 100644 index dc379f0f..00000000 --- a/kdlp.underground.software/course/fall2023/policies/submission_guidelines.md +++ /dev/null @@ -1,120 +0,0 @@ -### Assignment Submission Guidelines - -All homework submissions must be submitted in the form of an email patchset [0] generated by git format-patch [1] from commits [2] made in your local copy of [this repository](http://prod-01.kdlp.underground.software/cgit/KDLP_assignments.git/) with a cover letter [3] describing your work, and sent by git send-email [4] to our mailing list. - -As part of the peer-review process for coding assignments in this class (described [here](course_policies.md)) each assignment will require you to submit each patchset at least twice [5]. - -Try to start your assignments early. If you run into issues and get stuck it gives you time to ask questions and get help before the due date so you can submit something on time and get credit for the assignment, and if you finish early you can submit and potentially get feedback from us or other students that you can incorporate into a resubmission before the deadline in accordance with the resubmission policy [6]. - -#### [0] Patchset Guidelines - -The specific assignment document will specify which files to edit, and how many commits you should be making which will inform the overall structure of your patchset, but every patchset in this class must follow these general guidelines: - -* Each commit gets its own patch with a title and body - -* The patch series is introduced with one additional patch, the cover letter - -Fortunately, git format-patch can generate the appropriate files for you automatically: - -$ git format-patch -3 --cover-letter - -$ # generates patches from the three most recent commits (change -3 to whatever you need for the amount of commits you have) and a cover letter template. - -All of the patches must follow the patch guidelines [1]. Those generated from a commit should follow the commit guidelines [2] and the cover letter must follow the cover letter guidelines [3]. - -You will recieve an automatic zero on the assignment if any of the patches in your patchset are corrupt. This shouldn't be possible if you generate your patches with `git format-patch`, but if you edit the files manually they might get corrupted. You have been warned! The correct way to edit the patches is to edit the underlying commits (see `man git-rebase` and the `--amend` option from `man git-commit`) and then regenerate the patches. - -#### [1] Patch Guidelines - -Every patch in the patch series (including the cover letter) must end with a “Signed-off-by” line, called the DCO (Developer Certificate of Origin). The line must exactly match this format: - -Signed-off-by: Firstname Lastname - -The DCO line must be the final line of the email body right before the start of the patch diff. - -Fortunately, you can make git add this line automatically for you when you author a commit: - -$ git commit -s - -$ # include a DCO Signed-off-by line in the commit message automatically - -You will need to remember to add your DCO to the cover letter manually. - -You should use the checkpatch.pl script in the scripts/ directory of Linus’ kernel tree to make sure your patch is as close to upstream Linux kernel patch standards as possible. - -#### [2] Commit Guidelines - -Within the repository for this class there is a directory for each assignment (e.g. for A0, look in the A0 directory, for Mid 1, look in Mid1, etc.) and within that directory there should be everything you need to get started after you have read the specific document for the assignment you are working on. - -You will make the necessary changes or additions and turn them into commits using git commit. - -When you author a commit the first line(s) you type into your editor will become the title, and by hitting enter twice and leaving a blank line the subsequent text will become the full commit message. The git format-patch utility will automatically put the title and message of a commit into the title and body respectively of the corresponding patch email it generates. - -Your commits should have a title that is a short summary of the changes in this commit and you should include any further details in the commit message. - -You should make sure that the changes you are including in your commits are tidy. This means that code should follow the [kernel code style guidelines](https://www.kernel.org/doc/html/latest/process/coding-style.html), (tabs for indentation, tab width of 8, no lines exceeding 80 columns, etc). - -You must also avoid whitespace errors. These include whitespace at the end of a line, lines with only whitespace on them, extra blank lines at the end of a file, forgetting the newline on the last line of the file, etc. A good editor will highlight and/or automatically fix these for you, but git will also detect these when formatting and applying patches. - -When you format a patch, if you forgot the newline at the end of the file git will put this line at the end of the diff \ No newline at end of file. If you see this, you should adjust the file contents and fix your patch. You can check for the other whitespace errors by running git am to try and apply your patch. If git am prints a warning like this when you apply the patch: warning: 2 lines add whitespace errors. You should adjust the indicated lines and fix your patch. - -Your patches also must apply cleanly to HEAD commit on the master branch of the upstream repository. You can verify this by checking out that branch and trying to apply your patches. We should NOT need to apply your previous versions of the patch in order for the latest version of your patchset to apply. - -Sample workflow to check that your patchset applies cleanly: - -* Generate your patches and put them in a known location and take note of the filenames -* Make sure your git tree is up to date. You should do this each time you begin work within any git repository. - * Use `git remote update` to update all of your local copies of remote trees. -* Create and checkout a local branch based on the upstream `origin/master` branch by using: - * `git checkout -b origin/master` (branch name can be anything convenient) -* Apply your patchset to this branch using `git am ... ` -* If there are no errors that appear, congratulations, your patchset applies cleanly! - * If there are whitespace errors or corrupt patches, revise as needed by amending or rebasing your commits. - -#### [3] Cover Letter Guidelines - -When you open the cover letter file generated by git format-patch in your editor, it will contain a summary of all the changes made in the subsequent patches at the bottom and two filler lines indicating where you can add the title (\*\*\* SUBJECT HERE \*\*\*) and message (\*\*\* BLURB HERE \*\*\*) to the email. - -* You must replace the subject filler With your name and the assignment title. For example Charles Mirabile - Assignment 3 - -* You must replace the blurb filler with your writeup - -* Failure to remove these filler lines including the asterisks will result in lost points - -Your cover letter writeup should include a short write-up that specifies the following: - -* What you think your degree of success with this assignment is (from 0% to 100%) - -* A brief discussion of your approach to the assignment - -* A detailed description of any problems that you were not able to resolve before submitting. - -Failure to specifically provide this information will result in a 0 grade on your assignment. Further, If you do not disclose problems in your write-up and problems are discovered when your work is reviewed, you will receive a grade of 0. Note: These writeup guidelines are adapted from those of [Prof. Bill Moloney](https://cs.uml.edu/~bill/) - -#### [4] Submitting to the Mailing List - -Your patches should be sent to the address for the specific assignment. Each assignment will list the appropriate email to use, but in general e0 -> exercise0@kdlp.underground.software, p0 -> programming0@kdlp.underground.software f0 -> final0@kdlp.underground.software and likewise for the subsequent numbers. - -$ git send-email --to=whatever_assignment@kdlp.underground.software *.patch - -$ # send the emails to the mailing list. The list of patch files are passed as arguments - -$ # The *.patch wildcard expands to all of the files whose names end with .patch in the current directory - -#### [5] Peer Review Submission Process - -The patch format should be as follows: -* Send rfc patches for the submissions that are to be peer review (add the `--rfc` option when you format your patchset). -* Send non-rfc patches for the final submission (omit the `--rfc` option). - -#### [6] Late Assignment Submissions - -Late submissions will result in an automatic grade of zero. The only exception is the one-time per semester oopsie (refer [here](course_policies.md)). - -#### [7] Resubmissions - -If you notice a mistake in your submission (or have it pointed out to you by one of the instructors or another student) before the due date, you are welcome to resubmit as many times as you'd like until said deadline. If you choose to resubmit, you must regenerate the patches from your (potentially altered) commit(s), and send the full patch series as your resubmission (i.e. a cover letter and one email for each commit even if some of the emails are identical to what you sent before). - -Further, you must make clear that it is a resubmission by adding a version number after PATCH in the square brackets (i.e. Subject: [PATCH x/n] My title here would become Subject: [PATCH v2 x/n] My title here) incrementing the number as needed for however many resubmissions you make (you can use the `-v` option with an argument, denoting the version number, when reformatting your patchset). - -You must also document what changed since the last submission in your writeup, include a section with a title like “changes since vN” where N is the number of your last submission and you explain what changed. diff --git a/kdlp.underground.software/course/index.md b/kdlp.underground.software/course/index.md deleted file mode 100644 index 93182b34..00000000 --- a/kdlp.underground.software/course/index.md +++ /dev/null @@ -1,18 +0,0 @@ -#### Table of courses - -|Semester|Notes| -|---|---| -|[Fall 2023](fall2023/index.md)|Ongoing at [UMass Lowell](https://uml.edu) and [LFX](https://mentorship.lfx.linuxfoundation.org/)| -|[Spring 2023](spring2023/index.md)|Fourth Iteration| -|[Fall 2022](fall2022/index.md)|Third iteration| -|[Spring 2022](spring2022/index.md)|Second iteration| -|Fall 2021|First iteration| -|Spring 2021|Initial attempt| - -[Course Dashboard](/dashboard) - -[Course Slides](slides/index.md) - -[KDLP Git](/cgit) - -[Course Assignments Repository](/cgit/KDLP_assignments.git) diff --git a/kdlp.underground.software/course/slides/advanced_char_driver_ops.html b/kdlp.underground.software/course/slides/advanced_char_driver_ops.html deleted file mode 100644 index dc355f8c..00000000 --- a/kdlp.underground.software/course/slides/advanced_char_driver_ops.html +++ /dev/null @@ -1,269 +0,0 @@ -
-

Advanced Character Driver Operations

-

In Linux

-

LDD3: Chapter 6

-
-
-

ioctl

-
    -
  • int (*ioctl) (struct inode *inode, struct file *filp, unsigned int cmd, unsigned long arg);
    • -
  • Control hardware -
      -
    • Lock door, eject media, error information
      • -
    -
    • -
  • Big switch statement that selects the correct behavior according to ‘cmd’ argument
    • -
-
-
-

ioctl commands

-
    -
  • Make the numbers for commands unique -
      -
    • 4 bit fields (type, number, direction, size)
      • -
    • IO_TYPEBITS, _IOC_NRBITS ...
      • -
    -
    • -
  • _IO macro for combining a type and number into one value -
      -
    • See sensehat.h in repo for example
      • -
    -
    • -
-
-
-

ioctl return value

-
    -
  • POSIX standard states to return -ENOTTY when ioctl command is not valid
    • -
  • Although several kernel functions return -EINVAL
    • -
-
-
-

Capabilities & Restricted Operations

-
    -
  • A capability based system breaks down privileged operations into separate subgroups
    • -
  • Capset, capget
    • -
  • if (! capable (CAP_SYS_ADMIN))
    -return -EPERM;
    • -
-
-
-

ioctl argument

-
    -
  • Is unsigned long but can be an integer or a pointer -
      -
    • Depends on the particular ioctl
      • -
    -
    • -
  • Do appropriate checking before access if ‘arg’ is pointer -
      -
    • access_ok(type, *addr, size)
      • -
    • put_user(), __put_user(), get_user(), __get_user()
      • -
    -
    • -
-
-
-

Implementation

-
    -
  • Often just a big switch statement
    -switch(cmd) {
    -case SCULL_IOCRESET:
    -scull_quantum = SCULL_QUANTUM;
    -scull_qset = SCULL_QSET;
    -break;
    • -
-

case SCULL_IOCSQUANTUM:
-…
-}

-
-
-

Non-ioctl device control

-
    -
  • By writing control sequences -
      -
    • For example; running a remote tty session
      • -
    • Escape sequences are printed remotely but affect the local tty
      • -
    -
    • -
  • Controlling by write -
      -
    • For devices that just respond to commands
      • -
    -
    • -
-
-
-

Blocking I/O

-

Putting process to sleep until IO operation can happen

-
-
-

Sleeping

-
    -
  • When a process is put to sleep, it is removed from the scheduler’s run queue and put into a special state
    • -
  • Never sleep when running in an atomic context -
      -
    • Atomic context is where multiple steps must occur without any concurrent access ie while holding a spinlock, seqlock or RCU lock
      • -
    • When interrupts are disabled
      • -
    • No way to wake up
      • -
    -
    • -
  • Somebody has to wake up sleeping process -
      -
    • Wait queue
      • -
    -
    • -
-
-
-

Simple sleeping

-
    -
  • Wait_event(queue, condition);
    • -
  • wake_up(wait_queue_head_t *queue);
    • -
-
-
-

Blocking vs nonblocking I/O

-
    -
  • Sometimes process does not want to block whether or not it’s I/O can make progress
    • -
  • O_NONBLOCK flag for non blocking I/O
    • -
  • If read is called and no data, block until data arrives.
    • -
  • Similarly if write is called and no space in buffer, block until buffer is free
    • -
  • If O_NONBLOCK is used, read and write calls simply return -EAGAIN (“try again”) allowing the application to poll for data
    • -
-
-
-

Blocking I/O Example

-

Page 153 in LDD3

-
-
-

Advanced Sleeping

-
    -
  • First step: allocation and initialization of wait_queue_t structure followed by it’s addition to proper wait queue
    • -
  • Second step is to set the state of the process to mark it as being asleep -
      -
    • TASK_ITERRUPTIBLE and TASK_UNINTERRUPTIBLE indicate a process is asleep
      • -
    -
    • -
  • Final step is to give up the processor. -
      -
    • Check the condition you are sleeping for first. Failure to do this may result into a race condition
      -if (!condition)
      -schedule( );
      • -
    -
    • -
-
-
-

poll and select

-
    -
  • Poll, select and epoll used by apps that use nonblocking I/O
    • -
  • They all essentially have the same functionality -
      -
    • Each allow a process to determine whether it can read from or write to one or more files without blocking
      • -
    -
    • -
  • (*poll) (struct file *filp, poll_table *wait); -
      -
    • Called when user-space program performs poll, select or epoll
      • -
    -
    • -
-
-
-

Poll and select vs read and write

-
    -
  • Poll and select determine in advance if an I/O operation will block in that respect complementing read and write
    • -
  • Poll returns POLLIN|POLLRDNORM if there is data in the read buffer
    • -
  • Poll reports that device is unreadable if there is no data
    • -
  • Poll reports POLLHUP if we are at EOF
    • -
  • Poll returns POLLOUT|POLLWRNORM if writing won’t block
    • -
  • Reports file is not writable if writing would block
    • -
-
-
-

struct poll_table*

-
    -
  • Holds ‘struct file’ and ‘wait_queue_head_t’ pointers passed to poll_wait
    • -
  • A call to poll or select involves a handful of file descriptors so the cost of setting up the data structure is small
    • -
  • epoll allows for scaling to many file descriptors by setting up this data structure exactly once and to use it many times
    • -
-
-
-

Async Notifs

-
    -
  • Instead of calling poll to check for data, set async notifs for when data becomes available
    • -
  • Step 1: Specify process as owner of file
    • -
  • Step 2: Set FASYNC flag in device by means of the F_SETFL fcntl command
    • -
  • Input file can now request delivery of a SIGIO signal whenever new data arrives
    -signal(SIGIO, &input_handler); /* dummy sample; sigaction( ) is better */
    -fcntl(STDIN_FILENO, F_SETOWN, getpid( ));
    -oflags = fcntl(STDIN_FILENO, F_GETFL);
    -fcntl(STDIN_FILENO, F_SETFL, oflags | FASYNC);
    • -
-
-
-

POV: You are the driver

-
    -
  • When F_SETOWN is invoked, a value is assigned to filp->f_owner
    • -
  • When F_SETFL is executed to turn on FASYNC, the driver’s fasync method is called
    • -
  • When data arrives, all the processes registered for ansync notis must be sent a SIGIO
    • -
-
-
-

llseek

-
    -
  • Provide llseek method if seek operation corresponding to a physical operation on a device rather than using default
    • -
  • Some devices offer a data flow rather than a data area so seeking makes no sense. For example: serial ports of a keyboard
    • -
  • In this case, use ‘nonseekable_open’ for open
    • -
  • Set llseek method in ‘file_operations’ to no_llseek
    • -
-
-
-

Access control

-
    -
  • Sometimes only one authorized user should be allowed to open a device
    • -
  • Similar to ttys problem.
    • -
  • brute force: permit device to be opened by only one process at a time(single openness). AVOID THIS
    • -
  • Allow only one user to open device but can be opened by multiple processes -
      -
    • Same policy as ttys
      • -
    -
    • -
-
-
-

Cloning the Device on Open

-
    -
  • Create different private copies of the device depending on process opening it
    • -
  • Possible if not bound to hardware object
    • -
  • /dev/tty uses a similar technique
    • -
  • Copies of the driver are called “virtual devices” just like “virtual consoles” use a single tty device
    • -
  • Rarely needed
    • -
-
-
-

Block open instead of EBUSY

-
    -
  • Implement blocking ‘open’ so that the operation is slightly delayed rather than fail
    • -
  • Implementation is based on wait queue
    • -
  • Release is in charge of awakening any pending process
    • -
  • A blocking ‘open’ implementation is a problem for interactive user who has to keep guessing what’s going on -
      -
    • Would rather get a “resource busy” message
      • -
    -
    • -
-
- \ No newline at end of file diff --git a/kdlp.underground.software/course/slides/block_drivers.html b/kdlp.underground.software/course/slides/block_drivers.html deleted file mode 100644 index 7e4f7e81..00000000 --- a/kdlp.underground.software/course/slides/block_drivers.html +++ /dev/null @@ -1,114 +0,0 @@ -
-

Block Drivers

-

Chapter 16

-
-
-

Block Drivers

-
    -
  • Accessed by fixed sized blocks
    • -
  • Primarily used for disks drivers
    • -
  • Performance is more important than with char devices
    • -
  • Need to include in the driver:
    • -
-
#include <linux/genhd.h>
-#include <linux/blkdev.h>
-#include <linux/fs.h>
-
-
-
-

Data types

-
    -
  • Blocks are fixed size chunks of data, with size determined by the Kernel - usually 4096 bytes
    • -
  • Sectors are smaller fixed size chunks of data, with size determined by the attached hardware (i.e. disk) - usually 512 bytes
    • -
-
-
-

Registration

-
    -
  • Block devices are registered and unregistered using the following functions included from linux/fs.h
    • -
-
int register_blkdev(unsigned int major, const char *name);
-int unregister_blkdev(unsigned int major, const char *name);
-
-
    -
  • As with char devices, passing 0 as the major number will make the kernel assign an available major number
    • -
-
-
-

Block Drivers Operations

-
    -
  • Some of the operations work exactly like they do in char drivers:
    • -
-
int (*open)(struct inode *inode, struct file *filp);
-int (*release)(struct inode *inode, struct file *filp);
-int (*ioctl)(struct inode *inode, struct file *filp, unsigned int cmd, unsigned long arg);
-
-
-
-

Block Driver Operations

-
    -
  • Block driver specific methods:
    • -
-
int (*media_changed) (struct gendisk *gd);
-
-
    -
  • Checks if the user made a change to the removable media: returns non-zero value on change
    • -
-
int (*revalidate_disk) (struct gendisk *gd);
-
-
    -
  • Called when a user makes a change in the media, performs required work according to the change.
    • -
  • struct gendisk represents a single disk in the kernel
    • -
-
-
-

struct gendisk

-
    -
  • See linux/genhd.h
    • -
  • Initialized by the Block driver
    • -
  • Arguments:
    • -
-
int major; int first_minor; int minors; # each minor represents a partition
-char disk_name[32]; # cat /proc/partitions
-struct block_device_operations *fops;
-struct request_queue *queue; # used by kernel to manage I/O requests
-int flags; # bit fields flags describing the device (i.e. GENHD_FL_REMOVABLE)
-sector_t capacity; # the capacity of the device in 512-byte long sectors unit
-void *private_data; # to be used at the discretion of the block driver 
-
-
-
-

struct gendisk

-
    -
  • To allocate a struct gendisk:
    • -
-
struct gendisk *alloc_disk(int minors);
-
-
    -
  • Free using:
    • -
-
void del_gendisk(struct gendisk *gd);
-
-
    -
  • Finally, add the gendisk to the system using:
    • -
-
void add_disk(struct gendisk *gd);
-
-
    -
  • Set the capacity for the gendisk using:
    • -
-
void set_capacity(struct gendisk *gd, sector_t sectors)
-
-
- \ No newline at end of file diff --git a/kdlp.underground.software/course/slides/concurrency.html b/kdlp.underground.software/course/slides/concurrency.html deleted file mode 100644 index a63c58e9..00000000 --- a/kdlp.underground.software/course/slides/concurrency.html +++ /dev/null @@ -1,293 +0,0 @@ -
-

Concurrency and Race Conditions

-

In the Linux kernel

-

LDD3: Chapter 5

-
-
-

Review of Definitions

-
-
-

Thread

-
    -
  • Thread of execution
    • -
  • Stores execution state (registers + instruction pointer) and has its own stack
    • -
  • Each process has at least one (but maybe more) of them
    • -
  • Is the unit of program that actually is scheduled on to the CPU
    • -
-
-
-

Concurrency

-
    -
  • Multiple threads of execution happening (or appearing to happen) at the same time
    • -
  • Even with a single cpu core, task switching can give the same effects as true multiprocessing
    • -
  • Up to the OS scheduler to decide who runs when
    • -
-
-
-

Sleep

-

penguins

-
    -
  • When a linux process reaches a point where it cannot make further progress until something else is finished (e.g. waiting for data from disk or user io)
    • -
  • Scheduler yields the processor to another thread
    • -
-
-
-

Interrupt

-
    -
  • A signal emitted by hardware or software when something needs immediate attention
    • -
  • Scheduler gives the code responsible for handling these extremely high priority
    • -
-
-
-

Critical Section

-
    -
  • Section of code where shared resources (data like global variables or state like i/o) are accessed by a thread of execution
    • -
  • Cannot have multiple threads working on the same shared resources running their critical sections concurrently -
      -
    • That’s a race condition!
      • -
    -
    • -
-
-
-

Race Condition

-
    -
  • Multiple processes trying to access shared data at the same time
    • -
  • “Race” to see which thread reads/writes data first
    • -
  • Unexpected results when wrong access pattern happens -
      -
    • Memory leak
      • -
    • Corrupted data
      • -
    • System crash
      • -
    -
    • -
  • The asynchronous nature of interrupts means that code handling them must be extremely careful with global data. It also cannot block (sleeping) -
      -
    • Increased probability of race conditions
      • -
    -
    • -
-
-
-

Mutual exclusion

-
    -
  • Maintain concurrency for independent parts of execution (so we can enjoy speed or responsiveness benefits of parallel execution)
    • -
  • Stop multiple processes from running within a critical section simultaneously
    • -
  • Ensured using locking data structures like spin locks, semaphores, mutexes, event counters, sequencers, etc.
    • -
-
-
-

mutual exclusion 1
-image source

-
-
-

mutual exclusion 2
-image source

-
-
-

mutual exclusion 3
-image source

-
-
-

Deadlock

-

deadlock

-
    -
  • Mutual exclusion gone wrong
    • -
  • A situation where threads are holding some resources but also blocked waiting for others in a cycle preventing any forward progress.
    -image source
    • -
-
-
-

Locking Data Structures

-
    -
  • spinlock -
      -
    • locking primitive that can be used to implement all other forms of mutual exlcusion
      • -
    -
    • -
  • semaphore -
      -
    • Combination of integer value and two functions for increment and decrement (v,p)
      • -
    -
    • -
  • mutex -
      -
    • Special kind of semaphore with only two states locked/unlocked
      • -
    -
    • -
-
-
-

Concurrency Techniques within Linux

-
-
-

Spinlocks

-
    -
  • spinlock_t in kernel
    • -
  • Spinlocks must be held for minimum time possible
    -Only acceptable in atomic contexts
    • -
-
-
-

Spinlocks in Atomic Context

-
    -
  • Interrupts must be disabled on local cpu when a lock is used -
      -
    • Preemption disabled by locking implementation!
      • -
    -
    • -
  • Why? an interrupt might cause a different thread to be scheduled, which may require the lock you just took but haven’t yet released -
      -
    • Deadlock!
      • -
    -
    • -
  • Code must therefore be atomic (cannot sleep)
    • -
-
-
-

Semaphores and Mutexes

-
    -
  • struct semaphore
    • -
  • struct mutex
    • -
-
-
-

Reader/Write Semaphores

-

semaphores

-
    -
  • Allows multiple concurrent readers
    • -
  • rwsem -
      -
    • One writer or unlimited number of readers
      • -
    • Writers get priority
      • -
    • Allowing multiple readers Optimizes performance
      -image source
      • -
    -
    • -
-
-
-

Completions

-
    -
  • Allow a thread to tell another that a job is done
    • -
  • Better locking mechanism -
      -
    • Prevents performance issues when creation of user processes or new kernel threads are involved
      • -
    -
    • -
  • A: complete(xxx)
    • -
  • B: wait_for_completion(xxx)
    • -
  • B only proceeds when A had called complete
    • -
-
-
-

Reader/Writer Spinlocks

-
    -
  • Allow any number of readers into critical section simultaneously
    • -
  • Writers must have exclusive access
    • -
  • rwlock_t similar to rwsem semaphore
    • -
-
-
-

Ambiguous rules

-

Be careful not to double lock!
-e.g. foo() gets lock A, calls bar() which also gets lock A
-Common solution: modified _bar() that assumes A is already held

-
-
-

Lock Ordering Rules

-

lock ordering
-Total ordering!
-if you lock A then B then C
-You MUST unlock C then B then A
-And never lock B then A (etc) somewhere else!
-Otherwise, you get deadlock…

-
-
-

Fine Versus Coarse-Grained Locking

-
    -
  • Old kernels had one big lock -
      -
    • Not scalable
      • -
    -
    • -
  • Modern kernels have thousands of locks each protecting smaller resources
    • -
  • Can be too complex
    • -
  • lock_kernel() lmao 😂
    • -
-
-
-

Alternatives to locking

-

😳

-
-
-

Lock-Free Algorithms

-
    -
  • Circular buffer
    • -
  • Requires no locking in the absence of multiple consumers and producers
    • -
  • Only Producer thread can modify the write index and array location it points to
    • -
  • Only reader thread can access read index and value it points to
    • -
  • If the two pointers don’t overrun each other, the producer and consumer can access the buffer concurrently with no race conditions
    -https://stackoverflow.com/questions/871234/circular-lock-free-buffer
    • -
-
-
-

Atomic Variables

-
    -
  • Used for simple shared resources like an int value
    • -
  • o_op++; -
      -
    • Can be done in atomic manner
      • -
    -
    • -
  • Locking is overkill, use an atomic_t instead -
      -
    • atomic_inc(atomic_t *v);
      • -
    • atomic_dec(atomic_t *v);
      • -
    • atomic_add_return(int i, atomic_t *v);
      • -
    -
    • -
-
-
-

Bit Operations

-
    -
  • Used for shared flags
    • -
  • Manipulate individual bits in an atomic manner
    • -
  • Example functions -
      -
    • set_bit(nr, void *addr);
      • -
    • clear_bit(nr, void *addr);
      • -
    -
    • -
-
-
-

seqlocks

- -
-
-

Read-Copy-Update (RCU)

-
    -
  • When a data structure needs to be changed, the writing thread makes a copy, changes the copy, then aims the relevant pointer at the new version
    • -
  • Used for situations where, reads are common and writes are rare
    • -
  • Very complex: many memory barriers used
    • -
-
- \ No newline at end of file diff --git a/kdlp.underground.software/course/slides/debugging_techniques.html b/kdlp.underground.software/course/slides/debugging_techniques.html deleted file mode 100644 index c09d14cb..00000000 --- a/kdlp.underground.software/course/slides/debugging_techniques.html +++ /dev/null @@ -1,115 +0,0 @@ -
-

Debugging Techniques

-

For the Linux Kernel

-

LDD3: Chapter 4

-
-
-

Printk: the elite printf

-

Log levels

-
    -
  • KERN_EMERG
    • -
  • KERN_ALERT
    • -
  • ...
    -printk(KERN_XXXX "format string", ...);
    -NOT: printk(KERN_XXX, "format string", ...);
    -/proc/sys/kernel/printk
    • -
-
-
-

Querying

-

Massive use of printk slows down the system, query system info instead
-Can also rate limit
-Techniques for querying:

-
    -
  • Using /proc filesystem
    • -
  • ioctl
    • -
  • sysfs
    • -
-
-
-

Using the /proc filesystem

-

Each file under /proc is tied to a kernel function that generates it’s contents

-
    -
  • ps, top, uptime …, get info from /proc
    • -
  • Add/create files in /proc to export messages from drivers (discouraged)
    • -
  • Use sysfs instead (covered in ch-14)
    • -
-
-
-

ioctl

-
    -
  • System call that acts on a file descriptor
    • -
  • Implement ioctl commands tailored for debugging
    • -
  • Faster than /proc
    • -
  • But more work compared to using /pro
    • -
-
-
-

Watching

-

Watch behavior of driver in user space

-
    -
  • Call read() and watch it’s behavior
    • -
  • Run a debugger and step through function calls
    • -
  • strace
    • -
-
-
-

More on strace

-

A command that shows all system calls issued by a user-space program

-
    -
  • Shows function calls, their arguments and return values
    • -
  • -t (to display time call was made)
    • -
  • -e (to limit type of calls traced)
    • -
  • -o (to redirect output. Default is stderr)
    -Strace example next
    • -
-
-
-

strace ls /dev > /dev/scull0

-

open("/dev", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
-...
-getdents64(3, /* 141 entries */, 4096) = 4088
-…
-fstat64(1, {st_mode=S_IFCHR|0664, st_rdev=makedev(254, 0), ...}) = 0
-write(1, "MAKEDEV\nadmmidi0\nadmmidi1\nadmmid"..., 4096) = 4000
-write(1, "b\nptywc\nptywd\nptywe\nptywf\nptyx0\n"..., 96) = 96
-…
-close(1)
-exit_group(0)

-
-
-

System faults/Oops messages

-
    -
  • Caused by dereferencing NULL or invalid pointer
    • -
  • When an invalid pointer is dereferenced, the paging mechanism cannot map the pointer to a physical address and the processor signals a page fault
    • -
-
-
-

System Hangs

-

Sometimes there are no oops messages printed when bugs in the kernel hang the system

-
    -
  • Endless loop causes the kernel to stop scheduling aka hang
    • -
-
-
-

Tools for Advanced Debugging and More

-
    -
  • BPF: more than just the Berkeley Packet Filter
    • -
  • Ftrace: “function tracer” but much more
    • -
  • Gdb and friends (such as crash)
    • -
  • Ptrace: from one process to another
    • -
  • Perf: performance analysis tool
    • -
-
- \ No newline at end of file diff --git a/kdlp.underground.software/course/slides/git.html b/kdlp.underground.software/course/slides/git.html deleted file mode 100644 index 37b2a95a..00000000 --- a/kdlp.underground.software/course/slides/git.html +++ /dev/null @@ -1,361 +0,0 @@ -
-

Git

-

An open source version control system

-
-
-

Version Control & Git vs. GitHub

-
    -
  • The management of changes to a project
    • -
  • Git is specifically designed for version control of programming projects, provides a log of changes to your project. You can access previous instances of the project to help with collaboration, bug fixes, etc.
    • -
  • Git is not GitHub
    • -
  • GitHub is a website which can host your repositories to be available to the public. This is useful for group work and open source projects.
    • -
-
-
-

More Git

-
    -
  • Git allows you to keep track of the previous states of your or someone else’s repo
    • -
  • Each saved state of your repo has a hash identifier with which you can access said state
    • -
  • A repo that is managed with Git will have a .git directory which will have information about all saved changes
    • -
  • ls -a → To check hidden files not displayed by ls by default
    • -
-
-
-

Visualization

-

We will use a tree structure to illustrate the behavior of how saved states of a repository behave when manipulated with git commands. E.g. Below is a repository which has three commits to the main branch and one commit to the X branch. The blue pointer indicates that we are viewing the first commit of the X branch.
-git tree

-
-
-

Diagram Key

-

diagram key

-
-
-

Git Commands: git init

-
    -
  • Initialize the cwd as a git repository
    • -
  • In order to use most other git commands, you will need to be in a folder containing an initialized repo first
    • -
-
-
-

Git Commands: git clone

-

git clone diagram
-Copies a repository that is hosted on GitHub, or similar website, to your local directory

-
-
-

Git Commands: git add

-
    -
  • Adds a change, which was made to the repository, to a list of changes to be committed later on. This accounts for changes to an existing file or adding a new directory or file to the repo.
    • -
  • 'git add .' adds all new files and changes to existing files in current directory to the index
    • -
-
-
-

Git Commands: git commit

-

git commit diagram

-
    -
  • Saves the current state of your repository in Git. Changes are not live, need to push
    • -
  • git commit -m “somemessage” to add a short message to your commit and avoid opening your editor
    • -
-
-
-

Git Commands: git status

-

Shows which changes are tracked by Git and which changes still need to be committed.

-
-
-

SSH Keys

-
    -
  • To push your changes to your GitHub you need to link your local machine to your Github account.
    • -
  • Generate SSH key
    -$ ssh-keygen -t rsa -b 4096 -C “your github email”
    • -
  • Cat your public key file and highlight and copy it
    -$ cat ~/.ssh/id_rsa.pub
    • -
  • On GH go to Settings/SSH and GPG keys/New SSH key
    • -
  • Name your key and paste the public key in the key field
    • -
  • Now you can push to GitHub
    • -
-
-
-

Additional Details

-
    -
  • If you your local repo was created by cloning a remote repo, your local repo will already be linked with that remote.
    • -
  • If you want to publish a repo you made locally on github, you will need to make an empty GitHub repo and find it’s link and set a remote to that link manually
    -$ git remote add origin git@github.com:whoever/whatever.git
    -$ git push origin main
    • -
  • To set a default remote branch for future push operations on a given local branch do
    -$ git push -u origin main
    • -
  • And next time you push you can just do
    -$ git push
    • -
-
-
-

Git Commands: git push

-

git push diagram
-Uploads new Git commits to a remote repository. e.g. to GitHub

-
-
-

Git Commands: git pull

-

git pull diagram
-Opposite of git push. Download changes locally from remote repository

-
-
-

Branching: uses

-
    -
  • The main stem of the repo is the main branch, in a group project main will be the ‘final’ version of the project.
    • -
  • Branching is useful when group members have separate responsibilities to the project.
    • -
  • You can branch from main and work on your part of the project without disrupting main.
    • -
  • When you’re done and your code has been approved, your changes can be merged back into the main branch.
    • -
  • Delete a branch with git branch -d name
    • -
-
-
-

Branching

-

git branching d1
-$ git branch
-* main
-feature1

-
    -
  • Use git branch to check which branches exist and which one you are currently working in
    • -
-
-
-

git branching d2
-$ git checkout feature1
-(switch to an existing branch)
-$ git branch
-main
-* feature1

-
-
-

git branching d3
-$ git checkout main; git branch
-* main
-feature1
-$ git checkout -b feature2 (creates a new feature2 branch and brings you to it)
-$ git branch
-main
-feature1
-* feature2

-
-
-

Merging and pull requests(PR)

-
    -
  • If you are working on your own project you will use git merge to merge your branches into main.
    • -
  • If you are merging a feature branch on to main you are creating a new commit to main where all previous main and feature commits are combined.
    -$ git branch
    -main
    -* other
    -$ git checkout main; git merge other; git diff other
    • -
  • They will be the same if there are no conflicts
    • -
-
-
-

Git Commands: git diff

-
    -
  • To compare the changes between the current branch and the feature branch
    -$ git diff feature
    • -
  • To see what changes have been made to file.txt
    -$ git diff file.txt (if file.txt is staged)
    • -
  • To see changes of all staged files
    -$ git diff
    • -
  • For more advanced features
    -$ git diff --help
    • -
-
-
-

Merging and pull requests

-
    -
  • When working on a group project, you will want to open a pull request to make sure that your group is satisfied with merging your branch to main
    • -
  • First you will need to push your branch to GH
    -$ git push -u origin PRbranch
    • -
  • Here you are setting origin for your local branch PRbranch to be your GH PRbranch. So your local PRbranch will only push to the same GH PRbranch
    • -
  • Then, you can go on your GH and open a new pull request. You are requesting that your team review your proposed changes to main and then pull PRbranch into the main branch.
    • -
-
-
-

Merging and pull requests

-

merging d1
-$ git branch
-* main
-feature1
-feature2

-
-
-

merging d2
-$ git merge feature1

-
    -
  • If you use the '--squash' tag, it will merge but stop before making the merge commit
    -$ git merge --squash feature1
    -# Make some changes
    -$ git add . && git commit -m "some message"
    • -
  • This allows you to make some more changes before you commmit your merge
    • -
-
-
-

merging d3
-$ git merge feature2

-
-
-

Group project note

-

When new changes are pushed to main by other people in your group, in order to keep your branch up to date pull changes from your GH to your local machine and then merge local main into your branch. Just step one in the example.
-group project note diagram

-
-
-

Merge conflicts

-

You and your group members might modify the same file in the repo, if two people modify the same line of code and then you try to merge, git will not know how to handle the conflict. You will need to resolve that manually.

-
-
-

Git Commands: git log

-
    -
  • Allows you to see all commit history in your local git repo. Each commit will have a hash associated with it, so you could git checkout a particular commit down the tree if you wanted to.
    • -
  • The log is a good example of why you want to add descriptive messages when you commit, otherwise you will have no information about previous commits and will have to check each one individually to get any insight on them.
    • -
  • You can unstage a particular commit with git reset commitname/hash
    • -
  • Or fully remove the commit and set the new HEAD of the tree to the previous commit with git reset --hard commitname/hash
    • -
-
-
-

Forking a repository

-

You might not have direct access to working with someone else’s repo online, so to gain access to it you can fork it to your GH and then clone it to your local machine. This allows you to work on the same project and add new features, when you’re ready you can open a pull request against the original repo.

-
-
-

Advanced Git: git rebase

-

rebase 1 rebase 2
-$ git branch
-* main
-feature1
-$ git rebase feature1

-
-
-

Advanced Git: git cherry-pick

-

You use this to pick one particular commit to merge with main->HEAD
-cherrypick 1 cherrypick 2
-$ git branch
-* main
-feature1
-$ git cherry-pick #hash#
-Note: you can use -n to merge the commit with HEAD without committing

-
-
-

Advanced Git: git tag

-
    -
  • The ways to reference commits are by checking out a branch, which will bring you to branchname -> HEAD, or by referencing the hash of a particular commit
    • -
  • But if we have an interesting commit which we want easy access to later, we can tag it so we don’t have to look up it’s hash
    -git tag
    • -
-
-
-

Advanced Git: git log

-
    -
  • Like with any git command you can add flags to it to fine tune what you want the command to achieve
    • -
  • git log --help to check the flags. This applies to all git commands e.g. git cherry-pick --help
    • -
  • A useful variation on git log is
    -$ git log --graph --oneline --decorate
    • -
  • (--graph) shows a log in a graph format distinguishing the different active branches
    • -
  • (--oneline) compacts each commit to one line
    • -
-
-
-

Advanced Git: git bisect

-

Useful tool for finding the first commit which introduced a bug
-Steps:

-
    -
  1. git bisect start
    2. -
  2. git bisect good hash
    3. -
  3. git bisect bad (you can leave fourth arg blank to start from current commit)
    -Bisect will run a binary search based on your input, after step 3 bisect will drop you on a commit half way between the known good and bad commits
    4. -
  4. git bisect good/bad (depending on whether your test passed/failed)
    5. -
  5. Repeat step 4 until bisect finds the first bad commit
    6. -
  6. Fix the bug
    7. -
  7. git bisect reset (to get back to the commit where you ran step 1)
    8. -
-
-
-

Git Strategies: commit messages

-
    -
  • When staging a commit, try to organize each commit by topic. Do not cram unrelated changes into one commit
    • -
  • If you have modified a file for multiple different topics you can stage parts of the file with partial staging, use
    • -
  • git add -p file.txt
    • -
  • Terminal will prompt you which parts you would like to stage (y/n)
    • -
-
-
-

Commit Message Conventions

-

Formulating a commit message

-
    -
  1. The subject should be a concise summary of your changes
    -Write this on the first line of the git editor that pops up after a commit
    2. -
  2. The body can be more detailed
    -Write this on line 3 of the git editor
    3. -
-
    -
  • How did the file change
    • -
  • Why did you make the change
    • -
  • Any key notes on the change
    • -
-
-
-

Advanced Git: git rev-list

-
    -
  • Use this to show a custom view of the tree of commits
    -$ git rev-list feature1 feature2 main
    • -
  • Shows commits from feature1 and feature2 but not main
    -$ git rev-list --not feature1 feature2 ^main
    • -
  • Does the opposite
    -$ git-rev-list --help
    • -
-
-
-

Advanced Git: formatting and applying a patchset

-
    -
  • Another way to merge two branches is by applying a patchset
    • -
  • A patchset contains patches which is essentially a diff file in an email format
    • -
  • Gives your group the ability to review the patch before merging
    -format patch d1
    • -
-
-
-

Advanced Git: formatting and applying a patchset

-
    -
  • You can apply individual patches by patch name in the patches directory
    • -
  • If you do this, make sure that the parent of the commit that is to be applied as a patch is already in the branch you are applying to
    -format patch d2
    • -
-
-
-

git send-email

-

Follow this to configure your system to send email patches with your gmail address.
-Then you will need to create a patch set and run
-$ git send-email --to=address1,address2,etc ./path/to/patch(es)

-
-
-

Git Cheat Sheet

-

https://phoenixnap.com/kb/git-commands-cheat-sheet#ftoc-heading-10
-Useful Commands:
-rm -rf directory (to delete a directory and everything in it)
---abort (to abort any git process)
-e.g. git merge --abort

-
-
-

References

- -
- \ No newline at end of file diff --git a/kdlp.underground.software/course/slides/hardware_io.html b/kdlp.underground.software/course/slides/hardware_io.html deleted file mode 100644 index 88df320f..00000000 --- a/kdlp.underground.software/course/slides/hardware_io.html +++ /dev/null @@ -1,223 +0,0 @@ -
-

Hardware

-

In Linux

-

LDD3: Chapter 9

-
-
-

Device Registers

-
    -
  • Peripheral devices are controlled by reading from and writing to their registers.
    • -
  • The registers are accessed either through memory address space or I/O address space, both of which are equivalent interfaces in hardware, reading from and writing to the address bus, the control bus, and the data bus.
    • -
-
-
-

Compiler Optimizations

-
    -
  • I/O registers have side effects, while memory registers do not.
    • -
  • Because of this, developers need to be careful of compiler optimizations that can change the order of read/write I/O operations.
    • -
  • To prevent this issue, use memory barriers
    • -
-
-
-

Memory Barriers

-
    -
  • Read/write operations that appear before a relevant memory barrier execute before any read/write operations that appear after it -
      -
    • rmb() - Read memory barrier
      • -
    • wmb() - Write memory barrier
      • -
    • mb() - Memory barrier (for both)
      • -
    • read_barrier_depends() - weaker rmb
      • -
    -
    • -
  • read_barrier_depends() only blocks reordering reads when they have data dependencies upon one another
    • -
-
-
-

I/O Port Allocation

-
    -
  • To interact with device registers, the device is accessed through an I/O port
    • -
  • To use the I/O port, one must first acquire exclusive access to it using:
    • -
-
struct resource *request_region(unsigned long first, unsigned long n, const char *name);
-
-
    -
  • This requests <n> ports starting with <first>, for the device called <name>.
    • -
  • When you’re done with the ports release them using:
    • -
-
void release_region(unsigned long start, unsigned long n);
-
-
-
-

Manipulating I/O Ports

-
    -
  • To read from and write to I/O ports, you must use the appropriate sized function: -
      -
    • unsigned inb(unsigned port) - read one byte from an 8-bit port
      • -
    • void outb(unsigned char byte, unsigned port) - write one byte to an 8-bit port
      • -
    • unsigned inw(unsigned port) - read one word from a 16-bit port
      • -
    • void outw(unsigned port) - write one word to a 16-bit port
      • -
    • unsigned inl(unsigned port) - read one long from a 32-bit port
      • -
    • unsigned outl(unsigned port) - write one long to a 32-bit port
      • -
    -
    • -
  • No 64 bit port operations are defined, 64-bit architectures use a maximum of 32-bit operations on I/O ports
    • -
-
-
-

Manipulating I/O Ports in User Space

-
    -
  • inb(/w/l) and outb(/w/l) can be used in user-space as they’re defined in <sys/io.h>, to use them a program must be run as root, compiled with “-O” to force extension of inline functions, and acquire permissions using: -
      -
    • ioperm - to get permissions for individual ports
      • -
    • iopl - to get permissions for the entire I/O space
      • -
    -
    • -
  • Alternatively, user programs can access I/O ports using their device files in /dev/<device name>
    • -
-
-
-

String Operations on Ports

-
    -
  • I/O ports can be read from and written to using C-strings using the following functions:
    • -
-
void insb(unsigned port, void *addr, unsigned long count); 
-void outsb(unsigned port, void *addr, unsigned long count);
-void insw(unsigned port, void *addr, unsigned long count); 
-void outsw(unsigned port, void *addr, unsigned long count);
-void insl(unsigned port, void *addr, unsigned long count); 
-void outsl(unsigned port, void *addr, unsigned long count);
-
-
    -
  • When using string operations, look out for mismatches between byte-ordering rules of the host system and the I/O device (little endian vs big endian)
    • -
-
-
-

I/O Pausing

-
    -
  • If the device is slower than your processor, it can be overclocked and miss data, to solve for this, use pausing I/O functions which implement a wait in between I/O operations.
    • -
-
-
-

Platform Dependencies

-
    -
  • I/O port operations are highly platform dependent
    • -
  • For example, a port can be unsigned short in some architectures and unsigned long in others
    • -
-
-
-

Using I/O Memory

-
    -
  • A popular alternative to I/O ports is I/O memory
    • -
  • I/O memory is RAM regions that a device makes available to the processor through the bus
    • -
  • Technically, it’s possible to access I/O memory using pointers, much like regular memory, but this is highly discouraged for security and stability concerns
    • -
-
-
-

I/O Memory Allocation and Mapping

-
    -
  • I/O memory regions must be allocated prior to use, using:
    • -
-
struct resource *request_mem_region(unsigned long start, unsigned long len, char *name);
-
-
    -
  • This allocated a <len> long memory region starting at <start> for device <name>
    • -
  • When you’re done with a memory region, release it using:
    • -
-
void release_mem_region(unsigned long start, unsigned long len);
-
-
-
-

Accessing I/O Memory

-
    -
  • To access I/O memory regions, one must obtain their address using:
    • -
-
void *ioremap(unsigned long phys_addr, unsigned long size); 
-void *ioremap_nocache(unsigned long phys_addr, unsigned long size);
-
-
    -
  • The nocache version of ioremap gives a pointer to a read/write interface with no data caching, which drivers may sometimes need depending on the device
    • -
-
-
-

Read/Write I/O Memory

-
    -
  • To read from and write to I/O memory, use the functions:
    • -
-
unsigned int ioread8(void *addr); 
-unsigned int ioread16(void *addr); 
-unsigned int ioread32(void *addr);
-void iowrite8(u8 value, void *addr); 
-void iowrite16(u16 value, void *addr); 
-void iowrite32(u32 value, void *addr);
-
-
    -
  • Where addr is the address obtained from ioremap.
    • -
  • There are also versions of all of these functions that allow repeating read/write that look like:
    • -
-
void ioread8_rep(void *addr, void *buf, unsigned long count);
-
-
    -
  • Note: count is expressed as number of relevant data units (8-bit/16-bit/32-bit)
    • -
-
-
-

Read/Write I/O Memory

-
    -
  • You can also read from and write to I/O memory as a block using: -
      -
    • Fill a memory block with a repeating byte <value>
      • -
    -
    • -
-
void memset_io(void *addr, u8 value, unsigned int count); 
-
-
    -
  • Copy from I/O memory <source> to regular memory <dest> <count> bytes
    • -
-
void memcpy_fromio(void *dest, void *source, unsigned int count);
-
-
    -
  • Copy from regular memory <source> to I/O memory <dest> <count> bytes
    • -
-
void memcpy_toio(void *dest, void *source, unsigned int count);
-
-
-
-

Read/Write I/O Memory

-
    -
  • There are also old functions discouraged from use to operate on I/O memory:
    • -
-
unsigned readb(address); 
-void writeb(unsigned value, address);
-unsigned readw(address); 
-void writew(unsigned value, address);
-unsigned readl(address); 
-void writel(unsigned value, address);
-
-
-
-

Ports as I/O Memory

-
    -
  • The kernel provides functions to map I/O ports as I/O memory regardless of their actual implementation on the device:
    • -
-
void *ioport_map(unsigned long port, unsigned int count);
-
-
    -
  • This converts <count> ports starting at <port> to appear as I/O memory
    • -
  • Once you’re done with the ports, unmap the I/O memory using:
    • -
-
void ioport_unmap(void *addr);
-
-
- \ No newline at end of file diff --git a/kdlp.underground.software/course/slides/images/Concurrency/penguins.png b/kdlp.underground.software/course/slides/images/Concurrency/penguins.png deleted file mode 100644 index 708a053c..00000000 Binary files a/kdlp.underground.software/course/slides/images/Concurrency/penguins.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Concurrency/slide10.png b/kdlp.underground.software/course/slides/images/Concurrency/slide10.png deleted file mode 100644 index 9a7184e3..00000000 Binary files a/kdlp.underground.software/course/slides/images/Concurrency/slide10.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Concurrency/slide11.png b/kdlp.underground.software/course/slides/images/Concurrency/slide11.png deleted file mode 100644 index 2335de7b..00000000 Binary files a/kdlp.underground.software/course/slides/images/Concurrency/slide11.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Concurrency/slide12.png b/kdlp.underground.software/course/slides/images/Concurrency/slide12.png deleted file mode 100644 index cbd66004..00000000 Binary files a/kdlp.underground.software/course/slides/images/Concurrency/slide12.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Concurrency/slide13.png b/kdlp.underground.software/course/slides/images/Concurrency/slide13.png deleted file mode 100644 index 391c26c3..00000000 Binary files a/kdlp.underground.software/course/slides/images/Concurrency/slide13.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Concurrency/slide19.png b/kdlp.underground.software/course/slides/images/Concurrency/slide19.png deleted file mode 100644 index e9d1e4ad..00000000 Binary files a/kdlp.underground.software/course/slides/images/Concurrency/slide19.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Concurrency/slide23.png b/kdlp.underground.software/course/slides/images/Concurrency/slide23.png deleted file mode 100644 index 0f644c97..00000000 Binary files a/kdlp.underground.software/course/slides/images/Concurrency/slide23.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide13.png b/kdlp.underground.software/course/slides/images/Git/git_slide13.png deleted file mode 100644 index a2cc8686..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide13.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide14.png b/kdlp.underground.software/course/slides/images/Git/git_slide14.png deleted file mode 100644 index 1e0bed7e..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide14.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide16.png b/kdlp.underground.software/course/slides/images/Git/git_slide16.png deleted file mode 100644 index abe17ba9..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide16.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide17.png b/kdlp.underground.software/course/slides/images/Git/git_slide17.png deleted file mode 100644 index 5a15c26f..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide17.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide18.png b/kdlp.underground.software/course/slides/images/Git/git_slide18.png deleted file mode 100644 index 67b94dae..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide18.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide20_1.png b/kdlp.underground.software/course/slides/images/Git/git_slide20_1.png deleted file mode 100644 index 41812901..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide20_1.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide20_2.png b/kdlp.underground.software/course/slides/images/Git/git_slide20_2.png deleted file mode 100644 index ce45d084..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide20_2.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide20_3.png b/kdlp.underground.software/course/slides/images/Git/git_slide20_3.png deleted file mode 100644 index df826f94..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide20_3.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide21.png b/kdlp.underground.software/course/slides/images/Git/git_slide21.png deleted file mode 100644 index df0bd495..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide21.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide25_1.png b/kdlp.underground.software/course/slides/images/Git/git_slide25_1.png deleted file mode 100644 index 421fa18c..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide25_1.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide25_2.png b/kdlp.underground.software/course/slides/images/Git/git_slide25_2.png deleted file mode 100644 index 162ea3e9..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide25_2.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide26_1.png b/kdlp.underground.software/course/slides/images/Git/git_slide26_1.png deleted file mode 100644 index 68510272..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide26_1.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide26_2.png b/kdlp.underground.software/course/slides/images/Git/git_slide26_2.png deleted file mode 100644 index 0014ae76..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide26_2.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide27.png b/kdlp.underground.software/course/slides/images/Git/git_slide27.png deleted file mode 100644 index 6ee8f352..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide27.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide33.png b/kdlp.underground.software/course/slides/images/Git/git_slide33.png deleted file mode 100644 index a69b9986..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide33.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide34.png b/kdlp.underground.software/course/slides/images/Git/git_slide34.png deleted file mode 100644 index ec533744..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide34.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide4.png b/kdlp.underground.software/course/slides/images/Git/git_slide4.png deleted file mode 100644 index ed5615cd..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide4.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide5.png b/kdlp.underground.software/course/slides/images/Git/git_slide5.png deleted file mode 100644 index 21b015c5..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide5.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide7.png b/kdlp.underground.software/course/slides/images/Git/git_slide7.png deleted file mode 100644 index 0bef9fe9..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide7.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Git/git_slide9.png b/kdlp.underground.software/course/slides/images/Git/git_slide9.png deleted file mode 100644 index 01be4b79..00000000 Binary files a/kdlp.underground.software/course/slides/images/Git/git_slide9.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Kernel_Data_Types/ll.png b/kdlp.underground.software/course/slides/images/Kernel_Data_Types/ll.png deleted file mode 100644 index 6a2bd970..00000000 Binary files a/kdlp.underground.software/course/slides/images/Kernel_Data_Types/ll.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Linux_Crash_Course/Fedora-logo.png b/kdlp.underground.software/course/slides/images/Linux_Crash_Course/Fedora-logo.png deleted file mode 100644 index 9f1a4fec..00000000 Binary files a/kdlp.underground.software/course/slides/images/Linux_Crash_Course/Fedora-logo.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Linux_Crash_Course/RHEL-logo-hat.png b/kdlp.underground.software/course/slides/images/Linux_Crash_Course/RHEL-logo-hat.png deleted file mode 100644 index cf1f5b8a..00000000 Binary files a/kdlp.underground.software/course/slides/images/Linux_Crash_Course/RHEL-logo-hat.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Linux_Crash_Course/arch_linux_logo.png b/kdlp.underground.software/course/slides/images/Linux_Crash_Course/arch_linux_logo.png deleted file mode 100644 index 31d781b1..00000000 Binary files a/kdlp.underground.software/course/slides/images/Linux_Crash_Course/arch_linux_logo.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Linux_Crash_Course/grub_menu.png b/kdlp.underground.software/course/slides/images/Linux_Crash_Course/grub_menu.png deleted file mode 100644 index 5c1d2f06..00000000 Binary files a/kdlp.underground.software/course/slides/images/Linux_Crash_Course/grub_menu.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Linux_Crash_Course/system_stack_diagram.png b/kdlp.underground.software/course/slides/images/Linux_Crash_Course/system_stack_diagram.png deleted file mode 100644 index 00e70662..00000000 Binary files a/kdlp.underground.software/course/slides/images/Linux_Crash_Course/system_stack_diagram.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Linux_Crash_Course/ubuntu_logo.png b/kdlp.underground.software/course/slides/images/Linux_Crash_Course/ubuntu_logo.png deleted file mode 100644 index 8a16da83..00000000 Binary files a/kdlp.underground.software/course/slides/images/Linux_Crash_Course/ubuntu_logo.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Linux_ToGP/awk.png b/kdlp.underground.software/course/slides/images/Linux_ToGP/awk.png deleted file mode 100644 index e3183326..00000000 Binary files a/kdlp.underground.software/course/slides/images/Linux_ToGP/awk.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Linux_ToGP/if_statements.png b/kdlp.underground.software/course/slides/images/Linux_ToGP/if_statements.png deleted file mode 100644 index 0d40479f..00000000 Binary files a/kdlp.underground.software/course/slides/images/Linux_ToGP/if_statements.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Linux_ToGP/loops.png b/kdlp.underground.software/course/slides/images/Linux_ToGP/loops.png deleted file mode 100644 index d4e5e87d..00000000 Binary files a/kdlp.underground.software/course/slides/images/Linux_ToGP/loops.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Linux_ToGP/sed.png b/kdlp.underground.software/course/slides/images/Linux_ToGP/sed.png deleted file mode 100644 index aa49285b..00000000 Binary files a/kdlp.underground.software/course/slides/images/Linux_ToGP/sed.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Linux_ToGP/ttl_cable.png b/kdlp.underground.software/course/slides/images/Linux_ToGP/ttl_cable.png deleted file mode 100644 index 59534d73..00000000 Binary files a/kdlp.underground.software/course/slides/images/Linux_ToGP/ttl_cable.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Linux_ToGP/ttl_wiring.png b/kdlp.underground.software/course/slides/images/Linux_ToGP/ttl_wiring.png deleted file mode 100644 index 6fe9b8c6..00000000 Binary files a/kdlp.underground.software/course/slides/images/Linux_ToGP/ttl_wiring.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/MM_DMA/page_table.png b/kdlp.underground.software/course/slides/images/MM_DMA/page_table.png deleted file mode 100644 index 78bc71d9..00000000 Binary files a/kdlp.underground.software/course/slides/images/MM_DMA/page_table.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide14.png b/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide14.png deleted file mode 100644 index 13786acc..00000000 Binary files a/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide14.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide2.png b/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide2.png deleted file mode 100644 index 2f060f91..00000000 Binary files a/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide2.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide3.png b/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide3.png deleted file mode 100644 index 1eca54ea..00000000 Binary files a/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide3.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide4.png b/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide4.png deleted file mode 100644 index c58d0724..00000000 Binary files a/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide4.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide5.png b/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide5.png deleted file mode 100644 index 749224aa..00000000 Binary files a/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide5.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide9.png b/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide9.png deleted file mode 100644 index 093bf124..00000000 Binary files a/kdlp.underground.software/course/slides/images/Modules_Drivers/modules_drivers_slide9.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/images/fedora_confusion.png b/kdlp.underground.software/course/slides/images/fedora_confusion.png deleted file mode 100644 index f9b55515..00000000 Binary files a/kdlp.underground.software/course/slides/images/fedora_confusion.png and /dev/null differ diff --git a/kdlp.underground.software/course/slides/index.md b/kdlp.underground.software/course/slides/index.md deleted file mode 100644 index a44bf5b0..00000000 --- a/kdlp.underground.software/course/slides/index.md +++ /dev/null @@ -1,37 +0,0 @@ - -#### The latest course slides - -[Git](git.html) - -[Linux Crash Course](linux_crash_course.html) - -[Linux Tips of Great Power](linux_tips_of_great_power.html) - -[Kernel Compilation](kernel_compilation.html) - -[Debugging Techniques](debugging_techniques.html) - -[Modules and Drivers](modules_drivers.html) - -[Writing Modules](writing_modules.html) - -[Writing Character Drivers](writing_character_drivers.html) - -[Advanced Character Driver Operations](advanced_char_driver_ops.html) - -[Kernel Data Types](kernel_data_types.html) - -[Memory Allocation](memory_allocation.html) - -[Concurrency](concurrency.html) - -[Interrupts](interrupts.html) - -[Timing and Scheduling](timing_scheduling.html) - -[Memory Mapping and DMA](mm_dma.html) - -[Hardware I/O](hardware_io.html) - -[Block Drivers](block_drivers.html) - diff --git a/kdlp.underground.software/course/slides/interrupts.html b/kdlp.underground.software/course/slides/interrupts.html deleted file mode 100644 index 7a098605..00000000 --- a/kdlp.underground.software/course/slides/interrupts.html +++ /dev/null @@ -1,142 +0,0 @@ -
-

Interrupt Handling

-

Chapter 10

-
-
-

An interrupt is simply a signal that the hardware can send when it wants the processor’s attention

-
-
-

The /proc Interface

-
    -
  • Reported interrupts are shown in /proc/interrupts
    -root@montalcino:/bike/corbet/write/ldd3/src/short# m /proc/interrupts
    -CPU0 CPU1
    -0: 4848108 34 IO-APIC-edge timer
    -2: 0 0 XT-PIC cascade
    • -
  • First column shows IRQ number
    • -
  • The last two columns give information on the programmable interrupt controller that handles the interrupt
    • -
-
-
-

Installing an Interrupt

-
    -
  • Must tell kernel to expect an interrupt, if not, it simply acknowledges the interrupt and ignores it
    • -
  • int request_threaded_irq(unsigned int irq, irq_handler_t handler, irq_handler_t thread_fn, unsigned long flags, const char *name, void *dev);
    • -
  • thread_fn can be NULL if you only need a top half
    • -
  • Drivers can share handlers
    • -
-
-
-

Top and Bottom Halves

-
    -
  • Top half runs asynchronously (not in process context) with interrupts disabled
    • -
  • It has to make a snap judgement about the interrupt
    • -
  • The bottom half is a regular function to be executed later when we have more time
    • -
  • interrupts are enabled during execution of the bottom half
    • -
-
-
-

Implementing a Handler

-
    -
  • Can’t transfer data to and from user space. Doesn’t execute in context of a process
    • -
  • Can’t sleep
    • -
  • First step is often acknowledging the signal from the hardware
    • -
  • Might awaken processes sleeping if the interrupt signals an event they are waiting for
    • -
  • Should execute in a minimum amount of time -
      -
    • Use a bottom half for long computations
      • -
    -
    • -
-
-
-

Enabling and Disabling Interrupts

-
    -
  • void {disable_irq, disable_irq_nosync, enable_irq}(int irq); -
      -
    • Used to disable a single interrupt
      • -
    -
    • -
  • void local_irq_save(unsigned long flags);, void local_irq_disable(void); -
      -
    • Diable all interrupts
      • -
    -
    • -
  • void local_irq_restore(unsigned long flags);, void local_irq_enable(void); -
      -
    • Restores interrupts
      • -
    -
    • -
-
-
-

Interrupt Sharing

-
    -
  • Modern hardware has been designed to allow the sharing of interrupts; PCI bus requires it
    • -
  • Linux supports interrupt sharing on all buses
    • -
-
-
-

Installing a Shared Handler

-
    -
  • The SA_SHIRQ bit must be specified in the flags argument when requesting the interrupt
    • -
  • The dev_id argument must be unique. Any pointer into the module’s address space will do, but dev_id definitely cannot be set to NULL
    • -
  • irq_request succeeds when one of the following is true: -
      -
    • The interrupt line is free
      • -
    • All handlers already registered for that line have also specified that the IRQ is to be shared
      • -
    -
    • -
-
-
-

Running the Handler

-
    -
  • use the dev_id argument to determine which, of possibly many, devices might be interrupting
    • -
-
-
-

Handler Arguments and Return Value

-
    -
  • typedef irqreturn_t (*irq_handler_t)(int irq, void * dev_id);
    • -
  • Irq is the interrupt number, dev_id is a short of client data
    • -
  • Interrupt handlers should return a value indicating whether there was an actual interrupt to handle -
      -
    • IRQ_HANDLED if the IRQ was able to be handled by the top half
      • -
    • IRQ_WAKE_THREAD if the bottom half shold be scheduled
      • -
    • IRQ_NONE if there was nothing to handle (or IRQ wasn’t from device)
      • -
    -
    • -
-
-
-

Interrupt Driven I/O

-
    -
  • Buffer in case of read/write delay
    • -
  • Improve performance by handling part of action asynchronously
    • -
  • Read: -
      -
    • fill buffer from device during interrupt
      • -
    • copy to user from buffer on system call
      • -
    -
    • -
  • Write: -
      -
    • copy from user to buffer on system call
      • -
    • write from buffer to device on interrupt
      • -
    -
    • -
-
- \ No newline at end of file diff --git a/kdlp.underground.software/course/slides/kernel_compilation.html b/kdlp.underground.software/course/slides/kernel_compilation.html deleted file mode 100644 index d6075240..00000000 --- a/kdlp.underground.software/course/slides/kernel_compilation.html +++ /dev/null @@ -1,90 +0,0 @@ -
-

How can I, a regular citizen, compile the Linux Kernel

- -
-
-
    -
  • Once you have the source code, you need to acquire all of the necessary tools to compile the kernel. A short list of most of them is available inside the Kernel repository in the file Documentation/Changes
    • -
  • On fedora you can install them with the command: -
      -
    • sudo dnf -y install gcc flex make bison openssl openssl-devel elfutils-libelf-devel ncurses-devel bc git tar dwarves
      • -
    -
    • -
  • On other distributions you will need the same dependencies but the packages might have slightly different names
    • -
-
-
-
    -
  • The Kernel is extremely configurable, but most of the settings are not something you want to worry about. To make a kernel that will be compatible with your existing OS, you want to base the configuration on those from your current kernel. -
      -
    • Run the command cp /boot/config-$(uname -r)* .config from within the linux folder
      • -
    -
    • -
  • Now generate a new config based on those options. -
      -
    • Run make olddefconfig to copy the existing settings, and automatically pick default values for new options
      • -
    -
    • -
  • You’re now ready to begin the actual build. Run this command: -
      -
    • make -j$(nproc) vmlinux modules
      • -
    -
    • -
-

Expect this to take a few hours! And crash many times!

-
-
-
    -
  • During the compilation process, you will inevitably run into issues like missing dependencies. If make can’t find some command, try and install it using dnf. If that doesn’t work, try googling the error to see if someone else has encountered it. Send us a message on discord if you get stuck. This is tedious but to be expected.
    • -
  • Once make finishes, you have officially compiled the Kernel. But… what good is a Kernel if you aren’t using it?
    • -
  • To install the kernel, run this command: -
      -
    • sudo make modules_install install
      • -
    -
    • -
  • You now have built and installed the kernel! Congrats!
    • -
-
-
-

Useful Resources

- -
- \ No newline at end of file diff --git a/kdlp.underground.software/course/slides/kernel_data_types.html b/kdlp.underground.software/course/slides/kernel_data_types.html deleted file mode 100644 index 29f8c416..00000000 --- a/kdlp.underground.software/course/slides/kernel_data_types.html +++ /dev/null @@ -1,173 +0,0 @@ -
-

Data Types in the Kernel

-
-
-

Use of Standard C Types

-
    -
  • Standard C data types aren’t the same size on all architectures -
      -
    • Raises portability issues
      • -
    • e.g long: -
        -
      • x86: sizeof(long) == 4
        • -
      • x86_64: sizeof(long) == 8
        • -
      -
      • -
    -
    • -
  • Example: in misc-progs -
      -
    • Program shows different data types and their sizes
      • -
    -
    • -
  • Unused by linux: {u,}intptr_t
    • -
-
-
-

Assigning an Explicit Size to Data Items

-
    -
  • Sometimes kernel code requires data items of a specific type -
      -
    • u8, u16, u32, u64
      • -
    • Corresponding signed types exist but rarely used
      • -
    -
    • -
  • Double underscore used in user space -
      -
    • __u8, __u32, …
      • -
    -
    • -
  • Compare to stdint.h in userspace
    • -
-
-
-

Interface-Specific types

-
    -
  • Most of the commonly used data types in the kernel have their own typedef statements. Prevents portability problems -
      -
    • pid_t instead of int
      • -
    -
    • -
  • typedef defined types now less fashionable -
      -
    • devs would rather see the real type information
      • -
    • Hence things like: struct task_struct
      • -
    • Ambiguity: How to printf *_t types?
      • -
    -
    • -
-
-
-

Other Portability Issues : Time Intervals

-
    -
  • Don’t assume that there are 1000 jiffies per second -
      -
    • This is true for i386 platform
      • -
    -
    • -
  • Instead the time using HZ
    • -
  • For example to check a against a time out of half a second : -
      -
    • HZ / 2
      • -
    -
    • -
-
-
-

Other portability Issues : Page Size

-
    -
  • Memory page is PAGE_SIZE bytes and not necessarily 4 KB
    • -
  • Don’t hard code the value, might diff size depending on platform
    • -
  • To (portably) get a 16 KB page for temporary data :
    • -
-
#include <asm/page.h>
-int order = get_order(16\*1024); 
-buf = get_free_pages(GFP_KERNEL, order);
-
-
-
-

other Portability Issues : Byte Order

-
    -
  • Don’t make assumptions about byte ordering -
      -
    • Some are little-endian and others big-endian
      • -
    • Most PCs store multibyte values low-byte first and other platforms do this the other way around
      • -
    -
    • -
  • u32 cpu_to_le32(u32) and u32 le32_to_cpu(u32) -
      -
    • Above macros do conversions between little-endian and big-endian depending on the processor
      • -
    -
    • -
-
-
-

Other Portability Issues : Data Alignment

-
    -
  • Support for unaligned data access varies by platform
    • -
  • Use: {get,put}_unaligned()
    • -
  • Better to align your data -
      -
    • FYI: Compiler may rearrange and add padding
      • -
    -
    • -
-
-
-

Pointers and Error Values

-
    -
  • Returning a NULL pointer when a function fails doesn’t communicate what the problem is
    • -
  • Return an error code
    • -
  • Error code is encoded in a pointer -
      -
    • void *ERR_PTR(long error)
      • -
    • Use IS_ERROR to check if returned pointer is error code -
        -
      • long IS_ERR(const void *ptr);
        • -
      -
      • -
    -
    • -
  • long PTR_ERR(const void *ptr); -
      -
    • Extract actual error code
      • -
    -
    • -
-
-
-

Linked Lists

-
    -
  • Kernel developers developed standard circular doubly linked list to prevent duplication of code
    • -
  • struct list_head is used to maintain a list of data structures -
      -
    • Standard circular doubly linked list
      • -
    -
    • -
  • INIT_LIST_HEAD -
      -
    • Used to initialize list head prior to use
      • -
    -
    • -
-
struct list_head todo_list; 
-INIT_LIST_HEAD(&todo_list);
-
-
-
-

Linked Lists

-

ll image
-Source

-
- \ No newline at end of file diff --git a/kdlp.underground.software/course/slides/linux_crash_course.html b/kdlp.underground.software/course/slides/linux_crash_course.html deleted file mode 100644 index 73e1ab3c..00000000 --- a/kdlp.underground.software/course/slides/linux_crash_course.html +++ /dev/null @@ -1,239 +0,0 @@ -
-

Kickoff

-

Linux crash course

-
-
-

What is Linux

-
    -
  • Free and open source operating system -
      -
    • Licensed under the GNU GPL v2.0
      • -
    -
    • -
  • The focus of this class
    • -
-
-
-

Where is Linux

-
    -
  • All top performing supercomputers
    • -
  • Most embedded systems
    • -
  • Android phones
    • -
  • Almost all of the computers powering the internet
    • -
  • Not many home computers -
      -
    • Unless you count chromebooks or the steam deck
      • -
    -
    • -
  • On kernel.org, freely available
    • -
-
-
-

Who is Linux

-
    -
  • Linus Torvalds -
      -
    • Created the Linux Kernel in 1991 while studying CS at the University of Helsinki as a personal project
      • -
    • Originally he supported only his own hardware.
      • -
    • "it probably never will support anything other than AT-harddisks, as that's all I have" [0]
      • -
    • Also eventually created git to act as the VCS for the kernel
      • -
    -
    • -
  • Open source community project -
      -
    • Thousands of individual contributors
      • -
    • Corporate sponsors (including Red Hat)
      • -
    • The Linux Foundation
      • -
    -
    • -
-
-
-

Why use Linux

-
    -
  • Free software (free as in free speech) -
      -
    • No spying, telemetry, adware etc in most mainstream distributions
      • -
    • Can inspect and change the code as you please
      • -
    -
    • -
  • Fully customizable -
      -
    • Different choices for things like desktop environments, GUI programs, etc
      • -
    -
    • -
  • Can run on less powerful computers
    • -
-
-
-

Modularity

-
    -
  • An OS made from many interchangeable components
    • -
  • Kernel a (relatively) small piece
    • -
  • Distribution (Distro) comes with choices for some of these pieces -
      -
    • Desktop environment
      • -
    • Package manager
      • -
    • Default Applications
      • -
    -
    • -
-
-
-

Distro vs Distro

-
    -
  • Differences in package naming conventions -
      -
    • libfoo-dev (apt) vs libfoo-devel (dnf)
      • -
    -
    • -
  • Differences in target audience -
      -
    • Servers vs desktop computers
      • -
    • Gaming vs Software development
      • -
    -
    • -
  • Differences in philosophy -
      -
    • Easy to use with sane defaults vs configure it all yourself
      • -
    • FOSS only vs access to proprietary software
      • -
    -
    • -
-
-
-

Exclusively Open Source

-

fedora logo
-source

-
-
-

Beginner-friendlyish

-

ubuntu logo
-source

-
-
-

Do anything, but by hand

-

archlinux logo
-source

-
-
-

Open Source Software as a Service

-

rhel logo
-source

-
-
-

System Stack Overview

-

system stack diagram

-
-
-

Bootloader

-
    -
  • Grub
    -grub menu
    • -
-
-
-

Kernel

-
$ cat /home/rms/interjection.txt
-I'd just like to interject for a moment.  What you're referring to as Linux,
-is in fact, GNU/Linux, or as I've recently taken to calling it, GNU plus Linux.
-Linux is not an operating system unto itself, but rather another free component
-of a fully functioning GNU system made useful by the GNU corelibs, shell
-utilities and vital system components comprising a full OS as defined by POSIX.
-
-Many computer users run a modified version of the GNU system every day,
-without realizing it.  Through a peculiar turn of events, the version of GNU
-which is widely used today is often called "Linux", and many of its users are
-not aware that it is basically the GNU system, developed by the GNU Project.
-
-There really is a Linux, and these people are using it, but it is just a
-part of the system they use.  Linux is the kernel: the program in the system
-that allocates the machine's resources to the other programs that you run.
-The kernel is an essential part of an operating system, but useless by itself;
-it can only function in the context of a complete operating system.  Linux is
-normally used in combination with the GNU operating system: the whole system
-is basically GNU with Linux added, or GNU/Linux.  All the so-called "Linux"
-distributions are really distributions of GNU/Linux.
-
-
-
-

C Standard Library

-
    -
  • Syscall wrappers -
      -
    • man 2 vs man 3
      • -
    -
    • -
-
$ cat /proc/$$/maps
-561fa56c6000-561fa56f5000 r--p 00000000 00:1f 669281                     /usr/bin/bash
-...
-561fa6d44000-561fa705c000 rw-p 00000000 00:00 0                          [heap]
-...
-7f0275133000-7f0275135000 rw-p 001d1000 00:1f 669212                     /usr/lib64/libc.so.6
-...
-7fffe0211000-7fffe0232000 rw-p 00000000 00:00 0                          [stack]
-
-
-
-

Shared Libraries

-
    -
  • Some examples
    • -
-
$ cat /proc/$$/maps
-...
-7f027513d000-7f027514b000 r--p 00000000 00:1f 669207                     /usr/lib64/libtinfo.so.6.3
-...
-7f02751ba000-7f02751bc000 rw-p 00033000 00:1f 669209                     /usr/lib64/ld-linux-x86-64.so.2
-...
-
-
-
-

Storage Layout

-
$ lsblk
-NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
-sr0     11:0    1 1024M  0 rom
-zram0  251:0    0  7.7G  0 disk [SWAP]
-vda    252:0    0   10G  0 disk
-├─vda1 252:1    0    1M  0 part
-├─vda2 252:2    0    1G  0 part /boot
-└─vda3 252:3    0    9G  0 part /home
-                                /
-
-
-
-

File system layout (See FHS)

-
    -
  • / → root of file system
    • -
  • /root → home folder for root account
    • -
  • /bin → command binaries
    • -
  • /boot → boot loader files
    • -
  • /dev → device files
    • -
  • /proc → information about running processes
    • -
  • /etc → configuration files
    • -
  • /home → location of users' home folders
    • -
  • /lib → libraries
    • -
-
-
-

System Stack Overview

-

system stack diagram

-
-
-

The future

-

Where will linux go from here?

-
-
\ No newline at end of file diff --git a/kdlp.underground.software/course/slides/linux_tips_of_great_power.html b/kdlp.underground.software/course/slides/linux_tips_of_great_power.html deleted file mode 100644 index d15c6334..00000000 --- a/kdlp.underground.software/course/slides/linux_tips_of_great_power.html +++ /dev/null @@ -1,307 +0,0 @@ -
-

Linux Tips

-

Cool stuff

-
-
-

How can I, a regular citizen, install Linux

-
    -
  • Flash image to USB and boot
    • -
  • Dual booting
    • -
  • VM is an option
    • -
  • Some devices ship with linux by default (not many)
    • -
-
-
-

Quick CLI Review

-

ls - list the files in current directory
-cd - change current directory
-mkdir - make a new directory
-nano - open nano editor
-touch - create a file or update its modification time
-chmod - change access permissions
-rm - remove (permanently delete) a file
-rmdir - remove a directory
-pwd - print working directory

-
-
-

CLI tips

-
    -
  • !! -
      -
    • replaced with last command
      • -
    -
    • -
  • !$ -
      -
    • Replaced with last argument of last command
      • -
    -
    • -
  • cd - -
      -
    • cd back to the previous location you were
      • -
    -
    • -
  • man -
      -
    • read manual for a command
      • -
    • man -a intro (intro to linux)
      • -
    -
    • -
  • sudo -
      -
    • run command as root
      • -
    • sudo !! (run previous command as root if you forgot sudo)
      • -
    -
    • -
-
-
-

Keyboard Shortcuts

-
    -
  • Ctrl+C → kills running program
    • -
  • Ctrl+Z → pauses running program
    • -
  • Ctrl+D → send EOF
    • -
  • Ctrl+S & Ctrl+Q → start & stop text stream
    • -
  • Ctrl+R → to search for previous commands (fuzzy finder)
    • -
  • Ctrl+W → erase current word up to cursor
    • -
  • Ctrl+V → verbatim input
    • -
-
-
-

Useful tools

-
    -
  • Text Editor -
      -
    • Nano or even vim
      • -
    -
    • -
  • Cscope -
      -
    • Tool for browsing source code
      • -
    -
    • -
  • Man -
      -
    • Read the docs (manual) for every BASH command/C library
      • -
    -
    • -
-
-
-

sed - a text processing utility - Stream EDitor

-

sed

-
-
-

sed

-

$ sed -i -e ‘s/ *$//’ -e ‘s/[[:space:]]*$//’ -e ‘/^$/d’ file.sh

-
    -
  1. Delete all spaces at the end of every line in a file
    2. -
  2. Delete all tabs at the end of every line in a file
    3. -
  3. Delete all empty lines in a file
    -^ is the beginning of a line
    -$ is the end of a line
    4. -
-
-
-

vim/cscope - text based search utility

-
    -
  • cscope -R (in a parent directory to create a cscope database)
    • -
  • CTRL+D (to exit when the database is generated)
    • -
  • Open vim in the directory where cscope.out resides and you can search for text within the tree
    • -
  • Using cscope you can traverse the files that include the text you searched for
    • -
-
-
-

grep - command line util to find text

-

You can pipe the output of a command into grep or give a path to an input file
-dmesg | grep "text" or grep "text" file.sh
-If you want to find file.txt in your home directory
-ls ~ | grep file.txt #not entirely correct
-. → any character in regex
-ls /home | grep -F file.txt #-F means no regex, just text
-^ → start of line
-$ → end of line

-
-
-

awk - a text processing utility

-

awk

-
-
-

Basic Scripting

-
-
-

Quick Variable Review

-

$0...9 - $0 is filepath of the script and $1...9 are the arguments
-$# - the number of arguments
-$@ - an array of arguments
-$? - process’ exit code

-
-
-

Arithmetic Operations

-

x=1
-z=2
-$x+$z #3
-$z-$x #1
-$x*$z #2
-$z/$x #2
-$x%$z #1
-[ $x == $z ] #false
-[ $x != $z ] #true

-
-
-

Relational Operations

-

x=1
-z=2
-[ $x -eq $z ] #false (they aren’t equal)
-[ $x -ne $z ] #true (they are unequal)
-[ $x -gt $z ] #false (x is not greater than z)
-[ $x -lt $z ] #true (x is less than z)
-[ $x -ge $z ] #false (x is not greater than or equal to z)
-[ $x -le $z ] #true (x is less than or equal to z)

-
-
-

Boolean Operations

-

x=1
-z=2
-[ ! true ] #false
-[ $x -gt $z -o $x -ne $z ] #true
-[ $x -ne $z -a $x -lt $z ] #true

-
-
-

String Operations

-

x="1"
-z="2"
-[ $x == $z ] #false
-[ $x != $z ] #true
-[ -z $x ] #false (“1” is not of length zero)
-[ -n $x ] #true (“1” is of length non-zero)
-[ $x ] #true ($x is not an empty string)

-
-
-

File Operations

-

file=”secrets.txt”
-$ ls -l secrets.txt
---w------- 1 kdlp kdlp 106 Aug 22 04:01 secrets.txt
-[ -d $file ] #false ($file is not a directory)
-[ -f $file ] #true ($file is a regular file)
-[ -r $file ] #false ($file is not readable)
-[ -w $file ] #true ($file is writable)
-[ -x $file ] #false ($file is not executable)
-[ -s $file ] #true ($file has size greater than zero)
-[ -e $file ] #true ($file exists)

-
-
-

Getting set up

-

$ touch file.sh
-$ chmod 744 file.sh (chmod changes the permissions that people have for this file)
--rwxr--r-- (new permissions after chmod, now you can execute file.sh)
-r: read w: write x: execute
-$ ls -l
-Run ^ to check permissions in cwd
-First permissions index is a directory marker
-Then 1-3: owner permissions 4-6: group permissions 7-9: other user permissions

-
-
-

Getting set up

-

To write a bash script you can execute direction, you will need to include a shebang line to indicate that bash should be used to interpret the program
-On the first line you write #!/bin/bash
-Now you can use all commands you can use in your CLI in combination with conditionals, loops, and more

-
-
-

Command-line Arguments

-

You can reference the arguments passed to your script with $i where i is the index of the argument when the script is run
-$ ./script.sh dog cat
-$0 #./script.sh
-$1 #dog
-$2 #cat

-
-
-

If statements: integers and strings

-

if statements

-
-
-

Loops

-

loops

-
-
-

Functions

-

Functions in a bash script are declared with
-foo(){...}
-or
-function foo(){...}
-Function arguments come after calling the function, like passing arguments to a script e.g. foo arg1 arg2 ...
-Within foo() → $1=arg1, $2=arg2, ...

-
-
-

A && echo "true" || echo "false"

-

‘echo true’ fires if the prior command succeeds (echo $? = 0)
-‘echo false’ fires if the prior command fails
-What is echoed?

-
-
-

Debugging / Accessing Linux systems

-
-
-

Remote Access

- -
-
-

TTL cable

-
    -
  • Used to connect to serial console ports
    • -
  • Microcontrollers, Raspberry Pi, WiFi routers ...
    • -
  • Useful for low level debugging
    • -
  • DO NOT HAVE THE TTL RED WIRE AND USBC POWER SOURCE PLUGGED IN AT THE SAME TIME - -
    • -
-
-
-

ttl cable

-
-
-

ttl wiring

-
-
-

Tracing and bpftrace

-
    -
  • Tracing is most often used for debugging purposes. In the kernel, debugging can become overwhelming without the proper tools.
    • -
  • bpftrace is a tracing language which relies on the Linux BPF system and built-in Linux tracing capabilities, such as: kprobes(kernel level), uprobes(user-space level), and tracepoints
    • -
  • Which are all types of breakpoints in a stream of processes
    • -
-
-
-

Tracing and bpftrace

- -
- \ No newline at end of file diff --git a/kdlp.underground.software/course/slides/memory_allocation.html b/kdlp.underground.software/course/slides/memory_allocation.html deleted file mode 100644 index 08bd4246..00000000 --- a/kdlp.underground.software/course/slides/memory_allocation.html +++ /dev/null @@ -1,169 +0,0 @@ -
-

Allocating Memory

-

Chapter 8

-
-
-

Story of kmalloc

-
    -
  • Similar to malloc
    • -
  • Allocates contiguous memory
    • -
-
-
-

The Flags Argument

-

void *kmalloc(size_t size, int flags);

-
    -
  • GFP_ATOMIC -
      -
    • Allocate memory outside of a process context. Never sleeps
      • -
    -
    • -
  • GFP_KERNEL -
      -
    • Normal allocation of kernel memory
      • -
    -
    • -
  • GFP_USER -
      -
    • Used to allocate memory for user-space pages. It may sleep
      • -
    -
    • -
-
-
-

Memory zones

-
    -
  • DMA-capable memory -
      -
    • Used for DMA access
      • -
    -
    • -
  • Normal memory -
      -
    • Normal allocation
      • -
    -
    • -
  • High memory -
      -
    • Used to allow access to large amounts of memory on 32 bit platforms
      • -
    -
    • -
  • __GFP_DMA (only the dma zone is searched)
    • -
  • If no flags present, DMA and and normal memory are searched
    • -
  • __GFP_HIGMEM (all 3 zones are searched)
    • -
-
-
-

Size Argument

-
    -
  • Kernel memory available page sized chunks
    • -
  • Kernel creates set pools of memory objects of fixed size
    • -
  • Allocation is handled by returning a memory chunk from a sufficiently large pool back to the requester
    • -
  • memory allocated is fixed size byte arrays and requester will likely get back more memory than asked up to twice as much
    • -
-
-
-

Lookaside Caches

-
    -
  • Special memory pool for high volume objects
    • -
  • kmem_cache_t *kmem_cache_create(...) -
      -
    • creates a new cache object that can host any number of memory areas all of the same size, specified by the size argument
      • -
    -
    • -
  • void *kmem_cache_alloc(kmem_cache_t *cache, flags) -
      -
    • Allocate objects from cache object
      • -
    • cache ( cache created by kmem_cache_create )
      • -
    -
    • -
  • void kmem_cache_free(kmem_cache_t *cache, void *obj) -
      -
    • Free object memory into a cache
      • -
    -
    • -
  • kmem_cache_destroy(kmem_cache_t *cache) -
      -
    • Frees cache data structure; no longer usable
      • -
    -
    • -
-
-
-

Memory pools

-

Form of lookaside cache in kernel where memory allocation is not allowed to fail

-
-
-

get_free_page and friends

-

Used to allocate memory in page sized chunks

-
    -
  • for when driver needs big chunks of memory
    • -
  • void *get_zeroed_page(unsigned int flags); -
      -
    • Returns pointer to a new page and fills the page with zeros
      • -
    -
    • -
  • __get_free_page -
      -
    • Similar to get_zeroed_page, but doesn’t clear the page
      • -
    -
    • -
  • Flags argument works similarly to kmalloc -
      -
    • Usually GFP_KERNEL or GFP_ATOMIC is used
      • -
    -
    • -
  • free_page(...), free_pages(...)
    • -
-
-
-

vmalloc

-

Allocates contiguous memory in virtual address space

-
    -
  • Pages not consecutive. Each page retrieved by a call to alloc_page
    • -
  • Address range used by vmalloc is synthetic. Sets up page tables
    • -
  • void *vamalloc(unsigned long size);
    • -
  • void free(void * addr);
    • -
-
-
-

Per-CPU Variables

-
    -
  • When a per-cpu variable is created, each processor gets a copy
    • -
  • Used in networking subsystem. Require no locking and are fast
    • -
  • DEFINE_PER_CPU(type, name)
    • -
  • Dynamically allocated -
      -
    • void *alloc_percpu(type);
      • -
    • void * __alloc_percpu(size_t size, size_t align);
      • -
    -
    • -
-
-
-

Obtaining Large Buffers

-
    -
  • Allocations of large contiguous memory buffers are prone to failure
    • -
  • Best way of performing large I/O operations is through scatter/gather operations (chapter 15)
    • -
-
-
-

Acquiring Dedicated Buffer at Boot Time

-
    -
  • If you need a huge buffer of physical contiguous memory, request memory at boot time.
    • -
  • Dirty technique that bypasses memory management policies
    • -
  • Modules can’t allocate memory at boot time, only drivers directly linked to the kernel
    • -
-
- \ No newline at end of file diff --git a/kdlp.underground.software/course/slides/mm_dma.html b/kdlp.underground.software/course/slides/mm_dma.html deleted file mode 100644 index 6ca164e6..00000000 --- a/kdlp.underground.software/course/slides/mm_dma.html +++ /dev/null @@ -1,385 +0,0 @@ -
-

Memory Mapping and DMA

-

Chapter 15

-
-
-

Address Types

-
    -
  • vmlinux: Virtual Memory linux
    • -
  • Programs can allocate far more memory than is physically available
    • -
  • Address types : -
      -
    • User virtual addresses
      • -
    • Physical addresses
      • -
    • Bus addresses (on some arches: same as physical addresses)
      • -
    • Kernel logical addresses (often at constant offset from physical address)
      • -
    • Kernel virtual addresses (superset of logical addresses)
      • -
    -
    • -
  • Convert addresses using:
    • -
  • __{p,v}a() from asm/page.h
    • -
-
-
-

Physical Addresses and Pages

-
    -
  • Physical RAM divided into fixed-size “pages”
    • -
  • PAGE_SIZE may vary between arches -
      -
    • On x86_64: usually 4K
      • -
    -
    • -
  • Page tables! -
      -
    • The processor uses page tables to translate virtual addresses into corresponding physical addresses
      • -
    • Multi-level memory
      • -
    -
    • -
-
-
-

Page table diagram

-

page table
-source

-
-
-

High and Low Memory

-
    -
  • Low memory is memory for which logical address exists in kernel space -
      -
    • On most systems, all memory is low memory
      • -
    -
    • -
  • High memory: address physical memory outside of virtual address space -
      -
    • E.g. x86: 32 bit pointer => 4G addressable memory
      • -
    • But what if we want more?
      • -
    -
    • -
  • Must set up mapping as needed
    • -
  • Unnecessary on many systems (64-bit)
    • -
-
-
-

Memory Map and struct Page

-
    -
  • Struct Page -
      -
    • Keeps track of info about physical memory
      • -
    -
    • -
  • There is one struct page for each physical page
    • -
  • Some macros and functions that translate between struct page pointers and virtual addresses -
      -
    • struct page *virt_to_page(void *kaddr);
      • -
    • struct page *pfn_to_page(int pfn);
      • -
    • void *page_address(struct page *page);
      • -
    • void *kmap(struct page *page);
      • -
    • ...
      • -
    -
    • -
  • Useful when dealing with high/low mem situations (x86)
    • -
-
-
-

Virtual Memory Areas

-
    -
  • VMA is a kernel data structure used to manage distinct regions of a process’s address space
    • -
  • Memory area of a process $PID can be seen by -
      -
    • cat /proc/$PID/maps
      • -
    -
    • -
  • struct vm_area -
      -
    • A driver that supports mmap(2) needs to initialize a VMA
      • -
    • This structure corresponds to the memory userspace maps via mmap(2)
      • -
    -
    • -
-
-
-

The Process Memory Map

-
    -
  • struct mm_struct -
      -
    • Holds all the other data structures together
      • -
    -
    • -
  • Contains : -
      -
    • A process’s list of virtual memory areas
      • -
    • Page tables
      • -
    • A semaphore and a spinlock
      • -
    • …
      • -
    -
    • -
  • mm_struct can be shared between processes
    • -
-
-
-

The mmap Device Operation

-
    -
  • Allows direct userspace access to device memory
    • -
  • The mmap(2) ptr is part of struct file_operations
    • -
  • System call: mmap(caddr_t addr, size_t len, int prot, int flags, int fd, off_t offset)
    • -
  • File operation: int (*mmap)(struct file *filp, struct vm_area_struct *vma);
    • -
  • PAGE_SIZE granularity
    • -
-
-
-

Using remap_pfn_range

-
    -
  • This function along with io_remap_page_range are used to map a range of physical addresses
    • -
  • int remap_pfn_range(struct vm_area_struct *vma, unsigned long virt_addr, unsigned long pfn, unsigned long size, pgprot_t prot);
    • -
  • remap_pfn_range intended for use when pfn refers to RAM and io_remap_pfn is used phys_addr points to I/O memory
    • -
-
-
-

Adding VMA operations

-
    -
  • open(2) and close(2) -
      -
    • Called whenever a process opens or closes the VMA
      • -
    -
    • -
  • Installation is like struct file_operations
    • -
-
void my_vma_open(struct vm_area_struct \*vma) { … }
-void my_vma_close(struct vm_area_struct \*vma) { … }
-static struct vm_operations_struct my_vma_ops = {
-    .open = my_vma_open,
-    .close = my_vma_close, };
-
-
    -
  • Done in mmap method:
    • -
-
vma->vm_ops = my_vma_ops;
-
-
-
-

Mapping Memory with nopage

-
    -
  • Page not present in mapping (page fault)? Call nopage() to fix it -
      -
    • struct page *(*nopage)(struct vm_area_struct *vma, unsigned long address, int *type);
      • -
    -
    • -
  • mremap(2) can change the bounding addresses of a mapped region -
      -
    • Driver is not notified
      • -
    -
    • -
  • Called when user process tries to access a page that isn’t present in memory
    • -
  • Returns appropriate struct page *
    • -
  • Different struct vma_operations for driver when using nopage
    • -
-
-
-

Remapping Specific I/O Regions

-
    -
  • Most drivers only remap small range
    • -
  • Done by calculating offsets
    • -
  • User process can always use mremap to extend it’s mapping -
      -
    • Implement a simple nopage method that causes a bus signal to be sent to the faulting process to prevent extension of mapping
      • -
    -
    • -
-
-
-

Remapping RAM

-
    -
  • Reserved pages are the only ones that can safely be mapped to user space -
      -
    • Reserved pages are pages that aren’t available for memory management
      • -
    • E.g. 640KB to 1G on x86
      • -
    -
    • -
-
-
-

Remapping kernel Virtual Addresses

-
    -
  • Virtual address is an address returned by vmalloc -
      -
    • Mapped in kernel page tables, not user page tables
      • -
    -
    • -
  • No need to check the order parameter -
      -
    • Vmalloc allocates memory one page at a time. More likely to succeed
      • -
    -
    • -
-
-
-

Performing direct I/O

-
    -
  • Most I/O operations are buffered through the kernel
    • -
  • Direct I/O from user space could potentially speed things up -
      -
    • setup overhead for direct I/O is potential bottleneck
      • -
    -
    • -
  • Not necessary for a char driver
    • -
  • Done with function: get_user_pages()
    • -
-
-
-

Asynchronous I/O

-
    -
  • Allows applications to to do other processing while I/O is in progress -
      -
    • almost always involves direct I/O
      • -
    -
    • -
  • Necessary struct file_operations entries: -
      -
    • ssize_t (*aio_read)(...)
      • -
    • ssize_t (*aio_write)(...)
      • -
    • int (*aio_fsync)(...)
      • -
    -
    • -
-
-
-

Direct Memory Access

-

“DMA is the hardware mechanism that allows peripheral components to transfer their I/O data directly to and from main memory without the need to involve the system processor.” -- LDD3, page 440

-
-
-

Overview of a DMA Data Transfer

-
    -
  • Synchronous DMA read -
      -
    • Driver allocates DMA buffer, the hardware transfers data to buffer, process sleeps
      • -
    • On completion of write, hardware raises interrupt
      • -
    • The interrupt handler awakens the process and the actual reading takes place
      • -
    -
    • -
  • Asynchronous DMA read -
      -
    • Hardware raises an interrupt
      • -
    • Interrupt handler allocates DMA buffer
      • -
    • Peripheral device writes to buffer and raises another interrupt when done
      • -
    • Second handler handles new data and wakes up relevant process
      • -
    -
    • -
-
-
-

Allocating the DMA Buffer

-
    -
  • Buffer Size > PAGE_SIZE: -
      -
    • Buffer must occupy contiguous pages in physical memory because ISA and PCI carry physical addresses
      • -
    -
    • -
  • Can reserve physical memory with mem= kernel param -
      -
    • Use ioremap to get a usable virtual address
      • -
    -
    • -
-
-
-

Bus Addresses

-
    -
  • Device driver using DMA talks to interface bus that carries a physical address
    • -
  • DMA-based hardware actually uses bus addresses -
      -
    • Sometimes physical addresses
      • -
    • Sometimes special other addresses
      • -
    -
    • -
-
-
-

DMA mappings

-

A DMA mapping is a combination of allocating a DMA buffer and generating an address for that buffer that is accessible by the device

-
    -
  • Bounce buffers -
      -
    • Necessary when a device cannot access some area of physical memory, such as highmem
      • -
    -
    • -
  • Cache coherency -
      -
    • DMA mappings must account for this
      • -
    • dma_alloc_coherent()
      • -
    -
    • -
  • Other options -
      -
    • Streaming mappings
      • -
    • scatter/gather mappings
      • -
    -
    • -
-
-
-

DMA Pools

-
    -
  • For small, coherent DMA mappings -
      -
    • E.g. device needs DMA area smaller than one page
      • -
    -
    • -
  • Also used where dma is performed in areas embedded in a larger structure
    • -
  • dma_pool_{create,destroy,alloc,free}()
    • -
-
-
-

Registering DMA usage

-
    -
  • Similar to I/O ports and interrupt registry
    • -
  • To obtain and release DMA channel: -
      -
    • int request_dma(unsigned int channel, const char *name);
      • -
    • void free_dma(unsigned int channel);
      • -
    -
    • -
  • Take same care of DMA channels as interrupts and I/O ports -
      -
    • Request channel at open time
      • -
    • Delaying request allows sharing between drivers
      • -
    -
    • -
-
-
-

Talking to DMA controller

-
    -
  • Read asm/dma.h for details
    • -
  • Driver is in charge of configuring the DMA controller -
      -
    • When read or write is called or for asynchronous transfers
      • -
    -
    • -
  • Asynchronous transfers happen at open(2) or ioctl(2) call
    • -
  • DMA controller is a shared resource -
      -
    • Use: {claim,release}_dma_lock()
      • -
    -
    • -
  • Methods: -
      -
    • set_dma_mode() (read or write)
      • -
    • set_dma_addr() (set address of buffer)
      • -
    • set_dma_count() (how many bytes to transfer)
      • -
    -
    • -
-
- \ No newline at end of file diff --git a/kdlp.underground.software/course/slides/modules_drivers.html b/kdlp.underground.software/course/slides/modules_drivers.html deleted file mode 100644 index fcdb1e9a..00000000 --- a/kdlp.underground.software/course/slides/modules_drivers.html +++ /dev/null @@ -1,77 +0,0 @@ -
-

Kernel Modules and Device Drivers

-

In Linux

-

LDD3: Chapters 1 & 2

-
-
-

Questions to answer

-
    -
  • What are devices and drivers?
    • -
  • What are Linux kernel modules?
    • -
  • How are these concepts related?
    • -
-
-
-

What is a device?

-
    -
  • Examples: screen, keyboard, mouse, network interface, printer
    • -
  • The "organs" of the computer
    • -
  • A component of the computer that the cpu can interact with electronically
    • -
-
-
-

What is a device driver?

-
    -
  • Bridge OS/Hardware gap
    • -
  • Allow users to talk to device
    • -
  • OS ↔ driver ↔ CPU ↔ devices
    • -
-
-
-

Overview

-


-http://haifux.org/lectures/86-sil/kernel-modules-drivers/kernel-modules-drivers.html

-
-
-

What is a Kernel Module

-
    -
  • Snippet of code that can be added to a running kernel
    • -
-
-
-

Why Kernel Modules for Device Drivers?

-
    -
  • Keep kernel text/data minimal -
      -
    • Less error surface
      • -
    -
    • -
  • More customization -
      -
    • Can add hardware at runtime
      • -
    -
    • -
  • Easier to develop -
      -
    • Change text/data without reboot
      • -
    -
    • -
-
-
-

The End

-
-
\ No newline at end of file diff --git a/kdlp.underground.software/course/slides/timing_scheduling.html b/kdlp.underground.software/course/slides/timing_scheduling.html deleted file mode 100644 index 0debef9b..00000000 --- a/kdlp.underground.software/course/slides/timing_scheduling.html +++ /dev/null @@ -1,189 +0,0 @@ -
-

Measuring Time

-

Chapter 7

-
-
-

Measuring time lapses

-
    -
  • Timer interrupts
    • -
  • Generated by system hardware
    • -
  • Interval programmed at boot time by kernel
    • -
  • 100 or 1000 interrupts per second for most platforms -
      -
    • See HZ macro
      • -
    -
    • -
  • jiffies{_64,} (internal kernel counter) -
      -
    • Number of clock ticks since last boot for a given CPU
      • -
    • Has nothing to do with wall clock time
      • -
    -
    • -
-
-
-

Using jiffies

-
    -
  • Read only
    • -
  • Needed to calculate a future time stamp
    • -
  • time_{before,after}{,_eq}()
    • -
  • struct timeval & struct time_spec -
      -
    • Used to exchange time representations with user space programs
      • -
    -
    • -
-
-
-

Processor-Specific Registers

-
    -
  • Most modern CPUs have a counter register -
      -
    • Used to count clock cycles
      • -
    -
    • -
  • Depends on platform but this register may or may not be readable
    • -
  • Handle overflows if reg is not writable
    • -
  • rdtsc on x86 -
      -
    • Read (rd) time stamp counter (tsc) register
      • -
    • Different on other platforms
      • -
    -
    • -
-
-
-

Knowing the Current Time

-
    -
  • Drivers use jiffies to calculate time intervals across events -
      -
    • For example, to tell double-clicks from single-clicks
      • -
    -
    • -
  • Processor specific registers for more precise measurements
    • -
  • Use do_gettimeofday to get absolute timestamp in kernel -
      -
    • Uses struct timeval to export time stamp
      • -
    -
    • -
  • Otherwise real world time is best left for user space where the c library offers better support -
      -
    • clock_gettime(3) for example
      • -
    -
    • -
-
-
-

Long Delays

-

Chapter 7

-
-
-

Long Delays: Approach #1: Busy waiting

-
    -
  • A loop that monitors the jiffy counter
    • -
  • while (time_before(jiffies, j1))
    -cpu_relax();
    • -
  • Call to cpu_relax invokes a way of saying you’re not doing much with the processor right now
    • -
  • If kernel not configured for preemption, loop locks processor for duration of delay
    • -
  • Avoid: what does relax even mean?? How relaxed?? (not very)
    • -
  • On x86: cpu_relax <==> no-op
    • -
-
-
-

Long Delays: Approach #2: Yielding the processor

-
    -
  • Release the cpu when not interested in it
    • -
  • while (time_before(jiffies, j1))
    -schedule();
    • -
  • The current process does nothing but release the cpu but remains in run queue
    • -
  • On realtime: schedule just re-schedules the caller!
    • -
-
-
-

Long Delays: Approach #3: Timeouts

-
    -
  • Ask kernel to do a delay for you
    • -
  • wait_event _timeout(...); -
      -
    • Wait queue to wait for some other event but also run within a certain period of time
      • -
    • Returns after timeout
      • -
    -
    • -
  • wait_event_interruptable_timeout(...); -
      -
    • Interruptible sleep
      • -
    -
    • -
-
-
-

Short Delays

-
    -
  • Drivers deal with latencies using short delays
    • -
  • ndelay, udelay, mdelay -
      -
    • Nanoseconds, microseconds and milliseconds
      • -
    -
    • -
-
-
-

Kernel Timers

-
    -
  • Asynchronous execution
    • -
  • Call f(x,y,z) at some point in the future
    • -
  • Used to schedule an action to happen later without blocking the current process -
      -
    • For example, polling a device by checking its state at regular intervals
      • -
    -
    • -
-
-
-

Tasklets

-
    -
  • Data structure that must be initialized before use. -
      -
    • struct tasklet_struct { (*func)(u long); u long data; };
      • -
    -
    • -
  • Mostly used in interrupt management and run at interrupt time
    • -
  • Let kernel decide when to run the task
    • -
  • Run in atomic context
    • -
-
-
-

Workqueues

-
    -
  • Similar to tasklets with a few differences
    • -
  • workqueues run in kernel process context while tasklets run in atomic context
    • -
  • Kernel code can delay execution of workqueue functions
    • -
  • Key difference is that tasklets execute quickly and in atomic mode
    • -
  • Both tasklets and workqueues run on the same processor they were submitted
    • -
-
-
-

The Shared Queue

-
    -
  • Used occasionally to submit tasks if in driver
    • -
  • Tasks may take longer to execute since there’s sharing
    • -
  • Struct jiq_work. Actual code below:
    • -
-
prepare_to_wait(&jiq_wait, &wait, TASK_INTERRUPTIBLE); 
-schedule_work(&jiq_work); 
-schedule( ); 
-finish_wait(&jiq_wait, &wait);
-
-
- \ No newline at end of file diff --git a/kdlp.underground.software/course/slides/writing_character_drivers.html b/kdlp.underground.software/course/slides/writing_character_drivers.html deleted file mode 100644 index 14a74d84..00000000 --- a/kdlp.underground.software/course/slides/writing_character_drivers.html +++ /dev/null @@ -1,278 +0,0 @@ -
-

Character Drivers

-

In Linux

-
-
-

Live Coding

-

Bare bones module

-
-
-

Major and Minor

-
    -
  • Major: maps to driver associated with device
    • -
  • Minor: which device within the driver -
      -
    • typedef dev_t
      • -
    • MAJOR(dev_t)
      • -
    • MINOR(dev_t)
      • -
    • MKDEV(major, minor)
      • -
    -
    • -
  • ls -l /dev -
      -
    • Look for c in first column
      • -
    -
    • -
-
-
-

Allocating and Freeing Device numbers

-
    -
  • register_chrdev_region -
      -
    • N minors (static)
      • -
    -
    • -
  • alloc_chrdev_region -
      -
    • Allocates some minors and a major for a specified number of needed minors
      • -
    -
    • -
  • unregister_chrdev_region -
      -
    • Frees major and minor number
      • -
    -
    • -
  • /proc/devices
    • -
-
-
-

Dynamic allocation

-
    -
  • alloc_chrdev_region vs register_chrdev_region -
      -
    • Dynamic vs Static
      • -
    • Affects creation of device nodes
      • -
    • Static: easy node creation
      • -
    • Dynamic: bit more complex
      • -
    -
    • -
-
-
-

Device registration

-
    -
  • Register char dev -
      -
    • cdev_alloc
      • -
    • my_cdev->ops = &my_fops;
      • -
    • cdev_init
      • -
    • cdev_add
      • -
    -
    • -
  • Remove char dev -
      -
    • cdev_del
      • -
    -
    • -
  • The (much) older way -
      -
    • register_chrdev
      • -
    • unregister_chrdev
      • -
    -
    • -
-
-
-

Live Coding

-

Major + Minor, make it show up in /dev/

-
-
-

Important data structures

-
    -
  • struct file_operations -
      -
    • Holds pointer to the kernel methods (read, write, ioctl etc)
      • -
    -
    • -
  • struct file -
      -
    • Represents an open file (similar to FILE * or a file number in user space)
      • -
    -
    • -
  • struct inode -
      -
    • Represents a unique file within the directory structure (one or more paths may point to a single inode)
      • -
    • Many to one (file:inode)
      • -
    -
    • -
-
-
-

file_operations struct

-
    -
  • owner = THIS_MODULE
    -The Methods to the Madness
    • -
  • llseek(2)
    • -
  • read(2)
    • -
  • write(2)
    • -
  • ioctl(2)
    • -
  • open(2)
    • -
  • release(2)
    • -
  • ...
    • -
-
-
-

file struct

-
    -
  • NOT the same as C library FILE pointer -
      -
    • It is its equivalent within the kernel
      • -
    -
    • -
  • Represents an open file
    • -
  • Important fields -
      -
    • f_mode (opened for reading, writing etc (what you passed to open()))
      • -
    • f_pos (current location in file)
      • -
    • f_flags (flags you passed to open())
      • -
    • f_op (pointer to file operations the kernel will use with this file)
      • -
    • private_data (driver can use this to hold info it needs)
      • -
    -
    • -
-
-
-

inode struct

-
    -
  • Used by kernel to represent files
    • -
  • Two important fields -
      -
    • i_rdev: Contains device number
      • -
    • i_cdev: Contains pointer to char device
      • -
    -
    • -
-
-
-

Live Coding

-

Stubs for driver functions

-
-
-

Memory usage

-
    -
  • kmalloc (often devm_kmalloc) -
      -
    • Typically used in probe/init
      • -
    -
    • -
  • kfree (only needed if you aren’t using devm) -
      -
    • Typically used in remove/exit
      • -
    -
    • -
  • Kinda like malloc/free with extra steps -
      -
    • No glibc!
      • -
    • Need to tell the kernel memory zone
      • -
    • How long can you afford to wait?
      • -
    -
    • -
-
-
-

Open and Release

-
    -
  • Open -
      -
    • Initialize device if being used for the first time
      • -
    • Check for device specific errors
      • -
    • Update f_op
      • -
    • Set private data if needed
      • -
    -
    • -
  • Release -
      -
    • Deallocate anything allocated in open
      • -
    -
    • -
  • Kinda like kmalloc/kfree for the whole thing
    • -
-
-
-

Read and Write

-
    -
  • How do we get access to memory in user space?
    • -
  • Read (write from kernel side) -
      -
    • copy_to_user
      • -
    -
    • -
  • Write (read from kernel side) -
      -
    • copy_from _user
      • -
    -
    • -
  • __user to mark a userspace ptr -
      -
    • Don’t just dereference!
      • -
    -
    • -
-
-
-

Why not just dereference?

-
    -
  • In process context -
      -
    • Oops + death on a page fault
      • -
    -
    • -
  • Elsewhere, null ptr deref:
    -= panic() (big death) (not from system call)
    • -
  • Unchecked _* versions of copy*_user
    • -
-
-
-

Live Coding

-

Make it do something! Echo user input

-
-
-

Scull: Simple Character Utility for Loading Local Localities

- -
-
-

Linux Driver Development with Raspberry Pi - ch4

-
    -
  • Chapter 4 has examples of complete char dev including miscellaneous character module
    • -
  • miscdevice -
      -
    • Simpler way to write char device driver
      • -
    • Used for sense-hat display
      • -
    -
    • -
-
-
-

Discussion

-
- \ No newline at end of file diff --git a/kdlp.underground.software/course/slides/writing_modules.html b/kdlp.underground.software/course/slides/writing_modules.html deleted file mode 100644 index 58e400a7..00000000 --- a/kdlp.underground.software/course/slides/writing_modules.html +++ /dev/null @@ -1,176 +0,0 @@ -
-

How to Write Kernel Modules

-
    -
  • Kernel documentation -
      -
    • Hacking index
      • -
    • Driver API
      • -
    • Module building
      • -
    -
    • -
  • Source analysis & books - -
    • -
  • Practice!
    • -
-
-
-

Hello World Kernel Module

-
$ cat hello.c
-#include <linux/init.h>
-#include <linux/module.h>
-
-MODULE_LICENSE("GPL");
-
-static int hello_init(void)
-{
-    printk(KERN_ALERT "Hello, world\n");
-    return 0;
-}
-
-static void hello_exit(void)
-{
-    printk(KERN_ALERT "Goodbye, cruel world\n");
-}
-
-module_init(hello_init);
-module_exit(hello_exit);
-
-
-
-

We are not in Kansas anymore

-
    -
  • Goodbye main
    • -
  • Goodbye C standard library
    • -
  • Goodbye IEEE754
    • -
  • Hello ring 0 (x86_64) / EL1 (aarch64)
    • -
  • Hello asynchronous execution
    • -
  • Hello kernel API
    • -
-
-
-

Kernelspace vs Userspace

-
    -
  • Small, fixed-size stack
    • -
  • Each mode can have it’s own address space
    • -
  • Kernel space handles access to hardware
    • -
  • Execution is transferred to kernel space after a system call
    • -
-
-
-

Reentrant code

- -
-
-

Why not stick with userspace?

-
    -
  • Context switches makes response time slower
    • -
  • Access to hardware requires priveledged instructions -
      -
    • The most important drivers can’t be handled in user space
      • -
    -
    • -
-
-
-

Symbols, Versions & Platform dependence

-
    -
  • Kernel symbol table -
      -
    • Contains address of global kernel items (functions and variables)
      • -
    • Changes every time the kernel is recompiled
      • -
    • Modules can access symbols already exported by other code
      • -
    • Modules can introduce their own exported symbols
      • -
    -
    • -
  • A Module must be recompiled for each version of the kernel that wants to link with -
      -
    • Module has to be built with same understanding of target processor as kernel
      • -
    • Easier to distribute driver for general distribution by contributing to the mainline kernel
      • -
    -
    • -
-
-
-

Compiling Modules (out-of-tree)

-
    -
  • Develop your code in own folder
    • -
  • Compile by referencing the kernel makefile
    • -
  • If you build your own kernel, the makefile will already be installed
    • -
  • To get the makefile for a kernel from fedora run
    • -
-
sudo dnf install kernel-devel-`uname -r`
-
-
-
-

Hello World Kernel Module continued

-
$ cat Makefile
-obj-m += hello.o
-
-.PHONY: build clean load unload
-
-build:
-    make -C /lib/modules/$(shell uname -r)/build modules M=$(shell pwd)
-
-clean:
-    make -C /lib/modules/$(shell uname -r)/build clean M=$(shell pwd)
-
-load:
-    sudo insmod hello.ko
-unload:
-    -sudo rmmod hello
-
-
-
-

Loading and Unloading Modules

-
    -
  • Modules can be added/removed at runtime using insmod and rmmod
    • -
  • Adding a module calls its init function, and removing it calls its exit function.
    • -
-
$ make load
-sudo insmod hello.ko
-$ dmesg -t | tail -3
-hello: loading out-of-tree module taints kernel.
-hello: module verification failed: signature and/or required key missing - tainting kernel
-Hello, world
-$ make unload
-sudo rmmod hello
-$ dmesg | tail -1
-Goodbye, cruel world
-
-
-
-

Compiling Modules (in-tree)

-
    -
  • Add your code directly to the kernel, and add them to the Makefile(s)
    • -
  • Compile the kernel
    • -
  • Init is called at boot
    • -
-
-
-

The End

-
-
\ No newline at end of file diff --git a/kdlp.underground.software/course/spring2022/index.md b/kdlp.underground.software/course/spring2022/index.md deleted file mode 100644 index 65f9a254..00000000 --- a/kdlp.underground.software/course/spring2022/index.md +++ /dev/null @@ -1,36 +0,0 @@ -### Spring 2022 Course - -This page provides an overview of our course delivered in Spring 2022. -All resources are public or will be published by the end of Summer 2022. -Anything not linked on this page is available on request - -#### Recommended Hardware - -Raspberry Pi 4B - -#### Recommended Books - -[Linux Device Drivers, 3rd Edition (a.k.a. LDD3)](https://lwn.net/Kernel/LDD3/) -[Linux Driver Development with Raspberry Pi - Practical Labs (a.k.a LDDR) -](https://www.amazon.com/Linux-Driver-Development-Raspberry-Practical/dp/B096LPVG1X) - -#### Outline with Links - -|Week|Main Topic|Additional Content| -|---|---|---| -|[0](#week-0)|Linux Crash Course|Kernel patch review livestream| -|[1](#week-1)|Drivers and Modules|TODO| -|[2](#week-2)|Character Drivers|TODO| -|[3](#week-3)|Concurrency and Race Conditions|TODO| -|[4](#week-4)|Working with Hardware|TODO| -|[5](#week-5)|Interrupt Handling|TODO| -|[6](#week-6)|Memory Mapping, DMA, and Data Types|TODO| -|[7](#week-7)|Spring Break|TODO| -|[8](#week-8)|SenseHAT deep dive|TODO| -|[9](#week-9)|Platform devices and devres|TODO| -|[10](#week-10)|Live debugging a broken driver|TODO| -|[11](#week-11)|Color Click lab discussion|TODO| -|[12](#week-12)|Posting patches + oom killer|TODO| -|[13](#week-13)|Intro to GDB|TODO| -|[14](#week-14)|ACPI|TODO| -|[15](#week-15)|Final presentations|Final Presentations| diff --git a/kdlp.underground.software/course/spring2023/assignments/A0.md b/kdlp.underground.software/course/spring2023/assignments/A0.md deleted file mode 100644 index 39952dd6..00000000 --- a/kdlp.underground.software/course/spring2023/assignments/A0.md +++ /dev/null @@ -1,34 +0,0 @@ -### A0 - Git comfortable 🥲 - -Welcome to the class! Setting up a Linux environment is required for the course, use this assignment to get that set up and get your feet wet with the submission process we will use for this class. - -Please introduce yourself and learn how to submit email patches :) - -#### Outcomes: -* Set up your Linux environment for the rest of the class. -* Get familiar with git and git patches. -* Get familiar with how to submit to the mailing list (`git send-email`). -* Get familiar with the peer review process, which we will use for the rest of the course. - -#### Procedure: -* [Set up your VM and Linux environment](vm_instructions.md). -* Install `git` and `git-email`. Run `sudo dnf install -y git git-email`. -* Configure your `.gitconfig`, which lives in `~/.gitconfig`, using a text editor (e.g. `nano ~/.gitconfig` or `vi ~/.gitconfig`). - * You can pick your default editor by adding: 
[core]
-        editor = nano # Or which ever editor you prefer
-
- * You can set your identity information by adding: 
[user]
-        email = youremail@host.tld
-        name = Firstname Lastname
-
- * You can add settings for sending email patches using `git send-email`. If you are planning to use your gmail address, the following link will help you get set up: . Otherwise, you can find similar instructions online. -* Clone the class git repository: . Run `git clone https://github.com/underground-software/KDLP_assignments.git`. Change directory into `A0/` within the class repo. -* Add a file named `firstname_lastname.md` (e.g. `joel_savitz.md`), containing an introduction about yourself. The content can be whatever you want, whether it be why you are taking this class, your favorite ice cream flavor, or some random fact about yourself. -* Run `uname -a >> firstname_lastname.md` to append the details of your OS to your introduction file. -* Make a commit out of your changes. - -#### What to submit: -* Generate a patchset from your commit and cover letter. -* Send the patchset to the mailing list. - -[Submission Guidelines](submission_guidelines.html) diff --git a/kdlp.underground.software/course/spring2023/assignments/A1.md b/kdlp.underground.software/course/spring2023/assignments/A1.md deleted file mode 100644 index d93ce484..00000000 --- a/kdlp.underground.software/course/spring2023/assignments/A1.md +++ /dev/null @@ -1,80 +0,0 @@ -### A1 - Build a Shell 🐚 - -The objective of this assignment is to build a UNIX style shell. - -#### Outcomes: - -* Demonstrate your knowledge of C, specifically working with strings. -* Get an introduction to system calls. -* Make sure the course is a good fit for you. - -#### Procedure: - -* This assignment will guide you through the process of building a shell by iteratively improving your code and adding more features, starting with a very simple program. At the end you should have a shell that mostly works. -* You must make at least one commit per level completed, and label clearly at which commit you think you completed a particular level. -* You must pass all ten levels listed below. If you want to go further for an extra challange, see the list of ideas at the bottom. - -#### What to submit: - -* A patch which adds a directory named firstname_lastname to the assignment repo. - * The first patch should create this directory and add your Makefile. - * Make sure to have compiler warnings enabled (at least -Wall but ideally -Wextra -Wpedantic or even -Weverything if you use clang) and that your code doesn’t have any warnings or errors. -* The subsequent patches should add your shell. Each patch that completes a lvl should indicate that fact clearly. You can use as many patches as you'd like, just be sure to describe the purpose of each patch. Your program should compile after each patch is individually applied (you can use `git commit --amend` to fix mistakes). -* Don't forget a cover letter. - -[Submission Guidelines](submission_guidelines.html) - -#### Levels: - -* **lvl 0**: - * The shell prints a prompt, informing the user about which absolute path they are currently in (see `man 3 getcwd`) followed by a $ and a space (e.g. `/your/current/directory$ `). - * The shell then prints a new line and exits without any user interaction. -* **lvl 1:** - * The shell reads lines of user input, but doesn't need to do anything with them. It just prints a new prompt before each line. - * The shell loops until it gets EOF from user input (ctrl+d). - * The shell exits with code 0. -* **lvl 2:** - * If the user types any text, the shell prints "Unrecognized command" (but does not exit the loop). - * However, if the user just hits enter without typing anything, no error message is printed. -* **lvl 3:** - * The shell splits the line of input into pieces delimited by whitespace characters (see `man 3 isspace`). - * Instead of just printing "Unrecognized commmand" the shell shall include the name of the program in the error message (e.g. if the user types `cat shell.c`, the shell prints "Unrecognized commmand: cat"). -* **lvl 4:** - * the shell supports a few builtin commands (exit, cd and exec). If the first piece of the input is not "exit" or "cd" or "exec" it will still print the unrecognized command message, otherwise: - * exit: takes no arguments (prints error if they are provided) and closes the shell (return value of 0) - * cd: takes exactly one argument (otherwise it prints usage info) and changes the working directory of the shell process (see `man 2 chdir`). If chdir does not accept the path provided, an error message including a description of the errno is printed. The new working directory is reflected in the prompt. - * exec: takes at least one argument (maybe more) and replaces the shell with an instance of the command specified (see `man 3 execv`). If execution fails, an error message is printed including a description of the errno, and the shell continues. Otherwise, having been replaced, the shell does not return. -* **lvl 5:** - * The shell supports running executable files as commands within child processes. If the first piece of the input looks like a path (starts with . or /) a child process is created (see `man 2 fork`) and the command specified by the first argument is executed within the child using the provided arguments (see `man 3 execv`) similar to the exec builtin. If executing the command fails, the child process prints an error message including a description of the errno (don't forget to exit the child process). The shell waits for the child to finish running before printing the next prompt (see `man 2 waitpid`). -* **lvl 6:** - * In the case that the user types something that isn't a path or a builtin, before printing the unrecognized command error, the shell checks whether a file with that name exists in each of the directories listed in the `PATH` environment variable in order (see `man 3 getenv` and `man 2 stat`). If a file, with that name is found, the search can stop and that file is executed with arguments in a child process. Only if no file is found in any of the directories, should the unrecognized command error be printed (NOTE: you must do the path searching manually, and cannot rely on a member of the exec family that does path searching automatically e.g. `execvp`). -* **lvl 7:** - * Before processing the entered commands, the shell performs home directory substitution on the pieces (command name or arguments) that start with a `~`. - * The shell determines a username string by taking a substring of the piece after the `~` until the end of the string or the first `/`, whichever comes first. - * If the username string is empty, then the `~` is replaced with the value of the `HOME` environment variable (see `man 3 getenv`). - * If the username string is not empty, the shell attempts to locate the the user with that username and, if successful, replaces the `~` and the username substring with their home directory (see `man 3 getpwnam`). - * If `getpwnam` does not locate such a user, the shell leaves the piece unmodified. -* **lvl 8:** - * As the shell is processing the commands and arguments, if it finds a `<` or `>`, it skips any whitespace characters and attempts to treat the next piece of the input as a filename for redirection. If there is no filename before the end of the string, an error message is printed. - * A command can have more than one redirection (even more than one of the same type). If there are multiple of the same redirections, then the right-most one takes precedence. - * A redirection beginning with `<` causes the shell to open the corresponding file and replace the stdin of the child process with the file descriptor of the open file (see `man 2 dup`). - * A redirection beginning with `>` behaves similarly, but it replaces the stdout of the process. -* **lvl 9:** - * The shell supports the `|` (pipe) operator to chain multiple commands and their inputs and outputs. - * Each command seperated by a `|` is spawned as its own child process. The shell can handle more than one pipe. - * The shell creates a unidirectional pipe (see `man 2 pipe`) for each `|` and redirects the stdout from the left command to the writing end of the pipe and redirects the stdin of the right command to the reading end of the pipe. - * Any file redirections specified by the user take precedence over any implied redirections of the `|`. - -#### Bonus - -If you were inspired by completing all ten required level and you want to go further (for fun and bragging rights), feel free to add some or all of these features. - -* Add support for `;` to put multiple commands on one line and the `#` for comments. -* Add support for the `&&` and `||` operators to chain commands based on their return codes. -* Add support for special variables like `$?`, `$#`, `$*`, `$$`. -* Add support for reading environment variables using `$NAME` (i.e. that would be replaced with `getenv("NAME")`). -* Add support for modifying environment variables using an `export` builtin. -* Add support for background jobs (`jobs`, `fg`, `bg` builtins), and `&` for spawning a job in the background. -* Add support for the `source` builtin to run commands from a file. -* Add support for control flow operators (e.g. loops, conditionals, selection statements). - diff --git a/kdlp.underground.software/course/spring2023/assignments/A2.md b/kdlp.underground.software/course/spring2023/assignments/A2.md deleted file mode 100644 index bba2b4e4..00000000 --- a/kdlp.underground.software/course/spring2023/assignments/A2.md +++ /dev/null @@ -1,91 +0,0 @@ -### A2 - The Big Upgrade 🎢 - -Build, install, and boot the latest Linux kernel fresh off Linus Torvalds’ tree. - -#### Outcomes: -* Set up a kernel development environment. -* Get comfortable with building the kernel. - -#### Procedure: -* Get access to a Linux system on which you have root permissions. - - -* Install dependencies: - * On Fedora, you can use: - `sudo dnf builddep kernel` - * This roughly equivalent to: - - `sudo dnf -y install gcc flex make bison openssl openssl-devel elfutils-libelf-devel ncurses-devel bc git tar dwarves` - -* Install ccache to speed up subsequent kernel builds: - - `sudo dnf -y install ccache` - - *Be sure to restart your terminal after installing ccache so that it gets added to your path* - -* Clone the latest Linux kernel: . -* Compile the kernel with a custom version tag. - - # clone the kernel tree - - git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git - - # change the current directory to the kernel tree - - cd linux - - # change the current directory to the kernel tree - - cp /boot/config-$(uname -r)* .config - - # set any new config options to the defaults - - make olddefconfig - - # to add label to the new kernel version string, - - # edit ‘Makefile” and add something cool to the ‘EXTRAVERSION’ variable - - # you can print the new version string to test it by: - - make kernelversion - - # build the kernel - - # -j species the number of threads - - # nproc finds the correct number for your machine - - make -j $(nproc) - - # install the modules - - # you need to be root for this step so use sudo - - sudo make modules_install -j $(nproc) - - # install the new kernel - - sudo make install - - # reboot – hopefully the new kernel is set as the default :) - - reboot - - ... - - uname -a - - dmesg | head - - whoami -* Seems easy enough right? 😉 -* Expect the compilation to take a little while Make sure you have 5-10 GB of free space on the partition where you are storing the kernel tree and a bit of extra space on whatever partition /boot resides in to store the new kernel image and initramfs. - -#### What to submit: - -* The output of `uname -a` and `dmesg | head` and `whoami` on the system running your newly compiled Linux kernel. Make a copy of `template.md` in the assignment directory renamed to `firstname_lastname.md` and submit it as a patch. - -* Don’t forget a cover letter. - -[Submission Guidelines](submission_guidelines.html) diff --git a/kdlp.underground.software/course/spring2023/assignments/A3.md b/kdlp.underground.software/course/spring2023/assignments/A3.md deleted file mode 100644 index 31ccbadc..00000000 --- a/kdlp.underground.software/course/spring2023/assignments/A3.md +++ /dev/null @@ -1,56 +0,0 @@ -### A3 - Add a new syscall 🤫 - -Add a new system call to the kernel - -#### Outcomes: - -* Get comfortable with the concept of a syscall from both kernelspace and userspace - * Learn how userspace programs make systemcalls - * Learn how those calls are handled by the kernel -* Understand how C relates to assembly languge - * Write a C program to call your systemcall - * Write the same program in the assembly language of your platform - -#### Procedure: - -1. Using the appropriate SYSCALL_DEFINE\* macro, define a function that behaves as follows: - - a) The function takes two arguments, a userspace pointer to a buffer of type `char *`, an integer specifying the size of the buffer of type `size_t`. - - b) The function copies a string into the userspace buffer containing the student's name and the name of the executable binary that is running. - - c) The function returns the length of the string copied upon sucess. If there is an issue with the size of the provided buffer, checked by proxy of the `size_t` argument, then the function returns `-EFAULT` to the user. - - d) The function should take care to prevent any possibility of a buffer overflow. - -2. Add an `asmlinkage` declaration of your syscall in `include/linux/syscalls.h`. There are numerous examples of other syscalls in that file for ample student inspiration. - -3. Add an entry to the syscall table for your architecture. On x86_64, this is `arch/x86/entry/syscalls/syscall_64.tbl` and on aarch64 this is `include/uapi/asm-generic/unistd.h`. Take inspiration from the other syscalls. - -4. Write an assembly language program that invokes your syscall and prints the string that is placed in the buffer by the kernel to standard output using the write syscall. - -5. Write a C program that does exactly the same thing as the previous step. - -6. Compile and reboot into the newly modified kernel. Make sure your C and assembly language programs work correctly. Include the output in your cover letter. - -The following pages will be of interest to a student: - -* - -Conventions: - - 1. Assembly files that must be pre-processed use the extension \*.S - - 2. Assembly files that are ready to be assembled use the extension \*.s - -Refer to the [Linux kernel documentation](https://www.kernel.org/doc/html/latest/process/adding-syscalls.html) for further guidance. - -#### What to submit: - -* A patch which adds a directory named firstname_lastname to the assignment repo. - * The first patch should create this directory and add your Makefile. - * Make sure to have compiler warnings enabled (at least `-Wall` but ideally `-Wextra` and `-Wpedantic` too, or even `-Weverything` if you use clang) and that your code doesn’t have any warnings or errors. -* The second patch should add your kernel patches *as patches to our repo*. Generate a patch from your commit to the Linux kernel (make sure to use `--base=HEAD^` on git format patch so we can tell what commit your work was based on), and then take this file, put it in your named directory, and add *the email patch itself* to the staged git files and commit this. Then, generate a patch from this commit. This is the second patch of the assignment. -* The third patch adds your assembly program to your named directory. -* The fourth patch adds your C program to your named directory. -* Don't forget a cover letter. diff --git a/kdlp.underground.software/course/spring2023/assignments/A4.md b/kdlp.underground.software/course/spring2023/assignments/A4.md deleted file mode 100644 index 8d207497..00000000 --- a/kdlp.underground.software/course/spring2023/assignments/A4.md +++ /dev/null @@ -1,76 +0,0 @@ -### A4 - Scavenger Hunt 🔎 - -We will assign you a syscall and give you our specially designed program. - -#### Outcomes: - -* Learn basic Linux kernel tracing -* Get a basic understanding of BPF via `bpftrace` -* Strengthen understanding of the interaction between the kernel and userspace - -#### Procedure: - -1. Run `strace` on the provided program - - * Locate the lines corresponding to call(s) to your syscall in the output and put them into assignment.md - -2. Answer question 0 in assignment.md - - * Suggested: make use of the system’s manual pages - -3. Find the syscall definition within the kernel source - - * Put the location into assignment.md - - * Suggested: make use of [elixir](https://elixir.bootlin.com/) or [cscope](http://cscope.sourceforge.net/) - -4. Answer question 1 in the assignment.md file - - * Who last modified your syscall’s code? What were they trying to do? - -5. Write a `bpftrace` snippet to insert a probe that will print the syscall args and kernel stack in real time as the syscall is invoked - - * Ensure that the bpf probe is only triggered when our specific command is run - - * Suggested: [bpftrace/reference_guide.md at master](https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md#5-str-strings) - - * Hint: A script file must be marked as executable to be run - - * Hint: The first line of an executable `bpftrace` file must be "`#!/usr/bin/env bpftrace`" - -6. Re-run the program with the bpf probe inserted via your `bpftrace` code - - * Save the results into assignment.md - -7. Document your process in the cover letter as always - - -#### Tools of interest - -* `strace` - -* `man` - -* `cscope` - -* `git-blame` - -* `git-grep [ -n ]` - -* `bpftrace` and other BPF tooling - -#### What to submit - -* A cover letter following the assignment submission guidelines (as always) - -* A patch which adds a directory named `firstname_lastname` to the A4 folder of the class repo with a copy of `assignment.md`. - -* A patch that includes your changes to `assignment.md` - -* A patch that includes the new entries in log.txt from all the times you ran the program - -* A patch that includes two full logs from strace from running the program. Once as your regular user and once as running as root using `sudo`. Make sure you label which is which - -* A patch that adds your `*.bp` `bpftrace` program - -[Submission Guidelines](submission_guidelines.html) diff --git a/kdlp.underground.software/course/spring2023/assignments/A5.md b/kdlp.underground.software/course/spring2023/assignments/A5.md deleted file mode 100644 index 253182a4..00000000 --- a/kdlp.underground.software/course/spring2023/assignments/A5.md +++ /dev/null @@ -1,26 +0,0 @@ -### A5 - Building Character 💪 - -It't time to write your first driver! - -#### Outcomes: -* Get familiar with which system calls can be performed on a character device file. -* Get familiar with how the behavior of a character device is specified by its `file_operations` in the driver. -* Get familiar with how userspace code can invoke kernel code. - -#### Procedure: -* You will be assigned a random sequence of operations. -* Add the `ioctl` and `llseek` syscalls to your copy of the driver provided in the [assignment repo](https://github.com/underground-software/KDLP_assignments). - * Your implementations should be based on the implementations of the read and write operations. -* Write a userspace program to interact with the character driver from class. - * The userspace program will call the syscalls available through the driver in order of the sequence assigned to you. The userspace program should produce a matching output to what is assigned to you. - * Your program should check for system call errors. - -#### What to submit: -* A patch which adds a directory named `firstname_lastname/` to the assignment repo containing a copy of the provided driver and Makefile. -* The second patch should add the `ioctl` and `llseek` operations to the existing driver. -* The third patch should add your userspace program that implements the assignment and should add to the existing Makefile. - * Make sure to have compiler warnings enabled (at least `-Wall` but ideally `-Wextra` and `-Wpedantic` too, or even `-Weverything` if you use clang) and that your code doesn’t have any warnings or errors. -* The fourth patch should add a file containing the dmesg log from the driver upon a successful run of your program. -* Don't forget the cover letter. - -[Submission Guidelines](submission_guidelines.html) diff --git a/kdlp.underground.software/course/spring2023/assignments/A6.md b/kdlp.underground.software/course/spring2023/assignments/A6.md deleted file mode 100644 index 90f6394c..00000000 --- a/kdlp.underground.software/course/spring2023/assignments/A6.md +++ /dev/null @@ -1,23 +0,0 @@ -### A6 - Implement a spec 🫠 -You will practice developing a character driver and writing specifications and testing code for it. - -You will follow the steps taken in class to develop a specification, tests, and a character driver, and your driver -will behave similarly to what was shown in class, but with a small twist: your character driver should reverse the -string between when it receives it and when the user asks for it back. The lseek and ioctl operations should work identically. - -#### Outcomes: -* Get familiar with how a character driver spec is formatted and how thorough the tests need to be. -* Get comfortable following a spec in the context of writing a character driver. -* Get comfortable with writing simple character drivers. - -#### Procedure: -* Make a new new directory named `firstname_lastname/` within the A6 folder and write your specification in a markdown file. -* Write a test program and a makefile that compiles it. You can load our dummy character driver that just logs operations - while writing your test program so that you have a file you can interact with (obviously though it won't pass the tests). -* Create a character driver adhering to the spec and passing your test program, add support for compiling it to the makefile. - -#### What to submit: -* The first patch should create your directory with the spec inside. -* The second patch should add the test program and Makefile. -* The third patch should add the character driver and modify the Makefile so it can be compiled. -* Don't forget a cover letter. diff --git a/kdlp.underground.software/course/spring2023/assignments/course_application.md b/kdlp.underground.software/course/spring2023/assignments/course_application.md deleted file mode 100644 index b36e97e2..00000000 --- a/kdlp.underground.software/course/spring2023/assignments/course_application.md +++ /dev/null @@ -1,34 +0,0 @@ -## Course Application - -WELCOME APPLICANTS! - -Please thoroughly read and then complete the following instructions: - -Send an email to kdlp-application@googlegroups.com satisfying the following conditions: - -a) The subject of the email contains your full legal name followed by the text "-- [SESSION] Application for Introduction to Linux Kernel Development", where [SESSION] is replaced by the appropriate string, i.e. "Fall" or "Spring" followed by the current year. - -b) If you are enrolled at UMass Lowell, then send this email must be sent from your @student.uml.edu address. - -c) The email should state your current student and/or occupational status, i.e. if you are enrolled in a school, name the school, your course of study, your level of progress through this course of study, and if you are employed, name your employer (or business, if you own one), and your job title. - -d) The email should contain your response to the following question: - - What does the following C language expression evaluate to? - - "KDLP"[4] - - Explain your answer in fewer than 5 complete English sentences. - -e) If you have a resume, KDLP allows that it be attached to this email as a PDF file or text file. No other document in any other format may be attached to this email. - -f) The email must contain your discord username that you will use to complete step 2 - -g) If you prefer to use a different email to send and recieve email patches on our mailing list than the one you are using to send this message, you should specify this address. KDLP will provide instructions on the usage of gmail with our systems, but it is not required that all students use gmail. - -h) Include your time zone. - -i) Include a line certifying the [Developer Certificate of Origin](https://bssw.io/items/the-developer-certificate-of-origin) - -By submitting this email, you acknowledge that you have read and agree to the course policies listed [here](course_policies.html) - diff --git a/kdlp.underground.software/course/spring2023/assignments/fin1.md b/kdlp.underground.software/course/spring2023/assignments/fin1.md deleted file mode 100644 index e1a7efc9..00000000 --- a/kdlp.underground.software/course/spring2023/assignments/fin1.md +++ /dev/null @@ -1,39 +0,0 @@ -### Fin1 - Specification 📐 - -Assignment Summary: You should come up with the specification for a simple but interesting and non-trivial character device that does something entertaining. This is intentionally open ended. - -You will write a detailed specification that someone else could follow to successfully implement the driver and also write a userspace program that implements unit tests that a correct driver should pass to verify its compliance with the specification. - -

- -The specification should document and explain the following system calls: - -- Open - -- Read - -- Write - -- Close - -- At least one of ioctl / lseek - -The test program should exercise the behavior of all of the system calls in the specification to ensure the correctness of the driver. - -However, since you won’t have a completed driver yet, you won’t be able to fully run the test program and have the tests pass. We recommend using strace or another similar program to ensure that your test program is at least making the right system calls with the intended arguments. - -

- -What to submit: - -- A patch to the assignment directory containing your directory named firstname_lastname. - - - The first patch should create this directory and add your specification. - -- The second patch should include your test program and a Makefile to build it.. - - - Make sure to have compiler warnings enabled (at least -Wall but ideally -Wextra -Wpedantic or even -Weverything if you use clang) and that your code doesn’t have any warnings or errors. - -- Don’t forget a cover letter. - -[Submission Guidelines](submission_guidelines.html) diff --git a/kdlp.underground.software/course/spring2023/assignments/fin2.md b/kdlp.underground.software/course/spring2023/assignments/fin2.md deleted file mode 100644 index 512a64f7..00000000 --- a/kdlp.underground.software/course/spring2023/assignments/fin2.md +++ /dev/null @@ -1,23 +0,0 @@ -### Fin2 - Writing some actual kernel code at last 🕴️ - -You will be assigned someone else’s specification and tests. Your task is to implement the specification with a character device module. The existing tests should pass when run against your module. - -If you have a problem with the spec you are given, communicate with its author to work out potential areas of ambiguity, bugs in the test file, or even to add new features that make sense to you. If the spec author is uncooperative, let us know and we can help. We encourage you to have your discussions about each other’s code as responses to the patch emails from the previous assignment. - -

- -What to submit: - -- A patch to the assignment directory containing your directory named firstname_lastname. - - - The first patch should create this directory and add a Makefile to build your driver as well as the test program. - - - Make sure to have compiler warnings enabled (at least -Wall but ideally -Wextra -Wpedantic or even -Weverything if you use clang) and that your code doesn’t have any warnings or errors. - -- The second patch should add your test program. - -- The third patch should add your driver. - -- Don’t forget a cover letter. - -[Submission Guidelines](submission_guidelines.html) diff --git a/kdlp.underground.software/course/spring2023/assignments/final_pres_guide.md b/kdlp.underground.software/course/spring2023/assignments/final_pres_guide.md deleted file mode 100644 index 0702bc31..00000000 --- a/kdlp.underground.software/course/spring2023/assignments/final_pres_guide.md +++ /dev/null @@ -1,72 +0,0 @@ -### Final Presentation Guidelines 🎯 🏹 - -##### Overview - -This is your opportunity to reflect on what you've learned in the course -and give feedback. - -In your presentation you should demonstrate -your understanding of your work, -be able to explain what you did and why it matters to others, -and show your thought process when you are faced with a problem. - -##### Duration - -- Total hard time limit: 20 minutes - - - Main content duration: 10-12 minutes - - - Live coding and Q&A: 8-10 minutes - -##### Guiding questions - -- In your own words, what did you learn in this class? - -- What assignments did you particularly enjoy or not enjoy? Why? - -- What challenges did you face and how did you overcome them? - -- Are there any assignments you think you did an exceptional job on? - -- How does your work in this class fit into the big picture? - -##### Requirements - -- Answer the guiding questions - -- Make an initial submission for Fin2 by midnight the night before your presentation - - - During your presentation you must present this exact version of Fin2 as submitted - - - Like other initial submissions, you will only be graded on the subsequent final submission (due midnight 1 May 2023) - - -- You must have some visual support for your talking points - - - This can be a slide deck or something more creative - -- You must demonstrate your work on Fin2: - - - Show your code - - - Show compilation - - - Show module load - - - Show the tests passing - - - Show an example usage of your driver - - - Show module unload - -- If any of the above are incomplete, discuss what you tried and what you are planning to do next - -- We will give you a small coding challenge based on Fin2 to complete live during your presentation - -##### Recommendations - -- Refer back to your assignment write ups - -- When in doubt, explain your thought process clearly and in your own words - -*Date of publication: 17 April 2023* diff --git a/kdlp.underground.software/course/spring2023/assignments/index.md b/kdlp.underground.software/course/spring2023/assignments/index.md deleted file mode 100644 index 3174f43e..00000000 --- a/kdlp.underground.software/course/spring2023/assignments/index.md +++ /dev/null @@ -1,28 +0,0 @@ -## Pre-requisites -[A(-1)](course_application.md) - -## Assignments - -[A0](A0.md) - -[A1](A1.md) - -[A2](A2.md) - -[A3](A3.md) - -[A4](A4.md) - -[A5](A5.md) - -[A6](A6.md) - -[Fin1](fin1.md) - -[Fin2](fin2.md) - -## Presentations - -[P0](P0.md) - -[P1](P1.md) diff --git a/kdlp.underground.software/course/spring2023/assignments/mid_pres_guide.md b/kdlp.underground.software/course/spring2023/assignments/mid_pres_guide.md deleted file mode 100644 index 90c5bab8..00000000 --- a/kdlp.underground.software/course/spring2023/assignments/mid_pres_guide.md +++ /dev/null @@ -1,36 +0,0 @@ -**KDLP Spring 2023 Midterm Presentation Guidelines** - -* Overview: - * Short pre-recorded presentation about research into a topic of your choice - * Propose topic and outline ahead of time - * Plenty of room to be creative 🙂 -* Example midpoint presentation topics (pick one of these or propose your own): - * What is the OOM killer? - * Why are there multiple versions of some syscalls with numbers at the end? What issue were they added to address? - * pipe, pipe2, pipe3 - * dup, dup2, dup3 - * etc. - * Why do some syscalls have a version ending in `at`? What issue were they added to address? - * open vs openat - * fstat vs fstatat - * rename vs renameat - * symlink vs symlinkat - * etc - * What does .config do? How does it interact with the build process for the kernel? - * What is physical and virtual memory? Why is there a distinction between the two? - * What are all the ways that a process running on a cpu could be interrupted and switched to running code in the kernel? - * What is the history of git? How does it fit into the bigger picture of version control software? - * Or pick your own! -* Select a topic and send us your proposal with a short outline by 23:59 02/27/2023 EST -* Presentation requirements: - * 10 ± 1 minutes long - * Must include some sort of visual component: - * Slideshow - * Diagrams - * Etc. - * Must include some sort of live demo in the terminal - * Must include discussion of some snippet(s) kernel code that is relevant to your topic - * Upload a recording somewhere we can access it (e.g. an unlisted video on youtube and send us a link) - * You can use free software like OBS to record your screen or just make a meeting on zoom or google meet where you can share your screen and record it. - * Submit a link to your presentation by 23:59 03/13/2023 EST - diff --git a/kdlp.underground.software/course/spring2023/assignments/vm_instructions.md b/kdlp.underground.software/course/spring2023/assignments/vm_instructions.md deleted file mode 100644 index 2138c360..00000000 --- a/kdlp.underground.software/course/spring2023/assignments/vm_instructions.md +++ /dev/null @@ -1,72 +0,0 @@ -### How to set up your VM and Linux environment -Start downloading the standard ISO image for Fedora for your architechture (most likely it would be x86_64, unless you have an M1 Mac or you are running Linux on ARM in which case you want the aarch64 ISO): . This may take a while, depending on your connection, so get this started first. - -#### Get the appropriate software -* Windows - [VirtualBox](https://www.virtualbox.org/wiki/Downloads): - * Click the download link for Windows hosts. -* MacOS - [UTM](https://mac.getutm.app/): - * You don't need to buy it from the Apple Store for $10, there is a free download that is identical (press the download button at the above link). -* Linux - [virt-manager](https://virt-manager.org/): - * You can most likely get it from your system's package manager. - -#### Figure out how many resources to give your VM -* Depending on how powerful your machine is, you can give it more or less. - * Generally, leave at least half of the cores and RAM for your host system. - * The more you can give the Vm the better. - * For the class you would need at least 25-30G of hard-disk space to hold the VM. -* Balance it how you want. Feel free to reach out for specific advice, depending on the specs of your machine. - -#### Set up the VM -* Once the ISO image has finished downloading, you can begin the VM set-up process. -* Depending on the software you downloaded, the exact procedure will vary. But you want to create a new machine with the recourses you decided to allocate and you most likely just follow the set-up wizard of your software. Make sure to create a virtual hard drive for the machine (at least 25G is suggested). -* Attach the ISO image to the machine, this will allow it to boot into the Fedora installer. -* Start the VM for the first time and boot into the Fedora image. - -#### Install Fedora to your virtual machine -* Mostly follow the on-screen instructions, but there are some confusing steps. There are numerous tutorials online on how to proceed based on the VM you are using. -* This particular step, depicted below, might be especially confusing. Just click on "Installation Destination" and click "Done", you don't need to change anything you just need to confirm what was already selected by default. - -fedora confusion image - -* Once the installation has finished, Fedora will tell you to reboot. Shut down the machine and remove the ISO image from where you attached it to the VM (once this is done and you confirmed the installation was successful, you can remove the ISO image from your host machine to free up disk space). -* Boot the VM and you should be greeted with a Fedora login prompt, use the username "root" and then enter the root password you selected during installation (it won't look like you are typing the password, but that is just a security measure). -* If you logged in correctly, you should be greeted with a command prompt that looks something like: -``` -[root@localhost ~]# _ -``` -If you see this, you have installed Fedora correctly. Congrats! - -#### Optional steps -* Run a software update: - * Login with your root account. - * Run the command `dnf update`, this may take a while to complete. - * While you are waiting you can switch to a different console and continue following the steps. - * Press `ctrl+alt+f2` (on some keyboards you might need a function key to press f2). - * You can check back on the update any time by pressing `ctrl+alt+f1`. -* Create yourself a non-root account with sudo permissions. - * Login as root. - * Pick a username for your account. - * Run the command `useradd username` where *username* is the username you picked. - * Run the command `usermod -aG wheel username` to give the new user sudo permissions. - * Run `passwd username` and select a password. - * Log out of root (type `exit` or hit `ctrl+d`). - * Try logging in as your new user with the username and password you selected. - * Verify that you have root access via `sudo echo hello`, you should be prompted for a password and when you enter your password the `echo` command should execute. -* Set up SSH access: - * Run the command `ip -br a | grep UP`, read the ip address listed there. - * On your host machine, open a terminal and try running `ssh username@ip-address` where *username* is the username you picked earlier and the *ip-address* is the one you just found. - * You should be prompted for your password, after you enter it you will be able to access your vm from your host machine. - * If you want to obviate the need to enter the password every time you log in, you can set up SSH-keys. - * If you have never set up SSH-keys: - * Run the command `ssh-keygen` in a terminal on your host machine and accept the default values. - * If you already have SSH-keys or you just created them using the step above: - * If you are not on Windows, run `ssh-copy-id username@ip-address` to copy your keys to the VM. - * If you are on Windows, run `type $env:USERPROFILE\.ssh\id_rsa.pub | ssh username@ip-address "cat >> .ssh/authorized_keys"` - * You will be prompted for your password, enter it, and the command should finish. - * Try logging in with `ssh username@ip-address` and you should not need to enter your password again. -* Make the VM headless: - * Check back on the updates from the first step. Wait for them to finish. - * Once the updates are finished, run `shutdown now`. The VM should shut down and exit. - * Open the VM Settings and disable the graphics, the exact settings depends on the VM software you are using. - * Most of the time you will be able to keep the graphics off, since you already set up SSH access. - * This will lighten the load on your computer since it won't have to render the GUI and will make the VM run faster. diff --git a/kdlp.underground.software/course/spring2023/index.md b/kdlp.underground.software/course/spring2023/index.md deleted file mode 100644 index 057ec6b7..00000000 --- a/kdlp.underground.software/course/spring2023/index.md +++ /dev/null @@ -1,54 +0,0 @@ -## Spring 2023 Course - -### Course Summary and Purpose - -We aim to introduce students to the concepts, tools, and framework necessary for independent work on the Linux kernel. - -### Course description - -Linux is the most widely used operating system in the world. The core software component of the Linux operating system is the kernel. A couple of its roles include managing hardware interactions, virtualizing system resources, and enforcing security constraints. In effect, Linux kernel powers almost all of the world’s top supercomputers, android phones, and an innumerable variety of other computers. This course will introduce students to Linux kernel development by focusing on device driver development. This will give students hands-on experience working with internal Linux kernel APIs and provide an overview of some of the core features and components of the kernel. Gaining an understanding of the inner workings of the operating system and how to make changes to it will give students an invaluable perspective on how their computers work behind the scenes that will reveal a new layer of understanding to apply to any future software engineering practice. - -### Course objectives - -Students will: - -* Stretch their C programing skills to the limit - -* Create, develop, and test interesting and creative Linux kernel modules - -* Develop the ability to autonomously participate in the open source software community - -* Learn how to give and recieve feedback on code patches - -* Demonstrate the ability to explain your code and your knowledge of Linux in your own words - -### Required Materials - -1. A Linux environment to which you have root access and ideally physical access. - -2. Patience, determination, and an ability to focus on a frustrating task for sustained periods of time. - -### Course Schedule - -**All assignment due dates are in EST(UTC-5)** - -|#| week starting|S|M|T|W|R|F|S|Initial Submissions due @ 23:59 (Mondays unless stated otherwise)|Reviews due @ 23:59 (Tuesdays unless stated otherwise)|Final Submissions due @ 23:59 (Wednesdays unless stated otherwise)| -|--|--|--|--|--|--|--|--|--|--|--|--| -|0| 15 Jan||||[L0](lectures/L0.md)||||[A0](assignments/A0.md) due Thursday|A0 due Friday|A0 due Saturday| -|1| 22 Jan||[L1](lectures/L1.md)||[L2](lectures/L2.md)||||||| -|2| 29 Jan||[L3](lectures/L3.md)||[L4](lectures/L4.md)||||[A1](assignments/A1.md), [A2](assignments/A2.md)|A1, A2|A1, A2| -|3| 5 Feb||[L5](lectures/L5.md)||[L6](lectures/L6.md)||||||| -|4| 12 Feb||[L7](lectures/L7.md)||[L8](lectures/L8.md)||||[A3](assignments/A3.md)|A3|A3| -|5| 19 Feb|||[L9](lectures/L9.md)|[L10](lectures/L10.md)||||[A4](assignments/A4.md) due Tuesday|A4 due Wednesday|A4 due Thursday| -|6| 26 Feb||[L11](lectures/L11.md)||[L12](lectures/L12.md)||||[Choose Mid Pres Topic](assignments/mid_pres_guide.md)||| -|7| 5 Mar||||||||BREAK|BREAK|BREAK| -|8| 12 Mar||[L13](lectures/L13.md)||[L14](lectures/L14.md)||||[Mid Pres Recording](assignments/mid_pres_guide.md)||| -|9| 19 Mar||[L15](lectures/L15.md)||[L16](lectures/L16.md)||||[A5](assignments/A5.md)|A5|A5| -|10| 26 Mar||[L17](lectures/L17.md)||[L18](lectures/L18.md)||||||| -|11| 2 Apr||[L19](lectures/L19.md)||[L20](lectures/L20.md)||||[A6](assignments/A6.md)|A6|A6| -|12| 9 Apr||[L21](lectures/L21.md)||[L22](lectures/L22.md)||||||| -|13| 16 Apr||[L23](lectures/L23.md)||[L24](lectures/L24.md)||||[Fin1](assignments/fin1.md)|Fin1|Fin1| -|14| 23 Apr||[F0](lectures/F0.md)||[F1](lectures/F1.md)||||[Final Presentation Guidelines](assignments/final_pres_guide.md)||| -|15| 30 Apr||[F2](lectures/F2.md)||||||[Fin2](assignments/fin2.md)|Fin2|Fin2| - -[Course Policies](policies/course_policies.md) | [Submission Guidelines](policies/submission_guidelines.md) | [Peer Review Process](policies/peer_review.md) diff --git a/kdlp.underground.software/course/spring2023/lectures/F0.md b/kdlp.underground.software/course/spring2023/lectures/F0.md deleted file mode 100644 index 2bcc3207..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/F0.md +++ /dev/null @@ -1,10 +0,0 @@ -### 24 April 2023 -#### Presentation day 0 - -*Zeroth Group:* - -|Time|Name| -|--|--| -|5:05-5:25|Alexander Scheufele| -|5:30-5:50|Ian Sodersejrna| -|5:55-6:15|Hannah Guillen| diff --git a/kdlp.underground.software/course/spring2023/lectures/F1.md b/kdlp.underground.software/course/spring2023/lectures/F1.md deleted file mode 100644 index a2641cc2..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/F1.md +++ /dev/null @@ -1,10 +0,0 @@ -### 26 April 2023 -#### Presentation day 1 - -*First Group:* - -|Time|Name| -|--|--| -|5:05-5:25|Christopher Ober| -|5:30-5:50|Thomas Dorval| -|5:55-6:15|Ethan Rosenbaum| diff --git a/kdlp.underground.software/course/spring2023/lectures/F2.md b/kdlp.underground.software/course/spring2023/lectures/F2.md deleted file mode 100644 index 857de7c7..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/F2.md +++ /dev/null @@ -1,15 +0,0 @@ -### 1 May 2023 -#### Presentation day 2 - -*Second Group:* - -|Time|Name| -|--|--| -|3:30-3:50|Michael Burke| -|3:55-4:15|Tony Choma| -|4:20-4:40|Raul Olivares| -|4:45-5:05|Sahil Patel| -|5:10-5:30|Martin Nguyen| -|5:35-5:55|Vladmir Ventura| -|6:00-6:20|Seth Galasso| -|6:25-6:45|Preston Wilson| diff --git a/kdlp.underground.software/course/spring2023/lectures/L0.md b/kdlp.underground.software/course/spring2023/lectures/L0.md deleted file mode 100644 index c810f5ad..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L0.md +++ /dev/null @@ -1,72 +0,0 @@ -**L0 - 18 January 2023** - -[Lecture Recording](https://youtu.be/43FekcQlmN4) - -Notes on first class: - -* Do this right now: - * To get in discord server -* Online Syllabus: -* This is not like other classes!!! - * The course attempts to encapsulate the ethos of open source development in a classroom environment - * Introduce the tools, techniques and etiquette of submitting code to the kernel and provide practice with them - * Practical C coding, and plenty of it! - * Practical research -* This means: - * Public submissions - * Peer review - * Emphasis on presenting your work and explaining the process vs just getting the right answer -* Why should you care? Why is the grind worth it? - * This class will let you practice key skills for your career in a lower stakes environment: - * Practice communication skills that are absolutely invaluable in the tech industry - * Wisdom from people who already have careers in CS - * Experience with C programming on a large codebase - * Experience with git and other linux utilities - * And, if you are considering a career in kernel development: - * Instructors are the real deal - * Guest lectures from senior developers at Red Hat - * Linux Foundation Certificate - * Familiar with kernel code base / functions - * Familiar with mailing list workflow -* Requirements to be able to complete the class successfully (do, or have done this by add drop or you should drop the class): - * A Linux environment that you have root access to - * The following are not adequate: - * The UML CS servers (you do not have root access) - * WSL (need to be able to compile custom kernel) - * Recommended: - * Run Linux in a virtual machine - * Possible, but not recommended: - * Dual boot - * Ideal, but expensive (don’t go buying hardware for this class) - * Second machine - * Even if you use linux as a daily driver, you might still consider a VM or dual booting to protect your main install (e.g. from downtime or data loss in case you mess something up since we are working on the kernel) - * Setup VM ASAP! - * If you are on Windows or an intel (x64) mac use Virtual box - * If you are on linux, use QEMU - * If you are on an M1 (aarch64) mac use UTM - * We recommend installing [Fedora Linux](https://getfedora.org/en/server/download/) since it is a fully open source distro that upstream pretty closely - * Download the iso file for the right architecture (x86_64 for windows and intel mac, aarch64 for M1 mac) - * Setup vm with appropriate amount of resources - * Ensure you can connect to it using ssh from your host machine - * Shut it down and set it to headless mode (no window). Start and stop as needed for your course work -* Go over course policies: -* Introductory assignments - * Fill out the form on the website. A(-1), due end of day today! - * Git and git patches. A0, due tomorrow! - * Setting up git send email, A0 - * Shell. A1 due 11:59pm Jan 29 - * Compiling kernel. A2 due 11:59pm Jan 29 - * We will provide as much help and guidance as we can - * The lectures until Jan 30 will be reserved for questions about the assignments. If there are no questions, we will proceed with the scheduled lecture material. - * If you can’t handle the heat, get out of the kitchen… -* In conclusion - * This class is relatively frontloaded! - * If you can make it past A2, you will probably be fine. - * NO EXAMS! 😀 -* Do this right now if you have not - * [COURSE APPLICATION](https://kdlp.underground.software/course_application.html) - * Send to kdlp-application@googlegroups.com -* Mailing lists for A0: - * For UML students: kdlp-spring23-uml@googlegroups.com - * For LFX mentees: kdlp-spring23-lfx@googlegroups.com - diff --git a/kdlp.underground.software/course/spring2023/lectures/L1.md b/kdlp.underground.software/course/spring2023/lectures/L1.md deleted file mode 100644 index 92094a49..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L1.md +++ /dev/null @@ -1,19 +0,0 @@ -**L1 - 23 January 2023** - -[Lecture Recording](https://youtu.be/ywL066R6cOU) - -* Git lecture - * Demos start a bit after 1 hr in - * First charlie makes mistakes and fixes them - * [Slides](https://kdlp.underground.software/git.html) - -* Discussion about mutt starts at about 1:15 -* Example muttrc for gmail: - * -* Needed for gmail oauth - * -* For quick patch application in mutt: - * - -Some brief exploration of kernel init code afterwards - diff --git a/kdlp.underground.software/course/spring2023/lectures/L10.md b/kdlp.underground.software/course/spring2023/lectures/L10.md deleted file mode 100644 index 2c468e66..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L10.md +++ /dev/null @@ -1,9 +0,0 @@ -**L10 - 22 February 2023** - -[Lecture Recording](https://youtu.be/891uukIWyG0) - -Class Notes: - -* Writing modules: -* Makefile details -* `insmod`, `modprobe`, `rmmod` diff --git a/kdlp.underground.software/course/spring2023/lectures/L11.md b/kdlp.underground.software/course/spring2023/lectures/L11.md deleted file mode 100644 index a1caf740..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L11.md +++ /dev/null @@ -1,9 +0,0 @@ -**L11 - 27 February 2023** - -[Lecture Recording](https://youtu.be/qK0jzmyhdts) - -Class Notes: - -* Char dev demo -* Misc devices, major and minor numbers -* diff --git a/kdlp.underground.software/course/spring2023/lectures/L12.md b/kdlp.underground.software/course/spring2023/lectures/L12.md deleted file mode 100644 index 826cfe1e..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L12.md +++ /dev/null @@ -1,7 +0,0 @@ -**L12 - 01 March 2023** - -[Lecture Recording](https://youtu.be/XfnGgOwcuHA) - -Class Notes: - -* Continued with char driver demo diff --git a/kdlp.underground.software/course/spring2023/lectures/L13.md b/kdlp.underground.software/course/spring2023/lectures/L13.md deleted file mode 100644 index e01aa75f..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L13.md +++ /dev/null @@ -1,8 +0,0 @@ -**L13 - March 13 2023** - -[Lecture Recording](https://youtu.be/as7Iwq8GKj4) - -Class Notes: - -* More char dev demo -* File operations and beyond diff --git a/kdlp.underground.software/course/spring2023/lectures/L14.md b/kdlp.underground.software/course/spring2023/lectures/L14.md deleted file mode 100644 index a596bce4..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L14.md +++ /dev/null @@ -1,8 +0,0 @@ -**L14 - March 15 2023** - -[Lecture Recording](https://youtu.be/pOQ1Ad__WHs) - -Class Notes: - -* Get into details of A5 -* More char dev demo diff --git a/kdlp.underground.software/course/spring2023/lectures/L15.md b/kdlp.underground.software/course/spring2023/lectures/L15.md deleted file mode 100644 index e4ad11c9..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L15.md +++ /dev/null @@ -1,3 +0,0 @@ -**L15 - March 20, 2023** - -[Lecture Recording](https://youtu.be/go4L-xDhQ78) diff --git a/kdlp.underground.software/course/spring2023/lectures/L16.md b/kdlp.underground.software/course/spring2023/lectures/L16.md deleted file mode 100644 index f23e604a..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L16.md +++ /dev/null @@ -1,3 +0,0 @@ -**L16 - March 22, 2023** - -[Lecture Recording](https://youtu.be/sq7KhGaiAXQ) diff --git a/kdlp.underground.software/course/spring2023/lectures/L17.md b/kdlp.underground.software/course/spring2023/lectures/L17.md deleted file mode 100644 index d7299887..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L17.md +++ /dev/null @@ -1,9 +0,0 @@ -**L17 - March 27 2023** - -[Lecture Recording](https://youtu.be/nueSdUaSkIA) - -Class Notes: - -* Writing a char driver specification -* Writing a char driver test program based on spec -* Writing a char driver based on spec and test program diff --git a/kdlp.underground.software/course/spring2023/lectures/L18.md b/kdlp.underground.software/course/spring2023/lectures/L18.md deleted file mode 100644 index 5fde43bb..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L18.md +++ /dev/null @@ -1,9 +0,0 @@ -**L18 - March 29, 2023** - -[Lecture Re-Recording](https://youtu.be/zeLhUxmUsSk) - -Class notes: - -* This is a re-recording of the lecture, due to gmeet technical difficulties. -* The lecture goes over the end of the echo driver. -* We started a driver debugging excersize that we mention in the recording, the continuation will be on Monday's lecture. diff --git a/kdlp.underground.software/course/spring2023/lectures/L19.md b/kdlp.underground.software/course/spring2023/lectures/L19.md deleted file mode 100644 index 07a7e6f0..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L19.md +++ /dev/null @@ -1,8 +0,0 @@ -**L19 - April 03, 2023** - -[Lecture Recording](https://youtu.be/ePnwp5YGRAg) - -Class Notes: - -* Talk about the rest of the course, final presentations, and last two assignments. -* Finished the debugging excersize. diff --git a/kdlp.underground.software/course/spring2023/lectures/L2.md b/kdlp.underground.software/course/spring2023/lectures/L2.md deleted file mode 100644 index 33667d57..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L2.md +++ /dev/null @@ -1,13 +0,0 @@ -**L2 - 25 January 2023** - -[Lecture Recording](https://youtu.be/gtBNVjaN8bo) - -Class Notes: - -* Emphasize A1 & A2: get started ASAP. - * Explained the assignments more with some examples. -* De-weighing of A0 and emphasis on following submission guidelines. This is a one time exception. -* For A2: - * You can clone the kernel tree using the `--depth=1` option to make it quicker, but you will need to unshallow the tree for A4. -* Forgot to mention: - * UML students: Sign up on LFX! Maybe wait till add/drop date passes. diff --git a/kdlp.underground.software/course/spring2023/lectures/L20.md b/kdlp.underground.software/course/spring2023/lectures/L20.md deleted file mode 100644 index ab315027..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L20.md +++ /dev/null @@ -1,12 +0,0 @@ -**L20 - April 05, 2023** - -[Lecture Recording](https://youtu.be/EuSFOSFG97M) - -Class Notes: - -* Discuss final presentation schedule -* Misc device demo -* 1 month after last day of classes: ceasefire on any ways ppl circumvented challenges -* L24 may be guest lecture -* Misc devices -* hpet: diff --git a/kdlp.underground.software/course/spring2023/lectures/L21.md b/kdlp.underground.software/course/spring2023/lectures/L21.md deleted file mode 100644 index e8743e87..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L21.md +++ /dev/null @@ -1,11 +0,0 @@ -**L21 - April 10, 2023** - -[Lecture Recording](https://youtu.be/-3UhHV_Jts8) - -Class Notes: - -* Announce Wander for 21 April/L24/last day of class - * About memory barriers probably -* No peer review for Fin2 -* Spec due next Monday even though no class -* (starts around 5:18) Come up with spec for battleship game diff --git a/kdlp.underground.software/course/spring2023/lectures/L22.md b/kdlp.underground.software/course/spring2023/lectures/L22.md deleted file mode 100644 index 2c439a4a..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L22.md +++ /dev/null @@ -1,7 +0,0 @@ -**L22 - April 12, 2023** - -[Lecture Recording](https://youtu.be/VMcGlU2YXyk) - -Class Notes: - -* Explore the OOM killer in the kernel diff --git a/kdlp.underground.software/course/spring2023/lectures/L23.md b/kdlp.underground.software/course/spring2023/lectures/L23.md deleted file mode 100644 index 68c35c7e..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L23.md +++ /dev/null @@ -1,8 +0,0 @@ -**L23 - April 19, 2023** - -[Lecture Recording](https://youtu.be/Jc58sQE94Js) - -Class Notes: - -* Continued with the Battleship driver -* Last Charlie lecture :( diff --git a/kdlp.underground.software/course/spring2023/lectures/L24.md b/kdlp.underground.software/course/spring2023/lectures/L24.md deleted file mode 100644 index 592e05ee..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L24.md +++ /dev/null @@ -1,7 +0,0 @@ -**L24 - April 21, 2023** - -[Lecture Recording](https://youtu.be/m7c98vJ0gho) - -Class Notes: - -* Wander Costa presents on memory barriers diff --git a/kdlp.underground.software/course/spring2023/lectures/L3.md b/kdlp.underground.software/course/spring2023/lectures/L3.md deleted file mode 100644 index 6b2c5005..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L3.md +++ /dev/null @@ -1,26 +0,0 @@ -**L3 - 30 January 2023** - -[Lecture Recording](https://youtu.be/lttlHnINh4k) - -Class Notes: - -* Announcements: - * Today is the last day to drop the course - * If you haven’t started A1/A2 you should drop - * Let us know ASAP if you are dropping - * Brief: UML students: - * Sign up on LFX: - * - * - * - * - * Clarify due dates for first, reviews, and final - * A2: Copy template.md to firstname_lastname.md (Don’t delete it) -* Open floor to questions -* Show correct/incorrect A0 submissions -* Starting about 25 minutes in: go over git rebase -* Welcome new mentor: Erastus C. Toe -* `git rebase` examples by Charlie, see recording after ~00:30:00 -* Maybe linux tips: - -After class: send out announcement of changes to assignment diff --git a/kdlp.underground.software/course/spring2023/lectures/L4.md b/kdlp.underground.software/course/spring2023/lectures/L4.md deleted file mode 100644 index 550a7e06..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L4.md +++ /dev/null @@ -1,23 +0,0 @@ -**L4 - 1 February 2023** - -[Lecture Recording](https://youtu.be/92KsB0Rmrsw) - -Class Notes: - -* Reminder to do proper code review -* Explaining A3 -* Explaining executables ~00:18:00 -* Reference to LSB in `file` command: -* Showing `xxd` and `objdump` ~00:23:00 -* Showing assembly of executable -* Showing `ldd` ~00:25:00 -* Showing how to write assembly of a simple C program ~00:29:00 -* Exit syscall ~00:40:00 -* Syscall table: -* Adding functionality to assembly program ~00:51:00 -* Due date for A3 will be moved to a later date -* Adding write syscall to assembly ~01:04:00 -* Linux syscall calling conventions: -* Reference to kernel code: -* Descriptions of syscall instruction: -* Syscall table in the x86_64 kernel: diff --git a/kdlp.underground.software/course/spring2023/lectures/L5.md b/kdlp.underground.software/course/spring2023/lectures/L5.md deleted file mode 100644 index e2f9ef65..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L5.md +++ /dev/null @@ -1,19 +0,0 @@ -**L5 - 6 February 2023** - -[Lecture Recording](https://youtu.be/X9lzh4oUw20) - -Apply on [LFX](https://mentorship.lfx.linuxfoundation.org/project/958fe36a-d763-4422-81af-c5ecf2465957), if you have not yet. - -Class Notes: - -* A3 demo continued -* get_task_comm src -* -* Showed “Hello World” assembly example on aarch64 -* Showed C equivalent ~00:30:00 -* Pop quiz: - * Is `sizeof` a function or an operator? - * What is the type of `prompt`? (declared `static char prompt[] = “What is your name? “`) -* Showed strace ~00:48:00 -* Showed how to define syscall ~01:31:00 - diff --git a/kdlp.underground.software/course/spring2023/lectures/L6.md b/kdlp.underground.software/course/spring2023/lectures/L6.md deleted file mode 100644 index 8d2d4123..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L6.md +++ /dev/null @@ -1,8 +0,0 @@ -**L6 - 8 February 2023** - -[Lecture Recording](https://youtu.be/PvJzf78Gbxs) - -* Office hours -* ~00:10:00 Charle starts (final) A3 demo -* -* (the demo continues until the end of the recording) diff --git a/kdlp.underground.software/course/spring2023/lectures/L7.md b/kdlp.underground.software/course/spring2023/lectures/L7.md deleted file mode 100644 index 85df3873..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L7.md +++ /dev/null @@ -1,24 +0,0 @@ -**L7 - 13 February 2023** - -[Lecture Recording](https://youtu.be/ezgfAnD4oXM) - -Class Notes: - -* Shim demo - * [How to get process ID of background process? - Stack Overflow](https://stackoverflow.com/questions/1908610/how-to-get-process-id-of-background-process) -* Bpftrace and tracing demo - * -* Syscalls we are assigning for A4 (Note: you should have already been assigned to a syscall and a cohort. Let us know if that is not the case): - * execve - * openat - * fstat - * mmap - * close - * getuid - * munmap - * getrandom - * gettimeofday - * getpid - * exit -* Last process ran (`$!`): - diff --git a/kdlp.underground.software/course/spring2023/lectures/L8.md b/kdlp.underground.software/course/spring2023/lectures/L8.md deleted file mode 100644 index a001d534..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L8.md +++ /dev/null @@ -1,22 +0,0 @@ -**L8 - 15 February 2023** - -[Lecture Recording](https://youtu.be/frONY_C97lg) - -Class Notes: - -* Question about A4 -* Final questions about A3, final sub tonight -* Reflection on A1 - * Why did it go wrong? - * Is it too hard? - * Feedback for students: - * Read directions and plan ahead - * Start earlier - * Feedback for us? -* Feedback on pacing - * MWF cadence? - * When will you do the next assignment? Do we really want to drag it out? - * Peer review and final sub should be easy if you did a good job on initial submission -* Show base option on format patch -* Slides: - * [Linux Tips](https://kdlp.underground.software/linux_tips_of_great_power.html) diff --git a/kdlp.underground.software/course/spring2023/lectures/L9.md b/kdlp.underground.software/course/spring2023/lectures/L9.md deleted file mode 100644 index f4a18d61..00000000 --- a/kdlp.underground.software/course/spring2023/lectures/L9.md +++ /dev/null @@ -1,20 +0,0 @@ -**L9 - 21 February 2023** - -[Lecture Recording](https://youtu.be/vpIIkSaBgo8) - -Class Notes: - -* Mention change to A4 specs - * Add your own folder - * Make sure to git pull latest version of binary - * Make sure to git pull --unshallow kernel repo -* Questions on A4 -* Introduce Midpoint presentation - * Short pre-recorded presentation about research into a topic of your choice - * Topic and outline due next monday - * Actual recording due when we get back from spring break -* Show Midpoint guidelines - * -* Modules and drivers + writing drivers, if we have time - * - * diff --git a/kdlp.underground.software/course/spring2023/policies/course_policies.md b/kdlp.underground.software/course/spring2023/policies/course_policies.md deleted file mode 100644 index 681537bb..00000000 --- a/kdlp.underground.software/course/spring2023/policies/course_policies.md +++ /dev/null @@ -1,114 +0,0 @@ -## Course Policies - -**Assignment Procedures:** - -* Coding Assignments: - * Assignments must be submitted in the format of plaintext unless explicitly specified. - * Assignments must be submitted in the form of email code patches, and furthermore: - * Submissions must conform to all Linux kernel coding standards that are sane in a particular context. - * Submissions must not contain whitespace errors - * The instructors will provide detailed guidelines for automated checking of these requirements, and will gladly explain them upon request to any interested student - * Therefore the instructors reserve the right to refuse to grade any submission that does not conform to these standards. - * Students will participate in peer review for these assignments, as explained in the section below. - * Any code submission that fails to compile with zero warnings and zero errors will receive a zero. - -* Presentation Assignments: - * Students will be required to complete two (2) presentations throughout the course - * Students must deliver their presentations over a live video call - * Presentations must include original visual content, such as slides - * Students are required to include live demonstrations of their code by sharing their text editor and/or terminal during the presentation - -* Instructors reserve the right to change due dates within reason - * In such a case, we will provide sufficient advance notice to students - -**Peer Review Procedure (only for coding assignments)** - -* Students must participate in the class mailing list for submissions by giving other students feedback on their work. -* This is a part of each assignment grade and the tripartite scheme is as follows: - * The review process spans 2 days. - * Part 1: Initial submission - * The student makes their submission to the mailing list using git-send-email. - * This submission is due on the listed assignment "due date", see the [course schedule](course_spring2023.html). - * If the initial submission is late, the student will get a zero on the entire assignment. - * Part 2: Peer review - * A student is given two other student names (call them student A and student B). - * The student is assigned to review student A's and student B's submissions. - * If the student approves of a submission, then the student will reply to the approved email with what we call an "ack". - * An "ack" consists of a single line containing the following: Acked-by: Firstname Lastname . - * If the student sees problems with a submission, the student will reply to the problematic email with their feedback. - * In parallel, other students have been assigned the student's submission and the student should recieve feedback from two other students. - * These reviews are due one day past the initial submission, a late or no submission results in a zero for the review section of the assignment. - * Reviews are graded based on how many issues a student missed. - * The student receives 20% off for each unique issue not spotted with max penalty of 100%. - * Part 3: Final submission - * The student, if canny, will act on the feedback from the received reviews. - * Regardless of whether the student made changes to their initial submission, they must make a final submission. - * This is due two days after the initial submission, and due one day after receiving the reviews. - * Overall assignment grading scheme (percentages are relative to the total assignment grade): - * The initial submission is purely for the peer-review process. It is crucial that the initial submission is made on time, otherwise, as stated above, the student will recieve a zero on the entire assignment. - * Review 1: 10% - * Review 2: 10% - * Final Submission: 80% - -**Late Work Policy:** - -* There is no late work! - * Students receive a zero for assignments not submitted on time. -* One exception (a once per semester get out of jail free card): - * Students may choose to opt out of the peer review process. - * By bypassing the ack requirement (both giving and receiving) students receive the two days that would be used for that process as an extension before they make their first and only final submission. - * This means that you do you not get a chance to receive early feedback from your peers and so you get what you get for a final grade. - -**Mentorship Graduation:** - -* Students and mentees will be given the status of “Graduated” if we determine that their performance has been suitable to the degree that we would recommend as a strong candidate for an internship doing basic Linux kernel engineering work. - -
- -**Additional policies specific to LFX mentees:** - - -* Removal from the course - * In the case that a student fails to complete a task by a deadline with no prior notice, an instructor will reach out to you via email and/or direct message. If the student fails to respond to this message within seven (7) days of it’s receipt, the instructors reserve the right to remove the student from the program. - * Attendance is not required (because of timezones) but students should watch the recorded videos of class sessions -
- -
- -**Additional policies specific to UMass Lowell students:** - - -* Automatic Course Failure (Grade of F) - * In the case that a student fails to complete a task by a deadline with no prior notice, an instructor will reach out to you via email and/or direct message. If the student fails to respond to this message within seven (7) days of it’s receipt, the instructors reserve the right to give the student a failing grade (F) for the course. - * Attendance is... required. - -**UMass Lowell Grading Policy:** - -Student grades will be weighted according to the following scheme: - -| Category | Percentage | -|--|--| -| Assignments | 40% | -| Presentation 1 | 25% | -| Presentation 2 | 30% | -| Participation | 5% | - - -Students will be given a letter grade according to the following this scheme: - -|Letter grade|Percentage range| -|--|--| -|A |90 ~ 100| -|A-| 85 ~ 89.99| -|B+| 80 ~ 84.99| -|B |75 ~ 79.99| -|B-| 70 ~ 74.99| -|C+| 65 ~ 69.99| -|C |60 ~ 64.99| -|C-| 55 ~ 59.99| -|D+| 50 ~ 54.99| -|D |40 ~ 49.99| -|F |below 40| - -######A (4.0), A- (3.7), B+ (3.3), B (3.0), B- (2.7), C+ (2.3), C (2.0), C- (1.7), D+ (1.3), D (1.0), F (0.0) -
diff --git a/kdlp.underground.software/course/spring2023/policies/index.md b/kdlp.underground.software/course/spring2023/policies/index.md deleted file mode 100644 index 9904b79e..00000000 --- a/kdlp.underground.software/course/spring2023/policies/index.md +++ /dev/null @@ -1,10 +0,0 @@ - -## Latest Policy Documents - -[Course Policies](course_policies.md) - -[Submission Guidelines](submission_guidelines.md) - -[Peer Review Process](peer_review.md) - -[Presentation Guidelines](presentation_guidelines.md) diff --git a/kdlp.underground.software/course/spring2023/policies/peer_review.md b/kdlp.underground.software/course/spring2023/policies/peer_review.md deleted file mode 100644 index d453a05c..00000000 --- a/kdlp.underground.software/course/spring2023/policies/peer_review.md +++ /dev/null @@ -1,15 +0,0 @@ -### How to submit a code review - -* You will be assigned two students to peer review shortly after you complete your own initial submission. -* For each student, which you are assigned, you will do the following: - * Locate their submissions and download all of the patches in the patchset (besides the cover letter). - * Make a new branch off of master/main and switch over to that branch to apply the patches. - * Make sure each patch applies cleanly IN ORDER (a.k.a no corrupt patches, whitespace errors, etc.) - * You can run checkpatch.pl on the patches, ignore any errors pertaining to maintainers. - * Make sure the code compiles without warnings or errors. - * Make sure the program runs without crashing. - * Make sure that the output is correct (passes all tests, if applicable). - * If there are any problems with the submission, document said issues and report them in your email reply to the student who you are reviewing. - * If you determine that there are no issues with the submission, inform the recipient. - * If it turns out that there were issues with the submission that you missed, points will be deducted from your overall assignment grade. - * Refer [here](course_policies.md) for the grading policy regarding peer reviewing. diff --git a/kdlp.underground.software/course/spring2023/policies/presentation_guidelines.md b/kdlp.underground.software/course/spring2023/policies/presentation_guidelines.md deleted file mode 100644 index c91ba47d..00000000 --- a/kdlp.underground.software/course/spring2023/policies/presentation_guidelines.md +++ /dev/null @@ -1,43 +0,0 @@ -### Presentation Guidelines 🎯 🏹 - -Duration: Tentative till further notice. Midterm will be a bit shorter, final a bit longer, neither longer than 20m - -

- -Content: - -What have you learned so far by doing the assignments? - -- How did the assignments relate to what we learned in class in your own words - -What challenges have you faced and overcome working on these tasks? - -- Your assignment write ups should be useful here - -Assignments you think you did an exceptional job on - -- Why do you think so? Maybe demonstrate your competence/understanding - -What assignments did you enjoy or not enjoy and any feedback - -

- -Suggestions on what to prepare: - -- Visual demonstrations if you think they are helpful - -Live example of how your project functions - -- Could do something in the terminal live - -Maybe show your program’s inner workings - -- Explain your thought process - -

- -The presentation should demonstrate that you understand your work, you are able to explain what you did and why it matters to others, and you are able to show your thought process when you are faced with a problem. - -An example of a previous presentation can be found here. - -These are general guidelines for what we are looking for in a presentation. You do not need to follow it strictly but you should try to cover all of the areas at least somewhat. diff --git a/kdlp.underground.software/course/spring2023/policies/submission_guidelines.md b/kdlp.underground.software/course/spring2023/policies/submission_guidelines.md deleted file mode 100644 index e22e7c65..00000000 --- a/kdlp.underground.software/course/spring2023/policies/submission_guidelines.md +++ /dev/null @@ -1,122 +0,0 @@ -### Assignment Submission Guidelines - -All homework submissions must be submitted in the form of an email patchset [0] generated by git format-patch [1] from commits [2] made in your local copy of [this repository](http://prod-01.kdlp.underground.software/cgit/KDLP_assignments.git/) with a cover letter [3] describing your work, and sent by git send-email [4] to our mailing list. - -As part of the peer-review process for coding assignments in this class (described [here](course_policies.md)) each assignment will require you to submit each patchset at least twice [5]. - -Try to start your assignments early. If you run into issues and get stuck it gives you time to ask questions and get help before the due date so you can submit something on time and get credit for the assignment, and if you finish early you can submit and potentially get feedback from us or other students that you can incorporate into a resubmission before the deadline in accordance with the resubmission policy [6]. - -#### [0] Patchset Guidelines - -The specific assignment document will specify which files to edit, and how many commits you should be making which will inform the overall structure of your patchset, but every patchset in this class must follow these general guidelines: - -* Each commit gets its own patch with a title and body - -* The patch series is introduced with one additional patch, the cover letter - -Fortunately, git format-patch can generate the appropriate files for you automatically: - -$ git format-patch -3 --cover-letter - -$ # generates patches from the three most recent commits (change -3 to whatever you need for the amount of commits you have) and a cover letter template. - -All of the patches must follow the patch guidelines [1]. Those generated from a commit should follow the commit guidelines [2] and the cover letter must follow the cover letter guidelines [3]. - -You will recieve an automatic zero on the assignment if any of the patches in your patchset are corrupt. This shouldn't be possible if you generate your patches with `git format-patch`, but if you edit the files manually they might get corrupted. You have been warned! The correct way to edit the patches is to edit the underlying commits (see `man git-rebase` and the `--amend` option from `man git-commit`) and then regenerate the patches. - -#### [1] Patch Guidelines - -Every patch in the patch series (including the cover letter) must end with a “Signed-off-by” line, called the DCO (Developer Certificate of Origin). The line must exactly match this format: - -Signed-off-by: Firstname Lastname - -The DCO line must be the final line of the email body right before the start of the patch diff. - -Fortunately, you can make git add this line automatically for you when you author a commit: - -$ git commit -s - -$ # include a DCO Signed-off-by line in the commit message automatically - -You will need to remember to add your DCO to the cover letter manually. - -You should use the checkpatch.pl script in the scripts/ directory of Linus’ kernel tree to make sure your patch is as close to upstream Linux kernel patch standards as possible. - -#### [2] Commit Guidelines - -Within the repository for this class there is a directory for each assignment (e.g. for A0, look in the A0 directory, for Mid 1, look in Mid1, etc.) and within that directory there should be everything you need to get started after you have read the specific document for the assignment you are working on. - -You will make the necessary changes or additions and turn them into commits using git commit. - -When you author a commit the first line(s) you type into your editor will become the title, and by hitting enter twice and leaving a blank line the subsequent text will become the full commit message. The git format-patch utility will automatically put the title and message of a commit into the title and body respectively of the corresponding patch email it generates. - -Your commits should have a title that is a short summary of the changes in this commit and you should include any further details in the commit message. - -You should make sure that the changes you are including in your commits are tidy. This means that code should follow the [kernel code style guidelines](https://www.kernel.org/doc/html/latest/process/coding-style.html), (tabs for indentation, tab width of 8, no lines exceeding 80 columns, etc). - -You must also avoid whitespace errors. These include whitespace at the end of a line, lines with only whitespace on them, extra blank lines at the end of a file, forgetting the newline on the last line of the file, etc. A good editor will highlight and/or automatically fix these for you, but git will also detect these when formatting and applying patches. - -When you format a patch, if you forgot the newline at the end of the file git will put this line at the end of the diff \ No newline at end of file. If you see this, you should adjust the file contents and fix your patch. You can check for the other whitespace errors by running git am to try and apply your patch. If git am prints a warning like this when you apply the patch: warning: 2 lines add whitespace errors. You should adjust the indicated lines and fix your patch. - -Your patches also must apply cleanly to HEAD commit on the master branch of the upstream repository. You can verify this by checking out that branch and trying to apply your patches. We should NOT need to apply your previous versions of the patch in order for the latest version of your patchset to apply. - -Sample workflow to check that your patchset applies cleanly: - -* Generate your patches and put them in a known location and take note of the filenames -* Make sure your git tree is up to date. You should do this each time you begin work within any git repository. - * Use `git remote update` to update all of your local copies of remote trees. -* Create and checkout a local branch based on the upstream `origin/master` branch by using: - * `git checkout -b origin/master` (branch name can be anything convenient) -* Apply your patchset to this branch using `git am ... ` -* If there are no errors that appear, congratulations, your patchset applies cleanly! - * If there are whitespace errors or corrupt patches, revise as needed by amending or rebasing your commits. - -#### [3] Cover Letter Guidelines - -When you open the cover letter file generated by git format-patch in your editor, it will contain a summary of all the changes made in the subsequent patches at the bottom and two filler lines indicating where you can add the title (\*\*\* SUBJECT HERE \*\*\*) and message (\*\*\* BLURB HERE \*\*\*) to the email. - -* You must replace the subject filler With your name and the assignment title. For example Charles Mirabile - Assignment 3 - -* You must replace the blurb filler with your writeup - -* Failure to remove these filler lines including the asterisks will result in lost points - -Your cover letter writeup should include a short write-up that specifies the following: - -* What you think your degree of success with this assignment is (from 0% to 100%) - -* A brief discussion of your approach to the assignment - -* A detailed description of any problems that you were not able to resolve before submitting. - -Failure to specifically provide this information will result in a 0 grade on your assignment. Further, If you do not disclose problems in your write-up and problems are discovered when your work is reviewed, you will receive a grade of 0. Note: These writeup guidelines are adapted from those of [Prof. Bill Moloney](https://cs.uml.edu/~bill/) - -#### [4] Submitting to the Mailing List - -Your patches should be sent to the mailing list for your cohort you were given when you joined the class using the git send-email tool. You will need to configure git send email to be able to send patches from your email account, see [this guide](https://stackoverflow.com/questions/68238912/how-to-configure-and-use-git-send-email-to-work-with-gmail-to-email-patches-to) for instructions if you are using a gmail account. - -Once the configuration is correct, you will invoke git send-email to submit them: - -$ git send-email --to=your-cohort@googlegroups.com *.patch - -$ # send the emails to the mailing list. The list of patch files are passed as arguments - -$ # The *.patch wildcard expands to all of the files whose names end with .patch in the current directory - -#### [5] Peer Review Submission Process - -The patch format should be as follows: -* Send rfc patches for the submissions that are to be peer review (add the `--rfc` option when you format your patchset). -* Send non-rfc patches for the final submission (omit the `--rfc` option). - -#### [6] Late Assignment Submissions - -Late submissions will result in an automatic grade of zero. The only exception is the one-time per semester oopsie (refer [here](course_policies.md)). - -#### [7] Resubmissions - -If you notice a mistake in your submission (or have it pointed out to you by one of the instructors or another student) before the due date, you are welcome to resubmit as many times as you'd like until said deadline. If you choose to resubmit, you must regenerate the patches from your (potentially altered) commit(s), and send the full patch series as your resubmission (i.e. a cover letter and one email for each commit even if some of the emails are identical to what you sent before). - -Further, you must make clear that it is a resubmission by adding a version number after PATCH in the square brackets (i.e. Subject: [PATCH x/n] My title here would become Subject: [PATCH v2 x/n] My title here) incrementing the number as needed for however many resubmissions you make (you can use the `-v` option with an argument, denoting the version number, when reformatting your patchset). - -You must also document what changed since the last submission in your writeup, include a section with a title like “changes since vN” where N is the number of your last submission and you explain what changed. diff --git a/kdlp.underground.software/faq.md b/kdlp.underground.software/faq.md deleted file mode 100644 index 6e81891f..00000000 --- a/kdlp.underground.software/faq.md +++ /dev/null @@ -1,20 +0,0 @@ -### Frequently Asked Questions (FAQ): - -Q: How do I enroll in the "Introduction to Linux Kernel Development" course? - -A: Unless this program is offered at your university, apply for Introduction to Linux Kernel Development via [LFX](https://mentorship.lfx.linuxfoundation.org/project/958fe36a-d763-4422-81af-c5ecf2465957). - -Q: How can I stay informed about this program? - -A: [Join our mailing list](https://groups.google.com/forum/#!forum/rh-kdlp/join) to recieve our quarterly newsletter. - -Q: Who is behind this program? - -A: These people: - -|Name|Role|Contact| -|--|--|--| -|Joel Savitz|Software Engineer & Program Lead| [Joel's Website](https://joelsavitz.com)| -|Julia Denham|Software Engineer & Program Manager | [Julia's Linkedin](https://www.linkedin.com/in/julia-denham-4828a5120/)| -|Charles "Charlie" Mirabile|Software Engineer & Content Lead| [Charlie's Github](https://github.com/charliemirabile)| -|Denis Aleksandrov|TA & Intern|[Denis' GitHub](https://github.com/aleksandrov-denis)| diff --git a/kdlp.underground.software/images/Linux_Crash_Course/Fedora-logo.png b/kdlp.underground.software/images/Linux_Crash_Course/Fedora-logo.png deleted file mode 100644 index 9f1a4fec..00000000 Binary files a/kdlp.underground.software/images/Linux_Crash_Course/Fedora-logo.png and /dev/null differ diff --git a/kdlp.underground.software/images/Linux_Crash_Course/RHEL-logo-hat.png b/kdlp.underground.software/images/Linux_Crash_Course/RHEL-logo-hat.png deleted file mode 100644 index cf1f5b8a..00000000 Binary files a/kdlp.underground.software/images/Linux_Crash_Course/RHEL-logo-hat.png and /dev/null differ diff --git a/kdlp.underground.software/images/Linux_Crash_Course/arch_linux_logo.png b/kdlp.underground.software/images/Linux_Crash_Course/arch_linux_logo.png deleted file mode 100644 index 31d781b1..00000000 Binary files a/kdlp.underground.software/images/Linux_Crash_Course/arch_linux_logo.png and /dev/null differ diff --git a/kdlp.underground.software/images/Linux_Crash_Course/grub_menu.png b/kdlp.underground.software/images/Linux_Crash_Course/grub_menu.png deleted file mode 100644 index 5c1d2f06..00000000 Binary files a/kdlp.underground.software/images/Linux_Crash_Course/grub_menu.png and /dev/null differ diff --git a/kdlp.underground.software/images/Linux_Crash_Course/interjection.png b/kdlp.underground.software/images/Linux_Crash_Course/interjection.png deleted file mode 100644 index 1cac59c4..00000000 Binary files a/kdlp.underground.software/images/Linux_Crash_Course/interjection.png and /dev/null differ diff --git a/kdlp.underground.software/images/Linux_Crash_Course/slide15.png b/kdlp.underground.software/images/Linux_Crash_Course/slide15.png deleted file mode 100644 index c660ad3d..00000000 Binary files a/kdlp.underground.software/images/Linux_Crash_Course/slide15.png and /dev/null differ diff --git a/kdlp.underground.software/images/Linux_Crash_Course/slide16.png b/kdlp.underground.software/images/Linux_Crash_Course/slide16.png deleted file mode 100644 index edd69897..00000000 Binary files a/kdlp.underground.software/images/Linux_Crash_Course/slide16.png and /dev/null differ diff --git a/kdlp.underground.software/images/Linux_Crash_Course/storage_layout_slide17.png b/kdlp.underground.software/images/Linux_Crash_Course/storage_layout_slide17.png deleted file mode 100644 index a8c65df0..00000000 Binary files a/kdlp.underground.software/images/Linux_Crash_Course/storage_layout_slide17.png and /dev/null differ diff --git a/kdlp.underground.software/images/Linux_Crash_Course/ubuntu_logo.png b/kdlp.underground.software/images/Linux_Crash_Course/ubuntu_logo.png deleted file mode 100644 index 8a16da83..00000000 Binary files a/kdlp.underground.software/images/Linux_Crash_Course/ubuntu_logo.png and /dev/null differ diff --git a/kdlp.underground.software/images/fedora_confusion.png b/kdlp.underground.software/images/fedora_confusion.png deleted file mode 100644 index 5f1d12b4..00000000 Binary files a/kdlp.underground.software/images/fedora_confusion.png and /dev/null differ diff --git a/kdlp.underground.software/images/grub_menu.png b/kdlp.underground.software/images/grub_menu.png deleted file mode 100644 index bcdf483a..00000000 Binary files a/kdlp.underground.software/images/grub_menu.png and /dev/null differ diff --git a/kdlp.underground.software/images/kdlp_logo.png b/kdlp.underground.software/images/kdlp_logo.png deleted file mode 100644 index 1fce827c..00000000 Binary files a/kdlp.underground.software/images/kdlp_logo.png and /dev/null differ diff --git a/kdlp.underground.software/index.md b/kdlp.underground.software/index.md deleted file mode 100644 index 76184c31..00000000 --- a/kdlp.underground.software/index.md +++ /dev/null @@ -1,59 +0,0 @@ -### TL;DR: - -The Kernel Development Learning Pipeline (KDLP) program aims to create a comprehensive Linux kernel talent pipeline to address the current shortage of qualified entry level kernel candidates. - -### Mission statement - -Throughout the industry, there is a shortage of qualified entry-level software engineers focused on systems programming, especially within the niche of developing for the Linux kernel. - -As junior software engineers ourselves, this quickly became evident to us, and we are not the only people concerned about it. -Engineers and managers of all levels within our industry niche have observed that we must bring new talent into these spaces, and we must do so urgently. - -Time spares no one, not even the most legendary programmers of yore, and if we do not identify and train the next generation, -there will be no one to carry the torch of systems programming. -We have had the chance to work with many talented and experienced engineers firsthand, and we have benefited greatly from their guidance and mentorship, but we know that this opportunity will not last forever. - -In response, we have created the Kernel Development Learning Pipeline a program to bridge the gaps between academia and industry, between the novice and the legend, between the past and the future. - -### More detail - -Our program: - - - develops and teaches an "Introduction to Linux Kernel Development" course with curriculum designed to offer a smooth onramp for contributing to the linux kernel. - - participates in the wider linux community (sending patches upstream, attending and presenting at conferences, etc) - - recruits qualified students from our class to internship programs within Red Hat, personally recommending the best interns to hiring managers with the aim of bringing them into the industry as full time software engineers. - - innovates the role open source can play in the classroom: - - What if courses were run like an open source project, and students were encouraged to contribute to their learning by making their mark on the classes they take, leaving them better than they found them, and learning along the way. - - Science has shown, that the only way to learn is to process the information through your brain - to tell the story in your own words. - - By encouraging students to become active members of the course and take ownership of their learning, we can supercharge their growth. - -Current offerings: - - - We work with the Miner School of Computer and Information Sciences at the University of Massachuesetts Lowell, to offer [our course to their students](https://www.uml.edu/catalog/courses/COMP/3085) on an ongoing basis - - We are partnered with the Linux Foundation's LFX mentorship platform to offer our course as a [mentorship](https://mentorship.lfx.linuxfoundation.org/project/958fe36a-d763-4422-81af-c5ecf2465957). - -Past Work: - - - Spoke at Linux conventions: - - [LPC 2023](https://lpc.events/event/17/contributions/1554/) - - [Fosdem 2023](https://archive.fosdem.org/2023/schedule/event/kdlp_kernel_devel_learning_pipeline/) - - [devconf.cz 2023](https://devconfcz2023.sched.com/event/1MYmM/kernel-development-learning-pipeline-kdlp) - - [devconf.us 2021](https://devconfus2021.sched.com/event/lkgk/fedora-linux-for-the-raspberry-pi-platform) - - Upstream patches: - - [AT_EMPTY_PATH for linkat syscall](https://lore.kernel.org/all/20231110170615.2168372-1-cmirabil@redhat.com/) - - [flags for io_uring linkat support](https://lore.kernel.org/all/20231120105545.1209530-1-cmirabil@redhat.com/) - - [nolibc getauxval](https://lore.kernel.org/all/20240116181147.2230944-1-cmirabil@redhat.com/) - - [Raspberry Pi Sensehat Driver](https://lore.kernel.org/lkml/20220823174158.45579-1-cmirabil@redhat.com/) - - [RPi.GPIO2 python library](https://pypi.org/project/RPi.GPIO2/) - - [linux libgpiod library](https://lore.kernel.org/linux-gpio/20191203192305.31722-1-jsavitz@redhat.com/) - - [vchiq_arm](https://lore.kernel.org/all/20211014223230.451659-1-mguma@redhat.com/) - -Future Plans: - - - Scale our program - offer it at more universities - - improve options for self-study - - connections at more companies in this space for recruitment - -### Fun fact - -You can view this website in a CLI browser (such as [lynx](https://lynx.invisible-island.net/lynx2.8.9/index.html)). diff --git a/kdlp.underground.software/lpc2023.html b/kdlp.underground.software/lpc2023.html deleted file mode 100644 index 46d2eae4..00000000 --- a/kdlp.underground.software/lpc2023.html +++ /dev/null @@ -1,153 +0,0 @@ -
-

KDLP

-

Kernel Development Learning Pipeline

-

Joel Savitz

-

LPC 2023

-
-
-

What is KDLP?

-
    -
  • -

    A Red Hat sponsored open source project

    -
    • -
  • -

    An attempt to address the documentation problem

    -
    • -
  • -

    A smooth onramp for new Linux kernel engineers

    -
    • -
-
-
-

What do we do?

-
    -
  • -

    Publish course materials as freely available resource

    -
    • -
  • -

    Design and deliver a for-credit university course

    -
    • -
  • -

    Recruit the best and brightest as interns

    -
    • -
  • -

    Iterate on our curriculum using real student feedback

    -
    • -
-
-
- -

Produced by Joseph Brody, KDLP Creative Director

-
-
-

What are up to right now?

-

Finishing up the Fall Semester

-

All materials published at kdlp.underground.software

-
-
-

And now, on to the discussion

-
-
-

The documentation problem

-

and

-

The onboarding problem

-
-
-

Documentation

-
    -
  • -

    Quickly out of date

    -
    • -
  • -

    Does not meet students where they are

    -
    • -
  • -

    Sometimes less than thrilling to read

    -
    • -
-
-
-

Onboarding

-

Rhetorical questions, wait for next slide

-
    -
  • -

    How does an engineer break into kernel development?

    -
    • -
  • -

    Where do they go to start without guidance?

    -
    • -
  • -

    How can good resources find the correct eyes?

    -
    • -
-
-
-

Discussion questions

-
    -
  • -

    How did you get into kernel development?

    -
    • -
  • -

    How could your experience have been better?

    -
    • -
  • -

    How can we address that shortfall with our work?

    -
    • -
-
-
-

Shout-outs

-
    -
  • -

    Big thank you to Heidi Dempsey for her longtime support of the KDLP program. We wouldn't be here without her.

    -
    • -
  • -

    Thank you to Mike McGrath for his sponsorship of the program.

    -
    • -
  • -

    Thank you to the Red Hat Open Source Community Infrastructure team for providing hosting.

    -
    • -
-
-
-

More Credits

-
-

Charles Mirabile
-Julia Denham
-Michael Burke
-Conor Cowman
-Denis Aleksandrov
-Jake Correnti
-Joel Slebodnick
-Daniel Bauman
-Mwesigwa Guma
-Steve Meisner
-William Moloney
-Jeff Brown
-Val Cowan
-Ian Chen
-Holly Yanco
-Fred Martin
-Perry Myers
-Peter Martuccelli

-
-
-
-

Thank you

-

Visit our website:

-

kdlp.underground.software

-

My website: joelsavitz.com

-
-
\ No newline at end of file diff --git a/kdlp.underground.software/slides_src/advanced_char_driver_ops.md b/kdlp.underground.software/slides_src/advanced_char_driver_ops.md deleted file mode 100644 index f2f94414..00000000 --- a/kdlp.underground.software/slides_src/advanced_char_driver_ops.md +++ /dev/null @@ -1,197 +0,0 @@ ---- -theme: white ---- - -# Advanced Character Driver Operations -## In Linux -[LDD3](https://www.amazon.com/Linux-Device-Drivers-Jonathan-Corbet/dp/0596005903): [Chapter](https://lwn.net/Kernel/LDD3/) [6](https://static.lwn.net/images/pdf/LDD3/ch06.pdf) - ---- - -# ioctl -- int (\*ioctl) (struct inode \*inode, struct file \*filp, unsigned int cmd, unsigned long arg); -- Control hardware - - Lock door, eject media, error information -- Big switch statement that selects the correct behavior according to ‘cmd’ argument - ---- - -# ioctl commands -- Make the numbers for commands unique - - 4 bit fields (type, number, direction, size) - - IO_TYPEBITS, _IOC_NRBITS ... -- _IO macro for combining a type and number into one value - - See sensehat.h in repo for example - ---- - -# ioctl return value -- POSIX standard states to return -ENOTTY when ioctl command is not valid -- Although several kernel functions return -EINVAL - ---- - -# Capabilities & Restricted Operations -- A capability based system breaks down privileged operations into separate subgroups -- Capset, capget -- if (! capable (CAP_SYS_ADMIN)) -return -EPERM; - ---- - -# ioctl argument -- Is unsigned long but can be an integer or a pointer - - Depends on the particular ioctl -- Do appropriate checking before access if ‘arg’ is pointer - - access_ok(type, \*addr, size) - - put_user(), __put_user(), get_user(), __get_user() - ---- - -# Implementation -- Often just a big switch statement -switch(cmd) { -case SCULL_IOCRESET: -scull_quantum = SCULL_QUANTUM; -scull_qset = SCULL_QSET; -break; - -case SCULL_IOCSQUANTUM: -… -} - ---- - -# Non-ioctl device control -- By writing control sequences - - For example; running a remote tty session - - Escape sequences are printed remotely but affect the local tty -- Controlling by write - - For devices that just respond to commands - ---- - -# Blocking I/O -Putting process to sleep until IO operation can happen - ---- - -# Sleeping -- When a process is put to sleep, it is removed from the scheduler’s run queue and put into a special state -- Never sleep when running in an atomic context - - Atomic context is where multiple steps must occur without any concurrent access ie while holding a spinlock, seqlock or RCU lock - - When interrupts are disabled - - No way to wake up -- Somebody has to wake up sleeping process - - Wait queue - ---- - -# Simple sleeping -- Wait_event(queue, condition); -- wake_up(wait_queue_head_t \*queue); - ---- - -# Blocking vs nonblocking I/O -- Sometimes process does not want to block whether or not it’s I/O can make progress -- O_NONBLOCK flag for non blocking I/O -- If read is called and no data, block until data arrives. -- Similarly if write is called and no space in buffer, block until buffer is free -- If O_NONBLOCK is used, read and write calls simply return -EAGAIN (“try again”) allowing the application to poll for data - ---- - -# Blocking I/O Example -Page 153 in [LDD3](https://static.lwn.net/images/pdf/LDD3/ch06.pdf) - ---- - -# Advanced Sleeping -- First step: allocation and initialization of wait_queue_t structure followed by it’s addition to proper wait queue -- Second step is to set the state of the process to mark it as being asleep - - TASK_ITERRUPTIBLE and TASK_UNINTERRUPTIBLE indicate a process is asleep -- Final step is to give up the processor. - - Check the condition you are sleeping for first. Failure to do this may result into a race condition -if (!condition) -schedule( ); - ---- - -# poll and select -- Poll, select and epoll used by apps that use nonblocking I/O -- They all essentially have the same functionality - - Each allow a process to determine whether it can read from or write to one or more files without blocking -- (\*poll) (struct file \*filp, poll_table \*wait); - - Called when user-space program performs poll, select or epoll - ---- - -# Poll and select vs read and write -- Poll and select determine in advance if an I/O operation will block in that respect complementing read and write -- Poll returns POLLIN|POLLRDNORM if there is data in the read buffer -- Poll reports that device is unreadable if there is no data -- Poll reports POLLHUP if we are at EOF -- Poll returns POLLOUT|POLLWRNORM if writing won’t block -- Reports file is not writable if writing would block - ---- - -# struct poll_table\* -- Holds ‘struct file’ and ‘wait_queue_head_t’ pointers passed to poll_wait -- A call to poll or select involves a handful of file descriptors so the cost of setting up the data structure is small -- epoll allows for scaling to many file descriptors by setting up this data structure exactly once and to use it many times - ---- - -# Async Notifs -- Instead of calling poll to check for data, set async notifs for when data becomes available -- Step 1: Specify process as owner of file -- Step 2: Set FASYNC flag in device by means of the F_SETFL fcntl command -- Input file can now request delivery of a SIGIO signal whenever new data arrives -signal(SIGIO, &input_handler); /\* dummy sample; sigaction( ) is better \*/ -fcntl(STDIN_FILENO, F_SETOWN, getpid( )); -oflags = fcntl(STDIN_FILENO, F_GETFL); -fcntl(STDIN_FILENO, F_SETFL, oflags | FASYNC); - ---- - -# POV: You are the driver -- When F_SETOWN is invoked, a value is assigned to filp->f_owner -- When F_SETFL is executed to turn on FASYNC, the driver’s fasync method is called -- When data arrives, all the processes registered for ansync notis must be sent a SIGIO - ---- - -# llseek -- Provide llseek method if seek operation corresponding to a physical operation on a device rather than using default -- Some devices offer a data flow rather than a data area so seeking makes no sense. For example: serial ports of a keyboard -- In this case, use ‘nonseekable_open’ for open -- Set llseek method in ‘file_operations’ to no_llseek - ---- - -# Access control -- Sometimes only one authorized user should be allowed to open a device -- Similar to ttys problem. -- brute force: permit device to be opened by only one process at a time(single openness). AVOID THIS -- Allow only one user to open device but can be opened by multiple processes - - Same policy as ttys - ---- - -# Cloning the Device on Open -- Create different private copies of the device depending on process opening it -- Possible if not bound to hardware object -- /dev/tty uses a similar technique -- Copies of the driver are called “virtual devices” just like “virtual consoles” use a single tty device -- Rarely needed - ---- - -# Block open instead of EBUSY -- Implement blocking ‘open’ so that the operation is slightly delayed rather than fail -- Implementation is based on wait queue -- Release is in charge of awakening any pending process -- A blocking ‘open’ implementation is a problem for interactive user who has to keep guessing what’s going on - - Would rather get a “resource busy” message diff --git a/kdlp.underground.software/slides_src/block_drivers.md b/kdlp.underground.software/slides_src/block_drivers.md deleted file mode 100644 index 7f9ee81d..00000000 --- a/kdlp.underground.software/slides_src/block_drivers.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -theme: white ---- - -# Block Drivers -Chapter 16 - ---- - -# Block Drivers -- Accessed by fixed sized blocks -- Primarily used for disks drivers -- Performance is more important than with char devices -- Need to include in the driver: -```console -#include -#include -#include -``` - ---- - -# Data types -- Blocks are fixed size chunks of data, with size determined by the Kernel - usually 4096 bytes -- Sectors are smaller fixed size chunks of data, with size determined by the attached hardware (i.e. disk) - usually 512 bytes - ---- - -# Registration -- Block devices are registered and unregistered using the following functions included from linux/fs.h -```console -int register_blkdev(unsigned int major, const char *name); -int unregister_blkdev(unsigned int major, const char *name); -``` -- As with char devices, passing 0 as the major number will make the kernel assign an available major number - ---- - -# Block Drivers Operations -- Some of the operations work exactly like they do in char drivers: -```console -int (*open)(struct inode *inode, struct file *filp); -int (*release)(struct inode *inode, struct file *filp); -int (*ioctl)(struct inode *inode, struct file *filp, unsigned int cmd, unsigned long arg); -``` - ---- - -# Block Driver Operations -- Block driver specific methods: -```console -int (*media_changed) (struct gendisk *gd); -``` -- Checks if the user made a change to the removable media: returns non-zero value on change -```console -int (*revalidate_disk) (struct gendisk *gd); -``` -- Called when a user makes a change in the media, performs required work according to the change. -- struct gendisk represents a single disk in the kernel - ---- - -# struct gendisk -- See linux/genhd.h -- Initialized by the Block driver -- Arguments: -```console -int major; int first_minor; int minors; # each minor represents a partition -char disk_name[32]; # cat /proc/partitions -struct block_device_operations *fops; -struct request_queue *queue; # used by kernel to manage I/O requests -int flags; # bit fields flags describing the device (i.e. GENHD_FL_REMOVABLE) -sector_t capacity; # the capacity of the device in 512-byte long sectors unit -void *private_data; # to be used at the discretion of the block driver -``` - ---- - -# struct gendisk -- To allocate a struct gendisk: -```console -struct gendisk *alloc_disk(int minors); -``` -- Free using: -```console -void del_gendisk(struct gendisk *gd); -``` -- Finally, add the gendisk to the system using: -```console -void add_disk(struct gendisk *gd); -``` -- Set the capacity for the gendisk using: -```console -void set_capacity(struct gendisk *gd, sector_t sectors) -``` diff --git a/kdlp.underground.software/slides_src/concurrency.md b/kdlp.underground.software/slides_src/concurrency.md deleted file mode 100644 index 3fdc10c9..00000000 --- a/kdlp.underground.software/slides_src/concurrency.md +++ /dev/null @@ -1,229 +0,0 @@ ---- -theme: white ---- - -# Concurrency and Race Conditions -## In the Linux kernel - -[LDD3](https://www.amazon.com/Linux-Device-Drivers-Jonathan-Corbet/dp/0596005903): [Chapter](https://lwn.net/Kernel/LDD3/) [5](https://static.lwn.net/images/pdf/LDD3/ch05.pdf) - ---- - -# Review of Definitions - ---- - -# Thread -- Thread of execution -- Stores execution state (registers + instruction pointer) and has its own stack -- Each process has at least one (but maybe more) of them -- Is the unit of program that actually is scheduled on to the CPU - ---- - -# Concurrency -- Multiple threads of execution happening (or appearing to happen) at the same time -- Even with a single cpu core, task switching can give the same effects as true multiprocessing -- Up to the OS scheduler to decide who runs when - ---- - -# Sleep -![penguins](images/Concurrency/penguins.png) -- When a linux process reaches a point where it cannot make further progress until something else is finished (e.g. waiting for data from disk or user io) -- Scheduler yields the processor to another thread - ---- - -# Interrupt -- A signal emitted by hardware or software when something needs immediate attention -- Scheduler gives the code responsible for handling these extremely high priority - ---- - -# Critical Section -- Section of code where shared resources (data like global variables or state like i/o) are accessed by a thread of execution -- Cannot have multiple threads working on the same shared resources running their critical sections concurrently - - That’s a race condition! - ---- - -# Race Condition -- Multiple processes trying to access shared data at the same time -- “Race” to see which thread reads/writes data first -- Unexpected results when wrong access pattern happens - - Memory leak - - Corrupted data - - System crash -- The asynchronous nature of interrupts means that code handling them must be extremely careful with global data. It also cannot block (sleeping) - - Increased probability of race conditions - ---- - -# Mutual exclusion -- Maintain concurrency for independent parts of execution (so we can enjoy speed or responsiveness benefits of parallel execution) -- Stop multiple processes from running within a critical section simultaneously -- Ensured using locking data structures like spin locks, semaphores, mutexes, event counters, sequencers, etc. - ---- - -![mutual exclusion 1](images/Concurrency/slide10.png) -[image source](https://www.geeksforgeeks.org/mutual-exclusion-in-synchronization/) - ---- - -![mutual exclusion 2](images/Concurrency/slide11.png) -[image source](https://www.geeksforgeeks.org/mutual-exclusion-in-synchronization/) - ---- - -![mutual exclusion 3](images/Concurrency/slide12.png) -[image source](https://www.geeksforgeeks.org/mutual-exclusion-in-synchronization/) - ---- - -# Deadlock -![deadlock](images/Concurrency/slide13.png) -- Mutual exclusion gone wrong -- A situation where threads are holding some resources but also blocked waiting for others in a cycle preventing any forward progress. -[image source](https://www.geeksforgeeks.org/introduction-of-deadlock-in-operating-system/) - ---- - -# Locking Data Structures -- spinlock - - locking primitive that can be used to implement all other forms of mutual exlcusion -- semaphore - - Combination of integer value and two functions for increment and decrement (v,p) -- mutex - - Special kind of semaphore with only two states locked/unlocked - ---- - -# Concurrency Techniques within Linux - ---- - -# Spinlocks -- spinlock_t in kernel -- Spinlocks must be held for minimum time possible -Only acceptable in atomic contexts - ---- - -# Spinlocks in Atomic Context -- Interrupts must be disabled on local cpu when a lock is used - - Preemption disabled by locking implementation! -- Why? an interrupt might cause a different thread to be scheduled, which may require the lock you just took but haven’t yet released - - Deadlock! -- Code must therefore be atomic (cannot sleep) - ---- - -# Semaphores and Mutexes -- struct semaphore -- struct mutex - ---- - -# Reader/Write Semaphores -![semaphores](images/Concurrency/slide19.png) -- Allows multiple concurrent readers -- rwsem - - One writer or unlimited number of readers - - Writers get priority - - Allowing multiple readers Optimizes performance -[image source](https://sungju.github.io/kernel/internals/synchronization.html) - ---- - -# Completions -- Allow a thread to tell another that a job is done -- Better locking mechanism - - Prevents performance issues when creation of user processes or new kernel threads are involved -- A: complete(xxx) -- B: wait_for_completion(xxx) -- B only proceeds when A had called complete - ---- - -# Reader/Writer Spinlocks -- Allow any number of readers into critical section simultaneously -- Writers must have exclusive access -- rwlock_t similar to rwsem semaphore - ---- - -# Ambiguous rules -Be careful not to double lock! -e.g. foo() gets lock A, calls bar() which also gets lock A -Common solution: modified _bar() that assumes A is already held - ---- - -# Lock Ordering Rules -![lock ordering](images/Concurrency/slide23.png) -Total ordering! -if you lock A then B then C -You MUST unlock C then B then A -And never lock B then A (etc) somewhere else! -Otherwise, you get deadlock… - ---- - -# Fine Versus Coarse-Grained Locking -- Old kernels had one big lock - - Not scalable -- Modern kernels have thousands of locks each protecting smaller resources -- Can be too complex -- lock_kernel() lmao 😂 - ---- - -# Alternatives to locking -## 😳 - ---- - -# Lock-Free Algorithms -- Circular buffer -- Requires no locking in the absence of multiple consumers and producers -- Only Producer thread can modify the write index and array location it points to -- Only reader thread can access read index and value it points to -- If the two pointers don’t overrun each other, the producer and consumer can access the buffer concurrently with no race conditions -[https://stackoverflow.com/questions/871234/circular-lock-free-buffer](https://stackoverflow.com/questions/871234/circular-lock-free-buffer) - ---- - -# Atomic Variables -- Used for simple shared resources like an int value -- o_op++; - - Can be done in atomic manner -- Locking is overkill, use an atomic_t instead - - atomic_inc(atomic_t \*v); - - atomic_dec(atomic_t \*v); - - atomic_add_return(int i, atomic_t \*v); - ---- - -# Bit Operations -- Used for shared flags -- Manipulate individual bits in an atomic manner -- Example functions - - set_bit(nr, void \*addr); - - clear_bit(nr, void \*addr); - ---- - -# seqlocks -- Used for a frequently accessed, small, and simple resource -- Allows free access for readers but requires the readers to check for collisions with writers -- When a collision occurs, retry their access -[https://www.kernel.org/doc/html/latest/locking/seqlock.html](https://www.kernel.org/doc/html/latest/locking/seqlock.html) - ---- - -# Read-Copy-Update (RCU) -- When a data structure needs to be changed, the writing thread makes a copy, changes the copy, then aims the relevant pointer at the new version -- Used for situations where, reads are common and writes are rare -- Very complex: many memory barriers used diff --git a/kdlp.underground.software/slides_src/debugging_techniques.md b/kdlp.underground.software/slides_src/debugging_techniques.md deleted file mode 100644 index f5fba49f..00000000 --- a/kdlp.underground.software/slides_src/debugging_techniques.md +++ /dev/null @@ -1,99 +0,0 @@ ---- -theme: white ---- - -# Debugging Techniques -## For the Linux Kernel -[LDD3](https://www.amazon.com/Linux-Device-Drivers-Jonathan-Corbet/dp/0596005903): [Chapter](https://lwn.net/Kernel/LDD3/) [4](https://static.lwn.net/images/pdf/LDD3/ch04.pdf) - ---- - -# Printk: the elite printf -Log levels -- KERN_EMERG -- KERN_ALERT -- ... -printk(KERN_XXXX "format string", ...); -NOT: printk(KERN_XXX, "format string", ...); -/proc/sys/kernel/printk - ---- - -# Querying -Massive use of printk slows down the system, query system info instead -Can also rate limit -Techniques for querying: -- Using /proc filesystem -- ioctl -- sysfs - ---- - -# Using the /proc filesystem -Each file under /proc is tied to a kernel function that generates it’s contents -- ps, top, uptime …, get info from /proc -- Add/create files in /proc to export messages from drivers (discouraged) -- Use sysfs instead (covered in ch-14) - ---- - -# ioctl - -- System call that acts on a file descriptor -- Implement ioctl commands tailored for debugging -- Faster than /proc -- But more work compared to using /pro - ---- - -# Watching -Watch behavior of driver in user space -- Call read() and watch it’s behavior -- Run a debugger and step through function calls -- strace - ---- - -# More on strace -A command that shows all system calls issued by a user-space program -- Shows function calls, their arguments and return values -- -t (to display time call was made) -- -e (to limit type of calls traced) -- -o (to redirect output. Default is stderr) -Strace example next - ---- - -# strace ls /dev > /dev/scull0 -open("/dev", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3 -... -getdents64(3, /\* 141 entries \*/, 4096) = 4088 -… -fstat64(1, {st_mode=S_IFCHR|0664, st_rdev=makedev(254, 0), ...}) = 0 -write(1, "MAKEDEV\nadmmidi0\nadmmidi1\nadmmid"..., 4096) = 4000 -write(1, "b\nptywc\nptywd\nptywe\nptywf\nptyx0\n"..., 96) = 96 -… -close(1) -exit_group(0) - ---- - -# System faults/Oops messages -- Caused by dereferencing NULL or invalid pointer -- When an invalid pointer is dereferenced, the paging mechanism cannot map the pointer to a physical address and the processor signals a page fault - ---- - -# System Hangs -Sometimes there are no oops messages printed when bugs in the kernel hang the system -- Endless loop causes the kernel to stop scheduling aka hang - ---- - -# Tools for Advanced Debugging and More -- BPF: more than just the Berkeley Packet Filter -- Ftrace: “function tracer” but much more -- Gdb and friends (such as crash) -- [Ptrace](https://man7.org/linux/man-pages/man2/ptrace.2.html): from one process to another -- [Perf](https://en.wikipedia.org/wiki/Perf_(Linux)): performance analysis tool - diff --git a/kdlp.underground.software/slides_src/git.md b/kdlp.underground.software/slides_src/git.md deleted file mode 100644 index 556da58e..00000000 --- a/kdlp.underground.software/slides_src/git.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -theme: white ---- - -# Git -An open source version control system - ---- - -# Version Control & Git vs. GitHub -- The management of changes to a project -- Git is specifically designed for version control of programming projects, provides a log of changes to your project. You can access previous instances of the project to help with collaboration, bug fixes, etc. -- Git is not GitHub -- GitHub is a website which can host your repositories to be available to the public. This is useful for group work and open source projects. - ---- - -# More Git -- Git allows you to keep track of the previous states of your or someone else’s repo -- Each saved state of your repo has a hash identifier with which you can access said state -- A repo that is managed with Git will have a .git directory which will have information about all saved changes -- ls -a → To check hidden files not displayed by ls by default - ---- - -# Visualization -We will use a tree structure to illustrate the behavior of how saved states of a repository behave when manipulated with git commands. E.g. Below is a repository which has three commits to the main branch and one commit to the X branch. The blue pointer indicates that we are viewing the first commit of the X branch. -![git tree](images/Git/git_slide4.png) - ---- - -# Diagram Key -![diagram key](images/Git/git_slide5.png) - ---- - -# Git Commands: git init -- Initialize the cwd as a git repository -- In order to use most other git commands, you will need to be in a folder containing an initialized repo first - - ---- - -# Git Commands: git clone -![git clone diagram](images/Git/git_slide7.png) -Copies a repository that is hosted on GitHub, or similar website, to your local directory - ---- - -# Git Commands: git add -- Adds a change, which was made to the repository, to a list of changes to be committed later on. This accounts for changes to an existing file or adding a new directory or file to the repo. -- 'git add .' adds all new files and changes to existing files in current directory to the index - ---- - -# Git Commands: git commit -![git commit diagram](images/Git/git_slide9.png) -- Saves the current state of your repository in Git. Changes are not live, need to push -- git commit -m “somemessage” to add a short message to your commit and avoid opening your editor - ---- - -# Git Commands: git status -Shows which changes are tracked by Git and which changes still need to be committed. - ---- - -# SSH Keys -- To push your changes to your GitHub you need to link your local machine to your Github account. -- Generate SSH key -$ ssh-keygen -t rsa -b 4096 -C “your github email” -- Cat your public key file and highlight and copy it -$ cat ~/.ssh/id_rsa.pub -- On GH go to Settings/SSH and GPG keys/New SSH key -- Name your key and paste the public key in the key field -- Now you can push to GitHub - ---- - -# Additional Details -- If you your local repo was created by cloning a remote repo, your local repo will already be linked with that remote. -- If you want to publish a repo you made locally on github, you will need to make an empty GitHub repo and find it’s link and set a remote to that link manually -$ git remote add origin git@github.com:whoever/whatever.git -$ git push origin main -- To set a default remote branch for future push operations on a given local branch do -$ git push -u origin main -- And next time you push you can just do -$ git push - - ---- - -# Git Commands: git push -![git push diagram](images/Git/git_slide13.png) -Uploads new Git commits to a remote repository. e.g. to GitHub - ---- - -# Git Commands: git pull -![ git pull diagram](images/Git/git_slide14.png) -Opposite of git push. Download changes locally from remote repository - ---- - -# Branching: uses -- The main stem of the repo is the main branch, in a group project main will be the ‘final’ version of the project. -- Branching is useful when group members have separate responsibilities to the project. -- You can branch from main and work on your part of the project without disrupting main. -- When you’re done and your code has been approved, your changes can be merged back into the main branch. -- Delete a branch with git branch -d name - ---- - -# Branching -![git branching d1](images/Git/git_slide16.png) -$ git branch -\* main - feature1 -- Use git branch to check which branches exist and which one you are currently working in - ---- - -![git branching d2](images/Git/git_slide17.png) -$ git checkout feature1 -(switch to an existing branch) -$ git branch - main -\* feature1 - ---- - -![git branching d3](images/Git/git_slide18.png) -$ git checkout main; git branch -\* main - feature1 -$ git checkout -b feature2 (creates a new feature2 branch and brings you to it) -$ git branch - main - feature1 -\* feature2 - ---- - -# Merging and pull requests(PR) -- If you are working on your own project you will use git merge to merge your branches into main. -- If you are merging a feature branch on to main you are creating a new commit to main where all previous main and feature commits are combined. -$ git branch - main -\* other -$ git checkout main; git merge other; git diff other -- They will be the same if there are no conflicts - ---- - -# Git Commands: git diff -- To compare the changes between the current branch and the feature branch -$ git diff feature -- To see what changes have been made to file.txt -$ git diff file.txt (if file.txt is staged) -- To see changes of all staged files -$ git diff -- For more advanced features -$ git diff --help - ---- - -# Merging and pull requests -- When working on a group project, you will want to open a pull request to make sure that your group is satisfied with merging your branch to main -- First you will need to push your branch to GH -$ git push -u origin PRbranch -- Here you are setting origin for your local branch PRbranch to be your GH PRbranch. So your local PRbranch will only push to the same GH PRbranch -- Then, you can go on your GH and open a new pull request. You are requesting that your team review your proposed changes to main and then pull PRbranch into the main branch. - ---- - -# Merging and pull requests -![merging d1](images/Git/git_slide20_1.png) -$ git branch -\* main - feature1 - feature2 - ---- - -![merging d2](images/Git/git_slide20_2.png) -$ git merge feature1 -- If you use the '--squash' tag, it will merge but stop before making the merge commit -$ git merge --squash feature1 -\# Make some changes -$ git add . && git commit -m "some message" -- This allows you to make some more changes before you commmit your merge - ---- - -![merging d3](images/Git/git_slide20_3.png) -$ git merge feature2 - ---- - -# Group project note -When new changes are pushed to main by other people in your group, in order to keep your branch up to date pull changes from your GH to your local machine and then merge local main into your branch. Just step one in the example. -![group project note diagram](images/Git/git_slide21.png) - ---- - -# Merge conflicts -You and your group members might modify the same file in the repo, if two people modify the same line of code and then you try to merge, git will not know how to handle the conflict. You will need to resolve that manually. - ---- - -# Git Commands: git log -- Allows you to see all commit history in your local git repo. Each commit will have a hash associated with it, so you could git checkout a particular commit down the tree if you wanted to. -- The log is a good example of why you want to add descriptive messages when you commit, otherwise you will have no information about previous commits and will have to check each one individually to get any insight on them. -- You can unstage a particular commit with git reset commitname/hash -- Or fully remove the commit and set the new HEAD of the tree to the previous commit with git reset --hard commitname/hash - ---- - -# Forking a repository -You might not have direct access to working with someone else’s repo online, so to gain access to it you can fork it to your GH and then clone it to your local machine. This allows you to work on the same project and add new features, when you’re ready you can open a pull request against the original repo. - ---- - -# Advanced Git: git rebase -![rebase 1](images/Git/git_slide25_1.png) ![rebase 2](images/Git/git_slide25_2.png) -$ git branch -\* main - feature1 -$ git rebase feature1 - ---- - -# Advanced Git: git cherry-pick -You use this to pick one particular commit to merge with main->HEAD -![cherrypick 1](images/Git/git_slide26_1.png) ![cherrypick 2](images/Git/git_slide26_2.png) -$ git branch -\* main - feature1 -$ git cherry-pick \#hash\# -Note: you can use -n to merge the commit with HEAD without committing - - ---- - -# Advanced Git: git tag -- The ways to reference commits are by checking out a branch, which will bring you to branchname -> HEAD, or by referencing the hash of a particular commit -- But if we have an interesting commit which we want easy access to later, we can tag it so we don’t have to look up it’s hash -![git tag](images/Git/git_slide27.png) - ---- - -# Advanced Git: git log -- Like with any git command you can add flags to it to fine tune what you want the command to achieve -- git log --help to check the flags. This applies to all git commands e.g. git cherry-pick --help -- A useful variation on git log is -$ git log --graph --oneline --decorate -- (--graph) shows a log in a graph format distinguishing the different active branches -- (--oneline) compacts each commit to one line - ---- - -# Advanced Git: git bisect -Useful tool for finding the first commit which introduced a bug -Steps: -1. git bisect start -2. git bisect good hash -3. git bisect bad (you can leave fourth arg blank to start from current commit) -Bisect will run a binary search based on your input, after step 3 bisect will drop you on a commit half way between the known good and bad commits -4. git bisect good/bad (depending on whether your test passed/failed) -5. Repeat step 4 until bisect finds the first bad commit -6. Fix the bug -7. git bisect reset (to get back to the commit where you ran step 1) - ---- - -# Git Strategies: commit messages -- When staging a commit, try to organize each commit by topic. Do not cram unrelated changes into one commit -- If you have modified a file for multiple different topics you can stage parts of the file with partial staging, use -- git add -p file.txt -- Terminal will prompt you which parts you would like to stage (y/n) - ---- - -# Commit Message Conventions -Formulating a commit message -1. The subject should be a concise summary of your changes -Write this on the first line of the git editor that pops up after a commit -2. The body can be more detailed -Write this on line 3 of the git editor - - How did the file change - - Why did you make the change - - Any key notes on the change - ---- - -# Advanced Git: git rev-list -- Use this to show a custom view of the tree of commits -$ git rev-list feature1 feature2 main -- Shows commits from feature1 and feature2 but not main -$ git rev-list --not feature1 feature2 ^main -- Does the opposite -$ git-rev-list --help - ---- - -# Advanced Git: formatting and applying a patchset -- Another way to merge two branches is by applying a patchset -- A patchset contains patches which is essentially a diff file in an email format -- Gives your group the ability to review the patch before merging -![format patch d1](images/Git/git_slide33.png) - ---- - -# Advanced Git: formatting and applying a patchset -- You can apply individual patches by patch name in the patches directory -- If you do this, make sure that the parent of the commit that is to be applied as a patch is already in the branch you are applying to -![format patch d2](images/Git/git_slide34.png) - ---- - -# git send-email -Follow [this](https://stackoverflow.com/questions/68238912/how-to-configure-and-use-git-send-email-to-work-with-gmail-to-email-patches-to) to configure your system to send email patches with your gmail address. -Then you will need to create a patch set and run -$ git send-email --to=address1,address2,etc ./path/to/patch(es) - ---- - -# Git Cheat Sheet -[https://phoenixnap.com/kb/git-commands-cheat-sheet#ftoc-heading-10](https://phoenixnap.com/kb/git-commands-cheat-sheet#ftoc-heading-10) -Useful Commands: -rm -rf directory (to delete a directory and everything in it) ---abort (to abort any git process) -e.g. git merge --abort - ---- - -# References -- man git -- https://www.youtube.com/watch?v=RGOj5yH7evk -- [https://www.youtube.com/watch?v=_UZEXUrj-Ds](https://www.youtube.com/watch?v=_UZEXUrj-Ds) git rebase -- [https://www.youtube.com/watch?v=CRlGDDprdOQ](https://www.youtube.com/watch?v=CRlGDDprdOQ) git rebase vs git merge -- [https://www.youtube.com/watch?v=wIY824wWpu4](https://www.youtube.com/watch?v=wIY824wWpu4) git cherry-pick -- [https://www.youtube.com/watch?v=QtXj9tt-RUE](https://www.youtube.com/watch?v=QtXj9tt-RUE) Creating and applying git patches -- [https://www.youtube.com/watch?v=vSsypsDRiMU](https://www.youtube.com/watch?v=vSsypsDRiMU) Tags -- [https://www.youtube.com/watch?v=ecK3EnyGD8o](https://www.youtube.com/watch?v=ecK3EnyGD8o) useful techniques -- [https://www.youtube.com/watch?v=P3ZR_s3NFvM](https://www.youtube.com/watch?v=P3ZR_s3NFvM) git bisect diff --git a/kdlp.underground.software/slides_src/hardware_io.md b/kdlp.underground.software/slides_src/hardware_io.md deleted file mode 100644 index 393143ca..00000000 --- a/kdlp.underground.software/slides_src/hardware_io.md +++ /dev/null @@ -1,181 +0,0 @@ ---- -theme: white ---- - -# Hardware -## In Linux -[LDD3](https://www.amazon.com/Linux-Device-Drivers-Jonathan-Corbet/dp/0596005903): [Chapter](https://lwn.net/Kernel/LDD3/) [9](https://static.lwn.net/images/pdf/LDD3/ch09.pdf) - ---- - -# Device Registers -- Peripheral devices are controlled by reading from and writing to their registers. -- The registers are accessed either through memory address space or I/O address space, both of which are equivalent interfaces in hardware, reading from and writing to the address bus, the control bus, and the data bus. - ---- - -# Compiler Optimizations -- I/O registers have side effects, while memory registers do not. -- Because of this, developers need to be careful of compiler optimizations that can change the order of read/write I/O operations. -- To prevent this issue, use memory barriers - ---- - -# Memory Barriers -- Read/write operations that appear before a relevant memory barrier execute before any read/write operations that appear after it - - rmb() - Read memory barrier - - wmb() - Write memory barrier - - mb() - Memory barrier (for both) - - read_barrier_depends() - weaker rmb -- read_barrier_depends() only blocks reordering reads when they have data dependencies upon one another - ---- - -# I/O Port Allocation -- To interact with device registers, the device is accessed through an I/O port -- To use the I/O port, one must first acquire exclusive access to it using: -```console -struct resource *request_region(unsigned long first, unsigned long n, const char *name); -``` -- This requests ports starting with , for the device called . -- When you’re done with the ports release them using: -```console -void release_region(unsigned long start, unsigned long n); -``` - ---- - -# Manipulating I/O Ports -- To read from and write to I/O ports, you must use the appropriate sized function: - - unsigned inb(unsigned port) - read one byte from an 8-bit port - - void outb(unsigned char byte, unsigned port) - write one byte to an 8-bit port - - unsigned inw(unsigned port) - read one word from a 16-bit port - - void outw(unsigned port) - write one word to a 16-bit port - - unsigned inl(unsigned port) - read one long from a 32-bit port - - unsigned outl(unsigned port) - write one long to a 32-bit port -- No 64 bit port operations are defined, 64-bit architectures use a maximum of 32-bit operations on I/O ports - ---- - -# Manipulating I/O Ports in User Space -- inb(/w/l) and outb(/w/l) can be used in user-space as they’re defined in , to use them a program must be run as root, compiled with “-O” to force extension of inline functions, and acquire permissions using: - - ioperm - to get permissions for individual ports - - iopl - to get permissions for the entire I/O space -- Alternatively, user programs can access I/O ports using their device files in /dev/ - ---- - -# String Operations on Ports -- I/O ports can be read from and written to using C-strings using the following functions: -```console -void insb(unsigned port, void *addr, unsigned long count); -void outsb(unsigned port, void *addr, unsigned long count); -void insw(unsigned port, void *addr, unsigned long count); -void outsw(unsigned port, void *addr, unsigned long count); -void insl(unsigned port, void *addr, unsigned long count); -void outsl(unsigned port, void *addr, unsigned long count); -``` -- When using string operations, look out for mismatches between byte-ordering rules of the host system and the I/O device (little endian vs big endian) - ---- - -# I/O Pausing -- If the device is slower than your processor, it can be overclocked and miss data, to solve for this, use pausing I/O functions which implement a wait in between I/O operations. - ---- - -# Platform Dependencies -- I/O port operations are highly platform dependent -- For example, a port can be unsigned short in some architectures and unsigned long in others - ---- - -# Using I/O Memory -- A popular alternative to I/O ports is I/O memory -- I/O memory is RAM regions that a device makes available to the processor through the bus -- Technically, it’s possible to access I/O memory using pointers, much like regular memory, but this is highly discouraged for security and stability concerns - ---- - -# I/O Memory Allocation and Mapping -- I/O memory regions must be allocated prior to use, using: -```console -struct resource *request_mem_region(unsigned long start, unsigned long len, char *name); -``` -- This allocated a long memory region starting at for device -- When you’re done with a memory region, release it using: -```console -void release_mem_region(unsigned long start, unsigned long len); -``` - ---- - -# Accessing I/O Memory -- To access I/O memory regions, one must obtain their address using: -```console -void *ioremap(unsigned long phys_addr, unsigned long size); -void *ioremap_nocache(unsigned long phys_addr, unsigned long size); -``` -- The nocache version of ioremap gives a pointer to a read/write interface with no data caching, which drivers may sometimes need depending on the device - ---- - -# Read/Write I/O Memory -- To read from and write to I/O memory, use the functions: -```console -unsigned int ioread8(void *addr); -unsigned int ioread16(void *addr); -unsigned int ioread32(void *addr); -void iowrite8(u8 value, void *addr); -void iowrite16(u16 value, void *addr); -void iowrite32(u32 value, void *addr); -``` -- Where addr is the address obtained from ioremap. -- There are also versions of all of these functions that allow repeating read/write that look like: -```console -void ioread8_rep(void *addr, void *buf, unsigned long count); -``` -- Note: count is expressed as number of relevant data units (8-bit/16-bit/32-bit) - ---- - -# Read/Write I/O Memory -- You can also read from and write to I/O memory as a block using: - - Fill a memory block with a repeating byte -```console -void memset_io(void *addr, u8 value, unsigned int count); -``` - - Copy from I/O memory to regular memory bytes -```console -void memcpy_fromio(void *dest, void *source, unsigned int count); -``` - - Copy from regular memory to I/O memory bytes -```console -void memcpy_toio(void *dest, void *source, unsigned int count); -``` - ---- - -# Read/Write I/O Memory -- There are also old functions discouraged from use to operate on I/O memory: -```console -unsigned readb(address); -void writeb(unsigned value, address); -unsigned readw(address); -void writew(unsigned value, address); -unsigned readl(address); -void writel(unsigned value, address); -``` - ---- - -# Ports as I/O Memory -- The kernel provides functions to map I/O ports as I/O memory regardless of their actual implementation on the device: -```console -void *ioport_map(unsigned long port, unsigned int count); -``` -- This converts ports starting at to appear as I/O memory -- Once you’re done with the ports, unmap the I/O memory using: -```console -void ioport_unmap(void *addr); -``` diff --git a/kdlp.underground.software/slides_src/interrupts.md b/kdlp.underground.software/slides_src/interrupts.md deleted file mode 100644 index a5afa94d..00000000 --- a/kdlp.underground.software/slides_src/interrupts.md +++ /dev/null @@ -1,99 +0,0 @@ ---- -theme: white ---- - -# Interrupt Handling -[Chapter 10](https://static.lwn.net/images/pdf/LDD3/ch10.pdf) - ---- - -## An interrupt is simply a signal that the hardware can send when it wants the processor’s attention - ---- - -# The /proc Interface -- Reported interrupts are shown in /proc/interrupts -root@montalcino:/bike/corbet/write/ldd3/src/short# m /proc/interrupts -CPU0 CPU1 -0: 4848108 34 IO-APIC-edge timer -2: 0 0 XT-PIC cascade -- First column shows IRQ number -- The last two columns give information on the programmable interrupt controller that handles the interrupt - ---- - -# Installing an Interrupt -- Must tell kernel to expect an interrupt, if not, it simply acknowledges the interrupt and ignores it -- int request_threaded_irq(unsigned int irq, irq_handler_t handler, irq_handler_t thread_fn, unsigned long flags, const char \*name, void \*dev); -- thread_fn can be NULL if you only need a top half -- Drivers can share handlers - ---- - -# Top and Bottom Halves -- Top half runs asynchronously (not in process context) with interrupts disabled -- It has to make a snap judgement about the interrupt -- The bottom half is a regular function to be executed later when we have more time -- interrupts are enabled during execution of the bottom half - ---- - -# Implementing a Handler -- Can’t transfer data to and from user space. Doesn’t execute in context of a process -- Can’t sleep -- First step is often acknowledging the signal from the hardware -- Might awaken processes sleeping if the interrupt signals an event they are waiting for -- Should execute in a minimum amount of time - - Use a bottom half for long computations - ---- - -# Enabling and Disabling Interrupts -- void {disable_irq, disable_irq_nosync, enable_irq}(int irq); - - Used to disable a single interrupt -- void local_irq_save(unsigned long flags);, void local_irq_disable(void); - - Diable all interrupts -- void local_irq_restore(unsigned long flags);, void local_irq_enable(void); - - Restores interrupts - ---- - -# Interrupt Sharing -- Modern hardware has been designed to allow the sharing of interrupts; PCI bus requires it -- Linux supports interrupt sharing on all buses - ---- - -# Installing a Shared Handler -- The SA_SHIRQ bit must be specified in the flags argument when requesting the interrupt -- The dev_id argument must be unique. Any pointer into the module’s address space will do, but dev_id definitely cannot be set to NULL -- irq_request succeeds when one of the following is true: - - The interrupt line is free - - All handlers already registered for that line have also specified that the IRQ is to be shared - ---- - -# Running the Handler -- use the dev_id argument to determine which, of possibly many, devices might be interrupting - ---- - -# Handler Arguments and Return Value -- typedef irqreturn_t (\*irq_handler_t)(int irq, void \* dev_id); -- Irq is the interrupt number, dev_id is a short of client data -- Interrupt handlers should return a value indicating whether there was an actual interrupt to handle - - IRQ_HANDLED if the IRQ was able to be handled by the top half - - IRQ_WAKE_THREAD if the bottom half shold be scheduled - - IRQ_NONE if there was nothing to handle (or IRQ wasn’t from device) - ---- - -# Interrupt Driven I/O -- Buffer in case of read/write delay -- Improve performance by handling part of action asynchronously -- Read: - - fill buffer from device during interrupt - - copy to user from buffer on system call -- Write: - - copy from user to buffer on system call - - write from buffer to device on interrupt diff --git a/kdlp.underground.software/slides_src/kernel_compilation.md b/kdlp.underground.software/slides_src/kernel_compilation.md deleted file mode 100644 index 96e1d7b5..00000000 --- a/kdlp.underground.software/slides_src/kernel_compilation.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -theme: white ---- - -# How can I, a regular citizen, compile the Linux Kernel -- Start by getting the Kernel from its source, Linus Torvalds git repo - - [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git) -- You can download the source code as a zip from the website or by using the git clone command - - git clone "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git" - ---- - -- Once you have the source code, you need to acquire all of the necessary tools to compile the kernel. A short list of most of them is available inside the Kernel repository in the file Documentation/Changes -- On fedora you can install them with the command: - - sudo dnf -y install gcc flex make bison openssl openssl-devel elfutils-libelf-devel ncurses-devel bc git tar dwarves -- On other distributions you will need the same dependencies but the packages might have slightly different names - ---- - -- The Kernel is extremely configurable, but most of the settings are not something you want to worry about. To make a kernel that will be compatible with your existing OS, you want to base the configuration on those from your current kernel. - - Run the command cp /boot/config-$(uname -r)\* .config from within the linux folder -- Now generate a new config based on those options. - - Run make olddefconfig to copy the existing settings, and automatically pick default values for new options -- You’re now ready to begin the actual build. Run this command: - - make -j$(nproc) vmlinux modules - -Expect this to take a few hours! And crash many times! - ---- - -- During the compilation process, you will inevitably run into issues like missing dependencies. If make can’t find some command, try and install it using dnf. If that doesn’t work, try googling the error to see if someone else has encountered it. Send us a message on discord if you get stuck. This is tedious but to be expected. -- Once make finishes, you have officially compiled the Kernel. But… what good is a Kernel if you aren’t using it? -- To install the kernel, run this command: - - sudo make modules_install install -- You now have built and installed the kernel! Congrats! - ---- - -# Useful Resources -- Fedora has a guide on installing linux: - - [https://docs.fedoraproject.org/en-US/quick-docs/creating-and-using-a-live-installation-image/](https://docs.fedoraproject.org/en-US/quick-docs/creating-and-using-a-live-installation-image/) -- They also have a guide on how to build your own kernel: - - [https://docs.fedoraproject.org/en-US/quick-docs/kernel/build-custom-kernel/](https://docs.fedoraproject.org/en-US/quick-docs/kernel/build-custom-kernel/) -- Bootlin - browsable linux source code - - [https://elixir.bootlin.com/linux/latest/source](https://elixir.bootlin.com/linux/latest/source) - diff --git a/kdlp.underground.software/slides_src/kernel_data_types.md b/kdlp.underground.software/slides_src/kernel_data_types.md deleted file mode 100644 index 928580db..00000000 --- a/kdlp.underground.software/slides_src/kernel_data_types.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -theme: white ---- - -# Data Types in the Kernel - ---- - -# Use of Standard C Types -- Standard C data types aren’t the same size on all architectures - - Raises portability issues - - e.g long: - - x86: sizeof(long) == 4 - - x86_64: sizeof(long) == 8 -- Example: in misc-progs - - Program shows different data types and their sizes -- Unused by linux: {u,}intptr_t - ---- - -# Assigning an Explicit Size to Data Items -- Sometimes kernel code requires data items of a specific type - - u8, u16, u32, u64 - - Corresponding signed types exist but rarely used -- Double underscore used in user space - - __u8, __u32, … -- Compare to stdint.h in userspace - ---- - -# Interface-Specific types -- Most of the commonly used data types in the kernel have their own typedef statements. Prevents portability problems - - pid_t instead of int -- typedef defined types now less fashionable - - devs would rather see the real type information - - Hence things like: struct task_struct - - Ambiguity: How to printf \*_t types? - ---- - -# Other Portability Issues : Time Intervals -- Don’t assume that there are 1000 jiffies per second - - This is true for i386 platform -- Instead the time using HZ -- For example to check a against a time out of half a second : - - HZ / 2 - ---- - -# Other portability Issues : Page Size -- Memory page is PAGE_SIZE bytes and not necessarily 4 KB -- Don’t hard code the value, might diff size depending on platform -- To (portably) get a 16 KB page for temporary data : -```console -#include -int order = get_order(16\*1024); -buf = get_free_pages(GFP_KERNEL, order); -``` - ---- - -# other Portability Issues : Byte Order -- Don’t make assumptions about byte ordering - - Some are little-endian and others big-endian - - Most PCs store multibyte values low-byte first and other platforms do this the other way around -- u32 cpu_to_le32(u32) and u32 le32_to_cpu(u32) - - Above macros do conversions between little-endian and big-endian depending on the processor - ---- - -# Other Portability Issues : Data Alignment -- Support for unaligned data access varies by platform -- Use: {get,put}_unaligned() -- Better to align your data - - FYI: Compiler may rearrange and add padding - ---- - -# Pointers and Error Values -- Returning a NULL pointer when a function fails doesn’t communicate what the problem is -- Return an error code -- Error code is encoded in a pointer - - void \*ERR_PTR(long error) - - Use IS_ERROR to check if returned pointer is error code - - long IS_ERR(const void \*ptr); -- long PTR_ERR(const void \*ptr); - - Extract actual error code - ---- - -# Linked Lists -- Kernel developers developed standard circular doubly linked list to prevent duplication of code -- struct list_head is used to maintain a list of data structures - - Standard circular doubly linked list -- INIT_LIST_HEAD - - Used to initialize list head prior to use -```console -struct list_head todo_list; -INIT_LIST_HEAD(&todo_list); -``` - ---- - -# Linked Lists -![ll image](images/Kernel_Data_Types/ll.png) -[Source](https://static.lwn.net/images/pdf/LDD3/ch11.pdf) diff --git a/kdlp.underground.software/slides_src/linux_crash_course.md b/kdlp.underground.software/slides_src/linux_crash_course.md deleted file mode 100644 index e3659a36..00000000 --- a/kdlp.underground.software/slides_src/linux_crash_course.md +++ /dev/null @@ -1,197 +0,0 @@ ---- -theme: white ---- - -# Kickoff -Linux crash course - ---- - -# What is Linux -- Free and open source operating system - - Licensed under the GNU GPL v2.0 -- The focus of this class - ---- - -# Where is Linux -- All top performing supercomputers -- Most embedded systems -- Android phones -- Almost all of the computers powering the internet -- Not many home computers - - Unless you count chromebooks or the steam deck -- On kernel.org, freely available - ---- - -# Who is Linux -- Linus Torvalds - - Created the Linux Kernel in 1991 while studying CS at the University of Helsinki as a personal project - - Originally he supported only his own hardware. - - "it probably never will support anything other than AT-harddisks, as that's all I have" [[0]](https://www.cs.cmu.edu/~awb/linux.history.html) - - Also eventually created git to act as the VCS for the kernel -- Open source community project - - Thousands of individual contributors - - Corporate sponsors (including Red Hat) - - The Linux Foundation - ---- - -# Why use Linux -- Free software (free as in free speech) - - No spying, telemetry, adware etc in most mainstream distributions - - Can inspect and change the code as you please -- Fully customizable - - Different choices for things like desktop environments, GUI programs, etc -- Can run on less powerful computers - ---- - -# Modularity -- An OS made from many interchangeable components -- Kernel a (relatively) small piece -- Distribution (Distro) comes with choices for some of these pieces - - Desktop environment - - Package manager - - Default Applications - ---- - -# Distro vs Distro -- Differences in package naming conventions - - libfoo-dev (apt) vs libfoo-devel (dnf) -- Differences in target audience - - Servers vs desktop computers - - Gaming vs Software development -- Differences in philosophy - - Easy to use with sane defaults vs configure it all yourself - - FOSS only vs access to proprietary software - ---- - -# Exclusively Open Source -![fedora logo](images/Linux_Crash_Course/Fedora-logo.png) -[source](https://upload.wikimedia.org/wikipedia/commons/thumb/b/bd/Fedora-logo.svg/2048px-Fedora-logo.svg.png) - ---- - -# Beginner-friendlyish -![ubuntu logo](images/Linux_Crash_Course/ubuntu_logo.png) -[source](https://www.xilinx.com/content/xilinx/en/products/design-tools/embedded-software/ubuntu/_jcr_content/root/parsysFullWidth/xilinxflexibleslab/xilinxflexibleslab-parsys/xilinxcolumns_149128/childParsys-2/xilinximage.img.png/1629757312962.png) - ---- - -# Do anything, but by hand -![archlinux logo](images/Linux_Crash_Course/arch_linux_logo.png) -[source](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAYYAAACBCAMAAADzLO3bAAAArlBMVEX///8Xk9FNTU0Ai84Ajs8Aic1HR0cAj88/Pz9GRkZKSkoAiM09PT06Ojo2NjZBQUHl8PiJiYnS5fNra2vz+Pzg7ffL4fHn5+eTw+T4+/3MzMygyueKiorFxcXv7+/q8/mr0Orc3NyBgYG31uxnrtyYmJh9uODE3fBGodZ1dXXX19empqZeXl7P4/K7u7ufn5+JvuI0m9RcqtqxsbFjY2MuLi5HoddztN5WVlYAf8oaGq5tAAAOCElEQVR4nO2da3uiPBCGQQIICqxn0Xo+a9dW7Vbf///HXgIknDKIlsp2zXP1Q6sB09wkmZlMoiBwcXFxcXFxcXFxcXFxcXFxPZu6zaJrwOUI9YuuAZcgtCWx6CpwCcJclVZF14GrKYviuuhKcHVVUZR/FV2Lp5dDQVSXRdfi2bWSHAyiXHQ1nl1H3BtENC66Hs+tkSx6Kroiz62u6lHgNmuhQn5nEI9F1+SZNaMYuM1aoD4JBW6zFqiWRDGIUtGVeV4tUYCBx1kLU6gzcJu1MI1RmAK3WQuSGBWPsxaidnRMEuVW0TV6Sh3VKAZ1XnSNnlE0nBR0B54b8Hi9ozgG9F50nYg6+9rieqlW/x+wKqQ4BcdYsouulKPJaWoZVvnlWrmxJMnE1/ml/tRYTD/RGf4KF673p2oppVJJuYqhq9L6riQV/dABNQkBcyi6VsLAKrm6BYPtdGx0+PaqfYdmjDHpb1iFoxhq10qGesNIQj/U6Tkwe0PxEY27MAj18zdX65sUd93oJD0ruGL3YfipWgOdQSx6iH0qDM0KhEFqF1uzmzCoPxyDsAI5fBZbsefCAHOQfhdar9swdB9RpW8VyCGj6TfsnRabi2U6Uj5quy2zUGdAf7U7295gwCwV1hNhGInY5YQ4yPXrdxguSmbZ0nTdazNdsarGyyRcotM71V5N44/7x2S3KRvVatmkgaLhYFfbvL5eXhu1XW8Yuo5imJKSnc6QGWP58RjOyF1pqydCrK7UDClLe8MHEJJmToPmajiUFKeM4fy+fTU0r7jpNfhwdzEtTdFdKVrZ/DjRKwkG481BedpopmFUnZ/GW6K7ZcEwatXrrb80zIF3NEjY32kxwkp4drjeHRZaggIGoXVIgY3it6YgvJkEmfeED6emFoOoW+bC7xI+BmMnDDempdASWrW8GEbqEMXQH/m/2L/Ji+OjJEuOZDSPe0OjFcTGfhg0Nx8DqU6HaIosEBm6QyPZGdzGqpKWCjC8GfRtAz/RJ5ON0Dy5F3oYjJOwN5VEkUicKYSh2UUVH4MqSxXvbQnRZS0VhaM0zaMqV5gR2Xp3/Z9vodQZ8YRWngHopj8WSeuWIBxZ3rR0dTXUYDSky+HVL0AwaHsj+ua0ClxaKn/g/9LF4FCYsj5CK/eCOoQwzGRR8jHI3i6BUfwBk4Jna+RMiqR4VEdEDMV+hfEwimKOXSVITpLPI6HLmKivdocOhKFU3nslNqS7WKH3HDtpYzEvc6XgMQ1jsN6GOrPLOLPLiVaiH8KAYhjcPDgVyerncS3K7j+sUgtw5LwpMXvDUfUxuKTiTonTcCiD+ZJNzdDErMrnZh2pCQ7XkgMGtDF1RYmMT/rFK7FJjlq67swpKRRKesl2b61M9yZ70MMc9qQSKRhshP+348x7zV6tMRVE0kMzYBDaDodK1INq41uPb2jpVC0jvdUB0Y6nBlzvDjsNT5qOkao1NpuGblhBqxneLJ3AoGhOA27NFAqlMp5YHAyXaQDL+RDH0A3dn1hbqRjWqijPw8PODD97Ff/hyoJB6DrNJIXHIIwW5ZY00Yxbqaq0Tga9rxhLm7JjZC72E3/KsgevdAyxvKc1ikGrKtPdThBKkVcVq4ybmMzEput34I5GSullZTGYdDrb3aUc3Iv4Hn0EYEArSUSx0Bg2zknmCYxhHmDAKCNxTvdhvdq8WbVkG6lxNumu9CbhNE+VaCuFMehmzfPs9kFjOrzMzak3mWz3U8N9+E3Pww7Gu5KiBE73gM5G1K8LYVhFMIifanIuxctc/g6/dAyEn+00OwqWMcbORZVvmRnSlMmV3u53i8ViN3CGCduItlIIg/ZKvInX0ItaMNcKp7KCHQVXPYpKa4Q/bEI/gEQ5HAwklSSGQVQZFs2aRo+zYRBaeJoe+3/gbCKUn8++TM7HbF2LLE3eLjic4ciyzMaWNrGycd8OMGgbcknIvipPo3fbUJeAYtA/okXetHQMfrtiDDIrWO+U8Z2HZjYMQl8K5hO8ESS/2HMyRQzsDqmB1u2HoUWGnd0GwEA9Ceehp+ONlUhCon2DYjCiPrOwLWfFABgYMsUgZ8MgzPGs7M5/iQn7azpn7Qzp7KeJkBKFEsdgBAE/On8oG+jGAQa9EXsjOwbETiK7HQPOXnFnSXd8ym95+FfmzpC2DGdfAN/KbaXo3KCXgusoOnMI3FkIMCQC3dkxAEGAbBgiDHF7YXdDzNNWdT8mu8BV6Us82JOGIfRQm9EibFEM8UJFYHD9DWm1zNVWFVo3dAZ420ktpS+kYaAztLVn39gV3BuqmTBIuWIQztgfV/O0VZOZ9Hd1h07YE1Ysy4pGrWEMExLUq06YN/b0AAw22GOSGPx8rhxtVWiZB+4OzEnpJegMljE97U+Lj3DoGsawhWygiEAMk2IwNF13N888CTA3CRTrLjRQrRvEyhzWjGIxtAMMKGcM7gZBdlj8PkGJerBYGa2BDxYstTFiDV8flOKpxBQDeYPdG3LHgBsiz737UNZqitTkXah7ZUYak/i4mabotPyMmzCQ5nGesEwY/ARwEMM5iWHuDkpybvtZ2CncV7pDMh1rH8+c8ATFlJgGa1ruC8Gg3YlBzRcDtlhV7Ezn5ULfDkFk7XcgGOJWJ40peRgaDAw0gG2kVDM7hvEDMODwE3rvomsh58waZwpwJzAkDLUBgKFxHcMLDYa/wfV8AAbYxUvMDZ+e43bIbSPUbS4DVWI3HOTlkrBdCoZgKcHsCDFR46lQDJ9Oc4etdLwxDZ9whCOilTx2jbN2umWRGt8cSifaaFPuM2CwqeOnKzGbdVglMVcQQ6cae4ONQfwKBjmKoUUTA9yJlXnX25RCQUUo5d3EXmmCQb+EOsokbrCyMAi1IPtLiUCcWHrVb927MPzOBwM26cMYxCBo7hhMOQT33qExSZXX3dn4/QCaUWp8OxPJQCrpOl0J3ZHnXCdmEBNDKA6im8H8MHzBr/s7cCmG+JLEDRjY8Z8AgwTkrePIZwgDXhVA5FkTcwh1g0ufEslfaB+gHiHHPMgBXaXUjcagYw97C8siEDZ6GoZwIKSkGbXBZDic7Kd+LMRbkYMxGF/GoAa9gZX96kY+Awz4hFo5siQab4pbBa32yCG+S6BDJE7SCDWlbhmGUaYvlHGSURoGIZLUhNPAjWqQp2rh1aDbMPhdtZ4Rg9/2riObLOD+uxSDmxAT8p7xkujXdqRBqz3RNdsuwCGeO7aH0h/NziZ9UIqFZ5M32D0GAz6mIhmoWavoGMIQT5DBL3ztNAsgwB0f64A+k/BcNuwFB3MyqJauYEhNGLPwqPQIDO7oE3u6RgeE3p0plGDAD3/0NE4cayUPbmvZX3WF21w6aB9DYuZn94ZkdgBz+c2cuAGNKxiESRlau6veMTfch8FbhQwva9ldbKvivBUfAx5AaHaMr5UcTNmrZv3dvs10+mS3bvKcEmhBIjEkfiRSUXVzK3zoGTAIwwZjiwq+gRc2/yqGQxYMTXf8lT7dJFe7NZvL3gJbgOHASvBYBkGN2eh3ezlmfg6gFXvMZ3nnwFp1cjn0Lbb5wLp0/DRhLYxBV8ofiQ8RBpdqvEcoRsN3JEAMwxwxkAcOSbIjbyNEZSUEGJYoZKsGOtA871/2yK7f5FcDAW7WuavgYnWiZKdmWr7Zo2tly3mS96aiWVWz5GXfNZw/ymZpemJuTextzDJdO9U1y9zQYhgDzlL+w8SAy5L47FhGFX9YqDu/+m3iDPHAurEsSsEM21KjBjoS8UUOBm8kWsrMRC08VFXu22sCBLjZkz6wQMda/7H3tYvp2Kvm68Ld+7HQp7sB9Y0vl5fTNq3CvbdNCV9uGB8vg1DB3h+j2ljskytDwz9lw3DeoXdddVdkCq0f+8ToE+dj4Gil/w7LMJ93me4oUCU/hHmuSGuv9Vcys33GFfFO3wFwGdge/wxw4hjrP66Gw7QFzauyk5d3esAdhzvonawfFn9hdj7gIUk6LMmgWw8IAq19b2oAGNNjFwdGpZ9+OAWsx5ynZkO+MbC6egSg8RO8vyRoOwMCwlRQ50kEvLluEJjBDW32BNM3+OGgXxCYtAp9tVUsFrsWyVpEIuDNlVkpeXrQJbHe0P3dFSWXJXtDN1cGwXl6YDZgzNlT1ZnQ6qoOiSxHaXCxBIQxxLQdtwly6OA4Pe2jrGbZDsfFENgXUk5FZwTFZTwtjJaVgs8d+6lKSU26CYN3zInQTJzDwpVFKalJ8KDEXiJyz/2BHHyuNHXTUpPAhSNgVldF/h0bd+nKRnToMjDvO79TO55K6XtvwW8mhuFJ3IG7XVe2G0LBjFFK+j068IDGrUrdbqiiyph92arCOF4puIy7DrepDXYGFcnieQwdG2e3+0dJAklA9LjYAmZaVZLm46t2Z727lgEzS+bf4HqDmAvQqoSWWQ9Cb46PbBK5pPk/i1iusDy/bR9dc3xgkOARvuxKrKGp8mF8x31a78mtD+kn/HAFii9AI/l891Ay+5RjXcvP4Vt1V327ycNMsKIL0Ejtfyn9oHWOgiAZyH2hv1qNc6juP6pf4RNupcPXH9jmuxQB673aF8YzjgFWyHOT1/l8R4zdD00Sfg7fSmjXf/0D3wb5TQo8N3mdo9cbAlH8l/T9AB2+AwLWmKTf5nm60L8q33OT8oaARXoET1q6KtWbmL/pe8O67mSNeEjjivCaGxK/z5y3l9h8lXnSUqqasqh+8/erjuYyD2lc0Rmx90fkqvqnlOdJjP+eWv/NHzJ7rgr/7tC/WvOHPaRdHk0C9ZitK4//LC4uLi4uLi4uLi4uLi4urn9F/wP3GRT5wcv9lgAAAABJRU5ErkJggg==) - ---- - -# Open Source Software as a Service -![rhel logo](images/Linux_Crash_Course/RHEL-logo-hat.png) -[source](https://fossforce.com/wp-content/uploads/2021/10/RHEL-logo-hat.png) - ---- - -# System Stack Overview -![system stack diagram](images/Linux_Crash_Course/system_stack_diagram.png) - ---- - -# Bootloader -- Grub -![grub menu](images/Linux_Crash_Course/grub_menu.png) - ---- - -# Kernel -``` -$ cat /home/rms/interjection.txt -I'd just like to interject for a moment. What you're referring to as Linux, -is in fact, GNU/Linux, or as I've recently taken to calling it, GNU plus Linux. -Linux is not an operating system unto itself, but rather another free component -of a fully functioning GNU system made useful by the GNU corelibs, shell -utilities and vital system components comprising a full OS as defined by POSIX. - -Many computer users run a modified version of the GNU system every day, -without realizing it. Through a peculiar turn of events, the version of GNU -which is widely used today is often called "Linux", and many of its users are -not aware that it is basically the GNU system, developed by the GNU Project. - -There really is a Linux, and these people are using it, but it is just a -part of the system they use. Linux is the kernel: the program in the system -that allocates the machine's resources to the other programs that you run. -The kernel is an essential part of an operating system, but useless by itself; -it can only function in the context of a complete operating system. Linux is -normally used in combination with the GNU operating system: the whole system -is basically GNU with Linux added, or GNU/Linux. All the so-called "Linux" -distributions are really distributions of GNU/Linux. -``` - ---- - -# C Standard Library -- Syscall wrappers - - man 2 vs man 3 -``` -$ cat /proc/$$/maps -561fa56c6000-561fa56f5000 r--p 00000000 00:1f 669281 /usr/bin/bash -... -561fa6d44000-561fa705c000 rw-p 00000000 00:00 0 [heap] -... -7f0275133000-7f0275135000 rw-p 001d1000 00:1f 669212 /usr/lib64/libc.so.6 -... -7fffe0211000-7fffe0232000 rw-p 00000000 00:00 0 [stack] -``` - ---- - -# Shared Libraries -- Some examples -``` -$ cat /proc/$$/maps -... -7f027513d000-7f027514b000 r--p 00000000 00:1f 669207 /usr/lib64/libtinfo.so.6.3 -... -7f02751ba000-7f02751bc000 rw-p 00033000 00:1f 669209 /usr/lib64/ld-linux-x86-64.so.2 -... -``` - ---- - -# Storage Layout -``` -$ lsblk -NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS -sr0 11:0 1 1024M 0 rom -zram0 251:0 0 7.7G 0 disk [SWAP] -vda 252:0 0 10G 0 disk -├─vda1 252:1 0 1M 0 part -├─vda2 252:2 0 1G 0 part /boot -└─vda3 252:3 0 9G 0 part /home - / -``` - ---- - -# File system layout (See [FHS](https://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.pdf)) -- `/` → root of file system -- `/root` → home folder for root account -- `/bin` → command binaries -- `/boot` → boot loader files -- `/dev` → device files -- `/proc` → information about running processes -- `/etc` → configuration files -- `/home` → location of users' home folders -- `/lib` → libraries - ---- - -# System Stack Overview -![system stack diagram](images/Linux_Crash_Course/system_stack_diagram.png) - ---- - -# The future -Where will linux go from here? diff --git a/kdlp.underground.software/slides_src/linux_tips_of_great_power.md b/kdlp.underground.software/slides_src/linux_tips_of_great_power.md deleted file mode 100644 index 4252beb3..00000000 --- a/kdlp.underground.software/slides_src/linux_tips_of_great_power.md +++ /dev/null @@ -1,275 +0,0 @@ ---- -theme: white ---- - -# Linux Tips -Cool stuff - ---- - -# How can I, a regular citizen, install Linux -- Flash image to USB and boot -- Dual booting -- VM is an option -- Some devices ship with linux by default (not many) - ---- - -# Quick CLI Review -ls - list the files in current directory -cd - change current directory -mkdir - make a new directory -nano - open nano editor -touch - create a file or update its modification time -chmod - change access permissions -rm - remove (permanently delete) a file -rmdir - remove a directory -pwd - print working directory - ---- - -# CLI tips -- !! - - replaced with last command -- !$ - - Replaced with last argument of last command -- cd - - - cd back to the previous location you were -- man - - read manual for a command - - man -a intro (intro to linux) -- sudo - - run command as root - - sudo !! (run previous command as root if you forgot sudo) - ---- - -# Keyboard Shortcuts -- Ctrl+C → kills running program -- Ctrl+Z → pauses running program -- Ctrl+D → send EOF -- Ctrl+S & Ctrl+Q → start & stop text stream -- Ctrl+R → to search for previous commands (fuzzy finder) -- Ctrl+W → erase current word up to cursor -- Ctrl+V → verbatim input - ---- - -# Useful tools -- Text Editor - - Nano or even vim -- Cscope - - Tool for browsing source code -- Man - - Read the docs (manual) for every BASH command/C library - ---- - -# sed - a text processing utility - Stream EDitor -![sed](images/Linux_ToGP/sed.png) - ---- - -# sed -$ sed -i -e ‘s/ \*\$//’ -e ‘s/[[:space:]]\*\$//’ -e ‘/^\$/d’ file.sh -1. Delete all spaces at the end of every line in a file -2. Delete all tabs at the end of every line in a file -3. Delete all empty lines in a file -^ is the beginning of a line -$ is the end of a line - ---- - -# vim/cscope - text based search utility -- cscope -R (in a parent directory to create a cscope database) -- CTRL+D (to exit when the database is generated) -- Open vim in the directory where cscope.out resides and you can search for text within the tree -- Using cscope you can traverse the files that include the text you searched for - ---- - -# grep - command line util to find text -You can pipe the output of a command into grep or give a path to an input file -dmesg | grep "text" or grep "text" file.sh -If you want to find file.txt in your home directory -ls ~ | grep file.txt #not entirely correct -. → any character in regex -ls /home | grep -F file.txt #-F means no regex, just text -^ → start of line -$ → end of line - ---- - -# awk - a text processing utility -![awk](images/Linux_ToGP/awk.png) - ---- - -# Basic Scripting - ---- - -# Quick Variable Review -$0...9 - $0 is filepath of the script and \$1...9 are the arguments -\$\# - the number of arguments -\$@ - an array of arguments -\$\? - process’ exit code - ---- - -# Arithmetic Operations -x=1 -z=2 -\$x+\$z #3 -\$z-\$x #1 -\$x*\$z #2 -\$z/\$x #2 -\$x%\$z #1 -[ $x == $z ] #false -[ $x != $z ] #true - ---- - -# Relational Operations -x=1 -z=2 -[ $x -eq $z ] \#false (they aren’t equal) -[ $x -ne $z ] \#true (they are unequal) -[ $x -gt $z ] \#false (x is not greater than z) -[ $x -lt $z ] \#true (x is less than z) -[ $x -ge $z ] \#false (x is not greater than or equal to z) -[ $x -le $z ] \#true (x is less than or equal to z) - ---- - -# Boolean Operations -x=1 -z=2 -[ ! true ] #false -[ $x -gt $z -o $x -ne $z ] \#true -[ $x -ne $z -a $x -lt $z ] \#true - ---- - -# String Operations -x="1" -z="2" -[ $x == $z ] \#false -[ $x != $z ] \#true -[ -z $x ] \#false (“1” is not of length zero) -[ -n $x ] \#true (“1” is of length non-zero) -[ \$x ] \#true (\$x is not an empty string) - ---- - -# File Operations -file=”secrets.txt” -$ ls -l secrets.txt ---w------- 1 kdlp kdlp 106 Aug 22 04:01 secrets.txt -[ -d \$file ] \#false (\$file is not a directory) -[ -f \$file ] \#true (\$file is a regular file) -[ -r \$file ] \#false (\$file is not readable) -[ -w \$file ] \#true (\$file is writable) -[ -x \$file ] \#false (\$file is not executable) -[ -s \$file ] \#true (\$file has size greater than zero) -[ -e \$file ] \#true (\$file exists) - ---- - -# Getting set up -$ touch file.sh -$ chmod 744 file.sh (chmod changes the permissions that people have for this file) --rwxr--r-- (new permissions after chmod, now you can execute file.sh) -r: read w: write x: execute -$ ls -l -Run ^ to check permissions in cwd -First permissions index is a directory marker -Then 1-3: owner permissions 4-6: group permissions 7-9: other user permissions - ---- - -# Getting set up -To write a bash script you can execute direction, you will need to include a shebang line to indicate that bash should be used to interpret the program -On the first line you write \#!/bin/bash -Now you can use all commands you can use in your CLI in combination with conditionals, loops, and more - ---- - -# Command-line Arguments -You can reference the arguments passed to your script with \$i where i is the index of the argument when the script is run -\$ ./script.sh dog cat -\$0 \#./script.sh -\$1 \#dog -\$2 \#cat - ---- - -# If statements: integers and strings -![if statements](images/Linux_ToGP/if_statements.png) - ---- - -# Loops -![loops](images/Linux_ToGP/loops.png) - ---- - -# Functions -Functions in a bash script are declared with -foo(){...} -or -function foo(){...} -Function arguments come after calling the function, like passing arguments to a script e.g. foo arg1 arg2 ... -Within foo() → $1=arg1, $2=arg2, ... - ---- - -# A && echo "true" || echo "false" -‘echo true’ fires if the prior command succeeds (echo $? = 0) -‘echo false’ fires if the prior command fails -What is echoed? - ---- - -# Debugging / Accessing Linux systems - ---- - -# Remote Access -- Cockpit - - https://your.computers.ip.addr:9090 -- ssh - - Is often enabled by default - - Setup ssh keys so you can log in without a password - ---- - -# TTL cable -- Used to connect to serial console ports -- Microcontrollers, Raspberry Pi, WiFi routers ... -- Useful for low level debugging -- DO NOT HAVE THE TTL RED WIRE AND USBC POWER SOURCE PLUGGED IN AT THE SAME TIME - - YOU WILL FRY YOUR RPI -[purchase](https://www.adafruit.com/product/954) - ---- - -![ttl cable](images/Linux_ToGP/ttl_cable.png) - ---- - -![ttl wiring](images/Linux_ToGP/ttl_wiring.png) - ---- - -# Tracing and bpftrace -- Tracing is most often used for debugging purposes. In the kernel, debugging can become overwhelming without the proper tools. -- bpftrace is a tracing language which relies on the Linux BPF system and built-in Linux tracing capabilities, such as: kprobes(kernel level), uprobes(user-space level), and tracepoints -- Which are all types of breakpoints in a stream of processes - ---- - -# Tracing and bpftrace -- Some programmer determined action can be performed at these breakpoints, using bpftrace -- Refer to [https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md](https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md) -- For a bpftrace tutorial and reference guide diff --git a/kdlp.underground.software/slides_src/memory_allocation.md b/kdlp.underground.software/slides_src/memory_allocation.md deleted file mode 100644 index bdabdc8a..00000000 --- a/kdlp.underground.software/slides_src/memory_allocation.md +++ /dev/null @@ -1,108 +0,0 @@ ---- -theme: white ---- - -# Allocating Memory -[Chapter 8](https://static.lwn.net/images/pdf/LDD3/ch08.pdf) - ---- - -# Story of kmalloc -- Similar to malloc -- Allocates contiguous memory - ---- - -# The Flags Argument -void \*kmalloc(size_t size, int flags); -- GFP_ATOMIC - - Allocate memory outside of a process context. Never sleeps -- GFP_KERNEL - - Normal allocation of kernel memory -- GFP_USER - - Used to allocate memory for user-space pages. It may sleep - ---- - -# Memory zones -- DMA-capable memory - - Used for DMA access -- Normal memory - - Normal allocation -- High memory - - Used to allow access to large amounts of memory on 32 bit platforms -- __GFP_DMA (only the dma zone is searched) -- If no flags present, DMA and and normal memory are searched -- __GFP_HIGMEM (all 3 zones are searched) - ---- - -# Size Argument -- Kernel memory available page sized chunks -- Kernel creates set pools of memory objects of fixed size -- Allocation is handled by returning a memory chunk from a sufficiently large pool back to the requester -- memory allocated is fixed size byte arrays and requester will likely get back more memory than asked up to twice as much - ---- - -# Lookaside Caches -- Special memory pool for high volume objects -- kmem_cache_t \*kmem_cache_create(...) - - creates a new cache object that can host any number of memory areas all of the same size, specified by the size argument -- void \*kmem_cache_alloc(kmem_cache_t \*cache, flags) - - Allocate objects from cache object - - cache ( cache created by kmem_cache_create ) -- void kmem_cache_free(kmem_cache_t \*cache, void \*obj) - - Free object memory into a cache -- kmem_cache_destroy(kmem_cache_t \*cache) - - Frees cache data structure; no longer usable - ---- - -# Memory pools -Form of lookaside cache in kernel where memory allocation is not allowed to fail - ---- - -# get_free_page and friends -Used to allocate memory in page sized chunks -- for when driver needs big chunks of memory -- void \*get_zeroed_page(unsigned int flags); - - Returns pointer to a new page and fills the page with zeros -- __get_free_page - - Similar to get_zeroed_page, but doesn’t clear the page -- Flags argument works similarly to kmalloc - - Usually GFP_KERNEL or GFP_ATOMIC is used -- free_page(...), free_pages(...) - ---- - -# vmalloc -Allocates contiguous memory in virtual address space -- Pages not consecutive. Each page retrieved by a call to alloc_page -- Address range used by vmalloc is synthetic. Sets up page tables -- void *vamalloc(unsigned long size); -- void free(void * addr); - ---- - -# Per-CPU Variables -- When a per-cpu variable is created, each processor gets a copy -- Used in networking subsystem. Require no locking and are fast -- DEFINE_PER_CPU(type, name) -- Dynamically allocated - - void \*alloc_percpu(type); - - void \* __alloc_percpu(size_t size, size_t align); - ---- - -# Obtaining Large Buffers -- Allocations of large contiguous memory buffers are prone to failure -- Best way of performing large I/O operations is through scatter/gather operations (chapter 15) - ---- - -# Acquiring Dedicated Buffer at Boot Time -- If you need a huge buffer of physical contiguous memory, request memory at boot time. -- Dirty technique that bypasses memory management policies -- Modules can’t allocate memory at boot time, only drivers directly linked to the kernel diff --git a/kdlp.underground.software/slides_src/mm_dma.md b/kdlp.underground.software/slides_src/mm_dma.md deleted file mode 100644 index 9d0b1c3c..00000000 --- a/kdlp.underground.software/slides_src/mm_dma.md +++ /dev/null @@ -1,250 +0,0 @@ ---- -theme: white ---- - -# Memory Mapping and DMA -## Chapter 15 - ---- - -# Address Types -- vmlinux: Virtual Memory linux -- Programs can allocate far more memory than is physically available -- Address types : - - User virtual addresses - - Physical addresses - - Bus addresses (on some arches: same as physical addresses) - - Kernel logical addresses (often at constant offset from physical address) - - Kernel virtual addresses (superset of logical addresses) -- Convert addresses using: -- __{p,v}a() from asm/page.h - ---- - -# Physical Addresses and Pages -- Physical RAM divided into fixed-size “pages” -- PAGE_SIZE may vary between arches - - On x86_64: usually 4K -- Page tables! - - The processor uses page tables to translate virtual addresses into corresponding physical addresses - - Multi-level memory - ---- - -# Page table diagram -![page table](images/MM_DMA/page_table.png) -[source](https://0xax.gitbooks.io/linux-insides/content/Theory/linux-theory-1.html) - ---- - -# High and Low Memory -- Low memory is memory for which logical address exists in kernel space - - On most systems, all memory is low memory -- High memory: address physical memory outside of virtual address space - - E.g. x86: 32 bit pointer => 4G addressable memory - - But what if we want more? -- Must set up mapping as needed -- Unnecessary on many systems (64-bit) - ---- - -# Memory Map and struct Page -- Struct Page - - Keeps track of info about physical memory -- There is one struct page for each physical page -- Some macros and functions that translate between struct page pointers and virtual addresses - - struct page \*virt_to_page(void \*kaddr); - - struct page \*pfn_to_page(int pfn); - - void \*page_address(struct page \*page); - - void \*kmap(struct page \*page); - - ... -- Useful when dealing with high/low mem situations (x86) - ---- - -# Virtual Memory Areas -- VMA is a kernel data structure used to manage distinct regions of a process’s address space -- Memory area of a process $PID can be seen by - - cat /proc/$PID/maps -- struct vm_area - - A driver that supports mmap(2) needs to initialize a VMA - - This structure corresponds to the memory userspace maps via mmap(2) - ---- - -# The Process Memory Map -- struct mm_struct - - Holds all the other data structures together -- Contains : - - A process’s list of virtual memory areas - - Page tables - - A semaphore and a spinlock - - … -- mm_struct can be shared between processes - ---- - -# The mmap Device Operation -- Allows direct userspace access to device memory -- The mmap(2) ptr is part of struct file_operations -- System call: mmap(caddr_t addr, size_t len, int prot, int flags, int fd, off_t offset) -- File operation: int (\*mmap)(struct file \*filp, struct vm_area_struct \*vma); -- PAGE_SIZE granularity - ---- - -# Using remap_pfn_range -- This function along with io_remap_page_range are used to map a range of physical addresses -- int remap_pfn_range(struct vm_area_struct \*vma, unsigned long virt_addr, unsigned long pfn, unsigned long size, pgprot_t prot); -- remap_pfn_range intended for use when pfn refers to RAM and io_remap_pfn is used phys_addr points to I/O memory - ---- - -# Adding VMA operations -- open(2) and close(2) - - Called whenever a process opens or closes the VMA -- Installation is like struct file_operations -```console -void my_vma_open(struct vm_area_struct \*vma) { … } -void my_vma_close(struct vm_area_struct \*vma) { … } -static struct vm_operations_struct my_vma_ops = { - .open = my_vma_open, - .close = my_vma_close, }; -``` -- Done in mmap method: -```console -vma->vm_ops = my_vma_ops; -``` - ---- - -# Mapping Memory with nopage -- Page not present in mapping (page fault)? Call nopage() to fix it - - struct page \*(\*nopage)(struct vm_area_struct \*vma, unsigned long address, int \*type); -- mremap(2) can change the bounding addresses of a mapped region - - Driver is not notified -- Called when user process tries to access a page that isn’t present in memory -- Returns appropriate struct page \* -- Different struct vma_operations for driver when using nopage - ---- - -# Remapping Specific I/O Regions -- Most drivers only remap small range -- Done by calculating offsets -- User process can always use mremap to extend it’s mapping - - Implement a simple nopage method that causes a bus signal to be sent to the faulting process to prevent extension of mapping - ---- - -# Remapping RAM -- Reserved pages are the only ones that can safely be mapped to user space - - Reserved pages are pages that aren’t available for memory management - - E.g. 640KB to 1G on x86 - ---- - -# Remapping kernel Virtual Addresses -- Virtual address is an address returned by vmalloc - - Mapped in kernel page tables, not user page tables -- No need to check the order parameter - - Vmalloc allocates memory one page at a time. More likely to succeed - ---- - -# Performing direct I/O -- Most I/O operations are buffered through the kernel -- Direct I/O from user space could potentially speed things up - - setup overhead for direct I/O is potential bottleneck -- Not necessary for a char driver -- Done with function: get_user_pages() - ---- - -# Asynchronous I/O -- Allows applications to to do other processing while I/O is in progress - - almost always involves direct I/O -- Necessary struct file_operations entries: - - ssize_t (\*aio_read)(...) - - ssize_t (\*aio_write)(...) - - int (\*aio_fsync)(...) - ---- - -# Direct Memory Access -“DMA is the hardware mechanism that allows peripheral components to transfer their I/O data directly to and from main memory without the need to involve the system processor.” -- LDD3, page 440 - ---- - -# Overview of a DMA Data Transfer -- Synchronous DMA read - - Driver allocates DMA buffer, the hardware transfers data to buffer, process sleeps - - On completion of write, hardware raises interrupt - - The interrupt handler awakens the process and the actual reading takes place -- Asynchronous DMA read - - Hardware raises an interrupt - - Interrupt handler allocates DMA buffer - - Peripheral device writes to buffer and raises another interrupt when done - - Second handler handles new data and wakes up relevant process - ---- - -# Allocating the DMA Buffer -- Buffer Size > PAGE_SIZE: - - Buffer must occupy contiguous pages in physical memory because ISA and PCI carry physical addresses -- Can reserve physical memory with mem= kernel param - - Use ioremap to get a usable virtual address - ---- - -# Bus Addresses -- Device driver using DMA talks to interface bus that carries a physical address -- DMA-based hardware actually uses bus addresses - - Sometimes physical addresses - - Sometimes special other addresses - ---- - -# DMA mappings -A DMA mapping is a combination of allocating a DMA buffer and generating an address for that buffer that is accessible by the device -- Bounce buffers - - Necessary when a device cannot access some area of physical memory, such as highmem -- Cache coherency - - DMA mappings must account for this - - dma_alloc_coherent() -- Other options - - Streaming mappings - - scatter/gather mappings - ---- - -# DMA Pools -- For small, coherent DMA mappings - - E.g. device needs DMA area smaller than one page -- Also used where dma is performed in areas embedded in a larger structure -- dma_pool_{create,destroy,alloc,free}() - ---- - -# Registering DMA usage -- Similar to I/O ports and interrupt registry -- To obtain and release DMA channel: - - int request_dma(unsigned int channel, const char \*name); - - void free_dma(unsigned int channel); -- Take same care of DMA channels as interrupts and I/O ports - - Request channel at open time - - Delaying request allows sharing between drivers - ---- - -# Talking to DMA controller -- Read asm/dma.h for details -- Driver is in charge of configuring the DMA controller - - When read or write is called or for asynchronous transfers -- Asynchronous transfers happen at open(2) or ioctl(2) call -- DMA controller is a shared resource - - Use: {claim,release}_dma_lock() -- Methods: - - set_dma_mode() (read or write) - - set_dma_addr() (set address of buffer) - - set_dma_count() (how many bytes to transfer) diff --git a/kdlp.underground.software/slides_src/modules_drivers.md b/kdlp.underground.software/slides_src/modules_drivers.md deleted file mode 100644 index 6908c446..00000000 --- a/kdlp.underground.software/slides_src/modules_drivers.md +++ /dev/null @@ -1,53 +0,0 @@ ---- -theme: white ---- - -# Kernel Modules and Device Drivers -## In Linux -[LDD3](https://www.amazon.com/dp/0596005903): [Chapters](https://lwn.net/Kernel/LDD3/) [1](https://static.lwn.net/images/pdf/LDD3/ch01.pdf) & [2](https://static.lwn.net/images/pdf/LDD3/ch02.pdf) - ---- - -# Questions to answer -- What are devices and drivers? -- What are Linux kernel modules? -- How are these concepts related? - ---- - -# What is a device? -- Examples: screen, keyboard, mouse, network interface, printer -- The "organs" of the computer -- A component of the computer that the cpu can interact with electronically - ---- - -# What is a device driver? -- Bridge OS/Hardware gap -- Allow users to talk to device -- OS ↔ driver ↔ CPU ↔ devices - ---- - -# Overview -![height:500px](images/Modules_Drivers/modules_drivers_slide9.png) - - ---- - -# What is a Kernel Module -- Snippet of code that can be added to a running kernel - ---- - -# Why Kernel Modules for Device Drivers? -- Keep kernel text/data minimal - - Less error surface -- More customization - - Can add hardware at runtime -- Easier to develop - - Change text/data without reboot - ---- - -# The End diff --git a/kdlp.underground.software/slides_src/timing_scheduling.md b/kdlp.underground.software/slides_src/timing_scheduling.md deleted file mode 100644 index 75d9dd67..00000000 --- a/kdlp.underground.software/slides_src/timing_scheduling.md +++ /dev/null @@ -1,130 +0,0 @@ ---- -theme: white ---- - -# Measuring Time -[Chapter 7](https://static.lwn.net/images/pdf/LDD3/ch07.pdf) - ---- - -# Measuring time lapses -- Timer interrupts -- Generated by system hardware -- Interval programmed at boot time by kernel -- 100 or 1000 interrupts per second for most platforms - - See HZ macro -- jiffies{_64,} (internal kernel counter) - - Number of clock ticks since last boot for a given CPU - - Has nothing to do with wall clock time - ---- - -# Using jiffies -- Read only -- Needed to calculate a future time stamp -- time_{before,after}{,_eq}() -- struct timeval & struct time_spec - - Used to exchange time representations with user space programs - ---- - -# Processor-Specific Registers -- Most modern CPUs have a counter register - - Used to count clock cycles -- Depends on platform but this register may or may not be readable -- Handle overflows if reg is not writable -- rdtsc on x86 - - Read (rd) time stamp counter (tsc) register - - Different on other platforms - ---- - -# Knowing the Current Time -- Drivers use jiffies to calculate time intervals across events - - For example, to tell double-clicks from single-clicks -- Processor specific registers for more precise measurements -- Use do_gettimeofday to get absolute timestamp in kernel - - Uses struct timeval to export time stamp -- Otherwise real world time is best left for user space where the c library offers better support - - clock_gettime(3) for example - ---- - -# Long Delays -[Chapter 7](https://static.lwn.net/images/pdf/LDD3/ch07.pdf) - ---- - -# Long Delays: Approach #1: Busy waiting -- A loop that monitors the jiffy counter -- while (time_before(jiffies, j1)) -cpu_relax(); -- Call to cpu_relax invokes a way of saying you’re not doing much with the processor right now -- If kernel not configured for preemption, loop locks processor for duration of delay -- Avoid: what does relax even mean?? How relaxed?? (not very) -- On x86: cpu_relax <==> no-op - ---- - -# Long Delays: Approach #2: Yielding the processor -- Release the cpu when not interested in it -- while (time_before(jiffies, j1)) -schedule(); -- The current process does nothing but release the cpu but remains in run queue -- On realtime: schedule just re-schedules the caller! - ---- - -# Long Delays: Approach #3: Timeouts -- Ask kernel to do a delay for you -- wait_event _timeout(...); - - Wait queue to wait for some other event but also run within a certain period of time - - Returns after timeout -- wait_event_interruptable_timeout(...); - - Interruptible sleep - ---- - -# Short Delays -- Drivers deal with latencies using short delays -- ndelay, udelay, mdelay - - Nanoseconds, microseconds and milliseconds - ---- - -# Kernel Timers -- Asynchronous execution -- Call f(x,y,z) at some point in the future -- Used to schedule an action to happen later without blocking the current process - - For example, polling a device by checking its state at regular intervals - ---- - -# Tasklets -- Data structure that must be initialized before use. - - struct tasklet_struct { (\*func)(u long); u long data; }; -- Mostly used in interrupt management and run at interrupt time -- Let kernel decide when to run the task -- Run in atomic context - ---- - -# Workqueues -- Similar to tasklets with a few differences -- workqueues run in kernel process context while tasklets run in atomic context -- Kernel code can delay execution of workqueue functions -- Key difference is that tasklets execute quickly and in atomic mode -- Both tasklets and workqueues run on the same processor they were submitted - ---- - -# The Shared Queue -- Used occasionally to submit tasks if in driver -- Tasks may take longer to execute since there’s sharing -- Struct jiq_work. Actual code below: -```console -prepare_to_wait(&jiq_wait, &wait, TASK_INTERRUPTIBLE); -schedule_work(&jiq_work); -schedule( ); -finish_wait(&jiq_wait, &wait); -``` diff --git a/kdlp.underground.software/slides_src/writing_character_drivers.md b/kdlp.underground.software/slides_src/writing_character_drivers.md deleted file mode 100644 index cf11e99f..00000000 --- a/kdlp.underground.software/slides_src/writing_character_drivers.md +++ /dev/null @@ -1,181 +0,0 @@ ---- -theme: white ---- - -# Character Drivers -## In Linux - ---- - -# Live Coding -## Bare bones module - ---- - -# Major and Minor -- Major: maps to driver associated with device -- Minor: which device within the driver - - typedef dev_t - - MAJOR(dev_t) - - MINOR(dev_t) - - MKDEV(major, minor) -- ls -l /dev - - Look for c in first column - ---- - -# Allocating and Freeing Device numbers -- register_chrdev_region - - N minors (static) -- alloc_chrdev_region - - Allocates some minors and a major for a specified number of needed minors -- unregister_chrdev_region - - Frees major and minor number -- /proc/devices - ---- - -# Dynamic allocation -- alloc_chrdev_region vs register_chrdev_region - - Dynamic vs Static - - Affects creation of device nodes - - Static: easy node creation - - Dynamic: bit more complex - ---- - -# Device registration -- Register char dev - - cdev_alloc - - my_cdev->ops = &my_fops; - - cdev_init - - cdev_add -- Remove char dev - - cdev_del -- The (much) older way - - register_chrdev - - unregister_chrdev - ---- - -# Live Coding -## Major + Minor, make it show up in /dev/ - ---- - -# Important data structures -- struct file_operations - - Holds pointer to the kernel methods (read, write, ioctl etc) -- struct file - - Represents an open file (similar to FILE * or a file number in user space) -- struct inode - - Represents a unique file within the directory structure (one or more paths may point to a single inode) - - Many to one (file:inode) - ---- - -# file_operations struct -- owner = THIS_MODULE -The Methods to the Madness -- llseek(2) -- read(2) -- write(2) -- ioctl(2) -- open(2) -- release(2) -- ... - ---- - -# file struct -- NOT the same as C library FILE pointer - - It is its equivalent within the kernel -- Represents an open file -- Important fields - - f_mode (opened for reading, writing etc (what you passed to open())) - - f_pos (current location in file) - - f_flags (flags you passed to open()) - - f_op (pointer to file operations the kernel will use with this file) - - private_data (driver can use this to hold info it needs) - ---- - -# inode struct -- Used by kernel to represent files -- Two important fields - - i_rdev: Contains device number - - i_cdev: Contains pointer to char device - ---- - -# Live Coding -## Stubs for driver functions - ---- - -# Memory usage -- kmalloc (often devm_kmalloc) - - Typically used in probe/init -- kfree (only needed if you aren’t using devm) - - Typically used in remove/exit -- Kinda like malloc/free with extra steps - - No glibc! - - Need to tell the kernel memory zone - - How long can you afford to wait? - ---- - -# Open and Release -- Open - - Initialize device if being used for the first time - - Check for device specific errors - - Update f_op - - Set private data if needed -- Release - - Deallocate anything allocated in open -- Kinda like kmalloc/kfree for the whole thing - ---- - -# Read and Write -- How do we get access to memory in user space? -- Read (write from kernel side) - - copy_to_user -- Write (read from kernel side) - - copy_from _user -- __user to mark a userspace ptr - - Don’t just dereference! - ---- - -# Why not just dereference? -- In process context - - Oops + death on a page fault -- Elsewhere, null ptr deref: - = panic() (big death) (not from system call) -- Unchecked __\* versions of copy_\*_user - ---- - -# Live Coding -## Make it do something! Echo user input - ---- - -# Scull: Simple Character Utility for Loading Local Localities -- Example char driver used throughout LDD3 -- RAM simulation of external storage - - Platform/Hardware independent -- [scull/main.c at master ¡ starpos/scull ¡ GitHub](https://github.com/starpos/scull/blob/master/scull/main.c) - ---- - -# Linux Driver Development with Raspberry Pi - ch4 -- Chapter 4 has examples of complete char dev including miscellaneous character module -- miscdevice - - Simpler way to write char device driver - - Used for sense-hat display - ---- - -# Discussion diff --git a/kdlp.underground.software/slides_src/writing_modules.md b/kdlp.underground.software/slides_src/writing_modules.md deleted file mode 100644 index 32d29e60..00000000 --- a/kdlp.underground.software/slides_src/writing_modules.md +++ /dev/null @@ -1,146 +0,0 @@ ---- -theme: white ---- - -# How to Write Kernel Modules -- Kernel documentation - - Hacking index - - Driver API - - Module building -- Source analysis & books - - https://elixir.bootlin.com - - LDD3 - - The master documentation is the code -- Practice! - ---- - -# Hello World Kernel Module -```console -$ cat hello.c -#include -#include - -MODULE_LICENSE("GPL"); - -static int hello_init(void) -{ - printk(KERN_ALERT "Hello, world\n"); - return 0; -} - -static void hello_exit(void) -{ - printk(KERN_ALERT "Goodbye, cruel world\n"); -} - -module_init(hello_init); -module_exit(hello_exit); -``` - ---- - -# We are not in Kansas anymore -- Goodbye main -- Goodbye C standard library -- Goodbye IEEE754 -- Hello ring 0 ([x86_64](https://en.wikipedia.org/wiki/Protection_ring)) / EL1 ([aarch64](https://events.static.linuxfound.org/images/stories/pdf/lcna_co2012_marinas.pdf)) -- Hello asynchronous execution -- Hello kernel API - ---- - -# Kernelspace vs Userspace -- Small, fixed-size stack -- Each mode can have it’s own address space -- Kernel space handles access to hardware -- Execution is transferred to kernel space after a system call - ---- - -# Reentrant code -- Potential confusion: What is called when? -- Kernel code must be able to run in more than one context -- Mutexes mandatory -![reentrant code](images/Modules_Drivers/modules_drivers_slide14.png) - - ---- - -# Why not stick with userspace? -- Context switches makes response time slower -- Access to hardware requires priveledged instructions - - The most important drivers can’t be handled in user space - ---- - -# Symbols, Versions & Platform dependence -- Kernel symbol table - - Contains address of global kernel items (functions and variables) - - Changes every time the kernel is recompiled - - Modules can access symbols already exported by other code - - Modules can introduce their own exported symbols -- A Module must be recompiled for each version of the kernel that wants to link with - - Module has to be built with same understanding of target processor as kernel - - Easier to distribute driver for general distribution by contributing to the mainline kernel - ---- - -# Compiling Modules (out-of-tree) -- Develop your code in own folder -- Compile by referencing the kernel makefile -- If you build your own kernel, the makefile will already be installed -- To get the makefile for a kernel from fedora run -```console -sudo dnf install kernel-devel-`uname -r` -```` - ---- - -# Hello World Kernel Module continued -```console -$ cat Makefile -obj-m += hello.o - -.PHONY: build clean load unload - -build: - make -C /lib/modules/$(shell uname -r)/build modules M=$(shell pwd) - -clean: - make -C /lib/modules/$(shell uname -r)/build clean M=$(shell pwd) - -load: - sudo insmod hello.ko -unload: - -sudo rmmod hello -``` - ---- - -# Loading and Unloading Modules -- Modules can be added/removed at runtime using insmod and rmmod -- Adding a module calls its init function, and removing it calls its exit function. -```console -$ make load -sudo insmod hello.ko -$ dmesg -t | tail -3 -hello: loading out-of-tree module taints kernel. -hello: module verification failed: signature and/or required key missing - tainting kernel -Hello, world -$ make unload -sudo rmmod hello -$ dmesg | tail -1 -Goodbye, cruel world -``` - ---- - -# Compiling Modules (in-tree) -- Add your code directly to the kernel, and add them to the Makefile(s) -- Compile the kernel -- Init is called at boot - ---- - -# The End diff --git a/orbit/header.html b/orbit/header.html index 12055282..33fe1604 100644 --- a/orbit/header.html +++ b/orbit/header.html @@ -18,9 +18,7 @@

Kernel Development Learning Pipeline

Home - Course Git - FAQ Login