notes

8/26/2017
#########

- How do we write kernel space services?
- How do they differ from user space code?

1. Statically appending a service and part of bZimage.
2. Dynamically creating a modules. - module programming

	Most of service devs, FS , protocols etc prefer this interface of adding
	kmods (dynamically).  Without knowing much existing of source layout and
	using specific set of rules , we can insert a new service into the kernel.

	kmod goes into ring 0. So no APIs, no syscalls, no library invocations.
	Include files of particular modules should be from kernel headers.
	Userspace are typically from /usr/include.

	The files being included are in the following directory:
	/lib/modules/4.12.0-041200-generic/build/include/linux/module.h 

	kmod subsystem is responsible for cleanup and maintain the modules.  Using
	comments is not mandatory. Module is just mechanism (pkg) to insert code
	into kernel space.

	Makefile is not standalone Makefile.
	Take mod.c (mod.o) and make a mod out of it.

	EXPORT_SYMBOL_GPL - can't be used by proprietary modules.

	sysfs - logical file system and you will find called "module".
	We are interested in actually module body.

LDD-1
#####
	Insert the kernel services instead of a function.
	Drivers are virtualized services where applications interact with driver.

	DD in GPFS kernel is basically 2 pieces of code.
		- Interface to the application (kernel specific)
		Need to be aware of kernel architecture.
		- Hardware specific (device/bus specific).
		Need to be aware of how physically the h/w is connected to cpu.
		What protocol to use, directly connected ? via bus?
	
	Kernel specific driver can be implemented in 3 different ways
	These are 3 different models / ways of writing code. In other wods
	3 ways in which aplication can interact with the driver.
	1. Character driver approach (Sync devices).
	2. Block driver approach (Storage devices, flash, usb).
	3. Network driver (Network, WiFi etc).
	
1. Char drivers : Implemeted on all the devices where data exchange happens
synchronously. For eg mouse, keyboard. Not used for bulk data transfer.

Create a device file and allow the application to interact with the driver
using file operations. Kernel is going to view the driver like FS (vfs_ops).
Common API can be used using file* operations.

	Device files - 2 types
	1. Character devices
	2. Block devices


Check /proc/devices - for the available major,minor.
sudo mknod /dev/veda_cdrv c 300 0
	crw-r--r-- 1 root root 300, 0 Aug 26 13:49 /dev/veda_cdrv

Aquiring major number should be done dynamically. Otherwise
while porting may fail. Probe /proc/devices list and whichever
maj,min is free should be used to create a device file. This
will be a requirement in driver portability.


8/27/2017
#########

 alloc_chrdev_region() is used to dynamically reserver <maj,min> touple
 availability and return. This is used instead of register_chardev_region().

 Kernel allows 'n' number of devices to share same major number.  For eg: hard
 drive can have logical partitions. So we will have one device file for each
 logical partition. /dev/sda1, /dev/sda2 etc - they refer to multiple logical
 partitions with same major number but different minor numbers. Minor numbers
 are identifications file files/inodes of device.

 Depending on the type of devices, Maj,Min can be categorized.
 There is a way to create and delete device node can be done using 'devfs'. 
 Udev is generic framework (suite of scripts) that provide creation of device
 files. These are basedon events from hotplug subsystems.

 class_create() and device_create() - 
 device_destroy() and class_destroy() - These are 4 udev apis that are used to
 dyanmically create the device files.

 This will change the device tree and hotplug will generate an event.  udev
 will pickup this event generated from hotplug and creates a device file.  In
 the exit routine, device tree updated (deletion of virtual device) and hotplug
 subsystem will generate an event. udev will pickup this event and device file
 deletion happens.
 Check /sys/class/VIRTUAL - for device tree.
  
 hald - hald daemon maintains list of devices.  lshal - will list of devices
 that are currently plugged in.


8/28/2017
#########
#What is kernel?
	Kernel is set of routines that service the 2 generated events.
	Kernel code can be grouped in 2.
	1. Process events/context.
	2. Device/Interrupt events/context. Higher priority and cpu scheduler
    is disabled.

	Process context routine is of low priority and can be pre-empted.
	The example chr_drv_skel.c - the open is in process context.
	Single threaded driver.

#Deep drive of char dd open.
	open->sys_open->driver_open->char_dev_open()
	(normal file) > vfs open > driver specific open.
	When drivers are implemented, identify if it's concurrent i.e. available
	to 'n' number of applications parallely. Policy has to be coded that way.

	What driver open should consider during the implementation of service.
	1. Verify the caller request and enforce driver policy (concurrent or not).
	2. Verify if the target device (status) is ready or not.
	(Push to wakeup if needed).
	3. Check for need of allocation of resoureces within the driver.
	(Data structures, Interrupt service routines, memory  etc)
	4. Use-count to track the # of driver invocations if needed. May be useful
	during debugging drivers. "current" is pointer to current pcb. So use that
	to print pid of the caller etc. 

#Deep drive of char dd close.
	close->sys_close->driver_close->char_dev_release()
	(normal file close) > vfs close > driver specific close.

	1. Release of the resoureces
	2. Reduce the use-count etc.
	3. Undo basically all the operations done in open.

	Devices can be accessed via ports or memory (mapped, graphics card etc).

#Deep drive of char dd read.
	read->sys_read->driver_read->char_dev_read()
	(normal file read) > vfs read> driver specific read.

	1. Verify read arguments. Do validation on the read request.
	2. Read data from the device.(can be from port, device memory, bus etc).
	3. Whatever data device returns may need to be processed and then
	return to the appliation.
	4. Transfer the data to app buffer.
	copy_to_user will copy transfer the data from kernel to user address 
	space.
	5. Return # of bytes transferred.

#Deep drive of char dd write.
	write->sys_write->driver_write->char_dev_write()
	(normal file write) > vfs write> driver specific write.

	1. Verify / validate the write request.
	2. Copy data from app/userspace to driver specific buffer.
	3. Process data (may be format?)
	4. Write data to hardware.
	5. Increment *ppos
	6. Return nbytes (num of bytes that were written).

	Many frame buffer drivers (char) - they implement seek/lseek. It's not
	possible to move beyond the device space.

# concurrency
	When drivers are concurrent, there is a question of parallel access.  Take
	care of saftey of shared data/resources. So need to make driver routines
	re-entrant. There will be locking functions, etc.

	When drivers and kernel services - when multiple applications request
	request same driver functions parallely. When concurrent function
	guarrantees safty of shared  data, functions are called re-entrant.

	Even n++ is not guarateed to be atomic on all processors. Atomic
	instructions take memory lock and they are CPU specific. The atomic macros
	expand into CPU specific instructions. 

	2 main types of locking mechanism
	1. Wait type which inturn has 2 - Semaphore and Mutex
		If the lock is held and there is a contention on the lock by another
		thread, that thread is put to wait state.
	2. Poll type - Spinlocks.
		They keep spinning or keep polling without any wait. 
	
	mutex is more lightweight than semaphore.

	If there is a contention on a share resource between process context code
	ofthe driver and interrupt context code of the driver (for eg: Interrupt
	handler), use a special variant of spin_lock calls spin_lock_irqsave,
	spin_lock_irq, spin_lock_bh.

	Normal spin_lock need to be used only during contention between driver code
	within process context.

8/29/2017
#########
#ioctl - Used for special operations on device file.
	Special operations are device and vendor specific. So can't pre-declare
	the function pointers for such special operations.
	Up to 255 different cases/functionalities for single IOCTL.

	ioctl_list - will list all the available ioctls

	For implementing ioctl on the device - do the following
	1. Identify the special operations on the device.
	2. For each special ops, create a request command.
		Also create a header file so that applications can use ioctl.
	3. Implement all the cases (for request).

	Use ioctl encoding macros to create unique ioctl-request,command touple.
	There are 4 encoding macros that can be used.
		To arrive at unique numbers easily we use the following macros
   		_IO(type, nr);
   		_IOW(type, nr, dataitem) the fourth item calculated using sizeof 
   		_IOR(type, nr, dataitem)
   		_IOWR(type, nr, dataitem)
	
	inparam - application writes, driver reads. (_IOW,application writes)
	outparm - application reads, driver writes. (_IOR,application reads)

8/30/2017
#########
#ioctl - continued			aquire big_kernel_lock()
ioctl > sys_ioctl > do_ioctl ------------------> fops > ioctl > chr_drv_ioctl
							unlock()
	Because do_ioctl() aquires a big_kernel_lock(), ioctl is rendered single
	threaded. There is no way 2 apps/threads will enter ioctl function
	parallely.

	unlocked_ioctl = this is function pointer in the file_operations is used
	to make ioctl concurrent.

	/usr/src/linux-headers-4.12.0-041200/include/linux/capability.h - this
	defines various priviledge levels. They are called CAP* constants.
	capable() kernel macro - within ioctl implementation can check for the
	privilege levels being allowed or not.

8/31/2017
#########
#Semaphores.
 	Semaphore value=0 means semaphore is locked.
	value = 1, means semaphore is unlocked.

	down_interruptible - will try to aquire the lock and if it doesn't get the
	lock,it will go into interruptible wait state. It will create a wait-queue
	and pcb for the calling context will be enqued to the wait-queue.

	up() - in the write will make the semaphore available. (Unlock call of
	semaphore).

	The wait-queues that are part of semaphore are FIFO.  This type of locking
	using semaphores is discouraged because can lead to confusion.

#completion locks
	Another way to let the callers block if the resource is not available.

#Wait-queue
#	wait_event_interruptible - 
	WIll push the caller into interruptible wait state. Calling process's pcb
	is enqueued into wait-queue.
	wake_up_interruptible() is the routine to wake up.
	This will send a signal to all the PCBs that are in the wait-queue.

9/1/2017
########
	Linux provides a way that driver delivers messages to applications
	asynchronously.  Poll and select are 2 ways that applications can get the
	status of the device.

#async notifications:
	async is not used in all applications. When IO is not high priority, then
	this method is used. This will require applicatons to register with driver
	for async messages to be delivered.

	Applications get SIGIO from the driver and they can handle with special
	signal handler for SIGIO.

	kill_fasync() of sending SIGIO can be put in Interrupt Servie Routine (ISR).

	poll_wait() is a kernel routine that puts calling application into wait
	state (in the wait queue).

#Introducing delays in kernel routines.
	For kernel debugging, we may need to introduce delays.

	Busy wait - Infinite loop and cpu is busy.
	Yield cpu time - schedule() to relinquish the cpu time and is often used.
		this doesn't waste the cpu cycles.
	Timeouts - How long we want to be in delay by providing expiry timer.
		Process resumes after timer expires.


#### Interrupts
	Hardware devices are connected to interrupt controller via special line
	called IRQ. These devices are triggering interrupts using that IRQ line.
	X86 provides 16 lines. You can have up to 256 lines. 

	IRQ line assigned to particular to device need to be known to driver writer.
	IRQ descriptor table is linked list of IRQ descriptor.

9/2/2017
########
	lspci -v will list the pci devices.
	request_irq() is a kernel function that will register the IRQ handler
	with a specified IRQ line. It basically creates an instance of 'irqaction'
	that is hanging off from IRQ descriptor table.
	
	__init my_init() will call request_irq()
	__exit my_exit() will call free_irq();

	do_irq() is responsible for doing ISR routine.
	
	Refer interrupt.h
	Lists the possible flags while writing ISRs.
	If we want our ISR routine to be high priority, we can do IRQF_DISABLED can
	be used so that kernel won't do pre-emption.

	Exceptions are also sychronous interrupts in CPU level.
	Device Interrupts are asynchronous.

	Vector table will contain list of descriptors.
	0-32 - Are exceptions, (page faults etc).
	0-19 - are Watchdogs.
	20-31 - Intel reserved.
	IRQ0 = 32, IRQ1 = 33 (External device interrupts).
	For interrupts between 32-127, Linux has one response function called
	do_irq(). do_irq() is low level routine. For particular interrupt do_irq()
	check if there is a handler or not. ISRs are called from do_irq().
		
	
#How interrupts work in Linux.
	do_irq() is function of process 0. Process 0 (kernel) will respond to
	interrupt. Process 1 (init) is responsible for creation and management of
	processes.

	When interrupts occur, do_irq() queries the interrupt controller and
	finds which IRQ line is triggered. Linux kernel configures do_irq() routine
	as a default response function for all external interrupts. 

	do_irq() is routine of process 0, which is responsible for allocation of
	interrupt stack and invoking appropriate ISR routines. This interrupt stack
	is of 2 types. If kernel is configured to use 8k stack, then there is no
	separate stack. If kernel is configured with 4K stack, then interrupt stack
	is allocated. With 4k, there will be performance hit. By default 8K is the
	size of kernel stack. In Embedded linux and if there is a memory constraint
	4K may be needed.

	1. Find interrupt request line on which interrupt signal was triggered (by
	querying interrupt controller).

	2. Lookup IRQ descriptor table for addresses of registered interrupt
	service routines.

	3. Invoke registered ISRs.

	4. Enable IRQ line.

	5. Execute other priority work. 
	[More to come here]

	6. Invoke process scheduler. Until step6, process scheduler is disabled.

	Interrupt latency is total time spent by the system in response to
	interrupt. If interrupt latency is high, applications performance may be
	impacted. High priority will starve since system is spending more time in
	interrupts. One device may block other devices. When timer interrupt goes
	off, other interrupts are disabled.

Interrupt latency == Total amount of time system spends on the interrupt.
#Factors contributing to interrupt latency.
	a] H/W latency: Amount of time a processor is taking ack interrupt and
	invoke ISR.
	b] Kernel latency:  In Linux/Windows/or when process 0 is responding, how
	much time process 0 takes to start an ISR. This is called kernel latency.
	c] ISR Latency: When ISR routine invoked, how much time it takes.
	ISR are usually referred as INterrupt handlers.
	d] Soft interrupt latency (bottom half).
	e] Scheduler latency.
		e.1] Check if any high pri tasks waiting in queue.
		e.2] Signal handlers for the signals pending.
		e.3] Giving cpu to high pri task.

	RTOS has fixed time latency for interrupts. GPOS do not have fixed time
	latency.

#For NIC, both reception and transmission of pkt will trigger an interrupt.
#Sample pseudo-code for interrupt hander for NIC on 
#reception of pkt (on network device).
	1. Allocate buffer to hold the packet.
	2. Copy from NIC buffer to kernel or vice-versa.
	2. Process the packet, specially phsyical header.
	3. Queue pkt handing over to upper protocol layers.
	
9/3/2017
########
While designing ISRs the following issues are to be considered.
#Don't while writing ISR routine.
	1. Avoid calling dynamic memory allocation routines.
	2. Avoid transferring data (synchronous) between 2 buffer blocks.
	3. Avoid contending for access on global data structures because you need
	to go through locks.
	4. Avoid operations on user space addresses.
	5. Avoid call to scheduler. While ISR is running scheduler is disabled.
	Hence a call to scheduler may result in deadlock and need to be avoided.
	6. Avoid calls to operations which are non-atomic.

#Do's while writing ISR routine.
	1. Use pre-allocated buffers. (skb buffer in network drivers).
	2. Consider using DMA whenever data needs to be transferred between device
	and memory.
	3. Consider using per CPU data wherever needed. 
	4. Identify non-critical work and use appropriate deferred routines to
	execute them when system is idle or other scheduled time.

	If you are doing anything h/w specific within ISR is critical.
	Anything other than this is non-critical from Interrupt perspective.

Linux has bottom halves or RTOS call BH as "deferred functions". Basically
step 3 and 4 (processing and enque of the packets to higher layers) can be
deferred and run. It need not be run with interrupts disabled on the cpu.
ISR 
{
	schedule_bh(func);
}

func()                //Soft IRQ. 
{
	body;
}
	
Soft IRQ == Piece of code that runs with IRQ enabled but scheduler disabled.
THis is run in interrupt context. 

ISR terminates, IRQ enabled, scheduler enabled - Bottom half. 
Bottom half can be of 2 types. They can be running soft interrupts context.
They are called Soft IRQs or Work queue.

9/4/2017
########
#SOFTIRQs

Soft IRQ, tasklet and workqueue are 3 ways to do deffered execution of a
routine while servicing an interrupt. Non critical work can be scheduled.

#Linux 2.6 bottom halves.
1. Soft IRQ: It's a way where routine can be scheduled for deffered execution.
	It's available for static drivers and not dynamic drivers.  This is a
	bottom half which is runs with interrupts enabled, but pre-emption
	disabled.
	Refer include/linux/interrupt.h
	All softirqs are instances of softirq_action structure.
	Maximum of 32 soft irqs.

#Execution of softirq#
#
	1. There are maximum 32 allowed softirqs. There is a per-cpu softirq list.
	If ISR is running on cpu1, then softirq will run on cpu1 and so on.  When
	raise_softirq() is called, specified softirq instance is enqued to the list
	of pending softirqs (per cpu). 
	
	Both top-half and bottom-half need to be on same cpu otherwise there will be
	cache-line problem.

	2. Pending softirq list is cleared (run one by one) by do_irq() routine
	immediately AFTER isr is terminated with interrupt lines enabled. softirq
	run with pre-emption disabled but IRQ enabled. If interrupt goes off during
	softirq, then do_irq() is run as re-entrant and softirq will get
	pre-empted. Softirq pre-emption will happen for nested interrupts. 

	3. when softirqs can run?
	They can be run in 3 different ways. 
		3.1. softirq can run in the context of do_irq().

		3.2. softirq can also run in the after spin_unlock_bh().
		do_irq() can't run softirq, if the spin_lock_bh() is held.

		3.3. Softirq may execute in the context of ksoftirqd (Per cpu kernel
		thread). This is in process context of ksoftirqd - daemon. 

	static int __init() my_init()
	{
		open_softirq(mysoftirq, func)
		request_irq()
	}

	ISR {
		raise_softirq(mysoftirq);
	}

	func {
		..
		..
	}

# Another example of softirq.
# Both need to run on same cpu.
	func()
	{
		/* write to a list */
		while(1)
			raise_softirq(mysoftirq); <<< This is legal, though bad.
	}

	read() {
		spin_lock_bh()
		/* read from list */
		spin_unlock_bh();
	}

	Linux kernel allows softirq routine to reschedule itself within itself.
	It's possible to do raise_softirq() within softirq(). ksoftirqd - will also
	clear the pending softirqs when it gets the cpu slice.

#Question1: Why do we allow to reschedule softirq?
Consider 2 cpus as below
cpu1.							cpu2:
read()							func() 
{								{
	spin_lock_bh()					spin_lock_bh()
	/*read from list */				/*write to the list */
	spin_unlock_bh()				spin_unlock_bh()
}								}

	Suppose cpu1 has aquired the lock and is reading from the list, and
	interrupt is delivered to cpu2. cpu2 has some bh (softirq) func() and it
	tries to aquire the lock. The spin_lock_bh() in func() is going to spin on
	the cpu2 for lock to become available. This cpu2 is in interrupt context
	and going to waste cpu cycles which is not correct. Hence rescheduling of
	softirq is permitted from bh.

	Rescheduling bh is required to relinquish cpu from within bottom half when
	critical resource (like lock) is rquired for bottom half execution is not
	available. 

#Question2: Will softirq (bh) can run on 2 cpus run paralley?
#Yes, because the interrupt can be delivered to different cpus. 
	Use appropriate mutual exclusion locks while writing softirqs. 


#Limitations of softirqs.
	1. Softirqs are concurrent, i.e. same softirq could run on 'n' cpus
	parallely. 
	2. While implementing softirq code using mutual exclusion locks is
	mandatory wherever needed. 
	3. Using locks in interrupt context code will result in variable interrupt
	latency. 

	These are the reasons why softirqs are not available for modules.
	Concurrent execution in bottom half, only then consider softirqs. 

2. Tasklets.
	It's a softirq where tasklet is guarrateed to be serial. Because of serial
	execution,there is no need for locks. Tasklets are dyanmic softirqs that
	can be used from within module drivers without concurrency.  They are
	always executed serially.

	Tasklets = Dyanmic softirqs - concurrency. 

3. Work Queue
These are 3 different ways of deferring routines later during interrupt
processing.

9/5/2017
########
#TASKLETS
	Built on top of softirqs
	Represented by 2 softirqs
	tasklet_struct structure
	Created both statically and dynamically
	Less restrictive sync routines.

#Steps involved in tasklets.
	1) Declare tasklet
	DECLARE_TASKLET(name,func, data)
		OR
	struct tasklet_struct mytasklet;
	tasklet_init(mytasklet, func, data);

	2) Implement BH routine
	void func(unsigned long data);

	3) Schedule tasklet
	tasklet_schedule(&mytasklet); <<< Low priority
		OR
	tasklet_hi_schedule(&mytasklet); <<< High priority - use if you want high.
	
#	When tasklets are run? - Execution policy.  Tasklets are executed using
	same policy that is applied for softirqs since interrupt subystem of kernel
	views a tasklet either instance of type high_softirq or tasklet_softirq.

	Interrupt subsystem guarrantees the following with regards to execution of
	tasklet. 

# Tasklets --- multithreaded analogue of BHs. (From interrupt.h file).

   Main feature differing them of generic softirqs: tasklet
   is running only on one CPU simultaneously.

   Main feature differing them of BHs: different tasklets
   may be run simultaneously on different CPUs.

   Properties:
   * If tasklet_schedule() is called, then tasklet is guaranteed
     to be executed on some cpu at least once after this.
   * If the tasklet is already scheduled, but its execution is still not
     started, it will be executed only once.
   * If this tasklet is already running on another CPU (or schedule is called
     from tasklet itself), it is rescheduled for later.
   * Tasklet is strictly serialized wrt itself, but not
     wrt another tasklets. If client needs some intertask synchronization,
     he makes it with spinlocks.
	
	2 different tasklets between 2 different drivers can be run parallel and
	there may be need for synchronization. Tasklets are strictly serialized but
	not wrt to other tasklets. Inside a tasklet if you are accessing global
	data structure, then locking is required. 

	They can be run in 3 different ways. 
		3.1. softirq can run in the context of do_irq().

		3.2. softirq can also run in the after spin_unlock_bh().
		do_irq() can't run softirq, if the spin_lock_bh() is held.

		3.3. Softirq may execute in the context of ksoftirqd (Per cpu kernel
		thread). This is in process context of ksoftirqd - daemon. 

# Workqueues 
	Workqueues are 2.6 kernel only. Tasklets and Softirqs were there in 2.0.
	Workqueue are instances of work structure. 

# Timer bottomhalf - Executes whenever assigned timeout elapses.

9/6/2017
########
#Memory management in Linux
#There are 2 source directories for memory management.
	1] /usr/src/linux/mm - (Memory manager)
	2] /usr/src/linux/arch/i386/mm - (Memory initializer, runs at boottime)
	ppc/mm, mpis/mm, alpha/mm - Architecture specific source code in the kernel.

Processor views memory in real-mode as single array
Processor views memory in protected-mode as set of arrays/vectors.
	Each frame size is 4K.  The view changes depending on the type of mode.

When does the shift from real to protected mode happens?

# Different types of memory allocation algorithms. 

	Page allocator : THis is primary memory allocator and source of memory.

	slab allocator : Odd size memory allocation and always returns physically
	contiguous memory.
		kmalloc(),returns address and kfree(), frees addess.
		CAn be called from driver or kernel service.
		/proc/slabinfo - can view the slab allocation details.

		Slab allocation allows private cache (per driver/kernel service).
		Default allocators could be called from the cache list.
		kmem_cache_create()
			kmem_cache_alloc() 
			kmem_cache_free()
		kmem_cache_destroy()

	Cache allocator : Need to allocate data structures frequently. Reserver
	some pages as cache of objects so that drivers and FS can pre-create the
	objs and use them. They are not available to application directly.

	Fragmented memory allocator: Odd size requests and source of allocation is
	from various fragments. It's used when allocation request size is large and
	not called from applications directly.

	Boot memory allocators: startup drivers, BSP drivers - they aquire memory
	using boot mem allocator.

9/7/2017
########
	include <linux/mempool.h>
	pcb_task_struct_t, cdev etc - most of them are pre-allocated in pools.
	
	Cache is reserving pages which will be used for allocating memory later.
	Memory pool, create a cache and allocate instances. Mem pool is based on
	cache. Scsi, usb drivers, network drivers etc which are frequenty used
	structures that are used for data transfer are created using memory pool.

	Slab layer allows kernel services to create memory pools that can be used
	for pre-allocation of specific objects.

	1. Create a memory pool.
	2. Aquiring an object from the pool.  (mempool_alloc)
	3. Release the object (mempool_free)
	4. Destroy the pool (mempool_destroy).
	5. Destroy the cache (AFTER the pool destruction) - kmem_cache_destroy

	FS, protocol stack typically use this facility.

	Any request beyond > 128K - kmalloc() may fail. That is because 128K
	physically contigous block for kmalloc.

#mmap, munmap 
	Map IO cache buffer (from filesystem) into memory. 
	If application uses MAP_ANONYMOUS, then filedes is not considered and
	memory is mapped. 
	Can be used on device files. 
	open(/dev/file1) and then mmap().

	(low memory zone, normal zone, highmem zone)
	Implementing mmap callback in a character driver.
	Anything above 896M is called high memory zone.
	Process address space is also in normal zone (< 896M)

	1. Each process in the user space aquires set of pages into which process
	code, data and stack segments are mapped.
	2. Process pcb in the kernel space carries details about pages allocated to
	the process and segments to wihch they are mapped.
	(Refer mm18)

	For each process: (application)
		code segments go into few pages and need not be contiguous. 
			Each vm_area_struct correspond to each contiguous segment.
		stack segments go into few pages
		data segments go into few pages
			Each vm_area_struct correspond to each contiguous segment.

	/proc/pid/maps will give each of these vm_area_structs (block).
	block = set of pages.

# How linux tracks mappings?
	1. mm_struct_instance contains reference to a list of virtual memory blocks
	(vm_area_struct) that are mapped to application's code/data/stack segment.

	2. Each instance of vm_area_struct represents one block of the process
	address space.

	3. mm_struct also carries reference to the process page table with valid
	page table entries (PTEs). For protection reasons, we are mapping into
	different segments.

# When mmap is called, what happens?
	1. Application's mmap request on a file invokes do_mmap kernel routine via
	sys_mmap.
	2. do_mmap() allocates new instance of struct vm_area_struct & fills it with
	details of the new block attributes based on the arguments passed to mmap()
	routine by the application. malloc() calls do_mmap(). This is a key routine.
	3.do_mmap() invokes appropriate mmap support routine assigned to the file
	operations (fops) instance for eg: fops_mmap().

# How the driver's mmap works?
	1. Identify physical memory region (frames) that is required to be mapped.
	2. Map physical memory region to kernel's logical address space which is
	page+offset. [Physical memory == frame+offset].
	3. Set page reservation indicating that I/O activity should be disabled.
	4. Map physical address region to VMA instance.

9/8/2017
########
# Direct Memory Access (DMA) - memory allocation and management.

	Bus specific mode etc - they require dma allocations.

#Address translation using page tables.
	Newer Intel PAE extentions provide 36 bit addresses.  There are 3 patches
	availble which breakup the virtual address space.

	1. Each process carries it's own page table allocated by kernel at the
	process load time.

	2. Page tables contain entries mapping page to valid physical frame.
	Valid	virtual-page	modified	protection page-frame 
	1		140				1			RW			31
	1		20				0			R X			38

	Each entry is called page table entry (PTE).
	3. Processor's MMU (memory mgmt unit) at runtime looks up into page-table
	to translate a logical address to physical address. And reference of
	page-table is loaded into processor's register on every context switch.

# Multi level paging: where it's used?
Linux uses 3-level paging on desktop/server arch and 4-level paging on
NUMA/Enterprise architectures.

#Overhead / Limitation with page tables.
	1. As the number of processes increase kernel needs to set aside around 3MB
	of physical memory for each process to hold PTEs.

	2. Page TAble Indirection is one approach where there is no wastage of
	memory. Swap them out to disk when not required and swap in when required.
	This approach is implemented in many ways. 2-level, 3-level paging etc are
	different ways to manage page tables. The objective here is to enable page
	tables dynamically extensible. 

	3. Processor needs to spend 'n' amount of cycles looking up page-tables
	before the actual operations on the memory can be executed.

#How to optimize
	1. To optimize translation time, cpus provide specific cpu local buffers
	called Translation Lookaside Buffer (TLBs).  

	2. Processor < L1 < L2 < L3 < Memory  - 
	Processor is fastest access.

	3. TLBs can be managed in 2 ways
		3.1. Kernel / software managed: In s/w managed mode each TLB missed
		event triggers an exception which inturn is handled by kernel by
		updating TLB entries from page-tables.
		3.2. Hardware managed: In h/w managed mode each TLB miss event makes
		processor jump into physical page table in memory and load appropriate
		entries into TLB.

	4. Processors also provide high speed data instructions caches to optimize
	program execution by mirroring program's data/instructions to cache.

9/10/2017
########
#Memory Mapped IO (MMIO) Vs Port Mapped IO (PIO)
	PPIO:x86 They have port mapped IO.Separate set of instructions to read.
	In PC platform have 16 bit addressing.

	MMIO:risk	(ARM etc) have memory mapped IO. While memory bus is used
	for IO as well.

/proc/iomem - Details of memory mapped IO. Those devices that are memory mapped.
/proc/ioports - Details of port mapped IO.

#Accessing port mapped devices from user space.
	1. Port mapped addresses can be accessed by applications of Linux and
	kernel space services (modules). 
	2. Apps can access port mapped devices using either of the following ways.
		2.1] RUnning as root application using ioperm() routines.
		2.2] Using special device file using /dev/<..> - Will be disabled in
		future. 
	Refer /home/abk/Desktop/Work/Linux/veda/Code-2.6.34/devio/ioports.c 
	man ioperm, iopl - check 2 manpages.

# Kernel space port mapped device access.
	1. From kernel - you do request_region() and seek permission. Modules
	invoke request_region() routine to check for port access permissions and
	aquire port resource.

	2. Use kernel mode in/out family of functions for reads and writes on the
	IO port.

#Kernel APIs are:
	in[bwl], out[bwl] and string variants ins[bwl] and outs[bwl]

	linux/ioport.h and kernel/resource.c - Audit files.

# What is a bus.
	Bus = DAta path from transferring data between cpu and devices.
	3 kinds of buses
	1. Address bus  = Used to generate addresses.
	2. Data bus (parallel lines to carry data) = Used to transfer data.
	3. Control bus. = Used to transfer control signals. 
	IO bus is connection between IO devices and CPU.

#	Summary of PPIO
#	PPIO can be done in 2 ways
	From user space : 
		ioperm/in/out family of functions
		/dev/port - using driver's read write operations

	From kernel space:
		Using in/out family with request_region()
		Using ioport_map() - New way.

# 	MMIO 
	No way to do memory mapped IO from userspace.

# Memory alignment.
	Refer Documentation/unaligned-memory-access.txt in kernel source.  
	Memory alignment is storing data value at the address which is evenly
	divisible by it's size. Unaligned data causes exceptions on some
	architectures and is not supported on some archs. Intel x86 throws an
	exception when un-aligned data operation is initiated.

9/11/2017
########
#Block Device Drivers 
	Bulk and mass storage - they use block driver model for asynchronous mode.

	Buffer/page cache - cache requested file's data is found. I/O request from
	the applicatino will be synchronous if the data is found in buffer cache.
	If the I/O request doesn't find the data in the buffer/page cache, the
	application is put to wait (block) and a request is made to block layer.

	Block driver is per physical disk.
	There are 2 types of block driver.
		1. Logical block drivers - RAMDISK, emulation without physical device.
		2. Physical block drivers with I/O scheduler. REquest queue will be
		present between block layer and physical block driver that actually
		talks to the disk.
		
	vfs identifies block device as an instance of gendisk for block driver.
	vfs identifies char device as an instance of cdev for char device driver.

9/12/2017
########
#Block Device Drivers - Continued

	Number of bios involved in a request == number of contiguous sectors If
	whole file falls physically in continuous sectors, then there will be just
	one bio and num of bio_vec will be N number physically contiguous sectors.

9/13/2017
########
#PCI and network drivers.
	- PCI is typically found in PCs but not found in SoCs or embedded.
	2 types of bridges exist.
	- PCI to PCI bridge
	- PCI to ISA bridge (PCI2eISA, PCI2PCMCIA etc).
	
	Each PCI device carries 256 bytes of configuration information with some
	registers as part of an EEPROM/FIRMWARE.


	Inside device structure, configuration data is stored.
		First 64 bytes region carries a general header (structure) that
		provides details about device, class, vendor, status register, command
		register, port and memory information and interrupt line number (IRQ
		number). For all devices this header is common/required. REst will be
		device specific registers.

	For each PCI bus, there will be a struct called pci_bus
	For each PCI device, there will be struct called pci_device

	lspci -v will show the details of the pci.
	00:02.0 VGA compatible controller
	bus-id:device-id.function-id. Each PCI device can provide upto 7 functions.

#Steps involved in interaction with PCI devices

	1. Register with PCI bios.
	2. Enable the device. (Push to wakeup state).
	3. Probe device configuration. (IRQs, IO ports/memory regions).
	4. Allocate resources.
	5. Communicate with the device.

# Implementation of PCI NIC drivers (PCI and network code is new).
	1. Register the driver with PCI subsystem (PCI bios).
		pci_register_driver() - This function registers driver with PCI bios.
	
	2. static struct pci_driver nic_driver {
		.name = "nicdriver",
		.id_table = nic_idtable,
		.probe = nic_probe,
		.remove = nic_remove,
	};

	static int __init nic_init(void)
	{
		return pci_register_driver(&nic_driver);
	}

	static int __exit nic_cleanup(void)
	{
		pci_unregister_driver(&nic_driver);
	}

	PCI bios invokes the probe routine of the registered driver when the
	physical device specified in found. Probe is callback() and called when
	device is physically there.

	What we do in the probe function?

# Major steps in NIC driver.
	1. Carry out device initiazation operations.
		1.a] Enable the device using pci_enable_device()
		1.b] Enable bus master ring (if available).
			Bus master ring means, presence of DMA functionality. 
			pci_set_master()
		1.c] Extract IO/Memory information. 
			pci_resource_start()
			pci_resource_len()
			pci_request_regions()
		
		Verify if device's IO/Memory region is in use. 2 drivers should not use
		same registers.  
		Perform memory / IO mapping
		pci_iomap, pci_map() etc.

	2. Reigster the driver with appropriate subsystem. 
		Incase of network driver register it with common device layer.

		Char driver register with VFS
		Block driver registers with block layer and VFS (Optional)
		Network driver registers with Common Netdevice Layer. Int n/w
		terminology is called DLL. this comprises code for Mac layer and
		physical layer.

# Network drivers
	1. Network drivers are not enumerated as device files.
	(no VFS registration, no maj,min numbers).
	2. Net drivers register as an instance of net_device with common net device
	layer (CNDL)
	3. ifconfig is a userspace tool that shows all registered net_device
	objects.

9/14/2017
########
#Steps of network driver registration.
	Physical layer protocols define how frame needs to be created and
	transmitted. (Eth, Wireless, FDDI etc)

	1.  Wireless lan device driver, Eth driver etc  will register as net_device
	irrespective of the MAC Protocol it's supporting. The network driver is
	independent of physical layer protocol.
		1.a] Create an instance of type struct net_device. (alloc_netdev)
		INitialize the fields with device specific configuration.

		net_device instance uses a void pointer to refer to driver specific
		private structure which usually contains driver configuration info.
		(includes device usage statistics)

		1.b] Fill config data of network device to net_device fields.
		[pdev, iobase, reqs_len, irq etc]

		1.c] Initialize driver interface functions to function pointers of
		net_device instance. Register net_device with CNDL. (Mandatory step).

		ifconfig up eth0 << open will get from netdev_ops (structure that
		contains function pointers).

	2. Register ISR 

	3. Allocate transmission and reception DMA buffers within nic_open().

# Important note on addresses, DMA issues (bus,cpu,io)
	p = kmalloc() - which is page + offset (logical address).  CPU can
	translate this into frame+offset (physical address) using TLB, translation.
	But such allocation within NIC driver,  can't translate this to physical
	address.

	Whenever we need DMA, we need 2 addresses for the memory. One is physical
	address and one is logical address. In X86 architecture bus address, IO
	address and cpu addresses are same. Some archs like ARM etc, such addresses
	do not match. Hence there is another chip called I/O MMU.

	In PCI, DMA controller (bus master) is built-in. Pre-PCI, like ISA there
	used to be another DMA controller. DMA is capability of device to take over
	the address/data bus on the memory controller and drive transactions on it.

# DMA allocations
	1. PHysical and logical addresses of the buffer.
	2. Cache save buffers.

	Devices can't use logical addresses which usually are return types of
	kernel memory allocation routines (kmalloc).

	__pa(page+offset) will return  __va(frame+offset)
	__va(frame+offset) will return __pa(page+offset) 
	These can only be used in kernel space.

	Logical addresses of DMA buffer need to be translated to physical
	equivalents and configure the device controller with physical address.  In
	few cases there could be probable side effects when DMA memory is cached in
	CPU caches.