M3 Operating System Development

A chronicle of the development of the M3 operating system

Archive for the ‘Design & Development’ Category

Initialization Order of Operations – getting to pmode

leave a comment »

I was playing around with my initialization code and realized I had a problem. I had stripped down my boot sector so that it basically just loaded up my environment initialization code (which is a mixture of assembly and C) and threw control to that. I eliminated the code that enabled the A20 gate (since I moved this to the init code) and removed the code that put the processor into protected mode with a simple flat memory mode. However, when I tried to boot my code in the emulator, it would triple fault, which basically means the processor was put into a bad state and pooped out.

After thinking about it for a bit, I realized my error. By removing the code that put the processor in protected mode, I was keeping things running in 16-bit real mode. The initialization code that I was trying to pass control to was compiled as a 32-bit ELF file and converted to a 32-bit flat binary file. If I wasn’t putting the processor into 32-bit protected mode, then I obviously couldn’t hand it a bunch of 32-bit code to execute.


So I restored the code that puts the processor into 32-bit protected mode, using a very simple GDT that sets up a flat memory model. Voila! Everything was back in working order. I was hoping to delay enabling 32-bit protected mode until I had the environment completely set up, but instead I will have to go with the following strategy: 

1. In the boot sector, load up the initialization code into memory, enable the A20 gate, and enable pmode with a simple flat memory model. Then hand control to the initialization code.

2. In the initialization code, fully set up the operating environment – set up the proper GDT, IDT, TSS, setup paging, etc, etc. Load up the kernel. Then activate the new memory model, turn on paging, load up the interrupts, and refresh pmode. Once we’ve got a properly initialized environment, hand over control to the kernel.

Written by m3os

April 17, 2009 at 6:44 pm

Code Reorg

leave a comment »

Quick update.

I did a bit of code reorganization this week, in order to prepare to meet my new boot loader / environment initialization goals. I moved some of the initialization code out of the kernel folder and into the boot folder. That folder is where the boot sector and environment initialization code will live. I also added a file – init.c – that will mediate the environment initialization. I will try to do as much as possible in C, but some things will still need to be done in assembly, which is the reason for the init.asm file.

Right now, most of the code in the init files is stubbed out with lots of TODO comments. In the next couple of days, I will be filling out some of those portions.

Written by m3os

April 14, 2009 at 8:30 pm

New goals for the M3 boot loader and startup code

leave a comment »

I’ve taken a bit of a break from coding this past week, and have instead been reading, studying, and doing a bit of design.

After reading more deeply through the Intel IA-32 System Programmer manual and thinking about things a bit, I’ve changed my mind about the structure of the M3 startup code. I think that I’ve left out some things that need to be done before I jump to protected mode, and the GDT that I’ve currently set up doesn’t give me quite the memory model that I would like for M3.

It’s very interesting to read through the Intel documentation. As documentation goes, it’s well-written and contains many useful diagrams. It’s very tempting to try to cobble together an OS using only the various tutorials and source code available on the ‘net, but I think those that try to do this are really missing out, and their OSes are destined to be simple one-offs of existing systems at best. Without a full understanding of how the processor works, and what features are available, you’re handicapping yourself. Besides, this stuff is pretty fascinating.

OK, enough of the soapbox speech. As I was saying, I want to add some stuff to the M3 boot loader and OS initialization code, and do some code restructuring. Here’s a list of the things that I want to accomplish before I hand over control to my kernel:

  • Load kernel into memory
  • Enable A20
  • Setup GDT (for both system and user segments)
  • Setup protected-mode IDT
  • Setup TSS for multitasking
  • For paging, set up at least one page directory and page table.
  • Turn on pmode and paging at the same time (both in CR0)
  • Setup interrupt handlers
  • Jump to kernel main()

This stuff isn’t in order – it’s just a checklist.

As an additional goal, I’d like to do as much of this as possible in C, so that the code is more easily understood. Right now, I do things like setting up the GDT in assembly, and I’d like to rewrite that so that I have the bare minimum of assembly code.

There’s a lot of stuff here, and way too much to fit into a boot sector. So my thinking is that the boot sector will do very little – basically it will just load what I’m calling the OS environment initialization code and then hand over control to that code. 

The environment intialization code will be a mix of assembly and C, and will handle the tasks I listed above. It will also load the kernel into memory. When all its tasks are complete, it will hand over control to the kernel.

This design will keep the initialization code out of the kernel, since it really doesn’t belong in there, in my opinion. I think that this new design is clearer, and will therefore be more easily understood and easier to maintain.

Written by m3os

April 8, 2009 at 7:40 am

Boot Loaders from other Operating Systems

leave a comment »

I was in the bookstore last night, a place that I often find myself, and was happy to see that there were two books on operating system design and development. Of course, these were college-level textbooks, and therefore cost upwards of $100, so I was content to peruse them in the store instead of contributing to the madness that is textbook pricing.

One of the books was all theory, which wasn’t all that interesting to me, since I already own several books on OS theory. The other was theory mixed with examples from actual operating systems, which was much more useful to me.

Since I am currently trying to write the boot loader for M3, I paid close attention to the examples of boot loaders. I read about the booting process for Inferno, and the boot process for Linux. The Inferno example was very interesting to me, since Inferno basically initializes the system, loads up its kernel, and then launches a virtual machine on which all user-space processes run. This is actually very similar to what I’m thinking about for M3. I would ultimately like M3 to boot up and launch a Java Virtual Machine, so that all user applications could be written in Java or some other language that targets the JVM.

Inferno uses a VM named Dis that the developer (Bell Labs) wrote from scratch. Programs targeting the Dis VM are written in a language called Limbo, which has C-like syntax. For M3, I don’t want to write my own VM, and I don’t want my application developers to learn a completely new language in order to target my OS. That’s why I’m thinking about using the JVM. If I can get the JVM running on M3, then M3 will be able to run the 50 kajillion Java apps that are already out there.

Anyways, Inferno is a pretty cool OS, built by brilliant people with some interesting ideas. Unfortunately, Bell Labs is no more, but you can read about and download the Inferno OS here. It’s now open source, so you can dig into all the gritty coding details too.

OK – back to the boot loaders. After I read about Inferno’s boot process, I read about Linux’s. Boy, I never realized how involved the boot process was for Linux. It does a ton of stuff in assembly code before it jumps into the kernel. A lot of the complexity comes from the fact that the kernel code may be compressed, and so the boot loader needs to not only load the kernel into memory, but decompress it as well. The boot loader has many stages, and is much more involved than what I want M3’s loader to do. The other complexity comes from the fact that Linux was started back when the 386 was still new, and therefore the boot code needed to do many more hardware checks to determine what features were available to the system. With M3, I’m going to assume a minimum level of functionality, rather than testing for each and every specific feature.

For M3’s boot loader, I want to keep the code to the bare minimum necessary to get the kernel up and running. I want to keep the assembly code at a minimum, and get into C as soon as possible.

Written by m3os

March 7, 2009 at 3:37 pm

M3’s Boot Sector/Loader – Design & Goals

leave a comment »

Now that we’ve figured out how to build a boot sector and boot a virtual PC with it, it’s time to start thinking about fleshing it out into a boot sector that actually does something.

Since a boot sector is limited to 512 bytes, there isn’t a whole lot that we can do. The primary job of the boot sector is to load the kernel into memory and hand control to it. Since I’m planning to write the entire kernel in C, I’d also like to get the processor into protected mode before handing control to the kernel. I think that the C compiler will spit out binaries that expect protected mode.

What’s protected mode? 

There are two different modes that a CPU can operate in – real mode and protected mode. In real mode, the currently executing code has complete access to all the instructions of the CPU. In protected mode, a limited set of CPU functionality is accessible by the currently running code. If code running in protected mode needs some feature of the CPU that is protected, it can trigger an interrupt, which will hand over control to the kernel. Then the kernel can decide how best to service the request.

Protected mode allows the kernel to have exclusive control of critical functions of the CPU, such as those that manage memory and hardware. It will allow the CPU to help us with things like paging, virtual memory, and multitasking – all desirable features of modern operating systems.

How do we get the CPU into protected mode? Well, there are several steps that must be taken, including setting up a Global Descriptor Table (GDT), initializing some segments, etc. Intel’s IA-32 Intel Architecture Software Developer’s Manual, Volume 3 goes into the steps in some detail, and is a necessary resource for programming IA-32 CPUs. Actually, all the volumes are necessary reads, IMO. Here’s Intel’s page where you can download these manuals for free.

A20 gate

The other important thing that the boot sector needs to do is to enable the A20 gate, which is needed to enable protected mode. What is the A20 gate, you ask? Well, it’s a logic gate attached to the A20 line, a part of the system bus, and it’s typically disabled by the BIOS before our boot sector starts executing. For a good description of the A20 gate/line, read the Wikipedia article. Why try to explain it myself when others have already done such a good job?

The M3 boot sector – Goals

So here’s a summary of what we’re trying to accomplish with the M3 boot sector:

  1. Enable the A20 gate
  2. Load the C kernel to the proper location in memory (TBD)
  3. Get into 32-bit protected mode
  4. Hand over control to the kernel

Along the way, we’ll print relevant status messages, if we have the space. I’m not really sure how much code is going to be involved to do all these things, but I do know that 512 bytes is NOT a lot of space. I am fully expecting to have to break this functionality into two separate entities – the boot sector, and a boot loader. This is often called a multistage boot. The boot sector performs some initialization, loads the boot loader into memory, and then transfers control to the boot loader.

The boot loader will not have a 512 byte size restriction, and so it will likely handle the heavy lifting – getting the processor into pmode, loading up the kernel, and handing over control. We’ll see if I can get everything to fit in 512 bytes, but I’m not that proficient in assembly yet, so my code is probably more verbose than it needs to be.

At any rate, I’m looking forward to the challenge.

Written by m3os

March 4, 2009 at 9:08 am

Development Environment for M3 Operating System Development

leave a comment »

One of the first things that an operating system developer must do is decide on a development environment and a set of development tools. In this post, I will describe the development environment that I plan to use for M3 development.

In 2001, I bought my first Mac. I was intrigued by OS X. A really nice GUI built on top of rock-solid UNIX sounded like a winning combination, and oh boy is it a rockin’ combination! Since then, I have not purchased another PC. In recent years, many other developers have followed suit, and Macs have become quite a popular development platform. In my current job, I develop ASP.Net websites using a MacBook Pro running VMWare Fusion. I’ve got a virtual Windows 2003 server instance that I do my .Net development on, but for everything else, I use native OS X software. It’s a sweet setup.

So I decided right off the bat that my development environment would be OS X. Because it’s UNIX, there are a zillion dev tools available, many of them bundled with OS X on the included developer tools CD.

I also mentioned VMs. I decided that my life would be a whole lot easier if I ran M3 on a VM or an emulator during development, rather than try to run it on the actual target hardware. When I first started getting into OS development, about 10 years ago now, I remember having to write the boot loader and kernel to an actual 3.5″ floppy disk. To test, I would need to boot the computer off the floppy. If it didn’t work (as was often the case), I would need to figure out what went wrong, fix the code, compile, write the code to the floppy, and reboot the computer. Not an optimal development cycle by any stretch of the imagination.

Today, my Mac doesn’t even have a floppy drive. I haven’t bought or owned a floppy disk in years. That fact, combined with the painful floppy development cycle had me looking for more optimal ways to test operating system code. And I decided that what I needed was an emulator.

Right now, there seem to be two popular choices for IA32 emulation: Bochs and QEMU.

Bochs is an open-source IA32 emulator that runs on a variety of platforms. It is well-maintained, and there is plenty of help available out there if you need it.

QEMU is another open-source emulator that runs on a variety of platforms. For OS X, there is the Q project, which provides a nice Cocoa front-end to QEMU.

I haven’t yet decided which emulator I’ll use for my testing. I’m currently evaluating them to determine which will better suit my needs. 

Coding Tools

I’m getting a bit ahead of myself though. Before I can get to the testing stage, I need to have actually written and built some code.

The core M3 kernel will be written in C. Perhaps some of the other pieces will be written in a higher-level language, but the core – the task schedule, the memory manager, etc – will be written in C.

For C programming, I’m not going to look any further than the GNU C Compiler (gcc). It’s not even up for discussion. gcc is an excellent compiler, and available on a zillion platforms. There’s copious documentation, and it’s open-source. 

M3 will need to have certain parts written in assembly, primarily the boot loader. For assembly code, I’m currently evaluating two options: GNU Assembler (gas) and NASM.

GNU Assembler is the back-end assembler used by gcc. The benefit of using gas would be that I would remain within the GNU toolset. However, gas uses a different syntax than a lot of other assemblers (AT&T syntax as opposed to Intel), and I think that might make it more difficult for me to incorporate sample assembly code available on the Internet. I don’t want to have to translate from Intel to AT&T syntax.

Nasm – the Netwide Assembler is an open-source assembler project that is portable across many platforms and supports a wide variety of output formats. It is currently a very popular choice for assembly programming. 

I’m currently leaning towards using Nasm. I just like the assembly syntax a lot better.

Other Tools

For my text editor, I’ll most likely be using TextMate. It’s not free, but it’s well worth the money.

I’ll use make to script the build/linking process and make that easier. I don’t want to sit there and type a bunch of commands everytime I want to build M3. I’d rather just type

make m3

and have make do all the heavy lifting. Yeah, I’m lazy like that.

If I come across other tools that I find useful, I will be sure to note them in the blog.

Written by m3os

February 27, 2009 at 10:37 am