M3 Operating System Development

A chronicle of the development of the M3 operating system

Archive for April 2009

Initialization Order of Operations – getting to pmode

leave a comment »

I was playing around with my initialization code and realized I had a problem. I had stripped down my boot sector so that it basically just loaded up my environment initialization code (which is a mixture of assembly and C) and threw control to that. I eliminated the code that enabled the A20 gate (since I moved this to the init code) and removed the code that put the processor into protected mode with a simple flat memory mode. However, when I tried to boot my code in the emulator, it would triple fault, which basically means the processor was put into a bad state and pooped out.

After thinking about it for a bit, I realized my error. By removing the code that put the processor in protected mode, I was keeping things running in 16-bit real mode. The initialization code that I was trying to pass control to was compiled as a 32-bit ELF file and converted to a 32-bit flat binary file. If I wasn’t putting the processor into 32-bit protected mode, then I obviously couldn’t hand it a bunch of 32-bit code to execute.


So I restored the code that puts the processor into 32-bit protected mode, using a very simple GDT that sets up a flat memory model. Voila! Everything was back in working order. I was hoping to delay enabling 32-bit protected mode until I had the environment completely set up, but instead I will have to go with the following strategy: 

1. In the boot sector, load up the initialization code into memory, enable the A20 gate, and enable pmode with a simple flat memory model. Then hand control to the initialization code.

2. In the initialization code, fully set up the operating environment – set up the proper GDT, IDT, TSS, setup paging, etc, etc. Load up the kernel. Then activate the new memory model, turn on paging, load up the interrupts, and refresh pmode. Once we’ve got a properly initialized environment, hand over control to the kernel.


Written by m3os

April 17, 2009 at 6:44 pm

Code Reorg

leave a comment »

Quick update.

I did a bit of code reorganization this week, in order to prepare to meet my new boot loader / environment initialization goals. I moved some of the initialization code out of the kernel folder and into the boot folder. That folder is where the boot sector and environment initialization code will live. I also added a file – init.c – that will mediate the environment initialization. I will try to do as much as possible in C, but some things will still need to be done in assembly, which is the reason for the init.asm file.

Right now, most of the code in the init files is stubbed out with lots of TODO comments. In the next couple of days, I will be filling out some of those portions.

Written by m3os

April 14, 2009 at 8:30 pm

Modifications needed for M3 development under Windows

leave a comment »

For giggles, I downloaded the latest source code to my Windows box and attempted to build and run M3. It didn’t work. After some tinkering with build and linker settings, I have gotten my development environment working smoothly. I can now build and run M3 under Windows. I thought I would take a moment to explain the changes necessary to setup a M3 development environment on Windows.

Some assumptions:

  • You have installed cygwin, and the proper packages (nasm, gcc, binutils, etc).
  • You have installed Qemu and it is on the system path.

Once you’ve met these assumptions, make the following modifications to the M3 source files:

boot/init.asm :

  • you need to prefix an underscore to the extern symbol init_main. It should be _init_main, since gcc under cygwin prefixes function names with an underscore.

link.ld :

  • change OUTPUT_FORMAT from elf32-i386 to pe-i386. This is the default output format supported by the cygwin binutils.
  • change .rodata to .rdata . For some reason, this section is named differently under cygwin. If you don’t make this change, your code will still compile, but you won’t see any output to the screen, since the strings defined in your code won’t be linked into the binary.

Makefile :

  • change CFLAGS:
    • remove -fno-stack-protector . This isn’t supported by gcc under cygwin.
  • change EMUFLAGS:
    • add -L “c:\path\to\biosfolder” . There should be a Bios folder underneath the folder you installed Qemu into. Provide the path to this folder. Otherwise Qemu will complain about not being able to load BIOS.
    • add -no-kqemu
  • change OBJCOPY flags:
    • add -I pe-i386   This tells objcopy that object files you want to convert to binary are currently in PE format.

Once you make these changes, you should be good to go. You should be able to build M3 without errors and run it in the Qemu emulator.

EDIT: 4/12/09 – Modified to reflect the changes made during a small code reorg.

Written by m3os

April 10, 2009 at 8:33 am

New goals for the M3 boot loader and startup code

leave a comment »

I’ve taken a bit of a break from coding this past week, and have instead been reading, studying, and doing a bit of design.

After reading more deeply through the Intel IA-32 System Programmer manual and thinking about things a bit, I’ve changed my mind about the structure of the M3 startup code. I think that I’ve left out some things that need to be done before I jump to protected mode, and the GDT that I’ve currently set up doesn’t give me quite the memory model that I would like for M3.

It’s very interesting to read through the Intel documentation. As documentation goes, it’s well-written and contains many useful diagrams. It’s very tempting to try to cobble together an OS using only the various tutorials and source code available on the ‘net, but I think those that try to do this are really missing out, and their OSes are destined to be simple one-offs of existing systems at best. Without a full understanding of how the processor works, and what features are available, you’re handicapping yourself. Besides, this stuff is pretty fascinating.

OK, enough of the soapbox speech. As I was saying, I want to add some stuff to the M3 boot loader and OS initialization code, and do some code restructuring. Here’s a list of the things that I want to accomplish before I hand over control to my kernel:

  • Load kernel into memory
  • Enable A20
  • Setup GDT (for both system and user segments)
  • Setup protected-mode IDT
  • Setup TSS for multitasking
  • For paging, set up at least one page directory and page table.
  • Turn on pmode and paging at the same time (both in CR0)
  • Setup interrupt handlers
  • Jump to kernel main()

This stuff isn’t in order – it’s just a checklist.

As an additional goal, I’d like to do as much of this as possible in C, so that the code is more easily understood. Right now, I do things like setting up the GDT in assembly, and I’d like to rewrite that so that I have the bare minimum of assembly code.

There’s a lot of stuff here, and way too much to fit into a boot sector. So my thinking is that the boot sector will do very little – basically it will just load what I’m calling the OS environment initialization code and then hand over control to that code. 

The environment intialization code will be a mix of assembly and C, and will handle the tasks I listed above. It will also load the kernel into memory. When all its tasks are complete, it will hand over control to the kernel.

This design will keep the initialization code out of the kernel, since it really doesn’t belong in there, in my opinion. I think that this new design is clearer, and will therefore be more easily understood and easier to maintain.

Written by m3os

April 8, 2009 at 7:40 am

On the shoulders of others – standing or leaning?

leave a comment »

I’ve obviously been doing a lot of research recently, specifically into the entire booting and kernel loading process. Here are a few examples of such tutorials. But probably the most quoted and widely-used tutorial currently out there is the famous Bran’s Kernel Development Tutorial (bkerndev)

This tutorial is so successful because it is fairly well-written, and because it gently guides the reader through some fairly thorny tasks that are necessary when building an OS on the IA-32 architecture. The tutorial takes us through the booting process, through setting up protected mode, and then shows how to set up interrupt handling and basic I/O with hardware such as the keyboard. 

I haven’t done any number crunching, but I think it’s fair to assume that the publishing of Bran’s tutorial led to a bit of a population explosion in the OS Dev world. Since its release, there has been a proliferation of OS projects on OSdev.org and other similar sites. The reason for this is that many developers work their way through Bran’s tutorial, and having done this, start up their own OS projects.

In reading through the forums on osdev.org, I get the sense that new OS projects based on Bran’s tutorial are sorely frowned upon. Just recently, someone announced a new version of their OS, and they were promptly beat down for having done nothing but make a few changes to the bkerndev code. I can see how frustrating such projects must be for OS dev veterans, and such beatdowns are probably warranted in many cases, but I would like to take a moment to explore the pros and cons of “standing on the shoulders of those that have come before” as an approach to OS development.

The first time that I tried to develop M3, about 10 years ago, there was no Bran’s tutorial. There wasn’t anything like it. Information about boot sectors, and loading kernels, etc, was hard to come by. Besides the Intel manuals, there weren’t too many reliable sources of information for those interested in tinkering with OS dev. And in my case, the result of this lack of information was that I gave up. After getting a very rudimentary boot sector working, that basically just booted from an actual floppy disk and printed a welcome message to the screen in real mode, I gave up. As a one-man team, I didn’t have the time or resources to figure out how to set up the necessary scaffolding for my kernel. I was interested in memory management, and process scheduling, and file systems, but I never got anywhere near that point, because of the immense amount of work that needs to be done just to get the environment initialized and in a state such that control can be handed off to the kernel.

Fast forward to today. With Bran’s tutorial and others like it, I could use the code provided and build myself that necessary scaffolding in a matter of days, not months or years. Bran’s Kernel Dev tutorial does not teach you much of anything about developing the bulk of the kernel itself. Instead, it walks you through the environment setup that needs to be done to be at the point where you can really focus on kernel development. bkerndev is somewhat of a misnomer. It should be called bkernenvsetup or something like that.

Back to the forum beatdowns. If someone merely takes Brandon’s code, tweaks a line here or there, and tries to pass it off as his own OS, then absolutely he should be raked over the coals (or perhaps just ignored, if flamewars aren’t your MO). However, if someone is using such code as the means to an end, the end being that they get to start working on their actual kernel, which will run on top of all the Bran scaffolding, then I would argue that that’s a valid use of the code. In fact, Brandon himself says so, right in the introduction. He lists the steps that the tutorial will take, with the final step being:

12) …and the rest is up to you!

Using Bran’s code to get to the meat of kernel development is a legitimate and pragmatic approach to OS development. Such (re)usage of code is an example of standing on the shoulders of those that have come before, which has been a valid and acceptable (even encouraged) practice in the open source world for.. well, ever.

At this point (if you’ve actually read this far), you’re saying to yourself “Great, this guy is just trying to justify cutting corners in his OS project”. Not so. I’m not trying to do any such thing. I’ll admit that I have read Bran’s and many other tutorials, and have used them as reference implementations for some of my stuff, but I don’t plan to just copy the code and use it without having any understanding of what’s going on. Quite the opposite – I use such tutorials precisely so that I can understand what’s going on. And I think that’s the critical difference. By reading through the tuts, by going through the provided code line by line and really understanding what’s going on, you come to “own” the code. If you know the code back and forth, and you get to the point where you understand it as well as if you had written it yourself, then you are truly standing on the shoulders of those who have come before. They have helped to lift you up to a level of understanding. And that’s something great. That’s not something to beat down.

So is my plan with M3 to simply build on top of Bran’s environment? No – that’s not the plan. But I do plan to use such tutorials to give me a leg up, so that I have a fair chance of actually getting to the point where I’m working on the real meat of the kernel. I plan to use to the code as a reference implementation of the concepts explained in the Intel manuals and other reference books. Being able to read something in the manual, and then see some actual code that illustates the concept is a great way to learn and understand. 

I’d like to get to step 12 at some point in my lifetime, and standing on the shoulders of others is by far the most pragmatic way to do it.

Written by m3os

April 1, 2009 at 12:22 pm