Just to let you, the reader, know that I’m currently preoccupied at the moment with work. However, I’m still working on the netchannels design, and all signs indicate that I’ll be able to ping any host on the Internet by some time next week.
I’ve been away from my development laptop recently, and it’s actually given me the opportunity to come up with several new ideas. One of them is the netchannels design (which is the core of the userspace network stack), which I’ll detail on the Whitix wiki and some blog posts to come. The other is how filesystem metadata should be exposed in the virtual filesystem. Let me explain.
First of all, metadata is “data about data”. For a filesystem, it mainly relates to data about files. We’ve had standard attributes about files for over 40 years now; information like the file’s size, its permissions, and which user and group own it. We’ve got a set of standard system calls that cover all these different attributes, and they are well established. However, thinking about them more abstractly, they turn out to be (special kinds of) key value pairs. So why not extend the concept, and allow custom key value pairs (say, director for a movie file) to be stored along with the file itself?
Well, it’s possible! Filesystems, like ext3, support “extended attributes”. You can store a key and a value ( currently the size limit is the block size). Usually, you have to prefix the name of your attribute with a namespace, like “user” or “system”. Great! That surely means we can store a lot more information outside the file, where it isn’t locked into one of the many ten of thousands of file formats out there?
Well, problem is, nobody really uses it much.
Why? First of all, they’re relatively new to most people. In a lot of operating systems and filesystems, support was only recently introduced or made public. This means a lot of userspace programs can’t rely on them, since a lot of users don’t have extended attributes enabled (or are hardly used; NTFS on Windows is one example).
The second reason is that they’re not exposed that well. In many operating systems, there’s a separate set of system calls to deal with them. Also, they’re not easy to manipulate or list from the shell, since there’s another custom set of commands to list, create, delete and update them. Not good.
So, inspired by Reiser4, I believe Whitix should expose metadata openly via the filesystem. Imagine, to use some command line examples, creating a new attribute using ‘echo “Steven Spielberg” > /file/name@user.director’, or listing all of them . Now, before I expand upon the possibilities, there are several important differences between my proposal and that for Reiser4. First of all, files will not become directories, at least not without the tailing @. In that way, they still do operate in another namespace; you won’t be able to ‘cd’ into a file, so to speak.
Another thing is, you won’t be able to have directories in the extended attributes. They’re won’t be a metadata file named /file/name@a/b. It doesn’t make sense, as they’re won’t be that many attributes for each file; certainly not enough to warrant putting them into separate directories called ‘user’, ’system’ and the like. These features (or lack thereof!) hopefully avoid the downfalls and problems that were expressed with the Reiser4 proposal.
Now onto the cool stuff, and how it might work in a modern desktop environment. First of all, this really makes a lot of searching and finding much easier, especially on the command line. Searches like “find @user.director=’Steven Spielberg’ @user.file_type=’movie’”, or the natural language equivalent, “Find all films with director Steven Spielberg” become much more possible; movies in different formats have the same metadata.
Now the user won’t need to label all his files with these type of extended attributes manually. It’s really quite a chore, and really inefficient. Some of these attributes will be general (size, permissions etc. will be read-only and read-write attributes that are simulated by the VFS), filesystem-specific (a list of blocks under “system.block_map” for each file or directory perhaps?) and some will be created by software after downloading the file, like a web browser setting the mime-type, or any application that creates a file (to simplify things, there could be a core service that regularly scans the users’ files and updates the metadata).
At any rate, automatic metadata and attributes will be at the core of the Whitix desktop, and I’ll be talking soon about metadata will fit into this vision of mine.
So in the previous blog post, I’ve told you about what aims of the Linux driver layer are, as well as the advantages and disadvantages. Now I’ll let you know how you can join in and help provide Whitix with support for thousands of different devices. We’ll need help testing, writing functions and designing the layer. Let’s get started.
Check out the source code by typing svn co svn://www.whitix.org/svn/projects/Whitix/branches/netchannel . Build it (make) and test that the built ISO works on your computer or emulator before testing Linux drivers.
You can download the Linux drivers that you need from the linux_drivers directory on whitix.org. Find the Linux driver that supports your (PCI ethernet) hardware, and copy it to CdRoot/System/Modules/Network. Edit user/system/startup.c to load the driver (replace the Network/pcnet32 line), and type make to rebuild.
A lot of drivers will not be loaded because of undefined symbols. Get a list of imported symbols by typing nm CdRoot/System/Modules/Network/<driver name>.koLook up the definitions in the LXR, and add a stub defintion (print out the name of the function with KePrint) to the appropriate file in devices/linux. Don’t forget to add SYMBOL_EXPORT after the function.
Type make to build the code again, and keep repeating the process until the driver loads correctly. Then you can start implementing the functions as you see fit. Once you’ve got the driver to load, test it out by typing dhcp at the Burn command line, and if you get a IP, you’re successful! Although it’s an unusual development process, it’ll be made easier in future.
Ask us any questions by visting us at #whitix on irc.freenode.net. Enjoy!
So what is this Linux driver layer? It’s a new module in the Whitix kernel that aims to support a multitude of devices using their Linux drivers. The first class of drivers that the layer aims to support are PCI Ethernet drivers (which actually comprise most PC network devices), and slowly expanding to other PCI and USB devices.
One of the advantages of supporting Linux drivers follows from the fact that most operating system drivers are actually fairly OS-agnostic; that, is they do roughly the same set of functions. Network drivers across all operating systems, for example, register themselves, probe for any devices on the system’s buses, transmit and receive packets, and keep statistics about the card.
Using Linux drivers is this way means that a lot of development time is saved. We don’t have to reinvent the wheel by writing a Whitix driver for every device (that would end up working almost identically to the Linux equivalent). This also happily avoids the “there’s no drivers!” issue that many young operating systems have; Whitix’s driver support will be very similar to that of Linux.
One major issue about using the Linux drivers is that the Linux driver API is generally not stable. There’s an easy way to solve this; only allow drivers to be loaded from one particular kernel version. However, another problem arises. How about the different kernel configurations? There are thousands of possible configurations for each kernel versions. Another simple solution: use drivers from a particular distribution. In this case, we’ve chosen the Ubuntu drivers for 2.6.28-11-generic (to be precise). Any driver from that particular kernel build will work perfectly fine.
Of course, there’s the small disadvantage there that the newest devices won’t be supported unless we choose a new kernel version to support (then we’ve got the problem of testing the code again to make sure it’s stable enough for general use). However, this is a fairly small problem compared to writing Whitix versions of all the Linux drivers! It’s a small cost to pay; most desktop end-users won’t typically run exotic hardware that requires the newest Linux drivers.
So what’s the current status of the Linux driver layer? Well, I’ve been able to transmit and receive packets using the pcnet32 Linux driver (this is the driver that supports Ethernet cards like PcNet II, PcNet III and others), and I’m beginning experimenting with other Linux drivers. The process usually involves writing a lot of stub functions that are generally not used much; I typically implement a function if the driver calls it often or it’s needed for operating successfully. (As a technology preview, it’s an acceptable method of working). The more testing that takes place with a whole range of drivers, the stabler the Linux driver layer will be.
Hopefully this has been a short overview of what the Linux driver layer is, and its general aims. I’ll cover the code soon, along with a few diagrams to clarify things and what needs to be done! The wiki has some more information, as well as an incomplete list of the functions we need to implement. If you want to help, find out more by logging on to #whitix on irc.freenode.net. I’ll be committing my code in a few days, and talking more about this awesomely cool layer.
I’ll be busy until Friday, which means this blog won’t receive too many new posts until then. In the meantime, here’s a link about how sqllite is tested; it’s inspiration when you think something can’t be tested or that unit tests aren’t possible for a module of code.
In the meantime, if you want to jump into another language, try writing a web server in Haskell or a blog in Common Lisp. Both links are actually quite interesting reads, although they are quite advanced.
