How to Install NetBSD OpenBSD FreeBSD on the Same Disk

I have decided to document this because apparently the answer for the question how to install the 3 major BSDs on the same disk is either, no answer, it's impossible or it is difficult. Well in this tutorial I will show you that it is possible and not so difficult if you know what you are doing. It presented a great system engineering challenge for me as well.
With the era of virtualization you might ask why even bother with this why not just install them on 3 separate VMs, well because we can. Also you might need them for something like driver development where the OS needs direct access to the hardware or you would like to do a side by side benchmark of applications on the top of them on the exact same hardware. We are finally arrived to the point when just like Linuxes BSDs can be "dist-upgraded" as well instead of reinstalled at every new release.

This is a non-uefi guide for older machines but will work for most of the available VMs (KVM, Vmware, VirtualBox, Xen ...) as well so you can perform this whole setup in a single VM as well.

At the time of the writing the version of these OSes:

• OpenBSD 6.9
• NetBSD 9.1
• FreeBSD 13.0

I will present two different methods. One fully open source, using Linux GRUB2 as the main boot loader, the second is closed source using System Commander as the main boot loader. This guide is for MBR parition table, NOT for GPT. I will extend it with that later on.

First I will go through both methods, the needed components, the steps then at the end the caveats and some explanation. My point with this guide is that unlike the 7 years old half-baked answers what you can find online if you follow this step by step then you can complete the installation and have exactly the same working system as I do.

Required components:

• Open/Free/NetBSD
• Void Linux
• A live rescue Linux such as System Rescue CD

Steps:

1, Prepare the disk

In this method we start with an empty (zeroed out) disk but unlike in the other method first we will boot in with the Linux Rescue CD just to create the MBR partition table in the beginning of the disk. If you would start with NetBSD that would use BSD type disklabel on the disk without any question, we don't want that.

After you booted to the rescue CD all you need to do is:

  fdisk /dev/sda < or whatever your drive letter is

Fdisk will detect that there is no partition table on the disk and will create a DOS MBR based by default then just push 'w' to write it to the disk. DO NOT CREATE ANY PARTITIONS AT THIS POINT.

Why? Because NetBSD will not respect it, it will show the partitions and they will be unusable if you pre-created any.

1, NetBSD

It will say that the disk seems not to have been partitioned before ... select b (MBR), geometry correct, edit the mbr partition table now you can add partitions. I don't even bother with swap anymore because you can just use swap files later. (select Install-> YES, ACTIVE-> YES). Rest of the install is straight forward.

NetBSD again without question will write it's bootcode to the MBR, not really a problem.

At this point your partition table will look like this:

Disk /dev/sda: 120.0 GB, 120034123776 bytes, 234441648 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk label type: dosDisk identifier: 0x00000000   Device Boot      Start         End      Blocks   Id  System/dev/sda1   *          64    61440356    30720146+  a9  NetBSD

2, OpenBSD

Now you can make your life easier and before installing OpenBSD just boot to the rescue Linux and create a partition for it with id A6. It will detect it during the install and you can just push ok on the rest.

At this point your partition table will look like this:

Disk /dev/sda: 120.0 GB, 120034123776 bytes, 234441648 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk label type: dosDisk identifier: 0x00000000   Device Boot      Start         End      Blocks   Id  System/dev/sda1   *          64    61440356    30720146+  a9  NetBSD/dev/sda2        61440357   120031844    29295744   a6  OpenBSD

3, FreeBSD

Create a FREEBSD slice from the installer then create the rootfs inside. You will get bunch of "GEOM invalid disk labels" but the install will proceed.

4, Void Linux

I have lost most of my passion for Linux a long time ago. I have chosen Void due to it's clean (systemd free) design and it has a fast package manager. So the Void install is nothing special, install the boot loader to MBR. However at FS detection it will detect a lot of non-existent partitions due to the BSD slices, don't care about it, chose the last one which is /dev/sda4.

At this point your partition table will look like this:

Disk /dev/sda: 120.0 GB, 120034123776 bytes, 234441648 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk label type: dosDisk identifier: 0x00000000   Device Boot      Start         End      Blocks   Id  System/dev/sda1              64    61440356    30720146+  a9  NetBSD/dev/sda2        61440357   120031844    29295744   a6  OpenBSD/dev/sda3   *   120031848   182946407    31457280   a5  FreeBSD/dev/sda4       182946408   234441647    25747620   83  Linux

Now that Void Linux is installed edit the main grub config:

/etc/grub.d/40_custom

There are 2 ways to do this, we going to chose the easier one here which is chainloading the OS (this allows grub to boot an operating system’s boot loader). The other solution mentioned later, for example it can directly boot the FreeBSD kernel completely bypassing the BTX Loader, but I'm not sure that would work for OBSD/NBSD so why bother.

menuentry "NetBSD 9.1" {     insmod chain     chainloader (hd0,1)+1}menuentry "OpenBSD 6.9" {     insmod chain     chainloader (hd0,2)+1}menuentry "FreeBSD 13.0" {     insmod chain     chainloader (hd0,3)+1}

Issue: update-grub to update the entries.

That's it, wasn't that hard was it? :)

5, Final touches

If you want you can mount the BSD partitions in Linux but only as read only by default, here is the reasoning: Many Linux distros leave out write support for UFS when they compile their kernel, because the write support in the Linux UFS driver is not considered fully reliable. Rather than expose you to data loss if you mount a filesystem read-write, they prefer to warn you away.

Same goes with Void Linux, you can rebuild the kernel with CONFIG_UFS_FS_WRITE=y

Manual mounts:

mkdir -p /mnt/freebsd /mnt/netbsd /mnt/openbsdmount -t ufs -o ufstype=ufs2,ro /dev/sda1 /mnt/netbsd/mount -t ufs -o ufstype=ufs2,ro /dev/sda2 /mnt/openbsd/mount -t ufs -o ufstype=ufs2,ro /dev/sda3 /mnt/freebsd/

Automatic mounts:

You can add them permanently to /etc/fstab. (find out the proper UUIDs with running lsblk, ignore the bogous partitions what Linux finds)

UUID=INSERT UUID 1            /mnt/netbsd  ufs ufstype=ufs2,ro 0 0UUID=INSERT UUID 2            /mnt/openbsd ufs ufstype=ufs2,ro 0 0UUID=INSERT UUID 3            /mnt/freebsd ufs ufstype=ufs2,ro 0 0

Backup the MBR:

dd if=/dev/sda of=/root/sda-$(date +%Y-%m-%d).512 bs=512 count=1

This is basically a modification of the previously mentioned open source method with the difference of not using Linux at all since it is also possible to directly install Grub on one of the BSDs.

From the 3 I going to do this on NetBSD since that is which force written it's boot code to the MBR, so after you installed the 3 systems if you don't do any modifications it will boot NetBSD only.

If you follow this method the only difference with the aforementioned guide that you will not install Void Linux and in this case you might won't need the rescue Linux CD either.

On amd64 you can just do:

export PKG_PATH="http://ftp.NetBSD.org/pub/pkgsrc/packages/$(uname -s)/$(uname -m)/$(uname -r|cut -f '1 2' -d.)/All/"pkg_add grub2.

On i386 (and maybe other) architectures grub2 binary package is missing the reason probably is ... because it's broken. You can try to fix it and post a patch afterwards.

First if you did not install the compiler during the installation of NetBSD you can still do it with running sysinst on the main system and reinstall the sets (either go with full installation or custom and select the compiler).

cd /usrhttps://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc....tar xvzf pkgsrc.tar.gzcd /usr/pkgsrc/sysutils/grub2make install clean

Will not build :(

You need to find what you drive is, here are 2 easy methods how to do so:

dmesg  | grep -i drive...[     4.170022] wd0 at atabus0 drive 0[     4.170022] wd0: drive supports 16-sector PIO transfers, LBA48 addressing[     4.220040] wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)...

or

cat /etc/fstab | grep ffs/dev/wd0a               /       ffs     rw               1 1

Just delete the content of /usr/pkg/etc/grub/41_custom and replace it the attached boot config.

mkdir -p /grubgrub-mkconfig -o /grub/grub.cfg

Now, install GRUB into your hard drive's master boot record (MBR). You have to know its device name for this step (e.g. /dev/rwd0a). Exchange /dev/rwd0a with your desired device name, then change /dev/rXXXa to /dev/rwXXXd to access the raw disk, as in the following example:

Device name : /dev/rwd0a

Direct access : /dev/rwd0d

The appropriate grub-install command for this drive would be:

# grub-install --no-floppy /dev/rwd0d

Required components:

• MSDOS boot floppy
• Win98SE installer or MSDOS 6.22
• Open/Free/NetBSD
• System commander (create the rescue disks) < optional
• A rescue Linux such as System Rescue CD

PROTIP: if you don't want to mess around with floppies just get a Gotek, SC rescue disk is optional but it can also be PXE loaded.

Use a completely zeroed out hard drive, it will contain 4 PRIMARY PARTITIONS IN THIS ORDER:

1, Dos/Win98

2, NetBSD

3, OpenBSD

4, FreeBSD

Do NOT create any partitions in advance!

If your drive is not clean boot in the Linux rescue CD first and zero it out with:

dd if=/dev/zero of=/dev/sda bs=512 count=1

Steps:

1, Prepare the drive

Boot in with MSDOS 6.22 floppy create a 2GB (Max) FAT16 partition, do NOT create any other partitions!

fdisk ... reboot

format C: /u /s

At this point I just prefer booting in the linux rescue CD and pull the win98 installer, SC and other components from ftp and drop them on the DOS partition so no CD or anything else will be needed.

2, Install Win98SE (or DOS) and System Commander

I prefer going with 98SE, regardless that SC can be even installed on MSDOS 6.22 but if you feel like it's a waste of space and you just want the first OS to serve as bootloader then fine create a 100MB partition, not even full MSDOS setup is required as you will not use it for anything just a SYS C: of format C: /u /s to transfer the bootcode and COMMAND.COM and use the SC DOS install.

Another advantage of using good old MSDOS is that it runs on pretty much any x86 x64 boxes while you might run into issues with newer hardware with 98.

After both W98+SC setup properly you boot in with systemrescue and back up the MBR:

dd if=/dev/sda of=sda.446 bs=446 count=1

Put this file somewhere else like usb stick, I just ftped it to somewhere. The first 446 bytes with the remaining 64 bytes allocated to the partition table. Do not backup the whole 512 bytes because that is containing the partition table too not just the boot code and we are about to create more partitions!

3, Install NetBSD

Create a new partition for NetBSD from the installer (make sure you leave enough space for the other 2). From the 3 this is the only BSD which will forcefully overwrite the MBR, both OpenBSD/FreeBSD will not try to write anything in there if they see that here is a valid mbr.

4, Install OpenBSD

Again nothing special create a new partition for OpenBSD from the installer, leave space for FBSD. OpenBSD has the most I would say difficult to understand patitioner so if you are not sure how to create a slice for it then before doing this step boot back in the Linux Rescue and create a partition with cfdisk set the type to A6. OpenBSD will automatically recognize this and use it.

5, Install FreeBSD

Create a FREEBSD slice from the installer then create the rootfs inside.
You will get bunch of "GEOM invalid disk labels" but the install will proceed.

6, Fix the bootloaders

OK so this is the only tricky step. You boot in with your Linux rescue cd and if you haven't done it already first fix the boot code for SC in the MBR (the backup what you have made at Step 2):

dd if=sda.446 of=/dev/sda bs=446 count=1

Now mount the first DOS partition and create 3 boot record copies for the 3 OSes for System Commander to be able to use them.

  mount /dev/sda1 /mnt/dos   dd if=/dev/sda2 of=/mnt/dos/nbsd.512 bs=512 count=1  dd if=/dev/sda3 of=/mnt/dos/obsd.512 bs=512 count=1  dd if=/dev/sda4 of=/mnt/dos/fbsd.512 bs=512 count=1

The boot record files must be on the C:\ root drive for SC to be able to use them. Now reboot and in System Commander add them one by one by selecting ADD MBR and enter the filenames like C:\FBSD.512 and select they option BYPASS (This is important). At each when it asks for which partition to mark active always fill it properly eg. in this case 0-0 is the dos 0-1 nbsd, 0-2 obsd, 0-3 fbsd.

At the end you will end up with a partition table like this:

/dev/ada0: 310098 cyl 16 hd 63 secPart        Start        Size Type Flags   1:          63     4192902 0x06 0x00 (DOS)   2:     4194304    75776675 0xb9 0x00 (NET)   3:    79970979    72260370 0xb6 0x00 (OPEN)   4:   152231352   159383552 0xa5 0x80 (FREE)

After this you can rename them as you like and select cute icons for them voila your done.

I wanted to make the tutorial simpler so I did not present this option:

At METHOD 2 STEP 3 when you install NetBSD it will overwrite the MBR whether you like it or not so you won't be able to boot into SC or 98 anymore. At this point you could do a boot code restore to get back System Commander again with:

a, Restore the MBR as it's described in point 6: dd if=sda.446 of=/dev/sda bs=446 count=1

b, Boot in with SC's rescue floppy and do a CHECKMBR which essentially does the same it writes back the SC mbr

I just wanted to save some time with this because it's really not necessary to do a restore immediately.

All the BSDs had to be installed into primary partitions in the past however after some research for example I found some articles how to install FreeBSD on an extended partition:

# add custom menu entry for freebsd os installed on /dev/sda7menuentry "FreeBSD on SSD"{  insmod ufs2  insmod bsd  set root=(hd0,7)  kfreebsd /boot/kernel/kernel  kfreebsd_loadenv /boot/device.hints  set kFreeBSD.vfs.root.mountfrom=ufs:/dev/ada0s5a  set kFreeBSD.vfs.root.mountfrom.options=rw  set kFreeBSD.hw.psm.synaptics_support=1}

Even tho in this setup it's not particularly useful because that 1 EXT takes up a PRIMARY slot actually so it only makes sense if you want to install more other OSes into that extended partition.

LEAVE FREEBSD FOR LAST

From the 3 BSD distros FreeBSD is giving us the most trouble. The reason why FreeBSD intentionally left for last is because one of the others (and I think NetBSD) messes up the FreeBSD slice. Interestingly it is still mountable from Linux but the FreeBSD BTX loader will not see the slice anymore when you do "lsdev".

It is possible to boot the FreeBSD kernel directly with grub (Not using the BTX loader) but there are some things to consider: loader.conf is read by the BTX loader, which means it's not read when we boot from GRUB, aka those parameters it would pass on has to be defined in the GRUB config (see previous example).

It seems that FreeBSD after 8.4-Stable release switched from GEOM_MBR to GEOM_GPT.

Mounting from ufs: /dev/ad0s1a failed with error 19

adaXs - notation for slices in UFS

adaXp - GPT partition designation

Grub on other disk example:

Now let's say you stick in another PRIMARY MASTER disk before the current one with grub on it and you want to use that grub to load OSes on the second disk where you have SC and all these BSDs installed. This will work for getting to the MBR part but after booting these OSes, they won't work because regardless that you have faked the root drive with drivemap the drive letters will change inside the OS. With this OpenBSD wouldn't even get that far to load the boot loader and NetBSD would break after trying to mount the root fs. This might won't be a big deal for most of the current Linux distros because they all switched to using UUIDs for block devices so the naming does not matter anymore.

menuentry "NetBSD" {   set root=(hd1)   drivemap -s hd0 hd1   chainloader (hd1,2)+1}

Of course there is a workaround... which is that if you want to leave this system like this permanently with 2 disks, with another disk to be the PRIMARY MASTER then (no you don't need to reinstall anything) you need to boot in the installers of these BSDs again (why? because linuxes still in 2021 can't "safely" mount BSD partitions for rw only for ro :/) get a rescue shell RW mount their root fs and change the drive ids in fstab and other places accordingly and one day when your main PRIMARY drive dies or you decide to take it out guess what? Then you have to redo this whole procedure again with the changed drive letters this is why I considering multi-booting multiple disks is just way too much of a hassle.

Notes on System Commander: although I consider this software a 20+ years old abandonware if you are using the closed source method please buy DOS or Win98 as they are not free.

I will update this guide with a GPT version later on. MBR might be a 30 years old technology but it does it's thing and it does it well even Today and won't go away any time soon. Many of the computers produced Today can be set back to use pre-UEFI booting method and if for nothing else that part of the guide can help the growing retro pc community.
It would be interesting to reproduce all this on a regular pendrive, probably would present new challenges.

LINKS:

http://geodsoft.com/howto/dualboot/

https://www.howtogeek.com/193669/whats-the-differe...

https://wiki.gentoo.org/wiki/GRUB2/Chainloading