Installing NextStep 3.3 on VMWare 6

Update 2020: If you want to skip all this, I have a pre-made OpenStep 4.2 Virtual Appliance (OVA) for Oracle VirtualBox that is ready-to-use with apps, games and pre-configured networking. Download it here.

I’ve been hankering to learn more about the origins of OS X, so I figured the best place to start would be the OS at its core — NextStep (later named OpenStep.)

When Steve Jobs was ousted from Apple, he set out to create a new company that would deliver elite hardware, and a revolutionary operating system. NeXT was started to build powerful hardware and software that would leapfrog the Mac (and everyone else on the market.) They eventually had to kill off the hardware line (after making some very nice, uber-high-end machines) but the software they made changed the industry. It was on a NextStation computer, using the NextStep OS that Tim Berners Lee would invent the World Wide Web, and its the true Object Oriented approach of the OS that many companies have imitated or borrowed.

By the mid-90s Apple was in truly rough shape. Their archaic OS, once the greatest thing on the market, was showing its age, and Apple had tried, and failed (mostly due to mismanagement) to replace it. It was determined that they should buy an existing next-gen OS, and re-build on top of it. After flirting with IBM and BeOS, Apple settled on a deal with NeXT — a deal that grew until Apple acquired the company, re-acquiring Steve Jobs in the process. He became the iCEO (interim CEO) and the de-facto leader of the company. Existing ideas were rapidly brought over from the Mac onto NextStep/OpenStep, layered on top of its rock-solid BSD Mach microkernel, Object Oriented underpinnings, and a re-vamped version of its graphics layer, was a new Interface, and a Classic emulator to allow old Mac apps to run.

The end product, after many variations, pre-releases, betas, and one or two fairly ugly general releases, was what we know today as Mac OS X. Much of NextStep is still intact, including low-level OS APIs, visual and development concepts, and culture.

If you’re interested in getting NextStep up and running in VMWare (I’m using a Windows Vista 64-Bit Host OS, but these steps should work with most versions of VMWare — including Fusion) you’ll need to find an ISO of the NextStep OS somewhere, and maybe these steps will help you…

Configuring VMWare
After much frustration, I finally got past the initial hurdle of configuring it so the installer would run off a CD image. Here’s what I had to do:

Get the NextStep 3.3 floppy disc images from this great resource site, It has pretty much everything you need, plus some forums that you may or may not find helpful. You’re going to need at least:

  • Boot Disk
  • Driver Disk
  • Beta Drivers
  • Core Drivers

Here’s the basic setup steps:

  1. Configure a new VM, and tell VMWare that the Guest OS will be BSD.
  2. Create a virtual hard drive no larger than 2GB — and it must be pre-allocated, since NeXTStep will try to check the whole drive.
  3. 64MB of RAM will be more than enough
  4. “Insert” the Boot Disk floppy image (rename the extensions of all the disc images you downloaded to .flp to make VMWare find them easily) and “Insert” the ISO for the Install CD.
  5. Your Hard Drive and CD-ROM should be configured as IDE. The Hard Disk must be configured as IDE 0:0 and the CD-ROM MUST be configured as IDE 1:0 due to the primitive driver support.

Installing NextStep 3.3 and Drivers

  1. Start up the VM, following the prompts until you’re asked for driver discs.
  2. Start with the Core Drivers. It’ll only need that disk for a second to load keyboard support, and other basics.
  3. When asked for SCSI drivers, first use the “3.3 Driver Disk” and tell it to use option 4 — Adaptec 2940. This isn’t really present on your VM (no configuration in VMWare will let you use the SCSI drivers) but we just need to make the installer get past this step — note you may have to do it twice; just pick the same option again.
  4. You’ll be asked a second time for drivers. I assume the first time is looking for drives on the SCSI bus, and the second time is about IDE — which is what we’ll be using. That means this is the important driver step. Insert the “3.3 Beta Drivers” disc, and press 2 to list its drivers. Press 7 to go to the next page, and press 2 to choose “Dual IDE” option.
  5. Finally press 1 to skip loading any other drivers, and continue…
  6. The installer will ask a whole bunch of times if you really want to wipe out your hard drive (just the hard drive image, don’t worry!) and then it’ll copy the files it needs. When its done, it’ll ask you to remove (disconnect) the floppy and reboot. You should disconnect the floppy and CD at this point — and make sure they don’t automatically re-connect on boot, or you’ll get errors.
  7. When you boot from the hard drive, the OS will begin to load, and then notice it needs some drivers again. It’ll list the devices it doesn’t know about on its own. Supply the “3.3 Beta Drivers” disc, and any other disks it asks for.
  8. Soon you’ll see the Graphical OS Loader kick-in. It’ll take a few minutes and then it’ll want the “3.3 Beta Drivers” disc again. At this point you should have mouse support, so things get a little nicer. If you don’t get this far, make sure your hard drive image was pre-allocated, so that it passes the NeXTStep disk check.
  9. You’ll see the famous beach ball from OS X (but in grayscale instead of color) and then a Summary of Devices.
    Just to start out, I clicked on Monitors and added the Default VGA Adapter. I’ll have to mess with this, and what’s available online. I’ve heard its possible to get a much better resolution than VGA.
  10. The mouse was working, so I didn’t mess with that. Networking is another whole ball of wax, and I never got it working in NeXTStep.
  11. In sound, I added Sound Blaster 16, since that used to be very compatible — but it didn’t work. I didn’t touch the SCSI or Hard Drive drivers, since the ones I was using got me this far!
  12. In the next step, I unchecked the languages I didn’t want, and installed everything else so I could play with it. It struck me that much of the terminology used in the installer looked similar to that used in the OS X installer.
    When its done copying, remove the floppy (like it tells you to!) and go ahead and reboot.

Pretty soon you’ll be up and running inside NeXTStep! Check your e-mail — there’s a note there from our old friend Steve!

I have had some mouse problems — it behaves erratically, so I’ll be searching the forums for fixes on that. I found that if I hit “Ctrl + Alt” to release the mouse, then click inside NeXTStep again to send control back to the Guest OS, that clears up the problem… for a bit. Obviously I’ll be working on display and network drivers, but that’s outside of the scope of this article. It’s alive, and that’s all we were after today.

From here I’m going to point you to a couple links that should help you finish up a good install. I’ve also made a drivers ISO that will simplify things greatly.

Enjoy your bit of Apple/technology nostalgia…

24 thoughts on “Installing NextStep 3.3 on VMWare 6

  1. Just for interests sake, I’m posting this comment from OmniWeb in NextStep 3.3!!
    Display: 1280×1024 32-bit color
    Sound: Working!
    Network: WWW, FTP yes! SMB, partially.

  2. Hi, thanks for the how-to. I tried this on VMWare Fusion 2.0 but It tossed up the following on the console:
    vfs_mountroot: error=15
    panic: (Cpu 0) vfs_mountroot: cannot mount root
    panic: NeXT Mach 3.3 Mon Oct 24…
    System Panic:
    It seems this is the first time its attempting to boot from the 3.3 ISO image. I downloaded this image off the net (torrent) so perhaps it’s screwed up.
    I’m a NeXT programming veteran… wrote several commercial applications for NeXSTEP from 1990 to 1999. I moved over to Windows and Solaris afterwards, so it’s been quite the reunion with the iMac and Leopard. Getting a copy of NeXTSTEP 3.3 running in VM would be fantastic. Is there a chance you could share your vm image?

  3. A couple things to check:
    Ensure that the Guest OS is identified as BSD — using any other OS will, for some reason, cause a Kernel Panic. In VMWare 6.5 use the “Other” option, then choose “FreeBSD.”
    Don’t allocate too much RAM — remember that NeXTStep computers wouldn’t have had more than 512MB ever. I use 128MB and haven’t run into any issues.
    Your hard drive image should be pre-allocated — using a growable hard drive will cause problems — and 2GB or less. (No operating system of that era would know how to address anything over 2GB) Mine is 1.5GB.
    Your Hard Drive and CD-ROM both have to be IDE. This means a few CD-based things in NeXTStep won’t work (like the CD Player app) but you’ll have to live with that.
    Some of these options won’t be available if you do a “Typical (recommended)” setup when you build your VM. You’ll have to do a “Custom (advanced)” setup in VMWare to make sure you’re presented with all the right options.

  4. Your successful installation of NS 3.3 on VMware is very impressive, especially (if I am reading your post correctly) the use of video resolution above VGA. I have been experimenting for some with NS on virtual machines (tried VMware, VirtualBox, etc.), with absolutely no success getting video working above VGA due to the lack of VESA driver for NS, and/or the lack of a host machine “adapter” for the legacy video drivers in NS. My interest stems from wanting to use a very nice analysis application that I developed a number of year ago as a NS user (and lost when I shut down my last white NS 3.3 box several years ago). Is there any chance that you would be willing to help explain the route you took to solve the video problem? Thanks very much.

  5. Well I didn’t really solve it! The folks Atomic Object released these drivers that would perfectly with NextStep 3.3 and OpenStep 4.2:
    The only real trick is getting them into your NS/OS system. If you have networking working between your guest and your host, you could put the driver archive (the .compressed file) onto an FTP share, and use an FTP client to pull them in. Or you could do what I did: burn the drivers to a CD disk image (.iso file) and use that image as your CD-ROM in VMWare.
    There’s a bit of a trick to burning NS/OS-friendly ISOs though. You’ll have to use a program called MagicISO (or something equally capable):
    – Create a new, empty CD Disk image, then immediately go to File > Properties
    – Check the box in CD File System to ISO 9660
    – Check the box in CD File System to include the Rock Ridge extensions
    – Clear all other check boxes
    – Use the radio button to set the File / Directory names length to ISO 9660:1999 v2
    Save the image, or burn it to disc, and use VMWare to provide it to your guest OS. If you don’t feel like doing the burning, I’ve got a pre-made ISO that I put together linked in the post above. Then:
    – Copy the .compressed file off the disc onto your NS/OS file system, and double click to decompress it in place.
    – Now use to “Add…” a new Display Device Driver.
    – Click “Install Driver…” to find the decompressed file, and hit OK
    – Now choose ‘VMWare Display Adapter by Atomic Object” in the list.
    You can choose the display resolution you want under “Display Mode”
    The only remaining trick is that your mouse won’t know if you increase your screen size. You’ll need to go into the Mouse tab and:
    – Click “Expert…”
    – Scroll down to “X Size” and type in the width of your chosen screen resolution and HIT ENTER!
    – Scroll down to “Y Size” and type in the height of your screen resolution and HIT Enter!
    OK your way out and reboot. You should have full color at a great, modern resolution!

  6. Thank you *very* much for your help. I finally have NS working with 1024×768, with full mouse range. When I have networking, I will have a completely operational machine. It also appears to run fast on my hardware. The process still took a lot of trial-and-error, but the ease of cloning in VMware Workstation is helpful. The setup is still very tricky, and it looks like only a small subset combination of host settings, NS configuration, and installation order work. I found that setting the mouse range had to be absolutely last, and I believe that even the setting of the external window size in VMware may be important. When this is all done, I will carefully document all of the configuration params for future reference, and would be glad to share my results.

  7. Have been unable so far to get networking going, and there appears to be very little specific info on the net about networking on NS 3.3 under VMware. I am using the last version VMXnet driver which as I understand it, was developed for Openstep. The system hangs on booting for the host configs that I have tried so far: NAT, bridged network, and local network only. From the load log, the VMware virtual network “adapter” doesn’t appear to be recognized by NS, which is obviously a very bad sign. Any specifics on network configuration would be greatly appreciated.

  8. Well dangit…I HAD it working, then when I finally got the video driver decompressed, I started getting boot messages reading “Can’t find $LBL” Any thoughts?

  9. Bob, the furthest I got with network was manually assigning an IP, and getting FTP and WWW working. I messed briefly with SAMBA, but it wasn’t easy going, so I gave up. FTP covered my needs.
    Sorry Mark, I’ve never seen that one!

  10. I’m having issues getting the video driver to work. A message flys by (not recorded in /usr/adm/messages) about being unable to load …/Devices/VMWareFB.config/VMWareFB_reloc , and it reverts to default VGA.
    Is it possible that the driver on the ISO is OpenStep 4.x only, not compatible with NS 3.3?
    BTW, in your list of driver disks needed, the regular disk (3.3_driver_disk) isn’t listed in the “You’ll need at least” bullet list. Took me a little while to figure that out. 🙂

  11. Nevermind about the video driver — something happened when I decompressed the driver and the _reloc file got corrupted. Now it works, though running in vmware-server (on a linux box, using windows as a console) it’s REALLY dark looking. Running inside vmware-player on windows it’s much better.
    Thanks for your work documenting this. Been YEARS since I’ve had NeXTSTEP running on anything. Now I just have to get the network going and dig up a copy of OmniWeb…

  12. It’s time for an update on my progress. I am happy to report that I now have a complete NEXTSTEP (NS) 3.3 system (with Developer) running on VMWare Workstation 6.5. My VMWare Workstation is running on an x86_64 based dual processor Linux box running RHEL 5.2. It’s been a long road for me, but the result is quite amazing. My sincere thanks to Jonathan for providing critical information on this web site that got me going in the right direction.
    All of the problem areas: display, mouse, network, and sound are working. There are clearly some challenges in completing the installation. For example getting the VMMouse driver installed while the defective default driver is still running is one example. There are also several “non-obvious” bits of information that are needed, such as the need to change the “virtual network host adapter” to match the driver specifically written for NS. In the process I have bundled all of the needed drivers and packages (including the final NS updates) into a single CD iso for convenience of installation. I am guessing that with a bit of work it might even be possible to modify the initial installation data (on floppy images) to circumvent the need to post-install the needed drivers — or at least make it easier.
    One of the best aspects of running NS on modern hardware is that it performs (at least on VMWare) *much* better (meaning faster) than it ever did back in 1995… on either black or white hardware. The use of networking and NFS makes the 2 GB limit on the installation disk space much less important for real world applications. NS of doesn’t care about the size of the disk space on NFS mounts, and the network setup is very clean with the NFSManager application . NFS is also the most practical way to get data in and out of the virtual NS machine. It has been fun re-learning the amazing (still) advanced features of NS. I also have an entirely practical need in reviving an old but still useful NS application that I helped develop.
    I have documented my path to installation, and I would be glad to share this with interested folks (along with any assembled open-source/access drivers, etc.). With Jonathan’s help this can probably be arranged.
    A comment to David on the display darkness — the video adapter and driver on NS needs to be matched with the color depth actually running on your host machine. I noticed this color distortion affect with one installation that I made. The color depth cannot (afaik) be changed on the NS system, so it must be changed to match on the host system. When I corrected the color depth of the Linux host system (to:8/8/8-32) the problem went away. I would guess that your Windows system is by default using the correct color depth to match the virtual machine.

  13. For what it’s worth, I’ve got NS 3.3 up and running on VMware Fusion, with a little bit of help from this blog post (I’ve had it running in VMware previously—several years ago, but had long since forgotten the magic order of getting the drivers loaded).
    It’s worth noting that you’ll probably want to install the Y2K patch from Apple (which takes you to 3.3p3), so that your VM doesn’t think it’s 1994…
    My only snag now is that I’ve lost my copy of the Developer CD!

  14. Hi,
    I just tried to get NS 3.3 up and running on VMWare Fusion 2.01 – everything seems to work great, except the network. On booting it gives me a config space error in the boot log for the VMXNet driver (1.3 used) and aborts loading the driver. Had anyone the same problem and solved it?

  15. Got got the network working! My trouble came from importing an old VMWare image created on VMWare 3. In this image I used the AMD network driver and somehow Fusion did take over this setting. It could not be changed in the VMWare Fusion GUI however (like lots of special stuff VMWare choose not to confuse the mere mortals with it ;)).
    To resolve change a single line in the VMWare image (which is actually a hidden directory, use on MacOSX and cd into it). In IMAGENAME.vmx (where IMAGENAME is the name of your VMWare image) there is the line
    ethernet0.virtualDev = “e1000”
    just delete it and vmxnet loads happily. This should only be needed if you import an old VMWare image (pre 4.5) into a newer VM and you have used the AMD Network driver previously.

  16. Have been trying all suggestions on this site and a few other I could find by using google, I can’t get it to work :(, is it possible for someone to send me a clean installation of the vmware image of nextstep 3.3? e-mail is

  17. OPENSTEP4.2J is working fine on VMWare Fusion 3.0 with
    But NEXTSTEP3.3J on VMWare Fusion 3.0 is has a problem. It won’t find root device in reboot after the installation. But you can run a kernel from floppy disk with root device directive.
    Boot: fd()mach_kernel rootdev=hd0a
    but still PS2 driver is not so good.
    Or, you can quit software installer, at the final stage of the installation, it you give Terminal access.

  18. Got it working in VB under Snow Leopard, using OS type: Linux/Linux 2.4. Still trying to figure out how to get the screen resolution up beyond 640×480.

Leave a Reply

Your email address will not be published.