Cranky tools make for cranky developers

I’ve been compiling and testing the Open Porous Media (OPM) reservoir simulators for over a week now, and have discovered one very annoying ‘feature’ of the Ubuntu package tool.

I have had great success compiling OPM under Ubuntu 14.04, which is the ‘official’ version for OPM at this time.

I have had less than great success compiling OPM under Ubuntu 16.04 for the past week. Once I did a lot of manual tweaking and got it to compile, but every time since I’ve had the build fail. It isn’t easy to diagnose as the log files run to thousand of lines.

Tonight I figured out the problem.,, and it’s made me a bit cranky.

The issue is that there are package differences between Ubuntu 14 and 16, but I already knew that. It’s what the package manager does with some differences that’s annoying.

The command to install a package in Ubuntu is ‘apt-get install’, followed by one or more packages. So far, so good. But here’s the problem… if you specify multiple packages in the list, and any one of them no longer exists, the ENTIRE INSTALL fails. Worse, there’s no real error message. You only figure it out by the lack of messages about the packages that didn’t get installed.

For example, say you run ‘sudo apt-get install packa packb packd1.02 packe

If packd1.02 existed in U14 but now it’s packd1.11 (for example), you do get a message in the log telling you that packd1.02 cannot be found and thus cannot be installed. What you don’t get is any message telling you that it’s aborted the entire install and so packa, packb and packe are also NOT installed.

It took a few days to track down (why was BOOST missing all the time???) but finally I cracked the code, and it looks like the big compile is proceeding as expected.

It figures, Microsoft

I tend to keep everything. There’s a long story behind that going back to my dad and our move from Calgary to Red Deer when I was 10, but I’ll leave that for another time.

I’m not as much of a ‘physical stuff’ pack-rat as I used to be; moving a long distance cures a lot of that. But I am a digital pack-rat, and have been since my first experiences with computers. As a result, I have various documents I’ve written dating back to my undergraduate days. Many things were saved as text, but once word processors came about I started to save their documents as well. Other than one nasty episode where I’ve lost my entire M.Eng. Thesis because it was saved in Borland’s orphaned (and forever forgotten) “Sprint” editor, my stuff is mostly Microsoft Word documents.

After all, MS Word has been around since early Windows, and it’s still sold and maintained. What could possibly be wrong with saving old MS Word documents, correct?

Well, as it turns out, plenty. MS Word has dropped support for older products as time passed. Each new version seems to lose support for another older version. As a result, when I went to look at a 1992 MS Word document last night, it would not open. There were a few, and some would open but not display properly,  but some would not open at all. The error was “check the file protection settings in trust center”. I did that, but I already had as much prior support enabled as possible.

I even tried my older copy of MS Word on my 2005 Macbook. No joy there either.

I figured that the files were lost to time, thanks to Microsoft’s laziness and lack of care.

But.. this past week I’ve been playing with Open Source programs, most notably the FLOW reservoir simulator. I was compiling that from source on an Ubuntu 14.04 virtual machine running on VirtualBox on my Win7 PC with great success and fun. I also had created a VirtualBox Ubuntu 16.04 machine for compiling Tomcat and other Apache programs.

I remembered that all the Ubuntu machines were installed by default with a current copy of LibraOffice – one of the open source word processors.

On a whim, I fired up Ubuntu 16.04 on VirtualBox, used SMB to connect to my data server, and copied a directory of the older MS Word documents over to a local directory on Ubuntu. *** Programmers Rule #1: NEVER work on the original file.***

Once the documents were copied, I double-clicked the one that would not open at all in new MS Word, and … IT OPENED! No errors, not missing formatting. It was there, perfectly intact.

So, way to go Microsoft. You ditched compatibility with your own older document formats, but fortunately the Open Source world didn’t.

One more  win for Open Source.

Reservoir Simulation Revisited

Back in the early 1980’s, I worked for some years in Calgary writing reservoir simulators in FORTRAN for one company, and a few more years serving as technical support for another company. I really enjoyed the work and the intellectual stimulation of doing reservoir simulation.

Then my career moved to more traditional IT consulting work before I ended up working in education. I started teaching programming as a continuing education instructor, then became a daytime contract instructor before working full-time as an instructor with Athabasca University.

Over the years I’ve played around with modern reservoir simulators, even going as far to obtain an academic license for a commercial simulator from a company run by the son of one of the senior simulation researchers I had worked with years before.

When I started playing with FORTRAN again on my PiDP8 replica, I started to get interested my old Engineering work, going as far as to convert, compile and run all my old Engineering graduate studies programs.

But I wanted to play with reservoir simulation again.

So I started looking for open source reservoir simulators, and I found an excellent one in the Open Porous Media (OPM) project. They offer reservoir simulators and ancillary programs, all under the open license. Best of all, if you dig around a bit on the OPM website, you discover that it’s possible to compile everything from  source.

I have now obtained, built and installed the entire suite of simulation programs that OPM currently has, including FLOW (the main reservoir simulator) and ResInsight (a data visualizer). I have created repeatable scripts and documentation that allows me to build everything on a VirtualBox machine running Ubuntu 14.04, all running on my Windows7 PC. Ubuntu 14.04 is not the latest (that’s 16.04 currently), but that’s the version they recommend in the OPM wiki files.

It executes the sample simulations quite quckly, which is nice. They also have as examples some of the old  SPE Comparative Solution Project test cases that I used to run in the 1980s. It’s turning out to be really great fun.

My next steps are to expand the simulations I run beyond the simple test cases, and then explore some of the source code. Perhaps I can even contribute something to the project in time.

Virtualbox & Ubuntu

I now have three Ubuntu VM’s (virtual machines) on my Win7 PC running under VirtualBox.

Machine One is the ‘Firefox Build Environment’ that uses Ubuntu 16 and comes with Sublime-Text preinstalled with a host of macros to get the latest Firefox source and build the program. It works and was really fun to use. It’s also a bit boring as it’s kind-of single purpose for me. I really don’t want to mess with it as all this “stuff” is pre-configured and works.

So I obtained Ubuntu 16.04 and installed it as Machine Two. After installing it, I wrote a document about what I did. I then decided to try something radical (for me): I obtained Tomcat from source and compiled it. This required getting packages (apt-get) for Subversion, Java and other tools and libraries that Tomcat required. I was then able to get the source using SVN (subversion) and compile it following the Tomcat build documents. In the end it turned out to be very easy and fun, plus now I have Tomcat 9 (latest) running perfectly on Machine Two.

Now for the tough one. I started looking for an Open Source Reservoir Simulator, and found FLOW, from the OPM (Open Porous Media Project found at http://opm-project.org/. It was exactly what I was looking for: both Open Source and something I might be able to really get into again.

Although binaries are available, I really wanted to install from source, just for the fun of doing it all. After much reading, I obtained and installed Ubuntu 14.04 for Machine Three. This was the last version where OPM said everything compiled without problems.

After even more reading, I was able to obtain the sources and create shell scripts to download, configure and compile FLOW from source. I was also able to run the SPE test cases from my early days in industry successfully. Again, it was rather fun.

I also have other machines now – Fedora, OpenBSD, Debian and SAS, but the OPM machine is my favorite right now. I’ll write more on FLOW in another post.

VirtualBox – now on my Win7 machine

I have had VirtualBox running on my Athabascau University issued Macbook Pro for several months now. It’s a bit of a story…

I started down the road to virtualization on the Mac with Parallels. It worked well and wasn’t too costly… at first). Eventually I dumped Parallels as they kept wanting $69.99 more and more frequently for upgrades that didn’t really seem to offer much. Sadly for them, the last cash request came at the same time VMWare sent an offer to switch at a price that was too good to refuse. I dumped them for VMWare and that kept me happy for a few years.

However, VMWare also fell prey to the “cash for upgrade” mania and started doing the same thing. That’s when I searched for a free alternative and found VirtualBox.

Now VirtualBox has it’s problems. The biggest one is that it’s yet another Oracle product (like Java and Solaris and MySQL). Still, I use the three named products and so Oracle seems to be OK… for now.

I installed VirtualBox on the Mac and recreated my critical virtual machines – Win 7 (for netbeans marking), Win 98 (for LEGO Mindstorms), OpenBSD (for server testing), Minix 3 (Operating systems playing) and Windows 10 (again, for testing). It works well.

However, my latest tests failed without warning and without real recourse.

I just downloaded the Linux/Firefox development machine for VirtualBox from Mozilla.  It installed and started without problems, but when I went to build Firefox from the latest source as you are instructed to do in the guide, THE BUILD FAILED. Lots of error messages, but no help: the guide said it would be successful.

In an attempt to see if it was the VM or my Mac, I decided to install VirtualBox on my main Windows7 PC. It’s daunting as it warns you it will insert “stuff” into the network stack, and my Win7 network has always been a bit… weird.

Still, I could always uninstall. So I installed VirtualBox today and then gave it a try on the minix3 image (exported from the Mac), the SAS image I have and the Firefox image. BOOM!!!

Well, not boom, but *wimper*. It wouldn’t run. The error message was traced with google help to a BIOS switch that has virtual machines turned off for my system. I had to reboot, <F2> to modify BIOS settings to turn VM on, then save and reboot. However, once done VirtualBox worked.  I did have to re-import the images but then everything worked.

Now the acid test – build Firefox on the Ubuntu VM. After well over an hour, it was SUCCESS. All my networking still seems to work OK, and the VM machines all connect to the network without too much difficulty (DNS lookups seem to take some time but do work).

All in all, I’m pretty happy with VirtualBox. First on the Mac, and now on my Win7 PC.

Final thought: I was really rather excited to use the Ubuntu “FIrefox devel” machine to actually download and build Firefox from the latest source snapshots. I’ve been reading a great book “rebel code” about the whole GNU/Linux scene and have become interested in playing with both Linux and some of the open source programs as a developer rather than just a user. More on the book in another post.

PiDP8 boot script for MULTOS/8

The PiDP8 has one really neat feature; by setting the first left three white switches, you can control which of 8 boot scripts are used to start the SimH session running the PDP8/I emulation on the Pi.

Oscar provided 8 scripts, from 0.script to 7.script. According to the description file, 5.script is blank. It actually contains assembly instructions for ??? but I just saved that file as 5.script.save, then created a new 5.script which automatically boots the PiDP8 into MULTOS8 maintenance mode if the switches are set to octal 5 = 101.

Here’s my new 5.script:

reset
set cpu 32k
set cpu noidle
att rk0 ../imagefiles/multos8/rsh.rk05
att rk1 ../imagefiles/multos8/multos8.rk05
boot rk1

If you created a running MULTOS/8 system based on my last blog post, you can name the RKA0: disk pack you create to anything you want. I chose rsh.rk05, but any name will do. The .rk05 extension is not mandatory, but appropriate to keep things straight.T800_PiDP8 completed 2016-05-26

MULTOS/8 – a timeshare operating system for the PDP/8

As stated previously in my blog, I’ve been having tons of fun playing with my PDP8/I replica, the PiDP8 designed by Oscar Vermeulen and sold as a kit. The engine for the replica is a Raspberry Pi (3, in my case) running Simh, an awesome ‘old iron’ simulator that  runs on many platforms.

I was intrigued by the idea of MULTOS/8 after reading an inquiry about it on Oscar’s PiDP8 forum. I did a bit of digging, and found that a few folks were interested, but none (apparently) had it running on Simh.

I was able to obtain a copy from the PDP repository at https://www.pdp8online.com/images/images/os8.shtml and began.

The disk image is multos8.rk05, indicating it’s a disk pack much like the standard OS8 operating system. After some playing, I have been able to get MULTOS/8 running successfully on my PiDP8.

The general procedure is to boot OS8, then enter SIMH and attach the multos disk image to RK1, while detaching the OS8 image. In place of the OS8 disk image on RK0, you attach a new file as disk image (mine was called rsh.rk05). Then you boot to MULTOS8 in a maintenance mode. In that mode, you format (ZERO) the blank disk image and copy the MULTOS files to the blank disk. Now you can boot MULTOS into timeshare mode and play.

One quick note on paths for SIMH on the PiDP8:  I put my image files in /opt/pidp8/imagefiles/rsh to keep them separate from Oscar’s default files. SIMH finds files relative to /opt/pidp8, hence the ‘../imagefiles/rsh/rsh.rk05’ path. You are free to put your files where you want as long as SIMH can see them.

Here are the detailed steps:

Begin with the normal OS8 boot.
<ctrl-e> (to get into simh)

*Command                                                     Notes & comments

det rk0                                                     Detach the OS8 boot disk from rk0
att rk1 ../imagefiles/rsh/multos8.rk05                      Attach the MULTOS8 boot disk to rk1
att rk0 ../imagefiles/rsh/rsh.rk05                          Attach rk0 to a new file ‘rsh.rk05’. This will be the timeshare boot drive.
boot rk1                                                    Boot MULTOS/8 in maintenance mode

*Now booted to MULTOS/8 maintanance mode                     Looks and acts just like OS8 standard boot)

DIR RKA1:                                                   Directory listing of MULTOS/8 disk, first partition
DIR RKB1:                                                   Directory listing of MULTOS/8 disk, second partition
DIR RKA0:                                                   Will show nothing (or error) as the disk has not been ‘created’ yet.
ZERO RKA0:/Y                                           This initializes the RKA0: disk (rsh.rk05). You will be asked to confirm. **
COPY RKA0:<RKB0:*.*                           Copy all the files on the MULTOS/8 primary disk to the new disk.
DIR RKA0:                                                   Directory listing of RKA0:, now the same as RKA1:

DATE 23-JUN-70                                        Only dates in the ’70s work. This is needed to boot MULTOS8.
R MULTOS                                                  It will ask for a time in hhmm format, then boot to timeshare mode.

<ctrl-h>                                                        Log in user 1. There is no password so just press enter to get the ‘.’ prompt.
DIR                                                                This will show RKA0: listing, which is now the boot/default drive for timeshare.
R SYSTAT                                                      A command to show system status.
R LOGOFF                                                    A command to log off the user.

With this I was able to play on terminal 1 (the console) as user 1. I don’t have serial terminals set up for my PiDP8,  so can’t check other terminals. That’s a future project.

Critical NOTE: The /Y option on the zero command is critical as it makes the new disk bootable. If you omit the /y flag, then MULTOS will simply stall after you try and log in.

Here’s a sample session:

.R MULTOS

HELLO !

THIS IS THE MULTOS/8 MULTI-USER OS/8 TIMESHARING SYSTEM
CREATED BY COMPUTER METHODS
7822 OAKLEDGE ROAD
SALT LAKE CITY, UTAH 84121, USA
PHONE 801-942-8000

PLEASE INPUT TIME IN 24-HOUR FORMAT
(E.G. 0925 FOR 9:25 AM AND 1935 FOR 7:35 PM):
1022

THANK YOU !

THE SYSTEM IS NOW  TIMESHARING

TYPE CONTROL/H TO LOG ON.

PLEASE LOGON

.R SYSTAT
MULTOS-8   SYSTEM STATUS       1-JUN-1978    AT    12:27:10

UNASSIGNED MEMORY    16 K     SWAPPING FIELDS  6    MAX NO. USERS   2

JOB       ACTIVE        BUSY

SELF        1             0       TTY  I/O WAIT AT PC 6651

LPT         1             1

MEMORY USEAGE

FIELD   JOB  REL FIELD

LPT QUEUE EMPTY

.

 

PiDP8 – still fun after all these days…

I’m still having a blast with my PiDP8, which is a PDP8/I replica that uses a Raspberry Pi (3 in my case) to provide PDP8/I emulation inside a very nice box with replica PDP8/I switches and blinking leds. The PiDP8 was created by Oscar Vermeulen in the  Netherlands and is available from him as a kit. If you are interested, his website is http://obsolescence.wix.com/obsolescence#!pidp-8/cbie

I’ve attached a picture of my finished kit to this post as well.  I’m very happy with the build and with the overall unit. I only had one build problem – There’s a 40-pin header for the Raspberry Pi, and I always solder things in alternate pins/rows to spread the soldering heat evenly about the build. In this case, I managed totally miss one connection. Of course it was the main power connection between Pi and PiDP8, so when I finished, checked & double checked and finally connected it all up and applied power… nothing. After a careful re-scanning under a large magnifier knowing it had to be something really simple, I found the bare connection. Once soldered everything worked perfectly.

As I’ve said in other posts, I’ve been having a blast resurrecting my Engineering graduate studies FORTRAN programs and getting them to run on the PiDP8. I now have over 30 programs running, and all but a couple give results comparable to the same programs’s original output  and my modern 64-bit PC tests. In those cases, it’s due to single-precision floating point rounding errors that cannot be avoided. The originals were double precision, but the PDP8/I FORTRAN IV compiler does not support double precision. Still, I am satisfied with my results, and it was a lot of fun to do.

T800_PiDP8 completed 2016-05-26

PiDP8 (PDP8 replica) and FORTRAN: How suite it is :-)

Sorry for the pun, but I couldn’t resist. (the pun comes from program testing usually being done with a test framework and a “test suite”).

The past few days I have all of my smaller Engineering FORTRAN programs compiled and running on the PiDP8 – my PDP8 replica.  It was time to “work smarter” as they say. The basics of this involve replacing hand-typed commands with batch processes so that dozens of commands are stored in a file and invoked with one short command.

I began by reading all about the OS8 BATCH command and  batch files. I then created some smaller  test batch files to see how things worked, especially program input which usually requires keystrokes such as the following: “EXE RKB0:GAUSS.LD<esc>RKB0:GAUSS.DA/1<esc>” where <esc> means “press the escape key”. Batch files don’t typically allow control characters, so they rely on another mechanism. In the case of OS8, it’s the ‘$’ which tells BATCH that ‘the esc key was pressed’. In the end, the command line

EXE RKB0:GAUSS.LD<esc>RKB0:GAUSS.DA/1<esc>

becomes

.EXE RKB0:GAUSS.LD$

*RKB0:GAUSS.DA/1$

This works perfectly.

With such things resolved, I created two large batch files – one that compiles and loads all of my programs to date, and another which runs the programs. I no longer have to keep load files around as I can recreate them with one command:

SUB CMPALL

and likewise can run all but 3 programs with the command SUB RUNALL. It turns out one set of programs requires too much memory as the DIMENSION statements for some arrays consume 4000 bytes, just under the FORTRAN 4192 bytes allowed. However, this is enough to prevent the programs running under the BATCH system as it requires enough memory to prevent the programs from running (“INSUFFICIENT CORE”).

Still, pretty good stuff.

 

Playing with FORTRAN IV on the PiDP8

I’ve been spending a lot of my research time playing with my PIDP8. This is a replica of the PDP8 (slightly smaller form factor) complete with blinking lights and operational switches, but using a Raspberry Pi as the processing engine. The actual simulation of the PDP8 is handled by Simh, a  remarkable ‘ancient tech’ simulation package.

I’ve been resurrecting all my old Engineering grad school FORTRAN programs and getting them to compile and run on the PiDP8. It’s been a blast!

The PiDP8 runs OS8 as the nominal operating system, and the version of FORTRAN IV available is quite old and a bit quirky. The limits include no double precision, no lower case (at all!), strict adherence to column placement of code and strict 6 character limits on both file names (OS8) and variable names (FORTRAN IV). The quirks come when these limits are violated. Sometimes you get a compile error, sometimes a run-time error, but some times nothing except a hung program. As they say, “the lights are flashing, but no-one’s home”.

Some quick examples. Although there’s no double precision, the compiler accepts “DABS” and “DEXP” library function calls. No error messages anywhere, but the program will just hang waiting for the non-existent library. Using too-long variable names results in compiler error messages that have no apparent meaning until you ‘fine tooth’ the code and realize a variable name is longer than 6.

FORTRAN IV also does not allow free-format read & writes. “read(5,,end=9999,err=9999) x,y,z” is just full of not-allowed stuff. You need a proper format statement for every read & write.

In the end it took quite a while to get all of my programs running; around 35 in total. These include various matrix solvers, mass transfer and heat transfer programs, wellbore programs and flash calculations. As I said, much fun.

There were some differences in output between the original 1980’s runs (on a Honeywell MULTICS system), runs on a modern 64 bit PC running the GNU fortran compiler (f77) and the 12-bit PiDP8. However, with the exception of a couple of programs that I have not had time to scrutinize, all results were well within expected ranges given the differences in word size for single-precision calculations. The two cases that were substantially different were cases where the original program was double-precision on a 32-bit machine and convergence of the iterative solution was difficult, even in the original cases.