C++ Sockets Example

I hadn't done network programming in C++ in quite some time, so I did a search for some material to refresh my memory. It took longer than I thought it would to find a simple, working example, so I will post the one I eventually found here for future reference. Here is the post from StackOverflow that I found helpful: link.

To send a UDP message to the socket, run this from the command line:

• echo -n "my message" > /dev/udp/127.0.0.1/50037

Here is the code. To make the recvfrom() call blocking, change the MSG_DONTWAIT parameter to 0, and remove the check for the -1 return value, which is what the non-blocking recvfrom() returns when it doesn't receive anything.

#include <stdio.h>
#include <stdlib.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <string.h>
#include <unistd.h>

int main( void )
{
    int fd;
    if ( (fd = socket(AF_INET, SOCK_DGRAM, 0)) < 0 ) {
        perror( "socket failed" );
        return 1;
    }

    struct sockaddr_in serveraddr;
    memset( &serveraddr, 0, sizeof(serveraddr) );
    serveraddr.sin_family = AF_INET;
    serveraddr.sin_port = htons( 50037 );
    serveraddr.sin_addr.s_addr = htonl( INADDR_ANY );

    if ( bind(fd, (struct sockaddr *)&serveraddr, sizeof(serveraddr)) < 0 ) {
        perror( "bind failed" );
        return 1;
    }

    char buffer[200];
    //for ( int i = 0; i < 4; i++ ) {
    while( 1 ) {
        int length = recvfrom( fd, buffer, sizeof(buffer) - 1, MSG_DONTWAIT, NULL, 0 );
        if( length == -1 ) {
            sleep(1);
        }
        else if ( length < 0 ) {
            perror( "recvfrom failed" );
            break;
        } else {
            buffer[length] = '\0';
            printf( "%d bytes: '%s'\n", length, buffer );
        }
    }

    close( fd );
}

Compiling a Linux Kernel for gem5

It took me a while to find a combination of versions of the Linux kernel and gcc that would boot on gem5 simulation x86, so I wanted to take some notes on the version I finally got to work.

First, you need the full system files for x86 for gem5, as explained in step 8 of this post. I'll go ahead and repost those instructions here.

• To run in full system mode, you will need to download some more files from the gem5 website.
• First, you'll need to download the full system files and config files for X86. They are available here: http://www.m5sim.org/Download. Extract both of these to a directory where they will live (for our example, we'll assume this directory is /home/user123/gem5-util/). They need to be in a place where gem5 can see them while they are running. When you extract these files, you will get two directories, "binaries" and "disks".
• In the "disks" directory, there is a file called "linux-x86.img". This needs to be renamed to "x86root.img".
• gem5 also needs a file called "linux2-bigswap.img" to be in the disks directory. The only place I've found this is in the full system files for the Alpha architecture posted on the gem5 website, here:http://www.m5sim.org/Download. Download the Alpha system files, extract them, and move the file "linux2-bigswap.img" from this package to the "disks" directory containing the file "x86root.img".
• Now, set the environment variable M5_PATH to point to /home/user123/gem5-util (or wherever you put the "disks" and "config" directories).

Next, clone the Linux kernel repo using git.

$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git

The version of the kernel I've gotten to work with gem5 is 2.6.27.6, so checkout that one out.

$ git checkout v2.6.27.6.

Now, get the config files provided by gem5.

$ wget http://www.m5sim.org/dist/current/x86/config-x86.tar.bz2

$ tar -xjvf config-x86.tar.bz2

The only configuration I've gotten to work is "linux-2.6.22.9".

$ cp configs/linux-2.6.22.9 .config

I had to roll back to gcc-3.4.6 to get it to compile. You can get gcc-3.4.6 from here: link. I forced the kernel build system to use this version by doing:

$ ln -s <path to gcc-3.4.6> ./gcc

$ ln -s <path to gcc-3.4.6> ./g++

$ export PATH=:$PATH

Adding the colon to the beginning tells bash to look in the pwd first for programs. Now, build it.

$ make vmlinux -j

And you should be able to boot with this using gem5. It asks a bunch of questions regarding kernel configuration. I answer the default for all of them, and this works fine.

$ ./build/X86/gem5.opt ./configs/example/fs.py --cpu-type=timing \

--disk-image=<path to x86root.img> \

--kernel=<path to built vmlinux>

For a little more information on this topic, see this discussion on the gem5 mailing list: link.

Building a Toolchain for OR1K

I recently have started looking more into the site OpenCores.org, and came across its page on building a toolchain for OR1K.

I ran into a couple of problems when following the instructions for building the Newlib toolchain. One was this error when building stage 1 of gcc:

        "configure: error: cannot compute suffix of object files: cannot compile"

Instructions for resolving this error are given on the gcc website here. However, the tips there along with most help I found given on other sites suggest that the problem is that the locations of the MPC, MPFR, and GMP libraries are not in LD_LIBRARY_PATH. I am running Ubuntu, made sure that these libraries were installed and were in the LD_LIBRARY_PATH, and was still getting the error. I noticed that my versions of these libraries were newer than those listed in the or1k-gcc/contrib/download_prerequesites script, so I tried running it to get those versions. Unfortunately, it seems the download locations listed in that script don't exist anymore. So I downloaded the listed versions of these libraries from the projects' websites (MPC 0.8.1, MPFR 2.4.2, and GMP 4.3.2), commented out the commands from the script that downloaded the tarballs, and reran the script. After this, the building of stage 1 of gcc worked. I'll post updates here with further information that might be useful.

Notes on Cross-compiling

A quick post with some useful links and commands:

• Building binutils, gcc, and gdb: link.
• See the section "Roll-your-own".
• Page with a nice table of available targets: link.
• Comments on examining gcc's IR with the "-fdump-tree-all" and '-fdump-rtl-all" options: link.
• Useful clang commands.
• Generate assembly:
• clang -S ./helloworld.c -target x86
• Emit LLVM intermediate code:
• clang -S -emit-llvm ./helloworld.c -o ./helloworld.s
• Compile LLVM intermediate code into assembly code:
• llc -march x86 ./helloworld.s
• Use binutils to generate object file:
• as ./helloworld.s -o helloworld.o
• Or gcc to generate executable:
• gcc ./helloworld.s -o helloworld
• Command for disassembling object file:
• objdump -d helloworld.o
• Command to extract text from object file:
• objcopy -O binary --only-section=.text /path/firmware.ko /content.bin

Update 9 Aug. 2016.

I realized it could be useful to explain here how to generate a MIPS binary.

The MIPS assembly can be generated using the clang command above:

• clang -S ./helloworld.c -target mips -o helloworld.s

To generate a MIPS binary, we can build binutils for MIPS.

• Download latest binutils from here: link.
• Then (assuming you are on version 2.27 of binutils):
• tar xzvf binutils-2.27
• mkdir build-binutils
• cd build-binutils
• ../binutils-2.27/configure --TARGET=mips-el-unknown-linux-gnu --prefix=<installpath>
• make
• make install
• helloworld.s can be assembled like this:
• <installpath>/mipsel-unknown-linux-gnu-as helloworld.s -o helloworld
• You can examine the contents of the output file using the built objdump, as described above:
• <installpath>/mipsel-unknown-linux-gnu-objdump -d helloworld
• And extract the text:
• <installpath>/mipsel-unknown-linux-gnu-objcopy -O helloworld --only-section=.text ./helloworld helloworld.text

Much of the above was taken from here: link.

Building Octave 3.8.2 on CentOS 6.6

I recently ran into some trouble building Octave on CentOS.  After some searching of different message boards that offered steps towards resolution that didn't solve the problem for me, I came across this post which pointed me in the right direction.

I downloaded the BLAS source from here (click the "blas.tgz" link under the section "Reference BLAS Verstion 3.5.0"), and built it using the instructions given on the post linked above

gfortran -shared -O2 *.f -o libblas.so -fPIC

As suggested by that post, pointing the configure script to the lapack build at /usr/lib64/liblapack.so.3 didn't work. Instead, it required to download the "atlas" package from the yum repos, and point the configure script to the lapack library provided with it.

A couple of additional steps were required to resolve some problems with the configure script. Namely, I had to install the readline-devel package and the qt-devel package (to allow building of the gui). The command for configuration finally was

./configure --with-blas=/home/user123/BLAS/libblas.so --with-lapack=/usr/lib64/atlas/liblapack.so.3

Multiline String Matching in Perl

I came across a page here that offers a good explanation of how to use Perl for multiline search and replace. I thought I'd make a note here for future reference.

$ perl -pe 'undef $/; s/stringOnLineOne.*\nstringOnLineTwo//' inputFiles.*

See the section on that page entitled "The Field Record Separators."

A post at perlmonks shows how to use this in practice for multiline search. See the last answer to the question on this page.

For good explanations of the "-p" and "-e" flags, see this page. In brief, "-e" allows entering commands via the command line (as opposed to via a file), and "-p" allows looping over input files and printing the output.

gem5: Instructions for Enabling Full System Memory Tracing

1. Setup your proxy.
• export http_proxy=http://myproxy.myserver.com:1234
2. Clone the gem5-stable repo.
•  hg clone http://repo.gem5.org/gem5-stable
3. Install gem5's dependencies (http://www.m5sim.org/Dependencies).
• Note that on CentOS 6, the standard repos don't provide a new enough version of SWIG. I was able to download SWIG from the SWIG website and build it.
• On CentOS 6, the protobuf package doesn't include gzip_stream.h, which gem5 will ask for if you want to do memory tracing. I posted on StackOverflow about this problem, which eventually led me to the EPEL bug-tracking system. A bug was filed for this problem over two years ago, but it hasn't been fixed yet. To get around this problem, I downloaded protobuf from Google and built it.
4. Add the bin directories for SWIG and protobuf builds to your PATH. 
5. Add the lib directory for protobuf to your LIBRARY_PATH (note that this is not LD_LIBRARY_PATH).
6. Apply this patch to make necessary modifications to code: link.
• Copy it into the gem5-stable/ directory and run "hg import memTraceExample.patch".
• You will need to modify the path given on line 416 to point to your protobuf include directory:

main.Append(CPPPATH=[Dir('<path to protobuf install directory>/include/')])

• This patch adds the necessary modifications to insert CommMonitors to track all accesses to L1, to main memory, and to the data TLB. Because the L1 log will grow rapidly, for longer simulations, you will want the option to turn this off. To do so, you can edit configs/common/CacheConfig.py and change the call to the CommMonitor constructor for the L1 to make trace_enable=False.
• This patch also modifies the code that does virtual to physical memory translation so that is prints the mapping of a virtual to physical address when a new page is allocated. By default, the code doesn't print the mapping in this case.
7. Test in system call emulation mode.
• This won't be doing full system simulation. Although in the end we want full system simulation, doing a run in SE mode will verify that we have things working up to this point. Run:

./build/X86/gem5.opt --debug-flags=MMU --debug-file=mmu_trace.log ./configs/example/se.py --cpu-type=timing --caches --l2cache -c tests/test-progs/hello/bin/x86/linux/hello 

• The "--debug-flags=MMU --debug-file=mmu_trace.log" options will create the debug file "mmu_trace.log" in the m5out/ directory which logs the virtual to physical address translations.
• The trace files CT_mon{1,2,3}.trz.gz in the directory m5out. The debug file shows the mappings of virtual to physical addresses, while the other files show the traces of memory access to L1, main memory, and the data TLB.
• To view the traces of memory accesses in the CT_mon files, you will use a script provided by gem5.
• To use this script, you will need to setup the python protobuf library. Go to the directory containing the protobuf package you downloaded in the earlier step. Go into the python subdirectory, and follow the steps in the README file.
• To install to a directory other than the default, run an "export PYTHONPATH=<protobuf install directory>/lib/python2.6/site-packages/". When you get to the step to run "python setup.py install", instead run "python setup.py install --prefix=<protobuf install directory/lib/python2.6/site-packages/>".
• Go back to you gem5-stable directory. Run the commands:

export PYTHONPATH=$PYTHONPATH:<protobuf-2.5.0 package directory>/python/

./util/decode_packet_trace.py m5out/CT_mon1.trc.gz outputfile.log

• This should translate the protobuf output log into an ascii file named "outputfile.log" that contains the trace of L1 accesses.

8. Test in full system mode.
• To run in full system mode, you will need to download some more files from the gem5 website.
• First, you'll need to download the full system files and config files for X86. They are available here: http://www.m5sim.org/Download. Extract both of these to a directory where they will live (for our example, we'll assume this directory is /home/user123/gem5-util/). They need to be in a place where gem5 can see them while they are running. When you extract these files, you will get two directories, "binaries" and "disks".
• In the "disks" directory, there is a file called "linux-x86.img". This needs to be renamed to "x86root.img".
• gem5 also needs a file called "linux2-bigswap.img" to be in the disks directory. The only place I've found this is in the full system files for the Alpha architecture posted on the gem5 website, here: http://www.m5sim.org/Download. Download the Alpha system files, extract them, and move the file "linux2-bigswap.img" from this package to the "disks" directory containing the file "x86root.img".
• Now, set the environment variable M5_PATH to point to /home/user123/gem5-util (or wherever you put the "disks" and "config" directories).
• Run gem5 in full system mode: 

./build/X86/gem5.opt --debug-enable=TLB --debug-file=tlb_trace.log ./configs/example/fs.py --cpu-type=timing --caches --l2cache

• Here again, the "--debug-enable" and "--debug-file" options will trace virtual to physical address translations. We use the "TLB" option here to log these translations rather than the "MMU" option as we did earlier, because in full system simulation a different class is performing these translations. Note that because you are logging all virtual to physical address translations, the "tlb_trace.log" file will get very large very fast.
• To mitigate this problem of the exploding "tlb_trace.log" file, I modifed the arch/x86/tlb.cc file to log translations only when page faults occur. Logging these translations will be important when we start executing our own applications. If we want to print out an address and map it back to the memory trace, we'll need to see how these translations are taking place. The modifications to tlb.cc with restricted logging can be obtained through these diff files: tlb.cc, tlb.hh.
• In another terminal, connect to the gem5 simulation using the m5term utility:
• CD to the directory util/term/, and run "./m5term 3456". This will connect you to the simulation. After the simulator is done booting the system, you should see a linux command prompt. From this command prompt, you can run standard linux commands. Shut down the simulation and make sure that you got a TLB trace in m5out/tlb_trace.log. 
• Switch back over to the terminal where the gem5 simulation is running, and hit CTRL-C. This shuts down the simulator. Examine the log file in the top of the gem5-stable directory. This log file will have the name that you chose when you opened the ofstream in src/mem/comm_monitor.cc. The log file should contain a trace of memory accesses performed by the entire system that was simulated in the last run.

9. Run a custom program in the full system simulation.
• We will want to run our own custom programs with the gem5 full system simulator. Although we can compile programs from within the simulator, this will be fairly slow and tedious. Instead, we can compile the programs natively, mount the disk image from linux, and copy the compiled program from our native linux into the mounted disk image, where it can be accessed by the simulated system.
• The instructions for mounting the image are given on the gem5 website here: http://www.m5sim.org/Disk_images#Mounting_an_image. Run:

sudo mount -o loop,offset=32256 /home/user123/gem5-util/disks/x86root.img /home/user123/gem5-util/fsmount/

• (This assumes you've already created the fsmount directory).
• Now, within your natively running Linux, write a program - call it test.c, for example - that you want to run in the gem5 simulation. Compile it using gcc, and copy the binary into /home/user123/gem5-util/fsmount/bin/.
• Restart the gem5 simulation, and you should be able to run the test executable that you placed in the bin directory from within the gem5 simulation.
• After running your program, you will probably want to be able to look in the memory trace to see how it is using memory. However, if you print out addresses in your program, they will print out as virtual addresses. The memory traces log physical addresses. Hence, you need to use the tlb_trace.log file to find out what mappings take place. Say you printed out some address 0x2ac97c7a8010 used by your program. Look in tlb_trace.log for an address starting with 0x2ac97c7XXXXX to see where the page containing your address was faulted in. This will tell you which physical address your virtual address maps to.
• Now, using your decode_packet_trace.py python script described in a previous step, translate one of your CT_mon files to ascii. Now, you will search this ascii file for your physical address. However, these addresses are logged in decimal, not hex. So translate your physical address to decimal, and search the file for it. It should be in there. Note that these addresses are going to be accessed on boundaries (as the pages were during page faults). So, for instance, if you are looking at the log for accesses to external memory, and the block size of your last level cache is 64B, you will see in the log an access to the 64B block containing your physical address.

Caveats

1. Last time I tried, if you copy files into the disk image while gem5 is running, gem5 won't be able to see them. I had to reboot gem5 for it to see the files I copied in. 
2. I am still working on understanding the memory trace. For example, my log contains long strings of writes to the address 0x20000000000000b0, which I don't yet understand.

---

Results of Trials with other Tools

Before trying gem5, I tried to perform memory tracing with some other tools. Here are some notes on what I learned from trying those tools.

QEMU: The vCSIMx86 Version

There has been a version of QEMU released with modifications specifically for the purpose of gathering system-wide memory traces, available here. I was able to get it running and gather some traces, but I ran into some difficulties with it and found that it had some significant limitations, including the following:

1. It works only with a very old version of QEMU (v. 1.1.0).
2. After booting the virtual machine (which takes quite awhile, even on a machine with fairly good specs), only the first run of an application will successfully initiate tracing.
3. It isn't well-supported, as there doesn't appear to be a mailing list or other such resource for offering help with it.
4. I didn't ever gain confidence that it was actually tracing all memory accesses in the system.
5. Note that I contacted the author, and he confirmed that #1 and #2 are known limitations of the tool.