Problem Solving Assignment 2

Due: Thursday of week 4 (1/29/2009) at 11:59pm


UPDATE (1/28): When sorting the SensorData objects in the insertInOrder method, you do not need to worry about the order of different elements that have the same cookedValue. So for example, if you have two SensorData elements X and Y that both have the same cookedValue, then they could be ordered XY or YX, both are okay. This means that your output might be different from the sample solution in this way, but that's fine.

A common task in scientific programming is to evaluate the raw data that comes in from a device that is used to measure some physical quantity. It can be anything from a car's speedometer, to a digital thermometer, to an electron microscope. In each case the device will send data to a computer in a very simple format which then needs to be interpreted and processed before anything useful can be done with the data. Oftentimes the device will produce measurements very rapidly, possibly even thousands per second.

In this assignment you will get some practice at performing some very simple raw data processing, by reading sensor data from the internet. This sensor data is collected by devices all over California, by several different agencies affiliated with UCSD. Whenever a new set of data is collected, it is dumped to a particular webpage in real time. You will be reading the data from this webpage and processing it according to a very simple scheme we have devised.

The sensor data webpage is at the following URL: http://mercali.ucsd.edu/status.txt
It updates every minute or so. Here is an example of what the sensor data looks like:

As you can see, the data is divided into several columns, where each row represents a single measurement from a single device at a single station. The data columns may not appear to be aligned correctly, because they use the tab character ('\t') to separate columns. More on the format of the data is found below.

The starting point of this program will be the main method of the DataProcessor class. Your program is supposed to perform the following actions: (this is repeated in more detail in the description of the main method below)

  1. Check the command-line parameters to make sure they are in the correct format. This program should have 2 command-line parameters. The first is a String that is the URL of the sensor data webpage. The second is a String that specifies the name of a sensor data channel (i.e. like the third column of the sensor data).
  2. Create a SensorReader object that will open the URL given at the command line, and then read the data set from the webpage. This will give you an ArrayList full of SensorData objects.
  3. Filter the data list in two different ways, to get two new lists of data. More on this is described below.
  4. Compute the average value among the filtered data.
  5. Print out the results in a particular format.

The classes and methods described below are designed to help you achieve these goals.

This homework requires skills with file I/O, ArrayLists, and some string parsing. You will need to write the following classes:

  1. SensorData -- This class will simply be a container for the sensor data that we will read from the webpage. Every line of the webpage will correspond to one SensorData instance.
  2. SensorReader -- This class will contain methods for reading from the webpage and parsing out the list of SensorData objects.
  3. DataProcessor -- This class will contain methods for processing the list of SensorData objects that are created by the SensorReader class.

Details

You must put a comment at the top of your code that specifies your: real name, login name, email address, and PID. You must also include a two-to-four sentence description of the program. Use a multi-line comment (/* */). YOU MUST DO THIS FOR EVERY JAVA FILE!

Comments Comments in the style detailed in the book ARE required. Your code must be properly commented to receive full credit.

SensorData

Each line of the webpage will contain 10 items that we want to record:

The SensorData class is designed to hold these items. One SensorData instance will hold one line's worth of data, which will have all of these 10 items.

The SensorData class should include the following fields and methods:

  1. SensorData (constructor) - The constructor should take no parameters, and does not need to do any initialization.
  2. String network - This field holds the network of the sensor data. You will also need to write getNetwork and setNetwork methods.
  3. String station - This field holds the station of the sensor data. You will also need to write getStation and setStation methods.
  4. String channel - This field holds the channel of the sensor data. You will also need to write getChannel and setChannel methods.
  5. String location - This field holds the location of the sensor data. You will also need to write getLocation and setLocation methods.
  6. double time - This field holds the timestamp of the sensor data. You will also need to write getTime and setTime methods.
  7. double calibration - This field holds the calibration value of the sensor data. You will also need to write getCalibration and setCalibration methods.
  8. char type - This field holds the type code of the sensor data. You will also need to write getType and setType methods.
  9. double sampleRate - This field holds the sampleRate of the sensor data. You will also need to write getSampleRate and setSampleRate methods.
  10. double rawValue - This field holds the raw value of the sensor data. You will also need to write getRawValue and setRawValue methods.
  11. double cookedValue - This field holds the cooked value of the sensor data. You will also need to write getCookedValue and setCookedValue methods.
  12. getDate - This method will take no parameters, and return a new java.util.Date object that represents the same date as the time field of this SensorData object. More on this below.

SensorReader

The SensorReader class will be responsible for reading the webpage and parsing out a list of SensorData objects. The SensorReader object will be constructed and given a specific URL to read from. Then when the user calls the getDataList method, it will open the webpage for reading, read the lines, parse each line into a SensorData object, add each SensorData object to an ArrayList, and finally return the list of SensorData objects. The SensorReader class should have the following fields and methods:

  1. String url - This field will contain the URL that this SensorReader should try to read from.
  2. SensorReader (constructor) - This constructor should take a String parameter which is the URL to read from, and store it in the url field.
  3. openURL - This method is given for you in the code below. It will open a network connection to the URL given by the url field and return a BufferedReader which is setup to read from the webpage. Here is the source code for openURL, which YOU MUST COPY DIRECTLY INTO YOUR SensorReader.java FILE:
    private BufferedReader openURL() {
       BufferedReader result = null;
       try {
          URL urlObject = new URL(this.url);
          URLConnection connection = urlObject.openConnection();
          InputStream in = connection.getInputStream();
          result = new BufferedReader(new InputStreamReader(in));
       } catch (MalformedURLException murle) {
          System.out.println("URL string is invalid");
          System.exit(0);
       } catch (IOException ioe) {
          System.out.println("Error opening URL connection");
          System.exit(0);
       }
       return result;
    }
    The BufferedReader that this methods returns will be open and all ready to read from. You will still need to close it when you are done reading from it. Because of the wacky network code that we had to use, you will need to include an additional import statement at the top of your SensorReader file:
    import java.net.*;
  4. parseDataLine - This method will take a String as a parameter, which is meant to be a single line of the webpage. This method will parse the line into the data items described above. Once the data is parsed, you will use it to create a single SensorData instance. You will set all of the data in the SensorData object, and then return it.

    The data in each line is tab-delimited. This means that there is a tab character ('\t') in between each data item. In fact, some of the lines are missing certain pieces of data, so there will be two tab characters in a row. The only piece of data that might be missing is the location item. If it is missing, its value in the line will be the empty string ("").

    The best way to parse the individual items out of the line is to use the split method of the String class. Since the data items are tab-delimited, we will use "\t" as the delimiter we pass to split. This will return an array of strings, that is the result of chopping up the line at every place where there was a tab character. If there are two tab characters in a row (for example, if the location item is missing), then that element of the array will be the empty string (""). For example:

    "a\tb\tc\td".split("\t") == {"a", "b", "c", "d"}

    "a\t\tb".split("\t") == {"a", "", "b"}

    No special action needs to be taken if the location item is missing.

    Some of the fields of SensorData are doubles or chars. Two useful questions to ask yourself are:

    1. How do I convert a String to a double?
    2. How do I convert a String to a single char?

    NOTE: You may assume that the line you are given will always have the 10 tab-separated columns, and that we will not run your code on any webpage that does not have the right format.

  5. readDataLines - This method will take a BufferedReader as a parameter and read all the lines from it. For each line you read, you will pass it to parseDataLine, which will return you a SensorData object. You will add all the SensorData objects to an ArrayList, in the order that you read them. After all of the lines have been read and parsed, you will close the BufferedReader and then return the ArrayList. One VERY IMPORTANT THING TO REMEMBER is that the first line of the webpage is NOT A VALID DATA LINE. The first line instead contains the labels of the columns, so for our purposes it is garbage. You must remember to discard the first line and only try to parse the others.

    Since this method will only read from the BufferedReader and not try to open a file, you WILL NOT need to catch a FileNotFoundException. You WILL still need to catch a general Exception, and if one occurs you should print "Error reading from webpage" and then exit the program.

  6. getDataList - This method will combine the behavior of all the others. It will take no parameters. This method will first open the network stream and get a BufferedReader by calling openURL. Next, it will pass that BufferedReader to readDataLines and get back an ArrayList full of SensorData objects. Finally, it will return the ArrayList.

DataProcessor

This class will contain the main method, and will use the SensorReader class to perform some operations on the sensor data from the webpage. This class will read in the sensor data, then filter it in two different ways. It will then compute the average of the cooked values for all the data in the filtered list. Finally, it will print out both filtered lists, and the average that was computed.

The DataProcessor class should have the following methods:

  1. filter - This method will take an ArrayList and a String as parameters. The ArrayList parameter is assumed to contain SensorData objects. The String parameter will be the name of a channel, as in the third column of the sensor data webpage (i.e. it might be "Temp-out" or "windgust", etc). The filter method will create a new ArrayList and copy the elements of the input list into this new list, but ONLY the elements whose 'channel' fields are equal to the second input parameter.

    So, for example, if the second input parameter was the string "Temp-out", then you would loop over all the elements of the input list, and for each SensorData object that had a channel field equal to "Temp-out", you would add that object to the new list.

    At the end, this method will return the new filtered list.

  2. filterInOrder - This method is similar to filter, except that it requires you to order the resulting ArrayList in a particular way. Like filter, this method takes an ArrayList and a String as parameters. You will create a new ArrayList and copy elements from the input list into the new list when they have the correct channel field. The difference between this method and filter is that now when you add an element to the new list, you must add it in increasing order of the 'cooked value' of the SensorData objects. Inserting elements in the correct order will be achieved by making a call to the insertInOrder method, described below. At the end, this method returns the new list.
  3. insertInOrder - This method takes an ArrayList and a SensorData as parameters. It is assumed that the ArrayList contains SensorData objects. This method will insert the SensorData into the ArrayList, but in the correct location so that all of the SensorData objects in the list are sorted in increasing order by their 'cooked value' fields. You should assume that the input list is either empty or is already sorted by its 'cooked value' field. You will simply be inserting the new element into the list in such a way that it is still sorted afterwards. Here is an example:
    ArrayList list =
    cookedValue=1.0cookedValue=3.0cookedValue=4.0

    SensorData newdata = new SensorData();
    newdata.setCookedValue(2.0);
    insertInOrder(list, newdata);

    List after call to insertInOrder:
    cookedValue=1.0cookedValue=2.0cookedValue=3.0cookedValue=4.0

    The easiest way to write this method is to loop over the input list until you find the first element whose cooked value is >= the cooked value of the input SensorData object. Then you insert the object into the list at that location. If you get to the end of the list without finding such an element, you just add the new element onto the end of the list.

  4. computeAverage - This method will take an ArrayList of SensorData objects as a parameter. This method will compute the average of all the 'cooked value' fields of the elements in this list, and then return the average. Since the cooked values are all doubles, the average will also be a double. The average is computed by summing all of the 'cooked value' fields, and then dividing the sum by the number of elements in the list.
  5. printDataList - This method takes an ArrayList as a parameter. The ArrayList is full of SensorData objects, and this method will print them out in a particular way. Specifically, for every element E in the list, you should print out the following string:
    On <date>, the value at station <station> was <cooked value>
    Where the <date> string is the result of calling E.getDate().toString(), and the <station> string is the value of E.getStation(), and the <cooked value> string is the value of E.getCookedValue(). The SensorData objects should be printed in the same order they appear in the list.
  6. main - This is the main method of the program. It will tie together all the other methods of this class. This program will take 2 command-line parameters:
    1. The URL string of the webpage
    2. The 'channel' string that will be used for filtering
    If there are fewer than 2 command-line parameters given, the program should print the following message, and then exit:
    USAGE: java DataProcessor (url) (channel)
    If there are more than 2 command-line parameters given, the extras will be ignored and the program continues like normal.

    This method is fairly involved, so we will approach it piece by piece:

    Here is an example of what the final output of main could look like:

    > java DataProcessor  http://www.cse.ucsd.edu/classes/wi09/cse8b/psas/status.txt  Temp-out

    Average value of Temp-out is 22.739644733333332
    Filtered data list:
    On Tue Jan 13 13:27:59 PST 2009, the value at station HERMT was 24.111111
    On Tue Jan 13 13:25:59 PST 2009, the value at station NICMT was 24.444445
    On Tue Jan 13 13:27:59 PST 2009, the value at station IBShore was 19.222222
    On Tue Jan 13 12:13:59 PST 2009, the value at station WLAW1 was 26.722222
    On Tue Jan 13 13:13:59 PST 2009, the value at station USCMT was 24.833334
    On Mon Jan 12 12:05:59 PST 2009, the value at station DANMT was 29.666667
    On Tue Jan 13 04:11:59 PST 2009, the value at station GVAW1 was 20.333333
    On Tue Jan 13 13:27:59 PST 2009, the value at station SDSC was 24.444445
    On Tue Jan 13 13:27:59 PST 2009, the value at station ZUMMT was 22.944445
    On Tue Jan 13 13:25:59 PST 2009, the value at station PTDMT was 22.333333
    On Tue Jan 13 13:29:59 PST 2009, the value at station Solar1 was 27.928001
    On Tue Jan 13 13:26:59 PST 2009, the value at station CABMT was 25.833334
    On Tue Jan 13 13:27:59 PST 2009, the value at station VENMT was 23.555556
    On Wed Jan 07 13:16:59 PST 2009, the value at station TOPMT was 13.777778
    On Wed Jan 07 07:09:59 PST 2009, the value at station MALMT was 10.944445
    Filtered and sorted data list:
    On Wed Jan 07 07:09:59 PST 2009, the value at station MALMT was 10.944445
    On Wed Jan 07 13:16:59 PST 2009, the value at station TOPMT was 13.777778
    On Tue Jan 13 13:27:59 PST 2009, the value at station IBShore was 19.222222
    On Tue Jan 13 04:11:59 PST 2009, the value at station GVAW1 was 20.333333
    On Tue Jan 13 13:25:59 PST 2009, the value at station PTDMT was 22.333333
    On Tue Jan 13 13:27:59 PST 2009, the value at station ZUMMT was 22.944445
    On Tue Jan 13 13:27:59 PST 2009, the value at station VENMT was 23.555556
    On Tue Jan 13 13:27:59 PST 2009, the value at station HERMT was 24.111111
    On Tue Jan 13 13:25:59 PST 2009, the value at station NICMT was 24.444445
    On Tue Jan 13 13:27:59 PST 2009, the value at station SDSC was 24.444445
    On Tue Jan 13 13:13:59 PST 2009, the value at station USCMT was 24.833334
    On Tue Jan 13 13:26:59 PST 2009, the value at station CABMT was 25.833334
    On Tue Jan 13 12:13:59 PST 2009, the value at station WLAW1 was 26.722222
    On Tue Jan 13 13:29:59 PST 2009, the value at station Solar1 was 27.928001
    On Mon Jan 12 12:05:59 PST 2009, the value at station DANMT was 29.666667

Dates, Time, and this thing called the "Epoch"

Part of the sensor data we will be collecting involves a timestamp, which reflects the time at which the sensor data was collected. But how can you use a single number to record the second, minute, hour, day, month, and year that something happened?

The trick that computer scientists use is to encode a moment in time in terms of how much time has passed since some fixed reference point. So if we fixed our reference point to be May 8, 1981 at 3:15pm (Mike's birth date), then we could say that the turnin date for PSA1 (Jan 15, 2009 at 11:59pm) is 873884640, because that date is 873884640 seconds after our reference point.

So once we fix a reference point, we can express a moment in time exactly by recording the number of seconds that have passed since that reference point. So which reference point should we use? Not everyone recognizes Mike's birth date as the pivotal moment in history that it is, so computer scientists have decided to make the reference point be January 1st, 1970 at 12:00am (i.e. the first instant of the 1970's). This moment has become such a standard reference point that we give it a special name; "the Epoch".

The class java.util.Date is a representation of a particular date, including the year, month, day, hour, minute, second, and milliseconds. Internally, it just contains a single long integer value, which represents the date in terms of milliseconds since the Epoch, and the Date constructor just takes a long integer parameter. You will need to use this to implement the getDate method of SensorData.
(NOTE: The sensor data expresses times in terms of seconds since the Epoch, and the Date class is constructed by giving it a time in milliseconds since the Epoch. How do I convert from seconds to milliseconds?)

Our main reason for using the Date class is that it has a very nice toString method, which prints out the date in a very pretty format. This is used in the printDataList method of DataProcessor.

Testing/Grading

You can test your program by running it on the live sensor data webpage. The URL for this webpage is http://mercali.ucsd.edu/status.txt. THIS WEBPAGE CONTINUALLY CHANGES and so you might have trouble reproducing your results from one run to the next.

Since the data webpage is constantly updating, it is hard to get consistent results when testing your program. That is why we have made a copy of one particular version of the webpage and posted it here at http://www.cse.ucsd.edu/classes/wi09/cse8b/psas/status.txt so you can test your code on a page that doesn't change.

In addition, you can create your own copies of the webpage at some point in time and save them on your hard drive. Then you can run your program on them by using a file URL. A file URL is a URL that points to your hard drive, rather than the internet. It uses the following format:

file://<full-path-to-file>/filename
So for example, if I made a copy of the data webpage in my home folder on ieng6 and called it status.txt, then I could run my program on it as follows:
> java DataProcessor file:///home/linux/ieng6/cs8f/cs8fzz/status.txt ...
Notice that I have 3 slashes right after the file: part of the URL. This is because there are 2 slashes for the URL format, and 1 more slash from the path since it is an absolute path. You must use an absolute path when making a file URL.

Sample solution

We will once again have a sample solution that you can run on ieng6 to test to see if your code performs the correct way. You can run our sample solution by running:

runsamplePSA2 <url> <channel>
at the command line.

Turning in your homework

We will be using a Bundle script to do automated turnin for this assignment.

On ieng6 somewhere under your home directory you need to make a folder called PSA2. All of your files for this assignment must go in this folder. The files you need to turn in are:

(You do NOT need to copy bookClasses into this folder. We're not using bookClasses for this assignment.)
When you have completed the assignment and have all of these files in the PSA2 folder, you should cd into that folder and then run the bundleP2 command.


Here are several examples of the expected input/output behavior that your code should have. For each of these examples, your code should print EXACTLY the same output.

> java DataProcessor
USAGE: java DataProcessor (url) (channel)


> java DataProcessor http://mercali.ucsd.edu/status.txt
USAGE: java DataProcessor (url) (channel)


> java DataProcessor  http://not-existent-webpage.com  Temp-out
Error opening URL connection


> java DataProcessor  not-even-valid-url-format  Temp-out
URL string is invalid


> java DataProcessor  http://mercali.ucsd.edu/status.txt   Not-a-real-channel
Average value of Not-a-real-channel is NaN
Filtered data list:
Filtered and sorted data list:


> java DataProcessor  http://www.cse.ucsd.edu/classes/wi09/cse8b/psas/status.txt  Temp-out
Average value of Temp-out is 22.739644733333332
Filtered data list:
On Tue Jan 13 13:27:59 PST 2009, the value at station HERMT was 24.111111
On Tue Jan 13 13:25:59 PST 2009, the value at station NICMT was 24.444445
On Tue Jan 13 13:27:59 PST 2009, the value at station IBShore was 19.222222
On Tue Jan 13 12:13:59 PST 2009, the value at station WLAW1 was 26.722222
On Tue Jan 13 13:13:59 PST 2009, the value at station USCMT was 24.833334
On Mon Jan 12 12:05:59 PST 2009, the value at station DANMT was 29.666667
On Tue Jan 13 04:11:59 PST 2009, the value at station GVAW1 was 20.333333
On Tue Jan 13 13:27:59 PST 2009, the value at station SDSC was 24.444445
On Tue Jan 13 13:27:59 PST 2009, the value at station ZUMMT was 22.944445
On Tue Jan 13 13:25:59 PST 2009, the value at station PTDMT was 22.333333
On Tue Jan 13 13:29:59 PST 2009, the value at station Solar1 was 27.928001
On Tue Jan 13 13:26:59 PST 2009, the value at station CABMT was 25.833334
On Tue Jan 13 13:27:59 PST 2009, the value at station VENMT was 23.555556
On Wed Jan 07 13:16:59 PST 2009, the value at station TOPMT was 13.777778
On Wed Jan 07 07:09:59 PST 2009, the value at station MALMT was 10.944445
Filtered and sorted data list:
On Wed Jan 07 07:09:59 PST 2009, the value at station MALMT was 10.944445
On Wed Jan 07 13:16:59 PST 2009, the value at station TOPMT was 13.777778
On Tue Jan 13 13:27:59 PST 2009, the value at station IBShore was 19.222222
On Tue Jan 13 04:11:59 PST 2009, the value at station GVAW1 was 20.333333
On Tue Jan 13 13:25:59 PST 2009, the value at station PTDMT was 22.333333
On Tue Jan 13 13:27:59 PST 2009, the value at station ZUMMT was 22.944445
On Tue Jan 13 13:27:59 PST 2009, the value at station VENMT was 23.555556
On Tue Jan 13 13:27:59 PST 2009, the value at station HERMT was 24.111111
On Tue Jan 13 13:25:59 PST 2009, the value at station NICMT was 24.444445
On Tue Jan 13 13:27:59 PST 2009, the value at station SDSC was 24.444445
On Tue Jan 13 13:13:59 PST 2009, the value at station USCMT was 24.833334
On Tue Jan 13 13:26:59 PST 2009, the value at station CABMT was 25.833334
On Tue Jan 13 12:13:59 PST 2009, the value at station WLAW1 was 26.722222
On Tue Jan 13 13:29:59 PST 2009, the value at station Solar1 was 27.928001
On Mon Jan 12 12:05:59 PST 2009, the value at station DANMT was 29.666667


> java DataProcessor  http://www.cse.ucsd.edu/classes/wi09/cse8b/psas/status.txt  forecast
Average value of forecast is 46.5
Filtered data list:
On Tue Jan 13 13:27:59 PST 2009, the value at station HERMT was 51.0
On Tue Jan 13 13:27:59 PST 2009, the value at station WILMT was 61.0
On Tue Jan 13 13:25:59 PST 2009, the value at station NICMT was 51.0
On Tue Jan 13 13:27:59 PST 2009, the value at station IBShore was 9.0
On Tue Jan 13 13:13:59 PST 2009, the value at station USCMT was 1.0
On Mon Jan 12 12:05:59 PST 2009, the value at station DANMT was 44.0
On Tue Jan 13 13:27:59 PST 2009, the value at station SDSC was 51.0
On Tue Jan 13 13:27:59 PST 2009, the value at station ZUMMT was 51.0
On Tue Jan 13 13:25:59 PST 2009, the value at station PTDMT was 63.0
On Tue Jan 13 13:26:59 PST 2009, the value at station CABMT was 69.0
On Tue Jan 13 13:27:59 PST 2009, the value at station VENMT was 51.0
On Fri Jan 09 13:44:59 PST 2009, the value at station MALMT was 44.0
On Fri Jan 09 13:24:59 PST 2009, the value at station TOPMT was 44.0
On Tue Jan 13 13:26:59 PST 2009, the value at station TPSMT was 61.0
Filtered and sorted data list:
On Tue Jan 13 13:13:59 PST 2009, the value at station USCMT was 1.0
On Tue Jan 13 13:27:59 PST 2009, the value at station IBShore was 9.0
On Mon Jan 12 12:05:59 PST 2009, the value at station DANMT was 44.0
On Fri Jan 09 13:44:59 PST 2009, the value at station MALMT was 44.0
On Fri Jan 09 13:24:59 PST 2009, the value at station TOPMT was 44.0
On Tue Jan 13 13:27:59 PST 2009, the value at station HERMT was 51.0
On Tue Jan 13 13:25:59 PST 2009, the value at station NICMT was 51.0
On Tue Jan 13 13:27:59 PST 2009, the value at station SDSC was 51.0
On Tue Jan 13 13:27:59 PST 2009, the value at station ZUMMT was 51.0
On Tue Jan 13 13:27:59 PST 2009, the value at station VENMT was 51.0
On Tue Jan 13 13:27:59 PST 2009, the value at station WILMT was 61.0
On Tue Jan 13 13:26:59 PST 2009, the value at station TPSMT was 61.0
On Tue Jan 13 13:25:59 PST 2009, the value at station PTDMT was 63.0
On Tue Jan 13 13:26:59 PST 2009, the value at station CABMT was 69.0