Problem Solving Assignment 2

Due: Thursday of week 4 (4/23/2009) at 11:59pm

A common task in scientific programming is to evaluate the raw data that comes in from a device that is used to measure some physical quantity. It can be anything from a car's speedometer, to a digital thermometer, to an electron microscope. In each case the device will send data to a computer in a very simple format which then needs to be interpreted and processed before anything useful can be done with the data. Oftentimes the device will produce measurements very rapidly, possibly even thousands per second.

In this assignment you will get some practice at performing some very simple raw data processing, by reading sensor data from the internet. This sensor data is collected by oceanic buoy stations administered by Scripps Institution of Oceanography at UCSD. We will be using data from two local buoys, one in Oceanside and another in Torrey Pines. Whenever a new set of data is collected, it is dumped to a particular webpage in real time. You will be reading the data for each buoy station from each station's webpage and processing it according to a very simple scheme we have devised.

Buoy stations

It updates every hour or so. Here is an example of what the sensor data looks like:

As you can see, the data is divided into several columns, where each row represents a single measurement from a single buoy station. The data columns use white space characters ('\s') to separate columns. More on the format of the data is found below.

The starting point of this program will be the main method of the DataProcessor class. Your program is supposed to perform the following actions: (this is repeated in more detail in the description of the main method below)

  1. Check the command-line parameters to make sure they are in the correct format. This program should have 2 command-line parameters. The first is a String that is the URL of the Oceanside sensor data webpage. The second is a String that is the URL of the Torrey Pines sensor data webpage.
  2. Create a SensorReader object that will open the URL given at the command line, and then read the data set from the webpage. This will give you an ArrayList full of SensorData objects. (Do this for Oceanside and then for Torrey Pines)
  3. Filter the data list in two different ways, to get two new lists of data. More on this is described below. (Do this for Oceanside and then for Torrey Pines)
  4. Compute the average value among the filtered data. (Do this for Oceanside and then for Torrey Pines)
  5. Print out the results in a particular format. (Do this for Oceanside and then for Torrey Pines)

The classes and methods described below are designed to help you achieve these goals.

This homework requires skills with file I/O, ArrayLists, and some string parsing. You will need to write the following classes:

  1. SensorData -- This class will simply be a container for the sensor data that we will read from a webpage. Every line of the webpage will correspond to one SensorData instance.
  2. SensorReader -- This class will contain methods for reading from a webpage and parsing out a list of SensorData objects.
  3. DataProcessor -- This class will contain methods for processing the lists of SensorData objects that are created by the SensorReader class.
Below we provide a full specification of the design of the solution you should use. This gives you practice with the kinds of ``instructions'' one gets in industry. If you have trouble understanding the specification as described below, you should ask before you begin working.

Details

You must put a comment at the top of your code that specifies your: real name, login name, email address, and PID. You must also include a two-to-four sentence description of the program. Use a multi-line comment (/* */). YOU MUST DO THIS FOR EVERY JAVA FILE!

Comments Comments in the style detailed in the book ARE required. Your code must be properly commented to receive full credit.

SensorData

Each line of the webpage will contain 15 items that we want to record:

Note on the "height" items (overall wave height, swell wave height, wind wave height):
These items contain data about the highest one-third of the waves. You don't have to do any additional computation on this data to find the highest one-third of waves. Just use them as given.

The SensorData class is designed to hold these items. One SensorData instance will hold one line's worth of data, which will have all of these 15 items.

The SensorData class should include the following fields and methods:

  1. SensorData (constructor) - The constructor should take no parameters, and does not need to do any initialization.
  2. Calendar measurementDate - This field holds the date of the measurement. You will also need to write getMeasurementDate and setMeasurementDate methods.
  3. double overallWaveHeight - This field holds the overall wave height. You will also need to write getOverallWaveHeight and setOverallWaveHeight methods.
  4. double swellWaveHeight - This field holds the swell height. You will also need to write getSwellWaveHeight and setSwellWaveHeight methods.
  5. double swellWavePeriod - This field holds the swell period. You will also need to write getSwellWavePeriod and setSwellWavePeriod methods.
  6. double windWaveHeight - This field holds the wind wave height. You will also need to write getWindWaveHeight and setWindWaveHeight methods.
  7. double windWavePeriod - This field holds the wind wave period. You will also need to write getWindWavePeriod and setWindWavePeriod methods.
  8. String swellWaveDirection - This field holds the swell direction. You will also need to write getSwellWaveDirection and setSwellWaveDirection methods.
  9. String windWaveDirection - This field holds the wind wave direction. You will also need to write getWindWaveDirection and setWindWaveDirection methods.
  10. String steepness - This field holds the steepness. You will also need to write getSteepness and setSteepness methods.
  11. double overallWavePeriod - This field holds the overall wave period. You will also need to write getOverallWavePeriod and setOverallWavePeriod methods.
  12. int overallWaveDirection - This field holds the overall wave direction. You will also need to write getOverallWaveDirection and setOverallWaveDirection methods.

SensorReader

The SensorReader class will be responsible for reading the webpage and parsing out a list of SensorData objects. The SensorReader object will be constructed and given a specific URL to read from. Then when the user calls the getDataList method, it will open the webpage for reading, read the lines, parse each line into a SensorData object, add each SensorData object to an ArrayList, and finally return the list of SensorData objects. The SensorReader class should have the following fields and methods:

  1. String url - This field will contain the URL that this SensorReader should try to read from.
  2. SensorReader (constructor) - This constructor should take a String parameter which is the URL to read from, and store it in the url field.
  3. openURL - This method is given for you in the code below. It will open a network connection to the URL given by the url field and return a BufferedReader which is setup to read from the webpage. Here is the source code for openURL, which YOU MUST COPY DIRECTLY INTO YOUR SensorReader.java FILE:
    private BufferedReader openURL() {
       BufferedReader result = null;
       try {
          URL urlObject = new URL(this.url);
          URLConnection connection = urlObject.openConnection();
          InputStream in = connection.getInputStream();
          result = new BufferedReader(new InputStreamReader(in));
       } catch (MalformedURLException murle) {
          System.out.println("URL string is invalid");
          System.exit(0);
       } catch (IOException ioe) {
          System.out.println("Error opening URL connection");
          System.exit(0);
       }
       return result;
    }
    The BufferedReader that this methods returns will be open and all ready to read from. You will still need to close it when you are done reading from it. Because of the wacky network code that we had to use, you will need to include an additional import statement at the top of your SensorReader file:
    import java.net.*;
  4. parseDataLine - This method will take a String as a parameter, which is meant to be a single line of the webpage. This method will parse the line into the data items described above. Once the data is parsed, you will use it to create a single SensorData instance. You will set all of the data in the SensorData object, and then return it.

    The data in each line is white-space-delimited. This means that there is at least one space character ('\s') in between each data item.

    The best way to parse the individual items out of the line is to use the split method of the String class. Since the data items are delimited by one or more space characters, we will use "\\s+" as the delimiter we pass to split. This will return an array of strings, which is the result of chopping up the line at every place where there were one or more space characters in a row.

    "a\sb\sc\sd".split("\\s+") == {"a", "b", "c", "d"}

    "a\s\s\sb\s\s\sc\s\s\sd".split("\\s+") == {"a", "b", "c", "d"}

    Some of the fields of SensorData are doubles or ints. Two useful questions to ask yourself are:

    1. How do I convert a String to a double?
    2. How do I convert a String to an int?

    One of the fields (measurementDate) is a Calendar. The java.util.Calendar class is used to store and convert date information. You can read more information about this class in the Java API documenation for Calendar.

    To use the Calendar class, you will need to include an additional import statement at the top of your SensorData file:
    import java.util.*;

    5. To get a new Calendar object, you should call this method (instead of calling "new" since Calendar does not provide a public constructor):

    static Calendar getInstance()
    To set the time of the Calendar object, you should call this method (the date parameter is the day of the month):
    void set(int year, int month, int date, int hour, int minute, int second)
    NOTE:
    In the input data, the month is 1-based (month 1 is January).
    In the Calendar class, the month is 0-based (month 0 is January).
    You must subtract 1 from the input month before you set the Calendar month.
    For example, if you read in a line of data where the month is 4, you should set the month to 3 in setMeasurementDate

    NOTE:
    The "seconds" are not given in the input data, so you can set the seconds to 0 in measurementDate.

    Here is an example of creating a Calendar object and setting the time:

    Calendar calendar = Calendar.getInstance();
    calendar.set(2009, 4, 23, 11, 59, 0);

    NOTE: You may assume that the line you are given will always have the 15 space-separated columns, and that we will not run your code on any webpage that does not have the right format.

  5. readDataLines - This method will take a BufferedReader as a parameter and read all the lines from it. For each line you read, you will pass it to parseDataLine, which will return you a SensorData object. You will add all the SensorData objects to an ArrayList, in the order that you read them. After all of the lines have been read and parsed, you will close the BufferedReader and then return the ArrayList. One VERY IMPORTANT THING TO REMEMBER is that the first TWO LINES of the webpage are NOT VALID DATA LINES. Instead they are just header lines, which contain the labels of the columns, so for our purposes they are garbage. You must remember to discard the first TWO lines and only try to parse the others.

    Since this method will only read from a BufferedReader you will need to catch a general Exception, and if one occurs you should print "Error reading from webpage" and then exit the program.

  6. getDataList - This method will combine the behavior of all the others. It will take no parameters. This method will first call openURL to open the network stream and get a BufferedReader. Next, it will pass that BufferedReader to readDataLines and get back an ArrayList containing SensorData objects. Finally, it will return the ArrayList.

DataProcessor

This class will contain the main method -- this is a set of instructions to read and do some calculations on our two ocean bouys -- theoretically based on something our boss, the scientist might ask us to calculate from the data. This class will read in the sensor data (by calling methods of the SensorReader class), then perform a few calculations. You will perform these calculations twice: first for Oceanside, second for Torrey Pines.

  1. Find and print the average overall wave height.
  2. Find and print good times for sailing. More detail on how to determine the "good" sailing times is below.
  3. Create and print a list of the good sailing times (that you found in the previous step) ordered by swell wave height, from smallest swell wave height to largest swell wave height -- filtering out "faulty data".

The DataProcessor class should have the following methods:

  1. findGoodSailingTimes - This method will take an ArrayList as a parameter. The ArrayList parameter is assumed to contain SensorData objects. This method will return a NEW Arraylist that only contains those entries of the times when it would be good to sail near this buoy. We will define a good time to sail as one when both these conditions hold: This method should return a new ArrayList which contains only those SensorData entries which meet the criteria of a "good" sailing time. In other words, this method returns a new ArrayList with only a subset of the original list items in it.

  2. sortInOrder - This method has been requested by the scientists to allow them to see the data from each bouy ordered by swell wave height (they have some animals they are studying that are sensitive to swell wave height). They would like a new ArrayList created that is a copy of the input ArrayList ordered from smallest swell wave height to largest swell wave height.

    However, there is one more complication -- sometimes the sensor readings from the buoys are faulty (this can happen when an animal swims into them, for instance). Any entry with a swell wave height of 0.5 or higher should be filtered from (not included in) the ordered list. Additionally, this method should print out a message before it returns the ordered list to say how many elements of the original ArrayList were removed from the ordered list. The message that should be printed is (where x should be replaced by the number of elements discarded: 

    Number of sensor readings discarded due to overly high swell wave height: x

    In summary, sortInOrder should take an ArrayList input parameter (the unsorted, unfiltered list) and return an ArrayList (the sorted and filtered list).

    The way you will make an arrayList in sorted order is by simply inserting the new element into the list in such a way that the new list is still sorted afterwards. Here is an example (just showing how this would work with a list of ints: 

    Starting List: 15, 10, 13, 12

New List:

New List:
15

New List:
10, 15
 
New List:
10, 13, 15

New List:
10, 12, 13, 15

    The trick to making this easy is to decompose the problem into 2 parts. In this method (sortInOrder) do the first part -- loop over all the elements in the original list and for each element (that meets our swell wave height requirements) call a method insertInOrder sending two parameters -- the new list to insert into and the element of the starting list that you want to insert. Below see the method description of insertInOrder.

  3. insertInOrder - This method takes an ArrayList and an element to insert "in order" in that ArrayList. The value it uses to insert in order is the swell wave height. The easiest way to write this method is to loop over the new list until you find the first element whose swell wave height is >= the swell wave height of the of the SensorData object. Then you insert the object into the list at that location. If you get to the end of the list without finding such an element, you just add the new element onto the end of the list.
  4. computeAverage - This method will take an ArrayList of SensorData objects as a parameter. This method will compute the average of all the overall wave heights for the elements in the list and then return the average. Since the wave heights are all doubles, the average will also be a double.
  5. printDataList - This method takes an ArrayList as a parameter. The ArrayList contains SensorData objects, and this method will print them out in a particular way. Specifically, for every element E in the list, you should print out the following string:
    On <measurement date>, the overall wave height was <overall wave height>, the wind wave height was <wind wave height>, the swell wave height was <swell wave height>, and the steepness was <steepness>
    Where the <measurement date> string is the result of calling E.getMeasurementDate().getTime(), and the <overall wave height> string is the result of calling E.getOverallWaveHeight(), and the <wind wave height> string is the result of calling E.getWindWaveHeight(), and the <swell wave height> string is the result of calling E.getSwellWaveHeight(), and the <steepness> string is the result of calling E.getSteepness(), The SensorData objects should be printed in the same order they appear in the list.
  6. main - This is the main method of the program. It will tie together all the other methods of this class. This program will take 2 command-line parameters:
    1. The URL string of the webpage containing sensor data from Oceanside buoy station
    2. The URL string of the webpage containing sensor data from Torrey Pines buoy station
    If there are fewer than 2 command-line parameters given, the program should print the following message, and then exit:
    USAGE: java DataProcessor (urlOceanside) (urlTorreyPines)
    If there are more than 2 command-line parameters given, the extras will be ignored and the program continues like normal.

    After verifying the command-line parameters, the program should do the following steps for each buoy station (Oceanside and then Torrey Pines):

    To run your code on sample data, type the following on the command line:

    > java DataProcessor http://cs.ucsd.edu/classes/sp09/cse8b/psas/PSA2/oceanside.txt http://cs.ucsd.edu/classes/sp09/cse8b/psas/PSA2/torrey_pines.txt

    An example of what the final output of main could look like is here.

Testing/Grading

You can test your program by running it on the live sensor data on the buoy station webpages. THESE WEBPAGES CONTINUALLY CHANGE and so you might have trouble reproducing your results from one run to the next.

Since the data webpage is constantly updating, it is hard to get consistent results when testing your program. That is why we have made a copy of one particular version of the sensor data for each buoy and posted them so you can test your code on data that doesn't change:

http://cs.ucsd.edu/classes/sp09/cse8b/psas/PSA2/oceanside.txt
http://cs.ucsd.edu/classes/sp09/cse8b/psas/PSA2/torrey_pines.txt

Sample solution

We will once again have a sample solution that you can run on ieng6 to test to see if your code performs the correct way. You can run our sample solution by running:

runsamplePSA2 <urlOceanside> <urlTorreyPines>
at the command line.

Turning in your homework

We will be using a Bundle script to do automated turnin for this assignment.

On ieng6 somewhere under your home directory you need to make a folder called PSA2. All of your files for this assignment must go in this folder. The files you need to turn in are:

When you have completed the assignment and have all of these files in the PSA2 folder, you should cd into that folder and then run the bundleP2SA command.

Here are several examples of the expected input/output behavior that your code should have. For each of these examples, your code should print EXACTLY the same output.

> java DataProcessor
USAGE: java DataProcessor (urlOceanside) (urlTorreyPines)


> java DataProcessor http://cs.ucsd.edu/classes/sp09/cse8b/psas/PSA2/oceanside.txt
USAGE: java DataProcessor (urlOceanside) (urlTorreyPines)


> java DataProcessor http://not-existent-webpage.com http://not-existent-webpage.com
Error opening URL connection


> java DataProcessor not-even-valid-url-format not-even-valid-url-format
URL string is invalid


> java DataProcessor http://cs.ucsd.edu/classes/sp09/cse8b/psas/PSA2/oceanside.txt http://cs.ucsd.edu/classes/sp09/cse8b/psas/PSA2/torrey_pines.txt
Sample output is here.