A common task in scientific programming is to evaluate the raw data that comes in from a device that is used to measure some physical quantity. It can be anything from a car's speedometer, to a digital thermometer, to an electron microscope. In each case the device will send data to a computer in a very simple format which then needs to be interpreted and processed before anything useful can be done with the data. Oftentimes the device will produce measurements very rapidly, possibly even thousands per second.
In this assignment you will get some practice at performing some very simple raw data processing, by reading sensor data from the internet. This sensor data is collected by devices all over California, by several different agencies affiliated with UCSD. Whenever a new set of data is collected, it is dumped to a particular webpage in real time. You will be reading the data from this webpage and processing it according to a very simple scheme we have devised.
The sensor data webpage is at the following URL:
http://mercali.ucsd.edu/status.txt
It updates every minute or so. Here is an example of what the sensor data looks like:
As you can see, the data is divided into several columns, where each row represents a single measurement from a single device at a single station. The data columns may not appear to be aligned correctly, because they use the tab character ('\t') to separate columns. More on the format of the data is found below.
The starting point of this program will be the main method of the DataProcessor class. Your program is supposed to perform the following actions: (this is repeated in more detail in the description of the main method below)
This homework requires skills with file I/O, ArrayLists, and some string parsing. You will need to write the following classes:
You must put a comment at the top of your code that specifies your:
real name, login name, email address, and PID. You must also include a
two-to-four sentence description of the program. Use a multi-line
comment (/* */). YOU MUST DO THIS FOR EVERY JAVA
FILE!
Comments Comments in the style detailed in the book ARE required. Your code must be properly commented to receive full credit.
Each line of the webpage will contain 10 items that we want to record:
The SensorData class is designed to hold these items. One SensorData instance will hold one line's worth of data, which will have all of these 10 items.
The SensorData class should include the following fields and methods:
The SensorReader class will be responsible for reading the webpage and parsing out a list of SensorData objects. The SensorReader object will be constructed and given a specific URL to read from. Then when the user calls the getDataList method, it will open the webpage for reading, read the lines, parse each line into a SensorData object, add each SensorData object to an ArrayList, and finally return the list of SensorData objects. The SensorReader class should have the following fields and methods:
private BufferedReader openURL() {The BufferedReader that this methods returns will be open and all ready to read from. You will still need to close it when you are done reading from it. Because of the wacky network code that we had to use, you will need to include an additional import statement at the top of your SensorReader file:
BufferedReader result = null;
try {
URL urlObject = new URL(this.url);
URLConnection connection = urlObject.openConnection();
InputStream in = connection.getInputStream();
result = new BufferedReader(new InputStreamReader(in));
} catch (MalformedURLException murle) {
System.out.println("URL string is invalid");
System.exit(0);
} catch (IOException ioe) {
System.out.println("Error opening URL connection");
System.exit(0);
}
return result;
}
import java.net.*;
The data in each line is tab-delimited. This means that there is a tab character ('\t') in between each data item. In fact, some of the lines are missing certain pieces of data, so there will be two tab characters in a row. The only piece of data that might be missing is the location item. If it is missing, its value in the line will be the empty string ("").
The best way to parse the individual items out of the line is to use the split method of the String class. Since the data items are tab-delimited, we will use "\t" as the delimiter we pass to split. This will return an array of strings, that is the result of chopping up the line at every place where there was a tab character. If there are two tab characters in a row (for example, if the location item is missing), then that element of the array will be the empty string (""). For example:
"a\tb\tc\td".split("\t") == {"a", "b", "c", "d"}
"a\t\tb".split("\t") == {"a", "", "b"}
No special action needs to be taken if the location item is missing.
Some of the fields of SensorData are doubles or chars. Two useful questions to ask yourself are:
NOTE: You may assume that the line you are given will always have the 10 tab-separated columns, and that we will not run your code on any webpage that does not have the right format.
Since this method will only read from the BufferedReader and not try to open a file, you WILL NOT need to catch a FileNotFoundException. You WILL still need to catch a general Exception, and if one occurs you should print "Error reading from webpage" and then exit the program.
This class will contain the main method, and will use the SensorReader class to perform some operations on the sensor data from the webpage. This class will read in the sensor data, then filter it in two different ways. It will then compute the average of the cooked values for all the data in the filtered list. Finally, it will print out both filtered lists, and the average that was computed.
The DataProcessor class should have the following methods:
So, for example, if the second input parameter was the string "Temp-out", then you would loop over all the elements of the input list, and for each SensorData object that had a channel field equal to "Temp-out", you would add that object to the new list.
At the end, this method will return the new filtered list.
ArrayList list =
cookedValue=1.0 cookedValue=3.0 cookedValue=4.0
SensorData newdata = new SensorData();
newdata.setCookedValue(2.0);
insertInOrder(list, newdata);
List after call to insertInOrder:
cookedValue=1.0 cookedValue=2.0 cookedValue=3.0 cookedValue=4.0
The easiest way to write this method is to loop over the input list until you find the first element whose cooked value is >= the cooked value of the input SensorData object. Then you insert the object into the list at that location. If you get to the end of the list without finding such an element, you just add the new element onto the end of the list.
On <date>, the value at station <station> was <cooked value>Where the <date> string is the result of calling E.getDate().toString(), and the <station> string is the value of E.getStation(), and the <cooked value> string is the value of E.getCookedValue(). The SensorData objects should be printed in the same order they appear in the list.
USAGE: java DataProcessor (url) (channel)If there are more than 2 command-line parameters given, the extras will be ignored and the program continues like normal.
This method is fairly involved, so we will approach it piece by piece:
After verifying the command-line parameters, the main method will create a new SensorReader object (let's call it reader), giving it the URL string (the first command-line parameter). It will then call reader.getDataList() to get the list of SensorData objects (let's call our list L1).
We will then filter L1 in two different ways; one by calling filter on it and the other by calling filterInOrder on it (let's call the result of the first one L2 and the second one L3). Each of these method calls also needs a String parameter, and the value we should use is the value that we got as the second command-line parameter (the 'channel' string).
You will then compute the average cooked value for the filtered data
by calling computeAverage (let's call this value
average).
(QUESTION: should I call this on
L2 or L3? hmm...)
After all this work is done, you will simply print some output. You will begin by printing the following string:
Average value of <channel> is <average>where the <channel> string is the second command-line parameter, and the <average> string is the value of average.
Next, you will print the line:
Filtered data list:and then call printDataList on the list L2.
Filtered and sorted data list:and then call printDataList on the list L3.
Here is an example of what the final output of main could look like:
> java DataProcessor http://www.cse.ucsd.edu/classes/wi09/cse8b/psas/status.txt Temp-out
Average value of Temp-out is 22.739644733333332
Filtered data list:
On Tue Jan 13 13:27:59 PST 2009, the value at station HERMT was 24.111111
On Tue Jan 13 13:25:59 PST 2009, the value at station NICMT was 24.444445
On Tue Jan 13 13:27:59 PST 2009, the value at station IBShore was 19.222222
On Tue Jan 13 12:13:59 PST 2009, the value at station WLAW1 was 26.722222
On Tue Jan 13 13:13:59 PST 2009, the value at station USCMT was 24.833334
On Mon Jan 12 12:05:59 PST 2009, the value at station DANMT was 29.666667
On Tue Jan 13 04:11:59 PST 2009, the value at station GVAW1 was 20.333333
On Tue Jan 13 13:27:59 PST 2009, the value at station SDSC was 24.444445
On Tue Jan 13 13:27:59 PST 2009, the value at station ZUMMT was 22.944445
On Tue Jan 13 13:25:59 PST 2009, the value at station PTDMT was 22.333333
On Tue Jan 13 13:29:59 PST 2009, the value at station Solar1 was 27.928001
On Tue Jan 13 13:26:59 PST 2009, the value at station CABMT was 25.833334
On Tue Jan 13 13:27:59 PST 2009, the value at station VENMT was 23.555556
On Wed Jan 07 13:16:59 PST 2009, the value at station TOPMT was 13.777778
On Wed Jan 07 07:09:59 PST 2009, the value at station MALMT was 10.944445
Filtered and sorted data list:
On Wed Jan 07 07:09:59 PST 2009, the value at station MALMT was 10.944445
On Wed Jan 07 13:16:59 PST 2009, the value at station TOPMT was 13.777778
On Tue Jan 13 13:27:59 PST 2009, the value at station IBShore was 19.222222
On Tue Jan 13 04:11:59 PST 2009, the value at station GVAW1 was 20.333333
On Tue Jan 13 13:25:59 PST 2009, the value at station PTDMT was 22.333333
On Tue Jan 13 13:27:59 PST 2009, the value at station ZUMMT was 22.944445
On Tue Jan 13 13:27:59 PST 2009, the value at station VENMT was 23.555556
On Tue Jan 13 13:27:59 PST 2009, the value at station HERMT was 24.111111
On Tue Jan 13 13:25:59 PST 2009, the value at station NICMT was 24.444445
On Tue Jan 13 13:27:59 PST 2009, the value at station SDSC was 24.444445
On Tue Jan 13 13:13:59 PST 2009, the value at station USCMT was 24.833334
On Tue Jan 13 13:26:59 PST 2009, the value at station CABMT was 25.833334
On Tue Jan 13 12:13:59 PST 2009, the value at station WLAW1 was 26.722222
On Tue Jan 13 13:29:59 PST 2009, the value at station Solar1 was 27.928001
On Mon Jan 12 12:05:59 PST 2009, the value at station DANMT was 29.666667
Part of the sensor data we will be collecting involves a timestamp, which reflects the time at which the sensor data was collected. But how can you use a single number to record the second, minute, hour, day, month, and year that something happened?
The trick that computer scientists use is to encode a moment in time in terms of how much time has passed since some fixed reference point. So if we fixed our reference point to be May 8, 1981 at 3:15pm (Mike's birth date), then we could say that the turnin date for PSA1 (Jan 15, 2009 at 11:59pm) is 873884640, because that date is 873884640 seconds after our reference point.
So once we fix a reference point, we can express a moment in time exactly by recording the number of seconds that have passed since that reference point. So which reference point should we use? Not everyone recognizes Mike's birth date as the pivotal moment in history that it is, so computer scientists have decided to make the reference point be January 1st, 1970 at 12:00am (i.e. the first instant of the 1970's). This moment has become such a standard reference point that we give it a special name; "the Epoch".
The class java.util.Date is a representation of a
particular date, including the year, month, day, hour, minute, second,
and milliseconds. Internally, it just contains a single long
integer value, which represents the date in terms of milliseconds
since the Epoch, and the Date constructor just takes a
long integer parameter. You will need to use this to
implement the getDate method of SensorData.
(NOTE: The sensor data expresses times in terms of seconds
since the Epoch, and the Date class is constructed by giving
it a time in milliseconds since the Epoch. How do I convert
from seconds to milliseconds?)
Our main reason for using the Date class is that it has a very nice toString method, which prints out the date in a very pretty format. This is used in the printDataList method of DataProcessor.
You can test your program by running it on the live sensor data webpage. The URL for this webpage is http://mercali.ucsd.edu/status.txt. THIS WEBPAGE CONTINUALLY CHANGES and so you might have trouble reproducing your results from one run to the next.
Since the data webpage is constantly updating, it is hard to get consistent results when testing your program. That is why we have made a copy of one particular version of the webpage and posted it here at http://www.cse.ucsd.edu/classes/wi09/cse8b/psas/status.txt so you can test your code on a page that doesn't change.
In addition, you can create your own copies of the webpage at some point in time and save them on your hard drive. Then you can run your program on them by using a file URL. A file URL is a URL that points to your hard drive, rather than the internet. It uses the following format:
file://<full-path-to-file>/filenameSo for example, if I made a copy of the data webpage in my home folder on ieng6 and called it status.txt, then I could run my program on it as follows:
> java DataProcessor file:///home/linux/ieng6/cs8f/cs8fzz/status.txt ...Notice that I have 3 slashes right after the file: part of the URL. This is because there are 2 slashes for the URL format, and 1 more slash from the path since it is an absolute path. You must use an absolute path when making a file URL.
We will once again have a sample solution that you can run on ieng6 to test to see if your code performs the correct way. You can run our sample solution by running:
runsamplePSA2 <url> <channel>at the command line.
We will be using a Bundle script to do automated turnin for this assignment.
On ieng6 somewhere under your home directory you need to make a folder called PSA2. All of your files for this assignment must go in this folder. The files you need to turn in are:
Here are several examples of the expected input/output behavior that your code should have. For each of these examples, your code should print EXACTLY the same output.
> java DataProcessor
USAGE: java DataProcessor (url) (channel)
> java DataProcessor http://mercali.ucsd.edu/status.txt
USAGE: java DataProcessor (url) (channel)
> java DataProcessor http://not-existent-webpage.com Temp-out
Error opening URL connection
> java DataProcessor not-even-valid-url-format Temp-out
URL string is invalid
> java DataProcessor http://mercali.ucsd.edu/status.txt Not-a-real-channel
Average value of Not-a-real-channel is NaN
Filtered data list:
Filtered and sorted data list:
> java DataProcessor http://www.cse.ucsd.edu/classes/wi09/cse8b/psas/status.txt Temp-out
Average value of Temp-out is 22.739644733333332
Filtered data list:
On Tue Jan 13 13:27:59 PST 2009, the value at station HERMT was 24.111111
On Tue Jan 13 13:25:59 PST 2009, the value at station NICMT was 24.444445
On Tue Jan 13 13:27:59 PST 2009, the value at station IBShore was 19.222222
On Tue Jan 13 12:13:59 PST 2009, the value at station WLAW1 was 26.722222
On Tue Jan 13 13:13:59 PST 2009, the value at station USCMT was 24.833334
On Mon Jan 12 12:05:59 PST 2009, the value at station DANMT was 29.666667
On Tue Jan 13 04:11:59 PST 2009, the value at station GVAW1 was 20.333333
On Tue Jan 13 13:27:59 PST 2009, the value at station SDSC was 24.444445
On Tue Jan 13 13:27:59 PST 2009, the value at station ZUMMT was 22.944445
On Tue Jan 13 13:25:59 PST 2009, the value at station PTDMT was 22.333333
On Tue Jan 13 13:29:59 PST 2009, the value at station Solar1 was 27.928001
On Tue Jan 13 13:26:59 PST 2009, the value at station CABMT was 25.833334
On Tue Jan 13 13:27:59 PST 2009, the value at station VENMT was 23.555556
On Wed Jan 07 13:16:59 PST 2009, the value at station TOPMT was 13.777778
On Wed Jan 07 07:09:59 PST 2009, the value at station MALMT was 10.944445
Filtered and sorted data list:
On Wed Jan 07 07:09:59 PST 2009, the value at station MALMT was 10.944445
On Wed Jan 07 13:16:59 PST 2009, the value at station TOPMT was 13.777778
On Tue Jan 13 13:27:59 PST 2009, the value at station IBShore was 19.222222
On Tue Jan 13 04:11:59 PST 2009, the value at station GVAW1 was 20.333333
On Tue Jan 13 13:25:59 PST 2009, the value at station PTDMT was 22.333333
On Tue Jan 13 13:27:59 PST 2009, the value at station ZUMMT was 22.944445
On Tue Jan 13 13:27:59 PST 2009, the value at station VENMT was 23.555556
On Tue Jan 13 13:27:59 PST 2009, the value at station HERMT was 24.111111
On Tue Jan 13 13:25:59 PST 2009, the value at station NICMT was 24.444445
On Tue Jan 13 13:27:59 PST 2009, the value at station SDSC was 24.444445
On Tue Jan 13 13:13:59 PST 2009, the value at station USCMT was 24.833334
On Tue Jan 13 13:26:59 PST 2009, the value at station CABMT was 25.833334
On Tue Jan 13 12:13:59 PST 2009, the value at station WLAW1 was 26.722222
On Tue Jan 13 13:29:59 PST 2009, the value at station Solar1 was 27.928001
On Mon Jan 12 12:05:59 PST 2009, the value at station DANMT was 29.666667
> java DataProcessor http://www.cse.ucsd.edu/classes/wi09/cse8b/psas/status.txt forecast
Average value of forecast is 46.5
Filtered data list:
On Tue Jan 13 13:27:59 PST 2009, the value at station HERMT was 51.0
On Tue Jan 13 13:27:59 PST 2009, the value at station WILMT was 61.0
On Tue Jan 13 13:25:59 PST 2009, the value at station NICMT was 51.0
On Tue Jan 13 13:27:59 PST 2009, the value at station IBShore was 9.0
On Tue Jan 13 13:13:59 PST 2009, the value at station USCMT was 1.0
On Mon Jan 12 12:05:59 PST 2009, the value at station DANMT was 44.0
On Tue Jan 13 13:27:59 PST 2009, the value at station SDSC was 51.0
On Tue Jan 13 13:27:59 PST 2009, the value at station ZUMMT was 51.0
On Tue Jan 13 13:25:59 PST 2009, the value at station PTDMT was 63.0
On Tue Jan 13 13:26:59 PST 2009, the value at station CABMT was 69.0
On Tue Jan 13 13:27:59 PST 2009, the value at station VENMT was 51.0
On Fri Jan 09 13:44:59 PST 2009, the value at station MALMT was 44.0
On Fri Jan 09 13:24:59 PST 2009, the value at station TOPMT was 44.0
On Tue Jan 13 13:26:59 PST 2009, the value at station TPSMT was 61.0
Filtered and sorted data list:
On Tue Jan 13 13:13:59 PST 2009, the value at station USCMT was 1.0
On Tue Jan 13 13:27:59 PST 2009, the value at station IBShore was 9.0
On Mon Jan 12 12:05:59 PST 2009, the value at station DANMT was 44.0
On Fri Jan 09 13:44:59 PST 2009, the value at station MALMT was 44.0
On Fri Jan 09 13:24:59 PST 2009, the value at station TOPMT was 44.0
On Tue Jan 13 13:27:59 PST 2009, the value at station HERMT was 51.0
On Tue Jan 13 13:25:59 PST 2009, the value at station NICMT was 51.0
On Tue Jan 13 13:27:59 PST 2009, the value at station SDSC was 51.0
On Tue Jan 13 13:27:59 PST 2009, the value at station ZUMMT was 51.0
On Tue Jan 13 13:27:59 PST 2009, the value at station VENMT was 51.0
On Tue Jan 13 13:27:59 PST 2009, the value at station WILMT was 61.0
On Tue Jan 13 13:26:59 PST 2009, the value at station TPSMT was 61.0
On Tue Jan 13 13:25:59 PST 2009, the value at station PTDMT was 63.0
On Tue Jan 13 13:26:59 PST 2009, the value at station CABMT was 69.0