Shaoxuan's Engineering Log

PHP script for SSH access

2010-04-21T16:13:00.004-10:00

Today when I was struggling of slowly FTPing a PHP web application (Magento), which the FTP client estimates will take up to 4 hours for that 50+M of files, I start wondering if there is other way to FTP the package to server then unpack it there, not only for dev team, but also for customers. But we have only FTP access to the server, no SSH available, not if I login directly with root certification. After some discuss with colleagues, we can up of an idea of writing PHP code using the exec command to execute the unpack command. This sounds too simple and dangers, and what's worse, it works in our servers. Then I took a little additional effort to wrap up a relatively nicer PHP page to pretend SSH access from browser.

The code is here.

It is easy to see how dangers this could be in shared web hosting. It is even worse when the php is executed as the same user, like 'apache', because that means a client could get full access to others files, and can screw big things up.

Wasted an hour to fix "$(".datepicker").datepicker is not a function"

2010-04-02T09:25:00.002-10:00

I have been using datepicker in the jQuery UI package for quite a while and it was always nice and easy, until this morning when I was trying to add it to a page, it keep giving me the error "$(".datepicker").datepicker is not a function". I check from firbugs that, the jquery-ui-1.7.2.min.js is included, so as the jquery-1.3.2.min.js. I was copying the code from another page I wrote, where the datepicker works just well. I was haunting around that several lines of code, trying to use other version of jQuery and jQuery UI, but found nothing, until I roll down to the page and saw that there is another script reference of jquery-1.3.2.min.js, which I added 2 days ago for other purpose. As soon as I remove that second include, the datapicker come back to life.

The lesson is clear, yet is not frequently mentioned and would be a headless blocker for javascript newbies like me. Include the jQuery core library again will overwrite the functions added by the jQuery UI, thus the functions are missed in the final js object.

It is not a good practice to include these kind of library in the middle of a page. But it may still happen some where by some reasons. If you get weird javascript function not found error, try to look for these duplicated library include.

Use SearchControl and SearchForm of Google Search Ajax API

2009-11-10T20:24:00.003-10:00

When using the Google Search Ajax API, we may sometimes what to add functions when user initiate a search or clear the search result. the SearchForm is designed to do this. It provides .setOnSubmitCallback(object, method) and .setOnClearCallback(object, method) to let developer define method to call when user search or clear search result. But can we use the SearchControl, which provides easy-to-use and nice wrapped searcher, along with the SearchForm? The answer is yes. Here is the example code of using them together.

function OnLoad() {
var searchControl = new google.search.SearchControl();
searchControl.addSearcher(new google.search.WebSearch());
searchControl.addSearcher(new google.search.ImageSearch());

var drawOptions = new google.search.DrawOptions();
drawOptions.setDrawMode(google.search.SearchControl.DRAW_MODE_TABBED);
drawOptions.setSearchFormRoot(document.getElementById
("searchcontrol"));
searchControl.draw(document.getElementById("containerResults"), drawOptions);

var searchForm = new google.search.SearchForm(true, document
.getElementById('searchcontrol'));
searchForm.setOnSubmitCallback(searchControl, function() {
this.execute(searchForm.input.value);
});
searchForm.setOnClearCallback(searchControl, function() {
searchForm.input.value = '';
searchControl.clearAllResults();
});
searchForm.execute(QueryString);
}
google.setOnLoadCallback(OnLoad);

The HTML code is

< id="searchcontrol">
< id="containerResults">

The point to collect this two part together is drawOptions.setSearchFormRoot(document.getElementById("searchcontrol"));, which tells the SearchControl where to look for the associated SearchForm. Another thing to notice is that, in the SearchForm.SetOnCallBack, the execute() method to call should be the one of the SearchForm, like this.execute(searchForm.input.value);, not the one of the SearchControl (searchControl.execute(searchForm.input.value);), otherwise, the method set to OnClearCallBack will not be called when user clear search result.

Things to notice when testing form on Wicket

2009-11-10T20:03:00.005-10:00

Wicket, a Java-based web application framework, provides powerful tester to test the application. The tester can simulate activities of users on the web application. The form tester is a component of the tester, which use to test form behavior on the page. However, there are something one need to notice when using the form tester.

Let's look at this example.
We test a form in the Checkout page, which looks like this

Name
Street
Zip code
City
State

The test code is

FormTester formTester = tester.newFormTester("form");
formTester.setValue("name", "Philip");
formTester.setValue("street", "Main Street");
formTester.setValue("zipcode", "96822");
formTester.setValue("city", "Anchorage");
formTester.setValue("state-wmc:state", "Alaska");
formTester.submit();
tester.assertRenderedPage(Index.class);

But when run the test, the last assertion indicate that it still in the Checkout page. Did you find anything wrong? The problem lies in

formTester.setValue("state-wmc:state", "Alaska");

And the fix is

formTester.select("state-wmc:state", 1);

The reason is that, FormTester.setValue() method cannot be used to set the value of selection field, thus the "state-wmc:state" field is not filled, so when the form is submitted, an error of invalid value occurs. This is an usual mistake when using FormTester.

Additionally, in order to find out what goes wrong when the test fail, we can use the Tester.assertErrorMessages(String[]) to see the error message.

Final Summary of GSoC 2009

2009-08-24T13:03:00.004-10:00

As the fall semester 2009 begins, the GSoC 2009 goes to an end. During it I again successfully complete the project to the excepted goal.

At the beginning of the project, problems of the Issue sensor and its data structure design draw me quite back from the original schedule. However, after I settle it down, the further process become amazingly smooth. It took me like 2~3 weeks to finish Issue DailyProjectData analysis, and another week for Telemetry, and, several minutes to put it into Software ICU! I again experience how a well designed extensible system can boost new developments upon it. Though I am the main developer of the Software ICU, I really have not much to do in previous development of DPD and Telemetry, so it is not my previous experience lead to the boost development.

To add Issue analysis to Software ICU only involves editing a single configuration XML file(I was adding to the default configuration, but it is OK to add to your own configuration, if the developer did not upgrade the code in time =P). The main thing to newcomer may be to find out which file to edit and where it is. I just found that there is no documentation about it on hackystat-ui-wicket. I will add it in soon. Another maybe issue is that, none of the current stream classifier looks to fit the conception of Issue analysis. On the other hand, I do not yet have clear idea of how the Issue stream should be indicate the performance of developers. This can be one of the future study of Issue analysis.

In conclusion, last year, I followed the path of extensible system design to build the system. This year, I experience the benefit of building a system that way, and I am so thankful that we have done it that way. I am pretty sure this will be the way I keep following from now.

Almost done in summer project

2009-08-10T22:58:00.002-10:00

Till today, Telemetry analysis of Issue, and the Issue DailyProjectData page in Project Browser is finished. As the way Telemetry service and Telemetry page in Project Browser implement, the new issue telemetry chart will just show in Project Browser and no modify needed. That means most of expected feature of summer project is done. Only thing not yet finish is put Issue analysis into Software ICU.

What I need to do now most is to put Issue sensor in use in Hackystat's hudson service, so that daily issue data will be collected. When Issue Telemetry analysis available, Software ICU can easily utilize it by modifying the configuration xml file.

We are in good shape here.

Experiencing Telemetry

2009-08-03T21:45:00.003-10:00

Work of the past week is on development of hackystat-analysis-telemetry, to include telemetry analysis of Issue data into Hackystat system. Deal to the well constructed system as example and the sufficient documents, it is again quite straight forward to the goal. However, during coding, I found the telemetry language is kind of weird.

The language defines Telemetry Charts using some redefined language components, such as Reducers, which generate stream point value from DPD analysis, and axis. In the Chart definition, after define the name and parameters, the kernel part will look like this:

chart Issue(member, status) = {
"Issue invocations",
(IssueStream(member, status), yAxisZeroBased("Issue Count"))
};

Then we need to define the Stream called IssueStream, and after parameter description, the kernel part look like this:

streams IssueStream(member, status) = {
"Issue counts for the given mode for this Project",
Issue(member, status)
};

The definition of Streams seems redundant to me because it add no additional information to the reducer. So why not using the reducer directly in the Chart definition? I think it will be cooler that the telemetry language can directly use normal reducer like DevTime to generate the member level chart, so that no need to write in Java the member level reducer again.

Now I am coding the test cases for the Issue Telemetry analysis, need to make up Issue data again.

Finish milestone to DPD

2009-07-27T15:05:00.003-10:00

It is a critical week of the hackystat issue project that the functionalities from issue sensor up to DailyProjectData are ready to release, along with updated documentation found in
Ant Task Reference
SensorDataTypeSpecification
DPD API

The new build of Hackystat including these feature should be release very soon. And it will be deployed in our daily hudson build to start gathering issue data.

Next setp is to implement Telemetry analysis. Then put all these things into Project Browser, including Software ICU of course.

DPD almost done

2009-07-21T18:56:00.002-10:00

Issue analysis for DailyProjectData is almost done. The process is quite straight forward with the existed analyzes as examples. There is nothing interesting to mention in my opinion. It contains a total count of open issue on the dpd data for convenient.

When coding for DPD, I actually found some bugs in issue sensor. When testing the sensordata parser, I realize I can makeup some issue instance as CSV for testing. It should be used in tests for issue sensor as well.

Just keep working.

Moving step forward

2009-07-14T23:26:00.002-10:00

After a week's waiting, my little MacBook Pro had come back home on Sunday safe and sound, did not lose a single piece of useful data. As soon as I got him back, I start finishing up the issue sensor to ready for commit.

After a serious consideration, I decide to discard the RSS feed as data source completely, because firstly it does not provide much more useful information than extract data from issue summary table, secondly the RSS feed service seems more unstable than the issue tracking system,which make the sensor unstable as well, and thirdly it make the code more complex and hard to validate data completeness. My opinion is, if the user really don't want to miss a single update information, just run the sensor as often as he like.

I also found it kind of hard to test the issue sensor, because it is almost impossible to have a fully controllable and repeatable test environment. The content of Google Project Hosting will keep changing. For current state, I test it by run it twice, and make sure the first run generate some issue data, and the second run did not detect any changes. The better way to exam if the sensor working properly is to actually use it. So my job this week is to work on DailyProjectData to use the issue sensordata to generate issue DPD.

Commit Early, Commit Often

2009-07-06T22:00:00.002-10:00

I just get a painful lesson of the Commit Early, Commit Often principle. The thing is, my laptop (MacBook Pro bought at 07) died on Sunday. I brought it to Apple store and lucky I can get it fix for free, which will take 1 to 2 weeks. However, the code of the new version of issue sensor is still lying on its hard drive, and the data is not guarantee to be undamaged. That means my work for 3 weeks are gone, at least for the coming week or two. In fact, functional code is finished already. I was just waiting for completing the unit tests before commit the code. I am now so regret that I did not commit the code early.

Issue Sensor Redo

2009-06-25T20:22:00.002-10:00

Progress of issue sensor is behind expectation. After investigated SensorBase's mechanism, I started to redo the issue sensor. At first, I just want to modify the original code somehow to make it a singleton. However, I kind of mess it up and finally almost redo it from scratch. It is almost finish and just need some unit tests before commit the code.

The sensor is more difficult than I image, because it now actually do some of the analysis job during sensor data collection. In order to preserve as much information as possible, the sensor take not only RSS feed, but also information from issue tracking system directly. The Google Issue Tracking System provides a RESTful link to the issue summary table in CSV format. This gives information include almost as much everything as extracting from individual issue html page. I only keep the field interesting, they are id, type, status, priority, milestone and owner. Here is how the sensor do its job.

Firstly, it retrieve all issue sensor data from sensorbase (It would not be too much because only on piece of data per issue only.), and also get the issue summary table. Then match each issue from the table to a sensordata. If a issue does not have a sensordata for it, it will create a sensordata for it. Secondly, it check the RSS feed, and add update information to the sensor data. The field names, such as id or type, are used as property key, and the property value consists of the field value as well as the timestamp, collected with "--". Finally, new and modified sensor data will be sent to sensorbase.

Cannot put all data search in database

2009-06-15T17:25:00.002-10:00

Issue Sensor Data

The issue sensor data now is decided to use single instance per issue design. The owner of the sensordata, which is the major problem of the approach, is set by the issue sensor, before we can come up something better to decide the data owner. We just want to move on to the core part instead of sitting there thinking about the data owner. That is the easiest way, but require most attention of users to ensure things work good, so sufficient documentation may compensate somehow, hopefully.

New API to SensorBase

When coding the issue sensor as single instance per issue, it is important to get the data of the given issue efficiently. While the issue is identified by its id (number id usually), and that id is stored as a property in the sensordata, it will be nice to extend SensorBase's API to return sensordata which contains a given property with given value. So I start to try to do this. Unfortunely, I found it is not easy to accomplish, and should not be include to SensorBase's functionality.

Firstly I study about the mechanism of its API. I thought the API will look like http://hostname:port/sensorbase/sensordata/user/timestamp/?sdt=sensordatatype&propertyname=propertyvalue. It is actually possible to use dynamic property name, by using "{propertyname}={propertyvalue}" in route definition. Then in the resource, just get the property name and property value as two Strings, just the same way as usual property.

Secondly I try to extend the database interface to allow query sensordata with given property entry, when I found the current tables in database does not support this query. In the sensordata table, the values are (Owner, Tstamp, Sdt, Runtime TIMESTAMP, Tool, Resource, XmlSensorData, XmlSensorDataRef, LastMod), and the property list in stored in the XmlSensorData, which is a XML representation of the sensordata. In order to get the data, I need to query the XmlSensorData. However, in order to get a set of XmlSensorData, I need to first get their XmlSensorDataRef, then use that to query the XmlSensorData(It is not the only way to doing this, but it is the convention of SensorBase, probably to make it easier to separate data instances from query return.). When I get the XML data, I parse the XML and extract the property list, then do the comparison. Then this API query will return the XmlSensorDataRef. The user actually need to retrieve the data with the data reference again from SensorBase. As you can see, there are duplicate query from reference to data instance. That's because the SensorBase is trying to do the things that suppose to be done by data user. The most reasonable way is, the user get the data from the data references, then do the comparison to get the data he want by himself.

Therefore, the issue sensor will just get all the issue data(with the sensordatatype = issue), then compare the issue id to get the one it need to add data in.

Stuck in Issue Sensordata Design

2009-06-08T22:04:00.002-10:00

Issue sensordata is so different from other sensordata that it does not conceptually belong to a particular user. Instead, it belongs to a software project, but this project is not the same concept of project in Hackystat system. In Hackystat, a project is just a definition to group up users and data to represent a actual project, however, there does not necessary exist an associated actual project. The problem is, all Hackystat sensordata belongs to a user. We have to decide a user to process the data, and that user has to be sure to stay in all the projects which the issue may belong to for the whole life time of the projects. This is like a administrator in the project. But this administrator need to be managed by users, not the system. There is not an administrator defined in Hackystat users, and we dont want to make this exception for just one kind of sensordata(it is not necessary for others).

The first design is to store changes of an issue, starts from the creation of the issue, followed by its updates. Then the issue sensordata is assigned to the owner of the update/creation time. The major resource will be the RSS of the issue updates. The good of this is that it keep all the update in the help of RSS, and the owner of the data is reasonable. But the shortage is that the RSS provides limit information. In the creation thread, it only include the comment. In the update threads, it only include the state/labels that being changed. The current unchanged state is unknown from RSS and has to be found out from the issue tracking system(via http in most case). Another problem is when analyse the data, all data started from the project start time have to be gather together to get the view of the given time. It might be a lot computation if there is lots of updates.

The second design is to make a single sensordata associated to a single issue. The updates will be store in the properties list of that sensordata, from the same data resource: RSS. In the creation of an issue data, the current state/labels will be extract from issue tracking system, then it is easier to keep track of future changes. Also, it is easier to analyze, only need to go through that single data instance to figure out the state of a given time. However, the problem of this design is the owner of the sensordata, because it has to know the owner to get the data, and in project level analysis, that data owner has to be in the project which the issue should belong to. There is no a reasonable way to answer this question without making some hack or modifying/adding current system definition. It is possible to let user define who the data belongs to, but it is unsave because it require not only the user know excatly what he is doing, but also all sensors collecting data for the same project need to be configure excatly the same. Otherwise, there may exist mutilple copies of the data instance, which is a great fault of the single data assumption.

Summer Plan

2009-05-21T21:00:00.002-10:00

Google Summer of Code
During the summer, I will be doing the Google Summer of Code 2009. My project is to add Issues into Software ICU. The work include from collecting data from issue tracking system to produce end analysis to Software ICU. The data collection part is finished during spring. So the plan for this project is:

Review the issue data sensor and install it to Hackystat projects
Write DPD analysis for the issue analysis
Add Telemetry streams about issues
Add Issues to Software ICU
Revise the system from head to toe

I expect two weeks for the last three tasks. That is 10 weeks. Google Summer of Code last for 14 weeks, so I will have 4 weeks to spare to wherever needed or to catch up in progress if any delay. For the midterm goal, The first two tasks should be finished and the third one should on the half of the way.

Master Thesis
The thesis is another important task for me this summer, probably the most important one. There are six chapters in my thesis. I might finish each one in one to two weeks. So it will take probably two month to have it done. During this, Philip will revise each chapter for me after I finish it. And I will go over it on the fly. If everything goes as planned, my thesis will be finish before August.

The first chapter I will work on is the related work. I am looking for more relative research on empirical engineering to fill up this chapter.

Summary of Spring 09 Semester

2009-05-11T23:11:00.003-10:00

Brief Summary
The most of work relative to my thesis research or Hackystat was done in the first half of this semester. As the semester went on, I began to stuck in the two course I took, ICS606 and ICS621. The homework and projects began to accumulate and took me lots of time to accomplish. Thus, my total research and development output is somehow lower than previous semester, and the progress of my thesis is behind my expectation.

Achievement in Spring 09

Tech-report of Hackystat Classroom Evaluation in Fall 2008
In fall 2008, we deploy the Hackystat system in class projects of ICS413, together with a questionnaire survey near the end of the semester. Additionally, we gather log data of students' usage of the system. At the beginning of this semester, I started to review the result from the survey and analyze the log data, then wrote a tech-report of this evaluation. While the major component of this evaluation is the Software ICU, most of this tech-report will be able to go into somewhere of my thesis.

Seminar Presentation of Software ICU
This is one of the most important step in the progress to my thesis. It is a good chance to summarize the system. The slide can be found here. I spent a little more than a week to prepare the presentation. That was quite challenge to me because it was my first time to present to dozens of audiences. But the presentation came out to be a successful one. It really encourage me a lot.

Hackystat Manual Sensor
In Spring 09, my major development contribution to Hackystat is the manual sensor, relative posts can be find here, and here. It is a Java Swing application that let user manually input data to reflect their development activities which are not yet have Hackystat sensor attached. The current version is just as simple as a plain form plus a raw data viewer.

About My Thesis
In the last week, I have setup my plan to graduate in fall 2009. That means, the due day of my thesis will be sometime in the middle of October 2009. But I prefer to finish the stuff earlier, just in case accident happens. My plan is to finish my thesis before Auguest, and then defensive it in early Sepetember.

Here is the draft of my thesis, which would like to be considered as the technical report for this semester's independent study.

Report on Agent Development Stimulation Platform

2009-04-30T14:39:00.002-10:00

In the Assignment 3 of ICS606, I was supposed to install the Robocup Rescue system and do some experimental agent development upon it. However, even compile the system came out to be nothing trivial. The system is written in both C/C++ and Java. The latest version is back to Mar 2007. They use both GCC and JDK to compile the system, but since their latest release, both these two compilers have updated quite a bit. As a result, the code cannot be compiled using the latest version of the compilers. Moreover, as although the system is claimed to be platform-independent, its makefile do use some parameter that is not available to Macintosh (like -soname), and the source code contain some function that has no Macintosh version(like pthread_yield()). The only guaranteed is Unix, in the state when their latest version released.

I have tried to compile it in Mac OS X 10.5, Windows XP SP3 and Ubuntu 9.04, with the latest compilers. None of them has any luck to pass the compiling, and the error given in these three platform are different.

In order to successful compile the system, the user have three choices. The first way, also the hard way, is to fix all the compile failures to match the compiler of the user's enviornment. The second way, which is theorictical, is to configure the compile system to compatible to the state of the release day. The third, which is somehow tricky but most easy way, is to install a stand-alone Linux using the old build. The easier way is to install that in a virtual machine. It is proofed to work to install the Ubuntu 6.10(Edgy Eft).

On contrary, the Robocode, which I got to know from a classmate's presentation, is much user-friendly. The procedure from downloading to start using it is quite straight forward. The installation package execute correctly and I can start writing my own agent in 10 minutes!

In conculsion, the Robocup platform is ill designed and extremely user-unfriendly, while Robocode is a much better platform for agent stimulation and easy to use.

Boswell continue?

2009-04-13T23:31:00.002-10:00

Last week I started working on the course project of the agent class. The project I pick is based on Hackystat, and is a continue study of an existed topic: boswell/tickertape, a auto blogging agent for software development. While working on it, I am kind of lost my way. There is no directly related research about that. But there are many likely related fields, such as language processing, knowledge base, etc. The search field is so large that I am lost inside it. All potential related fields are so big to understand in short time. However, without understanding, it is hard to tell if concepts and technics from it will be useful or not. Currently, I just put my hope in language processing. There is some study of it and also using knowledge base concepts. I hope I can get a breakpoint from there.

This research is not directly related to my thesis. But it is an interesting research of Hackystat. Hopefully I can end up with some useful insight after finish the course project, if I can indeed finish it well....

Stuck in course work

2009-04-06T22:58:00.002-10:00

Another week done nothing about Hackystat nor thesis... But it was a busy week because I was stuck in my course work during the whole week.

In the first half of the week, I was preparing the midterm of ICS606 Autonomous Agent, which is on Wednesday. After that, I switch to the assignment of ICS621 Analysis of Algorithm. The assignment is to design an algorithm to solve the power grid load shedding problem. It is an NP-complete problem, and it is about directed weighted graph. It took me the whole week thinking of a better algorithm. Then at the end it came out to be worst than the brainless enumeration. That is quite frustrating.

This this week, I will have to start the almost-forgot course project of Autonomous Agent. The project I choose is to further develop the Tickertape in Hackystat project, to grant it more intelligence to act as human beings. The first step will be constructing "knowledge" about user's developing behaviours from hackystat sensor data. The hard part is how to design the structure of that knowledge in a way that it not only has good representation power, but also facilitate to generate human language sentence. I am not sure where to start, design the knowledge first then the way to express it, or the opposit way.

I hope I will be able to finish the course work faster and left some time for my other jobs.

Taking a break and gsoc 2009

2009-03-30T19:52:00.004-10:00

As the name tells, I took a break during spring break. =P

Ideas of Google Summer of Code

In the last meeting with Philip, he provides me an idea of utilizing Issues data into Hackystat, from the bottom of collecting and sending data to the top in Software ICU analysis. In the middle there are an associated DPD analysis and a set of Telemetry analyses. The Issues data is an essential sign of health of project management, but long missed in Hackystat version 8. Additionally, this is a good chance to work all through the system, which I did not experience yet. This proposal contains enough work for 3 months, so I used this idea in my application to GSoC 2009.

Before that, I have another idea: provide a facilitating way to deploy service of Hackystat. The first part is easy way to deploy sensorbase upon various database implementation. Currently, users have to implement the interface in Java code by themselves and complie the system. I want to find a way to separate the database access part into a standalone componenet that user did not have to complie sensorbase everytime there is update of sensorbase. The database access component requires much less update in functionality than sensorbase, thus less, if any expect bug fix, update/recomplie required. I may also do the implementation for some popluar database such as MySQL, IBM DB2, Oracle and MS SQL Server. The second part is an administration tool to lauch Hackystat service such as sensorbase, dpd, telemetry and projectbrowser. It provides GUI to configure settings, and capability to hide the command-line windows, make Hackystat runs in backgroud like most system services do. I did not familiar with either database implementation nor GUI configuration building. Thus estismated work is unknown for me. I expect more time in research that coding to accomplish this.

Revise my thesis

2009-03-23T22:44:00.003-10:00

State of manual report tool

The manual report tool is finished for basic but complete functionality. The UI is similar with the figures post before, except that the "Labels" field is removed and labels is expected to input in "Resource" field. This is actually the way to construct the resource field in sensor data. Making it this way makes user less confused when they see their "labels" shown in the resource field in history panel.

As discussed before, history panel manipulate sensor data piece by piece, no grouping function is provided.

Revise my master thesis

I start revising my thesis from the "portfolio" version to a "software ICU" version. The introduction is being totally rewritten to introduce the idea from different angle. I am reviewing recent tech-reports about Hackystat and software ICU, which including 09-02, 09-03 and 09-07 to get ideas of the start point.

Most of other parts, such as the related work section, can remain the same.

Self-report tool is ready

2009-03-16T21:43:00.002-10:00

After a little talk to Philip, I realize that showing the raw sensor data may even better than grouping them into events, because in that way we can give user more control over their data. The initial motivation of add that "event management" feature is in case of editing data. However, for the near goal, we will not provide the editing feature in the manage panel. If user think there is error in their input, just delete the data and resend it again. When managing the raw data, there is a possible trick to shorten the time by delete some of the data. The other reason I wanted that grouping feature is trying to make sure users will correctly delete their data, not leave unexpected data in sensorbase. However, in either way, users have to take responsibility to their data. The additional feature will not ensure they will make less mistake than without it. In contract, if the additional is less robust than it appear, users might pay less attention to their data's completeness than they will do with raw data, which may lead to more defect data. Therefore, after this second thought, I decide to leave the responsibility of the data's completeness back to the owners before a powerful and reliable enough manager exists.

Plan for this week is to finish the self-report tool. Everything is ready actually, just need more test of sending out data. After that, thesis!

More analysis on log data

2009-03-09T18:06:00.004-10:00

Accomplish of last week
1. Tech-report of Hackystat evaluation 2008 is revised. More analyses of data on per student bias are added in order to gain further insight into the logged and survey data. It is interesting to find that some students' responses are far from their reality: the use frequencies they claimed are much lower than the actual frequencies logged by the system. Though it is hard to verify if the error is intended or not, it does reveal some interesting point from their answers to other question: those students are the ones who concerned with sharing their data, especially DevTime and Commit. It is reasonable to infer that they were not happy with the data showing their laziness to their teammates.

2. All functionality for initial release of the self-report tool is complete. Currently the application takes a time period denoted by start and end time, resource file, tools and label to generate the DevTime data. The UI is shown below.
Also the application is capable to manage the data you sent. It is able to retrieve data in a given data, filter the data to a given tool if available, show it on a table, and allow user to delete data from it. The following image shows an example of this feature. The image is captured on the experimental implementation that the data is not filter to be self-reported only, which will be implemented in actual release.
The current state of the manager is actually not quite useful, or even annoying. This is because the approach of devtime sensor data: a piece of data make a five mintes period as active devtime. So for every event that user reported, there is not a single data relative to it, instead, there are a set of data, each departed by five mintes, to present the length of the action period. When editing, these pieces of data are shown and treated as separated instance, and are deletable separately. This does kind of distrupting the concept behind the sensor data. In order to avoid this, the data should grouped by events and managed as events. This will be the next priority of this project.

Plan for this week

I have a paper presentation on Wednesday morning, that will probably take all my time till then. But after that, I can have a small break from my course work and put more time to the lab projects and my thesis. So here is the plan for this week:

Continue the self-report tool. It is very close to it designed funtionality now.
Revise my thesis.

Presentation Finished!

2009-03-02T17:25:00.003-10:00

The biggest this for me last week is the presentation of my thesis project on seminar course. My slides can be found here. It was the biggest presentation I ever made and make me nervous to dead because I did not get much time to practice the talk. Fortunately it came out to be a successful one. I was planning to revise my thesis before start preparing the presentation but I found I was running out of time. So the thesis is still untouched yet. As discuss with Philip and Robert, my presentation did not include enough information about the evaluation result. I was thinking adding more to the slide but the result is too literal and long that it is hard to summarize good with a few slides. And insufficient analyze of the logging data is another reason of this difficulty. From Philip's inspiration, I started making more analysis charts over the data and they do reveal some additional information. They will soon be added to the evaluation tech-report.

Plan for this week:

Finish further analysis of the data and add the new result to the evaluation tech-report.
Finish the first release of self-report sensor. The remain thing for initial release is the ability to delete self-reported data. This is most involved with displaying an inner data object with JTable.
Revise my thesis. It is long postponed and should be caught up sooner. I still want to finish the thesis before submission deadline in order to keep the initiative of my graduation.

Common Lisp to my thesis

2009-02-23T18:03:00.002-10:00

During the last week, little work is been put into neither the self-report tool nor my thesis, because most of the time is put on the assignment of ICS606 Autonomous Agent. This assignment took me a whole week because it is based on Lisp. It requires to enhance a vacuum agent which is written in Lisp. Though I had taken the undergraduate class about Lisp, I were near to novice because firstly I did not experience writing functions, methods and structures in Lisp in that class, and secondly, I had nearly forgotten everything I learn in that class. It took me about two day to pick up Lisp and advance to object-oriented like feature. Meanwhile, I found the book Practical Common Lisp is pretty awesome, and it is available on web! It actually explain Lisp much better than the undergraduate class. After being familiar with Lisp, I did find Lisp is pretty good in program the agent because represent and interpret data is unbelievable facilitated. It gives more freedom than other language ever possible give, though it is sometime same as freedom to make mistake. =P

In this week, the priority is to revise my thesis. Actually the biggest task in this week is the seminar presentation on Thursday. Thus revising my thesis is one of the best way to coordinate ideas in the presentation. After not so many presentation in class, I feel my biggest problem is not practicing the start and detail besides slides. What I did in my previous experience is mostly like to outline the content to slides, and only review the detail over my head. When presenting, because of nervous, I often did not start the talk very good, that cause me even more nervous usually, and forgot what I had prepare in mind, then doing even worse. So I got stuck into a vicious circle and end up being reading the words in the slide. This time, I will trying to rehearsal some times before, in order to get prepared in both content and confidence.

Unit testing for Swing

2009-02-16T19:55:00.002-10:00

Effort put in my thesis last week is under my estimate because the assignment of ICS606 took more time than I expected. I will put more work in it this week.

I have started the self-report tool of Hackystat using MiG Layout. I am yet paying much attention in the UI, so it is too early to said if the MiG Layout is perfectly fit my requirement or not. But so far, it is handy to manage simple arrangement of components. And its Quick Start Guide and Cheat Sheet are very helpful and intuitive.

When starting the new project, I also searching for unit testing package for swing. After some digging, I settled to a package called FEST. Its assertions are quite flexible and readable. And it is an active project that running for a long time. After some trying, I finish my first test case with no difficult barrier. Then I notice that, what the tester do is to stimulate mouse/keyboard action on a real Swing application. I can see what it is doing during the test. That is both good and bad. The good thing is I can see what's going on, and what's going wrong in the test. And the bad thing is, I have to stop to watch it. If I move the mouse during the stimulation, the test will probably fail. I will no longer be able to run the unit test and do something else while waiting it finish. However, I think of an interesting idea from it: it may be possible to make "demo" test cases, that can serve as an example of usage in supplement of wiki guide.

Slow progress

2009-02-09T18:37:00.002-10:00

I am working on the self-report tool for hackystat(hackystat-sensor-manual). We decided to use MiG Layout for layout management. The project was just started, but progress is slow, because I am picking up with the Swing staff in Java, which I did not touch for quite a long time and never be very good at it.

The other thing on my priority list is my thesis. As the whole experiment become the classroom evaluation, my thesis topic therefore change from use software ICU to manage project and courage collaboration to use the software ICU to understand and teach software metrics. The principle is the same, just change the way to view and present it. The news is, the major part of my thesis is finished in tech-report of Hackystat classroom evaluation fall 2008 already.

Another waiting project is the SVN data collection service, respond to Google's new commit notification POST and generate sensor data. It will be a great service, but not in an urgent need because SVN sensor is now working well. But implement this new service will reduce the difficulty of setup svn sensor. And for users that don't use an automatic contiunous intergration engine like Hudson, they don't need to remember and run svn sensor very time after commit.

Plan for this week will be, build an intial framework of self-report tool, and merge the evaluation tech-report into my thesis.

More reports and papers

2009-02-02T21:51:00.003-10:00

This is a busy season of writing reports and papers. I am finishing the tech report of Hackystat fall 2008 evaluation and will then back to my thesis. Philip is just finish the draft paper of Hackystat SOA and I, as co-author, am revising it.

The first revision of Hackystat fall 2008 evaluation is finished. It is even more interesting with the usage logging data. When preparing the analysis data of system usage logging, I found that Pivot table of Excel is a very powerful tool and quite easy to use. Once the data is correctly imported(key is to set correct seqarator), Pivot table will do all the statistic analysis and generate a very nice summary table.

Last week I was provided a project to create a tool for users to self-report their un-tracked developing events. It will be a client-side application based in Java. But I did not spend much time on research the JGoodies and MigLayout, the visual layout packages for Java application. But once I get familiar with the layout thing, the primal functionality of the self-report tool should be a piece of cake. For further enhancement such as keyword remember/auto-completion, application/tool auto-selection, and event auto-generation, there will be a lot more study.

On the way to the Evaluation Report

2009-01-26T23:02:00.005-10:00

The first draft of Hackystat Classroom evaluation in fall 2008 is finished. Philip had proof-read once and give me feedback. I am now editing according to the feedback and finishing the future direction part.

This week I will start doing the analysis of the logging data collected during the evaluation period last semester. It will be interesting to compare it with the answers given by the students. This analysis and result of comparison will be included in the tech-report.

Meanwhile, there are still lots of learning for Latex to me. I am still quite new to Latex and not so familiar with it yet. One of my class's assignments are required to be written in Latex as well. There really a good time for me to practice it.

Currently, there is no planning for programming work. Project browser seems in quite a good shape now. Maybe after finishing the future direction part, I will do more improvement accordingly.

Plan for this semester

2009-01-19T17:36:00.002-10:00

Now it is the beginning of the second week of this semester. In this semester I continue to work in Philip's lab and write my thesis for Master plan A. Beside the ICS700 and ICS690(seminar), I took another two graduate courses: ICS621 Analysis of Algorithms & ICS606 Intelligence Autonomous Agents. Both of these course seem to be a busy class. The former one requires lots of paper writing, which may be good for me while I am writing my thesis. The later one will include quite amount of LISP programming, which I am not so familiar with. That seems to be a very busy semester.

Plan for RA, research and thesis

The first priority will be finishing the tech-report of the classroom evaluation of Hackystat in Fall 2008. Over a half of it is done. It should be finished in a few days.

The second priority, which will soon become the first, is writing the thesis about Software ICU. There are several tasks in it upon finishing the tech-report:

Process the usage data collected in the last semester;
Literature review on related work (software metrics, single and multiple software project management, software project portfolio, etc);
Review the former study of Hackystat such as Telemetry.

Other things related to CSDL research:

Philip has begun a new project called Devcathlon, which is a game about software development. It will be an interesting project to work on;
I also want to make some improvement to Hackystat system, it includes:
1. Enhance performance, java-based modularity system like Equinox and OSGi.
2. Enhance the loading process panel. Current implementation is to verbose. I want to make it more concise and/or use other approach such as loading bar.
3. Make improvement to let portfolio page better suit to the 9-LCD workspace. One idea will be make the input panel hiddable and introduce auto-refresh feature, but it will better with the performance enhancement.

Portfolio is in ICU now

2008-12-08T22:22:00.003-10:00

Actually, Portfolio is not in ICU at all. On contrary, it is in its best shape ever. It is just a kidding and the thing I want to say is that I decided to name my research on Portfolio as Software ICU, which is used in the ICS 413 class that I do my research on.

I have spent most of the past week on the issue sensor. It is mostly done now, as I believe. But I did not test it yet, neither manually nor via JUnit. I will test it manually in the coming days and make an intial release. As there is no way to run a Google Project Hosting locally, the only way to make an unit test is by using the public Google Project Hosting. Though we have less control of data than running a local server, the data in a certain period of time seems consist while there is no obvious way to change history data in Google Project Hosting (Unless some one hack it within Google, which is extremely impossible).

Now the sensor collect the following data:

Author - The author of the issue will become the owner of the sensor data.
Issue Number - The issue number in Google Project Hosting.
Update Number - The number of update of the issue. If the issue is just created, it will be 0.
Status - Actually this may better to call new status because it record the new status in the change, if the status is unchanged, this Status will be empty.
Issue link - The link to the page of this issue on Google Project Hosting website.
Comment - The comment user make, if any.

One major problem is that, from the issue updates Atom feed provided by Google, there is no way to extract the current status of the issue such as status, priority, types if they are not changed in the issue change event. There are two way to extract these data. The first is to retrieve the issue's page. The link of the page is collected in the sensor, so there is no trouble to locate the page, however, the page always represent the newest state of the issue, which may not equal to the state the sensor data represents. The other way is to get data from Google posting sent by Google Project Hosting. The posting will keep track of all change along with the initial state. It is ideal for make sensor data, but it require more work to setup the google group.

In a second thought, the Google Group posting approach seems somehow better. If switch to this way, there is no need to use the issue updates Atom feed any more, and all data can be extract from the postings. I will see into it later.

Portfolio enahcement and thesis draft

2008-12-02T00:15:00.003-10:00

Sortable table columns have been implemented. I implement this by making measure headers links that sort the internal data model according to the specific measure. It is accomplish somehow hacky because the internal data model store a list of MiniBarChart under each project and there is no name in the MiniBarChart instances. These MiniBarChart instances are generate from Telemetry stream data and the orders of the Telemtry analysis are identical in every project. Therefore, the sorting unit is not defined by name of the measure, though user think it is, but by the index of it in enabled measure.

After the intensive implementation of features of portfolio page, I took a small break around Thanksgiving. Then I work on the thesis. The draft of my thesis can be found here.

In the coming week, my major coding effort will be put on issue senor. The first experimental version should be available by the end of this week.

Big improves to Portfolio

2008-11-24T15:57:00.004-10:00

Big improves to Portfolio
Last week I have implemented a big improvement to Portfolio Page. The feature is to color the previous uncolored measures by the member participation. The idea is from here: http://code.google.com/p/hackystat-ui-wicket/issues/detail?id=118

Most of the previous uncolored measures are implemented using the member-level telemetry except FileMetric. Therefore, it is possible and useful to color the chart according to the participation of the members to the project according to these measures, because a health project anticipates equal contribution from each member.

When fixing Issue 97, I added a interface called StreamTrendClassifier to let user select different coloring method for a measure. I thought this would faciliate future development and it does in some extent. When implementing participation coloring, I am using the Classifier interface to specify new classifier to the measure. But the problem I did not realize in the beginning is that, configuration to the new classifier, which requires text fields of member percentage, threshold value and frequency percentage, is quite different from the stream trend one, which requires higher/lower threshold and higherbetter check box. So I modify the interface greatly to let each classifier provide its own configuration panel. And this idea may be apply to the DPD page, where each data model provides its own data panel instead of letting the session to create the panel for it.

After complete the functionality, I change the interface of the configuration panel greatly to make it looks better. The old format that each measure occupys a row of table and each field use a column works fine with uniform configuration fields, but not with various field between measures. For example the first three measures have higher/lower threshold and higherbetter, and the latter three have member percentage, threshold value and frequency percentage, then the higher threshold will aligned in the same column as member percentage, but they just have nothing in common. This may cost users' confusion in both concept and vision. After some tryings, I finally arrange fields of a stream classifier veritically inside a table cell. A screen shot of the new interface is like below:

This is much clear that different configuration fields are following the classifier selection, and do not implicit any common over different measures, or say classifiers. As a result of this change, the configuration panel now lays between the input panel and detail panel because it is too high and narrow to put it above the detail panel now.

Plan for this week
Two priority jobs for this week is issue 113, which provide sortable column in portfolio table. And the issue track for ant sensor.

Portfolio and Issue sensor

2008-11-17T22:32:00.002-10:00

Effort estimation of sortable portfolio columns

It is very useful to let the portfolio table sortable by value of the measures. There is two way to do it:

Make the table headers links that will sort the table according to the value of the measure.
Re-implement the portfolio detail table using DataTable, which support sorting.

Major effort of the first approach is to code the comparator, and the second's is to deploy the DataTable. The work to deploy DataTable is surely more because we have to implement the comparator after all. The benefit for that is, the DataTable looks more elegant. And we have more control over the content of each cell because each column can has its own class and html to define its display. However, as the current portfolio detail table looks good enough, I would prefer to go the first approach to add the feature with less work.

Issue Sensor

I start the sensor from the SVN sensor because it is using data from internet(svn repository) and therefore more similar to issue sensor than the other ant sensors which use data from local tools such as JUnit and EMMA. The issue sensor will grab data from feeds of Google Project Hosting, such as Feeds of hackystat-ui-wicket. I found a java api for RSS and ATOM named ROME to parse the feeds. The information of issue sensor data will include open/close an issue, change state of an issue and/or new comment to an issue. The initial implementation should be finished in this week.

Review of two papers

2008-11-12T21:48:00.002-10:00

[1]A risk based economical approach for evaluating software project portfolios

[2]Portfolio management of software development projects using COCOMO II

The software project portfolios mentioned in these two papers are quite different from the software portfolio I am working and doing research on. The software portfolio they talked about are the portfolio used to evaluation investments on software projects to maximum the profits and help manager to optimize resource distribution, while our software portfolio is used to monitor the health state of software projects and help users to improve their development practices. However, there is an opportunity that these two kind of portfolio can cooperate together.

When estimate a project, [1] use a term called Risk Level and [2] introduce a set of factors to represent the similar attribute. In our portfolio, the chance a project may fail is indicated by the health state of the project. It is more accurate and objective than the risk value given by the manager. If we combine these two kind of portfolio and use our portfolio to evaluate projects' risk level, the estimation of final profits should be more reasonable and accurate.

Every thing seems going well

2008-11-10T19:55:00.002-10:00

I am glad that students in ICS 413 are starting to use Hackystat now. I read some of their blog postings about it. They seems like the project portfolio quite a bit. Some of them did complain about the installation difficulties, but I complained it too when I am new to Hackystat, because we are all spoiled by the "click next to finish" install package.

Portfolio drill down to DPD

I was trying to add feature to portfolio that allow users to click on a value from detail table and retrieve a corresponding DPD analysis page. It will be an easy job if I can add REST API support for DPD page. But after some dig into codes of DPD page, I found it is not easy to accomplish because from there is no way to know what content sensitive menus are used in a specified DPD analysis programatically. So both when constructing the URL for a certain analysis, and parsing a URL to retrieve a page, we have to write hard-wired code to teach the system what argument to use for a certain analysis and how to parse/construct it. This will introduce a high risk of bugs for modifying DPD page. In my opinion,we should modify the logical structure of DPD page to better suit this and potential need of changes, if we have time to do so. At this momment, may be the best and fast way is to provide a incomplete functional REST API for DPD, which just include day, projects and analysis, not including the arguments. We can see how it feels before planning further steps.

Telemetry

Now users can specify the size of the chart in telemetry page. They can enter any size they want, the only restriction is the total size cannot be larger than 300,000 in pixel, which is a limitation in google chart.

Thesis

I am reading two papers and will write some review of them tommorrow probably.

To-date and portfolio

2008-11-03T22:32:00.002-10:00

Portfolio and ToDate

As function of portfolio page comes to a relative complete level, we start the To-Date page. I started it by copying the code from portfolio page and modify it a little to make it satisfy its goal. All difference between to-date and portfolio is the end date is fixed to today, and cumulative parameters are fixed to true. When test to date, I open the relative portfolio page to compare the result. Then I suddenly realize that, all functionality of to-date can be produced from portfolio, and the code is just a redundancy. Then why we bother to create another page for to-date analysis?

So we are thinking about remove to date page and direct potential users to use portfolio to retrieve "to date" analysis. We can add a button in portfolio to provide an easy way to get to-date analysis. Or even just make some documentation to tell user how to use portfolio to achieve their purposes. Either way, we will not need to continue the to-date page anymore at this moment.

Thesis

The need for coding for project browser is now reduced a lot. So it is a good time to seriously start again the long postponed thesis research work. The major task currently is to do more literature review. Also the students evaluation is about to start, so we should just sit tied and wait for the users' feedback before do more crazy coding to our system.

Portfolio and To date analysis

2008-10-27T22:19:00.002-10:00

Member-level telemetry in portfolio page

Now when user click a sparkline in portfolio page, it will go to the associated member level telemetry analysis, which provides more detail about that sparkline. Actually, the analysis used to generate the sparkline is the member level telemetry. These analysis will return multiple streams, then the sparkline is generated by merging these streams. New attribute "merge" is added to portfolio definitions. Currently the system support "sum", "avg", "min" and "max" merge methods, but only "sum" is actually being used.

At the beginning of trying to implement this feature for portfolio page, I was trying to associate a telemetry analysis to the portfolio measure, and when user click the sparkline, it will invoke the associated telemetry analysis rather than the analysis used to generate the sparkline. But after some coding I found that, the associated telemetry is most likely to have different parameters from the original analysis. That means either the associated analysis has to use default parameters, or we need to add parameter configuration for the associated telemetry. Neither of these choices satisfy us, so we discard this idea and use merging.

To Date Analysis

Portfolio page now is mostly done at current level, so it is good time to start to date analysis page now. Surprisingly, to-date page is pretty much the same as portfolio:

It has few input: projects only.
Each project we generate some analysis and list them in a table.
It has a configuration panel to configure these analysis.

So after copying code from portfolio, I just need to do some small change and it will work.

There are two major issue we need to address:

Which analysis should use cumulative value and which should use latest value.
When should the accumulation start.

The first one is easy because it can be tell from the cumulative parameter in telemetry definition. If a telemetry has cumulative parameter, that means accumulation make sense to it, therefore it should use cumulative value, and the others should use latest value.

The second one is more difficult to answer. A quick idea is to start from the start date of the project, but that will cause the serious performance issue, especially when requesting the Default project, which start date is set to 2000-01-01 by default. The other problem is, even if start from start date of the project is doable, is it make sense to do so? May be it will be better to let user select the start date? This is the first research question for to-date analysis.

Progress in project browser

2008-10-20T15:21:00.004-10:00

Ant task improvement:

After tolerate long verify time for quite a long, I finally start to fix this. When running verify.build.xml, Unit test and EMMA are both invoked. The unit test is to ensure all unit tests are passed, and the EMMA is to get a coverage of the current system. But both of them need to run junit to execute all unit tests in the system. Exactly same tests are executed twice. It is surely a waste of time, especially when these tests takes several minutes to complete and I have to wait until it finish then I can commit the code. So the idea it to combine these two task together. In the EMMA task, it requires to run the junit and track it to get coverage data. But in our old version, the EMMA task just get the coverage data and discard the test result. So I added code to let it monitor the test result as well and generate junit reports in build directory, which can be used to generate junit results and sensor data. Then I add a task in emma for verify, which invoke emma.tool, junit.report, junit.sensor, emma.report, emma.sensor, emma.echo to complete all task in Unit test and EMMA. Now, the verify.build.xml runs almost twice as fast as before.

Telemetry Improvement:

The color issue is fixed now. The random colors are selected in a way that colors have maximum difference from each other. This is done under the color wheel conception. If we need to pick n random color. The first color will be selected randomly, the other colors will have 360/n degree away from the preceding one, so that they will be distributed on the color wheel with maximum distance.

Moreover, the stream color mechanism is improve to take more use of coloring. The new rules are:

If there are more than 1 unit axis, streams will be colored according to the axes.
If there is only one axis, but have multiple stream names, streams will be colored according to the stream name.
If there is only one stream name as well. All stream will be colored separately.

So when users viewing member level telemetry or one telemetry over several projects, they will see a chart will stream of the same color. There will almost always be streams will different colors.

Portfolio Improvement:

Trend interpretation has been improved. Before, only the monotonous trends will be considered as stable, increasing or decreasing. All others will be considered as unstable. This is not the way people interpret trends. People accept a certain amount of vibrate in trends and still consider stable, increasing or decreasing. So I change the system this way. If a point has difference from the preceding point with certain amount, which is calculated as 5% of the average of the stream, will be considered same as its preceding. Otherwise if the point is higher, then it is increasing, if lower, then it is decreasing. Then the system go through the stream. If there is no increasing or decreasing points, the stream is stable. If it has increasing but decreasing points, it is increasing. If it has decreasing but increasing points, it is decreasing. If it has both increasing and decreasing points, it is unstable. In this way the system can accept trends with small vibrations and still classify them as how people usually do. Also the acceptable difference is discussable.

What's more, I added a classifier class to do this job. That means user can write their own classifier to interpret the stream trends, may be using some sophisticated mathematics methods as well.

Thesis outline

2008-10-13T22:21:00.005-10:00

The final tech report of 699 will be the draft of my thesis. The semester is half way through, it is time to think about it again to put something new to it.

Development on project portfolio page goes well. But my literature review is far behind schedule. I need to catch up with it now.

Here is the outline of my thesis:

1. Introduction

2. Related work

2.1 Software Product and Process Metrics

2.1.1 Product Metrics (Coverage, Complexity)
2.1.2 Process Metrics (LOC, Churn, DevTime)

2.2 Software Project Portfolios

2.2.1 Risk Based Software Project Portfolio
2.2.2 Some other Portfolios

3. Research Questions

3.1 How useful is this approach?
3.2 Does the system successful achieve its goal?

4. System Description

4.1 Overview
4.2 Portfolio Table

4.2.1 Sparklines
4.2.2 Evaluation color rules

4.3 User Configurations
4.4 Portfolio Definitions

5. Experiment Design and Analysis

5.1 Research Approach
5.2 Classroom Evaluation
5.3 Industry Approach

6. Future directions

Struggling on HTML stylesheet

2008-10-06T22:34:00.002-10:00

We lately found that, as the width of project selection field is limited and the multiple selection is not horizontal scrollable, it is no way to tell projects with long name such as hackystat-sensor-ant, hackystat-sensor-eclipse and hackystat-sensor-emacs, so we have to change it. And I found this interesting page Select Multiple Form Fields. It make me think more about the solution. But for Wicket, the multiple selection check box is supported natively. So that will be the easiest and acceptable solution. In order to scale the selection field properly into the input table, I got struggling on HTML stylesheet again. But I am going better with it now. There are too many properties in stylesheet that are overwhelming to me at the beginning. But finally I found this page http://www.w3.org/TR/CSS2/propidx.html to be very helpful. It is a full list of stylesheet properties and quite easy to find what you need.

Also, the user configuration persistence using UriCache and initial portfolio definition XML are implemented. Now the portfolio become more user friendly especially you used to customize your portfolio configuration. But currently there is no working sample for the XML definition, so I want to move some of the hard-wired measures into the sample XML, may be those uncolored process analyses such as Build, Commit and UnitTest.

As students in ICS 413 will use Hackystat soon, the logger for user usage will have the higher implementation priority. Hopefully we will obtain a useful set of evaluation data.

XML-based portfolio measure definition

2008-09-29T15:55:00.004-10:00

Software project portfolio page is under good developing progress. After a week of effort, most HTML issues in project browser pages are solved.

The portfolio measure definition XML has been added. Now user can use XML file to customize measure definitions in portfolio page. When new telemetry chart is available, user can just add it into the portfolio measure definition XML file and the portfolio page will be ready to use it. As the system is designed for easy configure, the work is quite straight forward to use a XML approach to replace the previous code-based one. The default setting remains the same. So if user don't create the setting file, he will not notice any difference.

I will implement the persisting the user's last configuration settings this week.

A week passes fast

2008-09-24T00:36:00.003-10:00

After parameter list is added to portfolio configuration panel, portfolio's setting becomes more similar to telemetry. Thus, at last, we change all portfolio page's setting to match that in telemetry page. Time interval in configuration panel has been replaced by start date and end date in input panel. The granularity is moved to input panel as well. And the order of input elements in input panel of both portfolio and telemetry has been modified so that the layout become more consistent over pages.

Before, I was anxious to give user the capacity to select any start/end date in portfolio page. Because I want to create an alternative mechanism that can emphasize the week granularity and its purpose of up-to-date state analysis. But I now realize that, even in stock portfolio system, where most attention is focus on current and future price, provides function to view historical data. Giving user more control power over the system will not harm the initial purpose of the system. Furthermore, users may discover usage that we never think of.

When arranging elements in input panel, I got a chance to look into the panels' layout in the page, and accidentally figure out how to control the layout. Now the pages are more compact and nice.

After added the date selections to portfolio page, the SimDate scenario of portfolio can be put in any time period as I wish. So SimDate is changed.

Weekly progress report 0915

2008-09-16T00:25:00.002-10:00

The parameter list in portfolio configuration panel is finish! The layout may still need to change, but the functionality is complete. I finally choose to use the well constructed methods in telemetry session to get the parameter definition list. The parameter definition will be needed when showing the configuration panel and when verifying parameters before retrieve data for detail panel. Therefore, if the definition is retrieved every time when it is needed, the reaction of the page will be slow down, especially when open the configuration panel. While there already a map of telemetry definitions, which contain the parameter definitions, managed in telemetry session, it is easier to just use it rather than manage another map of instances in portfolio sessions. But telemetry page and portfolio page are designed to be separated, is it better to keep the class in each page separated? However, there are too many connection between telemetry and portfolio page, both logically and physically, e.g. charts in portfolio has links to the associated telemetry pages. So it should be acceptable to connection the session between them.

After this critical implementation, it is much easier to finish the delayed portfolio simulation data. Before, I got trouble in constructing useful coverage data using the existed methods in SimData, because the granularity of the constructed coverage data is "line" and the default in parameter definition is "method", which is used for portfolio page. There is no reasonable way to change parameters in portfolio page before, so if I insist to do it before adding the parameter configuration, I will have to add some dirty code such as if (measure.name == "Coverage") {...}. I am glad that I don't have to worry about it any more.

For the literature review, I started last week, but did not do well so far. I only finish one article in the week. That article took me over 5 hours to finish. I really need to find more time for it and improve my reading speed.

Literatures Review: Development governance for software management

2008-09-11T18:45:00.004-10:00

The article: Development governance for software management

Summary:
Governance is an interesting concept that is different from management. Governance is the act of exerting management control to guide development practices to compliance, while management normally consist of overseeing inward personnel and operations and a set of outward-facing responsibilities: planning, budgeting and forecasting etc.

To measure that to governance, the article use the term key performance indicator(KPI). The two KPIs mainly discuss about are volatility KPI and volume KPI -- both are process KPIs . The volatility is good to measure and predict development processes. Monitoring t he volume will help to forecast project to adjust resource distribution ; assure quality; and measure the efficacy of a software design in programming aspect. It also talked a little bit about work-product KPIs such as coding guidelines and complexity.

Relevance to my research:
In Hackystat, volatility is measured as Churn and volume is measured as FileMetric. Hackystat also provides other measures like coupling, coverage, build, test, commits and code issue. They are now equal. But from the enlightening from the article, I realize they should be group into two: process measures and work-product measures. The formers show the performance of the develop team while laters show the quality of the product. The understanding of these two groups are different thus research to them should be somehow differentiated.

Process measures will include:

Build, Unit testing, Code Issues, Churn and File Metric

Work-product measures will include:

Coverage, Complexity and Coupling

An important idea from the article is that all these measures have to be monitor over time to give significant meaning. Even when talking about volume, what make sense is the change of the volume over time -- LOC increase relatively slow indicates efficacy design and bloated code usually brings negative impacts. The idea of looking at these measures is to look into their trend, in order to predict the processes. This match our idea in Project Portfolio. We show not only the current state of each measure, but also the historical trends. And we estimate projects performance from their analysis trend as well.

Weekly progress report

2008-09-08T15:10:00.004-10:00

Last week I have mainly worked on project portfolio. Validators has been added to configuration panel to ensure that higher threshold is always higher that lower threshold.

Now I am working on a "simpleportfolio" simulated dataset in SimData for the use of Tutorial Guided Tour of Portfolio analysis. Hopefully will finish today or tomorrow.

Future improvements for portfolio will include:

telemetry parameters for each measure
new measures from telemetry: dev time, build & unit test
show member of each project -- may not necessary to be an analysis, just show it near the project name, or in active members measure if available.
new analysis: active members -- inactive members are those have significantly less dev time than others, others are all active members.

I will start literature review this week. Read and summarize at least 2 articles per week. Each article will have a blog entry to summarize it.

Plan of thesis resreach

2008-09-01T19:47:00.007-10:00

This fall I am starting the research on my master thesis.

The topic of the thesis is about how to compare projects with the metrics provided by Hackystat.

In large companies, it is possible to have hundreds of projects at a same time. The abilities to understand and compare this large amount of project is a great challenges and opportunities for both project managers and developers. Managers will want to figure out how those projects are doing as well as how the develop groups are doing. Developers will like to find similar projects or groups for further communication on technique, tools, etc.

Utilizing Hackystat's auto-collected data of software development procedure, we build a Software Project Portfolio Management page on Project Browser. It demonstrates projects using a set of software procedure metrics with both present value and historical chart. These values and charts are colored according to the thresholds and trends. Users are able to quickly scan this portfolio and get fast understand from colors and narrow down their interest to several projects. Then they can have further investigation into those projects.

However, understanding of such a software project portfolio is still far from sufficient. We don't know if all metrics we using are useful, or there may be some interesting metrics we should introduce to the system. How to color the charts and value is also uncertain. So we give the system the capacity to adapt different situation and need. Users can define the value thresholds for each metrics, determine if the higher value means better, and select the color for good, bad and so so values and trends. In the future, users may even define more than 3 color spaces for value and more colors for more different trends.

To study this system, we are planning first evaluate it in classroom. We will:

let the students in ICS413 to use this system during their course program development
do several survey about their opinion of the system and their preferred customization
track down their usage of the system

Hopefully after research with these data, we will acquire deeper understanding of this Hackystat powered Software Project Portfolio Management.

Lessons Learned from GSoC 2008

2008-08-20T14:36:00.007-10:00

This summer I have participated the Google Summer of Code 2008(GSoC) program under the Project Hackystat. The project I do is to develop Telemetry and Project Portfolio Management pages for Hackystat Project Browser, the overall viewer for Hackystat, which uses Wicket web app framework.
It is now end and I can summarize it a little bit.

* Technical skills acquired

I have been doing Wicket development for Hackystat Project Browser before GSoC. After this summer with Wicket, I get more familiar with Wicket and love it more.
I learn lots of new things in Wicket during GSoC, while the most important one I think is unit testing on Wicket. Unit test on Wicket is said to be facilitating but I have not dig into it for quite a long time because I can't find a way to manipulate forms or assert values in table that base on ListView. During the summer, I dig into this field again and finally found the way to access component in ListView, using the path like "projectTable:1:projectStream:0:streamCheckBox". Therefore, I can now fully control the page with wicket tester and test most parts of the page. And test coverage has increased greatly since then.

* Lessons on software development

This is the first time I work in distance with my mentor. I gained lots of helpful experience during it.

The most important thing during this is keep good communication with my mentor. We have choose the weekly skype phone call but it never works because the time zone issue keep make us miss the right time of the meeting. That delay the communication and delay the progress as well. Finally, we settle down to email approach. Asynchronising way seems much easier when communicate cross time zones.

* Other

As it is a program during the summer vacation, I initially did it as I am really in vacation: lazy for work. I did not accomplish much until I realize that progress is behind schedule quite a bit. If I have an advise for future GSoC students, it will be don't consider you are really in vacation that summer, take the program as a full time one. Another advise will be to start familiar as soon as possible, even before GSoC started.

Blog entries of Shaoxuan's GSoC progress

2008-08-20T14:24:00.003-10:00

I have been back to China for the whole summer while I am working on the Google Summer of Code project for Hackystata.

I have kept writing blogs of my project's progress, but unfortunately I cannot access google blogger at my home. So I put all my blogs in one of the project's wiki pages. Here is the link:
http://code.google.com/p/hackystat-ui-wicket/wiki/ShaoxuanBlog

Brief review of Informative-workspace

2008-04-02T01:26:00.002-10:00

Informative workspace is a project to utilize a 9 monitors LCD screen to provide development team some kind of software engineering information. Detail can be found here.

It is mainly a Java web app project using Wicket framework. I am just start learning Wicket and found it amazingly cool. It turns web app elements such as page, form and text field into java classes. Developers can handle these components by handling java classes, which they are much more familiar with. Then the code turn out to be more resemble to Swing application. Another cool thing is Wicket does not introduce specially HTML tag. all its tags are simple standard HTML, which means we can use all kind of HTML editor such as Dreamweaver to design our pages. This is extremely cool.

Back to the informative workspace project. Their main component is a project overviewer, which intend to show some overview of a project we are interested in. The viewer looks pretty nice, and its code is good written, with rich documentation. But when importing the project into Eclipse, I found that the project's build path is empty. I have to add all the Wicket and Hackystat libraries manually. It is not a big deal to me, but might be a big deal to some new developer that not quite familiar of Wicket or Hackystat. Hope the development team will settle this problem.

Another issue I notice in that the developer just left his Hackystat account and password in their code. It is OK for a on going project does not have a external configuration component. But at least they should change or remove their account information temporarily before making the release distribution. It might not be a big deal in this case. It is just all about good developing habit.

About the functionality of the viewer, its idea, which shows the files editing state of the project, is good, but the implementation, based on file tree, does not emphasis the goal. I will suggest to separate the viewer into two: a file viewer and a developer viewer.

The file viewer shows when and by whom is the file last modified and how often it is modified, and the files should be able to group by the last modified date. Then we can tell what files are active and what are stable from change.

The developer viewer shows what file they are working on, and highlight it if there is some other developers are working on it as well.

Execution Permission in Macintosh/Unix

2008-04-01T18:44:00.002-10:00

Today I have downloaded a latest tomcat and tried to install it. But surprisingly I cannot get the tomcat run. The error messages are shown as followed:

delia:ambienthackystat ZsxKiNG$ startup.sh
-bash: /Applications/Develop/apache-tomcat-6.0.16/bin/startup.sh: Permission denied
delia:ambienthackystat ZsxKiNG$ catalina.sh
-bash: /Applications/Develop/apache-tomcat-6.0.16/bin/catalina.sh: Permission denied
delia:ambienthackystat ZsxKiNG$ sudo /Applications/Develop/apache-tomcat-6.0.16/bin/catalina.sh
Password:
sudo: /Applications/Develop/apache-tomcat-6.0.16/bin/catalina.sh: command not found
delia:ambienthackystat ZsxKiNG$

That's quite weird to me because I used to install tomcat before my last system reinstallation and it worked just fine. It seems to be some permission issue in Unix like OS.

After some google searching, I finally got the answer: I need to change the binary files' mode to executable for me. It is done by calling chmod a+x catalina.sh . Chmod is a Unix command that change files' mode, such as r(read), w(write) and x(execute). Users are only able to perform these action when the file is set to permit them to do so.

After this command, I did not get the permission denied error any more. Instead, I got a error from tomcat saying BASEDIR variable is not set correctly. This variable come out to should be set by default equal to CATALINA_HOME, but something seems prevent it from doing so. As there are lots of files in bin directory, it is reasonable to guess it is because tomcat need to execute some other file that does not have execution permission. So I just type in chmod a+x *.* and try again, tomcat ran.

That is a very important experience to me, a new comer to Macintosh from Windows. In windows, authorization is poor and trivial, but in Mac, it becomes dominating. Even the administrator/power user will not be allow to read, write or execute a file if they have not the permission to that file. It need a lot of practice before get used to it. Knowing how to change files' permission is an essential step.

Ideas of Thesis on Ambient Device

2008-03-19T23:30:00.005-10:00

About the Project

I am now working on a Hackystat related project call AmbientHackystat, which utilizes Hackystat with ambient device to show users some useful project states that indicated by data collected by Hackystat. For example, turn the ambient orb into red when a build fail and let the nabaztag(a bunny) tell you when new changes are committed to code repository. I am think of how will the thesis be if it is based on this.

The system is defined by a set of trigger action pairs, where trigger get data from Hackystat database and check it, then invoke some ambient devices' action when the data satisfy some defined condition, such as last build fail.

Current progress

For the current moment, the functionality is quite simple. The orb can change color and pulse speed, and the bunny can only talk. There are only two triggers available: SensorData and Coverage. Though SensorData trigger is able to configure to monitor all kind of sensordata from Hackystat, sensordata is too low level that only a few type of data will be meaningful to be monitored. My partner is working on the bunny and it will be fully utilized soon hopefully. I still keep working on the triggers and more will be built in the future.

Thesis research point

There are two research point within this project:

1. What kind of trigger will be useful

This part is more like a data mining process. How to generate meaningful data for the thousands of raw data. What kind of information will be interesting to developers and project managers. How much detail should be provided, such as should it tell the number of coverage or just show if the coverage is within a certain level. It has to be careful when abstracting information, too much detail will make it verbose while too little detail will make it meaningless. However, this part is mostly similar to another project in my lab that call Boswell, which is a auto micro biographer that generate informative message and send it to all kinds of information platform such as twitter, email, text message or even facebook. The core parts of these two project are the same: abstract information from raw data of hackystat. So make be we should work together in this part for research.

2. How to present a project state/development event on ambient devices.

This part is much unique. How to present things is really a matter. It has to be noticeable enough to make people to be able to aware of it. But it should not be too distracting. People need to be focus on their work and don't want to be scared by ambient devices. Hence color and animation seem more fit this situation than sounds and speech. However, some particular events may do worth a vocal alarm, such as build fail may be one of them. These issues are needed to be settled.

Problems of research

The most important issue for this research is the user. We do need a group of user to really try this out to tell if a certain function is useful. It might be OK even if we are the users ourselves. But in order to evaluate the system, we have to be a group that working on a same project and work with some certain behaviors that will make these ambient devices useful. For example, we should have people working at the same time that they will not be too easy to have the whole view of their project, then we can evaluate if it is useful to let the ambient devices tell them if the build and test fail or if someone is opening/editing the same file his is working on.

At this moment, we only have 3 people working on this project. We usually work at different time. And I, the only one get easy access to these ambient device, is the one write most of the project. So I already know what is happening before the devices change their state. I hope we could have some real users to try this mechanism and provide feedback so that we can evaluate and improve. I think this will be an essential part of the thesis.

Ant, Hudson and Continuous Integration

2008-03-19T22:17:00.003-10:00

Use Hudson

I have just hooked my project onto Hudson, a continuous integration build system. The process is amazingly easy that I don't even need to read any guideline before finish it. Just click the new job, field out the configuration form, save and I am done. The configuration form is good self-documented, and every entry is followed by a small question mark where you can click on it for more help information.

In my project's job, I use subversion for source code management and invoke ant task for build process. It is pretty straight forward that I have already these tools for quite a while.

Write Ant Task for Hudson

After hooked my project onto Hudson, I become more anxious about the used way my project being verified: run junit task for unit testing and emma task for coverage separately because emma does not fail even though the unit test fail and it does not send junit sensor data. Though I already feel it is not a good way to do it for a long time but I just did not brother to look into the ant file to fix it. But now as the Hudson is looking for new commit every minute and the Hudson's server is hooking up over 20 project in the same time, I am become kind of worried about wasting the machine's power for redundant work. Also, for one of the purpose of using Hudson is to let it generate useful project data for my AmbientDevice project, it is better to get the data sooner so that the triggers can react earlier. So I tried to look into the junit.build.xml and emma.build.xml to look for some solution.

What I found is, what makes the build fail is not directly because of the junit tests fail, but because a variable is set to false when a test fails. The variable is set to the failureproperty attribute in junit task, and it is not set in the junit task that inside the emma task. So I just add it back to emma one and make the build fail when tests fail. It make more sense because coverage of fail tests will not be accurate. Also I add the junit sensor to the task dependence to send data of unit tests. But then I noticed that even though the build fails when test fails, the coverage data is still computed and emma sensordata is sent. The horrible thing is this incorrect coverage data will corrupt the project data. For example a false alarm of coverage drop by 20% may be raised because of the junit tests fail. That will be quite misleading and confusing. The solution for this is to add unless="junit.failed" in emma.report and emma.sensor to stop it from running. But the unless attribute is actually occupied by a variable called emma.disable already. However, it does not have any usability about this variable so far. So I just removed it. I am also wondering if it is possible and how to put an AND operation in the attribute verification such as if and unless.

But anyway, the project is now verified in the way that I want: just run the unit test once and get both unit test results and coverage result if test pass. And emma will fail when test fails so that it will not generate some misleading coverage data. I just remember that I have met this kind of matters before that I run the emma after some small changes and suddenly found the coverage is much lower than I thought. I was thinking that I might have forget to put some test on some method then I looked into the test cases. But I found everything are there. It took me quite a while before noticing that it is because a test fail just when it start so all the rest of the test is not executed. That is quite a terrible experience and I am glad that I have fixed it by the way.

Code Review for iHacky

2008-03-19T11:14:00.002-10:00

Review project: iHacky for hackystat

This is a facebook application for software developers to generate their professional profile and get connected with other developers. It is based on the software engineering toolkit framework called Hackystat which collect and analyze software development process and product data in order to get a better view of the state of software development projects.

For user installation, there is a InstallationGuide wiki page in their project site. The process is described pretty clear in it. Moreover, the installation itself is good self documented that I finished the installation smoothly without go through the InstallationGuide.

iHacky is written in php, which I am not that familiar with. So I did not put much time to go over the code. I will focus in the functionality of the application.

Inside iHacky, I have to say that it is now almost empty. No actual useful function is provided. I did not use facebook much and not quite familiar with what it is capable of. Therefore, it is hard for me to determine if iHacky is useful or not. It is still under developing, however, it is still not proper to release a empty framework for code view. Though one of our view's goal is to provide some advise of what kind of functions will be suite for this application, if the developers(and also users) themselves have no idea of what can the application provide, how can they assume that other user will have? In my aspect, I find it useless at this moment, and have no idea what it might be capable of other than those of Hackystat ProjectViewer and TelemetryViewer.

For advise, the idea of showing what kind of tools, IDEs and programming languages are my favorite sounds cool. Then maybe I can find more people that using a particular tool and go ask for help and discuss things.

What's a good design of XML schema

2008-02-25T11:27:00.002-10:00

With the help of JAXB, playing with XML becomes quite a simple thing in Java.

But when doing my project ambienthackystat, I find it not that simple now. Because though using JAXB classes are easy, you have to had a well design XML schema to generate those facilitating classes. XML schema design is a thing that I had never think about before.

I am using XML to configure the system, to define the system's functions with trigger and action pairs. It works well in our first milestone. But when we going further, I find it somehow circumscribe our design. When I thinking of some interesting functions, I cannot help thinking how to describe it with our XML file. Then it become quite frustrating when I found it quite hard to do it. The XML file is limiting my design now... Maybe it is time to modify the XML schema, but it the system, which base on that XML, will probably need to redesign too. It comes out to be a huge task all in a sudden.

Though I still want the schema change, this time we will do it more carefully.

PrimeDirectivesRedux

2007-12-10T07:19:00.000-10:00

In this entry I will talk about how the following tools and techniques support one or more of the Three Prime Directives.

1. Interactive Development Environments (Eclipse, VisualStudio, etc.)
It support PD 3 by providing great help in developing. Good IDE can provide much more than a text editor. It can not only compile and debug programs, but also give out useful analysis such as package tree, class outline and documentation of classes and methods. Some of them can even provide useful auto edit function, such as refactor in Eclipse.

2. Coding Standards and Coding Standards Compliance Tools (Checkstyle, etc.)
It supports PD 3 by greatly increasing the codes' readability, thus helps developer to understand the program better. When all variables are given with meaningful name, developers will be able to read the code just as a article.

3. Build Systems (Ant, Make, etc.)
It supports PD 2 & 3. As a automatic mechanism, it hides many specified detail in building process that user could just type some simple commands to build up the whole system. To the developer, facilitate the building process is also helpful. And as build systems are IDE-dependent, it enable developer to choose whatever he like to develop.

4. Automated Quality Assurance Tools (PMD and FindBugs, etc.)
It support PD 3 by ensuring a better code quality. These tools could help clean many implicit bugs and ensure some more reasonable coding style, which may be more close to programmers' common sense. A system with fewer bugs are surely more friendly to developers.

5. Black and White Box Testing
Testing supports PD 1. It guarantees the correctness of the system so that it can run as expected and accomplishes the tasks successfully.

6. Automated Testing Tools (JUnit, HttpUnit, etc.)
This supports PD 3 by providing a automatic checking mechanism to make sure what the develop modified is proper. If the unit tests fails after some hacking, developer will immediately acknowledge that some hacking mush be wrong and therefore bugs can be fixed cheaper.

7. Configuration Management (CVS, SVN, etc.)
It supports PD 2 & 3. Users can easily check out the system from a repository and, with a well developed build system, build the system for use. Developer gain more benefit from CM system that it facilitates collaborating development. It tracks down the system's configuration all through its life-time so that errors introduced in some versions are more easier to found and fix or recover.

8. Issue Driven Project Management
It supports PD 3 by providing useful guide line for new developers that what is deficient right now in the system so that they will have better idea of what to do to contribute to the development.

9. Use Cases
It supports PD 1. Using use cases, developers can better understand what users expect and therefore build the system that way. Developers can have better communication with users with use cases and let users know what the system will be and properly do some changing, even before the system is built.

10. Software Review
It supports all 3 PDs. It provide another way to test all the 3 prime directives. Developers could get more feedback as an external user or developer so that they can do a better job to perfect the system.

11. Agile Methods (XP, Scrum, etc.)
It supports all 3 PDs. With better communication with customers, developers are able to ensure what they building can really satisfy users need. New developers will find it helpful to attend daily meeting to gain some general idea of development process. And simplicity also make it easier to understand and maintain.

12. Open Source Licenses (GPL, CPL, etc.)
It does not support prime directives directly. But it may have some contribution to PD 3 because new developer may feel easier to continue a software with proper license that he will not need to worry about legal responsibility in copyright to the original developer.

13. Online Hosting Services (Google Project Hosting, SourceForge, etc.)
It supports PD 3. Online Hosting Services provide developers a great place to share developing source, including codes and other materials. With CM tools, developers can easily check out others work and submit his to keep codes up to date. That much accelerate the development process.

MyIsern-1.3-review

2007-11-15T21:59:00.000-10:00

Group viewed: MyIsern-Purple

Installation Review:
The installation is not as smooth as previous projects. First failure is from the String.isEmpty() which is not supported in Java 5. I will have to keep fighting with this problem untill Java 6 for Macintonsh released. Then when I importing it into Eclipse, the buildpath fails. However, it is not difficult to fix and get the isEmpty thing fixed for me too.
Then PMD and Checkstyle run successfully. But the JUnit test fail. The first problem for its failure is because their working directory's name is different from project distribution directory's, which included the "purple", and the project then is deployed with the name myisernweb-purple. Therefore, the test get a 404 when accessing the page. After fixing this, 4 tests still fail, HTTP 500 are reported. In tomcat there are 2 exceptions pop out when tests fail. These may be some real bugs, but I did not see into the code to find out what is happening.
The installation clear enough for me to get the software run.

Code Convention Review:
Some unused private variables and methods are reported in Eclipse, which might be historical remaining.
Some class share a same java doc, which is probably copy-pasted.
The most important violation is that the class Id implements a raw type Comparable, which is highly not recommended in Java 5. The right interface should be Comparable. and then the compareTo method will take a String as parameter rather than an Object.

Test Case Review:
First I notice that the EMMA's setting is not correct that takes jaxb class into coverage comptation.
The test cases are far from enough. As reported in EMMA, most methods in ActionBean are not tested, and only 66% methods in model are tested. I am not sure if the coverage is effected by the failures in JUnit test.
One bug I meet is when click save with name field empty, the system will end up to an exception.

User Interface Review:
The user interface is pretty good! Not only the functionality, but the appearance. In researchers page, there are picture in front of each person. When adding/editing new researcher, I can easily choose an existed organization from a drop down list, or create a new one. In collaboration editing page, years and organizations can be selected from multi-select field.
However, one important function is missed: delete objects. Objects cannot be remove once being added.
The affiliate researchers field is missed in organization editing window. Maybe authors thought that is should be determine by researchers' organization. But there are no view page available, so I cannot see an organization's affiliate researchers by any means.
Though using multi-select field to deal with multi-value field is really a good idea, it will become hard for user when selections become numerous. When using last assignment's data as input, there will be tens of organizations in collaboration's selecting window. However, I don't have much idea of how to do it good, and this approach is actually the best I have seen so far. But I think it is necessary to figure out some better way to do it.

Summary:
MyIsern-Purple's 1.3.1114 distribution is overall amazing, in consideration of such a short develop time for it. I do see many things useful for my project in it. I am more convinced that user interface is significantly important. This application's functionality is better than mine. But when looking at its nice pages, I do feel that my application is not comparable to this one.

WebAppQuestions

2007-11-03T20:56:00.000-10:00

1. Explain in your own words the meaning of the web "request-response cycle".
A: User request a page by clicking or some other means. The brower prepare the request and send it to the server, then receive the response from server and show in to the user.

2. Explain how servlets facilitate processing of the request-response cycle.
A: Servlets are able to generate page dynamically, so that the server can return much more different pages accordingly, without storing them beforehand.

3. How do you login to the Tomcat Manager?
A: I access my Tomcat page from http://localhost:8080/, then click the Tomcat Manager on the left to login, with username and password.

4. What is the Tomcat Manager used for in the StackStripes application?
A: Through Tomcat Manager, we are able to deploy our StackStripes without stopping the server.

5. What is the directory and file layout of an installed web application like StackStripes? (This is NOT the same as the directory and file layout of the StackStripes distribution!)
A: the web applications are installed in $CATALINA_HOME/webapps. Inside each application's directory, there is a directory called WEB-INF, where stored the code of the application.

6. How do you build a war directory structure in Ant? Where is this accomplished in the StackStripes application?
A: We build it using war task in an Ant target. It is done in build.xml, in the target "war".

7. How do you install a war directory in a running Tomcat server? What information does the Ant task need to accomplish this?
A: By getting this page: http://localhost:8080/manager/deploy, with argument path and war.
Ant need the host's address, the system name to generate the path, and the war file's location.

8. What is the "Model2" architecture? What are its advantages?
A: MVC contains 3 part: Model, which run web application, access database and generate the response page; View: whatever you see in your browser; Control: connector between Model and View, process user's request from view, interact with Model, and return the page to be shown next.
Its advantage is separated program logic from user view, so that it is much easier to modify and maintain the web application.

9. What are JSP pages? How are they related to servlets?
A: JSP is a page that contain static text data such as HTML, and dynamic content such as scriptlets or stripes elements. It can generate the proper page according to user's interaction. JSP will be compiled into servlet when first time it is requested, and recompile later when needed.

10. Why is there a delay when retrieving a JSP page for the first time? Where is the Java code for that page stored?
A: Because JSP will be compile into servlet for the first time retrieving. The Java code is stored in server.

11. What are JSTL tags? Why are they useful? What is an example JSTL tag from the StackStripes system?
A: JSTL tags are a kind of xml tag in HTML to implement some kinds of functionalities. They are useful because of their simplicity, which enables people other than Java programmers to modify the code. Example JSTL tags in StackStripes system are:

< c:foreach var="element" items="${actionBean.stackIterator}">
...
</c:foreach>

12. What are Stripes tags? Why are they useful? What is an example Stripes tag from the StackStripes system?
A: Stripes tags are tags that present functional framework that defined by Stripes. With Stripes tags, we are able to separate the Java control codes from JSP pages. All Java codes can be put in Java classes and called by Stripes tags.
Example Stripes tags in StackStripes system include:

< stripes:form id="PushForm" action="/Stack.action">
...
< stripes:select size="1" name="numToPush">
< stripes:option value="1">1
...
</stripes:select>
< stripes:submit value="push" name="push">
</stripes:submit>
</stripes:form>
</p>

13. What is HttpUnit? How is it different from JUnit? Why is it useful? What is an example use of HttpUnit from the StackStripes system?
A: HttpUnit is a testing framework that emulates behavior of browser to test the function of web sites or web applications. Unlike JUnit which is a unit testing framework for all java programs, HttpUnit is focus in testing web applications. It makes it possible to verify web applications or web sites by automated testing.
An example in StackStripes system is :

WebConversation conversation = new WebConversation();
String Url = testHost + "stackstripes";
WebResponse response = conversation.getResponse(Url);
assertEquals("Checking index.jsp retrieval", pageTitle, response.getTitle());

which check if the page is retrieved correctly.

14. What needed to be changed in order to implement the Double It button? What didn't need to be changed? What did you learn about the MVC design pattern from this exercise?
A: I need to add a stripes:form for the Double It button, and all other things remain unchanged. MVC is very facilitated to use, as simple as to add a method in a class.

15. What are the equivalence classes that need to be tested for the Double It button?
A: empty stack and stack with some elements.

16. Provide two screen images of your new StackStripes application with the Double It button, one showing the page before and one showing the page after hitting the "Double It" button.
A: before:

after:

17. What is the singleton design pattern? What is an example of its use in StackStripes? Why is it needed?
A: singleton design pattern is a design pattern that ensure there is only one object of the class in runtime. In StackStripes, a static instance is defined, And the construction method is private so that users can not call it to create a new instance. Also, there is a getInstance method serve as the function as constructor, which always return this static singleton instance.
We use it because we need to ensure that users are interacting with the same instance wherever the open the page.

18. Some of the StackStripes tests exercise code on the "server side", while others exercise code on the "client" side. Which test classes exercise code on the "server", and which exercise code on the "client"? How does Emma deal with this to create appropriate coverage data?
A: Tests in TestStackModel exercise codes on the "client" side, and tests in TestStackActionBean exercise codes on the "server side". Emma track the running code in server side and put those data in coverage.ec. However, emma has to stop the server to make that file finalized onto disk so that it can be read to create appropriate coverage data.

19. Running 'ant -f junit.build.xml' results in the following target invocations: tomcat.check, tomcat.undeploy, compile, war, tomcat.deploy, junit.tool, junit.report, junit. Explain what each of these targets do.
A: First, tomcat.check target checks if tomcat is running and can be access properly. Then, tomcat.undeploy target undeploys the existed web application in the server. After that, compile target compiles the java codes and war target packs all web files into a war package. Then tomcat.deploy target is called to unwar the war package and deploy the web application onto server. After deploying, junit target awoke junit.tool and junit.report targets to run the unit tests in the web application and generate an HTML report of the JUnit tests.

StackStripesExtension

2007-11-03T19:52:00.000-10:00

Source code: stackstripes-shaoxuan-1.0.1104.zip

Thanks to the reading pages and the lectures in class, I did not encounter much problems in this assignment. The only one problem I faced is emma failed at the beginning. This problem is discussed in our google group. I finally found out that it is because the default directory where the coverage.ec will be save is relative to the directory where you launch the tomcat. So there seems no universal environment independent approach to this problem. Even if we change the output directory to an absolute path, it will still need to do different setting in different environment, at least in different OS such as Mac and Windows. As I am used to launch my tomcat from HOME directory, I modified the emma.build.xml to look for coverage.ec in HOME directory.

EMMA coverage:
[concat] Emma Coverage summary
[concat] class: 100% (2/2)
[concat] method: 100% (18/18)
[concat] block: 98% (141/144)
[concat] line: 96% (46/48)

The only 2 lines that are not tested is the statement that catch EmptyStackException in doubleIt when calling in action bean. However, these 2 lines will never been reach because the the empty stack is checked in doubleIt method in stackModel. But I still what to keep these 2 lines of code in case the program will break down in the future, and keep it that way rather than just throws it because "nothing should go wrong in the view".

UseCaseSpecification

2007-10-22T02:12:00.000-10:00

Project home: myisern-1-ivory
Use cases in Wiki

Use case is a very useful way to clear up my mine of what is need to be done in the system developing. Writing use case is to look into the system in terms of actor. Looking as user give me better comprehension of the system. The most important part of writing use case specification is to make it simple, as concise as possible.

MyISERN-1.2

2007-10-22T00:30:00.000-10:00

Project page: http://code.google.com/p/myisern-1-ivory/

All tasks have been completed.

This assignment is not a technical difficult one. All it needs is to read command line input in runtime and put it into data. There are many field in each of 3 objects and adding codes are very similar, but not similar enough to use some methods to group them all. Though we can divide our job by different input part, we cannot divide it into different file. So this comes the first time we try hacking into the same file in the same time, but with different part. Though we try our best to not modify any codes in others part, we still get a commit error. At last, I have to put my partner's hacking by copy and paste into my code to make it a whole.

After built the first runnable version, we got lots of bugs and fixed them. One of the most critical bug is cause by Java's parameter passing mechanism. When doing input in multi value fields, it comes out to input a List of String, so we use a method to deal with all these fields. At first, we send the original one as parameter, and try modify it directly inside the method, but the result came out to be incorrect. Then we found out that is because we use the expression like currentList = inputList. Later we change it to return the result List of String and it finally work well. This is because Java pass the parameter by pass the pointers of the object by value, not the object itself. So we doing currentList = inputList, it actually change the value of the pointer passed from parameter, but not the original object. Now I know that if we need to modify the object from parameter to a new one, we need to use the method within the instance, not the "=".

During the next milestone, we will probably add more help and make input more convenient.

MyISERN-1.1.Review

2007-10-16T22:27:00.000-10:00

Project reviewed: myisern-1-green

Installation:
At the first try, I was astonished that the code I downloaded cannot even compile. Then I found out that is was because I am using Java 1.5 while the code use the java.lang.String.isEmpty() method which is added in Java 1.6. Java is evil, and Macintosh with java 5 is evil too.
At the review perspective, my review cannot going any further now. Meanwhile, I hacked into the code and fixed it so that it can work with Java 5 and continue my review.

Code format and conventions
Violations:

File	Lines	Rule	Comment
MyIsernXmlLoader.java	43~56	EJS-39	Document all members
MyIsernXmlLoader.java	281	EJS-9	Use meaningful name

Test case
Whitebox perspective:
The test case did a very good coverage as much as 96%. Only some lines of printing out error message are not reached. However, the results are not checked. Tester can only look into the output to see if thing goes right.

Blackbox perspective:
There is no blackbox test. All test, except those come from example, are to call the main method with different arguments. There are actually many methods that return meaningful object, which should be test with quality.

Bugs:
I notice that the system did not maintain the Unique ID to be unique across the entire system. Only duplicate within Collaborations, Organizations or Researchers will be prevented, but not between them, such as a researcher and an organization with same name.

Summary:
What I feel is that things can go wrong in any possible way. I have never image the Mac with Java 1.5 will cause me trouble until I faced it today. However, try to avoid use the very new things will be a good idea unless it is necessary. In case of check empty String, String.length() = 0 will also finish the work. That will avoid some more problems.

MyISERN-1.1

2007-10-15T00:08:00.000-10:00

Project page: http://code.google.com/p/myisern-1-ivory/

All tasks have been completed.

The main difficulty of about this assignment is to deal with various commands. We did this by create a class contains enum fields and switch these fields. It works good, but not good enough. I think in our next Milestone, which need more commands, I will change it to instance factory model, producing sub class of the parser to separate different enums and situations.

We are using the issue driven method in this assignment. The first thing we did is to write issues, and then we pick issues to deal with. On Thursday, we met in library and work together in the afternoon. During that time, we finish building the whole main structure of this assignment. Working together like that is really great, and is of efficiency.

MyIsernReview

2007-10-08T21:22:00.000-10:00

Code Author: MyISERN-1-Blue: Marcius Bagwan, Sonwright M. Gomez and John Hauge
Project Page: http://code.google.com/p/myisern-1-blue/

Installation:
Installation is smooth. All ant check pass. But when I import the project into Eclipse, there is a warning of unused import: import static org.junit.Assert.assertTrue; in TestMyIsernXmlLoader.java

Code format and Convention:
There are a few violations:

File	Line	Rule	Comment
MyIsernXmlLoader.java	119,152	EJS-9	Use meaningful name
MyIsernXmlLoader.java	44,45,46	EJS-39	Document all members
MyIsernXmlLoader.java	218	EJS-48	One sentence summary

Test Case:
As all 4 methods, which are new from example, return void, there is nothing much to assert in test.
In white box perspective, the test coverage is 100% according to EMMA, so it is well done.
In black box perspective, there are only a few conditional statements and all of them all satisfied at least once.
However, I found some problems:
1. Two format mistake when printing organizations. First, it prints the country as "CounterUSA", which should be "Countery: USA", I think. Second, name is printed like "Name:University of Hawaii", where I think a space is missing because every other places in the table, ":" always follow by a space.
2. Though he program has a good command-line arguments and also help message, the program exit without any word when using error argument. Help message should come out in this situation.
3. If the xml file contain an empty element like , or missing a element which would contain other elements, the program will end with NullPointerException. However, these kind of xml files will fail when running validateXml.build.xml. Therefore, the mistake of the program will be running without validate the XML files first.

Summary:
I am surprise that though code format and convention has been used and emphasized for so long, there are still violations in our codes. It seems to be easy to follow exactly the rules required. But actually, it isn't. Lots of practice is needed.
About test, though methods return void is evil and hard to test with assertion, we still ought to do it seriously, at least to guarantee that the output looks right, such as correct message with proper format.
Null object are evil and can usually break down a program. Most of the reasons when it happens in actual application is because of illegal input, for example, miss some essential parts or mis-structure format. So we really need to make sure to validate input before processing it.

MyISERN-1.0

2007-10-07T22:01:00.000-10:00

Source and Distribution can be found at http://code.google.com/p/myisern-1-ivory/

Here is the first task of collaboration. My group have two people, me and Ben Karsin.
In Thursday, the day after last class, we meet as required in library. But, as none of us have actually initiate coding, we think the meeting is useless. Then we change our plan to keep in touch online, mostly by email.

In Friday and Saturday, we did not meet each other. Then problem came out. Our emails are replied with delay as long as 20 hours. Then our whole schedule has been postponed. I then realize how important the daily meeting is. However, as Sunday comes, we don't want to meet in school again. So we decided to keep contact online by Skype. But it works! We can communicate as well as face to face. Therefore, we complete most of our code in an afternoon.

In my opinion, daily meeting is significant, but real-time communication via internet is important too, or even better. Because meeting is only a short time in a fixed time, while communication can be all the time. However, it maybe just the case when only 2 people in a group, when members become more, meeting should be more efficient, especially in initial design.

In the first day we meet, we don't settle down an overall design. When I start to write codes, I have to worry about what my partner will write, will our code be able to combine? Though change may be applied latter, we do need to get a better design at the beginning. As least define some interfaces, which will guarantee usability in certain way.
The result of our collaboration is that I write codes and Ben test it.

Overall, this is a very helpful experience of group development. It is interesting to develop with others.

CM.Practice

2007-10-07T21:50:00.000-10:00

The Google Hosting Project I created : http://code.google.com/p/webspider-shaoxuan/

In this assignment, our goal is to install SVN, try it and create a new Google Hosting Project.

In my practice, everything goes well. I finish the first 2 tasks without problems.
When creating Google Hosting Project, I face a small problem that when modified codes, no log is sent to svn group. My problem is cause by adding wrong member in svn group. At first I thought that the "googlecode" in googlecode-nonreply@google.com should be replace by the project's name. The right way is to add "googlecode-nonreply@google.com" as a member.

What I learn, or say realize is that the experience in previous assignment is really helpful. Now I can install tools, modify environment variables and test code smoothly.

Review of Webspider-chiaofen-1.0.926

2007-09-30T11:08:00.000-10:00

Review package: webspider-chiaofen-1.0.926, Package Author: ChiaFen Zielinski
1. Installation Review:
Installation is smooth. I did not encounter any problem when installing and trying to use this package.
Junit, PMD, Checkstyle, Findbugs are all passed with no errors or warnings.
Emma Coverage summary
class: 100% (1/1)
method: 100% (9/9)block: 93% (726/784)
line: 92% (140.1/152)
This is a good coverage.
ant -f dist.build.xml also BUILD SUCCESSFUL.
However, there is a small problem that when running java -jar webspider-chiaofen.jar-totallinks http://hackystat.org 100, It printed out plenty of empty lines. It may because of the logging scheme, which was implemented with println().

2. Code format and conventions review
Violations:

Files	Lines	Violation	Comments
WebSpiderExample.java	25-31	EJS-39	Document all members.
WebSpiderExample.java	294	EJS-32	Document is unrelative to the method, which is copy-paste from another method.
WebSpiderExample.java	241	EJS-33	Keep comments and code in sync.
WebSpiderExample.java	299	EJS-9	Use meaningful names.

3. Test case review

Black box perspective.

traversePages(String urlLink, int numLink), findMostPopularPage() and findTotalPages(), these 3 method actually are not tested. The test method only call the main method once that will use these method and assert a meaningless assertion.

getNumLinks(), this method is tested properly.

isNum(String s), only method is called. Result is not verified properly.

logList (String urlLink, int numLink, String logging), same as above. But this is a method returns void, verify result may be difficult. However, I think do some digging into the data of the test object is able to verify as least a bit of the result, while author verify none.

White box perspective.

I have to say that, the test case is terrible. It actually does no meaningful test. What it does is only to run the methods and see if there is any exception or error comes out. Nearly no result is tested. 5 of the 8 test method use assertEquals ("test", testUrl, testWebSpider.startUrl) to verify the result, which is meaningless in most of the test. The test pass even when I add some bug code in the class code. I think the only propose of this test case is to pass EMMA with a high coverage.

Break da buggah.

When I run the program with arguments of -totallinks http://www.google.com, it ends with Array Index: java.lang.ArrayIndexOutOfBoundsException: 2.
When using -totallinks http://www.google.com 99999999999999999, it throw out a java.lang.NumberFormatException and break down.

4. Summary and Lessons Learned
We should really do out test right, use it to test our code, not to pass the QA tools.
My code does not do well when I test with the arguments above which cause problem. I didn't think of these things either. I only catch them and give out a warning. I think I can do it better by adding more useful statement that will help user use it right.

About the extension assignment of WebSpider

2007-09-26T00:48:00.001-10:00

In fact, I did not do much about it. However, I found a page that mention something similar to the matter. Here is the link: http://life.neophi.com/danielr/2005/10/httpunit_and_javascript.html. I have download and tried his code, but not act well enough. The retrieving seems to be unstable. Sometime it can retrieve some javascript pages but some time gives out error Invalid URL encoding or Unexpected end of ZLIB input stream. I don't know why, but this may have something to do with the search algorithm that I store the only the link to visit later, but not the WebConersation object. But I did not take time to modify my code for a further test.
I think hack into httpunit is the only way. I did find some crawler, but most of them are implementation like our webSpider, which also base on httpunit, so of course do not fully support javascript.
I read in httpunit's website that full support for javascript is one of their aim now. Maybe next version will fixed this problem. :)

WebSpider Extension

2007-09-25T23:39:00.000-10:00

webspider-shaoxuan-1.0.926.zip
Most part of what I did is about the testing.
First, I rewrote my test class and separate tests that one test method test one of the target's method. But there is one exception. I will talk about it later.
Second, I modified the method that return void. They are retrivePage(String) and getPages(int).
Both of them are modified to return an int value. The return of retrivePage(String) is the number of links found on this link. The return of getPages(int) is the number of pages that actually retrieved. After that, I found it easier to test them, but also better to know them that I did not know how much pages were actually retrieved before.
When I check for the pmd, it tell me I have to add a assert in testWebSpider() where I test the main method by only call it. In order to past PMD, I added a test of another method with a assert. So this is the only test method that test 2 methods.
I amitted that to do this in order to past PMD test is not a good way. But in the view of PMD, I think the proper way is remove that whole test method because it actually has nothing to assert. However, I would like to keep the test. I think keeping the test is more useful than satisfying PMD.

WebSpider

2007-09-24T03:31:00.000-10:00

Download: WebSpider-Shaoxuan.zip

Task 1~3 are accomplished.

Encountered troubles:

1. To saved URL from WebLink. The trouble is becase the WebLink object's getURLString method sometimes returns a incomplete URL. I have tried to fix it manually, but did not make it. Then I tried to store the WebLink object instead, but it still does seem to work. Finally I was told that webLink.getRequest().getURL() will return what I was looking for, so I used it.

2. Deal with exceptions while accessing pages. At first I tried to modify my code so that it can deal with more and more exception. But soon I found that the exception type various as much as pages on Internet and some of them are cause from javascript which httpunit support poorly. So, alternatively, I catch and log them instead of solve them. Also, I turn off httpunit's exception throwing when counter a javascript page.

3. To log. After reading in it, I found that the logging package is really convenient. I can log different message in different level that can be easily controlled and filtered. I only need to change setting in initializing. But then I encounter a problem when trying to print out the logs to Console. I tried nearly all classes in the package when found out that a new handler must be create in order to do what I want.

In testing, I found out that command arguments in main() method can be transfer by a String array. But as the main have not a return, I don't know how to assert the running result. The test just prove the code can be execute, but not guarantee the rightness. However, the test can cover over 90% method-lines. What are not covered are the exception catching expression. But if I change the argument to make it process a job of larger scale, it is sure that these line will be covered. I don't know whether testing like this is a right way. But it seems to be the only way to test a method without a return.

In this task, I really feel that the QAs tools are not additional jobs, but facilities. Used the xml files build in previous assignment, I nearly don't need to look into it anymore. Just use it to examine my code! That feels great. I think I will be greatly benefited from them in my future work.

Stack - Experience of Ant & QAs

2007-09-16T16:28:00.000-10:00

Stack-shaoxuan-5.0.916 is released. Distribution package can be download from stack-shaoxuan-5.0.916.zip.

The goal of this assignment is to learn Ant build system and Open source Java automated quality assurance tools (FindBugs, Checkstyle, PMD, etc). It is an interesting experience.

Task and Problems:
There are five tasks in the assignment. All of them has been completed. I have encountered some problems when installing QA tools and creating JavaNCSS project(Task 1 & 4). The other 3 tasks are easy.
problems in these tasks are mainly because I got my Macbook this week and it is the first time I use the Mac OS. I took a long time to figure out how to set the environment variables when I was trying to install those QA tools. Then when I trying to build the JavaNCSS project, the CLASSPATH thing held me for a while again. But now, when I solved all these task, I learnt things not only about Ant and QA tools, but also Mac OS.
The most important things in these tasks is to setup environment variables for every QA tools. It is significant to make then run right. In Mac, I need to run a terminal and use vi to create/modify a file named .profile in my home directory(where cd ~ will take you to). You cannot see the .profile in Finder, that is the reason why I confused. In the file I put all the definition of all the environment variables. When doing task 4, I got confused with the CLASSPATH because there is not a CLASSPATH definition when typing printenv, but the java compiler runs well. Then I know that it is because JAVA has enhanced in Mac OS and the compiler will search default directory for java classes. I can use CLASSPATH=./:{other directory} to add new destinations to CLASSPATH. But later, I found that the class required by javancss can be defined inside javancss.build.xml. That is a better way and I use it.
When improving the unit tests, I think it is easy that just add lines to test every methods and use the try/catch expressions to test the exceptions.

Impression of Ant and QA tools:
Ant is really a great tool that facilitate the automatic building procedure. Though its command structure is not so easy to learn from the beginning, its power enable us to build however we want to.

Difference between SCLC and JavaNCSS:
Though these two tools are both use to count lines in code files, SCLC count also blank line, which I think is useless, while JavaNCSS counts CCN(Cyclomatic Complexity Number) and average of NCSS, CCN and Javadoc per project/package/class/method, which make sense in knowing about the complex of your code. Therefore, JavaNCSS is what I prefer.

CodeRuler Revision

2007-09-01T22:31:00.000-10:00

New version of source code: http://www2.hawaii.edu/~sz/ics613/sz_v3.zip

After studying the score rules and testing codes from UCSD Spring 2004 Programming Contest. I made a lot of changes in my strategy.

Strategy of Knight:
By testing the codes of XXX_420_XXX, the second place of UCSD Spring 2004 Programming Contest, I surprisingly discover how excellent a simple brainless strategy can perform. Its idea is very simple: capture the nearest castle. It doesn't even group up knights when attack or annihilate enemy units when captured all castles. It just let every knight attack the castle that is nearest from it and let them stand still while all castles are captured. However, in most of the test matches, it does succeed to capture all castles! No one can ever beat it in castle capturing, though its stupid stand still method makes it some trouble because it doesn't notice when enemy knights are kill its peasants, as long as they don't capture castle.
Therefore, I decide to switch my strategy to a similar one: capture the nearest castle and annihilate enemy units when all castles are captured. That works pretty good. It acts just like a blitz, and is able to take all the castle within 150 turns when fight with samples! (500 turns per match)
I think one of the reason to its success is that it naturally separates its force in groups that maximum the possibility of surviving in raids and efficiency of capturing castles. The success of such simple idea is amazing.
I have also test the code of champion of UCSD Spring 2004 Programming Contest, its AI is much better. It will score every castle depend on the distance and nearby defence and select the best one to attack. It sounds nice, but it doesn't work so well in my test. One of the reason is its knights seems hesitant when selecting target, and this makes the capture procedure much slower.
Perhaps, to be simple is just a good idea.

Strategy of producing:
By studying the score rules, I found a key of getting high score when winning the match. That is to produce knight rather than peasant when your peasants are enough. The reason is, a knight scores 2 and a peasant only scores 1 at the end of the match. And, you don't need to continue to create peasants in order to claim all land. As I tested, even 40 peasants will be enough to claim more than 4000 land. So I continue to produce knights when I have 50 peasants. It just likes a cheat! I increase my winning score by 400 by just adding such simple expressions! That makes me possible to win the highest score event in case that I lost over a half of matches, and guarantees me the champion as long as I win a half of matches.

Strategy of peasant:
Nothing special. I just extended the searching area when choosing directions to improve the efficiency of claiming land.

CodeRuler Review

2007-08-31T19:12:00.000-10:00

I have reviewed Ben Karsin's code of CodeRuler from http://bkarsin.blogspot.com/2007/08/coderuler-may-best-bot-win.html. His AI strategy and algorithm is fantastic and his code is well written. His score system in direction decision is a very good idea. However, it seems a little bit too intelligent.
The first obvious problem is that his peasant will go up and down in a couple of fields when nearby fields are all claimed. It can be fixed simply by adding a random mechanism when all directions score the same.
The second problem is a bug that when his knight trace chased enemies' peasant to the world's angle, it will stop a block away from the target without attacking it. I don't know what's going wrong in the algorithm. This bug needs to be fixed.
The third one, which I think is most important, is a question about knight strength. In Ben Karsin's Ruler, knight runs away when its strength goes down to a degree. This strategy does protect knight from being killed, but it significant reduces the usable power of knight. Because when a knight runs for live when its strength goes below, for example, 10%, that means only 90% of a knight's power is usable. Although run-away knights can go to capture enemy knight, it makes enemies' knight able to attack anything as they wish, such as castle. In my opinion, warrior is honored to die in the battlefield bravely. So I don't agree in run-away strategy. Maybe let one or two knights go to capture enemies' peasant when strength gets low is OK, but most of the knight should fight to death.

Following is the summary of violations in the code.

Summary of violations

File	Lines	Violation	Comments
MyRuler.java	16,29,30,*	ICS-SE-Eclipse-2	Code indented 3 spaces or tab, not 2 throughout.
MyRuler.java	16,29,138,*	EJS - 7	Lack of space between operations and parenthesis.
MyRuler.java	104,106	EJS - 76	Use experssion instead of block statement.
MyRuler.java	114,116	EJS - 5	'{' in a new line, not in the last of upper line.
MyRuler.java	181,182,183	EJS - 6	Lines too long.
MyRuler.java	142,196,276	EJS - 9	Do not use a meaningful name.
MyRuler.java	74,81,90,*	EJS - 37	Used C-sytle comments to explain implementation, not one-line comments
MyRuler.java	250	N/A	The first if statement contains a blank expression, though is not prohibited by the rules, it made me somewhat confused.
MyRuler.java	224,50,54,*	EJS - 7	no blank lines to seperate each member of the class

Code Ruler Results

2007-08-29T02:15:00.000-10:00

Member: Shaoxuan Zhang

Source Code: http://www2.hawaii.edu/~sz/ics613/sz.zip

Strategy:
- Castle produce to maintain a certain proportion between knights and peasants.
- Peasants claim the land that isn't belongs to the rule and avoid being attacked. Otherwise, it move to a random direction.
- Knights are grouped into two, one assault team one castle capture team.
-- Castle capture team capture castle one by one, and destroy enemies' target around its way or near the castle.
-- Assault team capture castles only, do not attack any other enemy units. If no castles remain, they will wait in the center of the map for orders.

Test results:

	Test 1	Test 2	Test 3
sz vs. Migrate	820:0	784:0	797:0
sz vs. Gang Up	780:60	796:78	785:42
sz vs. Split Up	773:54	844:56	749:54

Lessons learnt:
CodeRuler: I now have understand some strategy of CodeRuler. Most of them have been implemented into my Ruler. Both my peasants and knights have advantage.
Peasant: Instead of move randomly, my peasants now know to move and claim an unclaimed land first. It is significant to accelerate the speed of claimming land, and contributes to mine beating Migrate Ruler to 0 points. In most of the test, my ruler claimed as much as 4000 land which means 85% of the world! In test of more player, this number can reach 4602, which just 6 fewer than the world. I think it is strong enough. Some intelligent strategy may farther improve this one, but the improvement will not worth the expense in calculation.
Another advantage is to avoid enemies' knight. The peasant will move away from the enemy knight if it is near. This feature not only make my peasants more safe, but also waste enemies' force to pursue the peasants. Sometimes, we can see other ruler's knight running after my peasants across over the world. Though my peasant sacrifices at last, it win me lot of time to capture the ruler's defendless castle.
Knight: The idea of spliting into two teams comes from the sample Split Up ruler, and it prove to be useful. The assault team can usually find and capture a empty castle.
However, the most important idea is to sweep out the way to the target castle, especially in one on one game when both rulers' knight is moving to others. When raiding not only capture enemy unit on the way, but also near the way, it will act as defending when attacking, especially for the knight that newly created from the castle. I think this idea is the most important one to help me win.
Castle: Castle's strategy is simple, just manage the proportion between knights and peasants and stop creating knights when enemy runs few.

Java, Eclipse and Programming: Actually, I did not get much new in Eclipse, just get more used to it. For Java, this assignment is a very good exercise of writing javadoc. After I generate my javadoc files, I do consider it helpful! I think I will continue practise and use it the future.
Then I would like to talk about programming. I got two experiences: the first is to design well before writing code, the second is to write more methods to group up the functions. Both these two are fundamental indeed, but now I have fully realized them. I have writen the ruler twice. The first time I wrote it without carefully design. So my ruler cannot even beat the Gang Up Ruler. The second time I rewrote it with careful design and used more nested method calling, that prove to be very helpful when lately I want to modify my strategy.

Working with another person? I worked alone in this project, so... no experience in this aspect.

Areca Backup -- an Open Soucre Software Experience

2007-08-26T10:50:00.000-10:00

Areca Backup is an open source software, you can see its project in http://sourceforge.net/projects/areca/.
Areca Backup is a file backup tool written in java. It supports data compression & encryption, incremental backup, file history explorer and many other features. Areca Backup also includes a transaction mechanism which guarantees your backups' integrity. --the description is copied from the link above.
Now, we come and see if this OSS satisfy the 3 Prime Directives.

Prime Directive 1. The system successfully accomplishes a useful task.
Areca Backup does do a good job in its task. It provides a very powerful and facilitative backup function. We can easily create a target of folder(s) and backup it. The archives will be save in the defined location and can be managed within the target. Also, targets can be grouped that makes us easier to manage them. When backup a second time, it will compare the existing files with the backup files, only the changes will be saved into the new archive. In recovering, we can either recover the whole folder or select some of the files to recover, that makes daily backup fast. What is more, we are able to choose in with archive we want to recover from. It is really a very helpful backup software.

Prime Directive 2. An external user can successfully install and use the system.
It installation is very easy, just need to run the install package and click next till finish.
However, the guide for new starter is deficient. There is no any guide of what to do to make the first backup archive, even in help website online. It took me several minutes to find out how to start and what a target means. But after beginning, it is easy to use. By adding a initial guide, the familiar time will be shorten from decades of minutes to couples of minutes.

Prime Directive 3. An external developer can successfully understand and enhance the system.
Though there is a source available in download, unfortunately, I did not find any develop-level documentation. I am able to create a project with the downloaded src, and the code is well organized. But it is difficult to read into it without guiding documentation. So I cannot do anything with the project. Thus, I think this Areca fails Prime Directive 3.

Conclusion.
Areca Backup is a useful software. It can help us in daily backup procedure. But as an open source package, it is not friendly to external developers. However, considering its value in future use, the investment will prove its worth. Maybe the package will be more developer friendly with some further information from the project admin.

FizzBuzz

2007-08-21T20:59:00.000-10:00

As an assignment, I just finished the FizzBuzz in Java on Eclipse. It took me 35min to do so. Since learning java for not so long time and not writing codes in java for month, I have totally forget even the basic syntax structure such as where the program begins and how to output a word. It took me most of the time to getting the memory back by searching in java's doc, but get nothing. At last, I found the answers in the "helloworld" tutorials... After that, I just spend several minutes to finish the program.
Fundamental is of most importance because you can nothing without it. Just like the beginning of my programming, I have all the idea even the codes in my brain in C++, but I don't know how to write the correct code in java. Maybe the basic elements of java & C++ are almost the same, but just almost, to program right, we need the exactly right one. Though much of basic codes can be generate by the platform automatic, it did not provide me one. Therefore, be familiar with the programming language is all the time important.

Following is the codes of my FizzBuzz:

public class FizzBuzz {
public static void main(String args[])
{
for (int i=1;i<=100;++i)
{
if (i%3==0)
{ System.out.print("Fizz"); }
if(i%5==0)
{ System.out.print("Buzz"); }
if(i%3!=0 && i%5!=0)
{ System.out.print(i); }
System.out.println();
}
}//end of main()
}

I have just read all others blog associated with FizzBuzz. I found that they all (i%15) to find out the case when we need a "FizzBuzz". But in my codes, there is not an (i%15) because the "FizzBuzz" is composed by "Fizz" and "Buzz". "Fizz" comes from (i%3) and "Buzz" comes from (i%5), so no (i%15) is needed. Meanwhile, as both (i%3) and (i%5) will be executed in any case, I cannot simply use a else to cover the remaining cases. So there comes to be an (i%3!=0 && i%5!=0) . In this FizzBuzz case, my implementation needs 4 compare statment while the other just take 3. But when the factor increase, such as FizzBuzzDozz(3,5,7), my way increase 2 compare statment every new factor while the other increase by P(n+1,2)-P(n,2)=n. That will be huge when factor become more. And, some of the item will easily be forgotten.
However, it looks like the algorithmic thing rather than the software engineering one... Anyway, just left it here :)

Frist log

2007-08-20T23:19:00.000-10:00

The beginning of the Blog....