WebSpider

Download: WebSpider-Shaoxuan.zip

Task 1~3 are accomplished.

Encountered troubles:

1. To saved URL from WebLink. The trouble is becase the WebLink object's getURLString method sometimes returns a incomplete URL. I have tried to fix it manually, but did not make it. Then I tried to store the WebLink object instead, but it still does seem to work. Finally I was told that webLink.getRequest().getURL() will return what I was looking for, so I used it.

2. Deal with exceptions while accessing pages. At first I tried to modify my code so that it can deal with more and more exception. But soon I found that the exception type various as much as pages on Internet and some of them are cause from javascript which httpunit support poorly. So, alternatively, I catch and log them instead of solve them. Also, I turn off httpunit's exception throwing when counter a javascript page.

3. To log. After reading in it, I found that the logging package is really convenient. I can log different message in different level that can be easily controlled and filtered. I only need to change setting in initializing. But then I encounter a problem when trying to print out the logs to Console. I tried nearly all classes in the package when found out that a new handler must be create in order to do what I want.

In testing, I found out that command arguments in main() method can be transfer by a String array. But as the main have not a return, I don't know how to assert the running result. The test just prove the code can be execute, but not guarantee the rightness. However, the test can cover over 90% method-lines. What are not covered are the exception catching expression. But if I change the argument to make it process a job of larger scale, it is sure that these line will be covered. I don't know whether testing like this is a right way. But it seems to be the only way to test a method without a return.

In this task, I really feel that the QAs tools are not additional jobs, but facilities. Used the xml files build in previous assignment, I nearly don't need to look into it anymore. Just use it to examine my code! That feels great. I think I will be greatly benefited from them in my future work.