Evidence of Josh's Barren Mindscape

Simple clojure string examples

2011-10-13T20:02:00.003-10:00

One of the easiest places to delve into a new programming language I've always found to be string manipulation. Not one of the most exciting aspects of most modern programming languages. However so many atomic processes either boil down to manipulating strings or rely on it at one level or another. Now in clojure as with lisp before it due to the fact that all data is code and all code is also data, clojure does not explicitly provide functions to manipulate strings. Clojure's lazy evaluation means that all functions are data type agnostic, which takes some real getting used to coming from such strongly typed languages such as Java or C#. Meaning that even though these examples use strings they work with any type of collection: sets, lists, etc. Here are some examples of some basic data transformations you can do with clojure:

01. First function example
(first "test") = t
Return the first item in the collection.

02. Rest function example
(rest "test") = (e s t)
Return all the items in the collection except the first element.

03. Str function example
(str "test") = test
Return the items in the collection as a string. If there is more than 1 item they are concatinated together, if a single item is passed then str is equivalent to the java toString() method.

04. Set function example
(set "test") = #{e s t}
Return a set of the distinct items in the collection.

05. Subs function example
(subs "test" 1) = est
Return a subset of the collection from the start or designated starting point (0 based) to either the collection's end or the designated ending point.

06. Nth function example
(nth "test" 1) = e
Return the nth item in the collection.

07. Reverse function example
(reverse "test") = (t s e t)
Return a collection with the original collection's items in reverse order.

08. Drop function example
(drop 2 "test") = (s t)
Returns a collection of all but the first n item in the collection.

09. Drop-last function example
(drop-last "test") = (t e s)
Drop last item in collection.

10. Count function example
(count "test") = 4
Return the size of the collection.

11. Cons function example
(cons "test" "01") = (test 0 1)
Cons function from lisp. Constructs a new collection with "test" as the first item in the final collection and treats "01" as a collection separating its elements.

12. Concat function example
(concat "test" "01") = (t e s t 0 1)
Returns a lazy sequence representing the concatenation of the first collection's elements with the elements of the second collection.

13. Lazy-cat function example
(lazy-cat "test" "01") = (t e s t 0 1)
Returns a lazy sequence of the supplied collections.

14. Take function example
(take 2 "test" = (t e)
Returns the first n items in the supplied collection.

15. Take-last function example
(take-last 2 "test" = (t e)
Returns the last n items in the supplied collection.

16. Take-nth function example
(take-nth 2 "test" = (t s)
Returns every nth item in the supplied collection.

See the clojure 1.3 api for full specifications: http://clojure.github.com/clojure/

Converting VARBINARY to VARCHAR yields only opening character

2011-07-12T16:16:00.010-10:00

Came across this interesting feature of SQL Server the other week. The system we're working on takes incoming text files and stores their contents in full as VARBINARY fields in order to maintain a complete collection of messages that have been uploaded to the system. Currently we're circumventing this default load process and loading directly into the VARBINARY columns from our external database.

DECLARE @foo NVARCHAR(3)

SET @foo = 'bar'

SELECT CONVERT(VARBINARY,@foo)

This yields the expected Unicode binary: 0x620061007200

However when unpacking these VARBINARY fields to VARCHARs we get the following:

SELECT CONVERT(VARCHAR,CONVERT(varbinary,@foo))

Yielding: 'b'

Did you see the error?

The problem arises when unknowingly boxing an NVARCHAR into a VARCHAR because you're unaware of what the initial datatype was before it got converted to binary. The variable @foo was originally an NVARCHAR that got converted to binary thus retaining the additional UTF-8 bytes. Now when unpacking that binary data into a VARCHAR expecting ASCII data we're only left with the opening character.

Say two different programmers wrote these two separate pieces. The first programmer loads NVARCHARs into the binary fields, and the second programmer keeps trying to extract these binary fields as VARCHARs only to find the data truncated down to a single character. Obviously the best thing to do in this case is to have the programmers communicate with each other in order to maintain data consistency. However if this proves impossible for whatever reason it may prove useful to check to make sure your fields aren't being inadvertently truncated when being unpacked from their binary fields.

tl;dr Just use NVARCHAR for everything and save yourself the headache

Getting emacs's rgrep working in windows

2011-02-06T22:30:00.008-10:00

In order to get many of the useful emacs utilities working in windows you'll need to install a UNIX emulator for windows. I chose to go with cygwin.

There are 3 different ways of grepping in emacs:
grep
lgrep (local grep - will only search the directory you're currently in)
rgrep (recursive grep - will search subdirectories)

Installing cygwin seemed to grant access to 2 types of grep: grep and lgrep. While rgrep continued to spout "Parameter format not correct" messages.

> rgrep -nH "meta" *.*

find . "(" -path "*/CVS" -o -path "*/.svn" -o -path "*/{arch}" -o -path "*/.hg" -o -path "*/_darcs" -o -path "*/.git" -o -path "*/.bzr" ")" -prune -o -type f "(" -iname "*.*" ")" -exec grep -i -nH -e "meta" {} /dev/null ";"
FIND: Parameter format not correct

Grep exited abnormally with code 2 at Sun Feb 06 23:12:00

It turns out that lgrep utilizes only the normal grep formatting. However, it appears that rgrep pipes its arguments to "find" in order to build the subdirectory tree where it runs grep on each folder individually. However emacs utilizes windows find by default and must be pointed to cygwin's find instead in order for rgrep's unix style arguments to work.

Please add the following line to your .emacs:

(setq find-program "C:\\path-to-cygwin\\bin\\find.exe")

*NOTE* You need to fully exit emacs and restart instead of merely running load-file ~/.emacs in order to overwrite many of the cached cygwin interfaces.

At this point rgrep sort of works. When running the same above search query I started receiving the following output (note how we are now running cygwin's find method:

> rgrep -nH "meta" *.*

C:\cygwin\bin\find.exe . "(" -path "*/CVS" -o -path "*/.svn" -o -path "*/{arch}" -o -path "*/.hg" -o -path "*/_darcs" -o -path "*/.git" -o -path "*/.bzr" ")" -prune -o -type f "(" -iname "*.*" ")" -exec grep -i -nH -e "meta" {} NUL ";"
/usr/bin/find: `grep': No such file or directory
/usr/bin/find: `grep': No such file or directory
/usr/bin/find: `grep': No such file or directory
.
.
.
etc

Emitting a "/usr/bin/find: `grep': No such file or directory" error for each unsuccessful search attempt.

After much searching I found a workaround here detailing how to pipe the warning message to "/dev/null" instead of "windows-null" in order to suppress the warning messages found in the grep output buffer.

Please add the following to your .emacs file:

;; Prevent issues with the Windows null device (NUL)
;; when using cygwin find with rgrep.
(defadvice grep-compute-defaults (around grep-compute-defaults-advice-null-device)
"Use cygwin's /dev/null as the null-device."
(let ((null-device "/dev/null"))
ad-do-it))
(ad-activate 'grep-compute-defaults)

This fixed rgrep for me. Nothing I seemed to do has gotten grep -r working however, and it still continues to only search the current directory. Supposedly installing ack will make all these problems go away but I haven't gotten it working quite yet. Hope this helps.

The Problem with IDE's

2008-04-24T10:14:00.022-10:00

I've always been a proponent of IDE's. I've always believed that the use of the correct IDE to solve a given problem increases productivity by vastly noticeable amounts.

My experience with IDE's in the past has been a tumultuous one. When first starting off in computer science we were introduced to such complex editing tools such as textpad and (shudder) notepad. These text editors gave way to an entire year writing C code in VI, which segued into using more robust editors such as Visual Studio and Eclipse. For in that infancy of IDE knowledge that was all the IDE's were and all they represented; a larger prettier looking text editor.

That was up until about 2 years ago when I took Software Engineering from Phillip Johnson. It was only then that the full scope of this Java language we had been using came into play. Now instead of dealing with five or ten classes we were dealing with hundreds of classes and thousands of method calls. Without the formal training with the Eclipse IDE, specifically tailored to the Java language, I would have been not only lost, but even worse, unproductive.

From that day forward I was sold on IDE's. Code completion and Intellisense were next to godliness, refactoring took just a few clicks, package structure, library imports, code reviews, Eclipse had it all. The future seemed to be a near infinite increase in productivity as I became ever more efficient in using eclipse's keyboard shortcuts and macros.

Then I started my first job in the software development industry, and about a week into it I had a conversation with a coworker which went something like:
Me: "Oh so what IDE do you guys typically use when you're working on projects?"
Coworker: "None."
Me: ".... what???"
Coworker: "I usually code in textpad or notepad and compile it with ant. Most people just pick a single text editor and get really good with it."
Me: ".... what???"

How could it possibly be that in a professional business environment there was no cohesion between developers and the tools that they used. Were they possibly ignorant about how much faster IDE's allow you to program? How was it even possible to be productive?

The IDE Divide, by Oliver Steele, makes the argument that the developer world is divided into two competing camps: The language mavens vs The tool mavens. His argument is basically thus; to the language mavens the real productivity increases occur when more powerful languages are introduced and all IDE's are functionally equivalent text editors. Conversely the tool maven would argue that all languages do functionally the same thing, implement the same methods, accomplish the same goals, and that the real productivity increase comes from the tools used to implement them. He goes on to argue that not only do these camps exist but that they are also mutually exclusive. That developers cannot (or in only the rarest of occasions) be both language and tool mavens simultaneously. This is due to the fact that learning either new developing tools or new languages makes it easier to learn the next developer tool or language that comes along, to the exclusion of the other. Therefore a positive feedback loop exists with respect to the individual camps, making it harder and harder to bridge the functional gap between the two.

Now Oliver Steele is quite obviously a language maven but his argument still stuns me. I've had the luck of having to migrate between two languages which are nearly identical (Java -> C#), but in the past I've noticed some problems when I venture outside of the languages I've formed some level of comfort with, let alone be left alone without my trusty eclipse editor. Luckily Visual Studio provides many of the feature that eclipse does and more.

Now I've seen Visual Studio developers be defined as "IDE Users" rather than software developers, which would be enough to enrage even the most open minded developer. But programming in immense languages like C# or Java is "clearly" (to me at least) easier and more efficiently done within an IDE. So many of the modern high level languages contain such immense API's and so many different libraries that it's seemingly impossible to find the method you're looking for within this giant pile of information.

Has programming become an intractable solution where one has to leaf through API's hundreds of pages long looking for a single method signature? Is it inevitable that the higher level programming languages will evolve to become more and more complex until they eventually become so obscure that they become essentially impossible to use without a specialized IDE?

I sometimes wonder if my coworker was right, are the more advanced IDE's actually useful or are IDE's really useless? Should we stay with the most barebones input possible? Is it even possible to proficiently code in gargantuan languages such as java or C# without the use of a proper IDE?

The alternative according to Oliver Steele (and its a compelling one) is to rely on languages to provide the functionality that we seek. Put the spare development effort into learning the features of new languages and how best to implement them into our projects.

Either way I spent my free time for the last few days finally deciphering how to use emacs. I know I'll be forced to use Visual Studio at the client job site I'm moving to tomorrow, but this article could possibly have changed my whole programming productivity philosophy (alliteration!). I should give it a few weeks to digest before I determine what IDE's really mean to me and how they figure into the grander scheme of being a complete developer.

The Requirements vs the Analyst

2008-04-17T16:55:00.001-10:00

I always wondered what the "Systems Analyst" job entailed because the job title just sounded so damn cool. Now that I've read Software Requirements by Karl Wiegers I have absolutely no envy for anyone with that job title, in fact I sympathize in the utmost with what must be one of the hardest jobs in the software engineering industry.

In his book Karl Wiegers goes into great depth and detail in an effort to outline the software requirements collection process, which many have touted as being the most important aspect of software engineering due to the fact that the cost in correcting flaws in the requirements left undetected scale exponentially later into the project. It goes into great detail breaking down the groups of users, differing techniques to elicit requirements from either users or managers, as well as the clarification and organization of these requirements into a comprehensible software requirements specification.

I've personally found requirements practice to be the absolute hardest part of software engineering. Not to say that working with other people is either undesirable for the average software engineer, but it is exhausting whenever the effort is to get a large number of people to agree on a single definition or interpretation of a business rule. The greatest tool that I found in Wiegers book was the 'education' concept. If time permits the ability to educate the users, managers, and all groups whom the finished product will be of value, is the greatest one has. Clear communication and education regarding the importance of precise requirements definition goes a long long way towards making the requirements gathering process bearable.

The ideas and processes I had in my mind whenever someone would mention "requirements gathering" involved a bunch of emails, hallway conversations, maybe a lunch or two, and possibly a meeting involving a few people. While Wiegers situations represent the absolute optimal situations given enough time and manpower, his vision of cooperation and communication between users and developers really would be a thing of beauty were anyone able to put it into practice.

Refactoring and Cyclomatic Complexity

2008-03-31T22:51:00.003-10:00

"Refactoring" by Martin Fowler is a great book that gives formal definition to a huge mishmash of informal programming procedures and habits. It not only argues the benefits and "whys" of the whole refactoring game (which are often informally understood and championed by many), but also gives formal definition to the types of code decay most often occurring as well as formal definitions and names to the possible solutions.

Fowler lovingly refers to this code decay as "code pungency" or "bad smells in code" when encountered by a developer. However in 1976 Thomas McCabe coined the term "Cyclomatic Complexity" in a paper outlining a software metric used to analyze the complexity of programs. The metric measures the number of linearly independent paths through a given chunk of source code.

"Cyclomatic complexity is computed using a graph that describes the control flow of the program. The nodes of the graph correspond to the commands of a program. A directed edge connects two nodes if the second command might be executed immediately after the first command." [wikipedia]

Basically as the number of conditional statements in a program grows, so do the number of corresponding paths and therefore the graphical representation of the program's control flow. A program with no if/else statements has a single path. Add in one 'if' statement and the number of paths grows to two. One path if the 'if' returns false, and another if it returns true. Any function call including some sort of boolean assessment will fork the program's path, not to mention 'while' and 'switch/case' statements, ad infinitum.

The software metric of cyclomatic complexity serves as a baseline indicator for when a program needs refactoring. From a top level viewpoint the analysis is useless because all programs must include some degree of complexity in order to be useful. Where the measurement become useful is at the method level where methods including some 30 or more linearly independent paths are clearly marked as overly complex and should be scheduled for refactoring.

Several tools exist which do good jobs of measuring this metric:
[JavaNCSS]
[Dependency Finder]

One which includes an excellent visual output of the state of a project's complexity is Panopticode, which also provides visual output of your given level of code coverage as a bonus.

Example: The complexity state of the Cruise Control 2.6 project (hit refresh until it loads).

All in all refactoring is an extremely necessary concept for every programmer to understand, even those who practice it now stand to gain a great deal by understanding why they are doing what they do. I hope to find other refactoring tools to aid in the developmental process soon.

The Power of Faith vs The Crushing grip of Reason

2008-03-17T19:33:00.004-10:00

A central theme in Lost is the modern personification of the conflict between Faith and Reason. I find this duality to be personified on a day to day basis through my experiences with modern technology and computer science.

The preparation for the upcoming first day at the new job has thus far been an attempt to balance the theoretical with the practical. Reading books such as Code Complete and The Mythical Man Month represent the theory side, while the practical side is a mishmash of Rails hacking and beginner .net, VB and C# tutorials. Should I have faith in my abilities to pickup new technologies and learn as much practical theory as possible, or should reason prevail and should I focus more on learning the physical construction tools personified by languages and IDE's. Of course in practice there is no picking and choosing because practicality must be combined with theory in order to be a complete programmer. So then is there also a balance to be found between faith and reason? Is there a difference??? Or is science just a higher form of religion in who's laws and postulates we steadfastly believe until they are overturned.

Again everything streams toward balance, like sliding down the strings held by some gigantic blindfolded woman of justice onto her scales. I must be empowered to balance myself.

"Trying to find a balance, trying to build a balance." - Slug

The Mythical Man Month

2008-03-16T20:32:00.003-10:00

It's interesting to reflect how much and how little has changed in the past 30 years since The Mythical Man Month was written. The current chapter I just finished reading, regarding the idea that change is an inevitability resonated with me very clearly. The idea that the actual task of any programmer is to realize a vision for a program consumer and deliver to them a semblance of satisfaction rather than any physical program, that the program itself as a physical object only quantizes and represents the customer's desire for change was fascinating.

So not only is change inevitable its also a product one must become accustomed to delivering.

Progress with becoming a prolific rails developer is proceeding at a decent pace. At first its quite unnerving how much work is done in the background via ruby scripts, but at the same time that nervousness is offset by excitement for the power of the rails platform. There is an unbelievable amount of things automatically built into rails, not only interoperability but programming practices as well. DB migrations are akin to version control, the entire MVC architecture. Working on session handling and error correction at the present moment. Proving to be very exciting.

The Great Job Hunt

2007-06-20T22:26:00.000-10:00

Ahh the good ole engineering log. More like my personal organizer and to-do list.

In preparation for the upcoming job interviews I thought it best to look over the following basic technologies just to ensure that I have answers to various questions, and to go over the things that I "supposedly" learned in 413.

Review List:
Servlets
Apache Tomcat
Apache Ant
Junit/Unit Testing
Rest API (via the Hackystat RestApiSpecification: http://code.google.com/p/hackystat-sensorbase-uh/wiki/RestApiSpecification)
Apache Struts
Ruby on Rails

Seems like a reasonable list to review for the present.

Zombie State

2007-05-04T17:30:00.000-10:00

Yes well at least I got the demo working a little while afterwards the meeting. Didn't go as well as it should have, but I'm pretty close to being finished, should be completely done sometime next week. Hopefully.

Wanted to blog about the help I got from Hong Bing before I forget what he told me. Basically the data that the jstl buttons receive come from within the form declaration on the jsp page. If there is no data declared between the form tags then the button and subsequent java class will have no data to handle. This is leading to a single button implementation. Also of note is that the user emails will need some kind of remove functionality. There are still some bugs with the xml persistence mechanism and I want to be sure that its happening correctly before I try to break something else.

Sleep Deprivation and H4's

2007-05-03T15:19:00.000-10:00

Last night was the first time I slept in almost 4 days.

It's probably a good thing that I hadn't blogged during the interval since it would've been incoherent rambling (much like how I normally blog I guess).

Monday was the first H4 I had done and was with Hong Bing working on my TimerTask implementation. The TimerTask makes a call to its own generateEmailIfTriggered method, making a reference to whether the Alert associated with it is enabled or not. The unfortunate result of my fooling with the TimerTask was general email spam being sent out to myself, Philip, and possibly Aaron and Austen although they said that they were unaffected. Last night even though I had disabled all alerts for every registered user the server was still attempting to send emails to Philip. Good thing that Road Runner is registered on the RBL List as an untrusted source of spam. So I can't send emails to test the module from home, but I can see where it is attempting to send emails to. If you received any spam from me, I apologize.

The hardest thing I'm having to deal with now is the transfer of data from the CommandRequest classes, as any data entered into them seems to get wiped out every time the page gets reloaded. The solution to this I'm assuming is JDOM persistence which is the last thing I need to get done.

Today's H4 was with Robert Brewer and dealt with going through the Agile Web Development with Rails book and the tutorials found therein. It highlighted the Ruby's interfacing to a database, creating and validating data and formatting the data for display on the web. I really hope I have some time in the near future to begin programming Rails although the sense that so many things are hidden in the implementations is slightly unnerving. I think its the same with any new programming language you encounter since Java often does the same thing.

This friday is the final deadline for the project and I'll be visiting Referentia in the morning. I still need to write up a user and administrator guide tonight. So its time to make the last burst for the finish line.

Tutorials bonanza

2007-04-29T12:22:00.000-10:00

Some of us are graduating soon so I thought I'd put some stuff together for us to keep out collective hacking skills together. For the past year or so I've been amassing a collection of tutorials from digg, of which I have actually gone through very few. Some of them are general computer science knowledge, but most do represent technologies which we have touched upon this semester and I thought I should share them in order to provide those of us who will soon be having a lot of free time something to keep our technological chops in good condition.

Top 12 Ruby on Rails tutorials

5 books every programmer should read

The Ajax toolkit framework for Eclipse

Over 140 Ajax tutorials

Python Tutorials (only because I know absolutely nothing about Python)

Computer Science Cheat Sheet

54 general programming exercises

Keep those coding skills up to snuff graduating class no matter what you're doing from here on out.

Echo2 and the REST Architecture

2007-04-27T19:49:00.000-10:00

I was really impressed with the REST architecture as outlined by Phillip today. Especially with the ease of the way in which modules interact via HTTP get, put, etc commands. Instead of the big black box program that it was before splitting up the system into smaller black boxes which communicate via such simple procedures seems infinitely easier than the hidden communication channels that you had to hunt down before. When I first started out developing for Hackystat I felt like it was this big black doorless house that I was stumbling around in the dark looking for a way in. Eventually some of the hidden doors revealed themselves through the wood paneling, or I jumped through some windows, but under the rest architecture its only a bunch of outhouse shaped buildings that have clearly defined paths between them. From a developer standpoint this is infinitely better than how it was before.

I'm have a probable entry point for my TimerTask to intrude upon the Hackystat architecture, and once its working will finish the alert portion of the module. I still need to find a way to set a different time interval at which the emails get triggered for testing purposes though.

Working on the UI as I write this.

Peer UI Programming

2007-04-25T23:56:00.000-10:00

The data gathering bug, which resulted in me gathering data for only my single user instance instead of gathering data for every user and consolidating them in a single daily project data instance, was a result of me not being logged in as the administrator of the project over which I was trying to gather data.

When logged in as the owner of a project the mail and all its formatting seems to work fine, although I will need more data in order to verify this. Right now have switched all efforts onto working on the alert implementation of the emailer which includes its own timer task implementation because the default Alert interface was dependent upon the default timer task, and therefore the default emailer which does not support html emails. Speaking of which I still need to figure out why the email does not display properly when viewed in gmail. As Robert says it is probably the external stylesheet and .gif links embedded. I will have to find a way to circumvent this eventually. With the TimerTask I need to figure out: 1) How to implement a second differential instead of a single set hour each day, 2) Where the alert makes calls to the actual Java code.

So the final to do list for the remainder of the semester looks like this:
1) Alert
2) Resolve Email issues
3) Fix the UI data gathering issues
4) Persist the data gathered from the UI

Wow I might actually finish this thing on time.

I have no idea what I'm going to talk about tommorrow.

Yay.

Houses that Open and Ghosts in the Shell

2007-04-20T16:53:00.000-10:00

The open house event was fun and informative since up until now I've never been privy to any of the products and services that they provide to the DoD. Models and Simulation, Autonomous Agent systems and Data Mining are all things that I love reading about and hope I have a chance to work on at some point later, if not for a company then certainly from a personal research perspective.

An autonomous agent system for Hackystat would be interesting. Sending the agents out into the web, to crawl and collect data, which they send back to the central Hackystat repository. Would be cool to see in action. I especially liked the microwave emitter demo and their dispersion algorithm. I still would like to do something modeling systems of objects in their relationship to one another. Like the swarming or flocking presentations I've given in other classes.

Would any of it work for Hackystat, who knows. I haven't even taken an AI course yet =\. Remind me to ask the guys in the lab for a good AI book.

Still still still trying to track down the dpd bug. I think it might have something to do with the fact that sending a single project instance to the data gatherer returns data, but iterating over all the projects owned by a user returns the indication that every project has zero data. Hmmm. The interface I don't think is a bug rather than I need to implement the request hooks instead of just trying to pull the data from the main UI java class. Should shoot for getting this done by this weekend.

Golgotha Tenement Blues

2007-04-18T00:41:00.000-10:00

Finally finally finally finished the html template after reducing the code to almost a quarter of what it was originally. I was making it much harder than it actually was and as a result of trying to gather too much metadata regarding the daily project data I revealed how little I actually knew about how the data was structured. Once I actually set out to see how the data was actually structured the answer to my formatting problems fell out immediately. Took way too much time figuring this out so now I really need to bust ass with regards to the UI. Hopefully I'll be able to figure out the data gathering problems quickly.

HTML Templating and Daily Project Data Collection

2007-04-15T17:59:00.000-10:00

There seems to be an issue in the gathering of the daily project data with regards to gathering data for all user for a given day, as opposed to just data for the user logged in. Right now the data is gathered via the DailyProjectDataCache, which I'm thinking might not be the right implementation for this project. When the Daily Project Data instance is returned from the cache the code calls the getDrilldowns method, which returns the drilldowns for the indicated day.

What I don't understand is why the getDrilldowns method uses the 'User' parameter. Right now I'm only able to collect data for the user logged in, but when the DailyProjectDetails module uses this method and invokes it in the same way through the data cache, it is able to grab data for all users.

I've dug through almost every Sdt extension of the Daily Project Data abstract class and can't see ANYWHERE that the user parameter is used.

If anyone knows the answer please let me know.

I should be finished templating the HTML code tonight but I need all of the data from all users to populate it, or else the whole purpose of this project is meaningless.

Structure Review and Project Status of the ProjectStatus Module

2007-04-13T19:27:00.000-10:00

I can say with confidence that the back end of the program is basically complete, with the exception of a few bits and pieces here and there. Most of the remaining back end has to do with the alert implementation.

What now remains to be done has almost primarily to do with the UI. The data flow of the UI and exception throwing pages. The alert enabling and the email entry. Persisting the email addresses entered and checking their validity also needs to be addressed. This persisting of the emails via JDOM and verifying their validity is probably the largest portion of the project that still needs to be addressed.

With the completion of the Java1.5 conversion complete i'll have more time to concentrate on these issues.

Class Cast Exceptions and Java 1.5

2007-04-11T00:28:00.000-10:00

Well i'm slightly happy to note that all of the frustration earlier today was due to the Java 1.5 conversion. The whole time I had been getting a sneaky ClassCastException which didn't show itself until I put some output code directly into the DailyProjectDetails class. The idea was that the getDailyProjectDrilldowns method is supposed to return a List of a List of String[] (blog won't let me post angle brackets due to html rejection). However it turns out that this was only happening for MOST of the SDT's, with some of them returning god knows what. When the code would hit these SDT's which did not return the expected Types the code would bomb without any indication why.

Thus all efforts are now turned to the Java 1.5 conversion (which I should have been spending more time on anyway). After all it turns out that the SDT's the code was bombing out on were the ones i'm responsible for... =-(

Drilldown Issues

2007-04-10T14:33:00.000-10:00

I wish subversion had a browser that cached all of your old versions, so that you could browse through your old code in case you changed something that you needed but didn't necessarily want to revert everything back to that previous version.

I didn't have a problem early getting the daily project details summary strings, but for some reason i'm having a much harder time getting the drilldowns. I can't tell if i'm trying to look at the wrong day's data, or if there is no data associated with this user for this particular day. Or if i'm just messing up on the data gathering side. Or maybe the project isn't getting loaded correctly for that particular user when hackystat gets deployed.

Now i'm just venting. I need to keep track of these issues and eliminate them one by one. If I could even determine that there is indeed data to look at before I try to manipulate it.

Use Cases and Data Compartmentalization

2007-04-09T19:13:00.000-10:00

Meant to blog yesterday but fell asleep trying to think of what to write.

I refactored the package structure to resemble an MVC design architecture since I was having trouble separating the view from the data collection. I wanted to have the View able to call the sendEmail method while passing the information it had gathered from the UI (such as email addresses) without needing to touch the data at all. I refactored and moved files around to reflect the data flow more accurately, only to discover after doing so that this is not MVC after all.

At the moment all the view does is gather data from the UI and call some sendEmail method in the mailer. The mailer gathers this data from the view and calls some getEmailBody method in the data manager which returns a String representing the email body. What the mailer doesn't need to know is that this String getting returned is completely made up of html tags. In essence the mailer shouldn't have to touch the data at all either.

All of the data collection and formatting should only be occuring in the data manager. The only thing the data manager may need is the User name, in order to get the names of the projects owned by the user. These projects with their associated data, and the html template page, are the only things the data manager should require (I think). It should take care of all formatting and only need to return a String, which the mailer cannot discern from any ordinary emailBody String.

I think this method of comparmentalizing the data should work out well.

Oh yes and I really need to write a detailed use case for this new module design in order to eventually finish the UI.

Pair Programming and Summary Grids

2007-04-05T09:18:00.000-10:00

The Pair Programming last night was extremely helpful. Much thanks again to Austen. Some of the items we discussed included:

1) HTML tag replacement and recognition. To dynamically generate the HTML page the module needs to recognize the specified HTML tags and replace them with the title followed by the data from the drill down lists.
2) The difference between how the summary strings are formed as opposed to the drill down data.
3) The typing of the List returned from the summary grid containing the drill down data and how to access it.
4) Issues regarding whether every sdt implements its drill down data uniquely.

Some miscellaneous things we went over included unit testing, proper java styles, and especially shortcuts and hotkeying things in eclipse. Which is just the greatest thing ever quite frankly.

This accessing and instantiating the drill down data remains the hardest thing left to do in the project. The remaining items to finish include:
1) Finish the UI/fix the UI bug(s).
2) Finish the new timer task implementation.
3) Figure out why gmail doesn't like linking to external style sheets.
4) Persist the entered email addresses and boolean "enable" value using JDOM.
5) I also need to update just about everything on the google project website.

At the very least I need to have at least a manual email generator which displays the drill down data in a formatted email finished by next week. I'm not sure what to package for the release this friday though.

3/3 - UI Work

2007-04-03T12:55:00.000-10:00

Trudging through the UI, learning more and more jsp tags and incorporation of java code into the jsp pages in order to dynamically name the variables which the command will be pulling data from. Ultimately i'm moving in the direction of working on Persisting the data with JDOM which i'll hopefully get to today.

Going proves slow though.

Mood: Resilient

3/30 - To Do List

2007-03-30T16:03:00.000-10:00

This list is the result of talks this week between myself Phillip Johnson, Austen and Aaron

1) Stream Daily Project Data (does it need to be cached?) and read it into the module.

2) Implement Timer Task (use 30 second intervals for development

3) Redo UI, new JSP pages closer to the project management UI

4) Persist emails entered into UI textfields using JDOM

5) Create unit tests

3/29 - Forks in the road

2007-03-29T17:11:00.000-10:00

This new direction of the project begs the question of what to do with the old code and whether it should still constitute the presentation to the class in terms of release packages.

Must be sure to clarify during the meeting tommorrow whether they still want this old functionality or if it should be scrapped.

3/29

2007-03-29T17:06:00.000-10:00

Points of interest from the Peer Programming session with Dr. Johnson:

1) Remember to use JDOM to persist the attributes and data defined by the users within the module.
2) The only capability that needs to be examined it probably the TimerTask which I haven't touched thus far.
3) First thing to do is probably to examine the structure of the ProjectManager to get an overview of how the user interface should be constructed. This is probably the best place to start.

...go

Go

2007-03-29T16:08:00.000-10:00

"The Ancient Japanese considered the Go board to be a microcosm of the Universe. Although when it is empty it appears to be simple and ordered, in fact, the possibilities of gameplay are endless. They say that no two go games have ever been alike. Just like snowflakes. So the Go board actually represents an extremely complex and chaotic universe."

- Sol Robinson from the movie Pi

2/27

2007-03-27T14:45:00.001-10:00

Didn't realize that I haven't been posting in a while.

Things finished:
1) Finally got the interval jsp page written and have successfully retrieved the interval selector from the jsp page. Now just need to figure out what to do with it, ie. how to implement only daily and/or weekly alert triggers. Doesn't really matter though because the weekly analysis is something else entirely.
2) Also successfully sent an html email, although the formatting is fucked because of the List.toString() method. Should be able to find some other way around this. concat(String)?

Things to do/goals for 2.0:
1) Need to figure out why the multipart email is throwing a MessagingException, only able to send the html email, and not the text/html emails both simultaneously.
2) Still need to port the system over to an alert instead of manual invocation. Possible topic for the peer programming but I should be able to do this one on my own.
3) Of especial interest, possibly a topic for the peer programming, need to find a way to cache the daily project detail html output and how to retrieve this html page to send it as an email.

These goals should comprise the v2.0 release for next week.

3/20

2007-03-20T11:16:00.000-10:00

At Referentia right now.

Next Phase will be the idea we discussed before about combining the different telemetry streams in order to consolidate the different chart reports into either 1 or 2 charts at the most.

Necessary things to accomplish:

1) Finish reading Cedric's thesis.
2) Decipher the telemetry stream language.
3) Complete the Alert module for next week's release.

Considering trying to make this into an AJAX implementation is this would prove even possible. Should propose this idea to Phillip.

Issues:

1) Lots of things are hard coded into the module at the moment. Must spend some time refactoring these before the release.
2) Considering just starting the project over from scratch, copying the core module code into a new alert module and keeping the existing code as a testing suite for manual testing. Reasons: converting the module to an alert while still retaining testing functionality is proving problematic.

3/19

2007-03-19T12:13:00.000-10:00

To Do List:

1) Get HTML into the emails.

2) Finish Refactoring the Module.

3) Write the XSL script and subsequent HTML --> email loading.

2/16

2007-03-16T12:23:00.000-10:00

Didn't get a chance to comment on the CSDL meeting wednesday.

Of especial interest was the idea of dynamic graphs, graphical data which although being displayed graphically in charts or graphs could also contain drill down abilities. Such as the pie charts, which initially represent project level statistics but when clicked upon separate into separate pie charts which represent package or module level statistics.

And yes the comet tail idea wherein a spectrum of colors in the shape of a comet tail could be representative of project level metrics.

3/12

2007-03-12T19:28:00.000-10:00

Things currently trying to get accomplished by the end of the week.

1) Finish HTML formatting of daily project details.

2) Finish JSP page design for the interval selector class.

3) Finish alert class with respect to the getDisplayUrl method.

Will add more if i can think of anything else. Need to go into the lab tommorrow to ask questions to Hong Bing and Phillip. Also need to think of some analysis questions for the CSDL presentation on wednesday.

3/10

2007-03-10T17:02:00.000-10:00

Starting work again after being sidelined by disease.

3/6

2007-03-06T11:46:00.000-10:00

Went to Referentia again around 11am.

Realized that I still have both the alert class to finish writing as well as getting the interval selector jsp page to work. Hoping to finish both of these today and start on the html page design tommorrow. Hopefully dreamweaver will be able to help come up with some design flow ideas. Possibly something to talk about in next week's CSDL meeting or in the lab.

Be sure to annotate the developer guides and use cases with screenshots before friday.

Trying to get back on track with the programming schedule. As the size of the program grows there's more and more stuff to tweak and bugs to fix.

Really really need to take some time to work on unit testing this thing.

3/5

2007-03-05T10:57:00.000-10:00

At referentia right now, working on hardcoding the data into an html file as a prototype to the dynamic html generation which will finalize the format of the emails.

3/1

2007-03-01T17:51:00.000-10:00

Collaboration with Referentia went well today. DailyProjectData collection is working but only in string format. Still need to port the main ProjectStatus implementation over to an Alert implementation instead of manual invocation. Also need to create the interval selector class and its associated jsp.

Must make time either monday or tuesday to go back to Referentia to do more collaboration.

2/27

2007-02-27T20:29:00.000-10:00

Spending some time figuring out if the alert should be project specific and how much harder that would be.

At the moment it would seem that a map dataset would need to be kept somewhere. Where the projectNames would represent the keys, and the values would be represented by boolean enabled/disabled values. Not sure if they would want this functionality but we'll try implementing it before thursday.

Maybe getting somewhat off track here, should probably go back to trying to collect the daily project details tommorrow and figuring out why that's not working first before implementing any new functionality.

2/25

2007-02-25T18:45:00.000-10:00

Currently working on completing 3 things before the next Referentia meeting this coming week.

1) Must retrieve the dailyProjectDetails data from the stdCmd alert class. Might not be able to use this method to retrieve the data.

2) Need to write the ProjectSelector class to select which project the data will represent.

3) Need to recreate the drop down list on the jsp page and populate it with projects.

2/23 Version 1.0 release class

2007-02-23T13:45:00.000-10:00

Sitting in class right now.

Issues:
1) Annotate the hackyStat statistics screenshots.

2) Add screenshots to the guides and use cases.

Demo's went well, need to do 2 things for the next week. Populate the body of the emails with data from dailyProjectDetails, and implement a timeframe selector for the jsp page.

Larger goals for the next week include html generation of data via xslt scripts to populate the body of the emails.

2/22

2007-02-23T06:00:00.000-10:00

Worked all night on overhauling the project from its old model to the newer integrated design which actually plugs into the hackystat system instead of standing on its own. Still stuck on the Apache bug documented on the news page however.

Be sure to ask Austen and Aaron about the concerns brought up at the CSDL meeting, as well as the idea that this email functionality already exists in another form within the hackyStat system.

Be sure to ask Phillip about the bug.

2/21

2007-02-21T14:06:00.000-10:00

Some future things of note to ask during next Referentia meeting.

What are the goals they wish to achieve with the new email functionality? Will they want to extend the alert system to other aspects of either the developer process or events within the projects they are working on.

2007-02-20T20:58:00.000-10:00

2/19-2/20

Make extensive revisions to the system, added a new package to the system which will house the Alert section of the program. Running into some workspace issues when trying to see data, or possibly some sensor issues when trying to send data, not sure which it is. Currently working on powerpoint presentation to give to CSDL tommorrow. Really not too sure what i'm going to say. I'll try reading Cedric's paper for some ideas.

2/15

2007-02-16T00:43:00.000-10:00

Got my first sense of agile programming today. Had a meeting with the Referentia staff today at 12:30. Upon review of the previous requirements by Aaron, Ryan and Austen they decided to take the project in a different direction for the time being.

Instead they now want me to concentrate on developing a branch of the current application module which can autogenerate emails on a to-be-determined frequency. The content of the emails is an alteration of the DailyProjectDetails and the recipient of the email will be the developer to whom the data pertains.

Basically they want to be able to report back statistics to individual developers on all the normal data collected. This is combining statistics reporting with a reformatting of the DailyProjectDetails.

For the past week I had been working on the MVC design for the managment reporting but Aaron says to put a hold on that for the time being.

I told them that's fine and that i'll start working on this new aspect of the project instead.

2/10-2/13

2007-02-13T22:27:00.000-10:00

For the past 4 days i've been working on updating the content of the google projects webpage so that it covers what i am doing as well as what i hope to accomplish extensively.

My primary goal at this point is to go back over the hackyStat documentation to gain further understanding of the hackystat system, and to find whether the implementations which i hope to incorporate into my application exist already. I'm just hoping at this point not to have to build the data collection and analysis portion extensively from scratch.

2/9

2007-02-09T17:07:00.000-10:00

Entered in the first use case and the issues hoping to accomplish by the 1.0 release.

2/8

2007-02-08T13:29:00.000-10:00

2/8 – Meeting with Referentia

Aaron wants reporting for: code issues, file metrics, coverage, commits.

Said to look at http://tsubaki/hackystat/controller?Key=S5ASQkXWuzPK&Page=analyses as a reference.

Create a confluence page.

Find a suitable App to model the ProjectStatus app after.

Ask Philip if I should be using telemetry for the trend lines or not.

First Post

2007-02-02T15:16:00.001-10:00

HAH!!