Forum OpenACS Q&A: Project-manager integration with logger
This is especially targetted at Lars and Peter, the original authors of logger.
Looking at the logger code, it's pretty clear that you were planning on integrating with project-manager once it is written:
"Once we have a dedicated project management package for OpenACS this table will be superseeded by tables in that package. In order to make such a change easier in the future we are not referencing the logger_projects table directly in the logger datamodel but instead reference acs_objects."
What I'd like to do in the next phase of project-manager is add in estimated and actual hours worked on a given task.
This is important for a whole bunch of reasons, including getting project status, making gantt or other style charts, and so on.
However, I don't want to create an integration problem in the future. How do we make sure that the hours for project manager and logger co-exist peacefully, or preferably overlap?
I wouldn't mind storing the hour information in another place, like logger. However, if we're going to do so, then...
- Hours should be able to be logged at the task level rather than the project level. Or perhaps both.
I realize that the code allows for this, because the logger tables reference acs_objects. However, it clearly is not the intent of the code.
- We're also going to need to take out the logger_projects table and link it in to pm_projects. I'd appreciate some help with this, if anybody has time (especially Lars or Peter).
- How are we going to deal with adding in PERT charts? PERT charts don't just require estimated hours of work, they require having estimated minimum and maximum hour estimates.
- We're also going to need to work out how the packages depend on each other in the .info file. I'm not sure how both of our packages can rely on each other, but they will, right? Can we do that in the APM?
I'm dealing with some tight deadlines this month, and I'm trying to get as much of the project-manager as possible done by the end of the month. So I'd appreciate some thoughts and help with this.
Thanks!
Yes, integrating with your project-manager was the intent.
1) Logging for tasks: While not the intent of the current code, this is the right direction to move it in.
2) This is the right thing to do, but we're not going to have time until October when Heidelberg has launched :(
3) Add those minimum and maximum estimates? Or what would a PART chart look like if all the min and max estimates were the same? So long as we don't require of all users all the time that they have to explicitly enter min and max estimates, fine with me.
4) Yes, I was thinking about that ... a circular dependency. Bad.
One way to resolve it would be to split the PM package at some level. Maybe PM is a data model and API package, on which logger can depend, and then there's a second package for PM-user-interface? We'll have to think about this a bit more.
/Lars
1) Wouldn't it make sense to link logger with bug-tracker as well. More to the point, is the project manager going to have it's own way to store tasks or will it reuse bug-tracker ?
2) Dependency. Is there any reason why logger has to rely on the PM? Why not leave logger as it is at the moment, but add code that allows it to check (mounted) PM instances as well?
If there are circular dependencies, I'd favour to integrate both packages. Because, there is no need to keep them seperate (if you have to install both anyway). And the fact that Collaboraid has written logger while Jade is writing the PM does not count as a reason :).
Re the circular dependency, I think integrating the two packages would probably be just fine, as long as the UI doesn't get too complex, i.e., if you don't care about tracking against tasks, we shouldn't force you to (even though there may be a dummy task in the DB).
/Lars
exciting to hear that you are working on a project manager!
Yes, it does seem like we'll have two way dependencies between the three packages project-manager, bug-tracker, and logger. I'm not sure this means we should merge them into one package though.
It sounds to me like the hour estimates belong in the bug-tracker where the tasks live.
Jade - are you going to commit your project manager code to the contrib CVS folder or somewhere else where we can see it? I think that would help feedback a lot.
1) Logging for tasks: glad to hear that this is in line with your vision.
2) Taking out logger_projects table and linking it to pm_projects: we'll have to see who can get to this first. I may end up doing it.
3) Good idea about minimum and maximum hour estimates. Making them equal to each other is a good idea. Yes, of course, using these is optional, and will probably be specified by a parameter.
4) Proposed solutions to circular dependency with packages:
a) split PM into data model and UI packages
b) integrate the two packages. Lars made a good point about retaining the flexiblility of both packages if we integrate them.
5) Malte: Linking logger with bug-tracker. Perhaps a more general question would be: is there a way logger could be extended so that it could act like a service, and any package could log hours? That's not a question for me to answer, I guess. My needs with project-manager are pretty immediate, however, so I can't wait for that to be developed.
6) Malte: Is project manager storing tasks its own way, or using bugtracker: I looked a lot at bug-tracker, and it doesn't really fit well with the project-manager domain. However, using workflow is definitely on the agenda, and I'll probably plunder heavily from bug-tracker when I put workflow in.
7) Malte questioned the need to rely on PM. You could allow it to check mounted PM instances instead of just relying on PM. However, someone will need to write scripts to transfer the projects back and forth. Not tough, just something to think about. I think this might be a good idea.
8) Peter said that he thought the hour estimates belong in the bug-tracker, where the tasks live. Since I'm not actually using bug-tracker, I'll take that to mean you think they belong in with the Tasks. This makes sense to me as well.
The question that remains for me is if the hours associated with the Task should be in the task as well. It should be an easy matter to compare real versus estimated hours. Many graphs and charts do this, for example.
If we keep the data in logger, then we need to make sure it can efficiently be accessed.
I'll need to do some more thinking about this as well.
9) Peter, the code has been in /contrib for a while now. Information on the project-manager, including links to the CVS directory via cvs.openacs.org, are at:
1) Merge logger into project-manager. Allow logging with projects, and perhaps the option to not do so. I'll definitely need some help with this.
2) Keep logger as a separate package, and perhaps write upgrade scripts to allow people to move over if anyone had the need for this?
2a) Project manager could be totally separate from logger.
2b) Project manager could try to integrate in some way with logger, by calling a new API or something.
There are some big disadvantages to keeping them separate, but I can see why some would want that.
It's also some work to merge them.
Project manager is getting pretty close to being able to be released as a beta. The last major bit of coding that needs to be done is to integrate it with logger in some fashion. (I also need to fix notifications, but that's pretty minor). And then there is a lot of cleanup to do. Anyway...
This is a pretty long post, but it describes integration between logger and project-manager. I'm directing this mostly to Lars and Peter, but comments from others are welcome!
-------------
Logger currently contains these tables:
---------------------------------
* logger_projects
These are the projects that one logs time to. Logger was designed to eventually replace this table with projects from project-manager.
However, some people (like Malte in this thread) have suggested making that optional, so that logger could also exist on its own.
Q: do we integrate them more closely, or allow logger to exist independently. Right now, I'm leaning towards letting it exist independently.
* logger_project_pkg_map
Allows the projects to be mounted in multiple package instances.
* logger_variables
Hours, or time, the units tracked within the projects.
* logger_project_var_map
Keeps track of which variables are tracked within each project.
* logger_projections
Projections are used to compare the variables logged versus what is expected. They have a variable, a start date, end date, and expected value. This isn't exposed much in the UI, however, so I'm not sure exactly how it is supposed to be used. Lars suggested using this for projected task completion times. I'm not sure exactly how that will work, for reasons that I detail later.
* logger_entries
This is where the time or expenses are actually stored. A timestamp, variable, and value.
Project Manager contains these tables that relate to logger:
--------------------------------------------------
* pm_projects
Inherits from cr_revisions, and keeps track of all the information of a project. Tables which inherit from PM projects should inherit from cr_items.
* pm_tasks
Inherits from cr_items. A task associated with a project.
* pm_tasks_revisions
Inherits from cr_revisions. The revisions of a task. Currently stores estimated completion time for tasks, and the total amount of time worked on each task.
Potential paths to integrating logger and project-manager
-------------------------------------------------
1) Replace the logger_projects table with the pm_projects table.
I would need to write (and test) upgrade scripts, and this would have the disadvantage of requiring both logger and project_manager to be installed. It would also get a little complicated, because both packages would depend on each other, creating a circular reference in the APM. Lars and I talked about this a little bit, and this could be resolved by splitting project-manager into two packages or possibly some other mechanism. This sounds like a lot of work, and I'm leaning away from this, partially because I only have 3-4 days to spend on this effort, and if it exceeds this, I may just have to postpone the integration.
2) Write wrappers around logger_projects in project-manager, that create both the logger_project and pm_project, and relate the two. logger_project has a foreign key reference on acs_objects. This would look something like this:
In pm::project::new
create pm_project
call logger::project::new with the same object_id
Issues: since logger::project::new creates a new object, I'm not sure they could share the same object_id. If they could, that would simplify things, because it would eliminate the need for a mapping table. If that doesn't work, then we could either create a mapping table, or change the logger::project::new function to only create a new object_id if it does not already exist. It may already do that -- I'll need to check that out.
This would mean that you can create extra (logger) projects in logger, and they wouldn't show up in project-manager. However, any project you create in project manager would show up in logger. Perhaps one solution to this would be for logger to check if project manager were installed, and if so, to redirect or suggest to the administrator that they need to do it in project-manager instead.
3) Each project-manager Project has a reference to a logger Project (add logger_project as a foreign key to the pm_projects table)
In pm::project::new
set logger_project [logger::project::new ...]
create pm_project with logger_project = $logger_project
This is a pretty easy solution, actually. It also allows us to directly get the logger_entries by using the value stories in pm_projects. That would be fairly convenient.
Note that in all we've talked about so far, logger_entries are logged against a project instead of a task. Currently, the project-manager data model logs all time against tasks.
What is the alternative? Well, we could say project-manager Tasks are equivalent to logger Projects.
Log to Projects versus Log to Tasks
------------------------------
What are the advantages of each?
There are several populations we're trying to serve, both of whom have slightly different interests.
Workers typically would want to log their time against Tasks, because that makes more sense. You do work on Something. However, if you don't have a Task to log against, I guess you would log against the Project.
Managers more typically would want to see the time logged per project.
The best solution would be to do both: log time against tasks, but allow managers to be able to see it broken down by project. Or to log time against the task if you're working with the task, but allow it the other way as well.
Alternate solutions include allowing logger projects to map to either one, or to only link logger projects to project-manager projects, but to have a mapping table that keeps track of what task logged a particular logger entry. That last solution may actually be the best of both world.
It seems to me that the best solution is this: create a pm_task_logger_proj_map table. This table is only used when the user logs time against a task. In that case, the time is logged, and an entry is added into pm_task_logger_proj_map, which contains the entry, and the task.
Then, when you are looking at a task, the UI pulls up all the hours logged against that task, and allows you to log more hours against that task.
However, when you're looking at a project, it pulls up all hours logged against that project.
Lars or Peter: any thoughts on this?
Storage of logged entries
---------------------
The basic idea is to move the data storage to logger, and out of project-manager. PM's pm_task_revisions table will keep the same entries, but they will just be denormalized from logger -- logger will have the definitive values. That will allow queries to be fast, but will slow down inserts. Probably not a big deal.
Projections
---------
The logger_projection table is not very exposed in the logger UI, so I'm not totally confident of how I'm going to use it.
I'm not totally sure how I would use logger_projections at this point, because they also map against projects, at least in how the tables are names. Theoretically, they could be used against Tasks, because they are a foreign key reference on acs_objects. Lars actually suggested this. Lars, is that what you would like me to do?
If we did use the logger_projections, then I'm not quite sure what to do with the start_time and end_time. I guess I would update those any time I got the chance. But with project-manager tasks, there no strict start_time and end_time. Well, there can be an end_time if a deadline is specified, but otherwise, these things are computed based on other tasks and based on today's date. So they would be changing all the time. I'm not sure if it would be important to keep the projections up to date as well.
Conclusion
---------
Well, hopefully soon we'll have this mess behind us. Right now, I'm thinking the best solution is as follows:
1) Add a column to pm_projects, which tracks which logger_project it is associated with.
2) Change the related project-manager functions to handle ::new and ::delete appropriately.
3) Add in the UI to the appropriate pages in project-manager to add in new logger_entries, and show logger_entries, and edit logger_entries. (Lars has spelled out in detail to me how to do this portion of it -- thank you, Lars).
When a logger entry is made from a task, associate it with that task via a new table pm_task_logger_proj_map.
4) Every pm_tasks will also keep track of an associated logger_projection, which tracks the estimated amount of work for that projection. I'm not sure if this allows for any sort of meaningful use of the logger_projection table, however. How would you compare the hours logged versus the hours projected, except manually?
I'd appreciate any comments on this. Sorry it's so exhaustive -- integration sucks and is often complicated. But at least it's not as bad as migrations are :)
Integration:
1) Added in a table pm_task_logger_proj_map
2) Added in a column to pm_projects: logger_project. This is a link to
the logger project.
3) Copied files from /logger/www/lib to project manager. Customized
them so that they would show the right context bar, and so that
links would be correct. A better solution would be to change logger
so that it can offer <includes> that have correct links for
packages that include it.
Note: the pm_projects table denormalizes the values it gets from
logger, and I haven't made any code changes to logger. So in the
following situation, there is a bug: a) create a project in
project-manager. b) go to logger, and associate the project with
that instance. c) log hours from logger d) look at the project in
project-manager. Until the project is edited, or a task under the
project is edited, or another log entry is made from
project-manager, the Hours completed will not include the hours
entered from logger.
4) Modified pm::project::new proc so that creating a new
project-manager project also creates a logger project, and so that
that ID is stored in the pm_projects table.
5) Added an include in the www/one.adp that shows last 30 days of
logger entries for a project. Links to add in new entries, to edit
entries, and to delete entries. ***Deleting entries does not work
right now.***
6) Added logged time to task-one page. Use pm_task_logger_proj_map table.
Show logged time on the task-one page.
Note: People can log as many hours as they want to projects, and
that doesn't affect the hours completed statistic for the
project. The hours completed statistic is based on hours project
and hours logged on TASKS.
7) Edited log.tcl and log.adp files so that edits made from tasks are
linked to tasks. They also update the denormalized tables in
pm_tasks_revisions, and pm_projects.
8) Modified task-add-edit so that you can add an entry while editing a
task.
9) Changed entries-table to show tasks instead of projects (you know
projects based on the context).
