Forum OpenACS Q&A: SQL-Modelling Costs and Invoices

Hi,

pretty boring stuff for most of you probably, but quite exciting if you're running your own company...

We have started working on extending the existing Invoicing module or Project/Open (http://www.project-open.com/product/modules/finance/) to include Quotes, Purchase Orders and Provider Bills. Also, we want to integrate this with travel costs and timesheet management to truely capture all costs of a project in order to calculate profit and loss on a per-project base.

Below please find the SQL definition of "im_invoices" as it is being used right now. Further below you can find a proposal for "im_costs" to be used as the base class for "im_invoices" in the future.

What do you think about the approach with "im_costs" as a superclass? I guess there are some very experienced database designers amongst the members of the OpenACS community...

Frank

mailto:frank_dot_bergmann_at_project_dash_open_dot_com
http://www.project-open.com/

create table im_invoices (
        invoice_id              integer
                                constraint im_invoices_pk
                                primary key
                                constraint im_invoices_id_fk
                                references acs_objects,
                                -- who should pay?
        customer_id            not null
                                constraint im_invoices_customer
                                references im_customers,
                                -- who get paid?
        provider_id            not null
                                constraint im_invoices_provider
                                references im_customers,
        customer_contact_id    integer
                                constraint im_invoices_contact
                                references users,
        invoice_nr              varchar(40)
                                constraint im_invoices_nr_un unique,
        invoice_date            date,
        due_date                date,
        invoice_currency        char(3)
                                constraint im_invoices_currency
                                references currency_codes(ISO),
        invoice_template_id    integer
                                constraint im_invoices_template
                                references im_categories,
        invoice_status_id      not null
                                constraint im_invoices_status
                                references im_categories,
        invoice_type_id        not null
                                constraint im_invoices_type
                                references im_categories,
        payment_method_id      integer
                                constraint im_invoices_payment
                                references im_categories,
        payment_days            integer,
        vat                    number,
        tax                    number,
        note                    varchar(4000)
);

-- Costs is the superclass for all financial items such as
-- Invoices, Quotes, Purchase Orders, Bills (from providers),
-- Travel Costs, Payroll Costs, Fixed Costs, Amortization Costs,
-- etc. in order to allow for simple SQL queries revealing the
-- financial status of a company.
--
-- Costs are also used for controlling, namely by assigning costs
-- to projects, customers and cost centers in order to allow for
-- (more or less) accurate profit & loss calculation.
-- This assignment sometimes requires to split a large cost item
-- into several smaller items in order to assign them more
-- accurately to project, customers or cost centers ("redistribution").

create table im_costs (
        cost_id                integer
                                constraint im_costs_pk
                                primary key,
                                constraint im_costs_item_fk
                                references acs_objects,
        name                    varchar(400),
        cost_status_id          integer
                                constraint im_costs_status_fk
                                references im_categories,
        cost_type_id            integer
                                constraint im_costs_type_fk
                                references im_categories,
        project_id              integer
                                constraint im_costs_project_fk
                                references im_projects,
        customer_id            integer
                                constraint im_costs_customer_fk
                                references im_customers,
        asset_id                integer
                                constraint im_costs_asset_fk
                                references im_assets,
        due_date                date,
        payment_date            date,
        amount                  number(12,3),
        currency                char(3) references currency_codes(iso),
        -- variable or fixed costs?
        variable_type_id        integer
                                constraint im_cost_variable_fk
                                references im_categories,
        -- cost has been split into several small cost items?
        redistributed_p        char(1)
                                constraint im_costs_var_ck
                                check redistributed_p in ('t','f'),
        -- points to its parent if the parent was "distributed"
        parent_cost_id          integer
                                constraint im_costs_parent_fk
                                references im_costs,
        -- "real cost", "planning" of "quote or purchase order"
        planning_type_id        integer
                                constraint im_costs_planning_type_fk
                                references im_categories,
        description            varchar(4000),
);

Collapse
Posted by Randy O'Meara on
Frank,

I don't know much about this subject, but I'm glad to see this discussion. And I'm looking forward to responses from others that are expert in this area.

/R

Collapse
Posted by Torben Brosten on
Frank,

It looks much simpler than how the data is abstracted at OpenForBiz[1], which would make for a more manageable system --at least from this perspective. =)

Have you considered supply chain management/operations from the various perspectives?

What are the performance considerations?

Since accounting is all about summaries, it seems that a practical approach would be to use summary tables with calculation/data-entry time-marks to allow a scheduled process to periodically update reports etc for scalability. You can think of this as a slow, large spreadsheet calculation, where recalcs are done periodically or on demand.

Can anyone comment on what approach(es) would scale best?

Speaking of which, creating a "thin client" based on Lars' Skinny-Table spec[2] may be a fast way of developing here as it could be immediately applicable to various accounting, ecommerce and bookkeeping packages, and extended to meet specific specialties. In regard to developing the thin client, I've been considering how to use ad_form to implement a gui similar to sql-ledger's[3] invoicing, where there are update and posting features for various table rows represented on one page --but that's another thread.

1. https://ofbiz.dev.java.net/servlets/ProjectDocumentList?folderID=236&expandFolder=236&folderID=236

2. http://www.collaboraid.biz/developer/

3. http://sql-ledger.org

Collapse
Posted by Frank Bergmann on
Hi Torben,

thank for your link, I'll definitely have a look at it.

Concerning performance: I'm not too concerned with that, because Oracle/Postgres will keep the first 10 million entries or so in RAM... (Maybe I should move the "description" out to a different table...).

My idea is to go along the lines of the new SAP philosophy and to do most reporting with a data ("business") warehouse.

<blockquote> Have you considered supply chain management/operations
</blockquote>

I'm going to go for small and medium _project_oriented_ organizations, so there won't be no modules to deal with physical goods moving. Instead, our focus is on human resources, project collaboration and KM as the "supply chain of the knowledge economy"...

Also, we don't aim to do "accounting". We want to leave all double-entry stuff to external programs, because they are cheap and they handle the country specific legal and other conventions.

Instead, we want to do "controlling", that means calculating profit and loss on a per-project or per-customer base. The point is that there are already modules for timesheet, payroll, invoices and travel costs (from the old 3.4 intranet and other past work). All these modules gather some kind of cost, which "just" needs to be integrated in order to get all costs incured by a project.

And maybe the most elegant solution for that is a common superclass that is (relatively) easy to sum...

Bests,
Frank

Collapse
Posted by Torben Brosten on

Okay, Frank. If performance and supply chain is not an issue, you might want to review sql-ledger's invoice and related tables as well as perhaps the whole system. SL seems to be the accounting complement of your application as I understand it. Also, it uses postgresql (with perl), so you can actually run it with openacs on the same server. Your application could pull data directly from the operational tables.

"..there are already modules for timesheet, payroll, invoices and travel costs (from the old 3.4 intranet and other past work)."

Really? The only intranet module I am aware of has no docs, so I am ignorant of any applicability to these efforts. =(

Where might I find more info?

'we want to do "controlling", that means calculating profit and loss on a per-project or per-customer base'

How would this package differentiate itself from accounting packages that handle data entry and relevant reports with customer and project summaries?

hmm.. maybe I can answer my own question. Historical, statistical analysis. Still, wouldn't it be easier to just add it to the accounting system used then? or will there be a data import feature? I digress.

Hope this helps,

Torben

Collapse
Posted by Malte Sussdorff on
This might be a little bit brutal, but why don't you just use the SQL Ledger Tables and port the functionality of it into an OpenACS Web based package. After all, it is reknown to be a very good accounting software *and* this would prevent inventing the wheel over and over again. Last but not least, if you go down that road, you might even be able to get some cross support from the developers.

Alternatively, just install SQL-Ledger on AOLserver along with your normal OpenACS installation and let your custom code write accounting information directly into SQL-Ledger by just porting from PERL to TCL the necessary ::create and ::get API.

Either way, I know it might be tempting to start with your own Accounting System, but give it some thought if it the best you can do with your development power, taking into account excellent software like SQL-Ledger.

Collapse
Posted by Frank Bergmann on
Hi Torben,

<blockquote> review sql-ledger's invoice and related tables as well
as perhaps the whole system
</blockquote>

This is what I'm doing right now. Also, the OfBiz data model looks cool. Loads of stuff. No way Project/Open could compete with that without the underlying OpenACS platform ... 😊

<blockquote>    'we want to do "controlling", that means calculating profit and
    loss on a per-project or per-customer base'

How would this package differentiate itself from
accounting packages that handle data entry and relevant
reports with customer and project summaries?
</blockquote>

Your're right, both is about sums. But the "accounting sums" have to comply with the famous GAP in the US and EU accounting right here. Let's say you buy a server for example. Accounting wise, this "investment" is "amortized" over 3 years. It's not a "cost" when you buy it. But every year you get a third if its price as cost in your books. This is done using "double entry", that means, the amortization cost is subtracted from the value of the computer (an "asset" that appears in your balance sheet) and added to the "amortization costs account".

Controlling wise, you don't care about amortization (well, not that much...). You add the cost of the server 50% to Project A and 50% to Project B (the server hosts the applications of both projects). Also, you add a percentage of the salary cost of the SysAdmin to both projects, according to timesheet information.

Important: Accounting doesn't know about projects and cost centers. That's a bit exaggerated because you can tweak accounting to include projects, but it's the basic idea. The summary is:

=> Accounting is boring! (get an accountant)
=> Controlling is sexy! (know what's left in your pocket)

Does this explain the difference? 😊

<blockquote>    "..there are already modules for timesheet, payroll, invoices and
    travel costs (from the old 3.4 intranet and other past work)."

Really? The only intranet module I am aware of has no
docs, so I am ignorant of any applicability to these
efforts. =(
</blockquote>

Timesheet was already part of the 3.4 (Philips own...) Intranet. It's life at http://ptdemo.dnsalias.com/. "Payroll" is a huge name for the "im_employees" table in the 3.4 Intranet. It contains an employee hierarchy and salary information. This cost needs to be "redistributed" to projects according to the timesheet information. Travel costs were added to the 3.4 Intranet at www.Competitiveness.com and I've got the kind permission to use the code. And "Fixed Costs" (such as office rent, aDSL line, ...) finally are just plain cost items that are distributed to projects according to the guts of the manager or maybe according to the hours spent on every project.

Bests,
Frank

Collapse
Posted by Frank Bergmann on
Hi Malte,

I think our posts crossed.

Let me reasure you that we're not going to go for a Yet Another Accounting System. Instead, P/O captures costs over the web and allows for the "controlling" stuff (redistributing costs to projects and customers) that is needed by project-oriented organizations (and not provided by SQL-ledger).

The idea is to export these items afterwards to SQL-ledger or similar packages (Quickbook, Conta-Plus, KHK-Kaufmann, ...) to facilitate the creation of balance sheets and income statements are required legally.

I'm already looking at the OfBiz and SQL-Ledger data models. It would be really cool to run OpenACS and SQL-Ledger on the same server...

Frank

Collapse
Posted by Torben Brosten on
Thanks, Frank, for the acs intranet package description and references.

By the way, sql-ledger can summarize transactions and income on cash and accrual basis as of version 2.4.0[1]

1. http://sql-ledger.org/cgi-bin/nav.pl?page=news.html&title=What's%20New

Collapse
Posted by Frank Bergmann on
Hi Torben,

checked SQL-Ledger a bit more in detail. Quite impressive, particularly the user community from a zillion different countries. However, they're using relatively complex accounting schemes which is a huge overkill for service-oriented companies...

I've seen your post on their mailing list. Are you sure you want to do the accounting of your consulting business with it? I'm running the company of my wife (about 1000 "projects" and invoices per year...) from an Excel sheet, including income statement, profit & loss and balance...

Frank

Collapse
Posted by Torben Brosten on
Hi Frank,

<blockquote> ..Are you sure you want to do the accounting of your
consulting business with it?
</blockquote>

Yes, for this and other "entities."

<blockquote> they're using relatively complex accounting schemes
which is a huge overkill for service-oriented companies...
</blockquote>

There's a general coa for service companies which automatically sets up assemblies as labor only.

Yes, its complex, but really there's a spreadsheet like simplicity to how it's built, which I believe helps make it so flexible.

I know many entrepreneurs and small organizations that are adversely affected by growth. Openacs scales for computing, and could provide a way to manage growth in other ways.

The plan is to do what Malte suggests above, namely to re-build sql-ledger as openacs packages that integrate with other packages. There will need to be some changes. The openacs kernel already provides many of the administrative processes, such as permissions, internationalization, and templating. Also, SL is built for flexibility over scalability, which is apparent since SL depends on perl and DBI which is not thread safe --not okay when operations get busy!

cheers,

Torben

Collapse
Posted by Nagita Karunaratne on
Maybe I'm a little naive but is it safe to keep your financial transaction data on a web server?

When you purchase a product/service from some organization isn't that information considered confidential?

Collapse
Posted by Torben Brosten on
A server can run in a LAN environment. and there are methods to reduce risks if doing it differently.
Collapse
Posted by Ben Koot on
Hi folks.

What if we would use logger as a kind of financial logbook. Anybody, without accounting knowledge, could log transaction details (we might need some additional (free format)input fields and the categories module, in the default logger entry page) in a matter of seconds, and it's already project specific. Logger allows for cvs export so that could be used to tie into SL. Or am I missing something?

I am trying to create a dead simple administration package for small business owners where 3rd party admin folks would take care of the "heavy" stuff, but the small business owner takes care of the input of stupid details, and create a birds eye view of his/her financial status. Some of the basic logger features already cover this.

As far as security, I remember Quickbooks used to a have a web interface that allowed you to upload financial info into your dedicated stand alone quick book installation, so I can't imagine that can be to complicated to set up.
That leaves us with one, but critical issue to solve; invoicing. I assume we could use CSS to create (default) invoicing templates. Personaly I feel the problem is not the accounting specific issues, at least in my admin for dumies model, but the ease of logging transactions for further prossessing that ought to be the focus, so this idea maybe oversimplyfying the concept Frank has in mind.

Just a thought

Cheers
Ben

P.s I am negoting funding for the project with  http://www.bom.nl/uk/index.html and am looking for a developper to build the concept. The goal is to use OACS as a hosted application businesstarter kit, providing Businessplanner software, corporate website, and admin centre. I have set a target for 150 - 200 startup businesses annualy, subsidized by the Dutch governement, Users pay a monthly fee for the infrastructure. Project is part of my involvement of http://www.seniorstart.org/showdocument.php?id=97&PHPSESSID=74fdd29c554ee585647bbe111ab1181che

Right now I lack insight into the financial inmplications to transform OACS in such a scenario, so any creative ideas are welcome. If you like to paticipate in the development, let me know.

Collapse
Posted by Jade Rubick on
I think that even for a small company, you want to use a real accounting package.
Collapse
Posted by Jarkko Laine on

Nagita,

The data doesn't sit on the web server, it's on the db server which doesn't have to be (and often isn't) the same box. Moreover, the db server is almost always behind a firewall. It's true that in this case you could access the data through the web server if you could crack into it. But like Torben said, if you keep it in your intranet behind a fw and not visible to the world, I don't see why it would be any less secure than any other software that works over the network (that is, almost every piece of software nowadays).

Frank,

I've been taught to call the thing you call controlling management accounting (as opposed to financial accounting). So I guess it's something like boring sex, which IMNSHO doesn't as a definition hit too far off the target. ;o)

The most important difference between the two is that in MA you don't have to do everything by the (varying) law, like four years amortization for this and three years for that. You can thus calculate the real costs and allocate them accurately.

As for the server case, I would be interested in amortization even in the realm of controlling (or MA). What if you use the same server for say ten later projects, too? The costs would be divided to the two earlier projects and if the cost is high, the profitability of the two would be largely skewed in contrast to the others, which also use the same server but don't defray any of its costs.

So I don't agree with the fact that the allocation of costs is easier in controlling, in fact it's one of the infamous problems of management accounting. But it's true that you have more leeway to fiddle with the allocation than in financial accounting.

Anyway, this is an interesting and worthwhile discussion 😊

Collapse
Posted by Frank Bergmann on
Great pieces!

Jarkko, you are right, assets need to be amortized. I think the subject has got much clearer after your explanation.

Ben, the idea of the logger is great. However, the entry of costs is going to be very application specific in order to allow for a great degree of "self services":
- Invoices can be posted by providers over the web
- Travel costs are logged by the employees themselves
- Payroll costs are distributed to projects according to the timesheet of the person.
...

So, we will need several different types of "loggers" and particular emphasis on permissions and security. This is why we are going for specific modules, apart from the fact that the code is already 70% there from previous projects.

<blockquote> dead simple
</blockquote>

(Financial) accounting is the contrary of "dead simple", because the accountant has to take a lot of decisions knowing the law (classify costs and everything else in 300+ "Chart of Accounts").

Actually, we need to make things "dead simple". The folks working in translation agencies (our main target market) are not very technically not very sophisticated, so the system needs to work out-of-the-box. Maybe there is some synergy between your business-starter-kit and P/O. Have a look at http://www.project-open.com/product/modules/finance and check the "Travel Costs Package" for example. A simplified version of it being used productively at Competitiveness.com during the last 3 years from everybody in the company.

<blockquote> re-build sql-ledger as openacs packages
</blockquote>

That sounds very, very interesting. I see our P/O functionality basicly as a "front-end" for something like SQL-Ledger, and an integration with it on the same server could solve several problems for us. Torben, did you already start actually working on it? Are you on Skype (my ID is frankxbergmann)?

Bests,
Frank

mailto:frank_dot_bergmann_at_project_dash_open_dot_com
http://www.project-open.com/
http://www.project-open.org/

Collapse
Posted by Torben Brosten on
Hi Frank,

Sorry, no skype here. I think discussing these topics via forums may be more productive as there's that automatic creation of a knowledgebase and others can pipe in.

Currently I am building resources and experience for starting this project. Less than 2% of the collected resources and notes are online. I need to familiarize myself with openacs package development, sql-ledger programming style, and spreadsheet calculation models (equation dependency hierarchies etc.).  Currently, there is a project framework consisting of some forums[1] so as to not harass openacs.org forums with OT discussion, incidental notes and incomplete thought processes.  The plan is to use Jade's project management package for implementation, and arch for development.

At this point I see these technical hurdles, mainly gui related.

a. making ad_form work with multiple db rows and maybe tables
b. creating a thin-client to handle basic editing of user tables to quickly develop a set of integerated service packages

I have to justify the development internally by upgrading the ecommerce package and adding features.  There's already a consensus, I think, to breaking ecommerce into smaller service packages and upgrading the code. For example, ad_form isn't used in it. Search for ecommerce in the forums and you can see a collection of related ideas and works from the openacs community.

Okay, I'm rambling now..

cheers,

Torben
AIM/YM/openacs irc: tekbasse

1. http://bitscafe.com/pub2/forums

Collapse
Posted by Frank Bergmann on
Hi Torben,

just checked the SQL-Ledger data model. Doesn't look vey different from the Project/Open model. But that's probably more an problem then a feature when trying to bring the two systems together:

- I definitely want to convert all major business objects to OpenACS objects: Invoice, project, customer, employee, vendor, department and warehouse.

- How to deal with duplicate objects? For example, P/O already knows about customers and invoices. OpenACS knows about users, addresses, translations and permissions. Which application should be used to maintain the information? Can triggers do the job to copy data between the tables and to maintain consistency?

I'm really not sure what to do: Incorporation (integrating the SQL-Ledger data model into OpenACS/Project/Open) or Integration (keeping applications separate and providing an export functionality from OpenACS/Project/Open to SQL-Ledger). Maybe we should honestly question if we are going to be able to port about half of the SL functionality to OpenACS (converting tables to OpenACS objects). In this case I would go for Incorporation and commit about one man-month of senior development time. Otherwise I would go for integration, which means using the Perl-GUI.

Are you available by phone? My mobile is: +34 609 953 751. Is there somebody else who would like to participate in one way or another? Malte?

Bests,
Frank


acc_trans    | table    | sql-ledger
ap            | table    | sql-ledger
ar            | table    | sql-ledger
assembly      | table    | sql-ledger
audittrail    | table    | sql-ledger
business      | table    | sql-ledger
chart        | table    | sql-ledger
customer      | table    | sql-ledger
customertax  | table    | sql-ledger
defaults      | table    | sql-ledger
department    | table    | sql-ledger
dpt_trans    | table    | sql-ledger
employee      | table    | sql-ledger
exchangerate  | table    | sql-ledger
gifi          | table    | sql-ledger
gl            | table    | sql-ledger
id            | sequence | sql-ledger
inventory    | table    | sql-ledger
invoice      | table    | sql-ledger
invoiceid    | sequence | sql-ledger
language      | table    | sql-ledger
makemodel    | table    | sql-ledger
oe            | table    | sql-ledger
orderitems    | table    | sql-ledger
orderitemsid  | sequence | sql-ledger
parts        | table    | sql-ledger
partscustomer | table    | sql-ledger
partsgroup    | table    | sql-ledger
partstax      | table    | sql-ledger
partsvendor  | table    | sql-ledger
pricegroup    | table    | sql-ledger
project      | table    | sql-ledger
shipto        | table    | sql-ledger
sic          | table    | sql-ledger
status        | table    | sql-ledger
tax          | table    | sql-ledger
translation  | table    | sql-ledger
vendor        | table    | sql-ledger
vendortax    | table    | sql-ledger
warehouse    | table    | sql-ledger
yearend      | table    | sql-ledger

Collapse
Posted by Malte Sussdorff on
I just got SQL-Ledger installed this morning, fixed a bug in the Lexware import script, got the German DATEV-SK03 running on AOLserver and am happy :). As Project Open is not available for Postgres yet, obviously there is not much of integration effort happening from my side, but I'm curious about the results and would be willing to help out later this year if someone finds the funding for it.

For my personal use, I'm happy as it stands now.

Collapse
Posted by Torben Brosten on
Hi Frank,

I'm just keeping the two systems separate for now, importing data from one to the other as needed.  I won't start developing an openacs version for at least a few months.

good luck!

Torben

Collapse
Posted by Frank Bergmann on
Hi,

<blockquote> I won't start developing an openacs version for at
least a few months.
</blockquote>

Sounds ok, and very clear, that's good. So my decision is to go ahead with the "Web Frontend" (Project/Open...) for the various types of costs, keeping in mind an integration interface.

Torben, how did you export the data?

Bests,
Frank

Collapse
Posted by Torben Brosten on
export from/to?

Some data is coming from excel worksheets (such as 1 worksheet per sale), so have had to build tables. In another case, from filemakerpro, and in another case from postgresql using the psql COPY..TO  All cases have been formated as tab-delimited tables, and processed with the sql-ledger generic import tool linked from the SL conversion[1] page.  I'm working to manually import the rest using psql COPY..FROM (being careful to populate keys etc.)

1. http://sql-ledger.org/cgi-bin/nav.pl?page=misc/conversions.html&title=Conversions

cheers,

Torben