The VIBES ISP Management System
$Id: vibes.txt,v 1.5 2008/07/30 16:58:23 darcy Exp $
Chapter 1 == ISP Management Systems
The title of this book is "The VIBES ISP Management System", but this system is
actually a conglomeration of several systems designed to work together. These
systems manage the physical system accounts and files on the system, perform
billing functions, and keep accounting records.
The VIBES ISP Management system is a robust and flexible system which can
manage all facets of a UNIX based ISP. Most of the system's interfaces are
web browser based, and so can be accessed remotely from virtually any platform,
and are encrypted through transport layer security (TLS/SSL).
Billing System
This system keeps track of subscriptions, discovers when it is time to bill
each customer, calculates how much a customer is to be billed (for resources
used, or for arbitrary services defined by the ISP), and creates and sends
invoices. It then makes billing information available to an accounts
receivable system.
The billing system can also include various billing related subsystems, such
as a live credit card transaction system, or an automated pre-authorized bank
withdrawal system.
System Management
The system management system (if you'll forgive the systemic over frequent
use of the word "system") examines the billing accounts, and creates the
actual physical resources required by the clients. This would include, for
example, creating web directories, maintaining passwords, making mailboxes
and aliases, setting up DNS records, and more.
Much of the system management information is filled into account records by
administrators, but there is also a web based client interface provided
to allow users to set some preferences and activate certain features.
Accounting System
As mentioned previously, the billing system makes the information about who
has been billed available to an accounts receivable system. The accounts
receivable system summarizes and keeps detailed accounting records which can
be used for financial statements and making sure all accounts are balanced.
Work Order System
If your ISP handles installations or other external jobs, the VIBES ISP
Management system can provide a work order scheduling system. Work orders
can be generated at the time accounts are created, or manually via a
calendar based interface. Work orders are provided to technicians, and
track what has been done.
1.1 Requirements of VIBES
The VIBES ISP Management system requires the following software:
- UNIX: We prefer to work on the NetBSD http://www.netbsd.org platform,
however the billing system should work fine on any UNIX-based operating
system which supports the software listed below.
- Python: A high level interpreted programming language. Most of the system
is put together with Python. Python is easy to understand and modify,
making the system extremely flexible and easy to customize.
- PostgreSQL 7 A professional Open Source relational database engine.
PostgreSQL supports, for free, all of the required features of expensive
commercial database systems, along with specialization and extensions not
found in other packages.
- Apache 1.3 The Apache web server provides the platform for the web based
interface. PyGreSQL Provides the interface between python and the
PostgreSQL database server. Miscellaneous Other small common packages
may provide additional services, such as the GD graphics package.
If there were interest in the billing/accounting parts of the system to be
supported on the Microsoft Windows platform, a port to that system may be
possible. The above software is available for other platforms, but the
billing system has not been tested or specifically adapted to them.
Chapter 2 == Client Accounts
2.1 Account Structure Overview
There are three main levels which may comprise an account. This three level
system allows for a an enhanced level of billing flexibility. It allows
clients to be billed in several different ways for any number of resources.
2.1.1 Client Level: Personal Information
The client level includes the client's name and contact information. Other
billing information such as credit card details are attached to this level.
This level can be thought of as representing a person or other entity, and
the various details relating to that person.
2.1.2 Billing Group Level: Account Type and Billing Dates
A billing group represents in the most general sense what the client (client
level) is being billed for. The billing group specifies an "account type"
within the system, and various details regarding the account such as the
home directory where the accounts for this group are based. The billing
group is also the level which keeps track of the billing period and term
dates, so that the system knows when the client is to be billed. Clients
can be disconnected at the billing level by marking them inactive, or
setting a disconnect date, causing them not to be billed any longer.
A client may have multiple billing groups attached to them. A Billing Group
may have many accounts attached to it. A billing group, however, may be
attached to only one client (or else how would you know which one to bill?).
2.1.3 Account Level: System Resources
The account level usually represents an actual system account. This includes
some sort of login ID and password, as well as an indication as to what
services the account may have access to. An account level record may
represent an individual web site, a mailbox, or specify whether a shell or
dial-up can be accessed. The account level specifies several other parameters,
some of which are accessible to the user via the client's web interface for
customization of things such as email forwarding, and shell program.
2.1.4 Recurring Billing Level: Anything Else?
Although the system is centered around the previously described three levels,
there is another type of record which may be regarded as another level. A
recurring billing record is similar to a billing group, except that it has
no account type associated with it. In fact a recurring billing record has
hardly any information associated with it beyond the basics of a name and a
description, and how much and how often a certain client is going to be
billed. Recurring billing can be attached to clients to bill them any amount
for any arbitrary service on a regularly recurring basis.
2.2 Creating System Accounts
Accounts can be created by system administrators, or they can be created by
users applying to the system. Accounts created by a system administrator.
The latter group requires the activation by a system administrator before
they become live.
2.2.1 Group Edit Privileges
In each billing group there is usually one special account to which a "group
edit" flag is attached (sometimes referred to as a "Group Leader" account).
This account is given some additional authority over the other accounts in
its group. Any account with the "group edit" privilege flagged will be able
to view the settings of the other accounts in the group (in the web-based
user login area), as well as change most of them1 . A group leader account
also has the potential to access functions such as viewing the billing
history for the group, adding mailboxes, changing credit card information,
and so on. Non-group leader accounts in the billing group will not see or
have access to these functions.
It has been said that there is usually only one group leader account in a
billing group: this is because the first system account created in a billing
group is given this "group edit" flag by default, while subsequent accounts
that may be added are not given it by default. The flag can be toggled
manually by an administrator on the system account page. There can
potentially be more than one group leader account in a billing group, or
even no accounts in a billing group with this privilege.
2.3 Terminating System Accounts
Accounts can be terminated or suspended on any of the levels of the system.
Deactivating a client will cause all billing groups beneath that client to
cease being billed. Deactivating a billing group will cause the client (who
may remain active) to stop being billed for any accounts or services beneath
that billing group. Individual accounts may also be deactivated, and then
will not be included in the tallying of resources being used by the billing
group they are attached to (and the client will of course no longer be able
to use that account as long as it remains deactivated).
When clients want their account terminated, deactivation of billing groups
can also be scheduled by filling in a deactivation date in the billing
group record. Effective immediately, as soon as a deactivation date is
filled in, the client attached will no longer be billed for that billing
group; however, the billing group will remain active until the deactivation
date is reached. With these behaviors in mind, one should not fill in the
deactivation date on a billing record until the billing group is within it's
last period of service.
2.4 Account Types and Attributes
The term "account" is at risk of vagueness. Sometimes when the term is used
it may refer to accounting-related accounts, or in an abstract sense it may
refer to a client. The more specific term "system account" may refer to a
specific service (such as a login shell, or a web site).
When an "account type" is referred to in relationship to the billing system,
however, it usually represents a service (or group of services) which the
ISP offers to a client, and which is specified in the client's "billing
group" record. The reader should be aware that the term "account" may be
used in any of the above senses, and the meaning only indicated by the
context. Be careful.
2.4.1 Defining/Creating New System Account Types
An "account type" to the billing system is a template which names and
defines how much a service, or group of services, will cost the client,
and how often they will be billed. Account types store basic information
such as which G/L accounts will be applied, whether "system accounts"
created under a billing group of the given account type will be allowed
a web directory, a static IP address, or a virtual domain assigned, etc.
Along with these basic properties, account types will list a number of
"attributes" attached, and give the details of how the resources
represented by each attribute is to be billed for, and how much will be
charged if extra resources of the given type of attribute are exceeded
in a billing period or term; and finally, how the attribute will be treated
on an invoice.
2.4.2 Account Attributes
Account attributes control the quantity of a given resource which is
included in an account type, as well as the rate at which extra charges
will be applied. For example, an account attribute for web traffic might
specify that a certain account type allow one gigabyte of traffic per
period, and that extra megabytes beyond that limit within the period be
billed at 5 cents each. Account attributes can also specify special
billing modes, or turn off billing for certain attributes entirely.
2.5 Relevant Billing Dates
In addition to recording the exact creation moment at all levels of a
client's account, the billing system tracks various dates for billing
purposes. These significant dates can be found in the client's billing
group record.
2.5.1 Start Date
The Start Date is the date upon which a billing group was started to be
billed. This may be different from the creation time of the account. On
systems using work orders for installations, this date will be the date
the installation was scheduled for.
2.5.2 Due Date
The Due Date is the date on which the billing group's current "term" ends.
This date is broken down into three parts. The month and the year are
obvious in meaning, and what is termed the "anniversary" is the day of the
month. This "anniversary" day of the month is used for calculations
regarding the billing group's "period.""""" At the end of a billing group's
term, the client is charged the subscription fee for the next term, along
with any extra services which are billed on a term basis.
2.5.3 Period Date
The Period Date is the date on which the billing group's current period
started. This date consists of a year and a month. The exact day of the
month is controlled by the "anniversary" part of the billing group's "term"
due date. This allows for billing groups whose billing begins on the 31st
of a month (or the 28th of February on a leap year) to be handled correctly
during shorter months.
The period start date indicates the next earliest date on which the client
will be invoiced; this will occur exactly one month (assuming the account's
period is set for one month) after the current period's start date. At the
end of a period, extra charges for most service overages are calculated
and billed for. Other charges (account transactions) that may have
accumulated will be billed at this time also.
(Note: certain account transactions may be invoiced on the next billing
day, bypassing the period date calculations).
Chapter 3 == Billing: Following the Money
3.1 Keeping Track of Amounts Owed
3.1.1 Account Transactions: Finding Out What Is Owed
The billing process runs automatically every day. It searches for clients'
billing groups for whom it is time to bill. This means it searches for
clients for whom the current date is on or beyond the client's billing
record's "Period End" or "Due Date"1 .
For each active billing group identified as being eligible for billing,
all relevant resources used by that client are examined. If the resources
used are found to exceed the amount of resources included in the billing
group's account type, then the customer is charged the extra fees set for
that resource type in that account type. Some extra fees apply only at the
end of the "term" (for example, extra mailboxes) to be billed when the
client pays their usual account fees; certain other fees (for example,
charges for extra web site traffic) are applied every billing "period"
whether the client's term is yearly, monthly, or some other timespan.
Additionally, the billing process searches for billing groups which have
had their "disconnect date" set, and the current date is now beyond the
"disconnect date". Billing groups matching this criteria are marked
inactive, and no further billing is done for inactive billing groups. It
is important to note that no billing is ever done for any billing group
which has a disconnect date specified; so one should never set a disconnect
date on a billing group except within the last "period" before the billing
group is to be deactivated.
For each item found where the customer owes money on (whether it be their
end-of-term re-subscription fees, or extra charges at the end of a period),
an itemized "account transaction" is created in the database, otherwise
known as a "line item". These line items are used to then create invoices.
3.1.2 Recurring Billing: Special Cases
Recurring billing records are handled separately from the regular billing
process. The reason for this is that recurring billing items are not
defined as a part of any particular account type, but represent any
arbitrary service for which you want to bill a client periodically.
3.1.3 Invoicing: Letting the Customer Know They Owe
The process of creating invoices is very simple. It is a automatic process,
that you start with the click of a button.
Invoices are created by examining all of the "account transaction" records
created by the automated daily billing process. All of the account
transactions for each customer are grouped together, summed up, and
converted into an "invoice record." The invoice record is then sent to
an invoice printing processes and the invoice is formatted and sent to
the customer.
There are several ways that the customer may receive an invoice, which
may be specified in the client's account record. The most common is simply
via email. Some clients, however, prefer to have their invoices printed
and mailed to them. The invoice printing process examines the client's
account preference, and sends out emails to those who prefer email, and
sends the invoice to the printer (ready to be mailed) for those who prefer
a hard copy.
3.1.4 Posting Invoices: Letting Accounts Receivable Know They Are Owed
After the invoices have been created for the customer, the accounting
system still needs proper records to keep track of what is owed to the
company. This process is called "posting the invoices", and is as simple
as the invoice creation: one merely requests the invoices to be posted
with a single click on a web page.
The postings do not immediately appear in the A/R system, but are actually
posted to a special holding area. Another process is then required to pick
up the newly posted accounts receivable records and put them into the A/R
system. These processes are separated so that if an external A/R system
is used, external processes can be created to import the A/R records from
the holding area into the foreign system. For companies that use the A/R
system built into the billing system, a process runs periodically (every
15 minutes by default) which then imports the A/R records into the final
A/R database.
3.2 Receiving Payments
There are various methods by which clients may render payment.
3.2.1 Processing Cash and Cheques
3.2.2 Processing Pre-authorized Credit cards
3.2.3 Automated Credit Card Billing
3.2.4 Manual Credit Card billing
3.2.5 Processing Pre-authorized Bank Withdrawals
3.3 Other Billing and Accounting Activities
3.3.1 Journaling
It is normal practice to "journal" general ledger accounts periodically:
usually at the end/close of every month to finalize all the accounting
for that period. Once G/L transactions have been posted to G/L they are
never allowed to be changed in the database. If there are errors in
posted records (over payments, forgotten data, etc), adjustments must be
made to the general ledger in following periods.
Journaling the accounts receivable transactions is as simple as accessing
the A/R journal web page, and selecting the "journalize" function. The
administrator will then be asked what the last date to include in the
journal will be (usually the last date of the previous month), and all
A/R transactions up to that date which have not previously been journaled
will be processed into the next journal.
Closed journals include some simple sums and counts to verify that the
journaled transactions have not been tampered with, should they ever
come into question.
Journals may be reviewed at any time, of course. In this same area a list
of journals can be displayed, and any journal accessed. Each journal is
summarized in two ways when displayed:
- Transaction Type Summary: A list summing the total monetary amount of
transactions in each A/R category. The categories currently include:
Adjustment Transactions, Credit Card Transactions, Notes (Credit/Debit),
Pre-Authorized Withdrawals, Receipt Transactions, and Invoices.
- G/L Summary: A list of all G/L account numbers and the monetary sum of
transactions for each. The sums are divided into credit and debit
columns, and the sums of each column, which should always balance out
equally, are printed at the bottom.
Detailed listings of the A/R transactions for each G/L account in the
journal, or of a specific transaction type, can be printed by simply by
clicking on the G/L account number or the transaction type label. (Note
that these detailed listings are rendered somewhat crudely as plain text
due to many web browsers having difficulty rendering very large tables.)
3.3.2 Credit and Debit Notes
3.3.3 Reversing Payments (NSF)
Unfortunately there are times where a payment has been recorded in the
billing system, but the money never materializes and so the payment needs
to be reversed. This may happen due to a client's cheque bouncing (for
reasons of non-sufficient funds in the account), a pre-authorized withdrawal
being rejected, or a credit card charge being reversed. There may also be
other special circumstances where a payment needs reversing.
The VIBES ISP Management system provides the ability to reverse the payment
on any invoice. This interface is usually accessed from the client's record:
a link appears near the top which accesses the "Reverse Payment (NSF)"
function. When this link is followed a list will be shown of all the
payments on invoices in the client's billing history. When the invoice
payment to be reversed is located and selected, an input appears allowing
the administrator to write a note giving the reason the payment is being
reversed.
Many companies want to charge clients a "processing" fee for NSF/reversed
payments. Depending on the system configuration, there will appear a
check box that may be selected which will charge the client such a fee,
when the payment reversal is confirmed (ie. the final submission button
pressed). This fee (set in the system configuration) will be added as a
line item in the client's account transactions, and the client will be
invoiced for the fee at the earliest opportunity.
3.3.4 Reports and Details
3.4 Collections
When a customer owes a significant amount of money for a long period of
time, some special action may need to be taken. The billing system refers
to this action as putting someone into "collections": that is, the
"collections" flag is set on the client's record.
3.4.1 Collections Modes
A process runs nightly and examines all of the clients to see who needs
to be put into collections. There are two modes that this process may
run in: "manual" and "auto". What mode the billing system is in is
controlled by the "Collections" setting on the system information page.
- Manual: The collections flag on a qualifying client's records is not
set. An email report for each client in this category is generated to
the address specified in the system information as "collect_email"
(with the reply-to set to the system information "billing_email"
setting). Note that unless action is taken (ie. the customer's balance
reduced, or the customer's record "collections" flag checked) this
report will be generated again every time the collections process runs.
- Auto: The collections flag on the qualifying client's record is set
immediately, as well as the report as described in the "manual" mode
is generated.
3.4.2 Qualifying for Collections
A client is considered to qualify to be given collections status if they
owe more than a specified amount for 90 days or more. The minimum amount
the client must owe in order to quality is set as "Collections Min" in
the system information.
3.5 General Ledger Accounts
For accounting purposes, all transactions will end up in the company's
general ledger. For systems using The VIBES ISP Management System's A/R
system, all billing transactions will be imported into specified general
ledger accounts.
The default general ledger account, if none other is assigned to a
transaction is the account known as "misc sales", which can be defined
in the system settings.
All G/L accounts can be defined in the "Chart of Accounts" area. This
area will list all G/L accounts defined on the system, allow them to be
edited, and new ones added.
Chapter 4 == Procedures
4.1 Customizing the System
4.1.1 Look and Feel
The system's overall appearance can be altered globally. The entire top
and bottom portions of the pages displayed may be designed in any way
desired; this includes colours, fonts, and menus. The contents of each
page is then simply inserted in the "middle".
The method of defining the surrounding "look and feel" of the system
is achieved by inserting the relevant HTML code into a special file.
These customizations will likely be done when the billing system is
installed, but can be altered at any time.
Since the interface is all web based, all of the menus can be completely
customized. One is free to organize links and function into sub-menus
or present them on a single page as desired, whichever is most effective
for your organization's needs.
4.1.2 sysinfo: Configuration and Settings
The central place for setting and modifying the configuration of the
system is the "sysinfo" (System Information) page. This page lists all
of the parameters and settings in the system. This page is divided into
a few sections, as follows:
- Client: All information regarding the system's company, including the
billing address, client identification code, and contact information.
- System: Settings related to the physical system, including network
settings, paths to files, and so on.
- G/L: The definition of all of the system general ledger account numbers.
This controls how the billing and accounting systems categorize
transactions.
- General: All the other settings for the system, such as report email
addresses, limits and defaults for various functions of the system.
The SYSINFO entries, one per row, list the entry label (or code), a brief
description of the entry, the entry data type, and finally the actual
value. In order to edit a value the label must be selected, which will
send the browser to an edit page. There are a few sysinfo entries which
may not be edited, these will not provide a link via their label to
the edit page.
The data types are mostly straight forward, but some are specialized.
They are as follows:
- String: Any words, or text; this is the most basic data type
- Integer: Any numbers (decimals will usually be chopped off)
- Float: Numbers with decimal points
- G/L Account: A general ledger account number; the format for this is
two groups of numbers (a major and a minor) separated by a dash.
Eg: 51001-1006.
- Text: Text is treated the same as a string, however a multi-line input
box is provided for editing/inputing this item.
- Colon List: Several strings separated by a colon(s); where these appear
there will be special instructions given in the description or help text
when you edit the item, describing the format required.
- Comma List Several string separated by commas; this is the same as the
colon list except that the comma is used to separate items.
Chapter 5 == Security Interface
5.1 Administrative Access Controls
5.1.1 The Admin Flag
There are several methods of restricting access to parts of the VIBES ISP
Management System interface. The most basic is the 'admin' flag. Anyone
who does not have an admin flag is completely locked out of the system
administration interface.
The admin flag is found in a person's account record.
At the time of writing this, there is no web based interface to the admin
flag. You must have your database administrator grant the admin flag
to desired accounts.
5.1.2 Group Permissions: Finer Access Controls
All accounts which have the admin flag set have the possibility of
accessing the web based administration interface, but the admin flag
alone does not necessarily grant access. There is a secondary security
system which can grant access to certain pages to defined groups of
administrators, either with full access, or restricted to read-only access.
To define which admins have permission for what pages Access Groups must
be created. The access group interface allows the creation and editing of
access groups. When an access group has been created, the group members
may be modified.
A list of all accounts with the admin flag is listed when an access group
is looked at. Beside each name are two check boxes. One check box indicates
that the accompanying admin is a member of the current access group, and
the second check box must be checked if the admin is to have 'update
privileges' in this access group. Update privileges allow pages which
are assigned to a given access group to be modified, and new records
created. If the admin does not have the update privilege flag, they will
not be able to make any changes to pages in the particular access group.
An admin must have both the "member" and "update" check boxes checked to
have full access to pages under a given access group.
The way pages are assigned to access groups is on a 'script-by-script'
basis. The web browser calls certain files/programs to preform certain
functions. Each of these programs presents the interface for certain
functions; each program can be assigned to
an access group.
An administrator with access to these access permission pages must be
careful not to remove their own access from them, or they themselves
will be locked out.
Since each program can only belong to one access group, access groups
may need to be organized with some thought to yield the desired
security results.
5.2 Client Interface Access
Clients can access their own password protected user page. However the
first system account created under each billing group has a special
"group edit" flag. This flag can be toggled, or given to other accounts
in the billing group, but by default only the first account has it. This
flag allows that account to also access and change settings for all
other accounts beneath the same billing group.
Accounts with the "group edit" flag also are potentially given other
options that accounts without the gedit flag will never see. These
accounts are allowed to see a list of all the accounts within the
billing group, to change the credit card information for the group,
and also add resources to the billing group (such as extra email boxes).
5.3 A Note About Passwords
Passwords are stored in an encrypted state within the database, and
(without resorting to cracking methods) the unencrypted password is not
available to anyone after it is set. This means that if a client loses
or forgets their password the only remedy is to set a new password for
them; it is effectively impossible to retrieve the passwords in the
database in a usable form.
Since the client interface requires users to know their old password in
order change it to a new one, if a client has lost their password, they
will have no recourse except to contact the ISP and have an administrator
reset their password to a new one.
| 
