                                                                                                                                                                                                                                                               
Delivered-To: david.cobas@gmail.com
Received: by 10.68.64.233 with SMTP id r9cs42520pbs;
        Fri, 24 Jun 2011 01:50:19 -0700 (PDT)
Received: by 10.204.138.136 with SMTP id a8mr1804484bku.106.1308905417812;
        Fri, 24 Jun 2011 01:50:17 -0700 (PDT)
Return-Path: <manohar.vanga@cern.ch>
Received: from CERNMX31.cern.ch (cernmx31.cern.ch [137.138.144.179])
        by mx.google.com with ESMTPS id q26si3339694faa.192.2011.06.24.01.50.17
        (version=TLSv1/SSLv3 cipher=OTHER);
        Fri, 24 Jun 2011 01:50:17 -0700 (PDT)
Received-SPF: pass (google.com: domain of manohar.vanga@cern.ch designates 137.138.144.179 as permitted sender) client-ip=137.138.144.179;
Authentication-Results: mx.google.com; spf=pass (google.com: domain of manohar.vanga@cern.ch designates 137.138.144.179 as permitted sender) smtp.mail=manohar.vanga@cern.ch
Received: from CERNFE21.cern.ch (137.138.144.150) by cernmxgwlb2.cern.ch
 (137.138.144.179) with Microsoft SMTP Server (TLS) id 14.1.270.1; Fri, 24 Jun
 2011 10:50:16 +0200
Received: from becoht-mvanga (137.138.192.18) by smtp.cern.ch
 (137.138.144.172) with Microsoft SMTP Server (TLS) id 14.1.270.2; Fri, 24 Jun
 2011 10:50:10 +0200
Date: Fri, 24 Jun 2011 10:47:47 +0200
From: Manohar Vanga <manohar.vanga@cern.ch>
To: <ht-drivers@cern.ch>
Subject: [ru@gnudd.com: Re: vme_intree_driver.txt]
Message-ID: <20110624084747.GA14398@becoht-mvanga>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
User-Agent: Mutt/1.5.21 (2010-09-15)
Return-Path: manohar.vanga@cern.ch
X-Originating-IP: [137.138.192.18]
List-ID: <ht-drivers.cern.ch>
Precedence: list
X-Auto-Response-Suppress: DR, OOF, AutoReply
Keywords: CERN SpamKiller Note: -50

Hi all,

I have been writing up a document describing the differences between the in-tree
driver and Emilio's (which we currently use). Below is the inlined document
followed by Alessandro's reply.

The document is still incomplete but this is where it is at so far. I am not
an expert at this and have many gaping holes in my knowledge and experience so
please _don't_ go easy on me with the criticism. I learn best that way :-)

----- BEGIN DOCUMENT -----

I have been studying the in-tree VME driver by Martyn Welsh and below are
the notes from my tests with it.

Note on Structure
-----------------

Just to mention beforehand, Emilio's driver (the one used at CERN currently
has one lesser level of indirection within most of the code when compared
with the in-tree driver. In this regard, the in-tree driver is better written
when compared with ours which was written for a specific purpose. For example,
here is an exerpt from the CERN driver (vme_dma.c):

    static int vme_dma_setup(struct dma_channel *channel, int to_user)
    {
        ...
        rc = tsi148_dma_setup(channel);
        ...
        return rc;
    }

And below is an example from the in-tree one:

    int vme_dma_list_exec(struct vme_dma_list *list)
    {
        struct vme_bridge *bridge;
        ...
        retval = bridge->dma_list_exec(list);
        ...
        return retval;
    }

This has an extra level of indirection through the vme_bridge structure which
makes it more generic.

Bridge Registration
===================

The TSI148 bridge driver is available as part of the in-tree driver. It
registers into the bridge framework of the VME driver. It does this using:

    static int tsi148_probe(struct pci_dev *pdev, const struct pci_device_id *id)
    {
        ...
        struct vme_bridge *tsi148_bridge;
        /* ... allocate and fill up the structure ... */
        vme_register_bridge(tsi148_bridge);
        ...
    }

vme_register_bridge assigns the bridge a bus number and registers VME_SLOTS_MAX
devices from within the bridge structure.

    struct vme_bridge {
        ...
        int num;
        ...
        struct device dev[VME_SLOTS_MAX];
        ...
    };

The bus number is assigned based on a bitvector that can hold 32 bits
(unsigned int). This means we can have upto 32 bridges registered.

One other thing to note is that the vme_bridge->dev[] structure contains devices
all pointing to the bridge itself from what I can see. This means when probe and
remove are called, the received dev in the parameters is the bridge and not the
device being probed or removed. Below is code from vme_register_bridge().

    for (i = 0; i < VME_SLOTS_MAX; i++) {
        dev = &bridge->dev[i];
        memset(dev, 0, sizeof(struct device));

        dev->parent = bridge->parent;
        dev->bus = &vme_bus_type;
        /*
         * We save a pointer to the bridge in platform_data so that we
         * can get to it later. We keep driver_data for use by the
         * driver that binds against the slot
         */
        dev->platform_data = bridge;
        dev_set_name(dev, "vme-%x.%x", bridge->num, i + 1);

        retval = device_register(dev);
        if (retval)
            goto err_reg;
    }

The parent of the device is being set to the parent of the bridge. I wasn't sure
about whether this was correct but from the the probe function of the vme_user
device, it is more clear.

    static struct device *vme_user_bridge;          /* Pointer to bridge device */

    static int __devinit vme_user_probe(struct device *dev, int cur_bus,
            int cur_slot)
    {
        ...
        /* Save pointer to the bridge device */
        vme_user_bridge = dev;
        ...

Emilio also seems to have found this. Below is the snippet from his LKML reply
at https://lkml.org/lkml/2010/10/22/43

> * .probe and .remove pass a pointer to a struct device representing a VME
>   bridge, instead of representing the device to be added/removed.
> 	* a bridge's module may be removed anytime and things do fall over;
> 	  there is no refcounting at all and thus all drivers attached to
> 	  the removed bus will oops.

Driver Registration
===================

VME drivers can register themselves using vme_register_driver. I wrote a driver
for the CVORA VME card to test things out and below is a snippet from its init
function.

    static int cvora_init(void)
    {
        ...
        retval = vme_register_driver(&cvora_driver);
        ...
    }

The cvora_driver above is of the type 'struct vme_driver'. In the very least,
we need to fill the 'name', 'probe' and 'release' fields. We also need to fill the
'bind_table' field (see below).

    struct cvora_driver {
        .name = "cvora",
        .probe = cvora_probe,
        .remove = __devexit_p(cvora_remove),

    };


Bind Table
----------

The in-tree driver seems to have taken its structure from the way PCI works and
requires each driver to provide a list of buses and slots within these buses
where it expects its card to be.

The problem is that if multiple bridges are ever present in the same
crate, the numbering of the buses/bridges depends on the load-order of the
bridge drivers. Not pretty. I don't think we have such a crate around (with
multiple bridges) so we can simply assume the bus number to be 0. If however,
we have such a configuration, we would have to ensure that the load-order of
the bridge drivers is always the same.

At CERN, we use a database to feed in the information to drivers (D:) so this
shouldn't be too big an issue.

The second issue however, is more "aesthetic". If we allow the slot numbers of 
the driver to be fed in through parameters, we have to write boring, repetitive
code that creates this bind table. The code in my CVORA driver, for example,
looks something like the one below.


    #define VME_BUS_MAX 32

    static int bus[VME_BUS_MAX];
    static int bus_num;
    MODULE_PARM_DESC(bus, "Enumeration of VMEbus to which the driver is connected");
    module_param_array(bus, int, &bus_num, 0);

    static int cvora_init(void)
    {
        int i;
        struct vme_device_id *ids;

        /* Dynamically create the bind table based on module parameters */
        ids = kmalloc(sizeof(struct vme_device_id) * (bus_num + 1), GFP_KERNEL);
        if (ids == NULL) {
                retval = -ENOMEM;
                goto err_id;
        }

        memset(ids, 0, (sizeof(struct vme_device_id) * (bus_num + 1)));

        for (i = 0; i < bus_num; i++) {
                ids[i].bus = bus[i];
                ids[i].slot = VME_SLOT_ALL;
        }

        cvora_driver.bind_table = ids;
        
        /* registration and error checking */
        ...
    }

The driver is loaded using the bus id of the TSI148 bridge (0, since its the
only one):

    $ insmod cvora.ko bus=0

Note that above, I have simply set the slot number in the bind table array
to VME_SLOT_ALL. This makes it _probe_ on all the slots. Note that this is
a probe and not a match. If we say a device will be in a specific slot, it is
fixed and no other device may be put in there (well, the wrong driver will get
probed based on the order of insmod). To avoid this, the slot numbers must
either be made unique for each driver (eg. if A has slots 1, 2 and 3, card B
should not have any of those). The other solution is to make these slot numbers
be per-driver rather than global.

The way that Emilio fixed this is to defer the matching to the driver itself.
So now the VME framework calls the match function of the driver which can then
do a device specific match (which cannot be done anywhere else as only the
driver knows how to identify its cards). This way, we don't need to provide any
bus/slot ids and we don't have ugly code for generating this table in each and
every VME driver. The driver simply gives the number of devices it wants to
match and these devices are created and the match is called for each one. The
device-specific matching is done in the driver and finally the non-matching
ones are removed.

Emilio's discussion of this issue on LKML: https://lkml.org/lkml/2010/10/22/43

We can also use VME_SLOT_CURRENT which takes the index of the device being
matched (in the vme_bridge->dev[] array) and compares with the geographical
address of the card being probed.

    static int vme_bus_match(struct device *dev, struct device_driver *drv)
    {
        ....
        while ((driver->bind_table[i].bus != 0) ||
                (driver->bind_table[i].slot != 0)) {
            ...
            if ((driver->bind_table[i].slot == VME_SLOT_CURRENT) &&
                    (num == vme_slot_get(dev)))
                return 1;
            ...
        }
        return 0;
    }

The vme_slot_dev() function just calls the slot_get field of the bridge
structure.

    int vme_slot_get(struct device *bus)
    {
        struct vme_bridge *bridge;
        bridge = dev_to_bridge(bus);
        ...
        return bridge->slot_get(bridge);
    }

This one requires the bridge to have geographical addressing capability (it
should be able to tell what slot a card is in). I don't know if this is
crate-specific or not but the bridge (TSI148) does support it. Below is the
code from 'bridges/vme_tsi148.c'.

    /*
     * Determine Geographical Addressing
     */
    static int tsi148_slot_get(struct vme_bridge *tsi148_bridge)
    {
        u32 slot = 0; 
        struct tsi148_driver *bridge;

        bridge = tsi148_bridge->driver_priv;

        if (!geoid) {
            slot = ioread32be(bridge->base + TSI148_LCSR_VSTAT);
            slot = slot & TSI148_LCSR_VSTAT_GA_M;
        } else 
            slot = geoid;

        return (int)slot;
    }

As can be seen in the code above, geographical addressing can be disabled
by passing a non-zero 'geoid' parameter to the vme_tsi148 driver during
load. If this is not supported, we cannot use VME_SLOT_CURRENT in the bind
table.

VME Resources
=============

The in-tree driver provides a way to request master, slave and/or DMA windows
and it does this through something called resources. This is simply a way to
identify and keep track of the different windows and DMA channels. So instead
of requesting a specific window or channel, we request a resource with the
specific attributes we want (eg. for slaves, A24 with MBLT).

This is managed by the vme_resource structure. It is a simple structure that
specifies the type of resource.

    /* Resource Type */
    enum vme_resource_type {
        VME_MASTER,
        VME_SLAVE,
        VME_DMA,
        VME_LM
    };

    ...

    struct vme_resource {
        enum vme_resource_type type;
        struct list_head *entry;
    };

NOTE: VME_LM in vme_resource_type is referring to the Location Monitor.

The bridge device registers a set of master, slave and DMA windows/channels
when it's driver is install and probe is run. Below is a snippet from
tsi148_probe().

    /* Add slave windows to list */
    for (i = 0; i < TSI148_MAX_SLAVE; i++) {
        slave_image = kmalloc(sizeof(struct vme_slave_resource),
            GFP_KERNEL);
        ...
        slave_image->address_attr = VME_A16 | VME_A24 | VME_A32 |
            VME_A64 | VME_CRCSR | VME_USER1 | VME_USER2 |
            VME_USER3 | VME_USER4;
        slave_image->cycle_attr = VME_SCT | VME_BLT | VME_MBLT |
            VME_2eVME | VME_2eSST | VME_2eSSTB | VME_2eSST160 |
            VME_2eSST267 | VME_2eSST320 | VME_SUPER | VME_USER |
            VME_PROG | VME_DATA;
        list_add_tail(&slave_image->list,
            &tsi148_bridge->slave_resources);
    }

Here the bridge is registering TSI148_MAX_SLAVE slave windows with the specific
attributes. The attributes are simply a set of bits being set in the specific
slave image. These are stored in the slave_resources list within the bridge
structure.

A similar procedure is done for masters and DMA channels in tsi148_probe().
Also it is useful to note that for the TSI148, the following are defined in
the header:

    #define TSI148_MAX_MASTER       8   /* Max Master Windows */
    #define TSI148_MAX_SLAVE        8   /* Max Slave Windows */
    #define TSI148_MAX_DMA          2   /* Max DMA Controllers */

From what I can make out:

Master resources: Those for drivers running on the master. Pretty much any
                  driver we use.
Slave resources : These are for drivers running on "intelligent slaves" that have
                  the kernel running on them.
DMA resources   : For any DMA transfers.

This document mostly concerns itself with master resources as these are what we
will be using for everything (we have no intelligent slaves running Linux at the
moment).

Requesting Resources
--------------------

As mentioned earlier, we can request resources based on a set of attributes. In
the case of the CVORA card, it has a 24 bit address space (A24). I requested a
master resource (the driver is running on a VME master, the main CPU) in the probe
function like so:

    card->resource = vme_master_request(card->bridge, VME_A24, VME_SCT, VME_D32);

In the above, VME_SCT refers to Single Cycle Transfer (or something like that...).
The VME_D32 refers to a data width of 32 bits. This was taken from the CVORA manual
available as a PDF at:

    http://isscvs.cern.ch/cgi-bin/cvsweb.cgi/cvora/doc/CVORA%20user%20Manual.pdf?cvsroot=abcofpga

There are similarly other functions for slaves and DMA.

    struct vme_resource * vme_master_request(struct device *dev,
        vme_address_t aspace, vme_cycle_t cycle, vme_width_t width);
    struct vme_resource * vme_slave_request(struct device *dev,
        vme_address_t aspace, vme_cycle_t cycle);
    struct vme_resource *vme_dma_request(struct device *dev,
        vme_dma_route_t route);

These functions loop through the bridge structures resource lists as registered
above and simply search for a free one with that requested bits set. Here is a
snippet from vme_slave_request():

    /* Loop through slave resources */
    list_for_each(slave_pos, &bridge->slave_resources) {
        slave_image = list_entry(slave_pos,
                struct vme_slave_resource, list);
        ...
        /* Find an unlocked and compatible image */
        if (((slave_image->address_attr & address) == address) &&
            ((slave_image->cycle_attr & cycle) == cycle) &&
            (slave_image->locked == 0)) {
            ...
            slave_image->locked = 1;
            allocated_image = slave_image;
            break;
        }
    }

They also set the "entry" pointer (to struct list_head) in the
vme_[master|slave|dma]_resource structure to point to &allocated_image->list
so that the vme resource structure (vme_[master|slave|dma]_resource) can be
retrieved later from the one passed by the user (struct vme_resource). Below
is an example of this from vme_master_set().

    int vme_master_set(struct vme_resource *resource, int enabled,
            unsigned long long vme_base, unsigned long long size,
            vme_address_t aspace, vme_cycle_t cycle, vme_width_t dwidth)
    {
        struct vme_master_resource *image;
        ...
        image = list_entry(resource->entry, struct vme_master_resource, list);
        ...
    }

Similar stuff happens in vme_[master|dma]_request(). Consequently, there are
also a set of vme_*_free() functions:

    void vme_master_free(struct vme_resource *res);
    void vme_slave_free(struct vme_resource *res);
    void vme_dma_free(struct vme_resource *res);

From the above, I think we can only register up to 8 master windows using this
driver. If we have more than 8 master drivers requesting windows, vme_request_master
would fail. I am not sure how people are using this driver but I suppose there is
a layer written on top to support more than 8 slaves.

In Emilio's driver, there is an abstraction over this to allow for more than 8
slave windows. Parts of one window are logically mapped allowing for different
devices to use the same window. For example if one window supports A24/D32 (24
bit address space and 32 bit data width), all devices requiring these window
attributes can map to different parts of this window (as long as there is
enough memory available).

Allocating Buffers
------------------

The driver documentation recommends using the vme_[alloc|free]_consistent() set
of functions to allocate and deallocate contiguous buffers in memory. These
functions simply do some checks and calls pci_[alloc|free]_consistent().

    void *vme_alloc_consistent(struct vme_resource *resource, size_t size,
            dma_addr_t *dma)
    {
        ...
        pdev = container_of(bridge->parent, struct pci_dev, dev);
        ...
        return pci_alloc_consistent(pdev, size, dma);
    }

    void vme_free_consistent(struct vme_resource *resource, size_t size,
            void *vaddr, dma_addr_t dma)
    {
        ...
        pdev = container_of(bridge->parent, struct pci_dev, dev);
        ...
        pci_free_consistent(pdev, size, vaddr, dma);
    }

One thing to note is that since it is calling pci_* functions, this means the
driver is assuming a PCI-VME bridge at all times (also the struct vme_bridge
contains a struct pci_dev as the parent device). I don't suppose
any other bridge apart from PCI is available or widely used so this is not a
problem. For us, it's fine since we use the TSI148 everywhere.

The vme_user driver (which allows user-space access for VME cards), which
is also the guide for my CVORA driver, however, doesn't use this stuff for
masters and instead uses it only for slaves.

For slaves:

    image[i].size_buf = PCI_BUF_SIZE;
    image[i].kern_buf = vme_alloc_consistent(image[i].resource,
        image[i].size_buf, &image[i].pci_buf);

For masters:

    image[i].size_buf = PCI_BUF_SIZE;
    image[i].kern_buf = kmalloc(image[i].size_buf, GFP_KERNEL);

For the CVORA driver (master driver), I used kmalloc in my code:

    card->size_buf = PCI_BUF_SIZE;
    card->kern_buf = kmalloc(card->size_buf, GFP_KERNEL);
    if (card->kern_buf == NULL) {
        printk(KERN_ERR PFX "kmalloc for buffer failed\n");
        goto vme_alloc_fail;
    }

Configuring the Master
----------------------

After requesting the required resources, we need to configure the channel
correctly by providing the base address, size of mapping etc.

In the case of masters and slaves, this is done using the
vme_[master|slave]_set() set of functions.

    int vme_master_set(struct vme_resource *, int, unsigned long long,
            unsigned long long, vme_address_t, vme_cycle_t, vme_width_t);
    int vme_slave_set(struct vme_resource *, int, unsigned long long,
            unsigned long long, dma_addr_t, vme_address_t, vme_cycle_t);

The first parameter is the resource we requested earlier. The second parameter
is whether to enable the master/slave or not (non-zero is to enable, zero to
disable). The third and fourth parameters are the base address and size of
mapping respectively. The last three parameters are the same as the ones passed
in the vme_*_request function. In the CVORA card, I do the following:

    card->base = bases[ndev];   /* array of bases passed as parameter to module */
    card->size = 0x8000;    /* Taken from CVORA manual */
    ...
    vme_master_set(card->resource, 1, card->base, card->size,
                    VME_A24, VME_SCT, VME_D32);

I then load the driver using:

    insmod cvora.ko bus=0 bases=0x3000000

Here, 0x3000000 is the default base address for the new card which I stuck into
the crate.

-----  END DOCUMENT  -----







----- Forwarded message from Alessandro Rubini <ru@gnudd.com> -----

Date: Thu, 23 Jun 2011 13:09:13 +0200
From: Alessandro Rubini <ru@gnudd.com>
To: manohar.vanga@cern.ch
Subject: Re: vme_intree_driver.txt

[btw: why doesn't this go to ht-drivers?]

> Just to mention beforehand, Emilio's driver (the one used at CERN
> currently has one lesser level of indirection within most of the
> code when compared with the in-tree driver. In this regard, the
> in-tree driver is better written when compared with ours which was
> written for a specific purpose.

One level less means more preformance -- is it useful? -- but
as your said it means it's tsi148-specific.

>         retval = bridge->dma_list_exec(list);

That's definitely better. You need structures of methods everywhere
except where perofmance is critical.

> This has an extra level of indirection through the vme_bridge structure which
> makes it more generic.

Exactly.

> The bus number is assigned based on a bitvector that can hold 32 bits
> (unsigned int). This means we can have upto 32 bridges registered.

This is suboptimal but I don't think people will ever exveed 32 buses.
If (when) it's possible to exceed it, a better mechanism should be used.

> One other thing to note is that the vme_bridge->dev[] structure
> contains devices all pointing to the bridge itself from what I can
> see. This means when probe and remove are called, the received dev
> in the parameters is the bridge and not the device being probed or
> removed.

This is bad. Can't is be a bus like other buses, with a list of
devices and drivers and a match function?  Then, each device should
host a pointer to the hosting bus, which has a bridge field.  Probe
and remove should receive the device itself, not the parent bus
controller.

> Below is code from vme_register_bridge().
> [...]
>         retval = device_register(dev);

Why does it register a device (0..slot-max) with no real device
being detected?

> Emilio also seems to have found this.

>> * .probe and .remove pass a pointer to a struct device representing a VME
>>   bridge, instead of representing the device to be added/removed.
>> 	* a bridge's module may be removed anytime and things do fall over;
>> 	  there is no refcounting at all and thus all drivers attached to
>> 	  the removed bus will oops.

Exactly.


> Driver Registration
> ===================
> 
> [...]

> The in-tree driver seems to have taken its structure from the way
> PCI works and requires each driver to provide a list of buses and
> slots within these buses where it expects its card to be.

Doesn't look like PCI. I mean, if you have no autodetection you must
say where your hardware is known to be. More like a platform device
than PCI/USB.

> The problem is that if multiple bridges are ever present in the same
> crate, the numbering of the buses/bridges depends on the load-order
> of the bridge drivers. Not pretty.

No, not pretty. It's like the eth0/eth1 problem (which has been solved
in the wrong way, in my opinion: I must remove the udev rules for that
or I have serious problems when switching boards).

Naming and numbering should depend on hardware address, not on load
order.  Like ether device should be named according to the pci slot
(and once I've done this trivial change for a client).

> I don't think we have such a crate around (with multiple bridges) so
> we can simply assume the bus number to be 0. If however, we have
> such a configuration, we would have to ensure that the load-order of
> the bridge drivers is always the same.

Not good.  But not easy to fix. I think existing drivers are using
the hardwired bus number, so changing it may be an issue.

But it can be done: the bus should be named by something which is not
a number, or at least not a zero, and specifying 0 means "all busses".
Existing drivers use 0, I assume, so that would work.  The probe
function for the driver should verify that the device is there,
if using "0" as a bus number. Anyway, I see no regression here as
currently everybody has one bus only, I suppose.

> At CERN, we use a database to feed in the information to drivers
> (D:) so this shouldn't be too big an issue.

But this database is a pain to maintain, isn't it?
 
> The second issue however, is more "aesthetic". If we allow the slot
> numbers of the driver to be fed in through parameters, we have to
> write boring, repetitive code that creates this bind table. The code
> in my CVORA driver, for example, looks something like the one below.

>     #define VME_BUS_MAX 32

isn't this elsewhere?


>         ids = kmalloc(sizeof(struct vme_device_id) * (bus_num + 1), GFP_KERNEL);
>         memset(ids, 0, (sizeof(struct vme_device_id) * (bus_num + 1)));

kzalloc please.


> The driver is loaded using the bus id of the TSI148 bridge (0, since its the
> only one):
> 
>     $ insmod cvora.ko bus=0

Then have this as a default. But yes, it's bad to go through this in each
driver.

> to VME_SLOT_ALL. This makes it _probe_ on all the slots. Note that this is
> a probe and not a match.

So you can look for the device and tell whether it's there or not, I suppose.

> The way that Emilio fixed this is to defer the matching to the
> driver itself.  So now the VME framework calls the match function of
> the driver which can then do a device specific match (which cannot
> be done anywhere else as only the driver knows how to identify its
> cards).

Unfortunately, like what happens on ISA, probing is not save. Even
reading has side effects on some devices, so you can't just touch
inside every slot. I mean: you have no other choice unless you have
more information, but usually you do, so you may specify bus+slot
(platform_device again).

So I think Emilio's approach is better but not best.

> We can also use VME_SLOT_CURRENT which takes the index of the device being
> matched (in the vme_bridge->dev[] array) and compares with the geographical
> address of the card being probed.

This should belong to the match function in the vme bus driver.

>     int vme_slot_get(struct device *bus)

Note that the *_get functins are usually increasing refcounts.  (and
*_put is the opposite). I suggest to choose a different name for this
function.

> This one requires the bridge to have geographical addressing capability (it
> should be able to tell what slot a card is in).

The bridge must know something in any case. Isn't the address
in VME uniquely identifying the slot?

> As can be seen in the code above, geographical addressing can be disabled
> by passing a non-zero 'geoid' parameter to the vme_tsi148 driver during
> load. If this is not supported, we cannot use VME_SLOT_CURRENT in the bind
> table.

I see.

> VME Resources
> =============
>
> The in-tree driver provides a way to request master, slave and/or
> DMA windows and it does this through something called
> resources. This is simply a way to identify and keep track of the
> different windows and DMA channels. So instead of requesting a
> specific window or channel, we request a resource with the specific
> attributes we want (eg. for slaves, A24 with MBLT).

resources in the kernel are continuous ranges you register in.
i.e., memory, I/O port (bleah!) and interrupts.  You remember the
resouce array associated to platform devices...

This looks like a similar thing, is it?

> The bridge device registers a set of master, slave and DMA windows/channels
> when it's driver is install and probe is run. Below is a snippet from
> tsi148_probe().

Ouch! Does this make sense?
 
>     /* Add slave windows to list */
>     for (i = 0; i < TSI148_MAX_SLAVE; i++) {
>         slave_image = kmalloc(sizeof(struct vme_slave_resource),
>             GFP_KERNEL);
>         ...
>         slave_image->address_attr = VME_A16 | VME_A24 | VME_A32 |
>             VME_A64 | VME_CRCSR | VME_USER1 | VME_USER2 |
>             VME_USER3 | VME_USER4;
>         slave_image->cycle_attr = VME_SCT | VME_BLT | VME_MBLT |
>             VME_2eVME | VME_2eSST | VME_2eSSTB | VME_2eSST160 |
>             VME_2eSST267 | VME_2eSST320 | VME_SUPER | VME_USER |
>             VME_PROG | VME_DATA;
>         list_add_tail(&slave_image->list,
>             &tsi148_bridge->slave_resources);
>     }

Fuck! If this is an array of TSI148_MAX_SLAVE things, why isn't this
an array+len instead of a list? Doesn't make too much sense to me.

I'd expect a resource thing like memory and i/o. The request should get
to the bridge, that knows its own capabilities.

> Requesting Resources
> --------------------
>
> [...]
>
> There are similarly other functions for slaves and DMA.
> 
>     struct vme_resource * vme_master_request(struct device *dev,
>         vme_address_t aspace, vme_cycle_t cycle, vme_width_t width);
>     struct vme_resource * vme_slave_request(struct device *dev,
>         vme_address_t aspace, vme_cycle_t cycle);
>     struct vme_resource *vme_dma_request(struct device *dev,
>         vme_dma_route_t route);

The kernel has a generic request_resource() thing, but I don't think
the set of resource is dynamic (there's mem_resource etc created at
boot time, I think).  We'd benefit from a similar approach. Actually,
we have an enum that names the different resources, you quoted it.
 
> These functions loop through the bridge structures resource lists [...]


The thing below is hairy, I'll quote it entirely:

> They also set the "entry" pointer (to struct list_head) in the
> vme_[master|slave|dma]_resource structure to point to &allocated_image->list
> so that the vme resource structure (vme_[master|slave|dma]_resource) can be
> retrieved later from the one passed by the user (struct vme_resource). Below
> is an example of this from vme_master_set().
> 
>     int vme_master_set(struct vme_resource *resource, int enabled,
>             unsigned long long vme_base, unsigned long long size,
>             vme_address_t aspace, vme_cycle_t cycle, vme_width_t dwidth)
>     {
>         struct vme_master_resource *image;
>         ...
>         image = list_entry(resource->entry, struct vme_master_resource, list);
>         ...
>     }

Uh! But "list_entry" is simply a "container_of". So why doesn't it
save directly the pointer to the wme_master_resource? It doesn't make
sense to have it like this!  If I misunderstood I'll get to the code
myself.

> From the above, I think we can only register up to 8 master windows
> using this driver. If we have more than 8 master drivers requesting
> windows, vme_request_master would fail. I am not sure how people are
> using this driver but I suppose there is a layer written on top to
> support more than 8 slaves.

Really, if they are windows they should be handled like the
mem_resource or ioport_resource.  And I wouldn't be sure people
use more than 8 slaves. Most users have small requirements, there's
only 1 LHS and a few other accelerators out there.
 
> In Emilio's driver, there is an abstraction over this to allow for
> more than 8 slave windows. Parts of one window are logically mapped
> allowing for different devices to use the same window. For example
> if one window supports A24/D32 (24 bit address space and 32 bit data
> width), all devices requiring these window attributes can map to
> different parts of this window (as long as there is enough memory
> available).

Seems saner.


> Allocating Buffers
> ------------------
>
> The driver documentation recommends using the
> vme_[alloc|free]_consistent() set of functions to allocate and
> deallocate contiguous buffers in memory. These functions simply do
> some checks and calls pci_[alloc|free]_consistent().

Does this mean that a VME bus has consistent memory directlty
and persistently mapped on PCI? It is quite a contraint, but maybe
the TSI bridge honors this -- no, I haven't yet read the whole TSI
manual, just a few pages.

>     void *vme_alloc_consistent(struct vme_resource *resource, size_t size,
>             dma_addr_t *dma)
>     {
>         ...
>         pdev = container_of(bridge->parent, struct pci_dev, dev);
>         ...
>         return pci_alloc_consistent(pdev, size, dma);
>     }

Also, this requires to bridge to be a PCI device. We'll need to lift
this requirement sooner or later -- shouldn't the function be
passed over to the bridge method? We _do_ have a bridge, this is
the extra layer advantage of this driver.

> One thing to note is that since it is calling pci_* functions, this means the
> driver is assuming a PCI-VME bridge at all times (also the struct vme_bridge
> contains a struct pci_dev as the parent device).

I should read all before beginning to write :)

> I don't suppose any other bridge apart from PCI is available or
> widely used so this is not a problem. For us, it's fine since we use
> the TSI148 everywhere.

Well, until somebody at CERN will etherbone the vme, you mean!

> The vme_user driver (which allows user-space access for VME cards), which
> is also the guide for my CVORA driver, however, doesn't use this stuff for
> masters and instead uses it only for slaves.
> 
> For slaves:
> 
>     image[i].size_buf = PCI_BUF_SIZE;
>     image[i].kern_buf = vme_alloc_consistent(image[i].resource,
>         image[i].size_buf, &image[i].pci_buf);
> 
> For masters:
> 
>     image[i].size_buf = PCI_BUF_SIZE;
>     image[i].kern_buf = kmalloc(image[i].size_buf, GFP_KERNEL);

I understand the former is a buffer within the VME memory space
as a DMA target and the latter is a normal buffer within the bridge
driver. Correct?


----- End forwarded message -----
