[Modeling-users] N-tier development and modeling

[Tom Hallman]

Greetings to all,

It is my hope to draw from the experience and expertise of those among
you who are more familiar with Modeling than I.

Here is the situation: my colleagues and I are attempting to put together
a framework using an N-tier methodology, with the client software
interfacing to the server/database backend via remote procedure calls (RPC).
For us, Modeling will serve as a large part of that backend, with an API
written overtop. The client will make calls to this API, and all will be grand.

The problem we are having is that of maintaining state between the server
and the client. For example, if we fetch a set of Person objects to be
displayed on the client, and then want to perform a delete on one particular
Person object, how do we know which Person to delete? Modeling seems to work
wonderfully up to where there is no RPC necessary, but once we add a split
over the RPC interface, Modeling seems to lack a natural way to refer to
objects across that divide.

One thought we had is to set the primary (and foreign?) keys as class
properties, such that we can refer to them across RPCs. This is somewhat of
a hybrid of Modeling and the old-fashioned "use the primary key to get what
we want" method.

Any thoughts on this matter would be greatly appreciated! Thanks so much
for your time.

Yours,
Tom

___________________________________________________________
Tom Hallman DiscipleMakers, Inc
***@dm.org http://www.dm.org/~hallmant
531 Brittany Dr * State College * PA * 16803 * 814-861-4591
Go therefore and make disciples of all nations (Matt 28:19)

[Sebastien Bigaret]

Hi Tom and all,

Post by Tom Hallman
Greetings to all,
It is my hope to draw from the experience and expertise of those among you
who are more familiar with Modeling than I.
Here is the situation: my colleagues and I are attempting to put together a
framework using an N-tier methodology, with the client software interfacing to
the server/database backend via remote procedure calls (RPC). For us, Modeling
will serve as a large part of that backend, with an API written overtop. The
client will make calls to this API, and all will be grand.

Nice!

Post by Tom Hallman
The problem we are having is that of maintaining state between the server
and the client. For example, if we fetch a set of Person objects to be
displayed on the client, and then want to perform a delete on one particular
Person object, how do we know which Person to delete? Modeling seems to work
wonderfully up to where there is no RPC necessary, but once we add a split
over the RPC interface, Modeling seems to lack a natural way to refer to
objects across that divide.
One thought we had is to set the primary (and foreign?) keys as class
properties, such that we can refer to them across RPCs. This is somewhat of a
hybrid of Modeling and the old-fashioned "use the primary key to get what we
want" method.
Any thoughts on this matter would be greatly appreciated! Thanks so much
for your time.

Working with PKs and FKs would work, but it is definitely not the
cleaniest way to identify objects wrt the framwork. There is a
dedicated component for that purpose: the objects' GlobalID (it can be
accessed through obj.globalID() or by asking the EC for the
globalIDForObject()); a GlobalID is a unique identifier for an object/a
row inside a framework.

Two different kinds of GlobalIDs exist:

- KeyGlobalIDs identify objects that exists in the database, and
uniquely identify such an object in whatever EditingContext.

- TemporaryGlobalID uniquely identifies a new object that is not saved
in the database yet. Because the objects is not saved, such an
globalID can identify the object only in its referring EC, or in one
of its children.

Both can be easily marshalled/unmarshalled, e.g. transformed to strings
and back to GlobalIDs; for example, a KeyGlobalID can be rebuilt from
its entityName() (string) and its keyValues() (dictionary).

Last, given that you have a gid, you get the corresponding object in a
EC simply by calling: obj=ec.faultForGlobalID(gid, ec)

Does this fulfill your needs?

I still have a question: will you also have an EC on the remote-side?
I'm asking this, because if this is the case the synchronization of the
client's EC and the server's one could be more complicated then simply
identifying objects by their globalIDs and passing some of their changed
attributes. And since I'm currently working on synchronization between
ECs and optimistic locking (working well, I think a first draft wil be
available this week-end), maybe I could also begin to think of such a
"remote" synchronization, esp. if you have a use-case and can consider
testing it in your own app.

-- Sébastien.

[Federico Heinz]

Post by Tom Hallman
Here is the situation: my colleagues and I are attempting to put together
a framework using an N-tier methodology, with the client software
interfacing to the server/database backend via remote procedure calls (RPC).
For us, Modeling will serve as a large part of that backend, with an API
written overtop. The client will make calls to this API, and all will be grand.

Maybe I'm missing someting here... ¿but are you sure that this ought to
be solved in Modeling itself? I can't help but refer to Modeling's
ancestor: EOF within OpenStep. In OpenStep, you would use EOF to get
object/relational persistence, and if you wanted to do n-tier deplyment
you'd resort to Distributed Objects, which were orthogonal to EOF (i.e.
any app could use either one, or both, or none).

So maybe what you are looking for is not a Modeling extension, but a
network-transparent message passing system for Python, which can be used
together with Modeling?

Fede

[Marcos Dione]

User-Agent: Mutt/1.3.28i

Post by Federico Heinz
So maybe what you are looking for is not a Modeling extension, but a
network-transparent message passing system for Python, which can be used
together with Modeling...

... just like pyro?

[Matthew Patton]

Hi,
Tom Hallman and I work together, so I speak for both of

Post by Tom Hallman
Here is the situation: my colleagues and I are attempting to put together
a framework using an N-tier methodology, with the client software
interfacing to the server/database backend via remote procedure calls (RPC).
For us, Modeling will serve as a large part of that backend, with an API
written overtop. The client will make calls to this API, and all will

be
grand.

Maybe I'm missing someting here... ?but are you sure that this ought to
be solved in Modeling itself? I can't help but refer to Modeling's
ancestor: EOF within OpenStep. In OpenStep, you would use EOF to get
object/relational persistence, and if you wanted to do n-tier deplyment
you'd resort to Distributed Objects, which were orthogonal to EOF (i.e.
any app could use either one, or both, or none).
So maybe what you are looking for is not a Modeling extension, but a
network-transparent message passing system for Python, which can be used
together with Modeling?

If I understand you correctly, I think you are talking about a similar
solution as the one Tom is proposing originally. To rephrase our idea, it
is to have to main components: a server and and a client, where the server
provides core data-accessing and data-manipulating functions, and the
client handles the interface to the user and the appropriate calling of
the functions on the server. For example, if a person had entered their
contact information through a web form, the client code to take that code
and save it to the server would be (rough sketch, sorry I don't know
python or modeling intimately yet):

# None of the following objects are explicitly related to Modeling

# Set the form data
address = Address()
address.setCity(formData["city"])
address.setState(formData["state"])
# etc.

person = Person()
person.setAddress(address)
address.setPerson(person)

# Initialize connection to the application server (this would be done
# once somewhere at the beginning of the client's initialization)
appServer = new AppServer(host="http://www.myappserver.com",
protocol=XML_RPC,
auth=userObj)

# Call add on the object PersonContainer, with person as the argument
# This is a "business logic" method, and may be the sort of transparent
# message-passing interface you were mentioning
appServer.PersonContainer.add(person)


On the server side (this would be served up with Zope)

class PersonContainer(ContainerBase):
def add(self, person):
""" This method would likely be abstracted, but I wrote it out for
the example. Here is where the Modeling code comes in.
One note: this person object *IS* a modeling-derived enterprise
object. The translation from a generic object on the client side
to a modeling object on the server side is a function of the
framework we are developing.
"""

self.ec.insert(person)
self.ec.saveChanges()


I hope this example is helpful and makes sense. It is our hope that by
separating out these common "business methods", that we can write numerous
different kinds of clients (web client for a public interface, gtk client
for an administrative interface) that will all utilize the same set of
business methods. Also, we can have the Application Server serve as a
common point for numerous kinds of clients to go to in order to access a
number of different data sources (LDAP servers, global
configuration files, and of, course, Modeling-accessed DB's).
I would be interested in any comments you all have as to how we
are going about doing this n-tiered approach and utilizing Modeling. As
far as we can tell, the only part of Modeling we lose is its faulting
capabilities on the client side. But the net gain of not having to write
any SQL, not having to care what kind of DB we are accessing, not having
to worry about locking or transactions are all enormous benefits that
Modeling gives us.

Thanks,
Matt

[Federico Heinz]

Post by Matthew Patton
Tom Hallman and I work together, so I speak for both of

Matt,

I think I understood what you were saying, but obviously you didn't
quite understand my response, which is my fault, of course. You will
find numerous references to NeXTstep and OpenStep in the Modeling
mailing list, as many of us come from that background, and Modeling
itself is a loose reimplementation of EOF (a technology first released
as part of NeXTstep) in Python.

Now, the code that you included would work quite differently in the
NeXTstep environment, for instead of doing the CORBA thing, we'd just
ask a server for a remote editing context. The local process would get a
proxy object, that works like it's local, but in fact forwards all
requests to a real editing context that lives in the server. We'd then
query the database using this editing context, instantiating objects in
the server, for which the Distributed Objects mechanism would create
proxies in the local address space. The whole IPC thing then happens
behind the scenes (aside from a few memory deallocation issues, the
programmer thinks he's dealing with local objects --- we can later argue
whether this is good or not).

My point was that, given that Modeling is an EOF reimplementation, it
might be rewarding to explore the possibilities of using something that
replicates the Distributed Objects functionality to achieve your goals.
Marcos pointed out in the list that Pyro might be such a beast...

Regards,

Fede

[Tom Hallman]

The local process would get a proxy object, that works
like it's local, but in fact forwards all requests to a real
editing context that lives in the server.

Fede,

Thanks for that input. A question - would using such a method (via Pyro,
etc.) bind us to using clients exclusively written in Python? For example,
if we wanted a client to be written in PHP or Java, we wouldn't be able to
use that Distributed Objects method. Is that correct?

~Tom

_________________________________________________________________
Frustrated with dial-up? Lightning-fast Internet access for as low as
$29.95/month. http://click.atdmt.com/AVE/go/onm00200360ave/direct/01/

[Federico Heinz]

Post by Tom Hallman
Thanks for that input. A question - would using such a method (via Pyro,
etc.) bind us to using clients exclusively written in Python? For example,
if we wanted a client to be written in PHP or Java, we wouldn't be able to
use that Distributed Objects method. Is that correct?

I'm not familiar with Pyro at all, so I'm afraid I can't comment on
that. If it were properly implemented, you ought to be able to cross
language boundaries (I remember java apps could call Distributed Objects
at some point in history), but certain assumptions would have to be true
in the target language. For instance, it would have to implement some
form of true message passing, and not mere polymorphism (i.e. no C++).

Fede

[John Lenton]

Post by Tom Hallman
Thanks for that input. A question - would using such a method (via Pyro,
etc.) bind us to using clients exclusively written in Python? For example,
if we wanted a client to be written in PHP or Java, we wouldn't be able to
use that Distributed Objects method. Is that correct?

pyro es python-specific; what it does is a pickle/unpickle,
essentially. Note that as modeling stands now you can't pickle the
objects (because they're actually weakrefs), so you'd have to write an
EditingContextProxy or something like it to hide that. Once you've
done that, you are a step away from being transport-independant. Pyro
is easy, and fast, so I'd recommend using it at first; however, once
you have that working, if you find you need to use corba or xmlrpc,
you'd just have to subclass the EditingContextProxy.

I'd like to point out that cimarron, the sub-proyect of PAPO I'm
currently busy with, aims to do what Matthew describes: in an MVC
triad, Modeling is the M (the classes Modeling generates are the
place to put the business logic), cimarron is the View, and you
program the Controller. The idea behind cimarron is rather like that
of anygui, in that you write the controller once, and cimarron is
able to render it via any of a series of concrete frontends (unlike
anygui, however, it's designed for strict MVC-like separation
:)). We currently only have a Gtk2 frontend, but we will be adding
at least curses and web (either via zope, or direct html generation)
at some point in the future. It is still rather green, but we're
already using it to reimplement PAPO's flagship proyect (Luca), so
it should be stabilizing quickly.

Let me point out, just in case it wasn't clear, that cimarron
neither knows nor cares if it's dealing with local modeling objects,
remote objects obtained via some kind of rpc, or plain python
objects; likewise, in whatever you decide to do, this separation is
crucial.

[Sebastien Bigaret]

Hi all,

Sorry for jumping in the discussion so lately, I've been quite busy those
last days.

I must say all this is really interesting. I unfortunately do not have
the time to investigate such things deeper, but i'm really interested in
a distributed layer. Had a very quick look at pyro, seems compact and
clean. If any of you experiment in this direction I'd like to hear from
the details, problems etc. As far as I can understand things, it seems
that Tom and John are speaking about similar things, if not the same.

BTW about weakrefs and pickle, I can effectively confirm that weakrefs
are used, mainly in objects (which hold a weakref to their EC) and in
to-many faults (AccessArrayFaultHandler) which for technical reasons
hold weakref of the object referencing them.

The other major problem will probably be that most operations, such as a
fetch(), are propagated through the object store hierarchy from an EC
down to the DatabaseContext(s), but at some point DBContext calls back
methods on the calling EC.


John: after a quick search I was not able to find any english-written
resources about papo and cimarron, is this available somewhere?



-- Sébastien.

[John Lenton]

Post by Sebastien Bigaret
I must say all this is really interesting. I unfortunately do not have
the time to investigate such things deeper, but i'm really interested in
a distributed layer. Had a very quick look at pyro, seems compact and
clean. If any of you experiment in this direction I'd like to hear from
the details, problems etc. As far as I can understand things, it seems
that Tom and John are speaking about similar things, if not the same.
BTW about weakrefs and pickle, I can effectively confirm that weakrefs
are used, mainly in objects (which hold a weakref to their EC) and in
to-many faults (AccessArrayFaultHandler) which for technical reasons
hold weakref of the object referencing them.

I've been thinking about this a bit (`in background', if you will),
and now I am of the opinion that actually using pyro on
editingcontexts and modeling objects is the wrong way to do this; I
think proxy classes (via overriding CustomObject?) is probably a saner
way to go. You probably don't want to shuttle your modeling objects
anyway: what you're wanting to distribute is the data, i.e. the result
of e.g. getFirstName, and possibly (although I doubt it) the effect of
setFirstName. Not the object itself; that would imply you are
distributing your business logic. I'll try to set up an example using
cimarron, so you can see what I mean.

Post by Sebastien Bigaret
John: after a quick search I was not able to find any english-written
resources about papo and cimarron, is this available somewhere?

papo is https://papo.vialibre.org.ar, but as you noted it's in
spanish. cimarron you can find in papo's savannah cvs; I'm afraid it's
lacking documentation and examples, and probably isn't much use
without a bit of handholding. However, it being a framework, we've
tried to keep it in english, so at least running pydoc on the files in
Generic/ should provide some insight. Consider it 'pre-alpha', because
it is.

[Ernesto Revilla]

Hi,

although this message does not respond exactly to the questions exposed here, I just wanted to comment what we do.

As with pyro, we designed our own protocol on top of XMLRPC, that is, objects and other data (None, DateTime) may be transmitted from server to client for whichever purpose. The client has a collection of 'dump' objects, in our words ObjectProxies, and nearly every call is a client-server-client transaction. Business logic resides on the server. The database can reside on another machine, of course. This is not very distributed, but it's a 3-tier architecture. We are still developing, so actually we are not in production phase. (In the future we may switch to pyro and/or Corba).

On the other hand, modeling is able to run at once different models on differen servers, and with the future optimistic locks, the same model could run on several machines.

In each way, if you work in a distributed environment, the implementation of an object will run on one or more 'server' machines, and the 'clients' will hold some proxies to these objects. All method calls not treated locally, will be send to the server. If you do not want to call the remote object, whenever an attribute is accessed, you can implement a proxy initialization which requests and stores locally all object attributes, or define a method in your framework (as we did), which requests a collection of attribute values for some objects (as a batch).


Best regsards
Erny

----- Original Message -----
From: "John Lenton" <***@vialibre.org.ar>
To: "Modeling users' mailing list" <modeling-***@lists.sourceforge.net>
Sent: Monday, March 15, 2004 2:00 PM
Subject: Re: [Modeling-users] N-tier development and modeling

Post by John Lenton

Post by Sebastien Bigaret
I must say all this is really interesting. I unfortunately do not have
the time to investigate such things deeper, but i'm really interested in
a distributed layer. Had a very quick look at pyro, seems compact and
clean. If any of you experiment in this direction I'd like to hear from
the details, problems etc. As far as I can understand things, it seems
that Tom and John are speaking about similar things, if not the same.
BTW about weakrefs and pickle, I can effectively confirm that weakrefs
are used, mainly in objects (which hold a weakref to their EC) and in
to-many faults (AccessArrayFaultHandler) which for technical reasons
hold weakref of the object referencing them.

I've been thinking about this a bit (`in background', if you will),
and now I am of the opinion that actually using pyro on
editingcontexts and modeling objects is the wrong way to do this; I
think proxy classes (via overriding CustomObject?) is probably a saner
way to go. You probably don't want to shuttle your modeling objects
anyway: what you're wanting to distribute is the data, i.e. the result
of e.g. getFirstName, and possibly (although I doubt it) the effect of
setFirstName. Not the object itself; that would imply you are
distributing your business logic. I'll try to set up an example using
cimarron, so you can see what I mean.

Post by Sebastien Bigaret
John: after a quick search I was not able to find any english-written
resources about papo and cimarron, is this available somewhere?

papo is https://papo.vialibre.org.ar, but as you noted it's in
spanish. cimarron you can find in papo's savannah cvs; I'm afraid it's
lacking documentation and examples, and probably isn't much use
without a bit of handholding. However, it being a framework, we've
tried to keep it in english, so at least running pydoc on the files in
Generic/ should provide some insight. Consider it 'pre-alpha', because
it is.
--
"In the long run, every program becomes rococo, and then rubble."
-- Alan Perlis
-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
Modeling-users mailing list
https://lists.sourceforge.net/lists/listinfo/modeling-users

[Sebastien Bigaret]

Hi,

Thinking about this in the background as well ;) I've also wondered
whether simple object-proxies could do the trick for a first step toward
a distributed data layer (which I now think should ultimately be built
on top of EC, but I may change my mind in a few hours:). And very,
very interestingly now comes Erny w/ something like a proof of concept
implementation on its side! That's very promising.

BTW is anyone here familiar w/ zodb's zeo? It's probably worth looking
at it as well.


John: I'm afraid i do not have the time to dig in cimarron's source code
now, so I think I'll wait for your short example :)

Erny: is your current impl. read-only on the client side or is it RW?
Which architecture are you using? Maybe GlobalIDs along with a map (one
client->one ec) to identify objects, and pickling (or alike) to transmit
an object's data?


-- Sébastien.

Post by Ernesto Revilla
Hi,
although this message does not respond exactly to the questions exposed here, I just wanted to comment what we do.
As with pyro, we designed our own protocol on top of XMLRPC, that is, objects and other data (None, DateTime) may be transmitted from server to client for whichever purpose. The client has a collection of 'dump' objects, in our words ObjectProxies, and nearly every call is a client-server-client transaction. Business logic resides on the server. The database can reside on another machine, of course. This is not very distributed, but it's a 3-tier architecture. We are still developing, so actually we are not in production phase. (In the future we may switch to pyro and/or Corba).
On the other hand, modeling is able to run at once different models on differen servers, and with the future optimistic locks, the same model could run on several machines.
In each way, if you work in a distributed environment, the implementation of an object will run on one or more 'server' machines, and the 'clients' will hold some proxies to these objects. All method calls not treated locally, will be send to the server. If you do not want to call the remote object, whenever an attribute is accessed, you can implement a proxy initialization which requests and stores locally all object attributes, or define a method in your framework (as we did), which requests a collection of attribute values for some objects (as a batch).
Best regsards
Erny
----- Original Message -----
Sent: Monday, March 15, 2004 2:00 PM
Subject: Re: [Modeling-users] N-tier development and modeling

Post by John Lenton

Post by Sebastien Bigaret
I must say all this is really interesting. I unfortunately do not have
the time to investigate such things deeper, but i'm really interested in
a distributed layer. Had a very quick look at pyro, seems compact and
clean. If any of you experiment in this direction I'd like to hear from
the details, problems etc. As far as I can understand things, it seems
that Tom and John are speaking about similar things, if not the same.
BTW about weakrefs and pickle, I can effectively confirm that weakrefs
are used, mainly in objects (which hold a weakref to their EC) and in
to-many faults (AccessArrayFaultHandler) which for technical reasons
hold weakref of the object referencing them.

I've been thinking about this a bit (`in background', if you will),
and now I am of the opinion that actually using pyro on
editingcontexts and modeling objects is the wrong way to do this; I
think proxy classes (via overriding CustomObject?) is probably a saner
way to go. You probably don't want to shuttle your modeling objects
anyway: what you're wanting to distribute is the data, i.e. the result
of e.g. getFirstName, and possibly (although I doubt it) the effect of
setFirstName. Not the object itself; that would imply you are
distributing your business logic. I'll try to set up an example using
cimarron, so you can see what I mean.

Post by Sebastien Bigaret
John: after a quick search I was not able to find any english-written
resources about papo and cimarron, is this available somewhere?

papo is https://papo.vialibre.org.ar, but as you noted it's in
spanish. cimarron you can find in papo's savannah cvs; I'm afraid it's
lacking documentation and examples, and probably isn't much use
without a bit of handholding. However, it being a framework, we've
tried to keep it in english, so at least running pydoc on the files in
Generic/ should provide some insight. Consider it 'pre-alpha', because
it is.
--
"In the long run, every program becomes rococo, and then rubble."
-- Alan Perlis