Comments on: Modules: should we support auto loading?

I need to survey the REBOL expert community for opinions on another important design issue for 3.0.

The REBOL definition of a module is: An standard form of independent context that extends the REBOL run-time environment.

Question: Should REBOL modules have the ability to auto-load from a default directory or URL?

For example, a script may specify:

REBOL [title: "A script" needs: [3.0.1 xml codec/mp3] ...]

This indicates that REBOL 3.0.1 is required, and that the XML and MP3 modules should be loaded. (The path used for codec/mp3 is just one possibility for organizing modules, and it is not finalized.)

Important points to consider:

1. Let it be clear that we are not changing the REBOL principle that most programs should be able to run "out of the box" with REBOL and no modules needed. That will remain as is. We are talking here about modules that are special extensions to REBOL. E.g. a web server module, charting module, special media loaders, new port schemes, etc.

2. Many/most other languages do this. When you ask to import a module, the runtime scans a few predetermined directories looking for the correct module to load.

3. We would need a way to specify the directory search path for finding such modules. Mostly likely a new system/options field.

4. Note that URL's could be allowed. For example, www.rebol.com/lib/ could be one of the default archive locations for modules.

5. Module files can be signed. This protects them from malware modifications. An security option will allow you to load non-signed modules.

6. Version resolution would be needed. Your code may depend on a more recent version of a module, or even a specific version.

7. This is not an essential requirement. You can always specify a file path or URL for importing modules. In fact, you could just use do and specify the module files.

8. Implementation is minor. The coding and test period should not be considered a factor.

I know we've discussed this before, but that was informal. If you want, you can refer to any of the above points by its number. (I won't change the order.)

Let's hear your thoughts.

17 Comments

Comments:

Henrik 31-Aug-2006 14:41:15	I like this a lot. It's time consuming to handle dependency problems on your own and a scenario that comes up for me often enough to want it in. Could this also be used to link scripts together to one big script for offline distribution? I use this very often as well and would eliminate the need for build scripts.
Carl Sassenrath 31-Aug-2006 14:51:20	Q: Could this also be used to link scripts together to [make] one big script for offline distribution? Previously, we've called that practice "include". However, there is a strong overlap. I would say, yes, it could be used in that fashion (just like it is possible today in many languages to specify static lib inclusion or dynamic lib loading). Note that it would still require a build tool of some kind, because you need to create the distribution "bundle".
Oldes 31-Aug-2006 14:56:36	To me it's not so important if they can be auto-loaded using the header as the possibility to load them if they are needed (later) and maybe uloaded since are not needed anymore. I'm really looking forwart to see first alpha versions.
Oldes 31-Aug-2006 14:59:38	I'm already using method to load dependent scripts if they are needed and I'm able to build one script from it. More important is to have the modules itself.
Carl Sassenrath 31-Aug-2006 16:02:06	I don't want to start an unload thread here, but plans are to allow unloading modules. (There will be some GC issues to consider. For instance, if your script still references a module, what is the result of an unload action?)
Volker 31-Aug-2006 16:04:14	I like it too, as Henrik for encapping. A header contains meta-informations, and dependencies are meta-informations. Nicer to have them there than somewhere in the script with 'do or #include.
Brian Hawley 31-Aug-2006 16:26:13	I like the module autoloading idea, go for it! As for module unloading, perhaps it could be done by the GC when all references to the module are unset. Then an unload function would just unset these references and trigger a GC. I'm more concerned about finalization. There have been many times when I have wrapped a resource in a fake port object to be able to put finalization code in the close function so that it would get called when the port object is collected. Some proper facility for module or object finalization would make this technique unnecessary.
Marco 31-Aug-2006 16:38:11	Many/most other languages do this. So don't do it.
Carl Sassenrath 31-Aug-2006 17:11:42	Volker: truth. Brian: you will be happy to know that finalize is part of the module spec (you have to be able to properly close ports, etc.) Marco: You are a true rebol! lol.
Gregg Irwin 31-Aug-2006 18:49:04	I'm all for it (in some form). I think the lack of this feature plays a big part in the dearth of standardized modules built outside RT; everybody does it differently. I also think it should be usable by a build system. Ladislav and I talked about this quite a bit, and there are pros and cons to requiring that INCLUDEs only be in the header. Flexbility is good. I know for what I'm doing we currently target 3 deployment models (script, encapped app, plug-in; sometimes the same app is deployed under all three). For libraries and distribution, look at other systems (e.g. RPM, Ruby Gems, DOAP, etc.) and build systems (MAKE, Rake, Ant/MSBuild, Maven, BOOM, et al), then throw away 80% of the crud in there (more in the case of Ant/MSBuild :). Building a module/library is another deployment model. This ties into everything else as well: e.g. testing and docs.
Robert 1-Sep-2006 0:09:51	Yes, add it. But than is's mandatory to use this information for the SDK as well. I don't want to write all the includes etc. once again to get an EXE. Just a static link (which I prefer anyway) where I know I have all I need included.
Maxim Olivier-Adlhoch 1-Sep-2006 0:34:49	please do all of the above :-) slim.r has been doing all of what you describe for 3 years now, (search path, version resolution, making single file distros, and much much more). Without all of these features... well, I'd have wasted weeks of time . might I add a point or two to above list? 9. NOT ENFORCING BINDING of the module's name or of exposed words. PLEASE allow the option to re-assign any automatic word binding module might perform as per automated features, simply to allow different conflicting modules to work together and adapt terminology within your code (if only to adapt language! :-) slim.r does this by having a refinement (/expose) on open. It supports submitting blocks containing word pairs [this that]. the first word is the one to find (within module). The second, the name it should take once exposed. 10.Allow concurent loading of the same module this allows modules to be used differently than simple "code loaders". If allowed to load a module more than once, we can use the finalize features to our advantage, like loading sessions and any error allows us to wipe out a user session, knowing the memory is freed forcibly. also, some binary includes within the module's source code, might need this feature... not supporting lib sharing, etc... .
Carl Sassenrath 1-Sep-2006 3:20:05	Gregg/Robert: yes, build/SDK is an important part. Modules are also relevant to REBOL plugin component architecture. There's a lot to modules, which is why we have to divide the subject into smaller parts for discussion purposes. So build approach will be separate thread. Maxim: Slim sounds great. I've not had the chance to use it, and I hope we can talk more about it at some point (in a different forum). Your additional requests are interesting. #9 is to remap the imported words. It is probably possible, but I'd like to see example of its use, because you can also use paths to access modules, which solves the conflicting module problem. (We should save this discussion for when I make that blog thread.) #10 is possible using MAKE module! You can make the same module as many times as you want. (But, perhaps such modules are not automatically added to the system modules list.)
Goldevil 3-Sep-2006 11:44:10	Module system, as described, is great. Maybe we need to specify a syntax of modules version and a syntax for specifying version compatibility. Here is an real example but transposed to rebol world. I need a precise version of a module Example: the codec/mp3 v1.0.0 doesn't support 192 kbps encoding, but the version 1.1.0 does it. More complicated, The version 1.1.0 has an annoying bug wich is always triggered by my script. I need the revision 1.1.3 that correct it. Worse! A new version of the codec (2.0.0) is not fully backward compatible. If a customer upgrade the codec, my script doesn't work anymore. For my projetcs I use this syntax for librairies : major version . minor version . Bug correction release The bug correction just says that a bug is corrected. The software has exactly the same specification that prior version. The minor version is always 100% backward compatible but it could include new or exented functionnalities. The major version change imply that this piece of code can be incompatible with prior major version. Back to my example. I had to specify something like REBOL [title: "A script" needs: [3.0.1 xml codec/mp3 1.1.3 ] ...] If the installed codec is 1.1.2, Rebol stop execution and throw an exception. If the installed codec is 1.4.8, Rebol continues. If the installed codec is 2.1.4, Rebol stop execution and throw an exception. Two other questions are coming in my mind : - Can I have more than 1 version of a module installed at the same time for compatibility issues ? - Can I use this to specify a version of rebol component (core 2.6.2, Draw System 1.1.0,...) ?
Cyphre 5-Sep-2006 7:58:13	Carl, would be also possible to sign/protect small scripts? Are you plannig some default built-in en/decryption support in R3 loader? It would be great to be able to protect Rebol script code without the need to be encapped.
Steven White 7-Sep-2006 15:35:56	Speaking as a beginner, I hope "plugin component architecture" refers to components that we write for ourselves and not components of REBOL that REBOL needs to run. I once tried to learn php, no problem with the basics, but then I tried to make it work with Oracle. It needed some component, I looked for it, either couldn't find it our couldn't understand how to "plug it in," and so finally gave up in frustration. I like REBOL because I can download/purchase it, and use it. I would hate to see a maze of interconnected components that must be found and installed in the correct combination just to get REBOL operating.
Porter Woodward 28-Nov-2006 11:33:35	I'd suggest taking a look at OSGi in the Java world. Knopflerfish is a pretty interesting example of what can be easily accomplished. http://www.knopflerfish.org/ and a pretty decent tutorial in pdf OSGi uses the term "bundle" or "service bundle" to describe a loadable module. That bundle has a version, and dependencies (including version dependencies) specified. These are specified in a manifest file - like so - Manifest-Version: 1.0 Bundle-Name: simplebundle Bundle-SymbolicName: simplebundle Bundle-Version: 1.0.0 Bundle-Description: Demo Bundle Bundle-Vendor: Vodafone Pilotentwicklung GmbH Bundle-Activator: de.vpe.simplebundle.impl.Activator Bundle-Category: example Import-Package: org.osgi.framework You'll notice this isn't dissimilar to the REBOL header in a REBOL script. There's some standard meta data to describe the author, etc. However there are also two lines of interest - the Bundle-Activator and Import-Package. These signal what object/method in the bundle activates or initializes it - and what additional packages might be required for the bundle to function properly respectively. I'm not saying it has to be done this way - but I would definitely suggest looking it over - as they have already considered a lot of the issues that come up with dynamically loadable and unloadable modules with multiple versions and dependencies. Even if it's Java based - the specification is open - so it might be possible to re-use some of the semantics for a REBOL based module system w/o reinventing the wheel.

Post a Comment:

You can post a comment here. Keep it on-topic.