25

Hello,

I've been working on a construction for a long time now, and te create
easy, functional access to all kind of resources
like SMB shares, FTP servers, SSH and USB devices while making use of
the best tools to mount them. In the case
of SMB shares that is mounting with cifs. Prepare for a long story, it's
not so easy to explain.

This construction is based upon two techniques I've integrated to work
together.
The first one is the automounter, which mounts targets on demand. In my
construction it is designed to work
with what I call resource records. These records contain all the
infromation about a resource and how to mount it.

It looks like:

/var/cache/mount.md5key/b4a0859ac795f50baf7e43ff4f49f314 contains

.directory
config
permissions
SMB_services


The name of this record is b4a0859ac795f50baf7e43ff4f49f314, all the
information is in the config file.
Other files do have a meaning, but it's to complicated to tell about
that here.

The config file looks like:

RECORD_type=smb
RECORD_security=private
RECORD_creator=discover-smb-resources.sh
LOCAL_user=sbon
SMB_workgroup=BONONLINE
SMB_name=LFS20060812
SMB_share=public
SMB_ip=192.168.0.2
SMB_auth_method=private
SMB_credential_file=/home/sbon/.smb/mount.cred


Now the automounter makes use of these records the following way
An apart auto.master (created for this purpose only) is in /var/run/autofs,
looking like:

/mnt/mount.md5key/sbon/mount /etc/autofs/auto.md5key MD5KEY_USER=sbon
/mnt/mount.md5key/root/mount /etc/autofs/auto.md5key MD5KEY_USER=root

(users sbon and root are making use this construction)

where the auto.md5key is:

* -fstype=md5key ":/&"

Very simple and straightforward, just passing every key to the mountcommand
of the nonexistent filesystem md5key. This mount.md5key is a wrapper, and
intelligent enought to do a lookup of the key, and launch the right mount
command, in the case of SMB this is mount.cifs. So when I do:

ls /mnt/mount.md5key/sbon/mount/b4a0859ac795f50baf7e43ff4f49f314

the share //LFS20060812/public is mounted there, using credentials
(which are stored in a file).

Maintenance of these resource records are done through scripts
(for SMB making use of utilities like nbtscan, nmblookup and smbclient)

Now this constrcution is not to be used directly, but in cooperation
with a fusemodule I've developed. But first, you can make use of
this mounting ability, by creating a tree with symlinks:

A "Windows Network" tree may look like:

/home/sbon/Network/Windows Network/BONONLINE/ADMIN/admin ->
/mnt/mount.md5key/sbon/mount/50b89942f697f095ac66359eae1b3a8f
/documentation ->
/mnt/mount.md5key/sbon/mount/68141c6ce3077c8e438f7b31c3002053
/public ->
/mnt/mount.md5key/sbon/mount/795e5d59a7901e91296cf6aae0987c94
/sbon ->
/mnt/mount.md5key/sbon/mount/8236f6c5eae783204a24f447c4ea8ec3

Now this works, but is not userfriendly in my opinion.
When browsing to these links and following them, most applications
forget the place where they are coming from, and the user ends up
browsing in /mnt/mount.md5key/... tree, for an average user a
strange place. Should not happen.
(there are more cons to this, every time an user enters
this directory all targets are mounted, cause most applications
want to "know" what symlinks are pointing to. Also most apps
in KDE look for a .directory file, also causing unnec. mounting)

So that's why I've created the FUSE modules fuse-workspace and
fuse-workpace-union. The first is a first try, with hardcoded paths.
The second one it way more advanced, with different parameters,
and has integration with the automounter, and can do some caching.

Now these modules are overlay filesystems, which "mirror" another
directory (the "bind" directory) into the mounted dir, just like
the fusexmp (part of the FUSE source) does with the root (/).
Next to mirror, they are able to make symlinks (to dirs) look like directories.

mkdir -p /home/sbon/.fuse/bind
mkdir -p /home/sbon/.fuse/conf
mv /home/sbon/Network /home/sbon/.fuse/bind
mkdir /home/sbon/Workspace

fuse-workspace-union --bind-directory=/home/sbon/.fuse/bind \
--conf-directory=/home/sbon/.fuse/conf \
/home/sbon/Workspace


Now without any action the symlinks above remain unchanged:

cd /home/sbon/Workspace/Network/Windows Network/BONONLINE/ADMIN
ls -l
lrwxrwxrwx 1 root root 61 2009-12-31 21:17 admin ->
/mnt/mount.md5key/sbon/mount/50b89942f697f095ac66359eae1b3a8f
lrwxrwxrwx 1 root root 61 2009-12-31 21:17 documentation ->
/mnt/mount.md5key/sbon/mount/68141c6ce3077c8e438f7b31c3002053
lrwxrwxrwx 1 root root 61 2009-12-31 21:17 public ->
/mnt/mount.md5key/sbon/mount/795e5d59a7901e91296cf6aae0987c94
lrwxrwxrwx 1 root root 61 2009-12-31 21:17 sbon ->
/mnt/mount.md5key/sbon/mount/8236f6c5eae783204a24f447c4ea8ec3


Now create directories in the conf tree, to instruct fuse-workspace-union
to translate the symlinks to shares to directories

install -d /home/sbon/.fuse/conf/Network/Windows\ Network/BONONLINE/ADMIN/admin
install -d /home/sbon/.fuse/conf/Network/Windows\ Network/BONONLINE/ADMIN/documentation
install -d /home/sbon/.fuse/conf/Network/Windows\ Network/BONONLINE/ADMIN/public
install -d /home/sbon/.fuse/conf/Network/Windows\ Network/BONONLINE/ADMIN/sbon


Now the same command

cd /home/sbon/Workspace/Network/Windows Network/BONONLINE/ADMIN
ls -l
drwxr-xr-x 2 sbon netgroup 61 2009-12-31 21:17 admin
drwxr-xr-x 2 sbon netgroup 61 2009-12-31 21:17 documentation
drwxr-xr-x 2 sbon netgroup 61 2009-12-31 21:17 public
drwxr-xr-x 2 sbon netgroup 61 2009-12-31 21:17 sbon

Now when listing the contents of this directory does not make all
targets to be mounted.

This does not only work for SMB shares, but also for FTP, SSH and I'm
convinced for IPX( Novell Netware) also.
And for local resources like USB devices. Right now there is mounting by
HAL of these devices, but I'm not so happy with that.
(soon to be replaced by udisk) The problem is that mounting of a device
by more than one user (MultiSeat!!) is not possible.
I've posted serveral meddages about this topic in de hal and udisk
maillist, no reply recieved)

With this construction, no problem. Each user has her/his own (autofs
managed) mountpoint in /mnt/mount.md5key/
with her/his own options, and this is mirrored into her/his homedirectory.
I' do not have investigated this futher, but by mirroring the
mountpoints to somewhere in your homedirectory, maybe it's also
possible to do some translation from charset to charset
Now, it's easy to add the automounted directory and mount the
fuse-workspace-union when a session starts through ConsoleKit
(and the at daemon) which has the ability to run scripts
when a session is added (and when a session is removed)
Futher is also not so hard to automate the proces of maintaining the
"resource records". Through a script which looks for SMB server with
nbtscan and determines the shares with smbclient for every user looged in,
and run that every 5 minutes.

And it's possible to automate the propagation of changes into the
connections trees in the homedirectories of users.

Now I write this idea to get this construction under your attention.
Gnome has something also with FUSE (the Gnome Virtual Filesystem/ gio).
I'm not working with Gnome (he KDE!) so I cannot say anything about that.
And in my opinion KDE needs something better than kio slaves!

This construction makes use of real paths, making resources available
just for every application, just like entering a directory.
It's making use of the cifs filesystem to access SMB shares, which has
more features than the kio slave for smb, like support of
ACL/.ATTR, oplocking, transparant uid/gid/permission in environments
with a shared id database like LDAP or ADS (with winbind).


And mentioned above more servives like FTP, IPX and local mounting is
possible.

More about this:

http://linux.bononline.nl/linux/mount.md5key/index.php

http://linux.bononline.nl/linux/runsessionscripts/index.php

http://linux.bononline.nl/linux/fuse-workspace/index.php

Screenshots:



Two networks available, FTP and SMB on my machine.



Browsing the Windows Network, there are two workgroups (and/or domains)



There are two servers in domain BONONLINE.



The server LFS20060812 offers five shares. Nothe the fine icons, and one blue, which
is still a bug...



Entering a share...



Viewing the ACL's of a file on a share which supports ACL/ATTR.


Note the icons, I'v put a lot of
effort in making fuse-workspace-union handle the .directory files right.
I hope I recieve at least a reaction what you think. Any reaction is
welcome. I've put a lot of effort in developing this constrcution,
learned C, and posted about this before, but very few reactions, while
I'm convinced that this construction has potential.
On my system it's working, and I'm very happy with it. It's so
convenient and functional. I know that the documentation is not up
to order, but I'm working on it.

Stef Bon

Pleaso to moderator: finally it's working! I get my hint on the list. But please check the format and will you remove the really stupid smiley!!!

Especcialy look at the pieces cut off(newlines removed) at a ls command.

Thanks

bonassis wrote:Pleaso to moderator: finally it's working! I get my hint on the list. But please check the format and will you remove the really stupid smiley!!!

Especcialy look at the pieces cut off(newlines removed) at a ls command.

Thanks

Sorry I can't. You have to do it yourself. Just edit your idea and check the no smiles or something like that...

I definately think this has potential, but here's a thought. Since the idea "conflicts" with mechanisms already in place in KDE, goes quite deep down the software stack, and that you already have a working prototype, it seems like it really should be discussed on one of the KDE-developer mailing-lists.

I'm no developer so I'm unable to say which one would be most appropriate.

The usual question for something like this is: "how does it handle asynchronous access?"

GVFS, like KIO, lets applications connect to the actual transfer handler and receive/send data asynchronously so they don't block when the actual transfer blocks.

First I've posted this on de kde maillist, no reaction.
Not even one. That's the reason I've posted the idea/construction here. I hope that developers also read this.

About asynchronous access?? It's based upon FUSE, so all the properties are that from FUSE.

I do not know what you exactly mean by asynchronous access?
I read in the help from fuse that it performs reads asynchronously.

I only can say that my module is an overlay fs, which simply mirrors existing content/data/files/dirs etc.

How can I test "being asynchronous"??

Stef

Asynchronous in this context means that the application requesting the operation will not block during the execution of the operation.

KIO and GVFS solve this by using a separate process for doing the actual I/O and the application connect to them through a local socket.

Applications send commands and get notified about results such as connection established, data received, ready for sending data, etc.

Usually mounted remote file systems like NFS or SMB have the problem that they look like local files with no easy way to application to know that they are actually remote.

So they treat them as local files, potentially trying performance boosting access methods like memory mapping the file, or data safety methods like writing to a second file and then renaming to the original file.

Solution that mount into the filesystem sometimes provide no standard way to pass metadata between application and the mount system, such as cookies or user agent string for webdav resources, or do not set the MIME type of files correctly, etc.

Hello,

I do not understand your reply.

My fs is a fuse module, an overlay fs which makes use of the automounter and (mount.)cifs to access SMB shares.
Up to my information FUSE offers multithreading, and I've made my module thread safe.
Futher according to me, it relies on the underlying fs. In the case of cifs, it can make use of the capabilities offered by cifs.
I always thought that mounting a SMB share with cifs is the best option to access the share from a Linux host. According to you that is not the case.. That's suprising really, to me cifs is the best way to go...
Stef

As I said, the primary need is an easy a deterministic way for applications to check whether a file is a local file or not.

Usually this is done by checking the file URL, i.e. it is a local file if the file URL starts with protocol "file:"

This breaks for remote resources mounted into the local file system, so there needs to be a different way to check that reliably.

The other issues are less important but nevertheless have to be considered and sovled in order not to regress in features of current solution (e.g. KIO).

Thanks for your reaction.

I'm beginning to understand the problems...
But I can say that the translation from hardcoded paths from this constrcution to URI (and vica versa) is possible.
For example

/home/sbon/Workspace/Network/Windows Network/BONONLINE/LFS2000812/bononline/testfile

in share bononline on server LFS20060812, domain BONONLINE

the module fuse-workspace-union is able to translate this path into the URI for this user

smb://LFS20060812/bononline/testfile

User specific, cause different users have different paths.
Futher vica versa is also possible.

I go reading about this topic/problems. Any advice where I can find good documentation about this issue (KIO/FUSE...)?

Stef Bon

KDE apps most commonly use KUrl::isLocalFile() to decide whether to use fast local access methods or handle a document through KIO.

You can use http://lxr.kde.org/ to find occurances of isLocalFile() to see how applications use this for different reasons.