I've had alot of problems using KDE to organize my files. I have several directories with over 100,000 files in them, and dolphin hangs when even opening the folder, let alone moving the files or clicking on anything. So, I was looking into using the command line as a more powerful way to handle my files, but I have a few questions.

Whats the best way to move all the files from one directory to another?

Code: Select all

mv /directory/* /directory2

Is this the best way?

And a big question. Is it possible to move the first 20,000 (or X number) files from a directory to another directory? Its impossible to open directories with so many files so Id like to break it into chunks but I want to keep the order the files are in.

Thanks for your help,
-Supreme1012

Not exactly answering your question, but something that I've often found useful. If a large number of files is involved it seems to be very much quicker to copy them to the new location, then select all and delete them as a batch. I could be wrong, but I suspect that mv copies the file then deletes it then reads the next file, and so on.

that depends on where the files are moved to.
if the files stay on the same filesystem, the mv command simply renames them.

True - I was thinking particularly of files that need to be moved across the LAN.

yea I have the moving the files command down now, but Im getting "bash: /bin/cp: Argument list too long" when trying to move the files to a different directory. Im assuming its b/c I have too many duplicates?

The most important thing I need is a way to move 20,000 files to a new directory. I believe I need a bash script to do this, but I dont know bash scripting language. Before I have to teach myself Bash, is there anyone with Bash skills that could write me a quick bash script to move X number files from /directory1 to /directory2?

you could try to move the files in batches, based on the filenames.

For example, move all files starting with ab with this command:

mv ab* /path/to/new_directory

that way, you avoid moving too much files at once..

Supreme1012 wrote:yea I have the moving the files command down now, but Im getting "bash: /bin/cp: Argument list too long" when trying to move the files to a different directory. Im assuming its b/c I have too many duplicates?

The most important thing I need is a way to move 20,000 files to a new directory. I believe I need a bash script to do this, but I dont know bash scripting language. Before I have to teach myself Bash, is there anyone with Bash skills that could write me a quick bash script to move X number files from /directory1 to /directory2?

yea I tried that. The problem is the file names are erratic, so I could end up with 500 files or 50,000 files if I do it that way. I need a way to control the (number) of files while keeping the file names preserved. I posted a thread in linuxquestions.org for help with a bash script if anyone has any bash scripting skills.

http://www.linuxquestions.org/questions/programming-9/

okay, so to summorize,

you want to split up a directory with 100,000 files into 5 directories with 20,000 files.
The files should stay in order (eg the first 20,000 go to directory1, second 20,000 to directory2, etc)
Names should be preserved.

i'll ask some local gurus if they can help.

Riinse wrote:okay, so to summorize,

you want to split up a directory with 100,000 files into 5 directories with 20,000 files.
The files should stay in order (eg the first 20,000 go to directory1, second 20,000 to directory2, etc)
Names should be preserved.

i'll ask some local gurus if they can help.

thanks! I would really appreciate it, and you seem to have all the relevant points correct. but please make sure to mention I want it split into 2 directories only. so that I have 1 directory of 80,000 files and 1 directory of 20,000 files

ok well I came up with a small script with some help that seems to do what I need it to do.

Code: Select all

#!/bin/bash

COUNTER=0
while [  $COUNTER -lt 5 ];
do
  FILE=$(ls -X ~/test/dir1/ | head -1)
  mv -n ~/test/dir1/$FILE ~/test/dir2/
  let COUNTER=COUNTER+1
done

My only conundrum is for how to use "mv -n" without it stopping the script once it gets a duplicate.

Well, i received a script that should do the job.
This script expects the new directories side by side with the original one, like:

/home/supreme/original
/home/supreme/new-with-20000
/home/supreme/new-with-80000

This is it:

Code: Select all

#!/bin/bash

##########
# CONFIG #
##########

# Directory which contains all the files, needed to be split to directories
SOURCEDIR='sourcedir'
# Directory which should get the first group of files
TARGETFIRST='targetdir1'
# Directory which should get the second group of files
TARGETSECOND='targetdir2'
# Number of files after which the switch is made from $TARGETFIRST to $TARGETSECOND
TARGETSWITCH=20000

#################
# ACTUAL SCRIPT #
#################
# Set the field seperator to only newlines. Otherwise a space and a tab also are
#  field seperators, having the effect that a file with spaces, each part gets
#  treated as a seperate filename, causing mv to fail (it can't find the file).
#  This is the only way I know to set it to newline. IFS="\n" doesn't work.
IFS='
'
# Initialise the counter of the number of files processed
counterFile=0

# Move to the sourcedir and start moving all the files
cd $SOURCEDIR

# Start looping through all the files. Files are in alphabetical order
for file in $(find -type f | sort) ; do

    # Check to see to which folder everything must go to
    if [ $counterFile -lt $TARGETSWITCH ] ; then
        # Haven't passed the switchpoint, go to $TARGETFIRST
        mv "$file" "../$TARGETFIRST/"
    else
        # Passed the switchpoint, go to $TARGETSECOND
        mv "$file" "../$TARGETSECOND/"
    fi

    # Increase the file counter with 1 (isn't there a nicer way for this?)
    counterFile=$(expr $counterFile + 1)
done

#################################################################################
# Copyright (c) 2010, Cybertinus                                                #
# All rights reserved.                                                          #
#                                                                               #
# Redistribution and use in source and binary forms, with or without            #
# modification, are permitted provided that the following conditions are met:   #
# 1. Redistributions of source code must retain the above copyright             #
#    notice, this list of conditions and the following disclaimer.              #
# 2. Redistributions in binary form must reproduce the above copyright          #
#    notice, this list of conditions and the following disclaimer in the        #
#    documentation and/or other materials provided with the distribution.       #
# 3. All advertising materials mentioning features or use of this software      #
#    must display the following acknowledgement:                                #
#    This product includes software developed by Cybertinus.                    #
# 4. Neither the name of Cybertinus nor the                                     #
#    names of its contributors may be used to endorse or promote products       #
#    derived from this software without specific prior written permission.      #
#                                                                               #
# THIS SOFTWARE IS PROVIDED BY CYBERTINUS ''AS IS'' AND ANY                     #
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED     #
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE        #
# DISCLAIMED. IN NO EVENT SHALL CYBERTINUS BE LIABLE FOR ANY                    #
# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES    #
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;  #
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND   #
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT    #
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS #
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.                  #
#################################################################################