Thread Problem, GUI freezes


#1

Hello everybody,

i have a problem with threads.

I’m trying to run the search for multimedia files (for example, * .avi, * .mkv, etc.) in a separate thread. The found files should be counted. I use the “The Genric” code from Martin’s book for thread execution.
The function performed in the thread looks like this.:

 @staticmethod
def count_files(f_dir, name_filters, filters, progress_callback):
    """
    Counts how many files match the search pattern
    :param f_dir: root directory for search
    :param name_filters: filter for filenames (e.g. *.avi)
    :param filters: filtering options available to QDir (e.g. QDir.Files)
    :param progress_callback: signal for progress function
    :type f_dir: str
    :type name_filters: list
    :type filters: int
    :type progress_callback: pyqtSignal
    :return: ammount of processed files
    :rtype int
    """
    print("In Count Files")
    try:
        if f_dir:
            it_count = QDirIterator(f_dir, name_filters, filters,
                                    QDirIterator.Subdirectories)
            # count files
            while it_count.hasNext():
                it_count.next()
                FdbCheckFiles.__processed_files += 1
                progress_callback.emit(FdbCheckFiles.__processed_files)
    finally:
        pass

    return FdbCheckFiles.__processed_files

My problem: In a search on an entire hard disk or partition is not permanently a file found and therefore no signal sent out (progress_callback). Without a signal, the GUI freezes.

How can I fix the problem?

All the examples I’ve gotten about threads in QT work with timers that send pretty regular signals to the mainthread. That always works great.

I also tried to create threads in different ways. Eg. inherit a worker from QObject and control it with “worker.moveToThread ()” and signals. The result is always the same.
Is not it possible to create a thread, let it work and receive only a result signal without blocking the main thread?

In my example run the QDirIterator and only if the iterator is ready to transfer the result?

Greetings Kai


#2

Hi Kai

The main thread shouldn’t be blocked by this function — it doesn’t depend on the progress_callback to be called, that’s just to send useful data out.

One thing that jumps out is that you’re emitting a signal for every single file which will lead to a lot of signals. Each of these signals must be handled by your main GUI code (anything that’s attached to the signal) and so effectively you’re repeatedly blocking your main GUI from you’re thread.

What you can do is to only emit the signal on a fraction of your loops. You already have a count variable, so you can just use modulo % against this number, to trigger the emit on each 1000 files, e.g.

     while it_count.hasNext():
         it_count.next()
         FdbCheckFiles.__processed_files += 1
         if FdbCheckFiles.__processed_files % 1000 == 0:
             progress_callback.emit(FdbCheckFiles.__processed_files)

You might want to add another emit at the end (before the return) to update your GUI with the final number.


#3

Hello Martin,

Thank you for your tip. But I can say with certainty that the problem (in this case) is not due to the amount of signals. I used this modulo construction (% 25) in the beginning, but during my trial I removed the lines. The behavior was always the same. Because of the name_filter used, only 600 files were processed during testing.

Somehow the problem seems to be related to a peculiarity of the QDirIterator. After doing the same thing with python’s os.walk, the worker thread works as expected.

Greetings Kai


#4

Great to hear you’ve sorted it out.

If switching to os.walk fixed it, it does sound like QDirIterator is blocking the main thread. Very strange behaviour indeed?!

Which version of Qt/PyQt are you using by the way?


#5

It’s PyQt5 5.10.1/PyQt5-sip 4.19.2 and Python 3.6.6

The new function looks like this:

 def process_import_files(self, rootdir, name_filters, progress_callback):
    """
    Counts how many files match the search pattern
    :param rootdir: root directory for search
    :param name_filters: filter for filenames (e.g. *.avi)
    :param progress_callback: signal for progress function
    :type rootdir: str
    :type name_filters: list
    :type progress_callback: pyqtSignal
    :return: ammount of processed files
    :rtype int
    """
    import_dirlist = []
    filecount = 0
    processed_files = 0
    rw_lock = QReadWriteLock()

    try:
        if rootdir:
            for root, dirs, files in os.walk(rootdir, topdown=True):
                for name in files:
                    finfo = QFileInfo(os.path.join(root, name))
                    suffix = '*.{0}'.format(finfo.suffix().lower())
                    rw_lock.lockForRead()
                    abort = self.thread_aborted
                    rw_lock.unlock()
                    if suffix in name_filters and not abort:
                        fdir = QDir.toNativeSeparators(
                            finfo.absoluteFilePath())
                        import_dirlist.append(fdir)
                        filecount += 1
                    elif abort:
                        return processed_files

        for importdir in import_dirlist:
            rw_lock.lockForRead()
            abort = self.thread_aborted
            rw_lock.unlock()
            if abort:
                break
            finfo = QFileInfo(importdir)
            fdir = QDir.toNativeSeparators(
                finfo.absoluteDir().absolutePath()
            )
            basename = finfo.completeBaseName()
            fname = finfo.fileName()
            suffix = finfo.suffix().lower()
            processed_files += 1
            ins_data = {'filecount': filecount,
                        'processed_files': processed_files,
                        'basename': basename,
                        'dir': fdir,
                        'fname': fname,
                        'suffix': suffix
                        }

            progress_callback.emit(ins_data)
            # Give the main threads event loop time to read out events.
            QThread.msleep(5)
    finally:
        pass

    return processed_files

progress_calback is used to call a routine in the main thread, which writes the data determined in the worker thread into an SQLite database.

The function in the main thread also uses part of the data in the sent dictionary to update the progress bar in the status bar of the main window.

The function that is called with the “result” signal indicates in a message box how many data have been processed and how many actually have been written to the database (multiple entries are not permitted).

As I see it now I could save a few lines and copy the finfo data directly in the dictionary.