Solvedkeras how to use fit_generator with multiple image inputs

Hello, I'm trying to use a model with paired input images through (in their own similar directory trees), augmented through ImageDataGenerator using also flow_from_directory (so the method infers the labels by the folder structure). I'm getting an error because keras can't handle it in this way.

How can I combine the generators (using flow_from_directory) to be accepted by fit_generator?

Here is a sample code

Model definition ***

#two classic CNN blocks are defined before these lines and then cocatenated
#create model
model = Model([input_img_1,input_img_2], out, name='colliculus_proto')

#compile model
model.compile(optimizer=Adam(lr=1e-3), 
              loss='categorical_crossentropy', 
              metrics=['accuracy'])

# image data generators for image inputs
def input_generator(train_dir,batchsize,img_height,img_width):
    train_generator = ImageDataGenerator(rescale = 1./255, 
                                       shear_range = 0.2, 
                                       zoom_range = 0.2,
                                       rotation_range=5.,
                                       horizontal_flip = True)
    training_set = train_generator.flow_from_directory(train_dir,
                                                 target_size = (img_height,img_width),
                                                 class_mode = 'categorical',
                                                 batch_size = batchsize,
                                                 shuffle=False)
    return training_set
                                             

def test_generator(test_dir,batchsize,img_height,img_width):
    test_gen = ImageDataGenerator(rescale = 1./255)
    test_set = test_gen.flow_from_directory(test_dir,
                                            target_size = (img_height,img_width),
                                            class_mode = 'categorical',
                                            batch_size = batchsize,
                                            shuffle=False)
    return test_set


#fit model
input1=input_generator(train_dir_1,batchsize,img_height,img_width)
input2=input_generator(train_dir_2,batchsize,img_height,img_width)
test1=test_generator(test_dir_1,batchsize,img_height,img_width)
test2=test_generator(test_dir_2,batchsize,img_height,img_width)

model.fit_generator([input1,input2],
                        steps_per_epoch=trainsetsize/batchsize,
                        epochs = epochs,
                        validation_data = [test1,test2],
                        validation_steps = testsetsize/batchsize,
                        use_multiprocessing=True,
                        shuffle=False)

The error I get is the following:

TypeError: Error when checking model input: data should be a Numpy array, or list/dict of Numpy arrays. Found: <keras.preprocessing.image.DirectoryIterator object at 0x7f824c5080f0>...

30 Answers

✔️Accepted Answer

Thanks. Here is how I solved it following some ideas from issue 3386, maybe someone might find it useful

input_imgen = ImageDataGenerator(rescale = 1./255, 
                                   shear_range = 0.2, 
                                   zoom_range = 0.2,
                                   rotation_range=5.,
                                   horizontal_flip = True)

test_imgen = ImageDataGenerator(rescale = 1./255)



def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):
    genX1 = generator.flow_from_directory(dir1,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False, 
                                          seed=7)
    
    genX2 = generator.flow_from_directory(dir2,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False, 
                                          seed=7)
    while True:
            X1i = genX1.next()
            X2i = genX2.next()
            yield [X1i[0], X2i[0]], X2i[1]  #Yield both images and their mutual label
            
            
inputgenerator=generate_generator_multiple(generator=input_imgen,
                                           dir1=train_dir_1,
                                           dir2=train_dir_2,
                                           batch_size=batch_size,
                                           img_height=img_height,
                                           img_width=img_height)       
     
testgenerator=generate_generator_multiple(test_imgen,
                                          dir1=train_dir_1,
                                          dir2=train_dir_2,
                                          batch_size=batch_size,
                                          img_height=img_height,
                                          img_width=img_height)              
          
 history=model.fit_generator(inputgenerator,
                        steps_per_epoch=trainsetsize/batch_size,
                        epochs = epochs,
                        validation_data = testgenerator,
                        validation_steps = testsetsize/batch_size,
                        use_multiprocessing=True,
                        shuffle=False)

Other Answers:

@laukun I would suggest write your own data generator like in this example:

class DataGenerator(keras.utils.Sequence):
    """Generates data for Keras."""
    def __init__(self, img_files, clinical_info, labels, ave=None, std=None, batch_size=32, dim=(300, 300), n_channels=3,
                 n_classes=2, shuffle=True):
        """Initialization.
        
        Args:
            img_files: A list of path to image files.
            clinical_info: A dictionary of corresponding clinical variables.
            labels: A dictionary of corresponding labels.
        """
        self.img_files = img_files
        self.clinical_info = clinical_info
        self.labels = labels
        self.batch_size = batch_size
        self.dim = dim
        if ave is None:
            self.ave = np.zeros(n_channels)
        else:
            self.ave = ave
        if std is None:
            self.std = np.zeros(n_channels) + 1
        else:
            self.std = std
        
        self.n_channels = n_channels
        self.n_classes = n_classes
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        """Denotes the number of batches per epoch."""
        return int(np.floor(len(self.img_files) / self.batch_size))

    def __getitem__(self, index):
        """Generate one batch of data."""
        # Generate indexes of the batch
        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]

        # Find list of IDs
        img_files_temp = [self.img_files[k] for k in indexes]

        # Generate data
        X, y = self.__data_generation(img_files_temp)

        return X, y

    def on_epoch_end(self):
        """Updates indexes after each epoch."""
        self.indexes = np.arange(len(self.img_files))
        if self.shuffle == True:
            np.random.shuffle(self.indexes)

    def __data_generation(self, img_files_temp):
        """Generates data containing batch_size samples."""
        # X : (n_samples, *dim, n_channels)
        # X = [np.empty((self.batch_size, self.dim[0], self.dim[1], self.n_channels))]
        X_img = []
        X_clinical = []
        y = np.empty((self.batch_size), dtype=int)

        # Generate data
        for i, img_file in enumerate(img_files_temp):
            # Read image
            img = skimage.io.imread(img_file)
            
            # Resize
            img = skimage.transform.resize(img, output_shape=self.dim, mode='constant', preserve_range=True)
            
            # Normalization
            for ch in range(self.n_channels):
                img[:, :, ch] = (img[:, :, ch] - self.ave[ch])/self.std[ch]
            
            if self.shuffle:
                # Some image augmentation codes
                ###### You can put your preprocessing codes here. #####

            X_img.append(img)
            X_clinical.append(self.clinical_info[img_file])
            y[i] = self.labels[img_file]
        X = [np.array(X_img), np.array(X_clinical)]
        return X, keras.utils.to_categorical(y, num_classes=self.n_classes)

And to call the generator:

train_datagen = DataGenerator(img_files=img_files, clinical_info=clinical_info, labels=labels, ave=ave, std=std, batch_size=batch_size)
val_datagen = DataGenerator(img_files=img_files, clinical_info=clinical_info, labels=labels, ave=ave, std=std, batch_size=batch_size)
hist = model.fit_generator(train_datagen, 
                           steps_per_epoch=len(img_files) / batch_size, 
                           epochs=num_epochs, 
                           verbose=1, 
                           validation_data=val_datagen, 
                           validation_steps=1)

The different thing I'm doing here is using two different kinds of input, an image and a numpy array. I saved image paths as a list, and created two dictionaries whose key words are the image paths. When you want to do the preprocessing, you can easily apply them right after reading the image using the image paths.

But this way, the speed will be definitely slower than loading data directly from .npy

Related Issues:

926
keras Loading model with custom loss function: ValueError: 'Unknown loss function'
I solved this problem by adding 'custom_bojects' my loss function: I trained and saved a model that ...
633
keras Tensorflow backend - bug in model._make_predict_function(...)
I had this problem when doing inference in a different thread than where I loaded my model ...
516
keras Creating Model variable throws "AttributeError: 'Tensor' object has no attribute '_keras_history'"
@zaid478 oh I assumed you were doing it on coursera Anyway Hello all setup: Keras 2.0 Tensorflow 1.0...
395
keras "<tensor> is not an element of this graph." when loading model.
I had a problem similar to that of @piraka9011 which was solved by calling model._make_predict_funct...
204
keras Quick Question: can a model be fit for multiple times?
Yes successive calls to fit will incrementally train the model. I'm now having something around 5000...
183
keras Error when checking model target: expected activation_2 to have shape (None, 10) but got array with shape (3, 1)
Your model has an output of shape (10,) however your outputs have dimension (1,) You probably want t...
178
keras Is there a way in Keras to apply different weights to a cost function?
Ok so I had the time to quickly test it Hi there I am trying to implement a classification problem w...
172
keras Keras model cannot be loaded if it contains a Lambda layer calling tf.image.resize_images
Try using custom_objects argument in load_model (or model_from_json) function For your case load_mod...
164
keras how to use fit_generator with multiple image inputs
Thanks Here is how I solved it following some ideas from issue 3386 maybe someone might find it usef...
151
keras Expected 3 dimensions but got array with shape (11, 2)
The problem is that you start with a three dimensional layer but never reduce the dimensionality in ...
148
keras AttributeError: 'ProgbarLogger' object has no attribute 'log_values'
This happens if steps_per_epoch is 0 Make sure that your batch size is not greater than the dataset ...
147
keras For large datasets, which to use: fit or train_on_batch?
I think the best way to fit large data which can not fit into memory is writing a customize generato...
147
keras Split train data into training and validation when using ImageDataGenerator and model.fit_generator
I just found this in the documentation given here Its okay if I am keeping my training and validatio...
145
keras using pre trained VGG16 for another classification task
One way to do this is to not include the fully-connected layers at the top of the network Then add n...
123
keras "Merge" versus "merge", what is the difference?
Merge is a layer Merge takes layers as input Merge is usually used with Sequential models merge is a...
115
keras load_model() with custom layers, and custom layers in general
For the people landing here by a Google search the code we should use: as can be found on kapre's Gi...
108
keras Retrained MobileNet and failed to load it for prediction
Just to make @fchollet's comment explicit for fellow n00bs like me...this works: ...
107
keras How does Embedding Layer work?
GloVe and word2vec DO NOT convert a word to an integer I'm curious about how embedding layers work E...
105
keras Running tensorflow backend on single gpu but on multi gpu machine
This is default behavior for TensorFlow If you want TensorFlow to not allocate memory for all of the...
105
keras TypeError: softmax() got an unexpected keyword argument 'axis'
Hi I get the same problem but I got a better solution Just downgrade the tensorflow and keras My pre...
101
keras Passing additional arguments to objective function
@rpinsler actually doing a closure in the manner you suggested seems to work (thanks!) ...
96
keras Option for using dropout in the predict phase (as an approximation to Bayesian DL)
There is this feature in Keras: it's the training argument in the call of the Dropout layer ...
89
keras loss function for multi-label problem
categorical_crossentropy: 1-of-N (one-hot) binary_crossentropy: 1-or-more 0/1 labels ...
87
keras Convert Keras model to TensorFlow
When you are using the TensorFlow backend your Keras code is actually building a TF graph You can ju...
82
keras experimental_list_devices in tensorflow_backend.py (line 506)
I solve this by installing an older version which version of Keras is downgraded? and which version ...
82
keras Removing layers with layers.pop() doesn't work?
I actually ran into this recently and I believe that this is an issue with how Model was rewritten t...
78
keras A concrete example for using data generator for large datasets such as ImageNet
I have some follow up questions @wongjingping : Response to 3 I should have been clearer I am not co...
72
keras ImportError: No module named h5py
Have you tried directly installing h5py? http://docs.h5py.org/en/latest/build.html Try running pip i...
71
keras Variable-size image to convolutional layer
You should use: input_shape=(1 None None) None in a shape denotes a variable dimension Note that not...
71
keras val_acc KeyError using ModelCheckpoint.
Use val_loss in place of val_acc otherwise for accuracy Use val_accuracy in place of val_acc ...
69
keras Combining Pretrained model with new layers
Okay @Vijethbv first of all the error that is raised is due to a misspelling It's model.layers with ...
67
keras LSTM/RNN input and output
For a feed-forward network your input has the shape (number of samples number of features) With an L...
64
keras Attention Mechanism Implementation Issue
@zzjin13 Here you go. Hope it helps. The problem is that I have an output a from a LSTM layer with t...
64
keras Each time I run the Keras, I get different result.
by default Keras's model.compile() sets the shuffle argument as True You should the set numpy seed b...
60
keras Callback self.validation_data is None, when fit_generator is used
I am facing the same issue too But when I look into the source codes of Keras Related: #2702 Check t...
60
keras ImageGenerator for multiple inputs
Ok I made it work! For anybody asking himself the same question here is my example solution: ...
51
keras Does Keras support using multiple GPUs?
Yes can run Keras models on multiple GPUs This is only possible with the TensorFlow backend for the ...
50
keras How to implement my own loss function?
(Copying the answer which I posted on Stackoverflow) http://stackoverflow.com/questions/33859864/how...
49
keras 'thread._local' object has no attribute 'value' error
I'm using keras 2.3.1 tensorflow 2.0.0 python 3.6 linux Mint 19.2 Cinnamon My project is based on Wa...
48
keras how to load weights trained on multi gpu model to the single gpu model?
@smthomas-sci Custom model checkpoint callback works very well The problem is that if I only have a ...
47
keras AttributeError: module 'pydot' has no attribute 'find_graphviz'
pydot-ng will give you pydot that satisfies the keras package use pip install pydot-ng to install it...
47
keras Unable to Load Custom Objectives from an H5 Model File
@StuartFarmer did you find a solution to your problem? all I had to do was to use load_model like be...
46
keras How to use a tensorflow metric function in keras?
For now it's sufficient to add variables created for some tf metric to GraphKeys.GLOBAL_VARIABLES co...
45
keras Confusion regarding class_weight
I have this simple function for computing the weights for each class: What i do is pick the majority...
43
keras Can't load_model with error “Optimizer weight shape (256, 32) not compatible with provided weight shape (4,)”
I am using Keras 1.1.1 and am having the same problem Deleting the optimizer weights as a workaround...
41
keras memory leak when using tensorflow
You can now use K.clear_session() when using TensorFlow which will clean up everything This is recom...
41
keras Generalized dice loss for multi-class segmentation
Hey guys I found a way to implement multi-class dice loss I get satisfying segmentations now ...
39
keras How to get learning rate during training ?
Use callback to access model.optimizer Notice: It only works for SGD If you want to do the same thin...