With function bellow using ImageDataGenerator and after that flow_from_directory. It’s gonna solve shuffle problem which exist when we predict with model using test_generator as data, giving us Y which is not matching to his pair X import math import numpy as np number_of_examples = len(test_generator.filenames) number_of_generator_calls = math.ceil(number_of_examples / (1.0 * batch_size)) x_final = np.empty((0,… Continue reading Image DataGenerator getting X and Y for each picture
Category: Large data
Loading large image data
In machine learning we can struggle with problems like not enough memory. There is a good solution, for that, and you can load the data without running out of RAM memory. The problem starts, when you load the data in one step and you holding it in a variable. That can full your memory easy.… Continue reading Loading large image data
What is Vaex?
Vaex is a python library for lazy Out-of-Core DataFrames (similar to Pandas), to visualize and explore big tabular datasets. It can calculate statistics such as mean, sum, count, standard deviation etc, on an N-dimensional grid up to a billion (109) objects/rows per second. Visualization is done using histograms, density plots and 3d volume rendering, allowing interactive exploration of big data. Vaex uses memory mapping, a zero memory copy… Continue reading What is Vaex?