Benchmarks
This is collection of projects and datasets for benchmarking. Some of them reproduce previously published as benchmark result for
comparison and the others compare accuracy and performance to MATLAB. Here we suggest multiple challenging tests that other
people can also use to test other forms of artificial intelligence or supporting it software.
The areas of random triangles
The features are random coordinates of vertices $(x_1, y_1), (x_2, y_2), (x_3, y_3)$, the target is area of triangle
$$ A = \frac{1}{2} \bigg|x_1(y_2-y_3) + x_2(y_3-y_1) + x_3(y_1-y_2)\bigg|. $$
The random coordinates have uniform distribution in any square field. The accuracy should not depend on the sizes of definition field.
The suggested training set is 10,000 records, validation set 2,000 records. The accuracy metric must be Pearson correlation
coefficient. The test must achieve accuracy of 99% on validation set and training time should not exceed few seconds.
Vector target
The targets are areas of 4 faces of random tetrahedron, the features are random coordinates of 4 vertices in a cube
$A(x_a, y_a, z_a), B(x_b, y_b, z_b), C(x_c, y_c, z_c), D(x_d, y_d, z_d)$.
Each area is computed by sampling 3 points, below is one for points $A,B,C$
$$ v_1 = (y_b - y_a)(z_c - z_a) - (z_b - z_a)(y_c - y_a) $$
$$ v_2 = (z_b - z_a)(x_c - x_a) - (x_b - x_a)(z_c - z_a) $$
$$ v_3 = (x_b - x_a)(y_c - y_a) - (y_b - y_a)(x_c - x_a) $$
$$ ABC = \frac{1}{2} \sqrt {v_1^2 + v_2^2 + v_3^2}. $$
Here we are dealing with correlated targets which share features and parts of internal structure. The size of training
set is 500,000 records, validation set 50,000. The training should not take longer than 5 minutes and Pearson correlation
coefficients for each individual component in vector target should be near 98.5%.
$5 \times 5$ determinants
The features are random matrices $5 \times 5$, the targets are their determinants. The training dataset is
10,000,000 records, validation set is 2,000,000. The training time should not exceed half an hour and
accuracy must be near 94% of Pearson correlation coefficient.
Remarks
These tests can be found under this menu block. All of them is much faster than MATLAB. The latter, for example, with
determinants is about 50 times faster, some others are 3 to 10 times faster. MATLAB is running very optimized
binaries in the background and uses all available resources of computer. It is easy to verify that by trying
any classic problem of linear algebra such as large matrix inversion or estimating eigenvalues. MATLAB is
always faster any library that some people tried to compare, so it can be used as all tests baseline.
|
|