Generating unit-test data with Julia
TDD the Easy Way!
Most of the development I do uses TDD and at present I’m writing some new Python classes to do basic 3D Maths which include Mat3, Mat4 and Vec3 and Vec4 classes.
These classes are very similar to the ones in my c++ NGL library and I started to test them using the same Unit test data.
For example the ngl::Mat4 inverse() method has the following test.
TEST(Mat4,inverse)
{
// test verified from wolfram alpha
// 1,0,0,0,0,0.4, -0.4, 0 ,0 , 0.1 , 0.4 0 ,0,0,0,1
ngl::Mat4 test(1,0,0,0,0,2,2,0,0,-0.5,2,0,0,0,0,1);
test=test.inverse();
ngl::Mat4 result(1,0,0,0,0,0.4f,-0.4f,0,0,0.1f,0.4f,0,0,0,0,1);
EXPECT_TRUE(test == result);
}
As you can see from the comments I used wolfram alpha to generate a simple test result then verify by running my test and comparing.
Whilst this works ok, it is only 1 simple case and generating more is time consuming using the web interface, and doesn’t scale well to multiple tests with different data.
I decided I needed another tool to help auto generate the tests for me.
What to use?
I’m lucky that I have access to quite a bit of software (being an academic), however I wanted to use something FOSS if possible. So I discounted MATLAB and Mathematica immediately.
I considered Octave which I have used before for projects as well as just using Numpy or GLM but in the end I decided I would try Julia as I quite fancied learning something new.
Julia
Julia is a dynamically typed programming language with a REPL. It’s syntax is similar to C like languages but has the feel of a scripting language. It’s is simple to install and works across Mac, Linux and Windows which are my target operating systems.
At it’s simplest level we can just run the REPL and do basic maths.
julia> 2*3*5
30
julia>
However for me the most useful thing is the built in Matrix types.
julia> zeros((4,4))
4×4 Matrix{Float64}:
0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0
julia>
We can also build a matrix by passing in values using newline to start a new row
julia> a=[1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16]
4×4 Matrix{Int64}:
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
julia> transpose(a)
4×4 transpose(::Matrix{Int64}) with eltype Int64:
1 5 9 13
2 6 10 14
3 7 11 15
4 8 12 16
This works well but what I really need is a more complex matrix of floating point values to do maths with. I investigated the random number generation routines here and found it was easy to generate a random 4x4 matrix using the following code
julia> rand(Float64,(4,4))
4×4 Matrix{Float64}:
0.604374 0.0912643 0.372395 0.203154
0.0834194 0.851173 0.754508 0.835495
0.336754 0.436165 0.255739 0.433446
0.333849 0.0981714 0.439347 0.331527
So using this basic process I can create some random 4x4 matrices and do some maths.
julia> a=rand(Float64,(4,4))
4×4 Matrix{Float64}:
0.155732 0.206663 0.0922702 0.827322
0.338653 0.399044 0.188655 0.168776
0.0293418 0.444229 0.816516 0.0736119
0.22251 0.255415 0.0859794 0.63177
julia> b=rand(Float64,(4,4))
4×4 Matrix{Float64}:
0.96249 0.462971 0.376175 0.908213
0.0793394 0.989267 0.154777 0.873103
0.931258 0.281352 0.293946 0.187418
0.518119 0.144784 0.585945 0.860775
julia> c=a*b
4×4 Matrix{Float64}:
0.680866 0.422288 0.602457 1.05131
0.620743 0.629062 0.343504 0.836611
0.862013 0.693432 0.362938 0.6309
0.64183 0.471351 0.518691 0.985017
Packages
One of the first issues I noticed was the random matrix generation is within the range 0-1, whilst this is fine for my tests, I felt it would be better to have negative values as well (as these are very common in 3D). This is something that is not supported by the default random generator and needs to use the Distributions package.
Julia has a
built in package manager which can be accessed via the ] key.
(@v1.7) pkg> add Distributions
This will install the package and dependencies We can then exit the pkg took by pressing backspace. We can then use it as follows
julia> using Distributions
julia> dist=Uniform(-100,100)
Uniform{Float64}(a=-100.0, b=100.0)
julia> rand(dist,(4,4))
4×4 Matrix{Float64}:
-48.6763 -63.7468 46.7309 45.7332
-64.3383 53.3985 27.7707 -77.1511
11.4589 -2.73745 -59.6267 -51.2212
10.7504 -16.7751 -94.7999 77.6484
Test Generator
Now I can do the basics I need to design my program. I decided that I needed to generate the following tests using 3x3 and 4x4 matrix values
a*ba+ba-binv(a)the inversedet(a)the determinant
I want to write this data out to a python file as a series of lists of float values that I can then use to construct my tests, something like this
a=[ [.......],[......]]
b=[ [.......],[......]]
a_times_b[ [.......],[......]]
The basic structure is a function called generateTests, this is passed in a few parameters which are generated from the command line of the script. size is the matrix size (3x3 or 4x4 for now), file is the name of the file to write and loopSize is the number of test to generate (the size of the list).
The overall structure of this function is documented below but a few things to mention :-
- Scope is usually delimited with
keywordthenend - Julia indexes from 1 not 0 (gets me a lot!)
!function names that end with an exclamation mark modify one or more of their arguments by convention
#=
size = Matrix size will be square usually 4 or 3
file = name of file to output to (.py)
loopSize = the number of list elements to generate
=#
function generateTests(size,file,loopSize)
# generate some empty lists for our data
a=Matrix{Float64}[]
b=Matrix{Float64}[]
a_times_b=Matrix{Float64}[]
a_plus_b=Matrix{Float64}[]
a_minus_b=Matrix{Float64}[]
a_inv=Matrix{Float64}[]
a_det=[]
# uniform distribution for rng
dist=Uniform(-100,100)
# generate our data (don't forget we index from 1!)
for i = 1:loopSize
# create our base matrix values a and b
push!(a,rand(dist, size, size))
push!(b,rand(dist, size, size))
# do our calculations and store
push!(a_times_b,a[i]*b[i])
push!(a_plus_b,a[i]+b[i])
push!(a_minus_b,a[i]-b[i])
push!(a_inv,inv(a[i]))
push!(a_det,det(a[i]))
end
# open our file for io
open(file,"w") do io
# println writes a line and \n
println(io,"# file generated by gen_mat4_tests.jl")
# dump our results to the file
writePythonVar(io,"a",a)
writePythonVar(io,"b",b)
writePythonVar(io,"a_times_b",a_times_b)
writePythonVar(io,"a_plus_b",a_plus_b)
writePythonVar(io,"a_minus_b",a_minus_b)
writePythonVar(io,"a_inv",a_inv)
writePythonVar(io,"a_det",a_det)
end # end io
end # end function
The main work is done by the function writePythonVar this will write the data to python lists.
#=
io = file pointer for the python file to write to
array = the array to write to file
=#
function printMatrix(io,array)
print(io,"[")
for i = 1:length(array)
print(io,array[i],",")
end
print(io,"],\n")
end
#=
io = file pointer for the python file to write to
var = the name of the variable to write to the file
data = the array of data to write
=#
function writePythonVar(io,var,data)
println(io,var,"=[")
for i = 1:length(data)
printMatrix(io,data[i])
end
println(io,"]")
end
I have also added command line parsing using the ArgParse module the full code can be seen here with the final data generated in the file mat4Data.py
Using the files
The files are placed in the same folder as my unit tests and imported as a python module.
import tests.mat4Data as mat4Data # this is generated from the julia file gen_mat4_tests.jl
A sample test will take the data values and perform a calculation on them and then compare to the result Julia created. The following test if for the mat4*mat4 (note the use of the new python
__matmult__ operator)
def compare_matrix(self, a, b, places=6):
for r, v in zip(a, b):
self.assertAlmostEqual(r, v, places=places)
def test_mat4_times_mat4(self):
for a, b, result in zip(mat4Data.a, mat4Data.b, mat4Data.a_times_b):
m1 = Mat4.from_list(a)
m2 = Mat4.from_list(b)
value = m1 @ m2
self.compare_matrix(value.get_matrix(), result)
Conclusions
I’m happy with the results of this, it was about a days coding and learning and it’s highlighted a few issues I had with my implementation for the Mat classes. I’m going to add a few more generated values (for example Mat * Vec) and I can now increase and decrease the test amounts when I need to try stress testing etc.
I have considered generating the test data fresh each time by using the unitest fixture and running the julia script each time. This could make the tests more interesting as they could be different each time you run the tests, but could also be problematic when trying to identify issues.
At present I hard code the seed by using Random.seed!(12345) as this helps to get predictable test values, but again this can be removed easily.
Now I have this working for python I may integrate the same data / setups into my NGL library for the C++ and PyNGL tests as well.