Much faster atomic column integration implementation
In similar fashion to !47 (merged), I have sped up the atomic column integration implementation. Particularly so for the Voronoi cell integration method.
For a 512px by 512px, the old code took 124 seconds. The new version takes 3.6 seconds. The new version makes it reasonable to perform Voronoi integration on 1024px (45 sec) and 2048px images (half-an-hour ish) as well, where the old version took a LOT longer.
I have achieved the speed-boost partly by rewriting some code and moving things that did not need to be inside loops, outside loops. But the main speed-increase has come from implementing the numba jit (just-in-time) compilation decorator around the function that calculates the shortest distance between points for the point-record image.
In addition, I have added a function that removes any cells that are near the edges of the image/dataset. These are often annoying in that they get mess up the contrast of the displayed integrated_record image, and they are complicated to manually remove in order to calculate correct statistics (think, mean of the atomic column intensity).
Finally, I have updated a few things on watershed, including switching to the new location for watershed in the skimage toolkit (previous was being deprecated).
Attached is an example file and example result, showing the raw data, Voronoi integration and integration with removed border cells.
%matplotlib widget
import atomap.api as am
import hyperspy.api as hs
import matplotlib.pyplot as plt
s = hs.load("20181005_2006_STEM_HAADF_2.7_Mx.tif")
s2 = s.isig[:1024,:1024] # change to 256, 256 for speed
#nav = hs.signals.Signal2D([[1,2,3], [4,5,6]]).T
S = s2#nav*s2.T
points_x, points_y = am.get_atom_positions(s2).T
i, ir, pr = am.integrate(S, points_x, points_y, method='Voronoi', remove_edge_cells=True, edge_pixels=5)
# Or alternatively, with `remove_edge_cells=False` in previous line
# i, ir, pr = remove_integrated_edge_cells(i, ir, pr, pixels=5)
fig = plt.figure(dpi=200)
hs.plot.plot_images([s2, ir, pr], fig=fig, colorbar=False)
Merge request reports
Activity
1124 1187 return (integrated_intensity, s_intensity_record, point_record.T) 1125 1188 1126 1189 1190 def _border_elems(image, pixels=1): 1191 """ 1192 Return the values of the edges along the border of the image, with 1193 border width `pixels`. 1194 """ 1195 arr = np.ones_like(image, dtype=bool) 1196 arr[pixels:-1-(pixels-1), pixels:-1-(pixels-1)] = False 1197 return image[arr] 1198 1199 1200 def remove_integrated_edge_cells(i_points, i_record, p_record, An example of how to use this functions should be added to https://gitlab.com/atomap/atomap/blob/master/doc/absolute_integrator.rst
994 def find_smallest_distance(i, j, points): 995 ''' 996 Finds the smallest distance between coordinates (i, j) 997 and a list of coordinates. 998 999 Parameters 1000 ---------- 1001 i : Integer 1002 j : Integer 1003 points : array like of shape (2,N) 1004 1005 Returns 1006 ------- 1007 distMin : Minimum distance 1008 minIndex : Index of minimum distance in points 1009 ''' changed this line in version 8 of the diff
990 990 return self 991 991 992 992 993 @nb.jit(nopython=True) I'm not too familiar with
numba, but based on the Numba installation page (https://numba.pydata.org/numba-doc/latest/user/installing.html) this should work fine when installing both using Conda and pip, on Windows, MacOS and Linux?changed this line in version 8 of the diff
1217 The output of the atomap integrate function or method 1218 1219 Returns 1220 ------- 1221 i_points : NumPy array 1222 Modified list of integrated intensities with either np.nan or 0 1223 on the removed values, which preserves the atom index. 1224 i_record : HyperSpy signal 1225 Modified integrated intensity record, with either np.nan or 0 1226 on the removed values, which preserves the atom index 1227 p_record : NumPy array, same size as image 1228 Modified points record, where removed areas have value = -1. 1229 1230 Example 1231 ------- 1232 points_x, points_y = am.get_atom_positions(s).T changed this line in version 8 of the diff
1029 ------- 1030 point_record : Voronoi array where equal values belong to 1031 the same Voronoi cell 1032 ''' 1033 for i, j in tqdm(np.ndindex(point_record.shape), 1034 desc="Calculating Voronoi", 1035 total=np.prod(point_record.shape)): 1036 minIndex, distMin = find_smallest_distance(i, j, points) 1037 if distMin >= max_radius: 1038 point_record[i][j] = 0 1039 else: 1040 point_record[i][j] = minIndex + 1 1041 return point_record 1042 1043 1044 def get_integrated_intensity(point_record, image, point_index): changed this line in version 8 of the diff
added 1 commit
- 6ddb9ed6 - fixed for all dimensions - changed convention to take Signal2Ds only
added 37 commits
-
6ddb9ed6...518fa616 - 16 commits from branch
atomap:master - 7558decc - fix bug when input is 4D!
- ab48fc7c - implement faster voronoi
- 8a89d3e4 - removed unnecessary line
- bb56be22 - faster doing one argmin and indexing than argmin and min
- a02e805b - much much faster with numba jit
- c0c51d77 - jittable code for 2D arrays with two tests
- c270b93e - added docstrings and removed timings
- 06186897 - Added progress bar descriptions and made values that are masked away = np.nan
- 0841006f - cleaning up
- 7b102de3 - faster masking by getting rid of loop
- f3154eff - cast watershed points to int, but by rounding first
- dc2b8568 - Added convenience function for removing cells near the edge of the image
- 7ae5acb4 - clarified docstring
- 0f99840f - flake8
- 642790aa - Add test
- 574278e4 - removed commented-out jit
- 6aa62e93 - fix test
- f0fc7ed6 - fixed rotated int
- 2b6acbdb - fixed for all dimensions - changed convention to take Signal2Ds only
- c5a6a35b - Merge branch 'INT' of gitlab.com:thomasaarholt/atomap into INT
- a9f1d186 - fix multidimensional edge removal
Toggle commit list-
6ddb9ed6...518fa616 - 16 commits from branch
For generating test data I'd recommend the
MakeTestDataclass: https://gitlab.com/atomap/atomap/blob/master/atomap/testing_tools.py#L10added 160 commits
-
a2faa2c6...11d0be12 - 159 commits from branch
atomap:master - 3efe169e - Bring up to speed with 2020
-
a2faa2c6...11d0be12 - 159 commits from branch
Okay @magnunor! One year later, I've added your suggested changes, rebased the whole thing (force push), and added documentation and fixed tests. I added image generation for the docs in a python function in the same way you've done - I'm gonna assume that they run automatically if they're in the
doc/imagesfolder.All tests pass locally here.
