@@ -24,7 +24,33 @@ Files may be distributed to begin with. Check the following documents and locati
...
@@ -24,7 +24,33 @@ Files may be distributed to begin with. Check the following documents and locati
*`gemini1-2:/mnt/gemini1-4/seales_uksr/`
*`gemini1-2:/mnt/gemini1-4/seales_uksr/`
*`lcc:/pscratch/seales_uksr/`
*`lcc:/pscratch/seales_uksr/`
# Generate/extract slices
# Determine crop
It is necessary to load the volume into a visualizer such as ImageJ to determine the crop bounding box. If the slices are already generated and on your local disk in one volume, which is likely if they came from a benchtop machine, those can be viewed directly. With the large .hdf files from Diamond, it is easiest to first extract some sample slices on LCC, transfer those to your machine and then view them there. Those can be extracted all at once from one or more .hdfs (multiple desirable if object scanned in slabs) like this:
The next step is the same for either the original full set of slices (if that is feasible and on your machine) or the subset sampled using something like the above.
Load the volume in Fiji/ImageJ, and select a rectangular bounding box. Scrub through the slices, adjusting the bounding box to make sure it is always outside the bounds of the object of interest. It is good to crop tightly to create smaller datasets, however it is important to be sure to contain the entire object. When in doubt, it's OK to make the bounding box a little bigger to be sure it includes the object.
Record the coordinates of the bounding box you have selected. You can either mouse over the edges and get close estimates from viewing the mouse coordinates in the ImageJ main window, or you can use `Analyze -> Tools -> ROI Manager`, add the current ROI, and use `More -> Specify...` withing the ROI manager to see the bounding box specs.
The goal of this step is to get a 16-bit .tif image stack. For benchtop sources, this is probably already done as part of the reconstruction process. For synchrotron scans, this step may be necessary. For example in the 2019 Diamond Light Source scans, the reconstruction output 32-bit float .hdf files from which .tif slices need to be extracted. Often, such as with the fragments scanned in that session, there is a separate .hdf for each "slab". Slices should be extracted from each, and then merged later.
The goal of this step is to get a 16-bit .tif image stack. For benchtop sources, this is probably already done as part of the reconstruction process. For synchrotron scans, this step may be necessary. For example in the 2019 Diamond Light Source scans, the reconstruction output 32-bit float .hdf files from which .tif slices need to be extracted. Often, such as with the fragments scanned in that session, there is a separate .hdf for each "slab". Slices should be extracted from each, and then merged later.
...
@@ -36,13 +62,13 @@ For particularly large datasets (such as those split into slabs) the entire data
...
@@ -36,13 +62,13 @@ For particularly large datasets (such as those split into slabs) the entire data
### One time setup:
### One time setup:
For convenience, a singularity container and skeleton slurm script have been placed in the DRI Datasets drive directory under Resources. These will allow for easy use of the relevant scripts from volume cartographer on the LCC servers, and should be copied to your scratch space before you begin:
For convenience, a singularity container and skeleton SLURM script have been placed in the DRI Datasets drive directory under Resources. These will allow for easy use of the relevant scripts from volume cartographer on the LCC servers, and should be copied to your scratch space before you begin:
The slurm script included should be lightly edited to be specific to the user. In particular the email field should be changed to the relevant address.
The SLURM script included should be lightly edited to be specific to the user. In particular the email field should be changed to the relevant address.
Now use sbatch and the previously copied slurm scripts to run extract/crop on the LCC system. The parameters passed to this script will be passed along to the hdf5_to_tif.py file included with volume cartographer, so should be treated in the same way:
Now use sbatch and the previously copied SLURM scripts to run extract/crop on the LCC system. The parameters passed to this script will be passed along to the hdf5_to_tif.py file included with volume cartographer, so should be treated in the same way:
@@ -76,10 +102,6 @@ This should run fairly quickly and multiple slabs can be processed at once. Now
...
@@ -76,10 +102,6 @@ This should run fairly quickly and multiple slabs can be processed at once. Now
Get the original volume or slices onto the machine you are using to process the dataset. This depends on context, but typically we use scp, rclone, etc.
Get the original volume or slices onto the machine you are using to process the dataset. This depends on context, but typically we use scp, rclone, etc.
## Extract slices (HDF5 files only)
\*\* Coming soon \*\*
## Crop slices (optional)
## Crop slices (optional)
Many of our datasets are too large to be processed efficiently in their native format. Cropping is the preferred method for reducing size as it maintains the spatial resolution of the scan. Scan through the slices to determine a good bounding box for the object in the scan. Test your crop using the `convert` utility provided by ImageMagick. The following command creates a 9060x1794 image starting at pixel (670,830) in the `full_slice_0000.tif` input image:
Many of our datasets are too large to be processed efficiently in their native format. Cropping is the preferred method for reducing size as it maintains the spatial resolution of the scan. Scan through the slices to determine a good bounding box for the object in the scan. Test your crop using the `convert` utility provided by ImageMagick. The following command creates a 9060x1794 image starting at pixel (670,830) in the `full_slice_0000.tif` input image:
* If you enabled dump-vis, a new directory called 'debugvis' will appear inside the working directory. Two directories inside, called 'mask' and 'skeleton', contain images that show what is being segmented. You can use these images as a reference to help you determine when to stop the segmentation.
* If you enabled dump-vis, a new directory called 'debugvis' will appear inside the working directory. Two directories inside, called 'mask' and 'skeleton', contain images that show what is being segmented. You can use these images as a reference to help you determine when to stop the segmentation.
To obtain a good-quality segmentation, the mask must cover the majority of the layer of interest, but it is fine if some small parts aren't covered or parts of neighboring pages get segmented too. This is an example of a good-quality segmentation: https://drive.google.com/file/d/1_qzL2L2gZpYHYUJznCZENbsW2ueUj8\_\\\_/view?usp=sharing
To obtain a good-quality segmentation, the mask must cover the majority of the layer of interest, but it is fine if some small parts aren't covered or parts of neighboring pages get segmented too. This is an example of a good-quality segmentation: https://drive.google.com/file/d/1_qzL2L2gZpYHYUJznCZENbsW2ueUj8\\\_\\\\\\\_/view?usp=sharing
Check the segmentation occasionally. If a lot of neighboring pages are getting segmented or if the segmentation loses the layer you are segmenting, use Ctrl-C to kill vc_segment **provided you ran vc_segment with `--save interval 1`**.
Check the segmentation occasionally. If a lot of neighboring pages are getting segmented or if the segmentation loses the layer you are segmenting, use Ctrl-C to kill vc_segment **provided you ran vc_segment with `--save interval 1`**.
...
@@ -235,7 +257,7 @@ The output of this process, `canny_raw.ply`, is a dense point set and requires f
...
@@ -235,7 +257,7 @@ The output of this process, `canny_raw.ply`, is a dense point set and requires f
1. Run `Filters/Point Set/Point Cloud Simplification` to reduce the point set to a reasonable size. If the surface is very smooth, use fewer points. Usually, within the order of 10k to 100k points typically retains enough detail while significantly speeding up later steps. Save this point set with the name: `01_simplified.ply`.
1. Run `Filters/Point Set/Point Cloud Simplification` to reduce the point set to a reasonable size. If the surface is very smooth, use fewer points. Usually, within the order of 10k to 100k points typically retains enough detail while significantly speeding up later steps. Save this point set with the name: `01_simplified.ply`.
2. Manually select and delete points that are not on the desired surface.
2. Manually select and delete points that are not on the desired surface.
3. Run `Filters/Selection/Select Outliers` and then delete the selected vertices. This cleans up groups of points that are not on the surface. It is recommended to enable the Preview option while tuning the selection options.
3. Run `Filters/Selection/Select Outliers` and then delete the selected vertices. This cleans up groups of points that are not on the surface. It is recommended to enable the Preview option while tuning the selection options.
4. Run `Filters/Point Set/Compute normals for point sets` to estimate surface normals for the point set. For dense, noisy point sets, adjust the `Neighbour num` value to a larger value, typically no more than 100. Save this point set to your working directory with the name: `canny_cleaned.ply`
4. Run `Filters/Point Set/Compute normals for point sets` to estimate surface normals for the point set. For dense, noisy point sets, adjust the `Neighbor num` value to a larger value, typically no more than 100. Save this point set to your working directory with the name: `canny_cleaned.ply`
5. Run `Filters/Remeshing, Simplification and Reconstruction/Surface Reconstruction: Screened Poisson` to triangulate the surface. This filter uses the surface normals generated in the previous step to fit a continuous surface to the point set. Increase the Reconstruction depth to make the surface fit more closely to the original point set at the expense of more faces and a rougher surface. Typically, use a reconstruction depth in the range of 8-10. Save this mesh to your working directory with the name: `canny_poisson.ply`
5. Run `Filters/Remeshing, Simplification and Reconstruction/Surface Reconstruction: Screened Poisson` to triangulate the surface. This filter uses the surface normals generated in the previous step to fit a continuous surface to the point set. Increase the Reconstruction depth to make the surface fit more closely to the original point set at the expense of more faces and a rougher surface. Typically, use a reconstruction depth in the range of 8-10. Save this mesh to your working directory with the name: `canny_poisson.ply`
6. Poisson will create faces which extend beyond the original point set. Run `Filters/Sampling/Hausdorff Distance` to add an attribute to each vertex of the new surface that is that vertex's distance to the nearest point in the original point set.
6. Poisson will create faces which extend beyond the original point set. Run `Filters/Sampling/Hausdorff Distance` to add an attribute to each vertex of the new surface that is that vertex's distance to the nearest point in the original point set.
7. Run `Filters/Selection/Select by Vertex Quality` to select those vertices in the Poisson surface which have large distances from the original point set. Use the Preview option to tune the selection. Delete the selected vertices and faces.
7. Run `Filters/Selection/Select by Vertex Quality` to select those vertices in the Poisson surface which have large distances from the original point set. Use the Preview option to tune the selection. Delete the selected vertices and faces.
