by Ryan Wicks | January 2023
Introduction
RTK GNSS receivers are one of many tools that are leveraged for conducting survey work, and with the addition of unoccupied aircraft systems (UASs) to supplement GNSS receivers, theodolites, and the like, new survey techniques are being used and further refined to collect data about the landscape and physical features of interest. A common workflow is to have several ground control point (GCP) markers distributed throughout an area of interest that is to be surveyed, to survey those few GCP locations with a RTK GNSS receiver or some other ground survey tool, and then fly a drone to collect aerial imagery of the entirety of the area of interest; so long as a significant portion of the images that the drone took have the GCP markers visible in the images, then the surveyed GCPs can be used to constrain a photogrammetry reconstruction from the drone images to have a similar level of accuracy throughout the survey area. I’ve touched on this technique in a few of my other articles. This can be an effective technique for attaining a high spatial accuracy over a large area at a resolution that wouldn’t be possible without instrumentation like LiDAR. While this workflow is sufficient in many cases, there is an additional role in this process that RTK GNSS receivers can play: assessing the accuracy of final photogrammetry products.
Orthomosaics, digital elevation models (DEMs), etc. have some level of error even when GCPs are used, and what’s more that error is not evenly distributed throughout the survey region. Understanding this error throughout the photogrammetry products can help a user have a clearer understanding of the accuracy of those products. How does one measure and quantify this error? A direct approach is to use test markers; if when a surveyor is placing GCP markers, if they also place a series of “test” markers and survey the locations of those markers, then the location of the test markers as observed by the ground survey can be compared to the location where the test markers appear in the photogrammetry products. The test marker locations as measured by the ground survey can be regarded as the “true” location of the markers – though in truth this too has error – and the marker location as observed by the orthomosaic product can be regarded as the observed location of the markers for purposes of assessing the accuracy of the photogrammetry products.
As one might expect, the error will not be the same for all markers, however the distribution of the error or disagreement in location between the ground-measured location and observed location in the photogrammetry products is indicative of the error throughout the photogrammetry products. The more test markers there are, the more one can get a sense for the distribution of the error, both across the entire survey region as well as errors in specific sections. In one of the UAS classes that we teach at UMass we have lab exercises where we try to quantify and qualify these hypothetical errors using these techniques.
There are 5 types of spatial accuracy or spatial error that I want to highlight:
- “X error” or “longitudinal error” or “easting error” – This is the difference between the Easting or Longitude measured for a given validation point, “i”, and the Easting or Longitude observed in a photogrammetry product of that validation point; i.e.:
- “Y error” or “latitudinal error” or “northing error” – This is the difference between the Northing or Latitude measured for a given validation point, “i”, and the Northing or Latitude observed in a photogrammetry product of that validation point.
- “XY error” or “lateral error” – This is the lateral Euclidean difference between the lateral position measured for a given validation point, “i”, and the lateral position observed in a photogrammetry product of that validation point.
- “Z error” or “vertical error” or “altitude error” – This is the difference between the altitude measured for a given validation point, “i”, and the altitude observed in a photogrammetry product of that validation point.
- “XYZ error” or “total error”- This is the Euclidean difference between the position measured in 3-dimensional space for a given validation point, “i”, and the position observed in 3-dimensional space in a photogrammetry product of that validation point.
You may note that equation (3) and equation (5) do not account for the curvature of the earth. This is generally okay so long as one is working on very small scales (i.e. a few hundred meters or smaller), and is working in a coordinate reference system with linear units such as meters or feet as is the case with UTM projections.
For each of these 5 ways to measure the error of a single validation point, we can calculate a bunch of metrics to get a sense of the error distribution of all of the validation points. Each validation point can be thought of as a discrete, independent sample of the error of photogrammetry data products such as orthomosaics or DEMs, and the more validation points that we have, the better an understanding we have of the error of any of these products. In the case of this lab we have 20 validation points, which should give us a fairly decent idea of the error distribution of the data products. In general it may be useful to produce histograms of the errors for each type of measurement of error in order to get a sense of the error distribution of that type of error, as well as calculate the root-mean-square error (RMSE) of each type of measurement of error, i.e. calculate the RMSE for longitudinal, latitudinal, vertical, lateral, and total error respectively. This provides a central estimate for errors in the orthmosaic product in question.
Note that this is the RMSE for a sample set, not a population set; the entire population is all of the pixels in a photogrammetry product, and we usually only have a dozen or so points to use for analysis as opposed to the tens or hundreds of millions of points that exist the photogrammetry product.
Class Exercise Data Collection
In one of our class exercises we place markers in a grid pattern as shown below. We surveyed these points within 3-4 cm. That is to say that the accuracy the measurements are such that they have a 1 standard deviation confidence interval that is no more than 3-4 cm (this is value varies somewhat from measurement to measurement). We used a Trimble R10 GNSS receiver with RTK corrections in this particular instance, though any particular device that has a similar accuracy or better could be used. Images were collected over the area around the region where the markers placed.
Image collection was done using a DJI Phantom 4 Pro on 24 Feb 2022 at around 3:15pm – 4:30pm on a calm day that was mostly overcast. Two sets of flights were done to collect two different image sets: one was done at roughly 157 feet (48 meters) above ground level, and the other was done at roughly 315 feet (96 meters) above ground level, and both were collected with 75% front overlap and side overlap between images.
30 markers were deployed before the flight to sever either as GCPs or test/validation markers. The positions of these markers were placed such that the images collected at 157 feet had minimum of 0.5 degrees of separation – i.e. the diagonal between GCPs was half that of the image footprint width. The images collected at 315 feet had a minimum of .25 degrees of separation – that is to say the diagonal between GCPs was a quarter that of the image footprint width.
Degrees of Separation and GCP Density
I think it useful to clarify what I mean by “degrees of separation”. As this exercise that we do in the class is intended to highlight, the density and the placement of GCPs in the survey area will generally impact the quality and accuracy of the final photogrammetry products. In general the accuracy of any one point in a photogrammetry product tends to decrease the further it is from a GCP; “degrees of separation” is a way to generalize the maximum distance that any point in the survey region can be from a GCP that is independent of any specific linear distance or scale.
I argue that in general it is more helpful to consider GCP spacing in terms of image frames, rather than any particular set linear distance because in the photogrammetry process error in the image alignment process can become compounded the more sets of tie points that there are between aligning any given image and a ground control point (GCP).
The graphic below may also help to show the geometric significance of the parameter “Degrees of Separation”: it is the lowest possible number of degrees of separation between any one image in the survey region and any other GCP in the survey region, where:
- an image that can see a GCP is said to have 1 degree of separation,
- …an image that can’t see a GCP but which has overlap with another image that can see a GCP is said to have 2 degrees of separation,
- … an image that can’t see a GCP but which has overlap with another image that has overlap with another image that can see a GCP is said to have 3 degrees of separation,
…
… etc.
Note, however, that if the GCPs in a survey region are spaced in a grid pattern, the orientation of the images taken in a nadir position (i.e. the camera is facing straight down, parallel to the pull of gravity) could be rotated in along the yaw axis to any arbitrary compass heading, which may or may not align with the axes of the grid pattern.
It is possible that the camera compass orientation places the width of the camera to be parallel to the a line between diagonal GCPs, and it is in this orientation where there is the lowest probability of any one given image directly seeing a GCP or the seeing the least number of GCPs if it can see 1 or more GCPs. Stated another way, it is the orientation where there will be the lowest percentage of all survey images that can see a GCP (provided even spacing of images over the survey region). Hence the degrees of separation are calculated assuming this geometry as a way of estimating a lower bound for the number of images that can see GCPs. Thus we could say that a GCP placement pattern has 2 degrees of separation if the distance between two GPCs along a diagonal in the grid is equal to 2 times the image footprint width of the imaging camera; 1 degree of separation if the distance of the diagonal is 1 image footprint width; 0.5 if it is half the image footprint width; etc. Note that the image footprint size will change depending on the altitude that the drone is flown at and the field of view of the camera that is used. Using standards for GCP placement that rely on linear units will only work for a given image footprint, but if we use “degrees of separation” we can generalize patterns of accuracy and error across different image footprint sizes. We can generalize the relationship between the size of an image footprint, the linear spacing between GCPs in a square grid, and the degrees of separation as:
The form of this equation is slightly different than the one shown in Figure 3, but they are algebraically identical.
Case Analyses
I want to demonstrate the calculations for some of these metrics in a few different cases to illustrate how both the degrees of separation between images and GCPs, as well as the distribution of GCPs throughout the survey region relate to the accuracy and precision of photogrammetry products in general. In all cases I want us to look at the accuracy of photogrammetry products that were derived from images that were captured during the flight that was done at 48 meters above ground level (AGL) for this analysis. At this altitude the width of the image footprint of images taken with the DJI FC6310 (the camera carried aboard the Phantom 4 Pro) is about 72 meters. While there were a few meters of deviation from a perfect rectangular grid patten for the markers, the diagonal between markers placed on the ground as shown in Figure 1 is approximately 36 meters, which corresponds to approximately half of the width of the image frame at 72 meters for that camera. Thus, if each marker is used as a GCP we would say that that GCP pattern has a maximum of 0.5 degrees of separation; if every other marker is used as a GCP we would say that that GCP pattern has a maximum of 1 degree of separation; if every 4th marker is used as a GCP we would say that the GCP pattern has a maximum of 2 degrees of separation; etc. In all cases we use Agisoft Metashape to produce the photogrammetry products from the images.
Let’s look at and compare four separate cases for how we could process the data from the images that were collected over the region that is shown in Figure 1:
Case 1: Use no GCPs and just use the position metadata from the cameras as measured by the aircraft GPS at the time of each image capture.
Case2: Use GCPs that are measured using an RTK GNSS receiver on the ground; use GCPs that are clustered tightly together in a square pattern in a corner, specifically using the following markers GCPs: 024, 025, 026, and 027. Maximum 0.5 Degrees of Separation.
Case 3: Use GCPs that are clustered less tightly together in square pattern in a corner, specifically using the following markers GCPs: 016, 018, 026, 028. Maximum 1 Degree of Separation.
Case 4: Use GCPs that are spaced in a square pattern that spreads the entire width of the survey region, specifically using the following markers as GCPs: 006, 010, 026, 030. Maximum 2 Degrees of Separation.
You may notice that the Degrees of Separation are not the only thing that changes between each of the cases; the distribution of the chosen GCPs changes too, specifically while the square pattern is similar between the pattern in all cases, cases 2, 3, and 4, have the linear distance between the GCPs and the relative spacing of the GCPs relative to the extent of the survey region change as well. We could have achieved varying Degrees of Separation between cases by flying and collecting image sets from different altitudes so that the image frame width would shrink compared to the spacing of the markers. Instead, we held altitude constant, and used the same image set in cases, and instead we have variability in the distribution of the GCPs that we are going to compare. While the Degrees of Separation (DoS) metric is useful, I want to introduce a couple of other metrics that we can use to assess the distribution of GCPs; while DoS is a good metric for the maximum distance between GCPs relative to the size of the image frame width and allows us to compare spacing of GCPs between surveys that may have different flight heights or image frame extents, it doesn’t provide any insight into how the GCPs are space throughout the survey region.
In a survey where we might be trying to achieve a high-accuracy product we would typically disperse the GCPs in a more uniform pattern across the entire survey region. (For this exercise, however, the non-uniformity and asymmetry is intentional to help illustrate a few key concepts.) Here the GCPs are clearly not uniformly dispersed, though, intuitively you may note that case 4 does a pretty good job, though I argue we could do an even more evenly distributed pattern with only 4 GCPs. Another way to put this would be do say that the GCP placement pattern in these cases have a spatial bias. While we can always look visually at the distribution of GCPs throughout a survey area and gain an intuitive sense if they are spaced evenly or not, I want to introduce some well-defined metrics that we can use to measure the distribution of GCPs in a more objective way.
There are a few ways to measure the spatial bias that exists in a GCP placement. For an arbitrary point, “P”, in a survey region we can measure the average distance from that point to n GCPs, D_P in the survey region:
Note that each point has a distance that can be calculated to each other GCP; Dp is the average of those distances for a given point, p, where the value of “p” is just and index for which point we are referencing. We could hypothetically calculate other statistics for the set of distances from a given point, p, to all of the GCPs. We could for instance calculate the distribution or standard deviation of the distance metric for any arbitrary point, p, but we would expect that if the GCPs were spaced out that then in general the standard deviation would be large; indeed if the standard deviation was small relative to the mean, that might suggest that GCPs were not spaced throughout the survey region, whereas a standard deviation on the same order of magnitude as the mean might suggest a better spacing, but such metrics aren’t quite conclusive because they depend not only on the positions of the GCPs, but also the position of the point “p”, so if we were to pick an arbitrary point, p, the standard deviation of distances to GCPs might look large or big not just because of the positioning of the GCPs, but also because of the position of point p relative to the GCP positions.
Alternatively, if we calculate the mean distance to GCPs for a sufficiently large number of points that are randomly or evenly distributed throughout the survey region, we can look at this distribution of means to get a more definitive sense of the distribution of GCPs. Once this has been done either for all possible points, p = {1, 2, 3,…,k} or at least a sufficient number of them, where k is the total number of points, we can look at the distribution of mean distances for all points that were calculated. If we say that Dp is the average of all Dp, that is to say the average of all the average distances for each point, p = {1,2,3,…,k}, then we can express the standards deviation of the mean distances to all GCPs for these points as:
The standard deviation of this distribution is a measurement of bias; the smaller the standard deviation, or the more tightly clustered the measurements of mean distance are, the less bias there is in the GCP distribution. This doesn’t necessarily tell us where or how the GCPs are clustered, but rather just a relative sense of how much bias there is in their placement.
Alternatively we could calculate the distance to the closest GCP for each point p = {1, 2, 3,…,k} , and then look at the distribution or standard deviation of those values, and that distribution would tell us similar information about the bias of GCP placement, which we will call SMin(D)P.
By looking at figures 4, 5, 6, and 7 we can see visually that lower values of SDpand SMin(D)P correspond to distributions of GCPs that are more evenly space throughout the survey area.
So how does GCP density – as measured by Degrees of Separation – and distribution – as measured by SDpand SMin(D)P – relate to photogrammetry accuracy? Let’s look at this case by case.
Case 1
As Figure 8 shows, the orthomosaic produced with no GCPs seems appropriate for the region – the road and land features from the Open Street Map base layer seem to align fairly well with the image of those features in the orthomosaic and there does not seem to be any obvious sign of distortion. Nonetheless, without GCPs the accuracy of the product is limited by the spatial accuracy of the GPS onboard the drone or whatever device is linking spatial data to the images. For the GPS onboard the Phantom 4 that was used we would expect a lateral accuracy of within a few meters of the true location. While the alignment process that Agisoft Metashape uses does do a good job of minimizing distortion in photogrammetry process by relying mostly on tie points and adjusting position, rotation, and lens parameters to minimize error in disagreement on position of those tie points, the spatial accuracy can regularly be shifted away from the true location without some kind of spatial input that has a high degree of accuracy such as surveyed GCPs.
Figure 9 shows a portion of the survey region at a smaller scale, and at this scale the accuracy of the orthomosaic can more readily be seen. The white markers, markers 017, 018, 023, and 024, are significantly displaced from where they were measured to be with the RTK GNSS receiver. You may note, however, that the displacement seems to be fairly consistent between those 4 GCPs, suggesting that while the precision of the photogrammetry product was preserved better than the accuracy; this is generally true for photogrammetry processes that don’t use GCPs, though GCPs also help improve the precision somewhat. The RMSE of the errors of each of the 30 markers was approximately 2.35 meters, and the range was 2.23 meters to 2.52 meters.
Case 2
In Case 2 when we introduce the use of GCPs with surveyed positions the orthomosaic looks very similar to the one in Case 1, however, if we look at the details in the region around markers 017, 018, 023, and 024 we see that the accuracy is much improved. Figure 10 shows the white markers so accurately aligned with he surveyed positions that they are occluded by the symbology at this scale.
If we look at an even smaller scale, as in Figure 11, we see that, at least here, the marker seems to align laterally to within a few centimeters. Clearly GCPs help dramatically improve the accuracy of photogrammetry products.
You may note that marker 024 was used as a GCP and was tagged with user input in the photogrammetry process, so if this did not align well it would suggest that there was some kind of technical or computing error in the photogrammetry process that we could address. So what does the accuracy of this orthomosaic and DEM (DEM is ought to be considered, too, to account for vertical accuracies) look like on a whole? We can calculate the metrics for error that we introduced in Equations 1-6 as shown in Figure 12:
| Image Frame Width (m) | Degrees of Separation | Standard Deviation (Sample) of Means of Distances to GCPs, S_(D ?_p ) | Standard Deviation (Sample) of Minimums of Distances to GCPs, S_(?Min(D)?_P ) | ||
| 72 | 0.5 | 34.442 | 30.614 | ||
| Northing Error (m) | Easting Error (m) | Vertical Error (m) | Lateral Error (m) | Total Error (m) | |
| Test Point RMSE | 0.057 | 0.018 | 1.488 | 0.060 | 1.490 |
| GCP RMSE | 0.001 | 0.002 | 0.000 | 0.002 | 0.002 |
By leveraging the GCPs in Case 2 we see by looking at the Test Point RMSE’s that the Northing and Easting errors are orders of magnitude smaller, however we see that the vertical errors, and thus consequently the total errors as well, are still much larger than we would want for an accurate survey. There are insights to be gained by looking not just at the global statistics for error, but also at how error is spatially distributed throughout the survey region. We can already see that the GCP errors themselves are quite small (though in truth this is an artifact of the reconstruction; they are not actually that accurate since there necessarily inherit the accuracy and error of the measurements of the R10 RTK GNSS receiver, but we will discuss that nuance later; for now let’s just at least assume that the GCPs mapped in the orthomosaic are the most likely to be the most accurate points in our survey region.), but we might get more insights if we look at the distribution of error for the other markers. We can investigate how error changes with respect to distance from a GCP.
Figure 13 shows the northing, easting, vertical, and total errors of the set of test point markers plotted against the distance to the nearest GCP. Analysis of comparison between the error of a test marker and its distance from the nearest GCP indicates that the error for any arbitrary point in the survey region increases the further it is from a GCP, or rather the accuracy is highest near a GCP and degrades with distance. This is most prominently seen in the vertical error in Figure 13, but this pattern also can be seen in Figure 14 where the northing error increasing with distance; this error can happen with either easting or northing errors. Furthermore, visual inspection would suggest that it seems that the error does not just grow linearly, but rather accelerates.
A solid explanation for these phenomena with the error can be expressed as such: the alignment of camera during the photogrammetry reconstruction uses tie points between images to align the images; naturally there is error in the position of the tie points in each of the image frames, as well as due to disagreements on position of those tie points between images, and that error propagates into the position, orientation, and lens transform parameters of the camera. Consequently this introduces error into the point cloud and the positions of discrete points such as the potions of the marker centers. There is error in between the alignment of each image frame, and so that error can compound as measurements of position in the point cloud are more and more image frames removed from a GCP; each image frame – each degree of separation – introduces new sources of error that can compound on top of each other. Hence understanding the GCP density and the distribution across the survey area – especially relative to the size of the image frame width – can help inform a sense of how error might vary over the survey region.
Another way to think intuitively about how GCP placement effects accuracy is to imagine a physical model such as where each of the image frames is a physical sheet of pliable metal or rubber, and the entire region is reconstructed by stapling these sheets together; GCPs can be thought of as pylons or supports that provide some rigidity to this somewhat pliable surface of these sheets. We can imagine that the further we move from one of these supports, we can imagine that the deflection and warping of the metal compounds more and more so that further points from these structural supports will have further deviations from the positions in space that they are intended to model.
While the easting error is somewhat reasonable, both the northing and vertical errors are generally larger than we could otherwise achieve if we had instead chosen a different, less biased, more evenly-distributed pattern for the GCPs. Potentially the easting error could have been high as well. The standard deviation of both the means of distances to GCPs for all test points and the standard deviation of the minimum of distances to GCPs or all test points are 34.4 meters and 30.6 meters respectively; both close the length of the diagonal between GCPs. The distribution and range of distances to the nearest GCP from each test point is also shown in Figure 15. These values – the standard deviations and range – are relatively large when compared to other cases, and suggest a relatively biased distribution of GCPs; thus the results of Case 2 match the expectation that significant spatial bias in the distribution of GCPs – i.e. GCPs that are not distributed evenly throughout the survey region – is related to larger error in the photogrammetry product when compared to other cases. What is more is that while the degree of separation for the GCP density, 0.5, is relatively quite low, as Martínez-Carricondo et al. show in their study (Patricio Martínez-Carricondo et al., “Assessment of UAV-photogrammetric mapping accuracy based on variation of ground control points”) it is possible to have a GCP density that is high enough (i.e. degree of separation that is low enough) that the accuracy is negatively impacted. It is possible that this happens because when the degree of separation is close to 0.5 or lower, many images in the image set can actually see 4 or more non-colinear GCPs; 3 non-colinear points define a plane, and so 3 non-colinear GCPs are sufficient to define the plane of any given image; any points beyond that might start to over-constrain the photogrammetry alignment and introduce more error.
In Case 2 we see that even points that are nearest to the GCPs have vertical errors around 0.2 meters, which is higher than we might otherwise be able to achieve, so something is negatively impacting the accuracy of this product; it could be because the GCP density being so high, it could be because the GCP spatial bias is so high, or it could be because of something else.
Case 3
| Image Frame Width (m) | Degrees of Separation | Standard Deviation (Sample) of Means of Distances to GCPs, S_(D ?_p ) | Standard Deviation (Sample) of Minimums of Distances to GCPs, S_(?Min(D)?_P ) | ||
| 72 | 1 | 26.950 | 22.829 | ||
| Northing Error (m) | Easting Error (m) | Vertical Error (m) | Lateral Error (m) | Total Error (m) | |
| Test Point RMSE | 0.012 | 0.019 | 0.035 | 0.022 | 0.041 |
| GCP RMSE | 0.001 | 0.001 | 0.000 | 0.002 | 0.002 |
Case 3 has more evenly distributed GCPs as can be seen in Figure 6, and is reflected in the standard deviation of distance metrics in Figure 16, and the distribution of minimum distances as displayed in Figure 19, however, it still is not as evenly distributed as it could be. Nonetheless we do see a significant increase in accuracy as shown in Figures 16, 17, and 18. The global RMSE for all errors is significantly lower than when compared to Case 2, and the values are almost as good as we could expect given the quality of the surveyed positions of the GCPs was to within 3-4 cm. You may notice that for northing, easting, vertical, and total error we see an increase in error the further we are from a GCP, though this error increase is not as steep as in Case 2. These are all patterns that we would expect for having more evenly spaced GCPs, and a GCP density that is about as low as it can get without introduce the potential for a at least partially over-constrained model in the alignment portion of the photogrammetry process.
The GCP and test marker arrangement of Case 3 does present a new dynamic in our potential analysis of the error and accuracy of the photogrammetry products; in Case 2 all of the test points that were measured were laterally exterior to the GCPs, that is to say the test points were laterally external to the GCPs. There are perhaps a few ways to mathematically explicitly define the laterally interior and laterally exterior region of the GCP placement, but intuitively we can understand what is considered laterally interior by looking at Figures 5, 6, and 7 for cases 2, 3, and 4 respectively. We can say that if we place a set GCPs laterally across a 2-Dimensional region, the subset of that region that is interior to those points is defined by the smallest region that can be defined by drawing straight lines connecting the GCPs which includes all of the GCPs either on that boundary or interior to/inside of that boundary. Figure 20 shows the region that is interior to the GCPs for Case 3.
In this case, will consider all of the following points to be interior to the GCPs: 016, 017, 018, 023, 024, 025, 026, 027, 028. While points 023 and 025 are technically on the exterior, had the markers been placed in a perfect grid pattern those points would have been along the boundary line and thus part of the interior. In any case, they are close enough to the interior that we can expect that their error will fit into the error distribution pattern of interior points.
How do the errors of interior regions and exterior regions compare? For any set of points for which the points in that set are a specific distance to the closest GCP we might expect that the points that fall within the interior region to have a higher accuracy and lower error. This is because interior error in general theoretically does not compound the same way exterior error does; whereas as the compounding error for interior points is bounded by the GCPs that surround them in more than one direction, exterior points have so such bound in at least a 1 pi radian arc. One can imagine the stappled sheets of metal or rubber again; for interior points in the model, those points are being suspended between at least 2 GCPs, whereas exterior points are like a cantilever that has no theoretical or upper bound the extent to which the error can compound as the distance from a GCP approaches infinity.
Let’s look at how the errors of the interior and exterior points in Case 3 compare. Figure 21 shows the total error of GCPs and test markers vs distance to the nearest GCP. (GCPs de facto have a minimum distance of 0.) Interior and Exterior points are plotted as separate sets.
What we would want to compare the nature of accuracy and error of interior vs exterior points would be measurements of error for each set (interior and exterior) that have roughly the same distribution of distances. As can be seen in Figure 21, the distribution of distances of the exterior points are different than the distribution of distances of interior points, so it is difficult to make a direct comparisons. While the interior points have a RMSE of 0.020 meters and the exterior points have a RMSE of 0.044 meters, we can’t directly compare those numbers to get a sense how the accuracy of interior points compare the accuracy of exterior points at similar distances in general. For those interior and exterior points that do have similar distances to the nearest GCP for each point respectively there does not seem to be a significant difference in error.
Case 4
| Image Frame Width (m) | Degrees of Separation | Standard Deviation (Sample) of Means of Distances to GCPs, S_(D ?_p ) | Standard Deviation (Sample) of Minimums of Distances to GCPs, S_(?Min(D)?_P ) | ||
| 72 | 2 | 9.730 | 14.986 | ||
| Northing Error (m) | Easting Error (m) | Vertical Error (m) | Lateral Error (m) | Total Error (m) | |
| Test Point RMSE | 0.008 | 0.010 | 0.055 | 0.012 | 0.056 |
| GCP RMSE | 0.001 | 0.001 | 0.000 | 0.001 | 0.001 |
In Case 4 the GCPs are spaced more evenly throughout the survey region than in Case 3, as can be seen by comparing the standard deviation of mean and minimum distances of test points to GCPs in each case, as well as by comparing the distribution of minimum distances in each case as shown in figures 25 and Figure 19. While the distribution of the GCPs through the survey region is more even, the GCP density drops from 1 Degree of Separation to 2 Degrees of Separation when comparing Case 3 to Case 4; while the more even distribution would generally lead to a more accurate orthomosaic, we would also expect to see a decrease in accuracy due the GCP density being lower in Case 4 compared to Case 3. In Case 4 the total accuracy of the test points is quite comparable to that of the total accuracy of Case 3, though the RMSE of total error in Case 4, 0.56 m, is actually greater than the RMSE of total error in Case 3, which is 0.41 m.
The distribution of interior versus exterior points is roughly reciprocal; whereas most of the test points in Case 3 were exterior, most of the test points in Case 4 are interior.
In Case 4 there is a noticeable difference the distribution of error when comparing interior points versus exterior points, though there are only 5 exterior points which are measured for comparison.
In both Case 3 and Case 4 a larger number of interior and exterior test points of at similar distances to the nearest GCP would likely reveal more evident patterns and differences between the effect of distance to the closest GCP on photogrammetry product accuracy when comparing regions interior to GCPs vs. regions exterior to GCPs. Additionally, if both Case 3 and Case 4 had more interior and exterior points at similar distance across cases, and if the distribution of GCPs was similarly even across cases then the effects of Degrees of Separation might be more strongly evident. When comparing Case 3 and Case 4, both the GCP density relative to the image frame and the GCP spacing changed, making it a little more difficult to attribute which influenced the differences in accuracy between Case 3 and Case 4. Nonetheless, for both interior and exterior points, the decrease in GCP density can be seen to have a negative effect on photogrammetry accuracy when the GCP density starts to decrease with Degrees of Separation higher than 1.
Comparing Interior and Exterior Errors vs. Degrees of Separation
The total error for regions interior to the GCPs was roughly the same between Case 3 with 1 Degree of Separation and Case 4 with 2 Degrees of Separation, as is shown in Figure 27.
As Figure 28 shows Case 2 with 0.5 Degrees of Separation had the greatest error for regions external to the GCPs by several orders of magnitude. While having GCP densities so high so that there is less than 1 Degree of Separation starts to introduce errors because of an over-constrained alignment in the photogrammetry process, I suspect that this does not account for all of the total error that we see in Case 2. The relatively extreme spatial bias of the GCP placement in the southeast corner of the survey area may have been the largest contributing factor to the error in the photogrammetry products. The GCPs were actually only imaged in a minority subset of the images and only extended across half of an image frame/footprint width and so the spatial information didn’t help inform the alignment process as much as it could have; even at that GCP density, if there had been more GCPs spread out, it seems probable that the external error would not have been as high at distances of 2-3 image frame/footprint widths away.
As we might expect from theory, Figure 29 shows the error in the region exterior to the GCPs seems to be greater for Case 4 with 2 degrees of separation than in Case 3 with 1 degree of separation.
Discussing other Sources of Error
There are a few other sources of error that we have not discussed. As is the general case for the standard deviation of a normal distribution that is the sum of 2 or more random variables that are normally distributed, the standard deviation of position errors in the photogrammetry product is the square root of the sum of the standard deviation of each of the sources of error:
One source of error is the error that comes from the process of tagging the center of a GCP to inform Agisoft Metashape or some other photogrammetry software package of the position of GCPs within the images. Whether this process is handled by a human or a computer image recognition algorithm, there is an expectation of non-zero error. In the cases that we discussed in this article the tagging of GCPs was done by hand on a sub-pixel scale, and I estimate S_tagging to be relatively small; on the order of 0.01 cm. In the cases we discussed in this article the GCPs had a S_measurement of about 0.02-0.03 meters. As such we could not expect the S_position values to be an smaller than roughly 0.035 or 0.03 meters. The standard deviations in errors that we see in excess of this are due to the errors in the alignment, which itself includes camera potions and orientation and lens parameter errors. Case 3 had the lowest errors, with a RMSE of 0.041 meters, and as such we can infer that S_alignment was a relatively small component of this error; contributing less than 0.02 meters. Theoretically 1 degree of separation in GCP placement is either the most optimal or very close to the most optimal in most cases for the highest level of accuracy. Given the accuracy of the GCP measurements in the field and the accuracy of the tagging process the analysis outlined in this article seems to suggest such, though more tests could offer more conclusive evidence of this.
As a final note, the RMSE’s in this analysis were mistakenly calculated using “n” in the denominator rather than “n-1” as shown in equation (6); “n” in the denominator would be an appropriate calculation for the RMSE of a population set, but the error measurements of the photogrammetry product are samples of the accuracy, and thus the RMSE calculation for a sample set that uses “n-1” in the denominator is more appropriately.. Consequently while the RMSE calculations listed in this article are not exactly correct, they are, I would argue, close enough to the more appropriate values to indicate similar patterns. In any case, I have not put in the time to recalculate the values and make all of the corresponding edits in this article.
Sources
All of the original images, marker locations, photogrammetry products, etc. that were used or referenced in this article can be found here:
https://drive.google.com/drive/folders/19JgP2sSGTg9zP4f2xOSUdnxjaKkh6b7_?usp=sharing
If you would like to provide feedback or have questions, you can contact the author, Ryan Wicks, via email: rwicks@umass.edu
References
Patricio Martínez-Carricondo, Francisco Agüera-Vega, Fernando Carvajal-Ramírez, Francisco-Javier Mesas-Carrascosa, Alfonso García-Ferrer, Fernando-Juan Pérez-Porras. Assessment of UAV-photogrammetric mapping accuracy based on variation of ground control points. International Journal of Applied Earth Observation and Geoinformation. Volume 72. 2018. Pages 1-10. ISSN 1569-8432. https://doi.org/10.1016/j.jag.2018.05.015. (https://www.sciencedirect.com/science/article/pii/S0303243418301545)