Commit bff67ac1 authored by Khaytsus's avatar Khaytsus

Update for https and get more data

Switched to htttps, http was redirecting.  That caused failures in curl.

Primarily switched to getting 310 prefixes separately from all the others, as it's such a much bigger pool of data than the rest.  So now it gets all of the US, then a fixed number from 310, then it figures out how many contacts are left from worldwide and gets those.

I also added a variable called 'wwextra' which lets you download more data than the radio can hold but when de-deuplicated is smaller than the maximum.  In other words, the radio with OpenGD77 can hold 13796 records but let's retrieve like 14500.  When de-duplicated, it'll still be below 13796.  But still do all the sanity checks so we don't go over.
parent a4c5333f
......@@ -5,22 +5,26 @@
datapath="/home/wally/ham/Radios/GD77/SW/DMR"
# Array of id,name which the script iterates over to get data, update for your needs
# Here I am getting all US (averages to about a week worth of contacts when limited
# to 500) plus some targeted states I'm interested in which pulls in a lot more calls
# per those states
# Here I am getting all US plus some targeted states I'm interested in which pulls
# in a lot more calls per those states. I do not pull in 310 here; I do that
# separately as it's a much larger dataset.
ids=(310,us310 311,us311 312,us312 313,us313 314,us314 315,us315 316,us316 \
ids=(311,us311 312,us312 313,us313 314,us314 315,us315 316,us316 \
317,us317 318,us318 319,us319 320,us320 3121,ky 3117,il 3118,in 3139,oh 3147,tn )
# Radio holds 10920, but I left this at 10900. If this changes in the future, update
max=10900
# Radio holds 13796
max=13790
# Max per area
count=500
count=700
# Generic Last heard count (add additional World Wide last-heard users)
# The data above is about 4000 entries, so this fills it out to about 10k
wwcount=6000
# 310 area users.. 3000 is about 200 days or so
uscount=3000
# How many extra for WW should we pull, there are always duplicates which
# winds up with not having nearly as many contacts as we can have, so pad
# the number a bit to get more contacts in once we de-duplicate everything
wwextra=800
# Shouldn't need to touch below here
......@@ -29,27 +33,46 @@ output=${ts}.csv
cd ${datapath}
rm *-???.csv *-????.csv
# Download each csv file
for i in ${ids[@]}; do
id=`echo $i | cut -f 1 -d ','`
name=`echo $i | cut -f 2 -d ','`
echo "Getting ${name} (id $id}) (${count} max)"
curl -s -o ${name}-${id}.csv "http://ham-digital.org/user_by_lh.php?id=${id}&cnt=${count}"
curl -s -o ${name}-${id}.csv "https://ham-digital.org/user_by_lh.php?id=${id}&cnt=${count}"
done
# Pull last heard from all around the world, with the genericcount limiting how many
# 5000 is about 6 days worth give or take
echo "Getting worldwide Last Heard data (${wwcount} max)"
curl -s -o lh-0000.csv "http://ham-digital.org/user_by_lh.php?cnt=${wwcount}"
# Pull last heard from the 310 area, limited to uscount
echo "Getting 310 Last Heard data (${uscount} max)"
curl -s -o us310-310.csv "https://ham-digital.org/user_by_lh.php?id=310&cnt=${uscount}"
# Combine all of them into a single file formatted for import
# Combine all of them into a single file to count how many we currently have
awk 'BEGIN { FS=";"; OFS=","; } {print $3,$2 " " $4,$4,",,,,<br/>"}' *-???.csv *-????.csv | sort | uniq | grep -v 'dmrid,callsign' >${output}
# Check to see if we exceeded the maximum this radio allows
lines=`wc -l ${output} | cut -f 1 -d " "`
if [ ${lines} -gt ${max} ]; then
echo "Warning: CSV contains more than ${max} lines"
echo "Warning 1: CSV contains more than ${max} lines"
fi
# Whatever is left, pull it from world-wide
wwcount=$(($max-$lines+$wwextra))
echo ""
echo "Have ${lines} of data, getting ${wwcount} more from world-wide"
curl -s -o lh-0000.csv "https://ham-digital.org/user_by_lh.php?cnt=${wwcount}"
# Combine all of them into a final file formatted for import
awk 'BEGIN { FS=";"; OFS=","; } {print $3,$2 " " $4,$4,",,,,<br/>"}' *-???.csv *-????.csv | sort | uniq | grep -v 'dmrid,callsign' >${output}
# Check to see if we exceeded the maximum this radio allows
lines=`wc -l ${output} | cut -f 1 -d " "`
if [ ${lines} -gt ${max} ]; then
echo "Warning 2: CSV contains more than ${max} lines"
fi
missedcount=$(($max-$lines))
echo ""
echo "${output} should be ready to import using ActiveClient.exe"
\ No newline at end of file
echo "${output} with ${lines} (${missedcount} less than ${max}) contacts should be ready to import using ActiveClient.exe or OpenGD77 CPS"
\ No newline at end of file
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment