Skip to content
GitLab
Projects
Groups
Snippets
Help
Loading...
Help
What's new
9
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Switch to GitLab Next
Sign in / Register
Toggle navigation
Open sidebar
DW
Mastodon Search Bot
Commits
6df1779d
Commit
6df1779d
authored
Sep 07, 2019
by
DW
🥑
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
일단 끝
parent
4c4cd280
Changes
3
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
41 additions
and
17 deletions
+41
-17
README.md
README.md
+3
-5
index.js
index.js
+37
-12
package.json
package.json
+1
-0
No files found.
README.md
View file @
6df1779d
...
...
@@ -4,9 +4,7 @@
[
보러 가기 (Beta)
](
https://planet.moe/@search
)
## 지금 안되는것들
1.
`!!from: 검색`
왠진 몰라도 고장남
2.
`해시태그 검색`
추가해야함
3.
CI/CD 셋업 (귀찮아...)
1.
CI/CD 셋업 (귀찮아...)
## 개발 플랜
<details>
...
...
@@ -47,8 +45,8 @@
*
500자 넘어가면 답글 여러개 이어서 다는거 구현
`✅`
5.
명령어들
`✅`
*
봇이 보낸건 무시
`✅`
*
!!help 같은거
`
✔️
`
*
대충 필요한게 !!help, !!forgetme, !!from: 빼고 또있으려나??
`
✔️
`
*
!!help 같은거
`
✅
`
*
대충 필요한게 !!help, !!forgetme, !!from: 빼고 또있으려나??
`
✅
`
6.
맛톤에서 개인이 백업 json 받을수있는거 DB에 추가해주는 기능 구현
`❌`
*
근데 어떻게 봇한테 json을 보내지?? 맛톤에 파일첨부 되던가???
...
...
index.js
View file @
6df1779d
...
...
@@ -16,8 +16,8 @@ const main = async () =>
[
new
winston
.
transports
.
File
({
filename
:
'
data/logs/error.log
'
,
level
:
'
warn
'
}),
new
winston
.
transports
.
File
({
filename
:
'
data/logs/combined.log
'
,
level
:
'
info
'
}),
new
winston
.
transports
.
File
({
filename
:
'
data/logs/combined_debug.log
'
,
level
:
'
debug
'
}),
new
winston
.
transports
.
Console
({
format
:
winston
.
format
.
simple
(),
level
:
'
silly
'
})
//
new winston.transports.File({ filename: 'data/logs/combined_debug.log', level: 'debug'}),
new
winston
.
transports
.
Console
({
format
:
winston
.
format
.
simple
(),
level
:
'
debug
'
})
]
});
...
...
@@ -56,9 +56,15 @@ const main = async () =>
const
db_search
=
db
.
prepare
(
oneLineTrim
`
SELECT * FROM search_database
WHERE content LIKE ?`
);
const
db_search_
from
=
db
.
prepare
(
oneLineTrim
`
const
db_search_
user
=
db
.
prepare
(
oneLineTrim
`
SELECT * FROM search_database
WHERE content LIKE ? AND user = ?`
);
WHERE (content LIKE ? AND user = ?)`
);
const
db_search_hash
=
db
.
prepare
(
oneLineTrim
`
SELECT * FROM search_database
WHERE (content LIKE ? AND hashtags LIKE ?)`
);
const
db_search_user_hash
=
db
.
prepare
(
oneLineTrim
`
SELECT * FROM search_database
WHERE (content LIKE ? AND user = ? AND hashtags LIKE ?)`
);
const
db_delete_url
=
db
.
prepare
(
oneLineTrim
`
DELETE FROM search_database WHERE url = ?`
);
logger
.
info
(
'
DB preparation successful!
'
);
...
...
@@ -218,10 +224,10 @@ const main = async () =>
logger
.
debug
(
'
Crawling Start
'
);
const
hash_exists
=
msg
.
data
.
tags
.
length
!==
0
;
const
cw_exists
=
Boolean
(
msg
.
data
.
spoiler_text
);
const
user_fullhandle
=
msg
.
data
.
account
.
acct
.
includes
(
'
@
'
)
?
msg
.
data
.
account
.
acct
:
msg
.
data
.
account
.
acct
+
`@
${
config
.
instance_url
}
`
;
const
user_fullhandle
=
(
msg
.
data
.
account
.
acct
.
includes
(
'
@
'
)
?
msg
.
data
.
account
.
acct
:
msg
.
data
.
account
.
acct
+
`@
${
config
.
instance_url
}
`
).
toLowerCase
()
;
logger
.
debug
(
`Cleaning up toot content - Original is:
${
msg
.
data
.
content
}
`
);
const
stripped_content
=
msg
.
data
.
content
.
replace
(
/<
[^
>
]
*>
?
/gm
,
'
'
).
replace
(
/
\s\s
+/g
,
'
'
).
replace
(
/
(
@
[
A-Za-z0-9.
\-
_
]
+
)
/g
,
''
).
trim
();
const
stripped_content
=
msg
.
data
.
content
.
replace
(
/<
[^
>
]
*>
?
/gm
,
''
).
replace
(
/
(
@
[
A-Za-z0-9.
\-
_
]
+
)
/g
,
'
'
).
replace
(
/
\s\s
+/g
,
'
'
).
trim
();
logger
.
debug
(
`Cleaned up - content is now:
${
stripped_content
}
`
);
logger
.
debug
(
'
Inserting Crawled toot to DB
'
);
...
...
@@ -238,7 +244,7 @@ const main = async () =>
else
{
// NOTE: 이거 웬지 SQLite 들어가면 검색할때 느릴것같은데 더 나은 방법 없나??
let
tags_plaintextified
=
msg
.
data
.
tags
.
reduce
((
acc
,
cur
)
=>
acc
+
cur
+
'
'
,
''
).
trim
();
let
tags_plaintextified
=
msg
.
data
.
tags
.
sort
((
a
,
b
)
=>
(
a
.
name
>
b
.
name
)
?
1
:
-
1
).
reduce
((
acc
,
cur
)
=>
acc
+
cur
.
name
+
'
'
,
''
).
trim
()
.
toLowerCase
()
;
if
(
!
cw_exists
)
// 해시태그만 있을때
{
logger
.
debug
(
`Toot's Tag is:
${
tags_plaintextified
}
`
);
...
...
@@ -312,18 +318,19 @@ const main = async () =>
logger
.
debug
(
'
Mention does not contain special command
'
);
// HTML 태그 제거, 공백/탭/기타 whitespace 여러개있는경우 공백 하나로 줄여버리기, @멘션 제거
const
stripped_content
=
msg
.
data
.
status
.
content
.
replace
(
/<
[^
>
]
*>
?
/gm
,
''
).
replace
(
/
\s\s
+
/g
,
'
'
).
replace
(
/
(
@
[
A-Za-z0-9.
\-
_
]
+
)
/g
,
'
'
).
trim
();
// HTML 태그 제거, 공백/탭/기타 whitespace 여러개있는경우 공백 하나로 줄여버리기, @멘션 제거
, 해시태그 제거
const
stripped_content
=
msg
.
data
.
status
.
content
.
replace
(
/<
[^
>
]
*>
?
/gm
,
''
).
replace
(
/
(
@
[
A-Za-z0-9.
\-
_
]
+
)
/g
,
''
).
replace
(
/
((
@search@planet
\.
moe
)
|
(
@search
))
/g
,
''
).
replace
(
/
(
#
[^\u
2000-
\u
206F
\u
2E00-
\u
2E7F
\s\\
'!"#$%&()*+,
\-
.
\/
:;<=>?@
\[\]
^`{|}~
]
*
)
/g
,
''
).
replace
(
/
\s\s
+/g
,
'
'
).
trim
();
logger
.
debug
(
`Stripped toot content is
${
stripped_content
}
`
);
let
username
=
''
;
let
tags
=
''
;
const
arr_words
=
stripped_content
.
split
(
'
'
);
// from 있는 검색
if
(
stripped_content
.
startsWith
(
'
!!from:
'
))
{
logger
.
debug
(
'
Stripped content starts with !!from:
'
);
username
=
arr_words
[
0
].
split
(
'
:
'
)[
1
];
username
=
(
arr_words
[
0
].
split
(
'
:
'
)[
1
]
).
toLowerCase
()
;
if
(
!
username
)
// from: 다음에 공백일경우
{
logger
.
debug
(
'
Username is empty!
'
);
...
...
@@ -339,6 +346,14 @@ const main = async () =>
}
logger
.
debug
(
`Username is:
${
username
}
`
);
}
// 해시태그 있는 검색
if
(
msg
.
data
.
status
.
tags
.
length
!==
0
)
{
logger
.
debug
(
'
Tag exists!
'
);
tags
=
msg
.
data
.
status
.
tags
.
sort
((
a
,
b
)
=>
(
a
.
name
>
b
.
name
)
?
1
:
-
1
).
reduce
((
acc
,
cur
)
=>
acc
+
cur
.
name
+
'
%
'
,
'
%
'
).
trim
().
toLowerCase
();
logger
.
debug
(
`Tag searchstring is:
${
tags
}
`
);
}
// 블랙리스트 / 글자수 처리
logger
.
debug
(
'
Evaluating if query is valid
'
);
...
...
@@ -371,7 +386,17 @@ const main = async () =>
logger
.
debug
(
'
Querying DB
'
);
const
startTime
=
new
Date
();
// SQL 쿼리
const
results
=
username
?
db_search_from
.
all
(
searchstring
,
username
)
:
db_search
.
all
(
searchstring
);
let
results
=
[];
if
(
username
)
{
if
(
tags
)
results
=
db_search_user_hash
.
all
(
searchstring
,
username
,
tags
);
else
results
=
db_search_user
.
all
(
searchstring
,
username
);
}
else
{
if
(
tags
)
results
=
db_search_hash
.
all
(
searchstring
,
tags
);
else
results
=
db_search
.
all
(
searchstring
);
}
const
endTime
=
new
Date
();
logger
.
debug
(
`DB result is:
${
JSON
.
stringify
(
results
,
null
,
2
)}
`
);
if
(
results
.
length
===
0
)
// 결과 없을 경우
...
...
@@ -429,7 +454,7 @@ const main = async () =>
logger
.
debug
(
'
User is in the same instance
'
);
const
response
=
await
mstdn_send_dm
(
msg
.
data
.
account
.
acct
,
stripIndent
`
검색봇을 팔로우해주셔서 감사합니다!
검색봇을 팔로우해주셔서 감사합니다!
'!!help'로 명령어 도움말을 보실 수 있습니다!
이 봇을 팔로우함으로써 여러분의 공개 / 타임라인에 비표시 툿이 검색을 위해 봇의 서버에 인덱싱된다는 점을 기억해주세요!
위 사항에 동의하신다면 이 툿에 별을 찍어주세요! (동의하기 전까지 검색 기능을 사용하실 수 없습니다.)
동의를 철회하고 싶으시다면 봇에게 DM으로 !!forgetme 라고 보냄으로써 봇에 인덱싱된 유저분의 모든 툿을 삭제할 수 있습니다!
...
...
package.json
View file @
6df1779d
...
...
@@ -4,6 +4,7 @@
"description"
:
"Search bot for Mastodon Instances without ElasticSearch"
,
"main"
:
"index.js"
,
"scripts"
:
{
"start"
:
"node index.js"
,
"test"
:
"echo
\"
Error: no test specified
\"
&& exit 1"
},
"repository"
:
{
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment